Automated Pegboard System: Reliability and Validity of a New Tool

Everett B. Lohman, DPTSc, PT, OCS*

Eric G. Johnson, DPTSc, PT*

Aaron M. Miguel, DPT*

Tim K. Cordett, DPT†

Danny Kang, DPT‡

 

*Department of Physical Therapy, School of Allied Health Professions, Loma Linda University, Loma Linda, California

†England Physical Therapy, Garden Grove, California

‡Ventura Orthopaedics, Ventura, California

 

KEY WORDS: Automated pegboard, motor control, motor learning

 

Abstract

Background and Purpose: Neurologic injury often affects a person’s motor function, and depending on the type of injury, one or both limbs can be affected. Objective measures after neurological injury are critical for rehabilitative treatment planning. The Automated Pegboard System (APB2000) is a new tool that objectively quantifies the temporal components involved with motor function. In this study we assessed the reliability and validity of the APB2000. The APB2000 and the Wolf Motor Function Test (WMFT) data were compared to assess criterion validity.

Subjects and Methods: Two groups of matched subjects participated in this study. Group 1 consisted of seven healthy adults (mean age of 67.40 years), and group 2 consisted of seven medically stable stroke survivors (mean age 69.17 years). The APB2000 test retest interval was 2-weeks.

Results: The single measure intraclass correlation coefficient (ICC) was 0.78 and 0.90 for the second and third trials, respectively. The average measure ICC was 0.93. The validity for the APB2000 was not supported (r = 0.101).

Discussion and Conclusions: The reliability of the APB2000 exceeded the 0.90 value to establish reliability when using the average measure ICC. Validity for the APB2000 was not supported. However, these results may be misleading, because of several outlying values that swayed the results considerably. The APB2000 appears to be a highly reliable tool to provide clinical measurements for baseline and progression testing in both the pathologic and nonpathologic population. Further validity testing with a larger sample size may result in establishing criterion validity.

Introduction

Neurologic injury often affects a person’s upper extremity motor function, and depending on the nature of the injury, one or both limbs can be affected. Upper extremity impairments after neurologic injury typically include weakness, tone abnormalities, sensory and perceptual deficits, incoordination, and general motor dyscontrol.1 Rehabilitation efforts aimed at achieving maximal functional restoration rely on theoretical concepts of motor control.

A popular theory of motor control or motor learning in neurological rehabilitation is Bernstein’s “systems theory”.2 The systems theory suggests that movement is the result of a complex interaction between the individual, task, and environment.2–4 The individual is required to first perceive and understand a particular task- or goal-directed movement. After this is accomplished, the individual develops a motor plan, given the constraints of the immediate environment. Finally, the individual generates a motor response to complete the task, which is carried out by the neuromusculoskeletal system.

Quantifiable upper extremity clinical measurements after neurologic injury are critical for rehabilitative treatment planning. The Automated Pegboard System (APB2000) is a new rehabilitation tool that proposes to objectively quantify the temporal component involved with motor function. The purpose of this study was to assess the reliability and validity of the APB2000. Data from the APB2000 and the Wolf Motor Function Test (WMFT) were compared.5 The WMFT was chosen as the criterion test because it has established reliability and validity and is specific to upper extremity movement.6

Methods

Subjects

Two groups of subjects aged 53 to 87 years of age participated in this study. Group 1 consisted of seven healthy adults (mean age, 67.40; SD, 11.76; age range, 56–84). Group 2 consisted of seven medically stable adults who had suffered a brain injury (cerebral vascular accident [CVA] or transient ischemic attack [TIA]) with a mean age of 69.17 years (SD, 12.92; age range, 53–87). Groups were matched for age, hand dominance, and gender (Table 1). A matched design was used in this study to enhance the degree of control over extraneous variables.7

Subjects were screened to rule out visual, cognitive, and upper extremity mobility impairments that would prevent them from performing the study tasks. The Loma Linda University Institutional Review Board-Human Review Committee approved all protocols. All procedures were explained to subjects before they signed a statement of informed consent. Subjects were excluded from the study if they did not have a minimal level of cognition, adequate vision, ability to read and follow simple commands, and adequate mobility and strength to perform simple study related tasks.

Strength Testing

Kendall’s method of manual muscle testing8 was used to determine the strength of the anterior deltoid, biceps brachii, triceps brachii, and pectoralis major muscles. Bilateral upper extremities were tested in both groups. Each muscle was assigned a grade from 0 (no palpable activity) to 5 (normal). A minimum grade of 4 of 5 (good grade) and 3 of 5 (fair grade) within the available range was required for groups one and two, respectively. Grip strength was assessed using a JAMAR grip dynamometer (grip setting 2). Subjects sat comfortably with their shoulder adducted and neutrally rotated, elbow flexed at 90˚, forearm and wrist in neutral. Subjects repeated each test three times, and we recorded the mean score. Subjects were required to demonstrate a minimum of 5 lb of force bilaterally during the JAMAR test to perform study-related tasks. Intrarater and interrater reliability was not determined for strength testing.

Range of Motion

Range of motion (ROM) for the shoulder joint was measured on bilateral upper extremities using standard goniometry.9 All subjects were required to have the following minimal upper extremity active ROM measurements: 1) shoulder flexion of 45˚, 2) shoulder abduction of 45˚, 3) wrist extension of 10˚ beyond the resting flexed position of the wrist, and 4) metacarpophalangeal (MCP) extension of 10˚ beyond the resting flexed position of the hand and fingers. Intra-rater and inter-rater reliability was not determined for ROM testing.

Vision Testing

Because one of the study tasks involved the subject reading a short command from a 2-inch high X 7 inch wide monitor screen, two scans for visual impairments were performed. First, each subject was asked to read a short sentence (“The red fox jumped over the box.”) that was the exact font size of the wording (1/2 inch high) that each subject was required to read from the monitor during the Automated Pegboard 2000 (APB2000) test at a distance of 44 inches and a height of 43 inches. Then, a Titmus Vision TesterTM, Test 11, was used to detect visual impairments. Test 11 contains 7 lines (line one being the easiest to read and line 7 the most difficult). To be included in the study, all subjects had to be able to read and correctly articulate the short sentence and all three of the alphabet letters on line two while the subject used the typical eyewear prescription as applicable.

Reading Testing

Each subject was timed as they read the short sentence described previously. To be included in the study, each subject was required to read the sentence correctly within a maximum of 8 seconds.

Instrumentation

Reflex Development & Testing’s (RD&T) Automatic Peg Board System APB2000 (Figs. 1 and 2) consists of a computer-driven panel containing 21 specially colored and shaped pegs in precisely located sockets. The subject was prompted by the computer to move specific pegs to specific new locations. The total time required for each stage of the process, including comprehension, execution, and completion of the instructions (perception, cognition, and action), are objectively measured, tracked, and reported analytically and graphically. The system measures mental comprehension as well as hand, arm, and upper body dexterity. RD&T has engineered a third generation model of its APB2000 system, which is ready for third-party verification.

The “gold standard” Wolf Motor Function Test (WMFT), which has demonstrated reliability and validity, served as our comparative clinical measure to establish criterion-related validity for subjects with and without brain injury.6 The WMFT manual served as our operational guideline.10 Because each subject was tested in a random order (block assignment) on the WMFT and the APB2000 during the same testing session, the validity testing for the APB2000 is considered a “concurrent criterion-related validity study.”11 The WMFT is an objective 17-task test (appendix) that quantifies joint movement and functional ability of the upper extremity.10 The WMFT is a timed, function-based test that uses a 16 X 43 inch laminated template secured to a 30 inch wide by 60 inch long by 29 inch high desk where tasks that simulate functional activities using familiar objects are performed (pencil, soup can, weighted sandbags, paper clips, playing cards, key, towel, laundry basket).

The Mini-Mental State Examination (MMSE) is a pencil and paper test that takes approximately 5 to 10 minutes to complete and measures cognition.12 The maximal total score is 30 points. To be included in the study, subjects had to have a minimum score of 24 to read and execute simple written commands. The MMSE was used to rule out major cognitive impairments. The MMSE has established interrater reliability, construct validity, and concurrent validity.12

Procedures

Before the study was begun, a researcher explained the study and an informed consent was signed by all subjects. Subjects then completed the MMSE. The subjects then completed a brief medical history questionnaire. Subjects then read the timed sentence and performed test 11 on the Titmus Vision TestTM. Next, subjects performed the Jamar grip strength test followed by a manual muscle strength test, and an upper extremity active ROM assessment bilaterally.

Data collection began once the subject had satisfied the inclusion criteria. On test day one, both the APB2000 and WMFT were administered randomly (randomized number scale) to both the right and left upper extremities. Each subject performed a practice 7-move sequenced trial bilaterally. Subjects then performed three 21-move trials with each upper extremity. The first trial on each side was used as an additional practice session while subsequent trials two and three were recorded (Appendix 1). The WMFT was also administered bilaterally (Appendix 2). Results from both tests were compared to establish validity.

Test day 2 consisted of the APB2000 testing only. The time interval between test day one and test day two was 2 weeks. Results from both tests were compared to establish reliability.

Data Analysis

Data were analyzed using the SPSS 10.0 program. Linear regression analysis of the results from the pretest and post-test sessions performed on separate days was used to determine test-retest reliability. Twenty-eight units of data were calculated (right and left upper extremities of all 14 subjects). Pearson product moment correlation coefficient was used to determine criterion-related concurrent validity. The level of significance was set at P = .05.

Using an average measure ICC, the investigators attempted to establish reliability through a single reliability index. Since there are no absolute standards by which to justify the reliability of an instrument, the investigators agreed to use the general guidelines as suggested by Portney & Watkins.7 In general, values above 0.75 are indicative of good reliability, and values below 0.75 represent poor to moderate reliability. “For most clinical measurements, reliability should exceed 0.90 to ensure reasonable validity.”7 Although both the stroke and normal subjects represented two separate homogeneous samples, when pooled together they represent one heterogeneous sample. Because these two samples were matched-pairs, other extraneous factors have been minimized. When performing a test-retest reliability study using ICC, it is essential to use a heterogeneous sample because “...the variability among subjects’ scores must be large to demonstrate reliability.”7 “Therefore, it is imperative that researchers be aware of the extent to which scores will naturally vary, and try to obtain heterogeneous samples whenever possible.”7

Results

Using the single measure ICC (3,1), pretest trial 2 compared with post-test trial 2 resulted in the ICC value of 0.78. Comparing pretest trial 3 with post-test trial 3, the ICC (3,1) value was 0.90. When the mean value of pretest trials 2 and 3 were compared with the mean value of post-test trials 2 and 3, the resulting average measure ICC (3,2) value was 0.93.

Concurrent validity for the APB2000 was determined by comparing the pretest trial 3 of the APB2000 to the WMFT. Using a Pearson product moment correlation, the value of r was 0.101, which did not exceed the critical value of .423 (2-tailed). Thus, the Ho was not rejected.

Discussion

Individual APB2000 trials on both upper extremities (trial 2 = 0.78, trial 3 = 0.86) resulted in greater than the priori determined ICC value of 0.75 for good reliability. By obtaining the mean value of trials 2 and 3, the reliability of the APB2000 exceeded the 0.90 value as suggested by Portney and Watkins7 to establish reliability, with a value of 0.93 for both upper extremities. For this reason, the investigators recommend that clinical testing and future research projects using the APB2000 perform a minimum of two recorded trials after adequate patient-subject familiarity with the equipment. Using average measure ICC, it would appear that the APB2000 is a highly reliable tool to provide clinical measurements for baseline and progression testing.

Validity for the APB2000 was not supported. Although there was not a strong correlation, these results may be misleading because there were several outliers that swayed the results considerably (Figure 1). A larger sample size may result in a stronger correlation because the influence of potential outliers would be reduced.

The APB2000 appears to be a highly reliable tool to provide clinical measurements for baseline and progression testing in both the pathological and non-pathological population. Further validity testing with a larger sample size may result in a stronger positive correlation.

A major difference between the WMFT and the APB2000 is the fact that the cognitive aspect of the task to be performed during the WMFT is explained to the patient ahead of time. The examiner then asks the patient if they have any questions regarding the task to be performed. This removes the cognitive processing time from the assessment and the focus of the test is purely motor planning and execution. During the APB2000 testing, written commands appear on a computer screen which initiate a timer that stops once the patient has completed the initial part of the test sequence. This provides the examiner with a quantitative measurement of cognitive processing time. Mental rehearsal or mental practice, has been repeatedly described in the literature as a cognitive tool that is effective in facilitating the acquisition of new motor skills.13–17 The APB2000 allows the examiner to place a temporal value on this process.

Conclusions

The reliability of the APB2000 exceeded the 0.90 value to establish reliability when using the average measure ICC. Validity for the APB2000 was not supported. There was not a strong correlation between the APB2000 and the WMFT. The APB2000 appears to be a highly reliable tool to provide clinical measurements for baseline and progression testing in both the pathological and nonpathological population. Further validity testing with a larger sample size may result in a stronger positive correlation.

Acknowledgments

The authors thank Grenith Zimmerman, PhD, and Donna Thorpe, MPH, PT, for their assistance with the data analysis.

References

1. Bennett S, Karnes J. Neurological disabilities: Assessment and treatment. Philadelphia, PA: Lipponcott; 1998.

2. Bernstein N. The coordination and regulation of movement. London: Pergamon, 1967.

3. Shumway-Cook A, Woolacott M. Motor control: Theory and practical applications. Philadelphia, PA: Lippincott; 2001.

4. Gordon J. Assumption underlying physical therapy intervention: theoretical and historical perspectives. In: Carr J, Shepherd R, eds. Movement Science: Foundations for Physical Therapy in Rehabilitation. Gaithersburg, MD: Aspen; 2000.

5. Wolf SL, Catlin PA, Ellis M, et al. Assessing Wolf motor function test as outcome measure for research in patients after stroke. Stroke 32:1635–1639, 2001.

6. Morris DM, Uswatte G, Crago JE, et al. The reliability of the wolf motor function test for assessing upper extremity function after stroke. Arch Phys Med Rehabil 82:750–755, 2001.

7. Portney LG, Watkins MP. Foundations of clinical research: Applications to practice 2nd ed. New Jersey: Prentice Hall; 2000.

8. Kendall FP, McCreary EK, Provance PG. Muscle testing and function 4th edition. Baltimore, Maryland: Williams and Wilkins, 1993.

9. Norkin CC, White DJ. Measurement of joint motion: A guide to goniometry. 2nd edition. Philadelphia, PA: FA Davis Company, 1995.

10. Wolf Motor Function Test Manual. Unpublished resource revised by University of Alabama at Birmingham and Birmingham Veteran’s Administration Medical Center, 2000.

11. Task Force on Standards for Measurement in Physical Therapy, 1991. New York: Riddle & Stratford, 1999.

12. Folstein MF, Folstein SE, McHugh PR. Mini mental state: A practical method for grading the cognitive state of patients for the clinician. J Psych Res 12:189–198, 1995.

13. Feltz D, Landers D. The effects of mental practice on motor skill learning and performance: A meta-analysis. J Sports Psychol 5:25, 1983.

14. Richardson A. Mental practice: a review and discussion (Part 1). Res Q 38:95, 1967.

15. Richardson A: Mental practice: a review and discussion (Part 2). Res Q 38:263, 1967.

16. Warner L, McNeill M. Mental imagery and it’s potential for physical therapy. Phys Ther 68:516, 1988.

17. MaringJ. Effects of mental practice on rate of skill acquisition. Phys Ther 70:165, 1990.

 

Table 1. Characteristics of Matched Pairs

 

                        Normal Subjects                                  Stroke Subjects

                                                        Hand                           Gender            Hand             Brain           Last
Pairs      Age (y)        Gender       Dominance     Age (y)    Involved*    Dominance/      Trauma      Trauma†                                                                        

1.              75            Female           Right             74          Female        Right/Left          CVA            32

2.              56            Female           Right             53          Female       Right/Right         CVA            52

3.              63              Male             Right             60            Male         Right/Right         CVA            16

4.              84            Female           Right             87          Female        Right/Left          CVA             5

5.              59            Female           Right             62          Female        Right/Left           TIA              39

6.              81            Female           Right             79          Female        Right/Left          CVA            48

7.              86            Female           Right             85          Female       Right/Right         CVA            23

 

*Side of upper extremity involvement.

†Time in months since the most recent brain trauma.

 

Figure 2. Automated Pegboard 2000 (APB2000): Right-to-Center Test.

 

Figure 1. Relationship between Wolf Motor Test scores and APB 2000 trial 3 total test scores (n=28).

 

Appendix 1. Operational definition and sequence of the APB2000:

 

First, each subject will be positioned in a super-incumbent posture facing the midportion of the APB2000. The APB2000 thigh stabilization bar will be position at the anterior aspect of the superior one-quarter of the subject’s thigh. This location will be determined by measuring the distance from the subject’s anterior superior iliac spine (ASIS) to the floor and multiplied by .75. The APB2000 pegboard angle of inclination can be positioned at increment gradients between 9˚ and 26˚. For this study, the gradient will be fixed at 9˚ or the position closest to the horizontal position (See Appendix 1 for APB2000 and subject positioning).

 

Next, the researcher will explain the APB2000 testing procedures to the subject. Once the subjects have had adequate practice time to familiarize themselves with the APB2000 test, the formal testing will begin. The subject will read and respond to a 21-sequence command on the monitor. Subjects will be asked to move various shapes across their body to various locations on the pegboard. To begin the task, the subject will press the start/stop or “home” button on the APB2000. At the end of each activity, the subject will press the same button to signify that the task is completed. Commands include but are not limited to using the left hand to move rectangle from right column one to center column two (Right-to-Center test). The computer will record each command, display time, distance, return time, and trip time per command. Tables and graphs will be used to display all data obtained. Three 21-move trials will be performed using each upper extremity. Those subjects selected for the test-retest reliability portion of this study will return at a time interval of one week (5 to 9 days following testing session one) for a second APB2000 testing session (Left-to-Center and Right-to-Center tests). Before this follow up testing procedure, each subject will be briefly questioned to establish that no major changes have occurred in the 1-week interval (such as medication changes, injury, illness, and so forth).

 

The test described below is called the “Left-to-Center” test. This test involves using the right hand to strike the home button as well as retrieval and relocation of the desired object. This test will functionally target the left hemisphere of the brain. As the name of the test indicates, once the subject is cued to do so by the visual monitor, the subject will move the desired object from the left hand side of the pegboard to the center using only the right upper extremity. The task is divided into two components: The motor planning and motor execution phases. The first phase starts the moment that the subject depresses the home button, continues as the subject reads the monitor for task instructions, then the subject reaches to retrieve the indicated object and stops the moment that the peg leaves its’ receptacle on the pegboard. The reaction time to complete this phase is recorded. The completion of phase one signals the start of the second phase that includes locating the indicated central pegboard resting position for the peg, and placement of the peg in the correct pegboard location. The time to complete this component of the task is recorded by the APB2000 computer. Next the subject releases the peg and depresses the home button. This concludes move one of the 21-move testing sequence. This instantly starts the sequence again until all 21 of the pegs have been successfully relocated from the left hand side of the board to the center of the automated pegboard. The computer tracks the total time to complete this task as well as the timing of the two phases. The “Right-to-Center” is the opposite process which involves the subject moving the pegs from the right-hand side of the pegboard to the center while using their left hand (Fig 2.). The color-coded basic shapes (diamonds, rectangles, stars, triangles, ovals, squares, and circles) are symmetrically arranged in three columns and seven rows on the left and right of the screen. The corresponding basic shaped receptacles in the center of the pegboard are located randomly and at various orientations so that some moves require that a subtle rotation movement be performed by the subject in order to successfully place the shape in the central receptacle.

 

Appendix 2. Operational Definition and Sequence for the WMFT

 

The tasks performed for the WMFT are listed below in the order of testing sequence:10

Task 1.    Forearm to table (side): The subject attempts to place their forearm on the table by active shoulder abduction.

Task 2.    Forearm to box (side): The subject attempts to place their forearm on the box by active shoulder abduction.

Task 3.    Extend elbow (side): The subject attempts to reach across the table through elbow extension.

Task 4.    Extend elbow with weight (side): The subject attempts to reach across the table through elbow extension while pushing a weighted sandbag with the ulnar border of the wrist.

Task 5.    Hand to table (facing): The subject attempts to place their hand on the table through active shoulder flexion.

Task 6.    Hand to box (facing): The subject attempts to place their hand on the box through active shoulder flexion.

Task 7.    Reach and retrieve (facing): The subject attempts to pull a 1-lb weight across the table with a “cupped hand” through elbow flexion.

Task 8.    Can lift (facing): The subject attempts to lift a soup can and bring it to their lips with a cylindrical grip.

Task 9.    Pencil lift (facing): The subject attempts to lift a pencil with a three-jaw chuck grasp.

Task 10. Paper clip lift (facing): The subject attempts to pick up a paper clip using a “pincer” grasp.

Task 11. Stack checkers (facing): The subject attempts to stack checkers onto the center checker.

Task 12. Flip cards (facing): The subject attempts to flip over each card using a “pincer” grasp.

Task 13. Turning the key in the lock (facing): The subject attempts to turn the key to both the left and the right using a “pincer” grasp.

Task 14. Towel fold (facing): The subject retrieves the towel then attempts to fold it length and the using only the test side hand, folds it in half again.

Task 15. Basket lift (facing): The subject lifts a laundry basket by grasping the handles and placing it on a “bedside table”.