Robots Items
The idea for Robots was part of the work to develop PSI tasks, but as the work evolved the TIMSS Robots did not seem to have the necessary requirements to become PSI tasks. Still the idea has potential for future digital assessments. As shown in this section, the TIMSS Robots were programmed to provide the value of y for any value of x, illustrating a particular type of digital assessment item. The first robot has only one item and the second robot has two items.
The Robots results for eighth grade are presented here, because the two item screens followed directly after Building in the assessment sessions. It seems that the eighth grade students found the combination of Building (geometry/algebra) followed by Robots (algebra) challenging. First, 15 percent of the students omitted Screen 1 of Robots on average, and then the average percent not reached rose from 3 percent for the first item to 11 percent for the last of the three.
Screen 1
The robot on Screen 1 provided a y value for each x value submitted by the students. The students were asked to use the number pad to enter some x values to determine the robot’s rule for determining y.
Click video to play
Maximum Score Points: 1
Content Domain: Algebra
Topic Area: Relationships and Functions
Cognitive Domain: Reasoning
Results 1
As shown in Exhibit 37, 20 percent of the students on average correctly identified the robot’s rule (1 point), which was y = 2x + 10. Korea (40%), Singapore (39%), and Chinese Taipei (37%) had the highest percentages of eighth grade students identifying the rule. Looking at the process data revealed that the most popular input strategy was entering sequential numbers, although some students entered multiples (2s, 5s, or 10s). On average across countries, boys had a higher percentage correct than girls.
As shown above on Screen 1, the table provided to the students included four rows for the students to accommodate a total of four entries, and then permitted reusing the boxes. Although further analysis showed that 53 percent of the students on average realized this and reused the boxes, looking at the process data suggested that some students were less familiar with these types of response spaces. Students’ uncertainty about how to use the boxes may have contributed to the omit rate, since 8 percent of the students on average did not make even one entry (another 1 percent made only one entry). Running out of entry boxes may also have had adversely affected achievement, because on average, 37 percent of the students made exactly four entries.
Screen 2
The second robot on Screen 2 had a different rule for determining the relationship between x and y. The relationship was somewhat more complicated, so the robot made a table of 6 pairs of x and y, but left out the y for the fourth pair and the x for sixth pair. As shown below, the numbers in the white boxes represent the answers, but were blank in the table provided to the students. The students were asked to complete the table and provide the robot’s rule. The non-response rates for Screen 2 were similar to those for Screen 1, but the omit rates increased with 16 percent omitting part A. The non-response rates for Screen 2 were similar to those for Screen 1, but the omit rates increased with 16 percent omitting part A. Again, the format most likely was a contributing factor. This item was not interactive. That is, based on the robot in Screen 1, students may have thought they could make additional entries into the table. But for the second robot, students needed to do their work on separate “scratch” paper.
Click video to play
Results 2A
Exhibit 38 contains the results for the percent of students receiving full credit (2 points) or partial credit (1 point) in 2A for finding either x or y, but not both. Eighth grade students in Chinese Taipei, Singapore, and Korea had the highest achievement—49, 46, and 45 percent of students were awarded full credit, respectively. However, there was a wide range in performance with 6 countries having less than 20 percent fully correct, such that the overall average for full credit was 25 percent. Another 13 percent on average received partial credit (1 point)—7 percent for finding y but not x and 6 percent for finding x but not y. Boys had higher achievement than girls on average across countries.
Results 2B
For writing the robot’s rule, where students were expected to type their answer in the response line, the omit rate jumped to 31 percent. Exhibit 39 presents the percentages of correct responses for 2B. Perhaps due to the high percent of omissions, the average percentage of correct responses—19 percent—was even lower than the average percentage completing the table. Korea (39%), Singapore (35%), and Chinese Taipei (34%) had the best performance. Boys had higher percentages of correct responses than girls on average across countries.