Problem Solving and Inquiry (PSI) Tasks in the TIMSS 2019 Assessment Design: A Look at Booklet Completion Rates

Basic Design of Blocks and Booklets in TIMSS 2019

The TIMSS 2019 assessment design encompassed eTIMSS and paperTIMSS as mirrored efforts of each other, such that within each, both fourth and eighth grades consisted of mathematics and science achievement items grouped into 28 item blocks (14 mathematics and 14 science).¹ The blocks were arranged into 14 booklets with four blocks per booklet. There were two parallel variations of the design: one for the paper version (paperTIMSS) and another adapted to digital presentation mode (eTIMSS). Because the special Problem Solving and Inquiry (PSI) tasks described herein are inherently computer based and had no paper counterpart, they were treated as a separate addition to the eTIMSS version.

Each TIMSS 2019 booklet or eBooklet consisted of two blocks of mathematics and two blocks of science items. Each item block appeared in two booklets, in a different position in each booklet to account for effects on achievement that can occur from items being earlier or later in the testing sessions (e.g., learning or fatigue). Each student completed one booklet. The 14 booklets were distributed among the students in participating classes according to a random assignment procedure, so that each booklet or eBooklet was assigned to approximately equal percentages of students.

Keeping Booklets 1–14 the same as possible in both paperTIMSS and eTIMSS enabled establishing a link between the paper and digital assessment modes, and made it possible to report student achievement on both paperTIMSS and eTIMSS together on the same TIMSS achievement scale in TIMSS 2019 International Results in Mathematics and Science.^2,3 These results which IEA released in December 2020 presented a comprehensive view internationally of mathematics and science achievement at the fourth and eighth grades as well as contextual data for 64 countries and 8 benchmarking systems.

The Problem Solving and Inquiry eBooklets

To explore how to increase the benefits of digital assessment for TIMSS 2023 (the subject of this report), at each grade, eTIMSS also included four blocks of problem solving and inquiry tasks and items, two of mathematics and two of science. Because the items in eBooklets 1–14 were the same as for paperTIMSS, the PSI item blocks were assigned to eBooklets 15–16. Similar to the paperTIMSS assessment, all 16 eBooklets were randomly distributed among students in the eTIMSS classes.

Allocating the PSI tasks to separate booklets (eBooklets 15–16) that were rotated along with eBooklets 1–14 provided the means to conduct a subsequent achievement scaling that also placed the PSI tasks on the TIMSS achievement scale.⁴ As described in the Introduction to this report, there essentially was no difference in average eTIMSS achievement as a result of including the PSI data. Although the PSI initiative was important for looking forward to TIMSS 2023, the PSIs only represented a small percentage of the nearly 900 items in the TIMSS 2019 assessment and the students who took them comprised just 12 percent of the 600,000 students.

The PSI tasks and items were allocated to four assessment blocks as shown in Exhibit A.1.

Exhibit A.1:  PSI Assessment Blocks with Subject and Task Label

Exhibit A.1 presents, for both fourth and eighth grades, the PSI item block labels from the TIMSS 2019 assessment design, the subjects that they address, and the PSI tasks that were assigned to them. For example, at fourth grade, block M1 is a mathematics block containing one longer task, Penguins, combined with a shorter Robots-4 task, whereas S1 is a science block containing the single Farm Investigation PSI task. Four of the PSI blocks, School Party and Farm Investigation at fourth grade and Building + Robots-8 and Pepper Plants at eighth grade, are published in full in this report. The other four blocks have been kept secure for use in future TIMSS assessments.

Exhibit A.2 shows, for both fourth and eighth grades, how the PSI assessment blocks were allocated to eBooklets 15 and 16 according to the eTIMSS 2019 assessment design. Each booklet has two mathematics and two science blocks, presented in two separately timed sessions. Blocks are not separately timed within sessions. At fourth grade, each session lasts for 36 minutes, with a 15-minute break in between. Eighth grade sessions last for 45 minutes.

Exhibit A.2:  PSI Block Assignments to eBooklets*

* Students had a 15-minute break between Session 1 and Session 2.

Note that eBooklet 15 and eBooklet 16 contain the same assessment material, but presented in different orders. For example, at fourth grade, eBooklet 15 contains the two mathematics blocks, Penguins + Robots-4 and School Party in the first session (Positions 1 & 2) and, then after the break, the two science blocks, Farm Investigation and Sugar Experiment, in the second session (Positions 3 & 4). eBooklet 16 begins with the two science blocks in the first session (Positions 1 & 2) but in the reverse order from eBooklet 15: Sugar Experiment followed by Farm Investigation. Session 2 (Positions 3 & 4) then contains the two mathematics blocks, but again in reverse order from eBooklet 15: School Party followed by Penguins + Robots-4. This block-booklet arrangement counterbalances the subject position effect (mathematics first, then science in eBooklet 15; science first, then mathematics in eBooklet 16). The arrangement also counterbalances the position-within-session effect to the extent possible (e.g., Penguins + Robots-4 followed by School Party in Position 1 & 2 of the first session in eBooklet 15; School Party followed by Penguins + Robots-4 in Position 3 & 4 of the second session).

Not Reached Items in the PSI Tasks

In reviewing the student response data for the PSIs, it became evident that students assigned PSI tasks (eBooklets 15 and 16) were not always completing their booklets at the same high rates as students assigned booklets containing regular, non-PSI items (eBooklets 1 to 14). As shown in Exhibit A.3, the percentage of students reaching all items in the regular eBooklets is very high, ranging from 92 percent for mathematics and science at fourth grade to 97 percent for eighth grade science. In contrast, students assigned PSI eBooklets had considerably lower completion rates, especially at fourth grade: 66 percent for mathematics and 76 percent for science.

Exhibit A.3:  Student Completion Rates for Regular eTIMSS and PSI eBooklets

Because eBooklets 15 and 16 contain the same PSI blocks but in different orders, it was possible to investigate whether the lower completion rates were related to the block position in the booklet, or more specifically if a block presented earlier in a session had higher completion thaN a block later in a session. Exhibit A.4 addresses this issue by comparing the percentages of students not reaching the last item in each PSI block when the block is at the beginning and end of a session.

Exhibit A.4:  Percentage of Students Not Reaching the Last Item in each PSI Block, by Block Position*

* Students had a 15-minute break between Session 1 and Session 2.

In eBooklet 15, the Penguins + Robots-4 fourth grade mathematics task is in the first block position and students had no difficulty reaching the end of the task, with only 4 percent not reaching the last item. However, in eBooklet 16 this task is in Position 4, the last position at the end of the testing session, and the percentage of students not reaching the last item increased to 30 percent. A similar situation is apparent for School Party, the other fourth grade mathematics task. In eBooklet 16 this task is in Position 3, which is the first position in Session 2 after the break. Position 3 is similar to Position 1 in that students have just had a break and are starting afresh in a new, separately timed session. Students had no difficulty completing School Party in this position, with just 1 percent not reaching the last item in the block. In contrast, School Party is in block Position 2 in eBooklet 15, which is the end of the first session. Students assigned this booklet would have worked through the Penguins + Robots-4 task before beginning School Party, and by the end of the task the percentage not reaching the last item increased to 39 percent.

The two fourth grade science PSI tasks show a block position effect similar to mathematics, although the percentage not reaching the last item is somewhat less: 19 and 28 percent for Sugar Experiment and Farm Investigation, respectively, compared with 30 and 39 percent for Penguins + Robots-4 and School Party.

The completion rates for eighth grade were somewhat better than for fourth grade, as shown in Exhibit A.3, and this was reflected in lower percentages of students failing to reach the last item in a block (Exhibit A.4). The two mathematics PSI blocks (Building + Robots-8 and Dinosaur Speed) had essentially no students not reaching the last item when these blocks were in the first position of a session (Position 1 and Position 3), but somewhat more (21% and 14%) when in the second session position (Position 2 and Position 4). The two eighth grade science PSI blocks showed the least not-reached effect, with essentially all students reaching all items when these blocks were in the first position of a session, and just 5 percent not reaching the last item in Pepper Plants and 8 percent in Sunken Ship when in the second position of a session.

Percent Correct in the PSI Report

Considering the relatively large percentages of students not reaching all of the items in the PSI blocks, it was clear that not all students had an opportunity to answer all of the items and that the usual TIMSS practice of treating not reached item responses as incorrect could introduce bias into the reported results. To avoid this situation, all not-reached responses were considered to be “not administered” rather than as incorrect, and not included in the computation of percent correct statistics. Accordingly, the base of the percent correct statistics presented in this report is the number of students that reached the item and had an opportunity to answer, rather than the number of students to whom the booklet was administered. The procedure of treating not-reached responses as not administered also was adopted in scaling the PSI data and scoring student responses.⁵
 

Notes

---

---