To: System Test Coordinators and School Principals
From: Montana Office of Public Instruction
Date: November 8, 2018
Subject: Smarter Balanced Grade 8 Math Rescore
Dear System Test Coordinators and School Principals:
On September 7th, the Office of Public Instruction (OPI) was informed by the Smarter Balanced Assessment Consortium that a member state had identified issues with one item in the Grade 8 Mathematics item package based on review of their state’s student data file. Upon further investigation into the Smarter Balanced 2017–18 test package, it was determined that there was a Grade 8 item with inaccurate difficulty parameters in place resulting in the algorithm estimating students’ abilities as too low. Additional checks were introduced to confirm the accuracy of the summative and interim operational item pools and no other issues were found.
On September 12th in preparation for the annual results presentation to the Board of Public Education (Board), the OPI released the 2018 assessment data for all six statewide assessments including the Math data set with the improper item scoring for this one Grade 8 Math item included. Schools were notified of the data issue in the statewide GEMS longitudinal database disclaimer (gems.opi.mt.gov) and within the PowerPoint shared with the Board.
Since the release the OPI has been working with its test delivery vendor Measured Progress to rescore this item for roughly one third of the Grade 8 students who received it. Measured Progress and American Institutes for Research (AIR) worked together to identify roughly 4,000 students in Montana who had received the item and to identify a strategy to rescore tests for students who received the item, holding harmless the students who got the item correct. To resolve the Grade 8 Math item issue, the test delivery vendor removed the item from the assessment package and rescored the assessment without the item included. This rescore procedure positively impacted these students’ performance resulting in some modest scale score increases.
Of the 4,000 students only about 10% who had received the item also changed proficiency levels. As a reminder student performance is reported by achievement levels, that is, Level 1 through 4.
Students who score a Level 3 or 4 are considered as performing at or above proficient.
Level 4 indicates that a student demonstrates advanced progress needed for success in future course work.
Level 3 indicates that a student demonstrates progress needed for success in future course work.
Students who score a Level 1 or 2 are not considered as proficient.
Level 2 indicates that a student requires further development to meet proficiency.
Level 1 indicates that the student needs substantial improvement to meet proficiency.
Should your school have students in this 10% category, you will be contacted with a separate e-mail instructing you on how to download new individual student score reports using the Online Reporting System (ORS) in online portal (mt.portal.airast.org). The statewide longitudinal database has also been updated with the correct Grade 8 data set (gems.opi.mt.gov). To see how your school is performing over time and compared to other schools in Montana, visit the OPI’s GEMS page at gems.opi.mt.gov.
Going forward, the Smarter Balanced Assessment Consortium has implemented additional quality control processes including an automatic check for large standard errors to prevent this issue from occurring again. Smarter Balanced staff will also institute additional checks to assure the quality of the item statistics.
If you have any questions about this password refresh, please contact the OPI Assessment Help Desk at 1-844-867-2569 or OPIAssessmentHelpDesk@mt.gov.
-OPI Assessment Division