Design-Based Research: The Ongoing Assessment Project (OGAP)

Authors: Robert Laird, Marge Petit, Stephanie Ratmeyer, Douglas Harris

Research to Date:

To date the primary research approach has been design-based research (Hake, 2004, Schoenfield, 2007, RAND Mathematics Study Panel, 2003). This approach involves designing, studying, and refining a theory-based intervention in the context of real classroom settings. The design-based research has encompassed development in these broad areas through interaction with hundreds of classrooms and thousands of students in multi interactions of development.

• Distillation of research related to student learning and misconceptions in specific mathematics content (fractions, multiplicative reasoning, and proportionality)

• Development of formative assessment tools and resources including a bank of formative assessment items designed to elicit student learning in each of the three content areas above
• Instructional decision-making based on knowledge of math education research and formative assessment information
• Assessment of student performance prior, during,  and following instruction
• Documenting the interaction of knowledge of math education research, increased math content knowledge, formative assessment, and instructional decision-making
• Development of professional development materials to communicate education research and use of OGAP materials and resources.
• Documenting the impact of OGAP participation on classroom teachers' knowledge and practice

Data collected and analyzed to date includes the following:

• Measures of student performance pre and post-intervention (e.g., pre and post assessment, retention study  (Examples 1 and 3 below)
• Collection and analysis of student work illuminating math education research in the targeted content areas (Example 2 below)
• Impact of professional development and use of OGAP on teachers' understanding of math education research and mathematics content
• Teacher perception of the impact of professional development and use of OGAP on teacher practice   (Example 4)  
• Analysis of teacher logs  

Example 1: Sample Student Performance Data - Student Pre-Post/Retention Study

A preliminary study of student retention was conducted in December 2005. The study involved administering the common items in the OGAP pre/post assessments that students completed during the 2005 OGAP Exploratory Study.

Fourth grade School B students completed the grade 3 pre-assessment items 1-12 on December 11, 2005.  This was the same assessment that was administered in January 2005 and at the end of the intervention in May 2005. Three students moved or were absent the day the follow-up assessment was given. Only the students who participated in OGAP in the winter of 2005 were included in the retention study.

Table 1: Samples Tested 2005 OGAP Retention Study

Table 2: School A OGAP Retention Study

Example 2 - Collection and Analysis of Student Work

OGAP 2005 Study - Use of inappropriate whole number reasoning (WNR) in OGAP pre and post assessments

This study involved analyzing a sample of student work in grade 4 pre to post assessments for evidence of applying whole number reasoning when solving fraction problems. The sample included 39 pre and post assessments and 442 items.

About 30.9% (137/442) of all responses analyzed for this error on the OGAP pre-assessment included solutions in which students incorrectly applied whole number reasoning to solving fraction problems(OGAP 2005 Exploratory Study). 

Table 3: OGAP Student Work Analysis - Use of Whole Number Reasoning

Teachers attributed the change from inappropriate whole number reasoning to reasoning with fractions as quantities to their increased emphasis on modeling, comparing and ordering fractions, and to the use of the number line. 

Example 3 - Controlled Study of Impact of OGAP - Whole Number Reasoning

Table 4: 2007 Action Research Data (C. Redmond, 2007)

Cohen's d Effect Size = .77

Example 4 - Impact of Professional Development and Use of OGAP on
Teacher Knowledge and Practice

Table 5: Teacher perception of the impact of OGAP on knowledge and practice 2006 - 2007 Fraction Scale-up

Further data collection in each of the categories above is necessary. Differential analysis of student groups formed by poverty, special education, and English Language Learners populations is recommended. Classroom observation and external review of teachers' plans and classroom materials are also in order.

Most of the data analysis has focused on fractions, then multiplicative reasoning, then proportional reasoning.