Challenges and Opportunities in Evaluating Two MSP Projects
Authors: Jane Butler Kahle, Yue Li, Sarah Woodruff

« Back to Poster Hall
4. Results
Next »

PennSTI
Comparisons Before and Three Years After Teacher Initial Participation in PennSTI
Differences were found between pre and final subscale means on the teaching ("What I do in class") and learning ("What my students do") subscales for teachers participating in the PennSTI (Table 3). These analyses used raw mean scores. Science teachers reported slightly more use of inquiry by their students, while chemistry teachers significantly increased their own ("What I do in class") use of inquiry.

 

Table 3. Comparison of Subscale Mean Scores for Penn Science and Chemistry Teachers: Pre (Spring 2005) and Final (Fall 2007).


Course

Subscale

Time

n

Mean

SD

t-value

df

p-value

n2

Science

What I do in class

Pre

16

3.51

0.39

-1.20

15

0.250

0.09

 

Final

16

3.65

0.38

 

 

 

 

What my students do

Pre

15

3.38

0.40

-1.86

14

0.084

0.20

 

Final

15

3.65

0.32

 

 

 

 

Chemistry

What I do in class

Pre

15

3.07

0.28

-3.55

14

0.003

0.47

 

Final

15

3.43

0.34

 

 

 

 

What my students do

Pre

15

3.10

0.46

-2.04

14

0.060

0.23

 

Final

15

3.40

0.38

 

 

 

 

1 = "Almost Never" and 5 = "Very Often"

During the spring of 2005, 2006, and 2007, the student questionnaire was administered to science or chemistry students in one typical class of each participating teacher. Although the student groups changed, they were taught by the same teacher. Item Response Theory (IRT)-Rasch Model-was used to analyze the student data. For the teaching and learning subscales that request frequency ratings, a Rasch measure of 0 indicates that the respondent selected "Almost Never" for all subscale items. A Rasch measure of 100 indicates that the respondent choose "Very Often" for all subscale items.

Science students reported more use of standards-based instruction on both the teaching ("What my teacher does") and learning ("What I do") subscales after teacher participation (Table 4). Likewise, chemistry students of PennSTI teachers reported increased frequency of inquiry learning after teacher participation (Table 5).

Table 4. Comparison of Rasch Mean Subscale Scores of Penn Science Students: Pre, Post Year 1, and Post Year 2.


Subscale

 Time

n

Rasch Mean

SD

F-value

df

p-value

n2

What I do in class

Pre

555

46.68 *

4.83

18.30

2, 1323

< 0.001

0.03

Post Yr 1

413

48.41

4.93

 

 

 

 

Post Yr 2

358

48.10

4.36

 

 

 

 

What my teacher does in class

Pre

552

60.50 *

12.31

7.20

2, 1316

0.001

0.01

Post Yr 1

410

62.89

13.10

 

 

 

 

Post Yr 2

357

63.29

10.98

 

 

 

 

* Mean scores for Post Year 1 and Post Year 2 are significantly higher than Pre mean scores.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Table 5. Comparison of Rasch Mean Subscale Scores of Penn Chemistry Students: Pre and Post Year 2.*


Subscale

 Time

n

Rasch Mean

SD

t-value

df

p-value

n2

What I do in class

Pre

362

46.70

5.87

-2.89

610

0.004

0.01

Post Yr 2

250

48.08

5.80

 

 

 

 

What my teacher does in class

Pre

362

57.16

12.19

-1.56

576

0.120

0.004

Post Yr 2

249

58.60

10.62

 

 

 

 

* The questionnaires were not administered Post Year 1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

 

MSPinNYC
Comparisons Before and One Year After Teacher Participation in the MSPinNYC Summer 2006 Institutes
Item Response Theory (IRT)-Rasch Model-was used in order to provide the project with rigorous and appropriate analyses. Comparisons were made between Rasch mean scores before the institute and one year after participating as a TRT teacher in an institute (Table 6). Significant differences were found (in favor of more frequent use of inquiry and other standards-based instruction) on both the teaching and learning subscales one year after teacher participation.

Table 6. Comparison of Rasch Mean Scores for Teaching and Learning Subscales for MSPinNYC TRT Teachers: Spring 2006 and Spring 2007.


Subscale

Time

n

Rasch Mean

SD

t-value

df

p-value

n2

What I do

Spring2006 (Pre)

17

57.77

8.81

-2.04

16

0.058*

0.21

Spring2007 (Post)

17

61.13

7.95

 

 

 

 

What my students do

Spring2006 (Pre)

17

49.45

7.21

-2.66

16

0.017

0.31

Spring2007 (Post)

17

53.31

6.23

 

 

 

 

* Using alpha = 0.1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Students in TRT teachers' classes completed the student questionnairepre and post teacher participation in MSPinNYC.  Four different institutes were offered in 2006 and the findings for the learning subscales are displayed in Table 7 by specific institute/class.

Table 7. Comparison of Rasch Mean Scores for Learning Subscale for Students of MSPinNYC TRT Teachers: Spring 2006 and Spring 2007.


Subscale

Time

n

Rasch Mean

SD

t-value

df

p-value

n2

Mathematics A

Spring2006 (Pre)

59

46.38

2.27

-2.36

130

0.020

0.04

Spring2007 (Post)

73

47.50

3.01

 

 

 

 

Mathematics B

Spring2006 (Pre)

37

46.32

2.39

-3.13

91

0.002

0.10

Spring2007 (Post)

56

47.99

2.59

 

 

 

 

Living Environment

Spring2006 (Pre)

99

45.77

2.64

1.72

182

0.088*

0.02

Spring2007 (Post)

85

45.01

3.35

 

 

 

 

Chemistry

Spring2006 (Pre)

68

44.97

3.17

-2.19

138

0.030

0.03

Spring2007 (Post)

73

46.12

3.00

 

 

 

 

* Using alpha = 0.1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Mathematics A and B and chemistry students indicated significantly more use of problem solving and other active learning strategies on the learning subscale after their teacher participated in an MSPinNYC institute. However, for students in Living Environment classes the reverse was true.

Comparison between Participant and Non-Participant Teachers in MSPinNYC Spring 2007

Comparisons also were made between the responses of the two groups of teachers and those of their students. No statistically significant differences were found for any of the subscale comparisons on the responses of teachers who participated in the Summer 2006 institutes (TRT team members) and non-participant Hub school colleagues. However, significant differences were found in comparing responses of the two groups of students (Table 8).

Table 8. Comparison of Rasch Mean Scores for the Learning Subscale for Students of MSPinNYC Participant and Non-Participant Teachers: Spring 2007.


Subscale

Time

n

Rasch Mean

SD

t-value

df

p-value

n2

Mathematics A

Participants

73

47.50

3.01

1.82

179

0.071*

0.02

Non-Participants

110

46.64

3.17

 

 

 

 

Mathematics B

Participants

56

47.99

2.59

2.94

97

0.004

0.08

Non-Participants

43

46.34

2.97

 

 

 

 

Living Environment

Participants

86

45.09

3.40

-1.65

322

0.101

0.01

Non-Participants

238

45.72

2.96

 

 

 

 

Chemistry

Participants

98

46.68

3.03

2.31

121

0.023

0.04

Non-Participants

25

45.05

3.56

 

 

 

 

* Using alpha = 0.1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Mathematics A and B students of TRT teachers indicated significantly more frequent use of active learning strategies than did students of non-participating teachers. No statistically significant differences were found between the responses of students in Living Environment classes taught by participating teachers and those taught by non-participating teachers.  Chemistry students of participating teachers, compared to students whose teachers had not participated, indicated significantly more frequent use of standards-based or inquiry activities on the learning subscale.