 |
 |
|||||||||||
 |
||||||||||||
 |
The Roadmap to Redesign (R2R)
Eastern Washington University Course Title: Introductory Psychology Project Abstract
Eastern Washington University plans to redesign its Introductory Psychology course. The course is presently taught as a five-credit lecture course by a mix of instructors at all ranks. Enrollments are growing. Ten sections of approximately 98 students each are expected for the 2004–2005 academic year. The course meets five times per week for 11 weeks (quarter system). The large classes in Introductory Psychology make it difficult to focus on the development of critical thinking and communication skills. The lecture format places students in a passive learning role, and there is little time for meaningful discussion among students. Moreover, legislative funding has not kept up with enrollment demand so faculty teaching loads are high. The faculty need release time for other expected professional activities such as research, grant writing, and curriculum development. Eastern Washington plans to use the Replacement Model in the redesigned course. Student learning will be improved by using technology to enhance lecture delivery, to engage students in active learning, and to provide opportunities for self-assessment. Class size will increase to 245 students, and the number of sections will decrease to four per year. There will be three lectures per week instead of five. Students will be required to attend a small seminar meeting every week (~12 students each). Graduate teaching assistants (GTAs) and undergraduate mentors will lead twenty scheduled seminar discussions per week. These small group discussions will facilitate the development of critical thinking and communication skills. Weekly online mastery quizzes will supplement bi-weekly in-class exams. Department faculty will decide on a common set of assessment objectives for Introductory Psychology. A set of test items corresponding to these common course objectives will be developed for inclusion on all final examinations administered in both the traditional sections (fall 2004 baseline) and the redesigned sections (winter 2005 and spring 2005 pilots). Performance on these questions will be used to compare the effectiveness of the redesigned course. In addition, students in all sections will be surveyed to determine the level of student satisfaction, and changes in student attendance and retention will be measured. In addition, the effectiveness of small discussion groups will be assessed using pre-test, post-test comparisons with specific emphasis on improvements in communication skills. The operational cost of the course will be reduced by decreasing the number of sections from ten to four per year and increasing the section size from 98 students to 245 students per section; decreasing the number of lectures delivered by faculty from five per week to three per week; decreasing the number of GTAs from five per year to four per year; and decreasing the number of in-class exams from eleven per term to six per term. The cumulative impact of these efficiencies will be to reduce the cost of course delivery from $100 per student to $40 per student, a 60% reduction. The cost may be reduced even further, to $35 per student, if full responsibility for teaching Introductory Psychology is eventually shifted to junior faculty. Interim Progress Report (as of 5/15/05) Eastern Washington University spent fall 2004 getting ready for the pilot. Two seminar rooms were constructed for the redesign, each equipped with a computer, LCD projector, and a Smartboard. The rooms are furnished with seminar-style tables that can be moved on glides over a carpeted floor to create separate working groups for special projects. An existing high-tech auditorium seating 240 students was updated with a Smartboard and a personal response system. The Enterprise version of Blackboard 6 was installed during winter 2005, permitting EWU to interface with Questionmark Perception for mastery quizzing. Conversion of the test bank for the selected text is underway as a joint undertaking by Worth Publishing and Questionmark. Included with the text is a CD supplement entitled PsychInquiry for activities and materials for student projects. Interim Progress Report (as of 4/15/06) In fall 2005, EWU fully implemented its redesign of introductory psychology. The project was scaled from one pilot section with about 180 students to two fully implemented sections of 250 students each. Thirty mentors were trained to lead seminar sections in a one-day workshop before the first day of class. Training continued at weekly meetings during the fall term. A new assistant professor was introduced to the redesigned format and assigned to one lecture section. The performance of his class was not distinguishable from that of the more experienced instructor. Each of the two instructors was assisted by a graduate teaching assistant (20 hours per week). The two assistants were formerly undergraduate mentors, so they had considerable experience with the redesigned format. The teaching assistants were able to assume all of the duties formerly handled by the preceptor. Seminar attendance was excellent and students rated the seminar discussions as the part of the course that they liked best. The team introduced a new generation of radio-frequency response pads (clickers) that performed much better than the system used during the pilot. Mentors were trained to use the clickers to lead question and answer sessions in which the answers were open-ended. The feedback from the distribution of responses was used to initiate further discussion using different types of arguments: opinion, authority, reason, and data. The mentors adapted quickly to the technique, and reported high levels of success. During the spring 2005 pilot, performance in the redesign sections and five traditional sections were compared using an information retention test developed by the assessment committee of the department of psychology. Although the pilot sections had approximately twice the enrollment of the traditional sections, there was no difference in mean performance of the two groups, indicating that increasing the course size did not adversely affect the overall performance of students. There were significantly more A grades in the redesigned section than in the traditional section, yet there were the same number of B and C grades. At the same time, there were more F’s in the redesigned section than in the traditional section. A close look at the grade distribution showed that the students earning F’s tended not to participate in scheduled learning activities (e.g., they tended not to take the online mastery quizzes.) An analysis of students who tended not to participate (roughly 30% of the class) revealed that 50% of students who did not log on for a mastery quiz during the first week failed the course. Of the students who had not logged on to take a mastery quiz by the second week, only one passed, and no one passed if they had not logged on for a quiz by the third week. In response to these findings, EWU developed a procedure that they call “scaffolding” to motivate non-participating students. Students who did not log on to take a mastery quiz in the first week were required to attend a workshop during a class hour to determine the nature of their problem and to be sure that they understood how to use the electronic technology to take the mastery quizzes. Students who did not log on during the second week were referred to their academic advisors for counseling, and students who did not log on by the third week were told that they would receive a failing grade if they missed another week without taking a quiz. They were asked to sign a contract agreeing to participate in the course by completing all remaining mastery quizzes. Non-participating students were informed that breaking the contract by missing a fourth week would result in their failing the course. The team hopes that the escalating scaffolding process will provide the students with support, solutions, and motivation to succeed. The scaffolding procedures seem to have worked. The number of non-participating students has been reduced to approximately 3%. Raw performance scores (% questions answered correctly) for both of the fully-implemented, fully-enrolled fall 2005 redesigned sections was superior to that of both the spring 2005 pilot redesigned sections. Student ratings for the fully-implemented sections were also substantially higher than either of the two pilot sections. Comparisons of the fully-implemented sections during fall 2005 have not yet been compared with the traditional sections offered in winter or spring 2005. During winter 2006, EWU expanded the redesign to include a third section, which was fully enrolled (250 students). EWU will start the fourth and final redesigned section during the spring 2006 quarter and expects full enrollment for that section, meeting their enrollment goal of approximately 1000 students for the academic year. In the redesign, did students learn more, less or the same compared to the traditional format? Improved Learning To compare student learning in the traditional and redesigned formats, the department of psychology decided to administer a 22-item knowledge test to all students without telling the instructors in either the traditional or the redesigned sections what would be covered. The test was developed using standards published by the American Psychological Association-–i.e., recommendations about what psychology majors should know. A slight difference in mean performance levels on this examination favored the traditional format (12.96 for the traditional students vs. 12.19 for the redesign students). But this difference was very small in that only one correct answer separated the means of the traditional and redesigned courses. Student performance in both types of sections was low compared to the maximum attainable score. Since the instructors were not consulted and did not know what questions would be used on the test, the examination did not test what was actually taught in the course. Instructors omitted coverage of some topics in order to progress at an orderly pace in the ten-week quarters, and instructors with different backgrounds make different choices about what material to omit. Also, one course in introductory psychology is not likely to provide the knowledge background expected of psychology majors, so low scores were to be expected. Improved Retention The overall DFW rate for the redesigned section was slightly lower at 23.0% than the 26.1% DFW rate experienced by the traditional sections. This difference was not significant. Other Impacts on Students The pilot sections revealed that students who consistently chose not to take mastery quizzes invariably failed the course. During the full implementations, a step-wise scaffolding procedure was implemented to engage these non-participating students by alerting them to the problem, providing technical assistance, and imposing immediate consequences for non-participation. The procedure was implemented by the second week of the term and was effective in changing the non-participants’ behavior, but not in ways that increased their overall chances of success in the course. Some withdrew while others engaged in perfunctory quiz activity in order to avoid consequences, e.g., taking multiple quizzes in rapid succession without time for study. EWU concluded that non-participating students tend to have very serious motivational problems that are not easily addressed by advice, warnings, or penalties. The university plans to commit resources to understand and address this problem via an integrated effort involving the Teaching and Learning Center , Student Affairs, R2R faculty, teaching assistants, and peer mentors. Were costs reduced as planned? EWU followed its cost reduction plan with some modification. The redesign of introductory psychology reduced the cost of the course by 47%. The savings was achieved mainly by increasing section size from 100 to 250 students. EWU’s original goal was to reduce the number of sections from ten to four, a savings of 60%. This goal could not be met because it was necessary to continue to offer two traditional sections in order to fulfill department obligations to individual faculty members. Consequently, the number of sections offered was six (two redesigned sections in fall 2005, one redesigned section and one traditional section in both winter and spring 2006.) Offering six sections rather than four would have reduced the savings to 40%, all else being equal, but the redesigned sections served approximately 100 more students than projected in the original proposal. EWU planned for course enrollment to grow from 900 to 1000 students, but it actually grew to more than 1100 students. Some of this growth was expected as the result of demographic trends, but most can be attributed to the increased capacity of the redesigned course. Adjusting for the increase in enrollment, the final cost-per-student was reduced from $100 in the traditional course to $53 in the redesigned course. Pedagogical Improvement Techniques What techniques contributed most to improving the quality of student learning? Online quizzing. The principal learning advantage of the redesigned course was that it provided objective diagnostic information about the relationship between student preparation and learning. Specifically, online quiz technology permitted the instructor to identify non-participating students at high risk for failure by the third week of the academic term. There were significant positive correlations between scores on the online mastery quizzes and scores on tests taken in class. There was a complex relationship between the frequency of mastery quiz efforts and subsequent in-class test scores. Students who distributed their mastery quiz efforts over several days earned higher test scores than students who concentrated their mastery quiz efforts on a single day, almost always the last day. A few students who took mastery quizzes as many as 20 times per week failed to improve much, while a few others did very well after taking a mastery quiz only once or twice. In addition, the technology revealed that students who prepare for a final examination by taking online mastery quizzes require fewer quizzes to achieve the same level of performance as they did earlier in the academic term. Whether the relationship between mastery quiz activity and tests scores is causal or only reflects individual differences in ability or motivation, it is clear that the detailed information that quizzing can provide is very useful for diagnostic and advising purposes. Radio-frequency clicker technology. Radio frequency clickers facilitated feedback to the instructor about class preparation and comprehension. They also facilitated the training of peer mentors as discussion leaders. As might be expected, peer mentors reported different degrees of success as discussion leaders. The clicker technology motivated discussions among students attending the seminars, but the quality of the discussions was quite variable. In some cases it appeared that mentors were not adequately prepared to lead the discussion. A more general problem was that the questions were not well suited to the task of stimulating discussions at a conceptual level. Questions developed to assess factual knowledge tended to close discussions prematurely, and questions developed to assess student attitudes tended to open discussions too much. What is needed to take advantage of the technology are questions that have answers that lead to further discussion–-e.g., questions for which the answer depends on prevailing circumstances. The mentor can follow-up such questions by asking how the answer would change if the circumstances were different. In addition, peer mentors must be trained to use the specialized question set so that the potential of such questions is realized. The development of specialized discussion questions and training resources for mentors is presently pending funding by the Teaching and Learning Center . Cost Savings Techniques What techniques contributed most to reducing costs? Consolidating sections and increasing class size. Most of the cost savings achieved were attributable to increasing the class size in order to reduce the number of sections taught by faculty. Online mastery quizzes. Perception software by QuestionMark provided students with a means of taking mastery quizzes from on- or off-campus sites at any time of day. This software accessed a very large data base of questions provided by the textbook publisher. Students could take unique mastery quizzes on multiple occasions. The selection of questions as well as the order of the distracters was randomized for every quiz. The student’s score on the quiz and an accompanying coaching report was provided immediately after each quiz. The high administrative cost of administering, correcting, and posting the results of thousands of mastery quizzes was avoided. Course management software. Blackboard provided the main link between the students and course resources, including access to Perception. This link reduced the cost normally associated with hardcopy materials. Radio-frequency clicker technology. These devices reduced the cost of mentor training because they provided a simple and systematic method for relatively inexperienced mentors to interact with their students in discussion groups. It was possible to train mentors to use the clickers in meaningful discussions after a preliminary four-hour workshop followed up by weekly one-hour meetings, provided that the questions were carefully designed to open discussions at a deeper level. Implementation Issues What implementation issues were most important? Administrative support. It is difficult to make significant changes in the way things are done. The redesign implementation at EWU involved acquiring new technologies, making changes to the way courses were scheduled, disrupting the teaching expectancies of colleagues, and, in general, convincing people at all levels that there were good reasons for not doing things the way they had been done before. Training. Developing methods to efficiently train and coordinate more than 30 peer mentors involved a great deal of time and patience. Trial and error. A system with integrated components involving rooms, schedules, mentors, textbooks, software, and hardware take time and money to build. Some choices, carefully considered, still do not perform as expected. Sometimes it is necessary to reverse direction or make changes on the fly. One step at a time. An approach to assessment and problem-solving developed at EWU was to take it one step at a time. The team assessed one or two things each term and made one or two adjustments in the following term, then assessed the result and started all over again. It’s an on-going process. Will the redesign be sustained now that the R2R program has concluded? The psychology department voted unanimously to make the redesigned course the standard model for introductory psychology at EWU. One or two traditional sections may be taught each year, as needed, but more than 80 percent of the students who enroll in Introductory Psychology will take the redesigned course. Two additional faculty members in the department have expressed interest in teaching the course in the redesigned format. After two years of effort, the tradition of peer mentoring is now well established in the department and the university. Students are asking their instructors how to become involved as peer mentors. The EWU Teaching and Learning Center has taken a lead role in developing a training model for peer mentors, and the psychology department is offering experienced mentors teaching assistantships for R2R courses when they are accepted into the graduate program. Will you apply the redesign methodology to other courses and programs on campus? The administration and the board of trustees have taken notice of the impact of the redesigned course and are encouraging deans in other colleges to adopt some form of “R2R” for their own large-enrollment courses. They are particularly interested in whether the technique of identifying non-participating students during the freshman year might impact the university’s efforts to retain students. Responding to that specific need, the Office of Student Affairs has committed personnel and funding to provide leadership to students identified as non-participants. The EWU Teaching and Learning Center is sponsoring efforts to develop redesign models for the departments of business, mathematics and biology. Pilot courses are underway, and internally funded grants are being awarded to faculty for developing redesign variants for their own courses.
|
 |
Quick Links: |
 |
||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|