 |
 |
|||||||||||
 |
||||||||||||
 |
The Roadmap to Redesign (R2R) Improving Learning and Reducing Costs: By Carol A. Twigg In September 2006, the National Center for Academic Transformation (NCAT) completed a three-year, FIPSE-funded program, The Roadmap to Redesign (R2R). The purpose of this institutional grant program was to encourage colleges and universities to redesign their instructional approaches using technology to achieve cost savings as well as quality enhancements. Redesign projects focused on large-enrollment, introductory courses, which have the potential of impacting significant student numbers and generating substantial cost savings. Building on the successes achieved in NCAT’s Program in Course Redesign (PCR), R2R sought to simplify the course redesign process and make it affordable within existing institutional resources. FIPSE funding was used to develop and provide a wide variety of resources that supported new institutions as they redesigned large-enrollment, introductory courses. R2R identified 20 institutions that wanted to redesign courses in four disciplines—precalculus mathematics, psychology, Spanish and statistics—in order to test a new model. The new model partnered experienced, successful institutions with new institutions and took advantage of best practices, virtual repositories of research-based learning materials and a streamlined redesign methodology. The redesign projects began in September 2004 and concluded in August 2006. (Detailed descriptions of the individual redesign projects and the outcomes each achieved can be found at http://www.thencat.org/R2R/R2R_RedProj.htm.) At the start, we said we could guarantee that if new institutions followed our advice—derived from the successes achieved in the Program in Course Redesign—they would improve student learning, increase retention and reduce instructional costs. And that is exactly what happened. Not all institutions followed our advice. Eight of the original 20 institutions did not complete the program. Here are the results for the 12 institutions that completed the program. Improved Learning Ten of the 12 R2R projects reported improved learning outcomes; two reported learning equivalent to the traditional format. Among the findings are the following: At Chattanooga State Technical Community College, students enrolled in the redesigned sections showed significant improvements in content knowledge as measured by pre- and post-tests as well as significantly better absolute post-test performances when compared to traditional sections. Students in the traditional course had a pre-test mean of 21 and a post-test mean of 26. Students in the redesigned course had a pre-test mean of 25 and a post-test mean of 36. At Georgia State University, students in the redesigned College Algebra pilot sections performed significantly better than students in traditional sections on five common examination questions (a mean score of 3.04 vs. 2.66 for traditional students.) In fall 2006 at Louisiana State University, the 1742 students in the redesigned College Algebra course had the lowest ever drop rate of 6% and an A-B-C rate of 75%. The final exam median of 78% was the highest ever achieved. In the First-Year Spanish full implementation at Texas Tech, redesign students performed significantly better than traditional students in vocabulary (a mean score of 81.04 vs. 74.18 for traditional students.) In the Second-Year Spanish pilot, redesign students performed significantly better than traditional students on assessments of vocabulary, grammar, reading composition and culture (74.04 vs. 60.52) and on an assessment of speaking skills (78.56 vs. 72.91). At the University of Alabama, a comparison of common final exam scores between the traditional Spanish courses and the redesigned courses indicated that mean final exam scores improved in all three of the redesigned courses: 66.81 vs. 65.54 in Spanish 101; 72.32 vs. 67.86 in Spanish 103; and 79.72 vs. 71.42 in Spanish 102. There was a significant increase in the number of students receiving a final grade of A in the redesigned Spanish 101 (40% vs. less than 12%) and in Spanish 102 (41% vs. 19%.) At the University of Missouri-Saint Louis, students learned more in the redesigned College Algebra course than in the traditional course as indicated by improved performance on common comprehensive final exams. There were more As and Bs and fewer Ds and Fs. Combined results for two academic years indicate that 56.6% of students in the redesigned sections achieved in the top two of six score ranges, while only 31.5% of students in the traditional sections did so. The fall 2005 pilot of Precalculus at the University of North Carolina-Chapel Hill compared student performance across three different formats: the traditional course format, a hybrid format and the redesigned format. Student gains in test performance were slightly greater with the redesigned and hybrid sections (average gain of 50.4% ± 1.4 vs. 47.6% ± 2.0, p=0.28.) At the University of North Carolina-Greensboro, the average score on common, comprehensive final examinations in Precalculus I increased significantly from 58.2 for the traditional course to 75.5. In Precalculus II, the final exam average increased from 65 for the traditional course to 69.6. The average score on common, comprehensive final examinations in Statistics increased from 63.5 for the traditional course to 68.5 for full implementation of the redesigned course. The Wayne State University team compared eight semesters of data from the traditional Beginning Algebra course to four semesters of data from the redesign using uniformly graded final examinations. The average score increased from 64.6% to 68.6%. The team also looked at success in the follow-on math course. The overall pass rate in the next course was slightly higher for those coming out of the redesigned course (44.0% vs. 42.9%). However, comparable final exam scores seem to produce better results for students coming from the redesign. For example, of those students who scored 90 -100% on the final exam, 86.7% of the redesign students passed the next course, while only 78.4% of the traditional students did so. Improved Retention Nine of the 12 R2R projects showed improvement in course completion rates; one reported no change and two produced invalid data. Among the findings are the following: In fall 2005, the Business Statistics course at Calhoun Community College enjoyed an 82% completion rate compared with a 65% rate in the traditional course. The overall course grade average was B (83%), and no D or F grades were given. The drop-failure-withdrawal (DFW) rate for the redesigned Introductory Psychology course at Eastern Washington University was 23.0% compared to 26.1% in the traditional sections. The DFW rate for the redesigned College Algebra course at Georgia State University was 12.2% in the fall 2004 pilot compared to 27.6% in the fall 2003 traditional sections. At Texas Tech, the number of Ws in the Spanish redesign full implementation dropped compared to a two-semester average of the traditional format. The number of Ds and Fs combined dropped despite more stringent grading standards in the redesign. The percentage of students who successfully completed (obtaining a final grade of C or better) two of the three redesigned Spanish courses at the University of Alabama increased compared to those in the traditional formats (in Spanish 101, 82% in the redesign vs. 75% in the traditional; in Spanish 102, 92% in the redesigned sections vs. 90% in the traditional sections.) At the University of Missouri-Saint Louis, retention rates improved in the redesigned College Algebra course relative to the traditional course. The successful completion rate (grade of C or better) for students increased from 63.3% in the traditional course to 78.4% in the redesign. At the University of North Carolina-Greensboro, student retention improved substantially in all redesigned courses. In College Algebra, the DFW rate dropped from 62% in the traditional course to 49% in the redesigned course, in Precalculus I from 77% to 38%, and in Precalculus II from 60% to 41%. In Statistics, the DFW rate dropped from 70% in the traditional course to 60% in the redesigned course. At Wayne State University, the student success rate in Beginning Algebra increased from 35.5% in the traditional course to 42.8% in the redesigned course. Approximately two-thirds of those who took the final exam passed it compared to only about half in the traditional course. Cost Savings With regard to cost savings, the R2R results are an unqualified success. All 12 of the completed R2R projects reduced their costs. Some saved more; others saved less; but all produced cost savings. Three universities saved more than they had planned. Georgia State University originally estimated that the cost-per-student in College Algebra would decrease from $96 to $87, a 9% savings. The actual cost-per-student was reduced to $80, producing a savings of about $24,000, a reduction of 17%. Louisiana State University originally estimated that the cost-per-student in College Algebra would decrease from $121 to $95, a 21% savings. Changes made in the full implementation further reduced the cost-per-student to about $78, an overall 36% savings. The University of North Carolina-Greensboro carried out its planned cost reduction strategy in Statistics and reduced the cost-per-student from $102 to $82. In addition, overall faculty presentation and test-proctoring hours were reduced by an additional 350 hours above what was initially planned, thus further reducing the workload of instructors. Two institutions saved less than they had planned. Calhoun Community College planned to increase the number of students in Statistics from 281 to 480 and the number of sections from 14 to 20. Final enrollment at the time of full implementation was 391, and the number of sections increased to 17. Calhoun also added an unanticipated half-time lab assistant with a background in statistics to the final redesign. With these changes, the cost- per-student declined from $170 to $144, a 15% decrease rather than the planned 35% decrease. Eastern Washington University’s original goal was to reduce the number of sections in Introductory Psychology from 10 to four, a savings of 60%. This goal could not be met because departmental obligations to individual faculty members made it necessary to offer two traditional sections. Approximately 100 more students than projected enrolled in the redesigned course. After adjusting for the increase in enrollment, the final cost-per-student was reduced from $100 in the traditional course to $53 in the redesigned course, a 47% reduction. Sustainability We asked the R2R institutions, will the redesign be sustained now that the R2R program has concluded? Here are their answers: At Calhoun Community College, the faculty members who teach the redesigned course have seen its benefits and are committed to making further improvements to their initial effort. Additionally, the second-semester Business Statistics course is scheduled to begin the redesign process in the 2006-07 school year. The team at Chattanooga State says, “We will be continuing our redesigned course and look forward to adding two new enthusiastic faculty members to our team.” Although East Carolina’s redesign project was negatively impacted by technology and infrastructure challenges as well as instructor family and health issues, the team plans to conduct a new redesign pilot in fall 2006 that will profit from the lessons learned during the past two years and then move to full implementation. At Eastern Washington, the psychology department voted unanimously to make the redesigned course the standard model. One or two traditional sections may be taught each year as needed, but more than 80 percent of students will take the redesigned version. At Georgia State, the sustainability of the redesign is not in jeopardy because of its success in increasing the number of As, Bs and Cs and reducing the DWF rate. LSU says, “There is absolutely no doubt that the redesign will be sustained. The university has already invested substantial resources in the existing 116-seat learning lab, and a second 122-seat learning lab will be operational in fall 2006.” At Seton Hall, the redesign will be continued since it has improved student learning and reduced the costs of the program. The team plans to do some fine-tuning to allow for further customization of the material. At Texas Tech, the course redesign is intended to be long-term. As long as the department retains its present chair, there is no likelihood that the redesign will be abandoned or substantially altered. The University of Alabama team is confident of the sustainability of the program. Several minor changes are planned for next year, which are intended to reduce even further the amount of time instructors spend on grading and improve student attitudes toward the new format. At UM-SL, the campus and the faculty are dedicated to continuing the successful redesign and expanding the model to other courses. The UNC-Chapel Hill math department is satisfied with the overall results of the initial implementation and is likely to continue it with some modifications. UNC-Greensboro will continue to explore the best format for the redesigned courses. In addition to offering the original replacement model, UNCG will also offer a fully online version and a supplementary version. During the 2006-2007 academic year, all three models will be compared. At Wayne State, the redesign has produced both improved learning and cost savings. Consequently, the university has made a commitment to build a new 150-station computer lab that will allow the team to expand the redesign to additional courses and to accommodate increased enrollments. We also asked the R2R teams, Will you apply the redesign methodology to other courses and programs on campus? Here again are their answers: Some are exploring options. Based on the encouraging results from the full implementation of the redesign in Spanish, the University of Alabama is currently studying the possibility of redesigning introductory French courses as well. Others report definite plans in addition to exploring other options. For example, at Chattanooga State, elements of the redesign have been included in Psychology of Personal Adjustment and Abnormal Psychology. Sociology is considering redesigning Introduction to Sociology Others have definite plans for expanding the redesign methodology to other courses. At Georgia State, the redesign will expand to the Calculus course sequence. At LSU, Precalculus will also be taught using the redesigned format, and plans are being made to redesign Trigonometry as well. The Texas Tech team has already piloted the redesign in second-year Spanish. In fall 2006 they will do a full implementation followed by other Spanish courses in spring 2007. During summer 2006, the German department converted their workbooks to online and adopted a supplemental model version for the fall semester. Texas Tech is encouraging other languages to follow suit. Based on the initial success of UM-SL’s redesign of College Algebra, the supplemental model was introduced in the Trigonometry course. Various faculty members are offering pilot sections for Basic Calculus, the Calculus series, and Statistics. If the results are positive, the changes will be made to all the sections of these courses. Based on the positive results achieved thus far, there is no reason not to continue this winning formula. By fall 2006, Wayne State will have redesigned three courses: the original Beginning Algebra, Intermediate Algebra, which was added in fall 2005, and Finite Math for the Social and Management Sciences. The team will also add a homework and quiz component to some other math courses that will not be fully redesigned. Finally, other institutions plan an even more elaborate expansion. Calhoun Community College has adopted NCAT’s course redesign process in its initiative to redesign 38 of its large enrollment courses over a five-year period. Redesign work is now underway in ten other courses. The Eastern Washington Teaching and Learning Center is sponsoring efforts to develop redesign models for the departments of business, mathematics and biology. Pilot courses are underway and internally funded grants are being awarded to faculty for developing redesign variants for their own courses. Lessons Learned R2R’s basic objective was to demonstrate the feasibility of redesigning large-enrollment introductory courses on a wider scale using a set of redesign tools and methods developed in the Program in Course Redesign (PCR). R2R lacked the infusion of $200,000 in external funds that each PCR course redesign project enjoyed. R2R participating campuses had to implement their redesigns using their own resources with support from NCAT and PCR participants. The essential ingredients of R2R were:
What Led to Success? When R2R began, we said that we could guarantee that new institutions could improve student learning, increase retention and reduce instructional costs if they followed our advice—derived from the successes achieved in the PCR. Did the R2R redesigns conform to the principles of effective redesign previously established by the PCR—that is, did participating campuses “follow the rules”? All 12 of the redesigns that fully implemented their redesigns used one of NCAT’s six course redesign models as did the eight that eventually withdrew or were dropped from the program. All of the redesigns incorporated technology and reduced lecture time. Most incorporated techniques that encouraged active learning and/or rested on mastery learning. Most improved student assessment systems, incorporated student tutorials and included on-demand help. The majority incorporated proactive intervention techniques designed to keep students on track such as assigning and grading practice problems or homework. Finally, most adopted learning resources that were either commercially available or had been developed through the PCR. R2R reaffirmed the lessons learned in the PCR. The pedagogical techniques (active learning, online tutorials, continuous assessment and feedback and on-demand support) that led to increased student learning in the PCR also led to increased student learning in R2R. Similarly, the cost reduction techniques (online tutorials, automated assessment, course management systems, shared resources and staffing substitutions) that led to reduced instructional costs in the PCR also led to reduced instructional costs in R2R. This second “proof of concept” in course redesign validated what was previously learned in the PCR. Learning from Experience While we now know what works well—and what we need to do in order to achieve the goals of course redesign—we also know that getting there means a lot of hard work. Course redesign is not a silver bullet. Putting the good ideas derived from the PCR into practice represents a real commitment on the part of participating institutions, especially when implementation challenges rear their ugly heads. At the outset, NCAT tried to provide an “early alert” to the R2R institutions by identifying the five most important implementation issues encountered by the PCR projects. NCAT encouraged the new institutions to pay special attention to how they would:
Indeed, once the redesign implementation phase was launched, participating campuses encountered implementation challenges corresponding to those to which they had been alerted. Preparing Students. About two thirds of R2R participants indicated that they experienced some difficulties in preparing students for the new format. Most common here were negative student reactions to the perception that the class was “an online class” that they did not think they had signed up for or that it “had no teacher.” These challenges were addressed by more up-front engagement with advisors to explain what the course would be like and the development of written materials and orientation sessions that explained the new format. Student complaints also diminished as they gained more experience with the new format and recognized that it was here to stay. Preparing the Campus. A majority of participants also reported encountering challenges associated with preparing others on campus for the redesigned format. Most of these were related to advising, where advisors did not provide correct information to students or simply misunderstood what the course was about. Project leaders learned that they needed to constantly and consciously “market” the redesign to key campus constituencies who knew little about the new course and how it differed from more traditional offerings. By the point of full implementation, these problems had been largely overcome. Training. About half of final participants reported challenges in training instructors, graduate teaching assistants and undergraduate peer tutors. As one participant observed, “We did not adequately plan for this and thought we could do it on the ‘cheap’…we have now changed course and will conduct our first paid training session for part-time faculty tomorrow.” Most participants reported similar up-front orientation sessions for all personnel in advance of the first course meeting as well as frequent updates throughout the term. By the point of full implementation, these problems had been addressed through greater structure and training. Technology Infrastructure. Sixty percent of final participants reported encountering challenges in ensuring an adequate technological infrastructure. Most of these occurred during early implementation and concerned adequate staffing or late-arriving equipment. Only one of the participants reported ongoing problems with technological infrastructure due to lack of support on the part of the CIO. Faculty Consensus. About two thirds of participants reported challenges in the area of achieving faculty consensus within the department about the redesign. Some of this was attributed to leadership issues—for example, interim department chairs who were reluctant to press resisting faculty. Others stressed the need for strong leadership and administrative support to overcome these challenges. Some project leaders thought they had solved the problem of faculty buy-in at the outset, but were surprised to find that they had not communicated as effectively as they had thought. This underlines the importance of constant communication to check signals and maintain momentum. The fact that all of the final redesigns will be continued testifies to the fact that these challenges were eventually overcome. Failing to achieve faculty consensus was the most important reason why eight of the original 20 institutions did not complete their redesign projects. Team leaders thought they had their colleagues’ support, but when the redesign got underway, they discovered that the opposition was stronger than anticipated. Building Institutional Commitment. About half of the final participants reported progress in building institutional commitment to redesign outside their home department. Participants frequently cited leadership and administrative support as factors in sustaining and expanding interest in redesign. In one case, redesign is being encouraged by system-level leadership and another noted support by trustees as a factor. Like building acceptance within the department, however, broadening institutional commitment required continuing attention and support even under favorable circumstances. R2R tools like the Five Critical Implementation Issues were designed to reduce implementation challenges, and many respondents emphasized that it was helpful to learn that others had encountered a particular problem and how they had overcome it. But sentiment was equally clear that many of these problems could not have been foreseen and that every campus may need to go through the experience of redesign on its own. As one participant put it, “I’m not sure you can change human nature…I can read the Implementation Issues, but I need to have the experience before I really understand the issue.” Another noted, “I do not see a way to avoid implementation issues…it is more a matter of one set of problems versus another set.” Participants stressed that foresight was helpful, but that there were no simple answers to complex organizational challenges. As one of them noted, “I think the Implementation Issues aren’t about the principles of course redesign as much as [they are about] the institutional context.” Providing materials and shared experiences were helpful to R2R participants in knowing that particular implementation issues would be likely to arise and in assuring them that they could work through them. But they could not provide them with ready-made “solutions.” Asked to sum up the implementation process as a whole, one participant emphasized, “It just takes hard work and time.” Although R2R participants valued examples, advice and interaction with experienced practitioners, they accomplished a great deal of the real work of redesign on their own. Based on their experience with the PCR, this was foreseen by NCAT staff. The NCAT course redesign methodology is not a silver bullet that can be quickly put in place on any campus. Even though most institutions thought they knew what they were getting into, they learned by experience how challenging it was to implement their plans. Building on What We’ve Learned R2R has provided a number of important lessons to inform next steps in large-scale course redesign—lessons that are already being incorporated into NCAT’s ongoing work. Among the most prominent lessons learned were the following: 1. It is feasible to undertake a successful course redesign without a substantial infusion of external resources. Testing this proposition, of course, was the fundamental premise of R2R: to determine if institutions could successfully complete redesigns without the $200,000 per-course allocation of grant funds provided by the PCR. This proposition was clearly sustained as 12 of 20 projects chosen to participate in the project did complete a successful redesign using their own funds (with one more still in process). Interestingly, those that persevered frequently used the language of external “grant” support when referring to the project on and off-campus, despite the fact that the “grant resources” provided were in the form of tools, convening, and consulting instead of money. 2. It is possible to streamline redesign to achieve more rapid implementation and adoption as long as one “follows the underlying rules.” R2R also sought to demonstrate that proven redesign models and implementation approaches could enable shortened timelines. R2R’s timeline was based on a faster cycle. In contrast to the PCR, the planning period was reduced from nine months to two months and the implementation period was reduced from two years to 15 months. Again, this proof of concept was fully demonstrated by the 12 redesigns that were completed within the project timeline. Above all, success required course teams to “follow the underlying rules” of the project with respect to proven redesign approaches and readily-available tools and resources. 3. Strong local leadership is critical to staying the course. Success in course redesign requires strong local project leadership—either at the department or at a higher administrative level. There was no one model of successful leadership in R2R. Some redesigns were managed collegially, others depended upon a core group of tenacious faculty, and still others were implemented in a “top-down” fashion by administration. Functions of top leadership mentioned especially were the willingness to stand up publicly on many occasions to talk about the project and its benefits and backing up the project team when they ran into trouble by providing resources or fixing administrative problems. But above all, local leaders had to ensure that the team stuck to the basic redesign plan. The Importance of Collaboration R2R participants valued collaborative work throughout the project. Those attending the initial workshop overwhelmingly felt that the opportunity to see real examples and interact with PCR “veterans,” as well as with project staff, were most valuable. Subsequent workshops were also highly praised for the interaction that took place. Such interaction not only disseminated ideas and techniques, but also built solidarity and mutual support through the knowledge that others were encountering (and overcoming) similar obstacles on their own campuses. Participants also valued collaboration within their own campus teams. The R2R redesign methodology emphasized the use of a team to implement the project, and all institutional respondents to the final project survey indicated that they had taken such an approach. Eight of the respondents gave the team approach the highest available rating, and all but one rated the team approach in the top two response categories. Yet the formally-established Academic Practices—the most formal collaborative feature of the R2R design—did not work as expected. Course teams instead tended to seek advice directly from NCAT staff rather than from the Academic Practices despite their initial enthusiasm for the Practice concept. Partly this may have been because doing so was easier and more comfortable, and it did give them the information they needed quickly and effectively. Many course teams preferred to “figure out things for themselves,” without consulting experienced partners, even though this sometimes meant “reinventing the wheel.” This happened repeatedly in the PCR as well. Going through the process of arriving at solutions to commonly-encountered problems—though not always efficient—may be necessary to achieving progress. Implementing solutions to problems on one’s own terms may help build longer-lasting commitment among those involved. R2R demonstrates that some kind of collaborative structure is beneficial to sustain multi-institutional approaches to redesign, but that the precise form (or forms) that it should take has not yet been found. This suggests a flexible approach for future efforts that combines face-to-face gatherings with on-demand mechanisms for exchange and collaboration electronically as well as a clear requirement for a larger pool of advisers, perhaps made available in a less formal and more flexible fashion than the Academic Practices. Conclusion The Roadmap to Redesign has achieved its goals. The pioneering institutions in the Program in Course Redesign established replicable models for those institutions that want to use technology to improve student learning while reducing instructional costs. Building upon that valuable experience, the R2R projects have adopted those models and achieved similar and, in some cases, even stronger results. Copyright 2007 The National Center for Academic Transformation
|
 |
Quick Links: |
 |
||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|