 |
 |
|||||||||||
 |
||||||||||||
 |
Colleagues Committed to Redesign (C2R) Improving Learning and Reducing Costs: Program Outcomes and Lessons Learned By the National Center for Higher Education Management Systems (NCHEMS) This report presents results of an external evaluation of a three-and-a-half-year FIPSE-funded project undertaken by the National Center for Academic Transformation (NCAT). The basic objective of the project was to promote widespread adoption of redesigned high-enrollment introductory college courses using tools and methods based on two earlier NCAT redesign projects—the Program in Course Redesign (PCR) funded by the Pew Charitable Trusts and the Roadmap to Redesign (R2R) project funded by FIPSE. Both prior projects were successful in a) demonstrating the feasibility of redesigning such courses to yield improved student learning at reduced cost and, b) creating a wide array of standard planning and implementation tools that could be effectively used by campus teams to undertake such redesigns. PROJECT OUTCOMES The Colleagues Committed to Redesign (C2R) program involved three rounds of course redesigns undertaken by campus teams in a particular course in a particular discipline. A total of 40 institutions participated in this component, 13 of which withdrew or were dropped from the project. Two institutions redesigned two separate courses, yielding a total of 28 redesigned courses. The most frequent subjects for redesign were in mathematics, statistics, and computer science with a total of 12 courses, followed by humanities with six courses, natural sciences with six courses, social sciences with two courses and professional studies with three courses. Experienced advisors were available to the C2R participants in the form of Redesign Scholars in multiple disciplines (psychology, mathematics, statistics, biology, physical science, and foreign languages), who were faculty and administrators experienced in course redesign in those disciplines through PCR and R2R. C2R was intended to disseminate more broadly the redesign methodology established by PCR and R2R. Like R2R, it lacked the infusion of $200,000 in external funds that each PCR course redesign project enjoyed, so campuses had to implement their redesigns using their own resources. C2R redesign participants also only had to carry the redesign through a pilot stage, with the decision to go to full implementation beyond the parameters of the project. The only grant resources provided by the project to participating institutions were travel support to attend project meetings and consultant support. Essential ingredients of C2R, therefore, were:
The intended logic of the project was that each C2R campus would create a redesign using the tools and advice provided, and would then pilot test this redesign. Master evaluation questions, given this logic, are:
1. Success in Soliciting Engagement in Course Redesign The C2R project built on a strong base of knowledge of redesign obtained through the previous PCR and R2R experiences and helped to increase awareness of the potential of redesign to increase learning and decrease cost. An indicator of success in soliciting engagement is the total number of institutions undertaking redesigns. C2R planned to accommodate a total of 60 redesigns, 20 for each of the three project years. By the project’s conclusion, only about half that number (29) had actually been implemented. There were a number of understandable reasons for this. First, unlike PCR, participating institutions received little monetary support from the project to underwrite their efforts. This contrasted with PCR, where participating institutions received $200,000 to redesign a course. As one campus representative summarized in evaluating their project, “trying to do a redesign without funding for faculty time was incredibly difficult.” Another reason for the shortfall in final numbers was not attributable to lack of initial interest but to lack of follow through. A number of institutions, for example, were not able to agree which course to redesign or how to do so or could not adequately document their costs. These were, of course, basic requisites of participation, and they were dropped from the project. 2. Effectiveness of Redesign Scholars The Redesign Scholar model has proven to be an unqualified success. Institutions participating in the three waves of redesign in C2R were given no direct monetary support to implement their projects. But they were given the opportunity to work with one or more Redesign Scholars at a low negotiated rate. All but one of the institutions responding to surveys on the implementation and effectiveness of their redesigns conducted a year after completion reported that they had taken advantage of the opportunity on one or more occasions. Of these, three reported that there has been ongoing contact and three more that the contact consisted of a single consultation, with the balance reporting several contacts or conversations during their period of engagement. And all but two report that they definitely will, or perhaps will, continue this contact beyond the completion of their project. Comments on the kinds of advice provided by Redesign Scholars were frequent and covered the gamut of implementation and substantive issues. The following comments provided by redesign leaders in each of the three rounds were illustrative:
Other topics addressed by Redesign Scholars mentioned as being especially helpful were a) the effective use of course preceptors and undergraduate teaching assistants, b) the need to constantly monitor the “vital statistics” of the redesign such as costs and student attendance, and c) the need to “advertise” the planned changes (and the reasons for making them) in advance to head off premature complaints by students and other faculty that things were changing. 3. Effectiveness of Redesign Resources Like R2R, the C2R project provided participants with multiple tools to assist them in undertaking course redesigns through publications and illustrations on the NCAT website, but also supplemented these resources through presentations and exhibits at the four annual Redesign Alliance conferences and associated Disciplinary Institutes. Data from project leaders in the institutions participating in the three rounds of sponsored redesigns under C2R reinforce these conclusions. All of the 22 respondents to the surveys administered to project leaders reported that they used the planning resources, and all of them reported them to be either “very useful” (14 respondents) or “somewhat useful” (8 respondents). When asked which they found most useful, the majority reported that the practical experiences of other successful institutions as portrayed in the case studies on the NCAT website or presented at the annual conferences and institutes helped them the most. Among the specific tools mentioned as being most helpful were the Course Planning Tool and the Five Principles of Successful Course Redesign. Participating institutions collectively covered the range of the six NCAT redesign models. Two used the Buffet model, six the Emporium model, one the Fully Online model, one the Linked Workshop model, twelve the Replacement model and seven the Supplemental model. Inspection of project final reports support the conclusion that these redesign efforts were implemented in a manner consistent with NCAT’s expectations about how to go about the redesign process. This is not surprising, because the 13 institutions of the original starting group of 40 that did not complete their redesigns withdrew or were removed from the project because they did not follow the rules. Participating institutions did encounter some implementation issues, but with ongoing assistance from both NCAT and the Redesign Scholars most were resolved. Institutions identified many of these issues on the e-surveys administered to them after the conclusion of their redesign, with the following results:
Many implementation issues were about technology. Answers to the open-ended questions posed in the e-survey indicated that most of these involved software glitches and problems with vendors. Several participants noted that “technical problems with the instructional platform for the courses” led to “students losing work or having to re-take tests.” Another noted that “working out the software bug was an issue” and another participant reported, “the textbook software package that we are using has shortcomings that are affecting students perception of the course.” Issues surrounding implementation that focused on achieving initial and ongoing faculty consensus within the department about the redesign were, for the most part, about communication. Some faculty resistance to the redesign was attributed to attitudes toward change in general. More pointedly, some noted that some faculty members were concerned about being replaced with technology. As one respondent reported, “There are faculty members who believe that using a course redesign could lead to dismissal of faculty members.” Experience with these and previous redesigns suggest that faculty issues such as these can be partially resolved through better communication and more time. Many respondents reported, and interviews with project teams during the annual Redesign conferences support the efficacy of ongoing communication throughout the implementation process and directly addressing the fear of potential faculty job loss. As faculty become more familiar with course redesign, moreover, they begin to see the benefits to the students and the department. 4. Improved Learning at Reduced Cost Costs were fully documented by project course teams using NCAT’s standard Course Planning Tool, which is based on the principles of activity-based costing. All 29 completed redesigns anticipated cost savings after full implementation, and all reported that they were on track to realize these savings. Student learning outcomes were investigated through direct assessments that compared student performance in redesign sections with performance on equivalent assessments in traditionally-configured sections. In most cases, the assessments used were common examinations or examination items. Fourteen of the 26 redesigns for which data are available completed redesigns yielding higher levels of student learning than the same course offered in a traditional format after full implementation of the redesign (11 of the 14 improvements were statistically significant). The rest of the redesigns yielded levels of learning that were not significantly different from traditional sections. Thirteen of the 26 redesigns for which data are available resulted in improved course completion rates in the pilot (seven of these were statistically significant). Seven of the redesigns experienced course completion rates that were no different from their traditionally-configured counterparts, while six experienced a lower rate (four of these were statistically significant). In assessing these results, however, it is important to recall that these were all pilot implementations in which not all the bugs had been worked out. Both R2R and PCR had several redesigns that did not achieve planned learning or completion goals during their pilots because of implementation difficulties, but achieved these goals later during full implementation. 5. Institutionalization and Generalization of the Redesigns Despite the fact that not all of these pilots demonstrated improved learning and course completion, the fact that virtually all of them did as well with respect to learning outcomes as the traditional mode of delivery at what were expected to be substantially reduced costs, convinced the majority of institutions to sustain the effort. All projects for which data were available reported that they intended to continue, most with full implementation and a few with an expanded range of pilots. And the one project uncertain about next steps with respect to implementation did not cite deficiencies in pilot performance as the reason for delay. Instead, reasons given were associated with the physical resources needed to implement the redesign at scale—in this case, the physical classroom space needed to support an Emporium. Even here, the project leader expected the difficulties to be worked out within a year. Adding to these favorable results with respect to institutionalization, seven projects reported that the redesign concepts piloted in their projects had yielded concrete efforts to generalize the redesign approach to other courses. Most of these were related courses in the same department, but some were further generalizations to other disciplines. One community college demonstrated good internalization of the NCAT approach by reporting that redesign methods would not be applied on an incremental basis, but would be applied systematically to the institution’s highest enrollment courses. Many of those institutions planning to extend redesign to other courses reported substantial attempts to engage the wider campus community through presentations, workshops, and seminars presented to their disciplinary colleagues or institution-wide. One created a website on redesign promoted to all faculty. Another held a summer institute on redesign open to faculty teams drawn from departments throughout the university. Taken together, C2R results on institutionalization are a bit stronger than those experienced in a prior FIPSE-funded program, R2R, and far stronger with regard to generalization. This—together with the fact that current fiscal conditions suggest greater institutional interest in generating cost savings—suggests that a take-off point for the NCAT redesign approach has been reached at many campuses. LESSONS LEARNED The FIPSE project provided a number of important lessons to further inform large-course redesign—lessons that continue to be used by NCAT in its subsequent project, Changing the Equation, funded by the Bill & Melinda Gates Foundation, in developmental mathematics. Among these lessons were the following: NCAT’s redesign methodology is mature and reliable in achieving results. More than ten years of experience with NCAT’s basic methodology, proven in scores of courses in different disciplines in different institutional settings, has resulted in an approach to redesign that can reliably produce positive academic outcomes at reduced costs. NCAT’s leadership reported, “at this point, we can say with certainty what works in redesigning mathematics and, to a certain extent, in foreign languages.” Steadily increasing numbers of success stories in psychology, humanities and the sciences are adding confidence to the conclusion that the redesign methods will work for any faculty team that “follows the rules” faithfully and that is located in a campus setting where the basic readiness criteria are met. This does not mean that they will pull off a successful redesign without difficulties. C2R campuses experienced some of the same mixed pilot results as their colleagues in the two preceding projects. And, like those who went before, most of them had to learn lessons for themselves rather than simply trusting the approach to work as directed. But the results of C2R emphasize the soundness of the basic NCAT approach and suggest that any willing campus can apply it to an appropriate course with confidence. It is possible to streamline application of the basic redesign methodology even further. NCAT’s R2R project demonstrated the feasibility of codifying the application of redesign through the use of common tools and templates like the six redesign models, the institutional readiness criteria, and the Course Planning Tool. These tools also proved effective in C2R. But experience in the later project also demonstrated that it was possible to simplify things even further. For example, changes have been made in the way costs and savings are estimated from the detailed activity-based cost analyses contained in the Course Planning Tool to a more aggregated approach to documenting course costs in NCAT’s Changing the Equation program. This has considerably alleviated the work involved in planning for redesign. Similarly, there has been a steady evolution of knowledge about which redesign models work best for which disciplines. For example, the Emporium Model is the most common approach in mathematics, as illustrated in the current project, Changing the Equation. Similarly, the Replacement Model appears to work best in foreign language instruction. These further refinements to the basic planning methodology, plus more practice in using them, meant further gains in the efficiency with which redesigns can be accomplished. The supporting infrastructure for redesign put in place through the FIPSE project has been extremely effective. In R2R and PCR, the center of gravity of project activity was at the institutional level. Each redesign was designed and implemented more or less one at a time, albeit with considerable and growing help from project personnel and a steadily expanding array of redesign tools. Indeed, one lesson of R2R, noted in the external evaluation report, was the need for concrete mechanisms to support inter-project collaboration. Through the annual redesign conferences, the disciplinary institutes, and an expanded network of Redesign Scholars, the project considerably increased the infrastructure available to support redesign at the participating campuses and expanded access to all institutions. Testimony of participants in the Redesign Alliance conferences and C2R disciplinary institutes showed these events to be an unqualified success, at least in terms of perceived outcomes. The fact that so many attendees reported that they have redesigns planned or under way (though we have no information about the quality of these efforts) also suggests that they have been effective in stimulating action. The other infrastructure component of C2R, the Redesign Scholars, has also proven a major success. This project expanded the network of experienced faculty prepared to assist campuses beyond R2R, and the Scholars gained considerably more experience in functioning in partnership with NCAT staff. As a result, they were able to operate more independently in providing assistance than in R2R. The fact that access to the Scholars was built into the redesign process undertaken by C2R campuses from the outset meant that such contact was far more likely to occur. Reported satisfaction with the advice provided by Redesign Scholars by participating campuses, as well as the successful outcomes of their projects, provides solid evidence that these individuals can meaningfully supplement the activities of NCAT staff. The latter report themselves “pretty happy with these results,” and plan to more than double the size of the pool of Redesign Scholars in the coming years. These infrastructure developments signal the transition of NCAT from a demonstration mode—with the objective to show how redesign can be accomplished successfully and the gains in learning and cost savings that can be realized—to a broader developmental role—with the objective to provide resources to a wider range of institutions that are undertaking redesigns outside the confines of a formal project. Institutions are embracing redesign more readily as they face budget shortfalls. Pilot results for C2R participants were no better than those achieved by participants in R2R and, in some cases, they were worse. Yet virtually all the 29 projects plan to go immediately to full implementation and some are reporting that they have no intention of going back. As noted, this is largely because of the cost savings achieved by the redesigns. Both PCR and R2R were to some extent dogged by faculty complaints that “this is just about saving money.” In C2R, participants appeared to welcome the fact that the redesign saved money. This is probably attributable to the fact that saving money in current budgetary conditions is more important than ever. When NCAT began its work more than a decade ago, getting more for less in higher education did not appear to be necessary, so redesign efforts were frequently viewed with suspicion. Now, they are a necessary condition for serving more students with fewer resources—a situation recognized by the several state systems of higher education that have sought NCAT’s assistance. In short, redesign is now broadly recognized in the policy community as a major tool for confronting the future. NCAT is now engaged in a project entitled Changing the Equation, centered on redesign in developmental mathematics courses at the nation’s community colleges. As it does so, it is making good use of all the lessons learned in C2R, as well as those amassed through the preceding demonstrations, R2R and PCR. The infrastructure of meetings, tools, and consultants put in place in all three of these projects, but for the most part launched and tested in C2R, also marks the transition of the NCAT’s role to broader institutional and policy support for course redesign. This new role is important and will be badly needed in higher education’s anticipated operating environment for the next decade.
|
 |
Quick Link |
 |
||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|