Program in Course Redesign: Round II

Improving Learning and Reducing Costs:
Lessons Learned from Round II of the Pew Grant Program in Course Redesign

By Carol A. Twigg

Download PDF version

Since April 1999, the Center for Academic Transformation at Rensselaer Polytechnic Institute has conducted a Program in Course Redesign with support from the Pew Charitable Trusts. The purpose of this institutional grant program is to encourage colleges and universities to redesign their instructional approaches using technology to achieve quality enhancements as well as cost savings. Redesign projects focus on large-enrollment, introductory courses, which have the potential of impacting significant student numbers and generating substantial cost savings. The Center has awarded $6 million in grants to 30 projects in three rounds of ten projects each.

The second round of redesign projects began in July 2000 and concluded in July 2002. (Detailed descriptions of the ten redesigns and the outcomes each achieved can be found at http://www.thencat.org/PCR.htm.) The ten institutions and the courses they redesigned are:

What follows is an analysis of the results of the Round II projects, with a focus on the most important quality improvement and cost reduction techniques used in the redesigns, the implementation issues they encountered, and the projected sustainability of the course redesigns. The Center has produced a similar analysis for Round I. Another analysis will be produced for the Round III projects when they are complete.

Quality Improvement Strategies and Successes

Nine of the ten Round II projects report improved learning outcomes; one reports no significant difference. Among the findings are the following:

  • Redesign students at CMU showed significant improvement over traditional students based on a comparison of exam questions. In solving problems, redesign students made fewer than one error per problem, whereas traditional students made about six errors per problem. Redesign students also demonstrated an enhanced ability to identify the appropriate statistical analysis to employ in a given real-world problem situation.
  • At Fairfield, redesign students performed significantly better than traditional students on benchmark exam questions. In a second-year genetics course, redesign students performed significantly better on questions that measured retention of key concepts: 79 percent in the traditional format, and 88 percent in the redesigned course.
  • RCC redesign students had significantly higher scores than traditional students in four of six content areas on a common final exam.
  • At UA, the sum of A and B grades based on comparable examinations and assignments was significantly higher for the redesigned course than for the traditional course. In subsequent math courses, redesign students outperformed traditional students.
  • At UD, performance in redesigned sections was significantly higher than in the traditional course.
  • The percentage of students at the University of Idaho earning A and B grades based on comparable examinations and assignments was higher in the redesigned course; the percentage of D's and F's was lower.
  • At the University of Iowa, redesign students outperformed (mean score 24.7) traditional students (mean score 19.2) and outscored them on 29 of 30 items on a common chemistry exam. In addition, redesign students outperformed the comparison group on two forms of an American Chemical Society standard exam (65.4 vs. 58.4 on the first and 61.0 vs. 52.4 on the second).
  • At UMass, in spite of more difficult questions, scores on exams in the redesigned course averaged 73 percent vs. 61 percent in the traditional course. Attendance averaged 89.9 percent in the redesign vs. 67 percent in the traditional with a significant correlation between attendance and performance on exams.
  • At UTK, oral skills among redesign students were significantly better than among traditional students.

Three of the projects note that exam questions in the redesigned courses have shifted to testing higher-level cognitive skills. Among the changes were the following:

  • At CMU, final exam questions asking students to choose an appropriate statistical test when the correct answer was either chi-square or t-test were added in the redesign. Previously, these questions were not posed to students because they were deemed too difficult. In the redesigned lab, 43 percent of student questions involved interpreting statistical analyses versus 34 percent in the traditional lab, and 25 percent involved a deep concept versus 11 percent respectively.
  • At Fairfield, questions on exams used in the redesigned course were changed to test higher order thinking and to allow students to synthesize material from the basic concepts.
  • At UMass, the vast majority of exam questions in the traditional course were designed to test recall of factual material or definitions of terms, and only 23 percent required reasoning or problem solving skills. In the redesigned course, 67 percent of the questions required problem-solving skills.

Six of the ten projects show improvement in course completion rates; two report no change; and two experienced problems with students dropping or withdrawing from the course. Among the findings were the following:

  • During the pilot at Cal Poly, the most enhanced outcomes were the retention of students (the drop-out and withdrawal rates were 6 percent) and the reduction of students receiving grades of D and F.
  • At Fairfield, the rate of course drops decreased from a rate of 8 percent in the traditional course to 3 percent in the redesigned course.
  • At UA, the average success rate (grades of C or better) for the redesigned course was 49 percent compared to 46 percent for the traditional. Females were consistently more successful than males in the redesigned course as were African-Americans when compared to Caucasians.
  • At the University of Idaho, the percentage of students earning a D or failing was cut by more than half. Hispanic students, who have historically been unsuccessful in math courses, had an 80 percent pass rate in Algebra.
  • The University of Iowa reduced the rate of Ds, Fs and Withdrawals from 24.6 percent to 13.1 percent.
  • At UMass, a higher percentage of students earned grades of C or better, moving from 63.2 percent in the traditional course to 68.1 percent in the redesigned course, a reassuring but statistically insignificant increase.

All ten projects have effected significant shifts in the teaching-learning enterprise, making it more active and learner-centered. The primary goal is to move students from a passive, note-taking role to an active, learning orientation. Lectures are replaced with a wide variety of learning resources, all of which involve more active forms of student learning or more individualized assistance. In moving from an entirely lecture-based to a student-engagement approach, learning is less dependent on words uttered by instructors and more dependent on reading, exploring, and problem-solving undertaken actively by students.

Among their most important quality improvement techniques, the Round II projects identify the same four cited by the Round I projects: continuous assessment and feedback, increased interaction among students, online tutorials, and undergraduate learning assistants (ULAs.) The Round II projects cite two additional techniques that contribute to improved student learning: individualized, on-demand support and structural supports that ensure engagement and progress. The following is a list of the most effective quality improvement techniques used by the Round II projects.

  • Continuous Assessment and Feedback. Shifting the traditional assessment approach in large introductory courses, which typically employs only midterm and final examinations, toward continuous assessment is an essential pedagogical strategy in these redesigns. Most of the ten projects include automated (computer-based) assessment and feedback into their redesigns in fields as diverse as chemistry, Spanish, biology, mathematics and statistics. Automating assessment and feedback enable both repetition (student practice) and frequent feedback, pedagogical techniques that research has consistently proven to enhance learning.

The Round II projects use quizzes from commercial sources as well as those they create themselves. Students are regularly tested on assigned readings and homework; quizzes probe their preparedness and conceptual understanding. These low-stakes quizzes motivate students to keep on top of the course material, structure how they study and encourage them to spend more time on task. Online quizzing encourages a "do it till you get it right" approach: Students are allowed to take quizzes as many times as they want to until they master the material. The Universities of Alabama and Idaho take advantage of the quizzing features of MyMathLab, a commercial software package that generates problems and offers immediate feedback. The University of Iowa makes heavy use of ChemSkillBuilder On-Line, a homework software program that replaces written homework and helps students practice problem-solving in an active learning environment.

UTK and UMass have developed their own online materials and quizzes. UTK's redesign of its Spanish course moves reading, writing and grammar exercises online, incorporating a rich array of learning resources and activities. Over 400 graphic, audio and video files have been keyed to course concepts. Students receive immediate (automated) feedback and detailed grammatical explanations about their work. Exercises are divided between practice exercises that can be taken as many times as needed and quizzes that can be taken only once for a grade. At UMass, quizzes provide powerful formative feedback to both students and faculty members. Faculty can quickly detect areas where students are not grasping concepts, enabling timely corrective intervention. Students receive detailed diagnostic feedback that points out why an incorrect response is inappropriate and directs them to material that needs review. Since students are required to complete quizzes before class, they are better prepared for higher-level activities once they get there. Consequently, the role of the instructor shifts from one of introducing basic material to reviewing and expanding what students have already been doing.

  • Increased Interaction among Students. Many of the projects have restructured their courses explicitly to increase discussion among students. Students in large lecture classes tend to be passive recipients of information, and student-to-student interaction is often inhibited by class size. Through smaller discussion forums established online, students can participate actively. Psychology students at UD participate in three online, small-group (10-12 students) activities. Groups read and comment on a relevant article in response to two questions posed by the instructor. A subset of the group then reviews the discussion and generates two group responses to the questions, which are reviewed by the whole group and sent to the instructor for evaluation. At UTK, collaborative homework assignments, completed in small groups, involve online discussions in Spanish. At the University of Iowa, chemistry students in pairs or groups work on thought questions in lecture; use interactive simulations and small group problem-solving in discussion sections; collect lab data in pairs, pooling data from larger groups, and analyze collective values for the whole section.

The redesign of introductory biology at UMass demonstrates how it is possible to create an active learning environment within a large lecture hall setting. Before class, UMass students review learning objectives, key concepts, and supplemental materials posted on the class Web site. During class, UMass uses ClassTalk, a commercial, interactive technology that compiles and displays students' responses to problem-solving activities. Class time is divided into ten- to fifteen-minute lecture segments followed by sessions in which students work in small groups applying concepts to solve problems posed by the instructor. Group responses are summarized and reported through ClassTalk. The instructor moderates the discussions and draws out key issues to reinforce specific ideas or reveal misconceptions. At Fairfield, lecture sessions in biology have been redesigned to support teams of students working at computer stations (iBooks), each connected to an instructor-controlled computer via wireless technology. Students are expected to peer-mentor each other during in-class discussions.

  • Online Tutorials. Nearly every one of the ten Round II projects relies heavily on instructional software, some of which has been created at the institution and some of which is available from commercial sources. The most sophisticated software development has occurred at CMU, whose redesign of statistics is based on SmartLab. SmartLab is an automated, intelligent tutoring system that monitors students' work as they go through lab exercises. SmartLab provides them with feedback when they pursue an unproductive path and closely tracks and assesses individual students' acquisition of skills in statistical inference—in effect, providing an individual tutor for each student. SmartLab supports a dynamic model of problem-solving in lab exercises by asking students to choose and categorize relevant variables and select the appropriate statistical package tools, thus making labs and homework more open-ended, exploratory, and active.

All three mathematics projects in Round II have built their redesigns around a commercial instructional software package called MyMathLab. The availability of this software has allowed each institution to avoid spending funds on software development and to direct all of their resources toward supporting student learning. The software is versatile--supporting verbal, visual, and discovery-based learning styles--and can be accessed anytime at home or in a lab. MyMathLab allows instructors to see what work students are actually doing and to easily monitor their progress. In all three cases, the project teams have been very pleased with the pedagogical quality of the commercial software. Similarly, the University of Iowa and Fairfield University were extremely satisfied with the quality of commercial software available in chemistry and biology respectively.

  • Undergraduate Learning Assistants (ULAs). The Universities of Alabama, Dayton and Idaho employed ULAs in lieu of graduate teaching assistants (GTAs.) All three universities found that ULAs do an excellent job of assisting their peers. As an example, UA's initial plan was to staff their Math Technology Learning Center (MTLC) primarily with instructors and to use graduate students and upper-level, undergraduate students for tutorial support. It soon became apparent that the undergraduate students were as effective as the graduate students in providing tutorial support, thus eliminating the need for graduate students. At the University of Idaho undergraduate tutors are given an overview of the upcoming week's material and the homework exercises that typically give students problems during a weekly, one-hour mandatory tutor training session. At that session, tutors relay important information about student difficulties to the course coordinator so that it can be properly relayed to leaders of student focus groups. These training sessions help maintain consistency in instruction, and the undergraduate tutors play an important role.

  • Individualized, On-Demand Support. The emporium model used by UA and Idaho eliminates all class meetings and replaces them with a learning resource center featuring online materials and on-demand personalized assistance. All learning experiences are designed to move students from a passive to an active learning experience in which the student controls and individualizes learning based on personal needs. Faculty, GTAs and peer tutors work with students individually and in groups. By moving away from the three-contact-hours-per-week norm, the emporium model significantly expands the amount of instructional assistance available to students: UA's MTLC is open 71 hours per week, and Idaho's Polya center is open 86 hours per week. At RCC, a Math Collaboratory has been established on its three campuses to move from a seat-time model to one based on subject mastery. Faculty and tutors work with students individually and in groups. Students are able to access the Math Collaboratory, staffed about 40 hours per week, on a drop-in basis.

  • Structural Supports that Ensure Student Engagement and Progress. Each of the three Round II mathematics projects began with the idea of open-ended, flexible attendance policies—that is, allowing students to come to learning labs voluntarily, at times most convenient to them, and benefit from faculty assistance without the need of a requirement. Each discovered that their students need more structure in order to succeed. UA, Idaho and RCC added mandatory attendance and required group meetings to ensure that students spend sufficient time on task. RCC requires students to participate in math lab activities for a minimum of two hours per week. Previously students were able to choose the amount of time they spent, depending on how much time was needed to complete the assignments. This lack of structure meant that some students did not spend enough time and were not able to benefit from the available feedback and assistance.

The instructional freedom provided by UA's original redesign format was a problem for students who were not self motivated and well organized. Alabama now requires students to spend a minimum of 3.5 hours per week in the MTLC. In spite of attendance requirements, some students do not spend enough time in the lab to meet learning objectives. To ensure that students invest adequate time in the course, student hours in the MTLC are tabulated weekly. An automated e-mail system is used to reward students who are meeting requirements and to encourage those who are falling behind. In addition, UA students are required to attend a thirty-minute group session each week that focuses on students' problems and allows instructors to follow up in areas where testing has defined weaknesses.

Idaho students are also assigned to focus groups of 40 to 50 students each that meet once a week to coordinate activities and discuss experiences and expectations. Students are assigned according to their majors so that particular applications can be emphasized. In response to student requests for more structure, the Idaho team has also created the weekly task list, a step-by-step breakdown of the week's assignment that shows the student precisely where to find the information that pertains to each specific problem. Instructors are able to use the task list to help each student devise a detailed study plan for the upcoming week. The task lists are Web-based with links to all of the necessary online lectures and to hints and other supplemental material providing more instruction. The task list has not only benefited students; it has also benefited undergraduate and graduate tutors, who can appropriately direct students to the instructional resource needed to complete any given homework problem.

People who are knowledgeable about proven pedagogies that improve student learning will find nothing surprising in the above list. Among the well-accepted Seven Principles for Good Practice in Undergraduate Education developed by Arthur W. Chickering and Zelda F. Gamson in 1987 are such items as "encourage active learning," "give prompt feedback," "encourage cooperation among students," and "emphasize time on task." Good pedagogy in itself has nothing to do with technology. What is significant about the faculty involved in these redesigns is that they were able to incorporate good pedagogical practice into courses with very large numbers of students—a task that would have been impossible without technology

Cost Reduction Strategies and Successes

There are a variety of ways to reduce costs. As a result, there are also a variety of strategies for pursuing instructional redesign, depending upon institutional circumstances. The approach most favored by the Round II projects is to maintain constant enrollments while reducing the total amount of resources devoted to the course. By using technology for those aspects of the course where it would be more effective and by engaging faculty only in tasks that require faculty expertise while transferring other tasks that are less academically challenging to those with a lower level of education, an institution can decrease costs per student even though the number of students enrolled in the course remains unchanged. Nine of the ten projects employ this approach, which makes sense when student demand for the course is relatively stable.

But if an institution is in a growth mode or has more demand than it can meet through existing course delivery, it may seek to increase enrollments while maintaining the same level of investment. Many institutions have escalating demand for particular subjects like Spanish or information technology that they cannot meet because they cannot hire enough faculty members. By using redesign techniques, they can increase the number of students they enroll in such courses and relieve these academic bottlenecks without changing associated costs.

The University of Tennessee, Knoxville has been able to increase student enrollment in an introductory Spanish course while reducing the amount of resources devoted to the course. By pairing experienced instructors with GTAs and using technology for those aspects of the course where it is most effective, UTK has doubled section size from 27 to 54 students and reduced the number of sections needed from 57 to 38. Previously, the traditional course enrolled 1539 students at a cost-per-student of $109. In the redesign, about 500 more students are being served for a total enrollment of 2052, and the cost-per-student has decreased from $109 to $28, a reduction of 74 percent. UTK offers an excellent model for those institutions facing greater student demand than can be met using conventional methods.

Another way to reduce costs is to decrease the number of course repetitions due to failure or withdrawal, so that the overall number of students enrolled each term is lowered and the required number of sections (and the faculty members to teach them) are reduced. At many community colleges, for example, it takes students about two-and-a-half tries to pass introductory math courses. If an institution can move students through in a more expeditious fashion by enabling them to pass key courses in fewer attempts, this will generate considerable savings--both in terms of institutional resources and in terms of student time and tuition. Six of the ten projects show a decrease in drop-failure-withdrawal (DFW) rates. Of those six, only the University of Iowa has calculated the cost savings that result. Iowa's reduction in its DFW rate from 24.6 percent to 13.1 percent has meant that 90 students each semester do not need to repeat the course. These students comprise three discussion sections and four laboratory sections. The personnel needed to cover these sections equates to 1.5 GTA, no longer necessary, a cost savings of $7,022. Clearly, the other five projects that reduced their DFW rates could calculate those savings, which, in turn, would produce a higher cost-per-student savings than we report.

What are the most effective cost-reduction techniques used by the redesign projects? Since the major cost item in instruction is personnel, reducing the time that faculty members and other instructional personnel invest in the course, and transferring some of these tasks to technology-assisted activities are key strategies. Some of the more predominant cost-reduction techniques used by the Round II projects include:

  • Online Course Management Systems. Course management systems play an important role in eight of the ten Round II redesigns. UA and UTK use Blackboard; Cal Poly, Fairfield, Iowa and UD use WebCT, Idaho uses a homegrown system created specifically for the redesigned course, and RCC uses instructional software that includes an integrated management system. Sophisticated course-management software packages enable faculty members to monitor student progress and performance, track their time on task, and intervene on an individualized basis when necessary.

Course management systems can automatically generate many different kinds of tailored messages that provide needed information to students. They can also communicate automatically with students to suggest additional activities based on homework and quiz performance, or to encourage greater participation in online discussions. Using course-management systems radically reduces the amount of time that faculty members typically spend in nonacademic tasks like calculating and recording grades, photocopying course materials, posting changes in schedules and course syllabi, sending out special announcements to students—as well as documenting course materials like syllabi, assignments, and examinations so that they can be used in multiple terms.

Both UA and Idaho have experienced problems related to limitations in the currently available course management systems, particularly in regard to the universities' need to track student activities in their large-enrollment, single-section courses. To address these problems, Idaho has developed its own database, and UA has relied on some manual data entry.

  • Online Automated Assessment of Exercises, Quizzes, and Tests. As noted above, most of the ten projects use automated grading of exercises, quizzes or tests for subjects that can be assessed through standardized formats, not only increasing the level of student feedback but also offloading these rote activities from faculty members and other instructional personnel. Some use the quizzing features built into commercial software products like MyMathLab and ChemSkillBuilder On-line; others use homegrown systems created specifically for the course like CMU's SmartLab. Online quizzing sharply reduces the amount of time faculty members or GTAs need to spend on the laborious process of preparing quizzes, grading them, and recording and posting the results. Automated testing systems that contain large numbers of questions in a database format enable individualized tests to be easily generated, then quickly graded and returned.

In the traditional general chemistry course at the University of Iowa, for example, 21 GTAs used to be responsible for grading more than 16,000 homework assignments each term. Because of the large number of assignments, GTAs could only spot-grade and return a composite score to students. Because the homework process has been automated through redesign, every problem is graded and students receive specific feedback on their performance. Four TAs are now available for other assignments-–a significant savings in personnel time.

  • Online Tutorials. The use of instructional software allows much of the time previously spent on instruction to be transferred to the technology and eliminates lecture time previously used to introduce content and review homework. At RCC, lecture time has been reduced from four to two hours per week. Class meetings have been reorganized and targeted to topics that students find particularly difficult. Faculty members spend more time interacting with students about questions and problems rather than repeating math concept information. At the University of Idaho, online course-delivery techniques supplemented with a textbook give students flexible access to course content. Every topic in the course is presented in a series of streaming-video lectures created by the departmental faculty. Individual faculty members no longer present the same content in duplicative efforts, nor do they replicate exercises and quizzes for each section. When coupled with one-on-one help, these techniques have proved to be more effective and less expensive than classroom lectures while increasing student interaction time.

  • Shared Resources. When the whole course is redesigned, substantial amounts of time that faculty members spend developing and revising course materials and preparing for classes can be considerably reduced by eliminating duplication of effort. All ten of the Round II projects benefit from using shared resources. At UMass, a wealth of online materials including an extensive database of questions that provides students with formative assessment are easily accessed and manipulated by faculty, leading to a significant reduction in preparation time. Since responsibility for improving and updating the materials is shared among instructors, each faculty member's workload has been reduced. In addition, the availability of a significant number of online resources allows GTAs to review the students' work more quickly and efficiently and to reduce their time devoted to course preparation. GTAs no longer attend faculty lectures since they are able to prepare for labs and office hours using the Web-based resources.

Fairfield has found that using computer-based resources allows more learning to take place within the classroom, thereby reducing the amount of time faculty need to spend in office hours and extra student appointments. Using biology Web sites and relevant software in class helps to illustrate difficult content, which traditionally required much more one-on-one faculty-student interaction outside of class. In addition, online lecture notes, review questions, activities associated with the text CD and Web sites are often used by students as learning resources instead of relying on faculty office hours.

Another benefit of creating shared course resources is the opportunity for continuous improvement of those resources. During each phase of implementation, redesign teams are able to modify, update and revise learning activities based on what works well and what does not. Student feedback on the clarity and number of assignments, as well as their expressed need for greater explanations and models, provides multiple indicators for areas needing change. The online environment permits flexibility in design and expansion where needed, and timely changes can be made. In addition, many teams have found that once the course resources have been developed, only a minimum amount of additional labor has been necessary to improve the course content and keep it current. The shared course materials not only save the original instructors involved in the redesign course preparation and maintenance time, but also enable their use by new faculty members who would have had to prepare the course during the first semester of teaching it.

  • Staffing Substitutions. By constructing a support system that comprises various kinds of instructional personnel, institutions can apply the right level of human intervention to particular kinds of student problems. Highly trained (and expensive) faculty members are not needed to support all of the many tasks associated with delivering a course. At Idaho, a team that includes faculty, graduate students and undergraduate peer tutors—each with very specific roles—provides highly competent assistance to students. The team has been able to increase the number of contact hours, while greatly decreasing the cost-per-hour of that contact. At UD, the expanded use of teaching assistants, particularly undergraduate assistants, has reduced faculty workload and ultimately the number of faculty members responsible for teaching the course. At UA, the initial redesign plan was to staff the MTLC primarily with instructors and to use graduate students and upper-level, undergraduate students for tutorial support. In the first semester of implementation, it became apparent that the undergraduate students were as effective as the graduate students in providing tutorial support, making it possible to replace the graduate students with lower cost undergraduates. In addition, data on student use of instructional staff was collected during the first semester of operation and refined on a semester-by-semester basis. Based on that usage data, it was possible to reduce the number of instructors and undergraduate tutors assigned to the MTLC by matching staffing levels to trends in student use.

The preceding five cost reduction techniques were also cited by the Round I projects. The Round II projects identify two additional cost reduction techniques:

  • Consolidation of Sections and Courses. Unlike participants in Round I, the Round II institutions were required to redesign the whole course. As a result, many have been able to realize cost savings by consolidating the number of sections offered or the number of courses offered. In the emporium model used at UA and Idaho, multiple sections of a course are combined into one large course structure, replacing duplicative lectures, homework, and tests with collaboratively developed online materials. UA combined 44 intermediate algebra sections of approximately 35 students each into one 1,500-student section; Idaho moved two precalculus courses, previously organized in 60 sections of approximately 40 students each, into its Polya learning center, treating each course as a coherent entity. Each university, by teaching multiple math courses in its facility, can share instructional person-power among courses, significantly reducing the cost of teaching these additional courses.

At Fairfield, the redesign of the introductory biology course involved the consolidation of four separate sections into a single large-lecture format, reducing the faculty by almost half. The number of faculty involved in the lecture portion of the course went from four in the fall and three in the spring, to two each semester. This change depended on using technology successfully to create dynamic learning environments for the students to make up for the larger class. Because of the success of the chemistry redesign at the University of Iowa, the department has been able to combine the general chemistry sequence with a separate chemical sciences sequence, previously required by the College of Engineering, and decrease the number of faculty members needed to teach those courses. Now the special sequence is no longer needed, and 1.5 faculty per term are available for other institutional assignments.

  • Lower Cost Course Materials. At Fairfield, replacing some dissection labs with computer-based activities has decreased laboratory costs by nearly 73 percent (from $2470 to $680) because fewer dissection organisms need to be purchased. In addition, using national and international Web sites allows faculty to reduce the number of required wet labs by half and expand the ability of students to study comparative anatomy, not easily accomplished before. By putting course materials online, the UTK team has reduced the cost of materials students needed to purchase. In the traditional format, students paid a total of $182.35 for the textbook ($65.75), a CD-ROM ($10.95), two workbooks ($67.90) and audio CDs to accompany the workbooks ($37.75). In the redesigned course, students pay only $96.00 or $81.15 for a customized version of the textbook ($59.25 new/$44.40 used) and an access card ($36.75) for the online material.

With regard to cost savings, the redesign methodology is an unqualified success. All ten of the Round II projects have reduced their costs. Some saved more than they had planned; others saved less. The Round II projects planned to reduce costs by about 44 percent on average, with a range of 20 to 84 percent. They actually reduced costs by 38 percent on average, with a range of 25 to 74 percent. Final results from Round II show a collective savings of $1,017,512 for ten courses, compared with the original projection of $1,043,821. (For a detailed comparison of planned versus actual savings, please see www.thencat.org/PCR/R2savings.html.)

Why is there such a large range in cost savings across the projects? Differences are directly attributable to the different design decisions made by the project teams, especially with respect to how to allocate expensive faculty members. Redesigns with lower savings tend to re-direct, not reallocate, saved faculty time. They keep the total amount of faculty time devoted to the course constant, but they change the way faculty members actually spend their time (for example, lecturing versus interacting with students.) Others substantially reduce the amount of time devoted to the course by non-faculty personnel like GTAs, but keep the amount of regular faculty time constant. Decisions like these reduce total cost savings.

Higher education has traditionally assumed that high quality means low student-faculty ratios and that large lecture-presentation techniques are the only low-cost alternatives. By using technology-based approaches and learner-centered principles in redesigning their courses, these ten institutions like the ten institutions involved in Round I are showing us a way out of higher education's historical trade-off between cost and quality. Each project carefully considered how best to use all available resources—including faculty time and technology—to achieve the desired learning objectives. Moving away from the current credit-for-contact paradigm of instruction and thinking systematically about how to produce more effective and efficient learning are fundamental conditions for success.

Implementation Issues

As part of the grant application process, the Center required institutions to assess and demonstrate their readiness to engage in large-scale redesign by responding to a set of institutional-readiness criteria and to a set of course-readiness criteria, both developed by Center staff. (For a full description of the program's readiness criteria, please see http://www.thencat.org/PlanRes/Readiness.htm.) Our experience in the program has taught us that some institutions, because of their prior investments and experiences, better understand what is required to create these new learning environments and are more ready to engage in redesign efforts. In addition, just as some institutions are more ready than others to engage in large-scale redesign, some faculty members and some courses are more ready than others to be the focus of that redesign effort. Prior experiences with technology-mediated teaching and learning and numerous attitudinal factors give them a head start on the process.

The experiences of the Round II projects, like those in Round I, corroborate the importance of readiness in completing a successful redesign project. The ten institutions involved in Round II exhibited a high degree of readiness, and all successfully completed their redesigns. When project teams encountered implementation problems, however, in almost every instance the problem was directly related to a lack of readiness. The description of implementation problems that follows is organized in relation to the program's readiness criteria; the italicized portions are taken from commentary about each criterion included in the grant program guidelines.

  • Course Readiness Criteria #3: Decisions about curriculum in the department, program, or school must be made collectively.

Decisions to engage in large-scale course redesign cannot be left to an individual faculty member. An institution's best chance of long-term success involves not a single individual but rather a group of people who, working together, are committed to the objectives of the project. Indicators that the faculty in a particular unit are ready to collaborate include the following: they may have talked among themselves about the need for change; they may have decided to establish common learning objectives and processes for the course in question; and they may have instituted pieces of a common approach, such as a shared final examination.

The biggest implementation issue for several of the Round I projects was achieving consensus among all faculty teaching the course about a variety of issues. In contrast, five of the ten Round II projects cite collective decision-making and departmental buy-in as key factors that led to the success of their redesigns, thus reinforcing the importance of this readiness criterion.

One of the greatest benefits of the redesign at UMass is that it forces more meaningful interaction among the course instructors and other key staff. There is more open dialog among instructors when each student activity is previewed prior to class and then evaluated for effectiveness after class. Feedback from what occurs in class leads to improvements to the course Web site. At RCC, the large number of faculty engaged in the redesign (24 spread among three campuses) led to a very complex redesign organization. Various committees created a common syllabus, common tests and final exams, ensuring that course outlines of record are followed. A common grading metric ensures that academic standards are upheld. The discussion among faculty at all three RCC campuses regarding student performance is another unexpected, positive outcome. At UD, five faculty members, each with a different specialty area, developed the virtual lecture content collaboratively. Developing the course online allowed the university to test a model of team teaching that focuses on content development and instructional design rather than course delivery.

Both UMass and Fairfield emphasize the need for the instructors teaching the course and the department as a whole to believe in the value of the new approaches. UMass instructors are motivated by improved student interaction during class and their ability to focus more on concept manipulation and less on factual recall. Like other redesign teams, the Fairfield team has been faced with the challenge of obtaining faculty buy-in from the entire department. Since some traditional lectures have been replaced by computer activities, less time is available to cover material. Consequently, some lecture and lab material has been eliminated. The team has strong backing from most of the department, including freedom and encouragement to redesign the course syllabus as appropriate. Thus far, they have been able to convince the majority that the changes will enhance learning without sacrificing content. The team has also concluded that complete departmental buy-in is not required if there is core support for change.

  • Course Readiness Criteria #4: The faculty must be able and willing to incorporate existing curricular materials in order to focus work on redesign issues rather than materials creation.

Faculty who are willing to use an appropriate blend of homegrown (created by local faculty) and purchased learning materials in a non-dogmatic fashion will have a head start. Faculty who are susceptible to the "not-invented-here syndrome"—that is, who believe that they must create everything themselves from scratch—will be consumed with materials development and will add large amounts of time to the redesign process. Courses taught by faculty who are willing to partner with other content providers, whether commercial software producers or other colleges or universities that have developed technology-based materials, make better candidates for a large-scale redesign project.

Most of the Round II projects use commercial software or materials developed by other institutions and cite the decision to do so as a key ingredient in the success of their redesigns. At Fairfield, for example, most of the redesign relies on pre-existing material. The instructors are able to use online lecture notes and review questions previously created by the faculty. The computer exercises used in both the classroom and laboratory utilize Web sites created and maintained mostly by other academic institutions. It has been critical, however, to review these Web sites thoroughly before using them with students to ensure ease of use and high-quality content. As a result of that careful evaluation, students encounter few problems accessing or using the Web sites. Finally, the instructors purchased the BioQuest software library, which includes more than 70 modules covering topics from all areas of the biological sciences.

In contrast, the UMass team had hoped to use online materials supplied by the textbook publisher for additional readings and student activities, but they encountered difficulties when students bought used textbooks and compatibility problems between platforms and hardware. Consequently, the team has de-emphasized publisher-provided materials and relies instead on a variety of free Web sites. A side benefit of this change is that introductory biology students are exposed to the wealth of high quality materials that are available online in their discipline. There are, however, only a small number of institutions that have made materials suitable for active student learning in biology widely available. Therefore, the UMass team has initially had to put a significant amount of time into creating new materials.

Carnegie Mellon is the only institution among the ten projects that has engaged in extensive software development. The expectation for this project was that the development time would be greater than desired because developing a sophisticated tool that improves pedagogy necessarily involves a design-test-redesign cycle. In fact, the development of SmartLab and its full implementation in the course have taken longer than other less programming-intensive strategies might take. There are still a number of lab assignments and a large number of homework assignments for which SmartLab lessons have not been constructed. This was, by design, an ambitious project and one that CMU is deeply committed to completing.

  • Institutional Readiness Criteria #3: The institution's goal must be to integrate computing throughout the campus culture.

Unlike institutions that have established "initiatives" without specific milestones, computing-intensive campuses know the numbers. They know the level of availability of network access and the level of personal computer ownership (or availability) for students and faculty on their campuses because their goal is saturation, and the numbers tell them how close they are to achieving that goal. Ubiquitous networked computing is a prerequisite to achieving a return on institutional investment. Until all members of the campus community have full access to IT resources, it is difficult to implement significant redesign projects.

All projects emphasize the importance of collaboration between instructional and technical support staff. As they ramped up, three of the projects encountered problems in providing adequate laboratory classroom space and equipment to offer the course in the redesigned format. These problems were eventually resolved. Idaho has found a particularly innovative way to deal with space constraints. Housing 71 computers in pods of four that are designed for as many as three students to work together at a single monitor, Idaho's Polya Center provides a learning environment for over 2400 students annually. To accommodate this large number of students, the team distributes the load of student use more evenly by spreading assignment deadline dates across each day of the week. Thus, 20 percent of students have deadline dates for assignments, tests and quizzes on Monday, 20 percent on Tuesday, and so on. The space is used more consistently rather than just before a test or assignment is due, allowing more students to be accommodated in a smaller lab and reducing the lab downtime.

Fairfield relies on supplying and using portable computers (iBooks) in lecture and laboratory. Moving the computers between spaces and securing them represents a unique challenge. With increased use, the computers have become more susceptible to damage and theft. In the future, the instructors plan to assign each student pair a particular computer to be used throughout the entire semester. Computers will be named after historically significant biologists, and each student will be assigned a specific computer. This will make students accountable for their own computers and should expedite computer pick-up at the beginning of class.

  • Institutional Readiness Criteria #7: The institution must have established ways to assess and provide for learners' readiness to engage in IT-based courses.

Learner readiness involves more than access to computers and to the network. It also involves access to technical support for using navigation tools and course-management systems. Students also need to be aware of what is required to be successful in technology-intensive courses. Making the change from face-to-face instruction to online learning involves far more than learning to use a computer. Many students are set in their ways after a lifetime (albeit brief) of passive instruction. They need preparation in making the transition to more active learning environments.

Preparing students (and their parents) for changes in the way a course is offered turned out to be an important factor in several project implementations. The radical change in instructional style at UA produced some unique issues not typically associated with the traditional course structure, what the team has called the "No Teacher Syndrome." During the first year of implementation, students were very concerned about the lack of a formal teacher for their course even though they had one-on-one instructional support available at all times. In an effort to develop a personal relationship between students and instructors, the team now schedules weekly 30-minute "class" sessions and uses an automated e-mail system to allow instructors to contact their students on a weekly basis. In addition, the time instructors spend in the lab is fixed and publicized to allow students to come to the lab at specific times and deal with the same instructional staff if they so desire.

At UD, student surveys revealed that a major contributor to students' pre-course attitudes toward online learning was the belief that the course would be impersonal and would lack opportunities for student-student and faculty-student interaction. The development team addressed this attitude through the use of regular e-mail communication with students (including a weekly newsletter), the use of electronic chat for exam reviews and office hours, and the incorporation of online collaborative activities. Post-course student evaluations revealed high to moderate levels of satisfaction with the communicative and collaborative aspects of the course. In addition, UD found that the course needs to be promoted among students, faculty, and staff. A Web site that includes a demonstration version of the course has proven to be an effective promotional tool.

  • Additional Implementation Problems: Several projects experienced problems because they underestimated the degree of instructor, GTA, and undergraduate tutor training that is required in order to implement their redesigns successfully. As the UA team noted, the one-on-one assistance the computer-based format requires is very different from the teaching format that instructors have used and/or experienced in the past. Both UA and Idaho have expanded training for all instructional personnel each semester to better equip them to provide assistance to students.

At UTK, many of the GTAs have no experience in an online environment and are not prepared to help the students when they ask questions or encounter problems. Although training was held prior to the start of the pilot term, the team discovered that there is a need for ongoing training and stronger continuing GTA support than was initially planned. Because many of the GTAs are Master's candidates with minimal or no teaching experience, their readiness to engage in a newly designed learning environment is also low. To address this combination of challenges, the personnel structure of subsequent terms has been changed. GTAs are now paired with adjunct instructors who have a higher level of readiness to work with students in the new design; GTAs no longer teach sections independently. This revised model has been much more successful than the original plan where GTAs were expected to teach independently.

Three of the projects experienced backsliding from their original project goals in regard to cost reduction, bringing to mind the importance of Institutional Readiness Criteria #1: The institution must want to reduce costs and increase academic productivity. It is questionable whether some of the ten institutions involved in Round II really want to reduce costs. In one case, the full redesign was successfully implemented with 3600 students, demonstrating increased student learning gains and decreased costs. Nevertheless, some faculty members prefer the old model. In response to that faculty preference, the institution began to offer students a choice of either the redesigned or traditional lecture format. Now half the students are enrolled in redesigned sections and half in traditional sections. These changes suggest a lack of departmental and institutional commitment to reducing costs and increasing student success. In order for the successes achieved in a redesign to have a sustained impact, administrative leadership needs to play an active and continuing role.

In the second case, a fully online redesign was implemented in fall 2001, realizing the planned cost reductions. While learning outcomes were similar to traditional sections, the drop rate was higher and the perceived attractiveness of the online course suffered. A number of students complained on the end-of-semester evaluations and argued that they should be given a choice. Several student advisors argued that a completely distributed, online course was inappropriate for incoming freshman students. In addition, a number of faculty members wanted to teach the course in the traditional manner. In response, the department re-instituted a traditional delivery option in winter 2002. About half the students enrolled in that option, and the other half enrolled in the redesigned format. By the fall 2002 term, only 10 percent of the students were taught using the redesigned format.

Rather than addressing the problems encountered in the initial redesign implementation by adding needed structure, modifying the design by interspersing face-to-face activities, and doing a better job of preparing students for the online experience, the institution simply abandoned the redesign, thus forgoing the cost savings benefits. The university allowed the problems encountered, which are common when making a transition to a new form of instruction, to determine the future of the course. Other institutions involved in redesigns experienced similar problems, but each of them worked on solving the problems and ended up with a successful redesign.

In the third case, the institution experienced an unexpected change in project leadership. Three days into the first quarter of full implementation, the project leader became ill and was forced to retire immediately. The faculty member who substituted for her was not familiar with the planned delivery mechanisms; consequently, the implementation was put on hold. By the next quarter, the project leader was replaced with another faculty member, who used the quarter to learn how to manage the course, using all of the previously developed materials. The new project leader lacked experience and disregarded the original plan. Rather than following through on the plan to reduce the labor-intensive aspects of the course, the new team tripled the total faculty time per student. The university was unable to manage the project, leaving it under the control of an inexperienced faculty member, indicating a lack of serious commitment to reducing cost.

Sustainability

One way to judge the success of a grant-funded project is to assess its potential to be sustained once the grant funding runs out. Seven of the Round II projects are firmly committed to sustaining their redesigns. (Even the three that have backslid are committed to partially sustaining their redesigns.) Comments include "the redesigned course is now an expected part of the schedule and is here to stay," "the sustainability of the project is not in question," and "the team is confident that the redesign will continue." As one team puts it, "The fact that the redesigned course is less expensive and more effective than the traditional course delivery method virtually guarantees that it will be sustained."

A second way to evaluate the success of a grant-funded project is to consider its impact on other courses within the department and within the institution. Again, most of the projects report that the original redesign is having an impact on other courses. The increased efficiencies of the redesign at CMU have instigated further course evolution, and a second two-semester course is now being consolidated into one semester. The tools and understanding built through the first redesign project are being applied directly to the new course. In addition, work is underway to incorporate SmartLab into other courses such as an upper level statistics course at Carnegie Mellon and an introductory statistics course at the University of Pittsburgh. Finally, a fully online introductory statistics course is being built at Carnegie Mellon that will use SmartLab as its core.

The Fairfield team has begun to change the entire introductory sequence for biology majors in response to the redesign successes. Four semesters of introductory biology course work have been consolidated into three semesters. The cost-effective nature of the new approach will ensure long-term sustainability. At Idaho, many students have expressed the wish that the techniques used in the precalculus courses be used in the same way in calculus. At UTK, the department has decided that the second-year Spanish course is now easier than the redesigned Spanish Transition course so they have selected a more difficult book for the second-year course.

The University of Alabama has committed substantial funding to expand the MTLC, which has allowed additional courses to be taught in the facility. As of fall 2002, all students enrolled in Introductory Algebra, Intermediate Algebra, Precalculus Algebra and Finite Mathematics are using the new techniques and the MTLC. The university intends to continue to expand the use of computer-based instruction to additional courses in the precalculus sequence. The UMass team reports that the innovations pioneered in the introductory course are being used in other courses in the biology department. The course redesign also served as a template for a grant by the Davis Educational Foundation that funds eight additional redesigns across campus. Finally, the redesign effort has fostered a campus community of individuals interested in active learning approaches, especially in large classrooms, and has resulted in more grant initiatives to continue the university's efforts to improve student learning.

To what do we attribute the high level of success achieved by the Round II projects? The Program in Course Redesign provided leadership in choosing the right participants, teaching them the planning methodology, actively supporting them as they developed their design plans, closely monitoring the implementation process, and insisting on ongoing and final progress reports that include measurable outcomes. The program followed a unique three-stage proposal process that required applicants to assess their readiness to participate in the program, develop a plan for improved learning outcomes, and analyze the cost of traditional methods of instruction as well as new methods of instruction utilizing technology. (Please click for a description of the Center's Course Planning Tool, which facilitates this analysis.)

Perhaps the most significant aspect of this process has been the need for the Center to teach the redesign methodology, especially in regard to cost savings, since neither faculty nor administrators traditionally employ this approach to restructuring courses using IT. Prospective grant recipients were supported throughout by a series of invitational workshops that taught these assessment and planning methodologies and by individual consultations with Center staff. Both faculty and administrators have repeatedly indicated that learning the methodology is key to the effectiveness of the process. Once learned, however, the methodology is easily transferable to other courses and disciplines.

The pioneering institutions from Round II have made improvements to the initial redesign efforts of the Round I institutions and have achieved stronger results. Like the Round I institutions, they have established replicable models for those institutions that want to use technology to improve student learning while reducing instructional costs. We look forward to producing an analysis similar to this one for Round III when their projects are complete.

 

Improving Learning and Reducing Costs: Lessons Learned from Round II of the Pew Grant Program in Course Redesign

By Carol A. Twigg

© 2004 Center for Academic Transformation
Sponsored by a grant from the Pew Charitable Trusts
Center for Academic Transformation
Rensselaer Polytechnic Institute
4th Floor, Walker Lab
110 8th Street, Troy, NY 12180
518-276-6519 (voice) / 518-695-5633 (fax)
http://www.center.rpi.edu

Return to Top

 

Program in Course Redesign Quick Links:

Program In Course Redesign Main Page...

Lessons Learned:
Round I...
Round III...

Savings:
Round I...
Round II...
Round III...

Project Descriptions:
Sorted by Discipline...
Sorted by Model...
Sorted by Success...
Sorted by Grant Rounds...