Arizona Board of Regents: Learner-Centered Education Course Redesign Initiative

Improving Learning and Reducing Costs:
A Summary of Program Outcomes

By Carol A. Twigg

From December 2006 to July 2009, the National Center for Academic Transformation (NCAT) conducted a program in course redesign in partnership with the Arizona Board of Regents (ABOR) called the Learner-Centered Education Course Redesign Initiative (LCE CRI). The goals of this partnership were to 1) build on the successful models and lessons learned from NCAT’s national programs to create a course redesign program within ABOR; 2) adopt new ways to improve student learning outcomes and demonstrate these improvements through rigorous assessment; 3) reduce instructional costs and free up instructional resources for other purposes; 4) produce a number of successful ABOR course redesigns that can be spread to institutions throughout the system; and, 5) develop the internal capacity of ABOR faculty and staff to continue the course redesign process. All of those goals have been achieved.

The LCE CRI awarded grants of $40,000 - $50,000, plus one grant of $100,000 for a project of exceptional merit, through a competitive process. NCAT engaged the ABOR institutions in an extensive planning process to teach applicants its principles of course redesign. The NCAT process required faculty teams to follow a highly structured, iterative course redesign development process. Of the 27 teams which began the process, 13 completed and were ultimately selected for funding for a total of $690,772. The final selection was made by a committee of Regents, provosts, and faculty leadership. The following proposals were funded: 1) Arizona State University: College Algebra*, Computing and Information Literacy, General Chemistry, Introduction to Geology, Organizational Management and Leadership, Public Speaking, Uses of Accounting Information* and Women in Society; 2) Northern Arizona University: Introductory Biology* and Introduction to Psychology; 3) University of Arizona: A Geological Perspective, General Chemistry and Introductory Biology. Ten of the 13 projects completed the redesign process and fully implemented their redesigns in spring 2009. (Those asterisked did not complete the process and were only given half of the original award.)

The results achieved by the LCE CRI were very strong. The redesigns impacted approximately 21,726 students. Seven of the 10 projects demonstrated improved student learning as measured by direct comparisons of content mastery. The remaining three produced learning equivalent to the traditional format. Of the eight projects that wanted to improve course completion rates (as measured by a final grade of C or better), five succeeded. (Two projects had course completion rates of 90% and 93% in the traditional format and demonstrated equally strong completion rates in the redesign.) All 10 projects reduced their instructional costs—on average by 37%, and five projects saved more than originally projected. The annual savings for the 10 projects was $1,380,767. Eight of the redesigns will definitely be sustained after the grant period is over; the future of two of them is uncertain.

NCAT’s recommendations to the ABOR for future action included: 1) the ABOR should find ways to publicize the results of the LCE CRI among the three campuses; 2) the ABOR should conduct a second round of the LCE CRI; 3) the ABOR should engage in a transition process from NCAT’s managing a course redesign initiative to doing so itself; 4) provosts’ offices on each campus should be more purposeful and more actively involved in generating course redesign project possibilities and should think about incorporating redesign as a factor in campus resource allocations; and, 5) the ABOR should consider requiring that each campus integrate course redesign into its campus allocation strategies.  

Project Outcomes

What follows is a summary of the LCE CRI Course Redesign Projects’ Final Reports.

1) Did student learning improve (as measured by direct comparisons of content mastery)?

5 Yes
2 Yes – scores showed no difference, but the course is significantly more difficult
3 No difference (learning was equivalent to traditional format)

2) Did course completion rates improve (as measured by comparisons of final grades)?

5 Yes
1 No difference but significantly more difficult course
2 No difference (course completion was not an issue and was equivalent to traditional format: 90% and 93%)
2 No

3) Were instructional costs reduced?

10 Yes
  (In addition, 5 projects saved more than originally projected.)

4) Will the redesign be sustained after the grant period is over?

8 Yes
2 Questionable

1) Did student learning improve (as measured by direct comparisons of content mastery)?

Yes

1. ASU Computer Literacy (compared final grades using common criteria)

  • The redesigned course included a significant shift in content to ensure student preparation in computing and the use of modern tools and techniques to solve real world problems. Content shifted from learning a few basic concepts and common tools, to covering a broader range of technology advances, and familiarity with a wide variety of tools to support future self-learning.
  • In six prior terms of the course taught in the traditional format, an average of 26% of students earned 70%+ (a C or better); in the redesigned format, 65% of students earned 70+ (a C or better) on a demonstrably more difficult course.

2. ASU Organizational Management and Leadership (compared common exam questions)

  • Students in the redesigned course performed better compared to the traditional format. Specifically, scores on common exam questions in the traditional course averaged 67.4%, while in the redesigned course, exam scores averaged 76%.

3. ASU Women in Society (compared common midterms and common exams)

  • In each of the fall 2008 redesign sections, the average midterm grade was higher than the average of the spring 2007 traditional course.

--In Section A, the average midterm grade was 2.79% higher;
--In Section B, the average midterm grade was 14.61% higher; and,
--In Section C, the average midterm grade was 9.92% higher

  • Final exam grades in the fall 2008 redesign courses were also higher than in the spring 2007 traditional course.

--In Section A, the average exam grade was 6.13% higher;
--In Section B, the average exam grade was 7.83% higher; and,
--In Section C, the average exam grade was 7.49% higher.

4. UA Biology (compared common pre-tests and post-tests)

  • Students in the traditional fall 2007 course gained 7.47% in dealing with core course objectives; students in the redesigned fall 2008 course gained 8.64%. This difference is significant at the 0.001 level.

5. UA Chemistry (compared common American Chemical Society final exams)

  • Students in the first semester of the redesigned course sequence performed significantly better (p<0.05) than their counterparts in the previous years. The average final exam grade in the two semesters that the redesigned first-semester course has been offered was 59.3 ± 15.1% compared to an average of 54.0 ± 16.3% in the two previous years.
  • Similar results were obtained for the second course in the General Chemistry sequence. The average grade for the final exam in the fall 2008 semester of the redesigned course was 49.9 ± 11.6% compared to an average of 45.2 ± 7.69% in the equivalent off-sequence semesters in the two previous years.
  • Comparison of the final grade distribution between the traditional and the redesigned courses also showed a significant improvement in student performance (67.1% to 70.7% in the first semester, and 64.4% to 68.1% in the second semester.)
  • Development and implementation of common midterm and final exams based on a common set of learning objectives for the two courses has helped decrease the wide variability in the depth and extent of the material covered in different sections of the same course.

Yes – outcomes show no difference, but the course is significantly more difficult

1. ASU Geology (compared concept sketches on exams)

  • The final class averages were 74.3% for the traditional course and 73.4% for the redesigned course, a difference that is not statistically significant. The team considers such comparisons to have limited validity because of the major changes in how the course was taught.
  • The redesign changed the kind of knowledge that students gained compared to the traditional course. Previously this course used multiple-choice questions during exams to assess a thin veneer of factual knowledge spread across many aspects of geology. In contrast, the redesigned course used (1) online quizzes to assess a breadth of knowledge, (2) concept sketches on exams to assess deep conceptual knowledge in 40 or so of the most important geologic topics, and (3) online investigations that allowed students to solve an authentic scientific problem.
  • The fruits of this approach were clearly visible in the excellent concept sketches produced by students during exams designed to demonstrate student mastery of conceptual knowledge, which averaged better than 80% on nearly every exam. The team considers these concept sketches to be the best indication of how well students understand geological concepts and systems.

2. NAU Psychology (compared common pre-tests and post-tests)

  • In the redesigned section, performance improved from a mean of 31.2% correct on the pre-test to 40.2% for the post-test. This represents an improvement in .72 of one standard deviation, which is the second best ever obtained for a face-to-face section of this class since the department began using the knowledge assessment.
  • The traditional section in fall 2006 used as a comparison also showed a large increase across the pre- and post-test (from 32.02% to 47.76%).
  • When expressed in terms of proportion of one standard deviation, the improvement in the redesigned section is closer to that in the traditional section based on the mean scores alone.
  • These findings must be considered in light of the substantial increase in the amount and challenge level of required course work compared to the traditional mode of delivery. Traditionally, students completed few or no assignments outside of the standard four exams, and class participation was not required or assessed. In the redesigned course, students completed four required web assignments, daily class participation questions, 14 required online quizzes, and a required research survey.

No difference

1. ASU Chemistry (compared performance on common final exams)

NCAT note: A formal report has not been received, but the PI has conveyed this general information.

2. ASU Public Speaking (compared performance on final speeches)

  • Overall, the redesigned course allowed for the same learning levels to be achieved by students, while tripling course capacity.
  • The largest impact on students was an increase in comfort when speaking in a public setting. Students in the redesign course were much more likely to voluntarily engage in public speaking at an earlier point during the semester. Students were quick to engage in class discussions and showed more excitement when about to perform their speech assignments than in the traditional course.

3. UA Geology (compared common pre-tests and post-tests and common exam questions)

  • The Geosciences Concept Inventory ( GCI v1.0) was given as a pre-test/post-test to students in the traditional course and the redesigned course to compare student learning in terms of gain scores.
  • Students in the traditional course had a pre-test GCI percentage correct of 28.67 (SD=11.45, n=96) which increased to 45.26 (SD=11.76, n=84) on the post-test. After course redesign, students had a pre-test GCI percentage correct of 38.62 (SD=8.42, n=144) which increased to a post-test GCI score of 48.73 (SD=7.49, n=132).
  • Although the gains from pre-test to post-test were statistically significant, the different in post-test GCI scores between the two groups was not. Students’ knowledge levels were equivalent in both courses.
  • An analysis of common four separate essay-style exam questions was conducted to compare students in both the traditional and the redesigned courses. In all categories, the student-supplied responses from the two courses were indistinguishable.
  • Based on additional qualitative data collected, the team believes that these assessment instruments had insufficient resolving power to detect differences in student achievement or the student experience rather than that there were no actual differences in the student experience.

2) Did course completion rates improve (measured by comparing final grades)?

Yes

1. ASU Chemistry

  • The DFW rate in the first-semester course for science majors dropped by 4%.
  • The DFW rate did not change in the non-science-majors course.

2. ASU Computer Literacy

  • In six prior terms of the course taught in the traditional format, an average of 26% of students earned 70%+ (a C or better); in the redesigned format, 65% of students earned 70+ (a C or better) on a demonstrably more difficult course.
  • Similar results in the number of students withdrawing from the course remained between traditional and redesigned sections.

3. ASU Geology

  • Student success rate (C or higher in the course) was 85% in the spring 2008 traditional sections compared to 86.3% in the redesigned full implementation in fall 2008.
  • This improvement accompanied an increase in student workload from the traditional course, where students were only required to take five exams during the semester, to the fully redesigned course, where students completed five exams, 25-30 online quizzes, and 10 online investigations. In other words, the success rate went up slightly even though the workload and depth of coverage in the course increased from the traditional course to the fully redesigned course.

4. UA Biology

  • The DFW rate for traditional sections was 38.41%; the DFW rate for the redesigned sections was 33.83%, a difference of 4.58%.
  • The decrease in DFW rates was accompanied by an increase of 5.22% in C grades, with rates of As and Bs relatively unchanged.

5. UA Chemistry

  • In the past two years, 11.4% of the students enrolled in the first semester of General Chemistry failed the course in the traditional format. This percentage dropped to an average of 6.5% in the two semesters that the redesigned course has been offered.
  • Similar trends were observed for the second course in the series (a drop from 14.4% to 4.8% when comparing equivalent off-sequence semesters in the past two years).

No difference but significantly more difficult course

1. NAU Psychology

  • Similar results in the number of students withdrawing from the course remained between traditional and redesigned sections.
  • These findings must be considered in light of the substantial increase in the amount and challenge level of required course work compared to the traditional mode of delivery. Traditionally, students completed few or no assignments outside of the standard four exams, and class participation was not required or assessed. In the redesigned course, students completed four required web assignments, daily class participation questions, 14 required online quizzes, and a required research survey.

No difference

1. ASU Organizational Management and Leadership

  • Completion rates in this course are high, averaging around 93%.

2. UA Geology

  • Completion rates in this course are high, averaging around 90%.

No

1. ASU Women in Society

  • The student success rate (C or higher in the course) was 85% in the spring 2007 traditional sections compared to 75% in the redesigned full implementation in fall 2008. Although students did better on exams, low assignment scores--including assignments not completed--impacted final grades.

2. ASU Public Speaking

  • Student success rate (C or better) went from 92% in the traditional course to 89% in the redesigned course.

3) Were instructional costs reduced?

Yes

1. ASU Chemistry

  • The number of GTAs involved in the course was reduced from 99 to 75 annually.
  • The mix of personnel teaching the course also changed: 8.5 GTAs were replaced with five full-time instructors.
  • Together, these changes produced a savings of $411,575, a 20% reduction, which was slightly more than the projected 19% reduction.
  • There was also a considerable reduction in the use of space. The need for 74 recitation rooms for 24 students each was replaced by one newly designed room for 72 students devoted exclusively to general chemistry recitations.

2. ASU Computer Literacy

  • The plan was to decrease the cost-per-student from $50 to $38, a 24% reduction. The design worked better than expected. Delivery of the redesigned course is possible with less support than originally anticipated, leading to a cost reduction from $50 to $28 per student, a 44% savings.
  • The school has chosen to sustain the model with a higher level of faculty support which lessens the savings somewhat, although they are still substantial; a reduction from $50 to $33 per student, a 34% savings.
  • It was challenging to convince administrators that the redesign objectives could be met and that the course should be delivered as designed. It was difficult to get them to understand the substantially different way in which the course would be delivered and to convince them that it was possible to deliver the course with reduced costs. (NCAT note: This comment is from the lead faculty member!)

3. ASU Geology

  • The plan was to reduce the time spent by instructional personnel by 26%. Overall, the team estimated that the redesign effort resulted in a time savings of 30% for instructors and 35% for teaching assistants.
  • As a result of lessons learned during deployment of the fully redesigned course, the amount of time spent by instructors and teaching assistants during the current (spring 2009) semester is much less, and so the final time savings will be higher than these estimates.

4. ASU Organizational Management and Leadership

  • The team planned to increase the number of students served from 270 to 360 students and to increase section size from ~45 to ~60 students, reducing the cost-per-student from $373 to $159, a 57% decrease. The actual cost-per-student was $154, which represents a 59% savings.
  • Now that the course is taught in a new building with larger classrooms, the team believes they will be able to increase enrollment to as high as 250 students per term, because one classroom will hold 150 and a second classroom will hold 100. This means that they can increase the annual enrollment to ~500 (from the current 360) without additional resources.

5. ASU Public Speaking

  • The original cost savings plan was implemented. The university saved $200 per student enrolled in the redesigned course. The enrollment capacity has the ability to triple, although because of university restrictions currently it has only doubled.
  • In previous semesters, the course was traditionally taught with eight sections, capped at twenty-five each, a total of 200 students enrolled. In contrast, the redesigned course provided six sections capped at 100 students. This provided an opportunity for 600 students to complete the course annually.
  • No additional resources were required. With the increased enrollment in the redesigned course, the number of faculty members did not change.

6. ASU Women in Society

  • As planned, the redesign reduced the cost-per-student from $78 to $57, a 27% decrease. The redesign achieved this cost savings by increasing class size from 150-200 to 400 and reducing the number of sections from nine to four in the redesign.
  • The program was able to serve 1,200 students (903 in WST 100 and 300 in WST 300) with only three faculty (one tenure track and two instructors.)
  • The cost savings will allow the program to accommodate new student growth during a time of retrenchment in new hiring and to meet demand of a new graduate program.

7. NAU Psychology

  • The operational cost of the course was reduced by decreasing the number of sections from 11 to eight, increasing section size, and reducing the number of people teaching the course from seven to five.
  • Face-to-face section size increased from 175 per section to 400 in three sections and 200 in three sections. During the redesign, the team also added online sections of the course. Two online sections of 100 students each were also added, increasing the total number of students served from 1925 to 2000.
  • The team saved more than anticipated. The team anticipated that the reduced cost-per-student would be $48, a 23% reduction. The redesign actually reduced the cost-per-student from $60 to $42, a 30% reduction.
  • The cost savings were used to address budget cuts and to expand the department of psychology's course offerings to include an honors section of Introduction to Psychology, special topics courses and individual undergraduate research experiences.
  • Going forward, full-time faculty will do 90% of the teaching of PSY 101 without its dominating resource use in the department.

8. UA Biology

  • The plan was to achieve cost savings by decreasing the number of instructors in the fall term from six to four, reducing the number of GTAs and increasing the number of undergraduate learning assistants (ULAs) who are not paid while supporting increased enrollment.
  • The fully implemented redesign reduced the number of faculty in the fall term from six to five. The changes for the GTAs and ULAs were implemented as planned.
  • This change during full implementation means that the cost-per-student has declined from $266 in the traditional course to $178 in the redesigned course (rather than to the planned cost-per-student of $130). This is a savings of 33% rather than the planned 51%.

9. UA Chemistry

  • Four faculty and eight lecturers supported by 85 GTAs taught the traditional course at a cost of $188 per student. The redesigned course was staffed with four faculty, six lecturers and 85 GTAs at a cost of $171 per student. These changes result in a cost-per-student reduction of 13% and a projected savings of ~$100,000 per year.
  • As anticipated, the number of hours both faculty and lecturers spent in planning and delivering the course were reduced, f reeing up to 20 hours per week of faculty time that can be used for other activities including curriculum development, mentoring, supervision, course assessment etc.
  • Thanks to these cost savings, the department of chemistry has been able to sustain, and even increase, the average enrollment in the General Chemistry courses despite the major budget cuts that we have suffered in the past three years.

10. UA Geology

  • The team had a two-fold strategy to reduce costs.

--decreasing the number of GTAs each term from seven to four and replacing many of them with undergraduate preceptors.
--reducing the number of hours spent by faculty and graduate teaching assistants on preparation, class time and grading.

  • The bulk of our savings came from using twice as many undergraduate preceptors (from 20 to 40) and fewer half-time graduate teaching assistants.
  • Instructors found that they inevitably found themselves spending more time than they planned on preparation for teaching and on grading purely by choice, not out of necessity.

4) Will the redesign be sustained after the grant period is over?

Yes

1. ASU Chemistry

  • Sustainability is high because of the construction and monopoly of the new cooperative learning room for the general chemistry classes. The provost toured in April 2009 to see the progress, and visitors from outside the department are impressed with this new learning environment.

2. ASU Computer Literacy

  • It is certain that the redesigned course will be sustained.
  • A new faculty member was recruited to deliver the redesigned course and has successfully completed a semester. Her feedback from is positive, and she plans to maintain the course as designed.
  • The transition of the course to the new faculty member was easily facilitated. Alongside the redesigned course, the team created substantial documentation to support faculty, TA and undergraduate learning assistant (ULA) training.
  • The new faculty member is continuing to implement minor changes in an effort to further improve the quality of the course and address the challenges the team was unable to focus on due to the technical challenges faced.

3. ASU Geology

  • The redesign approach is being used in all large Introduction to Geology classes. The redesign effort has resulted in improved student understanding of geologic concepts, as assessed with concept sketches, and improved rates of student success. The time and cost savings are significant and are consistent with those originally estimated.
  • Since the start of the redesign project, all instructors have used the same textbook, PowerPoints, online quizzes, online investigations, online movies, and other materials. Instructors are largely pleased with this new approach, which has now been adopted by seven faculty members–every instructor who has taught a large Introduction to Geology class since the start of the redesign project. The efforts of the redesign project will be sustained for the foreseeable future.

4. ASU Organizational Management and Leadership

  • The redesign initiative is a success. Student learning has increased while the cost-per-student in the course had decreased. There is no question that the team is committed to the redesign and that it will be sustained. Other faculty members who might be assigned to teach the course in the future have also expressed their intent to employ the course redesign in their sections. Additionally, the redesign has also become a model for other courses taught by professors on the team as well as by other faculty in the school.
  • The team believes the redesign will enable them to accommodate even more students, and without adding new sections, as enrollment in the school continues to grow. With the availability of larger and suitable classrooms (that is, classrooms suitable for grouping students on occasion), the team believes that class size could increase by about 20% with perhaps graduate teaching assistant and student-grader hours being the only modest cost increase.

5. ASU Women in Society

  • The redesign will be very easy to sustain and to replicate for new faculty. This redesign allowed the program to standardize its survey courses. In the past, each instructor taught her own version, and the results were often uneven. Now the course is delivered in a consistent manner.
  • The redesign was such a success that the program may adapt the model for some other large classes.

6. NAU Psychology

  • The team is confident that the redesign will be sustained over time.
  • In fact, the department will have full-time faculty teaching all face-to-face sections of PSY 101 while maintaining a $42 cost per student.
  • The transition of a new faculty member into the co-teaching team highlights the flexibility of the redesign.
  • Additionally, the impressive learning outcomes and cost savings of the redesign has resulted in the placement of one of the team members at the university-level to serve as a campus redesign scholar in an attempt to continue the university's efforts to improve student learning and foster a campus community of individuals interested in active-learning approaches, especially in large classrooms.

7. UA Biology

  • The faculty who participated in the course redesign were uniformly positive about the impact of the redesign. Although the learning gains, as measured by pre- to post-test gains, were relatively modest, there is increased enthusiasm for “optimizing” the approaches and working together to address some of the issues that arose from the post-course analysis.
  • The instructional team has continued monthly meetings throughout the spring semester, unheard of prior to the redesign, and several are working with the publisher to suggest modifications and improvements to the Mastering Biology materials so that the students will be even better supported in their learning.

8. UA Chemistry

  • The department of chemistry fully supports the changes that were implemented and is committed to provide the resources needed to sustain the project.
  • At this moment, all of the General Chemistry courses offered by the department are being taught following the new format.

Questionable

1. ASU Public Speaking

  • The PI says, “The course redesign has proven to be sustainable. It is able to maintain a consistent curriculum and cost savings while course content is delivered effectively to the students. However, because the university is making a wide spectrum of changes to its academic programs, currently ASU is not planning to keep the redesign program in place.”

2. UA Geology

  • The PI says, “The sustainability of the project is not in question unless the general education program at the university sees some significant changes in the wake of the transformation and budget cuts. During the current semester (spring 2009) all sections are being run essentially as they were during our fully implemented redesign semester.”
  • However, the two strategies used to reduce costs were to replace graduate teaching assistants with undergraduate teaching assistants and reduce instructor time in preparing and teaching the course.
  • The team does not seem to be committed to cost reduction:

1) Instructors found that they spent more time than they planned on preparing for teaching and on grading purely by choice, not out of necessity.
2) The PI feels that having a graduate teaching assistant in break-out sessions leads to better exam and quiz scores. “If this is the case we will strive to get more TA support and give the students the value of this experience. We will continue to look for ways to improve learning that may require the use of more graduate teaching assistants in the future.”

  • Thus, both cost reduction strategies may be abandoned.

Final reports from each completed redesign are at http://www.thencat.org/States/AZ/ABOR Project Descriptions.htm. Final reports include learning outcome data, course completion data, cost reduction data, a discussion of the most important pedagogical techniques that led to increased learning, a discussion of the most important cost reduction techniques that led to reduced costs, a discussion of implementation issues encountered during the redesign process, and a discussion of future sustainability of the redesign.

 

Quick Links:

Arizona Board of Regents Main Page..