One year after our CS department returned back to in-person classes, we reflect on what happened, what persisted, and what may be transformative in the world of CS assessments.
Here are the slides we're presenting at the Pandemic Pedagogy Research Symposium:
This past year of learning and teaching CS online has led to new insights and innovations in the classroom. Here are our recommendations at a glance, whether you're teaching in person or remote. To learn more, read the About Page.
Make concise pre-recorded videos available so that students can cover the material at their own pace and can be better prepared to utilize synchronous class activities to maximize their learning.
💬 "Being able to look for a specific bit of information that I am confused about, being able to rewind, process information at a different pace, etc. are invaluable. PLEASE let that stay for the future!" - sophomore studying SymSys.
📌 Try This: To ensure that students are completing asynchronous materials, you may consider including concept checks to be completed before synchronous classes.
One clear trend is that students strongly value the scheduling flexibility of asynchronous video presentations, as well as opportunities for review. What was once seen as a “nice to have” addition to a course appears to becoming an expectation for all CS courses, according to students who were polled at the end of Spring 2021:
During the pandemic, asynchronous video was particularly important for students in varying time zones. However, students participating in athletics or other extracurriculars cite the same advantages for residential course offerings.
In addition, the ability to watch short lecture videos prior to synchronous course meetings increases the value of the time spent as a group and may reduce the number of synchronous class meetings needed per week.
Instructor: Troccoli, N.; Enrolled Students: 303 in Spring Quarter; Link here
By active learning we mean activities, dialogue and practice problems that make learning more social and provide opportunities for human interaction. It is also helpful to survey and collect frequent feedback so that you can continue to iterate with new formats for “active learning” methodologies.
💬 "If a course demands lots of synchronous student time, it needs to actually have students participating during that time." - junior studying CS.
📌 Try this: Some example uses of synchronous time include...
1. Real-time questions answered by course assistants over chat or online discussion
2. Problem sessions, often dividing the class in smaller groups or sections
3. Fireside chats with high-level overviews, guest speakers, and Q&A
Synchronous class time provides an opportunity to synthesize and apply material from asynchronous lectures.
💬 "One of my classes had prerecorded lectures, but the synchronous part was like a pleasant recap - I thought this went very well." - Student
The experiments at our institution did not reveal one “best way” to engage students during synchronous class meetings. However, the four quarters of instruction did reveal that students preferred classes in which they were:
(1) actively solving practice problems in class,
(2) able to ask questions of the instructor and the TAs,
(3) getting targeted practice on specific topics with the instructor and TAs,
(4) able to learn about real world applications of relevant course topics, including from guest speakers who are professionals in the field.
Making the course responsive to student needs and questions, while creating opportunities to take advantage of the precious human resources and opportunities for social interaction appears to greatly benefit learning (Sarma & Yoquinto, 2020)
"The in-class problem solving activities in CS166 were great because we could anonymously message Keith through zoom chat and then through what he said see what others are thinking/feeling." - Student
In the future, “active learning” might need to be more explicitly defined in CS to encompass a broad range of activities in which students are engaged with the instructor, the TAs, their classmates and/or guest speakers during synchronous meetings.
Instructors: Fatahalian, K. and Olukotun, O.; Students Enrolled: 106 in Autumn Quarter; Link here
💬 "Last quarter was, surprisingly, one of my most enjoyable quarters teaching. Lecture attendance was up, student-staff interaction was up, and my co-instructor Kunle Olukotun and I felt we had a better handle than ever before on whether students were following the lecture. By the end of the quarter, a higher percentage of the class was attending live lectures than previous quarters." - Fatahalian, K.
Read more about the virtual classroom designed for CS 149 - Virtual Teaching Doesn’t Mean Giving Up on the Live Lecture
Instructors: Anari, N. and Finn, C; Students Enrolled: 199 in Spring Quarter; Link here
Instructors: Frank, M. and Lassiter, D.; Students Enrolled: 120 in Autumn Quarter; Link here
Use frequent assessment to reduce student stress and scaffold learning effectively. Consider instituting smaller exams, partner assessments, or concept checks. Each of these three options is described in more detail in connection with case studies below.
High stakes exams are not the only way to assess student learning. Frequent lower stakes exams can reduce the stress on students and allow for faster learning cycles. More varied formats such as projects, group assignments and homeworks all help to evaluate student understanding of the material. Projects and group assignments also let students take on more peer teaching responsibilities, which increases their learning. Frequent concept checks help students assess how they are doing in a course and adjust their effort accordingly.
Student response:
70% of students considered it beneficial or very beneficial to have multiple smaller exams in lieu of one final exam.
69% of students found partner / group assignments beneficial or very beneficial.
56% of students considered it beneficial or very beneficial to have weekly concept checks.
Student response: 70% of students considered it beneficial or very beneficial to have multiple smaller exams in lieu of one final exam.
💬 I also enjoyed the lower amount of pressure placed on the exams and higher weight given to the homework. - Student.
🔍 Case study: CS110, Spring 2021
Instructors: Cain, J. and Cyrus, R.; Students Enrolled: 192 in Spring Quarter; Link here
💬 "We're adopting this grading scheme so self-assessments can be as challenging and thought-provoking as on-campus exams generally are, but without the added pressure to get every single point. It also means you're not competing against each other and fighting to sit in the better half of the bell curve." - Cain, J.
🔍 Case study: CS 161, Winter 2021
Instructors: Anari, N. and Charikar, M.; Students Enrolled: 360 in Winter Quarter; Link here
💬 "For this course, having a limited time when you have to get all the answers right in a limited amount of time, it’s asking a lot. Given more time, they might have gotten the right answers. So it’s better to have a longer period of time. Having multiple assessments gives students the opportunity to do poorly once or whatever." - Charikar, M.
💬 "Not sure of the right frequency - students do feel like they’re being tested all the time. Reminded students that they are only being tested on a portion of the material (just 2 weeks worth)." - Charikar, M.
Student response: 69% of students found partner / group assignments beneficial or very beneficial.
💬 "SymSys 1 had mini-group projects! Much less stress than collaborating on an entire class project and a chance to get to know people." - Student
💬 "Having a pset partner can also be helpful to bounce ideas off of, study with, help each other debug, and also have a friend to learn with in the class during this virtual time!" - Student
However, this came with difficulties, especially in a remote setting.
💬 "In CS 166, the requirement to do a group project with at least THREE people is pretty unreasonable in a virtual scenario. The point of grades is to assess individual achievement and understanding, and group projects rarely do that, instead measuring the ability to find friends in a class, make schedules/deal with logistic, nag people to get them to work, and manage other logistics. I'm fine with the option to do projects in groups, but there should ALWAYS be the option to do a final project by yourself and the expectations for such projects should be titrated accordingly." - Student
💬 "One of the biggest issue with remote classes was the absence of in-person interactions making it harder to from new connections and hence often making it hard to choose project partners for course projects." - Student
🔍 Case study: CS 229, Spring 2021
Instructors: Charikar, M. and Re, C.; Students Enrolled: 279 in Spring Quarter; Link here
🔍 Case study CS 190, Winter 2021
Instructor: Ousterhout, J.; Students Enrolled: 18 in Winter Quarter; Link here
Student response: 56% of students considered it beneficial or very beneficial to have weekly concept checks.
💬 Weekly concept checks in CS 107 also helped me stay on track with the asynchronous lecture videos, and I thought that was a great idea because I never got behind on lecture which I often do. - Student
💬 Having concept checks or mini pre-lecture videos each week gave us an opportunity to see the material another time / get an intro to the material before the actual lecture. - Student
🔍 Case study CS 107, Spring 2021
Instructor: Troccoli, N.; Students Enrolled: 303 in Spring Quarter; Link here
🔍 Case study CS 109, Spring 2021
Instructor: Cain, J.; Students Enrolled: 203 in Spring Quarter; Link here
"No doubt we’ll continue the concept checks when back on campus - will do this for every future class." Cain, J.
Students who had the opportunity to “revise and resubmit” homework and assessments found that they were better able to learn the content because they felt supported in continuing to learn the material until they proved they had mastered it.
Revise and resubmit policies that support mastery of course material reduce students stress and provide powerful opportunities for student learning and success. Support of these policies suggests that a forced grading curve is not necessary, particularly in introductory courses.Mastery-based or competency-based learning has been shown to improve student outcomes by shifting the focus away from exams to learning (Voorhees, 2001).
The incredible response by students and instructors alike to these policies suggest that both groups see tremendous benefits in this approach, with opportunities to support learners without putting too much of a burden on instructors.
Instructor: Lee, C. and Schwarz, K.; Students Enrolled: 206
💬 I've always wanted to experiment with some form of mastery learning, and this was my chance! We have 3 take-home exams, and after each take-home exam is graded, students whose work was unsatisfactory have an opportunity (with extensive feedback and staff support, though not the actual solutions) to revise and resubmit their work a week later. Students whose work is still unsatisfactory can revise & resubmit again, with more staff support."- Lee, C.
Student response: 75% of students polled considered it beneficial or very beneficial to have the option to revise and resubmit work.
💬 Dr. Lee’s revise and resubmit policy in CS103 was fantastic. It meant I could engage with topics until I reached mastery, rather than just settling for “good enough.” - Student
💬 My learning experience feels so much more whole now that I’ve worked out the correct answer to every single midterm problem; I know this method of revision might be controversial because it doesn’t provide a memorization-based meritocracy grade distribution, but I feel like the CS program here has explained it’s goal to be to make all of us excellent programmers through hard work - which I think revision rewards, getting us student back on their feet after what might’ve been a demoralizing fall from grace. - Student
The key takeaway for instructors and course designers is to find out what different students need and be responsive so that everyone has an equal shot at success in their course regardless of their prior background.
💬 Jerry Cain had a Slack workspace and he and the TA's were super active on it. I felt like I could receive and offer help to classmates, and I really felt like a part of a community.
💬 In CS107, the responsiveness on Ed was extremely beneficial to my learning because I always felt supported as I was doing my assignments.
📌 Try this: Before you launch a new asynchronous forum, make sure to communicate with your TAs the added workload of managing one.
Instructor: Troccoli, N.; Students Enrolled: 303 in Spring Quarter; Link here
Used Nooks for office hours
Used Ed for the discussion forum
Here are selected student responses:
💬 My course used Nooks, and I really enjoyed it. Each Nooks room corresponded to a problem on the pset, and you could go there to work on it with other students. This made it easier to connect with other students, more so than doing the same thing on Zoom. Even at in-person Stanford, I enjoy going to general study halls, so if classes have an option like Nooks, it creates more of a community.
💬 From CS107: I liked how this class had a question thread on Ed for each lecture. My CS106A class had a TA answer questions live via Zoom chat which I absolutely loved (which that wouldn't be possible in an in-person lecture). However, I realized that CS107's design was even better because asking on Ed "records" the questions everyone else asks (and you can search it for keywords!). This means I can read through their questions after lecture, which has been helpful for my understanding.
💬 Virtual OH with queues are, in my opinion, the best thing about remote courses and I hope continue when things go back to normal...You can login to them from anywhere, which made it far easier to attend OH and get support (especially as a part-time student)...TAs felt less pressured to stay longer than they would in a physical location because it was easier to just enforce the time boundaries and log off. - Student.
For some that don’t feel comfortable in hectic environments or may have restrictions on their mobility on campus (from a disability, injury, etc.), online office hours will continue to be their preferred method for seeking help.
💬 Virtual OH with queues are, in my opinion, the best thing about remote courses and I hope continue when things go back to normal...You can login to them from anywhere, which made it far easier to attend OH and get support (especially as a part-time student)...TAs felt less pressured to stay longer than they would in a physical location because it was easier to just enforce the time boundaries and log off.
Students responded positively to new platforms as well:
💬 My course used Nooks, and I really enjoyed it. Each Nooks room corresponded to a problem on the pset, and you could go there to work on it with other students. This made it easier to connect with other students, more so than doing the same thing on Zoom. Even at in-person Stanford, I enjoy going to general study halls, so if classes have an option like Nooks, it creates more of a community.
Instructor: Manning, C.; Students Enrolled: 466 in Winter Quarter; Link here
Students were encouraged to choose which Office Hours session to attend based on the expertise of the Course Assistant hosting the session, the calendar clearly listed who would be hosting the office hours and their area of expertise
The student response:
💬 Office hours- Online OH are so much more accessible and I actually feel like I got a lot more help this year than in previous years. Huang basement is chaotic and CAs are often running around like crazy and attention is always divided. CS224N group office hours functioned really well online and my other classes like CS168, CS224S, etc had individual online OH that worked super, super well. Nooks is definitely better than Zoom for office hours.