Pandemic Innovations & the Future of CS Assessments
One year after our CS department returned back to in-person classes, we reflect on what happened, what persisted, and what may be transformative in the world of CS assessments.
Here are the slides we're presenting at the Pandemic Pedagogy Research Symposium:
What did we learn about CS pedagogy last year?
This past year of learning and teaching CS online has led to new insights and innovations in the classroom. Here are our recommendations at a glance, whether you're teaching in person or remote. To learn more, read the About Page.
#1: Make asynchronous videos available.
Make concise pre-recorded videos available so that students can cover the material at their own pace and can be better prepared to utilize synchronous class activities to maximize their learning.
💬 "Being able to look for a specific bit of information that I am confused about, being able to rewind, process information at a different pace, etc. are invaluable. PLEASE let that stay for the future!" - sophomore studying SymSys.
📌 Try This: To ensure that students are completing asynchronous materials, you may consider including concept checks to be completed before synchronous classes.
One clear trend is that students strongly value the scheduling flexibility of asynchronous video presentations, as well as opportunities for review. What was once seen as a “nice to have” addition to a course appears to becoming an expectation for all CS courses, according to students who were polled at the end of Spring 2021:
During the pandemic, asynchronous video was particularly important for students in varying time zones. However, students participating in athletics or other extracurriculars cite the same advantages for residential course offerings.
In addition, the ability to watch short lecture videos prior to synchronous course meetings increases the value of the time spent as a group and may reduce the number of synchronous class meetings needed per week.
Instructor: Troccoli, N.; Enrolled Students: 303 in Spring Quarter; Link here
- Posted short "byte-size" pre-recorded videos in advance of the live lectures
- Videos were 4 - 15min. in length
- The course CAs facilitated optional "lecture watch parties" which enabled students to come together and watch the recorded videos while having the opportunity to stop and ask questions or discuss the content with others. Some students enjoyed these, others didn't; the course staff met the students where they were at.
- Live lectures were shorter, optional, and primarily focused on reviewing concepts, answering questions, and doing exercises
#2: Structure synchronous class time for "active learning."
By active learning we mean activities, dialogue and practice problems that make learning more social and provide opportunities for human interaction. It is also helpful to survey and collect frequent feedback so that you can continue to iterate with new formats for “active learning” methodologies.
💬 "If a course demands lots of synchronous student time, it needs to actually have students participating during that time." - junior studying CS.
📌 Try this: Some example uses of synchronous time include...
1. Real-time questions answered by course assistants over chat or online discussion
2. Problem sessions, often dividing the class in smaller groups or sections
3. Fireside chats with high-level overviews, guest speakers, and Q&A
Synchronous class time provides an opportunity to synthesize and apply material from asynchronous lectures.
💬 "One of my classes had prerecorded lectures, but the synchronous part was like a pleasant recap - I thought this went very well." - Student
The experiments at our institution did not reveal one “best way” to engage students during synchronous class meetings. However, the four quarters of instruction did reveal that students preferred classes in which they were:
(1) actively solving practice problems in class,
(2) able to ask questions of the instructor and the TAs,
(3) getting targeted practice on specific topics with the instructor and TAs,
(4) able to learn about real world applications of relevant course topics, including from guest speakers who are professionals in the field.
Making the course responsive to student needs and questions, while creating opportunities to take advantage of the precious human resources and opportunities for social interaction appears to greatly benefit learning (Sarma & Yoquinto, 2020)
"The in-class problem solving activities in CS166 were great because we could anonymously message Keith through zoom chat and then through what he said see what others are thinking/feeling." - Student
In the future, “active learning” might need to be more explicitly defined in CS to encompass a broad range of activities in which students are engaged with the instructor, the TAs, their classmates and/or guest speakers during synchronous meetings.
Instructors: Fatahalian, K. and Olukotun, O.; Students Enrolled: 106 in Autumn Quarter; Link here
💬 "Last quarter was, surprisingly, one of my most enjoyable quarters teaching. Lecture attendance was up, student-staff interaction was up, and my co-instructor Kunle Olukotun and I felt we had a better handle than ever before on whether students were following the lecture. By the end of the quarter, a higher percentage of the class was attending live lectures than previous quarters." - Fatahalian, K.
Read more about the virtual classroom designed for CS 149 - Virtual Teaching Doesn’t Mean Giving Up on the Live Lecture
- Live lectures took place in a virtual classroom within a new platform called OhYay and was designed by Prof. Kayvon Fatahalian, one of the instructors.
- Fatahalian's goal in building the virtual classroom was to incorporate the four elements he believes are most valuable in a live lecture:
- Lecture should feel like an event
- The lecturer needs to be able to constantly read the room
- Students should feel comfortable asking and answering questions
- Small group problem solving increases participation - The live lecture experience included:
- Student video off by default
- A raise-hand box with an option to enable video and appear on screen to ask a question
- Chat box with text or emoji responses
- Question board with upvote capabilities
- "Reaction storms" - animated audience reactions that float up the screen
- Breakout rooms
Instructors: Anari, N. and Finn, C; Students Enrolled: 199 in Spring Quarter; Link here
- One synchronous class session per week consisted of a live lecture with faculty which involved a introduction to the module for the week and a Q&A
- Devoted the other synchronous class session per week to a problem session in which a CA guided students to work together through practice problems
- Outside of the synchronous class times, students were expected to watch pre-recorded course videos (modules)
Instructors: Frank, M. and Lassiter, D.; Students Enrolled: 120 in Autumn Quarter; Link here
- Dedicated their Friday synchronous seminars to a general discussion or they invited a guest speaker to talk about a specific topic
- Students were required to submit one question or comment for each lecture video and reading, those questions and comments were then used to launch the weekly discussion during the Friday seminar, additional questions could also be submitted in advance or asked during the live discussion
#1: Formative and summative assessments can take many forms.
Use frequent assessment to reduce student stress and scaffold learning effectively. Consider instituting smaller exams, partner assessments, or concept checks. Each of these three options is described in more detail in connection with case studies below.
High stakes exams are not the only way to assess student learning. Frequent lower stakes exams can reduce the stress on students and allow for faster learning cycles. More varied formats such as projects, group assignments and homeworks all help to evaluate student understanding of the material. Projects and group assignments also let students take on more peer teaching responsibilities, which increases their learning. Frequent concept checks help students assess how they are doing in a course and adjust their effort accordingly.
Student response:
70% of students considered it beneficial or very beneficial to have multiple smaller exams in lieu of one final exam.
69% of students found partner / group assignments beneficial or very beneficial.
56% of students considered it beneficial or very beneficial to have weekly concept checks.
Student response: 70% of students considered it beneficial or very beneficial to have multiple smaller exams in lieu of one final exam.
💬 I also enjoyed the lower amount of pressure placed on the exams and higher weight given to the homework. - Student.
🔍 Case study: CS110, Spring 2021
Instructors: Cain, J. and Cyrus, R.; Students Enrolled: 192 in Spring Quarter; Link here
- Three self assessments were offered in the Spring quarter to gauge the student's comfort with and understanding of the course material at specific points in the course
- Used as a tool for students to see how much they've learned and to see what work they had left to do
- Assessments were to be done individually and were open-book, open-note, and completed on Gradescope
- Made up of short answer, multiple choice, code reading, code writing questions
- For assessment-specific questions, students could post a private question to the discussion forum
- Assessment logistics: 72-hr window in which students must start the exam and they were allowed a 3-hour time period to complete it
- Grading: Scores were initially a percentage of 100% but for the final grades, the instructors took the square root of the percentage and multiplied it by 10 to result in a different percentage.
💬 "We're adopting this grading scheme so self-assessments can be as challenging and thought-provoking as on-campus exams generally are, but without the added pressure to get every single point. It also means you're not competing against each other and fighting to sit in the better half of the bell curve." - Cain, J.
🔍 Case study: CS 161, Winter 2021
Instructors: Anari, N. and Charikar, M.; Students Enrolled: 360 in Winter Quarter; Link here
- Four exams, each exam was designed to take students 1.5 hrs to complete
- Open book, closed collaboration
- Instructors made changes to the timing constraints as the quarter progressed based on feedback:
- Exam #1 - Timed (2 hrs)
- Exam #2 - Timed (36 hrs), students had 36 hrs to take the exam within a 96-hr period
- Exam #3 and Exam #4 - Timed (48 hrs), students had the full 48 hr window to complete the exam
💬 "For this course, having a limited time when you have to get all the answers right in a limited amount of time, it’s asking a lot. Given more time, they might have gotten the right answers. So it’s better to have a longer period of time. Having multiple assessments gives students the opportunity to do poorly once or whatever." - Charikar, M.
💬 "Not sure of the right frequency - students do feel like they’re being tested all the time. Reminded students that they are only being tested on a portion of the material (just 2 weeks worth)." - Charikar, M.
Student response: 69% of students found partner / group assignments beneficial or very beneficial.
💬 "SymSys 1 had mini-group projects! Much less stress than collaborating on an entire class project and a chance to get to know people." - Student
💬 "Having a pset partner can also be helpful to bounce ideas off of, study with, help each other debug, and also have a friend to learn with in the class during this virtual time!" - Student
However, this came with difficulties, especially in a remote setting.
💬 "In CS 166, the requirement to do a group project with at least THREE people is pretty unreasonable in a virtual scenario. The point of grades is to assess individual achievement and understanding, and group projects rarely do that, instead measuring the ability to find friends in a class, make schedules/deal with logistic, nag people to get them to work, and manage other logistics. I'm fine with the option to do projects in groups, but there should ALWAYS be the option to do a final project by yourself and the expectations for such projects should be titrated accordingly." - Student
💬 "One of the biggest issue with remote classes was the absence of in-person interactions making it harder to from new connections and hence often making it hard to choose project partners for course projects." - Student
🔍 Case study: CS 229, Spring 2021
Instructors: Charikar, M. and Re, C.; Students Enrolled: 279 in Spring Quarter; Link here
- Students could choose to do some or all assignments solo and were not tied to the same partner for the whole quarter, if they chose to work with a partner they were told to submit only once with both their names
- If working with a partner, students were told they must fully understand the material and shouldn't just divide up the work.
🔍 Case study CS 190, Winter 2021
Instructor: Ousterhout, J.; Students Enrolled: 18 in Winter Quarter; Link here
- All three projects were assigned as group projects with students working in teams of two for the entirety of the quarter
Student response: 56% of students considered it beneficial or very beneficial to have weekly concept checks.
💬 Weekly concept checks in CS 107 also helped me stay on track with the asynchronous lecture videos, and I thought that was a great idea because I never got behind on lecture which I often do. - Student
💬 Having concept checks or mini pre-lecture videos each week gave us an opportunity to see the material another time / get an intro to the material before the actual lecture. - Student
🔍 Case study CS 107, Spring 2021
Instructor: Troccoli, N.; Students Enrolled: 303 in Spring Quarter; Link here
- Every pre-recorded lecture video had an accompanying lecture check-in quiz that served as an understanding check for the content that was covered
- Students were permitted to take the quiz up to three times
- The check-in quiz had to be completed by the live lecture session, no late submissions were accepted
- Lecture check-in quizzes were worth 10% of a student's grade
🔍 Case study CS 109, Spring 2021
Instructor: Cain, J.; Students Enrolled: 203 in Spring Quarter; Link here
- Concept checks included material from previous lectures and pre-recorded videos
- Posted at least two days before the live lecture and were due just before the live lecture began, late submissions permitted up to a week after the deadline for 50% credit
- Students were allowed to submit an unlimited number of times, only the final submission was graded
- Concept checks made up 10% of a student's grade
"No doubt we’ll continue the concept checks when back on campus - will do this for every future class." Cain, J.
#2: Consider adopting policies that support mastery learning methods.
Students who had the opportunity to “revise and resubmit” homework and assessments found that they were better able to learn the content because they felt supported in continuing to learn the material until they proved they had mastered it.
Revise and resubmit policies that support mastery of course material reduce students stress and provide powerful opportunities for student learning and success. Support of these policies suggests that a forced grading curve is not necessary, particularly in introductory courses.Mastery-based or competency-based learning has been shown to improve student outcomes by shifting the focus away from exams to learning (Voorhees, 2001).
The incredible response by students and instructors alike to these policies suggest that both groups see tremendous benefits in this approach, with opportunities to support learners without putting too much of a burden on instructors.
Instructor: Lee, C. and Schwarz, K.; Students Enrolled: 206
- To certify that students were competent in the course material, they had to earn a 90% or higher on exams to pass
- If students didn't earn a 90% or higher on an exam, they were asked to revise and resubmit it by the following Sunday
- If students earned a 90% or higher on their resubmitted work they got the full credit, if they did not earn a passing grade, they were given more opportunities to revise and resubmit but at a cost to their grade
- Course staff offered coaching and mentoring throughout the revise and resubmit process
💬 I've always wanted to experiment with some form of mastery learning, and this was my chance! We have 3 take-home exams, and after each take-home exam is graded, students whose work was unsatisfactory have an opportunity (with extensive feedback and staff support, though not the actual solutions) to revise and resubmit their work a week later. Students whose work is still unsatisfactory can revise & resubmit again, with more staff support."- Lee, C.
Student response: 75% of students polled considered it beneficial or very beneficial to have the option to revise and resubmit work.
💬 Dr. Lee’s revise and resubmit policy in CS103 was fantastic. It meant I could engage with topics until I reached mastery, rather than just settling for “good enough.” - Student
💬 My learning experience feels so much more whole now that I’ve worked out the correct answer to every single midterm problem; I know this method of revision might be controversial because it doesn’t provide a memorization-based meritocracy grade distribution, but I feel like the CS program here has explained it’s goal to be to make all of us excellent programmers through hard work - which I think revision rewards, getting us student back on their feet after what might’ve been a demoralizing fall from grace. - Student
#1: Meet students where they are by creating multiple formats for support, including asynchronous.
The key takeaway for instructors and course designers is to find out what different students need and be responsive so that everyone has an equal shot at success in their course regardless of their prior background.
💬 Jerry Cain had a Slack workspace and he and the TA's were super active on it. I felt like I could receive and offer help to classmates, and I really felt like a part of a community.
💬 In CS107, the responsiveness on Ed was extremely beneficial to my learning because I always felt supported as I was doing my assignments.
- For students that like group environments, an in-person setting might be preferred.
- For students without strong social connections in their classes, facilitated “homework parties” can provide invaluable access to peer learners.
- Due to Zoom fatigue, students responded well to asynchronous forums such as Slack, Ed, and Piazza during the remote learning.
📌 Try this: Before you launch a new asynchronous forum, make sure to communicate with your TAs the added workload of managing one.
Instructor: Troccoli, N.; Students Enrolled: 303 in Spring Quarter; Link here
Used Nooks for office hours
- Students signed up in the queue and joined the Nooks room for the problem they were working on, this encouraged students to work with other students
- Course staff tried to utilize group rooms to help as many people as possible, students were encouraged to ask questions that were appropriate in a group setting
- If a student needed a private room, they were limited to 15 minutes
Used Ed for the discussion forum
- Students were encouraged to engage with other students and the course staff by asking and and answering questions and sharing resources on the discussion forum
- The discussion forum was not meant to be used for specific questions related to code, private posts were disabled
- Students were encouraged to participate openly and non-anonymously
Here are selected student responses:
💬 My course used Nooks, and I really enjoyed it. Each Nooks room corresponded to a problem on the pset, and you could go there to work on it with other students. This made it easier to connect with other students, more so than doing the same thing on Zoom. Even at in-person Stanford, I enjoy going to general study halls, so if classes have an option like Nooks, it creates more of a community.
💬 From CS107: I liked how this class had a question thread on Ed for each lecture. My CS106A class had a TA answer questions live via Zoom chat which I absolutely loved (which that wouldn't be possible in an in-person lecture). However, I realized that CS107's design was even better because asking on Ed "records" the questions everyone else asks (and you can search it for keywords!). This means I can read through their questions after lecture, which has been helpful for my understanding.
#2: Offer both virtual and in-person office hours.
💬 Virtual OH with queues are, in my opinion, the best thing about remote courses and I hope continue when things go back to normal...You can login to them from anywhere, which made it far easier to attend OH and get support (especially as a part-time student)...TAs felt less pressured to stay longer than they would in a physical location because it was easier to just enforce the time boundaries and log off. - Student.
For some that don’t feel comfortable in hectic environments or may have restrictions on their mobility on campus (from a disability, injury, etc.), online office hours will continue to be their preferred method for seeking help.
💬 Virtual OH with queues are, in my opinion, the best thing about remote courses and I hope continue when things go back to normal...You can login to them from anywhere, which made it far easier to attend OH and get support (especially as a part-time student)...TAs felt less pressured to stay longer than they would in a physical location because it was easier to just enforce the time boundaries and log off.
Students responded positively to new platforms as well:
💬 My course used Nooks, and I really enjoyed it. Each Nooks room corresponded to a problem on the pset, and you could go there to work on it with other students. This made it easier to connect with other students, more so than doing the same thing on Zoom. Even at in-person Stanford, I enjoy going to general study halls, so if classes have an option like Nooks, it creates more of a community.
Instructor: Manning, C.; Students Enrolled: 466 in Winter Quarter; Link here
Students were encouraged to choose which Office Hours session to attend based on the expertise of the Course Assistant hosting the session, the calendar clearly listed who would be hosting the office hours and their area of expertise
- Students could make an appointment with the instructor through Calendly
- Used Nooks for office hours with TAs
- Students were encouraged to choose which Office Hours session to attend based on the expertise of the Course Assistant hosting the session, the calendar clearly listed who would be hosting the office hours and their area of expertise
- Office Hours were held Monday - Saturday at different times of day to accommodate different time zones
The student response:
💬 Office hours- Online OH are so much more accessible and I actually feel like I got a lot more help this year than in previous years. Huang basement is chaotic and CAs are often running around like crazy and attention is always divided. CS224N group office hours functioned really well online and my other classes like CS168, CS224S, etc had individual online OH that worked super, super well. Nooks is definitely better than Zoom for office hours.
