Documenting Teaching Effectiveness

Documenting Teaching Effectiveness
Robert M. Anthony
Graduate Teaching Associate
Department of Sociology
Winner of the 2005 Graduate Associate Teaching Award

In the following summary I will discuss both positive and negative feedback generated from the course that I was nominated for, Introduction to Classical Sociological Theory. I will also discuss the steps that I have taken to improve my teaching and my theory course in the wake of my own reflections and the reflections of others. This will be accomplished through a presentation of my overall experience with developing and refining this course, and by recounting the events of my teaching experience in the last year.

While taking a seminar on independent teaching offered in our department, I informed the department’s teaching coordinator that I would like the opportunity to teach a theory course. I hoped to teach theory to undergraduates because it was the first course that I enrolled in as an undergraduate student. It was that experience as an undergraduate student that inspired me to pursue sociology and teaching as a graduate student. At the time of the seminar, I had already taught Sociology 101 and the Sociology of Sport 655 independently. I felt that I was ready to teach a more advanced course. At my request, in the Winter Quarter of 2004, the department’s teaching coordinator assigned me to my own theory section. However, after discussing it with my advisor we decided that it would be better for me to hold-off preparing for a theory course until after I helped him as a TA with his graduate level theory course. We believed that this would allow me to sharpen my knowledge of the course material and gain valuable grading experience for the course.

I spent the Winter Quarter of 2004 grading graduate student’s essays, taking notes from my advisor, and re-orienting myself with social theory. During that time I was able to gain a new perspective on the history and development of social theory that would be very useful in constructing my own course.

In the Spring of 2004 I was initially scheduled to teach Sociology 101. But three weeks before Spring Quarter started unforeseen circumstances in the department left a theory section open that needed to be filled and I was offered to teach the theory course. Since I felt confident having just helped my advisor with a graduate level theory course I accepted. Little was I aware of how difficult it would be to prepare and teach social theory in just a few short weeks.

To put it bluntly, I did not do as well as I had hoped the first time I taught this class. This is reflected in the mean SEI score that I received for the class which was a 3.4. The average department score for this course in the previous 10 years was a 4.4, and I had never received a score below 4.0 for any of my previous classes. So what went wrong?

To get an idea of the mistakes that I made in prepping and presenting a course on social theory, I turn to the qualitative feedback that I received from this course. The first comment that I received read as follows:

“This is one of the few courses I’ve taken where the instructor really goes out of their way to make sure students understand the material. From the Jeopardy review to the class-wide e-mails, he really made things easier to understand. If something really didn’t seem to get across to us, he would make a review sheet and send it out in an e-mail. He really cared how we did.”

About four others had perceived the class the same, writing similar comments. But after reading through the rest of the comments it became clear that I did not do nearly as well as I had initially thought. Over and over again the comments from other students began to express the same concerns about the structuring and presentation of the material. Their concerns can be summed up with a comment from one of my students:

“The instructor seemed like he was really excited about teaching, which is good…However, I thought his lectures were very hard to follow. My suggestion would be to repeat main points 3 times to cue the students into what is needed to know. Also, the instructor jumped around some. Maybe if he could weed out the useless info from the useful. It was very difficult to study from the lecture notes because he presented so much information and was very vague as to what was needed for the exam.”

Besides the comments highlighted above, other themes emerged from the feedback and included: speaking too fast, reading directly from the notes and not making eye contact, offering confusing definitions and examples, having poor guidance for note taking, and finally, failing to offer students a comprehensive way to review for exams. It was clear that if I was given the opportunity to teach the course again it would need major revisions.

With such disappointing feedback and low SEI scores I went to my advisor to find out if I would be allowed to teach the course again given my performance. He informed me that more than likely I would receive another chance. And I wanted another chance. I had something to prove to myself. Fortunately I received that chance in the Fall of 2004.

When I started to revise my class I pulled out the comments once again and began to reflect upon the experience. The first thought that I had was that I needed to recognize who I was teaching. The theory course is not an elective course like Sociology of Sport, nor is it an introductory class like Sociology 101. It is a required course for Sociology and Criminology majors and students must pass the class to graduate.

So what did this mean? It meant that most of the students who take the course are enrolled because they have to be. I had a very different student population than I had in any of my previous courses due to the role that the course plays for Sociology and Criminology majors in their graduation. In addition, theory is difficult for most students to begin with, making it a real challenge to teach. To understand theory and to do well in this course students need to have good analytical, critical, and logic skills as well as a knack for abstract thinking. Understanding or creating theory is not something that comes naturally to most people. Most of the students taking the course are there to get by, not because they love theory like I do, and most have heard from others that it is difficult and come in expecting the worst.

With this in mind, it became clear from my SEI’s and qualitative feedback that I had taught the class in such a way that those who were good abstract thinkers and note takers benefited from my presentations, while those who do not posses these skills were left with little or no direction. This led to frustration among a majority of the students and resulted in them giving up on the class and on me. So the first thing that I needed to do was develop a way to make the class more appealing to students who are not good at abstract thinking while at the same time making sure that I do not hold back those who are. And that is what I attempted to do as I restructured the course.

So what steps did I take? The first step that I took was to create a student course packet that included a more detailed outline of my lectures, diagrams, charts, and overheads. This was in response to the many who wanted more concrete direction for taking notes during lectures. After putting together a free course packet, I also made strides to reduce the amount of information presented in lectures. One problem that I had while constructing the course was that I had just come out of being a TA for a graduate level theory course. In my neurotic haste I included more information than needed to be presented in a class for undergraduates. For instance, I included a great deal of information on the intellectual influences of the social theorists covered in the course. After re-reading the student comments it became clear that the detail I offered for the intellectual influences had confused the students about what was really important; the ideas of the main theorists. Although not all of the intellectual influences could be cut out (for they are important), I was able to drastically reduce the discussions and information on them to a few main points.

Another step that I took was to re-write the lectures themselves. Some students had complained that there was too much information in the lectures which made it difficult to know what was important for the exams. With exams counting for 50% of their total grade I had to do something about this. So I condensed the material that I had originally compiled by focusing on the most important ideas that each theorist contributed to sociology. I also revised the class writing assignments to make them clearer for those who have problems structuring their essays. I made the assignments open enough for students with writing talent to be creative while retaining a clear structure that aids in guiding students who have problems organizing their ideas. Finally, I cut out some of the required readings that were tangential to classical social theory and made them supplemental for those who find theory exciting like I do. Again, looking back I think the major source of my downfall was structuring the class too close to that of a graduate level course and the qualitative feedback made me aware of this pitfall.

What were the results of my revisions? Well to my surprise I was given the opportunity to teach the course again in the Fall of 2004. I entered the course with some new found confidence (and a new strategy) and I was successful. Previously I had received an SEI mean score of 3.4 in the Spring of 2004, with my new revisions and some experience my SEI’s for the Fall of 2004 were 4.6 for a morning theory class! Needless to say I was extremely happy (and relieved) when I received them. The qualitative feedback was overwhelmingly positive, with many more informed critical suggestions (instead of angst) that I have since implemented in my Winter 2005 class. Here are a few of the comments I received from the Fall 2004 course:

Thank you for being an energetic, effective instructor at 9:30 in the morning. Occasionally you went over the material very rapidly but it wasn’t ever a problem to ask you to repeat or clarify a point.

I really enjoyed the class- it was well organized and I learned a lot. My only suggestion is to have more discussion- it’s interesting to see what others think.

I would like to suggest that the tests not be so word specific but based on general concepts and ideas. But considering the difficult subject matter, Rob did an excellent job explaining it.

I really enjoyed the class. It was helpful when you could break things down into more modern terms that we could relate to.

Like the course a lot actually. At first I was really intimidated at the sound of a core class, and even after the first few days I was still nervous. Bu the print-off lecture notes helped a lot…they were crucial in my understanding of the subject matter.

Overall I think you did a good job. The only thing you might consider doing differently is how you billed the tests (how much they are worth of the total grade). I think that the papers do a good job gauging student’s understanding of the course material and should be worth more.

The ideas that my Fall class suggested were implemented this past quarter. They included: a study guide for the tests, more class discussion over the readings and lectures, more time between paper due dates, and slowing the pace of the lectures down. I have continued to make revisions of my own in hopes of making the class more accessible to more students. Throughout the quarter I composed questions specific to the topics of each lecture and to the readings. The questions were meant to aid students in focusing their thoughts on the central issues in the readings and lectures to improve their understanding of each main theorist. Like math, theory cannot be learned if one does not use it and that’s what the papers and now the questions are designed to do, foster independent abstract thinking. What is also good about the questions is that I use them as a source to generate more class discussion and as a way to reiterate important concepts.

Although I have not received my feedback from my most recent theory class, I have already thought of ways to make improvements for future classes. One area that I hope to revise is the student lecture notes. Up to this point the student course pack has essentially been a blank outline of my lectures notes. Although students have found them to be helpful, I have come to find that students are very concerned about the headings that do not get covered in lectures. In some instances I have to skip over non-essential materials that I prepared to make room for the discussions I now implement. What I intend to do is restrict the student notes only to material that I know I will cover in class. This will serve two purposes. One it will decrease the amount of writing that students will engage in during lectures so that they listen and ask more questions. And second, it will help them focus more on the central theories that the class aims to teach. Needless to say I am anxious to see how this new strategy works in the future.

In conclusion, it has been an eventful year. I went from receiving disappointing reviews and questioning my teaching abilities to being nominated for GATA. I can honestly say that just being nominated is reward enough. It is no secret that graduate students do not do this for the money since there is little in it. Most of us are still idealistic and hope to receive the reward of appreciation. Knowing that a student took their own time to nominate me is humbling and rewarding. It would also be great if my peers recognized my hard work as well. I hope to have the opportunity to share my teaching philosophy as well as my course materials with you. Thanks for your consideration in advance.

Documenting Teaching Effectiveness
Bora Bosna
Graduate Teaching Associate
Department of Mathematics
Winner of the 2012 Graduate Associate Teaching Award

Feedback helps me learn to make math more human. It reminds me I am teaching actual human beings, not learning machines and that the time we spend together really does affect their lives. I’m also greedy about feedback because it continues to make me a much better person.

I score high on “created learning atmosphere” because I tried different teaching methods with upper level courses. I taught 131,132,151,152 until Spring 2008 which were “coordinated” courses with the largest numbers in OSU with centralized quizzes, homeworks and exams. I signed up for 153,254,255 and 415 from then on. These courses have higher technicality and thinking so the time issues are even more challenging, but they are less structured so they give me more say and freedom. One thing I tried was: I solve an exercise, explain the major ways of thinking and themes involved in it, then write on the board a new exercise which the students practice in their seats. I borrow from theater rehearsals what is called “exquisite pressure” which means to give them a time limit to put pressure on them but not so short that it ends up making them panic. I go around the room, visit students one by one and indirectly give pointers if they made a mistake. If too many people got stuck on the same crucial step, at say the 3 minute mark I would announce that step and partially spoil the solution to get everyone on the same page so the class can move forward and the exquisite pressure doesn’t turn into a rut. I encourage guesses, wrong answers, mistakes. I say “I heard a 5, someone said -2, anyone else? Going once…” It turns into a fun auction where they forget about the fear of making mistakes, the lack of confidence in their answers and the embarrassment that keeps them quiet. Students said they learned a lot from me this way and never found recitation useless or boring.

Another method I employ that results in high scores on “created learning atmosphere” is “tell me what to do.” I jokingly tell them “okay everyone, I am a dummy who doesn’t understand math, you will tell me what to do in this exercise and I will write it down.” This is a fun way for them to enjoy mathematical writing and activity more humanly and to get them thinking. It’s rare to see smiles or hear laughter in a math class, if they’re having fun and giving bodily responses they are learning. Students reported these worked great and added humor to class, and scored me high on this category. These methods helped me make class more student-oriented.

I get high scores on “communicated subject matter clearly” because I changed the way I use language in class. We math teachers tend to do what we call “handwaving,” which is to “explain away” certain things and leave some details to students, but they think that’s accurate argument. I avoid handwaving language and use formal terminology as much as possible for it to stick. Instead of saying “set this equal to this and solve for this” I say “What is the exercise asking? For the tangent line find the derivative and evaluate it for the slope.” We give human names to abstract concepts like “faithful action,” “regular space,” “normal subgroup,” “f dominates g” and so on. I invent new language to show students this side of math. When they ask “Do I need to simplify this?” I say “now you need to tailor this to best suit your needs at hand.” But I try not to make it sound boring. I call some infinite series we use often “celebrities.” I joke and say “integration by parts is good for you. It’s like oatmeal.”

Responding to comments like “sometimes the explanations were not clear” I realized that we math teachers tended to explain math in a way that made sense to us rather than to the students but they were struggling with much more basic issues we were “handwaving.” We lacked empathy. I remember seeing the poster by Cindy Bernlohr’s office door for the first time: “Teaching: It’s about Learning. A Celebration of Students.” I realized that teaching is not about my understanding but about how students learned. I had to teach in a more student centered way. So I learned to think like a student by analyzing the papers I graded; so I could explain math in a way they could understand, not me. My scores on “communicated subject matter clearly” went up from 4.2 and 4.0 in Autumn 2006 to 4.8 and 4.8 by Spring 2007.

My high scores on “well organized” and “genuinely interested in teaching” categories are due to aiming for the absolute utilization of physicality (time and space) for the students’ benefit. You will see the moment I walk into the classroom I remove all objects like chairs, tables, carts, monitors that block the board view. I mark off the bottom of the board and write as high as possible. In L-shaped classrooms I mark off the parts students cant’ see. I pass graded papers back with minimum recitation time spent; I alphabetize to find papers quickly, coming early and passing papers them before class begins, or after class, or asking students questions and passing papers back while letting them think. I strengthened my lower body and got a larger “teacher-sized” eraser and so now I clean a board in 5 seconds. I treat teaching as a performance. I warm up before it, I built up stamina to last through so that my voice, eye contact and my “intensity” don’t give in. I treat it as choreography so I utilize class time very well; there were several times when I put the final period exactly the same moment the bell rang, it’s quite a magical moment, I heard students go “wow.” Since class time is never enough I walk with students after class, provide help by email and extra office hours, especially for out of town people and people with tough work shedules. Students’ feedback has been “he genuinely cares that his students learn and do well.”

My recitations are so effective that I get lots of comments like “Bora filled in what was missing in lecture,” “Bora served as the primary instructor” and “I learned more from Bora than the lecturer, he should have taught lecture.” Mathematics is learned by doing, so when I was the sole instructor in Summer 2008, 2009 and 2011 I addressed this by moving away from the usual lecture to a “lecture recitation hybrid” 5 days a week with more doing and less passive receiving. The feedback was great. Students told me they really loved lecture, that it felt like recitation 5 days a week. These were the times students told me of my passion of math the most and I got the most amount of unsolicited emails; since I met them 5 days a week instead of 2, the personal connection was stronger and the “rubbing off” much better.

But I wasn’t always such an effective teacher. When I first started in Autumn 2006, my “overall” scores were 4.4 and 4.2. Students commented that I should not assume that they knew a certain material from their background. I learned to work around such gaps in knowledge, to provide “reminders” throughout class but keep them short enough that the students who had the background did not feel bored or treated dumb. The comments changed to “Bora never assumes you know this or that like other TAs and he is always patient.” My scores on “interested in helping students” went up from 4.3 and 4.4 to 4.9 and 5.0 by Winter 2008. Then I began to receive comments like “learned more in recitation than in lecture” since I was incorporating little snippets of the lectures into recitations. These comments continue to this day.

The real challenge came after Winter 2008. I was already nominated for the Phil Huneke Excellence in Teaching Award (which I won the next year). The feedback was almost entirely positive and affirming. The student feedback got me so far, but I had always wanted to go further. So I created a self-feedback mechanism. At the end of each quarter I have an “SEI ritual.” I read all forms, I write down in bullet points what worked well for students, think of what could be done even better, what wasn’t absolutely 100% in my opinion even if the student feedback was great. I also use midquarter evaluations and the Math Department evaluations as a safety measure since the SEIs went online. The student feedback is great but it can only take me so far. I think I can go even further. Teaching is far deeper than we think.

Documenting Teaching Effectiveness
Monali Chowdhury
Graduate Teaching Associate
Department of Psychology
Winner of the 2011 Graduate Associate Teaching Award

Feedback provides a vital channel of information that informs me of my “hits and misses” in class, allowing me to reinforce the “hit” and modify the “miss” to hopefully transform it into a future “hit.”

Collecting and Summarizing Feedback

Student Feedback: Feedback from my students, which I solicit throughout the quarter, is indispensable in my self reflection and self improvement as a teacher.

• 4th week evaluation: After students have become fairly familiar with my teaching style, I ask them to complete what I have named the “Yum and Yuck!” open-ended evaluation, which indicates the things they like and do not like about my teaching style, respectively. This brief instrument gives me a snapshot of  how students are responding (e.g., “the candy- memory experiment we did in class was cool”, “more breaks”), and its timing allows me to adapt my style to meet the specific needs of students in that section.
• Online Student Evaluation of Instruction (SEI): I use the OSU online SEI, an elective instrument that students are encouraged to complete, to collect feedback at the end of the quarter. The SEI consists  of 10 items relating to the instructor’s effectiveness that students rate on a 1(low)-to-5(high) Likert scale.  In addition to the numerical ratings, I particularly appreciate the comments section as students’ own words often tell me more about their holistic experience in my class. It gives me confidence that I am successful in meeting the goals outlined in my Teaching Statement when I see my thoughts and  intentions resonate among student comments. For example, “Monali made lecture actually FUN to attend, she relates the material to real-world events, this helps me to both retain the information and apply it outside the classroom,” “loved how she knew all 80 of our names, felt like she truly cared for each of us,” “answered questions thoroughly, if she didn’t know an answer, she would find it for us by the next class,” “encouraged us to critically think by asking questions,” “what she taught stayed with me because of all the examples and her in class experiments.” My average SEI response rate of 69% (over the last four quarters) stands higher than the average SEI response rate of 57% across all sections of Psych 100.
• Open-ended student evaluation: I also use an end-of-quarter open-ended student evaluation, provided by the Psych 100 program, for additional qualitative feedback. The overwhelming majority of positive remarks is always very rewarding and continues to motivate me! Whenever I see a concern (e.g.,  “she seemed to repeat herself quite a bit,” “goes fast sometimes”), I reflect on my approach and make changes as described later in this section. I have included three representative open-ended student evaluations and a summary of my cumulative online SEI scores in Part 2 of this section.
• Rose and Thorn: Starting this quarter, I have implemented a strategy to closely monitor student feedback on specific course material. At the end  of every class, I ask students to write down their “rose” for that lecture  – their favorite or most easily understood topic, and the “thorn” – the most difficult or poorly  understood topic. I go through these responses to find the most commonly cited “thorn” which then I explain further, and give more examples of, in the next class period.

Mentor feedback: One of the most useful forms of constructive feedback comes from class observations made by Dr. Melissa Beers, Faculty Director of the Psych 100 program. Dr. Beers has frequently observed my teaching and commended me on my strong classroom management, expressive presentation style, and thoughtful use of active learning exercises. Our continued discourse has helped me to build on and refine my pedagogical innovations and skills. Last quarter, my PFF mentor Dr. Robert Weis at Denison University, invited me to teach one of his Introductory Psychology classes. I was very excited at this opportunity for teaching a smaller class in a liberal arts college environment. In his feedback, Dr. Weis complimented my conversational style of teaching, and ability to engage student participation. He also gave me some helpful tips on increasing student participation in quieter groups of students. 

Peer feedback: I find it very useful to have my peers visit my class and vice versa. Such exchanges help me get a fresh perspective on topics, and learn about new examples or activities for particular concepts. Peers who have visited my class have commented on my energy and enthusiasm, easy-going rapport with students, and stimulating PowerPoint lectures. 

Course Assistant feedback: Most quarters I get the opportunity of working with an undergraduate course assistant (CA). The Psych 100 CA program is aimed at Psychology majors (mostly honors students) who want to get involved in teaching. This not only presents a mentoring experience for me, but  also provides an opportunity to solicit feedback on my teaching from an undergraduate’s perspective. CAs who have regularly sat in on my classes have found my teaching style “interesting, lively while still being organized and informative,” and to provide “the perfect balance of humor and seriousness.” 

Facilitator Feedback:  Being a facilitator at the UCAT university-wide teaching orientation this past Fall was a valuable and novel experience as first, I was no longer teaching psychology but teaching about teaching, and second, the audience who consisted of graduate students and early-career teachers, was very different from my usual audience of freshmen. It was very satisfying to see the core features of my teaching style being reflected in the feedback of the group even in this setup – “Chowdhury was very  engaging. I enjoyed her manner of presenting,” “Funny, organized, upbeat,” “Excellent presenter –  I would want to be in her class!”, “Excellent, clean and quick debate/rapport w/ class.”

Integrating Feedback into Teaching

My first quarter of teaching was in Fall 2005 – a year after I had arrived from India to pursue my graduate studies at OSU. Though I thoroughly enjoyed teaching that first quarter, my online SEI scores were the poorest they have ever been. In their comments, students consistently noted that while they found me knowledgeable on psychology, they felt that I “had little rapport with students,” and could “use more activities and in-depth examples.” Reflecting on that quarter, I realized that I had inadvertently created the kind of classroom atmosphere that I had grown up experiencing in my high school and undergraduate college classes in India. Cultural differences in educational styles make the points of focus very different in a typical US and Indian classroom. Going through my student comments, I realized that I had unintentionally created for myself an image of an authoritarian and detached instructor. Though I had all the good intentions of being approachable and friendly to my students, my demeanor did not explicitly reflect that. I took these student evaluations to heart, and the next quarter I took active steps in developing student rapport. Some of the changes were deceptively simple like smiling more often! I integrated my humorous personality in my teaching style, engaged in more small talk with students, and verbally expressed frequently that I was there to help them and they could come to me anytime with questions. In an attempt to use more class activities, I sought suggestions from my peers, sat in on classes of advanced TAs, and spent considerable time reading up (online and books on teaching psychology) on ideas for class demonstrations. This was also when I started my own Psych 100 video library where I continue to collect relevant clips from TV shows and movies. My students in subsequent quarters, Winter and Spring 2006, actively responded to these changes. My online SEI scores steadily progressed and became comparable to those of the Psychology Department. These changes also made teaching a more fulfilling and enjoyable experience for me! I had to take a hiatus from teaching to work as a Research Assistant for my advisor from Fall 2006 to Summer 2009. I returned to teaching in Fall 2009 wanting to get back to the excitement of presenting in front of a class. In the intervening time, I had taken several graduate classes and had become familiar with the classroom environments and teaching styles of some of the best faculty at OSU. My presentation style had also evolved to become more confident and engaging. I redid my lecture PowerPoints and assignments from scratch, constantly being mindful of the comments students had made back in Fall 2005. That quarter, Dr. Beers congratulated me on having the highest online SEI score course-wide – a perfect 5.0. Since then, my online SEI scores (found in the cumulative report included) have consistently been in the range of 4.7 to 4.9 – higher than the Psychology Department average and similar reference groups in the University. I received a Meritorious Teaching Award from the Psych 100 program in recognition of my teaching excellence in the 2009– 10 academic years. Recent changes I have made based on student feedback is in the use of examples in class. In Winter 2010, a few of my student comments were along the same lines as “sometimes I totally got a concept but still we went through a ton of examples of it.”  I discussed with my colleagues and Dr. Beers the best ways to approach this comment. We agreed that students often inaccurately assume that they have understood a topic and consequently feel that they do not need additional examples. To create a balance, I have become more active in using student-generated examples in class (as outlined in my Teaching Statement). This has proved to be a good way of breaking the monotony of me providing multiple examples, while still furnishing students with real-life applications of concepts from their peers. My evolution as a successful teacher is constantly guided by the response it evokes from my audience, and I try to keep my teaching style fresh and exciting for both myself and my students.