George Clooney’s 2009 movie Up in the Air showed us one way in which 21st-century online job assessments are conducted. Employees sit in a room and receive the summative evaluation of their work (in the movie they are all terminations) delivered via Skype by someone whom the employee has never met.

Most moviegoers were aghast at the prospect of the evaluation process we saw in the movie, but they considered it a very real possibility. Technology is changing everything, from job assessment to online teaching and learning. And it has empowered new groups of stakeholders. Our research on the online faculty teaching evaluation system at the Johns Hopkins School of Education (SOE) reveals that technology has given us the policy opportunity and political forces to create a 21st-century faculty assessment system. We believe that it is possible, through a collaboration of newly empowered faculty and student stakeholders, to design and implement new assessments that will improve the teaching and learning at SOE and other institutions of higher education.

Organizational change through politics and power

One approach to organizational change is to seize a moment in time when an organizational problem merges with that organization’s policy needs and political opportunity for its stakeholders (McLendon, Cohen-Vogel, and Wachen 2015). It is particularly important to seize these strategic moments in higher education, where bureaucracies and institutional traditions mitigate against rapid change. The problem under discussion, SOE’s teaching evaluation system, has emerged due to the development of SOE’s new online master’s and doctoral programs. The faculty believe that the current evaluation system does not accurately reflect their online instructional responsibilities. At the same time, SOE online students, empowered by their online relationships with faculty, feel they should have a bigger impact on the design of instruction and on the use of student feedback. This merging of policy problems and political uncertainty has led to an organizational moment that calls for a policy change for instructional evaluation and, at the same time, to empower SOE faculty and students.

Student evaluation of teaching policy problem

The continued growth of online education presents new technological and pedagogical challenges, among other policy problems, requiring new thinking about assessing effective instruction. The increasing pressure from the accountability movement, the emergence of rapid growth in online education, and the current challenges associated with commonly used student evaluations of teaching (SETs) are forcing higher education leaders and faculty to change assessments in order to improve university-level teaching in online courses. Technology also makes it possible for a new stakeholder partnership of faculty and students to address this particular policy problem together and to advocate for 21st-century online assessment systems that will have the political support to make them sustainable.

In order to analyze the current SOE instructor evaluation process and to identify alternative assessment policies, the researchers interviewed school leadership and faculty and surveyed current students in the new online education doctorate (EdD) program. Like over 90 percent of North American higher education institutions, SOE relies on SETs. Our findings show that SOE faculty still debate the merits of the student evaluation of teaching, often not using this data to inform their teaching practices. One reason for the lack of faculty support is that online instruction, like face-to-face, remains of secondary importance to many university faculty. Our results showed that even those School of Education faculty on a clinical track (instructional), as opposed to the research track, continue to undervalue teaching as a promotional tool. Further, regardless of the SET survey data they see, SOE faculty are not very inclined to improve their teaching methods.

School of Education student evaluation of teaching

The SOE currently uses a process for evaluating research and clinical faculty teaching effectiveness that includes the individual development and educational assessment (IDEA) student survey and an EdD program-administered teacher evaluation survey. To administrative and faculty leadership, the steps of the evaluation process were clear. However, the data suggested that the use and value of the survey findings were not clear to faculty or students.

From those faculty interviewed, we learned that a key problem for the personalized student feedback is that the SET is of varying importance to the faculty. Many voiced frustration, saying that leadership uses the survey findings when ratings are bad and ignores them when ratings are good. SOE leadership indicated that it is still possible for faculty to advance through the promotional process, even with average or marginal teaching ratings. Research faculty suggested that evaluation survey findings increase in importance directly preceding or during the promotion year, while clinical faculty reported that as long as the ratings stay above the minimum requirement, no action is taken or required. Furthermore, one semester of poor adjunct teacher evaluation data at SOE typically signals the end of an adjunct’s job. Full-time faculty who receive poor ratings, defined as less than average, are placed on a performance improvement plan.

SOE leadership also acknowledged that faculty teaching evaluations can be a source of dissatisfaction among faculty, particularly among the faculty who identify as researchers who teach. Many of these faculty members believe that research is most important in the promotional process, and they strive to minimize their teaching time. Those researchers who teach that we interviewed reasoned that the small sample size of student respondents to SET “reflects the small minority that are unhappy” (Borkoski and Kobett 2015); they maintain that happy students do not submit evaluations. Despite their skepticism of potential rewards, there are faculty who consider instruction to be an important part of the job of an online professor (on the clinical track). These teachers who do research believe that “research matters, but not as much as quality teaching and service” (Borkoski and Kobett 2015). The recognition of this division of faculty between researchers who teach and teachers who research contributes to a political debate that opens the door to policy alternatives.

Students unhappy with format

Interestingly, another political force emerged in our analysis of the SOE faculty assessment program. Over 50 percent of the EdD students surveyed reported disagreement or a neutral response to the statement, “My feedback is utilized to improve courses” (Borkoski and Kobett 2015). In other words, students feel unable to influence the instructional process. This suggests that new policy alternatives are needed to improve transparency with SOE students and also to communicate the importance of their instructor evaluations. In a technological age, student unhappiness with a program is communicated quickly and widely. This reality also increases student power in this political situation.

Variation in evaluation support and implementation, by students and by the two faculty groups, could empower an SOE faculty/student coalition to drive policy change to implement more effective teacher evaluations and student impact on that process. Furthermore, such a stakeholder coalition—empowered by intimate online relationships between teachers and learners, and seeking technology-driven policy alternatives—could alter the power relationships in higher education for the foreseeable future. 

Conclusion

In the 21st century, student needs are changing, and higher education faculty need “effective pedagogical skills for delivering student learning outcomes [to build] new relationships regarding access to teachers, and a wider range of communication and collaborative working through learning platforms” (Hénard and Roseveare 2012). Numerous policy options for the present SOE teacher evaluation system, offering better response to student needs, were suggested by those interviewed in our survey. One faculty member discussed the importance of consistent and ongoing reflection throughout the semester. Another discussed giving frequent and consistent feedback to online students, engaging in personal phone calls, and facilitating topical synchronous sessions with students. Still another faculty member identified the need to develop an online personality, including the ability “to lighten up the tension and concerns within the community” (Borkoski and Kobett 2015) to ease students’ stress and anxiety. As one member of the faculty put it: Successful online instructors must adopt student-centered strategies including an ability to personalize feedback, engage within a community as a member of the group, and establish presence throughout the course (Borkoski and Kobett 2015).

Because the online EdD program is new, the political environment is opportune for SOE faculty and students to join forces and design a system that will change the constructs of instruction, for SOE and for other higher-education institutions. This coalition can develop and implement a new system that includes effective online teaching competencies, hiring practices that reflect them, and an evaluation system connected to these identified traits and feedback mechanisms. A clearly connected and integrated system of evaluation steps will lead to an evaluation process that not only collects data but uses the feedback in ways that improve teaching and learning.

Michel Foucault (1984) points out that it is our duty to break the bonds of professional constraints, reallocate power, and change our environments. Beyond reforming the School of Education, faculty and students could infuse value into college instruction in research universities through the design and implementation of a new instructional evaluation system. This merger of SOE’s policy needs and political opportunity means that the time is now for a coalition for policy change. Getting to satisfactory policy alternatives for instructional evaluation and reform will be more complex than simply firing poorly performing faculty via Skype. However, the resulting policy alternatives will do a great deal more for higher education teaching and learning as well as faculty and student empowerment.

References

Borkoski, Carey, and Beth M. Kobett. “Politics, Power, and Teacher Evaluation.” Johns Hopkins University, 2015.

Foucault, Michel. The Foucault Reader. Edited by Paul Rabinow. New York: Pantheon Books, 1984.

Hénard, Fabrice, and Deborah Roseveare. Fostering Quality Teaching in Higher Education: Policies and Practices: An IMHE guide for higher education institutions. OECD Institutional Management in Higher Education, 2012. https://www.oecd.org/edu/imhe/QT%20policies%20and%20practices.pdf

McLendon, Michael, Lora Cohen-Vogel, and John Wachen. “Understanding Education Policy Making and Policy Change in the American States: Learning from Contemporary Policy Theory.” In Handbook of Education Politics and Policy, 2nd edition, edited by Bruce S. Cooper, James Cihulka, and Lance Fusarelli. New York: Routledge, 2015.

Up in the Air. Directed by Jason Reitman. Universal City, CA: DreamWorks, 2009.

All Contributors

Carey Borkoski

Assistant Professor, Schools of Education and Public Health at Johns Hopkins University

Beth Kobett

Assistant Professor, Stevenson University

Henry Smith

Assistant Professor of Education Policy and Politics, Johns Hopkins School of Education