By Nathan Kracklauer
The US Federal Reserve is once again debating a change of course in interest rates. Whether we can or cannot afford a car or home, or whether we will have a job next week, could hinge on the decision they make. So yeah, we kinda want them to have mastered the full range of cognitive skills related to monetary policy.
How do economists develop those skills? And how can they and we—the beneficiaries or victims of their decisions—feel confident in their expertise?
Before getting anywhere near the Federal Reserve Building, economists go through years of academic training. As a monetary policymaker, a person ought to know a thing or two about the history of the Great Depression, for example. But the Fed economists don’t just know that the stock market crashed in 1929. They comprehend that fact and can apply it in arguments they make as they analyze the current economy to see if there are similarities to 1929. They can synthesize all those arguments into a plan of what to do with interest rates and evaluate that plan to reach a decision.
Professional educators understand that knowledge, comprehension, application, analysis, synthesis, and evaluation are developed through different learning methods and have to be assessed through different methods as well.
You can validate acquisition of the factoid “Black Thursday took place on October 24, 1929” in a simple multiple-choice question. But to get beyond mere knowledge takes something else.
How have students traditionally developed higher-order cognitive capabilities and validated their attainment? By writing the college paper (and its bigger sisters, theses and dissertations).
Thanks to generative AI, that methodology is now dead.
A Eulogy for the Essay
I always enjoyed writing papers. That’s probably because I was extremely lucky with my professors. Some teachers lose the plot and grade based on how well you follow writing guidelines like “Don’t use passive voice” and “Tell them what you’re going to tell them…” The guidelines are not recipes for writing a brilliant essay. Truly brilliant essays break those “rules” all the time. Those guidelines are really there to make your essay easier to read for the poor shmuck who has to evaluate a truckload of them.
I had teachers who understood that the point of writing a paper is not the paper. They assigned papers to help their students develop a deep understanding of a subject, learn how to unmask bad arguments and articulate good ones, and learn to freshly illuminate a well-known subject from their unique perspective.
Large language models (LLMs) have changed everything. Why spend hours researching a topic and cobbling together prose you know will be mediocre when generative AI can do it for you in minutes?
Because the point is not the production of a paper.
In a comment on a blog post about using LLMs for academic cheating, computer science professor Kevin Zatloukal offered a priceless description of how brain-dead the act of using LLMs to generate your college paper is:
“The reason we ask students to write [essays] is that, for a human brain, the most efficient way to produce a good essay is to first understand the topic and then think critically about it. The classroom is a factory for producing human understanding and critical thinking skills. The essays … are actually a waste product.
The situation is as if someone saw a factory that was pumping waste into the river and decided to make a new factory that pumps waste into the river even faster and more cheaply, not noticing that the point of the factory was to produce bicycles not waste.”
What Comes Next for Assessing Cognitive Skills?
In the face of this truly catastrophic development, educators are scrambling to find an alternative. Blue Books and oral examinations are in again.
As the program director of a certificate program in business management leadership (The 12-Week MBA by Abilitie), I’m in the thick of this evolution. The advent of generative AI happened to coincide with the moment we were experimenting with better ways to assess how well our program was serving our participants as they developed their business acumen, people leadership, influencing, and collective decision-making capabilities.
We were doing penny-ante assessments like knowledge checks, of course. But we had been relying on our experiential methodology—in which participants play business simulations and interact with AI characters in managerial conversations—to do the heavy lifting when it came to developing application, analysis, synthesis, and evaluation-level capabilities. One option for raising the perceived rigor of our program would have been to establish clear rubrics and grade performance in our simulations, using them to both develop and assess.
But our learning philosophy is founded on the principle that our simulations are opportunities for people to make mistakes. We want our learners to treat our simulations not (only) as games you can win but as laboratories in which you run experiments, including experiments you think will fail in interesting ways. Grading simulation performance went against our principles and our compact with our learners. We needed a different demonstration of learning acquisition.
We thought about instituting a “term paper” as a final individual demonstration of learning. But we discarded that idea immediately. We saw where LLMs were taking things. It wasn’t that we distrusted our participants to do the right thing. It’s more that we knew outside parties—potential employers of our graduates, prospective candidates, media representatives—would question the rigor of something that relied on a validation approach that would soon be obsolete.
In-class “Blue Book” exams were out of the question for us with our virtual delivery. Instead, we opted for two other approaches to validation.
- First, we made a group project—previously optional— required. In it, small teams of participants perform quantitative and qualitative analysis of real-world companies and formulate an investment strategy.
- Second, we instituted a concluding interview. It’s something like an oral examination in which participants prepare answers to some complex questions about core program content and deliver them live to Abilitie executives. We grill them with follow-up questions, focusing on how the concepts apply in their current roles, or how they imagine applying in the roles they aspire to.
Along with the standard knowledge checks, the group project and the concluding interview have taken us toward a rigorous validation of the achievement of the learning objectives for us, for third parties and, most importantly, for the graduates themselves. We have further progress to make in reinventing learning measurement and evaluation. And so does the entire world of education.
Conclusion
The nature of work will transform as we avail ourselves of the tools of large language models. That could include managerial decision-making, even at the level of the Fed. I’m sure some reader is thinking right now, “Hey, I bet AI could do a better job setting interest rates than a board of humans.”
Sadly, that sentiment betrays the erosion of critical thinking capacity that has already taken place. How would we tell whether the AI’s monetary policy is “better” without the deep understanding, critical thinking, and capacity for value judgment we used to develop with essay writing?
If anything, those capacities will be more important than ever. And if we don’t find good alternatives to the essay, we’ll come to regret its demise.
Explore Approaches to Evaluating the Impact of Learning
The upcoming Communicating L&D’s Impact online conference takes you on a deep exploration of how modern learning professionals are evaluating learning and showing its impact. Don’t miss this two-day online event, April 15 & 16. Explore tools for identifying and tracking essential metrics, aligning learning to business priorities, and, critically, communicating your results and impact to stakeholders. Register today!
Image credit: James Ogden
