Chatbot Checklist: Reflection on an AI-Based Writing Lesson

 So… I tried something out.

Like I said in the last post, I’ve been wanting to start assessing students' writing using AI chatbots (specifically SchoolAI) or figuring out ways to incorporate and assess students’ use of it. This week, I had the opportunity to carry out and participate in my first full AI-based lesson plan.

Background

Right now, students are nearing the end of their Holocaust research paper. My CT planned to have a day where students completed an AI support activity. However, he missed a couple of days due to a family emergency, so I was in a position to lead this first class using AI. Being on my own, I also took the liberty of modifying the lesson a bit, adding onto what we had originally planned.

Lesson Breakdown

After a quick spiel about my personal use and philosophy on AI, I directed students to open their iPads and click on the SchoolAI link in Canvas. We spent the next little bit walking through this Chatbot Checklist (I love alliteration):

A “section” means at least one sentence, and at most one paragraph.

1. Clean-Up

1. Ask SchoolAI to edit a section of your essay, keeping the language the same, but revising the punctuation, usage, and grammar.

2. Adjust Length

1. Ask SchoolAI to either lengthen or shorten a section of your essay.

3. Change Tone

1. Ask SchoolAI to revise a section of your essay to change the tone, maybe more “formal” or “academic.”

4. Ask for General Feedback

1. Paste your entire draft into SchoolAI, and request general feedback tips to make it stronger. Manually change your draft in response to SchoolAI’s suggestions.

I mirrored my iPad screen onto the TV and took students through the Chatbot Checklist step by step. As I wrote prompts and copied sections, they followed along on their own devices, getting a hands-on feel for the process.

Reactions

Almost immediately, issues became apparent. The combination of us trying to set certain parameters, using outside files to try to guide the AI, and it being free and relatively small made for a confusing product.

First, we tried to engineer the bot to prevent it from taking full drafts and rewriting or editing them entirely. That did not work. Students pretty easily managed to "break" the bot, circumventing the guardrails and generating full rewrites.

While this really powerful tool was able to effectively and efficiently rewrote full drafts, it also became clear that there were shortcomings in it's abilities. In my little preamble at the start of class, I had warned students that AI sometimes "hallucinates," generating inaccurate information or content that violates the parameters of a prompt—I was right, and it did so more than I had anticipated. It provided feedback that was not only inconsistent with our in-class instruction but also with the draft template that students had used and that we had fed to the AI. Sometimes, it suggested changes that students had already incorporated or that simply didn’t make sense. For many students, the bot generated more confusion and frustration than clarity and assistance.

Some were outright pissed:

“This thing is so STUPID!! It keeps telling me the wrong thing! I'll keep doing it because you want me to, but I hate it.” punches iPad screen

Others described an overall discomfort with using it, saying things like “I like my brain better” or that “it feels weird.” Despite being given the freedom to use this tool, some still felt like it was cheating. One particular student announced that they would “never use AI unless it’s 'for funny'.” When I asked him to elaborate, he responded, “like generating an image of Evander Holyfield’s ear running away from him.”

Probably the lengthiest and most in-depth conversation I had was with the sub in the room. She expressed genuine concern about kids not using their brains and how this might affect their development. She was skeptical and said outright that she worried all kids would stop learning and developing the ability to read and write. While she was giving feedback to me, she was also voicing a more general concern about the state of society, which I thought was totally fair and important.

While the negative reactions were much more vocal and visible, there were also reactions on the other side of the spectrum. For some kids, the tool was relatively new, and a lightbulb kind of went off in their heads: “Oh wow, I could use this to write the WHOLE THING!” When I was asked if they could just copy and paste everything the bot generated, I realized I didn’t have a solid, prepared answer. I gave a pretty wishy-washy response, using the phrase "gray area" a lot.

Just walking around the room and checking in, there were a handful of kids who said, “Yeah, this is super helpful actually.”

Overall I'd say it was a pretty solid experiment with its ups and downs—we'll see where we go from here.