AI Won’t Destroy Tests

When calculators first started coming out, people said they would be used to cheat and students wouldn’t learn anything. Instead, we changed how testing works to focus on learning what’s important – broader concepts and implications – instead of “what is 232+47”. With AI tools, we again need to change how tests work. This time, instead of asking if a student can regurgitate information in a way that aligns with the teacher, we can start to see if students are actually paying attention to the work. The difference between AI answers and real answers is a level of understanding deeper than the surface.

YouTube Censorship Made Me Write a Script

YouTube’s been forcing creators to censor their works more and more, and often times after a successful publish of said content. More history and valuable information is being lost every day because a corporation controls the largest source of video content freely available.

At the same time, I’ve been running commands using yt-dlp over and over again for my own purposes, aside from this censorship. The syntax is relatively easy to forget despite being very clearly defined, so I finally made a script to handle it for me.

It’s in Lua because that’s what I prefer to use, and available on GitHub’s gists. Because it is based on yt-dlp, it works for any website supported by yt-dlp. Here’s how to use it:

  ./video-dl.lua [action] <url>
[action]: What is desired.
  video (default): Highest quality video (maximum 720p).
  backup, clone, copy: English subtitles (including automatic subtitles), thumbnail, description, highest quality video (maximum 720p).
  music, audio: Highest quality audio only.
  metadata, meta: English subtitles (including automatic subtitles), thumbnail, description.
<url>: Source. YouTube URL expected, but should work with anything yt-dlp works with.

Information wants to be free. Help it.

How to Use ChatGPT

I’m late to the party, but maybe that’s better. I’ve forgotten some of the hype around AI, and the pace of innovation has settled down a little. Think of ChatGPT as a thinking tool with access to an internet-sized – but imprecise – database. That database was last updated in September 2021, and is imprecise because due to how neural networks work. The thinking part of this tool is rudimentary, but powerful. It does many things well with the correct input, but also fails spectacularly with the “wrong” input.

I separate the idea of what ChatGPT is from how it functions and where its knowledge comes from, because it helps me think of uses while remembering its limits. For example, I used it to help me journal more effectively, but when I tried to probe its knowledge of Havana Syndrome – a conspiracy theory commonly presented as fact by USA officials, it expressed useless information, because it has no conception of how it knows anything, or where its knowledge comes from.

Things ChatGPT is Good At

This list is presented in no particular order, but it is important to stress that AI often lie and hallucinate. It is important to always verify information received from AI. This list is based on my experiences over the past month, and will be updated as I use ChatGPT more. It is not comprehensive, but is intended to be what I find most useful.

  • Socratic method tutoring: The Socratic method is essentially “Asking questions helps you learn.” ChatGPT is very good at explaining topics, just make sure you verify its explanations are factual. (Questions I asked: Why are smooth-bore tanks considered more advanced while rifling in guns was an important innovation? Why do companies decrease the quality of tools over time?)
  • Writing: ChatGPT tends to be too verbose, but you can make it simplify and rewrite statements, and it can help you find better ways to write. (I asked it to explain the Socratic method a few times, then wrote my own version.)
  • Scripting: I created a utility script for file statistics in 2-3 hours by refining output from ChatGPT. The end result is more reusable, better written, and more functional than it would’ve been if I had worked on it alone. And that’s ignoring the fact that I got something I liked far faster than I would’ve on my own. (Just.. you need a programmer still. It can do some pretty cool things on its own, but also forgets how to count often.)
  • Planning: This is a todo item for me. I haven’t successfully used it for planning yet, but I intend to, and have heard of good results from others.

Things ChatGPT is Bad At

  • Facts & math: AI hallucinate. Check everything they teach you.
  • Finding sources: ChatGPT’s knowledge is formed by stripping the least useful data out of most of the internet, and who said what is far less important than specific pieces of knowledge – like how do you make a heading in HTML?
  • An unbiased viewpoint: While ChatGPT is fairly good at avoiding most bias, everything is biased. Removing bias completely is impossible. Discussing anything where there is strong motive to present a specific viewpoint will lead to that viewpoint being presented more often than an unbiased viewpoint.
  • Violent, illegal, and sexual content: While it is possible to bypass OpenAI’s strict handling of content, it is difficult, inconsistent, and can lead to having access revoked. Sadly, this prevents many ethical use cases due to a heavy-handed approach, and embeds the bias of OpenAI’s team into the model directly. There are ways around this with non-ChatGPT models.
  • What to do in Minecraft: I tried so many TIMES to get interesting ideas. It just can’t do it.

Things ChatGPT is Okay At

It’s important to know where AI can be a useful tool, but must be used carefully due to mixed results, so I am also including a list of things that work sometimes.

  • Advice: Similar to the Socratic method, a back and forth conversation can help you with your thoughts. Just be aware that ChatGPT can give some really bad advice too. For example, I wanted to see what it had to say on turning hobbies into jobs, and it covered none of the downsides, only talking about it as a purely positive experience.
  • Game design: I have spent too much time telling ChatGPT to design games for me. It will generate an infinite rabbit-hole of buzzwords and feature ideas, but cannot understand the concepts of limited time or scope. If you try to follow its designs, you will never complete anything.
  • Summarizing: If given text as input directly, when it is short enough, a summary can be reliably generated. If asked to summarize something extremely popular before its data cut-off, the summary can be okay. The drop-off on this is insane. Try asking it about Animorphs for example, something talked about occasionally, and certainly known about, but not something it can summarize.

This draft sat around for about 2/3rd of a month nearly complete. I would like it to have even more information, but I would like it more for it to be public. Apologies if it was a little short for you, but hopefully someday I’ll make a better version.

File Size Statistics Script (Lua)

I used ChatGPT to write a script for generating a list of file statistics based on everything within the directory it is placed in. It uses LuaFilesystem, and generates a final output like the following after it’s done processing through the files:

2359    files found.
Average (mean) file size:       44842.524374735 bytes
Standard deviation:     320478.50592438
Multiple modes:
Mode 1: 126     bytes
Mode 2: 204     bytes
Frequency:      7
[####################] 0.00 - 199271.16: 2245 files
[##########          ] 199271.16 - 398542.33: 59 files
[#######             ] 398542.33 - 597813.49: 16 files
[#######             ] 597813.49 - 797084.65: 14 files
[#####               ] 797084.65 - 996355.82: 6 files
[#####               ] 996355.82 - 1195626.98: 8 files
[##                  ] 1195626.98 - 1394898.14: 2 files
[#                   ] 1394898.14 - 1594169.31: 1 files
[#                   ] 1594169.31 - 1793440.47: 1 files
[                    ] 1793440.47 - 1992711.63: 0 files
[                    ] 1992711.63 - 2191982.80: 0 files
[#                   ] 2191982.80 - 2391253.96: 1 files
[                    ] 2391253.96 - 2590525.12: 0 files
[                    ] 2590525.12 - 2789796.29: 0 files
[                    ] 2789796.29 - 2989067.45: 0 files
[##                  ] 2989067.45 - 3188338.61: 2 files
[                    ] 3188338.61 - 3387609.78: 0 files
[                    ] 3387609.78 - 3586880.94: 0 files
[                    ] 3586880.94 - 3786152.10: 0 files
[                    ] 3786152.10 - 3985423.27: 0 files
[                    ] 3985423.27 - 4184694.43: 0 files
[#                   ] 4184694.43 - 4383965.59: 1 files
[                    ] 4383965.59 - 4583236.76: 0 files
[                    ] 4583236.76 - 4782507.92: 0 files
[                    ] 4782507.92 - 4981779.08: 0 files
[                    ] 4981779.08 - 5181050.24: 0 files
[#                   ] 5181050.24 - 5380321.41: 1 files
[                    ] 5380321.41 - 5579592.57: 0 files
[                    ] 5579592.57 - 5778863.73: 0 files
[                    ] 5778863.73 - 5978134.90: 0 files
[                    ] 5978134.90 - 6177406.06: 0 files
[                    ] 6177406.06 - 6376677.22: 0 files
[#                   ] 6376677.22 - 6575948.39: 1 files
[                    ] 6575948.39 - 6775219.55: 0 files
[                    ] 6775219.55 - 6974490.71: 0 files
[                    ] 6974490.71 - 7173761.88: 0 files
[                    ] 7173761.88 - 7373033.04: 0 files
[                    ] 7373033.04 - 7572304.20: 0 files
[                    ] 7572304.20 - 7771575.37: 0 files
[                    ] 7771575.37 - 7970846.53: 0 files
[                    ] 7970846.53 - 8170117.69: 0 files
[                    ] 8170117.69 - 8369388.86: 0 files
[                    ] 8369388.86 - 8568660.02: 0 files
[                    ] 8568660.02 - 8767931.18: 0 files
[                    ] 8767931.18 - 8967202.35: 0 files
[                    ] 8967202.35 - 9166473.51: 0 files
[                    ] 9166473.51 - 9365744.67: 0 files
[                    ] 9365744.67 - 9565015.84: 0 files
[#                   ] 9565015.84 - 9764287.00: 1 files
0th percentile: 0       bytes
10th percentile:        167     bytes
20th percentile:        317     bytes
30th percentile:        476     bytes
40th percentile:        692     bytes
50th percentile (median):       986     bytes
60th percentile:        1428    bytes
70th percentile:        2101    bytes
80th percentile:        3650    bytes
90th percentile:        38917   bytes
100th percentile:       9764287 bytes

With minimal effort, you could change it quite a bit, because it’s written as pure functions. I wouldn’t have achieved this myself, nor produced it so quickly, if I didn’t have ChatGPT do the easy stuff for me. I found the experience quite helpful. While ChatGPT did once forget that Lua indexes tables starting with 1, and made a few weird decisions and downright inefficient code in some places, it allowed me to focus on making it work exactly how I wanted it to, instead of just mostly correct or “good enough for now”.

(Btw, the example output above is from my Obsidian vault. You can read a bit more about how I use Obsidian to organize my notes here.)