ChatGPT: My Test Drive

Alert Gazette readers may recall that I’ve plumbed the shallows of so-called Artificial Intelligence (AI) with some reviews of AI-powered image generators, notably DALL-E 2 and, to a lesser extent, DreamStudio. I was semi-impressed with the capabilities of both of these services, and I’ve used them on occasion to create throwaway images…pictures that aren’t the focus of a post, but might relate to the subject at hand. The image at the top of this post is an example.

Of course, we’ve all been hearing about the massive amounts of time and money being thrown at AI by virtually every major — and more than a few minor — tech company on the planet, and the project that’s getting the most publicity is ChatGPT, developed by OpenAI, the same research lab that developed the aforementioned DALL-E 2. GPT is the acronym for generative pre-trained transformer, which in my Neanderthalic understanding refers to a computer system that, based on scanning billions of words collected from the internet and other sources, attempts to (1) understand those words and then (b) use its understanding to predict what comes next in a “conversation.” It’s an iterative process whereby the system theoretically learns as it goes along by interacting with humans.

That’s where the “Chat” part of the name comes in. If you’ve ever used the chat feature on a company’s website to get tech or customer support, you’ve likely begun the conversation with a chatbot, a computer program designed to figure out what your question or problem is and where to direct you to get it addressed. Chances are you finally end up texting back and forth with a real human, because those chatbots are actually pretty dumb…they’re the machine equivalent of a switchboard operator (and I mean no offense to actual switchboard operators, if any still exist).

But these new chatbots, such as ChatGPT, supposedly get smarter as they continue to “converse” with people and pull in additional data from the internet. I say “supposedly,” because I remain unconvinced of the reliability of these services, given that they’re “learning” from massive numbers of humans, and you could fill a galaxy with the amount of biases and misconceptions and downright lies that the average human-type person can generate.

For example, I can tell ChatGPT that I’m an expert in quantum mechanics and it wouldn’t be able to know for sure that I’m lying. Fortunately, it’s sufficiently self-aware (?) to recognize this limitation, because here’s what happened when I did just that:

I probed this line of reasoning a bit further:

Now, one might logically wonder why my credentials wouldn’t be suspect if ChatGPT has really consumed an estimated “hundreds of billions” of words from the internet and other sources. I mean, I’ve published hundreds of thousands of words over the past twenty years, and not a single one of them offered even the slightest indication that I know squat about quantum mechanics. [Ed. — Or anything else, for that matter.] So, I asked the chatbot if it knew about the Fire Ant Gazette. Its answer was pretty fascinating to me*:

There are a couple of factual errors in that response, and I decided to test the chatbot’s willingness/ability to correct them:

It’s safe to assume that ChatGPT is not omniscient, and in fairness, its creators admit that right up front. Here’s the disclaimer that appears the first time you log in:

When I decided to write about ChatGPT, I wanted to approach it from the perspective of using it to generate the full text of a blog post, to see how it might turn out. I’m sure you’ve read accounts of students using it to write their term papers, or employees using it to create their reports to management, or even jobseekers using it to construct/fabricate their résumés. I assumed that these people just input something like write a 1,000 word report on how chaos theory relates or informs the decision of pro football teams to go for a first down on fourth down in their own territory, or something equally descriptive.

So I requested ChatGPT write 500 word blog post about snakes in Horseshoe Bay, Texas, in the style of a typical post found on www.ericsiegmund/fireant. The results were well-written and on-topic, but they contained a number of inaccuracies or non-useful information (e.g. rattlesnakes can range in color from light brown to dark brown). Also, while the chatbot honored the 500 word limitation, it did so in the middle of a sentence. I later learned that the free version has a 500 word limitation, and the bot apparently doesn’t know how to style its results to fit neatly into that box.

Based on these disappointing results, I initially entitled this post “ChatGPT: Hard Pass.” I confess that this was due to my misunderstanding of what ChatGPT is and how it’s designed to work. The “chat” portion of its name should have been a clue, but I’ve never professed to not be clueless.

ChatGPT is most useful when you enter into a dialog with it…a question and answer session that not only provides you with information you need, but also informs and guides the bot, which hopefully results in additional building blocks for whatever problem you’re attempting to solve, or question you’re attempting to answer, or term paper you’re attempting to cheat on.

At the risk of oversimplifying things more than I already have throughout this post, ChatGPT (and probably all the other pretenders to the throne) is a tool, and its usefulness depends to some extent on the skill and knowledge — even if that knowledge is limited to knowing what questions to ask — of the wielder. That’s why I’ve changed the post title to something more neutral.

Nevertheless, I doubt that you’ll ever read anything on the Gazette that was generated by ChatGPT…but I can’t guarantee that at some point, I’ll find it useful for research purposes.

I’m now seriously considering a post about quantum mechanics…

*An hour or so after this conversation, I posed a follow-up question to ChatGPT: What can you tell me about a blogger named Eric Siegmund? Its response was humbling and also evidence of some limitations in its learning skills: I’m sorry, but I don’t have any information about a specific blogger named Eric Siegmund. There are millions of bloggers around the world, and it’s possible that Eric Siegmund is one of them. And, BTW, if in the future you run across the term “VanityAIChatting,” well…you heard it here first. [Return]