Apple’s Image Recognition AI Could Use Some Work

Howdy, partners! I’m your friendly neighborhood Texas cowboy, and I’m here to talk to you about the wild world of artificial intelligence. Now, I know what you’re thinking – AI ain’t exactly the kind of thing you’d expect to hear a cowpoke like me talkin’ about. But let me tell you, this stuff is fascinating! It’s like a whole new frontier, full of possibilities and challenges. In this blog post, we’ll be takin’ a closer look at some of the latest developments in the field of AI, and how it’s changin’ the way we live and work. So put on your boots, grab your hat, and let’s ride into the future together!

You may have heard about ChatGPT, the AI-powered text generator from OpenAI, the same outfit that brought us DALL-E 2, the text-to-image generator that I reviewed a while back. I used ChatGPT to generate the lame-ish opening paragraph above by asking it to write an opening paragraph in the style and with the vernacular of a Texas cowboy for a post about artificial intelligence. [I have no idea why it assumed that I am a Texas cowboy and thus used first-person styling, but there ya go, podnuh.]

The results are actually pretty decent, so I’m contemplating putting this blog on autopilot from here on out, and letting a machine do the work for me while I concentrate on perfecting my napping technique. Although it would be pretty depressing if the posts turn out to be more thoughtful, engaging, and grammatically correct than mine. [Ed. You really think that’s a bridge too far?]

Not every AI application is as successful. It’s kind of surprising to me that one which is falling a bit short comes from Apple, via its image recognition feature built into its Photos app (for both MacOS and iOS devices).

You may not be familiar with that feature or use it that often; if not, here’s a pretty good primer on how it works…or should work. Note that this introduction focuses on the iOS version, but the desktop version is essentially the same.

If you’re too busy with your fantasy bocce ball league machinations to read the article to which I’ve painstakingly linked, here’s a Q&D overview. When you browse your Photos library, you may come across a picture where the circled “I” (for “Information”) icon has a couple of tiny stars next to it, as shown below.

When you click or tap on the icon, a pop-up appears that shows information (duh) about the photo, and also a new icon overlaying a part of the image, along with an option to look up something about that part of the image. In the example below, Photos is telling me that it recognizes the big honkin’ tree, and gives me the option to look it up. In this case, clicking on the “look up” option will tell me that the tree in question is a red maple. I have no reason to doubt the veracity of that claim, not being Paul Bunyan or Neil Sperry.

So far, so good. And Apple’s ID bot doesn’t limit itself to plants. It also purports to identify animals, birds, insects, book covers, famous artwork (in case you can’t remember what the Mona Lisa looks like), and well-known places or landmarks, like Amarillo’s Cadillac Ranch. Well, I’m just guessing at that last one; if you have a photo in your collection, check it out and let me know.

In reality, the image recognition AI falls a wee bit short in the reliability/accuracy realms, and some of its failings are amusing.

Sometimes, you’ll run across a photo with the starred “I” icon, but there’s nothing in the photo that’s flagged to look up. Other times, the flagged portion of the image is completely illogical.

Take the following photo, for example. This is a portion of an image I created in Photoshop, using a photo of an actual ant mound in which I added smoke plumes rising from the holes. The tiny icon that Apple’s AI added indicates that it thinks it’s a bird. But it gets better; rather than attempting to identify this “bird,” the AI presents a half dozen additional photos which I assume it thinks will help me know more about the “bird,” and one of them is a photo of the feet of a French poodle.

Occasionally, a photo where the primary subject is a very recognizable animal, the AI ignores it completely, perhaps thinking, “only an idiot won’t know what this is; I have better things to do.” Here’s an example:

I mentioned above that the Photos AI purports to identify book covers, which, now that I think about it, seems like a dumb feature. I mean, won’t the cover of the book tell you what book it is? Anyway, maybe it works…and maybe it doesn’t. Here’s a photo of a plaque that’s mounted at our local Nature Park; Debbie and I made a nominal donation to its development and this is a very nice bit of recognition. But it’s not a book, regardless of what Apple says.

That’s not to say that it doesn’t do its best to match it up to an actual book…and the book it finds is, well, interesting.

Incidentally, the description attached to this book on openlibrary.org is intriguing enough that I might actually have to read it: Two boys plan to catch the watersnake that lives in the pond for their science project, but the snake has other ideas.

Last, and possibly least, here’s photo that you might, in your limited understanding of the natural world, identify as a section of concrete at a construction site, but which in reality — as defined by Apple’s AI — contains a striking image of a great white egret.

I could go on and on — the mountain climber identified as “la cucaracha”; the blank space on our under-construction deck identified as an Indian “neem tree”; the paving stone flagged as a type of moth (even though the bird icon is employed), while the nearby lizard remains completely anonymous; a pergola support post shown to be a pomegranate tree; a back porch door ID’d as a Japanese snowbell.

There were a few wins. The corgi that visited us was properly identified, and most of the felines in the recent cat meme post were correctly-if-not-helpfully shown as “cat.” Oh, and one of our Japanese maples was properly identified…a huge victory.

One problem for Apple, as I see it, is that their focus on privacy and security limits their ability to get the user feedback that would allow the AI to learn with use. The makers of the text generation and text-to-image AIs state clearly that anything we input into the front end of their machine will become data points for improving their products.

For now, Apple’s image recognition feature in its Photos app remains an occasionally amusing oddity, as far as I’m concerned. YMMV.