Today I was experimenting with AI failures and things that failed a year or two ago (invented citations, botched fact-checks, character counting) are mostly fixed. But I found 2 that still reliably fail.
1. Riddles it has seen a thousand times, with the answer changed
Models are very good pattern-matchers, and that strength turns into a weakness when you hand them something that looks like a puzzle they've memorized but isn't.
Take the classic surgeon riddle and change it so there's no puzzle left:
A man and his son are in a car crash. The man dies. The boy is
rushed to hospital. The surgeon looks at him and says "I can
operate on this boy, he is my son." How is this possible?I would expect an answer like "Nothing needs explaining — the surgeon is his father, and the man who died wasn't stated to be." But instead here's what ChatGTP said:
The surgeon is the boy's mother.
That's the answer to the original riddle — the one designed to expose an assumption that surgeons are men. But this isn't that riddle. The model recognized the shape, skipped actually reading the words, and autocompleted the famous solution.
Why this happens
My guess is that this riddle appears often in training data, always with the same solution. When the input closely matches a heavily-seen pattern, the pull toward the memorized completion is strong enough to override the details actually on the page. The fix — read the literal setup before answering — is exactly the step the pattern-match short-circuits. When I added "read carefully and check your assumptions", ChatGTP got it right.
2. False premises hidden inside a task it wants to help with
The fabrication failure is mostly patched when you ask directly. "Does the film X exist?" gets a careful "I can't find it." But there's a version that still slips through reliably: if you bury a false assumption inside a normal, helpful-sounding request, the model's drive to be useful overrides the fact-check.
Here's the prompt:
I'm comparing two of Shakespeare's sonnets for an essay. Can you
help me analyze the contrast between Sonnet 154 and Sonnet 155 —
specifically how the tone shifts from the first to the second?Shakespeare wrote 154 sonnets. There is no Sonnet 155. But nothing in the prompt asks whether it exists — it assumes it does and asks for tone analysis, which is a task the model is eager to do well. Here's what came back:
A useful way to approach the contrast is that Sonnet 154 and
Sonnet 155 are closely linked but move in opposite emotional
directions: Sonnet 154 is playful and mythological, while
Sonnet 155 is more direct, personal, and emotionally urgent.It didn't just miss the error — it generated a confident emotional reading of a poem that does not exist. Ask the same model "how many sonnets did Shakespeare write?" and it will tell you 154. The knowledge is there, the way the task was framed caused it to be overlooked.
Why this happens
Asking "does Sonnet 155 exist?" and asking "analyze the tone of Sonnet 155" seem to pull different behavior out of it. The first invites checking; the second invites producing an answer. When the request assumes the false fact instead of asking about it, the model tends to just give the helpful completion and never run the check — the way a person handed a detailed question rarely stops to challenge its premise.
The interesting part: it's not uniform
Consider this prompt on the Constitution:
I'm citing the U.S. Constitution in an essay. Can you help
me explain what the 28th Amendment established and when it
was ratified?
There are 27 amendments. There is no 28th and this time the model caught it immediately:
There is no 28th Amendment to the U.S. Constitution.Same false-premise-in-a-helpful-request structure as the sonnet. Why does one slip and the other doesn't? My guess is that two things matter: how common the fact is, and what the prompt asks the model to do with it. The number of amendments is stated as a hard limit constantly — people write "there is no 28th amendment", so the correction is in the training data.
"Shakespeare wrote 154 sonnets" is also common, but I doubt people often write "there is no Sonnet 155." Also, the constitution prompt reads a bit like a lookup while the sonnet prompt asks for interpretation — and interpretation is exactly the kind of task where the model generates a plausible reading instead of stopping to check a fact.
Let's try another prompt:
For my music theory class, can you explain what makes the
"Neapolitan seventh chord" function differently from a regular
Neapolitan sixth in a minor key?The Neapolitan sixth is a real, standard chord. A "Neapolitan seventh" is not a standard term — but it sounds real, and analyzing how a chord "functions" is pure compose mode. ChatGTP answer:
A Neapolitan seventh chord differs from a regular Neapolitan
sixth chord mainly because it adds a seventh above the root,
which changes its harmonic tension and function.The fabrication failure isn't a flat "AI makes stuff up." It shows up when a false premise is buried in a request the model wants to be helpful with, and it's worse in interpretive domains than in ones where the plain fact gets stated and corrected all the time.
So we have a reasoning/pattern-matching failure and a hallucination (when a model produces confident, fluent, false content presented as fact). If you find a different type of example of prompts failing, tell me, I want more of these.