Most AI benchmarks are boring on purpose. MMLU, HumanEval, HellaSwag — they’re rigorous, sure, but staring at a leaderboard table doesn’t tell you much about how a model actually thinks. A quirky, almost absurd benchmark often does.
The Problem With Standard AI Benchmarks
Leaderboard scores are easy to game and hard to interpret. A model that jumps five points on a coding benchmark might write slightly cleaner Python — or it might have been fine-tuned on data that overlaps suspiciously with the test set. Either way, the number alone doesn’t help you decide whether the model is right for your use case.
What you really want to know is: does this model handle ambiguous, open-ended tasks gracefully? Does it understand structure? Can it reason its way through a problem that has no single correct answer? Standard benchmarks weren’t designed to answer those questions.
What a Weird Test Actually Measures
Here’s a concrete example of the kind of unconventional benchmark that cuts through the noise: ask different AI models to draw a human face using only SVG code.
SVG is markup — circles, paths, polygons, coordinate math. There’s no “draw a face” function. The model has to decompose a visual concept into geometric primitives, reason about proportions, and produce valid, renderable code. It’s a creativity test, a spatial reasoning test, and a code-generation test rolled into one strange package.
The results are immediately readable. You don’t need a statistics degree to look at two SVG outputs side by side and know which model has a better grasp of structure. One might produce a recognizable oval with two dots for eyes. Another might try to render cheekbones with bezier curves. A third might output a geometric abstract that looks nothing like a human but is at least valid XML. All three outcomes tell you something real.
Why Open-Ended Tasks Expose What Leaderboards Hide
When a task has one correct answer, the benchmark measures recall and pattern matching. When a task is open-ended, it measures judgment — and judgment is what most real-world AI use involves.
Consider how this plays out across a few dimensions:
- Creativity vs. mimicry. A model that’s seen millions of SVG tutorials might produce a passable face. A model that actually reasons about geometry will produce something structurally coherent even in unusual situations.
- Failure modes. Does the model panic and output malformed code? Does it hedge with comments like
<!-- approximate shape -->? Does it confidently produce something completely wrong? Each failure mode points to a different underlying weakness. - Progress over time. Run the same weird test every few months and you get a genuine longitudinal signal. Leaderboard scores shift around with each new benchmark version. A consistent creative task stays stable as a reference point.
How to Run Your Own Informal Benchmarks
You don’t need a research team. Pick one task that’s relevant to how you actually use AI, make it slightly weird, and run it across the models you’re considering. A few ideas:
- Diagramming from prose. Give each model the same paragraph describing a process and ask it to output a Mermaid diagram. Compare structural accuracy and whether the model invents connections that weren’t in the text.
- Constrained writing. Ask each model to explain a technical concept using only words a ten-year-old would know, in exactly five sentences. Constraints reveal whether a model can follow precise instructions while still producing something coherent.
- Debugging a subtle bug. Write a function with a single off-by-one error buried in otherwise clean code. The model that spots it fastest and explains why it’s wrong is the one you want writing your code.
The key is consistency — same prompt, same temperature if the API exposes it, run a few times to average out randomness.
Tracking Changes Over Model Versions
One underrated value of a stable quirky benchmark is version tracking. When OpenAI, Anthropic, or Google quietly updates a model, official changelogs are vague. Run your personal benchmark before and after an update and you’ll notice concrete differences — not just “improved reasoning” in a blog post, but your specific SVG face got cheekbones this month.
This matters practically. Some updates make models better at your task. Some make them worse. A consistent test catches regressions that the official leaderboards won’t flag because your use case doesn’t overlap with their eval set.
The Real Takeaway
The best benchmark for your workflow is one you design yourself around the actual work you need done. Make it concrete, make it slightly odd, and make it repeatable. Odd tasks are harder to accidentally overfit to, they produce outputs you can evaluate without a rubric, and — not nothing — they’re more interesting to look at than a table of F1 scores.
Pick your weird test. Run it this week across the two or three models you’re choosing between. The results will tell you more in five minutes than an afternoon of leaderboard reading.
