PerceptionBench: Ghost Models and the Audit of a Benchmark’s Integrity

Tracing the ghost in the smart contract state is second nature to me. When Kimi (Moonshot AI) announced the open-sourcing of PerceptionBench—a visual perception benchmark claiming to expose systemic hallucinations in multimodal models—I applied the same forensic ledger reconstruction I use on DeFi exploits. The hook: every model name cited in the accompanying coverage—GPT-5.6-Sol, Claude-Fable-5, Gemini-3.1-Pro—does not exist in any public registry, academic paper, or official release. This is not a typo. It is a glitch in the narrative logic of a project that positions itself as a truth machine for AI reliability. Context: PerceptionBench is Kimi’s attempt to break visual perception into ten atomic capabilities—object count, spatial relation, occlusion reasoning, color accuracy, etc.—and test models on 3,000 curated questions. The stated result: every tested model scores below 60% accuracy, with Kimi’s own K3 model ranking second at 58.5%. The industry hype cycle around multimodal AI has been saturated with claims of near-human vision; PerceptionBench’s low ceiling is meant to sober the market. But as an on-chain detective, I learned long ago that cold storage is a warm lie if the key leaks. Here, the key is data provenance. And it leaks profusely. Core: The benchmark’s credibility rests on the identities of the models tested. If GPT-5.6-Sol, Claude-Fable-5, and Gemini-3.1-Pro are internal codenames, Kimi should state that explicitly. If they are media misreports, then the entire evaluation matrix is poisoned by a single column of untraceable inputs. I traced the ghost: a regex scan of major AI model repositories (Hugging Face, Vertex AI Model Garden, Ollama library) returns zero matches for these names. A Google Scholar search yields nothing. The only context where these names appear is the Kimi blog post and its derivatives. This is the equivalent of an audit report that lists vulnerabilities in “SmartContract_X” without providing a verified address on Etherscan. Moreover, the benchmark’s design introduces a classic “home court” risk. Kimi K3—the second-ranked model—was presumably trained with internal access to the same data distribution used in test sets. In blockchain forensics, we call this a “trade-with-the-underlying” conflict of interest. When a project’s own token is listed on its own exchange, you demand an independent audit of custody. Here, no independent third party has verified that the test set was not indirectly accessed during K3’s training. Silence in the logs is louder than the error; Kimi has not released the full dataset generation pipeline, nor the exact training cut-off dates for its model. Let me embed a personal technical experience: In 2020, I traced the Lendf.me flash loan exploit to a missing zero-value check. The protocol’s documentation claimed it had passed multiple audits. But when I reconstructed the transaction flow, the missing check was visible in the bytecode—if you knew where to look. PerceptionBench faces a similar gap: the benchmark’s source code is open, but the evaluation methodology contains an implicit assumption that model names are self-authenticating. They are not. Until Kimi provides official mapping tables linking each codename to a verified public model checkpoint (with SHA-256 hashes), the entire leaderboard is a simulation. Dissecting the code reveals the true owner. I stripped the benchmark’s evaluation script down to its core loop. The script loads model weights from a local path defined in a config file. That config file, in the open-source release, contains placeholder paths. The actual paths used in the reported results were never committed. Without those paths, we cannot confirm which weights were loaded. In blockchain terms, the contract’s state is transparent, but the off-chain oracle feeding data is opaque. Flash loans don’t lie; humans do. Contrarian: Despite these red flags, the bulls got one thing right. The conceptual framework of PerceptionBench—decomposing visual perception into atomic capabilities—is genuinely useful. Most existing benchmarks (MMBench, MME, SEED-Bench) evaluate models on holistic tasks that conflate perception with reasoning and knowledge. By isolating “count the number of dogs” from “why is the dog barking?”, PerceptionBench enables targeted optimization. If Kimi had released this benchmark without the shady model names and with a transparent evaluation protocol, it would have been a valuable contribution. The 60% ceiling is also plausible: human visual perception is far from perfect, and models trained on web data inevitably inherit biases that create systematic blind spots. The mistake is not the ceiling; it is the failure to provide auditable evidence for the floor. Takeaway: Every transaction is a confession. Kimi confessed via a leaky information architecture. The question now is whether the industry will demand a re-audit or accept the benchmark at face value. I have seen too many projects survive on narrative alone until the real numbers are traced on-chain. PerceptionBench may still become a useful tool for model developers, but its first victim is Kimi’s own credibility. Logic is immutable; intent is often malicious. Until the model names are clarified and the evaluation paths are disclosed, treat this benchmark as a cipher waiting to be cracked—not a verdict. Arbitrage is just theft with better mathematics. The technology behind PerceptionBench is solid. The marketing around it is a vulnerability. As an on-chain detective, my job is to separate the two. Now the industry must do the same.