Six Rounds of Independent Testing: Hardening the Sensbench AI Builder

Over the past week, an independent third party ran six rounds of deep, adversarial testing against the Sensbench AI Builder and the Sensbench MCP (Model Context Protocol) server. Their brief was simple: try to break it. They generated projects, compiled firmware, re-ran identical prompts, poked at caches, wired up real MQTT brokers, and audited our billing. Here is what happened, what we fixed, and what the numbers look like now.

How the tests were run

Every claim below was verified in a real environment:

• Real compilation — every generated sketch was compiled on our actual arduino-esp32 toolchain (ESP-IDF build), not a linter or a fake pass.
• Real MQTT — generated Home Assistant sketches were flashed into the simulator, connected to a real public MQTT broker, and driven end-to-end: mosquitto_pub ON → device callback → relay pulse → state published back. Discovery messages were captured on homeassistant/switch/…/config.
• Cache semantics — identical prompts, semantically different prompts, board changes and constraint changes were replayed to verify that cached answers only ever match the exact request that produced them.
• Billing reconciliation — every generation and compile was reconciled against the quota ledger: successes charged, cache hits free, failed compiles free.

Round one: the honest number was 20%

The first round was humbling. Out of five generated projects, four failed to compile — including a garage-door opener, one of the most common real-world requests. The failures were classic beginner C++ mistakes: helper functions used before declaration, hand-rolled protocol stacks, a missing-library include. The verdict was fair: “the pipeline is real, but the generator is unreliable.”

What we changed

Rather than argue, we built guardrails into the pipeline itself:

• Automatic prototype injection — the compile toolchain does not run Arduino’s auto-prototype step, so we now inject function prototypes at the correct position automatically before every compile.
• Internal self-check compile — every generated project is compiled internally before it is ever shown to a user. If it fails, the compiler error is fed back and the model regenerates, up to twice. Users only ever see verified results.
• Mature libraries over hand-rolled stacks — MQTT now uses the battle-tested PubSubClient library; the model is explicitly forbidden from re-implementing the MQTT protocol from scratch.
• Requirement-completeness checks — compiling is not enough. If the prompt asks for Home Assistant MQTT, the code must actually contain a working MQTT client with subscribe/publish/callback logic — a local-only demo dressed up as the real thing is rejected and regenerated.
• Verified-only caching — the answer cache only ever stores projects that passed both compile and requirement checks. Repeating a popular prompt always returns the verified build.
• Simulator-reality rules — generated sketches now default to the simulator’s real access point name, a reachable default MQTT broker, bidirectional ON/OFF handling, retained initial state for Home Assistant, and exact sensor types (a reed switch stays a reed switch).

The numbers now

• Initial pass rate on real-world requests: 1 in 6.
• Pass rate after the fixes, across repeated independent rounds: 6 of 6 consecutive compiles + real MQTT logic — including cache-hit responses.
• Cache semantic testing: 6 requests, no mismatches. Identical prompts return byte-identical code; changed prompts, boards, or constraints always generate fresh.
• End-to-end MQTT: device → broker → Home Assistant-style subscription → command → relay pulse → state publish, all confirmed over a real broker, including authenticated brokers.
• MCP server: 8/8 tools exercised end-to-end; quota billing reconciled exactly (success charged, cache hits and failures free).

What we are still open about

Honest limitations we documented during testing:

• Home Assistant integration is manual MQTT by default (discovery config is published, but HA auto-discovery setup depends on your HA MQTT configuration).
• The simulator sandbox has an outbound allowlist — some public brokers are reachable (e.g. broker.emqx.io), others are not. Check our docs for what works.
• Hardware selection matches the request semantically, not always to the letter — the model may pick an equivalent sensor class; we flag this in the generated plan’s notes.

Why publish this

Anyone can publish a demo video. We chose instead to publish the full arc: the failing rounds, the fixes, and the re-tests — with reproducible methodology. If you want to try to break it yourself, the AI Builder is free for one build a day, and the simulator runs entirely in your browser. The MCP server is available with real quota accounting. Bring your own prompt, your own broker, your own skepticism.

— The Sensbench team