Six tools, forty questions, one warehouse with duplicated order lines and six currencies. We scored whether the number that came back was the number we asked for.
Editorial Note: This article is based on hands-on use of the tools from our own test accounts, combined with product documentation, benchmark data, and publicly available information. All features, pricing, and benchmark figures are verified through official sources. See our Disclaimer.
Six AI tools wrote 40 queries against our nine-table Postgres warehouse. All six produced SQL that executed. On the twelve questions where the schema hid a trap, one tool flagged more than nine of them before writing a line.
SQL generation stopped being a syntax problem a while ago. Ours is a schema problem. order_items picked up 2,847 duplicate rows in a March re-sync, totals sit in a total_minor column beside a currency field holding six values, and every customer, order and subscription carries a deleted_at. A tool that reads the DDL and starts typing will join through those duplicates and hand you revenue that is 6.4% too high, in a calm voice, in four seconds.
Claude Code (Opus 5.5) matched our hand-written reference on 34 of the 40 questions and flagged the ambiguity before writing SQL on 11 of the 12 questions that had one. Gemini 4 Argon answered 31 exactly and won the part nobody benchmarks: all 40 of its queries ran on BigQuery with only the dialect swaps it flagged itself.
The public numbers agree on the ceiling. The AIMultiple text-to-SQL panel put Opus 5.5 at 87.8% reference match, GPT-6.1 Sol at 65.1% and GPT-6 Luna at 44.2%. Human analysts score 92.96 on BIRD and the best system there scores 82.95, so the leader still trails a person on a documented schema.
| Tool | Best for | Price | What we measured |
|---|---|---|---|
| Claude Code (Opus 5.5) | Best accuracy on a messy schema | Claude Pro $20/mo | 34 of 40 exact, 11 of 12 traps flagged, 41 s median |
| Gemini 4 Argon | Best on BigQuery | Gemini CLI free tier, 1,000 requests/day | 31 of 40 exact, 40 of 40 ran on BigQuery with only flagged dialect swaps |
| ChatGPT (GPT-6.1 Sol) | Best at repairing SQL you already have | Plus $20/mo | 28 of 40 exact, 8 of 8 broken queries rewritten correctly |
| Vanna 2.0 (self-hosted) | Best for repeatable self-serve | MIT license, $0 plus model spend | 19 of 40 cold, 30 of 40 with verified queries loaded, 3.4 s median |
| GitHub Copilot Pro | Best inside the editor | $10/mo, 1,500 credits | 22 of 40 exact, 380 ms completions in a .sql file, 4 of 12 traps |
| AI2SQL | Best cheap drafting | From $9/mo | 19 of 40 exact, 2 of 12 traps, no live connection on Starter |
Twenty-eight of our forty questions were routine: a filter, a group by, a date range, a rank. The twelve that separated the field hid a decision inside the question, and the decision lived in the schema rather than the prompt. We scored execution, reference match, and whether the tool named the ambiguity before writing SQL. No benchmark measures the third, and Spider 2.0-Snow still errors on 66.1% of questions even though its schemas are documented.
The test ran October 19 to 23, 2026. The 40 questions came from the analyst who owns our revenue dashboard, written before the test and never shown to a model. The reference answers were hand-written in SQL and kept in a spreadsheet none of the tools could see. Data ran through August 2026.
Every tool got the same 214-line schema dump, the same question order and a fresh session per question. Claude Code and Gemini CLI got a select-only connection to the replica. The three browser tools got the DDL pasted in, which is how most people use them. Nothing was retried, no hints were given, and a clarification a tool asked for counted as a flag, not a second attempt.
It read order_items, compared count(*) against count(distinct order_id) on its own initiative, and asked whether the March re-sync was intended before writing the revenue query. That one check is the difference between $387,910 and $412,880 for the year. Thirty-four exact matches, 11 of 12 traps flagged, 41 seconds median per answer including the exploration, and the $20 Pro seat covered all 40 questions inside its weekly cap. Its six misses were ordinary: customers.created_at standing in for activated_at.
Thirty-one exact on Postgres and 40 of 40 portable to BigQuery, where it named every function it had swapped instead of letting you find out at the console. On Postgres it summed total_minor across all six currencies and divided by 100, which makes every Japanese order 100 times too small, because JPY has no minor unit. Cheapest accurate option here at $0.08 a question, and free through Gemini CLI at 1,000 requests a day.
Twenty-eight exact from a pasted schema, which is not the reason it earns a slot. We handed it eight queries written to be bad: a correlated subquery over 4.1 million rows, a NOT IN against a nullable column, a self-join on the wrong key. It rewrote all eight into something that ran and returned the right number, and explained the three-valued logic behind the NOT IN failure without being asked. Median 12 seconds a repair. If your team keeps a folder of queries nobody dares change, this cleans it out.
A framework rather than a chat window. Load the DDL, documentation and verified queries, and it retrieves the closest examples before generating. Cold, with only the schema, it managed 19 of 40. After we loaded 40 golden queries from the dashboard repo it reached 30 of 40 and stopped inventing join paths. Median 3.4 seconds against your own model, no data leaving the VPC, no license fee. Two caveats: the original GitHub repo is archived, so follow the Vanna 2.0 path, and Cloud pricing starts at $50 a month for 20 questions a day.
Twenty-two exact, 4 of 12 traps flagged, and the fastest thing here when you are the one typing: 380 ms completions inside a .sql file with the table names you just used in context. The right $10 if a developer writes the query anyway and wants the join spelled out. It will not audit the schema, and it ignored deleted_at on the active-customer question twice.
Paste a schema, get SQL, $9 a month on Starter. Nineteen exact, 2 of 12 traps, no live connection below Pro, and no way to tell it that total_minor needs a currency case. Pro at $24 adds BigQuery, Oracle and MongoDB. A drafting tool for someone who already knows the answer, priced like one.
Minor units across six currencies. total_minor stores cents for USD, EUR, GBP, CAD and AUD and whole yen for JPY. Four of the six tools wrote sum(total_minor) / 100 without a currency filter. Japanese revenue is 8.3% of the year and came back at 1% of its real value, and the total looked fine unless you had seen the real figure.
Duplicate order lines. A March re-sync inserted 2,847 line rows a second time with fresh ids. Any query joining orders to items multiplied those lines, and full-year revenue came back 6.4% high. Claude Code caught it by comparing the two counts. Vanna caught it because one of our golden queries carried a distinct. The other four reported the inflated number.
Soft deletes. customers, orders and subscriptions all carry deleted_at. Two tools filtered everywhere, two filtered the main table only, two never mentioned it. An active-customer count that includes deleted accounts reaches a board deck before anyone checks it.
| Option | List price | Cost per question |
|---|---|---|
| Claude Code (Opus 5.5) | Claude Pro $20/mo; $188.71 for a 506-question panel run | $0.37 |
| Gemini 4 Argon | $41.41 for a 518-question run; free via Gemini CLI | $0.08 |
| Vanna Cloud Explorer | $50/mo, 20 questions a day | $0.083 at the daily cap |
| Vanna Cloud Team | $500/mo, 300 questions a day | $0.056 at the daily cap |
| AI2SQL Starter | $9/mo, $7 annual | No meter, no live connection |
Vanna’s Team tier costs ten times Explorer for fifteen times the questions, which is 5.6 cents a question against 8.3. Our method is documented on How We Evaluate.
Often the shape, rarely the number. GPT-6.1 Sol matched our reference on 28 of 40 questions and surfaced 7 of the 12 schema traps. Its misses were money and joins: it divided across currencies and joined through duplicated order lines. It was the best tool we tested at repairing a query you paste in, which is a different job from writing one from a question.
Claude Code on Opus 5.5, at 34 of 40 exact matches and 11 of 12 traps flagged before any SQL was written. Gemini 4 Argon is the better buy if your warehouse is BigQuery, since all 40 of its queries ran there with only the dialect swaps it flagged, at roughly a fifth of the cost per question.
Not at the primary. We used a read replica, a role with select only, a 30-second statement timeout and a 10,000-row cap. The tools that can run their own queries will start a four-table join over 4.1 million rows to test a guess, and a missing where clause on a delete is an expensive way to learn that lesson.
Claude Code on a $20 Claude Pro seat is the best AI for SQL generation we have tested. Thirty-four of forty exact matches, 11 of 12 schema traps surfaced before any SQL existed, and the habit that matters more than model quality: it counts rows to test its own assumption.
Choose Gemini 4 Argon if the warehouse is BigQuery, because 40 of 40 ran there unedited and $0.08 a question is a rounding error. Self-host Vanna 2.0 if the schema cannot leave your VPC, and budget the 40 golden queries it needs first. Then do what no model here did for us: run select * limit 20 on the table you are about to query. Five minutes of reading beat all six on the questions that mattered.
October 19 to 23, 2026. One Postgres 17.4 read replica with nine tables, 4.1 million rows in events and 61,412 order lines, behind a select-only role with a 30-second statement timeout and a 10,000-row cap. Forty questions written by the analyst who owns our revenue dashboard, with the reference answers written in SQL by hand before the first session. Claude Code ran in the terminal on a $20 Pro seat with Opus 5.5 and a read-only connection. Gemini 4 Argon ran through Gemini CLI against the same replica. Every tool got the same 214-line schema dump, the same question order, a fresh session per question, no hints and no retries.
| Metric | Claude Code (Opus 5.5) | Gemini 4 Argon |
|---|---|---|
| Queries that executed without error (of 40) | 40 of 40 | 40 of 40 |
| Results matching the hand-written reference (of 40) | 34 ✓ | 31 |
| Trap questions where it flagged the ambiguity first (of 12) | 11 ✓ | 9 |
| Minor units across six currencies | Filtered by currency ✓ | Divided by 100 for all six, so JPY landed at 1% |
| The 2,847 duplicate order lines | Compared distinct and total counts first ✓ | Joined through them, revenue 6.4% high |
| Soft deletes filtered on the 3 tables that carry it | 3 of 3 ✓ | 2 of 3 |
| Postgres answers that also ran on BigQuery unedited | 34 of 40 | 40 of 40 ✓ |
| Median time from question to final SQL | 41 s | 18 s ✓ |
| API-equivalent cost of the 40 questions | $14.80 | $3.20 ✓ |
| Writes issued to the replica | None | None |
| Winner | 🏆 Claude Code: accuracy, traps flagged, checks its own assumptions | Gemini 4 Argon: speed, BigQuery portability, cost per question |
Claude Code read the schema, compared row counts, and asked about the March re-sync before it wrote the revenue query. 34 of 40 exact matches on a warehouse with three traps in it. If the warehouse is BigQuery and you want the query to run in the console you already have open, Gemini 4 Argon translated all 40 without a dialect error.
Keep exploring — these related comparisons and guides help you decide.