AI for Data Analysis: What It Gets Right, Where It Breaks, and How to Check Its Numbers

AI for data analysis is the use of machine learning and large language models to prepare data, write and run queries or code, summarise results, flag anomalies and produce forecasts, usually through a plain-English interface. You ask a question about your data, and the system decides which tables to read, computes an answer and explains it.

AI analysts return fluent answers whether or not they are right. This guide puts numbers on that gap using published text-to-SQL and data-agent benchmarks, shows where accuracy holds and where it falls apart, gives five checks for an AI analyst's output and works through a plausible ROAS answer that turns out to be wrong.

What is AI for data analysis?

Today the term usually means a large language model (LLM) connected to a database, a spreadsheet or a Python environment, doing part of an analyst's job.

What the term covers

Six jobs sit under the label.

• Code generation. The model writes Python, SQL or spreadsheet formulas from a plain-English request, and often runs them.
• Natural-language querying and text-to-SQL. A question such as "what were sales by region last month?" becomes a database query.
• Summarisation. A result table or a report becomes prose.
• Anomaly detection. A statistical method flags values that break from the expected pattern.
• Forecasting. A time-series model projects a metric forward.
• Conversational analytics. A chat interface on a business intelligence layer that handles follow-up questions.

These jobs do not suit an LLM equally. The generative layer mostly writes code and words. Anomaly detection and forecasting are statistical jobs. An LLM can call the method and describe the output, but the estimate should come from a model built for it.

AI data analyst versus a data analyst using AI

An AI data analyst is a software agent that takes a question and returns an analysed answer, choosing the data, running the query and writing the conclusion. A data analyst using AI delegates pieces of that chain and keeps the judgement, including the check on each number before it reaches a decision.

Agentic AI for data analysis goes further. An agent plans a sequence of steps, queries the data, inspects intermediate results and iterates.

Conversational analytics, defined

Conversational analytics means asking a business intelligence layer questions in plain English and getting charts and tables back. The better implementations sit on governed metric definitions rather than raw tables. Google's documentation for Conversational Analytics in Looker describes its version as grounded in the Looker semantic layer.

How AI analyses data, from question to answer

Most tools walk a question through the same five stages.

Stage	What the AI does	What can go wrong
1. Interpret the question	Decides which metric and period are meant	"Revenue" read as gross, not net; "last quarter" as the last 90 days
2. Link to the schema	Picks tables, columns and joins	A wrong or stale table, or a join that duplicates rows
3. Generate code	Writes SQL or Python	Code runs but filters the wrong dates or drops a segment
4. Execute and read	Runs the code and reads the output	Results truncated, sampled or read from the wrong column
5. Write the narrative	Describes the result in prose	Rounds, embellishes or states a number the output does not contain

Syntax errors and failed runs announce themselves, while the wrong metric, join or filter, or a narrative that drifts from the output, returns a clean answer with the same confidence as a correct one.

What AI for data analysis gets right, with evidence

Simple, well-specified queries

On clean schemas with clear questions, text-to-SQL is close to solved. In the Spider 2.0 paper presented at ICLR 2025, OpenAI's o1-preview solved 91.2% of tasks on Spider 1.0, a widely used academic text-to-SQL benchmark. A question that maps to one table, one metric and one filter will usually get the right query.

Speed on tasks inside the frontier

In a pre-registered field experiment with 758 consultants at Boston Consulting Group, Dell'Acqua and colleagues, in a Harvard Business School working paper with co-authors from Wharton, Warwick Business School, MIT Sloan and BCG, found that consultants using GPT-4 on tasks inside the AI's capability frontier completed 12.2% more tasks and finished them 25.1% faster, with significantly higher quality. The authors call the frontier jagged because tasks of similar apparent difficulty fall on different sides of it. The study did not test analysis tasks directly, but the same logic suggests the safest gains in analysis come from bounded tasks such as drafting a query or charting a table.

Code you can read

OpenAI's guidance on data analysis says ChatGPT writes and runs the Python for you, and tells users to review the generated code and output tables. Visible code matters because code can be checked line by line and a narrative cannot.

Summarising results it has been given

Turning a result table into a paragraph is a narrower job than computing it, and the easiest output to check, because every figure in the paragraph can be matched against the table. A summary can still round a figure, merge segments or add a cause the table does not show, so check every number in it.

Where AI for data analysis fails: the benchmark evidence

Several of the studies below tested models that have since been superseded, so read the figures for the pattern they show rather than their exact level.

Same model, harder databases

Lei and colleagues report o1-preview results on three benchmarks of increasing realism. It solved 91.2% of Spider 1.0 tasks and 73.0% of BIRD tasks, while a code agent framework built on o1-preview solved 21.3% of Spider 2.0 tasks. Spider 2.0 contains 632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases, on databases that often have more than 1,000 columns. Its tasks are also longer, often needing several SQL queries that exceed 100 lines, so the drop reflects harder workflows as well as messier data.

The leaderboards have moved, and what they leave out

Specialised systems have since closed much of the gap on BIRD. On the BIRD execution accuracy leaderboard, checked on 18 October 2026, the top test score is 82.95% (GrainSQL, 7 September 2026) against human performance of 92.96%. Two caveats apply. The leading entries use oracle knowledge, meaning the benchmark supplies expert hints about the data that your business would have to write down itself. And a leaderboard top is an engineered system tuned for that benchmark, not a general chat assistant pointed at your warehouse. Even with the hints, the best system sits about 10 points below people.

Multi-step analysis is harder still

DABstep, built by authors from Adyen and Hugging Face from more than 450 real challenges on a financial analytics platform, tests agents on multi-step problems. In the paper's original evaluation, the best agent reached only 14.55% accuracy on the hardest tasks. Later agents score higher, with Google's DS-STAR reporting 45.2% accuracy on DABStep in 2025, which is still under half. Each step is another chance to pick the wrong table, misread an intermediate result or carry an earlier error forward.

Numbers pulled from documents

FinanceBench is a set of 10,231 questions about publicly traded companies. Tested on a 150-question sample, GPT-4-Turbo used with a retrieval system answered incorrectly or refused to answer 81% of them. Adding retrieval did not make the model a dependable reader of financial documents.

Outside the frontier, AI made people less accurate

The same BCG experiment that found the speed gain also found its cost. On a task selected to sit outside the AI's frontier, consultants using AI were 19 percentage points less likely to produce correct solutions than those working without it.

Accuracy falls as schemas get messier, as questions need more steps and as definitions become specific to one business, which is the normal state of marketing data.

Why AI produces plausible numbers that are wrong

Google's documentation for Looker Conversational Analytics warns that Gemini "can generate output that seems plausible but is factually incorrect", and recommends validating all output before use. Wrong answers from an AI analyst rarely look wrong, and the same seven causes recur.

1. Ambiguous metric definitions. Gross or net revenue, platform-reported or modelled conversions. The model picks one without saying which.
2. Schema linking to the wrong place. A deprecated copy of the sales table, or a spend column that excludes agency fees.
3. Join fan-out. Joining orders to a table with several rows per order repeats each order's revenue once per row.
4. Silent filters and date windows. A WHERE clause that drops refunds, a test market or the last week of the quarter.
5. Grain mismatches. Daily spend joined to weekly sales, or campaign spend divided by channel revenue.
6. Truncation and sampling. A large upload read in part, with the answer computed on whatever fitted.
7. Narration drift. The prose states a number that is not in the output, or calls a 2% movement a sharp rise.

A language model is trained to produce fluent text, and when the data cannot answer a question it tends to fill the gap rather than say so. Vendors whose AI is real describe failure modes like these in specific terms, which makes a useful test when telling real AI apart from marketing theatre.

Types of AI data analysis tools

AI data analysis tools fall into four categories, each with its own failure mode.

General chat assistants with code execution

You upload a file and the assistant writes and runs Python to answer. The strength is ad hoc exploration with code you can read. OpenAI's own guidance tells users: "Always verify critical results yourself before sharing, since ChatGPT can make mistakes." The risk is that nothing holds shared metric definitions, so each session reinterprets "revenue" from the column names it sees.

Spreadsheet copilots

Assistants built into spreadsheet software produce formulas, tables and charts you can inspect cell by cell. The common errors are formulas applied to the wrong range, lookups that quietly return the first match, and summary text that goes stale when the data changes. Audit the formulas as you would a junior analyst's workbook.

BI conversational analytics

These tools sit on a governed BI layer, so metric definitions are fixed centrally rather than guessed per question. Looker's version is grounded in its semantic layer, and its documentation says it does not yet support questions that need prediction and forecasting, or advanced statistical analysis including correlation and anomaly detection. A vendor scoping its own tool to descriptive questions is a fair signal of where the technology is dependable.

Domain-grounded assistants

These read answers from a validated domain model instead of computing them from raw tables. In marketing measurement, that means reporting contributions, ROAS and saturation from a fitted marketing mix model. It answers fewer kinds of question, and each answer comes from a model that has already been checked.

Category	Best for	Main failure risk	Check first
Chat assistant with code execution	One-off exploration of a file	No shared definitions; narrative drift	The code and the row count
Spreadsheet copilot	Modelling inside a workbook	Wrong ranges; stale summary text	The formulas and their ranges
BI conversational analytics	Recurring descriptive questions	Questions outside the semantic layer	Whether the metric is defined there
Domain-grounded assistant	Questions the model was built for	Questions outside the model's scope	That every number traces to the model

Which AI is best for data analysis?

There is no single best AI for data analysis. The right choice depends on the type of question, and one enterprise benchmark shows how much depends on how the data is described to the model. In it, Sequeda, Allemang and Jacob found that GPT-4 answered 16% of questions correctly when prompted directly against SQL databases, and 54% when the same questions were posed over a knowledge graph representation of the data. The study tested only GPT-4, so it cannot rank assistants, but the same model more than tripled its accuracy once the data was described properly.

Choose by question type rather than by a vendor's ranking.

Question type	Example	Use	Condition
One-off look at a file	"Which products had the most returns in this CSV?"	Chat assistant with code execution	Read the code it ran
Recurring descriptive question	"What were sales by region last month?"	Conversational analytics on a semantic layer	The metric is defined centrally
Spreadsheet modelling	"Build a monthly cohort table from this sheet"	Spreadsheet copilot	Audit every formula
Causal or budget question	"Did TV drive the Q3 lift? Where should the next $1M go?"	A validated MMM or experiment, with AI as the interface	The AI reads the model and never estimates the effect

How to use AI for data analysis

Before you ask: definitions, data dictionary, known answers

Write down what each metric means before any AI sees the data: which revenue counts (gross or net), which conversions count (platform-reported, CRM or modelled) and what grain the data is at (daily or weekly, campaign or channel). Give that data dictionary to the tool with the data, and fix duplicate rows and missing weeks first. Then prepare two or three questions whose answers you already know, such as last quarter's revenue from finance.

While you ask: prompting for checkable output

Prompt for output you can check.

Ask for	Example prompt line
The query or code alongside the answer	"Show the SQL you ran, then the result."
Stated assumptions	"List every assumption about metric definitions before answering."
Row count and date range	"State the number of rows used and the exact date range."
One question per prompt	"Answer only this: total Q2 revenue."
Permission to say no	"If the data cannot answer this, say so and say what is missing."

After it answers: decide what can be delegated

Sort every question into one of three types.

1. Descriptive (what happened). Delegate it, with the checks below.
2. Diagnostic (why did it change). Let the AI propose hypotheses and pull the data, then test them yourself.
3. Causal (what did the spend cause, and what if we change it). This needs a model or an experiment. Scenario questions belong in a model built for what-if analysis, where the AI moves the inputs and the model computes the outcome.

How to check an AI analyst's work

Five checks catch the failure modes described above. Each takes minutes for a single answer, and most of them can be automated for recurring questions.

1. Reproduce it

Ask the same question again, rephrase it, then ask it in a fresh session with no history. A correct answer survives all three. One that changes between runs gives you no basis for a decision.

2. Trace every number to its source

Every number in the narrative must appear in an output table, and every output table must come from a query you can see. Delete anything you cannot trace, however reasonable it sounds.

3. Run sanity checks

• Totals reconcile to the finance or source-system figure for the same period.
• Row counts before and after each join match expectations. A join that adds rows is a warning.
• Units, currency and the date range are the ones you asked for.
• The order of magnitude is plausible against the previous period, and segment shares add up to 100%.

4. Ask for the code or query, and read it

Check the joins (does any of them multiply rows?), the filters (is anything excluded that should be in?), the aggregation grain (are daily and weekly figures mixed?) and the metric definition (is this the revenue you meant?). Even without writing SQL, anyone can confirm that the dates in the WHERE clause match the question and that the table names are the ones the data team uses.

5. Test it on questions with known answers

Keep a golden set of ten to twenty questions with verified answers from real reporting. Run it when you adopt a tool and after any change of tool, model, prompt or data source. It is the same discipline we recommend for evaluating a RAG chatbot, applied to numbers instead of documents.

A plausible answer that is wrong: a worked example

The numbers here are illustrative, built to show the mechanism. A marketer uploads two Q2 tables to a chat assistant.

• orders: 4,000 orders and $1,200,000 of revenue. To keep the arithmetic simple, every order is worth $300.
• touchpoints: 6,000 rows, one per recorded marketing touch, so 1.5 touches per order on average, with every order touched at least once.

Meta spend was $150,000. The question is "What was Meta's ROAS in Q2?" The assistant joins the tables on order ID, sums revenue where the channel is Meta, and replies: "Meta ROAS in Q2 was 4.2, from $630,000 of Meta-driven revenue on $150,000 of spend."

SELECT SUM(o.revenue) AS meta_revenue
FROM orders o
JOIN touchpoints t ON t.order_id = o.order_id
WHERE t.channel = 'Meta';

The answer is wrong. In the data, 1,500 orders had a Meta touch, and those orders carry 2,100 Meta touch rows because some customers saw several Meta ads. The join repeats each order's revenue once per Meta touch, so the sum is 2,100 x $300 = $630,000. Revenue from orders with a Meta touch is 1,500 x $300 = $450,000, and ROAS on that basis is $450,000 / $150,000 = 3.0.

Two of the five checks catch it directly.

• Sanity check. The same join without the Meta filter totals 6,000 x $300 = $1,800,000, against $1,200,000 in the orders table.
• Reading the query. Nothing deduplicates on order_id. Counting each order once fixes it.

SELECT SUM(revenue) AS meta_touched_revenue
FROM orders
WHERE order_id IN (
  SELECT order_id FROM touchpoints WHERE channel = 'Meta'
);

A known-answer test would probably miss it. Asked for total Q2 revenue, the assistant would most likely sum the orders table alone and return the correct $1,200,000, because that question never touches the join. A golden set protects only the query paths it exercises, so include questions that run through your joins.

Even the corrected $450,000 is attributed revenue, from orders a Meta ad touched, and some of those customers would have bought anyway. In this illustration, a marketing mix model estimates Meta's incremental Q2 revenue at $210,000, so incremental ROAS is $210,000 / $150,000 = 1.4. Our guide to ROAS covers why attributed and incremental returns differ.

A chat assistant can be taught to produce the 3.0. It cannot produce the 1.4 from these tables, because one quarter of orders and touches holds no spend variation over time and no control variables from which a counterfactual could be estimated.

Causal questions need a model or an experiment

Why a chatbot cannot answer "did this campaign cause that sale?"

Causal questions ask what sales would have been without the campaign, and your data records only the world where it ran. A sales lift during a TV burst can come from seasonality, a price cut, a promotion or wider distribution, and in observational data all of them move with media spend. Separating them needs a marketing mix model with adstock, saturation and control variables, or an experiment. A holdout or geo experiment gives a causal anchor for one channel at a time, and a model calibrated against it extends that evidence across every channel and period.

Language models are weak at this inference. On Corr2Cause, a benchmark of more than 200,000 samples presented at ICLR 2024, seventeen LLMs performed close to random when asked to infer causation from correlation. They do carry general causal knowledge (advertising can drive branded search, price cuts lift unit sales), which helps when proposing hypotheses. That knowledge describes the world in general, and a chat assistant asked "did Meta cause the Q2 lift?" is likely to answer from the general pattern unless it is made to read a model.

For this reason the statistical core of measurement should stay human-led and validated, with AI working around it, the split set out in Peripheral Agentic MMM.

Forecasting is a modelling job too

Chat interfaces will produce a forecast on request, so forecasting often gets lumped in with AI analysis. Tan and colleagues at NeurIPS 2024 ran ablation studies on three popular forecasting methods built around a pretrained LLM, removing the language model or replacing it with a basic attention layer. Performance did not degrade, and in most cases it improved. Pretrained LLMs did no better than models trained from scratch, despite their computational cost. The study tested forecasting models with an LLM inside rather than chat assistants, but it gives little reason to expect a language model to forecast well on its own.

A purpose-built time-series forecasting model should therefore produce the forecast, with intervals, while the LLM explains it and moves the inputs for scenario runs. When an AI tool hands you a forecast, ask which model produced it. If the answer is the language model itself, treat the number as a sketch.

Grounding: make every number come from your own model

Grounding means giving the AI governed definitions or precomputed values to work from, so it retrieves and explains numbers that a validated model or governed data layer has already produced instead of computing new ones from raw tables each time. The closest published evidence is the knowledge-graph benchmark described above, where a richer description of the same data more than tripled GPT-4's accuracy. Looker applies the governed-definitions half of the idea, grounding its conversational analytics in the semantic layer.

In marketing measurement, contributions, ROAS, saturation curves and adstock should come from a marketing mix model that analysts have fitted and validated. The AI reads those values as they currently stand, compares them, runs the scenarios the model supports and explains what they mean. In a grounded setup, it should never derive a channel's ROAS from a join of spend and sales tables, which is where the worked example went wrong.

A model's number is valid only under the model's assumptions, and an AI that retrieves it without its caveats will mislead a budget owner fluently. We made the same argument for grounding a RAG chatbot in business data rather than documents, where each number has to stay linked to the conditions under which it holds.

Observability: trace every answer

If you cannot see what an AI analyst did, you cannot check it at scale. For every answer, log the question, the generated query or code, the values retrieved or computed, the model and prompt version, and the final narrative.

With that record, the five checks can be run after the fact, and failures separate into two kinds. Retrieval failures pulled the wrong data. Reasoning failures pulled the right data and drew the wrong conclusion. They need different fixes, and without traces they look identical.

Re-run the golden set after every model or prompt change and compare it with the logged baseline. Our post on LLM observability for MMMGPT shows what such a trace looks like, and the same discipline applies to any AI analyst.

Frequently asked questions

Can AI do data analysis?

Yes, for well-specified descriptive work. AI for data analysis writes queries and code, summarises tables and drafts charts quickly. Accuracy falls on realistic enterprise databases, multi-step problems and business-specific metrics, and AI cannot estimate causal effects from observational data on its own. Delegate descriptive questions with checks, and put causal questions to a model or an experiment.

Is ChatGPT good for data analysis?

It is good for exploring an uploaded file, because it writes and runs Python you can read. OpenAI's own guidance tells users to review the generated code and output tables and to verify critical results before sharing. It has no shared metric definitions, and its narratives can drift from the output, so read the code and reconcile totals first.

How accurate is AI for data analysis?

It depends on the data and the question. OpenAI's o1-preview solved 91.2% of tasks on the clean Spider 1.0 benchmark, while an agent built on it solved 21.3% on Spider 2.0, which uses real enterprise databases. On the BIRD leaderboard, the best specialised system scores 82.95% against 92.96% for humans, with expert hints supplied. Expect high accuracy on simple questions and much lower accuracy on messy, multi-step ones.

What is agentic AI for data analysis?

Agentic AI for data analysis uses an agent that plans several steps, runs queries, inspects intermediate results and iterates, instead of answering in one pass. Every step adds a chance of error. On the DABstep benchmark, the best agent in the original evaluation reached 14.55% accuracy on the hardest tasks, and Google's DS-STAR later reported 45.2%. Log each step so the work can be checked.

Will AI replace data analysts?

The evidence points to a change in the job. In a field experiment with 758 BCG consultants, those using AI finished tasks inside its frontier 25.1% faster, but on a task outside it they were 19 percentage points less likely to produce a correct solution. As routine query writing is automated, the work moves towards defining metrics, checking AI output and making causal judgements.

Is it safe to upload company data to an AI tool?

That depends on the vendor's data-use terms. Before uploading, check whether the vendor trains on your data, where it is stored and for how long, and whether an enterprise agreement changes those terms. Remove personal data the question does not need. For recurring analysis, prefer a governed connection to your warehouse or BI layer over repeated file uploads.

How we approach it at Aryma

An AI analyst is only as trustworthy as the source of its numbers, so Aryma Dimension and eMMMy read from your own marketing mix model. Aryma Dimension is an AI meeting analyst. eMMMy joins your measurement call, presents your marketing mix model out loud, answers the room live from your own model's results, draws charts such as saturation curves mid-conversation and runs budget scenarios. Every number it quotes, from contribution and ROAS to saturation and adstock, comes from your model. It asks before running an optimiser or sending an email, and every meeting stays searchable afterwards.

Inside ArymaEdge, our MMM platform, eMMMy reads your MMM charts, runs simulator scenarios, explains budget optimiser recommendations and interprets saturation curves. Describe a spend plan in plain language and it moves the simulator to those values, with expected sales updating instantly. For questions about method, MMMGPT answers from a decade of marketing mix modeling knowledge with sourced, grounded responses.

To put an AI analyst on top of your own measurement, get in touch with us.