There is an obvious way to build AI over an ERP: describe the schema to a language model, ask it to write a query, run the query, show the answer. It demos beautifully. We built the first version that way and then spent a long time understanding why it could not be trusted.
The failure is not that the model writes invalid queries. Invalid queries are the good case: they error, you see the error, you fix something. The failure is that it writes valid queries against the wrong thing.
What a wrong answer looks like
Ask an SAP system how much discount you gave last month. There are thirty-six entity sets whose
name contains some form of “sales order”. One of them is the order header. It carries a field called
TotalNetAmount, which is a real field holding a real number.
A model asked to write that query will very often sum TotalNetAmount, label it “discount”, and
return it. Every part of that is well-formed. The entity exists, the field exists, the aggregation
is correct, the period filter is right. The number that comes back is a genuine measurement, of
something else entirely.
Nothing downstream can catch it. It is not an anomaly, it does not fail a sanity check, and it is the right order of magnitude for an answer somebody was expecting. It is simply a different figure wearing the name of the one you asked for.
The rule we settled on
The model reasons. Code queries.
A question is compiled (in one model call) into a typed specification. That specification is not a query string. It cannot hold one. It holds an entity label chosen from a list the platform built, a measure, a grouping, a set of structured predicates and a period token.
Four things follow, and each closes a failure we actually measured:
It cannot name the wrong entity. The candidate list is assembled by code from the catalogue and ranked. The model picks a label out of it. An entity outside that list has no way to be expressed; this is different in kind from being told not to.
It cannot write a filter. Predicates are structured, and each field is resolved against the live
catalogue. There is no slot anywhere in the format for a filter string. This one came from watching
a model spend fifteen minutes cycling through in (...), padded keys, substring(), endswith
and finally eq '4': thirty reads, twenty-two of which returned byte-identical empty results.
It cannot write a date. Periods arrive as tokens (this_month, last_quarter, ytd) and are
resolved against the server clock at execution. This is why a saved view called “last 30 days” still
means the last 30 days a year later. We found eight saved dashboards carrying hardcoded date ranges,
including one titled “Last 30 Days” showing a window from the previous November.
A wrong field name is repaired or the question escalates. Resolution requires both a similarity
floor and a margin over the runner-up, because near-identical field names are real: there are
systems carrying AdditionalCustomerGroup1 through 5, which score 0.958 against each other with a
margin of zero. A repair is shown in the answer. Ambiguity is escalated, not guessed.
What it costs
Expressiveness. There are questions this cannot compile, and they escalate to a slower path that reasons. We consider that the correct trade: the slow path is a cost in seconds, and the fast wrong answer is a cost in trust that you only discover much later.
It also costs a candidate list. Building one means indexing the whole SAP surface into something searchable, ranking it well, and accepting that ranking is a real engineering problem rather than an embedding call. That is most of the work.
The part that is not solvable this way
One thing does not yield to structure. Whether the word “discount” describes the field
TotalNetAmount is a judgement about meaning, and no similarity threshold separates it cleanly;
we measured the distributions and they overlap where it matters.
So we do not judge it. We state it. Every figure prints what it actually computed:
Source: zsd_sales_order / ZA_SalesOrder · discount = sum of TotalNetAmount · period all timeWrong in a way a reader can see at a glance, rather than wrong in a way that rests entirely on the model’s choice of words. That is a weaker guarantee than the other four, and it is the honest one.