Concept Programming

Passing the benchmark is not dependability

An AI system that approves an unauthorised expense because a senior manager insists has not failed at reasoning. It has failed at dependability: behaving as specified across the whole operating range, including under pressure, urgency and unfamiliar phrasing.

The common fixes teach a model what not to say. They penalise bad outputs or filter them at the boundary, and leave the model's inner picture of the task untouched, which is why rephrasing and role-play keep getting around them. Animal learning shows the same pattern: punishment suppresses a response in one setting and teaches no alternative.

Teach the model where to go

A language model does not start empty. Its training text has already laid down the distinctions it can draw, so every new concept lands on material that already has a shape, and the shape differs for every concept.

Concept Programming works with that. It measures what the model already holds, supplies the pieces it lacks, and trains the concept as a positive target paired with an explicit response: what to do instead, not only what to avoid. Training examples are drawn from the full range of ways the situation is expressed, so that wordings the model has never seen still land in the same place.

Five steps

1. 01Name and define. State the concept as a plain proposition, with the conditions under which it applies.
2. 02Probe the parts. Test the base model on each condition separately, before any training. Parts it cannot tell apart get their own lessons first.
3. 03Generate varied cases. Build training cases by varying who, what, how much and where, across the domains and pressures the system will meet.
4. 04Pair each case with a response. A prohibition gets a refusal that says which rule applies. An obligation gets the required action. A value gets reasoning that applies it.
5. 05Measure on what was withheld. Test on fresh cases, on a whole style of wording kept out of training, and on ordinary unrelated requests.

Where the errors live

Four experiments ran on small open models from two families, all on one laptop.

The central finding is about how to evaluate. In the fine-tuning experiment, every error fell in the one style of wording deliberately kept out of training, and none fell in the styles training had covered. With zero misses in 400 to 480 covered cases, the true miss rate on covered wording is below 0.75% with 95% confidence.

So a single accuracy figure hides the real picture. A deployment claim becomes a coverage question: how much of real traffic arrives in wording the training never covered? Concept Programming turns that into a threshold an engineer can audit against.

A second finding concerns side effects. A model trained on a rule without ordinary requests in the mix kept its accuracy on the rule but began flagging 18.7% of unrelated requests. Including ordinary requests as counter-examples brought that to zero.