Foundation Models in Production: What They Cost, Where They Break, and How to Ship Them Offline

A demo runs once. A pipeline runs at 3am, forty times, on a box you're paying for by the second.

This is the part that decides whether zero-shot ML actually ships: what it costs to run, where the cliffs are, and how to deploy it somewhere with no internet.

Part 3 of a series. Part 1 introduced tabfm_classify / tabfm_regress; part 2 picked a model. Everything below is measured on one 8-core CPU (Ryzen 9 3950X) against the telco churn dataset.

The Single Most Important Thing to Know

Scoring more rows is nearly free. Adding context rows is not.

That one sentence will save you more money than any tuning flag, and it's the opposite of how classical ML batching intuition works.

Here's the same 500-row context, scoring 50 → 600 rows:

scored rows	mitra	tabicl-v2
50	11.1 s	1.3 s
150	14.5 s	1.0 s
300	18.5 s	1.2 s
600	27.2 s	1.5 s

Twelve times the output costs Mitra 2.5× the time, and TabICL essentially nothing — its curve is flat within noise. Mitra's marginal cost works out to roughly 29 ms per scored row on top of a ~9.6 s fixed cost, and that fixed cost is the context.

Now compare that to growing the context, from part 1, scoring a constant 300 rows:

context rows	time
100	8 s
500	23 s
2,000	78 s

Context is superlinear. Output is nearly free.

The practical rules:

• -> Never score row by row. One call scoring 600 rows costs 27 s; 600 calls scoring one row each would cost 600 × ~10 s. That is the difference between a nightly job and a non-starter.
• -> Don't reflexively feed it all your labelled data. Part 1 measured 500 context rows beating 2,000 on quality and being 3.4× faster. More context is not more better.
• -> Batch across groups. If you're scoring per region or per plant, one call per group with a modest context beats one giant call.

Load Time vs Inference Time

These are different costs with different fixes, and lumping them together is how people conclude "big models are slow" when the truth is "big models take a while to load".

model	size	cold (first call)	warm	loading cost
tabicl-v2	110 MB	2.3 s	2.3 s	~0 s
mitra	303 MB	39.3 s	19.3 s	~20 s
tabfm-v1	6.56 GB	37.2 s	9.7 s	~27 s

Loading tracks size. Inference does not — the 6.56 GB model scores twice as fast as the 303 MB one.

The load cost is a one-off per session, and you can move it off the critical path:

CALL tabfm_load('classification');   -- pay the 27 s at startup, not on the first user query

Anything long-lived — a warm DuckDB process behind an API, a scheduled job that scores many groups — should do this once and then never think about it again. Anything short-lived pays it every time, which is a strong argument for tabicl-v2 in a CLI or a Lambda-style worker.

To free it again:

CALL tabfm_unload();     -- memory returns to baseline

Memory

Peak resident memory for a session with both mitra and tabicl-v2 loaded, plus DuckDB and the dataset: 4.5 GB.

That is the number to plan container limits against, and note what it implies: weights are not the whole story. Mitra and TabICL together are 413 MB on disk, but the runtime footprint is ten times that — ONNX Runtime allocates working buffers proportional to context × features × model width, not just to the weights.

Budget generously, and if you're running tabfm-v1 (6.56 GB of weights before any working set), give the container real headroom.

Does anofox_tabfm_threads Help?

Short answer, on our hardware: no, and we can't explain why.

The setting is documented as controlling ONNX Runtime's intra-op thread count, defaulting to half your cores. We set it before the first predict, in four separate processes, scoring the same 300 rows against the same 500-row context with mitra:

anofox_tabfm_threads	wall time	CPU time
1	17.2 s	256 s
4	18.0 s	285 s
8	17.9 s	285 s
16	18.9 s	299 s

Wall time is flat within 4% — that part could be explained by the workload being memory-bandwidth bound. What can't be explained away is the CPU column: at threads = 1 the process still burned 256 seconds of CPU in 17 seconds of wall clock, which is about fifteen cores. Whatever the setting is doing, it is not restricting the extension to one thread.

So treat it as non-functional until proven otherwise rather than as a tuning knob, and don't plan on it to make TabFM a polite neighbour on a shared box. If you need to cap CPU, cap it outside the database — cgroups, container limits, taskset. We've reported the finding.

This is the honest state of a young extension: the scaling behaviour is excellent and thoroughly measurable, and one of the four tuning settings doesn't visibly do anything.

The Weight Cache

Everything downloaded lives under ~/.cache/anofox-tabfm, keyed by Hugging Face repo and revision. Ours, after this series:

 43M  Prior-Labs__TabPFN-v2-reg@main/
 56M  Prior-Labs__TabPFN-v2-clf@main/
289M  autogluon__mitra-classifier@main/
289M  autogluon__mitra-regressor@main/
429M  jingang__TabICL@main/
 13G  google__tabfm-1.0.0-pytorch@main/
─────
 23G  total

Three things worth noticing:

1. Classification and regression are separate downloads. Mitra is 303 MB, but both tasks is 578 MB. TabFM is 6.56 GB per task — 13 GB for both. Only download the task you use.
2. tabfm-v1 is 96% of the cache. If disk matters, the commercial-only lineup from part 2 fits in under 800 MB.
3. It's inspectable from SQL:

SELECT model, task, bytes, loaded, license FROM tabfm_models();

Cleanup is SQL too:

CALL tabfm_remove('regression', model := 'tabfm-v1');

Air-Gapped Deployment

This is where a lot of in-database ML falls over: production has no internet, and the model wants to phone Hugging Face.

The cache is an ordinary directory, and its location is a setting. That's the entire mechanism — there's no credential store, no registry service, no sidecar.

SET anofox_tabfm_cache_dir = '/opt/models/anofox-tabfm';

So the deployment recipe is: populate that directory somewhere with network access, ship it, point the setting at it.

On the connected machine, download once and copy the directory:

# connected build box
duckdb -c "INSTALL anofox_tabfm FROM community; LOAD anofox_tabfm;
           INSTALL httpfs; LOAD httpfs;
           CALL tabfm_download('classification', model := 'mitra');"

cp -r ~/.cache/anofox-tabfm/autogluon__mitra-classifier@main  /opt/models/anofox-tabfm/
# 289 MB — one directory, no other artifacts

We tested that this actually holds, rather than assuming it. A fresh cache root containing only the Mitra classifier, and the query run in a network namespace with no interfaces at all (unshare -rn — not a firewall rule, no route to anywhere):

LOAD anofox_tabfm;
SET anofox_tabfm_cache_dir = '/tmp/airgap-models';

SELECT yhat, round(yhat_score, 3) AS conf
FROM tabfm_classify('tr', 'Churn', test := 'te', model := 'mitra');

yhat	conf
false	0.988
false	0.991
false	0.736

60 rows scored in 8.7 seconds with no network stack. No download attempt, no timeout, no degraded path — the extension simply found its weights where the setting pointed and ran.

Note what wasn't needed: no httpfs, no Hugging Face token, no config file. On the offline box, httpfs is only required if your data lives behind a URL.

For a private or gated Hugging Face repo — on the machine that does have network — a DuckDB secret carries the token, so no credentials end up in the extension's own config:

CREATE SECRET hf (TYPE http, BEARER_TOKEN 'hf_…', SCOPE 'https://huggingface.co');

Practical notes for an air-gapped rollout:

• -> Bake it into the image. The cache is content, not state — put it in the container layer and the model is warm from the first request.
• -> Mount it read-only. Nothing writes to the cache after download; a read-only mount is a decent tamper check.
• -> Pin the revision. Cache keys include the HF revision, so a pinned revision means the bytes you tested are the bytes you ship.

Guardrails and Failure Modes

Two settings exist to stop a runaway query from eating the box:

SET anofox_tabfm_max_rows = 10000;      -- default; per predict / per group
SET anofox_tabfm_max_features = 500;    -- default

These are deliberately conservative. Given the scaling curve above, a 10,000-row context is already a multi-minute call — the guardrail is doing you a favour.

Every failure names its own fix, which matters at 3am:

Invalid Input Error: tabfm: model 'classification' is not downloaded.
Run: CALL tabfm_download('classification');

And a property worth relying on: repeated identical calls returned byte-identical predictions for all three models we tested. Same input, same seed, same output — you can diff yesterday's scores against today's and any difference is a real data change, not model jitter.

When You Need a GPU

Everything above is CPU. At some point the scaling curve stops being acceptable — a 10,000-row context on CPU is minutes, not seconds — and that's the case for GPU.

anofox_tabfm ships four build flavors from one codebase: cpu (the default, and the one on the community repository), cuda for NVIDIA, rocm for AMD, and coreml for Apple Silicon. Selection is a setting, and discovery is a query:

SELECT * FROM tabfm_devices();   -- what this machine actually offers
SET anofox_tabfm_device = 'auto';  -- auto / cpu / cuda / rocm / coreml
SET anofox_tabfm_gpu_precision = 'bf16';

GPU builds come from the anofox repository rather than the community one, because community extensions are CPU-only by policy:

SET custom_extension_repository = 'https://ext.anofox.com/tabfm/rocm';

We have not benchmarked the GPU path in this post, and we're not going to quote numbers we didn't measure. The machine these benchmarks ran on has a CPU-flavored build installed; producing trustworthy GPU figures means a ROCm build plus a per-shape compile step, and that deserves its own post rather than a footnote here. What we can tell you is the shape of the decision: GPU matters when context is large, and it does nothing for the load-time cost, which is disk-bound either way.

What It Actually Costs

Take the measured throughput at a 500-row context and put a price on it. An 8-core cloud instance is roughly $0.35/hour, so about$0.0001 per second.

model	measured	throughput	1,000 rows	cost per 1,000
tabicl-v2	600 rows in 1.5 s	~390 rows/s	~2.6 s	~$0.0003
mitra	600 rows in 27.2 s	~22 rows/s	~45 s	~$0.004

(The 1,000-row figures extrapolate from the measured 600-row calls at the same context size; treat them as indicative, not measured.)

Fractions of a cent per thousand predictions, with no training run, no model registry, and no serving infrastructure. The comparison that matters isn't against a GPU cluster — it's against the two weeks of engineering time you'd otherwise spend building a pipeline for that column.

The Production Checklist

• -> Pick the model on licence and ceilings first (part 2), quality second
• -> CALL tabfm_load() at startup for anything long-lived
• -> Batch scoring into as few calls as possible; keep the context deliberately small
• -> Bake the weight cache into your image; set anofox_tabfm_cache_dir; pin the revision
• -> Plan memory around the working set, not the file size — ours peaked at 4.5 GB
• -> Leave max_rows alone unless you've measured what raising it costs
• -> Diff yesterday's predictions against today's; it's deterministic, so any change is real

---

In this series:

1. Zero-Shot Machine Learning in SQL — in-context learning and the two shapes
2. Which Tabular Foundation Model Should You Use? — the bake-off
3. Foundation Models in Production — cost, memory, air-gapped deployment (this post)

---

Reference

📖 anofox-tabfm on GitHub → — settings reference, flavors, and the benchmark scripts behind these numbers.