A language model can pick up a new task from a few examples in its prompt. Foundation models for structured data cannot yet do the same. To predict well on a database they have never seen, today's models still need tens of thousands of labeled examples in context. RT-J closes much of that gap. It is a relational foundation model that makes accurate predictions from a few hundred examples or fewer, built by pairing the Relational Transformer architecture with a training recipe designed for learning from little context.
Why context efficiency matters
Most of an organization's data lives in relational databases, and the questions people want answered from that data, such as which customers will churn or how much revenue a product will bring next quarter, are exactly the ones that drive decisions. Yet strong relational predictors have usually needed task-specific training, large labeled datasets, or hand-built features. That puts good predictions out of reach precisely when they are most useful, when only a handful of labels exist and an answer is needed quickly.
Relational foundation models take a promising step toward this, pretraining once across many databases and adapting to a new schema at inference time. RT-J builds on this line of work and focuses it on the few-shot regime, showing that a pretrained Relational Transformer becomes a strong context-efficient predictor once the recipe around it is designed for the job. Three ingredients make the difference.
Broad pretraining data
The public data for pretraining relational models is small and thin on the hard cases, rare events, cold starts, and heavy-tailed targets. RT-J is trained on THE JOIN, the largest open corpus of relational data assembled to date.
Training at inference context lengths
Standard pretraining trains at one short context length, but at inference the model faces contexts of every size. RT-J samples the context length during training, so one model is strong across the whole range it will actually be run at.
Relevant context
The examples a model sees should be the ones most relevant to the question. Prior methods either ignored the database structure or expanded neighborhoods blindly. RT-J uses a lightweight random-walk retriever that surfaces rows both close to the target and likely to carry a useful label.
THE JOIN
A foundation model is only as broad as the data it learns from, and existing relational collections were simply too small and too uniform. THE JOIN is a corpus of 639 real-world databases spanning e-commerce, sports, media, finance, healthcare, and more, paired with roughly 6,000 forecasting tasks that deliberately include the hard cases, rare events, cold starts, and heavy-tailed targets, alongside 10,600 autocompletion tasks over ordinary database rows. Building it took four steps.
- Collection. Databases were gathered from public repositories, APIs, domain portals, community data dumps, and existing benchmarks, all under permissive licenses. Anything overlapping with RelBench was left out so that evaluation stays honest.
- Standardization. Real-world databases are messy, so a set of automatic fixes repairs the common problems, coercing boolean and numeric columns, inferring missing foreign keys, and normalizing timestamps, until every database shares a clean relational structure.
- Task generation. A language model proposes realistic forecasting tasks for each database, and a deterministic checker discards any proposal that does not hold up against the schema.
- Task filtering. A gradient-boosted model scores each task and keeps the ones that are genuinely learnable or that make useful training challenges.
Open corpus. THE JOIN is released on Hugging Face as stanford-star/the-join, so anyone can pretrain and evaluate relational foundation models on the same data.
How RT-J is trained
RT-J keeps the Relational Transformer backbone, which turns every database cell into a token and attends across columns, rows, and foreign-key links. What changes is the recipe around it, designed so the model learns to predict well from little context.
Varying context length
How much context the model sees is the biggest lever on accuracy, but training on very long contexts is expensive. RT-J varies the context length during training, so a single model works well at any length and can spend more compute for more accuracy when it counts.
Masking only the target
RT-J hides exactly one cell per example, the one it has to predict. Denser masking is the obvious way to get more supervision per step, and we tried it: masking non-target cells at any rate costs both accuracy metrics, because the masked cells corrupt the very context the prediction is read from.
Random-walk retrieval
Borrowing an idea from large-scale graph recommendation, RT-J runs many short random walks out from the target row and keeps the rows it visits most often as in-context examples. The whole process runs on the fly, with nothing precomputed, so it scales to thousands of databases.
The released model has 85M parameters, with 12 layers, a hidden size of 512, and 8 attention heads. It is pretrained in two phases, first on the synthetic PluRel corpus and then on THE JOIN, each phase taking under a day on a single node of up to 8 B200 GPUs. The released checkpoint is the stochastic weight average after 9k steps of THE JOIN phase, and the same single model handles both classification and regression.
Spending more compute at inference
Because training randomizes over context length and retrieval settings, RT-J can be run in many configurations at inference. Two of them turn extra compute into better predictions.
How well it works
RT-J is evaluated on RelBench, 21 forecasting tasks across seven databases, none of which appear during pretraining. The comparison is deliberately demanding. Every baseline is a strong in-context pipeline, a featurizer such as RDBLearn or an LLM agent that writes SQL, paired with a tabular predictor, and each method is handed the same labeled rows so that the comparison is fair at every budget.
Label-efficient regression
At every label budget RT-J is more accurate than every label-matched baseline, by 6–13% relative nMAE. The reason is that RT-J reads the raw relational context around each label instead of collapsing it into flat features. Adding per-task context tuning and ensembling brings RT-J to a mean of 32.4 nMAE with only a few hundred in-context examples per task, within one point of RDBLearn trained on 10,000 labels per task (31.4) while using about 25× fewer labels. The closed-source KumoRFM-2 remains ahead on this mean, at 29.1.
Classification when labels are scarce
Classification is harder to win on few labels, because it depends more on label volume, so the strongest baseline eventually catches up at the largest budget. Still, when labels are scarce RT-J leads by up to 3 AUROC points, and its classification accuracy keeps improving all the way out to the 8,192-cell context it was trained for.
What drives the gains
Ablations trace RT-J's performance back to its recipe. Random-walk retrieval is the most label-efficient way to build context. Masking only the target cell beats every denser masking rate we swept. Mixing forecasting and autocompletion tasks during pretraining beats either family alone. And the meaning of real table and column names matters throughout, more than it might seem.
- Shuffling table and column names before prediction hurts accuracy across the board, by up to 2.4% relative on regression and up to 2.1 AUROC points on classification.
- The gap does not shrink with more labels, a sign that RT-J absorbs prior world knowledge from real schema names during pretraining, not just patterns from the examples in front of it.
Key takeaways
- Context-efficient relational prediction is a recipe, not an architecture. Matching the full foundation-model recipe to the few-shot regime turns a pretrained Relational Transformer into a strong in-context learner.
- Data, context length, and retrieval each matter. THE JOIN supplies diverse pretraining data, sampling the context length during training matches pretraining to inference, and random-walk retrieval supplies relevant relational context.
- Test-time compute scales quality. Context ensembling and context tuning improve predictions further, and let RT-J match or surpass baselines that consume far more labels.
RT-J is evaluated on RelBench's 21 forecasting tasks and is bounded by its 8,192-cell context. Natural next steps include scaling both the corpus and the model, adding supervised fine-tuning and SQL-style interfaces, and extending the recipe to link prediction and transfer across databases.