A language model can pick up a new task from a few examples in its prompt. Foundation models for structured data cannot yet do the same. To predict well on a database they have never seen, today's models still need tens of thousands of labeled examples in context. RT-J closes much of that gap. It is a relational foundation model that makes accurate predictions from a few hundred examples or fewer, built by pairing the Relational Transformer architecture with a training recipe designed for learning from little context.
Why context efficiency matters
Most of an organization's data lives in relational databases, and the questions people want answered from that data, such as which customers will churn or how much revenue a product will bring next quarter, are exactly the ones that drive decisions. Yet strong relational predictors have usually needed task-specific training, large labeled datasets, or hand-built features. That puts good predictions out of reach precisely when they are most useful, when only a handful of labels exist and an answer is needed quickly.
Relational foundation models take a promising step toward this, pretraining once across many databases and adapting to a new schema at inference time. RT-J builds on this line of work and focuses it on the few-shot regime, showing that a pretrained Relational Transformer becomes a strong context-efficient predictor once the recipe around it is designed for the job. Three ingredients make the difference.
Broad pretraining data
The public data for pretraining relational models is small and thin on the hard cases, rare events, cold starts, and heavy-tailed targets. RT-J is trained on THE JOIN, the largest open corpus of relational data assembled to date.
Dense supervision
Standard pretraining hides a single value at a time in a short context, but at inference the model faces long contexts full of labeled examples. RT-J instead masks many cells at once over long contexts, so each training step teaches far more.
Relevant context
The examples a model sees should be the ones most relevant to the question. Prior methods either ignored the database structure or expanded neighborhoods blindly. RT-J uses a lightweight random-walk retriever that surfaces rows both close to the target and likely to carry a useful label.
THE JOIN
A foundation model is only as broad as the data it learns from, and existing relational collections were simply too small and too uniform. THE JOIN is a corpus of 650 real-world databases spanning e-commerce, sports, media, finance, healthcare, and more, paired with roughly 6,000 forecasting tasks that deliberately include the hard cases, rare events, cold starts, and heavy-tailed targets, alongside autocompletion tasks over ordinary database rows. Building it took four steps.
- Collection. Databases were gathered from public repositories, APIs, domain portals, community data dumps, and existing benchmarks, all under permissive licenses. Anything overlapping with RelBench was left out so that evaluation stays honest.
- Standardization. Real-world databases are messy, so a set of automatic fixes repairs the common problems, coercing boolean and numeric columns, inferring missing foreign keys, and normalizing timestamps, until every database shares a clean relational structure.
- Task generation. A language model proposes realistic forecasting tasks for each database, and a deterministic checker discards any proposal that does not hold up against the schema.
- Task filtering. A gradient-boosted model scores each task and keeps the ones that are genuinely learnable or that make useful training challenges.
Open corpus. THE JOIN is released on Hugging Face as stanford-star/the-join, so anyone can pretrain and evaluate relational foundation models on the same data.
How RT-J is trained
RT-J keeps the Relational Transformer backbone, which turns every database cell into a token and attends across columns, rows, and foreign-key links. What changes is the recipe around it, designed so the model learns to predict well from little context.
Varying context length
How much context the model sees is the biggest lever on accuracy, but training on very long contexts is expensive. RT-J varies the context length during training, so a single model works well at any length and can spend more compute for more accuracy when it counts.
Masking many cells
Instead of hiding one value per example, RT-J hides many at once. Every long forward pass then produces many predictions to learn from, which makes training far more efficient and the model more robust.
Random-walk retrieval
Borrowing an idea from large-scale graph recommendation, RT-J runs many short random walks out from the target row and keeps the rows it visits most often as in-context examples. The whole process runs on the fly, with nothing precomputed, so it scales to thousands of databases.
The released model has 85M parameters, with 12 layers, a hidden size of 512, and 8 attention heads. It was pretrained for 100k steps on 32 H100 GPUs over about two days, and the same model handles both classification and regression.
Spending more compute at inference
Because training randomizes over context length and retrieval settings, RT-J can be run in many configurations at inference. Two of them turn extra compute into better predictions.
How well it works
RT-J is evaluated on RelBench, 21 forecasting tasks across seven databases, none of which appear during pretraining. The comparison is deliberately demanding. Every baseline is a strong in-context pipeline, a featurizer such as RDBLearn or an LLM agent that writes SQL, paired with a tabular predictor, and each method is handed the same labeled rows so that the comparison is fair at every budget.
Label-efficient regression
The regression story is striking. At every label budget RT-J is more accurate than every baseline, and its best result uses about 16× fewer labels than any baseline needs to reach its own best. The reason is that RT-J reads the raw relational context around each label instead of collapsing it into flat features. Adding per-task context tuning and an 8-seed ensemble brings RT-J to a mean of 23.0 nMAE with only about 400 examples per task, ahead of RDBLearn trained on 10,000 labels and of the closed-source KumoRFM-2, the top result on the RelBench in-context regression leaderboard.
Classification when labels are scarce
Classification is harder to win on few labels, because it depends more on label volume, so the strongest baseline eventually catches up at the largest budget. Still, when labels are scarce RT-J leads by up to 3 AUROC points. Interestingly, its regression accuracy saturates by a 2,048-cell context while classification keeps improving all the way to 8,192 cells. Longer contexts help classification most.
What drives the gains
Ablations trace RT-J's performance back to the three pillars of its recipe. Random-walk retrieval is the strongest way to build context. Masking many cells at once reaches low error the fastest and keeps long-context training affordable. And the meaning of real table and column names matters throughout, more than it might seem.
- Shuffling table and column names before prediction hurts accuracy across the board, by 7 to 10% relative on regression and 1.8 to 2.8 AUROC points on classification.
- The gap does not shrink with more labels, a sign that RT-J absorbs prior world knowledge from real schema names during pretraining, not just patterns from the examples in front of it.
Key takeaways
- Context-efficient relational prediction is a recipe, not an architecture. Matching the full foundation-model recipe to the few-shot regime turns a pretrained Relational Transformer into a strong in-context learner.
- Data, supervision, and retrieval each matter. THE JOIN supplies diverse pretraining data, multi-cell masking supplies dense supervision, and random-walk retrieval supplies relevant relational context.
- Test-time compute scales quality. Context ensembling and context tuning improve predictions further, and let RT-J match or surpass baselines that consume far more labels.
RT-J is evaluated on RelBench's 21 forecasting tasks and is bounded by its 8,192-cell context. Natural next steps include scaling both the corpus and the model, adding supervised fine-tuning and SQL-style interfaces, and extending the recipe to link prediction and transfer across databases.