RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

Lay summary for non-technical audience

Much of the world's important data, from bank transactions to hospital records to online shopping, is stored in databases made of many linked tables. Organizations often want to make predictions from this data, such as which customers will leave or how a clinical trial will turn out. Today this usually requires specialist effort and tens of thousands of labeled past examples. We built RT-J, an AI model that learns a new prediction task from just a hundred or so examples shown on the spot, like a person picking up a task from a few worked examples. To teach it, we used AI agents to gather hundreds of real public databases and pose thousands of practice prediction questions about them. RT-J needs about ten times fewer examples than strong existing methods to match their accuracy, putting fast, low-cost forecasting within reach of small businesses, scientists and analysts.

Pretrained models for language and vision solve new tasks from a handful of examples in their prompt. Models for relational databases, where much of the world's enterprise and scientific data lives, still need task-specific training or tens of thousands of labels. RT-J is an 86M-parameter relational foundation model that makes accurate predictions on an unseen database from hundreds of in-context examples or fewer, in a single forward pass.

86M

parameters, pretrained on contexts of up to 8,192 database cells

639

web-crawled databases in THE JOIN, with 6,023 LLM-generated forecasting tasks

10×

fewer in-context examples than strong feature-engineering pipelines need to match it on RelBench regression

RelBench regression nMAE versus number of in-context examples. RT-J reaches about 33 nMAE with around 225 examples; RDBLearn + TabICLv2 and LLM Agent + TabICLv2 need roughly 2,300 examples, 10 times more, to reach the same level. — **In-context regression on RelBench.** Given the same labeled examples, RT-J beats the best feature-engineering pipelines at every example count, and the pipelines need roughly 10× more examples to catch up.

Same architecture, new recipe

RT-J keeps the Relational Transformer architecture, which reads cells of a database directly as tokens instead of flattening it into a feature table. What changes is everything around the model.

RT-J pretraining recipe in six steps: sample a database and task from THE JOIN, sample a target row and drop later rows, rank candidate examples by random-walk visit count, expand each with a local bounded-width BFS, form the context window of cell tokens, and predict the masked target cell. — **The RT-J pretraining recipe.** Random walks from the target row pick which labeled rows enter the context, and a local expansion around each adds the rows linked to it.

Real pretraining data

THE JOIN is the largest open corpus of real relational data to date: 639 databases across e-commerce, sports, finance, healthcare, and more, with forecasting tasks proposed by an LLM and filtered by deterministic checks. RT-J is pretrained first on synthetic PluRel databases, then on THE JOIN.

Long-context training

Training samples context sizes from 512 to 8,192 cells, so the context holds hundreds of labeled examples and one model works at every size.

Random-walk retrieval

Instead of expanding blindly from the target row, random walks rank the rows that are close to it and carry useful labels. Retrieval is cheap, needs no precomputation, and beats uniform sampling by 4.1 nMAE and 6.1 AUROC points.

Results

On RelBench's 21 forecasting tasks across 7 held-out databases, RT-J is compared with the strongest in-context pipelines: RDBLearn or an LLM agent writing SQL to engineer features, followed by TabICLv2 or LightGBM. Every pipeline sees exactly the labeled rows in RT-J's context, plus the rest of the database.

RT-J versus four featurizer plus predictor pipelines as the number of in-context examples grows. On regression RT-J has the lowest nMAE at every example count. On classification RT-J leads when examples are scarce, and LLM Agent + TabICLv2 catches up at the largest context. — **RT-J against in-context pipelines on RelBench.** RT-J points are labeled with the context size in cells.

Regression. RT-J beats every pipeline at every context size, by 2.3–5.0 nMAE points.
Classification. RT-J leads by 3.1 AUROC points when examples are scarcest, and the strongest pipeline catches up at the largest context.
Test-time compute helps. Tuning the context per task and averaging over several sampled contexts improve RT-J further.

Each pretraining phase helps

Synthetic pretraining on PluRel improves a randomly initialized RT, and continuing on THE JOIN improves it again. Real data matters most in context, where RT-J beats RT-PluRel by 0.6 nMAE and 1.5 AUROC points.

Mean nMAE over 9 RelBench regression tasks, as submitted to the RelBench leaderboard. RT is not pretrained, so it has no in-context result. Hover for values.

Mean AUROC over 12 RelBench classification tasks. Hover for values.

RelArena-alpha combined model and system leaderboard, Elo anchored at 1000: RT-PluRel 1861, TabPFN-Rel (API) 1826, TabPFN-Rel (OSS) 1727, GraphSAGE 1655, RelGT 1584, RDBLearn 1554, RelGNN 1519, constant per-entity 1256, constant global 1000. — **Fine-tuned RT-PluRel ranked first on RelArena-α at the time of submission.** Elo over the 21 RelBench forecasting tasks, anchored at 1000 for the global constant predictor. RT-PluRel is a system submission (hatched) with its own tuning protocol. See the RelArena-α page for the benchmark and full leaderboard.

Pretraining on real databases also teaches RT-J what table and column names mean. Shuffling those names costs up to 2.1 AUROC points, and the gap grows with more examples, so the schema carries knowledge that examples alone do not replace.

Takeaways

Context-efficient predictions emerge from large-scale pretraining. Real data, long contexts, and smarter retrieval turn RT into a strong in-context learner.
Modeling the database natively beats feature engineering. Given the same labels, RT-J needs about 10× fewer of them.
Real data teaches semantics. Table and column names carry knowledge that examples alone do not replace.

Release

We release:

the RT-J and RT-PluRel models,
THE JOIN data (alongside the PluRel synthetic data used for the first pretraining phase),
the THE JOIN agentic curation pipeline, and
our pretraining, fine-tuning and inference code.

Cite

@inproceedings{ranjan2026rtj,
  title     = {{RT-J}: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions},
  author    = {Rishabh Ranjan and Vignesh Kothapalli and Harshvardhan Agarwal and Charilaos Kanatsoulis and Roshan Upendra and Tom Palczewski and Carlos Guestrin and Jure Leskovec},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026}
}