Relational learning has produced many strong methods, but the community has had no reliable, reproducible way to compare them. RelArena-α is a unified benchmarking framework, released by Prior Labs in collaboration with STAR, that re-runs every method on RelBench under one API, one data state, and one evaluation protocol, in the spirit of TabArena for tabular data. On its first leaderboard, RT-PluRel, our Relational Transformer pretrained on PluRel synthetic data, is the #1 system in end-to-end predictive performance.
Why a new benchmark
Results reported on RelBench have become hard to trust and harder to compare. Several of the strongest-reported methods cannot be independently reproduced because training scripts or tuning procedures were never released. Even where results can be reproduced, they were obtained under different evaluation regimes and tuning budgets, so a gap between two papers may say more about the surrounding system than about the model. Copying self-reported baseline numbers has become common practice, there is no shared standard for aggregating across tasks, the feature-engineering baselines that dominate industry are under-represented, and published methods rarely come with an interface a practitioner could apply to their own database.
What RelArena-α standardizes
The α-release focuses on RelBench v1's 21 entity-level forecasting tasks, by far the most widely
used task type in the literature, and redistributes no data: every database is fetched at runtime
through the relbench package under its original license. Within that scope, the
framework owns everything around the model.
One data state
Every method sees exactly the same database during training, tuning, and evaluation. The database is frozen at the validation cut-off while tuning and at the test cut-off for the final evaluation, so no method can quietly peek past the boundary.
Models vs. systems
Following TabArena, a model submission declares only a search space; RelArena-α samples configurations, schedules runs, and selects the final candidate under a shared budget. A system submission brings its own tuning regime. Models isolate methodological effects; systems track the best end-to-end pipelines.
Shared aggregation
Results flow into TabArena's bencheval library, which produces bootstrapped Elo ratings with confidence intervals, average ranks, critical-difference diagrams, and win-rate matrices, so headline claims are computed the same way for everyone.
Strong, honest baselines
GNNs (GraphSAGE, RelGT, RelGNN), flattening approaches (RDBLearn, TabPFN-Rel), the RT-PluRel relational foundation model, a trivial LightGBM, and two learning-free constant predictors, all re-run rather than copied from papers.
The first leaderboard
Elo ratings are fit with a Bradley-Terry model over pairwise task outcomes and anchored so that the global constant predictor sits at 1000, where a 400-point gap implies roughly a 91% win probability. Two boards are reported: models alone under the standardized tuning regime, and models together with systems. Among models, TabPFN-Rel ranks first. Once systems are included, RT-PluRel takes the top spot in end-to-end predictive performance.
| Rank | Method | Submission | Elo |
|---|---|---|---|
| 1 | RT-PluRel #1 system | system | 1861 |
| 2 | TabPFN-Rel (API) | model | 1826 |
| 3 | TabPFN-Rel (OSS) | model | 1727 |
| 4 | GraphSAGE | model | 1655 |
| 5 | RelGT | model | 1584 |
| 6 | RDBLearn | model | 1554 |
| 7 | RelGNN | model | 1519 |
| 8 | Constant (per-entity) | learning-free | 1256 |
| 9 | Constant (global) | learning-free | 1000 |
- Tabular models are highly competitive. Flattening the database into a table, as TabPFN-Rel and RDBLearn do, holds its own against specialized relational architectures.
- Constant predictors are not trivially beaten. The per-entity constant, which uses no features and no model, beats RelGNN and RelGT on four tasks each. Only TabPFN-Rel and RT-PluRel exceed it on all 21.
- Text matters on a few tasks. Dropping text features moves TabPFN-Rel from 1821 to 1706 Elo, driven almost entirely by
rel-event/user-ignoreandrel-avito/user-clicks. - Everything is expensive. The single-seed board took hundreds of wall-clock hours. RT-PluRel's fit is on average about 5× slower than TabPFN-Rel via the API and 30× slower than the open-source variant.
RT-PluRel: the first system submission
RT-PluRel is STAR's entry and the first system on RelArena-α. It builds on the 85M-parameter Relational Transformer, which represents a database as cell tokens linked by column, row, and foreign-key attention, using the public checkpoints pretrained under the PluRel synthetic-data protocol, one each for classification and regression. Each prediction's context is assembled from the entity's relational neighborhood by breadth-first expansion and random-walk ranking, bounded by the split's cut-off timestamp so that tuning and evaluation respect RelArena-α's data states. It registers an empty search space and performs all model selection internally, which is why it is reported as a system rather than a model.
Fine-tuning with early stopping
The pretrained checkpoint is delta-fine-tuned on the training split, learning a zero-initialized additive delta on frozen weights so that weight decay regularizes toward the pretrained model. Training mixes context configurations from a grid over context size, neighborhood width, and recency preference, and stops after 10k steps without validation improvement.
Context selection
With the selected checkpoint frozen, 60 context configurations are scored on validation rows with four sampling seeds each, and the best one under the task's primary metric is kept.
Refit and ensemble
Following RelArena-α's refit protocol, the model is retrained from the pretrained checkpoint on train + validation with the step count scaled to the data size. Test predictions ensemble eight context-sampling seeds.
Open weights and protocol. The pretrained checkpoints are on Hugging Face as stanford-star/rt-plurel, and the full RT-PluRel tuning protocol ships inside the RelArena-α repository, so the #1 result can be reproduced and scrutinized end to end. Its strong showing points to the value of researching model and tuning-regime improvements together.
TabPFN-Rel and RPI
TabPFN-Rel is Prior Labs' relational harness for TabPFN-3 and the top-ranked model submission. It inherits RDBLearn's recipe, aggregating features along every join path up to a tuned depth with deep feature synthesis, and improves on it with a leakage-free tuning split, an order of magnitude more context rows, re-attached text columns from the entity table, and a context-selection scheme that trades off recency and diversity at no extra cost. A text-free open-source variant runs without the hosted API.
RPI, the Relational Predictive Interface, closes the gap from paper to practice. A database and a forecasting task are specified entirely in YAML, with no Python, turning a folder of CSV or Parquet files into a RelArena-α task on which any implemented method, including TabPFN-Rel and RT-PluRel, can be tuned and run in two lines of code. It ships with example specifications for all 21 RelBench v1 tasks, a Kaggle example, and an agent skill.
Open issues
- Running models is hard. Most competitive methods need hours of CPU-bound preprocessing per database, and runtimes differ by orders of magnitude, which makes fully standardizing tuning an open problem.
- Missing baselines. The current set is not complete, and stronger automated-feature-engineering tabular baselines are not yet represented. Adding a method is documented, with skill files for agentic coding tools.
- Task types and data quality. The α-release covers only entity-level forecasting; recommendation, attribute prediction, foreign-key prediction, and autocomplete tasks remain to be integrated, and whether the existing tasks reflect practitioners' needs is an open question.
- Timestamp boundaries. GNN baselines and RT-PluRel may use rows at the entity's own timestamp as context while flattening methods do not, a difference to be aligned in future releases.
RelArena-α is intended as a starting point rather than a definitive benchmark. STAR and Prior Labs will continue to grow the baselines, standardize tuning, and extend task coverage with the community. Contributions and feedback are welcome on GitHub.