RelBench: A Benchmark for Deep Learning on Relational Databases

RelBench is an open benchmark for predictive machine learning on relational databases. It pairs curated multi-table databases with well-defined predictive tasks and a single standardized evaluation, so methods can be compared head to head, helping Relational Deep Learning and relational foundation model research move quickly. The original benchmark (NeurIPS 2024) was extended by RelBench v2 (ICLR 2026 DATA-FM Workshop) into a large-scale repository, and RelBench 3 (the current PyPI release) loads every database and task straight from the Hugging Face Hub. The benchmark, datasets, and tooling are open source under an MIT license.

Relational deep learning

Most of the world's data lives in relational databases, several tables linked through primary and foreign keys. Turning that data into a prediction has traditionally meant flattening the tables into one feature matrix by hand, a slow and lossy process that throws away much of the relational structure. Relational Deep Learning instead treats the database directly as a graph, where rows are nodes and foreign keys are edges, and learns predictive models end to end with no manual feature engineering. Because the model reads the schema itself, one architecture can serve many tasks over the same database, and increasingly transfer across databases as a relational foundation model.

The paradigm was set out in the Relational Deep Learning position paper (ICML 2024), and RelBench is the benchmark built to measure it. It gives the field a common yardstick, the same way ImageNet did for computer vision, so that progress on relational prediction is concrete and reproducible.

Datasets and tasks

The benchmark gathers eleven real-world databases from very different domains, each shipped with a fixed temporal split and a set of predictive tasks. Every prediction is made at a reference time and may use only information available before it, so each task is a realistic forecast rather than a leaky lookup. The seven original databases (30 tasks) are hosted at stanford-star/relbench-v1.

RelBench v2 adds four databases and 36 tasks, including Autocomplete tasks that predict existing columns of the original seven databases, hosted at stanford-star/relbench-v2-extra.

Tasks fall into four types, each scored with a single metric so results stay directly comparable across methods and databases.

A growing repository

RelBench v2 expands the benchmark into a single entry point for a much larger collection of 90 community databases, all loaded with the same relbench.load_dataset("<org>/<repo>/<name>") call.

• CTU Prague repository. 71 relational databases from the CTU Prague Relational Learning Repository, integrated through ReDeLEx and hosted at stanford-star/redelex.
• 4DBInfer. Seven databases from the 4DBInfer benchmark for graph-centric predictive modeling, hosted at stanford-star/dbinfer.
• Temporal Graph Benchmark. Twelve datasets from the Temporal Graph Benchmark, with time-stamped event streams expressed as relational schemas, hosted at stanford-star/tgb, so temporal graph models and Relational Deep Learning can be compared directly. TGB and 4DBInfer tasks that use their own scoring protocols load as data only and are evaluated with the original benchmarks' evaluators.

Leaderboard

RelBench tracks test-set results across its tasks, submitted as prediction tables through the RelBench repository and validated automatically, and kept current as the field shifts toward relational foundation models. There are three independent boards, classification (12 tasks), regression (9), and recommendation (10), each ranking methods by their mean test metric over every task on the board, and in-context submissions are marked so zero-shot and task-specific results stay easy to read side by side.

Get started

RelBench installs from PyPI and loads any database and task straight from the Hub, with no per-dataset code.

pip install relbench             # data + task loading
pip install "relbench[example]"  # + PyTorch Geometric & PyTorch Frame, for the GNN examples

Loading a dataset and a task takes only a few lines, and the relational schema comes back ready to model.

import relbench

dataset = relbench.load_dataset("rel-f1")   # or a HuggingFace 'org/repo[/subdir]', or a local path
db = dataset.get_db()                  # rows after test_timestamp are hidden

task = dataset.load_task("driver-position")   # dataset.get_task_names() lists them
train_table = task.get_table("train")  # train / val / test label tables
test_table  = task.get_table("test")   # the target column is hidden on test

# ... train any model on db + train_table, predict on the test entities ...
metrics = task.evaluate(test_pred)     # standard metric for the task type

Two short notebooks walk through the rest and open directly in Google Colab with no local setup.

Cite

If you use RelBench, please cite the position and benchmark papers. If you use RelBench v2, please also cite the v2 paper.