About Ithaca
Ithaca benchmark is a composite benchmark with tasks and datasets from classical and historical humanities. This first version of the Ithaca benchmark includes tasks in ancietnt Greek, Latin, Sanskirt, Tibetan, classical Chinese, classical Korean Hanja, and ____. Ithaca benchmark is our attempt to start a new conversation around benchmarking AI for the humanities (for humanities sake). We hope to build a community around this benchmark and to expand it with more tasks and datasets in the future.
News
- Sept 24, 2026: We are live! We will be making updates to improve benchmarking AI progress for the humanities -- Join Ithaca benchmark!
Submission
To submit, please follow the submission guideline here. Please send your submissino to:
unsojo@cornell.edu. We will get back to you in a few days.
Join the Ithaca Benchmark!
Ithaca benchmark solicits more datasets! We will continue to publish updated versions of Ithaca benchmark with more submissions. Join our email list for latest updates and join our community!
Citation
@article{ithacabench,
title={xxx},
author={xxxx},
journal={xxx},
volume={36},
year={2024}
}
Leaderboard - Execution Accuracy (EX)
| Model | Code | Size | Oracle Knowledge | Dev (%) | Test (%) | |
|---|---|---|---|---|---|---|
|
TBD Data Engineers + DB Students |
✔️ | 92.96 | ||||
| Sep 07, 2026 | TBD TBD [TBD] |
UNK | ✔️ | 78.10 | 82.39 | |
| Aug 22, 2026 | TBD TBD |
UNK | ✔️ | 77.77 | 82.28 | |
| Dec 16, 2025 | TBD TBD [TBD] |
UNK | ✔️ | 77.64 | 81.95 | |
| Sep 25, 2025 | TBD TBD [TBD] |
[link] | UNK | ✔️ | 74.90 | 81.67 |
| Jun 19, 2026 | TBD TBD |
UNK | ✔️ | 75.74 | 81.33 | |
| May 27, 2026 | TBD TBD |
UNK | ✔️ | 73.66 | 80.83 | |
| Aug 19, 2026 | TBD Adya AI |
UNK | ✔️ | 72.49 | 79.82 | |
| Jul 14, 2026 | TBD TBD TBD |
[link] | UNK | ✔️ | 74.49 | 79.09 |
| Jul 04, 2026 | TBD Anonymous |
UNK | ✔️ | 75.10 | 78.70 | |
| Jul 10, 2026 | TBD HKUST(GZ) [Boyan Li et al. '25] |
[link] | 27B | ✔️ | 74.49 | 78.42 |
| Jul 11, 2026 | TBD Amazon Ads - SpektrBot [Kishore Kumar et al.] |
[link] | UNK | ✔️ | 73.09 | 78.31 |
| July 14, 2025 | TBD LongShine AI Research |
UNK | ✔️ | 74.32 | 77.53 | |
| Aug 03, 2026 | TBD UST |
[link] | 31B | ✔️ | 76.86 | |
| Jul 07, 2026 | TBD MR Tech |
UNK | ✔️ | 75.95 | 76.80 | |
| Jan 02, 2026 | TBD China Telecom, TeleAI |
UNK | ✔️ | 73.53 | 76.63 | |
| Jan 26, 2026 | TBD HKUST(GZ) [Boyan Li et al. '25] |
[link] | UNK | ✔️ | 73.53 | 76.58 |
| Dec 4, 2025 | TBD AWS-Quick Science [Ravi Shankar et al.'25] |
30B-3B-MoE | ✔️ | 72.99 | 76.47 | |
| Feb 6, 2026 | TBD2-SQL Anonymous |
UNK | ✔️ | 74.45 | 76.41 | |
| Jul 14, 2026 | TBD Institute of Dataspace, Hefei Comprehensive National Science Center-IDATA |
UNK | ✔️ | 76.36 | ||
| Sep 07, 2026 | TBD Anonymous |
UNK | ✔️ | New Dev 72.69 |
76.08 |
|
| Apr 16, 2025 | TBD Google Cloud [Pourreza et al. '24] |
UNK | ✔️ | 74.90 | 76.02 | |
| Apr 3, 2026 | TBD wenyuai |
UNK | ✔️ | 72.75 | 75.96 | |
| Feb 21, 2026 | TBD South China Normal University |
30B | ✔️ | 74.19 | 75.91 | |
| Sep 22, 2025 | TBD JD:CHO-JDT-JDL |
[link] | UNK | ✔️ | 74.25 | 75.85 |
| Oct 23, 2025 | TBD Sinovatio AI Lab |
UNK | ✔️ | 73.72 | 75.80 | |
| May 30, 2025 | TBD Tencent Cloud |
UNK | ✔️ | 74.12 | 75.74 | |
| Feb 27, 2025 | TBD Contextual AI |
[link] | UNK | ✔️ | 73.50 | 75.63 |
| Dec 17, 2024 | TBD Alibaba Cloud [Yifu Liu et al. '24] |
[link] | UNK | ✔️ | 73.34 | 75.63 |
| Sep 07, 2026 | TBD Anonymous |
UNK | ✔️ | New Dev 72.23 |
75.41 |
|
| Sep 1, 2025 | TBD Anonymous |
UNK | ✔️ | 73.66 | 75.35 | |
| May 30, 2025 | TBD Tencent Cloud / Fudan University |
UNK | ✔️ | 73.47 | 75.35 | |
| Jan 8, 2026 | TBD Anonymous |
32B | ✔️ | 72.88 | 75.29 | |
| May 17, 2026 | TBD AIC-Lab, Chungbuk National University |
[link] | UNK | ✔️ | 77.10 | 75.13 |
| Oct 30, 2025 | TBD Amazon Ads-SpektrBot [Ravi Shankar et al.] |
30B-3B-MoE | ✔️ | 72.10 | 74.85 | |
| Sep 02, 2026 | TBD AMS CIR |
32B | ✔️ | 73.40 | 74.51 | |
| Sep 10, 2025 | TBD CMCC, JIUTIAN Research |
32B | ✔️ | 73.60 | 74.06 | |
| May 27, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 32B | ✔️ | 71.33 | 73.67 |
| May 18, 2026 |
TBD Kuaishou Business Team |
UNK | ✔️ | 70.67 | 73.34 | |
| Oct 27, 2024 |
TBD IBM Research AI |
34B | ✔️ | 72.43 | 73.17 | |
| Mar 31, 2025 | TBD Google Cloud / Stanford [Pourreza et al. '25] |
14 B | ✔️ | 72.29 | 72.78 | |
| Apr 28, 2026 | TBD OpenAI |
[link] | UNK | ✔️ | 72.55 | |
| Aug 06, 2025 |
TBD Gena Co. [Dönder et al. '25] |
[link] | UNK | ✔️ | 70.53 | 72.28 |
| Aug 21, 2024 | TBD Alibaba Cloud [Xiangjin Xie et al. '25] |
[link] | UNK | ✔️ | 69.30 | 72.28 |
| Aug 17, 2026 | TBD Tampere University |
[link] | 27B | ✔️ | 71.64 | 72.16 |
| Mar 4, 2025 | TBD Renmin University of China + ByteDance Infra Lab [Li et al. '25] |
[link] | 32B | ✔️ | 69.23 | 72.05 |
| Jul 22, 2024 | TBD Distyl AI Research [Maamari et al. '24] |
UNK | ✔️ | 67.21 | 71.83 | |
| Sep 22, 2025 | TBD HKU [Qu et al. ACL Main'25] |
UNK | ✔️ | 65.45 | 71.83 | |
| May 27, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 7B | ✔️ | 69.19 | 71.72 |
| Jun 30, 2025 | TBD JXUFE |
[link] | 14B | ✔️ | 69.43 | 71.60 |
| Apr 10, 2025 |
TBD Conrad Labs queryosity.io |
UNK | ✔️ | 69.40 | 71.16 | |
| May 21, 2024 |
TBDIR +CG +UT Stanford [Talaei et al.'24] |
[link] | UNK | ✔️ | 68.31 | 71.10 |
| May 08, 2025 | TBD |
[link] | 32B | ✔️ | 70.08 | 70.60 |
| Aug 1, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 1.5B | ✔️ | 67.08 | 70.49 |
| Aug 28, 2024 | TBD Uber Freight |
UNK | ✔️ | 72.16 | 70.26 | |
| Feb 24, 2025 | TBD HKUST(GZ) [Li et al. '25] |
[link] | 32B | ✔️ | 69.70 | 70.26 |
| Aug 30, 2024 | TBD Fudan University + Transwarp Technology |
UNK | ✔️ | 68.12 | 70.21 | |
| Feb 21, 2026 | TBD Baseline |
UNK | ✔️ | New Dev 68.77 |
70.15 |
|
| Nov 10, 2024 |
TBD Seoul National University |
UNK | ✔️ | 68.64 | 69.26 | |
| Jan 9, 2025 |
TBD Anonymous |
[link] | UNK | ✔️ | 66.88 | 69.26 |
| Dec 18, 2024 |
TBD mindflow.ai |
UNK | ✔️ | 70.93 | 69.20 | |
| Jan 09, 2025 | TBD Alibaba Cloud [Yifu Liu et al. '24] |
[link] | 32B | ✔️ | 67.01 | 69.03 |
| Jul 14, 2024 |
TBD Google Cloud |
UNK | ✔️ | 66.95 | 69.03 | |
| Jul 2, 2024 |
TBD ByteDance Infra Lab |
33B | ✔️ | 65.45 | 68.87 | |
| Nov 18, 2024 |
TBD Anonymous [Cao et al.'24] |
[link] | UNK | ✔️ | 67.21 | 68.70 |
| Aug 24, 2026 | TBD Alio Leuchtmann |
[link] | 2B | ✔️ | New Dev 61.40 |
68.47 |
| Feb 21, 2026 | TBD Baseline |
UNK | ✔️ | New Dev 66.17 |
68.14 |
|
| Aug 06, 2026 | TBD Anonymous |
UNK | ✔️ | 70.93 | 68.03 | |
| Mar 4, 2025 | TBD TBD [TBD] |
[link] | 7B | ✔️ | 69.04 | 67.97 |
| May 14, 2024 |
TBD TBD |
20B | ✔️ | 65.38 | 67.86 | |
| Nov 11, 2024 | TBD TBD TBD |
UNK | 65.91 | 67.41 | ||
| Feb 21, 2026 | TBD Baseline |
UNK | ✔️ | New Dev 67.34 |
66.85 |
|
| May 21, 2024 |
TBDIR +SS +CG Stanford [Talaei et al.'24] |
[link] | UNK | ✔️ | 65.00 | 66.69 |
| Jun 1, 2026 |
TBD Independent Researcher |
120B | ✔️ | 62.40 | 66.46 | |
| Sep 23, 2024 |
TBD Bilkent University [Caferoğlu et al.'24] |
[link] | UNK | ✔️ | 65.58 | 66.29 |
| Aug 29, 2024 | TBD Arcwise |
UNK | ✔️ | 67.99 | 66.21 | |
| Sep 10, 2025 |
TBD MathCo |
UNK | ✔️ | 63.95 | 65.90 | |
| Apr 30, 2025 |
TBD Cohere AI [Team Cohere et al.'25] |
[link] | 111B | ✔️ | 63.49 | 65.68 |
| Nov 18, 2024 |
TBD Anonymous [Cao et al.'24] |
[link] | UNK | ✔️ | 63.56 | 65.51 |
| Jan 14, 2024 |
TBD Dunamu [Lee et al. '24] |
UNK | ✔️ | 63.36 | 65.45 | |
| Aug 20, 2024 | TBD Xffuture |
UNK | ✔️ | 64.73 | 65.23 | |
| Apr 08, 2024 |
TBD Alibaba Cloud |
UNK | ✔️ | 61.34 | 64.95 | |
| Jun 7, 2024 |
TBD Soochow University |
UNK | ✔️ | -- | 64.62 | |
| Aug 30, 2024 | TBD Fudan University + Transwarp Technology |
UNK | ✔️ | 62.97 | 64.51 | |
| Oct 10, 2024 | TBD Wuhan University of Technology |
236B | ✔️ | 66.82 | 64.00 | |
| Feb 6, 2025 | TBD Northeastern University |
UNK | ✔️ | 64.60 | 63.50 | |
| Feb 21, 2024 |
TBD Anonymous |
13B | ✔️ | 55.48 | 63.39 | |
| Mar 11, 2025 |
TBD onekq.ai |
[link] | 32B | ✔️ | 64.60 | 63.33 |
| Apr 10, 2024 |
TBD Tencent CDP-youpu |
UNK | ✔️ | 62.58 | 63.22 | |
| Feb 21, 2026 | TBD Baseline |
UNK | ✔️ | New Dev 63.82 |
62.94 |
|
| Jun 1, 2024 |
TBD HKUST(GZ) [Li et al. '24] |
[link] | UNK | ✔️ | 58.50 | 62.66 |
| Aug 1, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 0.5B | ✔️ | 56.87 | 61.82 |
| Mar 27, 2024 |
TBD PingCAP |
[link] | UNK | ✔️ | 58.15 | 60.98 |
| Feb 21, 2026 | TBD Baseline |
UNK | ✔️ | New Dev 61.67 |
60.93 |
|
| Nov 16, 2023 |
TBD Mercator Technologies |
UNK | ✔️ | 59.71 | 60.71 | |
| Jan 30, 2026 |
TBD Crater Labs [Thaker et al. CAIAC'26] |
[link] | 4B | ✔️ | 54.63 | 60.42 |
| Oct 12, 2023 |
TBD Renmin University of China [Li et al. SIGMOD'24] |
[link] | 15B | ✔️ | 58.47 | 60.37 |
| Feb 27, 2024 |
TBD University of Alberta [Pourreza et al. '24] |
[link] | 7B | ✔️ | 55.8 | 60.31 |
| Feb 21, 2026 | TBD Baseline |
UNK | ✔️ | New Dev 60.63 |
59.87 |
|
| Sep 23, 2024 |
TBD Bilkent University [Caferoğlu et al.'24] |
[link] | UNK | ✔️ | 61.60 | 59.81 |
| Nov 21, 2023 |
TBD BUAA & Tencent [Wang et al. '23] |
UNK | ✔️ | 57.56 | 59.59 | |
| Jul 17, 2026 |
TBD Martin Luther University Halle-Wittenberg |
[link] | 0.8B | 53.52 | 59.59 | |
| Oct 12, 2023 |
TBD Renmin University of China [Li et al. SIGMOD'24] |
[link] | 7B | ✔️ | 57.17 | 59.25 |
| May 27, 2024 |
TBD HKU [Qu et al. ACL Findings'24] |
[link] | UNK | ✔️ | 56.19 | 59.14 |
| Nov 09, 2023 |
TBD Alibaba Group [Gao and Wang et al. VLDB'24] |
[link] | UNK | ✔️ | 54.76 | 57.41 |
| May 24, 2024 |
TBD IBM Research AI |
20B | 51.69 | 57.13 | ||
| Aug 10, 2024 |
TBD Baseline |
[link] | 236B | ✔️ | 56.13 | 56.68 |
| Aug 15, 2023 |
TBD University of Alberta [Pourreza et al. '23] |
[link] | UNK | ✔️ | 50.72 | 55.90 |
| Aug 08, 2024 |
TBD Baseline |
[link] | 123B | ✔️ | 53.52 | 55.84 |
| Jul 01, 2023 |
TBD Baseline |
[link] | UNK | ✔️ | 46.35 | 54.89 |
| Nov 8, 2024 |
TBD Peking University [Xiong et al. CIKM'25] |
[link] | UNK | 54.56 | 54.11 | |
| Sep 19, 2024 |
TBD Prem AI |
[link] | 1B | ✔️ | - | 51.54 |
| Jul 16, 2023 |
TBD Baseline |
[link] | UNK | ✔️ | 42.70 | 49.02 |
| Nov 23, 2023 |
TBD Anonymous |
7B | ✔️ | 37.68 | 47.74 | |
| Mar 17, 2023 |
TBD HKU & DAMO [Li et al. NeurIPS'23] |
[link] | UNK | ✔️ | 36.64 | 40.08 |
| Mar 17, 2023 |
TBD Baseline |
UNK | ✔️ | 37.22 | 39.30 | |
| Feb 17, 2023 |
TBD Baseline |
175B | ✔️ | 34.35 | 36.47 | |
| Aug 24, 2026 | TBD [Shiven Khurdi '26] |
[link] | 394M | ✔️ | 23.51 | 33.15 |
| Jul 16, 2023 |
TBD Baseline |
[link] | UNK | ✔️ | 27.38 | 33.04 |
| Mar 17, 2023 |
TBD HKU & DAMO [Li et al. NeurIPS'23] |
[link] | UNK | 25.88 | 28.95 | |
| Aug 24, 2026 | TBD [Shiven Khurdi '26] |
[link] | 394M | ✔️ | 19.23 | 27.33 |
| Mar 17, 2023 |
TBD Baseline |
UNK | 24.05 | 26.77 | ||
| Feb 17, 2023 |
TBD Baseline |
175B | 25.42 | 24.86 | ||
| Feb 5, 2023 |
TBD Baseline |
3B | ✔️ | 23.34 | 24.05 | |
| Feb 3, 2023 |
TBD Baseline |
770M | ✔️ | 19.75 | 20.94 | |
| Feb 3, 2023 |
TBD Baseline |
220M | ✔️ | 11.54 | 12.89 | |
| Feb 5, 2023 |
TBD Baseline |
3B | 10.37 | 11.17 | ||
| Feb 3, 2023 |
TBD Baseline |
770M | 9.71 | 10.38 | ||
| Feb 3, 2023 |
TBD Baseline |
220M | 6.32 | 7.06 |
Leaderboard - Reward-based Valid Efficiency Score (R-VES)
| Model | Code | Size | Oracle Knowledge | Test | |
|---|---|---|---|---|---|
| TBD Data Engineers + DB Students |
✔️ | 83.26 | |||
| Sep 07, 2026 | TBD TBD [TBD] |
UNK | ✔️ | 77.64 | |
| Sep 25, 2025 | TBD TBD TBD |
[link] | UNK | ✔️ | 77.00 |
| Dec 16, 2025 | TBD TBD [TBD] |
UNK | ✔️ | 76.31 | |
| Aug 19, 2026 | TBD TBD |
UNK | ✔️ | 74.95 | |
| Aug 22, 2026 | TBD TBD |
UNK | ✔️ | 74.76 | |
| Sep 07, 2026 | TBD Anonymous |
UNK | ✔️ | 74.20 | |
| Jul 04, 2026 | TBD Anonymous |
UNK | ✔️ | 74.68 | |
| May 27, 2026 | TBD TBD |
UNK | ✔️ | 73.93 | |
| Sep 07, 2026 | TBD Anonymous |
UNK | ✔️ | 73.77 | |
| Jun 19, 2026 | TBD TBD |
UNK | ✔️ | 73.75 | |
| Jul 10, 2026 | TBD HKUST(GZ) [Boyan Li et al. '25] |
[link] | 27B | ✔️ | 72.26 |
| July 14, 2025 | TBD LongShine AI Research |
UNK | ✔️ | 71.89 | |
| May 18, 2026 | TBD AIC-Lab, Chungbuk National University |
UNK | ✔️ | 71.45 | |
| Dec 17, 2024 | TBD Alibaba Cloud [Yifu Liu et al. '24] |
[link] | UNK | ✔️ | 71.41 |
| Oct 27, 2024 |
TBD IBM Research AI |
34B | ✔️ | 71.37 | |
| Feb 21, 2026 | TBD South China Normal University |
30B | ✔️ | 71.18 | |
| Apr 3, 2026 | TBD wenyuai |
UNK | ✔️ | 70.85 | |
| Jan 26, 2026 | TBD HKUST(GZ) [Boyan Li et al. '25] |
[link] | UNK | ✔️ | 70.43 |
| Oct 23, 2025 | TBD Sinovatio AI Lab |
UNK | ✔️ | 70.41 | |
| Jul 07, 2026 | TBD MR Tech |
UNK | ✔️ | 70.29 | |
| Sep 22, 2025 | TBD JD:CHO-JDT-JDL |
UNK | ✔️ | 70.16 | |
| Feb 27, 2025 | TBD Contextual AI |
UNK | ✔️ | 70.02 | |
| Apr 16, 2025 |
TBD Google Cloud [Pourreza et al. '24] |
UNK | ✔️ | 69.94 | |
| Jul 14, 2026 | TBD HKUST(GZ) [Boyan Li et al. '26] |
[link] | UNK | ✔️ | 69.85 |
| Sep 1, 2025 | TBD Anonymous |
UNK | ✔️ | 69.54 | |
| Aug 21, 2024 | TBD Alibaba Cloud |
UNK | ✔️ | 69.36 | |
| May 18, 2026 | TBD Kuaishou Business Team |
UNK | ✔️ | 69.32 | |
| Mar 31, 2025 | TBD Google Cloud / Stanford [Pourreza et al. '25] |
14 B | ✔️ | 68.67 | |
| Aug 03, 2026 | TBD UST |
[link] | 31B | ✔️ | 68.35 |
| May 27, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 32B | ✔️ | 67.84 |
| May 27, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 7B | ✔️ | 67.47 |
| Jul 22, 2024 | TBD Distyl AI Research |
UNK | ✔️ | 67.41 | |
| Mar 4, 2025 | TBD Renmin University of China + ByteDance Infra Lab [Li et al. '25] |
[link] | 32B | ✔️ | 67.05 |
| May 21, 2024 | TBDIR +CG +UT Stanford [Talaei et al.'24] |
[link] | UNK | ✔️ | 66.53 |
| Jul 5, 2024 | TBD Uber Freight |
UNK | ✔️ | 66.39 | |
| May 14, 2024 | TBD IBM Research AI |
20B | ✔️ | 66.25 | |
| Aug 17, 2026 | TBD Tampere University |
[link] | 27B | ✔️ | 65.90 |
| Jul 14, 2024 | TBD Google Cloud |
UNK | ✔️ | 65.70 | |
| Aug 30, 2024 | TBD Fudan University + Transwarp Technology |
UNK | ✔️ | 65.62 | |
| Aug 06, 2025 | TBD Gena Co. gena.co |
UNK | ✔️ | 65.52 | |
| Aug 1, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 1.5B | ✔️ | 65.25 |
| Dec 18, 2024 | TBD mindflow.ai |
UNK | ✔️ | 65.19 | |
| Mar 4, 2025 | TBD Renmin University of China + ByteDance Infra Lab [Li et al. '25] |
[link] | 7B | ✔️ | 65.04 |
| Apr 10, 2025 | TBD queryosity.io |
UNK | ✔️ | 64.73 | |
| Jan 9, 2025 | TBD Anonymous |
[link] | UNK | ✔️ | 64.41 |
| Jan 09, 2025 | TBD Alibaba Cloud [Yifu Liu et al. '24] |
[link] | 32B | ✔️ | 64.65 |
| Aug 29, 2024 | TBD Arcwise |
UNK | ✔️ | 63.68 | |
| Nov 11, 2024 | TBD AT&T CDO - DSAIR [Shkapenyuk et al. '25] |
UNK | 63.34 | ||
| May 21, 2024 | TBDIR +SS +CG Stanford [Talaei et al.'24] |
[link] | UNK | ✔️ | 62.77 |
| Sep 23, 2024 | TBD Bilkent University [Caferoğlu et al.'24] |
[link] | UNK | ✔️ | 62.43 |
| Aug 06, 2026 | TBD Anonymous |
UNK | ✔️ | 62.06 | |
| Jun 7, 2024 | TBD Soochow University |
UNK | ✔️ | 61.37 | |
| Aug 20, 2024 | TBD Xffuture |
UNK | ✔️ | 61.28 | |
| Jan 14, 2024 | TBD Dunamu [Lee et al. '24] |
UNK | ✔️ | 61.23 | |
| Sep 10, 2025 | TBD MathCo |
UNK | ✔️ | 60.51 | Feb 27, 2024 | PB-SQL Seoul National University |
UNK | ✔️ | 60.36 |
| Aug 30, 2024 | TBD Fudan University + Transwarp Technology |
UNK | ✔️ | 60.35 | |
| Feb 6, 2025 | TBD Northeastern University |
UNK | ✔️ | 60.13 | |
| Mar 11, 2025 | TBD onekq.ai |
[link] | 32B | ✔️ | 60.02 |
| Apr 30, 2025 |
TBD Cohere AI [Team Cohere et al.'25] |
[link] | 111B | ✔️ | 59.82 |
| Oct 10, 2024 | TBD Wuhan University of Technology |
236B | ✔️ | 59.42 | |
| Nov 21, 2023 | TBD BUAA & Tencent [Wang et al. '23] |
UNK | ✔️ | 57.60 | |
| Aug 1, 2025 | TBD Wuhan University of Technology + University of Science and Technology of China [Lei Sheng et al. '25] |
[link] | 0.5B | ✔️ | 57.11 |
| Oct 12, 2023 | TBD Renmin University of China [Li et al. SIGMOD'24] |
[link] | 15B | ✔️ | 56.73 |
| Apr 10, 2024 | TBD Tencent CDP-youpu |
UNK | ✔️ | 56.63 | |
| Nov 16, 2023 | TBD Mercator Technologies |
UNK | ✔️ | 56.63 | |
| May 24, 2024 | TBD IBM Research AI |
20B | 56.11 | Mar 27, 2024 | {Chat2Query} (GPT-4 + data entity modeling) (PingCAP) PingCAP |
[link] | UNK | ✔️ | 56.06 |
| Oct 12, 2023 | TBD Renmin University of China [Li et al. SIGMOD'24] |
[link] | 7B | ✔️ | 55.69 |
| Sep 23, 2024 | TBD Bilkent University [Caferoğlu et al.'24] |
[link] | UNK | ✔️ | 55.64 |
| Nov 09, 2023 | TBD Alibaba Group [Gao and Wang et al. VLDB'24] |
[link] | UNK | ✔️ | 54.02 |
| Jul 17, 2026 | TBD Martin Luther University Halle-Wittenberg |
[link] | 0.8B | 53.43 | |
| Aug 10, 2024 | TBD Baseline |
[link] | 236B | ✔️ | 53.25 |
| Aug 15, 2023 | TBD University of Alberta [Pourreza et al. '23] |
[link] | UNK | ✔️ | 53.07 |
| Aug 08, 2024 | TBD Baseline |
[link] | 128B | ✔️ | 52.59 |
| Jul 01, 2023 | TBD Baseline |
[link] | UNK | ✔️ | 51.75 |
BIRD Mini-Dev
A Lite version of developtment dataset, which is designed to facilitate efficient and cost-effective development cycles, especially for testing and refining SQL query generation models. For more details, please visit the GitHub repository. For updating Leaderboard, please make sure your paper or resource is public available and submit a PR.
Mini Dev - Execution Accuracy (EX)
| Model | Code | Size | Oracle Knowledge | SQLite | MySQL | PostgreSQL | |
|---|---|---|---|---|---|---|---|
| Aug 21, 2026 |
TBD DeepLethe [report] |
[link] | UNK | ✔️ | 70.20 | - | 65.80 |
| Jul 22, 2026 |
TBD Quanam SmartQuery |
UNK | ✔️ | - | - | 56.80 | |
| Jun 31, 2024 |
TBD HKU [Qu et al. ACL Findings'24] |
[link] | UNK | ✔️ | 58.00 | 49.20 | 50.80 |
| Jun 31, 2024 | TBD | UNK | ✔️ | 47.80 | 40.80 | 35.80 | |
| Jun 31, 2024 | TBD | UNK | ✔️ | 47.00 | 43.20 | 35.00 | |
| Jun 31, 2024 | TBD | UNK | ✔️ | 45.80 | 41.00 | 36.00 | |
| Jun 30, 2026 |
TBD Crater Labs [Thaker et al. CAIAC'26] |
[link] | 4B | ✔️ | 45.00 | - | - |
| Jun 31, 2024 | TBD | 70B | ✔️ | 40.80 | 37.00 | 29.40 | |
| Jun 31, 2024 | TBD | UNK | ✔️ | 38.00 | 36.00 | 27.40 | |
| Jun 31, 2024 | TBD | UNK | ✔️ | 33.60 | 31.20 | 26.60 | |
| Jun 31, 2024 | TBD | 13B | ✔️ | 30.60 | 25.00 | 21.60 | |
| Jun 31, 2024 | TBD | 8B | ✔️ | 24.40 | 24.60 | 18.40 | |
| Jun 31, 2024 | TBD | 46.7B | ✔️ | 21.60 | 13.60 | 12.40 |