Register for the benchmark

This is a partial and plain-language summary meant to help you understand some of the key points of the terms of engagement. The full Terms of Access and Participation control and is the authoritative and complete version.

What you're agreeing to

By requesting and engaging with the benchmark and its materials, you agree to Schemantic's Terms of Access and Participation.

License

You get a limited, revocable license to use the materials for evaluating your systems and internal research.

As-is

Everything — materials, scoring, infrastructure — is provided as-is with no guarantees. Some content is LLM-generated and may contain errors. The benchmark package includes executable code (scoring harness, SQL queries). You are responsible for reviewing and running this code in an appropriate environment. Schemantic is not liable for any damage resulting from executing benchmark materials.

Liability

The benchmark is free and provided as-is. You assume the risks described in the full terms and agree to indemnify Schemantic and its related parties and entities.

Your data

We collect registration info to run the benchmark. Submissions may be retained up to 60 months for scoring and research, or longer as the team otherwise decides. You can request deletion per our Privacy Policy.

Attribution

You must cite the benchmark properly when publishing results, using the citation format at schemantic.io/benchmark.

Submission limits

One submission per test set. Rate-limited to one test set per quarter, unless otherwise permitted by the Schemantic team.

Export controls

You may not use benchmark materials in violation of U.S. export control laws or sanctions regulations.

Indemnification

You agree to indemnify and hold harmless Schemantic, its founders, employees, and affiliates from claims arising from your use of benchmark materials or violation of these terms.

Scoring timeline

Scoring is done on a volunteer basis. Results may take up to 100 days or longer. We're a small team and appreciate your patience.

Changes

We may modify, suspend, or discontinue the benchmark, scoring methodology, or these terms at any time. Continued participation after changes constitutes acceptance.

Disputes

Washington state law. Binding arbitration in King County, WA. No class actions, no jury trials. This benchmark is a free resource we maintain for the research community — these terms exist to keep it that way.