By checking "I Agree" and accessing the (S)uper (H)ard r(EL)ati(O)nal (B)enchmark (the "Benchmark") materials, you ("Participant") enter a binding agreement with Schemantic, LLC, a Washington limited liability company ("Company"). You affirm that you are at least 18 years of age or the age of majority in your jurisdiction, whichever is greater, and that you have authority to bind any organization on whose behalf you access these materials.
1. Definitions.
"Benchmark Materials" means all artifacts provided to Participant, including without limitation generated Parquet files, schemas, query history, prompts, gold SQL, scoring harnesses, documentation, updates, and any other materials made available in connection with the Benchmark. "Submission" means any SQL output, result sets, metadata, or other artifacts Participant provides for evaluation. "Leaderboard" means any public or private ranking of Participant scores. "Generator" means the proprietary system, including all components, dependencies, and auxiliary systems, that produces Benchmark Materials. "Difficulty Profile" means the target distribution of evaluation parameters, including without limitation entity counts, source system counts, table counts, mutation coverage, bridge table quality tiers, prompt difficulty, and any other parameters Company may add, modify, or remove at its discretion, across the difficulty matrix. The Difficulty Profile is a statistical target, not a guarantee of identical difficulty across individual evaluations. Company may modify the Difficulty Profile at any time without notice. "Site" means schemantic.io and any successor domains operated by Company. "Personal Information" means any information that identifies or could reasonably be used to identify Participant, including without limitation name, email address, IP address, account identifier, organizational affiliation, and any other data collected during registration or use of the Benchmark. "Third-Party Services" means any third-party platforms, services, or infrastructure used in connection with the Benchmark, including without limitation cloud hosting providers, data warehouse providers, LLM API providers, content delivery networks, and authentication services. "Affiliate" means any entity that directly or indirectly controls, is controlled by, or is under common control with Company, where "control" means ownership of more than fifty percent (50%) of the voting securities or equivalent ownership interest. Where Participant accesses the Benchmark on behalf of an organization, all references to "Participant" in these Terms apply equally to the organization and to each individual authorized by the organization to access the Benchmark; the organization is jointly and severally liable for the acts and omissions of its authorized individuals.
2. License Grant.
Company grants Participant a limited, non-exclusive, non-transferable, non-sublicensable, revocable license to use Benchmark Materials for the purpose of evaluating text-to-SQL and semantic discovery systems against the Benchmark and for related internal research purposes, including the uses permitted under Section 3(a), subject to the restrictions in Section 3. The Generator, including all algorithms, processes, techniques, parameters, configurations, prompt templates, difficulty calibration methods, noise taxonomies, and any other methodology used to produce Benchmark Materials, constitutes the proprietary trade secrets and confidential information of Company. No right, title, or interest in the Generator or its underlying processes is granted or implied by this license.
This license terminates automatically upon breach of any term herein. Company may revoke this license and terminate Participant's access at any time, for any reason or no reason, with or without notice, at Company's sole discretion. No advance notice, cure period, or explanation is required. For the avoidance of doubt, participation in the Benchmark by commercial entities, including those operating competitive products in the text-to-SQL, semantic layer, or data analytics markets, is permitted and encouraged. The Prohibited Activities section of the Schemantic Terms of Service does not restrict Benchmark participation or use of Benchmark Materials as authorized under these Terms.
3. Restrictions.
Participant shall not: (a) redistribute, sublicense, sell, or publicly host Benchmark Materials or any derivative thereof for commercial purposes beyond Participant's own internal research and Benchmark participation; Participant may use Benchmark Materials for research purposes, including training or fine-tuning machine learning models, provided that any such use is entirely at Participant's own risk and subject to the disclaimers and limitations of liability herein; (b) publicly attribute Benchmark results to Company as an endorsement of any product or system; (c) publish, disclose, or publicly reference Benchmark scores without including the Benchmark version, date of evaluation, and unmodified difficulty distribution as reported in the test package; (d) misrepresent Benchmark methodology, scoring, difficulty parameters, or results, including through selective disclosure, misleading contextualization, or omission of material evaluation parameters; (e) submit evaluations more frequently than the submission frequency limits set forth in Section 6, or circumvent such limits by any means; (f) use Benchmark Materials or Benchmark results in any publication, presentation, press release, marketing material, or public disclosure without attribution to the (S)uper (H)ard r(EL)ati(O)nal (B)enchmark and Schemantic using the citation format specified at schemantic.io/benchmark or a substantially equivalent attribution; (g) interfere with, manipulate, or attempt to influence another Participant's evaluation, Submission, score, or Leaderboard position, whether by technical means, social engineering, or any other method; (h) access, use, distribute, or re-export Benchmark Materials in violation of any applicable U.S. export control laws, sanctions regulations, or trade compliance requirements, including the Export Administration Regulations (EAR) and regulations administered by the Office of Foreign Assets Control (OFAC).
4. Participant Representations and Warranties.
Participant represents and warrants that: (a) Participant has the legal capacity and authority to enter into these Terms and to perform its obligations hereunder; (b) if Participant is accessing the Benchmark on behalf of an organization, Participant has the authority to bind that organization; (c) all information provided by Participant during registration and Benchmark use is accurate, complete, and not misleading; (d) Participant's Submissions do not contain any malicious code, including without limitation viruses, worms, Trojan horses, SQL injection payloads, or any code designed to disrupt, disable, damage, or gain unauthorized access to Company's systems or the systems of any other Participant or third party; (e) Participant's Submissions and Participant's use of Benchmark Materials do not infringe, misappropriate, or violate any third party's intellectual property rights, privacy rights, or other legal rights; (f) Participant's access to and use of the Benchmark complies with all applicable laws and regulations, including without limitation export control laws, data protection laws, and computer fraud and abuse laws; and (g) Participant is not accessing the Benchmark primarily for the purpose of monitoring availability, performance, or functionality, or for any benchmarking or competitive analysis of Company's infrastructure (as opposed to evaluating Participant's own systems against the Benchmark). Breach of any representation or warranty in this Section constitutes a material breach of these Terms and grounds for immediate termination of Participant's access.
5. Intellectual Property.
All rights, title, and interest in the Generator, including all algorithms, source code, processes, techniques, parameters, configurations, prompt engineering methods, difficulty calibration systems, noise taxonomies, scoring methodology, difficulty parameterization, and all associated documentation remain exclusively with Company and constitute Company's proprietary trade secrets. Participant retains ownership of its own systems, methods, and source code. No intellectual property rights are transferred by this agreement in either direction.
Participant retains ownership of its Submissions. Participant grants Company a non-exclusive, royalty-free, worldwide, perpetual license to use Submissions solely for the purposes of scoring, Leaderboard publication (if Participant opts in), and aggregated research analysis. Company acquires no rights to Participant's underlying systems, methods, or intellectual property through receipt of Submissions.
Company does not intend to publish Participant's individual Submission content (i.e., Participant's actual SQL queries) without Participant's express consent; however, this statement of intent does not constitute a binding obligation, and Company shall not be liable for any incidental or inadvertent disclosure of Submission content. This statement does not apply to scores, rankings, or aggregated and anonymized data, which Company may publish without restriction. Company may use aggregated, anonymized Benchmark submission data for research and publication. Company does not intend to publish individual results without Participant's express opt-in consent, but this statement of intent does not create an independent obligation enforceable by Participant.
Company may retain Submissions for a period of up to sixty (60) months following the date of Submission for purposes of scoring verification, dispute resolution, and aggregated research. After such period, Company may, but is not obligated to, delete Submission data. Company assumes no obligation to retain, archive, back up, or return any Submission or associated data, and shall not be liable for loss or deletion of Submissions at any time.
Any public use of Benchmark Materials, Benchmark results, or derivative analysis thereof must include attribution to the (S)uper (H)ard r(EL)ati(O)nal (B)enchmark and Schemantic in the form specified at schemantic.io/benchmark. This requirement survives termination of this agreement.
6. Evaluation and Scoring.
The Benchmark is operated on a volunteer basis at no cost to Participant. Company does not guarantee any particular level of service, uptime, or availability. The Benchmark website, scoring infrastructure, data warehouse, and associated services may be unavailable, degraded, or discontinued at any time without notice. Company assumes no obligation to maintain, update, or restore any component of the Benchmark infrastructure.
Company may, in its sole discretion, score Submissions. Company is under no obligation to score any Submission and may decline to score any Submission for any reason or no reason. If Company elects to score a Submission, Participant acknowledges that Company may take up to one hundred (100) calendar days or longer from the date of Submission to return scored results. The 100-day period is an estimate, not a commitment; failure to score within 100 days or any other timeframe does not constitute a breach of these Terms. Company shall not be liable for delays in scoring, failure to score, or decisions not to score, regardless of the reason.
Participant may submit one (1) evaluation per ninety (90) calendar day period. The 90-day period begins on the date Company acknowledges receipt of a Submission. Submissions must be delivered by email to the address designated on the Site; Submissions delivered by any other means (including without limitation web upload, API, or physical media) will not be accepted. Company may, at its sole discretion, waive or modify the submission frequency limit for individual Participants. Submission of an evaluation does not guarantee that it will be scored.
Benchmark Materials, including schemas, prompts, gold SQL, and scoring harnesses, are generated in whole or in part using large language models ("LLMs") and other automated systems. Company does not warrant the correctness, completeness, consistency, or fitness of any LLM-generated content. Errors, hallucinations, inconsistencies, or artifacts introduced by LLMs may appear in any Benchmark Material, including gold SQL used for scoring. Company is not responsible for, and disclaims all liability arising from, any defect, inaccuracy, or bias attributable to LLM-generated content. Participant accepts Benchmark Materials as delivered, with full knowledge that automated generation may introduce errors that affect scoring outcomes.
The scoring harness is provided as-is. Company does not warrant that the scoring harness is free from bugs, logic errors, or implementation defects. If a scoring harness error is discovered, Company may, at its sole discretion, re-score affected Submissions, adjust Leaderboard entries, or take no action. Company is under no obligation to investigate, disclose, or remediate scoring harness errors, and Participant waives any claim arising from scoring harness defects.
All data in Benchmark Materials is synthetically generated. Any resemblance to real persons, organizations, products, schemas, trade secrets, proprietary systems, or actual business data is coincidental and unintentional. Company makes no representation that generated data is or is not similar to any real-world data, and disclaims all liability for any claim based on perceived similarity between Benchmark Materials and any Participant's or third party's proprietary information.
7. Leaderboard Governance.
The Leaderboard is a limited-capacity resource. Company reserves the right to cap the total number of entries displayed on the Leaderboard at any time and at its sole discretion. Company is not obligated to display all Participants who have submitted evaluations. If the Leaderboard is at capacity, Company may, at its sole discretion, remove lower-ranked entries, archive older entries, or decline to add new entries. Each Participant (or organization, if Participant is participating on behalf of an organization) may hold a maximum of one (1) Leaderboard entry at a time, regardless of the number of individuals within that organization who access the Benchmark. Multiple individuals within the same organization share the organization's single-entry allocation. Company may grant exceptions at its sole discretion. If a Participant submits an additional evaluation that results in a qualifying score, the new score replaces the existing entry, at Company's discretion.
Leaderboard entries must display the Participant's legal name or the legal name of the organization on whose behalf the Participant is participating. Pseudonyms, handles, screen names, nicknames, trade names not registered as legal entities, and any other non-verifiable identifiers are prohibited. Company may, at its sole discretion, require Participant to provide documentary verification of identity, including without limitation government-issued identification, articles of incorporation, or other evidence of legal name, before publishing or maintaining a Leaderboard entry. Failure to provide requested verification within thirty (30) calendar days of Company's request constitutes grounds for removal of the Leaderboard entry and suspension of Participant's access. Company may reject or remove any Leaderboard entry that, in Company's sole judgment, uses an unverifiable, misleading, or inappropriate identifier.
The Benchmark is not a contest, sweepstakes, lottery, promotion, or competition within the meaning of any federal, state, or foreign law. No prizes, awards, compensation, financial incentives, or other things of value are offered, promised, or implied in connection with the Benchmark or the Leaderboard. Leaderboard placement confers no entitlement, property right, or financial interest. Company makes no representation that Leaderboard placement will result in any commercial, employment, reputational, or other benefit to Participant.
Leaderboard publication is separate from and not required for Benchmark participation. Participant may access and use Benchmark Materials without opting in to the Leaderboard. Leaderboard opt-in occurs at the time of Submission through a separate, affirmative selection. By opting in to Leaderboard publication, Participant grants Company a non-exclusive, royalty-free, worldwide, perpetual license to publish, display, reproduce, and distribute Participant's legal name (or organizational name), Leaderboard ranking, Benchmark scores, evaluation date, and Benchmark version in connection with the Leaderboard, the Site, Company's marketing and promotional materials, research publications, and press communications. This license survives termination of these Terms. Participant waives any right of publicity, right of privacy, or moral rights claim arising from Company's publication of Leaderboard data as described in this Section.
Leaderboard rankings reflect scored output against the evaluation as generated and do not constitute Company's opinion, endorsement, recommendation, or assessment of the quality, fitness, reliability, or commercial viability of any Participant's system, product, or service. Leaderboard rankings are not statements of fact regarding the absolute or relative capabilities of any system. Company disclaims all liability for any interpretation of Leaderboard rankings as endorsements, recommendations, or qualitative assessments.
If Participant uses Leaderboard rankings, Benchmark scores, or any data derived from the Benchmark in advertising, marketing, sales materials, press releases, or any other public communication, Participant is solely responsible for compliance with all applicable laws and regulations, including without limitation the Federal Trade Commission Act (15 U.S.C. § 41 et seq.), FTC Endorsement Guides (16 C.F.R. Part 255), and any applicable state consumer protection and truth-in-advertising laws. Company is not responsible for, and disclaims all liability arising from, Participant's use of Benchmark data in marketing or advertising, including any claim that such use constitutes a false, misleading, or deceptive trade practice.
Participant may request removal of Participant's name from the Leaderboard at any time by written notice to Company. Company may process removal requests at its discretion and is not bound to any specific processing timeline. Upon removal, Company may retain anonymized score data (with Participant's name and identifying information removed) for aggregated research and historical statistical purposes. Removal of a Leaderboard entry does not terminate these Terms or release Participant from any surviving obligation. Company is not required to remove references to Participant's Leaderboard ranking from previously published materials, cached pages, third-party reproductions, or archived content outside Company's control.
Company reserves the right to modify the Leaderboard format, ranking methodology, display criteria, entry limits, and verification requirements at any time without notice. Participant waives any claim arising from changes to Leaderboard governance, including removal of entries, changes to ranking methodology, or imposition of new eligibility or verification requirements.
8. Data Collection and Communications.
To access the Benchmark, Participant must provide certain Personal Information, including name, email address, and organizational affiliation. Participant consents to Company's collection, storage, processing, and use of Personal Information for the purposes of: (a) administering the Benchmark, including account creation, authentication, scoring, and Leaderboard management; (b) communicating with Participant regarding Submissions, scores, Leaderboard status, and Benchmark updates; (c) aggregated analytics and research regarding Benchmark usage and participation; and (d) compliance with legal obligations, including export control verification and consent documentation. Company's collection and use of Personal Information is further governed by the Privacy Policy at schemantic.io/privacy.
By accessing the Benchmark, Participant opts in to receive marketing and promotional communications from Company, including announcements regarding Company products, services, events, and partner offerings, delivered via email or other electronic means. Participant may unsubscribe from marketing communications at any time by using the unsubscribe mechanism provided in each communication or by contacting Company at the address specified on the Site. Unsubscription from marketing communications does not affect Company's right to send transactional, operational, or Benchmark-related communications, including scoring notifications, Leaderboard updates, Terms modifications, security notices, and account-related messages. Company will process unsubscribe requests within ten (10) business days. All marketing communications will comply with applicable laws, including the CAN-SPAM Act (15 U.S.C. § 7701 et seq.) and, where applicable, the General Data Protection Regulation (EU 2016/679).
Participant consents to receive all notices, disclosures, and communications related to these Terms and the Benchmark by electronic means, including email to the address provided during registration. Electronic delivery of any notice satisfies any legal requirement that such notice be in writing. It is Participant's responsibility to maintain a current email address on file with Company. Company is not responsible for notices undelivered due to Participant's failure to maintain a current email address or due to email filtering, spam blocking, or other factors outside Company's control.
Participant represents that all Personal Information provided to Company is accurate and current, and agrees to update such information promptly if it changes. Company is not liable for any consequence arising from Participant's provision of inaccurate, outdated, or incomplete Personal Information. Company may suspend or terminate Participant's access if Company reasonably believes that Participant's Personal Information is materially inaccurate.
Company may share Personal Information with third-party service providers solely for the purposes described in this Section, subject to the Privacy Policy. Company will not sell Personal Information to third parties. In the event of a merger, acquisition, or sale of substantially all of Company's assets, Personal Information may be transferred to the successor entity, and Participant consents to such transfer.
Participant may request deletion of Participant's Personal Information by submitting a written request to Company at the contact address specified on the Site or via email to the address designated in the Privacy Policy. Company will process verified deletion requests in accordance with applicable law, including any legally mandated response periods. Company may require Participant to verify identity before processing a deletion request. Upon completion of a deletion request, Company will delete or de-identify Participant's Personal Information from its active systems, subject to the following exceptions: (a) Company may retain Personal Information required for compliance with applicable law, including without limitation tax, legal reporting, and audit obligations; (b) Company may retain acceptance records as described in Section 26 (Acceptance Records) for the retention period specified therein; (c) Company may retain data necessary to resolve pending disputes, enforce these Terms, or exercise legal claims; (d) Company may retain anonymized and aggregated data from which Participant cannot reasonably be identified; and (e) Company may retain Submission data for the retention period specified in Section 5 (Intellectual Property), subject to anonymization upon Participant's request. Deletion of Personal Information does not obligate Company to purge data from backup systems, disaster recovery archives, or offline storage, provided that such data will be deleted in the ordinary course of backup rotation and will not be used for active processing following the deletion request. Company may, but is not obligated to, direct Third-Party Service providers to delete Participant's Personal Information. Company does not warrant or guarantee third-party compliance with deletion requests and shall not be liable for any Third-Party Service provider's failure to delete data.
In the event of a security incident affecting Participant's Personal Information, Company will comply with applicable data breach notification laws, including RCW 19.255.010 and any other applicable state or federal requirements. Company's obligations with respect to any security incident are limited to those required by applicable law. Company disclaims all liability for consequential, incidental, or indirect damages arising from any security incident, subject to the limitations in Section 10. Company is not obligated to provide credit monitoring, identity theft protection, or any other remedial service beyond what is required by applicable law.
9. Disclaimer of Warranties.
ALL BENCHMARK MATERIALS ARE PROVIDED "AS IS" AND "AS AVAILABLE." COMPANY AND ITS AFFILIATES AND LICENSORS MAKE NO WARRANTY OF ANY KIND, WHETHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, INCLUDING WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, SATISFACTORY QUALITY, NON-INFRINGEMENT, AND QUIET ENJOYMENT, AND ANY WARRANTIES ARISING OUT OF ANY COURSE OF DEALING OR TRADE USAGE. COMPANY DOES NOT WARRANT THAT BENCHMARK MATERIALS WILL BE UNINTERRUPTED, ACCURATE, ERROR-FREE, OR FREE OF HARMFUL COMPONENTS. COMPANY DOES NOT WARRANT THAT BENCHMARK RESULTS CONSTITUTE A DEFINITIVE OR REPRODUCIBLE MEASURE OF ANY SYSTEM'S PERFORMANCE.
The Benchmark depends on Third-Party Services, including cloud hosting, data warehouse infrastructure, and LLM APIs. Company does not warrant the performance, availability, accuracy, security, or reliability of any Third-Party Service. Company disclaims all liability for any failure, error, delay, outage, data loss, or security incident attributable to any Third-Party Service. Participant acknowledges that Third-Party Services are governed by their own terms and conditions, over which Company has no control.
COMPANY DOES NOT PROVIDE ANY INDEMNIFICATION, DEFENSE, OR HOLD-HARMLESS OBLIGATION TO PARTICIPANT FOR ANY CLAIM ARISING FROM PARTICIPANT'S USE OF BENCHMARK MATERIALS, INCLUDING WITHOUT LIMITATION CLAIMS THAT BENCHMARK MATERIALS OR ANY LLM-GENERATED CONTENT THEREIN INFRINGE, MISAPPROPRIATE, OR VIOLATE ANY THIRD PARTY'S INTELLECTUAL PROPERTY RIGHTS, PRIVACY RIGHTS, OR OTHER RIGHTS. PARTICIPANT USES BENCHMARK MATERIALS AT ITS SOLE RISK AND IS SOLELY RESPONSIBLE FOR EVALUATING AND BEARING ALL RISKS ASSOCIATED WITH SUCH USE.
Participant acknowledges that Benchmark results may vary due to factors outside Company's control, including model versioning, inference configuration, data warehouse load, and query execution variability.
Participant acknowledges that each Benchmark evaluation is generated procedurally and that no two evaluations share the same schema, entity names, join structure, or prompts. While the Generator targets a consistent Difficulty Profile across evaluations, Company does not warrant that any individual evaluation is of identical difficulty to any other individual evaluation. Difficulty consistency is a statistical property of the generation process, not a per-instance guarantee. Participant accepts that variance in scores across evaluations may reflect generation variance, system variance, or both, and that Company bears no obligation to investigate, explain, or remediate score differences between evaluations or between Participants.
Leaderboard placement is determined solely by scored output against the evaluation as generated. Company reserves the right to withhold, revoke, or modify any Leaderboard entry at its sole discretion and without obligation to provide justification. Participant waives any claim arising from perceived or actual differences in evaluation difficulty.
Company makes no representation regarding how third parties, including employers, investors, customers, partners, or any other entity, may interpret, weight, or act upon Leaderboard results or Benchmark scores. Participant assumes all risk associated with the publication of Benchmark scores, including reputational, commercial, and employment-related consequences. Participation in the Leaderboard is voluntary; Participant may decline Leaderboard publication at the time of submission.
10. Limitation of Liability.
TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, COMPANY, ITS OFFICERS, DIRECTORS, EMPLOYEES, AGENTS, AND AFFILIATES SHALL NOT BE LIABLE FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL, OR PUNITIVE DAMAGES, INCLUDING DAMAGES FOR LOSS OF PROFITS, GOODWILL, USE, DATA, BUSINESS OPPORTUNITY, OR REVENUE, ARISING OUT OF OR RELATED TO: (A) PARTICIPANT'S ACCESS TO OR USE OF (OR INABILITY TO USE) THE BENCHMARK MATERIALS; (B) ANY DECISIONS MADE OR ACTIONS TAKEN IN RELIANCE ON BENCHMARK RESULTS; (C) DATA WAREHOUSE UNAVAILABILITY, DATA LOSS, OR CORRUPTION; (D) ANY UNAUTHORIZED ACCESS TO OR ALTERATION OF PARTICIPANT'S SUBMISSIONS OR PERSONAL INFORMATION; (E) LEADERBOARD PLACEMENT OR NON-PLACEMENT; (F) ANY OTHER PARTICIPANT'S CONDUCT, INCLUDING MANIPULATION, INTERFERENCE, CHEATING, OR ANY ACTION THAT AFFECTS PARTICIPANT'S EVALUATION, SCORE, OR LEADERBOARD POSITION; (G) ANY THIRD PARTY'S INTERPRETATION OF, RELIANCE ON, OR ACTIONS TAKEN BASED ON BENCHMARK RESULTS, LEADERBOARD RANKINGS, OR ANY DATA DERIVED THEREFROM, INCLUDING WITHOUT LIMITATION HIRING DECISIONS, PROCUREMENT DECISIONS, INVESTMENT DECISIONS, OR REPUTATIONAL ASSESSMENTS; (H) ERRORS, INACCURACIES, HALLUCINATIONS, OR DEFECTS IN LLM-GENERATED CONTENT WITHIN BENCHMARK MATERIALS, INCLUDING GOLD SQL, PROMPTS, SCHEMAS, OR SCORING HARNESSES; (I) SERVICE UNAVAILABILITY, INTERRUPTIONS, OR DISCONTINUATION OF THE BENCHMARK, WEBSITE, OR ANY ASSOCIATED INFRASTRUCTURE; (J) ANY MARKETING OR PROMOTIONAL COMMUNICATIONS SENT TO PARTICIPANT, INCLUDING CONTENT, FREQUENCY, OR DELIVERY FAILURES; (K) ANY SECURITY INCIDENT AFFECTING PARTICIPANT'S PERSONAL INFORMATION; (L) ANY FAILURE, ERROR, OUTAGE, OR SECURITY INCIDENT ATTRIBUTABLE TO ANY THIRD-PARTY SERVICE, INCLUDING CLOUD HOSTING, DATA WAREHOUSE, OR LLM API PROVIDERS; OR (M) ANY OTHER MATTER RELATING TO THE BENCHMARK MATERIALS, REGARDLESS OF THE THEORY OF LIABILITY, EVEN IF COMPANY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
COMPANY'S TOTAL AGGREGATE LIABILITY FOR ALL CLAIMS ARISING FROM OR RELATED TO THESE TERMS OR THE BENCHMARK MATERIALS SHALL NOT EXCEED THE GREATER OF (I) THE TOTAL AMOUNTS, IF ANY, ACTUALLY PAID BY PARTICIPANT TO COMPANY UNDER THESE TERMS DURING THE TWELVE (12) MONTHS IMMEDIATELY PRECEDING THE EVENT GIVING RISE TO THE CLAIM, OR (II) ONE U.S. DOLLAR ($1.00). PARTICIPANT ACKNOWLEDGES THAT THE BENCHMARK IS PROVIDED AT NO COST AND THAT THIS LIMITATION REFLECTS THE ALLOCATION OF RISK BETWEEN THE PARTIES.
PARTICIPANT AGREES THAT PARTICIPANT'S SOLE AND EXCLUSIVE REMEDY FOR ANY DISPUTE WITH COMPANY ARISING OUT OF OR RELATING TO THE BENCHMARK MATERIALS IS TO CEASE USE OF THE BENCHMARK MATERIALS. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, PARTICIPANT WAIVES ANY RIGHT TO SEEK INJUNCTIVE OR OTHER EQUITABLE RELIEF AGAINST COMPANY IN CONNECTION WITH THESE TERMS OR THE BENCHMARK MATERIALS. THIS WAIVER DOES NOT LIMIT COMPANY'S RIGHT TO SEEK INJUNCTIVE OR OTHER EQUITABLE RELIEF AGAINST PARTICIPANT, INCLUDING WITHOUT LIMITATION EMERGENCY OR PRELIMINARY INJUNCTIVE RELIEF WITHOUT THE REQUIREMENT OF POSTING A BOND, TO ENFORCE SECTIONS 3 (RESTRICTIONS), 4 (PARTICIPANT REPRESENTATIONS), 5 (INTELLECTUAL PROPERTY), OR 12 (EXPORT CONTROLS).
In any proceeding arising from or relating to these Terms, the arbitrator or court shall consider the extent to which Participant's own actions, configuration choices, model selection, query strategy, or other conduct contributed to the damages claimed, and shall reduce any award accordingly. Participant bears the burden of proving that any alleged harm was caused solely by Company's conduct and not by Participant's own actions or the actions of third parties.
THE LIMITATIONS IN THIS SECTION APPLY ONLY TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW. Some jurisdictions do not allow the disclaimer of certain warranties or the limitation of certain damages, so some or all of the terms above may not apply to Participant, and Participant may have additional rights. In that case, these Terms limit Company's responsibilities only to the maximum extent permissible in Participant's jurisdiction.
11. Assumption of Risk.
PARTICIPANT EXPRESSLY ACKNOWLEDGES AND ASSUMES ALL RISKS ASSOCIATED WITH ACCESSING AND USING THE BENCHMARK MATERIALS, INCLUDING RISKS RELATED TO: (a) the accuracy, completeness, and reliability of Benchmark results and Leaderboard rankings; (b) the availability and continuity of Benchmark services; (c) the potential for LLM-generated errors in any Benchmark Material; (d) the publication of Benchmark scores and associated reputational, commercial, or employment-related consequences; (e) the actions of third parties, including other Participants, employers, investors, and customers, in response to Benchmark results; (f) the performance, availability, and security of Third-Party Services; and (g) any other risk arising from or relating to the Benchmark.
Participant's use of the Benchmark Materials is voluntary and at Participant's sole risk. Participant's access to the Benchmark at no cost constitutes sufficient consideration for this assumption of risk and the other obligations undertaken by Participant in these Terms.
12. Indemnification.
Participant agrees to indemnify, defend, and hold harmless Company, its affiliates, officers, directors, employees, and agents from and against any third-party claims, liabilities, damages, losses, and expenses (including reasonable attorneys' fees) arising out of or related to: (a) Participant's use of Benchmark Materials in violation of these Terms; (b) Participant's public statements, publications, or representations regarding Benchmark results; (c) any claim that Participant's systems, methods, or Submissions infringe a third party's intellectual property rights; (d) Participant's breach of Section 3 (Restrictions) or Section 4 (Participant Representations and Warranties); (e) Participant's violation of applicable export control laws, sanctions regulations, or trade compliance requirements; (f) any claim arising from Participant's provision of inaccurate Personal Information, including claims by third parties whose information Participant provided without authorization; or (g) any damage to Company's systems, infrastructure, or data caused by malicious code, SQL injection payloads, or other harmful content contained in Participant's Submissions. A party seeking indemnity will provide the indemnifying party with prompt written notice, reasonable cooperation, and sole control of defense and settlement.
13. Export Controls and Compliance.
Participant represents and warrants that Participant is not (a) located in, organized under the laws of, or ordinarily resident in any country or territory subject to comprehensive U.S. sanctions (currently Cuba, Iran, North Korea, Syria, and the Crimea, Donetsk, and Luhansk regions of Ukraine); (b) identified on the U.S. Department of the Treasury's Specially Designated Nationals and Blocked Persons List, the U.S. Department of Commerce's Entity List or Denied Persons List, or any other applicable restricted party list; or (c) owned or controlled by any person or entity described in (a) or (b). Participant shall comply with all applicable U.S. export control laws, sanctions regulations, and other trade compliance requirements in connection with access to and use of Benchmark Materials. Participant shall not re-export, transfer, or make available Benchmark Materials, directly or indirectly, to any destination, entity, or person prohibited by applicable law. Company may terminate access immediately and without notice if Company reasonably believes Participant is in violation of this Section.
14. Disqualification.
Company reserves the right, at its sole and absolute discretion, to disqualify any Participant, void any Submission, remove or modify any Leaderboard entry, and revoke access to Benchmark Materials, without notice, for any reason, including but not limited to: (a) suspected cheating, gaming, or manipulation of evaluation results; (b) circumvention or attempted circumvention of evaluation limits or scoring methodology; (c) interference with another Participant's evaluation or Submission; (d) misrepresentation of identity, affiliation, or evaluation conditions; or (e) any conduct that Company determines, in its sole judgment, undermines the integrity of the Benchmark. Disqualification decisions are final and not subject to appeal. Company is under no obligation to disclose the basis for any disqualification. Participant waives any claim arising from disqualification, including claims for damages, reputational harm, or lost opportunity.
15. Suspension and Discontinuation.
Company may, at any time and in its sole discretion, suspend, modify, or permanently discontinue the Benchmark, the Leaderboard, the scoring infrastructure, the data warehouse, or any associated service, in whole or in part, with or without notice and for any reason or no reason. No suspension or discontinuation shall create any liability to Participant or any third party, entitle Participant to any refund, credit, or compensation, or obligate Company to maintain, archive, or return any Submission or Benchmark data. Participant's sole remedy upon suspension or discontinuation is to cease use of Benchmark Materials.
16. Force Majeure.
Company shall not be liable for any failure or delay in performing any obligation under these Terms caused by events beyond Company's reasonable control, including acts of God, natural disasters, pandemics, epidemics, war, terrorism, riots, embargoes, acts of governmental authorities, fire, flood, power outage, internet or telecommunications failure, third-party service provider outages (including cloud infrastructure, data warehouse, and LLM API providers), cyberattacks, or labor disputes. During any force majeure event, Company's obligations are suspended without liability for the duration of the event.
17. Dispute Resolution.
Any dispute arising from or relating to these Terms shall be governed by the laws of the State of Washington, without regard to conflict-of-law principles. The parties agree to attempt informal resolution for 60 days before initiating formal proceedings. If informal resolution fails, disputes shall be resolved through binding arbitration administered by the American Arbitration Association under its Commercial Arbitration Rules, conducted in King County, Washington, or such other location as Company may designate at its sole discretion. PARTICIPANT WAIVES ANY RIGHT TO A JURY TRIAL AND TO PARTICIPATE IN ANY CLASS, CONSOLIDATED, OR REPRESENTATIVE ACTION. The prevailing party in any arbitration or legal proceeding arising from or relating to these Terms shall be entitled to recover its reasonable attorneys' fees, costs, and expenses from the non-prevailing party.
18. Limitations Period.
ANY CLAIM OR CAUSE OF ACTION ARISING OUT OF OR RELATING TO THESE TERMS OR THE BENCHMARK MATERIALS MUST BE COMMENCED WITHIN ONE (1) YEAR AFTER THE EVENT GIVING RISE TO THE CLAIM. ANY CLAIM NOT BROUGHT WITHIN THIS PERIOD IS PERMANENTLY BARRED. This limitations period applies regardless of the form of action, whether in contract, tort, strict liability, or otherwise, and regardless of when Participant discovered or should have discovered the facts giving rise to the claim.
19. Assignment.
Participant may not assign, transfer, or delegate any of Participant's rights or obligations under these Terms, in whole or in part, without Company's prior written consent, and any attempted assignment without such consent is void. Company may assign, transfer, or delegate its rights and obligations under these Terms, in whole or in part, without Participant's consent, including in connection with a merger, acquisition, corporate reorganization, or sale of all or substantially all of Company's assets. These Terms shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns.
20. Modification.
Company may modify these Terms at any time by posting updated Terms on this page with a revised effective date. Continued access to Benchmark Materials after the effective date constitutes acceptance of the modified Terms. If Participant does not agree to modified Terms, Participant's sole remedy is to discontinue use.
21. Severability.
If any provision of these Terms is held invalid or unenforceable by a court of competent jurisdiction, that provision shall be enforced to the maximum extent permissible, and the remaining provisions shall continue in full force and effect.
22. No Third-Party Beneficiaries.
These Terms are for the sole benefit of Company and Participant and do not confer any rights, remedies, or benefits on any third party. No third party may enforce or rely upon any provision of these Terms.
23. No Waiver.
Company's failure to enforce any right or provision of these Terms shall not constitute a waiver of such right or provision. Any waiver of any provision of these Terms shall be effective only if in writing and signed by Company.
24. Governing Language.
These Terms are drafted in English. If these Terms are translated into any other language, the English-language version shall control in the event of any conflict or inconsistency.
25. Survival.
Sections 3 (Restrictions), 4 (Participant Representations and Warranties), 5 (Intellectual Property), 6 (Evaluation and Scoring), 7 (Leaderboard Governance), 8 (Data Collection and Communications), 9 (Disclaimer), 10 (Limitation of Liability), 11 (Assumption of Risk), 12 (Indemnification), 13 (Export Controls), 14 (Disqualification), 15 (Suspension and Discontinuation), 16 (Force Majeure), 17 (Dispute Resolution), 18 (Limitations Period), 19 (Assignment), 21 (Severability), 22 (No Third-Party Beneficiaries), 23 (No Waiver), 24 (Governing Language), 26 (Acceptance Records), and 27 (Relationship to General Terms) survive termination of this agreement.
26. Acceptance Records.
By clicking "I Agree," Participant acknowledges that Company will record the following information at the time of acceptance: Participant's full legal name, email address, organizational affiliation (if any), IP address, browser user agent string, timestamp of acceptance, agreement version identifier, and account identifier. This record constitutes Company's evidence of Participant's informed consent to these Terms. Company may retain acceptance records for such period as Company deems appropriate for purposes of legal compliance, dispute resolution, and audit. Participant agrees that clicking "I Agree" constitutes an electronic signature within the meaning of the Electronic Signatures in Global and National Commerce Act (15 U.S.C. § 7001 et seq.) and the Washington Uniform Electronic Transactions Act (RCW 19.360), and that this electronic acceptance has the same legal force and effect as a handwritten signature.
27. Relationship to General Terms.
These Terms supplement and are in addition to the Schemantic Terms of Service (schemantic.io/terms) and Privacy Policy (schemantic.io/privacy), which govern general use of the Site. In the event of a conflict between these Terms and the general Terms of Service with respect to the Benchmark Materials, these Terms control. Together with the Terms of Service and Privacy Policy, these Terms constitute the entire agreement between Participant and Company with respect to the Benchmark Materials and supersede all prior agreements and understandings.