Research contract v1

Expert AI benchmark study protocol

This protocol packages a blinded reviewer study without treating automation as expert evidence. The public research submission contains only aggregated pseudonymous reviews; raw responses, identity keys, consent records and transport metadata stay private.

Preregister and freeze

  1. Create a draft submission with stable AI- position IDs, decided pilot/tuning/holdout splits and empty expertReviews arrays.
  2. Store the dated protocol and corpusHash in a third-party timestamped system or signed study record before review. Repository tooling cannot prove when a hash was created.
  3. Prepare and verify the private package:
node scripts/prepare-mahjong-ai-review.mjs prepare DRAFT.json PRIVATE_PACKAGE_DIRECTORY
node scripts/prepare-mahjong-ai-review.mjs verify DRAFT.json PRIVATE_PACKAGE_DIRECTORY/corpus.manifest.json PRIVATE_PACKAGE_DIRECTORY/reviewer.packet.json

The reviewer packet uses opaque ITEM- IDs and omits source IDs, split assignments, evidence IDs, existing reviews, measured latency and model predictions. Do not send the manifest or prediction file to reviewers.

Collect privately

Give each reviewer separate packet and response-template copies. Keep the real-person-to-REV- key, consent, recruitment record, messages and raw returns outside the public repository. Reviewers should work independently from only the acting hand, public state and legal actions.

node scripts/prepare-mahjong-ai-review.mjs check-response PRIVATE_MANIFEST.json REVIEWER_PACKET.json RESPONSE.json

This rejects missing or duplicate items, illegal actions, contradictory labels and unsupported fields, but cannot prove who completed a response or what they saw.

Assemble the public aggregate

Merge only pseudonymous reviews and explicitly choose draft, pilot or final:

node scripts/prepare-mahjong-ai-review.mjs assemble DRAFT.json PRIVATE_MANIFEST.json REVIEWER_PACKET.json PUBLIC_SUBMISSION.json pilot RESPONSE-1.json RESPONSE-2.json RESPONSE-3.json
node scripts/validate-mahjong-research.mjs PUBLIC_SUBMISSION.json

Score predictions only after frozen labels are assembled:

node scripts/score-mahjong-ai-benchmark.mjs PUBLIC_SUBMISSION.json PREDICTIONS.json

What automation cannot prove

Hashes do not prove preregistration timing or the absence of another corpus. Opaque IDs do not prove blinding. Anonymous IDs do not prove reviewers are distinct, independent, human, qualified or consenting. File checks do not prove the model avoided holdout labels, reviewers used only visible information, rationales are truthful or recruitment was unbiased. These claims require a documented human process and retained private evidence.

Back to Singapore Mahjong methodology