Run private ROM and reference tests on the server runner #66

Closed
opened 2026-10-03 19:30:24 +02:00 by kawa · 4 comments
Owner

Use the configured CRYSTAL_ROM_URL secret on Docker-Builder1 for trusted main-branch code. Download privately, verify the Crystal 1.1 hash, build the pinned PRET reference with RGBDS 1.0.3, and execute unit and Chromium/WebKit reference/gameplay tests. Keep synthetic pull-request CI, clean up private files, and do not upload game data or gameplay artifacts. Record the server result and remaining validation boundaries.

Commits: 03aa913, 0d8ff44, 60fb65b, 15a31f9

Status (2026-10-06)

Commits: 03aa913 (private job), 0d8ff44 (synthetic suites without a ROM), 60fb65b (one browser worker), 15a31f9 (90 s per test, 60 min job).

  • HTTPS secret, download, hash check, PRET build and unit tests with the ROM pass on Docker-Builder1.
  • With two browser workers the runner dropped off mid-run; with one it stays online.
  • Latest private run 138: 157 passed, 21 skipped, 12 failed: all are the WebKit start-menu regression #70. Synthetic run 137 is green.

Blocked by #70. Close once a private run on main is green.

Use the configured CRYSTAL_ROM_URL secret on Docker-Builder1 for trusted main-branch code. Download privately, verify the Crystal 1.1 hash, build the pinned PRET reference with RGBDS 1.0.3, and execute unit and Chromium/WebKit reference/gameplay tests. Keep synthetic pull-request CI, clean up private files, and do not upload game data or gameplay artifacts. Record the server result and remaining validation boundaries. Commits: 03aa913, 0d8ff44, 60fb65b, 15a31f9 ## Status (2026-10-06) Commits: 03aa913 (private job), 0d8ff44 (synthetic suites without a ROM), 60fb65b (one browser worker), 15a31f9 (90 s per test, 60 min job). - HTTPS secret, download, hash check, PRET build and unit tests with the ROM pass on Docker-Builder1. - With two browser workers the runner dropped off mid-run; with one it stays online. - Latest private run [138](https://git.azuze.fr/kawa/Crystal/actions/runs/138): 157 passed, 21 skipped, 12 failed: all are the WebKit start-menu regression #70. Synthetic run [137](https://git.azuze.fr/kawa/Crystal/actions/runs/137) is green. Blocked by #70. Close once a private run on main is green.
kawa added the P0 - now label 2026-10-03 19:30:24 +02:00
kawa added this to the Pokemon Crystal - Web project 2026-10-03 19:30:24 +02:00
kawa moved this to En cours in Pokemon Crystal - Web on 2026-10-03 19:30:24 +02:00
Author
Owner

Status after the Codex session stopped:

  • Private run 122 failed at the download step: CRYSTAL_ROM_URL must use HTTPS. The secret needs a direct HTTPS URL.
  • Runs 123–130 (commits cdee576…ce81297) are queued: runner Docker-Builder1 is offline.
  • Locally at ce81297 with the ROM: 643 unit tests passed (3 skipped), 169 Chromium/WebKit tests passed (21 skipped), builds and reference/vendor/release checks passed.

Remaining: runner online + HTTPS secret, then a green private-checks run on main.

Status after the Codex session stopped: - Private run [122](https://git.azuze.fr/kawa/Crystal/actions/runs/122) failed at the download step: `CRYSTAL_ROM_URL must use HTTPS.` The secret needs a direct HTTPS URL. - Runs 123–130 (commits cdee576…ce81297) are queued: runner `Docker-Builder1` is offline. - Locally at ce81297 with the ROM: 643 unit tests passed (3 skipped), 169 Chromium/WebKit tests passed (21 skipped), builds and reference/vendor/release checks passed. Remaining: runner online + HTTPS secret, then a green private-checks run on main.
kawa was assigned by PestoRosso 2026-10-04 01:09:16 +02:00
Author
Owner

Update:

  • HTTPS secret works: private run 128 passed end to end.
  • 0d8ff44 fixes the synthetic job (atlas/Pokédex suites read the ROM at collection time); run 135 is green.
  • 60fb65b runs the private browser step with one worker: the runner no longer drops off, but run 136 has 25 failures:
    • 13 Chromium parity/palette tests hit the 30 s timeout on the slow host; they pass locally.
    • 12 WebKit tests fail because of a regression from 525ff09 (#67): on a phone viewport the start menu overflows the top of the game screen, so POKéDEX/POKéMON/PACK are clipped and can't be tapped. Reproduced locally and bisected (5546e57 passes, 525ff09 fails).
Update: - HTTPS secret works: private run [128](https://git.azuze.fr/kawa/Crystal/actions/runs/128) passed end to end. - 0d8ff44 fixes the synthetic job (atlas/Pokédex suites read the ROM at collection time); run [135](https://git.azuze.fr/kawa/Crystal/actions/runs/135) is green. - 60fb65b runs the private browser step with one worker: the runner no longer drops off, but run [136](https://git.azuze.fr/kawa/Crystal/actions/runs/136) has 25 failures: - 13 Chromium parity/palette tests hit the 30 s timeout on the slow host; they pass locally. - 12 WebKit tests fail because of a regression from 525ff09 (#67): on a phone viewport the start menu overflows the top of the game screen, so POKéDEX/POKéMON/PACK are clipped and can't be tapped. Reproduced locally and bisected (5546e57 passes, 525ff09 fails).
Author
Owner

Run 138 at 15a31f9 (one worker, 90 s per test): the runner stayed online, and the 13 Chromium parity tests now pass. Unit tests with the ROM pass; browser step: 157 passed, 21 skipped, 12 failed in 33.1 min. The 12 failures are exactly the WebKit start-menu regression tracked in #70. The synthetic run 137 is green.

Remaining for this card: a green private run once #70 is fixed.

Run [138](https://git.azuze.fr/kawa/Crystal/actions/runs/138) at 15a31f9 (one worker, 90 s per test): the runner stayed online, and the 13 Chromium parity tests now pass. Unit tests with the ROM pass; browser step: 157 passed, 21 skipped, 12 failed in 33.1 min. The 12 failures are exactly the WebKit start-menu regression tracked in #70. The synthetic run [137](https://git.azuze.fr/kawa/Crystal/actions/runs/137) is green. Remaining for this card: a green private run once #70 is fixed.
Author
Owner

Private ROM checks are green on main at 62fe4bf: run 142 (unit and playthrough 790 passed, 3 skipped; Chromium and WebKit 193 passed, 21 skipped, 0 failed in 15.2 min). The 12 WebKit failures were #70, fixed in 62fe4bf. The public checks (run 141) also passed. Closing.

Private ROM checks are green on main at 62fe4bf: run [142](https://git.azuze.fr/kawa/Crystal/actions/runs/142) (unit and playthrough 790 passed, 3 skipped; Chromium and WebKit 193 passed, 21 skipped, 0 failed in 15.2 min). The 12 WebKit failures were #70, fixed in 62fe4bf. The public checks (run [141](https://git.azuze.fr/kawa/Crystal/actions/runs/141)) also passed. Closing.
kawa closed this issue 2026-10-13 12:00:08 +02:00
kawa moved this to Fait in Pokemon Crystal - Web on 2026-10-13 12:00:09 +02:00
Sign in to join this conversation.