diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 76a7e7c..54eab3e 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -10,10 +10,10 @@ ### Real-World Runtime Validation (RTVAL) -- [ ] **RTVAL-01**: A generated `.intunewin` package is uploaded to a real Microsoft Intune tenant and accepted (no format errors), with byte-level conformance confirmed against tenant ingestion -- [ ] **RTVAL-02**: A generated install script runs successfully under SYSTEM context on a real Intune-managed Windows endpoint, with `pnputil` driver staging and `$PSScriptRoot` path resolution verified -- [ ] **RTVAL-03**: Generated detect script returns the expected exit code on a real endpoint after install (Intune detection rule succeeds) -- [ ] **RTVAL-04**: Uninstall script removes the printer cleanly under SYSTEM context on a real endpoint +- [x] **RTVAL-01**: A generated `.intunewin` package is uploaded to a real Microsoft Intune tenant and accepted (no format errors), with byte-level conformance confirmed against tenant ingestion +- [x] **RTVAL-02**: A generated install script runs successfully under SYSTEM context on a real Intune-managed Windows endpoint, with `pnputil` driver staging and `$PSScriptRoot` path resolution verified +- [x] **RTVAL-03**: Generated detect script returns the expected exit code on a real endpoint after install (Intune detection rule succeeds) +- [x] **RTVAL-04**: Uninstall script removes the printer cleanly under SYSTEM context on a real endpoint - [x] **RTVAL-05**: A signed-off RUNTIME-VALIDATION.md report records tenant, device, OS build, driver vendor(s) tested, screenshots/logs, and any issues found ### UX Tech Debt (UX) @@ -52,10 +52,10 @@ Carried forward from v1.0 Out of Scope — no change. | Requirement | Phase | Status | |-------------|-------|--------| -| RTVAL-01 | Phase 10 | Pending | -| RTVAL-02 | Phase 10 | Pending | -| RTVAL-03 | Phase 10 | Pending | -| RTVAL-04 | Phase 10 | Pending | +| RTVAL-01 | Phase 10 | Complete | +| RTVAL-02 | Phase 10 | Complete | +| RTVAL-03 | Phase 10 | Complete | +| RTVAL-04 | Phase 10 | Complete | | RTVAL-05 | Phase 10 | Complete | | UX-01 | Phase 9 | Complete | | UX-02 | Phase 9 | Complete | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 7fe0bd9..94ed114 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -93,5 +93,5 @@ Full details: [`milestones/v1.0-ROADMAP.md`](milestones/v1.0-ROADMAP.md) | 7. Dashboard & Nav Polish | v1.0 | 1/1 | Complete | 2026-04-13 | | 8. Nyquist Validation Track | v1.1 | 0/? | Not started | — | | 9. UX Tech Debt Closure | 3/3 | Complete | 2026-04-13 | — | -| 10. Real-World Runtime Validation | 1/3 | In Progress| | — | +| 10. Real-World Runtime Validation | 2/3 | In Progress| | — | | 11. Real-World Rollout & Feedback | v1.1 | 0/? | Not started | — | diff --git a/.planning/STATE.md b/.planning/STATE.md index e2b293e..a11714d 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,10 +2,10 @@ gsd_state_version: 1.0 milestone: v1.1 milestone_name: Hardening & Validation -status: in-progress — plan 10-02 advanced; RTVAL-01 PASS, RTVAL-02 PASS (attestation-only), RTVAL-03 PASS (attestation-only — 2nd consecutive); paused at Task 4 (RTVAL-04 uninstall under SYSTEM) -stopped_at: 10-02-live-intune-runtime-validation-PLAN.md — Task 4 (RTVAL-04) human-action checkpoint awaiting technician uninstall evidence +status: in-progress — plan 10-02 COMPLETE; all four RTVAL runtime checks recorded (RTVAL-01 artifact-backed PASS, RTVAL-02/03/04 attestation-only PASSes — 3 consecutive, audit trail weakened); ready for plan 10-03 (sign-off) +stopped_at: 10-02 complete — next is 10-03 sign-off and phase closure last_updated: "2026-04-13T00:00:00.000Z" -last_activity: 2026-04-13 — RTVAL-03 recorded PASS (attestation-only, 2nd consecutive; user warned about weakened audit trail and approved); plan 10-02 paused at Task 4 (RTVAL-04) +last_activity: 2026-04-13 — Plan 10-02 completed. RTVAL-04 recorded PASS (attestation-only, 3rd consecutive; user warned twice about weakened audit trail and approved). SUMMARY.md created with prominent attestation-only debt section for wave-3 verifier and phase verifier. progress: total_phases: 4 completed_phases: 1 @@ -26,10 +26,10 @@ See: .planning/PROJECT.md (updated 2026-04-13 after v1.0 milestone) Milestone: v1.1 Hardening & Validation Phase: 10 — Real-World Runtime Validation -Plan: 02 in progress — Task 1 PASS (RTVAL-01), Task 2 PASS (RTVAL-02, attestation-only), Task 3 PASS (RTVAL-03, attestation-only — 2nd consecutive); paused at Task 4 (RTVAL-04) human-action checkpoint -Status: in-progress — awaiting technician evidence for RTVAL-04 (Intune uninstall removes printer cleanly under SYSTEM) -Decision: Resume plan 10-02 after HMAC scope + Detection.xml generator fixes landed (commits 74535ea, 7716246); ISSUE-01 resolved. RTVAL-02 and RTVAL-03 both accepted as attestation-only PASSes per explicit user approval 2026-04-13; user was warned on RTVAL-03 that a second consecutive attestation-only check weakens the audit trail and still chose to proceed. -Last activity: 2026-04-13 — RTVAL-03 PASS (attestation-only, commit 475d593); plan 10-02 paused at Task 4 checkpoint +Plan: 02 COMPLETE — all 4 tasks recorded (RTVAL-01 artifact-backed PASS; RTVAL-02/03/04 attestation-only PASSes, 3 consecutive). Next plan: 10-03 (sign-off and phase closure). +Status: in-progress — plan 10-02 done, plan 10-03 ready to start +Decision: Plan 10-02 complete with a structurally weakened runtime audit trail. RTVAL-02/03/04 all accepted as attestation-only PASSes per explicit, repeated user approval. User was warned twice (on RTVAL-03 and again on RTVAL-04) that consecutive attestation-only checks damage the audit trail and chose to proceed both times. Plan 10-03 sign-off must explicitly address whether to re-run RTVAL-02/03/04 with full artifact capture before closing the phase. +Last activity: 2026-04-13 — RTVAL-04 PASS (attestation-only, commit 2c912ca); plan 10-02 SUMMARY.md created with prominent attestation-only debt section; plan 10-02 closed ## Milestone History @@ -72,6 +72,8 @@ Full decision log in PROJECT.md Key Decisions table. Milestone v1.0 decisions ar - [Phase 10-real-world-runtime-validation]: 10-02: RTVAL-01 PASS on re-test (2026-04-13) — ISSUE-01 resolved by commits 74535ea (HMAC over IV+ciphertext) and 7716246 (Detection.xml alignment with IntuneWinAppUtil.exe reference format); plan resumed at Task 2 - [Phase 10-real-world-runtime-validation]: 10-02: RTVAL-02 accepted as attestation-only PASS (2026-04-13) — technician verbally confirmed install succeeded on ARES-5CG5220YTM but did NOT provide IntuneManagementExtension.log excerpt or portal screenshot; user explicitly approved "Pass without evidence"; audit trail weakened for this check and flagged in RUNTIME-VALIDATION.md Notes - [Phase 10-real-world-runtime-validation]: 10-02: RTVAL-03 accepted as attestation-only PASS (2026-04-13) — second consecutive attestation-only check; no rtval-03-detection.png and no rtval-03-detect-manual.txt captured; user was explicitly warned that a second consecutive attestation-only check further weakens the audit trail and still chose to proceed; flagged in RUNTIME-VALIDATION.md Notes as soft PASS requiring re-run with full artifact capture before phase sign-off +- [Phase 10-real-world-runtime-validation]: 10-02: RTVAL-04 accepted as attestation-only PASS (2026-04-13) — **third consecutive attestation-only check**; no rtval-04-uninstall-log.txt and no rtval-04-uninstall-status.png captured; user was warned a SECOND time about cumulative audit trail damage and still chose to proceed. Together, RTVAL-02/03/04 constitute an attestation-only runtime half for Phase 10: only RTVAL-01 (tenant ingestion) is artifact-backed. Plan 10-03 sign-off must explicitly address whether to re-run RTVAL-02/03/04 with full evidence before closing the phase. +- [Phase 10-real-world-runtime-validation]: 10-02: Plan 10-02 COMPLETE (2026-04-13) — SUMMARY.md created with prominent "Attestation-Only Audit Trail Damage" section for the wave-3 verifier and phase verifier ### Active Blockers @@ -86,5 +88,5 @@ None. BLOCKER-01 resolved 2026-04-13 via commits 74535ea (HMAC over IV+ciphertex ## Session Continuity Last session: 2026-04-13T00:00:00.000Z -Stopped at: 10-02-live-intune-runtime-validation — Task 4 (RTVAL-04) human-action checkpoint; awaiting technician evidence for uninstall under SYSTEM (uninstall log SYSTEM context + exit 0, printer removed via Get-Printer, Intune portal "Not installed", uninstall status screenshot) -Resume file: .planning/phases/10-real-world-runtime-validation/10-02-live-intune-runtime-validation-PLAN.md +Stopped at: Completed 10-02-live-intune-runtime-validation-PLAN.md — next is plan 10-03 (sign-off and phase closure) +Resume file: .planning/phases/10-real-world-runtime-validation/10-03-*-PLAN.md (to be drafted or existing) diff --git a/.planning/phases/10-real-world-runtime-validation/10-02-live-intune-runtime-validation-SUMMARY.md b/.planning/phases/10-real-world-runtime-validation/10-02-live-intune-runtime-validation-SUMMARY.md new file mode 100644 index 0000000..b78cb2f --- /dev/null +++ b/.planning/phases/10-real-world-runtime-validation/10-02-live-intune-runtime-validation-SUMMARY.md @@ -0,0 +1,182 @@ +--- +phase: 10-real-world-runtime-validation +plan: 02 +subsystem: testing +tags: [intune, intunewin, runtime-validation, rtval, pnputil, system-context, attestation-only] + +# Dependency graph +requires: + - phase: 10-real-world-runtime-validation + provides: "Plan 10-01 — locked .intunewin package (Copieur_2eme), tenant + device metadata, RUNTIME-VALIDATION.md scaffold" +provides: + - "RTVAL-01 PASS (artifact-backed) — real Intune tenant accepts generated .intunewin without format errors" + - "RTVAL-02 PASS (attestation-only) — install under SYSTEM context confirmed verbally by technician" + - "RTVAL-03 PASS (attestation-only) — Intune detection rule reports Installed confirmed verbally" + - "RTVAL-04 PASS (attestation-only) — uninstall under SYSTEM confirmed verbally; printer removed cleanly" + - "Two structural .intunewin generator defects discovered and fixed during RTVAL-01 (ISSUE-01): HMAC scope + Detection.xml format" +affects: + - 10-03-sign-off-and-phase-closure + - 11-real-world-rollout-and-feedback + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Attestation-only PASS is permitted but MUST be flagged in RUNTIME-VALIDATION.md Notes and called out in plan SUMMARY for downstream verifiers" + - "Debug-then-resume flow: RTVAL failure → /gsd:debug on generator → commit fixes → re-test same check against fresh build" + +key-files: + created: [] + modified: + - .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md + +key-decisions: + - "RTVAL-01 initial FAIL (blocker) debugged to root cause: two .intunewin generator defects (HMAC over ciphertext-only instead of IV+ciphertext; Detection.xml not matching IntuneWinAppUtil.exe reference format). Fixed in commits 74535ea and 7716246; re-test PASS." + - "RTVAL-02, RTVAL-03, and RTVAL-04 accepted as attestation-only PASSes per explicit, repeated user approval. User was warned on RTVAL-03 about consecutive attestation-only checks weakening audit trail, and warned AGAIN on RTVAL-04, and chose to proceed both times." + - "Phase 10 runtime audit trail is now structurally weakened: 3 of 4 RTVAL runtime checks are verbal-only. Plan 10-03 sign-off must either accept this or force re-run of RTVAL-02/03/04 with full artifact capture." + +patterns-established: + - "SUMMARY.md must surface attestation-only debt prominently so the wave-3 verifier and phase verifier cannot miss it" + - ".intunewin generator defects discovered during real-tenant testing are captured as ISSUE-## entries in RUNTIME-VALIDATION.md with resolution commit refs" + +requirements-completed: + - RTVAL-01 + - RTVAL-02 + - RTVAL-03 + - RTVAL-04 + +# Metrics +duration: "~1 day (wall-clock, spanning debug + re-test + three attestation checkpoints)" +completed: 2026-04-13 +--- + +# Phase 10 Plan 02: Live Intune Runtime Validation Summary + +**All four RTVAL runtime checks recorded against live tenant rubis.fr on ARES-5CG5220YTM — RTVAL-01 artifact-backed PASS after fixing two .intunewin generator defects; RTVAL-02/03/04 attestation-only PASSes (user approved despite repeated warnings about weakened audit trail).** + +## Performance + +- **Duration:** ~1 day wall-clock (includes debug cycle for ISSUE-01 + three human-action checkpoints) +- **Started:** 2026-04-13 +- **Completed:** 2026-04-13 +- **Tasks:** 4 (all `checkpoint:human-action`) +- **Files modified:** 1 (RUNTIME-VALIDATION.md — incrementally across 4 task commits) + +## Accomplishments + +- **RTVAL-01 (artifact-backed):** Real Intune tenant (rubis.fr) accepted the generated `Copieur_2eme.intunewin` package without format errors on re-test against the fixed build. Two screenshots committed (`rtval-01-tenant-upload.png`, `rtval-01-app-assigned.png`). +- **RTVAL-01 debug cycle:** Initial run FAILED with empty wizard fields and greyed-out OK button. Root cause debugged to two structural defects in the ImpTune .intunewin generator: + 1. HMAC was computed over ciphertext only instead of IV+ciphertext (commit `74535ea`) + 2. Detection.xml did not match the IntuneWinAppUtil.exe reference format (commit `7716246`) + Both fixes landed; RTVAL-01 re-tested PASS. +- **RTVAL-02 (attestation-only):** Technician verbally confirmed install script ran under SYSTEM on ARES-5CG5220YTM, driver deployed, Intune portal showed Installed. No log excerpt or screenshot captured. +- **RTVAL-03 (attestation-only, 2nd consecutive):** Technician verbally confirmed Intune reports Installed and detection rule passes. No screenshot or detect transcript captured. User warned about weakened audit trail and approved. +- **RTVAL-04 (attestation-only, 3rd consecutive):** Technician verbally confirmed Intune uninstall ran under SYSTEM, exited cleanly, printer removed from ARES-5CG5220YTM. No uninstall log or portal screenshot captured. User warned a second time about the cumulative audit-trail damage and approved. + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: RTVAL-01 — Tenant ingestion (FAIL → debug → PASS)** + - `403a0a5` (fix): initial FAIL recorded — Intune cannot parse generated .intunewin + - `46cfde0` (chore): STATE.md updated — plan blocked at RTVAL-01 FAIL + - `44a4f2c` (docs): debug session recorded + - `7716246` (fix): Detection.xml aligned with IntuneWinAppUtil.exe reference format + - `74535ea` (fix): HMAC computed over IV+ciphertext + - `35d4edc` (docs): debug session updated — HMAC scope bug identified + - `00b709d` (docs): debug session resolved + - `cf3b86a` (docs): RTVAL-01 PASS recorded on re-test against fixed build + - `bcd12a3` (docs): BLOCKER-01 cleared, plan resumed at Task 2 +2. **Task 2: RTVAL-02 — Install under SYSTEM** — `870158b` (docs) — attestation-only PASS + - `d86199f` (docs): STATE.md advanced to Task 3 +3. **Task 3: RTVAL-03 — Detection rule reports Installed** — `475d593` (docs) — attestation-only PASS (2nd consecutive) + - `b0be3d3` (docs): STATE.md advanced to Task 4 +4. **Task 4: RTVAL-04 — Uninstall under SYSTEM** — `2c912ca` (docs) — attestation-only PASS (3rd consecutive) + +**Plan metadata:** (this commit) `docs(10-02): complete live-intune-runtime-validation plan` + +## Files Created/Modified + +- `.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md` — Status line updated to reflect all four RTVAL checks recorded; RTVAL-01 through RTVAL-04 sections populated with results, evidence (or attestation disclaimer), and Notes. +- `imptune/generators/intunewin_builder.py` (via debug cycle, commits `74535ea`, `7716246`) — HMAC scope fix + Detection.xml format alignment; unblocked RTVAL-01. + +## Decisions Made + +- **Debug-before-abandon:** RTVAL-01 initial FAIL was not deferred to a gap-closure plan — instead, the .intunewin generator was debugged inline (ISSUE-01) because the root cause was clearly in-scope for the package under test. Two commits fixed it and the same check was re-run on the fresh build. +- **Attestation-only accepted for RTVAL-02/03/04:** User explicitly, repeatedly chose verbal attestation over artifact capture. The executor flagged each successive choice (warning on RTVAL-03, second warning on RTVAL-04) and proceeded per user instruction. This is documented here, in RUNTIME-VALIDATION.md Notes for each check, and in STATE.md decisions log — it MUST be visible to plan 10-03 (sign-off) and the phase verifier. + +## Deviations from Plan + +### Rule 1 (Bug) — `.intunewin` generator defects discovered during RTVAL-01 + +**1. [Rule 1 - Bug] HMAC computed over ciphertext only instead of IV+ciphertext** +- **Found during:** Task 1 (RTVAL-01) — Intune wizard rejected the package, debugged to the root cause +- **Issue:** `intunewin_builder.py` computed HMAC over the ciphertext alone, but the Intune Win32 package format requires HMAC over `IV || ciphertext`. Intune's package parser rejected the file silently (empty wizard fields). +- **Fix:** Updated HMAC computation to include the IV prefix. +- **Files modified:** `imptune/generators/intunewin_builder.py` +- **Committed in:** `74535ea` (standalone fix commit, not inside a task commit) + +**2. [Rule 1 - Bug] Detection.xml did not match IntuneWinAppUtil.exe reference format** +- **Found during:** Task 1 (RTVAL-01) — same debug cycle +- **Issue:** Embedded Detection.xml used a slightly different schema/element ordering than IntuneWinAppUtil.exe emits. Intune parses the reference format strictly, so the wizard could not populate metadata fields. +- **Fix:** Aligned Detection.xml generation with the IntuneWinAppUtil.exe reference output byte-for-byte. +- **Files modified:** `imptune/generators/intunewin_builder.py` +- **Committed in:** `7716246` (standalone fix commit, not inside a task commit) + +--- + +**Total deviations:** 2 auto-fixed (both Rule 1 bugs in the generator under test) +**Impact on plan:** Both fixes were essential and in-scope — the whole point of RTVAL-01 is to discover exactly this class of defect. Plan was paused via debug session and resumed without re-planning. + +## ⚠️ CRITICAL: Attestation-Only Audit Trail Damage + +**Read this before signing off Phase 10 in plan 10-03.** + +Three of the four RTVAL runtime checks in this plan are **attestation-only** (verbal confirmation from the technician, no log excerpt, no screenshot, no transcript): + +| Check | Status | Artifact-backed? | Warning at time of acceptance | +| -------- | ----------------------------- | ---------------- | -------------------------------------------------------------------------- | +| RTVAL-01 | PASS | **Yes** | — | +| RTVAL-02 | PASS (attestation-only) | No | User approved "Pass without evidence" | +| RTVAL-03 | PASS (attestation-only, 2nd) | No | User warned about weakened audit trail, approved anyway | +| RTVAL-04 | PASS (attestation-only, 3rd) | No | User warned a **second time** about weakened audit trail, approved anyway | + +**What this means:** + +- Only tenant ingestion (RTVAL-01) is independently verifiable from committed evidence. +- The entire **runtime half** of Phase 10 (install under SYSTEM, detection, uninstall under SYSTEM) rests on the technician's verbal report. None of the required observations (`$PSScriptRoot` resolved, `pnputil /add-driver` success line, `NT AUTHORITY\SYSTEM` banner, exit code 0, `Get-Printer` shows/hides the printer, Intune portal Installed/Not installed) were captured to disk. +- For any future regression, incident investigation, or compliance audit, RTVAL-02/03/04 should be treated as **soft PASSes** and re-run with full artifact capture before the phase is considered closed. + +**Recommended action for plan 10-03 (sign-off):** + +1. Either: re-run RTVAL-02/03/04 with full artifact capture before signing off, OR +2. Explicitly accept the attestation-only audit trail in the sign-off block and record the technician's name + the compensating control (e.g., known test device, known tenant, same session). + +The sign-off checklist in RUNTIME-VALIDATION.md should **not** be ticked silently — the reviewer needs to actively acknowledge the attestation-only nature of 3 of 4 checks. + +## Issues Encountered + +- **ISSUE-01 (resolved):** Initial RTVAL-01 FAIL — Intune wizard could not parse `Copieur_2eme.intunewin`. Root-caused during a /gsd:debug session to two `.intunewin` generator defects (HMAC scope + Detection.xml format). Both fixed in `74535ea` and `7716246`. RTVAL-01 re-tested PASS against the fixed build. See RUNTIME-VALIDATION.md "Issues Found → ISSUE-01" for the full trail. +- **Attestation debt:** Three consecutive runtime checks went attestation-only. Not an "issue" in the traditional sense (user chose it), but documented here, in STATE.md decisions, and in RUNTIME-VALIDATION.md Notes as a **persistent risk** that plan 10-03 must explicitly address. + +## User Setup Required + +None — all configuration was performed by the technician during the four human-action checkpoints. + +## Next Phase Readiness + +- **Plan 10-03 (sign-off):** Ready to start. RTVAL-01..04 all have recorded statuses (no PENDING lines). Sign-off agent MUST: + 1. Read the "CRITICAL: Attestation-Only Audit Trail Damage" section of this summary + 2. Decide whether to re-run RTVAL-02/03/04 with evidence OR explicitly accept the attestation-only runtime half + 3. Populate the "Signed off by" and "Signed off date" fields in RUNTIME-VALIDATION.md +- **Phase 11 (real-world rollout) readiness:** Conditionally ready. The .intunewin generator is known-good (fixed in commits 74535ea + 7716246, confirmed by RTVAL-01). The runtime behavior (install/detect/uninstall under SYSTEM) is attested but not artifact-proven — Phase 11 rollout will be the real stress test. +- **Blockers:** None for plan 10-03. BLOCKER-01 (ISSUE-01) resolved. + +## Self-Check: PASSED + +All referenced commits verified to exist in git history (`403a0a5`, `7716246`, `74535ea`, `cf3b86a`, `870158b`, `475d593`, `2c912ca`). SUMMARY.md written to expected path. + +--- +*Phase: 10-real-world-runtime-validation* +*Plan: 02-live-intune-runtime-validation* +*Completed: 2026-04-13*