183 lines
13 KiB
Markdown
183 lines
13 KiB
Markdown
---
|
|
phase: 10-real-world-runtime-validation
|
|
plan: 02
|
|
subsystem: testing
|
|
tags: [intune, intunewin, runtime-validation, rtval, pnputil, system-context, attestation-only]
|
|
|
|
# Dependency graph
|
|
requires:
|
|
- phase: 10-real-world-runtime-validation
|
|
provides: "Plan 10-01 — locked .intunewin package (Copieur_2eme), tenant + device metadata, RUNTIME-VALIDATION.md scaffold"
|
|
provides:
|
|
- "RTVAL-01 PASS (artifact-backed) — real Intune tenant accepts generated .intunewin without format errors"
|
|
- "RTVAL-02 PASS (attestation-only) — install under SYSTEM context confirmed verbally by technician"
|
|
- "RTVAL-03 PASS (attestation-only) — Intune detection rule reports Installed confirmed verbally"
|
|
- "RTVAL-04 PASS (attestation-only) — uninstall under SYSTEM confirmed verbally; printer removed cleanly"
|
|
- "Two structural .intunewin generator defects discovered and fixed during RTVAL-01 (ISSUE-01): HMAC scope + Detection.xml format"
|
|
affects:
|
|
- 10-03-sign-off-and-phase-closure
|
|
- 11-real-world-rollout-and-feedback
|
|
|
|
# Tech tracking
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- "Attestation-only PASS is permitted but MUST be flagged in RUNTIME-VALIDATION.md Notes and called out in plan SUMMARY for downstream verifiers"
|
|
- "Debug-then-resume flow: RTVAL failure → /gsd:debug on generator → commit fixes → re-test same check against fresh build"
|
|
|
|
key-files:
|
|
created: []
|
|
modified:
|
|
- .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
|
|
|
|
key-decisions:
|
|
- "RTVAL-01 initial FAIL (blocker) debugged to root cause: two .intunewin generator defects (HMAC over ciphertext-only instead of IV+ciphertext; Detection.xml not matching IntuneWinAppUtil.exe reference format). Fixed in commits 74535ea and 7716246; re-test PASS."
|
|
- "RTVAL-02, RTVAL-03, and RTVAL-04 accepted as attestation-only PASSes per explicit, repeated user approval. User was warned on RTVAL-03 about consecutive attestation-only checks weakening audit trail, and warned AGAIN on RTVAL-04, and chose to proceed both times."
|
|
- "Phase 10 runtime audit trail is now structurally weakened: 3 of 4 RTVAL runtime checks are verbal-only. Plan 10-03 sign-off must either accept this or force re-run of RTVAL-02/03/04 with full artifact capture."
|
|
|
|
patterns-established:
|
|
- "SUMMARY.md must surface attestation-only debt prominently so the wave-3 verifier and phase verifier cannot miss it"
|
|
- ".intunewin generator defects discovered during real-tenant testing are captured as ISSUE-## entries in RUNTIME-VALIDATION.md with resolution commit refs"
|
|
|
|
requirements-completed:
|
|
- RTVAL-01
|
|
- RTVAL-02
|
|
- RTVAL-03
|
|
- RTVAL-04
|
|
|
|
# Metrics
|
|
duration: "~1 day (wall-clock, spanning debug + re-test + three attestation checkpoints)"
|
|
completed: 2026-04-13
|
|
---
|
|
|
|
# Phase 10 Plan 02: Live Intune Runtime Validation Summary
|
|
|
|
**All four RTVAL runtime checks recorded against live tenant rubis.fr on ARES-5CG5220YTM — RTVAL-01 artifact-backed PASS after fixing two .intunewin generator defects; RTVAL-02/03/04 attestation-only PASSes (user approved despite repeated warnings about weakened audit trail).**
|
|
|
|
## Performance
|
|
|
|
- **Duration:** ~1 day wall-clock (includes debug cycle for ISSUE-01 + three human-action checkpoints)
|
|
- **Started:** 2026-04-13
|
|
- **Completed:** 2026-04-13
|
|
- **Tasks:** 4 (all `checkpoint:human-action`)
|
|
- **Files modified:** 1 (RUNTIME-VALIDATION.md — incrementally across 4 task commits)
|
|
|
|
## Accomplishments
|
|
|
|
- **RTVAL-01 (artifact-backed):** Real Intune tenant (rubis.fr) accepted the generated `Copieur_2eme.intunewin` package without format errors on re-test against the fixed build. Two screenshots committed (`rtval-01-tenant-upload.png`, `rtval-01-app-assigned.png`).
|
|
- **RTVAL-01 debug cycle:** Initial run FAILED with empty wizard fields and greyed-out OK button. Root cause debugged to two structural defects in the ImpTune .intunewin generator:
|
|
1. HMAC was computed over ciphertext only instead of IV+ciphertext (commit `74535ea`)
|
|
2. Detection.xml did not match the IntuneWinAppUtil.exe reference format (commit `7716246`)
|
|
Both fixes landed; RTVAL-01 re-tested PASS.
|
|
- **RTVAL-02 (attestation-only):** Technician verbally confirmed install script ran under SYSTEM on ARES-5CG5220YTM, driver deployed, Intune portal showed Installed. No log excerpt or screenshot captured.
|
|
- **RTVAL-03 (attestation-only, 2nd consecutive):** Technician verbally confirmed Intune reports Installed and detection rule passes. No screenshot or detect transcript captured. User warned about weakened audit trail and approved.
|
|
- **RTVAL-04 (attestation-only, 3rd consecutive):** Technician verbally confirmed Intune uninstall ran under SYSTEM, exited cleanly, printer removed from ARES-5CG5220YTM. No uninstall log or portal screenshot captured. User warned a second time about the cumulative audit-trail damage and approved.
|
|
|
|
## Task Commits
|
|
|
|
Each task was committed atomically:
|
|
|
|
1. **Task 1: RTVAL-01 — Tenant ingestion (FAIL → debug → PASS)**
|
|
- `403a0a5` (fix): initial FAIL recorded — Intune cannot parse generated .intunewin
|
|
- `46cfde0` (chore): STATE.md updated — plan blocked at RTVAL-01 FAIL
|
|
- `44a4f2c` (docs): debug session recorded
|
|
- `7716246` (fix): Detection.xml aligned with IntuneWinAppUtil.exe reference format
|
|
- `74535ea` (fix): HMAC computed over IV+ciphertext
|
|
- `35d4edc` (docs): debug session updated — HMAC scope bug identified
|
|
- `00b709d` (docs): debug session resolved
|
|
- `cf3b86a` (docs): RTVAL-01 PASS recorded on re-test against fixed build
|
|
- `bcd12a3` (docs): BLOCKER-01 cleared, plan resumed at Task 2
|
|
2. **Task 2: RTVAL-02 — Install under SYSTEM** — `870158b` (docs) — attestation-only PASS
|
|
- `d86199f` (docs): STATE.md advanced to Task 3
|
|
3. **Task 3: RTVAL-03 — Detection rule reports Installed** — `475d593` (docs) — attestation-only PASS (2nd consecutive)
|
|
- `b0be3d3` (docs): STATE.md advanced to Task 4
|
|
4. **Task 4: RTVAL-04 — Uninstall under SYSTEM** — `2c912ca` (docs) — attestation-only PASS (3rd consecutive)
|
|
|
|
**Plan metadata:** (this commit) `docs(10-02): complete live-intune-runtime-validation plan`
|
|
|
|
## Files Created/Modified
|
|
|
|
- `.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md` — Status line updated to reflect all four RTVAL checks recorded; RTVAL-01 through RTVAL-04 sections populated with results, evidence (or attestation disclaimer), and Notes.
|
|
- `imptune/generators/intunewin_builder.py` (via debug cycle, commits `74535ea`, `7716246`) — HMAC scope fix + Detection.xml format alignment; unblocked RTVAL-01.
|
|
|
|
## Decisions Made
|
|
|
|
- **Debug-before-abandon:** RTVAL-01 initial FAIL was not deferred to a gap-closure plan — instead, the .intunewin generator was debugged inline (ISSUE-01) because the root cause was clearly in-scope for the package under test. Two commits fixed it and the same check was re-run on the fresh build.
|
|
- **Attestation-only accepted for RTVAL-02/03/04:** User explicitly, repeatedly chose verbal attestation over artifact capture. The executor flagged each successive choice (warning on RTVAL-03, second warning on RTVAL-04) and proceeded per user instruction. This is documented here, in RUNTIME-VALIDATION.md Notes for each check, and in STATE.md decisions log — it MUST be visible to plan 10-03 (sign-off) and the phase verifier.
|
|
|
|
## Deviations from Plan
|
|
|
|
### Rule 1 (Bug) — `.intunewin` generator defects discovered during RTVAL-01
|
|
|
|
**1. [Rule 1 - Bug] HMAC computed over ciphertext only instead of IV+ciphertext**
|
|
- **Found during:** Task 1 (RTVAL-01) — Intune wizard rejected the package, debugged to the root cause
|
|
- **Issue:** `intunewin_builder.py` computed HMAC over the ciphertext alone, but the Intune Win32 package format requires HMAC over `IV || ciphertext`. Intune's package parser rejected the file silently (empty wizard fields).
|
|
- **Fix:** Updated HMAC computation to include the IV prefix.
|
|
- **Files modified:** `imptune/generators/intunewin_builder.py`
|
|
- **Committed in:** `74535ea` (standalone fix commit, not inside a task commit)
|
|
|
|
**2. [Rule 1 - Bug] Detection.xml did not match IntuneWinAppUtil.exe reference format**
|
|
- **Found during:** Task 1 (RTVAL-01) — same debug cycle
|
|
- **Issue:** Embedded Detection.xml used a slightly different schema/element ordering than IntuneWinAppUtil.exe emits. Intune parses the reference format strictly, so the wizard could not populate metadata fields.
|
|
- **Fix:** Aligned Detection.xml generation with the IntuneWinAppUtil.exe reference output byte-for-byte.
|
|
- **Files modified:** `imptune/generators/intunewin_builder.py`
|
|
- **Committed in:** `7716246` (standalone fix commit, not inside a task commit)
|
|
|
|
---
|
|
|
|
**Total deviations:** 2 auto-fixed (both Rule 1 bugs in the generator under test)
|
|
**Impact on plan:** Both fixes were essential and in-scope — the whole point of RTVAL-01 is to discover exactly this class of defect. Plan was paused via debug session and resumed without re-planning.
|
|
|
|
## ⚠️ CRITICAL: Attestation-Only Audit Trail Damage
|
|
|
|
**Read this before signing off Phase 10 in plan 10-03.**
|
|
|
|
Three of the four RTVAL runtime checks in this plan are **attestation-only** (verbal confirmation from the technician, no log excerpt, no screenshot, no transcript):
|
|
|
|
| Check | Status | Artifact-backed? | Warning at time of acceptance |
|
|
| -------- | ----------------------------- | ---------------- | -------------------------------------------------------------------------- |
|
|
| RTVAL-01 | PASS | **Yes** | — |
|
|
| RTVAL-02 | PASS (attestation-only) | No | User approved "Pass without evidence" |
|
|
| RTVAL-03 | PASS (attestation-only, 2nd) | No | User warned about weakened audit trail, approved anyway |
|
|
| RTVAL-04 | PASS (attestation-only, 3rd) | No | User warned a **second time** about weakened audit trail, approved anyway |
|
|
|
|
**What this means:**
|
|
|
|
- Only tenant ingestion (RTVAL-01) is independently verifiable from committed evidence.
|
|
- The entire **runtime half** of Phase 10 (install under SYSTEM, detection, uninstall under SYSTEM) rests on the technician's verbal report. None of the required observations (`$PSScriptRoot` resolved, `pnputil /add-driver` success line, `NT AUTHORITY\SYSTEM` banner, exit code 0, `Get-Printer` shows/hides the printer, Intune portal Installed/Not installed) were captured to disk.
|
|
- For any future regression, incident investigation, or compliance audit, RTVAL-02/03/04 should be treated as **soft PASSes** and re-run with full artifact capture before the phase is considered closed.
|
|
|
|
**Recommended action for plan 10-03 (sign-off):**
|
|
|
|
1. Either: re-run RTVAL-02/03/04 with full artifact capture before signing off, OR
|
|
2. Explicitly accept the attestation-only audit trail in the sign-off block and record the technician's name + the compensating control (e.g., known test device, known tenant, same session).
|
|
|
|
The sign-off checklist in RUNTIME-VALIDATION.md should **not** be ticked silently — the reviewer needs to actively acknowledge the attestation-only nature of 3 of 4 checks.
|
|
|
|
## Issues Encountered
|
|
|
|
- **ISSUE-01 (resolved):** Initial RTVAL-01 FAIL — Intune wizard could not parse `Copieur_2eme.intunewin`. Root-caused during a /gsd:debug session to two `.intunewin` generator defects (HMAC scope + Detection.xml format). Both fixed in `74535ea` and `7716246`. RTVAL-01 re-tested PASS against the fixed build. See RUNTIME-VALIDATION.md "Issues Found → ISSUE-01" for the full trail.
|
|
- **Attestation debt:** Three consecutive runtime checks went attestation-only. Not an "issue" in the traditional sense (user chose it), but documented here, in STATE.md decisions, and in RUNTIME-VALIDATION.md Notes as a **persistent risk** that plan 10-03 must explicitly address.
|
|
|
|
## User Setup Required
|
|
|
|
None — all configuration was performed by the technician during the four human-action checkpoints.
|
|
|
|
## Next Phase Readiness
|
|
|
|
- **Plan 10-03 (sign-off):** Ready to start. RTVAL-01..04 all have recorded statuses (no PENDING lines). Sign-off agent MUST:
|
|
1. Read the "CRITICAL: Attestation-Only Audit Trail Damage" section of this summary
|
|
2. Decide whether to re-run RTVAL-02/03/04 with evidence OR explicitly accept the attestation-only runtime half
|
|
3. Populate the "Signed off by" and "Signed off date" fields in RUNTIME-VALIDATION.md
|
|
- **Phase 11 (real-world rollout) readiness:** Conditionally ready. The .intunewin generator is known-good (fixed in commits 74535ea + 7716246, confirmed by RTVAL-01). The runtime behavior (install/detect/uninstall under SYSTEM) is attested but not artifact-proven — Phase 11 rollout will be the real stress test.
|
|
- **Blockers:** None for plan 10-03. BLOCKER-01 (ISSUE-01) resolved.
|
|
|
|
## Self-Check: PASSED
|
|
|
|
All referenced commits verified to exist in git history (`403a0a5`, `7716246`, `74535ea`, `cf3b86a`, `870158b`, `475d593`, `2c912ca`). SUMMARY.md written to expected path.
|
|
|
|
---
|
|
*Phase: 10-real-world-runtime-validation*
|
|
*Plan: 02-live-intune-runtime-validation*
|
|
*Completed: 2026-04-13*
|