kawa/ImpTune
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
main
ImpTune/.planning/phases/10-real-world-runtime-validation/10-02-live-intune-runtime-validation-SUMMARY.md
T
kawa 55516ee10f Commit initial
2026-04-15 17:57:12 +02:00

13 KiB
Raw Permalink Blame History

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, duration, completed
phase plan subsystem tags requires provides affects tech-stack key-files key-decisions patterns-established requirements-completed duration completed
10-real-world-runtime-validation 02 testing
intune
intunewin
runtime-validation
rtval
pnputil
system-context
attestation-only
phase provides
10-real-world-runtime-validation Plan 10-01 — locked .intunewin package (Copieur_2eme), tenant + device metadata, RUNTIME-VALIDATION.md scaffold
RTVAL-01 PASS (artifact-backed) — real Intune tenant accepts generated .intunewin without format errors
RTVAL-02 PASS (attestation-only) — install under SYSTEM context confirmed verbally by technician
RTVAL-03 PASS (attestation-only) — Intune detection rule reports Installed confirmed verbally
RTVAL-04 PASS (attestation-only) — uninstall under SYSTEM confirmed verbally; printer removed cleanly
Two structural .intunewin generator defects discovered and fixed during RTVAL-01 (ISSUE-01): HMAC scope + Detection.xml format
10-03-sign-off-and-phase-closure
11-real-world-rollout-and-feedback
added patterns
Attestation-only PASS is permitted but MUST be flagged in RUNTIME-VALIDATION.md Notes and called out in plan SUMMARY for downstream verifiers
Debug-then-resume flow: RTVAL failure → /gsd:debug on generator → commit fixes → re-test same check against fresh build
created modified
.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
RTVAL-01 initial FAIL (blocker) debugged to root cause: two .intunewin generator defects (HMAC over ciphertext-only instead of IV+ciphertext; Detection.xml not matching IntuneWinAppUtil.exe reference format). Fixed in commits 74535ea and 7716246; re-test PASS.
RTVAL-02, RTVAL-03, and RTVAL-04 accepted as attestation-only PASSes per explicit, repeated user approval. User was warned on RTVAL-03 about consecutive attestation-only checks weakening audit trail, and warned AGAIN on RTVAL-04, and chose to proceed both times.
Phase 10 runtime audit trail is now structurally weakened: 3 of 4 RTVAL runtime checks are verbal-only. Plan 10-03 sign-off must either accept this or force re-run of RTVAL-02/03/04 with full artifact capture.
SUMMARY.md must surface attestation-only debt prominently so the wave-3 verifier and phase verifier cannot miss it
.intunewin generator defects discovered during real-tenant testing are captured as ISSUE-## entries in RUNTIME-VALIDATION.md with resolution commit refs
RTVAL-01
RTVAL-02
RTVAL-03
RTVAL-04
~1 day (wall-clock, spanning debug + re-test + three attestation checkpoints) 2026-04-13

Phase 10 Plan 02: Live Intune Runtime Validation Summary

All four RTVAL runtime checks recorded against live tenant rubis.fr on ARES-5CG5220YTM — RTVAL-01 artifact-backed PASS after fixing two .intunewin generator defects; RTVAL-02/03/04 attestation-only PASSes (user approved despite repeated warnings about weakened audit trail).

Performance

  • Duration: ~1 day wall-clock (includes debug cycle for ISSUE-01 + three human-action checkpoints)
  • Started: 2026-04-13
  • Completed: 2026-04-13
  • Tasks: 4 (all checkpoint:human-action)
  • Files modified: 1 (RUNTIME-VALIDATION.md — incrementally across 4 task commits)

Accomplishments

  • RTVAL-01 (artifact-backed): Real Intune tenant (rubis.fr) accepted the generated Copieur_2eme.intunewin package without format errors on re-test against the fixed build. Two screenshots committed (rtval-01-tenant-upload.png, rtval-01-app-assigned.png).
  • RTVAL-01 debug cycle: Initial run FAILED with empty wizard fields and greyed-out OK button. Root cause debugged to two structural defects in the ImpTune .intunewin generator:
    1. HMAC was computed over ciphertext only instead of IV+ciphertext (commit 74535ea)
    2. Detection.xml did not match the IntuneWinAppUtil.exe reference format (commit 7716246) Both fixes landed; RTVAL-01 re-tested PASS.
  • RTVAL-02 (attestation-only): Technician verbally confirmed install script ran under SYSTEM on ARES-5CG5220YTM, driver deployed, Intune portal showed Installed. No log excerpt or screenshot captured.
  • RTVAL-03 (attestation-only, 2nd consecutive): Technician verbally confirmed Intune reports Installed and detection rule passes. No screenshot or detect transcript captured. User warned about weakened audit trail and approved.
  • RTVAL-04 (attestation-only, 3rd consecutive): Technician verbally confirmed Intune uninstall ran under SYSTEM, exited cleanly, printer removed from ARES-5CG5220YTM. No uninstall log or portal screenshot captured. User warned a second time about the cumulative audit-trail damage and approved.

Task Commits

Each task was committed atomically:

  1. Task 1: RTVAL-01 — Tenant ingestion (FAIL → debug → PASS)
    • 403a0a5 (fix): initial FAIL recorded — Intune cannot parse generated .intunewin
    • 46cfde0 (chore): STATE.md updated — plan blocked at RTVAL-01 FAIL
    • 44a4f2c (docs): debug session recorded
    • 7716246 (fix): Detection.xml aligned with IntuneWinAppUtil.exe reference format
    • 74535ea (fix): HMAC computed over IV+ciphertext
    • 35d4edc (docs): debug session updated — HMAC scope bug identified
    • 00b709d (docs): debug session resolved
    • cf3b86a (docs): RTVAL-01 PASS recorded on re-test against fixed build
    • bcd12a3 (docs): BLOCKER-01 cleared, plan resumed at Task 2
  2. Task 2: RTVAL-02 — Install under SYSTEM — 870158b (docs) — attestation-only PASS
    • d86199f (docs): STATE.md advanced to Task 3
  3. Task 3: RTVAL-03 — Detection rule reports Installed — 475d593 (docs) — attestation-only PASS (2nd consecutive)
    • b0be3d3 (docs): STATE.md advanced to Task 4
  4. Task 4: RTVAL-04 — Uninstall under SYSTEM — 2c912ca (docs) — attestation-only PASS (3rd consecutive)

Plan metadata: (this commit) docs(10-02): complete live-intune-runtime-validation plan

Files Created/Modified

  • .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md — Status line updated to reflect all four RTVAL checks recorded; RTVAL-01 through RTVAL-04 sections populated with results, evidence (or attestation disclaimer), and Notes.
  • imptune/generators/intunewin_builder.py (via debug cycle, commits 74535ea, 7716246) — HMAC scope fix + Detection.xml format alignment; unblocked RTVAL-01.

Decisions Made

  • Debug-before-abandon: RTVAL-01 initial FAIL was not deferred to a gap-closure plan — instead, the .intunewin generator was debugged inline (ISSUE-01) because the root cause was clearly in-scope for the package under test. Two commits fixed it and the same check was re-run on the fresh build.
  • Attestation-only accepted for RTVAL-02/03/04: User explicitly, repeatedly chose verbal attestation over artifact capture. The executor flagged each successive choice (warning on RTVAL-03, second warning on RTVAL-04) and proceeded per user instruction. This is documented here, in RUNTIME-VALIDATION.md Notes for each check, and in STATE.md decisions log — it MUST be visible to plan 10-03 (sign-off) and the phase verifier.

Deviations from Plan

Rule 1 (Bug) — .intunewin generator defects discovered during RTVAL-01

1. [Rule 1 - Bug] HMAC computed over ciphertext only instead of IV+ciphertext

  • Found during: Task 1 (RTVAL-01) — Intune wizard rejected the package, debugged to the root cause
  • Issue: intunewin_builder.py computed HMAC over the ciphertext alone, but the Intune Win32 package format requires HMAC over IV || ciphertext. Intune's package parser rejected the file silently (empty wizard fields).
  • Fix: Updated HMAC computation to include the IV prefix.
  • Files modified: imptune/generators/intunewin_builder.py
  • Committed in: 74535ea (standalone fix commit, not inside a task commit)

2. [Rule 1 - Bug] Detection.xml did not match IntuneWinAppUtil.exe reference format

  • Found during: Task 1 (RTVAL-01) — same debug cycle
  • Issue: Embedded Detection.xml used a slightly different schema/element ordering than IntuneWinAppUtil.exe emits. Intune parses the reference format strictly, so the wizard could not populate metadata fields.
  • Fix: Aligned Detection.xml generation with the IntuneWinAppUtil.exe reference output byte-for-byte.
  • Files modified: imptune/generators/intunewin_builder.py
  • Committed in: 7716246 (standalone fix commit, not inside a task commit)

Total deviations: 2 auto-fixed (both Rule 1 bugs in the generator under test) Impact on plan: Both fixes were essential and in-scope — the whole point of RTVAL-01 is to discover exactly this class of defect. Plan was paused via debug session and resumed without re-planning.

⚠️ CRITICAL: Attestation-Only Audit Trail Damage

Read this before signing off Phase 10 in plan 10-03.

Three of the four RTVAL runtime checks in this plan are attestation-only (verbal confirmation from the technician, no log excerpt, no screenshot, no transcript):

Check Status Artifact-backed? Warning at time of acceptance
RTVAL-01 PASS Yes —
RTVAL-02 PASS (attestation-only) No User approved "Pass without evidence"
RTVAL-03 PASS (attestation-only, 2nd) No User warned about weakened audit trail, approved anyway
RTVAL-04 PASS (attestation-only, 3rd) No User warned a second time about weakened audit trail, approved anyway

What this means:

  • Only tenant ingestion (RTVAL-01) is independently verifiable from committed evidence.
  • The entire runtime half of Phase 10 (install under SYSTEM, detection, uninstall under SYSTEM) rests on the technician's verbal report. None of the required observations ($PSScriptRoot resolved, pnputil /add-driver success line, NT AUTHORITY\SYSTEM banner, exit code 0, Get-Printer shows/hides the printer, Intune portal Installed/Not installed) were captured to disk.
  • For any future regression, incident investigation, or compliance audit, RTVAL-02/03/04 should be treated as soft PASSes and re-run with full artifact capture before the phase is considered closed.

Recommended action for plan 10-03 (sign-off):

  1. Either: re-run RTVAL-02/03/04 with full artifact capture before signing off, OR
  2. Explicitly accept the attestation-only audit trail in the sign-off block and record the technician's name + the compensating control (e.g., known test device, known tenant, same session).

The sign-off checklist in RUNTIME-VALIDATION.md should not be ticked silently — the reviewer needs to actively acknowledge the attestation-only nature of 3 of 4 checks.

Issues Encountered

  • ISSUE-01 (resolved): Initial RTVAL-01 FAIL — Intune wizard could not parse Copieur_2eme.intunewin. Root-caused during a /gsd:debug session to two .intunewin generator defects (HMAC scope + Detection.xml format). Both fixed in 74535ea and 7716246. RTVAL-01 re-tested PASS against the fixed build. See RUNTIME-VALIDATION.md "Issues Found → ISSUE-01" for the full trail.
  • Attestation debt: Three consecutive runtime checks went attestation-only. Not an "issue" in the traditional sense (user chose it), but documented here, in STATE.md decisions, and in RUNTIME-VALIDATION.md Notes as a persistent risk that plan 10-03 must explicitly address.

User Setup Required

None — all configuration was performed by the technician during the four human-action checkpoints.

Next Phase Readiness

  • Plan 10-03 (sign-off): Ready to start. RTVAL-01..04 all have recorded statuses (no PENDING lines). Sign-off agent MUST:
    1. Read the "CRITICAL: Attestation-Only Audit Trail Damage" section of this summary
    2. Decide whether to re-run RTVAL-02/03/04 with evidence OR explicitly accept the attestation-only runtime half
    3. Populate the "Signed off by" and "Signed off date" fields in RUNTIME-VALIDATION.md
  • Phase 11 (real-world rollout) readiness: Conditionally ready. The .intunewin generator is known-good (fixed in commits 74535ea + 7716246, confirmed by RTVAL-01). The runtime behavior (install/detect/uninstall under SYSTEM) is attested but not artifact-proven — Phase 11 rollout will be the real stress test.
  • Blockers: None for plan 10-03. BLOCKER-01 (ISSUE-01) resolved.

Self-Check: PASSED

All referenced commits verified to exist in git history (403a0a5, 7716246, 74535ea, cf3b86a, 870158b, 475d593, 2c912ca). SUMMARY.md written to expected path.

Phase: 10-real-world-runtime-validation Plan: 02-live-intune-runtime-validation Completed: 2026-04-13

Reference in New Issue View Git Blame Copy Permalink