Commit initial

This commit is contained in:
2026-04-15 17:57:12 +02:00
parent 005d8e797e
commit 55516ee10f
269 changed files with 26854 additions and 0 deletions
@@ -0,0 +1,156 @@
---
phase: 10-real-world-runtime-validation
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
- .planning/phases/10-real-world-runtime-validation/evidence/.gitkeep
autonomous: false
requirements:
- RTVAL-05
must_haves:
truths:
- "A real .intunewin package generated by the current ImpTune build exists on disk and its SHA256 is recorded"
- "A RUNTIME-VALIDATION.md scaffold exists with tenant, device, OS build, and driver vendor fields ready to fill"
- "An evidence/ folder exists under the phase directory for screenshots and logs"
artifacts:
- path: ".planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md"
provides: "Scaffolded evidence report with metadata table and empty result sections"
contains: "## Tenant & Environment"
- path: ".planning/phases/10-real-world-runtime-validation/evidence/.gitkeep"
provides: "Evidence folder placeholder so git tracks the directory"
key_links:
- from: "imptune package export UI"
to: "RUNTIME-VALIDATION.md metadata"
via: "technician records printer name, driver vendor, SHA256, build commit"
pattern: "Package SHA256:.*[a-f0-9]{64}"
---
<objective>
Produce the artifact under test and the evidence scaffold before any real-world testing begins.
Purpose: You cannot validate what you did not generate. Lock the exact commit, package, and metadata that will be sent to the tenant so the final report is reproducible.
Output: A real .intunewin package generated by the running ImpTune instance and a RUNTIME-VALIDATION.md scaffold with all environment fields waiting to be filled in.
</objective>
<execution_context>
@C:/Users/SebastienQUEROL/.claude/get-shit-done/workflows/execute-plan.md
@C:/Users/SebastienQUEROL/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/REQUIREMENTS.md
</context>
<tasks>
<task type="auto">
<name>Task 1: Scaffold RUNTIME-VALIDATION.md and evidence folder</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md, .planning/phases/10-real-world-runtime-validation/evidence/.gitkeep</files>
<action>
Create `RUNTIME-VALIDATION.md` under the phase directory with the following sections (empty values to be filled by technician in later tasks):
```
# Runtime Validation Report — ImpTune v1.1
**Status:** DRAFT (in progress)
**Started:** <YYYY-MM-DD>
**Signed off by:** <name>
**Signed off date:** <YYYY-MM-DD>
## Tenant & Environment
| Field | Value |
|-------|-------|
| Intune tenant (domain) | |
| Test device hostname | |
| Windows OS build | |
| Device is Intune-managed | yes / no |
| Driver vendor(s) tested | |
| ImpTune commit SHA | |
| Package file name | |
| Package SHA256 | |
| Generated at | |
## RTVAL-01 — Tenant ingestion
Status: PENDING
Evidence: (screenshot path)
Notes:
## RTVAL-02 — Install under SYSTEM (pnputil + $PSScriptRoot)
Status: PENDING
Evidence: (IntuneManagementExtension.log excerpt path)
Notes:
## RTVAL-03 — Detection rule reports Installed
Status: PENDING
Evidence: (screenshot path)
Notes:
## RTVAL-04 — Uninstall under SYSTEM
Status: PENDING
Evidence: (log path + screenshot)
Notes:
## Issues Found
(list any defects; link to v1.1 tickets or defer rationale)
## Sign-off
- [ ] All five RTVAL criteria PASS or have documented rationale
- [ ] Evidence files committed under `evidence/`
- [ ] Signed by: ___________________
```
Also create `.planning/phases/10-real-world-runtime-validation/evidence/.gitkeep` as an empty file so git tracks the evidence directory.
</action>
<verify>
<automated>test -f .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md && test -f .planning/phases/10-real-world-runtime-validation/evidence/.gitkeep && grep -q "RTVAL-01" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md && grep -q "Sign-off" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md</automated>
</verify>
<done>RUNTIME-VALIDATION.md scaffold exists with all five RTVAL sections and a sign-off block; evidence/ folder exists in git.</done>
</task>
<task type="checkpoint:human-action" gate="blocking">
<name>Task 2: Technician generates a real .intunewin package from running ImpTune</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md</files>
<action>
PAUSE for human technician. Claude cannot run Docker + click through the export UI + read a local file's hash on the user's box. The executor agent must present the following instructions and wait.
Technician MUST perform the following:
1. Start ImpTune from the current commit (`docker compose up` or the local dev command).
2. Record the current commit SHA: `git rev-parse HEAD` and paste into RUNTIME-VALIDATION.md "ImpTune commit SHA".
3. In the UI, pick or create a test printer using a driver vendor representative of the target MSP environment (e.g., HP Universal, Konica Minolta, Brother). Record the vendor in the report.
4. Export the package via the package export button. Save the resulting `.intunewin` file locally.
5. Compute its SHA256: `certutil -hashfile <file>.intunewin SHA256` (Windows) or `sha256sum <file>.intunewin` (Linux/WSL). Paste into report.
6. Fill in tenant domain, test device hostname, Windows OS build (`winver` on device), and generation timestamp in the report metadata table.
7. Copy the generated `.intunewin` file into `.planning/phases/10-real-world-runtime-validation/evidence/` (or note its archived location if too large for git).
Expected outcome: RUNTIME-VALIDATION.md metadata table is FULLY populated before proceeding to plan 10-02.
Resume signal: reply "metadata filled" (with the package SHA256) or describe blockers.
</action>
<verify>
<automated>! grep -E "^\| (Intune tenant|Test device hostname|Windows OS build|Driver vendor\(s\) tested|ImpTune commit SHA|Package file name|Package SHA256|Generated at) \| *\|" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md</automated>
</verify>
<done>Every row in the Tenant & Environment metadata table has a non-empty value; technician has replied "metadata filled".</done>
</task>
</tasks>
<verification>
- RUNTIME-VALIDATION.md exists with scaffold
- Metadata table populated (commit SHA, tenant, device, OS build, vendor, package SHA256)
- A real .intunewin file is generated and located (path or archive recorded)
</verification>
<success_criteria>
Technician signals "metadata filled"; report metadata table has no empty fields; package file exists and its SHA256 is recorded.
</success_criteria>
<output>
After completion, create `.planning/phases/10-real-world-runtime-validation/10-01-preflight-package-and-scaffold-SUMMARY.md`
</output>
@@ -0,0 +1,120 @@
---
phase: 10-real-world-runtime-validation
plan: 01
subsystem: testing
tags: [intune, intunewin, runtime-validation, evidence, ricoh]
# Dependency graph
requires:
- phase: 09-ux-tech-debt-closure
provides: polished ImpTune build that generates the .intunewin package under test
provides:
- RUNTIME-VALIDATION.md scaffold with all Tenant & Environment metadata populated
- Copieur_2eme.intunewin evidence file committed to evidence/
- Baseline commit SHA and package SHA256 locked for reproducible validation
affects:
- 10-02-tenant-ingestion
- 10-03-system-install-pnputil
- 10-04-detection-rule
- 10-05-uninstall
# Tech tracking
tech-stack:
added: []
patterns:
- "RUNTIME-VALIDATION.md as living evidence report — filled incrementally across Phase 10 plans"
- "evidence/ directory under phase for screenshots, logs, and generated artifacts"
key-files:
created:
- .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
- .planning/phases/10-real-world-runtime-validation/evidence/.gitkeep
- .planning/phases/10-real-world-runtime-validation/evidence/Copieur_2eme.intunewin
modified: []
key-decisions:
- "Package under test: Copieur_2eme.intunewin (Ricoh PCL6 Universal Print driver) committed to evidence/ for traceability"
- "Commit SHA locked at 1c3f458583d826bdcde7ffa5192de6f2fb8e1202 — all Phase 10 RTVAL results reference this exact build"
patterns-established:
- "Lock commit SHA + package SHA256 before any runtime testing begins — prevents report drift"
requirements-completed:
- RTVAL-05
# Metrics
duration: ~30min (human-action gate included)
completed: 2026-04-13
---
# Phase 10 Plan 01: Preflight Package and Scaffold Summary
**Ricoh PCL6 .intunewin package (Copieur_2eme) generated and committed to evidence/ with full tenant/environment metadata locked in RUNTIME-VALIDATION.md**
## Performance
- **Duration:** ~30 min (includes human-action gate for technician to generate package and fill metadata)
- **Started:** 2026-04-13
- **Completed:** 2026-04-13
- **Tasks:** 2
- **Files modified:** 3
## Accomplishments
- Scaffolded RUNTIME-VALIDATION.md with five RTVAL test sections and a sign-off block
- Technician populated all Tenant & Environment metadata rows: tenant domain (arescom.fr), device hostname, OS build 26200.7171, Ricoh PCL6 driver vendor, ImpTune commit SHA, package filename, package SHA256, and generation date
- Committed Copieur_2eme.intunewin (33 MB) into evidence/ so the exact artifact under test is reproducible
## Task Commits
Each task was committed atomically:
1. **Task 1: Scaffold RUNTIME-VALIDATION.md and evidence folder**`1c3f458` (chore)
2. **Task 2: Technician generates a real .intunewin package from running ImpTune**`88cf53d` (chore)
## Files Created/Modified
- `.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md` — evidence report scaffold with all metadata populated, Started date set, five RTVAL sections ready
- `.planning/phases/10-real-world-runtime-validation/evidence/.gitkeep` — git placeholder for evidence directory
- `.planning/phases/10-real-world-runtime-validation/evidence/Copieur_2eme.intunewin` — real package generated by ImpTune at commit 1c3f458
## Decisions Made
- Package under test is the Ricoh PCL6 Universal Print driver (ricoh.zip), representative of the target MSP environment
- ImpTune commit SHA locked at `1c3f458583d826bdcde7ffa5192de6f2fb8e1202` — all subsequent RTVAL plans reference this exact build
- .intunewin file committed directly to evidence/ (33 MB); acceptable for this use case since it is the artifact under test, not a build artifact
## Deviations from Plan
### Auto-fixed Issues
**1. [Rule 2 - Missing] Set Started date and added Generated at row**
- **Found during:** Task 2 completion (continuation agent review)
- **Issue:** RUNTIME-VALIDATION.md still had `<YYYY-MM-DD>` placeholder for Started, and the "Generated at" row was absent from the filled metadata table
- **Fix:** Set Started to 2026-04-13 and added "Generated at | 2026-04-13" row to close the table
- **Files modified:** RUNTIME-VALIDATION.md
- **Committed in:** 88cf53d (Task 2 commit)
---
**Total deviations:** 1 auto-fixed (missing field completeness)
**Impact on plan:** Minor completeness fix. No scope creep.
## Issues Encountered
None beyond the expected human-action gate (technician generating the package and filling metadata).
## User Setup Required
None — this plan is itself the human-action setup for Phase 10.
## Next Phase Readiness
- RUNTIME-VALIDATION.md metadata is fully locked; ready for Plan 10-02 (Tenant Ingestion)
- The Copieur_2eme.intunewin package is committed to evidence/ for reference
- Tenant: arescom.fr, Device: ARES-5CG5220YTM (Intune-managed), OS: 26200.7171
- No blockers for proceeding to 10-02
---
*Phase: 10-real-world-runtime-validation*
*Completed: 2026-04-13*
@@ -0,0 +1,207 @@
---
phase: 10-real-world-runtime-validation
plan: 02
type: execute
wave: 2
depends_on:
- 10-01
files_modified:
- .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
- .planning/phases/10-real-world-runtime-validation/evidence/
autonomous: false
requirements:
- RTVAL-01
- RTVAL-02
- RTVAL-03
- RTVAL-04
must_haves:
truths:
- "A real Intune tenant accepted the generated .intunewin without format errors (RTVAL-01)"
- "A real Intune-managed Windows endpoint ran the install script under SYSTEM with pnputil staging drivers and $PSScriptRoot resolving (RTVAL-02)"
- "Intune detection rule reports 'Installed' on the endpoint after install (RTVAL-03)"
- "Intune uninstall removes the printer cleanly under SYSTEM context (RTVAL-04)"
artifacts:
- path: ".planning/phases/10-real-world-runtime-validation/evidence/rtval-01-tenant-upload.png"
provides: "Screenshot of Intune showing successful package upload"
- path: ".planning/phases/10-real-world-runtime-validation/evidence/rtval-02-install-log.txt"
provides: "IntuneManagementExtension.log excerpt showing install success + pnputil + $PSScriptRoot"
- path: ".planning/phases/10-real-world-runtime-validation/evidence/rtval-03-detection.png"
provides: "Screenshot of Intune device install status = Installed"
- path: ".planning/phases/10-real-world-runtime-validation/evidence/rtval-04-uninstall-log.txt"
provides: "Log + printer-list verification showing printer removed under SYSTEM"
key_links:
- from: "Intune tenant"
to: "generated .intunewin from plan 10-01"
via: "Win32 app upload"
pattern: "Package (uploaded|accepted)"
- from: "install script ($PSScriptRoot)"
to: "bundled driver folder inside .intunewin"
via: "pnputil /add-driver staging under SYSTEM"
pattern: "pnputil.*Published Name|Driver package added successfully"
---
<objective>
Drive the four real-world RTVAL behavioral checks end-to-end on a live Intune tenant against a real Windows endpoint, and capture hard evidence (screenshots + device logs) for each.
Purpose: This is the phase's whole reason for existing. Every check here is a behavior no unit test can prove — it has to happen on real hardware with a real tenant.
Output: evidence/ folder populated with RTVAL-01..04 artifacts and RUNTIME-VALIDATION.md status lines flipped from PENDING to PASS/FAIL.
</objective>
<execution_context>
@C:/Users/SebastienQUEROL/.claude/get-shit-done/workflows/execute-plan.md
@C:/Users/SebastienQUEROL/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
@.planning/REQUIREMENTS.md
</context>
<tasks>
<task type="checkpoint:human-action" gate="blocking">
<name>Task 1: RTVAL-01 — Upload .intunewin to real Intune tenant</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md, .planning/phases/10-real-world-runtime-validation/evidence/rtval-01-tenant-upload.png, .planning/phases/10-real-world-runtime-validation/evidence/rtval-01-app-assigned.png</files>
<action>
PAUSE for human technician. Requires Intune tenant admin credentials — no complete free CLI path exists.
Technician MUST:
1. Sign in to https://intune.microsoft.com with tenant admin.
2. Go to Apps → Windows → Add → App type: Windows app (Win32).
3. Upload the `.intunewin` file from plan 10-01.
4. OBSERVE: Intune parses the file and presents the app metadata form without format errors.
5. Fill in minimal app info (name = "ImpTune RTVAL Test - <vendor>"), install command, uninstall command, detection rule (use the generated detect script: File exists / custom script as produced by ImpTune), requirement = Windows 10/11 x64, assignment = the single test device only.
6. Save the app. OBSERVE: app appears in the apps list with "Waiting for install status" or similar.
7. Capture screenshots:
- `evidence/rtval-01-tenant-upload.png` — upload success page showing the parsed .intunewin metadata
- `evidence/rtval-01-app-assigned.png` — app assignment screen showing the test device group
8. Update RUNTIME-VALIDATION.md section "RTVAL-01 — Tenant ingestion":
- Status: PASS if tenant accepted the package with NO format errors. FAIL otherwise.
- Evidence: paths to the two screenshots.
- Notes: any warnings Intune surfaced.
PASS: Intune accepted the upload and allowed assignment WITHOUT a format/extraction error.
FAIL: tenant rejected the file, surfaced a parse error, or could not load detection metadata — record the exact error text verbatim in Notes.
Resume signal: reply "RTVAL-01 PASS" + screenshot paths, or "RTVAL-01 FAIL" with exact tenant error text.
</action>
<verify>
<automated>test -f .planning/phases/10-real-world-runtime-validation/evidence/rtval-01-tenant-upload.png && ! grep -A1 "RTVAL-01 — Tenant ingestion" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md | grep -q "Status: PENDING"</automated>
</verify>
<done>RTVAL-01 status is PASS or FAIL in the report; tenant upload screenshot committed; any FAIL has verbatim error text in Notes.</done>
</task>
<task type="checkpoint:human-action" gate="blocking">
<name>Task 2: RTVAL-02 — Install on device under SYSTEM; verify pnputil + $PSScriptRoot</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md, .planning/phases/10-real-world-runtime-validation/evidence/rtval-02-install-log.txt, .planning/phases/10-real-world-runtime-validation/evidence/rtval-02-install-status.png</files>
<action>
PAUSE for human technician. Requires physical/virtual access to the test Intune-managed Windows endpoint.
Technician MUST:
1. On the test device, force Intune sync: Settings → Accounts → Access work or school → Info → Sync (or restart `IntuneManagementExtension`).
2. Wait for Intune to push the assignment. Typical latency 530 minutes.
3. While install runs, open `C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\IntuneManagementExtension.log` and `AgentExecutor.log`.
4. After install completes, extract the block covering this app's install run and save as `evidence/rtval-02-install-log.txt`.
5. OBSERVE in the log — every item below MUST be visible:
a. The install script ran under `NT AUTHORITY\SYSTEM` (script banner or process token).
b. `$PSScriptRoot` resolved to the Intune app staging directory (not empty, not `C:\Windows\system32`).
c. `pnputil /add-driver` (or equivalent) executed against the bundled driver folder and reported success (`Driver package added successfully` or `Published Name:`).
d. Final script exit code is 0.
6. Also capture Intune portal view: Apps → the test app → Device install status → test device → Install status = "Installed". Screenshot as `evidence/rtval-02-install-status.png`.
7. Update RUNTIME-VALIDATION.md section "RTVAL-02":
- Status: PASS only if ALL four observations hold. Otherwise FAIL with which observation failed.
- Evidence: both file paths.
PASS: SYSTEM context + $PSScriptRoot resolved + pnputil staged drivers + exit 0.
FAIL: ANY of the above missing. Record the exact log excerpt showing the failure in Notes; open a defect ticket in "Issues Found".
Resume signal: reply "RTVAL-02 PASS" + log path, or "RTVAL-02 FAIL" with failing observation and excerpt.
</action>
<verify>
<automated>test -f .planning/phases/10-real-world-runtime-validation/evidence/rtval-02-install-log.txt && ! grep -A1 "RTVAL-02 — Install under SYSTEM" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md | grep -q "Status: PENDING"</automated>
</verify>
<done>RTVAL-02 status is PASS or FAIL in the report; install log committed; for PASS the log shows SYSTEM, pnputil success, $PSScriptRoot resolved, and exit 0.</done>
</task>
<task type="checkpoint:human-action" gate="blocking">
<name>Task 3: RTVAL-03 — Confirm Intune detection rule reports Installed</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md, .planning/phases/10-real-world-runtime-validation/evidence/rtval-03-detection.png, .planning/phases/10-real-world-runtime-validation/evidence/rtval-03-detect-manual.txt</files>
<action>
PAUSE for human technician.
Technician MUST:
1. In Intune portal: Apps → the test app → Device install status → test device row → confirm "Installation status" = **Installed** (this is driven by the generated detect script).
2. On the endpoint itself, confirm the printer is present:
- PowerShell (user context OK for verification only): `Get-Printer | Where-Object Name -eq "<printer name>"` returns a row.
- OR open Settings → Bluetooth & devices → Printers & scanners → confirm printer visible.
3. Run the generated detect script manually once to cross-check:
- Open admin PowerShell and execute the detect script; confirm exit code 0.
4. Screenshot Intune "Installed" status as `evidence/rtval-03-detection.png`.
5. Save detect script manual run transcript as `evidence/rtval-03-detect-manual.txt`.
6. Update RUNTIME-VALIDATION.md section "RTVAL-03":
- Status: PASS if Intune shows Installed AND manual detect script exits 0 AND printer is visible. Otherwise FAIL.
PASS: Intune "Installed" + printer visible + detect script exit 0.
FAIL: Intune shows "Not detected", manual detect non-zero, or printer missing. Record in Notes.
Resume signal: reply "RTVAL-03 PASS" + evidence paths, or "RTVAL-03 FAIL" with observed status.
</action>
<verify>
<automated>test -f .planning/phases/10-real-world-runtime-validation/evidence/rtval-03-detection.png && ! grep -A1 "RTVAL-03 — Detection rule" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md | grep -q "Status: PENDING"</automated>
</verify>
<done>RTVAL-03 status is PASS or FAIL; detection screenshot and manual-run transcript committed.</done>
</task>
<task type="checkpoint:human-action" gate="blocking">
<name>Task 4: RTVAL-04 — Intune uninstall removes printer cleanly under SYSTEM</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md, .planning/phases/10-real-world-runtime-validation/evidence/rtval-04-uninstall-log.txt, .planning/phases/10-real-world-runtime-validation/evidence/rtval-04-uninstall-status.png</files>
<action>
PAUSE for human technician.
Technician MUST:
1. In Intune portal: change the app assignment for the test device from Required to Uninstall (or remove assignment and add as Uninstall).
2. Force Intune sync on the device again. Wait for the uninstall to run.
3. Extract the uninstall run from `IntuneManagementExtension.log` and save as `evidence/rtval-04-uninstall-log.txt`.
4. OBSERVE in the log:
a. Uninstall script ran under `NT AUTHORITY\SYSTEM`.
b. Final script exit code 0.
5. On the endpoint, confirm the printer is GONE:
- `Get-Printer | Where-Object Name -eq "<printer name>"` returns nothing.
- OR Settings → Printers & scanners → printer no longer listed.
6. Confirm Intune portal shows "Not installed" for the test device.
7. Screenshot as `evidence/rtval-04-uninstall-status.png`.
8. Update RUNTIME-VALIDATION.md section "RTVAL-04":
- Status: PASS if SYSTEM context + exit 0 + printer gone + Intune "Not installed". Otherwise FAIL with which check failed.
PASS: clean removal under SYSTEM, zero leftover printer artifacts.
FAIL: printer still present, non-zero exit, or tenant shows install failure. Record in Notes.
Resume signal: reply "RTVAL-04 PASS" + evidence paths, or "RTVAL-04 FAIL" with failing check.
</action>
<verify>
<automated>test -f .planning/phases/10-real-world-runtime-validation/evidence/rtval-04-uninstall-log.txt && ! grep -A1 "RTVAL-04 — Uninstall under SYSTEM" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md | grep -q "Status: PENDING"</automated>
</verify>
<done>RTVAL-04 status is PASS or FAIL; uninstall log and status screenshot committed; printer confirmed removed on the device for PASS.</done>
</task>
</tasks>
<verification>
- evidence/rtval-01-tenant-upload.png exists
- evidence/rtval-02-install-log.txt exists and shows SYSTEM + pnputil + $PSScriptRoot
- evidence/rtval-03-detection.png exists and shows Installed
- evidence/rtval-04-uninstall-log.txt exists and shows clean removal
- RUNTIME-VALIDATION.md status lines for RTVAL-01..04 are PASS or FAIL (no PENDING)
</verification>
<success_criteria>
All four RTVAL checkpoint tasks have signaled a final status (PASS or FAIL) with evidence files committed to evidence/. Any FAIL has a corresponding "Issues Found" entry in the report with defect ticket or deferral rationale.
</success_criteria>
<output>
After completion, create `.planning/phases/10-real-world-runtime-validation/10-02-live-intune-runtime-validation-SUMMARY.md`
</output>
@@ -0,0 +1,182 @@
---
phase: 10-real-world-runtime-validation
plan: 02
subsystem: testing
tags: [intune, intunewin, runtime-validation, rtval, pnputil, system-context, attestation-only]
# Dependency graph
requires:
- phase: 10-real-world-runtime-validation
provides: "Plan 10-01 — locked .intunewin package (Copieur_2eme), tenant + device metadata, RUNTIME-VALIDATION.md scaffold"
provides:
- "RTVAL-01 PASS (artifact-backed) — real Intune tenant accepts generated .intunewin without format errors"
- "RTVAL-02 PASS (attestation-only) — install under SYSTEM context confirmed verbally by technician"
- "RTVAL-03 PASS (attestation-only) — Intune detection rule reports Installed confirmed verbally"
- "RTVAL-04 PASS (attestation-only) — uninstall under SYSTEM confirmed verbally; printer removed cleanly"
- "Two structural .intunewin generator defects discovered and fixed during RTVAL-01 (ISSUE-01): HMAC scope + Detection.xml format"
affects:
- 10-03-sign-off-and-phase-closure
- 11-real-world-rollout-and-feedback
# Tech tracking
tech-stack:
added: []
patterns:
- "Attestation-only PASS is permitted but MUST be flagged in RUNTIME-VALIDATION.md Notes and called out in plan SUMMARY for downstream verifiers"
- "Debug-then-resume flow: RTVAL failure → /gsd:debug on generator → commit fixes → re-test same check against fresh build"
key-files:
created: []
modified:
- .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
key-decisions:
- "RTVAL-01 initial FAIL (blocker) debugged to root cause: two .intunewin generator defects (HMAC over ciphertext-only instead of IV+ciphertext; Detection.xml not matching IntuneWinAppUtil.exe reference format). Fixed in commits 74535ea and 7716246; re-test PASS."
- "RTVAL-02, RTVAL-03, and RTVAL-04 accepted as attestation-only PASSes per explicit, repeated user approval. User was warned on RTVAL-03 about consecutive attestation-only checks weakening audit trail, and warned AGAIN on RTVAL-04, and chose to proceed both times."
- "Phase 10 runtime audit trail is now structurally weakened: 3 of 4 RTVAL runtime checks are verbal-only. Plan 10-03 sign-off must either accept this or force re-run of RTVAL-02/03/04 with full artifact capture."
patterns-established:
- "SUMMARY.md must surface attestation-only debt prominently so the wave-3 verifier and phase verifier cannot miss it"
- ".intunewin generator defects discovered during real-tenant testing are captured as ISSUE-## entries in RUNTIME-VALIDATION.md with resolution commit refs"
requirements-completed:
- RTVAL-01
- RTVAL-02
- RTVAL-03
- RTVAL-04
# Metrics
duration: "~1 day (wall-clock, spanning debug + re-test + three attestation checkpoints)"
completed: 2026-04-13
---
# Phase 10 Plan 02: Live Intune Runtime Validation Summary
**All four RTVAL runtime checks recorded against live tenant rubis.fr on ARES-5CG5220YTM — RTVAL-01 artifact-backed PASS after fixing two .intunewin generator defects; RTVAL-02/03/04 attestation-only PASSes (user approved despite repeated warnings about weakened audit trail).**
## Performance
- **Duration:** ~1 day wall-clock (includes debug cycle for ISSUE-01 + three human-action checkpoints)
- **Started:** 2026-04-13
- **Completed:** 2026-04-13
- **Tasks:** 4 (all `checkpoint:human-action`)
- **Files modified:** 1 (RUNTIME-VALIDATION.md — incrementally across 4 task commits)
## Accomplishments
- **RTVAL-01 (artifact-backed):** Real Intune tenant (rubis.fr) accepted the generated `Copieur_2eme.intunewin` package without format errors on re-test against the fixed build. Two screenshots committed (`rtval-01-tenant-upload.png`, `rtval-01-app-assigned.png`).
- **RTVAL-01 debug cycle:** Initial run FAILED with empty wizard fields and greyed-out OK button. Root cause debugged to two structural defects in the ImpTune .intunewin generator:
1. HMAC was computed over ciphertext only instead of IV+ciphertext (commit `74535ea`)
2. Detection.xml did not match the IntuneWinAppUtil.exe reference format (commit `7716246`)
Both fixes landed; RTVAL-01 re-tested PASS.
- **RTVAL-02 (attestation-only):** Technician verbally confirmed install script ran under SYSTEM on ARES-5CG5220YTM, driver deployed, Intune portal showed Installed. No log excerpt or screenshot captured.
- **RTVAL-03 (attestation-only, 2nd consecutive):** Technician verbally confirmed Intune reports Installed and detection rule passes. No screenshot or detect transcript captured. User warned about weakened audit trail and approved.
- **RTVAL-04 (attestation-only, 3rd consecutive):** Technician verbally confirmed Intune uninstall ran under SYSTEM, exited cleanly, printer removed from ARES-5CG5220YTM. No uninstall log or portal screenshot captured. User warned a second time about the cumulative audit-trail damage and approved.
## Task Commits
Each task was committed atomically:
1. **Task 1: RTVAL-01 — Tenant ingestion (FAIL → debug → PASS)**
- `403a0a5` (fix): initial FAIL recorded — Intune cannot parse generated .intunewin
- `46cfde0` (chore): STATE.md updated — plan blocked at RTVAL-01 FAIL
- `44a4f2c` (docs): debug session recorded
- `7716246` (fix): Detection.xml aligned with IntuneWinAppUtil.exe reference format
- `74535ea` (fix): HMAC computed over IV+ciphertext
- `35d4edc` (docs): debug session updated — HMAC scope bug identified
- `00b709d` (docs): debug session resolved
- `cf3b86a` (docs): RTVAL-01 PASS recorded on re-test against fixed build
- `bcd12a3` (docs): BLOCKER-01 cleared, plan resumed at Task 2
2. **Task 2: RTVAL-02 — Install under SYSTEM**`870158b` (docs) — attestation-only PASS
- `d86199f` (docs): STATE.md advanced to Task 3
3. **Task 3: RTVAL-03 — Detection rule reports Installed**`475d593` (docs) — attestation-only PASS (2nd consecutive)
- `b0be3d3` (docs): STATE.md advanced to Task 4
4. **Task 4: RTVAL-04 — Uninstall under SYSTEM**`2c912ca` (docs) — attestation-only PASS (3rd consecutive)
**Plan metadata:** (this commit) `docs(10-02): complete live-intune-runtime-validation plan`
## Files Created/Modified
- `.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md` — Status line updated to reflect all four RTVAL checks recorded; RTVAL-01 through RTVAL-04 sections populated with results, evidence (or attestation disclaimer), and Notes.
- `imptune/generators/intunewin_builder.py` (via debug cycle, commits `74535ea`, `7716246`) — HMAC scope fix + Detection.xml format alignment; unblocked RTVAL-01.
## Decisions Made
- **Debug-before-abandon:** RTVAL-01 initial FAIL was not deferred to a gap-closure plan — instead, the .intunewin generator was debugged inline (ISSUE-01) because the root cause was clearly in-scope for the package under test. Two commits fixed it and the same check was re-run on the fresh build.
- **Attestation-only accepted for RTVAL-02/03/04:** User explicitly, repeatedly chose verbal attestation over artifact capture. The executor flagged each successive choice (warning on RTVAL-03, second warning on RTVAL-04) and proceeded per user instruction. This is documented here, in RUNTIME-VALIDATION.md Notes for each check, and in STATE.md decisions log — it MUST be visible to plan 10-03 (sign-off) and the phase verifier.
## Deviations from Plan
### Rule 1 (Bug) — `.intunewin` generator defects discovered during RTVAL-01
**1. [Rule 1 - Bug] HMAC computed over ciphertext only instead of IV+ciphertext**
- **Found during:** Task 1 (RTVAL-01) — Intune wizard rejected the package, debugged to the root cause
- **Issue:** `intunewin_builder.py` computed HMAC over the ciphertext alone, but the Intune Win32 package format requires HMAC over `IV || ciphertext`. Intune's package parser rejected the file silently (empty wizard fields).
- **Fix:** Updated HMAC computation to include the IV prefix.
- **Files modified:** `imptune/generators/intunewin_builder.py`
- **Committed in:** `74535ea` (standalone fix commit, not inside a task commit)
**2. [Rule 1 - Bug] Detection.xml did not match IntuneWinAppUtil.exe reference format**
- **Found during:** Task 1 (RTVAL-01) — same debug cycle
- **Issue:** Embedded Detection.xml used a slightly different schema/element ordering than IntuneWinAppUtil.exe emits. Intune parses the reference format strictly, so the wizard could not populate metadata fields.
- **Fix:** Aligned Detection.xml generation with the IntuneWinAppUtil.exe reference output byte-for-byte.
- **Files modified:** `imptune/generators/intunewin_builder.py`
- **Committed in:** `7716246` (standalone fix commit, not inside a task commit)
---
**Total deviations:** 2 auto-fixed (both Rule 1 bugs in the generator under test)
**Impact on plan:** Both fixes were essential and in-scope — the whole point of RTVAL-01 is to discover exactly this class of defect. Plan was paused via debug session and resumed without re-planning.
## ⚠️ CRITICAL: Attestation-Only Audit Trail Damage
**Read this before signing off Phase 10 in plan 10-03.**
Three of the four RTVAL runtime checks in this plan are **attestation-only** (verbal confirmation from the technician, no log excerpt, no screenshot, no transcript):
| Check | Status | Artifact-backed? | Warning at time of acceptance |
| -------- | ----------------------------- | ---------------- | -------------------------------------------------------------------------- |
| RTVAL-01 | PASS | **Yes** | — |
| RTVAL-02 | PASS (attestation-only) | No | User approved "Pass without evidence" |
| RTVAL-03 | PASS (attestation-only, 2nd) | No | User warned about weakened audit trail, approved anyway |
| RTVAL-04 | PASS (attestation-only, 3rd) | No | User warned a **second time** about weakened audit trail, approved anyway |
**What this means:**
- Only tenant ingestion (RTVAL-01) is independently verifiable from committed evidence.
- The entire **runtime half** of Phase 10 (install under SYSTEM, detection, uninstall under SYSTEM) rests on the technician's verbal report. None of the required observations (`$PSScriptRoot` resolved, `pnputil /add-driver` success line, `NT AUTHORITY\SYSTEM` banner, exit code 0, `Get-Printer` shows/hides the printer, Intune portal Installed/Not installed) were captured to disk.
- For any future regression, incident investigation, or compliance audit, RTVAL-02/03/04 should be treated as **soft PASSes** and re-run with full artifact capture before the phase is considered closed.
**Recommended action for plan 10-03 (sign-off):**
1. Either: re-run RTVAL-02/03/04 with full artifact capture before signing off, OR
2. Explicitly accept the attestation-only audit trail in the sign-off block and record the technician's name + the compensating control (e.g., known test device, known tenant, same session).
The sign-off checklist in RUNTIME-VALIDATION.md should **not** be ticked silently — the reviewer needs to actively acknowledge the attestation-only nature of 3 of 4 checks.
## Issues Encountered
- **ISSUE-01 (resolved):** Initial RTVAL-01 FAIL — Intune wizard could not parse `Copieur_2eme.intunewin`. Root-caused during a /gsd:debug session to two `.intunewin` generator defects (HMAC scope + Detection.xml format). Both fixed in `74535ea` and `7716246`. RTVAL-01 re-tested PASS against the fixed build. See RUNTIME-VALIDATION.md "Issues Found → ISSUE-01" for the full trail.
- **Attestation debt:** Three consecutive runtime checks went attestation-only. Not an "issue" in the traditional sense (user chose it), but documented here, in STATE.md decisions, and in RUNTIME-VALIDATION.md Notes as a **persistent risk** that plan 10-03 must explicitly address.
## User Setup Required
None — all configuration was performed by the technician during the four human-action checkpoints.
## Next Phase Readiness
- **Plan 10-03 (sign-off):** Ready to start. RTVAL-01..04 all have recorded statuses (no PENDING lines). Sign-off agent MUST:
1. Read the "CRITICAL: Attestation-Only Audit Trail Damage" section of this summary
2. Decide whether to re-run RTVAL-02/03/04 with evidence OR explicitly accept the attestation-only runtime half
3. Populate the "Signed off by" and "Signed off date" fields in RUNTIME-VALIDATION.md
- **Phase 11 (real-world rollout) readiness:** Conditionally ready. The .intunewin generator is known-good (fixed in commits 74535ea + 7716246, confirmed by RTVAL-01). The runtime behavior (install/detect/uninstall under SYSTEM) is attested but not artifact-proven — Phase 11 rollout will be the real stress test.
- **Blockers:** None for plan 10-03. BLOCKER-01 (ISSUE-01) resolved.
## Self-Check: PASSED
All referenced commits verified to exist in git history (`403a0a5`, `7716246`, `74535ea`, `cf3b86a`, `870158b`, `475d593`, `2c912ca`). SUMMARY.md written to expected path.
---
*Phase: 10-real-world-runtime-validation*
*Plan: 02-live-intune-runtime-validation*
*Completed: 2026-04-13*
@@ -0,0 +1,144 @@
---
phase: 10-real-world-runtime-validation
plan: 03
type: execute
wave: 3
depends_on:
- 10-01
- 10-02
files_modified:
- .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
- .planning/REQUIREMENTS.md
- .planning/ROADMAP.md
autonomous: false
requirements:
- RTVAL-05
must_haves:
truths:
- "A reviewer can open RUNTIME-VALIDATION.md and read a complete signed-off report with tenant, device, OS build, driver vendor, screenshots/logs, and any issues"
- "Every RTVAL-01..04 result is PASS/FAIL with cited evidence (no PENDING)"
- "Any FAIL has either a v1.1 defect ticket or a written deferral rationale"
- "REQUIREMENTS.md and ROADMAP.md mark Phase 10 complete"
artifacts:
- path: ".planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md"
provides: "Final signed-off runtime validation report"
contains: "Signed off by:"
key_links:
- from: "RUNTIME-VALIDATION.md Sign-off checklist"
to: "evidence/*.png and evidence/*.txt files from plan 10-02"
via: "markdown relative links"
pattern: "\\]\\(evidence/.*\\)"
---
<objective>
Convert the draft RUNTIME-VALIDATION.md into a signed-off, reviewable report and update tracking docs so Phase 10 can be marked complete.
Purpose: RTVAL-05 requires a reviewer-grade report, not just scattered evidence files. This plan closes the loop.
Output: A final RUNTIME-VALIDATION.md committed to the repo plus REQUIREMENTS.md/ROADMAP.md ticks.
</objective>
<execution_context>
@C:/Users/SebastienQUEROL/.claude/get-shit-done/workflows/execute-plan.md
@C:/Users/SebastienQUEROL/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
@.planning/REQUIREMENTS.md
@.planning/ROADMAP.md
</context>
<tasks>
<task type="auto">
<name>Task 1: Finalize RUNTIME-VALIDATION.md report body</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md</files>
<action>
Read the current RUNTIME-VALIDATION.md (populated by plans 10-01 and 10-02). Then:
1. For each RTVAL-01..04 section, ensure:
- Status line is PASS or FAIL (not PENDING).
- Evidence paths use markdown relative links like `[screenshot](evidence/rtval-01-tenant-upload.png)`.
- Notes field has at least one sentence (either "no anomalies" or description of issues).
2. For each FAIL, confirm there is a corresponding entry under "## Issues Found" formatted as:
```
- **[RTVAL-0X]** <short description> — <defect ticket ID or "Deferred to v1.2: <rationale>">
```
If a FAIL has no Issues Found entry yet, add one noting "TODO: triage — see Task 2 checkpoint".
3. Update the "Status" field at the top from `DRAFT (in progress)` to `READY FOR SIGN-OFF`.
Do NOT fabricate results. If any section is still PENDING because plan 10-02 did not actually run, STOP and return an error to the executor — 10-03 must not be run before 10-02 is complete.
</action>
<verify>
<automated>! grep -q "Status: PENDING" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md && grep -q "READY FOR SIGN-OFF\|SIGNED OFF" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md</automated>
</verify>
<done>Report has zero PENDING status lines, all evidence links are markdown-valid, Issues Found is consistent with FAILs, and top-level status is READY FOR SIGN-OFF.</done>
</task>
<task type="checkpoint:human-verify" gate="blocking">
<name>Task 2: Human review and sign-off</name>
<files>.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md</files>
<action>
PAUSE for human reviewer. The reviewer (user acting as technician/owner) MUST:
1. Open `.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md`.
2. Walk every RTVAL-01..04 section and click every evidence link. Confirm screenshots open and log excerpts are readable.
3. For every FAIL, confirm the "Issues Found" entry is present AND has either:
- a v1.1 defect ticket ID (open a ticket if needed), OR
- an explicit deferral to v1.2 with written rationale.
4. If satisfied, edit the report's Sign-off block:
- Change `Signed off by:` to the reviewer's name.
- Change `Signed off date:` to today's date.
- Tick the three sign-off checkboxes.
- Change the top-level Status from `READY FOR SIGN-OFF` to `SIGNED OFF`.
5. If NOT satisfied, describe exactly which section is lacking and which additional evidence is required. The plan loops back to plan 10-02 (or a gap-closure plan).
Resume signal: reply "signed off" (after editing the file) or describe missing evidence.
</action>
<verify>
<automated>grep -q "^\*\*Status:\*\* SIGNED OFF" .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md</automated>
</verify>
<done>Report top-level status = SIGNED OFF; Signed off by/date populated; all three sign-off checkboxes ticked.</done>
</task>
<task type="auto">
<name>Task 3: Tick RTVAL requirements and mark Phase 10 complete in tracking docs</name>
<files>.planning/REQUIREMENTS.md, .planning/ROADMAP.md</files>
<action>
Only run after Task 2 signals "signed off". Verify by grepping the report for `Status: SIGNED OFF` — if absent, STOP.
Then:
1. In `.planning/REQUIREMENTS.md`:
- Change `- [ ] **RTVAL-01**` through `- [ ] **RTVAL-05**` to `- [x]`.
- In the Traceability table, change status for RTVAL-01..05 from `Pending` to `Complete`. If any RTVAL ended FAIL and was deferred, mark it `Deferred (v1.2)` instead and add a footnote link to the report's Issues Found entry.
2. In `.planning/ROADMAP.md`:
- Change `- [ ] **Phase 10: Real-World Runtime Validation**` to `- [x]` and append `(completed <today>)`.
- In the Progress table row for Phase 10, set `Plans Complete` to `3/3`, `Status` to `Complete`, and `Completed` to today's date.
Do not touch unrelated rows. Preserve existing formatting exactly.
</action>
<verify>
<automated>grep -q "\[x\] \*\*RTVAL-05\*\*" .planning/REQUIREMENTS.md && grep -q "\[x\] \*\*Phase 10" .planning/ROADMAP.md</automated>
</verify>
<done>REQUIREMENTS.md has RTVAL-01..05 ticked (or explicitly deferred); ROADMAP.md Phase 10 ticked with completion date and 3/3 plans.</done>
</task>
</tasks>
<verification>
- RUNTIME-VALIDATION.md top-level status = SIGNED OFF
- Sign-off name + date populated
- Zero PENDING status lines
- REQUIREMENTS.md RTVAL-01..05 ticked
- ROADMAP.md Phase 10 ticked
</verification>
<success_criteria>
A fresh reviewer can clone the repo, open RUNTIME-VALIDATION.md, follow every evidence link, and reach the same PASS/FAIL verdicts. Tracking docs reflect phase completion.
</success_criteria>
<output>
After completion, create `.planning/phases/10-real-world-runtime-validation/10-03-report-signoff-SUMMARY.md`
</output>
@@ -0,0 +1,87 @@
---
phase: 10-real-world-runtime-validation
plan: 03
subsystem: validation/reporting
tags: [runtime-validation, sign-off, phase-closure, rtval]
requirements: [RTVAL-05]
dependency_graph:
requires: [10-01, 10-02]
provides: [signed-off RUNTIME-VALIDATION.md, Phase 10 closure]
affects: [.planning/REQUIREMENTS.md, .planning/ROADMAP.md, .planning/STATE.md]
tech_stack:
added: []
patterns: [human sign-off checkpoint, attestation gap acknowledgement]
key_files:
created:
- .planning/phases/10-real-world-runtime-validation/10-03-report-signoff-SUMMARY.md
modified:
- .planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md
- .planning/ROADMAP.md
- .planning/REQUIREMENTS.md
decisions:
- Phase 10 closed with RTVAL-02/03/04 as attestation-only PASSes (only RTVAL-01 artifact-backed); reviewer explicitly acknowledged the audit-trail gap at sign-off rather than blocking closure for a re-run.
metrics:
completed_date: 2026-04-13
---
# Phase 10 Plan 03: Report Sign-off Summary
Finalized RUNTIME-VALIDATION.md, obtained human sign-off with explicit attestation-gap acknowledgement, and flipped Phase 10 tracking docs to complete.
## What Was Done
### Task 1 — Finalize RUNTIME-VALIDATION.md report body
Completed in the 10-02 → 10-03 handoff window: all RTVAL-01..04 sections carry concrete PASS/FAIL verdicts, evidence links, and notes; top-level Status was flipped to READY FOR SIGN-OFF by the prior agent.
### Task 2 — Human review and sign-off (checkpoint)
Reviewer Sébastien QUEROL read the report, acknowledged the attestation-only audit-trail gap for RTVAL-02/03/04, and signed off on 2026-04-13:
- Top-level **Status:** flipped to `SIGNED OFF`
- **Signed off by:** Sébastien QUEROL
- **Signed off date:** 2026-04-13
- All three sign-off checkboxes ticked `[x]`
- Reviewer explicitly accepted that only RTVAL-01 is artifact-backed; RTVAL-02/03/04 rest on technician verbal attestation. Compensating controls listed in the report (known device, known tenant, single session, known-good generator) stand in for missing log/screenshot evidence.
Commit: `5685fd9``docs(phase-10): human sign-off on RUNTIME-VALIDATION.md (attestation gap acknowledged)`
### Task 3 — Tick requirements, mark Phase 10 complete
- `.planning/REQUIREMENTS.md` already had RTVAL-01..05 ticked `[x]` and the Traceability table showing `Complete` for each (landed in the 10-02 completion commit `206648c`). Idempotent confirmation only — no edits needed.
- `.planning/ROADMAP.md`:
- Phase 10 entry flipped from `[ ]` to `[x]` with `(completed 2026-04-13)` appended.
- Progress table row for Phase 10 set to `v1.1 | 3/3 | Complete | 2026-04-13`.
## Verification
- `grep "^\*\*Status:\*\* SIGNED OFF" RUNTIME-VALIDATION.md` → hit
- `grep "Signed off by:\*\* Sébastien QUEROL" RUNTIME-VALIDATION.md` → hit
- Zero `Status: PENDING` lines in the report
- `grep "\[x\] \*\*RTVAL-05\*\*" REQUIREMENTS.md` → hit
- `grep "\[x\] \*\*Phase 10" ROADMAP.md` → hit
- ROADMAP.md Progress row for Phase 10 shows `3/3 | Complete | 2026-04-13`
All verification checks from the plan pass.
## Deviations from Plan
None. Task 3 REQUIREMENTS.md edits were already present from plan 10-02's completion commit, making that step a no-op confirmation rather than a mutation. No deviation rules (1-4) triggered.
## Phase 10 Closure Note — Attestation-Only Audit Trail
Phase 10 closes with a structurally weakened runtime audit trail that downstream consumers must be aware of:
- **RTVAL-01** (tenant ingestion): PASS, artifact-backed (screenshots + exact .intunewin package committed under `evidence/`), re-tested on the fixed build after ISSUE-01 was resolved (commits `74535ea` + `7716246`).
- **RTVAL-02** (install under SYSTEM): PASS, **attestation-only** — no `IntuneManagementExtension.log` excerpt, no portal screenshot.
- **RTVAL-03** (detection rule): PASS, **attestation-only, 2nd consecutive** — no portal screenshot, no manual detect transcript.
- **RTVAL-04** (uninstall under SYSTEM): PASS, **attestation-only, 3rd consecutive** — no uninstall log, no portal screenshot.
RTVAL-02/03/04 together form an attestation-only runtime half for the phase. The user was warned twice during plan 10-02 (on RTVAL-03 and again on RTVAL-04) that consecutive attestation-only checks damage the audit trail, and chose to proceed both times. At sign-off the user again explicitly acknowledged the gap and accepted Phase 10 closure on that basis.
**Implication for future work:** If a regression, incident, or customer escalation touches SYSTEM-context install, detection, or uninstall, RTVAL-02/03/04 must be treated as soft PASSes — the "it worked once" claim for this build cannot be independently re-derived from evidence files and must be re-validated with full artifact capture. Phase 11 rollout proceeds at the reviewer's risk.
## Self-Check: PASSED
- FOUND: `.planning/phases/10-real-world-runtime-validation/RUNTIME-VALIDATION.md` (SIGNED OFF)
- FOUND: `.planning/phases/10-real-world-runtime-validation/10-03-report-signoff-SUMMARY.md`
- FOUND: REQUIREMENTS.md RTVAL-01..05 ticked + Traceability Complete
- FOUND: ROADMAP.md Phase 10 ticked with 3/3 Complete 2026-04-13
- FOUND commit: `5685fd9` (Task 2 sign-off)
@@ -0,0 +1,130 @@
---
phase: 10-real-world-runtime-validation
verified: 2026-04-13T00:00:00Z
status: gaps_found
score: 2/5 must-haves artifact-verified (3/5 if attestation-only PASSes counted)
gaps:
- truth: "Install script verified under SYSTEM on a real endpoint with pnputil + $PSScriptRoot confirmed in device log (RTVAL-02)"
status: partial
reason: "ROADMAP.md Success Criterion #2 literally requires 'verified in the device log'. No IntuneManagementExtension.log excerpt or portal screenshot was captured; PASS rests entirely on technician verbal attestation. User was warned and explicitly accepted the gap at sign-off, but goal-as-written is not met."
artifacts:
- path: ".planning/phases/10-real-world-runtime-validation/evidence/"
issue: "Missing rtval-02 IntuneManagementExtension.log excerpt and portal 'Installed' screenshot"
missing:
- "evidence/rtval-02-install-log.txt (SYSTEM context banner + $PSScriptRoot resolution + pnputil /add-driver success + exit 0)"
- "evidence/rtval-02-install-status.png (Intune portal device install status = Installed)"
- truth: "Detection rule reports Installed with evidence captured (RTVAL-03)"
status: partial
reason: "Second consecutive attestation-only check. No portal screenshot, no manual detect transcript. Goal sentence 'with evidence recorded' is not met; PASS is verbal only."
artifacts:
- path: ".planning/phases/10-real-world-runtime-validation/evidence/"
issue: "Missing rtval-03 detection evidence"
missing:
- "evidence/rtval-03-detection.png (Intune portal Installed status)"
- "evidence/rtval-03-detect-manual.txt (manual detect script run with exit 0 and Get-Printer output)"
- truth: "Uninstall under SYSTEM cleanly removes printer with evidence (RTVAL-04)"
status: partial
reason: "Third consecutive attestation-only check. No uninstall log or portal screenshot. User was warned twice and accepted; goal-as-written is not met."
artifacts:
- path: ".planning/phases/10-real-world-runtime-validation/evidence/"
issue: "Missing rtval-04 uninstall evidence"
missing:
- "evidence/rtval-04-uninstall-log.txt (IntuneManagementExtension.log SYSTEM context + exit 0)"
- "evidence/rtval-04-uninstall-status.png (Intune portal 'Not installed' after uninstall)"
---
# Phase 10: Real-World Runtime Validation — Verification Report
**Phase Goal (ROADMAP.md):** Generated `.intunewin`, install, detect, and uninstall artifacts are proven to work end-to-end on a real Intune tenant against a real Windows endpoint, **with evidence recorded**.
**Verified:** 2026-04-13
**Status:** gaps_found (signed-off, but goal-as-written not fully artifact-backed)
**Re-verification:** No — initial verification
## Goal Achievement
### Observable Truths (derived from ROADMAP.md Success Criteria)
| # | Truth (RTVAL-ID) | Status | Evidence |
|---|------------------|--------|----------|
| 1 | `.intunewin` uploaded to live Intune tenant and accepted without format errors, with tenant ingestion confirmation captured as evidence (RTVAL-01) | PASS (artifact-backed) | `evidence/rtval-01-tenant-upload.png` (9.3 KB), `evidence/rtval-01-app-assigned.png` (21 KB), `evidence/Copieur_2eme.intunewin` (33 MB, SHA256 8818124a...3ddc). Re-tested on fixed build (commits 74535ea + 7716246) against tenant rubis.fr. |
| 2 | Install script succeeds under SYSTEM with `pnputil` driver staging and `$PSScriptRoot` path resolution verified in the device log (RTVAL-02) | FAIL — attestation-only | No `IntuneManagementExtension.log` excerpt captured. No portal screenshot. Technician verbal attestation only. SC#2 explicitly requires "verified in the device log". |
| 3 | Detection rule reports Installed on real endpoint after install, evidence captured from Intune portal (RTVAL-03) | FAIL — attestation-only | No portal screenshot, no manual detect transcript. Second consecutive attestation-only check. |
| 4 | Uninstall from Intune removes printer cleanly under SYSTEM (RTVAL-04) | FAIL — attestation-only | No uninstall log, no portal screenshot. Third consecutive attestation-only check. |
| 5 | `RUNTIME-VALIDATION.md` is signed-off with tenant/device/OS build/vendor/screenshots/logs/issues (RTVAL-05) | PASS | `RUNTIME-VALIDATION.md` Status = SIGNED OFF, signed by Sébastien QUEROL on 2026-04-13, tenant (rubis.fr), device (ARES-5CG5220YTM), OS build (26200.7171), vendor (Ricoh PCL6 Universal), ISSUE-01 documented + resolved, three sign-off checkboxes ticked. Logs subsection of the report is empty (no rtval-02/03/04 logs linked) but the document structurally exists and is signed. |
**Score:** 2/5 truths artifact-verified; 3/5 attestation-only; 5/5 PASS if attestation is accepted (as reviewer did at sign-off).
### Required Artifacts
| Artifact | Expected | Status | Details |
|----------|----------|--------|---------|
| `RUNTIME-VALIDATION.md` | Signed-off report with full metadata | PASS | Exists, SIGNED OFF, reviewer = Sébastien QUEROL, 2026-04-13 |
| `evidence/rtval-01-tenant-upload.png` | Tenant upload wizard screenshot | PASS | 9,389 bytes, committed |
| `evidence/rtval-01-app-assigned.png` | App assignment screenshot | PASS | 21,618 bytes, committed |
| `evidence/Copieur_2eme.intunewin` | Exact package under test | PASS | 33,913,101 bytes, SHA256 recorded |
| `evidence/rtval-02-install-log.txt` | Device log excerpt (SYSTEM + pnputil + $PSScriptRoot) | MISSING | Not captured; SC#2 literal requirement unmet |
| `evidence/rtval-02-install-status.png` | Intune portal Installed status | MISSING | Not captured |
| `evidence/rtval-03-detection.png` | Portal Installed screenshot | MISSING | Not captured |
| `evidence/rtval-03-detect-manual.txt` | Manual detect transcript | MISSING | Not captured |
| `evidence/rtval-04-uninstall-log.txt` | Uninstall log excerpt | MISSING | Not captured |
| `evidence/rtval-04-uninstall-status.png` | Portal Not-installed screenshot | MISSING | Not captured |
### Key Link Verification
| From | To | Via | Status | Details |
|------|----|----|--------|---------|
| Generator commit 00b709d | Uploaded .intunewin | SHA256 match in report | WIRED | SHA256 8818124a...3ddc recorded, matches committed artifact |
| ISSUE-01 fix | Re-test PASS | Commits 74535ea + 7716246 referenced | WIRED | Both commits present in git log |
| RUNTIME-VALIDATION.md sign-off | RTVAL-02/03/04 attestation acknowledgement | Sign-off checklist item #1 | WIRED | Checkbox #1 explicitly references "attestation-only nature... explicitly read and acknowledged" |
### Requirements Coverage
| Requirement | Source Plan | Description | Status | Evidence |
|-------------|-------------|-------------|--------|----------|
| RTVAL-01 | 10-02 | `.intunewin` accepted by live tenant, byte-level conformance | SATISFIED | Two screenshots + committed package; re-test on fixed build |
| RTVAL-02 | 10-02 | Install under SYSTEM, pnputil + $PSScriptRoot verified in device log | BLOCKED (accepted) | No log artifact; SC#2 "verified in the device log" literally unmet. Reviewer accepted at sign-off. |
| RTVAL-03 | 10-02 | Detect script returns expected exit code on real endpoint | BLOCKED (accepted) | No portal screenshot or detect transcript. Reviewer accepted. |
| RTVAL-04 | 10-02 | Uninstall removes printer cleanly under SYSTEM | BLOCKED (accepted) | No uninstall log or portal screenshot. Reviewer accepted. |
| RTVAL-05 | 10-01, 10-03 | Signed-off RUNTIME-VALIDATION.md with tenant/device/OS/vendor/screenshots/logs/issues | SATISFIED (structurally) | Document signed, all required metadata fields populated, ISSUE-01 documented + resolved. "Logs" portion of the metadata checklist is empty because rtval-02/03/04 never produced logs — this is consistent with the accepted gap. |
All 5 RTVAL IDs accounted for in plan frontmatters (RTVAL-01 in 10-02; RTVAL-02/03/04 in 10-02; RTVAL-05 in 10-01 + 10-03). No orphaned requirements. REQUIREMENTS.md traceability table shows all five Complete.
### Anti-Patterns Found
| File | Pattern | Severity | Impact |
|------|---------|----------|--------|
| `RUNTIME-VALIDATION.md` | "PASS (attested, no artifact)" on 3 consecutive runtime checks | Warning | Documented and acknowledged by reviewer; not silently hidden — but represents a structurally weakened audit trail for the runtime half of the phase. |
| `evidence/` | Missing log files and screenshots for RTVAL-02/03/04 | Warning | Known-and-accepted gap; not a hidden stub. |
No blockers: the gap is explicit, documented at multiple levels (report body, sign-off checklist, 10-03 SUMMARY closure note), and reviewer acknowledgement is recorded in writing.
### Human Verification Required
None — all decisions that needed human judgement already happened at sign-off. The reviewer:
1. Read the attestation-only warning in RUNTIME-VALIDATION.md
2. Read the equivalent warning in 10-02 SUMMARY
3. Was warned twice during plan 10-02 (RTVAL-03 and RTVAL-04)
4. Signed off with explicit acknowledgement and named compensating controls (known device, known tenant, single session, known-good generator)
## Gaps Summary
Phase 10 closes with **documented, reviewer-accepted evidentiary gaps** on RTVAL-02/03/04. The phase goal as literally written in ROADMAP.md — "proven to work end-to-end... **with evidence recorded**" plus Success Criterion #2's "verified in the device log" — is NOT met for the install/detect/uninstall half of the runtime checks. Only RTVAL-01 (tenant ingestion) is artifact-backed. RTVAL-05 (report) is structurally satisfied and signed.
This verifier records the gap as **gaps_found** rather than **passed** for audit-trail integrity: the goal sentence demands evidence, the reviewer chose to accept attestation instead, and that choice must remain visible to any future consumer (regression, incident response, customer escalation, Phase 11 rollout decisions). The gap is NOT a blocker for Phase 11 — the reviewer's sign-off is valid and recorded — but the verification report must not paper over the evidentiary debt.
**Recommended disposition:**
- Accept Phase 10 as closed (sign-off is valid and explicit).
- Treat RTVAL-02/03/04 as **soft PASSes** going forward.
- If any regression touches SYSTEM-context install, detect, or uninstall, re-run RTVAL-02/03/04 with full artifact capture before drawing conclusions from the "it worked once" attestation.
- Phase 11 rollout proceeds at the reviewer's accepted risk.
### Grouped root cause
All three failed truths share one root cause: **no log/screenshot capture discipline during the live 10-02 session**. A single gap-closure plan (`/gsd:plan-phase 10 --gaps`) could re-run all three checks in one endpoint session and produce the six missing evidence files in ~30 minutes. This is offered as an option, not a requirement — the reviewer has already accepted the current state.
---
_Verified: 2026-04-13_
_Verifier: Claude (gsd-verifier)_
@@ -0,0 +1,88 @@
# Runtime Validation Report — ImpTune v1.1
**Status:** SIGNED OFF — all four RTVAL runtime checks recorded (1 artifact-backed, 3 attestation-only): RTVAL-01 PASS (re-test 2026-04-13, artifact-backed); RTVAL-02 PASS (attestation-only, no artifact); RTVAL-03 PASS (attestation-only, no artifact — 2nd consecutive); RTVAL-04 PASS (attestation-only, no artifact — 3rd consecutive). Signed off 2026-04-13 with explicit acknowledgement of the attestation-only audit-trail gap.
**Started:** 2026-04-13
*(Sign-off name and date are populated in the Sign-off section at the bottom of this file.)*
## Tenant & Environment
| Field | Value |
|-------|-------|
| Intune tenant (domain) | rubis.fr |
| Test device hostname | ARES-5CG5220YTM |
| Windows OS build | 26200.7171 |
| Device is Intune-managed | yes |
| Driver vendor(s) tested | ricoh.zip
- PCL6 Driver for Universal Print |
| ImpTune commit SHA | 00b709d25aa20a3d5fbdabb728f6083d4f9f958c |
| Package file name | Copieur_2eme.intunewin |
| Package SHA256 | 8818124aa97ed3da24bf73a1f08f43065e6efea46f44a3abffc9983b097d3ddc |
| Generated at | 2026-04-13 |
## RTVAL-01 — Tenant ingestion
Status: PASS
Tested: 2026-04-13 (re-test against fixed build)
Evidence:
- [tenant-upload screenshot](evidence/rtval-01-tenant-upload.png) — Intune Win32 app wizard with parsed .intunewin metadata populated (name, platform, size, MAM enabled)
- [app-assigned screenshot](evidence/rtval-01-app-assigned.png) — app assignment screen showing the test device group
- [Copieur_2eme.intunewin package under test](evidence/Copieur_2eme.intunewin) — the exact artifact uploaded to tenant rubis.fr
Notes: Initial run on 2026-04-13 failed with empty wizard fields and greyed-out OK button (see ISSUE-01). Root cause was two structural defects in the .intunewin generator: (1) HMAC was computed over ciphertext only instead of IV+ciphertext, and (2) Detection.xml did not match the IntuneWinAppUtil.exe reference format. Both fixed in commits 74535ea and 7716246. Re-test on the fixed build against live tenant rubis.fr: package parses cleanly, all wizard fields populate, OK button enabled, assignment saved successfully.
Package tested: Copieur_2eme.intunewin (SHA256 8818124aa97ed3da24bf73a1f08f43065e6efea46f44a3abffc9983b097d3ddc, ImpTune commit 00b709d)
## RTVAL-02 — Install under SYSTEM (pnputil + $PSScriptRoot)
Status: PASS (attested, no artifact)
Tested: 2026-04-13
Evidence: Attested by technician — no log excerpt or portal screenshot captured. Audit trail weakened for this check; technician verbally confirmed install succeeded on ARES-5CG5220YTM, driver deployed, Intune portal showed Installed state.
Notes: ATTESTATION-ONLY — this check is NOT artifact-backed. No `IntuneManagementExtension.log` excerpt and no `Device install status = Installed` screenshot were captured on the test endpoint. The four required observations (SYSTEM context banner, `$PSScriptRoot` resolution, `pnputil /add-driver` success line, exit code 0) were NOT independently verified from logs; PASS rests entirely on the technician's verbal report ("checked on the device and everything is deployed correctly"). User explicitly approved proceeding without evidence on 2026-04-13. For any future regression or incident investigation, treat RTVAL-02 as a soft PASS and re-run with full log capture before signing off the phase.
## RTVAL-03 — Detection rule reports Installed
Status: PASS (attested, no artifact)
Tested: 2026-04-13
Evidence: Attested by technician — no portal screenshot or detect transcript captured. Audit trail weakened. Technician verbally confirmed Intune reports Installed and detection rule passes on ARES-5CG5220YTM.
Notes: ATTESTATION-ONLY — this check is NOT artifact-backed. This is the **second consecutive attestation-only check** (RTVAL-02 was also attestation-only), which further weakens the audit trail for plan 10-02. No `evidence/rtval-03-detection.png` (Intune portal "Installed" screenshot) and no `evidence/rtval-03-detect-manual.txt` (manual detect script run transcript with exit code 0) were captured. The three required observations (Intune portal status = Installed, manual detect script exit 0, printer visible via `Get-Printer`) were NOT independently verified; PASS rests entirely on the technician's verbal report. User was explicitly warned that a second consecutive attestation-only check weakens the audit trail and still chose to proceed without evidence on 2026-04-13. For any future regression, incident investigation, or phase sign-off, RTVAL-02 and RTVAL-03 should be treated as soft PASSes and re-run with full artifact capture before the phase is closed.
## RTVAL-04 — Uninstall under SYSTEM
Status: PASS (attested, no artifact)
Tested: 2026-04-13
Evidence: Attested by technician — no uninstall log or portal screenshot captured. Audit trail weakened. Technician verbally confirmed Intune uninstall ran under SYSTEM, exited cleanly, and printer was removed from ARES-5CG5220YTM.
Notes: ATTESTATION-ONLY — this check is NOT artifact-backed. This is the **third consecutive attestation-only check** (RTVAL-02, RTVAL-03, and RTVAL-04 are all attestation-only). Together, RTVAL-02/03/04 constitute an **attestation-only runtime half for Phase 10**: only RTVAL-01 (tenant ingestion) is artifact-backed for this plan. No `evidence/rtval-04-uninstall-log.txt` (IntuneManagementExtension.log excerpt showing SYSTEM context + exit 0) and no `evidence/rtval-04-uninstall-status.png` (Intune portal "Not installed" screenshot) were captured. The four required observations (uninstall script ran under `NT AUTHORITY\SYSTEM`, exit code 0, printer gone via `Get-Printer`, Intune portal shows "Not installed") were NOT independently verified; PASS rests entirely on the technician's verbal report. User was explicitly warned twice — first on RTVAL-03 and again on RTVAL-04 — that consecutive attestation-only checks significantly weaken the audit trail for Phase 10, and still chose to proceed without evidence on 2026-04-13. For any future regression, incident investigation, or phase sign-off, RTVAL-02/03/04 should all be treated as soft PASSes and re-run with full artifact capture before the phase is closed.
## Issues Found
### ISSUE-01: .intunewin package metadata unreadable by Intune Win32 app wizard
- **Discovered:** 2026-04-13 during RTVAL-01
- **Severity:** Blocker — prevents all downstream RTVAL checks (02, 03, 04)
- **Symptom:** Win32 app creation wizard fields (name, platform, size, MAM enabled) remain empty after file upload; OK button stays greyed out; no portal error shown
- **Root cause hypothesis:** Detection.xml embedded inside the .intunewin archive is missing, malformed, or uses an unexpected schema version. The IntuneWin32App packager requires a valid `detection.xml` at the root of the archive alongside the content folder.
- **Affected file:** Copieur_2eme.intunewin (evidence/Copieur_2eme.intunewin)
- **Affected code:** ImpTune .intunewin generator (commit 1c3f458)
- **Resolution path:** Debug the generator to verify Detection.xml is being created, embedded, and follows the expected Intune Win32 app manifest schema. Recommended: run `/gsd:debug` on the .intunewin generator, OR let phase verification surface this as a gap and use `/gsd:plan-phase 10 --gaps` to draft a gap-closure plan.
- **Status:** Resolved in commits 74535ea (HMAC over IV+ciphertext) and 7716246 (Detection.xml alignment with IntuneWinAppUtil.exe reference format). Re-tested 2026-04-13 against live tenant rubis.fr — package now parses; wizard fields populate; OK button enabled.
## Sign-off
**Reviewer acknowledgement required — DO NOT tick silently.**
Before ticking the checkboxes below, the reviewer MUST read the "Attestation-only audit trail" warning in this section and in `10-02-live-intune-runtime-validation-SUMMARY.md` ("CRITICAL: Attestation-Only Audit Trail Damage"). Only RTVAL-01 (tenant ingestion) is artifact-backed. RTVAL-02, RTVAL-03, and RTVAL-04 are verbal attestations from the technician, and the user was warned twice (on RTVAL-03 and again on RTVAL-04) that this weakens the audit trail. Signing off here means either (a) accepting the attestation-only runtime half with the compensating controls listed below, or (b) re-running RTVAL-02/03/04 with full artifact capture before ticking.
### Attestation-only audit trail (reviewer must acknowledge)
- **RTVAL-02:** PASS (attestation-only) — no `IntuneManagementExtension.log` excerpt, no portal screenshot
- **RTVAL-03:** PASS (attestation-only, 2nd consecutive) — no `rtval-03-detection.png`, no `rtval-03-detect-manual.txt`
- **RTVAL-04:** PASS (attestation-only, 3rd consecutive) — no `rtval-04-uninstall-log.txt`, no `rtval-04-uninstall-status.png`
Compensating controls (if accepting attestation-only):
- Known test device: `ARES-5CG5220YTM` (Windows build 26200.7171)
- Known tenant: `rubis.fr`
- Single session (technician was on the device while attesting)
- Known-good .intunewin generator (RTVAL-01 artifact-backed after ISSUE-01 fix)
- Technician name: _to be filled in at sign-off_
### Sign-off checklist
- [x] All four RTVAL runtime checks (RTVAL-01..04) have a PASS/FAIL status with documented rationale, and the attestation-only nature of RTVAL-02/03/04 has been explicitly read and acknowledged (not silently ticked)
- [x] All available evidence files are committed under `evidence/` (RTVAL-01 screenshots + package); RTVAL-02/03/04 evidence gaps are explicitly accepted OR a re-run is scheduled before phase closure
- [x] RTVAL-05 report is reviewer-grade: tenant, device, OS build, driver vendor, evidence links, and issues found are all populated
**Signed off by:** Sébastien QUEROL
**Signed off date:** 2026-04-13