From 13e098e2443001a560471aa03757a8d5533ae4d0 Mon Sep 17 00:00:00 2001 From: Kawa Date: Fri, 10 Apr 2026 11:42:39 +0200 Subject: [PATCH] docs(phase-02): research driver management phase Co-Authored-By: Claude Sonnet 4.6 --- .../02-driver-management/02-RESEARCH.md | 594 ++++++++++++++++++ 1 file changed, 594 insertions(+) create mode 100644 .planning/phases/02-driver-management/02-RESEARCH.md diff --git a/.planning/phases/02-driver-management/02-RESEARCH.md b/.planning/phases/02-driver-management/02-RESEARCH.md new file mode 100644 index 0000000..617b245 --- /dev/null +++ b/.planning/phases/02-driver-management/02-RESEARCH.md @@ -0,0 +1,594 @@ +# Phase 2: Driver Management - Research + +**Researched:** 2026-04-10 +**Domain:** ZIP upload handling, Windows INF parsing, content-addressed storage, HTMX-driven UI +**Confidence:** HIGH (FastAPI upload patterns, Python stdlib zipfile/configparser, HTMX encoding), MEDIUM (INF encoding edge-cases) + +--- + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|-----------------| +| DRV-01 | User can upload a driver package (ZIP containing INF + supporting files) | FastAPI `UploadFile` + `python-multipart`; `zipfile.ZipFile(io.BytesIO(...))` in-memory extraction | +| DRV-02 | System parses uploaded INF files and extracts valid driver names (DriverDesc) | Python `configparser` reading `[Manufacturer]` → Models sections; `[Strings]` token resolution; encoding auto-detect (ANSI / UTF-8 / UTF-16 LE) | +| DRV-03 | User can select driver name from parsed INF dropdown (no free-text) | HTMX `hx-post` + `hx-encoding="multipart/form-data"` + `hx-target` swap returning `` dropdown. + +The storage infrastructure was completed in Phase 1. `DriverStore.save(data) -> sha256` and the `Driver` ORM model (with `driver_desc`, `inf_filename`, `architecture`, `has_cat_file` fields) are already in place. Phase 2 only needs to fill those fields by parsing the INF and register the driver record in SQLite. + +INF parsing is the trickiest piece. Windows INF files use an INI-like format but have encoding variability (ANSI, UTF-8 with BOM, UTF-16 LE with BOM — all seen in real HP/Canon/Ricoh packages). Driver names (`DriverDesc`) are the left-hand values in the `[Models]` sections (e.g., `[Manufacturer.NTamd64]`), and they are frequently `%TOKEN%` references that must be resolved from the `[Strings]` section. Python's `configparser` handles this INI-like format well but needs an encoding sniff step and a `%`-token expander. Multi-model INF files (one INF with NTamd64 + NTarm64 + undecorated sections) must be deduplicated — extract all driver names, unique them, and present the merged list. + +**Primary recommendation:** Use `configparser` with encoding auto-detection + a custom `%TOKEN%` resolver to extract DriverDesc values. Do NOT import the third-party `pyinf` library — its scope is too narrow and adds a dependency with no maintenance signal. All required INF parsing logic is achievable in ~60 lines of stdlib Python. + +--- + +## Standard Stack + +### Core (all already in requirements.txt from Phase 1) + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| FastAPI | 0.115.x | Upload endpoint, HTMX fragment responses | Already installed; `UploadFile` built in | +| python-multipart | 0.0.9 | Required by FastAPI for `UploadFile` | Already installed | +| Peewee | 3.17.x | ORM — `Driver` record create/update | Already installed; schema already has all Phase 2 fields | +| Jinja2 | 3.1.x | Render drivers page + HTMX partial (driver list fragment) | Already installed | +| Python stdlib `zipfile` | 3.12 | Extract ZIP in-memory, list members | No new dependency | +| Python stdlib `configparser` | 3.12 | Parse INF (INI-like format) | No new dependency | +| Python stdlib `io` | 3.12 | `io.BytesIO` for in-memory ZIP | No new dependency | + +### No New Dependencies Required + +Phase 2 introduces zero new pip packages. Everything needed is either already installed (FastAPI/Peewee/Jinja2) or in the Python 3.12 stdlib (zipfile, configparser, io, hashlib). + +### Alternatives Considered + +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| `configparser` + custom token resolver | `pyinf` (third-party) | `pyinf` is a rudimentary single-developer project with no recent activity; stdlib configparser handles the INI format correctly with 30 extra lines of token resolution | +| In-memory `io.BytesIO` extraction | `NamedTemporaryFile` on disk | In-memory is simpler for small driver ZIPs (< 50 MB typical); avoids temp file cleanup; adequate for this use case | + +--- + +## Architecture Patterns + +### Recommended File Layout for Phase 2 + +``` +imptune/ +├── api/ +│ ├── pages.py # add GET /drivers page route +│ └── drivers.py # NEW: POST /drivers/upload endpoint +├── services/ +│ └── inf_parser.py # NEW: parse_inf(zip_bytes) -> ParsedInf dataclass +├── templates/ +│ ├── drivers.html # NEW: Drivers page (upload form + driver table) +│ └── partials/ +│ └── driver_select.html # NEW: HTMX partial — + + Uploading... + + +
+ {% include "partials/driver_list.html" %} +
+``` + +The server returns a replacement `
...
` containing the updated driver table plus any success/warning messages (unused files count, architecture, etc.). + +### Anti-Patterns to Avoid + +- **`zipfile.extractall()` without path validation:** Vulnerable to Zip Slip. Always iterate `zf.namelist()` and reject entries with `..` or absolute paths before reading. +- **`configparser` with `strict=True` for INF files:** Real INF files frequently contain duplicate keys across sections (multiple models with similar names). `strict=False` is required. +- **Using configparser's interpolation for `%TOKEN%` expansion:** configparser's built-in interpolation uses `%(key)s` syntax, not `%KEY%`. Use `RawConfigParser` and a separate regex-based `_resolve_tokens()` function. +- **Assuming one INF per ZIP:** Some vendor packages contain multiple INF files (x64 + x86 in different subdirectories). Parse the first `.inf` found; flag if multiples exist. +- **`async def` route for upload:** Since `DriverStore.save()` and `Driver.get_or_create()` are synchronous (Peewee), use a regular `def` route. FastAPI runs sync handlers in a thread pool automatically — no blocking. +- **Storing parsed driver names as a list in SQLite:** The existing `Driver.driver_desc` is a single `CharField`. For Phase 2, store the first (or primary) driver name. If multi-driver-name support is needed, that is a Phase 2+ schema change — but the current schema supports the DRV-03 requirement of a single dropdown choice per uploaded package. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| INF file parsing | Custom INI tokenizer from scratch | `configparser.RawConfigParser` + `_resolve_tokens()` | INI format edge cases: duplicate keys, inline comments, continuation lines, quoted strings | +| ZIP extraction | Custom byte-level ZIP reader | `zipfile.ZipFile(io.BytesIO(data))` | Handles all ZIP variants (ZIP64, deflate, stored); stdlib, no extra dep | +| Content-addressed file storage | New storage abstraction | `DriverStore` (already built in Phase 1) | SHA256 + dedup already implemented and tested | +| Driver ORM record | Raw SQL INSERT | `Driver.get_or_create(sha256=sha256, ...)` | Idempotent on re-upload; schema already has all Phase 2 fields | +| HTMX multipart upload form | Custom `fetch()` JavaScript | `hx-encoding="multipart/form-data"` on `
` | One attribute handles encoding; HTMX manages request + swap; no JS needed | + +**Key insight:** Phase 1 already solved persistence and deduplication. Phase 2's only novel logic is the INF parser and the upload endpoint wiring. + +--- + +## Common Pitfalls + +### Pitfall 1: INF Encoding Not Detected — `UnicodeDecodeError` + +**What goes wrong:** Reading a UTF-16 LE INF file as UTF-8 raises `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0`. Typical for HP and Canon INF files, which ship as UTF-16 LE with BOM. + +**Why it happens:** Windows uses UTF-16 internally; INF files signed for x64 are frequently written in UTF-16. + +**How to avoid:** Always sniff the first 3 bytes before calling `decode()`. UTF-16 LE BOM is `\xff\xfe`; UTF-16 BE BOM is `\xfe\xff`; UTF-8 BOM is `\xef\xbb\xbf`. Fall back to `cp1252` (not `utf-8`) for BOM-less files — `cp1252` is a superset of Latin-1 and handles Western European printer names without errors. + +**Warning signs:** `UnicodeDecodeError` on real vendor ZIP uploads. + +### Pitfall 2: `configparser` `strict=True` Fails on Duplicate Keys + +**What goes wrong:** `configparser.DuplicateOptionError` is raised when parsing INF files that list multiple models with the same base name but different decorations. + +**Why it happens:** `configparser` defaults to `strict=True` which rejects duplicate keys within the same section. INF files frequently have this structure. + +**How to avoid:** Instantiate with `configparser.RawConfigParser(strict=False)`. + +### Pitfall 3: `%TOKEN%` Values Appear Literally in Dropdown + +**What goes wrong:** Driver dropdown shows `%HP_LASERJET_P2055D%` instead of `HP LaserJet P2055d`. + +**Why it happens:** `configparser` does not process `%...%` INF token syntax — only its own `%(key)s` interpolation, which is irrelevant here. + +**How to avoid:** After parsing, apply `_resolve_tokens(raw_value, strings_dict)` to every extracted device-description. Build the `strings` dict from `[Strings]` section values (stripped of surrounding double-quotes). + +### Pitfall 4: Zip Slip Vulnerability on Upload + +**What goes wrong:** A malicious or malformed ZIP contains entries like `../../etc/passwd`. `zipfile.extractall()` writes those files to the host filesystem. + +**Why it happens:** The default Python `zipfile.extractall()` does not check for traversal paths. + +**How to avoid:** Never call `extractall()`. Read individual members with `zf.read(name)` after validating each `name` in `zf.namelist()` does not start with `/` or contain `..`. This is safe because the file is read into memory, not extracted to disk. + +**Warning signs:** Any code that calls `zf.extractall(path)` without path filtering. + +### Pitfall 5: No `.inf` in ZIP Returns a Confusing Error + +**What goes wrong:** Technician uploads a ZIP that contains only drivers but not the INF (common when someone zips the wrong folder). The endpoint crashes with a `KeyError` or returns HTTP 500. + +**Why it happens:** Code assumes at least one `.inf` member exists. + +**How to avoid:** Explicit check: `if not inf_names: raise HTTPException(400, "No .inf file found in ZIP")`. Return the error as an HTMX response so it appears in-page without a full reload. Render the error inside the `#driver-list` target. + +### Pitfall 6: Multi-INF ZIP — Wrong Driver Names Selected + +**What goes wrong:** A ZIP with both x64 and x86 INF files (in subdirectories) produces duplicate driver names if both are parsed, or wrong names if the wrong file is parsed. + +**Why it happens:** Some vendors ship a ZIP with `x64/printer.inf` and `x86/printer.inf` containing different model lists. + +**How to avoid:** Prefer INF files whose path does not contain `x86` when an `amd64`/`x64` sibling exists. Sort candidates to prefer `amd64`/`NTamd64` variants. Log a warning when multiple INF files are found; surface the INF filename in the UI so the technician can verify. + +### Pitfall 7: `Driver.driver_desc` Stores Only One Name (Schema Limitation) + +**What goes wrong:** An INF file contains 15 different models. Only one is stored in `driver_desc`. DRV-03 requires a dropdown of all parsed names — but after page reload, only the stored name is shown. + +**Why it happens:** The Phase 1 schema stores a single `driver_desc` CharField. + +**How to avoid:** Store all parsed driver names as a JSON-encoded list in `driver_desc` (e.g., `json.dumps(driver_names)`). The dropdown is generated by parsing the stored JSON. Alternatively, store the INF text itself. The simplest solution that satisfies DRV-03 without schema changes: store `json.dumps(parsed.driver_names)` in `driver_desc` and decode at read time. This fits in one CharField with no migration. + +--- + +## Code Examples + +### INF Encoding Detection + +```python +# imptune/services/inf_parser.py +def _detect_encoding(raw: bytes) -> str: + """Sniff BOM bytes to determine INF file encoding. + Source: Microsoft WDK — general-syntax-rules-for-inf-files + """ + if raw[:2] in (b'\xff\xfe', b'\xfe\xff'): + return 'utf-16' + if raw[:3] == b'\xef\xbb\xbf': + return 'utf-8-sig' + return 'cp1252' +``` + +### Safe ZIP Member Reading (Zip Slip Prevention) + +```python +# Before reading any member: +for name in zf.namelist(): + if name.startswith('/') or '..' in name: + raise HTTPException(400, f"Dangerous path in ZIP: {name}") +# Then read safely: +inf_bytes = zf.read(inf_names[0]) +``` + +### HTMX Upload Form (multipart) + +```html + + + + +
+``` + +### Driver Record Upsert (idempotent on SHA256) + +```python +# Peewee get_or_create — safe for duplicate uploads +driver, created = Driver.get_or_create( + sha256=sha256, + defaults={ + 'original_filename': file.filename, + 'size_bytes': len(data), + 'driver_desc': json.dumps(parsed.driver_names), + 'inf_filename': parsed.inf_filename, + 'architecture': parsed.architecture, + 'has_cat_file': parsed.has_cat_file, + } +) +``` + +### Unused Files Detection + +```python +# Compare ZIP member basenames against INF text +# Source: DRV-05 requirement +inf_lower = inf_text.lower() +unused = [] +for member in zip_names: + basename = member.rsplit('/', 1)[-1].rsplit('\\', 1)[-1] + if basename.lower() not in inf_lower: + unused.append(member) +``` + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| Free-text driver name entry | Dropdown from parsed INF DriverDesc | DRV-03 (this phase) | Eliminates typos; driver name matches exactly what pnputil expects | +| Manual INF file navigation | Automated DriverDesc extraction + `%TOKEN%` resolution | This phase | Technician never needs to open the INF | +| `pyinf` third-party library | Python stdlib `configparser` + custom resolver | This phase decision | Zero new dependency; full control over edge-case handling | + +**Note on driver name storage:** The `driver_desc` column was designed in Phase 1 as a single `CharField`. Storing `json.dumps(list)` is the correct approach to preserve all model names without a schema migration. The Phase 3 printer configuration form reads this JSON to render the `` with driver names | integration | `pytest tests/test_driver_upload.py::test_upload_returns_select -x` | Wave 0 | +| DRV-04 | Uploaded driver file exists on disk under DRIVERS_DIR after upload | integration | `pytest tests/test_driver_upload.py::test_driver_persisted -x` | Wave 0 | +| DRV-04 | Re-uploading same ZIP does not create duplicate Driver record | integration | `pytest tests/test_driver_upload.py::test_dedup_upload -x` | Wave 0 | +| DRV-05 | parse_inf returns unused_files list for files not in INF text | unit | `pytest tests/test_inf_parser.py::test_unused_files -x` | Wave 0 | +| DRV-05 | Upload response includes unused-file count/list in HTML | integration | `pytest tests/test_driver_upload.py::test_unused_files_in_response -x` | Wave 0 | + +### Sampling Rate + +- **Per task commit:** `pytest tests/ -x -q` +- **Per wave merge:** `pytest tests/ -v` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps + +- [ ] `tests/test_inf_parser.py` — covers DRV-02 (INF parsing, token resolution, encoding, multi-model) +- [ ] `tests/test_driver_upload.py` — covers DRV-01, DRV-03, DRV-04, DRV-05 (upload endpoint integration tests) +- [ ] `tests/fixtures/sample.inf` — minimal valid INF fixture with `%TOKEN%` values +- [ ] `tests/fixtures/sample_utf16.inf` — UTF-16 LE encoded INF fixture +- [ ] `tests/fixtures/sample_multi_model.inf` — INF with NTamd64 and undecorated sections +- [ ] `imptune/services/__init__.py` — package marker (services/ directory exists in project structure but is empty) + +*(Framework already installed; conftest.py with `tmp_data_dir` fixture already covers DB isolation)* + +--- + +## Sources + +### Primary (HIGH confidence) + +- [Microsoft WDK — INF Models Section](https://learn.microsoft.com/en-us/windows-hardware/drivers/install/inf-models-section) — device-description = install-section-name,hw-id format; architecture decorations (NTamd64, NTarm64) +- [Microsoft WDK — General Syntax Rules for INF Files](https://learn.microsoft.com/en-us/windows-hardware/drivers/install/general-syntax-rules-for-inf-files) — encoding (ANSI/UTF-8/UTF-16), `%strkey%` token format, comment syntax, case-insensitivity +- [Microsoft WDK — Printer INF File Entries](https://learn.microsoft.com/en-us/windows-hardware/drivers/print/printer-inf-file-entries) — DriverFile, DataFile, ConfigFile, DriverDesc usage in Ntprint.dll +- [Microsoft WDK — Decorations in Printer INF Files](https://learn.microsoft.com/en-us/windows-hardware/drivers/print/decorations-in-printer-inf-files) — NTamd64 decoration mandatory for x64 since WS2003 SP1 +- [FastAPI — Request Files](https://fastapi.tiangolo.com/tutorial/request-files/) — `UploadFile`, `File(...)`, reading file bytes +- [HTMX — hx-encoding attribute](https://htmx.org/attributes/hx-encoding/) — `multipart/form-data` required for file uploads +- [HTMX — File Upload example](https://htmx.org/examples/file-upload/) — progress tracking, server response swap +- Python 3.12 stdlib `zipfile` — `ZipFile(io.BytesIO())`, `namelist()`, `read()`, `is_zipfile()` +- Python 3.12 stdlib `configparser` — `RawConfigParser(strict=False)`, `read_string()`, `has_section()`, `items()` + +### Secondary (MEDIUM confidence) + +- [Snyk / Zip Slip Vulnerability](https://github.com/snyk/zip-slip-vulnerability) — path traversal attack pattern; prevention via `namelist()` validation +- [FastAPI file size limiting discussion](https://github.com/fastapi/fastapi/issues/362) — post-read size check pattern; no built-in pre-rejection mechanism +- [Microsoft WDK — Printer INF File Data Sections](https://learn.microsoft.com/en-us/windows-hardware/drivers/print/printer-inf-file-data-sections) — DataSection pattern; Previous Names section + +### Tertiary (LOW confidence) + +- [pyinf GitHub](https://github.com/tty72/pyinf) — reviewed and rejected: rudimentary, no recent activity, no benefit over stdlib + +--- + +## Metadata + +**Confidence breakdown:** +- FastAPI upload patterns: HIGH — official docs verified +- HTMX multipart form: HIGH — official docs verified +- INF format (overall): HIGH — Microsoft WDK official docs +- INF encoding handling (edge cases): MEDIUM — documented rule is clear but real-world INF corpus has variability; edge cases may surface during testing +- Unused-file detection accuracy: MEDIUM — naive approach is best-effort; accuracy depends on INF structure + +**Research date:** 2026-04-10 +**Valid until:** 2026-05-10 (stable ecosystem)