From 9a55c9e7d075b639af1e395c6011aea19581ae49 Mon Sep 17 00:00:00 2001 From: Dev Date: Thu, 2 Apr 2026 15:33:47 +0200 Subject: [PATCH] =?UTF-8?q?docs(03-04):=20complete=20SearchService=20and?= =?UTF-8?q?=20DuplicatesService=20plan=20=E2=80=94=202/2=20tasks,=205=20Ma?= =?UTF-8?q?keKey=20tests=20pass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .planning/ROADMAP.md | 2 +- .planning/STATE.md | 14 +- .planning/phases/03-storage/03-04-SUMMARY.md | 128 +++++++++++++++++++ 3 files changed, 138 insertions(+), 6 deletions(-) create mode 100644 .planning/phases/03-storage/03-04-SUMMARY.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 5fd1ce0..87713a5 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -125,6 +125,6 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 |-------|----------------|--------|-----------| | 1. Foundation | 8/8 | Complete | 2026-04-02 | | 2. Permissions | 7/7 | Complete | 2026-04-02 | -| 3. Storage and File Operations | 4/8 | In Progress| | +| 3. Storage and File Operations | 5/8 | In Progress| | | 4. Bulk Operations and Provisioning | 0/? | Not started | - | | 5. Distribution and Hardening | 0/? | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 2c10f12..4a8588e 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,14 +3,14 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: executing -stopped_at: Completed 03-06-PLAN.md — Phase 3 EN/FR localization keys -last_updated: "2026-04-02T13:32:33.562Z" +stopped_at: Completed 03-04-PLAN.md — SearchService and DuplicatesService +last_updated: "2026-04-02T13:33:35.151Z" last_activity: 2026-04-02 — Plan 03-02 complete — StorageService CSOM scan engine implemented progress: total_phases: 5 completed_phases: 2 total_plans: 23 - completed_plans: 19 + completed_plans: 20 percent: 65 --- @@ -79,6 +79,7 @@ Progress: [██████░░░░] 65% | Phase 03-storage P01 | 10min | 2 tasks | 22 files | | Phase 03-storage P03 | 2min | 2 tasks | 2 files | | Phase 03-storage P06 | 5min | 1 tasks | 3 files | +| Phase 03-storage P04 | 2min | 2 tasks | 2 files | ## Accumulated Context @@ -138,6 +139,9 @@ Recent decisions affecting current work: - [Phase 03-storage 03-02]: StorageService.LastModified uses StorageMetrics.LastModified with fallback to Folder.TimeLastModified — StorageMetrics.LastModified may be DateTime.MinValue for empty libraries - [Phase 03-storage 03-02]: System folder filter uses Forms/ and _-prefix heuristic — matches SharePoint standard hidden folder naming convention - [Phase 03-storage]: Explicit System.IO using required in StorageCsvExportService and StorageHtmlExportService — WPF project does not include System.IO in implicit usings (established project pattern) +- [Phase 03-storage 03-04]: SearchService uses SelectProperties.Add per-item loop — StringCollection has no AddRange(string[]) overload in this SDK version +- [Phase 03-storage 03-04]: DuplicatesService.MakeKey internal static method matches inline test helper in DuplicatesServiceTests exactly — deliberate design to ensure test parity +- [Phase 03-storage 03-04]: DuplicatesService file mode re-implements pagination inline — avoids coupling between services with different result models (DuplicateItem vs SearchResult) ### Pending Todos @@ -150,6 +154,6 @@ None yet. ## Session Continuity -Last session: 2026-04-02T13:32:33.560Z -Stopped at: Completed 03-06-PLAN.md — Phase 3 EN/FR localization keys +Last session: 2026-04-02T13:33:35.149Z +Stopped at: Completed 03-04-PLAN.md — SearchService and DuplicatesService Resume file: None diff --git a/.planning/phases/03-storage/03-04-SUMMARY.md b/.planning/phases/03-storage/03-04-SUMMARY.md new file mode 100644 index 0000000..cdfc514 --- /dev/null +++ b/.planning/phases/03-storage/03-04-SUMMARY.md @@ -0,0 +1,128 @@ +--- +phase: 03-storage +plan: "04" +subsystem: search +tags: [csom, sharepoint-search, kql, duplicates, pagination] + +# Dependency graph +requires: + - phase: 03-01 + provides: ISearchService, IDuplicatesService, SearchOptions, DuplicateScanOptions, SearchResult, DuplicateItem, DuplicateGroup, OperationProgress models and interfaces + +provides: + - SearchService: KQL-based file search with 500-row pagination and 50,000-item hard cap + - DuplicatesService: file duplicates via Search API + folder duplicates via CAML FSObjType=1 + - MakeKey composite key logic for grouping duplicates by name+size+dates+counts + +affects: [03-05, 03-07, 03-08] + +# Tech tracking +tech-stack: + added: [] + patterns: + - "KeywordQuery + SearchExecutor pattern: executor.ExecuteQuery(kq) registers query, then ExecuteQueryRetryHelper.ExecuteQueryRetryAsync executes it" + - "StringCollection.Add loop: SelectProperties is StringCollection, not List — must add properties one-by-one" + - "StartRow pagination: += BatchSize per iteration, hard stop at MaxStartRow (50,000)" + - "goto done pattern for early exit from nested pagination loop when MaxResults reached" + +key-files: + created: + - SharepointToolbox/Services/SearchService.cs + - SharepointToolbox/Services/DuplicatesService.cs + modified: [] + +key-decisions: + - "SearchService uses SelectProperties.Add per-item loop — StringCollection has no AddRange(string[]) overload in this SDK version" + - "DuplicatesService.MakeKey internal static method matches inline test helper in DuplicatesServiceTests exactly — deliberate design to ensure test parity" + - "DuplicatesService file mode re-implements pagination inline (not delegating to SearchService) — avoids coupling between services with different result models" + +patterns-established: + - "KQL SelectProperties: Add each property in a foreach loop, never AddRange with array" + - "Search pagination: do/while with startRow <= MaxStartRow guard, break on empty table" + - "Folder CAML: FSObjType=1 (not FileSystemObjectType) — wrong name returns zero results" + +requirements-completed: [SRCH-01, SRCH-02, DUPL-01, DUPL-02] + +# Metrics +duration: 2min +completed: 2026-04-02 +--- + +# Phase 03 Plan 04: SearchService and DuplicatesService Summary + +**KQL file search with 500-row StartRow pagination (50k cap) and composite-key duplicate detection for files (Search API) and folders (CAML FSObjType=1)** + +## Performance + +- **Duration:** 2 min +- **Started:** 2026-04-02T14:09:25Z +- **Completed:** 2026-04-02T14:12:09Z +- **Tasks:** 2 +- **Files modified:** 2 created + +## Accomplishments + +- SearchService implements full KQL builder (extension, date range, creator, editor, library filters) with paginated retrieval up to 50,000 items +- DuplicatesService supports both file mode (Search API) and folder mode (CAML FSObjType=1) with client-side composite key grouping +- MakeKey logic matches the inline test scaffold from Plan 03-01 DuplicatesServiceTests — 5 pure-logic tests pass + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Implement SearchService** - `9e3d501` (feat) +2. **Task 2: Implement DuplicatesService** - `df5f79d` (feat) + +## Files Created/Modified + +- `SharepointToolbox/Services/SearchService.cs` - KQL search with pagination, vti_history filter, regex client-side filter, KQL length validation +- `SharepointToolbox/Services/DuplicatesService.cs` - File/folder duplicate detection, MakeKey composite grouping, CAML folder enumeration + +## Decisions Made + +- `SelectProperties` is a `StringCollection` — `AddRange(string[])` does not compile. Fixed inline per-item `foreach` add loop (Rule 1 auto-fix applied during Task 1 first build). +- DuplicatesService re-implements file pagination inline rather than delegating to SearchService because result types differ (`DuplicateItem` vs `SearchResult`) and the two services have different lifecycles. +- `MakeKey` is `internal static` to match the test project's inline copy — enables verifying parity without a live CSOM context. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] StringCollection.AddRange(string[]) does not exist** +- **Found during:** Task 1 (SearchService build) +- **Issue:** `kq.SelectProperties.AddRange(new[] { ... })` — `SelectProperties` is `StringCollection` which has no `AddRange` taking `string[]`; extension method overload requires `List` receiver +- **Fix:** Replaced with `foreach` loop calling `kq.SelectProperties.Add(prop)` for each property name +- **Files modified:** `SharepointToolbox/Services/SearchService.cs`, `SharepointToolbox/Services/DuplicatesService.cs` +- **Verification:** `dotnet build` 0 errors after fix; same fix proactively applied in DuplicatesService before its first build +- **Committed in:** `9e3d501` (Task 1 commit) + +--- + +**Total deviations:** 1 auto-fixed (Rule 1 - bug) +**Impact on plan:** Minor API surface mismatch in the plan's code listing; fix is purely syntactic, no behavioral difference. + +## Issues Encountered + +- `dotnet test ... -x` flag not recognized by the `dotnet test` CLI on this machine (MSBuild switch error). Removed the flag; tests ran correctly without it. + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +- SearchService and DuplicatesService are complete and compile cleanly +- Wave 2 is now ready for 03-05 (Search/Duplicate exports) and 03-06 (Localization) to proceed in parallel with 03-03 (Storage exports) +- 5 MakeKey tests pass; CSOM integration tests will remain skipped until a live tenant is available + +--- +*Phase: 03-storage* +*Completed: 2026-04-02* + +## Self-Check: PASSED + +- SharepointToolbox/Services/SearchService.cs: FOUND +- SharepointToolbox/Services/DuplicatesService.cs: FOUND +- .planning/phases/03-storage/03-04-SUMMARY.md: FOUND +- Commit 9e3d501 (SearchService): FOUND +- Commit df5f79d (DuplicatesService): FOUND