From 97fc29c15e802325607d7d257c89bf6e33dc5f3a Mon Sep 17 00:00:00 2001 From: Dev Date: Fri, 3 Apr 2026 09:17:41 +0200 Subject: [PATCH] docs(04): research phase domain for Bulk Operations and Provisioning Co-Authored-By: Claude Opus 4.6 (1M context) --- .../04-RESEARCH.md | 675 ++++++++++++++++++ 1 file changed, 675 insertions(+) create mode 100644 .planning/phases/04-bulk-operations-and-provisioning/04-RESEARCH.md diff --git a/.planning/phases/04-bulk-operations-and-provisioning/04-RESEARCH.md b/.planning/phases/04-bulk-operations-and-provisioning/04-RESEARCH.md new file mode 100644 index 0000000..2b4881f --- /dev/null +++ b/.planning/phases/04-bulk-operations-and-provisioning/04-RESEARCH.md @@ -0,0 +1,675 @@ +# Phase 4: Bulk Operations and Provisioning - Research + +**Researched:** 2026-04-03 +**Domain:** SharePoint CSOM bulk operations, PnP Framework provisioning, Microsoft Graph group management, CSV parsing +**Confidence:** HIGH + +## Summary + +Phase 4 introduces five new tabs (Transfer, Bulk Members, Bulk Sites, Folder Structure, Templates) that perform write operations against SharePoint Online and Microsoft 365 Groups. The core challenge is implementing reliable per-item error handling with continue-on-error semantics, CSV import/validation/preview, and cancellation support -- all following the established `FeatureViewModelBase` pattern. + +The file transfer feature uses CSOM's `MoveCopyUtil` for cross-site file copy/move operations. Bulk member addition requires Microsoft Graph SDK (new dependency) for M365 Group operations, with CSOM fallback for classic SharePoint groups. Site creation uses PnP Framework's `SiteCollection.CreateAsync`. Template capture reads site structure via CSOM properties, and template application manually recreates structure rather than using the heavy PnP Provisioning Engine. CSV parsing uses CsvHelper (new dependency). Folder structure creation uses CSOM's `Folder.Folders.Add`. + +**Primary recommendation:** Use a shared `BulkOperationRunner` pattern across all bulk operations to enforce continue-on-error, per-item reporting, cancellation, and retry-failed semantics consistently. Keep template capture/apply as manual CSOM operations (not PnP Provisioning Engine) to match the lightweight scope defined in CONTEXT.md. + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions +- Continue all items on error, report summary at end -- never stop on first failure +- Errors logged live in log panel (red) AND user can export CSV of failed items after completion +- Failed-items CSV includes: original row data, error message, timestamp -- user can fix and re-import +- "Retry Failed" button appears after completion with partial failures, re-runs only failed items +- User can also manually re-import the corrected failed-items CSV +- Always show confirmation summary dialog before any bulk write operation starts +- File transfer: user chooses Copy or Move; conflict policy per operation: Skip/Overwrite/Rename +- Move deletes source files only after successful transfer to destination +- Source and destination selection: site picker + library/folder tree browser on both sides +- Metadata preservation: best effort via CSOM; if rejected, log warning and continue +- Template capture: user selects what to capture via checkboxes +- Available capture options: Libraries, Folders, Permission groups, Site logo, Site settings +- No custom columns, content types, or navigation links in v1 +- Template remembers source site type and creates same type on apply +- Templates persisted as JSON locally +- Dedicated Templates tab with list, capture, apply, rename/delete +- Bundled example CSV templates shipped with app for each bulk operation +- After CSV import: full validation pass + DataGrid preview with valid/invalid indicators +- CSV encoding: UTF-8 with BOM detection +- Each operation gets its own top-level tab: Transfer, Bulk Members, Bulk Sites, Folder Structure, Templates +- Each tab follows FeatureViewModelBase pattern (AsyncRelayCommand + IProgress + CancellationToken) + +### Claude's Discretion +- Exact CSV column names and schemas for each bulk operation type +- Tree browser component implementation details for file transfer source/destination +- Template JSON schema structure +- PnP Provisioning Engine usage details for template apply +- Confirmation dialog layout and wording +- DataGrid preview grid column configuration +- Retry mechanism implementation details + +### Deferred Ideas (OUT OF SCOPE) +None -- discussion stayed within phase scope + + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|-----------------| +| BULK-01 | Transfer files/folders between sites with progress tracking | MoveCopyUtil CSOM API for cross-site copy/move; download-then-upload fallback pattern from existing PowerShell | +| BULK-02 | Add members to groups in bulk from CSV | Microsoft Graph SDK batch API (up to 20 members per PATCH); CSOM fallback for classic SP groups | +| BULK-03 | Create multiple sites in bulk from CSV | PnP Framework SiteCollection.CreateAsync with TeamSiteCollectionCreationInformation / CommunicationSiteCollectionCreationInformation | +| BULK-04 | All bulk operations support cancellation | CancellationToken threaded through BulkOperationRunner; check between items | +| BULK-05 | Bulk errors reported per-item | BulkOperationResult model with per-item status; failed-items CSV export | +| TMPL-01 | Capture site structure as template | CSOM Web/List/Folder/Group property reads; manual recursive folder enumeration | +| TMPL-02 | Apply template to create new site | SiteCollection.CreateAsync + manual CSOM library/folder/group/settings creation | +| TMPL-03 | Templates persist locally as JSON | System.Text.Json serialization following SettingsRepository pattern | +| TMPL-04 | Manage templates (create, rename, delete) | TemplateRepository with same atomic write pattern as SettingsRepository | +| FOLD-01 | Create folder structures from CSV | CSOM Folder.Folders.Add with parent-first ordering | +| FOLD-02 | Example CSV templates provided | Existing /examples/ CSV files already present; bundle as embedded resources | + + +## Standard Stack + +### Core (already in project) +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| PnP.Framework | 1.18.0 | SharePoint CSOM extensions, site creation, folder/file operations | Already used; provides SiteCollection.CreateAsync, MoveCopyUtil wrappers | +| CommunityToolkit.Mvvm | 8.4.2 | MVVM pattern, ObservableProperty, AsyncRelayCommand | Already used for all ViewModels | +| Microsoft.Identity.Client | 4.83.3 | MSAL authentication | Already used; Graph SDK will share token cache | +| Serilog | 4.3.1 | Structured logging | Already used for all operations | +| System.Text.Json | (built-in) | JSON serialization for templates | Already used in SettingsRepository/ProfileRepository | + +### New Dependencies +| Library | Version | Purpose | Why Needed | +|---------|---------|---------|------------| +| CsvHelper | 33.1.0 | CSV parsing with type mapping, validation, BOM handling | De facto standard for .NET CSV; handles encoding, quoting, type conversion automatically. Avoids hand-rolling parser | +| Microsoft.Graph | 5.x (latest stable) | M365 Group member management, batch API | Required for BULK-02 (adding members to M365 Groups); CSOM cannot manage M365 Group membership directly | +| Microsoft.Graph.Core | 3.x (transitive) | Graph SDK core/auth | Transitive dependency of Microsoft.Graph | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| CsvHelper | Manual parsing (Split + StreamReader) | CsvHelper handles RFC 4180 edge cases, BOM detection, type mapping. Manual parsing risks bugs on quoted fields, encoding | +| Microsoft.Graph SDK | Raw HTTP to Graph API | SDK handles auth token injection, batch splitting, retry, serialization. Not worth hand-rolling | +| PnP Provisioning Engine for templates | Manual CSOM property reads + writes | Provisioning Engine is heavyweight, captures far more than needed (content types, navigation, custom actions). Manual approach matches PS app behavior and v1 scope | + +**Installation:** +```bash +dotnet add SharepointToolbox/SharepointToolbox.csproj package CsvHelper --version 33.1.0 +dotnet add SharepointToolbox/SharepointToolbox.csproj package Microsoft.Graph --version 5.74.0 +``` + +## Architecture Patterns + +### Recommended Project Structure (new files) +``` +SharepointToolbox/ + Core/ + Models/ + BulkOperationResult.cs # Per-item result: Success/Failed/Skipped + error message + BulkMemberRow.cs # CSV row model for member addition + BulkSiteRow.cs # CSV row model for site creation + TransferJob.cs # Source/dest site+library+conflict policy + FolderStructureRow.cs # CSV row model for folder structure + SiteTemplate.cs # Template JSON model + SiteTemplateOptions.cs # Capture options (booleans for each section) + TemplateLibraryInfo.cs # Library captured in template + TemplateFolderInfo.cs # Folder tree captured in template + TemplatePermissionGroup.cs # Permission group captured in template + Infrastructure/ + Persistence/ + TemplateRepository.cs # JSON persistence for templates (like SettingsRepository) + Services/ + IFileTransferService.cs # Interface for file copy/move operations + FileTransferService.cs # CSOM MoveCopyUtil + download/upload fallback + IBulkMemberService.cs # Interface for bulk member addition + BulkMemberService.cs # Graph SDK batch API + CSOM fallback + IBulkSiteService.cs # Interface for bulk site creation + BulkSiteService.cs # PnP Framework SiteCollection.CreateAsync + ITemplateService.cs # Interface for template capture/apply + TemplateService.cs # CSOM property reads for capture, CSOM writes for apply + IFolderStructureService.cs # Interface for folder creation from CSV + FolderStructureService.cs # CSOM folder creation + ICsvValidationService.cs # Interface for CSV validation + preview data + CsvValidationService.cs # CsvHelper-based parsing, schema validation, row validation + Export/ + BulkResultCsvExportService.cs # Failed-items CSV export (re-importable format) + ViewModels/ + Tabs/ + TransferViewModel.cs + BulkMembersViewModel.cs + BulkSitesViewModel.cs + FolderStructureViewModel.cs + TemplatesViewModel.cs + Views/ + Tabs/ + TransferView.xaml(.cs) + BulkMembersView.xaml(.cs) + BulkSitesView.xaml(.cs) + FolderStructureView.xaml(.cs) + TemplatesView.xaml(.cs) + Dialogs/ + ConfirmBulkOperationDialog.xaml(.cs) # Pre-write confirmation + FolderBrowserDialog.xaml(.cs) # Tree browser for library/folder selection +``` + +### Pattern 1: BulkOperationRunner (continue-on-error with per-item reporting) +**What:** A generic helper that iterates items, catches per-item exceptions, tracks results, and checks cancellation between items. +**When to use:** Every bulk operation (transfer, members, sites, folders). +**Example:** +```csharp +// Core pattern shared by all bulk operations +public static class BulkOperationRunner +{ + public static async Task> RunAsync( + IReadOnlyList items, + Func processItem, + IProgress progress, + CancellationToken ct) + { + var results = new List>(); + for (int i = 0; i < items.Count; i++) + { + ct.ThrowIfCancellationRequested(); + progress.Report(new OperationProgress(i + 1, items.Count, $"Processing {i + 1}/{items.Count}...")); + try + { + await processItem(items[i], i, ct); + results.Add(BulkItemResult.Success(items[i])); + } + catch (OperationCanceledException) { throw; } // cancellation propagates + catch (Exception ex) + { + results.Add(BulkItemResult.Failed(items[i], ex.Message)); + } + } + return new BulkOperationSummary(results); + } +} +``` + +### Pattern 2: CSV Import Pipeline (validate -> preview -> execute) +**What:** Three-step flow: parse CSV with CsvHelper, validate all rows, present DataGrid preview with valid/invalid indicators, then execute on user confirmation. +**When to use:** Bulk Members, Bulk Sites, Folder Structure tabs. +**Example:** +```csharp +// CsvHelper usage with BOM-tolerant configuration +public List> ParseAndValidate(Stream csvStream) +{ + using var reader = new StreamReader(csvStream, Encoding.UTF8, detectEncodingFromByteOrderMarks: true); + using var csv = new CsvReader(reader, new CsvConfiguration(CultureInfo.InvariantCulture) + { + HasHeaderRecord = true, + MissingFieldFound = null, // handle missing fields gracefully + HeaderValidated = null, // custom validation instead + DetectDelimiter = true, // auto-detect ; vs , + TrimOptions = TrimOptions.Trim, + }); + + var rows = new List>(); + csv.Read(); + csv.ReadHeader(); + while (csv.Read()) + { + try + { + var record = csv.GetRecord(); + var errors = Validate(record); + rows.Add(new CsvValidationRow(record, errors)); + } + catch (Exception ex) + { + rows.Add(CsvValidationRow.ParseError(csv.Context.Parser.RawRecord, ex.Message)); + } + } + return rows; +} +``` + +### Pattern 3: Failed-Items CSV Export (re-importable) +**What:** Export failed items as CSV identical to input format but with appended Error and Timestamp columns. User can fix and re-import. +**When to use:** After any bulk operation completes with partial failures. +**Example:** +```csharp +// Failed-items CSV: same columns as input + Error + Timestamp +public async Task ExportFailedItemsAsync( + IReadOnlyList> failedItems, + string filePath, CancellationToken ct) +{ + await using var writer = new StreamWriter(filePath, false, new UTF8Encoding(true)); + await using var csv = new CsvWriter(writer, CultureInfo.InvariantCulture); + + // Write original columns + error column + csv.WriteHeader(); + csv.WriteField("Error"); + csv.WriteField("Timestamp"); + await csv.NextRecordAsync(); + + foreach (var item in failedItems.Where(r => !r.IsSuccess)) + { + csv.WriteRecord(item.Item); + csv.WriteField(item.ErrorMessage); + csv.WriteField(DateTime.UtcNow.ToString("o")); + await csv.NextRecordAsync(); + } +} +``` + +### Pattern 4: File Transfer via CSOM (download-then-upload approach) +**What:** The existing PowerShell app downloads files to a temp folder, then uploads to the destination. This is the most reliable cross-site approach when using CSOM directly. MoveCopyUtil is an alternative for same-tenant operations. +**When to use:** BULK-01 file transfer. +**Example:** +```csharp +// Approach A: MoveCopyUtil (preferred for same-tenant, simpler) +var srcPath = ResourcePath.FromDecodedUrl(sourceFileServerRelativeUrl); +var dstPath = ResourcePath.FromDecodedUrl(destFileServerRelativeUrl); +MoveCopyUtil.CopyFileByPath(ctx, srcPath, dstPath, overwrite, new MoveCopyOptions +{ + KeepBoth = false, // or true for "Rename" conflict policy + ResetAuthorAndCreatedOnCopy = false, +}); +await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct); + +// Approach B: Download-then-upload (reliable cross-site fallback) +// 1. Download: ctx.Web.GetFileByServerRelativeUrl(url).OpenBinaryStream() +// 2. Upload: destFolder.Files.Add(new FileCreationInformation { ContentStream, Overwrite, Url }) +``` + +### Anti-Patterns to Avoid +- **Stopping on first error in bulk operations:** All bulk ops MUST continue-on-error per CONTEXT.md. Never throw from the item loop. +- **Using PnP Provisioning Engine for template capture/apply:** It captures far more than needed (content types, custom actions, navigation, page layouts). The v1 scope only captures libraries, folders, permission groups, logo, settings. Manual CSOM reads are simpler, lighter, and match the PS app behavior. +- **Parsing CSV manually with string.Split:** CSV has too many edge cases (quoted fields containing delimiters, embedded newlines, BOM). Use CsvHelper. +- **Creating M365 Groups via CSOM for member addition:** CSOM cannot manage M365 Group membership. Must use Microsoft Graph API. +- **Blocking UI thread during file download/upload:** All service methods are async, run via FeatureViewModelBase.RunOperationAsync. + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| CSV parsing | Custom StreamReader + Split | CsvHelper 33.1.0 | RFC 4180 quoting, BOM detection, delimiter detection, type mapping, error handling | +| M365 Group member management | Raw HTTP calls to Graph | Microsoft.Graph SDK 5.x | Token management, batch splitting (20-per-request auto), retry, deserialization | +| Bulk operation error handling | Copy-paste try/catch in each ViewModel | Shared BulkOperationRunner | Ensures consistent continue-on-error, per-item tracking, cancellation, and retry-failed across all 5 bulk operations | +| CSV field escaping on export | Manual quote doubling | CsvHelper CsvWriter | RFC 4180 compliance, handles all edge cases | +| Template JSON serialization | Manual JSON string building | System.Text.Json with typed models | Already used in project; handles nulls, escaping, indentation | + +**Key insight:** The bulk operation infrastructure (runner, result model, failed-items export, retry) is shared across all five features. Building it as a reusable component in Wave 0 prevents duplicated error handling logic and ensures consistent behavior. + +## Common Pitfalls + +### Pitfall 1: MoveCopyUtil fails on cross-site-collection operations with special characters +**What goes wrong:** `MoveCopyUtil.MoveFileByPath` / `CopyFileByPath` can fail silently or throw when file/folder names contain special characters (`#`, `%`, accented characters) in cross-site-collection scenarios. +**Why it happens:** SharePoint URL encoding differences between site collections. +**How to avoid:** Use `ResourcePath.FromDecodedUrl()` (not string URLs) for all MoveCopyUtil calls. For files with problematic names, fall back to download-then-upload approach. +**Warning signs:** Sporadic failures on files with non-ASCII names. + +### Pitfall 2: M365 Group member addition requires Graph permissions, not SharePoint permissions +**What goes wrong:** App registration has SharePoint permissions but not Graph permissions. Member addition silently fails or returns 403. +**Why it happens:** M365 Groups are Azure AD objects, not SharePoint objects. Adding members requires `GroupMember.ReadWrite.All` or `Group.ReadWrite.All` Graph permission. +**How to avoid:** Document required Graph permissions. Detect permission errors and surface clear message: "App registration needs Group.ReadWrite.All permission." +**Warning signs:** 403 Forbidden on Graph batch requests. + +### Pitfall 3: Site creation is asynchronous -- polling required +**What goes wrong:** `SiteCollection.CreateAsync` returns a URL but the site may not be immediately accessible. Subsequent operations (add libraries, folders) fail with 404. +**Why it happens:** SharePoint site provisioning is asynchronous. The site creation API returns before all components are ready. +**How to avoid:** After site creation, poll the site URL with retry/backoff until it responds (up to 2-3 minutes for Teams sites). The existing `ExecuteQueryRetryHelper` pattern can be adapted. +**Warning signs:** 404 errors when connecting to newly created sites. + +### Pitfall 4: CancellationToken must be checked between items, not within CSOM calls +**What goes wrong:** Cancellation appears unresponsive because CSOM `ExecuteQueryAsync` doesn't accept CancellationToken natively. +**Why it happens:** CSOM's `ExecuteQueryAsync()` has no CancellationToken overload. +**How to avoid:** Check `ct.ThrowIfCancellationRequested()` before each item in the bulk loop. For long-running single-item operations (large file transfer), check between download and upload phases. +**Warning signs:** Cancel button pressed but operation continues for a long time. + +### Pitfall 5: CSV delimiter detection -- semicolon vs comma +**What goes wrong:** European Excel exports use semicolons; North American exports use commas. Wrong delimiter means all data lands in one column. +**Why it happens:** Excel's CSV export uses the system's list separator, which varies by locale. +**How to avoid:** CsvHelper's `DetectDelimiter = true` handles this automatically. The existing PS app already does manual detection (checking for `;`). CsvHelper is more robust. +**Warning signs:** All columns merged into first column during import. + +### Pitfall 6: TeamSite creation requires at least one owner +**What goes wrong:** `SiteCollection.CreateAsync` with `TeamSiteCollectionCreationInformation` fails if no owner is specified. +**Why it happens:** M365 Groups require at least one owner. +**How to avoid:** Pre-flight CSV validation must flag rows with Type=Team and empty Owners as invalid. The PS app already checks this (line 5862). +**Warning signs:** Cryptic error from Graph API during site creation. + +### Pitfall 7: Template capture -- system lists must be excluded +**What goes wrong:** Capturing all lists includes system libraries (Style Library, Site Pages, Form Templates, etc.) that fail or create duplicates on apply. +**Why it happens:** CSOM `Web.Lists` returns all lists, including hidden system ones. +**How to avoid:** Filter out hidden lists (`list.Hidden == true`) and system templates (BaseTemplate check). The PS app filters with `!$_.Hidden` (line 936). Also exclude well-known system lists by name: "Style Library", "Form Templates", "Site Assets" (unless user-created). +**Warning signs:** Error "list already exists" when applying template. + +## Code Examples + +### File Copy/Move with MoveCopyUtil (CSOM) +```csharp +// Source: https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.movecopyutil +public async Task CopyFileAsync( + ClientContext ctx, + string sourceServerRelativeUrl, + string destServerRelativeUrl, + ConflictPolicy conflictPolicy, + IProgress progress, + CancellationToken ct) +{ + var srcPath = ResourcePath.FromDecodedUrl(sourceServerRelativeUrl); + var dstPath = ResourcePath.FromDecodedUrl(destServerRelativeUrl); + + var options = new MoveCopyOptions + { + KeepBoth = conflictPolicy == ConflictPolicy.Rename, + ResetAuthorAndCreatedOnCopy = false, // preserve metadata best-effort + }; + + bool overwrite = conflictPolicy == ConflictPolicy.Overwrite; + MoveCopyUtil.CopyFileByPath(ctx, srcPath, dstPath, overwrite, options); + await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct); +} +``` + +### Download-then-Upload Fallback (cross-site) +```csharp +// Source: existing PowerShell app pattern (Sharepoint_ToolBox.ps1 lines 5362-5427) +public async Task TransferFileViaStreamAsync( + ClientContext srcCtx, string srcServerRelUrl, + ClientContext dstCtx, string dstFolderServerRelUrl, string fileName, + bool overwrite, + IProgress progress, CancellationToken ct) +{ + // Download from source + var fileInfo = Microsoft.SharePoint.Client.File.OpenBinaryDirect(srcCtx, srcServerRelUrl); + using var memStream = new MemoryStream(); + await fileInfo.Stream.CopyToAsync(memStream, ct); + memStream.Position = 0; + + // Upload to destination + var destFolder = srcCtx.Web.GetFolderByServerRelativeUrl(dstFolderServerRelUrl); + var fileCreation = new FileCreationInformation + { + ContentStream = memStream, + Url = fileName, + Overwrite = overwrite, + }; + destFolder.Files.Add(fileCreation); + await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(dstCtx, progress, ct); +} +``` + +### Site Creation with PnP Framework +```csharp +// Source: https://pnp.github.io/pnpframework/api/PnP.Framework.Sites.SiteCollection.html +public async Task CreateTeamSiteAsync( + ClientContext adminCtx, string title, string alias, + string? description, CancellationToken ct) +{ + var creationInfo = new TeamSiteCollectionCreationInformation + { + DisplayName = title, + Alias = alias, + Description = description ?? string.Empty, + IsPublic = false, + }; + + using var siteCtx = await adminCtx.CreateSiteAsync(creationInfo); + siteCtx.Load(siteCtx.Web, w => w.Url); + await siteCtx.ExecuteQueryAsync(); + return siteCtx.Web.Url; +} + +public async Task CreateCommunicationSiteAsync( + ClientContext adminCtx, string title, string siteUrl, + string? description, CancellationToken ct) +{ + var creationInfo = new CommunicationSiteCollectionCreationInformation + { + Title = title, + Url = siteUrl, + Description = description ?? string.Empty, + }; + + using var siteCtx = await adminCtx.CreateSiteAsync(creationInfo); + siteCtx.Load(siteCtx.Web, w => w.Url); + await siteCtx.ExecuteQueryAsync(); + return siteCtx.Web.Url; +} +``` + +### Graph SDK Batch Member Addition +```csharp +// Source: https://learn.microsoft.com/en-us/graph/api/group-post-members +// Source: https://martin-machacek.com/blogPost/0b71abb2-87c9-4c88-9157-eb1ae4d6603b +public async Task AddMembersToGroupAsync( + GraphServiceClient graphClient, string groupId, + IReadOnlyList userPrincipalNames, + CancellationToken ct) +{ + // Graph PATCH can add up to 20 members at once via members@odata.bind + foreach (var batch in userPrincipalNames.Chunk(20)) + { + ct.ThrowIfCancellationRequested(); + var memberRefs = batch.Select(upn => + $"https://graph.microsoft.com/v1.0/users/{upn}").ToList(); + + var requestBody = new Group + { + AdditionalData = new Dictionary + { + { "members@odata.bind", memberRefs } + } + }; + + await graphClient.Groups[groupId].PatchAsync(requestBody, cancellationToken: ct); + } +} +``` + +### Template Capture via CSOM +```csharp +// Source: existing PowerShell pattern (Sharepoint_ToolBox.ps1 lines 909-1024) +public async Task CaptureTemplateAsync( + ClientContext ctx, SiteTemplateOptions options, + IProgress progress, CancellationToken ct) +{ + var web = ctx.Web; + ctx.Load(web, w => w.Title, w => w.Description, w => w.Language, + w => w.SiteLogoUrl, w => w.WebTemplate, w => w.ServerRelativeUrl); + await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct); + + var template = new SiteTemplate + { + Name = string.Empty, // set by caller + SourceUrl = ctx.Url, + CapturedAt = DateTime.UtcNow, + SiteType = web.WebTemplate == "GROUP" ? "Team" : "Communication", + Options = options, + }; + + if (options.CaptureSettings) + { + template.Settings = new TemplateSettings + { + Title = web.Title, + Description = web.Description, + Language = (int)web.Language, + }; + } + + if (options.CaptureLogo) + { + template.Logo = new TemplateLogo { LogoUrl = web.SiteLogoUrl }; + } + + if (options.CaptureLibraries) + { + var lists = ctx.LoadQuery(web.Lists.Where(l => !l.Hidden)); + await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct); + + foreach (var list in lists) + { + ct.ThrowIfCancellationRequested(); + // Load root folder + enumerate folders recursively + ctx.Load(list, l => l.Title, l => l.BaseType, l => l.BaseTemplate, l => l.RootFolder); + await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct); + + var folders = await EnumerateFoldersRecursiveAsync(ctx, list.RootFolder, progress, ct); + template.Libraries.Add(new TemplateLibraryInfo + { + Name = list.Title, + BaseType = list.BaseType.ToString(), + BaseTemplate = (int)list.BaseTemplate, + Folders = folders, + }); + } + } + + return template; +} +``` + +### Folder Structure Creation from CSV +```csharp +// Source: existing PowerShell pattern (Sharepoint_ToolBox.ps1 lines 6162-6193) +public async Task CreateFoldersAsync( + ClientContext ctx, string libraryTitle, + IReadOnlyList folderPaths, // sorted parent-first + IProgress progress, CancellationToken ct) +{ + var list = ctx.Web.Lists.GetByTitle(libraryTitle); + ctx.Load(list, l => l.RootFolder.ServerRelativeUrl); + await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct); + + var baseUrl = list.RootFolder.ServerRelativeUrl.TrimEnd('/'); + + for (int i = 0; i < folderPaths.Count; i++) + { + ct.ThrowIfCancellationRequested(); + progress.Report(new OperationProgress(i + 1, folderPaths.Count, + $"Creating folder {i + 1}/{folderPaths.Count}: {folderPaths[i]}")); + + // Resolve creates all intermediate folders if they don't exist + var folder = ctx.Web.Folders.Add($"{baseUrl}/{folderPaths[i]}"); + await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct); + } +} +``` + +### Recommended CSV Schemas + +**Bulk Members (bulk_add_members.csv):** +``` +GroupName,GroupUrl,Email,Role +Marketing Team,https://contoso.sharepoint.com/sites/Marketing,user1@contoso.com,Member +Marketing Team,https://contoso.sharepoint.com/sites/Marketing,manager@contoso.com,Owner +``` +Note: The existing example only has Email. Extending with GroupName/GroupUrl/Role enables bulk addition to multiple groups. + +**Bulk Sites (bulk_create_sites.csv) -- matches existing:** +``` +Name;Alias;Type;Template;Owners;Members +``` +Keep semicolon delimiter for Excel compatibility. CsvHelper's DetectDelimiter handles both. + +**Folder Structure (folder_structure.csv) -- matches existing:** +``` +Level1;Level2;Level3;Level4 +``` +Hierarchical columns. Non-empty cells build the path. Already present in /examples/. + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| PnP Sites Core (OfficeDevPnP.Core) | PnP.Framework 1.18.0 | 2021 migration | Same API surface, new namespace. Project already uses PnP.Framework | +| SP.MoveCopyUtil (JavaScript/REST) | CSOM MoveCopyUtil (C#) | Always available | Same underlying SharePoint API, different entry point | +| Microsoft Graph SDK v4 | Microsoft Graph SDK v5 | 2023 | New fluent API, BatchRequestContentCollection, breaking namespace changes | +| Manual Graph HTTP calls | Graph SDK batch with auto-split | v5+ | SDK handles 20-per-batch splitting automatically | + +**Deprecated/outdated:** +- PnP Sites Core (`OfficeDevPnP.Core`): Replaced by PnP.Framework. Do not reference the old package. +- Graph SDK v4 `BatchRequestContent`: Replaced by v5 `BatchRequestContentCollection` which auto-splits beyond 20 requests. + +## Open Questions + +1. **Graph SDK authentication integration with MSAL** + - What we know: The project uses MsalClientFactory to create PublicClientApplication instances. Graph SDK needs a TokenCredentialProvider. + - What's unclear: Exact wiring to share the same MSAL token cache between PnP Framework and Graph SDK. + - Recommendation: Create a `GraphClientFactory` that obtains tokens from the same MsalClientFactory. Use `DelegateAuthenticationProvider` or `BaseBearerTokenAuthenticationProvider` wrapping the existing PCA. Research this during implementation -- the token scopes differ (Graph needs `https://graph.microsoft.com/.default`, PnP uses `https://{tenant}.sharepoint.com/.default`). + +2. **MoveCopyUtil vs download-then-upload for cross-site-collection** + - What we know: MoveCopyUtil works within the same tenant. The PS app uses download-then-upload. + - What's unclear: Whether MoveCopyUtil handles all cross-site-collection scenarios reliably (special characters, large files). + - Recommendation: Implement MoveCopyUtil as primary approach (simpler, server-side). Fall back to download-then-upload if MoveCopyUtil fails. The fallback is proven in the PS app. + +3. **Bulk member CSV schema -- Group identification** + - What we know: The current example CSV only has Email column. For bulk addition to *multiple* groups, we need group identification. + - What's unclear: Whether users want to add to one selected group (UI picker) or multiple groups (CSV column). + - Recommendation: Support both -- UI group picker for single-group scenario, CSV with GroupUrl column for multi-group. The CSV schema is Claude's discretion per CONTEXT.md. + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | xunit 2.9.3 + Moq 4.20.72 | +| Config file | SharepointToolbox.Tests/SharepointToolbox.Tests.csproj | +| Quick run command | `dotnet test SharepointToolbox.Tests --filter "Category!=Integration" --no-build -q` | +| Full suite command | `dotnet test SharepointToolbox.Tests --no-build` | + +### Phase Requirements to Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| BULK-01 | File transfer service handles copy/move/conflict policies | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~FileTransferService -x` | Wave 0 | +| BULK-02 | Bulk member service processes CSV rows, calls Graph API | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~BulkMemberService -x` | Wave 0 | +| BULK-03 | Bulk site service creates team/communication sites from CSV | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~BulkSiteService -x` | Wave 0 | +| BULK-04 | BulkOperationRunner stops on cancellation | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~BulkOperationRunner -x` | Wave 0 | +| BULK-05 | BulkOperationRunner collects per-item errors | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~BulkOperationRunner -x` | Wave 0 | +| TMPL-01 | Template service captures site structure correctly | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~TemplateService -x` | Wave 0 | +| TMPL-02 | Template service applies template to new site | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~TemplateService -x` | Wave 0 | +| TMPL-03 | TemplateRepository persists/loads JSON correctly | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~TemplateRepository -x` | Wave 0 | +| TMPL-04 | TemplateRepository supports CRUD operations | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~TemplateRepository -x` | Wave 0 | +| FOLD-01 | Folder structure service creates folders from parsed paths | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~FolderStructureService -x` | Wave 0 | +| FOLD-02 | Example CSVs parse correctly with CsvValidationService | unit | `dotnet test SharepointToolbox.Tests --filter FullyQualifiedName~CsvValidation -x` | Wave 0 | + +### Sampling Rate +- **Per task commit:** `dotnet test SharepointToolbox.Tests --filter "Category!=Integration" --no-build -q` +- **Per wave merge:** `dotnet test SharepointToolbox.Tests --no-build` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] `SharepointToolbox.Tests/Services/BulkOperationRunnerTests.cs` -- covers BULK-04, BULK-05 +- [ ] `SharepointToolbox.Tests/Services/CsvValidationServiceTests.cs` -- covers FOLD-02, CSV parsing +- [ ] `SharepointToolbox.Tests/Services/TemplateRepositoryTests.cs` -- covers TMPL-03, TMPL-04 +- [ ] `SharepointToolbox.Tests/Services/BulkResultCsvExportServiceTests.cs` -- covers failed-items export +- [ ] CsvHelper package added to test project if needed for test CSV generation + +## Sources + +### Primary (HIGH confidence) +- [PnP Framework API - SiteCollection](https://pnp.github.io/pnpframework/api/PnP.Framework.Sites.SiteCollection.html) - Site creation methods +- [PnP Framework API - CommunicationSiteCollectionCreationInformation](https://pnp.github.io/pnpframework/api/PnP.Framework.Sites.CommunicationSiteCollectionCreationInformation.html) - Communication site creation +- [CSOM MoveCopyUtil.CopyFileByPath](https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.movecopyutil.copyfilebypath?view=sharepoint-csom) - File copy API +- [CSOM MoveCopyUtil.MoveFileByPath](https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.movecopyutil.movefilebypath?view=sharepoint-csom) - File move API +- [Microsoft Graph - Add members to group](https://learn.microsoft.com/en-us/graph/api/group-post-members?view=graph-rest-1.0) - Graph member addition +- [CsvHelper official site](https://joshclose.github.io/CsvHelper/) - CSV parsing library +- [CsvHelper NuGet](https://www.nuget.org/packages/csvhelper/) - Version 33.1.0 + +### Secondary (MEDIUM confidence) +- [Provisioning modern team sites programmatically](https://learn.microsoft.com/en-us/sharepoint/dev/solution-guidance/modern-experience-customizations-provisioning-sites) - Site creation patterns +- [PnP Core SDK - Copy/Move content](https://pnp.github.io/pnpcore/using-the-sdk/sites-copymovecontent.html) - CreateCopyJobs pattern documentation +- [Graph SDK batch member addition example](https://martin-machacek.com/blogPost/0b71abb2-87c9-4c88-9157-eb1ae4d6603b) - Batch API usage +- Existing PowerShell app (Sharepoint_ToolBox.ps1) - Proven patterns for all operations + +### Tertiary (LOW confidence) +- Graph SDK authentication integration with existing MSAL setup - needs validation during implementation + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - PnP.Framework already in use; CsvHelper and Graph SDK are de facto standards with official documentation +- Architecture: HIGH - Patterns directly port from working PowerShell app; BulkOperationRunner is a well-understood generic pattern +- Pitfalls: HIGH - Documented from real-world experience in the PS app and Microsoft issue trackers +- Template capture/apply: MEDIUM - Manual CSOM approach is straightforward but may hit edge cases with specific site types or regional settings +- Graph SDK auth wiring: LOW - Needs validation; sharing MSAL tokens between PnP and Graph SDK has nuances + +**Research date:** 2026-04-03 +**Valid until:** 2026-05-03 (stable APIs, PnP Framework release cycle is quarterly)