Sharepoint-Toolbox/.planning/milestones/v1.0-phases/04-bulk-operations-and-provisioning/04-06-PLAN.md

---
phase: 04
plan: 06
title: TemplateService + FolderStructureService Implementation
status: pending
wave: 1
depends_on:
  - 04-01
files_modified:
  - SharepointToolbox/Services/TemplateService.cs
  - SharepointToolbox/Services/FolderStructureService.cs
  - SharepointToolbox.Tests/Services/TemplateServiceTests.cs
  - SharepointToolbox.Tests/Services/FolderStructureServiceTests.cs
autonomous: true
requirements:
  - TMPL-01
  - TMPL-02
  - FOLD-01

must_haves:
  truths:
    - "TemplateService captures site libraries (non-hidden), folders (recursive), permission groups, logo URL, and settings via CSOM"
    - "TemplateService filters out hidden lists and system lists (Forms, Style Library, Form Templates)"
    - "TemplateService applies template by creating site (Team or Communication), then recreating libraries, folders, and permission groups"
    - "Template capture honors SiteTemplateOptions checkboxes (user selects what to capture)"
    - "FolderStructureService creates folders from CSV rows in parent-first order using CSOM Folder.Folders.Add"
    - "Both services use BulkOperationRunner for per-item error reporting"
  artifacts:
    - path: "SharepointToolbox/Services/TemplateService.cs"
      provides: "Site template capture and apply"
      exports: ["TemplateService"]
    - path: "SharepointToolbox/Services/FolderStructureService.cs"
      provides: "Folder creation from CSV"
      exports: ["FolderStructureService"]
  key_links:
    - from: "TemplateService.cs"
      to: "SiteTemplate.cs"
      via: "builds and returns SiteTemplate model"
      pattern: "SiteTemplate"
    - from: "TemplateService.cs"
      to: "PnP.Framework.Sites.SiteCollection"
      via: "CreateAsync for template apply"
      pattern: "CreateSiteAsync"
    - from: "FolderStructureService.cs"
      to: "BulkOperationRunner.cs"
      via: "per-folder error handling"
      pattern: "BulkOperationRunner.RunAsync"
---

# Plan 04-06: TemplateService + FolderStructureService Implementation

## Goal

Implement `TemplateService` (capture site structure via CSOM property reads, apply template by creating site and recreating structure) and `FolderStructureService` (create folder hierarchies from CSV rows). Both use manual CSOM operations (NOT PnP Provisioning Engine per research decision).

## Context

`ITemplateService`, `IFolderStructureService`, `SiteTemplate`, `SiteTemplateOptions`, `TemplateLibraryInfo`, `TemplateFolderInfo`, `TemplatePermissionGroup`, and `FolderStructureRow` are from Plan 04-01. BulkSiteService pattern for creating sites is in Plan 04-05.

Key research findings:
- Template capture reads `Web` properties, `Lists` (filter `!Hidden`), recursive `Folder` enumeration, and `SiteGroups`
- Template apply creates site first (PnP Framework), then recreates libraries + folders + groups via CSOM
- `WebTemplate == "GROUP#0"` indicates a Team site; anything else is Communication
- Must filter system lists: check `list.Hidden`, skip Forms/Style Library/Form Templates
- Folder creation uses `Web.Folders.Add(serverRelativeUrl)` which creates intermediates

## Tasks

### Task 1: Implement TemplateService

**Files:**
- `SharepointToolbox/Services/TemplateService.cs`

**Action:**

```csharp
using Microsoft.SharePoint.Client;
using PnP.Framework.Sites;
using Serilog;
using SharepointToolbox.Core.Models;
using SharepointToolbox.Infrastructure.Auth;

namespace SharepointToolbox.Services;

public class TemplateService : ITemplateService
{
    private static readonly HashSet<string> SystemListNames = new(StringComparer.OrdinalIgnoreCase)
    {
        "Style Library", "Form Templates", "Site Assets", "Site Pages",
        "Composed Looks", "Master Page Gallery", "Web Part Gallery",
        "Theme Gallery", "Solution Gallery", "List Template Gallery",
        "Converted Forms", "Customized Reports", "Content type publishing error log",
        "TaxonomyHiddenList", "appdata", "appfiles"
    };

    public async Task<SiteTemplate> CaptureTemplateAsync(
        ClientContext ctx,
        SiteTemplateOptions options,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        progress.Report(new OperationProgress(0, 0, "Loading site properties..."));

        var web = ctx.Web;
        ctx.Load(web, w => w.Title, w => w.Description, w => w.Language,
                      w => w.SiteLogoUrl, w => w.WebTemplate, w => w.Configuration,
                      w => w.ServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        var siteType = (web.WebTemplate == "GROUP" || web.WebTemplate == "GROUP#0")
            ? "Team" : "Communication";

        var template = new SiteTemplate
        {
            Name = string.Empty, // caller sets this
            SourceUrl = ctx.Url,
            CapturedAt = DateTime.UtcNow,
            SiteType = siteType,
            Options = options,
        };

        // Capture settings
        if (options.CaptureSettings)
        {
            template.Settings = new TemplateSettings
            {
                Title = web.Title,
                Description = web.Description,
                Language = (int)web.Language,
            };
        }

        // Capture logo
        if (options.CaptureLogo)
        {
            template.Logo = new TemplateLogo { LogoUrl = web.SiteLogoUrl ?? string.Empty };
        }

        // Capture libraries and folders
        if (options.CaptureLibraries || options.CaptureFolders)
        {
            progress.Report(new OperationProgress(0, 0, "Enumerating libraries..."));
            var lists = ctx.LoadQuery(web.Lists
                .Include(l => l.Title, l => l.Hidden, l => l.BaseType, l => l.BaseTemplate, l => l.RootFolder)
                .Where(l => !l.Hidden));
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

            var filteredLists = lists
                .Where(l => !SystemListNames.Contains(l.Title))
                .Where(l => l.BaseType == BaseType.DocumentLibrary || l.BaseType == BaseType.GenericList)
                .ToList();

            for (int i = 0; i < filteredLists.Count; i++)
            {
                ct.ThrowIfCancellationRequested();
                var list = filteredLists[i];
                progress.Report(new OperationProgress(i + 1, filteredLists.Count,
                    $"Capturing library: {list.Title}"));

                var libInfo = new TemplateLibraryInfo
                {
                    Name = list.Title,
                    BaseType = list.BaseType.ToString(),
                    BaseTemplate = (int)list.BaseTemplate,
                };

                if (options.CaptureFolders)
                {
                    ctx.Load(list.RootFolder, f => f.ServerRelativeUrl);
                    await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);
                    libInfo.Folders = await EnumerateFoldersRecursiveAsync(
                        ctx, list.RootFolder, string.Empty, progress, ct);
                }

                template.Libraries.Add(libInfo);
            }
        }

        // Capture permission groups
        if (options.CapturePermissionGroups)
        {
            progress.Report(new OperationProgress(0, 0, "Capturing permission groups..."));
            var groups = web.SiteGroups;
            ctx.Load(groups, gs => gs.Include(
                g => g.Title, g => g.Description));
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

            foreach (var group in groups)
            {
                ct.ThrowIfCancellationRequested();

                // Load role definitions for this group
                var roleAssignments = web.RoleAssignments;
                ctx.Load(roleAssignments, ras => ras.Include(
                    ra => ra.Member.LoginName,
                    ra => ra.Member.Title,
                    ra => ra.RoleDefinitionBindings.Include(rd => rd.Name)));
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

                var roles = new List<string>();
                foreach (var ra in roleAssignments)
                {
                    if (ra.Member.Title == group.Title)
                    {
                        foreach (var rd in ra.RoleDefinitionBindings)
                        {
                            roles.Add(rd.Name);
                        }
                    }
                }

                template.PermissionGroups.Add(new TemplatePermissionGroup
                {
                    Name = group.Title,
                    Description = group.Description ?? string.Empty,
                    RoleDefinitions = roles,
                });
            }
        }

        progress.Report(new OperationProgress(1, 1, "Template capture complete."));
        return template;
    }

    public async Task<string> ApplyTemplateAsync(
        ClientContext adminCtx,
        SiteTemplate template,
        string newSiteTitle,
        string newSiteAlias,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        // 1. Create the site
        progress.Report(new OperationProgress(0, 0, $"Creating {template.SiteType} site: {newSiteTitle}..."));
        string siteUrl;

        if (template.SiteType.Equals("Team", StringComparison.OrdinalIgnoreCase))
        {
            var info = new TeamSiteCollectionCreationInformation
            {
                DisplayName = newSiteTitle,
                Alias = newSiteAlias,
                Description = template.Settings?.Description ?? string.Empty,
                IsPublic = false,
            };
            using var siteCtx = await adminCtx.CreateSiteAsync(info);
            siteCtx.Load(siteCtx.Web, w => w.Url);
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(siteCtx, progress, ct);
            siteUrl = siteCtx.Web.Url;
        }
        else
        {
            var tenantHost = new Uri(adminCtx.Url).Host;
            var info = new CommunicationSiteCollectionCreationInformation
            {
                Title = newSiteTitle,
                Url = $"https://{tenantHost}/sites/{newSiteAlias}",
                Description = template.Settings?.Description ?? string.Empty,
            };
            using var siteCtx = await adminCtx.CreateSiteAsync(info);
            siteCtx.Load(siteCtx.Web, w => w.Url);
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(siteCtx, progress, ct);
            siteUrl = siteCtx.Web.Url;
        }

        // 2. Connect to the new site and apply template structure
        // Need a new context for the created site
        var newCtx = new ClientContext(siteUrl);
        // Copy auth cookies/token from admin context
        newCtx.Credentials = adminCtx.Credentials;

        try
        {
            // Apply libraries
            if (template.Libraries.Count > 0)
            {
                for (int i = 0; i < template.Libraries.Count; i++)
                {
                    ct.ThrowIfCancellationRequested();
                    var lib = template.Libraries[i];
                    progress.Report(new OperationProgress(i + 1, template.Libraries.Count,
                        $"Creating library: {lib.Name}"));

                    try
                    {
                        var listInfo = new ListCreationInformation
                        {
                            Title = lib.Name,
                            TemplateType = lib.BaseTemplate,
                        };
                        var newList = newCtx.Web.Lists.Add(listInfo);
                        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(newCtx, progress, ct);

                        // Create folders in the library
                        if (lib.Folders.Count > 0)
                        {
                            await CreateFoldersFromTemplateAsync(newCtx, newList, lib.Folders, progress, ct);
                        }
                    }
                    catch (Exception ex)
                    {
                        Log.Warning("Failed to create library {Name}: {Error}", lib.Name, ex.Message);
                    }
                }
            }

            // Apply permission groups
            if (template.PermissionGroups.Count > 0)
            {
                progress.Report(new OperationProgress(0, 0, "Creating permission groups..."));
                foreach (var group in template.PermissionGroups)
                {
                    ct.ThrowIfCancellationRequested();
                    try
                    {
                        var groupInfo = new GroupCreationInformation
                        {
                            Title = group.Name,
                            Description = group.Description,
                        };
                        var newGroup = newCtx.Web.SiteGroups.Add(groupInfo);

                        // Assign role definitions
                        foreach (var roleName in group.RoleDefinitions)
                        {
                            try
                            {
                                var roleDef = newCtx.Web.RoleDefinitions.GetByName(roleName);
                                var roleBindings = new RoleDefinitionBindingCollection(newCtx) { roleDef };
                                newCtx.Web.RoleAssignments.Add(newGroup, roleBindings);
                            }
                            catch (Exception ex)
                            {
                                Log.Warning("Failed to assign role {Role} to group {Group}: {Error}",
                                    roleName, group.Name, ex.Message);
                            }
                        }
                        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(newCtx, progress, ct);
                    }
                    catch (Exception ex)
                    {
                        Log.Warning("Failed to create group {Name}: {Error}", group.Name, ex.Message);
                    }
                }
            }

            // Apply logo
            if (template.Logo != null && !string.IsNullOrEmpty(template.Logo.LogoUrl))
            {
                try
                {
                    newCtx.Web.SiteLogoUrl = template.Logo.LogoUrl;
                    newCtx.Web.Update();
                    await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(newCtx, progress, ct);
                }
                catch (Exception ex)
                {
                    Log.Warning("Failed to set site logo: {Error}", ex.Message);
                }
            }
        }
        finally
        {
            newCtx.Dispose();
        }

        progress.Report(new OperationProgress(1, 1, $"Template applied. Site created at: {siteUrl}"));
        return siteUrl;
    }

    private async Task<List<TemplateFolderInfo>> EnumerateFoldersRecursiveAsync(
        ClientContext ctx,
        Folder parentFolder,
        string parentRelativePath,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        ct.ThrowIfCancellationRequested();
        var result = new List<TemplateFolderInfo>();

        ctx.Load(parentFolder, f => f.Folders.Include(sf => sf.Name, sf => sf.ServerRelativeUrl));
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        foreach (var subFolder in parentFolder.Folders)
        {
            // Skip system folders
            if (subFolder.Name.StartsWith("_") || subFolder.Name == "Forms")
                continue;

            var relativePath = string.IsNullOrEmpty(parentRelativePath)
                ? subFolder.Name
                : $"{parentRelativePath}/{subFolder.Name}";

            var folderInfo = new TemplateFolderInfo
            {
                Name = subFolder.Name,
                RelativePath = relativePath,
                Children = await EnumerateFoldersRecursiveAsync(ctx, subFolder, relativePath, progress, ct),
            };
            result.Add(folderInfo);
        }

        return result;
    }

    private static async Task CreateFoldersFromTemplateAsync(
        ClientContext ctx,
        List list,
        List<TemplateFolderInfo> folders,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        ctx.Load(list.RootFolder, f => f.ServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);
        var baseUrl = list.RootFolder.ServerRelativeUrl.TrimEnd('/');

        await CreateFoldersRecursiveAsync(ctx, baseUrl, folders, progress, ct);
    }

    private static async Task CreateFoldersRecursiveAsync(
        ClientContext ctx,
        string parentUrl,
        List<TemplateFolderInfo> folders,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        foreach (var folder in folders)
        {
            ct.ThrowIfCancellationRequested();
            try
            {
                var folderUrl = $"{parentUrl}/{folder.Name}";
                ctx.Web.Folders.Add(folderUrl);
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

                if (folder.Children.Count > 0)
                {
                    await CreateFoldersRecursiveAsync(ctx, folderUrl, folder.Children, progress, ct);
                }
            }
            catch (Exception ex)
            {
                Log.Warning("Failed to create folder {Path}: {Error}", folder.RelativePath, ex.Message);
            }
        }
    }
}
```

**Verify:**
```bash
dotnet build SharepointToolbox/SharepointToolbox.csproj --no-restore -q
```

**Done:** TemplateService compiles. Captures site structure (libraries, folders, permission groups, logo, settings) respecting SiteTemplateOptions checkboxes. Applies template by creating site + recreating structure. System lists filtered out.

### Task 2: Implement FolderStructureService + unit tests

**Files:**
- `SharepointToolbox/Services/FolderStructureService.cs`
- `SharepointToolbox.Tests/Services/TemplateServiceTests.cs`
- `SharepointToolbox.Tests/Services/FolderStructureServiceTests.cs`

**Action:**

1. Create `FolderStructureService.cs`:
```csharp
using Microsoft.SharePoint.Client;
using Serilog;
using SharepointToolbox.Core.Models;
using SharepointToolbox.Infrastructure.Auth;

namespace SharepointToolbox.Services;

public class FolderStructureService : IFolderStructureService
{
    public async Task<BulkOperationSummary<string>> CreateFoldersAsync(
        ClientContext ctx,
        string libraryTitle,
        IReadOnlyList<FolderStructureRow> rows,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        // Get library root folder URL
        var list = ctx.Web.Lists.GetByTitle(libraryTitle);
        ctx.Load(list, l => l.RootFolder.ServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        var baseUrl = list.RootFolder.ServerRelativeUrl.TrimEnd('/');

        // Build unique folder paths from CSV rows, sorted parent-first
        var folderPaths = BuildUniquePaths(rows);

        return await BulkOperationRunner.RunAsync(
            folderPaths,
            async (path, idx, token) =>
            {
                var fullPath = $"{baseUrl}/{path}";
                ctx.Web.Folders.Add(fullPath);
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, token);
                Log.Information("Created folder: {Path}", fullPath);
            },
            progress,
            ct);
    }

    /// <summary>
    /// Builds unique folder paths from CSV rows, sorted parent-first to ensure
    /// parent folders are created before children.
    /// </summary>
    internal static IReadOnlyList<string> BuildUniquePaths(IReadOnlyList<FolderStructureRow> rows)
    {
        var paths = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

        foreach (var row in rows)
        {
            var parts = new[] { row.Level1, row.Level2, row.Level3, row.Level4 }
                .Where(s => !string.IsNullOrWhiteSpace(s))
                .ToArray();

            // Add each level as a path (e.g., "Admin", "Admin/HR", "Admin/HR/Contracts")
            var current = string.Empty;
            foreach (var part in parts)
            {
                current = string.IsNullOrEmpty(current) ? part.Trim() : $"{current}/{part.Trim()}";
                paths.Add(current);
            }
        }

        // Sort by depth (fewer slashes first) to ensure parent-first ordering
        return paths
            .OrderBy(p => p.Count(c => c == '/'))
            .ThenBy(p => p, StringComparer.OrdinalIgnoreCase)
            .ToList();
    }
}
```

2. Create `FolderStructureServiceTests.cs`:
```csharp
using SharepointToolbox.Core.Models;
using SharepointToolbox.Services;

namespace SharepointToolbox.Tests.Services;

public class FolderStructureServiceTests
{
    [Fact]
    public void FolderStructureService_Implements_IFolderStructureService()
    {
        Assert.True(typeof(IFolderStructureService).IsAssignableFrom(typeof(FolderStructureService)));
    }

    [Fact]
    public void BuildUniquePaths_FromExampleCsv_ReturnsParentFirst()
    {
        var rows = new List<FolderStructureRow>
        {
            new() { Level1 = "Administration", Level2 = "Comptabilite", Level3 = "Factures" },
            new() { Level1 = "Administration", Level2 = "Comptabilite", Level3 = "Bilans" },
            new() { Level1 = "Administration", Level2 = "Ressources Humaines" },
            new() { Level1 = "Projets", Level2 = "Projet Alpha", Level3 = "Documents" },
        };

        var paths = FolderStructureService.BuildUniquePaths(rows);

        // Should contain unique paths, parent-first
        Assert.Contains("Administration", paths);
        Assert.Contains("Administration/Comptabilite", paths);
        Assert.Contains("Administration/Comptabilite/Factures", paths);
        Assert.Contains("Administration/Comptabilite/Bilans", paths);
        Assert.Contains("Projets", paths);
        Assert.Contains("Projets/Projet Alpha", paths);

        // Parent-first: "Administration" before "Administration/Comptabilite"
        var adminIdx = paths.ToList().IndexOf("Administration");
        var compIdx = paths.ToList().IndexOf("Administration/Comptabilite");
        Assert.True(adminIdx < compIdx);
    }

    [Fact]
    public void BuildUniquePaths_DuplicateRows_Deduplicated()
    {
        var rows = new List<FolderStructureRow>
        {
            new() { Level1 = "A", Level2 = "B" },
            new() { Level1 = "A", Level2 = "B" },
            new() { Level1 = "A", Level2 = "C" },
        };

        var paths = FolderStructureService.BuildUniquePaths(rows);

        Assert.Equal(4, paths.Count); // A, A/B, A/C + dedup
    }

    [Fact]
    public void BuildUniquePaths_EmptyLevels_StopsAtLastNonEmpty()
    {
        var rows = new List<FolderStructureRow>
        {
            new() { Level1 = "Root", Level2 = "", Level3 = "", Level4 = "" },
        };

        var paths = FolderStructureService.BuildUniquePaths(rows);

        Assert.Single(paths);
        Assert.Equal("Root", paths[0]);
    }

    [Fact]
    public void FolderStructureRow_BuildPath_ReturnsCorrectPath()
    {
        var row = new FolderStructureRow
        {
            Level1 = "Admin",
            Level2 = "HR",
            Level3 = "Contracts",
            Level4 = ""
        };

        Assert.Equal("Admin/HR/Contracts", row.BuildPath());
    }

    [Fact(Skip = "Requires live SharePoint tenant")]
    public async Task CreateFoldersAsync_ValidRows_CreatesFolders()
    {
    }
}
```

3. Create `TemplateServiceTests.cs`:
```csharp
using SharepointToolbox.Core.Models;
using SharepointToolbox.Services;

namespace SharepointToolbox.Tests.Services;

public class TemplateServiceTests
{
    [Fact]
    public void TemplateService_Implements_ITemplateService()
    {
        Assert.True(typeof(ITemplateService).IsAssignableFrom(typeof(TemplateService)));
    }

    [Fact]
    public void SiteTemplate_DefaultValues_AreCorrect()
    {
        var template = new SiteTemplate();
        Assert.NotNull(template.Id);
        Assert.NotEmpty(template.Id);
        Assert.NotNull(template.Libraries);
        Assert.Empty(template.Libraries);
        Assert.NotNull(template.PermissionGroups);
        Assert.Empty(template.PermissionGroups);
        Assert.NotNull(template.Options);
    }

    [Fact]
    public void SiteTemplateOptions_AllDefaultTrue()
    {
        var opts = new SiteTemplateOptions();
        Assert.True(opts.CaptureLibraries);
        Assert.True(opts.CaptureFolders);
        Assert.True(opts.CapturePermissionGroups);
        Assert.True(opts.CaptureLogo);
        Assert.True(opts.CaptureSettings);
    }

    [Fact(Skip = "Requires live SharePoint tenant")]
    public async Task CaptureTemplateAsync_CapturesLibrariesAndFolders()
    {
    }

    [Fact(Skip = "Requires live SharePoint admin context")]
    public async Task ApplyTemplateAsync_CreatesTeamSiteWithStructure()
    {
    }
}
```

**Verify:**
```bash
dotnet build SharepointToolbox.slnx --no-restore -q && dotnet test SharepointToolbox.Tests --no-build --filter "FullyQualifiedName~FolderStructureService|FullyQualifiedName~TemplateService" -q
```

**Done:** FolderStructureService tests pass (5 pass, 1 skip). TemplateService tests pass (3 pass, 2 skip). Both services compile and the BuildUniquePaths logic is verified with parent-first ordering.

**Commit:** `feat(04-06): implement TemplateService and FolderStructureService`