Sharepoint-Toolbox/.planning/milestones/v1.0-phases/04-bulk-operations-and-provisioning/04-06-PLAN.md

phase, plan, title, status, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	title	status	wave	depends_on	files_modified	autonomous	requirements	must_haves
04	06	TemplateService + FolderStructureService Implementation	pending	1	04-01	SharepointToolbox/Services/TemplateService.cs SharepointToolbox/Services/FolderStructureService.cs SharepointToolbox.Tests/Services/TemplateServiceTests.cs SharepointToolbox.Tests/Services/FolderStructureServiceTests.cs	true	TMPL-01 TMPL-02 FOLD-01	truths artifacts key_links TemplateService captures site libraries (non-hidden), folders (recursive), permission groups, logo URL, and settings via CSOM TemplateService filters out hidden lists and system lists (Forms, Style Library, Form Templates) TemplateService applies template by creating site (Team or Communication), then recreating libraries, folders, and permission groups Template capture honors SiteTemplateOptions checkboxes (user selects what to capture) FolderStructureService creates folders from CSV rows in parent-first order using CSOM Folder.Folders.Add Both services use BulkOperationRunner for per-item error reporting path provides exports SharepointToolbox/Services/TemplateService.cs Site template capture and apply TemplateService path provides exports SharepointToolbox/Services/FolderStructureService.cs Folder creation from CSV FolderStructureService from to via pattern TemplateService.cs SiteTemplate.cs builds and returns SiteTemplate model SiteTemplate from to via pattern TemplateService.cs PnP.Framework.Sites.SiteCollection CreateAsync for template apply CreateSiteAsync from to via pattern FolderStructureService.cs BulkOperationRunner.cs per-folder error handling BulkOperationRunner.RunAsync

Plan 04-06: TemplateService + FolderStructureService Implementation

Goal

Implement TemplateService (capture site structure via CSOM property reads, apply template by creating site and recreating structure) and FolderStructureService (create folder hierarchies from CSV rows). Both use manual CSOM operations (NOT PnP Provisioning Engine per research decision).

Context

ITemplateService, IFolderStructureService, SiteTemplate, SiteTemplateOptions, TemplateLibraryInfo, TemplateFolderInfo, TemplatePermissionGroup, and FolderStructureRow are from Plan 04-01. BulkSiteService pattern for creating sites is in Plan 04-05.

Key research findings:

• Template capture reads Web properties, Lists (filter !Hidden), recursive Folder enumeration, and SiteGroups
• Template apply creates site first (PnP Framework), then recreates libraries + folders + groups via CSOM
• WebTemplate == "GROUP#0" indicates a Team site; anything else is Communication
• Must filter system lists: check list.Hidden, skip Forms/Style Library/Form Templates
• Folder creation uses Web.Folders.Add(serverRelativeUrl) which creates intermediates

Tasks

Task 1: Implement TemplateService

Files:

• SharepointToolbox/Services/TemplateService.cs

Action:

using Microsoft.SharePoint.Client;
using PnP.Framework.Sites;
using Serilog;
using SharepointToolbox.Core.Models;
using SharepointToolbox.Infrastructure.Auth;

namespace SharepointToolbox.Services;

public class TemplateService : ITemplateService
{
    private static readonly HashSet<string> SystemListNames = new(StringComparer.OrdinalIgnoreCase)
    {
        "Style Library", "Form Templates", "Site Assets", "Site Pages",
        "Composed Looks", "Master Page Gallery", "Web Part Gallery",
        "Theme Gallery", "Solution Gallery", "List Template Gallery",
        "Converted Forms", "Customized Reports", "Content type publishing error log",
        "TaxonomyHiddenList", "appdata", "appfiles"
    };

    public async Task<SiteTemplate> CaptureTemplateAsync(
        ClientContext ctx,
        SiteTemplateOptions options,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        progress.Report(new OperationProgress(0, 0, "Loading site properties..."));

        var web = ctx.Web;
        ctx.Load(web, w => w.Title, w => w.Description, w => w.Language,
                      w => w.SiteLogoUrl, w => w.WebTemplate, w => w.Configuration,
                      w => w.ServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        var siteType = (web.WebTemplate == "GROUP" || web.WebTemplate == "GROUP#0")
            ? "Team" : "Communication";

        var template = new SiteTemplate
        {
            Name = string.Empty, // caller sets this
            SourceUrl = ctx.Url,
            CapturedAt = DateTime.UtcNow,
            SiteType = siteType,
            Options = options,
        };

        // Capture settings
        if (options.CaptureSettings)
        {
            template.Settings = new TemplateSettings
            {
                Title = web.Title,
                Description = web.Description,
                Language = (int)web.Language,
            };
        }

        // Capture logo
        if (options.CaptureLogo)
        {
            template.Logo = new TemplateLogo { LogoUrl = web.SiteLogoUrl ?? string.Empty };
        }

        // Capture libraries and folders
        if (options.CaptureLibraries || options.CaptureFolders)
        {
            progress.Report(new OperationProgress(0, 0, "Enumerating libraries..."));
            var lists = ctx.LoadQuery(web.Lists
                .Include(l => l.Title, l => l.Hidden, l => l.BaseType, l => l.BaseTemplate, l => l.RootFolder)
                .Where(l => !l.Hidden));
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

            var filteredLists = lists
                .Where(l => !SystemListNames.Contains(l.Title))
                .Where(l => l.BaseType == BaseType.DocumentLibrary || l.BaseType == BaseType.GenericList)
                .ToList();

            for (int i = 0; i < filteredLists.Count; i++)
            {
                ct.ThrowIfCancellationRequested();
                var list = filteredLists[i];
                progress.Report(new OperationProgress(i + 1, filteredLists.Count,
                    $"Capturing library: {list.Title}"));

                var libInfo = new TemplateLibraryInfo
                {
                    Name = list.Title,
                    BaseType = list.BaseType.ToString(),
                    BaseTemplate = (int)list.BaseTemplate,
                };

                if (options.CaptureFolders)
                {
                    ctx.Load(list.RootFolder, f => f.ServerRelativeUrl);
                    await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);
                    libInfo.Folders = await EnumerateFoldersRecursiveAsync(
                        ctx, list.RootFolder, string.Empty, progress, ct);
                }

                template.Libraries.Add(libInfo);
            }
        }

        // Capture permission groups
        if (options.CapturePermissionGroups)
        {
            progress.Report(new OperationProgress(0, 0, "Capturing permission groups..."));
            var groups = web.SiteGroups;
            ctx.Load(groups, gs => gs.Include(
                g => g.Title, g => g.Description));
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

            foreach (var group in groups)
            {
                ct.ThrowIfCancellationRequested();

                // Load role definitions for this group
                var roleAssignments = web.RoleAssignments;
                ctx.Load(roleAssignments, ras => ras.Include(
                    ra => ra.Member.LoginName,
                    ra => ra.Member.Title,
                    ra => ra.RoleDefinitionBindings.Include(rd => rd.Name)));
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

                var roles = new List<string>();
                foreach (var ra in roleAssignments)
                {
                    if (ra.Member.Title == group.Title)
                    {
                        foreach (var rd in ra.RoleDefinitionBindings)
                        {
                            roles.Add(rd.Name);
                        }
                    }
                }

                template.PermissionGroups.Add(new TemplatePermissionGroup
                {
                    Name = group.Title,
                    Description = group.Description ?? string.Empty,
                    RoleDefinitions = roles,
                });
            }
        }

        progress.Report(new OperationProgress(1, 1, "Template capture complete."));
        return template;
    }

    public async Task<string> ApplyTemplateAsync(
        ClientContext adminCtx,
        SiteTemplate template,
        string newSiteTitle,
        string newSiteAlias,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        // 1. Create the site
        progress.Report(new OperationProgress(0, 0, $"Creating {template.SiteType} site: {newSiteTitle}..."));
        string siteUrl;

        if (template.SiteType.Equals("Team", StringComparison.OrdinalIgnoreCase))
        {
            var info = new TeamSiteCollectionCreationInformation
            {
                DisplayName = newSiteTitle,
                Alias = newSiteAlias,
                Description = template.Settings?.Description ?? string.Empty,
                IsPublic = false,
            };
            using var siteCtx = await adminCtx.CreateSiteAsync(info);
            siteCtx.Load(siteCtx.Web, w => w.Url);
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(siteCtx, progress, ct);
            siteUrl = siteCtx.Web.Url;
        }
        else
        {
            var tenantHost = new Uri(adminCtx.Url).Host;
            var info = new CommunicationSiteCollectionCreationInformation
            {
                Title = newSiteTitle,
                Url = $"https://{tenantHost}/sites/{newSiteAlias}",
                Description = template.Settings?.Description ?? string.Empty,
            };
            using var siteCtx = await adminCtx.CreateSiteAsync(info);
            siteCtx.Load(siteCtx.Web, w => w.Url);
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(siteCtx, progress, ct);
            siteUrl = siteCtx.Web.Url;
        }

        // 2. Connect to the new site and apply template structure
        // Need a new context for the created site
        var newCtx = new ClientContext(siteUrl);
        // Copy auth cookies/token from admin context
        newCtx.Credentials = adminCtx.Credentials;

        try
        {
            // Apply libraries
            if (template.Libraries.Count > 0)
            {
                for (int i = 0; i < template.Libraries.Count; i++)
                {
                    ct.ThrowIfCancellationRequested();
                    var lib = template.Libraries[i];
                    progress.Report(new OperationProgress(i + 1, template.Libraries.Count,
                        $"Creating library: {lib.Name}"));

                    try
                    {
                        var listInfo = new ListCreationInformation
                        {
                            Title = lib.Name,
                            TemplateType = lib.BaseTemplate,
                        };
                        var newList = newCtx.Web.Lists.Add(listInfo);
                        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(newCtx, progress, ct);

                        // Create folders in the library
                        if (lib.Folders.Count > 0)
                        {
                            await CreateFoldersFromTemplateAsync(newCtx, newList, lib.Folders, progress, ct);
                        }
                    }
                    catch (Exception ex)
                    {
                        Log.Warning("Failed to create library {Name}: {Error}", lib.Name, ex.Message);
                    }
                }
            }

            // Apply permission groups
            if (template.PermissionGroups.Count > 0)
            {
                progress.Report(new OperationProgress(0, 0, "Creating permission groups..."));
                foreach (var group in template.PermissionGroups)
                {
                    ct.ThrowIfCancellationRequested();
                    try
                    {
                        var groupInfo = new GroupCreationInformation
                        {
                            Title = group.Name,
                            Description = group.Description,
                        };
                        var newGroup = newCtx.Web.SiteGroups.Add(groupInfo);

                        // Assign role definitions
                        foreach (var roleName in group.RoleDefinitions)
                        {
                            try
                            {
                                var roleDef = newCtx.Web.RoleDefinitions.GetByName(roleName);
                                var roleBindings = new RoleDefinitionBindingCollection(newCtx) { roleDef };
                                newCtx.Web.RoleAssignments.Add(newGroup, roleBindings);
                            }
                            catch (Exception ex)
                            {
                                Log.Warning("Failed to assign role {Role} to group {Group}: {Error}",
                                    roleName, group.Name, ex.Message);
                            }
                        }
                        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(newCtx, progress, ct);
                    }
                    catch (Exception ex)
                    {
                        Log.Warning("Failed to create group {Name}: {Error}", group.Name, ex.Message);
                    }
                }
            }

            // Apply logo
            if (template.Logo != null && !string.IsNullOrEmpty(template.Logo.LogoUrl))
            {
                try
                {
                    newCtx.Web.SiteLogoUrl = template.Logo.LogoUrl;
                    newCtx.Web.Update();
                    await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(newCtx, progress, ct);
                }
                catch (Exception ex)
                {
                    Log.Warning("Failed to set site logo: {Error}", ex.Message);
                }
            }
        }
        finally
        {
            newCtx.Dispose();
        }

        progress.Report(new OperationProgress(1, 1, $"Template applied. Site created at: {siteUrl}"));
        return siteUrl;
    }

    private async Task<List<TemplateFolderInfo>> EnumerateFoldersRecursiveAsync(
        ClientContext ctx,
        Folder parentFolder,
        string parentRelativePath,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        ct.ThrowIfCancellationRequested();
        var result = new List<TemplateFolderInfo>();

        ctx.Load(parentFolder, f => f.Folders.Include(sf => sf.Name, sf => sf.ServerRelativeUrl));
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        foreach (var subFolder in parentFolder.Folders)
        {
            // Skip system folders
            if (subFolder.Name.StartsWith("_") || subFolder.Name == "Forms")
                continue;

            var relativePath = string.IsNullOrEmpty(parentRelativePath)
                ? subFolder.Name
                : $"{parentRelativePath}/{subFolder.Name}";

            var folderInfo = new TemplateFolderInfo
            {
                Name = subFolder.Name,
                RelativePath = relativePath,
                Children = await EnumerateFoldersRecursiveAsync(ctx, subFolder, relativePath, progress, ct),
            };
            result.Add(folderInfo);
        }

        return result;
    }

    private static async Task CreateFoldersFromTemplateAsync(
        ClientContext ctx,
        List list,
        List<TemplateFolderInfo> folders,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        ctx.Load(list.RootFolder, f => f.ServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);
        var baseUrl = list.RootFolder.ServerRelativeUrl.TrimEnd('/');

        await CreateFoldersRecursiveAsync(ctx, baseUrl, folders, progress, ct);
    }

    private static async Task CreateFoldersRecursiveAsync(
        ClientContext ctx,
        string parentUrl,
        List<TemplateFolderInfo> folders,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        foreach (var folder in folders)
        {
            ct.ThrowIfCancellationRequested();
            try
            {
                var folderUrl = $"{parentUrl}/{folder.Name}";
                ctx.Web.Folders.Add(folderUrl);
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

                if (folder.Children.Count > 0)
                {
                    await CreateFoldersRecursiveAsync(ctx, folderUrl, folder.Children, progress, ct);
                }
            }
            catch (Exception ex)
            {
                Log.Warning("Failed to create folder {Path}: {Error}", folder.RelativePath, ex.Message);
            }
        }
    }
}

Verify:

dotnet build SharepointToolbox/SharepointToolbox.csproj --no-restore -q

Done: TemplateService compiles. Captures site structure (libraries, folders, permission groups, logo, settings) respecting SiteTemplateOptions checkboxes. Applies template by creating site + recreating structure. System lists filtered out.

Task 2: Implement FolderStructureService + unit tests

Files:

• SharepointToolbox/Services/FolderStructureService.cs
• SharepointToolbox.Tests/Services/TemplateServiceTests.cs
• SharepointToolbox.Tests/Services/FolderStructureServiceTests.cs

Action:

1. Create FolderStructureService.cs:

using Microsoft.SharePoint.Client;
using Serilog;
using SharepointToolbox.Core.Models;
using SharepointToolbox.Infrastructure.Auth;

namespace SharepointToolbox.Services;

public class FolderStructureService : IFolderStructureService
{
    public async Task<BulkOperationSummary<string>> CreateFoldersAsync(
        ClientContext ctx,
        string libraryTitle,
        IReadOnlyList<FolderStructureRow> rows,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        // Get library root folder URL
        var list = ctx.Web.Lists.GetByTitle(libraryTitle);
        ctx.Load(list, l => l.RootFolder.ServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        var baseUrl = list.RootFolder.ServerRelativeUrl.TrimEnd('/');

        // Build unique folder paths from CSV rows, sorted parent-first
        var folderPaths = BuildUniquePaths(rows);

        return await BulkOperationRunner.RunAsync(
            folderPaths,
            async (path, idx, token) =>
            {
                var fullPath = $"{baseUrl}/{path}";
                ctx.Web.Folders.Add(fullPath);
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, token);
                Log.Information("Created folder: {Path}", fullPath);
            },
            progress,
            ct);
    }

    /// <summary>
    /// Builds unique folder paths from CSV rows, sorted parent-first to ensure
    /// parent folders are created before children.
    /// </summary>
    internal static IReadOnlyList<string> BuildUniquePaths(IReadOnlyList<FolderStructureRow> rows)
    {
        var paths = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

        foreach (var row in rows)
        {
            var parts = new[] { row.Level1, row.Level2, row.Level3, row.Level4 }
                .Where(s => !string.IsNullOrWhiteSpace(s))
                .ToArray();

            // Add each level as a path (e.g., "Admin", "Admin/HR", "Admin/HR/Contracts")
            var current = string.Empty;
            foreach (var part in parts)
            {
                current = string.IsNullOrEmpty(current) ? part.Trim() : $"{current}/{part.Trim()}";
                paths.Add(current);
            }
        }

        // Sort by depth (fewer slashes first) to ensure parent-first ordering
        return paths
            .OrderBy(p => p.Count(c => c == '/'))
            .ThenBy(p => p, StringComparer.OrdinalIgnoreCase)
            .ToList();
    }
}

2. Create FolderStructureServiceTests.cs:

using SharepointToolbox.Core.Models;
using SharepointToolbox.Services;

namespace SharepointToolbox.Tests.Services;

public class FolderStructureServiceTests
{
    [Fact]
    public void FolderStructureService_Implements_IFolderStructureService()
    {
        Assert.True(typeof(IFolderStructureService).IsAssignableFrom(typeof(FolderStructureService)));
    }

    [Fact]
    public void BuildUniquePaths_FromExampleCsv_ReturnsParentFirst()
    {
        var rows = new List<FolderStructureRow>
        {
            new() { Level1 = "Administration", Level2 = "Comptabilite", Level3 = "Factures" },
            new() { Level1 = "Administration", Level2 = "Comptabilite", Level3 = "Bilans" },
            new() { Level1 = "Administration", Level2 = "Ressources Humaines" },
            new() { Level1 = "Projets", Level2 = "Projet Alpha", Level3 = "Documents" },
        };

        var paths = FolderStructureService.BuildUniquePaths(rows);

        // Should contain unique paths, parent-first
        Assert.Contains("Administration", paths);
        Assert.Contains("Administration/Comptabilite", paths);
        Assert.Contains("Administration/Comptabilite/Factures", paths);
        Assert.Contains("Administration/Comptabilite/Bilans", paths);
        Assert.Contains("Projets", paths);
        Assert.Contains("Projets/Projet Alpha", paths);

        // Parent-first: "Administration" before "Administration/Comptabilite"
        var adminIdx = paths.ToList().IndexOf("Administration");
        var compIdx = paths.ToList().IndexOf("Administration/Comptabilite");
        Assert.True(adminIdx < compIdx);
    }

    [Fact]
    public void BuildUniquePaths_DuplicateRows_Deduplicated()
    {
        var rows = new List<FolderStructureRow>
        {
            new() { Level1 = "A", Level2 = "B" },
            new() { Level1 = "A", Level2 = "B" },
            new() { Level1 = "A", Level2 = "C" },
        };

        var paths = FolderStructureService.BuildUniquePaths(rows);

        Assert.Equal(4, paths.Count); // A, A/B, A/C + dedup
    }

    [Fact]
    public void BuildUniquePaths_EmptyLevels_StopsAtLastNonEmpty()
    {
        var rows = new List<FolderStructureRow>
        {
            new() { Level1 = "Root", Level2 = "", Level3 = "", Level4 = "" },
        };

        var paths = FolderStructureService.BuildUniquePaths(rows);

        Assert.Single(paths);
        Assert.Equal("Root", paths[0]);
    }

    [Fact]
    public void FolderStructureRow_BuildPath_ReturnsCorrectPath()
    {
        var row = new FolderStructureRow
        {
            Level1 = "Admin",
            Level2 = "HR",
            Level3 = "Contracts",
            Level4 = ""
        };

        Assert.Equal("Admin/HR/Contracts", row.BuildPath());
    }

    [Fact(Skip = "Requires live SharePoint tenant")]
    public async Task CreateFoldersAsync_ValidRows_CreatesFolders()
    {
    }
}

3. Create TemplateServiceTests.cs:

using SharepointToolbox.Core.Models;
using SharepointToolbox.Services;

namespace SharepointToolbox.Tests.Services;

public class TemplateServiceTests
{
    [Fact]
    public void TemplateService_Implements_ITemplateService()
    {
        Assert.True(typeof(ITemplateService).IsAssignableFrom(typeof(TemplateService)));
    }

    [Fact]
    public void SiteTemplate_DefaultValues_AreCorrect()
    {
        var template = new SiteTemplate();
        Assert.NotNull(template.Id);
        Assert.NotEmpty(template.Id);
        Assert.NotNull(template.Libraries);
        Assert.Empty(template.Libraries);
        Assert.NotNull(template.PermissionGroups);
        Assert.Empty(template.PermissionGroups);
        Assert.NotNull(template.Options);
    }

    [Fact]
    public void SiteTemplateOptions_AllDefaultTrue()
    {
        var opts = new SiteTemplateOptions();
        Assert.True(opts.CaptureLibraries);
        Assert.True(opts.CaptureFolders);
        Assert.True(opts.CapturePermissionGroups);
        Assert.True(opts.CaptureLogo);
        Assert.True(opts.CaptureSettings);
    }

    [Fact(Skip = "Requires live SharePoint tenant")]
    public async Task CaptureTemplateAsync_CapturesLibrariesAndFolders()
    {
    }

    [Fact(Skip = "Requires live SharePoint admin context")]
    public async Task ApplyTemplateAsync_CreatesTeamSiteWithStructure()
    {
    }
}

Verify:

dotnet build SharepointToolbox.slnx --no-restore -q && dotnet test SharepointToolbox.Tests --no-build --filter "FullyQualifiedName~FolderStructureService|FullyQualifiedName~TemplateService" -q

Done: FolderStructureService tests pass (5 pass, 1 skip). TemplateService tests pass (3 pass, 2 skip). Both services compile and the BuildUniquePaths logic is verified with parent-first ordering.

Commit: feat(04-06): implement TemplateService and FolderStructureService