Sharepoint-Toolbox/.planning/milestones/v1.0-phases/04-bulk-operations-and-provisioning/04-03-PLAN.md

phase, plan, title, status, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	title	status	wave	depends_on	files_modified	autonomous	requirements	must_haves
04	03	FileTransferService Implementation	pending	1	04-01	SharepointToolbox/Services/FileTransferService.cs SharepointToolbox.Tests/Services/FileTransferServiceTests.cs	true	BULK-01 BULK-04 BULK-05	truths artifacts key_links FileTransferService copies files using CSOM MoveCopyUtil.CopyFileByPath with ResourcePath.FromDecodedUrl FileTransferService moves files using MoveCopyUtil.MoveFileByPath then deletes source only after success Conflict policy maps to MoveCopyOptions: Skip=catch-and-skip, Overwrite=overwrite:true, Rename=KeepBoth:true Recursive folder enumeration collects all files before transferring BulkOperationRunner handles per-file error reporting and cancellation Metadata preservation is best-effort (ResetAuthorAndCreatedOnCopy=false) path provides exports SharepointToolbox/Services/FileTransferService.cs CSOM file transfer with copy/move/conflict support FileTransferService from to via pattern FileTransferService.cs BulkOperationRunner.cs per-file processing delegation BulkOperationRunner.RunAsync from to via pattern FileTransferService.cs MoveCopyUtil CSOM file operations MoveCopyUtil.CopyFileByPath|MoveFileByPath

Plan 04-03: FileTransferService Implementation

Goal

Implement FileTransferService for copying and moving files/folders between SharePoint sites using CSOM MoveCopyUtil. Supports Copy/Move modes, Skip/Overwrite/Rename conflict policies, recursive folder transfer, best-effort metadata preservation, and per-file error reporting via BulkOperationRunner.

Context

IFileTransferService, TransferJob, ConflictPolicy, TransferMode, and BulkOperationRunner are defined in Plan 04-01. The service follows the established pattern: receives ClientContext as parameter, uses ExecuteQueryRetryHelper for all CSOM calls, never stores contexts.

Key CSOM APIs:

• MoveCopyUtil.CopyFileByPath(ctx, srcPath, dstPath, overwrite, options) for copy
• MoveCopyUtil.MoveFileByPath(ctx, srcPath, dstPath, overwrite, options) for move
• ResourcePath.FromDecodedUrl() required for special characters (Pitfall 1)
• MoveCopyOptions.KeepBoth = true for Rename conflict policy
• MoveCopyOptions.ResetAuthorAndCreatedOnCopy = false for metadata preservation

Tasks

Task 1: Implement FileTransferService

Files:

• SharepointToolbox/Services/FileTransferService.cs

Action:

Create FileTransferService.cs:

using System.IO;
using Microsoft.SharePoint.Client;
using Serilog;
using SharepointToolbox.Core.Models;
using SharepointToolbox.Infrastructure.Auth;

namespace SharepointToolbox.Services;

public class FileTransferService : IFileTransferService
{
    public async Task<BulkOperationSummary<string>> TransferAsync(
        ClientContext sourceCtx,
        ClientContext destCtx,
        TransferJob job,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        // 1. Enumerate files from source
        progress.Report(new OperationProgress(0, 0, "Enumerating source files..."));
        var files = await EnumerateFilesAsync(sourceCtx, job, progress, ct);

        if (files.Count == 0)
        {
            progress.Report(new OperationProgress(0, 0, "No files found to transfer."));
            return new BulkOperationSummary<string>(new List<BulkItemResult<string>>());
        }

        // 2. Build source and destination base paths
        var srcBasePath = BuildServerRelativePath(sourceCtx, job.SourceLibrary, job.SourceFolderPath);
        var dstBasePath = BuildServerRelativePath(destCtx, job.DestinationLibrary, job.DestinationFolderPath);

        // 3. Transfer each file using BulkOperationRunner
        return await BulkOperationRunner.RunAsync(
            files,
            async (fileRelUrl, idx, token) =>
            {
                // Compute destination path by replacing source base with dest base
                var relativePart = fileRelUrl;
                if (fileRelUrl.StartsWith(srcBasePath, StringComparison.OrdinalIgnoreCase))
                    relativePart = fileRelUrl.Substring(srcBasePath.Length).TrimStart('/');

                // Ensure destination folder exists
                var destFolderRelative = dstBasePath;
                var fileFolder = Path.GetDirectoryName(relativePart)?.Replace('\\', '/');
                if (!string.IsNullOrEmpty(fileFolder))
                {
                    destFolderRelative = $"{dstBasePath}/{fileFolder}";
                    await EnsureFolderAsync(destCtx, destFolderRelative, progress, token);
                }

                var fileName = Path.GetFileName(relativePart);
                var destFileUrl = $"{destFolderRelative}/{fileName}";

                await TransferSingleFileAsync(sourceCtx, destCtx, fileRelUrl, destFileUrl, job, progress, token);

                Log.Information("Transferred: {Source} -> {Dest}", fileRelUrl, destFileUrl);
            },
            progress,
            ct);
    }

    private async Task TransferSingleFileAsync(
        ClientContext sourceCtx,
        ClientContext destCtx,
        string srcFileUrl,
        string dstFileUrl,
        TransferJob job,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        var srcPath = ResourcePath.FromDecodedUrl(srcFileUrl);
        var dstPath = ResourcePath.FromDecodedUrl(dstFileUrl);

        bool overwrite = job.ConflictPolicy == ConflictPolicy.Overwrite;
        var options = new MoveCopyOptions
        {
            KeepBoth = job.ConflictPolicy == ConflictPolicy.Rename,
            ResetAuthorAndCreatedOnCopy = false, // best-effort metadata preservation
        };

        try
        {
            if (job.Mode == TransferMode.Copy)
            {
                MoveCopyUtil.CopyFileByPath(sourceCtx, srcPath, dstPath, overwrite, options);
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(sourceCtx, progress, ct);
            }
            else // Move
            {
                MoveCopyUtil.MoveFileByPath(sourceCtx, srcPath, dstPath, overwrite, options);
                await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(sourceCtx, progress, ct);
            }
        }
        catch (ServerException ex) when (job.ConflictPolicy == ConflictPolicy.Skip &&
                                          ex.Message.Contains("already exists", StringComparison.OrdinalIgnoreCase))
        {
            Log.Warning("Skipped (already exists): {File}", srcFileUrl);
        }
    }

    private async Task<IReadOnlyList<string>> EnumerateFilesAsync(
        ClientContext ctx,
        TransferJob job,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        var list = ctx.Web.Lists.GetByTitle(job.SourceLibrary);
        var rootFolder = list.RootFolder;
        ctx.Load(rootFolder, f => f.ServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        var baseFolderUrl = rootFolder.ServerRelativeUrl.TrimEnd('/');
        if (!string.IsNullOrEmpty(job.SourceFolderPath))
            baseFolderUrl = $"{baseFolderUrl}/{job.SourceFolderPath.TrimStart('/')}";

        var folder = ctx.Web.GetFolderByServerRelativeUrl(baseFolderUrl);
        var files = new List<string>();
        await CollectFilesRecursiveAsync(ctx, folder, files, progress, ct);
        return files;
    }

    private async Task CollectFilesRecursiveAsync(
        ClientContext ctx,
        Folder folder,
        List<string> files,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        ct.ThrowIfCancellationRequested();

        ctx.Load(folder, f => f.Files.Include(fi => fi.ServerRelativeUrl),
                         f => f.Folders);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);

        foreach (var file in folder.Files)
        {
            files.Add(file.ServerRelativeUrl);
        }

        foreach (var subFolder in folder.Folders)
        {
            // Skip system folders
            if (subFolder.Name.StartsWith("_") || subFolder.Name == "Forms")
                continue;
            await CollectFilesRecursiveAsync(ctx, subFolder, files, progress, ct);
        }
    }

    private async Task EnsureFolderAsync(
        ClientContext ctx,
        string folderServerRelativeUrl,
        IProgress<OperationProgress> progress,
        CancellationToken ct)
    {
        try
        {
            var folder = ctx.Web.GetFolderByServerRelativeUrl(folderServerRelativeUrl);
            ctx.Load(folder, f => f.Exists);
            await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);
            if (folder.Exists) return;
        }
        catch { /* folder doesn't exist, create it */ }

        // Create folder using Folders.Add which creates intermediate folders
        ctx.Web.Folders.Add(folderServerRelativeUrl);
        await ExecuteQueryRetryHelper.ExecuteQueryRetryAsync(ctx, progress, ct);
    }

    private static string BuildServerRelativePath(ClientContext ctx, string library, string folderPath)
    {
        // Extract site-relative URL from context URL
        var uri = new Uri(ctx.Url);
        var siteRelative = uri.AbsolutePath.TrimEnd('/');
        var basePath = $"{siteRelative}/{library}";
        if (!string.IsNullOrEmpty(folderPath))
            basePath = $"{basePath}/{folderPath.TrimStart('/')}";
        return basePath;
    }
}

Verify:

dotnet build SharepointToolbox/SharepointToolbox.csproj --no-restore -q

Done: FileTransferService compiles. Implements copy/move via MoveCopyUtil with all three conflict policies, recursive folder enumeration, folder auto-creation at destination, best-effort metadata preservation, per-file error handling via BulkOperationRunner, and cancellation checking between files.

Task 2: Create FileTransferService unit tests

Files:

• SharepointToolbox.Tests/Services/FileTransferServiceTests.cs

Action:

Create FileTransferServiceTests.cs. Since CSOM classes (ClientContext, MoveCopyUtil) cannot be mocked directly, these tests verify the service compiles and its helper logic. Integration testing requires a real SharePoint tenant. Mark integration-dependent tests with Skip.

using SharepointToolbox.Core.Models;
using SharepointToolbox.Services;

namespace SharepointToolbox.Tests.Services;

public class FileTransferServiceTests
{
    [Fact]
    public void FileTransferService_Implements_IFileTransferService()
    {
        var service = new FileTransferService();
        Assert.IsAssignableFrom<IFileTransferService>(service);
    }

    [Fact]
    public void TransferJob_DefaultValues_AreCorrect()
    {
        var job = new TransferJob();
        Assert.Equal(TransferMode.Copy, job.Mode);
        Assert.Equal(ConflictPolicy.Skip, job.ConflictPolicy);
    }

    [Fact]
    public void ConflictPolicy_HasAllValues()
    {
        Assert.Equal(3, Enum.GetValues<ConflictPolicy>().Length);
        Assert.Contains(ConflictPolicy.Skip, Enum.GetValues<ConflictPolicy>());
        Assert.Contains(ConflictPolicy.Overwrite, Enum.GetValues<ConflictPolicy>());
        Assert.Contains(ConflictPolicy.Rename, Enum.GetValues<ConflictPolicy>());
    }

    [Fact]
    public void TransferMode_HasAllValues()
    {
        Assert.Equal(2, Enum.GetValues<TransferMode>().Length);
        Assert.Contains(TransferMode.Copy, Enum.GetValues<TransferMode>());
        Assert.Contains(TransferMode.Move, Enum.GetValues<TransferMode>());
    }

    [Fact(Skip = "Requires live SharePoint tenant")]
    public async Task TransferAsync_CopyMode_CopiesFiles()
    {
        // Integration test — needs real ClientContext
    }

    [Fact(Skip = "Requires live SharePoint tenant")]
    public async Task TransferAsync_MoveMode_DeletesSourceAfterCopy()
    {
        // Integration test — needs real ClientContext
    }

    [Fact(Skip = "Requires live SharePoint tenant")]
    public async Task TransferAsync_SkipConflict_DoesNotOverwrite()
    {
        // Integration test — needs real ClientContext
    }
}

Verify:

dotnet build SharepointToolbox.slnx --no-restore -q && dotnet test SharepointToolbox.Tests --no-build --filter "FullyQualifiedName~FileTransferService" -q

Done: FileTransferService tests pass (4 pass, 3 skip). Service is fully implemented and compiles.

Commit: feat(04-03): implement FileTransferService with MoveCopyUtil and conflict policies