How do I upload hundreds of local folders to Sharepoint via an automated process?

Darrell Porter 6 Reputation points
2024-09-23T07:45:33.9033333+00:00

Copy literally hundreds of thousands of files across thousands of folders using automation, such as Powershell using native Microsoft tools, not using OneDrive, into an existing structure on Sharepoint Online

SharePoint Development
SharePoint Development
SharePoint: A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.Development: The process of researching, productizing, and refining new or existing technologies.
2,968 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Sakina Murtuza Bagichawala 0 Reputation points
    2024-09-23T12:45:17.7+00:00

    To copy a large number of files and folders into SharePoint Online using PowerShell, you can utilize the SharePoint PnP (Patterns and Practices) PowerShell module. This method leverages native Microsoft tools and is suitable for your requirements.

    Prerequisites

    Install PnP PowerShell: Open PowerShell as an administrator and run:

    powershell
    Copy code
    Install-Module -Name PnP.PowerShell
    

    Connect to SharePoint Online: You need to connect to your SharePoint Online site. Replace the URL with your site’s URL.

    powershell
    Copy code
    Connect-PnPOnline -Url "https://yourtenant.sharepoint.com/sites/yoursite" -UseWebLogin
    

    Script to Copy Files

    Here’s a sample script that copies files from a local directory to SharePoint Online:

    powershell
    Copy code
    # Define the local directory and SharePoint destination
    $localSource = "C:\Path\To\Local\Folder"
    $sharePointDestination = "/sites/yoursite/Shared Documents/TargetFolder"
    
    # Get all files and subfolders
    $items = Get-ChildItem -Path $localSource -Recurse
    
    # Loop through each item
    foreach ($item in $items) {
        $relativePath = $item.FullName.Substring($localSource.Length + 1) -replace '\\', '/'
        $targetPath = "$sharePointDestination/$relativePath"
    
        # Check if it's a directory
        if ($item.PSIsContainer) {
            # Create the directory in SharePoint
            New-PnPFolder -Name $relativePath -Folder $sharePointDestination -List "Documents"
        } else {
            # Upload the file to SharePoint
            Add-PnPFile -Path $item.FullName -Folder $targetPath
        }
    }
    

    Key Points

    • Adjust Paths: Ensure you adjust $localSource and $sharePointDestination to match your structure.
    • Error Handling: Consider adding error handling to manage any issues that arise during the upload.
    • Performance: For very large operations, you might want to implement throttling or batch uploads to prevent overwhelming SharePoint.

    Run the Script

    1. Open PowerShell.
    2. Paste the script above, modify the paths as needed, and run it.

    This script will recursively copy all files and folders from the specified local directory to the designated SharePoint Online document library.

    Additional Considerations

    • Permissions: Ensure you have the necessary permissions to write to the target SharePoint library.
    • Large File Sizes: Be aware of SharePoint’s file size limits and adjust your approach if needed.

    This method should efficiently automate the copying of a vast number of files and folders into SharePoint Online using PowerShell.To copy a large number of files and folders into SharePoint Online using PowerShell, you can utilize the SharePoint PnP (Patterns and Practices) PowerShell module. This method leverages native Microsoft tools and is suitable for your requirements.

    Prerequisites

    Install PnP PowerShell: Open PowerShell as an administrator and run:

    powershell
    Copy code
    Install-Module -Name PnP.PowerShell
    

    Connect to SharePoint Online: You need to connect to your SharePoint Online site. Replace the URL with your site’s URL.

    powershell
    Copy code
    Connect-PnPOnline -Url "https://yourtenant.sharepoint.com/sites/yoursite" -UseWebLogin
    

    Script to Copy Files

    Here’s a sample script that copies files from a local directory to SharePoint Online:

    powershell
    Copy code
    # Define the local directory and SharePoint destination
    $localSource = "C:\Path\To\Local\Folder"
    $sharePointDestination = "/sites/yoursite/Shared Documents/TargetFolder"
    
    # Get all files and subfolders
    $items = Get-ChildItem -Path $localSource -Recurse
    
    # Loop through each item
    foreach ($item in $items) {
        $relativePath = $item.FullName.Substring($localSource.Length + 1) -replace '\\', '/'
        $targetPath = "$sharePointDestination/$relativePath"
    
        # Check if it's a directory
        if ($item.PSIsContainer) {
            # Create the directory in SharePoint
            New-PnPFolder -Name $relativePath -Folder $sharePointDestination -List "Documents"
        } else {
            # Upload the file to SharePoint
            Add-PnPFile -Path $item.FullName -Folder $targetPath
        }
    }
    

    Key Points

    • Adjust Paths: Ensure you adjust $localSource and $sharePointDestination to match your structure.
    • Error Handling: Consider adding error handling to manage any issues that arise during the upload.
    • Performance: For very large operations, you might want to implement throttling or batch uploads to prevent overwhelming SharePoint.

    Run the Script

    1. Open PowerShell.
    2. Paste the script above, modify the paths as needed, and run it.

    This script will recursively copy all files and folders from the specified local directory to the designated SharePoint Online document library.

    Additional Considerations

    • Permissions: Ensure you have the necessary permissions to write to the target SharePoint library.
    • Large File Sizes: Be aware of SharePoint’s file size limits and adjust your approach if needed.

    This method should efficiently automate the copying of a vast number of files and folders into SharePoint Online using PowerShell.

    0 comments No comments

  2. Darrell Porter 6 Reputation points
    2024-09-23T19:50:07.4266667+00:00

    Trying this solution now.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.