DESCRIPTION: SharePoint Online is, as its name suggests, Microsoft’s online version of SharePoint on-premise. It provides much of the infrastructure and features available in SharePoint on-premise in a cloud based platform. The Administrator Account version of the SharePoint Online system must be used for account mapping. A global admin user from the tenant is required for the oauth of that system.
RECOMMENDED POPS: Azure.
Identify Paths, Folder and File Names That Are Too Long
Two of the most common reasons for file transfer errors during SharePoint migrations are ‘Name too long’ and ‘Path too long.’ SharePoint has a limit of 255 characters for any file or folder, and a limit of 400 characters for the full URL path to the item. The URL includes the full SharePoint site name. So if the path for your document on SharePoint is this:
The path length calculation is based on this string:
Note that characters such as spaces, slashes, ampersands and unicode characters will all be converted in URL encoding, and the resulting encoding for a single character may be several characters in length.
If you have not named your SharePoint tenant before starting the migration, it is immensely helpful to give it a short name to minimize the incidence of path length errors. If you are moving your data to a specific folder on SharePoint, keeping the folder name short will likewise minimize errors and the need to rename paths to a smaller length.
Running a simulation prior to the actual data transfer will identify paths, folder names and file names that are too long for SharePoint. You can then adjust the paths prior to running the actual migration. Path renaming done after the map is generated can result in mapping errors and other issues with your migration, so re-running the map after renaming is recommended. Typically, data that has paths too long is clustered into a few folders, and is reasonably easy to address. Contact email@example.com for advice on whether you need to re-run a map. Simulation will also detect files greater than 15GB, which is the limit for Sharepoint.
You can get a rough idea of paths that are too long by running an analysis at the very outset of the project and then running a (LEN) function in Excel for all of the paths in the analysis report. Any character count that is > ~250 characters or so is in the risk zone for path too long errors.
Many OneNote notebooks, and the associated items such as Notebook packages, may not transfer from or to SharePoint due to the facts that they are highly dependent on the ecosystem in which they were created, and they do not always have well defined properties as files. If they do transfer, they may not open in the same state as on the source, for reasons independent of CFP. Please make other provisions to transfer these items.
Global Admin Selection
Microsoft’s api makes the oauth user an owner for all data transferred to OneDrive and SharePoint sites. For those users who do not want the oauth to have this access, create a global admin specifically for the migration. Once the migration is complete, delete the oauth user, and they will be concurrently deleted as an owner for all migrated content.
Email Notifications for Shared Items
INTERNAL SHAREES: Cloud FastPath suppresses sharing notifications automatically for internal sharees.
EXTERNAL SHAREES: External sharees will receive an email notification when an item is shared with them on a given tenant. The email is a security measure to verify the user’s identity, but it also is a means to provide access to the shared item(s). Once the external sharee clicks on a link in the email to accept the share, they will be registered with the tenant as an external user with a user id on that tenant. At that point, all further emails to that user will be suppressed.
A very helpful article in understanding some of the ACL limits in SPO/ODFB is https://docs.microsoft.com/en-us/office365/servicedescriptions/sharepoint-online-service-description/sharepoint-online-limits#items-in-lists-and-libraries. There are two types of ACL limits on Microsoft platforms:
- ACL quotas
- Number of items that have a given ACL
SharePoint Online has a limit of 50,000 ACLs per document library. For best performance on SharePoint after migration completes, Microsoft recommends that you stay below 5000 ACLs, and, in many instances, a fraction of this number. The number of ACLs that will be applied by CFP is roughly equivalent to the number of lines in the Paths tab of the spreadsheet – but the specifics are:
- each item (file or folder) that is shared counts as one ACL, regardless of the number of users and groups that are sharees for that item
- if a folder is shared, that counts as one ACL, but any child items that inherit permissions from the folder do not count against the ACL quota
- you can have a maximum of 5000 users and groups that share a given item
If a map has more than 50K lines for a given document library, you can use the ‘Ignore ACLs Below This Depth’ and ‘Ignore Permissions on Files’ settings in the Settings page of the APM map wizard. These options are not available for all data sources, and using these options may result in giving users unintended access to data, so it’s important to understand how they work and what the full ramifications are for using them. Note that after you change the settings, you can just click the REGENERATE Button on the Define Maps screen to just recrunch the data for the map job that has already been run, and create a smaller map. You do not have to launch a new map generation job.
Be aware that if users are applying ACLs to data in a given account by mechanisms other than CFP, that counts toward the total number of ACLs. Consult your Microsoft representative with any questions about the recommended number of ACLs on each SharePoint subsite and document library.
Number of Items That Have a Given ACL
The maximum number of items that can have a given ACL is 100,000. What that means is that if you have a folder that contains more than 100,000 total files and folders (total items, and not just the immediate children), adding an ACL to the top level folder will fail because it exceeds Microsoft’s limit. Moreover, for SharePoint sites, where the site owners, members and visitors are inherited down to the doc library contents, this can cause issues with any folder in the doc library that contains more than 100,000 items.
If you identify a folder that has more than 100,000 items, here are some things you can do to address the limitation.
- Long term, the best strategy is to break the folder up into smaller pieces, and migrate it as a set of smaller folders. firstname.lastname@example.org can show you the best way to do this. This has the added benefit of decreasing load times for the folder contents in the SPO web app once migration has completed.
- Deselect the Advanced Option “Keep Parent Permissions” to prevent the site permissions from inheriting down to the folder. You’ll need to add those permissions back for individual subfolders of the main folder so that the site users have access to it – and make sure that those subfolders each have less than 100,000 items. Move the data over in one pass, and apply the ACLs in a separate pass.
- Keep sharing strategies as simple as possible. Reduced permissions on large subfolders such as these – where one or two items have reduced permissions and the other items have the same permissions as the parent – can result in exceeding the 50,000 total ACL quota.
Rate Limiting, and How to Avoid It
Rate limiting is a mechanism that most cloud services apply to ensure there are adequate system resources for the web app they provide to their end users, and to ensure that data is protected from malicious acts such as DOS attacks. Unfortunately, it can significantly slow down a migration. Some steps you can take to mitigate rate limiting are:
- Before migration starts, notify your Tervela sales rep if the migration is more than 10TB. Tervela sales will work with Microsoft to ensure that your SharePoint account is fully provisioned with all of the resources it needs for the migration. Tervela sales will need the tenant name and tenant ID to forward to Microsoft.
- Run migrations after normal business hours for where your SharePoint data is stored. If the data is stored in the Pacific time zone, run jobs after 6PM PT and on weekends.
- If the migration is large, split it up into several jobs. Make sure that each job has an SPO system with a different oauth user. That means that if you want to run five concurrent jobs, you will need five global admin users. For best results, you will also need five different oauth users for the source, so you don’t get rate limiting from the source.
- Ensure that no other backup utilities, syncs, DLPs, migration tools, or anything else is running on your site that is imposing a large request load on the site.
- If you are seeing retry rates that are greater than 30% on your job, contact Tervela. In some cases, we will ask you to file a ticket with Microsoft support to investigate. For fastest service, send the ticket number to email@example.com. Following up on the ticket is still the customer’s responsibility, but we can sometimes expedite resolution if we know the ticket number.
- Batch retries (i.e., the “JobImportant” errors seen in the event log) are Microsoft errors uploading content from Azure to Sharepoint. They are entirely separate from the retry rates seen in the job metrics balloon, which typically are due to 429 errors. Batch retries happen after CFP has transferred the data to Azure on the tenant, and CFP has no control over these batch retries.
SHAREPOINT ONLINE SITE URL: Enter your Sharepoint site URL. If your company is companyName, the standard URL would be https://companyName.sharepoint.com. Leave this field blank unless advised to complete it by Tervela support.
TENANT URL: Enter your O365 Tenant URL (e.g., https://companyName-admin.sharepoint.com). This field is primarily for users who have custom URLs for their tenant that deviate from the standard format. Leave this field blank unless advised to complete it by Tervela support.
SHAREPOINT ONLINE TEAM SITE NAME: [Deprecated] You may optionally select a Team Site name from the dropdown to transfer a subset of your SharePoint data. You will only see this option on Legacy jobs. New jobs will select SharePoint subsites via the file chooser.
DOCUMENT LIBRARY: [Deprecated] You must select a Document Library. If selecting a Team Site and Document Library, select the Team Site first and wait for that to validate. Then select the Document Library and wait for that to validate. You will only see this option on Legacy jobs. New jobs will select SharePoint doc libraries via the file chooser.
COMPRESSED CONTENT: This provides optimization for sending compressed files such as .zip and .tar files. Leave this checkbox deselected unless you are sending compressed files.
Use SPO Migration API: Uses the Sharepoint API rather than the OneDrive API when selected. Leave this field selected unless advised to deselect it by Tervela support.
Use SPO Migration API to Create Directories: Uses the Sharepoint API to apply user metadata to folders on OneDrive and SPO targets. It must be selected to send folder Last Modifier (“Use Last Modifier” option must be set and configured; available only on select source platforms) to OneDrive folders. If “Use Last Modifier” is not set, the “Use SPO Migration API to Create Directories” should still be selected, as it ensures that the OneDrive target account user will be listed as the last modifier of all their folders. If this option is not selected, the admin used to oauth the OneDrive system will be listed as the last modifier of all folders transferred to OneDrive, which can confuse and concern individual end users.
ON SELECTING DATA WITH THE FILE CHOOSER:
- At this time, Cloud Fastpath cannot create new teamsites or document libraries, so the only valid action for creating a new directory via the file chooser is for an actual directory within a document library.
- You must select either a subsite and doc library, a doc library off the root, or a folder within a doc library as the target destination. Sharepoint cannot write directories to the root of the site or to a subsite.
- You may select multiple sites or doc libraries for jobs where SharePoint is the source. The selected sites and doc libraries on the source will be written as directories on the target. Be aware that CFP only transfers folders and files; if you select other content, such as lists and pages, that are not actual folders and files, CFP will not transfer them.
- “Forms” is a reserved word in SPO. You cannot write to /Forms subsites. You can create – and transfer – folders called “Forms,” but CFP will convert them to the UTF equivalent name (_u0066__u006f__u0072__u006d__u0073_) on the SPO target if they are the top folder in the subsite. Folders further down – e.g., /sites/subsite/docLib/FolderA/Forms – will transfer as is.
Account Mapping Considerations
Paths are structured relative to the path defined by the Team Site and Document Library configured in the SPO system. So if you have a target SPO system with a Team Site ‘Team1’ and a Document Library of ‘Documents,’ the path defined by that system will be https://company.sharepoint.com/Team1/Shared%20Documents. If the target directory is root (/), a source folder of /My Folder will have a target path of /My Folder in the transfer report, but the full url will still be https://company.sharepoint.com/Team1/Shared%20Documents/My%20Folder.
You can add redirects in your spreadsheet to multiple Sharepoint subsites in a single mapping spreadsheet, thereby filling several subsites from a single job. You can also send data to OneDrive personal sites. Be aware of the following:
- To send data to a Sharepoint subsite, enter the full path to a document library in the subsite. The Target Path in the paths tab of your mapping spreadsheet should look like /sites/subsite/doc library or /subsite/doc library. You cannot write directly to a Microsoft subsite, so if you don’t include a doc library in the path, that portion of the job will result in a write error. Enter the global admin email in the Target Owner field.
- You do not need to make any URL encoding conversions, such as entering <span=class=”helpCode”>%20 for spaces.
- To send data to a OneDrive account, enter a Target Path in the format /personal/account_email/documents. For example, to send data to a OneDrive account firstname.lastname@example.org, enter /email@example.com/documents. Also enter the same email (e.g., firstname.lastname@example.org) as the Target Owner.
- Run a simulation to ensure that you will get the expected results from your redirects in the spreadsheet.