DESCRIPTION: SharePoint Online is, as its name suggests, Microsoft’s online version of SharePoint on-premise. It provides much of the infrastructure and features available in SharePoint on-premise in a cloud based platform. The Administrator Account version of the SharePoint Online system must be used for account mapping. A global admin user from the tenant is required for the oauth of that system.
RECOMMENDED POPS: Azure.
What’s New: CFP has added advanced permissions to OneDrive, and there is a new role, Restricted Viewer, that is listed in the mapping spreadsheet. Restricted Viewers have view rights but no download rights. Along with this addition, Write (Users) and Write (Groups) will now be called Contribute (Users) and Contribute (Groups) respectively to be consistent with Microsoft terminology in the Advanced Permissions settings.
Identify Paths, Folder and File Names That Are Too Long
Two of the most common reasons for file transfer errors during SharePoint migrations are ‘Name too long’ and ‘Path too long.’ SharePoint has a limit of 255 characters for any file or folder, and a limit of 400 characters for the full URL path to the item. The URL includes the full SharePoint site name. So if the path for your document on SharePoint is this:
The path length calculation is based on this string:
Note that characters such as spaces, slashes, ampersands and unicode characters will all be converted in URL encoding, and the resulting encoding for a single character may be several characters in length.
If you have not named your SharePoint tenant before starting the migration, it is immensely helpful to give it a short name to minimize the incidence of path length errors. If you are moving your data to a specific folder on SharePoint, keeping the folder name short will likewise minimize errors and the need to rename paths to a smaller length.
Running a simulation prior to the actual data transfer will identify paths, folder names and file names that are too long for SharePoint. You can then adjust the paths prior to running the actual migration. Path renaming done after the map is generated can result in mapping errors and other issues with your migration, so re-running the map after renaming is recommended. Typically, data that has paths too long is clustered into a few folders, and is reasonably easy to address. Contact firstname.lastname@example.org for advice on whether you need to re-run a map. Simulation will also detect files greater than 15GB, which is the limit for Sharepoint.
You can get a rough idea of paths that are too long by running an analysis at the very outset of the project and then running a (LEN) function in Excel for all of the paths in the analysis report. Any character count that is > ~250 characters or so is in the risk zone for path too long errors.
Many OneNote notebooks, and the associated items such as Notebook packages, may not transfer from or to SharePoint due to the facts that they are highly dependent on the ecosystem in which they were created, and they do not always have well defined properties as files. If they do transfer, they may not open in the same state as on the source, for reasons independent of CFP. Please make other provisions to transfer these items.
Email Notifications for Shared Items
INTERNAL SHAREES: Cloud FastPath suppresses sharing notifications automatically for internal sharees.
EXTERNAL SHAREES: External sharees will receive an email notification when an item is shared with them on a given tenant. The email is a security measure to verify the user’s identity, but it also is a means to provide access to the shared item(s). Once the external sharee clicks on a link in the email to accept the share, they will be registered with the tenant as an external user with a user id on that tenant. At that point, all further emails to that user will be suppressed.
SharePoint Online has a limit of 50,000 ACLs for the entire main site. For best performance on SharePoint after migration completes, Microsoft recommends that you stay below 5000 ACLs, and, in many instances, a fraction of this number. The number of ACLs that will be applied by CFP is roughly equivalent to the number of lines in the Paths tab of the spreadsheet – but the specifics are:
- each item (file or folder) that is shared counts as one ACL, regardless of the number of users and groups that are sharees for that item
- if a folder is shared, that counts as one ACL, but any child items that inherit permissions from the folder do not count against the ACL quota
If a map has more than 50K lines for a given user, you can use the ‘Ignore ACLs Below This Depth’ and ‘Ignore Permissions on Files’ settings in the Settings page of the APM map wizard. These options are not available for all data sources, and using these options may result in giving users unintended access to data, so it’s important to understand how they work and what the full ramifications are for using them. Note that after you change the settings, you can just click the REGENERATE Button on the Define Maps screen to just recrunch the data for the map job that has already been run, and create a smaller map. You do not have to launch a new map generation job.
Be aware that if users are applying ACLs to data in a given account by mechanisms other than CFP, that counts toward the total number of ACLs. Consult your Microsoft representative with any questions about the recommended number of ACLs on each SharePoint subsite.
Rate Limiting, and How to Avoid It
Rate limiting is a mechanism that most cloud services apply to ensure there are adequate system resources for the web app they provide to their end users, and to ensure that data is protected from malicious acts such as DOS attacks. Unfortunately, it can significantly slow down a migration. Some steps you can take to mitigate rate limiting are:
- Before migration starts, notify your Tervela sales rep if the migration is more than 10TB. Tervela sales will work with Microsoft to ensure that your SharePoint account is fully provisioned with all of the resources it needs for the migration. Tervela sales will need the tenant name and tenant ID to forward to Microsoft.
- Run migrations after normal business hours for where your SharePoint data is stored. If the data is stored in the Pacific time zone, run jobs after 6PM PT and on weekends.
- If the migration is large, split it up into several jobs. Make sure that each job has an SPO system with a different oauth user. That means that if you want to run five concurrent jobs, you will need five global admin users. For best results, you will also need five different oauth users for the source, so you don’t get rate limiting from the source.
- Make sure your SPO system credentials were entered after August 1, 2018. Tervela has made significant enhancements to the SPO adapter, but you will need to enter credentials after the above date to take advantage of them.
- Ensure that no other backup utilities, syncs, DLPs, migration tools, or anything else is running on your site that is imposing a large request load on the site.
- If you are seeing retry rates that are greater than 30% on your job, contact Tervela. In some cases, we will ask you to file a ticket with Microsoft support to investigate. For fastest service, send the ticket number to email@example.com. Following up on the ticket is still the customer’s responsibility, but we can sometimes expedite resolution if we know the ticket number.
- Batch retries (i.e., the “JobImportant” errors seen in the event log) are Microsoft errors uploading content from Azure to Sharepoint. They are entirely separate from the retry rates seen in the job metrics balloon, which typically are due to 429 errors. Batch retries happen after CFP has transferred the data to Azure on the tenant, and CFP has no control over these batch retries.
SHAREPOINT ONLINE SITE URL: Enter your Sharepoint site URL. If your company is companyName, the standard URL would be https://companyName.sharepoint.com. Leave this field blank unless advised to complete it by Tervela support.
TENANT URL: Enter your O365 Tenant URL (e.g., https://companyName-admin.sharepoint.com). This field is primarily for users who have custom URLs for their tenant that deviate from the standard format. Leave this field blank unless advised to complete it by Tervela support.
SHAREPOINT ONLINE TEAM SITE NAME: [Deprecated] You may optionally select a Team Site name from the dropdown to transfer a subset of your SharePoint data. You will only see this option on Legacy jobs. New jobs will select SharePoint subsites via the file chooser.
DOCUMENT LIBRARY: [Deprecated] You must select a Document Library. If selecting a Team Site and Document Library, select the Team Site first and wait for that to validate. Then select the Document Library and wait for that to validate. You will only see this option on Legacy jobs. New jobs will select SharePoint doc libraries via the file chooser.
COMPRESSED CONTENT: This provides optimization for sending compressed files such as .zip and .tar files. Leave this checkbox deselected unless you are sending compressed files.
Use SPO Migration API: Uses the Sharepoint API rather than the OneDrive API when selected. Leave this field selected unless advised to deselect it by Tervela support.
ON SELECTING DATA WITH THE FILE CHOOSER:
- At this time, Cloud Fastpath cannot create new teamsites or document libraries, so the only valid action for creating a new directory via the file chooser is for an actual directory within a document library.
- Verify that your SharePoint system oauth is dated 7/20/2018 or later.
- You must select either a subsite and doc library, a doc library off the root, or a folder within a doc library as the target destination. Sharepoint cannot write directories to the root of the site or to a subsite.
- You may select multiple doc libraries for jobs where SharePoint is the source. Be aware that CFP only transfers folders and files; if you select other content, such as lists and pages, that are not actual folders and files, CFP will not transfer them.
Account Mapping Considerations
Paths are structured relative to the path defined by the Team Site and Document Library configured in the SPO system. So if you have a target SPO system with a Team Site ‘Team1’ and a Document Library of ‘Documents,’ the path defined by that system will be https://company.sharepoint.com/Team1/Shared%20Documents. If the target directory is root (/), a source folder of /My Folder will have a target path of /My Folder in the transfer report, but the full url will still be https://company.sharepoint.com/Team1/Shared%20Documents/My%20Folder.
You can add redirects in your spreadsheet to multiple Sharepoint subsites in a single mapping spreadsheet, thereby filling several subsites from a single job. You can also send data to OneDrive personal sites. Be aware of the following:
- To send data to a Sharepoint subsite, enter the full path to a document library in the subsite. The Target Path in the paths tab of your mapping spreadsheet should look like /sites/subsite/doc library or /subsite/doc library. You cannot write directly to a Microsoft subsite, so if you don’t include a doc library in the path, that portion of the job will result in a write error.
- You do not need to make any URL encoding conversions, such as entering %20 for spaces.
- To send data to a OneDrive account, enter a Target Path in the format /personal/account_email/documents. For example, to send data to a OneDrive account firstname.lastname@example.org, enter /email@example.com/documents.
- Run a simulation to ensure that you will get the expected results from your redirects in the spreadsheet.