Regardless of whether your migration has Google Drive as a source, target or both, Google Drive jobs may run into rate limiting. You can request a quota increase from Google to mitigate this, but your request may be partially or completely denied. Fortunately, the errors that result from Rate Limiting – such as ENOTFOUND and ETIMEDOUT – are usually transient, meaning the affected files will transfer if the job is run again.
The following items need more requests and may increase the incidence of rate limiting:
- Google docs > Google docs migrations (Google to Google migrations only)
- Files and folders that are extensively shared (~25 or more ACLs per item). In these instances, the item will typically transfer – completely – but some or all of the shares will not, and the item will be listed in transfer report as partial-success because not all shares are applied.
- Using the Retrieve Google Docs Sizes option. This option should only be used if you are instructed to do so by Tervela support.
If you do run into issues, the following measures will mitigate rate limiting:
- Run Google Drive jobs after normal business hours, or when users are not accessing Google Drive.
- If you are running an APM job, transfer the data in a data only job on one run, and apply the ACLs in a separate job.
- If you have extensive sharing (>25 ACLs) on a folder or file, consider using a group for those users. One group will consolidate the ACLs of many users into one.
- If your jobs are showing signs of rate limiting – either errors or high retry rates that slow the job down considerably – splitting the job up into smaller pieces. Contact Tervela support for further info.
- For jobs going to Google Drive targets, Google has an absolute, non-configurable upload limit of 750 GB per day per user across all drives. Characteristically, job history charts in the Status tab present with file and byte progress that suddenly flatline for several hours when a user has been affected by this quota. Job progress will generally resume when the 24-hour period has lapsed, so the job does not need to be restarted. See https://support.google.com/a/answer/172541 for details. If you are sending data to Team Drives, you may replace the quota’ed target user in the job with a different one; the new user will need Manager access to the Team Drive.
- If you are running two concurrent jobs, it’s imperative to have different oauth users for the two jobs.