Google Drive (Admin)

Description:  Google Drive is a cloud provider operating from drive.google.com.  To authorize the Google Drive Admin system, you will generate a .json file in your Google account, which is uploaded to CFP. The Google Drive (Administrator/Service Account) system is required for account mapping (in which data from multiple accounts on a source is transferred to corresponding accounts on the target); it can also be used to send data to Google Team/Shared Drives in non account mapping jobs. You must have Google admin credentials to use this system.

Preparing Your Google Account:

The five steps to make your Google Drive account CFP ready are:

  1. Create a project
  2. Create a service account for the project, and generate the json file with your key pair
  3. Enable the requisite API’s for the project
  4. Enable the project service account
  5. Request quota increase from Google Drive to reduce rate limiting

Once you have set up your Google Drive account, the following information and tips will help you set up your job.

Configure System:

Enter your admin email into the Administrator Email field.  Several additional fields will display:

Select Choose File to upload the json file you created earlier.  Once it’s uploaded, the Service Account Client Email and Private Key fields will fill with the masked  key pair from your json file:

About Google Docs

Google has many pseudo files, such as Google Docs, that are really just data stored in a database, and then rendered as files in the Google Drive UI.  The Google Drive adapter will automatically export and transfer docx, xlsx, pptx and svg files from Google Docs, Sheets, Slides and Draw “files” respectively.  Other Google pseudo files, such as sites and forms, will not be exported and will not be transferred.

Google Drive supports files and folders with duplicate names. The first of these will be transferred over to the target with its original name (e.g., “duplicate.txt”). Subsequent duplicates will be transferred with the file/folder name plus an appended id (e.g., “duplicate (ABC123).txt”).

Google docs, sheets, slides and draw files may be transferred directly from one Google account to another for account mapping jobs.  When you build the job in the APM wizard, do the following to maintain Google docs in their native state:

  1. Add the source Google Drive system.
  2. Add the target Google Drive system.  As soon as you add the Google Drive Target system, there will be a Migrate Google Docs checkbox on the Target System screen.  Leave this box deselected if you want to migrate Google docs as Google docs, but also migrate .docx files as .docx files. Selecting this checkbox will also convert .docx files on the source to Google docs on the target.
  3. Go back to the Source System screen and you will now see a Migrate Google Docs checkbox on this screen as well.  Select the Migrate Google Docs checkbox on the source.
  4. Continue to configure your migration as usual.

NOTE:  Do not select the Migrate Google Docs checkbox unless you are running a Google to Google migration. Only select the Retrieve Google Docs Sizes option if you have been instructed to do so by Tervela support.

Selecting Google Data with the File Chooser

My Drives

If you select multiple Google source accounts for transfer, only the contents of each person’s My Drive will be transferred.  Team Drives will not be transferred.  If you are selecting data from a single account, you may select /My Drive to select the root of the account.

Team/Shared Drives

You must select a specific Team Drive folder for the job to validate.  The oauth user must have Full access to a Team Drive folder to transfer data to it, or View access to transfer data from it.  Simply selecting /Team Drive will cause the job to fail validation when you try to run it.

Google recently changed the name of Team Drives to Shared Drives. While ‘Team Drives’ has been deprecated, the Google API still supports it, and ‘Team Drive’ is the terminology supported throughout CFP.

If you want to transfer data to multiple Team Drive folders in a single job, you can do so by selecting any folder on Google Drive as the target, and then use the mapping spreadsheet to apply redirects for the different Team Drive folders:

SOURCE PATH TARGET OWNER TARGET PATH RENAME? (Y/N)
/Projects/Marketing admin@yourCompany.com /Team Drives/Marketing Y
/Projects/Engineering admin@yourCompany.com /Team Drives/Engineering Y

At present, Team Drive folders must be created directly on Google Drive.  You can apply permissions to the various Team Drive folders by entering them in the Target Paths tab.  Specify the Google admin as the owner and the full path – i.e., /Team Drives/Folder name – for Target Owner and Target Path respectively.  You can then add permissions to the desired columns.

If you do not select Google source or target that begins with /My Drive or /Team Drives/folder, the job will not validate.

When adding redirects in the Paths tab for Team/Shared Drives, use ‘/Team Drives/[team folder]’ and not the newer terminology, ‘/Shared Drives/[team folder].’

Mapping Roles

My Drives and Team Drives support different mapping roles for Google targets.  Below is a table of the roles listed in the Paths / Path Conflicts tabs of the spreadsheet, the resulting permissions in Google Drive, and where they are supported (i.e., My Drives, Team Drives, or both).

Be aware that for account mapping jobs going to a Google Drive target, the mapping spreadsheet will list all of the permissions for the selected data.  However, folder permissions are not supported for Google Team Drive targets.  If it is important to preserve permissions, send the data to a personal drive (/My Drive) instead.

SPREADSHEET ROLE GOOGLE DRIVE ROLE NOTES
ORGANIZER Team Drive top level folders:
Full access
Supported for Team Drive top level folders only.
OWNER Team Drive top level folders:
OwnerMy Drive folders and files:
Owner
WRITER Team Drive top level folders only:
Edit accessTeam Drive files:
Can editMy Drive folders:
Organize, add & editMy Drive files:
Can edit
Not supported for Team Drive folders except for top level
COMMENTER Team Drive top level folders:
Comment accessTeam Drive files:
CommentMy Drive files:
Can comment
Not supported for Team Drive folders except top level

Not supported for My Drive folders

READER Team Drive top level folders:
View accessTeam Drive files:
ViewMy Drive folders:
View onlyMy Drive files:
Can view
Not supported for Team Drive folders except top level

Account Mapping Target Path Configuration – Google Drive Sources

For multi-user account mapping jobs that have a Google Drive source, the contents of /My Drive will be selected for each account.  This results in a target path of /target_destdir/My Drive/[my drive content].  For example, if you specify a target folder of “Files from Google,’ and you have source folder on Google Drive called /My Drive/My Documents, the resulting target path will be “/Files from Google/My Drive/My Documents.”

To eliminate ‘My Drive’ from all target paths, enter a line in the paths tab of your mapping spreadsheet for each account selected for transfer, as shown here:

SOURCE OWNER SOURCE PATH TARGET OWNER TARGET PATH RENAME? (Y/N)
Janet.Alfonso@company.com /My Drive jalfonso@company.com /My Drive/Migration Files Y
Robert.Chu@company.com /My Drive rchu@company.com /My Drive/Migration Files Y

Owners vs. Accounts

Google Drive source migrations move data explicitly by owner rather than by account. Here’s an example to show what that means:

Janet has a folder called “/Janet’s folder” in the root of her Google Drive account, and she owns it. She also owns a folder “/Janet’s folder/Janet’s child folder.” Sara owns a child folder, “/Janet’s folder/Sara’s child folder,” that also resides in Janet’s account.

“/Janet’s folder” and “/Janet’s folder/Janet’s child folder” will transfer to the mapped target account for Janet. “/Janet’s folder/Sara’s child folder” will move to the mapped target account for Sara.

The reason that CFP takes this approach is that every item in Google drive has a single owner. Transferring based on owner ensures that each item is only transferred once. The folder “/Janet’s folder/Sara’s child folder” will be listed in both Janet’s account – because she owns the parent folder – and in Sara’s account – because she owns the folder itself. So if we applied an account-centric approach, that folder would be transferred twice. This not only results in confusing duplicate data, but it also incurs unnecessary additional data charges.

Note that CFP has a very flexible mapping configuration, so that if you wanted to retain the folder structure on the target that you had on Google and transfer “/Janet’s folder/Sara’s child folder” to Janet’s target account, this can be done with redirect commands.

Forward Slashes in File and Folder Names

Forward slashes (/) are commonly used on Google Drive in file and folder names. This presents an issue when moving to other service providers, as the forward slash character is not only invalid, but reserved for use as a delimiter of directories and files in paths for the data. For that reason, Cloud FastPath converts forward slashes to divider slashes (U+2215) for all Google sourced migrations. You will see that character replacement in the spreadsheet, file chooser and transfer reports.

Rate Limiting

Regardless of whether your migration has Google Drive as a source, target or both, Google Drive jobs may run into rate limiting. You can request a quota increase from Google to mitigate this, but your request may be partially or completely denied. Fortunately, the errors that result from Rate Limiting – such as ENOTFOUND and ETIMEDOUT – are usually transient, meaning the affected files will transfer if the job is run again.

The following items need more requests and may increase the incidence of rate limiting:

  • Google docs > Google docs migrations (Google to Google migrations only)
  • Files and folders that are extensively shared (~25 or more ACLs per item).  In these instances, the item will typically transfer – completely – but some or all of the shares will not, and the item will be listed in transfer report as partial-success because not all shares are applied.
  • Using the Retrieve Google Docs Sizes option. This option should only be used if you are instructed to do so by Tervela support.
That said, most migrations will run fine with Google docs migrations, and many intensively shared files will transfer as well.  So the best thing to do here is to try running the migration as is.

If you do run into issues, the following measures will mitigate rate limiting:

  • Run Google Drive jobs after normal business hours, or when users are not accessing Google Drive.
  • If you are running an APM job, transfer the data in a data only job on one run, and apply the ACLs in a separate job.
  • If you have extensive sharing (>25 ACLs) on a folder or file, consider using a group for those users. One group will consolidate the ACLs of many users into one.
  • If your jobs are showing signs of rate limiting – either errors or high retry rates that slow the job down considerably – splitting the job up into smaller pieces.  Contact Tervela support for further info.
  • If you are running two concurrent jobs, it’s imperative to have different oauth users for the two jobs.
Updated on July 26, 2019

Was this article helpful?

Related Articles

Leave a Comment