Microsoft Office 365
On This Page
- 1 Microsoft Office 365 Connection Overview
- 2 Create Connection | User Interface
- 3 Features and Limitations
- 3.1 Files/Folders
- 3.1.1 File and Folder Name Restrictions
- 3.1.2 Connection Pooling
- 3.1.3 Delta Job Runs
- 3.1.4 Disabled Users
- 3.1.5 HTML Files
- 3.1.6 Invalid Characters and Spaces
- 3.1.7 Job Target Paths
- 3.1.8 Language Translation
- 3.1.9 Link Remediation
- 3.1.10 Lock Events
- 3.1.11 Mapping
- 3.1.12 Metadata Mapping
- 3.1.13 OneNote Files
- 3.1.14 Permissions Preservation
- 3.1.14.1 Inherited Library Permissions
- 3.1.15 Timestamps
- 3.1.16 Version Deletes
- 3.1 Files/Folders
- 4 Corrupt Files
- 5 Platform Item Exception Errors
- 6 Transfer Content to SharePoint Online Shared Document Library | Connection URI and Path Example
- 7 Create Connection | REST API
- 8 Create Job | Impersonation with Admin Connection
- 9 Create Job | with Tenant Level Connection
Microsoft Office 365 Connection Overview
DryvIQ connections to the Microsoft Office 365 platform can be made by using an administrator account with the proper privileges to manage Office 365 configurations. The connection can be made to a single site or to a site collection/tenant root. DryvIQ creates the connection using the OAuth 2.0 flow to simplify login and connection management.
Create Connection | User Interface
Select Connections > Add connection.
Select Microsoft Office 365 as the platform on the Add connection modal.
Enter the connection information. Reference the table below for details about each field.
Select Sign in with Microsoft Office 365.
Enter the email for the account being used to create the connection and click Next. You must use an admin account with the proper privileges to manage Office 365 configurations.
Enter the password for the account and select Sign in.
You will see a "Connection test succeeded" message on the Add connection modal. (If you don't see this message, repeat the sign-in and authorization steps above. If the connection fails, verify the URL. DryvIQ cannot create the connection if the URL is incorrect.)
Select Done to finish creating the connection.
Add Connection Modal - Microsoft Office 365
Field | Description | Required |
|---|---|---|
Display as | Enter the display name for the connection. If you will be creating multiple connections, ensure the name readily identifies the connection. The name displays in the application, and you can use it to search for the connection and filter lists. If you do not add a display name, the connection will automatically be named using the URL. For example, Microsoft Office 365 (https://mycompany.sharepoint.com/). If it will be helpful for you to reference the connection by account, you should use the default name. | Optional |
URL | Enter the URL of your Office 365 account. You can find the URL by logging into Microsoft Office 365 using the account you want to use to create the connection. The URL in the address bar is the correct URL to use. It will look something like "https://mycompany.sharepoint.com/_layouts/15/viewlsts.aspx?view=14," where mycompany will be the site name of your company. You can copy and paste the full URL or only the part before "layouts" (https://mycompany-my.sharepoint.com/). DryvIQ will ignore the extra part of the URL in most cases. If the connection fails, verify the URL. DryvIQ cannot create the connection if the URL is incorrect. | Required |
Token endpoint | This setting is rare for most Microsoft connections and can be left blank. Your administrator will provide this value if needed. | Optional |
Graph API endpoint | This setting is rare for most Microsoft connections and can be left blank. Your administrator will provide this value if needed. | Optional |
External users notification | This setting indicates whether you want to send notifications to external users. The default value is No, so notifications will not be sent. If you select Yes, notifications will be sent to external users when they have been granted access to content based on permissions. | Optional |
Set Tenant Level Connection | This setting indicates whether the connection should be set at the tenant level. The default value is No, so the connection will be set based on the URL provided. If you select Yes, the connection will override the URL provided and connect to {tenant}-admin.sharepoint.com. Using a tenant-level connection allows one connection to be used to create jobs that point at different site collections. When creating a job using the Microsoft Office 365 connection, you will need to make the root an actual site collection since DryvIQ cannot create a new site. Currently, SharePoint Online does not prevent you from creating a document library (subsite) named “sites.” If you will be using a tenant-level connection to SharePoint, you must ensure your site does not contain a document library (subsite) named “sites.” Otherwise, all data for the migration will be transferred to the “sites” library rather than the top-level site. Since “sites” is a managed path in SharePoint, you will not be able to browse to this location to locate the data. | Optional |
Behavior When Deleting Items | Select the delete DryvIQ should perform when deleting items: Permanent or Soft. Soft delete is the default delete behavior; however, Permanent is the recommended behavior. A soft delete marks items as deleted. You can still access them to restore or permanently delete the items. A permanent delete removes the items. This deletion is not reversible. | Optional |
Microsoft Sign In Modal
Connection Test Succeeded
Features and Limitations
Platforms all have unique features and limitations. DryvIQ’s transfer engine manages these differences between platforms, allowing you to configure actions based on Job Policies and Behaviors. Utilize the Platform Comparison tool to see how your integration platforms may interact regarding features and limitations.
The service account you use to connect to your SharePoint Online must be a Site Collection Admin on every SharePoint site to which DryvIQ needs to migrate content.
SharePoint Tenant Connections must have full control of the SharePoint admin site ({tenant}-admin) as well. This control is granted by adding the Service Account to the SharePoint admin role in the Office 365 Admin Center (https://admin.microsoft.com).
OneDrive for Business Connections are automatically configured to the Documents library.
When configuring your job JSON, do not include "Documents" in the location path to point to the built-in documents library; you must use “/Shared Documents” for this particular built-in document library.
The correct configuration path is /FolderName.
If you will be using a tenant-level connection to SharePoint, you must ensure your site does not contain a document library (subsite) named “sites.” Otherwise, all data for the migration will be transferred to the “sites” library rather than the top-level site. Since “sites” is a managed path in SharePoint, you will not be able to browse to this location to locate the data.
Files/Folders
Office 365 has the following file/folder restrictions.
Supported Features | Unsupported Features | Other Features/Limitations |
|---|---|---|
Invalid characters: | " \ / : * ? < > | ||
File size maximum: 250 GB Microsoft limit | ||
Segment path length: N/A | ||
Path length maximum: 400 | ||
(See Mapping below.) | Maximum number of files per folder: 5000 | |
Maximum Enterprise Keyword length: 255 characters | ||
Restricted characters in Enterprise Keywords: < and > | ||
(See Metadata Mapping below.) | No leading whitespace | |
No trailing periods and whitespace. | ||
If a file extension is present, trailing periods and whitespace are allowed before the extension | ||
No non-printable ASCII characters | ||
Transferring Microsoft Lists is not supported. |
File and Folder Name Restrictions
The following are restricted file/folder names.
_vti_test
CON
PRN
AUX
NUL
COM0
COM1
COM2
COM3
COM4
COM5
COM6
COM7
COM8
COM9
LPT0
LPT1
LPT2
LPT3
LPT4
LPT5
LPT6
LPT7
LPT8
LPT9
For more information on Office 365 restrictions, see Microsoft’s official documentation.
Connection Pooling
Microsoft Office 365 connections using OAuth 2.0 authentication may experience bandwidth throttling from Microsoft when using connection pooling.
Delta Job Runs
After the initial transfer of an item using a Microsoft SharePoint connection, DryvIQ will leverage Microsoft’s change tracking API on delta job runs to identify changes. This results in fewer Graph API requests and shorter job execution times.
Disabled Users
The Microsoft Office 365 Connector supports author/owner preservation for disabled users in the following cases:
When content is being migrated using the SharePoint Migration API (which happens by default during the initial migration job execution), DryvIQ will preserve author/owner for disabled users.
When the migration job is creating the destination root folder (the root folder selected for the job doesn’t exist and needs to be created), DryvIQ will preserve the owner/author for the disabled user only if it can resolve the user through the
EnsureUserCSOM API method or the_api/web/siteusersREST API call. Depending on the state of the disabled user's information in SharePoint, these API calls may not return the required IDs DryvIQ requires to set the user as the owner/author.
HTML Files
SharePoint Online does not support uploading HTML files larger than 256 MB. If your migration includes HTML files that exceed this limit, the file transfer will fail, and DryvIQ will log an entry in the audit with the following message: "Unable to interpret the contents of this page because it exceeds the maximum page size of 268435456 bytes.”
Invalid Characters and Spaces
DryvIQ verifies file and folder names to identify unsupported characters based on the platform. DryvIQ handles invalid characters according to the Allow unsupported file names to be changed setting for the job.
If the setting is disabled, DryvIQ will throw an exception when it encounters a file or folder that it cannot migrate due to an unsupported character in the name.
If enabled, DyvIQ will replace invalid characters with an underscore (_) or UTF-8 encoding based on the option selected. This setting allows DryvIQ to transfer files and folders. The logic includes leading and trailing spaces in file and folder names. DryvIQ replaces the space rather than trimming it because trimming the space could cause duplicate file names. Adding the underscore ensures the name remains unique.
DryvIQ will sanitize file names that contain combined Unicode characters by replacing the characters with an underscore (_).
Job Target Paths
When entering the job target path manually, you can use either “%252F” or forward slashes (/) between “sites” and the site name since DryvIQ supports both formats (for example, “/sites%252FTestSite” or “/sites/TestSite”).
Language Translation
DryvIQ does not support translation when migrating between SharePoint sites that have different default language configurations.
Link Remediation
DryvIQ currently supports file-based links only for OneDrive and SharePoint. This means DryvIQ will remediate links that point directly to documents (such as Word documents, Excel workbooks, and PowerPoint presentations). However, DryvIQ does not remediate links to folders or entire SharePoint sites. If your documents contain folder paths or site URLs, you’ll need to update those links manually after the migration to ensure they point to the correct location.
Microsoft Link Type | Microsoft Link Format |
|---|---|
Links to personal or user-owned files stored in the OneDrive document library | <https://<tenant>-my.sharepoint.com/:w:/r/personal/<UserOneDrive>/_layouts/15/Doc.aspx?sourcedoc={<PlatformId>}&file=<Filename>&action=default&mobileredirect=true |
Links to team-based or site-owned files stored in libraries managed by SharePoint sites |
|
Lock Events
The Graph API does not support lock event detection without the use of a separate API call, which can slow down change detection. The workaround is to turn off native event detection to transfer locks in each job run or to use a soft reset to transfer locks as needed.
Mapping
When creating CSV mapping files for import, usernames must be in lowercase to adhere to the search requirements of the connector properly.
Metadata Mapping
If a library requires specific metadata but the metadata is missing from a file being copied into the library, the file will be flagged and will not transfer on subsequent job runs. If you have files stuck in a flagged state due to missing metadata, you may need to transfer the files and add the required metadata manually.
OneNote Files
Microsoft has specific standards regarding migrating OneNote files (defined in the Microsoft documentation), and DryvIQ adheres to these standards. DryvIQ converts the OneNote files during the initial folder transfer. If a folder is a valid OneNote folder, DryvIQ will convert it to a OneNote file on Microsoft 365. The item status in the DryvIQ Platform will be “Revised” to indicate DryvIQ converted it. If it is not a valid OneNote folder, DryvIQ will not convert the folder and will transfer it as a regular folder to Microsoft 365.
Updates to the files will not result in the folder type changing, so DryvIQ will not update the folder type based on events in the folder after initial transfer. For example, if a folder contains a non-OneNote file, DryvIQ transfers it as a regular folder on the initial transfer. If you remove the non-OneNote files from the source folder, DryvIQ will not detect the change and will not attempt to convert the folder on subsequent job runs.
Permissions Preservation
DryvIQ uses the following rules when transferring permissions to Office 365.
If a user account has permissions to a file and the user is a member of a group that has greater or equal permissions to that file, DryvIQ will not make any permissions changes. The operation will show as succeeded, and no permissions will have changed. This behavior applies to both inherited and unique permissions scenarios.
If a user account has permissions to a file and the user is a member of a group that has lesser permissions to that file, DryvIQ will break inheritance, and the file will gain unique permissions for the user. DryvIQ will add the user account with the requested permissions.
If a user account has permissions to a file and the user is not a member of a group that has permissions to that file, DryvIQ will break inheritance, and the file will gain unique permissions. DryvIQ will add the user account with the requested permissions.
If a group has permissions to a file, DryvIQ will break inheritance, and the file will gain unique permissions. DryvIQ will add the group with the requested permissions.
Inherited Library Permissions
If DryvIQ is not able to obtain the inheritance settings for a document library (due to an error during the request or other unknown application error), DryvIQ will produce an ”Unable to determine Document Library permissions inheritance. Retry the item to resolve the issue. Contact support if the problem persists” exception for all items in the batch request. DryvIQ sets items to a retry status. This status allows DryvIQ to retry obtaining inherited permission settings rather than making assumptions about the library’s unique permissions and enabled inheritance.
Timestamps
There is a discrepancy in timestamps for the SharePoint Online Folder Created Date when using CSOM and Batch Mode. This behavior is a known issue within OneDrive for Business/Office 365.
DryvIQ will attempt to preserve timestamps on folders when using both CSOM and the batch API. However, SharePoint Online updates the folder's modified dates whenever a file is uploaded into it. As a result, when using CSOM, timestamps are preserved when the folder is initially created but are updated after every file is uploaded. When using the batch API, it preserves the timestamps on the folders after all of the files within the batch are committed. This process is the cause of the discrepancy between the two methods.
Version Deletes
Version deletes are supported.
Corrupt Files
If you see “Error=Value=CobaltAllZerosDetected” errors in the logs, the file that triggered the error will not be uploaded to SharePoint Online. Microsoft produces this error for files it considers to be corrupt.
Platform Item Exception Errors
DryvIQ uses a batch job (uploading multiple files and committing them to the destination asynchronously) to migrate files that do not exist to Microsoft Office 365 or OneDrive for Business connections since this is the best practice for storage providers that offer batch support. If Microsoft responds with an “item already exists” during a batch migration job, DryvIQ compares the latest set of item properties (modified, versions, and number of bytes) to the item properties in the batch job. If the properties on the existing destination file do not match the properties in the batch job, DryvIQ will generate an error identifying the conflicting properties as in the example below.
PlatformItemOutOfDateException: The item changed during processing. Retry the item to resolve the issue. Contact support if the problem persists. /TD_7bcd1ca33f274b0bb7a8698f99936f6b/Folder/File.txt with changed properties: timestamp - original:7/3/2024 4:09:49 PM +00:00, latest:7/3/2024 4:09:56 PM +00:00; number of bytes - original:3, latest:8;
In the error, original is the item property in the batch job, and latest is the item property on the destination. The item will be flagged as “Retry,” and DryvIQ will handle it like a regular retry item. After selecting the retry action (Retry, Ignore, etc.), the item should be resolved.
If the batch job and destination have the same properties, no error will be generated, and the item will be transferred.
Transfer Content to SharePoint Online Shared Document Library | Connection URI and Path Example
PlatformType in Connection: office365
URI Pattern in Connection: https://company.sharepoint.com/[SiteNameWhereLibraryExists]/
Target/Path in Job:Connection Config: /[LibraryName]
This configuration will transfer data to the destination library specified. The connection will take it down to the site-specific level.
Path in the 'Job:Connection:Target' should be the Document Library.
Use the Library name found in the URI (you do not have to escape/encode spaces, just enter a space if that exists in the library name).
If a path is not specified, DryvIQ will transfer files to the default library for the selected site.
Create Connection | REST API
You will need to add the relevant connection information to the GET request. The following GET request will return a Microsoft login link. Use this link to complete logging into your account and grant DryvIQ access to the account.
GET {{url}}v1/connections/platforms/office365-graph/new?domain={{YOUR URL}}&name={{YOUR CONNECTION NAME}}&client_id={{YOUR CLIENT ID}}&client_secret={{YOUR CLIENT SECRET}}
Create Job | Impersonation with Admin Connection
As an Office 365 administrator, you can impersonate a user using the path that relates to their content. Then, use their information in the impersonate_as block. (See Impersonation for more information on how to impersonate users.)
Impersonation with Office 365 Connection
{
"name":"Basic job with impersonation",
"kind": "transfer",
"transfer": {
"audit_level": "trace",
"transfer_type": "copy",
"source": {
"connection": { "id": "{{nfs_source_connectionID}}" },
"target": {
"path": "/sourcePath"
}
},
"destination": {
"connection": { "id": "{{O365_destination_connectionID}}" },
"impersonate_as": {
"id": "00",
"name": "Joe Smith",
"email": "jsmith@company.onmicrosoft.com"
},
"target": {
"path": "/destinationPath"
}
},
"simulation_mode": false
},
"schedule": {
"mode": "manual"
},
"stop_policy": {
"on_success": 5,
"on_failure": 5,
"on_execute": 25
},
"category": {
"name": "category name"
}
}Create Job | with Tenant Level Connection
When creating a job that is mapped to the default site collection of the tenant the value "Default" must be used even though the URL does not have a ../sites/{site_name} representation. When setting the root site collection the name should not be prefixed with "sites/".
Impersonation with Office 365 Connection
{
"name":"Basic job with tenant level connection",
"kind": "transfer",
"transfer": {
"audit_level": "trace",
"transfer_type": "copy",
"source": {
"connection": { "id": "{{nfs_source_connectionID}}" },
"target": {
"item": {
"parent": {
"parent": {
"root": true,
"name": "{{site_collection_name}}"
},
"name": "{{site_name}}"
},
"name": "{{library_name}}"
}
}
},
"destination": {
"connection": { "id": "{{O365_destination_connectionID}}" },
"impersonate_as": {
"id": "00",
"name": "Joe Smith",
"email": "jsmith@company.onmicrosoft.com"
},
"target": {
"path": "/destinationPath"
}
},
"simulation_mode": false
},
"schedule": {
"mode": "manual"
},
"stop_policy": {
"on_success": 5,
"on_failure": 5,
"on_execute": 25
},
"category": {
"name": "category name"
}
}