User Migration Guides | SharePoint to Office 365

Owned by Lindsey Byers (Deactivated)
Last updated: Aug 04, 2022 by Andrea Harvey
35 min read

 

 

 

Contents

Introduction 

DryvIQ is an enterprise data integration platform that enables organizations to maximize business value and productivity from their content.  It connects disparate storage platforms and business applications together, allowing organizations to move, copy, synchronize, gather, and organize files as well as their related data across any system. DryvIQ empowers your users with unified access to the most relevant, complete, and up-to-date content—no matter where it resides.    

DryvIQ delivers a user-friendly, web-based experience optimized for PC, tablet, and mobile phone interfaces, so you can monitor and control your file transfers anywhere, from any device.   

DryvIQ’s true bi-directional hybrid/sync capabilities enable organizations to leverage and preserve content across on-premises systems and any cloud service. Seamless to users, new files/file changes from either system are automatically reflected in the other.  


How does DryvIQ Work? 

Cloud storage and collaboration platforms continue to be the driving force of digital transformation within the enterprise. However, users need to readily access the content that resides within your existing network file systems, ECM, and other storage platforms—enabling them to be productive, wherever they are.  DryvIQ is purpose-built to provide boundless enterprise content integration possibilities. The DryvIQ Platform is 100% open and provides a highly-scalable architecture that enables enterprises to meet evolving technology and user demands—no matter how complex. 

The DryvIQ platform provides: 

  • A low risk approach to moving content to the cloud while maintaining on-premises systems

  • No impact to users, IT staff, business operations, or existing storage integrations

  • The ability to extend cloud storage anywhere/any device capabilities to locally-stored content

  • Easy integration of newly acquired business storage platforms into existing infrastructures

The Engine 

DryvIQ’s bi-directional synchronization engine enables your enterprise to fully-integrate and synchronize your existing on-premises platforms with any cloud service. It empowers your users to freely access the content they need while IT staff maintains full governance and control. DryvIQ integrates with each system's published Application Program Interface (API) at the deepest level—optimizing transfer speeds and preserving all file attributes. 

Security 

DryvIQ’s 100% security-neutral model does not incorporate or use any type of proxy cloud service or other intermediary presence point. All content and related data is streamed directly via HTTPS [256-bit encryption] from the origin to the destination system(s). Additionally, DryvIQ works with native database encryption. For more information, please see DryvIQ Platform | Security Whitepaper

Analyzer | Simulation Mode

The DryvIQ analyzer is a powerful enterprise file transfer simulation that eliminates the guesswork. You will gain granular insight into your entire content landscape including its structure, usage, age, type, applicable metadata, and more no matter where the files are located (in local storage, remote offices, or on user desktops). 

Simulation mode allows you to create a job with all desired configuration options set and execute it as a dry-run. In this mode, no data transfers, no permissions are set, and no changes are made to either the source or the destination. This can be useful in answering several questions about your content prior to running any jobs against your content.

Command and Control 

Via highly-advanced web, DryvIQ Command-line Interface (CLI), RESTful API, and/or .NET interfaces, DryvIQ administrators can easily integrate systems, control interactions and behaviors, or create and control end-user experiences for advanced self-service capabilities. 

Features and Functionality 

The DryvIQ Platform enables you with complete integration and control over: 

  • User accounts 

  • User networked home drives 

  • User and group permissions 

  • Document types, notes, and file attributes 

  • Timestamps 

  • Versions 

  • Departmental, project, and team folders 

  • Defined and custom metadata 

Architecture and Performance 

The DryvIQ platform is built on a pluggable, content–streaming architecture that enables highly–automated file/data transfer and synchronization capabilities for up to billions of files. File bytes stream in from one connection endpoint (defined by the administrator) and across a customer-owned and operated network and/or cloud service. They are then streamed out to a second connection endpoint. Content can also flow bidirectionally across two endpoints rather than solely from an "origin" to a "destination."

Supported Features

The DryvIQ Platform Comparison tool allows you to compare platform features and technical details to determine which are supported for your transfer scenario. Viewing the Platform Comparison results for your integration will display a list of features of each platform and provide insight early in the integration planning process on what details may need further investigation.

The Platform Comparison tool is available through the Platforms menu options. For further information, see Platform Comparison.

Connection Setup

DryvIQ is built on a concept of connections. A connection is made to the source platform, and another connection is made to the destination platform. A job is created to tie the two platforms together. When DryvIQ connects to a content platform, it does so by using the publicly available Application Programming Interface (API) for the specific platform. This ensures DryvIQ is “playing by the rules” for each platform. 

Connections “connect” to a platform as a specific user account. The user account requires the proper permissions to the platform to read, write, update, and/or delete the content based on the actions the DryvIQ job is to perform. The connection user account should also be set up so the password does not expire; otherwise, the connection will no longer be able to access the platform until the connection has been refreshed with the new password. Most connections require a specific user account and its corresponding password. The user account is typically an email address. 

Authenticated Connections

Authenticated Connections are accounts that have been verified with the cloud-based or network-based platform when created. The connection can be user/password-based or done through OAuth2 flow, where a token is generated based on the granting authorization to DryvIQ through a user login. This authorization allows DryvIQ access to the user's drive information (files and folder) on the platform. These connections are used as the source or the destination authentication to transfer your content.

 

OAuth2 Interactive (Web) Flow

Connectors such as Box, Google Drive, and Dropbox use the OAuth2 interactive (or web) flow.

 

OAuth2 Client Credentials Flow

Connections such as Syncplicity and Google Workspace uses the OAuth2 client credentials flow.

 

SharePoint

SharePoint (all versions, CSOM) uses a custom username/password authentication model.


OAuth2 Interactive (Web) Flow

You will need the following information when creating a connection to Network File System, Box, Dropbox, and Dropbox for Business:

  • A name for the connection

  • The account User ID (such as jsmith@company.com)

  • The password for the User ID.

Create Connections

Creating a connection in the DryvIQ Platform user interface is easy! Simply add a connection, select your platform, and enter the requested information. DryvIQ will securely validate your credentials and connect to your source content. 

Microsoft SharePoint On-Premise

DryvIQ connections to SharePoint 2013, 2016, and 2019 on-premise platforms can be made by using any valid SharePoint user account with applicable permissions to access content. There are several ways to transfer content, this document outlines each of those situations and how to configure them.

Microsoft SharePoint supported versions: 2013, 2016, and 2019

Create Connection | User Interface

  1. Select Connections > Add connection.

  2. Select Microsoft SharePoint as the platform on the Add connection modal.

  3. Enter the connection information. Reference the table below for details about each field.

  4. Select Test connection to ensure DryvIQ can connect using the information entered.

  5. You will see a "Connection test succeeded" message on the Add connection modal when DryvIQ establishes connection. (If you don't see this message, verify the information you entered.)

  6. Select Done.

 

Add connection modal - Microsoft SharePoint

 

Field

Description

Required

Field

Description

Required

Display As

Enter the display name for the connection. If you will be creating multiple connections, ensure the name readily identifies the connection. The name displays in the application, and you can use it to search for the connection and filter lists.

If you do not add a display name, the connection will automatically be named using the SharePoint URL. For example, Microsoft SharePoint (http://sharepoint.url.com/). If it will be useful for you to reference the connection by account, you should use the default name. 

Optional

URL

Enter the URL for SharePoint including the site collection where the target document library exists.

Required

User Name

Enter the SharePoint username. It can coincide with the Active Directory Universal Principal Name (UPN).

Required

Password

Enter the password for the username added in the previous field.

Required

SSO Provider

Only Active Directory Federation Services (ADFS) is supported as an SSO provider for SharePoint at this time. 

Optional

SSO URL

Enter the URL of the configured SSO provider.

Optional

Authentication type

Select the authentication method the server uses: Basic, Forms, or NTLM. You can select the Detect automatically option to have DryvIQ automatically detect the authentication type when testing the connection. This is the default selection. 

Optional

 

Connection Test Succeeded

 

 

Features and Limitations 

Platforms all have unique features and limitations. DryvIQ’s transfer engine manages these differences between platforms and allows you to configure actions based on Job Policies and Behaviors. Utilize the Platform Comparison tool to see how your integration platforms may interact regarding features and limitations. For more information on SharePoint restrictions, see Microsoft’s official documentation.

 

Features and limitations may vary between SharePoint versions. Refer to Microsoft's official documentation for your specific version of SharePoint for a full list of supported features and limitations.

Files and Folders

Below is list of SharePoint’s supported and unsupported features as well as additional file/folder restrictions. 

Supported Features

Unsupported Features

Supported Features

Unsupported Features

Version preservation

Mirror lock ownership

Timestamp preservation

 

Author/Owner preservation

 

File lock propagation

 

Restricted types

 

Account map

 

Group map

 

Permission preservation

 

User impersonation

 

Metadata map

 

Tags map

 

 

Other Features/Limitations

SharePoint 2013

SharePoint 2016

SharePoint 2019

Other Features/Limitations

SharePoint 2013

SharePoint 2016

SharePoint 2019

File size maximum

2 GB

2 GB

15 GB

Path length maximum

260

260

400

Segment path length

128

128

260

Invalid characters

| # { } % & \ ~ + / : * ? < > \\ “

| # % \ \\ / : * ? < > “

| \ \\ / : * ? < >  "

Restrictions

  • Consecutive periods

  • Leading tildes

  • Leading periods

  • Trailing periods before extension

  • Trailing periods after extension

  • Leading whitespace

  • Trailing whitespace before extension

  • Trailing whitespace after extension

  • Non-printable

  • Leading tildes without an extension

  • Trailing periods after extension

  • Leading whitespace

  • Trailing whitespace after extension

  • Non-printable

  • Trailing periods after extension

  • Leading whitespace

  • Trailing whitespace after extension

  • Non-printable

Check Out

DryvIQ does not support Libraries that have Require Check-out enabled.

Date Preservation

Date preservation works with SharePoint if you are using author-preservation.

Enterprise Keywords

SharePoint doesn't allow duplicate Enterprise Keywords. The longest string allowed in an Enterprise Keyword is 255 characters. 

File Extensions

SharePoint restricts certain file extensions in different versions. Click here for the complete list.

Invalid Characters and Spaces

DryvIQ verifies file and folder names to identify unsupported characters based on the platform. It then replaces invalid characters with an underscore (_) so the files and folders can be transferred. 

The logic includes leading and trailing spaces in file and folder names. DryvIQ replaces the space rather than trimming it because trimming the space could cause duplicate file names. Adding the underscore ensures the name remains unique.

Okta

DryvIQ does not support Okta with SharePoint.

Restricted File/Folder Names

The following are restricted file and folder names:

  • _vti_test

  • _w

  • _t

The following are restricted folder names:

  • Test_files

Folder names cannot end with the following:

.files

_elemei

_files

_ficheiros

_Dateien

_arquivos

_fichiers

_dosyalar

_bestanden

_datoteke

_file

_fitxers

_archivos

_failid

_filer

_fails

_tiedostot

_bylos

_pliki

_fajlovi

_soubory

_fitxategiak

URLs

The transformation of URLs into SharePoint can sometimes increase the overall URL length, which can cause either 400 or 414 URL failures. (Spaces are the primary way this can happen.) DryvIQ recommends you remove the spaces or shrink the URL length to resolve this error.

Versioning

Versioning is supported if versioning is enabled on the library.

SharePoint 2019 supports version deletes. 

Create Connection | REST API

Create a Basic Connection

Create a basic SharePoint connection at the following endpoint.

GET {{url}}v1/connections/platforms/SharePoint/new

 

Create a Connection with Applicable Authorization Type

Verify your instance support forms. In rare instances, there can be a misconfiguration with SharePoint that can falsely report that Forms are supported. If you are experiencing connection issues with SharePoint, you may need to configure for ntlm authentication for the auth_type parameter.

SharePoint Connection with Explicit Auth Type

{   "name": "SharePoint 2016",   "platform": {     "id": "sharepoint"   },   "auth": {     "uri":"http://url.com",     "username":"USER",     "password":"PASSWORD",     "auth_type":"ntlm"      } }

SharePoint Connection | API Job Configuration Examples

The example below shows a simple job for a SharePoint connection created as standard user without impersonation.

POST {{url}}v1/jobs

{     "name":"Simple Job",     "kind": "transfer",     "transfer": {       "transfer_type": "copy",         "source": {             "connection": { "id": "{{SharePoint_Connection_sourceID}}" },             "target": {                 "path": "/sourceDocumentLibrary"             }         },         "destination": {             "connection": { "id": "{{Box_Connection_destinationID}}" },             "target": {                 "path": "/destinationFolder"             }         },         "simulation_mode": false     },     "schedule": {         "mode": "manual"     },     "stop_policy": {         "on_success": 5,         "on_failure": 5,         "on_execute": 25     },     "category": {       "name": "Report {Name}"     } }

Microsoft Office 365

DryvIQ connections to the Microsoft Office 365 platform can be made by using an administrator account with the proper privileges to manage Office 365 configurations. The connection can be made to a single site or to a site collection/tenant root. DryvIQ creates the connection using the OAuth 2.0 flow to simplify login and connection management.

Create Connection | User Interface

  1. Select Connections > Add connection.

  2. Select Microsoft Office 365 as the platform on the Add connection modal.

  3. Enter the connection information. Reference the table below for details about each field.  

  4. Select Sign in with Microsoft Office 365.

  5. Enter the email for the account being used to create the connection and click Next. You must use an admin account with the proper privileges to manage Office 365 configurations. 

  6. Enter the password for the account and select Sign in.

  7. You will see a "Connection test succeeded" message on the Add connection modal. (If you don't see this message, repeat the sign in and authorization steps above. If the connection fails, verify the URL. DryvIQ cannot create the connection if the URL is incorrect.)

  8. Select Done to finish creating the connection. 

 

Add Connection Modal - Microsoft Office 365 

 

Field

Description

Required

Field

Description

Required

Display as

Enter the display name for the connection. If you will be creating multiple connections, ensure the name readily identifies the connection. The name displays in the application, and you can use it to search for the connection and filter lists.

If you do not add a display name, the connection will automatically be named using the URL. For example, Microsoft Office 365 (https://mycompany.sharepoint.com/). If it will be useful for you to reference the connection by account, you should use the default name.

Optional

URL

Enter the URL of your Office 365 account. 

You can find the URL by logging into Microsoft Office 365 using the account you want to use to create the connection. The URL in the address bar is the URL you need to use. It will look something like "https://mycompany.sharepoint.com/_layouts/15/viewlsts.aspx?view=14," where mycompany will be the site name of your company. You can copy and paste the full URL or only the part before layouts "layouts" (https://mycompany-my.sharepoint.com/). DryvIQ will ignore the extra part of the URL in most cases.

If the connection fails, verify the URL. DryvIQ cannot create the connection if the URL is incorrect.

Required

Token endpoint

This setting is rare for most Microsoft connections and can be left blank. If needed, this value will be provided by your administrator.

Optional

Graph API endpoint

This setting is rare for most Microsoft connections and can be left blank. If needed, this value will be provided by your administrator.

Optional

External users notification

This indicates if you want to send notifications to external users. The default value is No, so no notifications will be sent. 

If you select Yes, notifications will be sent to external users when they have been granted access to content based on permissions.

Optional

Set Tenant Level Connection

This indicates if the connection should be set at the tenant level. The default value is No, so the connection will be set based on the URL provided.

If you select Yes, the connection will override the URL provided and connect to {tenant}-admin.sharepoint.com.

Using a tenant level connection allows one connection to be used to create jobs that point at different site collections.  When creating a job using the Microsoft Office 365 connection, you will need to make the root an actual site collection since DryvIQ cannot create a new site.

Optional

Behavior When Deleting Items

Select the type of delete DryvIQ should perform when deleting items: Permanent or Soft. Soft delete is the default delete behavior; however, Permanent is the recommended behavior.

A soft delete marks items as a deleted. You can still access them to restore or permanently delete the items.

A permanent delete removes the items. This delete is not reversible.

Optional

 

Microsoft Sign In Modal

 

 

Connection Test Succeeded

 

Features and Limitations 

Platforms all have unique features and limitations. DryvIQ’s transfer engine manages these differences between platforms and allows you to configure actions based on Job Policies and Behaviors. Utilize the Platform Comparison tool to see how your integration platforms may interact regarding features and limitations.

  • SharePoint Online Connections must have full control of all SharePoint site collections.

  • OneDrive for Business Connections are automatically configured to the Documents library. When configuring your job JSON, do not include "Documents" in the location path, such as “/Documents/FolderName.” The correct configuration path is “/FolderName.”

  • SharePoint Tenant Connections must have full control of the SharePoint admin site ({tenant}-admin) as well as full control of all SharePoint site collections that are to be accessed.

Files and Folders

Office 365 has the following file/folder restrictions.

Supported Features

Unsupported Features

Other Features/Limitations

Supported Features

Unsupported Features

Other Features/Limitations

Version preservation

Mirror lock ownership

Invalid characters:  |   "   \   /   :   *   ?   <   >

Timestamp preservation

User impersonation

File size maximum: 250 GB Microsoft limit
DryvIQ uses a file maximum size of 249.999999 GB files to ensure transfer. 

Author/Owner preservation

 

Segment path length: N/A

Account map

 

Path length maximum: 400

Group map

 

Maximum number of files per folder: 5000 

Permissions preservation

 

Maximum Enterprise Keyword length: 255 characters

Metadata map

 

Restricted characters in Enterprise Keywords: < and >

Tags map

 

No leading whitespace 
See Invalid Characters and Spaces below

 

 

No trailing periods and whitespace. 
See Invalid Characters and Spaces below

 

 

If a file extension is present, trailing periods and whitespace are allowed before the extension

 

 

No non-printable ASCII characters

 

 

Transferring Microsoft Lists is not supported.

File and Folder Name Restrictions

The following are restricted file/folder names.

  • _vti_test

  • CON

  • PRN

  • AUX

  • NUL

  • COM0

  • COM1

  • COM2

  • COM3

  • COM4

  • COM5

  • COM6

  • COM7

  • COM8

  • COM9

  • LPT0

  • LPT1

  • LPT2

  • LPT3

  • LPT4

  • LPT5

  • LPT6

  • LPT7

  • LPT8

  • LPT9

  • For more information on Office 365 restrictions, see Microsoft’s official documentation.

Connection Pooling

Microsoft Office 365 connections using OAuth 2 authentication may experience bandwidth throttling from Microsoft when using connection pooling.

HTML Files

SharePoint online does not support uploading HTML files larger than 256 MB. If your migration includes HTML files that exceed this limit, the file transfer will fail, and an entry will be logged in the audit with the following message: "Unable to interpret the contents of this page because it exceeds the maximum page size of 268435456 bytes.”

Invalid Characters and Spaces

DryvIQ verifies file and folder names to identify unsupported characters based on the platform. It then replaces invalid characters with an underscore (_) so the files and folders can be transferred. 

The logic includes leading and trailing spaces in file and folder names. DryvIQ replaces the space rather than trimming it because trimming the space could cause duplicate file names. Adding the underscore ensures the name remains unique.

DryvIQ will sanitize file names that contain combined Unicode characters by replacing the characters with an underscore (_).

Lock Events

Graph API does not support lock event detection without the use of a separate API call, which will slow down change detection. The workaround is to disable native event detection to transfer locks in each job run or to use a soft reset to transfer locks as needed. 

Mapping

When creating CSV mapping files for import, the usernames must be lowercase to properly adhere to the search requirements for the connector.

Metadata Mapping

If a library requires specific metadata but the metadata is missing from a file being copied into the library, the file will be flagged and will not transfer on subsequent jobs runs. If you have files stuck in a flagged state due to missing metadata, you may need to manually transfer the files and add the required metadata.  

Permissions Preservation

The following rules will be used when transferring permissions to Office 365.

  • If a user account is granted permissions to a file and the user is a member of a group that has greater or equal permissions to that file, no permissions changes will be made. The operation will show as succeeded, and no permissions will have changed. This behavior applies to both inherited and unique permissions scenarios.

  • If a user account is granted permissions to a file and the user is a member of a group that has lesser permissions to that file, inheritance will be broken, and the file will gain unique permissions for the user. The user account will be added with the requested permissions.

  • If a user account is granted permissions to a file and the user is not a member of a group that has permissions to that file, inheritance will be broken, and the file will gain unique permissions. The user account will be added with the requested permissions.

  • If a group is granted permissions to a file, inheritance will be broken, and the file will gain unique permissions. The group will be added with the requested permissions.

Timestamps

There is a discrepancy in timestamps for the SharePoint Online Folder Created Date when using CSOM and Batch Mode. This behavior is a known issue within OneDrive for Business/Office 365.

DryvIQ will attempt to preserve timestamps on folders when using both CSOM and the batch API. However, SharePoint Online updates the folder's modified dates whenever a file is uploaded into it. As a result, when using CSOM, the timestamps will be preserved when the folder is initially created but then updated after every file that gets uploaded. When using batch API, it preserves the timestamps on the folders after all of the files within the batch are committed. This is the cause for the discrepancy between the two methods.

Version Deletes

Version deletes are supported. 

Transfer Content to SharePoint Online Shared Document Library | Connection URI and Path Example

  • PlatformType in Connection: office365

  • URI Pattern in Connection: https://company.sharepoint.com/[SiteNameWhereLibraryExists]/

  • Target/Path in Job:Connection Config: /[LibraryName]

This configuration will transfer data to the destination at the library specified. Connection will take it down to Site specific level. 

Path in the 'Job:Connection:Target' should be the Document Library.

Use the Library name found in the URI (you do not have to escape/encode spaces, just enter a space if that exists in the library name).