Contents
Introduction
SkySync is an enterprise data integration platform that enables organizations to maximize business value and productivity from their content. It connects disparate storage platforms and business applications together, allowing organizations to move, copy, synchronize, gather, and organize files as well as their related data across any system. SkySync empowers your users with unified access to the most relevant, complete, and up-to-date content—no matter where it resides.
SkySync delivers a user-friendly, web-based experience optimized for PC, tablet, and mobile phone interfaces, so you can monitor and control your file transfers anywhere, from any device.
SkySync’s true bi-directional hybrid/sync capabilities enable organizations to leverage and preserve content across on-premises systems and any cloud service. Seamless to users, new files/file changes from either system are automatically reflected in the other.
How does SkySync Work?
Cloud storage and collaboration platforms continue to be the driving force of digital transformation within the enterprise. However, users need to readily access the content that resides within your existing network file systems, ECM, and other storage platforms—enabling them to be productive, wherever they are. SkySync is purpose-built to provide boundless enterprise content integration possibilities. The SkySync Platform is 100% open and provides a highly-scalable architecture that enables enterprises to meet evolving technology and user demands—no matter how complex.
The SkySync platform provides:
A low risk approach to moving content to the cloud while maintaining on-premises systems
No impact to users, IT staff, business operations, or existing storage integrations
The ability to extend cloud storage anywhere/any device capabilities to locally-stored content
Easy integration of newly acquired business storage platforms into existing infrastructures
The Engine
SkySync’s bi-directional synchronization engine enables your enterprise to fully-integrate and synchronize your existing on-premises platforms with any cloud service. It empowers your users to freely access the content they need while IT staff maintains full governance and control. SkySync integrates with each system's published Application Program Interface (API) at the deepest level—optimizing transfer speeds and preserving all file attributes.
Security
SkySync’s 100% security-neutral model does not incorporate or use any type of proxy cloud service or other intermediary presence point. All content and related data is streamed directly via HTTPS [256-bit encryption] from the origin to the destination system(s). Additionally, SkySync works with native database encryption. For more information, please see SkySync Platform | Security Whitepaper
Analyzer | Simulation Mode
The SkySync analyzer is a powerful enterprise file transfer simulation that eliminates the guesswork. You will gain granular insight into your entire content landscape including its structure, usage, age, type, applicable metadata, and more no matter where the files are located (in local storage, remote offices, or on user desktops).
Simulation mode allows you to create a job with all desired configuration options set and execute it as a dry-run. In this mode, no data transfers, no permissions are set, and no changes are made to either the source or the destination. This can be useful in answering several questions about your content prior to running any jobs against your content.
Command and Control
Via highly-advanced web, SkySync Command-line Interface (CLI), RESTful API, and/or .NET interfaces, SkySync administrators can easily integrate systems, control interactions and behaviors, or create and control end-user experiences for advanced self-service capabilities.
Features and Functionality
The SkySync Platform enables you with complete integration and control over:
User accounts
User networked home drives
User and group permissions
Document types, notes, and file attributes
Timestamps
Versions
Departmental, project, and team folders
Defined and custom metadata
Architecture and Performance
The SkySync platform is built on a pluggable, content–streaming architecture that enables highly–automated file/data transfer and synchronization capabilities for up to billions of files. File bytes stream in from one connection endpoint (defined by the administrator) and across a customer-owned and operated network and/or cloud service. They are then streamed out to a second connection endpoint. Content can also flow bidirectionally across two endpoints rather than solely from an "origin" to a "destination."
Supported Features
The SkySync Platform Comparison tool allows you to compare platform features and technical details to determine which are supported for your transfer scenario. Viewing the Platform Comparison results for your integration will display a list of features of each platform and provide insight early in the integration planning process on what details may need further investigation.
The Platform Comparison tool is available through the Platforms menu options. For further information, see Platform Comparison.
Connection Setup
SkySync is built on a concept of connections. A connection is made to the source platform, and another connection is made to the destination platform. A job is created to tie the two platforms together. When SkySync connects to a content platform, it does so by using the publicly available Application Programming Interface (API) for the specific platform. This ensures SkySync is “playing by the rules” for each platform.
Connections “connect” to a platform as a specific user account. The user account requires the proper permissions to the platform to read, write, update, and/or delete the content based on the actions the SkySync job is to perform. The connection user account should also be set up so the password does not expire; otherwise, the connection will no longer be able to access the platform until the connection has been refreshed with the new password. Most connections require a specific user account and its corresponding password. The user account is typically an email address.
Authenticated Connections
Authenticated Connections are accounts that have been verified with the cloud-based or network-based platform when created. The connection can be user/password-based or done through OAuth2 flow, where a token is generated based on the granting authorization to SkySync through a user login. This authorization allows SkySync access to the user's drive information (files and folder) on the platform. These connections are used as the source or the destination authentication to transfer your content.
OAuth2 Interactive (Web) Flow
Connectors such as Box, Google Drive, and Dropbox use the OAuth2 interactive (or web) flow.
OAuth2 Client Credentials Flow
Connections such as Syncplicity and GSuite uses the OAuth2 client credentials flow.
SharePoint
SharePoint (all versions, CSOM) uses a custom username/password authentication model.
OAuth2 Interactive (Web) Flow
You will need the following information when creating a connection to Network File System, Box, Dropbox and Dropbox for Business:
A name for the connection
The account User ID (such as jsmith@company.com)
The password for the User ID.
Create Connections
Creating a connection in the SkySync Platform user interface is easy! Simply add a connection, select your platform, and enter the requested information. SkySync will securely validate your credentials and connect to your source content.
Box
The Box connector in SkySync allows you to analyze, migrate, copy, and synchronize files to your Box account from cloud storage repositories and on-premise network file shares. The first step is to create the Box connection by providing the connection information required for SkySync to connect to the server. The connector can be created using a standard Box account or a Box Service Account.
Please refer to Box Service Account for information regarding how to create a Box Service Account connection.
Even though simulation mode doesn’t move data, Box will identify activity on accounts during simulation mode. Therefore, your Box administrator should turn off Box security notifications when Box is the source connector on copy jobs or when Box is used in sync jobs. This prevents users from getting security notifications about activity on their accounts.
Create Connection | SkySync Application User Interface
Select Connections > Add connection.
Select Box as the platform on the Add connection modal.
Enter the connection information. Reference the table below for details about each field.
Select Sign in with Box.
On the Box Customer Log In modal, enter the Email Address and Password required to log in to the Box account and select Authorize.
Select Grant access to Box when prompted to authorize the connection.
You will see a "Connection test succeeded" message on the Add connection modal. (If you don't see this message, repeat the sign in and authorization steps above.)
Select Done to finish creating the connection.
Add Connection Modal - Box
Field | Description | Required |
|---|---|---|
Display as | Enter the display name for the connection. If you will be creating multiple connections, ensure the name readily identifies the connection. The name displays in the application, and you can use it to search for the connection and filter lists. If you do not add a display name, the connection will automatically be named using the account owner's name. For example, Box (John Doe). If it will be useful for you to reference the connection by account, you should use the default name. | Optional |
User Type | Required | |
Connect as standard user | Select this option to create a standard connection to access a user's files and folders. This is the default selection. | |
Connect as account administrator | Select this option to create an administrator connection. This requires administrator privileges and grants access to all accounts within the organization. This option is often used along with impersonation to simplify transferring multiple user accounts. When connected as an administrator, the first level of folders will be user names. | |
Platform API client credentials | Required | |
Use the system default client credentials | Select this option to use the default SkySync client application. | |
Use custom client credentials | Select this option to use custom client credentials provided by your administrator. When selected, two additional fields will be available to enter the credentials. Your administrator can use the information provided in the following link to obtain the credentials: Box Documentation - Setting up a JWT app. | |
Client ID | This field displays only when you select Use custom client credentials. This value will be provided by your administrator. Your administrator can use the information provided in the following link to obtain the credentials: Box Documentation - Setting up a JWT app. | Optional |
Client Secret | This field displays only when you select Use custom client credentials. This value will be provided by your administrator. Your administrator can use the information provided in the following link to obtain the credentials: Box Documentation - Setting up a JWT app. | Optional |
Behavior When Deleting Items | Select the type of delete SkySync should perform when deleting items: Permanent or Soft. Soft delete is the default delete behavior; however, Permanent is the recommended behavior. A soft delete marks items as a deleted. You can still access them to restore or permanently delete the items. A permanent delete removes the items. This delete is not reversible. | Optional |
Box Customer Log In
Grant Access to Box
Connection Test Succeeded
Features and Limitations
Platforms all have unique features and limitations. SkySync’s transfer engine manages these differences between platforms and allows you to configure actions based on Job Policies and Behaviors. Utilize the Platform Comparison tool to see how your integration platforms may interact regarding features and limitations.
Files and Folders
Below is list of Box's supported and unsupported features as well as additional file/folder restrictions. The information below the tables provides additional details about supported and unsupported features as well as best practices when using this connector.
Supported Features | Unsupported Features | Other Features/Limitations |
|---|---|---|
Invalid characters: \ / | ||
File size maximum: 15 GB | ||
Path length maximum: n/a | ||
Segment path length: 255 | ||
No leading spaces in file name/folder names | ||
No trailing spaces in folder names, file names, or file extensions | ||
No non-printable characters | ||
Box has download limitations for the number of folders and files contained in one folder. Please consult Box documentation for further details. | ||
Box accounts that do not have administrator-level access cannot remove group permissions on files during a job transfer. | ||
Google document types natively created on Box can be moved and will maintain formatting. However, they will have the native Google file extensions (.gdoc, .gsheet, etc.). | ||
The maximum tag size in Box is 255 characters. You can enter more characters than the maximum, but Box will truncate it down to 255 characters. |
Box Comments
SkySync does not support transferring Box comments from folders and files.
Box Notes
SkySync offers the option to convert Box Notes when migrating to other platforms. The Allow Rendition option can be enabled on the Behaviors page when creating a new job.
When enabled, Box Notes will be transferred to the destination as .docx files.
When disabled, Box Notes will be transferred to the destination in .JSON format.
Warning! Enabling renditions will cause your job to run slower since Box doesn't support native event detection for Box Note files and must crawl to detect changes.
While edits to Box Notes on the source will be detected and transferred to the destination on subsequent job runs, edits to the .docx file on the destination are not transferred to the original Box Note on the source for sync jobs.
Unsupported Box Notes Features
Preview images are not supported when converting Box Notes to .docx files since there is not enough information available for SkySync to download the image.
Box Notes to SharePoint Online
When transferring Box Notes from Box to SharePoint Online without the Rendition feature enabled (transferring the Box Notes in .json format), Box Note files will transfer; however, edits to Box Notes will not be picked up, and manual intervention is required. Below is an outline of how Box Notes are handled on the initial and subsequent job runs.
First Job Run
Box Note files transfer to SharePoint Online as expected, preserving the extension.
The file version history transferred to the destination reflects every edit that occurred on the Box Note file in the Box source application.
Second Job Run
If changes are made to the Box Note file on the source in between job runs, the new version is not transferred to the destination.
Resolution
Perform a soft job reset and run the job again. This will re-evaluate the new edits/versions that occurred on the source Box Note file. (See Job | Reset Options for information on how to reset jobs.)
Character Sanitization
SkySync will sanitize file names that contain combined Unicode characters by replacing the characters with an underscore (_).
File Size Limits
The maximum file size limit for uploads to Box can be between 250 MB to 32 GB depending on your Box account. There is no maximum file size limit coded with SkySync. Instead, SkySync will evaluate the user profile and retrieve the max_upload_size parameter when the Box connection is created. SkySync will respect the limit provided as the maximum file size for this connection. See Box's Understand the Maximum File Size You Can Upload to Box support document for more information.
Group Maps
When creating a group map where Box is the destination, you must provide the Box ID in order for the map to work.
Invalid Characters and Spaces
SkySync verifies file and folder names to identify unsupported characters based on the platform. It then replaces invalid characters with an underscore (_) so the files and folders can be transferred.
The logic includes leading and trailing spaces in file and folder names. SkySync replaces the space rather than trimming it because trimming the space could cause duplicate file names. Adding the underscore ensures the name remains unique.
Owner Permissions
SkySync doesn’t expose owner permissions when migrating from Box. When the account running the job is the owner of the content but the user map between that account and the destination account don’t match, SkySync won’t grant privileges to the audit trail creator, so the owner will not be able to access the content.
Path Lengths
Box does not impose restrictions for the total path length.
Segment path lengths are limited to 255 character. Segments are delimited by a forward slash (/). For example, <max 255 characters>/<max 255 characters>.
Server System Clock
Your Box connection from SkySync will fail to make a successful connection if the SkySync server system clock time is ahead of the Box platform time because the access token will be expired by the time it is returned from Box. Therefore, you must ensure the time on the server running SkySync is set to the same time as the Box platform or, preferably, a minute or two behind.
The Box platform uses UNIX time; you can find the UNIX timestamp by visiting https://www.unixtimestamp.com/ . Enter the current UTC time of the SkySync server on this site to get the UNIX time of Box and the application server.
You can find the current UTC time by doing a simple Internet search for "current UTC time" or by visiting timeanddate.com.
UNIX Timestamp Converter
Shared Links
When transferring from Box, SkySync will filter out links shared to specific users and log an entry in the audit log that the shared links were not preserved due to the global sharing policy.
Version Preservation
Each Box account type is limited to the maximum number of file versions that can be accessed. This means that while the UI shows as many versions as you upload, you can only access a certain number of versions. In addition to the default version maximum, account administrators can set the maximum number of file versions to save and track for the account. Both the default file version maximum and any custom version settings added to the Box account affect version transfers. Please refer to Box’s version history documentation to understand the version limitations of your account.
Create Connection | SkySync API
The following GET will return a Box login link. Use the link to complete logging into the account and to grant SkySync access to the account.
GET {{url}}v1/connections/platforms/box/new
GET {{url}}v1/connections/platforms/box/new?admin_mode=1
Box Connection | API Job Configuration Examples
Path Example: Box connection created as standard user, no impersonation
Connect as a Box user. The path relates to my source content.
{
"name":"Simple Job",
"kind": "transfer",
"transfer": {
"transfer_type": "copy",
"source": {
"connection": { "id": "{{Box_connection_AsStandardUser_sourceID}}" },
"target": {
"path": "/sourceFolder"
}
},
"destination": {
"connection": { "id": "{{OneDriveForBusiness_connection_destinationID}}" },
"target": {
"path": "/Documents/destinationFolder"
}
},
"simulation_mode": false
},
"schedule": {
"mode": "manual"
},
"stop_policy": {
"on_success": 5,
"on_failure": 5,
"on_execute": 25
},
"category": {
"name": "Report {Name}"
}
}
Path Example: Box Connection Created as Standard User with Impersonation
As a Box administrator, impersonate a user in the system where the path relates to their content. This requires administrator privileges to use the impersonation feature, such as administrator or co-administrator.
{
"name":"Simple Job",
"kind": "transfer",
"transfer": {
"transfer_type": "copy",
"source": {
"connection": { "id": "{{Box_connection_AsStandardUser_sourceID}}" },
"impersonate_as": { "email": "user@company.com" },
"target": {
"path": "/sourceFolder"
}
},
"destination": {
"connection": { "id": "{{OneDriveForBusiness_connection_destinationID}}" },
"target": {
"path": "/Documents/destinationFolder"
}
},
"simulation_mode": false
},
"schedule": {
"mode": "manual"
},
"stop_policy": {
"on_success": 5,
"on_failure": 5,
"on_execute": 25
},
"category": {
"name": "Report {Name}"
}
}
Path Example: Box Connection created as an Account Administrator
Connect as an account administrator. The path must include the user of the content you are transferring. If impersonation is used for this scenario, permissions will be ignored.
{
"name":"Simple Job",
"kind": "transfer",
"transfer": {
"transfer_type": "copy",
"source": {
"connection": { "id": "{{Box_connection_AsAdmin_sourceID}}" },
"target": {
"path": "/user@company.com/sourceFolder"
}
},
"destination": {
"connection": { "id": "{{OneDriveForBusiness_connection_destinationID}}" },
"target": {
"path": "/Documents/destinationFolder"
}
},
"simulation_mode": false
},
"schedule": {
"mode": "manual"
},
"stop_policy": {
"on_success": 5,
"on_failure": 5,
"on_execute": 25
},
"category": {
"name": "Report {Name}"
}
}
Dropbox for Business
The Dropbox connector in SkySync allows you to analyze, migrate, copy, and synchronize files to your Dropbox account from cloud storage repositories and on-premise network file shares. The first step is to create the Dropbox connection by providing the connection information required for SkySync to connect to the server. The connector can be created using a standard Dropbox account or a Dropbox for Business account. However, when creating a connection using a Dropbox for Business account, you must use an Administrator account. Standard accounts will return an error from the Dropbox for Business platform.
For more information on using a standard Dropbox account, please refer to the Dropbox connector page.
Create Connection | SkySync Application User Interface
Select Connections > Add connection.
Select Dropbox for Business as the platform on the Add connection modal.
Enter the connection information. Reference the table below for details about each field.
Select Sign in with Dropbox for Business.
On the Dropbox API Request Authorization modal, enter the Email and Password required to log in to the account and select Sign In. Only an Administrator account can be used for Dropbox for Business. Standard accounts will return an error from the Dropbox for Business platform.
Select Allow when prompted to authorize the connection.
You will see a "Connection test succeeded" message on the Add connection modal. (If you don't see this message, repeat the sign in and authorization steps above.)
Select Done to finish creating the connection.
Add Connection Modal - Dropbox for Business
Field | Description | Required |
|---|---|---|
Display as | Enter the display name for the connection. If you will be creating multiple connections, ensure the name readily identifies the connection. The name displays in the application, and you can use it to search for the connection and filter lists. If you do not add a display name, the connection will automatically be named, "Dropbox for Business." SkySync recommends that you edit the name to more readily identify the connection. | Optional |
Platform API client credentials | Required | |
Use the system default client credentials | Select this option to use the default SkySync client application. | |
Use custom client credentials | Select this option to use custom client credentials provided by your administrator. When selected, two additional fields will be available to enter the credentials. | |
Client ID | This field displays only when you select Use custom client credentials. This value will be provided by your administrator. | Required when using custom credentials |
Client Secret | This field displays only when you select Use custom client credentials. This value will be provided by your administrator. | Required when using custom credentials |
Dropbox API Request Authorization modal
Allow Access to Dropbox
Connection Test Succeeded
Features and Limitations
Platforms all have unique features and limitations. SkySync’s transfer engine manages these differences between platforms and allows you to configure actions based on Job Policies and Behaviors. Utilize the Platform Comparison tool to see how your integration platforms may interact regarding features and limitations.
Files and Folders
Below is list of Dropbox's supported and unsupported features as well as additional file/folder restrictions.
Supported | Unsupported | Other Features/Limitations |
|---|---|---|
< > \\ / : ? * \ | ~ | ||
File size maximum: 50 GB or smaller | ||
Path length maximum: n/a | ||
Segment path length: 255 | ||
Maximum number of files per folder: 10,000 | ||
No trailing spaces after file extensions | ||
Author Preservation
The Dropbox connector uses "per-request impersonation." This means SkySync makes requests to the platform on behalf of the account owner, not the administrator. Therefore, files created by other users will fail to upload unless the account owner mounts the shared parent folder into their Dropbox drive before uploading the file. (See https://help.dropbox.com/files-folders/add-shared-folder for more information.) Failed uploads will be logged with a message similar to the following example: "[PermissionFailure] The account 'Joe Smith' is not authorized to perform the requested operation."
Business Team Folders
Team folders transfer automatically with any transfer job; however, Team and Shared folders must first be mounted in Dropbox (Dropbox for Business API only surfaces mounted Team and Shared folders). If you include Job Filters | Filter Shared Content, Team Folders will not be transferred. The Dropbox for Business connector creates shared user folders by default. To create shared folders in a Team Folder, the root Team Folder must be created in the Dropbox UI and selected as the root of the job. Shared folders that are created in a Team Folder support permission disinheritance. Shared folders created in a user folder do not support disinheritance and will log a warning.
Character Sanitization
SkySync will sanitize file names that contain combined Unicode characters by replacing the characters with an underscore (_).
Dropbox Paper
SkySync does not support Dropbox paper. Dropbox Paper is a separate work space that stores Paper documents in your account on http://paper.dropbox.com . SkySync doesn't support have access to these documents as part of your Dropbox connection.
Path Lengths
Dropbox does not impose restrictions for the total path length.
Segment path lengths are limited to 255 character. Segments are delimited by a forward slash (/). For example, <max 255 characters>/<max 255 characters>.
Versioning
The Dropbox platform treats content version history differently than other cloud platforms, with each native move and rename tracked as a new version. In order to only transfer true version history, enable the Scripting option on step 6 during job creation and copy the following JSON into the scripting box:
{
"transfer": {
"versioning": {
"preserve": "native",
"select": "published"
}
}
}
For more information about how to use Advanced Scripting, refer to the Advanced Scripting | Additional Configuration Options page.
Version Limits
While Dropbox allows you to store unlimited versions, the platform restricts version downloads to 100. This means only the most recent 100 versions for a given file will be transferred to the destination. Refer to the Dropbox API Documentation for more information. You can also find additional information in the Dropbox Forum.
If you want to transfer just teams folders, you can use the Dropbox for Business Teams Folders connector to create a connection that will transfer only teams folders.
Create Connection | SkySync API
Create a New Dropbox for Business Connection
The following GET will return a target URL. Use this URL to log in to the Dropbox for Business account to authenticate and create the connection. Make sure you connect using an Administrator account.
GET {{url}}v1/connections/platforms/dfb/new
Create a New Dropbox for Business Connection Using Custom Credentials
Create a new connection using custom credentials with the example call below. Replace the name, client_id, and client_secret with information relevant to your connection.
{
"name": "Dropbox for Business",
"platform": {
"id": "dfb"
},
"auth": {
"client_id": "{{clientID}}",
"client_secret": "{{clientSecret}}"
}
}
Dropbox for Business with Single Sign-On (SSO)
Create a new connection using single sign-on with the example call below. You will need to obtain the applicable access token for your Dropbox for Business account.
{
"name": "Dropbox for Business",
"platform": {
"id": "dfb"
},
"auth": {
"access_token": "{{access_token}}"
}
}
Create Job | Dropbox for Business Account Admin Connection
Create a simple transfer job using the example call below. Replace the information with information relevant to your job and connectors.
Note that the following are known issues when creating a job for a Dropbox for Business account using an Administrator account connection.
Connection-based Impersonation should be used due to a caching issue which may incorrect shared folder detection
Connection-based Impersonation is shown in the user-interface as "Run As...." option on the Locations step when creating a new job.
Path-based impersonation should not be used, first 'folder' is a user name such as "path": "/user@company.com/"
{
"name":"Simple Job",
"kind": "transfer",
"transfer": {
"transfer_type": "copy",
"source": {
"connection": { "id": "{{DropboxForBusiness_Admin_ConnectionID}}" },
"impersonate_as": { "email": "user@company.com" },
"target": {
"path": "/sourceFolder"
}
},
"destination": {
"connection": { "id": "{{OneDriveForBusiness_connection_destinationID}}" },
"target": {
"path": "/Documents/destinationFolder"
}
},
"simulation_mode": false
},
"schedule": {
"mode": "manual"
},
"stop_policy": {
"on_success": 5,
"on_failure": 5,
"on_execute": 25
},
"category": {
"name": "Report Category {Name}"
}
}
Transferring Dropbox for Business Team Folders
Team folders will transfer automatically with any transfer job; however, Team and Shared folders must first be mounted in Dropbox (Dropbox for Business API only surfaces mounted Team and Shared folders)
If you include Job Filters | Filter Shared Content, Team Folders will not be transferred
The Dropbox for Business connector creates shared user folders by default. To create shared folders in a Team Folder, the root Team Folder must be created in the Dropbox UI and selected as the root of the job.
Shared folders that are created in a Team Folder support permission disinheritance. Shared folders created in a user folder do not support disinheritance and will log a warning.
If you want to transfer just teams folders, you can use the Dropbox for Business Teams Folders connector to create a connection that will transfer only teams folders.
Connection Pooling
When transferring data between a source and destination, there are a number of factors that can limit the transfer speed. Most cloud providers have rate limitations that reduce the transfer rate, but if those limits are account based and it supports impersonation, SkySync can create a pool of accounts that issues commands in a round-robin format across all the accounts connected to the pool. Any modifications to the connection pool will used on the next job run.
For example, if a connection pool has two accounts, all commands will be alternated between them. If a third account is added to the pool, the next run of the job will use all three accounts.
Connection pooling is not supported with My Computer and Network File System (NFS) connection.
User and Group Maps
A user account or group map provides the ability to explicitly associate users and groups for the purposes of setting ownership and permissions on items transferred. These mappings can happen automatically using rules or explicitly using an exception. Accounts or groups can be excluded by specifying an exclusion, and unmapped users can be defaulted to a known user. For more information, see Permissions | Account Map / Group Map.
Here are a few things to consider when creating an account or group map:
A source and destination connection are required and need to match the source and destination of the job that will be referencing the user or group map.
A map can be created before or during the creation of the job.
A map can be used across multiple jobs.
Once updated, the updates will not be reapplied to content that has already been transferred.
User and Group Map Import Templates
Please see Account Map / Group Map | CSV File Guidelines for map templates and sample downloads. You can use these samples to help you build you own mapping files.
User & Group Map Exceptions
A user or group map exception provides the ability to explicitly map a specific user from one platform to another. These are exceptions to the automatic account or group mapping policies specified. User account or group map exceptions can be defined during the creation of the map or can be imported from a comma-separated values (CSV) file. See Account Map / Group Map | Exceptions for more information.
User and Group Map Exclusions
A user or group map exclusion provides the ability to explicitly exclude an account or group from owner or permissions preservation. User account or group map exclusions can be defined during the creation of the map or can be imported from a comma-separated values (CSV) file. See Account Map / Group Map | Exclusions for more information.
Transfer Planner
At the start of a project, it is common to begin planning with questions like "How long should I expect this to take?" Transfer Planner allows you to outline the basic assumptions of any integration, primarily around the initial content copy at the beginning of a migration or first synchronization. It uses basic assumptions to begin visualization of the process without requiring any setup of connections or jobs.
The tool estimates and graphs a time line to complete the transfer based on the information entered in the Assumptions area. The timeline assumes a start date of today and uses the values in the Assumptions section to model the content transfer. The window displays projected Total Transfer in dark blue and Daily Transfer Rate in light blue. Hovering the mouse pointer over the graph displays estimated transfer details for that day. The timeline recalculates if you change any of the values, making simple “what if?” scenario evaluations. If necessary, you can reset the timeline to display the default values for the transfer planner tool.
Note that the Transfer Planner is primarily driven by the amount of data needing to be processed. SkySync has various tools for transferring versions of files (if the platform supports this feature), which can increase the size of your data set. It also has the ability to filter out specific files by their type or by other rules you set. At this stage, a rough estimate of total size is recommended as it can refined later using Simulation Mode.
Simulation Mode
Simulation mode allows you to create a job with all desired configuration options set and execute it as a dry-run. In this mode, no data transfers, no permissions are set, and no changes are made to either the source or the destination. This can be useful in answering several questions about your content prior to running any jobs against your content.
How much content do I have?
An important first step in any migration is to determine how much content you actually have since this can help determine how long a migration will take.
What kinds of content do I have?
Another important step in any migration is to determine what kinds of content you actually have. Many organizations have accumulated a lot of content, and some of that may not be useful on the destination platform. The results of a simulation mode job can help you determine if you should introduce any filter rules to narrow the scope of the job. An example would be if you should exclude executable files (.exe or .bat files) or exclude files older than 3 years old.
What kinds of issues should I expect to run into?
During the course of a migration, there are many things to consider and unknown issues that can arise, many of which will only present themselves once you start doing something with the source and destination. Running a job in simulation mode can help you identify some of those issues before you actually start transferring content. Examples can include:
Are my user mappings configured correctly?
Does the scope of the job capture everything I expected it to capture?
Do I have files that are too large for the destination platform?
Do I have permissions that are incompatible with the destination platform (i.e. ACL vs waterfall)?
Do I have files or folders that are too long or contain invalid characters that the destination platform will not accept?
Creating a Simulation Job
During the last stage of the job creation process, you have the option to enable simulation mode. When a job is in simulation mode, it can be run and scheduled like any other job, but no data will be transferred.
Transitioning a Simulation Job to Transfer Content
After review, a simulation job can be transitioned to a live job that will begin to transfer your content to the destination platform.
Creating a Job
SkySync delivers a user-friendly, web-based experience optimized for PC, tablet, and mobile phone interfaces, so you can monitor and control your file transfers anywhere, from any device.
SkySync’s true bi-directional hybrid/sync capabilities enable organizations to leverage and preserve content across on-premises systems and any cloud service. Seamless to users, new files/file changes from either system are automatically reflected in the other.
SkySync uses jobs to perform specific actions between the source and destination platforms. The most common type of jobs are copy and sync; please see Create New Job | Transfer Direction for more information. All jobs can be configured to run manually or on a defined schedule. This option will be presented as the last configuration step.
To create a job, select the Jobs option from the left menu and click on Create Job. SkySync will lead you through a wizard to select all the applicable options for your scenario.
The main job creation steps include:
Selecting a Job Type
Configuring Locations
Defining Transfer Policies
Defining Job Transfer Behaviors
Selecting Advanced Options
Reviewing the job settings summary
Creating the job and defining the job schedule.
Job Type
Job type defines the kind of job and the actions the job will perform with the content. There are four job types available: basic transfer, folder mapping, user job mapping, and network home drive mapping.
Transfer Direction
The transfer direction determines if content is copied in one direction from the source to the destination or if the content syncs between the source and destination. There is also an option to replicate the folder structure of the source onto the destination. See Create New Job | Transfer Direction for more information.
Define Source & Destination Locations
All platform connections made in the SkySync Platform application will be available in the locations drop-down lists when creating a job. The source defines the current location of the content you wish to transfer, and destination defines the location of where you would like to transfer the content. If your connections were created with administrative privileges, you may have the ability to impersonate another user within your organization.
Configuring Your Locations | Impersonation
Impersonation allows a site admin access to all the folders on the site, including those that belong to other users. With SkySync, a job can be set up using the username and password of the site admin to sync/migrate/copy files to or from a different user's account without ever having the username or password of that user. This is done by enabling the Run as user toggle. Then choose the user.
Job Category
The category function allows for the logical grouping of jobs for reporting and filtering purposes. The category is optional and does not alter the job function in any way. SkySync comes with two default job categories:
Maintenance: SkySync maintenance jobs only. This category allows you to view the report of background maintenance jobs and is not intended for newly created transfer jobs.
Default: When a category is not defined during job creation, it will automatically be given the default category. This option allows you to create a report for all jobs that a custom category was not assigned.
Create Job Category
You can create custom categories to group jobs as needed. You can then filter the Jobs list by category to quickly find groups of jobs.
Job Policies
Job policies define what should happen once items have been successfully transferred and set up rules around how to deal with content as it is updated on the platforms while the job is running. SkySync works on the concept of “deltas,” where the transfer engine only transfers files after they have been updated. File version conflicts occur when the same file on the source and destination platforms have been updated between job executions. Policies define how SkySync handles file version conflicts and determine whether or not it persists a detected file deletion. Each job has its own defined policies, and the settings are NOT global across all jobs.
Conflict Policy | File Version Conflicts
When a conflict is detected on either the source or the destination, the Conflict Policy determines how SkySync will behave. See Conflict Policy for more information.
Delete Policy | Deleted Items
When a delete is detected on either the Source or the Destination, the Delete Policy determines how SkySync will behave. See Delete Policy for more information.
Behaviors
Behaviors determine how this job should execute and what course of action to take in different scenarios. Most behaviors are enabled by default as recommended settings to ensure content is transferred successfully to the destination.
Zip Unsupported Files / Restricted Content
Enabling this behavior tells SkySync to compress any file that is not supported on the destination into a ZIP file format before being transferred. This will be done instead of flagging the item for manual remediation to allow transfer of the file. For example, if you attempt to transfer the file "db123.cmd" from a Network File Share to SharePoint, SkySync will compress the file to "db123.zip" before transferring it over, avoiding an error message.
Allow unsupported file names to be changed
The Segment Transformation policy controls if SkySync can change folder and file names to comply to platform restrictions. This behavior allows SkySync to change the names of folders and files that don't comply with the platform rules for leading and trailing characters or that contain characters that are not supported by the destination. This will be done instead of flagging the file for manual remediation and preventing it from being transferred. There are two options for supporting characters that are not supported by the destination.
The Replace with underscore option replaces invalid characters with underscores. For example, if you attempted to transfer the file "Congrats!.txt" from Box to NFS, SkySync would rename the file to "Congrats_.txt" on the destination.
The Encode invalid character option encodes invalid characters instead of replacing them with an underscore. The UTF8 bytes for invalid characters are used and converted to a hex string. For example, a file names "123白雜.txt" would be converted to "123E799BDE99B9CE8.txt."
Allow truncation
Each platform has limits for path and segment lengths. The truncation option determines how to handle paths and segments that exceed these limits. If disabled (the default option), an item with a path or segment length that exceeds the platform limits will not transfer and will be be flagged for remediation. When enabled, truncation will shorten a path or segment name when it exceeds the maximum number of characters a platform allows by removing the right-most characters until the path meets the length requirements. Enabling truncation reduces the potential for flagged items.
Preserve file versioning between locations
Some platforms keep snapshots of every change made to files. Version preservation determines how file versions are handled in the transfer. If version preservation is enabled for a job, the application will attempt to preserve all or the specified number of versions of the files if supported on both platforms. See Version Preservation for more information.
Allow link detection on supported files
Link detection will scans files and identifies any links in the files. It will run for both simulation and transfer jobs. The Job reports will display link information when available for the job. See Link Detection for more information.
Advanced
These optional job configurations allow you to determine what features you want to preserve, filter, or add during your content transfer.
Filtering
Filtering defines rules for determining which items are included or excluded during transfer. There are multiple custom filter options available. See Job Filters and the following for more information.
Permission Preservation
This setting enables SkySync to determine how permissions are transferred across platforms. See the following resources for more information:
Permissions | Author/Owner Preservation
Permissions | Permissions Preservation
Permissions | Permissions Import
Permissions | Preserve Shared Links
Metadata Mapping
Metadata mapping allows you to import a CSV file that documents source metadata and how you want it applied to the destination. See Metadata Import for more information.
Scripting
Some SkySync features are not available in the user interface. The scripting feature allows the advanced SkySync user to enter advanced transfer features by inserting JSON formatted job controls. Enabling this option will allow you to leverage these features and apply it during job creation. See Advanced Scripting | Additional Configuration Options for more information.
Job Summary
Before you create your job, review all your configurations and adjust them as needed. The Edit option will take you directly to the configuration to make changes. Modifying your job after creation is not supported; however, the option to duplicate your current job will allow you to make any adjustments without starting from the beginning.
Saving the Job
Once you have verified the configuration options, you need to save the job. This allows you to specify a name for the job so it can be distinguished from any other jobs you need to create.
You also have the option of setting a schedule and stop policies for a job. If you choose not to schedule the job, it will run every 15 mins after the last execution completes.
Defining the Job Schedule
A job schedule allows you to control when the job runs. The schedule manages the job runs for you so you don't have to manually execute job runs. The schedule can be set to determine both how often a job runs and when a job should stop running. You have the option of editing schedules for jobs as needed once the job is created. See Scheduling a Job for more information.
Defining Stop Policies
Stop policies determine when a job should stop running. If none of the stop policies are enabled, a scheduled job will continue to run until it is manually stopped or removed. See Job Stop Policy for more information.
Reports
Reporting is paramount with the SkySync Platform. Whether you chose to utilize the SkySync manager application, CLI, or REST API, reporting options are available to help you manage and surface data about your content in real-time.
Job Overview Report
This report provides detailed transfer information for the individual job. The percentages displayed are intended to provide a quick summary of the content. In the event rounding applies to one of the values, the total of the percentages may not equal exactly 100%.
In some circumstances, bytes on the destination can be higher than listed on the source. This discrepancy is caused by property promotion on Word documents. See Report Values | Potential Differences due to Post Processing for more information.
Schedule: Provides information on how many times the job has executed, when the job will run again, and progress towards meeting the job stop policy defined
Transfer Details | Identified Chart: Reflects content identified on the source platform and the status summary for items
Transfer Details | Revised Chart: Reflects content that SkySync revised during transfer to meet destination requirements and user-defined job configurations
Transfer Details | Flagged Chart: Reflect content that SkySync could not transfer. Manual remediation is required
Run Breakdown Report: Provides job history information for each execution for the given job
Values in the run breakdown may differ from values presented in the charts. This is because the run breakdown tracks each individual occurrence where as an item can only exist in a single chart category.
Example: When an item is both truncated AND ignored, it would not show up in the "Revised" chart but would show up in the "Revised" run breakdown
The run breakdown also shows both files and folder values. The charts display files and folder values separately, with the "Transfer Details" dropdown available to switch between display values.
Run Breakdown | Select a specific Job Run
Run Breakdown | Last Activity
Job Content Insights Report
The Content Insight report presents a breakdown of your transferred data for your given job. Where applicable, SkySync will use the content modified date when supported by a given platform. This ensures accurate modified date timestamps when viewing the Content Insights charts. Use the drop-down options to change the chart views.
Job Sharing Insights Report
This report offers a breakdown of all permissions associated to your content. The values presented are based on the source content.
On the Shared Insights tab for a job, the value "Not Shared" represents both items that have no permissions as well as content shared by inheritance from the parent folder. At this time, SkySync only tracks permissions applied during transfer, not permissions that result from inheritance within the hierarchy.
Job User Mappings Report
The User Mappings report for a given job presents the permission breakdown of your content. If any of the following features are enabled, User Mapping report will populate:
Lock Propagation | Mirror Owner
Job Validation Report
Control the level of tracking and reporting for content that exists on both the source and destination platform, including content that has been configured to be excluded from transfer and content that existed on the destination prior to the initial transfer. Items that have been ignored / skipped by policy or not shown because they already existed on the destination can now be seen on reports with the defined categories.
The default validation option is inspect none. This option does not need to be configured in the application user-interface or through the REST API; it is the system default.
This configuration will not track all items but will offer additional tracking with performance in mind. Inspect none will track all items on the source at all levels of the hierarchy but not including those configured to be ignored/skipped through policy. For the destination, all content in the root (files and folders) that existed prior to the initial transfer will be tracked as destination only items and reported as ignored/skipped. For more information, see Job Validation | Item Inspection Policy.
This option has the following features:
Source: All content (files and folders) at all levels in the hierarchy, but not including those configured to be ignored/skipped through policy. However, if the connection does not have access to a given folder in the hierarchy, we cannot track and report these items.
Destination: All content in the root (files and folders) that existed prior to the initial transfer will be tracked as destination only items and reported as ignored/skipped.
Destination: All content (files and folders) lower depths of the directory (sub-folders) that existed prior to the initial transfer will not be tracked.
If the connection does not have access to a given folder in the hierarchy, we cannot track and report these items.
Job Reports - Validation tab: At any time, you may run a validation run, which will trigger a full inspection of all content relating to the option you select for the next run only.
Generate Job Reports
SkySync Reports provide several options to combine multiple jobs into a single report for review. Reports are generated by category, individually selected jobs, or by convention job parent (user job mapping, network home drive mapping, or folder mapping job types). Reports are separated into two categories (report types): transfer jobs and simulation jobs.
Generate Report
Report Type
The report type determines which jobs can be selected for the report and on which tab the information displays. Transfer jobs are the jobs actively transferring content. These reports show under the Reports tab. Simulation jobs imitate transfer. These reports show under the Simulation jobs tab. If you don't specify the report type when creating the report, it is automatically created as a transfer job report and will be added to the Reports tab.
Report Contents
The report contents define which jobs the report includes.
All jobs: All jobs for the selected job type (transfer or simulation) will be added to the report.
Jobs under specific categories: This option allows you to include only jobs that have been assigned to specific categories. Select the category from the list that appears. All available categories display in the list. Only the jobs that match the selected job type (transfer or simulation) will be included. Select Add another category to add additional categories. You can click Remove if you want to remove a selected category.
Children under a specific parent job: This option will add all child jobs under the selected parent job to the report. Only jobs that match the selected job type (transfer or simulation) will display in the list.
Manually select jobs: This option allows you to select individual jobs you want to include on the report. Only jobs that match the job type selected for the report (transfer or simulation) display in the list. For convention jobs, the jobs display in the list as "Parent Job Name | Child Job Name" so you can easily identify each job. You must select at least two jobs. Select Add another job to add additional jobs. If you select more than two jobs, you can click Remove next to jobs to remove a job. (This option is only active for jobs three and on. If you need to remove one of the first two jobs, simply select a different job from the list.)
Remediation
Each item is a package, consisting of the media itself, version history, author, sharing, and any other metadata. SkySync ensures all pieces of the item package are transferred to the destination to preserve data integrity. If SkySync can't transfer an item, the item is set to retry. By default, SkySync will attempt three times to reconcile items that are in pending/retry status. Depending on your job configuration, this may occur with the defined schedule, or you can start the job manually. When an item is in a "flagged" status, this means SkySync has made all attempts to transfer the file without success, and it requires manual remediation. These items can be retried or ignored on subsequent job runs. If retried and successfully transferred, the item will be removed from the flagged items.
All migrations require some amount of manual user intervention to move content that fails to transfer automatically. One of the uses of simulation mode is to get an understanding prior to a live transfer of how many files might fail to transfer and the reasons. This can be used to adjust the job parameters to achieve a higher number of automatic remediation successes.
General Reasons Content does not Transfer
Errors from Source and Destination Platforms
This is a broad error category that indicates SkySync was prevented from reading, downloading, uploading, or writing content during content transfer by either the source or destination platform provider. Each situation is dependent on the storage provider rejection reason and will require manual investigation to resolve.
Insufficient Permissions
Many platforms require additional permissions in order to perform certain functions, even for site administrator accounts. These permissions typically require a special request from the storage provider. For example, content that has been locked, hidden, or flagged to disable download may require this special permission request from your storage provider.
Scenario-Specific Configuration
Content on your source storage platform is diverse, and users across your business will structure their data in a variety of ways. A single one-size-fits-all project configuration may not be suitable and can result in some content not transferring to the destination platform. SkySync Professional Services and Customer Support can assist in assessing these situations to help provide custom, scenario-specific configuration that may workaround the issue that is preventing the transfer.
Disparate Platform Features
Each platform provider has a given set of features that are generally shared concepts in the storage business industry. However, each storage platform may have behavioral or rule differences within these features, and aligning these discrepancies can be challenging. Features such as permission levels (edit, view, view+upload, etc.) may not align as an exact match from the source platform to the destination platform. File size restrictions or file names may need to be altered to conform to meet the destination platform's policies. SkySync will attempt to accommodate these restrictions through configurations in the system, but not all scenarios can be covered in a diverse data set.
Interruption in Service
SkySync must maintain connection to the database at all times during the transfer process. If there is an interruption in service, SkySync will fail the transfer as it is unable to track/write to the database.
How do I validate my content transferred successfully?
Verify the destination
SkySync will report all content that has transferred to the destination. Log into your destination platform and verify the content is located as expected.
SkySync is reporting items in "pending" or "retrying" status, what are my next steps?
Run the job again
SkySync defaults to retrying the job 3 times to reconcile items that are in pending/retry status. Depending on your job configuration, this may occur with the defined schedule or you can start the job manually.
Review the log message
SkySync logs a reason why the item is in pending/retry status. On the job Overview tab, click on the Transfer Details breakdown status "retrying." This will take you to the Items tab with the content filtered to show just items set for retry. Select the item and click the View item history link in toolbox on the right side of the page.
SkySync is reporting items in "Flagged" status. What are my next steps?
When an item is in "flagged" status, this means SkySync has made all attempts to transfer the file without success, and it requires manual remediation.
Review the log message
SkySync logs a reason why the item is in pending/retry status. On the job Overview tab, click on the Transfer Details breakdown status "flagged." This will take you to the Items tab with the content filtered to show just the flagged items. Select the item and click the View item history link in toolbox on the right side of the page. Review the message and determine if you can resolve the issue on the source platform.
Export the flagged items for a job
To export the a flagged item report,
Select Jobs and select the job with the flagged items.
Go to the Items tab.
Filter the list to show only items with the Status “Flagged.”
Select Export this report. This saves the list of flagged items to a CSV file for review.
Review Flagged Items page
The Flagged items menu option will display the flagged items across all jobs. Use the export option to export the report to a CSV file. You can then use the file to remediate issues by correcting issues that are preventing items from transferring and identifying flagged items that don't need to be transferred. Once you’ve worked through the export, return to the Flagged items page to set the job to retry an item or ignore it on the next job run. You can also choose to view the history of the item. You can work with one item at a time or select multiple items and select the appropriate action.
SkySync enables the world’s most successful enterprises with total control over and unbounded access to their content, empowering their evolving modern workforce. We connect disparate storage platforms, unite silos of information, and synchronize content spread across the enterprise—at scale. We enable our customers to achieve singular access and unified control over their content, so they can support the evolving needs of a modern workforce.
To dive in further, view our support documentation or contact us at info@skysync.com for more information about how SkySync can help you connect and transform your content.