Folder Mapping Jobs
On This Page
Overview
Convention jobs, such as folder mapping, are ideal for migrations where you wish to control the transfer at a granular level without the effort of creating individual jobs. DryvIQ will automatically create a unique job for each folder in your hierarchy, inheriting configurations from the parent/master job.
Folder mapping jobs are a type of convention or master job that creates child jobs a single level deep in the folder hierarchy at the path you specify. The folder mapping job may be controlled and manipulated just like a transfer job, but when executed, it will not transfer data. Instead, each execution creates, modifies, or deletes its child jobs, which are then responsible for transferring data. For example, if your source contains three subfolders, a folder-mapping job would create a child job for each of those three folders in the specified path. As new folders are created in the source, additional child jobs are automatically created for them. Data will be transferred when the child jobs are run.
Performance Considerations
The parent for a folder mapping job has minimal impact on performance. However, the child jobs it creates follow the same performance consideration as any job type available in DryvIQ.
If DryvIQ is installed on a single instance, Parallel Writes will be a limiting factor, regardless of how many child jobs are created. In a multi-node scenario, more jobs can run concurrently without impacting performance.
Please review our DryvIQ Platform | Scalability and Performance Whitepaper for more information.
Creating a Folder Mapping Job
Folder Mapping supports Impersonation, and all job features defined while creating the parent job will be applied to the child jobs it creates.
Defining Your Source and Destination Paths
If you are an administrator using Impersonation, enable Run as user..., and choose the user you wish to access.
Source / Destination Path: If you wish to transfer all content, leave the source path blank. A child job will be created for every top-level folder. If a folder is selected for the source path, a child job will be created for every subfolder within the parent folder.
Child Job Source/Destination Path: This directory within each folder will be used as the source.
Target the root of each folder: The child job will be created for the first-level folder relative to the source path.
Target a specific directory within each folder: If a folder exists in every directory, you can define it with this option.
The Create child job for files located in the root setting allows you to migrate files that exist directly in the root of the selected source location when running a folder-mapping migration job. When this option is enabled, DryvIQ creates a separate child job specifically for root-level files. This child job uses the same filters, transfer settings, and configuration options applied to all other child jobs in the migration, ensuring consistent processing across the entire folder structure. You can enable this option regardless of whether child jobs are being created from the target root or from a specific folder, providing additional flexibility when working with source locations that contain important files at the top level of the directory. The root-level child job appears in the Jobs list with the label “{source directory} [Root Files]” so you can easily identify files that were located at the top level of the source folder.
Create the Folder Mapping Job
After configuring policies, behaviors, and advanced features, you will be prompted to schedule the job. This schedule will be applied to the child jobs. The parent job will run immediately to create the child jobs. After the child jobs are created, the default schedule is set to run every 6 hours to review the source for any new content. The parent and child job schedules can be changed at any time.
Creating a Folder Mapping Job Through the REST API
In general, creating a folder mapping job is not much different than creating a transfer job. The "kind" parameter specifies the job type. It must be set to "folder_mapping."
Adding a Schedule
To set the schedule for the child jobs, set the schedule parameter within the transfer block. If no schedule is provided for the child jobs, they will run every 15 minutes by default. The schedule parameter outside the transfer block applies only to the parent job. If no schedule is provided for the parent job (convention job), it will run every six hours by default.
In the example below, the parent job is set to run automatically after creation and run every 15 minutes thereafter. The child jobs are set to a manual schedule.
Example JSON
POST {{url}}v1/jobs |
|---|
{
"name":"Convention Job | Folder Mapping",
"kind": "folder_mapping",
"transfer": {
"transfer_type": "copy",
"audit_level": "trace",
"batch_mode": "always",
"conflict_resolution": "latest",
"delete_propagation": "ignore_destination",
"failure_policy": "continue",
"large_item": "skip",
"lock_propagation": "ignore",
"max_items_per_container": 10000,
"performance": {
"parallel_writes": {
"requested": 2
}
},
"permissions": {
"policy": "add",
"links": true,
"failures": "none"
},
"preserve_owners": true,
"timestamps": true,
"empty_containers": "create",
"duplicate_names": "rename",
"item_overwrite": "overwrite",
"restricted_content": "convert",
"segment_transform": true,
"versioning": {
"preserve": "native",
"select": "all"
},
"account_map": {
"id": "{{account_map_id}}",
"type": "account_map"
},
"filter": {
"source": [
{
"action": "exclude",
"rules": [
{
"type": "filter_hidden"
}
],
"type": "filter_rule"
}
],
"destination": [
{
"action": "exclude",
"rules": [
{
"type": "filter_hidden"
}
],
"type": "filter_rule"
}
]
},
"source": {
"connection": { "id": "{{cloud_connection_source}}" },
"target": {
"path": "/SourcePath"
}
},
"destination": {
"connection": { "id": "{{cloud_connection_destination}}" },
"target": {
"path": "/DestinationPath"
}
},
"simulation_mode": false,
"schedule": {
"mode": "manual"
}
},
"convention": {
"include_root": true
},
"schedule": {
"mode": "auto"
},
"stop_policy": {
"on_success": 5,
"on_failure": 5,
"on_execute": 25
},
"category": {
"name": "Convention Jobs | Folder Mapping"
}
} |
Duplicate Folder Names
If there are duplicate folder names, DryvIQ will only create the child job for the first folder it encounters and skip the duplicate folder even if the "duplicate_names" policy is set to "rename." Therefore, it is important that you verify there are no duplicate folder names before creating your folder mapping job.
Creating a Folder Mapping Job by Including Specific Folders
POST {{url}}v1/jobs |
|---|
{
"name":"Folder Mapping Convention",
"kind": "folder_mapping",
"transfer": {
"transfer_type": "copy",
"source": {
"connection": { "id": "{{nfs_connection}}" }
},
"destination": {
"connection": { "id": "{{cloud_connection}}" },
"target": {
"path": "/FolderMappingDestination"
}
},
"schedule": {
"mode": "manual"
}
},
"schedule": {
"mode": "manual"
},
"convention": {
"inclusions": [
"folder1",
"folder2",
"folder4"
]
}
} |
Creating a Folder Mapping Job by Excluding Specific Folders
POST {{url}}v1/jobs |
|---|
{
"name":"Folder Mapping Convention",
"kind": "folder_mapping",
"transfer": {
"transfer_type": "copy",
"source": {
"connection": { "id": "{{nfs_connection}}" }
},
"destination": {
"connection": { "id": "{{cloud_connection}}" },
"target": {
"path": "/FolderMappingDestination"
}
},
"schedule": {
"mode": "manual"
}
},
"schedule": {
"mode": "manual"
},
"convention": {
"exclusions": [
"folder1",
"folder2",
"folder4"
]
}
} |
Deleting Child Jobs
If you delete a child job from a folder mapping job, the job will eventually be recreated when the parent job runs again. However, the child job will not be recreated until the “Delete Pending Jobs” system job runs. You can filter the Jobs list to show just the system jobs and manually run this job. Once this job completes, the deleted child job for the folder mapping job will be recreated the next time the parent job runs.