Versions Compared

Version Old Version 24 New Version Current
Changes made by

Andrea Harvey

Andrea Harvey

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Convention jobs such as folder mapping are ideal for migrations where you wish to control the transfer at a granular level without the effort of creating individual jobs. SkySync DryvIQ will automatically create a unique job for each folder in your hierarchy, inheriting configurations from the parent/master job. 

...

The parent for a folder mapping job has minimal impact on performance. However, the child jobs it creates follow the same performance consideration as any job type available in SkySyncDryvIQ. 

If SkySync DryvIQ is installed on a single instance, Parallel Writes will be a limiting factor, regardless of how many child jobs are created. In a multi-node scenario, more jobs running concurrently may be configured without impacting performance. 

Please review our SkySync DryvIQ Platform | Scalability and Performance Whitepaper for more information.

...

To set the schedule for the child jobs, set the schedule parameter within the transfer block.  The schedule parameter outside the transfer block is the schedule for the parent job only. In the example below, the parent job is set to run automatically after creation and run every 15 mins thereafter. The child jobs are set to a manual schedule.

Example JSON

POST {{url}}v1/jobs

Code Block
{
    "name":"Convention Job | Folder Mapping",
    "kind": "folder_mapping",
    "transfer": {
      "transfer_type": "copy",
      "audit_level": "trace",
      "batch_mode": "always",
      "conflict_resolution": "latest",
      "delete_propagation": "ignore_destination",
      "failure_policy": "continue",
      "large_item": "skip",
      "lock_propagation": "ignore",
      "max_items_per_container": 10000,
      "performance": {
                "parallel_writes": {
                    "requested": 2
                }
      },          
      "permissions": {
    		"policy": "add",
    		"links": true,
    		"failures": "none"
      },
      "preserve_owners": true,
      "timestamps": true,
      "empty_containers": "create",
      "duplicate_names": "rename",
      "item_overwrite": "overwrite",
      "restricted_content": "convert",
      "segment_transform": true,
      "versioning": {
    		"preserve": "native",
    		"select": "all"
      },
      "account_map": {
         "id": "{{account_map_id}}",
         "type": "account_map"
    },
    "filter": {
                "source": [
                    {
                        "action": "exclude",
                        "rules": [
                            {
                                "type": "filter_hidden"
                            }
                        ],
                        "type": "filter_rule"
                    }
                ],
                "destination": [
                    {
                        "action": "exclude",
                        "rules": [
                            {
                                "type": "filter_hidden"
                            }
                        ],
                        "type": "filter_rule"
                    }
                ]
        },
        "source": {
            "connection": { "id": "{{cloud_connection_source}}" },
            "target": {
                "path": "/SourcePath"
            }
        },
        "destination": {
            "connection": { "id": "{{cloud_connection_destination}}" },
            "target": {
                "path": "/DestinationPath"
            }
        },
        "simulation_mode": false,
        "schedule": {
            "mode": "manual"
        }
    },
    "schedule": {
        "mode": "auto"
    },
    "stop_policy": {
        "on_success": 5,
        "on_failure": 5,
        "on_execute": 25
    },
    "category": {
      "name": "Convention Jobs | Folder Mapping"
    }
}

Info

Duplicate Folder Names

If there are duplicate folder names, SkySync DryvIQ will only create the child job for the first folder it encounters and skip the duplicate folder even if the "duplicate_names" policy is set to "rename." Therefore, it is important that you verify there are no duplicate folder names before creating your folder mapping job. 

Creating a Folder Mapping Job by Including Specific Folders

POST {{url}}v1/jobs

Code Block
{
  "name":"Folder Mapping Convention",
  "kind": "folder_mapping",
  "transfer": {
      "transfer_type": "copy",
      "source": {
        "connection": { "id": "{{nfs_connection}}" }
      },
      "destination": {
        "connection": { "id": "{{cloud_connection}}" },
        "target": {
          "path": "/FolderMappingDestination"
        }
      },
      "schedule": {
        "mode": "manual"
      }
  },
  "schedule": {
    "mode": "manual"
  },
  "convention": {
    "inclusions": [
      "folder1",
      "folder2",
      "folder4"
    ]
  }
}


Creating a Folder Mapping Job by Excluding Specific Folders

POST {{url}}v1/jobs

Code Block
{
  "name":"Folder Mapping Convention",
  "kind": "folder_mapping",
  "transfer": {
      "transfer_type": "copy",
      "source": {
        "connection": { "id": "{{nfs_connection}}" }
      },
      "destination": {
        "connection": { "id": "{{cloud_connection}}" },
        "target": {
          "path": "/FolderMappingDestination"
        }
      },
      "schedule": {
        "mode": "manual"
      }
  },
  "schedule": {
    "mode": "manual"
  },

  "convention": {
    "exclusions": [
      "folder1",
      "folder2",
      "folder4"
    ]
  }
}

Note

Deleting Child Jobs

If you delete a child job from a folder mapping job, the job will eventually be recreated when the parent job runs again. However, the child job will not be recreated until the “Delete Pending Jobs” system job runs. You can filter the Jobs list to show just the system jobs and manually run this job. Once this job completes, the deleted child job for the folder mapping job will be recreated the next time the parent job runs.

...