You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 37
Next »
Version: 5.8
Release Date: September 4, 2024
DryvIQ Platform version 5.8 was available on September 4, 2024. This release includes several new features for Discover that allow for even more customized scan setup and viewing options for scan results. In addition, it includes new connector offerings and improved feature functionality for both Discover and Migrate. The release also includes several bug fixes. For detailed information about individual items, refer to the DryvIQ Platform Documentation Center.
Version 5.8 includes a Catalog database index replacement. Replacing the index takes roughly 1 minute per 10 million Catalog items. To avoid the process from timing out during the upgrade, you need to run a script to replace the index before upgrading to 5.8. Please follow the steps below.
Stop all jobs and scans that are running.
Back up the database.
Run the following script to replace the index.
DROP INDEX catalog.idx_in_item_hierarchy_path;
CREATE INDEX idx_in_item_hierarchy_path
ON catalog.in_item_hierarchy USING btree
(path)
INCLUDE(id, parent_id)
TABLESPACE pg_default;
Run the DryvIQ upgrade.
New user account Discover results: A new Accounts page has been added to Discover so you can instantly view Discover scan results by account to help you identify your largest users by both file count and size. This page provides the same filter, refine, customization, saved views, and export options as the Results page, so you have all the same great functionality available to you while viewing your scan results by account. Learn more here.
New Discover scan settings: A new Settings step has been added to the Discover Scan Setup. This step allows you to add filters based on path, pattern, file type, and files size and offers settings to specify how much of a file gets scanned. You can even enable optical character recognition (OCR) if you want PDFs and images to be scanned for text. The settings allow you to have full control over what is included and excluded in your Discover scans. Learn more here.
Enhanced scan editing capabilities: Several editing capabilities have been added to allow to you update scans after you create them. This includes renaming scans and duplicating scans to use as templates, so you don’t have to create new scans from scratch. You can even edit the scan categories once the scan has been created to adjust them as needed. Learn more here.
PII label change for clarity: When using the PII entity type, the “Entity Type” label was renamed “Organization Name” to accurately reflect the information being displayed.
Filter label change for clarity: The “Date Range” filter in the left filter panel has been renamed “Item Date Filter” to clarify that the dates refer to the “created on” and “modified on” dates for the items.
Discover Scan Setup Review page formatting updates: Several formatting updates were made to the Discover Scan Setup Review page to improve readability. The header font for each section was increased and the items under each section now display as bulleted lists.
Simplified save options for new Discover scans: When saving a new scan, the save options will change based on the data source you select. Learn more here.
Improved job filter logic: DryvIQ now applies the job filters to all items that have changed on subsequent job runs to improve transfer results. This means that an item that did not transfer on a previous job run will transfer on a subsequent run if a change causes the item to now meet an include filter criteria. Similarly, an item that originally transferred will be skipped if the update causes the item to now meet an exclude filter criteria.
Improved link remediation performance: Updates were made to improve the link remediation queries. This will result in overall faster performance of the link remediation feature.
New Docusign connector: DryvIQ now offers a read-only Docusign connector that can be used to scan files and templates in your Docusign account and migrate them out of your account and into your preferred content storage platform. Learn more here.
New separate managed Network File System connector: DryvIQ now offers two Network File System connections in Linux environments. The Network File System (SMB) connector is the native connector that uses SMB protocol to access files. It can be used for Migrate, Discover, and Govern. The new Network File System (Managed) connector is a read-only connector that uses SMBLibrary to make the connection, which performs faster than native Samba. The managed connector can be used as a source connection for Migrate jobs or for Discover scans and Govern policies where DryvIQ only needs to read content. The managed connector cannot be used as the destination connection for Migrate jobs or for Govern policies that require actions to be taken against content.
Owner preservation for disabled users on root folder creation for Microsoft Office 365 will now be preserved if the information needed to resolve the user is available through either the EnsureUser CSOM API call or the _api/web/siteusers REST API call from Microsoft.
External email invitations are now properly sent for Microsoft Office 365 connections.
A fix was made to prevent the “Object reference not set to an instance of an object" error could occur during link remediation.
5.7.4 (August 9, 2024)
The Google Drive connector was updated to accommodate paging through file versions from the Google API to ensure all versions transfer when version preservation is enabled.
Updates were made to link remediation to prevent jobs from hanging when certain link patterns are present in documents.
The Links tab for a parent job now displays the links identified for all its child jobs.
The Network File System (SMB) connector was updated to correct a "The path 'xxxx' cannot be made relative to parent 'yyyy'" error that would occur when scanning items for this connector.
5.7.3 (July 11, 2024)
New Jobs Grid and Jobs Export Headers: The Jobs grid was enhanced to add additional column headers to provide more options for viewing information on the page. In addition, the Jobs Export was updated to include the new fields to provide more comprehensive reporting on your jobs. The new fields include:
last_execution
next_execution
current_execution_duration
category
created_on
modified_on
successful_bytes
successful_bytes_all_versions
revised_bytes
revised_bytes_all_versions
revised_items
retry_bytes
retry_bytes_all_versions
retry_items
flagged_bytes
flagged_bytes_all_versions
flagged_items
skipped_bytes
skipped_bytes_all_versions
skipped_items
transfer_rate_bytes_per_second
Duplicate entries are no longer added to the audit logs when access rules are applied to items.
Logs now respect applied filter dates.
Dropbox connections can now successfully be created.
Duplicated jobs using Google Shared Drives as the source platform now start without error.
The Microsoft Office 365 connector can now access classic picture library types.
The Box connector was updated to correct an issue that affected creating impersonation jobs through the REST API. A job will no longer produce an error when created through the REST API with only the username field in the impersonate_as block.
If Microsoft responds with an “item already exists” during a batch migration job, DryvIQ now compares the latest set of item properties (modified, versions and number of bytes) to the item properties in the batch job. If the properties on the existing destination file do not match the properties in the batch job, DryvIQ will generate an error identifying the conflicting properties. In the error, original is the item property in the batch job, and latest is the item property on the destination. The item will be flagged as “Retry” and will be handled like a normal retry item. After selecting the retry action (Retry, Ignore, etc.), the item should be resolved.
The OpenText and ShareFile connectors were updated to ensure folder renames migrate when one of these connectors is used as the source connection for a migration job.
Updates were made to link remediation to prevent malformed links from affecting job performance or preventing jobs from finishing.
5.7.2 (June 7, 2014)
The Jobs export now includes the last failure message and status for each job on the export.
Items flagged for retry that no longer exist will now be marked as “Ignored,” and an audit log message will be added for the item.
Improvements were made to Simulation Jobs to prevent jobs failing with an "Object reference" error when targeting a destination folder that doesn't exist.
The Microsoft Office 365 connector was updated to fix an issue with connecting to a site using a tenant level connection and running a policy that requires access rules.
Updates were made to prevent reporting duplicate items in the policy scan results for PostgreSQL databases when data sources are assigned to more than one policy.
The SharePoint connectors were updated to correct an issue that would sometimes correlate an old version of an item as the current version.
Encoded characters can now be used the destination path when creating a SharePoint connector in DryvIQ.
When paging sites at the Microsoft Office 365 tenant level, all sites for a tenant will be available in the Locations picker.
The Dropbox for Business and Dropbox for Business Teams Folders connectors were updated to produce better error messages when attempting to impersonate an invalid or unknown Dropbox user account during author preservation.
The HCP Anywhere connector was updated to prevent "The application was unable to obtain a valid access token to the storage platform" errors in jobs when HCP Anywhere source system is under load.
The NFS connector was updated to ensure Execute access rights were properly applied to NFS items.
Updates were made to prevent 500 errors and rate limit exceptions from disabling a connector.
DryvIQ will return null to prevent an error for files with unparsable created on, modified on, or last accessed time.
Link remediation was updated to remediate links to files outside the current job.
Improved error messaging was added to identify when there is an issue with the user or group map configuration.
When DryvIQ encounters a write/delete collision during a copy job (one-way transfer), it will prefer the write and treat it as a new item, generating a new platform ID.
5.7.1 (May 17, 2024)
Insights now includes a new Managed Custom Insights option. Custom Insights allow you to create your own custom dashboards to report on the information most relevant to your data discovery needs. Learn more here.
Discover scan setup is fully available through the user interface and includes the ability to customize everything from the data sources and discover categories that should be included in the scan to the schedule the scan will use. Learn more here.
You can now save a view as a document set and use the saved document set as the data source for a new Discover scan. Learn more here.
The Locations page was renamed Scans to reflect that this page lists the Discover scans that have been created.
The Refine feature on the Scans page (formally Locations) has two new options: Has child scans? and Is a Document set? so you can quickly filter the list for scan for mapped locations and document sets. Learn more here.
The “Scan All” options has been removed to prevent all scans from being run accidentally. You can still select and run multiple scans.
Improvements were made to how you filter the Results by location. This option was removed from the filter panel and was added to the new Select scans option on the top of the Results page. The new selection modal simplifies how you select the scans you view by allowing you to search for a specific scan, view all scans, or browse through the 50 most recent scans. Learn more here.
The Results export now defaults to a maximum of 100,000 items to ensure performance. The export will truncate anything selected over this value when generating the export. The maximum value can be adjusted using the catalog:query_max_export value in the configuration settings if a different default value is preferred. Learn more here.
The date fields in the Results export now use ISO 8601 formatting. Learn more here.
The Self-Service Migration Management charts were updated so “Processed” is now “Revised in Transfer” to match the charts on the Overview page of the individual job reports.
The export for jobs that failed to start now includes the last failure message and status for the jobs.
The SharePoint connector was updated to correct an issue that would sometimes correlate an old version as the current when reading an item.
Items that were marked for Retry but no longer exist will now be marked as Ignored with an audit log message.
Updates were made to prevent 500 errors and rate limit exceptions from disabling connections.
When a write event occurs on the source and a move out of job scope event on the destination is detected, the moved file will be deleted.
All sites for a tenant will be available when paging through the Locations picker at the Office 365 tenant level.
Execute access rights are properly applied to NFS items.
Link remediation for files outside the job now properly remediates.
DryvIQ will return null to prevent an error for files with an unparsable created on, modified on, or last accessed date.
5.6.3 (April 4, 2024
A new File Versions entity type is available that provides file version details such as the number of versions for a file and the the total amount of bytes for all versions. In addition to being used as part of the Discover Results to provide version insights, this entity type can be used as part of a Govern policy or you can upload individual files against the entity type if you prefer.
Updates were made to the Validation page to prevent “infinity%” from displaying.
A fix was made to correct an issue with the URL endpoint not providing connection data to the URL resolver.
Items with more than 200 versions will be transferred.
To prevent audit trail issues, the SharePoint Online connectors were updated to delete folder content individually when the retention policy is applied.
The SharePoint Online connectors were updated to fix an issue that prevents DryvIQ from creating libraries that have the same name as the site name.
The Box connector was updated to correct issues with Box Notes rendering due to list items and headings that used advanced formatting.
An update was made to prevent Google native docs from showing with 0 bytes during simulation mode when versioning is enabled. Note that only the size of the current version can be calculated from the API during simulation.
5.6.2 (March 7, 2024)
Additional options have been added on the in the Locations page that allow you to view results and insights for each location. Learn more.
All location pickers have been updated to display the connection user or impersonated user under the path.
A new job setting can be configured for delta job runs to skip destination events in scenarios where the subsequent job run occurs after an initial run into an empty destination. This setting can only be set using the API and must be set per job run. Learn more here.
Records in the Transfer Item audit will now be purged after 90 days. The default value can be updated in the configuration options.
The Items job report page now allows you to display Mime type as one of the column headers. It displays the mime type for five types of Google documents: Google Jamboard, Google Sites, Google Maps, Google Forms, and Lucidchart. Learn more here.
Items reported with an unknown status are now reported on the Overview and Items reports.
The roll up reports now include a User Mappings page like the individual job report. This allows you to review all the user mappings for the jobs included in the report instead of having to go to the individual job to review the user mapping results.
Updates were made to improve the performance of the Links export when an large number of links are included in the exported report.
When you cancel the location picker modal for a Dropbox connection when creating a job, DryvIQ now properly sends Dropbox a cancelation token to cancel any additional paging requests.
The Dropbox connector was updated to request and use refresh tokens for authentication.
Folders with permissions set to Deny now show up as Flagged.
Graphs were updated to show 0 instead of negative numbers.
An error no longer occurs when duplicating jobs in the user interface without editing the root paths.
An update was made to ensure the the metadata filter reason is populated in database when an item is excluded by an “Include” policy.
Sorting by job duration now displays in proper chronological order.
A null check was added to the Google connector to ensure links located on the root of the Google connection can be detected and resolved.
The Job column on the Items page now sorts properly.
5.6 (January 25, 2024)
Discover now supports adding a location as a mapped location to make adding locations to the Discover scan easier. When you execute the mapping against the location, DryvIQ creates a child locations for every folder one level under the primary location. You can then run the Discover scan at the child location. Learn more here.
The DryvIQ Discover results now include duplicate detection. Learn more here.
A new Self-Service Migration Management feature was added so migration administrators can monitor each migration wave separately and easily manage the jobs for the migration waves as needed. Learn more here.
Link remediation has been added to the Link Detection feature, so DryvIQ can now update the identified links, eliminating the need for manual remediation. Learn more here.
An update was made to the “Move” transfer type to support moving duplicate files (multiple files on the source with the same name and extension) to the destination if the platform supports duplicate names.
For SharePoint Online migrations, desktop.ini and .ds_store files will be revised and included in the revised file count.
The Amazon S3 connector now supports permissions. Learn more here.
The SMB Network File System connector now has a Connection Mode setting that allows you to create the connection with read-only access. Learn more here.
DryvIQ now offers a Hitachi HCP Anywhere connector. Learn more here.
5.5.6.4043 (December 7, 2023)
DryvIQ tested and verified the Dropbox connectors (Dropbox for Business and Dropbox for Business Teams Folders) are compatible with the new Dropbox for Business Teams Spaces 2.0. You do not need to update existing Dropbox connections as DryvIQ works with both versions of Dropbox for Business Teams Spaces.
The Google Workspace connector was updated with a new new setting that allows DryvIQ to impersonate a file owner when deleting an item. This prevents items from being flagged when the file owner is different than the current impersonated user, requiring less manual remediation. Learn more here.
Because Google does not allow downloading of native link files that appear on the Google platforms, DryvIQ now filters these files out of the migration job and logs the skipped reason for reference.
A change was made to the SharePoint Online connectors to refresh the server time zone to ensure proper timestamps for on-premises and customer-hosted deployments.
Restoring deleted items to the destination no longer results in multiple copies being created with # appended to the name.
A fix was made to ensure the Reports page in the DryvIQ Cloud environment loads.
Filtered items are now properly labeled as “ignored” rather than being added to the flagged items.
5.5.5 (November 2, 2023)
A new option was added to the Author Preservation settings that allows you to preserve authorship of the destination root folder. Learn more here.
The Box connector was updated to improve how DryvIQ pages large amounts of content to improve transfer performance.
Changes were made to how DryvIQ handles sharing to external users for Microsoft SharePoint Online connectors to ensure requests show up in External Access Invitations.
An update was made to the Box connector to ensure duplicate files are not created when DryvIQ encounters errors when transferring content.
5.5.3 (October 13, 2023)
A new “Account/Group Unmapped” Revision filter has been added to the Items page so the grid can be filtered to show just unmapped files. Learn more here.
To ensure performance, the permissions export available on the Sharing Insights page has been limited to one million rows. Learn more here.
The Network File System (NFS) connector was updated to prevent errors from being produced when using read-only accounts for the connection.
The SharePoint Online connectors were updated with the following fixes:
Microsoft CSOM will be used to ensure external user notifications are sent when this option is enabled for the connection.
Permissions added using shared links will be preserved when migrating between SharePoint connections.
The Office 365 application registration has been updated to show DryvIQ instead of “SkySync.”
The Egnyte connector was updated to fix the following issues with metadata mapping:
Metadata mapping no longer fails.
An issue with setting and retrieving date fields in Egnyte metadata templates has been fixed. This included adding support for metadata timestamp values using the yyyymmdd format.
Requesting the metadata schema for an Egnyte connection through the REST API no longer produces an error.
Root permissions are now properly transferred from the source to a SharePoint Online document library.
Jobs will now correctly read the source and destination targets when only the platform ID is used to create the job through the REST API.
An update was made to the account mapping logic to include the username in the hashing calculation to improve performance for account maps where only the username is available.
5.5.1 (September 13, 2023)
DryvIQ now includes an Intelligent Migration feature that allows you to assign items in the Discover Results to an intelligent migration job. This allows you to take action against your Discover Results to ensure your content is migrated exactly where you want it to be moved based on its make up and sensitivity. Learn more here.
You now have the option of making a saved view private or public when creating it. Private views are available only on the machine where the view was originally saved. Public views are available to all users, so you can create a saved view and allow others to use it to review the results and take actions against the content as needed. Learn more here.
Updates were made to SharePoint online connections to properly manage duplicate folder names on move events.
OneDrive for Business connections now properly display user folders so the file path can be specified when creating a job.
DryvIQ now successfully creates the target folder when the folder name matches the SharePoint Site name.
Migration jobs using SharePoint Online connections now handle European local settings datetime formats correctly on Windows Servers.
Scheduling transfer or policy jobs now correctly adds them to the intended scheduler, eliminating duplicate entries in the database.
The "Downloading content is not supported for file" message is now an error rather than a warning. Items that receive this message will be added to the Flagged Items so you can choose to retry or ignore the items. This change is not retroactive, so items that received this message previous to 5.5.1 will still be flagged with a warning.
Items flagged as “Retry” in a job no longer produce a disconnected tracking error.
5.5 (August 16, 2023)
A new Stale Data insights page was added so you can identify your outdated data, also referred to as redundant, obsolete, and trivial (ROT) data. Learn more here.
The forced column setup on first access has been removed. The page now defaults to showing all columns and content for the columns. You need to click Edit columns and adjust the display to reflect the preferred display.
The column order was changed to display the most important file details first. Now, the Item Name column includes the path, and this column is followed by the File Details elements in a new order: Bytes, Created on, Created by, Modified on, Modified by, Owner, and Last Touched.
A new Owner column was added to the display grid to show who is assigned as the owner of the file. For platforms that don't support a separate owner, this column will default to the author (created by).
The Results grid now includes four new sharing columns as part of the Permissions entity type:
1. Shared via External Links to identify files shared with users outside of the internal domain,
2. Shared via Internal Links to identify files shared with users in the internal domain,
3. Shared Link User Addresses to identify the email addresses for the users to whom the files is shared, and
4. Shared Link Group Names to show the group names with whom the file has been shared.
Bytes formatting was updated to follow a more standard byte format. Instead of reporting the size in just bytes, we now format the size into a corresponding unit (B, KB, MB, etc.).
The Sharing and Permissions filter was renamed Sharing and Access Control. As part of this rename, the sections were reorganized and updated to make the filters more effective. Part of the filter updates was the addition of operators that allow you to add more specific filtering for the user and group sharing. Learn more here.
The Filters and Search buttons were renamed Filter and Refine (respectively). In addition, highlighting was added to the buttons to make it clear they have been selected.
The filter pills for a location only include the contextual path as this is the most important part of the location, and it should be visible. This applies to all places in Discover where locations are displayed.
You can now choose to save a view as a new view or an existing view. If no saved views exist, you have to save the view as a new view. If there are existing views saved, you can choose to update a view (save the current view as an existing view) or create a new saved view. Learn more here.
DryvIQ now uses natural sorting. Learn more about how natural sorting works here.
The scan status is now available on the Discover Locations page. The Scan status column display when the last scan ran for each location. If a scan is in progress, this field will display a progress bar to indicate the scan progress. If scans are running for multiple locations, each location will display its own progress bar. The Discover Locations page refreshes every minute. Learn more here.
The last scanned date has been added to the top of the Policy reporting page (next to the active toggle). This allows you to see how old the scan results are when reviewing the policy reports. This date does not automatically refresh. The data only updates if you navigate to another page or manually refresh the page.
Updates were made so you don't lose your place when editing a policy. Before, when you clicked Apply changes, the current tracking group would close, and the screen would reset to the top of the page. Now, the tracking group remains open, and the screen stays at the same location on the on the page.
When editing policies, new logic was added so the Apply changes button is unavailable if no changes have been made to the page. This provides a visual indicator whether edits have taken place and if saving is required.
The scan status is now available on the Govern Data Sources page. The Status column display when the last scan ran for each location. If a scan is in progress, this field will display a progress bar to indicate the scan progress. If scans are running for multiple locations, each location will display its own progress bar. The Govern Data Sources page does not refresh, so a manual refresh is required to view progress. Learn more here.
A new Active User Status entity type was added. It will identify if the owner and/or creator of a file is active and can identify if users with whom the file has been shared are active.
The File Details entity type includes two new fields:
1. Owner: This will identify the owner of a file. Some platforms (such as Box and Google) allow an owner who is different from the creator to be assigned to a file. If owner is not supported, the creator will be considered the owner.
2. Last Touched: This is the last time the file was touched. It uses the most recent accessed date or last modified date based on which date the platform supports.
The Licensing page now displays the byte limits set for both Migrate and Govern/Discover. We also optimized how we calculate the volume used, which may cause your volume used to go up slower than it did previously. There will be no retroactive changes to your calculations, but this will affect all jobs run after you upgrade to the 23.08.16162131 release. Please see the License page for additional information on how usage will be calculated going forward.
For jobs using Google Drive as the source connection, a permission reset will now properly reapply permissions to converted Google Docs.
Updates were made to ensure newly uploaded items on the source are transferred on delta runs.
If a job that is part of a connection pool contains an invalid URL, the job will not run and will be flagged as “Failed to start.” This message will be listed in the log, and the failed job will not impact the other jobs associated with the connector.
Upgrade scripts no longer fail due to a NULL extension.
Updates were made to the Box connector to correct an issue where only 100 permissions (collaborations in Box) were being returned.
5.4.5 (August 3, 2023)
When a permissions reset is performed after an initial job execution, newly applied permissions are now correctly recorded in the audit log.
If a job exceeds the maximum number of items per container based on the platform limit, the containers will be flagged as “Limit Failures” in the user interface and in the logs.
Updates were made to prevent JSON files in private cloud environments from including duplicate and empty keys.
5.4.4 (June 29, 2023)
Connector credentials can now be updated to fix a disconnected connection.
Samba (Linux NFS): The connector was updated to prevent link detection from failing.
Samba (Linux NFS): Updates were made to the connector to improve its handling of locked files so it properly reports them as locked. Update logging was also added as part of this fix to assist with troubleshooting for this connector.
