Exporting Large Data Sets in Chunks

Large data sets will cause the exports to timeout. Another issue with large data sets is that Microsoft Excel only supports two million rows, so not all entries will be included in the CSV file if export exceeds this limit. If you need to export a large data set, you can use the following steps to export the report into sperate CSV files based on an offset value.

1. Navigate to the page that contains the information you want to export.

2. Right-click the Export link on the page.

3. Click Copy link address in the menu that displays. This copies the export URL to your clipboard.
   
   

   Open

   

4. Paste this link into a text editor. It will look something like this: http://localhost:9090/v1/transfers/items.csv?job_categories=9&simulation_mode=0&active=1&access_token=

5. Edit the link to remove the access_token= text.

6. Add &Limit=[value]&offset=[value] to the end of the URL, where the Limit [value] is the number of records you want the export to include and the offset [value] is the record in the data set where the export should start. For example, the URL http://localhost:9090/items.csv?job_categories=9&simulation_mode=0&active=1&Limit=900000&offset=0 will export an file that includes 900,000 records starting with the first record in the data set, where the URL http://localhost:9090/v1/transfers/items.csv?job_categories=9&simulation_mode=0&active=1&Limit=900000&offset=900000 will export a file that includes 900,000 records starting with the 900,000th record in the data set.

7. Paste the URL into your browser. An export will be generated.

8. Once the export is generated, edit the URL again and increase the offset by the number you used as the limit.

9. Repeat steps 7-8 until all the items have been exported.
   For example, if you wanted to chunk the export in increments of 900,000, your edits would look like the following. The Limit value is 900000, and the offset gets incremented by 900000 for each URL:
   Call 1: http://localhost:9090/v1/transfers/items.csv?job_categories=9&simulation_mode=0&active=1&Limit=900000&offset=0

   Call 2: http://localhost:9090/v1/transfers/items.csv?job_categories=9&simulation_mode=0&active=1&Limit=900000&offset=900000

   Call 3: http://localhost:9090/v1/transfers/items.csv?job_categories=9&simulation_mode=0&active=1&Limit=900000&offset=1800000

   Call 4: http://localhost:9090/v1/transfers/items.csv?job_categories=9&simulation_mode=0&active=1&Limit=900000&offset=2700000