Download files with the bulk restore tool

Updated

Overview

The bulk restore tool is a Python application you run in your environment to restore files from multiple archives at once. For example, you can use it to efficiently download all of the data backed up from many devices. This article describes how to install and use the bulk restore tool. 

The bulk restore tool is designed for restoring many archives
The bulk restore tool requires several steps in the Incydr API to complete the initial setup and configuration. If you are restoring less than 50 archives, it is more efficient to perform multiple device restores via the administration console instead.

Requirements

Best practices

Follow these best practices for optimal performance:

  • Keep jobs small: Run small batches of users per job, ideally between 1 to 10 users. This increases the likelihood of job completion without errors. Also, in the case of a failure, it makes it easier to identify and retry specific users.
  • Process users individually: For optimal results, consider processing users one at a time. You can also script a process to iterate through a list of users, executing one job at a time. 
  • Restore files locally: For best performance, restore files to a local drive rather than a network share. If you must restore files to a network drive, use the --database-target-folder option to keep the metadata database on the local drive where the bulk restore tool is running.
  • Use multiple machines: For larger jobs that require quick completion, run smaller jobs across multiple machines simultaneously instead of running a single large job on one machine.

There are other factors that can also affect restore performance, including the size of the archive, the size and number of versions of individual files, and the restore destination. For example, a restore job for 4 users with large archives could take significantly longer than a job for 10 users with small archives.

Create an API client

The bulk restore tool requires an API client for authentication. To create an API client with bulk restore permissions:

  1. Sign in to the Incydr console.
  2. Select Administration > Integrations > API Clients.
  3. Click Create new API client.
  4. Enter a client name. For example, "Bulk restore client." Optionally, enter a description.
  5. In the Bulk Restore row, select both Read and Write.
  6. Click Save.
  7. Save the Client ID, Secret, and Base URL in a secure location. They are required in the configuration steps below.

Video

Watch the video below for an in-depth training and demo of the bulk restore tool.

Install and configure the bulk restore tool

Use the bulk restore tool to efficiently download all of the data backed up from several devices.

Step 1: Install the bulk restore tool

Use the following command to install the bulk restore tool: 

python3 -m pip install --extra-index-url=https://pypi.us.code42.com bulk-restore-tool

Step 2: Define configuration 

Create a configuration file that allows the bulk restore tool to authenticate with the server and defines other settings. Update the brt_url value to the URL of your Incydr console.

cat <<EOT >> .env
brt_url="https://example.code42.com"
brt_api_client_id="API_CLIENT_NAME"
brt_api_client_secret="API_CLIENT_SECRET"
EOT

Alternatively, you can define these variables as environment variables. 

Use cases

Use the bulk restore tool to restore files from multiple archives at once. For example: 

  • Get all the data for a defined list of users
  • Restore data that matches a regular expression, for example, all PDF files 

Restore files

Use the create-job and start-job commands to restore files using the bulk restore tool. For full details about commands and how to use them, use the brt --help command in the application. For example, brt create-job --help   

Retrieve archive metadata

Before running the restore job, use the following command to retrieve and process the archive metadata. The command creates a .json file in the same directory with the job definition restore_----.json.

brt create-job --type username eemil.patrizia@example.com --include-deleted --target-folder restored_data

Run the restore job 

The following command restores the files:

brt start-job restore_f840f215-409d-49fd-9748-8db1a22019c0.json

File system structure

The following is an example of the directory structure, which includes where the data is saved and in what format. In this example, the bulk restore tool was run against a single user with a single device.  

.
├── bulk_restore.log
├── output                                      <-- Output folder as defined in the "create-job" command
│   └── ExampleRestoreJob                       <-- User-defined job "name" or system defined job ID
│       ├── 1097277766535665992                 <-- Device guid owning the files being restored
│       │   └── C                               <-- Location where files were restored to
│       │       └── Users
│       │           └── epatrizia
│       │               └── Desktop
│       │                   └── back_me_up
│       │                       ├── image-1.png
│       │                       ├── image-2.png
│       │                       ├── image-3.jpg
│       │                       ├── image-3.png
│       │                       ├── image-5.jpg
│       │                       └── image-6.jpg
│       ├── eemil.patrizia@example.com_BUTTERCUP_1097277766535665992.db                       <-- SQLite database of files to be restored (per user-device pair)
│       ├── eemil.patrizia@example.com_BUTTERCUP_1097277766535665992.log                      <-- Log of files that were restored
│       ├── eemil.patrizia@example.com_BUTTERCUP_1097277766535665992_1097277766219362796.json <-- Raw metadata for file versions returned by the server
│       └── eemil.patrizia@example.com_BUTTERCUP_1097277766535665992_error.log                <-- Log of files that errored during the restore
└── restore_ExampleRestoreJob.json

View bulk restores

To view past bulk restore jobs, search the audit log in the Incydr console for the API client used to authenticate the bulk restore. 

Upgrade

To upgrade to the latest version of the bulk restore tool, use the following command: 

python3 -m pip install --extra-index-url=https://pypi.us.code42.com --upgrade bulk-restore-tool

Uninstall

Use the following command to uninstall the bulk restore tool: 

pip3 -m pip uninstall bulk-restore-tool

Troubleshooting

Database lock errors

When targeting a file system that does not properly support file locking (such as many SMB share implementations), you may encounter errors with the bulk restore tool database. To avoid errors, use the --database-target-folder option to store the database elsewhere (for example, on the local disk where the tool is running). For example:

brt create-job --name example --type device 12345 --target-folder /path/to/remote/share --database-target-folder /path/to/local/disk

The filename or extension is too long

When restoring files to Windows devices, some attempts may fail if the file path is too long. These appear in the bulk_restore.log with the error The filename or extension is too long. To work around this issue, follow these steps to enable long paths on the target device:

  1. Open the Windows Registry Editor:
    1. Press Win + R.
    2. Type regedit, then press Enter.
  2. Navigate to the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
  3. Find the LongPathsEnabled Value:
    • If it exists and is set to 1, long path support is already enabled.
    • If it is set to 0 or does not exist, long path support is disabled. To enable it, create a new DWORD value with the name LongPathsEnabled and value 1.

Release history

Version Release date Notes
1.8.3 July 8, 2025
  • Improved restore session stats reporting.
  • Handle folder and file names that consist of only periods when restoring to Windows.
  • Improved logging around how long certain IO operations take.
  • Improved resilience against network instability.
1.8.2 June 9, 2025 Fixed an issue that could occur during error logging.
1.8.1 May 5, 2025 Improved error messaging to make it clear when restoring to a Windows device failed because the file path is too long.
1.8.0 Apr 1, 2025
  • The brt_url now accepts values with a trailing slash.
  • In addition to simply .env, the configuration file now accepts these additional file names:
    • .env.txt
    • env.txt
    • brt.env
    • brt.env.txt
    • .brt.env.txt
1.7.15 Mar 24, 2025
  • Increased the number of automatic retry attempts if a file fails to restore.
  • Improved handling of cross-operating system restores for files and folders that end with a period.
1.7.14 Mar 14, 2025 Fixed another issue where a file backed up from a Mac failed to restore to a Windows device if the filename contained unsupported Windows characters. 
1.7.13 Mar 12, 2025
  • Fixed a rare issue where a file backed up from a Mac with a carriage return in the filename could not be restored to a Windows device.
  • Improved handling of cases where a file selected for restore has the same name as an existing file or directory in the restore location.
  • Improved handling of files with control characters in the filename. 
1.7.12 Feb 24, 2025 Fixed an issue where a file backed up from a Mac failed to restore to a Windows device if the filename contained unsupported Windows characters. 
1.7.11 Feb 18, 2025
  • Improved logging and performance of restore jobs for archives currently undergoing archive maintenance.
  • Improved logging for files that fail to restore.
  • Fixed a rare issue where files could fail to restore.
1.7.10 Feb 12, 2025 Improved handling of restore jobs for archives currently undergoing archive maintenance.
1.7.9 Feb 5, 2025
  • Better handling of files without metadata, such as alternate data streams and resource forks.
  • Improved logging of file paths containing multibyte unicode characters.
1.7.8 Jan 28, 2025
  • Improved handling of files that fail to restore.
  • The log file now includes more details about completed restore jobs.
  • The brt_url now accepts either the URL of your Code42 console or the API client base URL.
1.7.7 Jan 13, 2025
  • Fixed an issue where filenames with special characters did restore properly to FAT32 file systems.
  • Log files for a specific user and device now only contain entries for files that succeeded or failed to restore. Logging for retry attempts moved to the bulk_restore.log file.
1.7.6 Jan 7, 2025
  • Fixed an issue where restore jobs created in version 1.7.4 or earlier did not run in version 1.7.5.
  • Improved performance when restoring a file that previously failed to download.
1.7.5 Jan 2, 2025 Better handling and logging of restore jobs if metadata is unavailable for one or more archives.
1.7.4 Dec 12, 2024 Fixed a rare issue where an error could occur if a restore job included a user with no archives. 
1.7.3 Dec 11, 2024 Fixed a macOS issue where some files without file extensions were not restored properly.
1.7.2 Nov 14, 2024 Fixed a rare issue where some files originally backed up from a Mac could not be restored to a Windows device.
1.7.1 Oct 31, 2024 Fixed an issue introduced in version 1.7.0 where some archives could not be restored.
1.7.0 Oct 29, 2024 Added--include-empty-directories option to restore empty directories.
1.6.2 Oct 2, 2024 Fixed an issue where the log file did not support a custom name if the bulk restore tool was launched via a library in a script (instead of from the command line).
1.6.1 Sept 26, 2024 Better handling of cases where the selected user or device does not have any archives to restore.
1.6.0 Sept 20, 2024
  • Improvements to logging to help with troubleshooting.
  • Removed option to restore to a ZIP file.
  • Removed option to restore all files for all users on a legal hold. Instead, restore files for individual users (not the entire legal hold).
1.5.7 May 7, 2024
  • Fixed an issue where a restore job could fail if a custom location was specified for the bulk restore tool database.
  • Fixed an issue where the log level defined on the command line was not being applied.
1.5.6 May 3, 2024 Improved handling when restoring from devices whose names contain invalid Windows filesystem characters.
1.5.3 Feb 29, 2024 Improved handling of empty files.
1.5.2 Feb 22, 2024 Improved handing of folders with Windows reserved filenames.
1.5.1 Jan 24, 2024 Fixed a rare issue where a restore job could fail with the error "Archive has no file versions, skipping." In these cases, the files did exist in the archive and were available to restore. Now, the restore job completes successfully (as expected).
1.5.0 Jan 2, 2024
  • Regex exclusions and inclusions can now be defined in a file instead of being individually included on the command line.
  • Downloaded JSON metadata for the files in the archive is now stored side-by-side with the metadata database.
  • When running the bulk restore tool within a Python app, the status of the tool as well as the work completed during the current run of the tool is available.
  • Fixed a rare issue where all files were not restored from an archive.
  • Fixed an issue with name collisions if a restored file had the same name as a restored directory.
1.4.0 Dec 13, 2023
  • Improved performance when restoring to a network drive.
  • Fixed a bug that could cause an out-of-memory error when processing very large files.
  • Fixed a rare issue where files in some specific Windows system directories could not be restored.
  • Improved handing of files with Windows reserved filenames.
1.3.3 Nov 9, 2023 Improved error messaging when server_url or api client is missing.
1.3.2 Nov 1, 2023 Added --include-only-deleted option to restore only files marked as "deleted."
1.3.1 Oct 23, 2023 Fixed an issue where files with modification dates before 1980 could not be restored to a zip file.
1.3.0 Oct 3, 2023 Performance improvements.
1.2.0 Sept 27, 2023 Added support for restoring macOS resource forks.
1.1.3 Sept 19, 2023 Fixed an issue where files backed up from a Mac could not be restored to a Windows device if the filename included unsupported Windows characters.
1.1.1 Sept 6, 2023
  • Added the option to store the database in a separate location from the restored files.
  • Various improvements to speed and efficiency.
  • Fixed a bug that could cause the tool to fail when attempting to restore an empty archive.
1.0.9 Aug 21, 2023 Improved error messaging.
1.0.8 Aug 16, 2023 Performance and stability improvements.
1.0.7 July 16, 2023 Performance and stability improvements.
1.0.2 May 22, 2023
  • Event loop improvements.
  • Added support for restoring extended file metadata.
  • Added support for custom logging.
1.0 Mar 8, 2023 Initial release

External resources

Related topics

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.