As administrators, we may find ourselves in a situation where we need to download backups of courses from previous years in order to comply with our institution's data retention policies.
These downloaded backups can then be kept for as long as necessary in another location enabled for this purpose, such as an Amazon S3 bucket, a folder on a server at our institution, or a hard drive, without affecting the space contracted for Blackboard.
In SaaS, backups are created automatically on a daily basis and are stored in the Content Collection for access for 30 days.
To make downloading them easier, Blackboard has created a script that allows us to download backups in batches to a local folder or an S3 bucket before they disappear from the server.
Downloads are performed through the REST API and therefore require a few preliminary steps before use, but afterward they can be downloaded without using user credentials. This script can download backups of the courses included in a list provided in a file, or of all courses created between two specified dates.
The script does not create the backups; it only downloads them. If a backup does not exist for a course, it will not be downloaded, even if the course falls between the specified dates or is included in the provided file.
Preparing the REST Application
First, we will register an application in the Developer Portal (you can learn how to register applications and much more about the REST API here) and save the three parameters it provides: Application key, Secret, and Application ID.
Preparing Blackboard
Now, go to Administrator Panel > Integrations > REST API Integrations and create a new integration by adding the Application ID you copied and assigning a user with sufficient permissions to run this integration.
We recommend creating a specific role for this integration, with the minimum privileges necessary to perform the task, and using a user created for this purpose who is assigned the system role we created.
The required privileges are:
- Read permission in Content Collection
- Archive permission for courses and organizations
- Archive permission that includes passwords for courses and organizations
Leave the remaining options as they are by default and Save.
Preparing the PC
You will need to install Python 3.8 or later, pip, and the requests and boto3 packages. Download the script from this support article.
In a few steps, the script receives a token with your key and secret, lists the contents of /internal/autoArchive, and downloads the latest archive for each of the selected courses, along with its log file. Once it finishes, it displays a summary of what was done.
If you set the secret as an environment variable, you will not have to enter it every time you run the script, and it will not appear in your shell history. If you prefer not to add it, the script will ask you for it each time it runs.
On Linux, export AUTO_ARCHIVE_SECRET='<application secret>'
On Windows, $env:AUTO_ARCHIVE_SECRET = '<application secret>'.
Running the Script
The basic invocation would therefore be:
python3 auto_archive_bulk_download.py [--help] --site SITE --key KEY
[ --file FILE | --beginDate YYYY/MM/DD --endDate YYYY/MM/DD ]
[--debug] --output OUTPUT
Where:
--help Print the help
--site SITE URL of the Blackboard environment we want to access, including the protocol.
--key Application key generated in the Developer Portal
--file Path to the text file containing the list of course identifiers. Required if the dates are omitted.
-- beginDate Course creation date from which to download, in YYYY/MM/DD format
--endDate Course creation date through which to download, in YYYY/MM/DD format
--debug Debug mode
--output Path where the backups will be saved. This can be local or an S3 path.
Keep in mind that:
- the script will ask for the secret at the beginning of execution unless you save it in an environment variable
- the file must contain only one courseID per row, with no other characters or separators
- if dates are going to be used, both must be specified. All backups of courses created between the two specified dates will be downloaded.
- local output paths can be absolute or relative, but it is important that the directory where the download will take place already exists.
- Backup files must exist in /internal/autoArchive/COURSE-ID. If they do not exist, the script will not download anything and will show which courses have this situation when it finishes.
- the script will download the newest backup for each course
- remember not to run multiple instances of the script in parallel, as this would cause performance issues.
At the end, a summary of what was done will be displayed on screen:
- How many backups were downloaded
- Courses with errors and the reason for the error
- Courses with an empty folder.
- Courses with a folder but no backups. Remember that only automatic backups are downloaded; all other files are ignored.
- Backups downloaded without their corresponding log file.
- Courses in the file specified in --file that were not found, or courses that were found but fell outside the date range.
- Whether the script stopped earlier than expected and how many courses were not processed
Examples
Downloading a List of Courses in a File
python3 auto_archive_bulk_download.py --site https://mysite.blackboard.com
-- key <application key> --output ~/Downloads/autoArchives
--file ~/Documents/feedFile.txt
This mode requires a feed file. The script processes a file containing a list of Course IDs and downloads the backups for those courses.
The feed file is a plain-text file containing the IDs of the courses to download. One course ID per row, with no other information, delimiter, or header. For example,
2021-ENG001 2021-ENG002 2020-MAT005
It is important to use the Course_ID and not other values.
Downloading Courses Created Between Dates
python3 auto_archive_bulk_download.py --site https://mysite.blackboard.com
-- key <application key> --output ~/Downloads/autoArchives
--beginDate 2020/01/20 --endDate 2021/01/20
Downloading to S3 Buckets
The script can be used so that instead of downloading files to a local directory, the files are uploaded directly to an S3 bucket. You will need to create a bucket and an access key with sufficient permissions to upload files.
To use the bucket:
- You must export the AWS Access Key ID and Secret key as environment variables using the standard AWS names.
- The path to the bucket is specified using standard S3 notation instead of the local path. Example: s3://mybucket/myfolder/
- The slash at the end of the bucket address is important; do not omit it.
Remember to run only one instance of the script to avoid degrading server performance.
Limitations
Remember that REST API applications are limited to 10,000 calls per day. This script uses one call per course plus a few more to find the folders, so approximately 9,900 courses can be downloaded in one day if the calls are not used for anything else. If the limit is reached, the script will stop, indicate how many courses remain unprocessed, and display code 1. Here you can see how to increase the API request limit.
Comments
0 comments
Please sign in to leave a comment.