v0.2.0 (2023-03-22)
This release marks the introduction of the S3Watcher into the cloud data processing pipeline, in preparation for the upcoming second dataflow test. A file watcher system that can be configured to watch an AWS S3 bucket new files and then download them onto a local machine. This release includes the following features:
- This is a filewatcher utility that polls an SQS queue for bucket events and then downloads the files to a defined local directory.
- It is designed to be used and run within a docker container, but it is not required.
- It is also required that you have the correct AWS credentials set up for the user that will be running the script, and a valid access policy for your SQS queue, that allows the S3 bucket to send messages to the queue.
Full Changelog: https://github.com/HERMES-SOC/s3watcher/commits/v0.2.0