Skip to content

acme-s3

Efficient operations on S3 with retries, progress tracking and parallel processing

Problem

Interactive with boto3 client can at times lack some features. This library aims to provide a more user-friendly interface to S3 operations.

Features

  • Multithreaded opeations for opeations on multiple files
  • Progress tracking for all operations
  • Retries for recoverable errors
  • Benchmarking tool to test the performance of the operations
  • Graceful error handling with useful messages

Dev environment

The project comes with a python development environment. To generate it, after checking out the repo run:

chmod +x create_env.sh

Then to generate the environment (or update it to latest version based on state of uv.lock), run:

./create_env.sh

This will generate a new python virtual env under .venv directory. You can activate it via:

source .venv/bin/activate

If you are using VSCode, set to use this env via Python: Select Interpreter command.

Example usage

as3 bench --bucket $TEST_AWS_BUCKET_NAME --prefix s3-benchmark

Project template

This project has been setup with acme-project-create, a python code template library.

Required setup post use

  • Enable GitHub Pages to be published via GitHub Actions by going to Settings-->Pages-->Source
  • Create release environment for GitHub Actions to enable uploads of the library to PyPi
  • Setup auth to PyPI for the GitHub Action implemented in .github/workflows/release.yml via Trusted Publisher uv publish doc
  • Once you create the python environment for the first time add the uv.lock file that will be created in project directory to the source control and update it each time environment is rebuilt

  • In order not to replicate documentation in docs/docs/index.md file and README.md in root of the project setup a symlink from README.md file to the index.md file. To do this, from docs/docs dir run:

    ln -sf ../../README.md index.md