aria_s3_copy - Linux

Overview

aria_s3_copy is a command-line utility used to copy objects between Amazon Simple Storage Service (S3) buckets. It relies on the aria2c engine to handle multi-threaded downloads and uploads, optimizing transfer speeds and reliability.

Syntax

aria_s3_copy [options] <source-bucket-path> <source-object-name> <destination-bucket-path> <destination-object-name>

Options/Flags

  • -a, –access-key-id : Specify the S3 access key ID.
  • -k, –secret-access-key : Specify the S3 secret access key.
  • -b, –bucket-name : Define the bucket name.
  • -e, –endpoint : Set the S3 endpoint URL.
  • -r, –region : Indicate the S3 region.
  • -p, –process-number : Specify the number of concurrent threads (default: 1).
  • -m, –max-connection-per-server : Set the maximum number of connections per server (default: 10).
  • -v, –verbose: Enable verbose output.
  • –help: Display help information.

Examples

Simple Copy:

aria_s3_copy my-source-bucket my-source-object my-destination-bucket my-destination-object

Copy with Credentials:

aria_s3_copy \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  my-source-bucket my-source-object my-destination-bucket my-destination-object

Copy with Region and Endpoint:

aria_s3_copy \
  --region us-west-1 \
  --endpoint s3.us-west-1.amazonaws.com \
  my-source-bucket my-source-object my-destination-bucket my-destination-object

Common Issues

  • Invalid Credentials: Ensure you provide valid S3 access key ID and secret access key.
  • Insufficient Permissions: Verify that you have the necessary permissions to copy the objects in the source and destination buckets.
  • Network Connectivity: Check your network connectivity and ensure that you can access the S3 endpoints.

Integration

Combine with rsync for Incremental Updates:

rsync --dry-run source-folder/ s3://destination-bucket/folder/ | aria_s3_copy --process-number 4

Use as a Download Manager with wget:

wget -O - https://s3.example.com/file.txt | aria_s3_copy --destination-bucket-path local-dir --destination-object-name file.txt

Related Commands

  • aws s3 cp: AWS CLI command for copying S3 objects.
  • s3cmd: Open-source command-line tool for managing S3 objects.