How to transfer files to Object storage (S3)?

Direct AWS CLI integration

Where?

Data is delivered on AWS S3 under a bucket with the tenant's name. Credentials for the tenant's bucket on Things Solver DOO infrastructure are delivered through email and SMS.

All data should be stored under the path s3://workforce-hub-content/{tenant_id}/raw-data/.

How often?

All data is delivered once per day, ideally during the night (around 2–3am).

Data storage (S3) instructions

We create the {tenant_id} folder within our workforce-hub-content bucket on S3. Your data should be stored in the folder raw-data. If you have never used S3, there are 4 steps that are useful:

1. The first step is to install the AWS Command Line Interface (aws-cli). On the following link you can find installation steps for each OS. For Windows, you will use the graphical interface; for Linux, it is enough to copy the first block from the documentation.

Installation steps: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

2. After installation, add the user profile via the command:

aws configure import --csv file:///path/to/file/workforce-hub-content-{tenant_id}.csv

The user profle is placed within the mentioned CSV file, and will be provided by Things Solver team after the initial environment setup. This file is used for accessing the S3 bucket.

3. It’s possible to check if the profile is added with the next command:

aws configure list-profiles

The output of the command should be a list of all profiles (if there are more than one), and one of them should be workforce-hub-content-{tenant_id}.

4. To upload files to S3, it is necessary to use the command cp in the next format:

aws s3 cp /path/to/file/data_to_upload.csv s3://workforce-hub-content/{tenant_id}/raw-data/ --profile workforce-hub-content-{tenant_id}     

After setup, it is possible to upload all files from a local directory at once. For CSV files, use the following command:

aws s3 cp /path/to/folder s3://workforce-hub-content/{tenant_id}/raw-data/ --recursive --exclude "*" --include "*.csv" --profile workforce-hub-content-{tenant_id}     

As the bucket has a predefined structure, make sure you are sending data to the correct location. All data you send should be placed under s3://workforce-hub-content/{tenant_id}/raw-data/.