Using the S3 Proxy

Once the S3 proxy is configured and running, switching your code to point to the proxy instead of S3 should involve only minimal changes. Documented below is an overview of those changes with code samples.

Bucket addressing

When using AWS S3, the bucket name and region are typically part of the URL as part of Amazon's virtual bucket addressing. With that, an S3 bucket named "antimatter" in the "us-west-2" region would be addressable at https://antimatter.s3.us-west-2.amazonaws.com

Path style bucket addressing (e.g. https://s3.us-west-2.amazonaws.com/antimatter) is possible, but Amazon has deprecated support for this.

AWS client library setup

Python

import boto3

# This assumes that the proxy can be accessed at "http://s3proxy:9234" and a bucket
# named "test_bucket" exists in region "us-west-2"
PROXY_ENDPOINT = "http://s3proxy:9234"
BUCKET = "test_bucket"
REGION = "us-west-2"
TEST_FILE = "test_file"
TEST_TEXT = b"Antimatter test"

# Create client, pointing at the proxy
# Note: to configure an access key and secret, pass the "aws_access_key_id" and
# "aws_secret_access_key" args.
client = boto3.client("s3", region_name=REGION, endpoint_url=PROXY_ENDPOINT)

# Test client connection and functionality by creating a file in the bucket, then
# downloading the file and comparing the contents
client.put_object(Bucket=BUCKET, Key=TEST_FILE, Body=TEST_TEXT)
response_text = client.get_object(Bucket=BUCKET, Key=TEST_FILE).get("Body").read()
assert response_text == TEST_TEXT

# Check that the file was encrypted on S3 by downloading it with a regular client
client_plaintext = boto3.client("s3", region_name=REGION)
response_plaintext = client_plaintext.get_object(Bucket=BUCKET, Key=TEST_FILE).get("Body").read()
assert response_plaintext != TEST_TEXT

NodeJS

import {
    GetObjectCommand,
    PutObjectCommand,
    S3Client,
} from "@aws-sdk/client-s3";
import { strict as assert } from "node:assert";

// This assumes that the proxy can be accessed at "http://s3proxy:9234" and a bucket
// named "test_bucket" exists in region "us-west-2"
const PROXY_ENDPOINT = "http://s3proxy:9234";
const BUCKET = "test_bucket";
const REGION = "us-west-2";
const TEST_FILE = "test_file";
const TEST_TEXT = "Antimatter test";

// Create client, pointing at the proxy
/*
 Note: to configure an access key and secret, add
 credentials: {
    accessKeyId: <your access key>,
    secretAccessKey: <your secret>,
 },
 to the configuration passed to the S3Client
 */
const client = new S3Client({
    region: REGION,
    endpoint: PROXY_ENDPOINT,
    forcePathStyle: true,
});

// Test client connection and functionality by creating a file in the bucket, then 
// downloading the file and comparing the contents
await client.send(
    new PutObjectCommand({
        Bucket: BUCKET, 
        Key: TEST_FILE,
        Body: TEST_TEXT,
    })
);
const { Body } = await client.send(
    new GetObjectCommand({
        Bucket: BUCKET,
        Key: TEST_Key,
    })
);
assert.equal(TEST_TEXT, Body);

// Check that the file was encrypted on S3 by downloading it with a regular client
const plaintextClient = new S3Client({ region: REGION });
const { Body: plaintextBody } = await plaintextClient.send(
    new GetObjectCommand({
        Bucket: BUCKET,
        Key: TEST_Key,
    })
);
assert.notEqual(TEST_TEXT, plaintextBody);

Generating presigned URLs

Generating a presigned URL can be performed by building it client side, or by calling the correct endpoint in the S3 Proxy (for presigned GETs and PUTs; support for POSTs is not yet implemented).

For more about presigned URLs on S3, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html

Code examples

The code examples assume the following:

• the S3 Proxy can be reached at http://s3proxy:9234
• the presigned endpoints on S3 Proxy can be reached at http://s3proxy:9235
• the bucket 'test_bucket' exists
• the file 'test_file' can be accessed and/or created
• for Python examples, an S3 client named 'client_plaintext' that points to AWS directly has been created, as in the code examples above
• for NodeJS examples, an S3 client named 'clientPlaintext' that points to AWS directly has been created, as in the code examples above

Generating a presigned GET request:

HTTP with curl

curl --request GET \
  --url http://s3proxy:9234/am-gen-presign/get/test_bucket/test_file

Python

from datetime import datetime, timedelta
import hashlib
import re

ttl = 100  # time in seconds before the presigned URL expires
ANTIMATTER_PRESIGN_SECRET = "my_secret"  # the same secret configured on the S3 Proxy
domain_id = "dm-xxxxxxxx"  # your domain ID

amz_url = client_plaintext.generate_presigned_url(
    ClientMethod="get_object",
    Params={
        "Bucket": "test_bucket",
        "Key": "test_file",
    },
    ExpiresIn=100
)

token_pattern = re.compile(r".*x-amz-signature=([a-f0-9]+).*", re.I)
m = token_pattern.match(amz_url)
amz_token = m.group(1)
timestamp = (datetime.now() + timedelta(seconds=ttl)).isoformat()
token_hash = hashlib.sha256(f"{ANTIMATTER_PRESIGN_SECRET}{timestamp}{amz_token}".encode("utf-8")).hexdigest()

presigned_url = f"http://s3proxy:9235/am-presigned/get/?dm={domain_id}&ts={timestamp}&tk={token_hash}&amz={amz_url}"

NodeJS

import { GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { createHash } from 'node:crypto';

const ttl = 100; // time in seconds before the presigned URL expires
const ANTIMATTER_PRESIGN_SECRET = "my_secret"; // the same secret configured on the S3 Proxy
const domain_id = "dm-xxxxxxxx"; // your domain ID

const command = new GetObjectCommand({
    Bucket: "test_bucket",
    Key: "test_file",
});
const amzUrl = await getSignedUrl(clientPlaintext, command, { expiresIn: ttl })

const reToken = new RegExp('.*x-amz-signature=([a-f0-9]+).*', 'i');
const matches = reToken.exec(amzUrl);
const amzToken = matches[1];
const timestamp = new Date(new Date().getTime() + ttl * 1000).toISOString();
const tokenHash = createHash('sha256').update(`${ANTIMATTER_PRESIGN_SECRET}${timestamp}${amzToken}`).digest('hex');

const presignedUrl = `http://s3proxy:9235/am-presigned/get/?dm=${domainId}&ts=${timestamp}&tk=${tokenHash}&amz=${amzUrl}`;

Generating a presigned PUT request:

HTTP with Curl

curl --request GET \
  --url http://s3proxy:9234/am-gen-presign/put/test_bucket/test_file

Python

from datetime import datetime, timedelta
import hashlib
import re

ttl = 100  # time in seconds before the presigned URL expires
ANTIMATTER_PRESIGN_SECRET = "my_secret"  # the same secret configured on the S3 Proxy
domain_id = "dm-xxxxxxxx"  # your domain ID

amz_url = client_plaintext.generate_presigned_url(
    ClientMethod="put_object",
    Params={
        "Bucket": "test_bucket",
        "Key": "test_file",
    },
    ExpiresIn=ttl
)

token_pattern = re.compile(r".*x-amz-signature=([a-f0-9]+).*", re.I)
m = token_pattern.match(amz_url)
amz_token = m.group(1)
timestamp = (datetime.now() + timedelta(seconds=ttl)).isoformat()
token_hash = hashlib.sha256(f"{ANTIMATTER_PRESIGN_SECRET}{timestamp}{amz_token}".encode("utf-8")).hexdigest()

presigned_url = f"http://s3proxy:9235/am-presigned/put/?dm={domain_id}&ts={timestamp}&tk={token_hash}&amz={amz_url}"

NodeJS

import { PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { createHash } from 'node:crypto';

const ttl = 100; // time in seconds before the presigned URL expires
const ANTIMATTER_PRESIGN_SECRET = "my_secret"; // the same secret configured on the S3 Proxy
const domain_id = "dm-xxxxxxxx"; // your domain ID

const command = new PutObjectCommand({
    Bucket: "test_bucket",
    Key: "test_file",
});
const amzUrl = await getSignedUrl(clientPlaintext, command, { expiresIn: ttl })

const reToken = new RegExp('.*x-amz-signature=([a-f0-9]+).*', 'i');
const matches = reToken.exec(amzUrl);
const amzToken = matches[1];
const timestamp = new Date(new Date().getTime() + ttl * 1000).toISOString();
const tokenHash = createHash('sha256').update(`${ANTIMATTER_PRESIGN_SECRET}${timestamp}${amzToken}`).digest('hex');

const presignedUrl = `http://s3proxy:9235/am-presigned/put/?dm=${domainId}&ts=${timestamp}&tk=${tokenHash}&amz=${amzUrl}`;

Generating a presigned POST request:

HTTP with Curl

# Not supported; this must be generated client side

Python

from datetime import datetime, timedelta
import hashlib
import re

ttl = 100  # time in seconds before the presigned URL expires
ANTIMATTER_PRESIGN_SECRET = "my_secret"  # the same secret configured on the S3 Proxy
domain_id = "dm-xxxxxxxx"  # your domain ID

amz_req = client_plaintext.generate_presigned_post(
    Bucket="test_bucket",
    Key="test_file",
    ExpiresIn=ttl,
)

fields = amz_req["fields"]
amz_token = fields["x-amz-signature"]
timestamp = (datetime.now() + timedelta(seconds=ttl)).isoformat()
token_hash = hashlib.sha256(f"{ANTIMATTER_PRESIGN_SECRET}{timestamp}{amz_token}".encode("utf-8")).hexdigest()

fields["amz"] = amz_req["url"]
fields["dm"] = domain_id
fields["ts"] = timestamp
fields["tk"] = token_hash

# This would then be POSTed to http://s3proxy:9235/am-presigned/post
# If using the 'requests' library, this would look like:
#   `requests.post("http://s3proxy:9235/am-presigned/post", data=fields, files={"file": b"file content"})

NodeJS

// This examples uses the aws-s3-form library for generating presigned POSTs
import AwsS3Form from "aws-s3-form";

import { createHash } from 'node:crypto';

const ttl = 100; // time in seconds before the presigned URL expires
const ANTIMATTER_PRESIGN_SECRET = "my_secret"; // the same secret configured on the S3 Proxy
const domain_id = "dm-xxxxxxxx"; // your domain ID
const AWS_ENDPOINT = "https://test_bucket.us-west-2.amazonaws.com"

const date = new Date();
date.setTime(date.getTime() + 60 * 60 * 1000);
const timestamp = date.toISOString();

const uploadData = {
    policyExpiration: date,
    keyPrefix: `${BUCKET}/`,
    acl: "private",
};

// Note this requires the access key and secret when creating the AWS S3 form generator
let awsS3Form = new AwsS3Form({
    accessKeyId: ACCESS_KEY,
    secretAccessKey: SECRET_KEY,
    region: REGION,
    bucket: BUCKET,
});
let formData = awsS3Form.create(TEST_KEY, uploadData);
const amzToken = formData.fields["X-Amz-Signature"];
const tokenHash = createHash("sha256").update(`${ANTIMATTER_PRESIGN_SECRET}${timestamp}${amzToken}`).digest("hex");

// Set the S3 Proxy specific form fields
formData.fields["amz"] = `${AWS_ENDPOINT}`;
formData.fields["key"] = TEST_KEY;
formData.fields["dm"] = domain;
formData.fields["ts"] = timestamp;
formData.fields["tk"] = tokenHash;

// Create the FormData and add the file
const form = new FormData();
Object.keys(formData.fields).forEach((x) => form.append(x, formData.fields[x]));

// This would then be POSTed to to http://s3proxy:9235/am-presigned/post
// If using the 'ky' library, this would look like:
// form.append("file", "file content");
// await ky.post("http://s3proxy:9235/am-presigned/post", {body: form});