S3 Policies

You can manage access to your buckets and objects using policies. Policies are JSON files. In a policy you specify a set of targets (buckets or objects), users, and operations the users are alowed to perform on the targets. A list of supported bucket and object operations are listed in the Ceph docs.

Open/public access

S3 buckets can be opened to the public, allowing read and/or write functionality to any user.

In this article we will explore how this can be achieved using two popular command line clients: s3cmd and the MinIO client, mc.

Since the MinIO client comes with some out-of-the-box “canned” policies we will start with this and then move on to the s3cmd later in the article.

Note: before we start it would be appropriate to warn that allowing public upload to an S3 bucket is something which should be used with utmost caution. Please be careful if you enable this.

Using MinIO client

We assume that you already have the MinIO client installed and correctly configured. Here we assume mpcdf is the name of your storage alias.

Calling mc anonymous documents the clients features for public sharing.

From this manual we can see that three canned policies are available:

public = read/write
download = read only
upload = write only

You can remove all policy from a bucket by using none (i.e. allow only owner access) as the policy.

It is also possible to set a custom policy by using the mc set-json command.

Setting and Viewing Policies

Create a test bucket

mc mb mpcdf/publictest

Set the bucket to public (read/write)

mc anonymous set public mpcdf/publictest

View the policy

mc anonymous get mpcdf/publictest
mc anonymous list mpcdf/publictest

To gain a more detailed view of the policy (the actual json policy)

mc anonymous get-json mpcdf/publictest

This will return the json policy to stdout. You can pipe this into a file and use it as a basis for more complex policies which you can apply using mc anonymous set-json policy.json mpcdf/custombucket. These reference json policies can also be used as a source of policies for the s3cmd, as we will see later.

As a bucket owner you can still manage objects in the bucket using s3 commands.

For public (unauthenticated) users simple curl commands can be used to access the bucket.

To upload files to a public writable bucket.

head -c 1M /dev/random > test.1mib
curl https://objectstore.hpccloud.mpcdf.mpg.de/publictest/test.1mib \
    -X PUT -T test.1mib
rm -f test.1mib

To download files from a public readable bucket.

curl https://objectstore.hpccloud.mpcdf.mpg.de/publictest/test.1mb \
    -o download.file

Information about the bucket itself, including the contents, can be found by accessing the bucket URL via a web browser or curl

curl https://objectstore.hpccloud.mpcdf.mpg.de/publictest/ | xmllint --format -

This will return a XML document.

UsingS3cmd

The s3cmd can be used to set bucket policies but requires that the policy be provided as a json document (no canned policies are available with s3cmd).

Assuming that the json policy is stored in public-policy.json (which can be obtained via the minio client get-json command).

Firstly, to ensure that no policy is set, we can use the s3cmd info command.

s3cmd info s3://publictest

Create the policy file, public-policy.json, here for public read and write access:

{
  "Statement": [
    {
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::publictest"
      ],
      "Sid": ""
    },
    {
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:ListMultipartUploadParts",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::publictest/*"
      ],
      "Sid": ""
    }
  ],
  "Version": "2012-10-17"
}

Now set the policy for the bucket

s3cmd setpolicy public-policy.json s3://publictest

Check the policy

s3cmd info s3://publictest