S3 Usage Howto

There are several ways to view your storage utilization in S3.

Here we’ll summarize how to view usage on a global level and also per container/bucket using standard S3 tools. In addition a recipe is provided to allow access to S3 usage statistics following the installation of an extension to the standard aws-cli.

The standard tools tend to be slower than the extension but allow for querying of individual buckets rather than the total account quota.

See the note at the end of this article about storage quotas.

Using standard S3 tools

Firstly using the openstack command:

To view global storage statistics

$ openstack object store account show
+------------+---------+
| Field      | Value   |
+------------+---------+
| Account    | v1      |
| Bytes      | 1048604 |
| Containers | 2       |
| Objects    | 2       |
+------------+---------+

To view container level statistics:

$ openstack container show mycontainer
+--------------+-------------+
| Field        | Value       |
+--------------+-------------+
| account      | v1          |
| bytes_used   | 28          |
| container    | mycontainer |
| object_count | 1           |
+--------------+-------------+

$ openstack container show mycontainer2
+--------------+--------------+
| Field        | Value        |
+--------------+--------------+
| account      | v1           |
| bytes_used   | 1048576      |
| container    | mycontainer2 |
| object_count | 1            |
+--------------+--------------+

Using the s3cmd command:

To gain a global view

$ s3cmd du 
          28       1 objects s3://mycontainer/
     1048576       1 objects s3://mycontainer2/
------------
1048604      Total

$ s3cmd du s3://mycontainer/
          28       1 objects s3://mycontainer/

To get human readable numbers

$ s3cmd du -H
   28       1 objects s3://mycontainer/
1024K       1 objects s3://mycontainer2/
------------
1024K        Total

Using the minio client mc:

$ mc du -h

To gain a global view.

$ mc du cephs3
1.0MiB

To view global usage (split into containers/buckets)

$ mc du -r cephs3
28B     mycontainer
1.0MiB  mycontainer2
1.0MiB

To view usage per container/bucket

$ mc du -r cephs3/mycontainer2
1.0MiB  mycontainer2

Note: working with sub-containers/buckets

Although S3 containers/buckets are the real units of storage many clients allow you to address data within these buckets in a similar manner to a filesystem (with / being used as a separator for sub-containers/buckets similar to a directory).

To view the storage usage of sub-containers/buckets you can use the s3cmd and/or the minio command.

$ s3cmd ls s3://mycontainer2/phase2/
2021-08-31 09:37            0  s3://mycontainer2/phase2/
2021-08-31 09:40      1048576  s3://mycontainer2/phase2/1mb_testfile_1
2021-08-31 09:40      1048576  s3://mycontainer2/phase2/1mb_testfile_2

Using the s3cmd:

$ s3cmd du s3://mycontainer2/
     3145728       4 objects s3://mycontainer2/

Viewing the sub-container/bucket

$ s3cmd du s3://mycontainer2/phase2/
     2097152       3 objects s3://mycontainer2/phase2/

For human readable output

$ s3cmd -c s3cfg.dev2 du s3://mycontainer2/phase2/ -H
2048K       3 objects s3://mycontainer2/phase2/

Using the minio client:

$ mc du cephs3/mycontainer2/phase2/
2.0MiB  mycontainer2/phase2

Using the usage-stats extension

Ceph’s RadosGW does have a custom S3 API endpoint that can be used to efficiently retrieve that information (like df instead of du): https://docs.safespring.com/storage/usage-statistics/

Once installed the extension can be used from the aws client as follows:

$ aws --profile <my-profile> --endpoint-url https://objectstore.hpccloud.mpcdf.mpg.de s3api get-usage-stats
{
    "Summary": {
        "QuotaMaxBytes": 214748364800,
        "QuotaMaxBuckets": 1000,
        "QuotaMaxObjCount": 51200,
        "QuotaMaxBytesPerBucket": -1,
        "QuotaMaxObjCountPerBucket": -1,
        "TotalBytes": 1168113870,
        "TotalBytesRounded": 1168130048,
        "TotalEntries": 16
    }
}

The extension can be enabled by placing the ceph-specific json api definition file from [1] in ~/.aws/models/s3/2006-03-01/ as described in [2].

The following python code can also be used to gather the usage-stats:

#!/usr/bin/python3

import boto3
import json

# S3 connection details
session = boto3.session.Session(profile_name='<my-profile>')

s3 = session.client(
    service_name='s3',
    endpoint_url='https://objectstore.hpccloud.mpcdf.mpg.de'
)

print(json.dumps(s3.get_usage_stats(), indent=2))

To enable the usage or profiles please set your profile in ~/.aws/credentials.

Note: Storage Quotas.

It is important to remember that the S3 quotas apply to a whole openstack project and not individual users which are granted access to that project.

Also note that Quotas are applied to:

Storage Volume
Number of Objects

[1] https://raw.githubusercontent.com/ceph/ceph/master/examples/rgw/boto3/service-2.sdk-extras.json

[2] https://github.com/ceph/ceph/tree/main/examples/rgw/boto3