Temporary File Sharing

A common use-case for web applications is the need to asynchronously run a long running process for a user and then provide the user with a download link where they may collect their data.

This recipe will combine two of the more advanced s3 functionalities to address this use-case. Namely:

Temporary urls for short lived shares
Bucket lifecycles to enable automatic object deletion after a set lifetime

The s3 object storage service allows users to generate a temporary url which can be shard with colleagues and collaborators, enabling anonymous access to a data object for a restricted amount of time. This can be thought of as a short-lived-share.

Bucket lifecycles can be implemented to provide a lifetime for uploaded objects. Any uploaded object will be deleted n days after upload (where n is configurable).

In this recipe we will step through the creation of a bucket with a lifecycle, uploading of objects and generation of temporary URLs. Although this can all be achieved using command line clients we will eyplore using a python script for the object upload and temporary URL generation.

Create a bucket with a lifetime policy

The lifecyle policy documented here marks any object added to the tempstore bucket for deletion after one day. The lifecycle application is run in the background and asynchronously. That means the time of deletion is fuzzy, but always happens some time after the expiry date is reached.

Using s3cmd

Make a bucket

s3cmd mb s3://tempstore

Create a lifecycle policy called delete-1day.xml. Here is an example for a policy where objects will be deleted after one day:

<?xml version="1.0"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>Delete-After-1-Day</ID>
    <Prefix/>
    <Status>Enabled</Status>
    <Expiration>
      <Days>1</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

And apply the policy to you bucket:

s3cmd setlifecycle delete-1day.xml s3://tempstore

Check that the policy has been applied:

s3cmd getlifecycle s3://tempstore
s3cmd info s3://tempstore

Using MinIO client

Similarly with the minio-clients (where the alias is mys3)

Create a bucket with a lifecycle

mc mb mys3/tempstore

Set a lifcycle policy where objects are marked to be deleted after one day:

mc ilm add mys3/tempstore --expiry-days "1"

Check the lifecycle

mc ilm ls mys3/tempstore
mc ilm export mys3/tempstore

Create temporary URLs

The commands below generate is a URL which may be used to fetch the file via a browser of command line tool such as curl. The URL is only valid for a limited amount of time. Downloads attempted after the expiry date of the URL will result in an Access Denied message.

The s3cmd od MinIO client commands could easily wrapped in a script to create temporary URLs. It is also possible for us to make use of the python API via boto3. Below is a small example script which will both upload an object and create a temporary URL.

Using s3cmd:

To generate a temporary URL which is valid for the next 10 mins

s3cmd signurl s3://tempstore/temporary_file.txt +600

Note: It is advisable to set https support in the .s3cfg file

use_https = True
signurl_use_https = True

Using the MinIO clients:

For info see:

mc share download -h

To create a temporary URL allowed to read the object:

mc share download mys3/mctempstore/temporary_file.txt

Using Boto

#!/usr/bin/env python3

import boto3
import click

# temp-share - example - upload object and generate temp url
# John Alan Kennedy 2022


def put_object(s3, bucket, key):
    s3.put_object(Bucket=bucket, Key=key, Body=open(key, "rb"))


def generate_presigned_url(s3, bucket, key, expires):
    """Generate pre-signed URL for a key in bucket.

    :param s3: s3_client connection
    :param key: Name of the S3 object
    :param bucket: Name of the S3 bucket
    :param expires: Time to expiration of temp url
    """
    url = s3.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": bucket, "Key": key},
        ExpiresIn=expires,
    )
    print("{0!s}{1!s}".format(key, url))


@click.command()
@click.argument("bucket", required=True)
@click.argument("key", required=True)
@click.option("--expires", "-e", default=600, help="Time to expire in seconds.")
def main(bucket, key, expires):
    # S3 connection details
    session = boto3.session.Session(profile_name="rdoopenstack")
    s3 = session.client(
        service_name="s3", endpoint_url="https://objectstore.hpccloud.mpcdf.mpg.de"
    )
    put_object(s3, bucket, key)
    generate_presigned_url(s3, bucket, key, expires)


if __name__ == "__main__":
    main()

The script can be called as follows:

temp-share tempstore temporary_file.txt

The --expires option can be added to define the URL validity lifetime.

The script is very basic. However, it is easy to see how this could form the basis of a more complex application.