Chapter 5
Highlighted features

5.1 Virtual hosting

Buckets in LibreS3 can be accessed in three ways:

path-style
libres3.example.com/bucket/object

A direct access to the object, which works even if the bucket name is not DNS compatible.

virtual-hosted subdomain style
bucket.libres3.example.com/object

Requires that bucket names are DNS compatible and a DNS/hosts entry for each bucket or a wildcard DNS record (see 3.1)

virtual-hosted FQDN
bucket.example.net/object

Equivalent to bucket.example.net.libres3.example.com.

The client should send a Host: bucket.example.net header, which usually requires a CNAME DNS entry, see 3.1. Requires that bucket names are DNS compatible, and that you use your FQDN as the bucket name, for example:

s3cmd mb s3://bucket.example.net

The latter can be useful for public buckets, see 6.4.

5.2 Streaming and range-requests

LibreS3 supports streaming files from SX by downloading blocks as they are needed (streaming a large file begins as soon as the first batch of blocks is downloaded by LibreS3).

It is also possible to begin streaming a file from an arbitrary position, and retrieve less than the entire file, i.e. range requests.

This works both for authenticated (s3cmd get --continue) and public requests (wget --continue), see 6.5.

5.3 Access logs

Since version 1.1 LibreS3 uses the Apache/NCSA Combined Log Format for its access.log file, for example:

196.168.1.2 - edwin [06/Jul/2015:08:38:20 +0200] "GET /volpublic/?policy HTTP/1.1" 200 190 "-" "Boto/2.34.0 Python/2.7.9 Linux/3.16.0-4-amd64" 
192.168.1.2 - - [06/Jul/2015:09:58:26 +0200] "GET /slider_logo.png HTTP/1.1" 200 11436 "http://volpublic.libres3.example.com:8008/" "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0 Iceweasel/38.1.0"

The first line is an example for an authenticated request, and the second line is an example for an anonymous request (using directory indexing according to the referer).

The format is:

192.168.1.2
IP address of client
-
always unavailable (the identd identity)
edwin
name of the authenticated SX user. If the HTTP result code is 401 or 5xx then it cannot be trusted because the user may not be authenticated yet
[06/Jul/2015:09:58:26 +0200
] the server’s local date, time and timezone
“GET /volpublic/?policy HTTP/1.1”
the request method, path and HTTP version
200
the HTTP result code
190
the size of the reply body sent to the client. Note that if the connection is closed/reset then this shows the truncated size, not the size that would have been sent. This is shown as - when the body is empty.
“-”
no HTTP referer available
“Boto/2.34.0 Python/2.7.9 Linux/3.16.0-4-amd64”

User-Agent of the client (not all S3 clients send a user-agent, for example s3cmd doesn’t)

Access.log entries are written asynchronously, after the entire reply body is sent to the client.

5.4 Listing buckets

s3cmd ls s3:// shows only the buckets that are owned by your user, what is consistent with S3’s behaviour. In case you have access to some buckets in read or write mode and would like to use them with s3cmd or other clients, you can enable listing of all accessible buckets by adding the following to libres3.conf and restarting LibreS3:

allow_list_all_volumes=true

5.5 IPv6 support

LibreS3 supports IPv6 communication with SX version 1.2 or newer (note that an SX node can have either an IPv4 or an IPv6 public address, but not both at the same time). LibreS3 can be reached either by IPv4 or IPv6, depending on how your DNS wildcard record is set up, and whether your S3 client and OS support IPv6.

5.6 Quota support

Each volume in SX has a maximum size (its quota). There is no equivalent concept in S3, thus you need to specify the default volume size (and replica count) for buckets created through S3 in libres3.conf using the volume_size field. This is done automatically if you used libres3_setup to configure LibreS3.

In case a new file would exceed the SX volume’s quota, LibreS3 should report an HTTP 413 error. The quota errors will be reported for both normal and multipart uploads, however in the case of multipart ones, the quota error will only be reported after all parts have been uploaded to LibreS3.

LibreS3 creates an internal volume for each user to store the parts uploaded during a multipart upload. This volume’s name is prefixed by libres3- and is created with the size specified in the volume_size field and replica count 1. You have to ensure that the size is sufficient to hold all in-progress (and interrupted) multipart uploads for a user, otherwise one may run out of space during multipart upload even if the destination volume has plenty of space available.

Interrupted multipart uploads are not automatically removed (because one can resume by providing the upload ID), to clean them first run s3cmd multipart on each bucket in turn to find out the objects and upload IDs, and s3cmd abortmp on each upload ID in turn to actually delete it:

$ s3cmd multipart -c s3://bucket 
$ s3cmd abortmp s3://bucket/object Id