Chapter 3
Configuration

3.1 Requirements

LibreS3 by default listens on ports 8008 and 8443, which need to be available on a given IP address.

LibreS3 connects to the SX cluster via HTTP(S). You can run LibreS3 and SX on the same or different hosts.

LibreS3 doesn’t store state on the local filesystem, when it does need to share data with other LibreS3 instances (e.g. multipart uploads) it will use an SX volume for that purpose. You can run any number of LibreS3 instances, to provide load-balancing or failover.

LibreS3 1.2+ requires at least SX version 2.0 for all the features to work correctly. libres3_setup will check if the SX cluster is running a supported version.

DNS zone entry

S3 buckets require a wildcard A record pointing to the IP address (1.2.3.4 below) of the host running LibreS3, for example:

*.libres3.example.com. A 1.2.3.4 
libres3.example.com. A 1.2.3.4

If you don’t have control over the DNS you’ll have to modify the /etc/hosts file of each client machine and add a line for each bucket you want to access:

libres3.example.com 1.2.3.4 
bucket1.libres3.example.com 1.2.3.4 
bucket2.libres3.example.com 1.2.3.4

You can also add CNAME for virtual hosting buckets:

bucket.example.net CNAME bucket.example.net.libres3.example.com.

You can of course just add directly A or AAAA entries that point directly to the LibreS3 server, it only matters that clients send the desired bucket name in the Host: header.

3.2 Setting up a LibreS3 node

Setting up LibreS3 is as simple as running the interactive tool libres3_setup. If you provide the path to an existing SX cluster configuration file created by sxsetup, most of the settings will be done automatically.

Please make sure the default volume replica count setting is less or equal to the number of nodes in the SX cluster.

In the examples below we assume that you have an SX cluster already up and running and you want to deploy LibreS3 on libres3.example.com.

Example setup with sxsetup.conf:

# libres3_setup --sxsetup-conf /etc/sxserver/sxsetup.conf 
Successfully loaded SX configuration from ’/etc/sxserver/sxsetup.conf 
 
S3 (DNS) name: libres3.example.com 
 
Locking LibreS3 private settings on SX cluster: sx://admin@192.168.1.192:443/ 
libres3_setup: main: locking cluster for changes 
 
Generating default SSL certificate and key in /etc/ssl/certs/libres3.pem and /etc/ssl/private/libres3.key 
Generating a 2048 bit RSA private key 
.....+++ 
..+++ 
writing new private key to ’/etc/ssl/private/libres3.key 
----- 
Uploaded generated SSL certificate and key to SX cluster 
libres3_setup: main: unlocking cluster for changes 
Settings completed 
 
S3 HTTPS port: 8443 
 
S3 HTTP port: 8008 
 
Default volume size [use K, M, G and T suffixes]: 100G 
 
Default volume replica count: 1 
 
Generating ’/etc/libres3/libres3.conf 
File ’/etc/libres3/libres3.conf already exists, overwriting 
libres3_setup: main: locking cluster for changes 
libres3_setup: main: Updating cluster metadata 
libres3_setup: main: unlocking cluster for changes 
volume_size = 100G 
s3_https_port = 8443 
s3_http_port = 8008 
s3_host = libres3.example.com 
replica_count = 1 
allow_volume_create_any_user = true 
Updating ’/etc/libres3/libres3.sample.s3cfg 
Generating ’/etc/libres3/libres3.sample.boto 
Updating ’/etc/libres3/libres3-insecure.sample.s3cfg 
Generating ’/etc/libres3/libres3-insecure.sample.boto 
 
Do you want to start LibreS3 now? [Y/n] Y 
[....] Restarting LibreS3: libres3No libres3_ocsigen found running; none killed. 
 
Loading configuration from /etc/libres3/libres3.conf 
Waiting for server to start (5s) ... OK

Example without sxsetup.conf: 1

# libres3_setup 
Admin key or path to key-file: 0DPiKuNIrrVmD8IUCuw1hQxNqZfJ0hlBUgyckAolodd4C/4r4ecY3QAA 
 
SX server IP/DNS name: sx.example.com 
 
SX server HTTPS port: 443 
 
Run as user: nobody 
 
Run as group: nogroup 
 
S3 (DNS) name: libres3.example.com 
 
Locking LibreS3 private settings on SX cluster: sx://admin@192.168.1.192:443/ 
libres3_setup: main: locking cluster for changes 
 
Generating default SSL certificate and key in /etc/ssl/certs/libres3.pem and /etc/ssl/private/libres3.key 
Generating a 2048 bit RSA private key 
.....+++ 
..+++ 
writing new private key to ’/etc/ssl/private/libres3.key 
----- 
Uploaded generated SSL certificate and key to SX cluster 
libres3_setup: main: unlocking cluster for changes 
Settings completed 
 
S3 HTTPS port: 8443 
 
S3 HTTP port: 8008 
 
Default volume size [use K, M, G and T suffixes]: 100G 
 
Default volume replica count: 1 
 
Generating ’/etc/libres3/libres3.conf 
File ’/etc/libres3/libres3.conf already exists, overwriting 
libres3_setup: main: locking cluster for changes 
libres3_setup: main: Updating cluster metadata 
libres3_setup: main: unlocking cluster for changes 
volume_size = 100G 
s3_https_port = 8443 
s3_http_port = 8008 
s3_host = libres3.example.com 
replica_count = 1 
allow_volume_create_any_user = true 
Updating ’/etc/libres3/libres3.sample.s3cfg 
Generating ’/etc/libres3/libres3.sample.boto 
Updating ’/etc/libres3/libres3-insecure.sample.s3cfg 
Generating ’/etc/libres3/libres3-insecure.sample.boto 
 
Do you want to start LibreS3 now? [Y/n] Y 
[....] Restarting LibreS3: libres3No libres3_ocsigen found running; none killed. 
 
Loading configuration from /etc/libres3/libres3.conf 
Waiting for server to start (5s) ... OK

You can use --batch and provide all settings on the command line, for the full list of options see libres3_setup --help

To start/stop LibreS32:

# libres3 start 
Starting LibreS3 
LibreS3 started successfully 
# libres3 status 
--- LibreS3 STATUS --- 
LibreS3 is running (PID 28245) 
 
--- LibreS3 INFO --- 
SSL private key: /etc/ssl/private/libres3.key 
LibreS3 logs: /var/log/libres3/ 
# libres3 stop 
Loading configuration from /etc/libres3/libres3.conf 
Sending TERM to PID 28245 ... 
Waiting for PID 28245 ...

If the server doesn’t start, please check the log files for details.

That’s it — your LibreS3 cloud storage is already up and running! You can now connect to it with your favorite S3 client.

3.3 Advanced settings

Internal IP addresses

If you’ve configured SX with public and internal IP addresses LibreS3 can only access SX via its public IP address, so make sure it is reachable from LibreS3 and that you set the public IP address of SX in sx_host.

libres3.conf

When you run libres3_setup it will generate a file /etc/libres3/libres3.conf and store settings in SX cluster metadata.

LibreS3 tries to be compatible with the S3 protocol by default, however sometimes there are features outside of the S3 protocol that would be useful. These can be configured in libres3.conf and all have the allow_ prefix:

allow_list_all_volumes (default=true)
only volumes owned by you are listed by default, see 5.4.
allow_public_bucket_index (default=false)
by default you can’t list a bucket or subdirectory from a browser, see 6.6.
allow_volume_create_any_user (default=true)
by default any authenticated user can create a bucket that doesn’t exist yet. In SX only admin users can perform such an action, and you can configure LibreS3 to provide that behaviour, however some S3 applications may not behave correctly anymore.

There are more settings that tune various aspects of LibreS3, an up-to-date list3 can always be found in the generated libres3.conf: each configuration option has a comment briefly describing its purpose, and settings that are disabled by default are commented out.

You can customize libres3.conf and restart LibreS3 for the changes to take effect. However if you are running multiple LibreS3 nodes a more convenient way to update the settings on the entire cluster is:

# libres3_setup --update key=value sx://admin@sx.example.com

And then run libres3 reload on each node. This will store the LibreS3 settings in SX cluster metadata, which LibreS3 reads on startup or when it receives a SIGHUP signal. The settings from the cluster override the settings from libres3.conf.

For example to enable public bucket indexing:

# libres3_setup --update allow_public_bucket_index=true sx://admin@sx.example.com 
Previous configuration: 
      volume_size = 10G 
      s3_http_port = 8008 
      s3_host = libres3.example.com 
      replica_count = 2 
      allow_volume_create_any_user = true 
      allow_public_bucket_index = false 
 
libres3_setup: main: locking cluster for changes 
libres3_setup: main: Updating cluster metadata 
libres3_setup: main: unlocking cluster for changes 
 
New configuration: 
      volume_size = 10G 
      s3_http_port = 8008 
      s3_host = libres3.example.com 
      replica_count = 2 
      allow_volume_create_any_user = true 
      allow_public_bucket_index = true

SSL certificates

When you run libres3_setup on the first node a self-signed wildcard SSL certificate is generated, and uploaded to the SX cluster private settings (accessible only to admin users). On the next invocation of libres3_setup it will download the SSL certificate and key from the SX cluster (if available), to ensure that the same certificate is used on all LibreS3 nodes.

To reset the SSL certificate and allow generating a new one:

# sxadm cluster --set-param libres3_private= sx://admin@sx.example.com

1you can use sxadm node --info /var/lib/sxserver/storage/ on the SX node to find out the required information

2LibreS3 and SX will communicate using TLS by default. For debugging purposes you can configure SX with sxsetup --no-ssl

3this list doesn’t include settings used for debugging/tuning low-level aspects