Chapter 3
Configuration
3.1 Requirements
LibreS3 by default listens on ports 8008 and 8443, which need to be
available on a given IP address.
LibreS3 connects to the SX cluster via HTTP(S). You can run LibreS3 and
SX on the same or different hosts.
LibreS3 doesn’t store state on the local filesystem, when it does need to
share data with other LibreS3 instances (e.g. multipart uploads) it will use
an SX volume for that purpose. You can run any number of LibreS3
instances, to provide load-balancing or failover.
LibreS3 1.2+ requires at least SX version 2.0 for all the features to work
correctly. libres3_setup will check if the SX cluster is running a supported
version.
DNS zone entry
S3 buckets require a wildcard A record pointing to the IP address (1.2.3.4
below) of the host running LibreS3, for example:
*.libres3.example.com. A 1.2.3.4 libres3.example.com. A 1.2.3.4
If you don’t have control over the DNS you’ll have to modify the
/etc/hosts file of each client machine and add a line for each bucket you
want to access:
libres3.example.com 1.2.3.4 bucket1.libres3.example.com 1.2.3.4 bucket2.libres3.example.com 1.2.3.4
You can also add CNAME for virtual hosting buckets:
bucket.example.net CNAME bucket.example.net.libres3.example.com.
You can of course just add directly A or AAAA entries that point directly to
the LibreS3 server, it only matters that clients send the desired bucket
name in the Host: header.
3.2 Setting up a LibreS3 node
Setting up LibreS3 is as simple as running the interactive tool libres3_setup.
If you provide the path to an existing SX cluster configuration file created by
sxsetup, most of the settings will be done automatically.
Please make sure the default volume replica count setting is less or
equal to the number of nodes in the SX cluster.
In the examples below we assume that you have an SX cluster already
up and running and you want to deploy LibreS3 on libres3.example.com.
Example setup with sxsetup.conf:
# libres3_setup --sxsetup-conf /etc/sxserver/sxsetup.conf Successfully loaded SX configuration from ’/etc/sxserver/sxsetup.conf’ S3 (DNS) name: libres3.example.com Locking LibreS3 private settings on SX cluster: sx://admin@192.168.1.192:443/ libres3_setup: main: locking cluster for changes Generating default SSL certificate and key in /etc/ssl/certs/libres3.pem and /etc/ssl/private/libres3.key Generating a 2048 bit RSA private key .....+++ ..+++ writing new private key to ’/etc/ssl/private/libres3.key’ ----- Uploaded generated SSL certificate and key to SX cluster libres3_setup: main: unlocking cluster for changes Settings completed S3 HTTPS port: 8443 S3 HTTP port: 8008 Default volume size [use K, M, G and T suffixes]: 100G Default volume replica count: 1 Generating ’/etc/libres3/libres3.conf’ File ’/etc/libres3/libres3.conf’ already exists, overwriting libres3_setup: main: locking cluster for changes libres3_setup: main: Updating cluster metadata libres3_setup: main: unlocking cluster for changes volume_size = 100G s3_https_port = 8443 s3_http_port = 8008 s3_host = libres3.example.com replica_count = 1 allow_volume_create_any_user = true Updating ’/etc/libres3/libres3.sample.s3cfg’ Generating ’/etc/libres3/libres3.sample.boto’ Updating ’/etc/libres3/libres3-insecure.sample.s3cfg’ Generating ’/etc/libres3/libres3-insecure.sample.boto’ Do you want to start LibreS3 now? [Y/n] Y [....] Restarting LibreS3: libres3No libres3_ocsigen found running; none killed. Loading configuration from /etc/libres3/libres3.conf Waiting for server to start (5s) ... OK
Example without sxsetup.conf:
# libres3_setup Admin key or path to key-file: 0DPiKuNIrrVmD8IUCuw1hQxNqZfJ0hlBUgyckAolodd4C/4r4ecY3QAA SX server IP/DNS name: sx.example.com SX server HTTPS port: 443 Run as user: nobody Run as group: nogroup S3 (DNS) name: libres3.example.com Locking LibreS3 private settings on SX cluster: sx://admin@192.168.1.192:443/ libres3_setup: main: locking cluster for changes Generating default SSL certificate and key in /etc/ssl/certs/libres3.pem and /etc/ssl/private/libres3.key Generating a 2048 bit RSA private key .....+++ ..+++ writing new private key to ’/etc/ssl/private/libres3.key’ ----- Uploaded generated SSL certificate and key to SX cluster libres3_setup: main: unlocking cluster for changes Settings completed S3 HTTPS port: 8443 S3 HTTP port: 8008 Default volume size [use K, M, G and T suffixes]: 100G Default volume replica count: 1 Generating ’/etc/libres3/libres3.conf’ File ’/etc/libres3/libres3.conf’ already exists, overwriting libres3_setup: main: locking cluster for changes libres3_setup: main: Updating cluster metadata libres3_setup: main: unlocking cluster for changes volume_size = 100G s3_https_port = 8443 s3_http_port = 8008 s3_host = libres3.example.com replica_count = 1 allow_volume_create_any_user = true Updating ’/etc/libres3/libres3.sample.s3cfg’ Generating ’/etc/libres3/libres3.sample.boto’ Updating ’/etc/libres3/libres3-insecure.sample.s3cfg’ Generating ’/etc/libres3/libres3-insecure.sample.boto’ Do you want to start LibreS3 now? [Y/n] Y [....] Restarting LibreS3: libres3No libres3_ocsigen found running; none killed. Loading configuration from /etc/libres3/libres3.conf Waiting for server to start (5s) ... OK
You can use --batch and provide all settings on the command line, for
the full list of options see libres3_setup --help
To start/stop LibreS3:
# libres3 start Starting LibreS3 LibreS3 started successfully # libres3 status --- LibreS3 STATUS --- LibreS3 is running (PID 28245) --- LibreS3 INFO --- SSL private key: /etc/ssl/private/libres3.key LibreS3 logs: /var/log/libres3/ # libres3 stop Loading configuration from /etc/libres3/libres3.conf Sending TERM to PID 28245 ... Waiting for PID 28245 ...
If the server doesn’t start, please check the log files for details.
That’s it — your LibreS3 cloud storage is already up and running! You
can now connect to it with your favorite S3 client.
3.3 Advanced settings
Internal IP addresses
If you’ve configured SX with public and internal IP addresses LibreS3 can
only access SX via its public IP address, so make sure it is reachable from
LibreS3 and that you set the public IP address of SX in sx_host.
libres3.conf
When you run libres3_setup it will generate a file /etc/libres3/libres3.conf
and store settings in SX cluster metadata.
LibreS3 tries to be compatible with the S3 protocol by default, however
sometimes there are features outside of the S3 protocol that would be
useful. These can be configured in libres3.conf and all have the allow_
prefix:
-
allow_list_all_volumes (default=true)
- only volumes owned by you are
listed by default, see 5.4.
-
allow_public_bucket_index (default=false)
- by default you can’t list a
bucket or subdirectory from a browser, see 6.6.
-
allow_volume_create_any_user (default=true)
- by
default any authenticated user can create a bucket that doesn’t
exist yet. In SX only admin users can perform such an action,
and you can configure LibreS3 to provide that behaviour, however
some S3 applications may not behave correctly anymore.
There are more settings that tune various aspects of LibreS3, an up-to-date
list
can always be found in the generated libres3.conf: each configuration
option has a comment briefly describing its purpose, and settings that are
disabled by default are commented out.
You can customize libres3.conf and restart LibreS3 for the changes to
take effect. However if you are running multiple LibreS3 nodes a
more convenient way to update the settings on the entire cluster is:
# libres3_setup --update key=value sx://admin@sx.example.com
And then run libres3 reload on each node. This will store the LibreS3
settings in SX cluster metadata, which LibreS3 reads on startup or when it
receives a SIGHUP signal. The settings from the cluster override the
settings from libres3.conf.
For example to enable public bucket indexing:
# libres3_setup --update allow_public_bucket_index=true sx://admin@sx.example.com Previous configuration: volume_size = 10G s3_http_port = 8008 s3_host = libres3.example.com replica_count = 2 allow_volume_create_any_user = true allow_public_bucket_index = false libres3_setup: main: locking cluster for changes libres3_setup: main: Updating cluster metadata libres3_setup: main: unlocking cluster for changes New configuration: volume_size = 10G s3_http_port = 8008 s3_host = libres3.example.com replica_count = 2 allow_volume_create_any_user = true allow_public_bucket_index = true
SSL certificates
When you run libres3_setup on the first node a self-signed wildcard SSL
certificate is generated, and uploaded to the SX cluster private settings
(accessible only to admin users). On the next invocation of libres3_setup it
will download the SSL certificate and key from the SX cluster (if
available), to ensure that the same certificate is used on all LibreS3
nodes.
To reset the SSL certificate and allow generating a new one:
# sxadm cluster --set-param libres3_private= sx://admin@sx.example.com