Chapter 4
Client configuration
4.1 Authentication
S3 clients require that you provide an “Access key” and a “Secret key”. The
“Access key” is your SX username, and your “Secret key” is your SX token
(key).
In case you are accessing your SX cluster using a username and
password, you need to find out your SX token by running sxinit if you
haven’t already done so:
$ sxinit sx://username@clustername $ cat $HOME/.sx/clustername/auth/username
4.2 s3cmd
You can use the generated s3cfg config file
or
configure s3cmd from scratch. Below we assume that your LibreS3 is
running on libres3.example.com and it supports TLS. The important s3cmd
configuration settings are:
use_https Truehost_base libres3.example.com:8443host_bucket %(bucket)s.libres3.example.com:8443access_key <your-sx-username>secret_key <your-sx-key>ca_certs_file s/etc/ssl/certs/libres3.pem
If you don’t use TLS, please use the port 8008 instead of 8443, and set
use_https to False. Once you’ve configured s3cmd check that it properly
connects to LibreS3:
$ s3cmd ls --debug 2>&1 | grep host
Supported s3cmd commands:
-
Bucket
- mb, rb, ls, la, du, info, setpolicy, delpolicy, multipart
-
Object
- put, get, del, rm, sync, info, cp, modify, mv, abortmp, listmp,
sign, signurl
Encrypted files
You can set gpg_passphrase in .s3cfg and use s3cmd --encrypt to upload/download
encrypted files .
Caveats
Certificate verification changed in Python version 2.7.9+, and you’ll need
s3cmd version 1.5.1 or newer to match. The .s3cfg must contain a
ca_certs_file entry pointing to the certificate of the LibreS3 server,
otherwise certificate verification (and thus HTTPS connections) will
fail.
Note that wildcard TLS certificates only match one level, hence you
should avoid using bucket names which contain dots.
For example a certificate for *.s3.example.com is:
- valid for a.s3.example.com
- NOT valid for a.b.s3.example.com
For s3cmd version 1.6.1 you’ll have to use --no-check-hostname if you use
an HTTPS port different from 8443.
4.3 Python-boto
S3 clients using Python boto are configured in ~/.boto, or you can use the
generated
file. A typical configuration is:
[Credentials] aws_access_key_id=<your-sx-username> aws_secret_access_key=<your-sx-key> s3_host=libres3.example.com s3_port=8443 [Boto] is_secure = True ca_certificates_file=/etc/libres3/libres3.pem https_validate_certificates=True
Note that setting s3_host will override the hostname you give to
applications on the command-line. If you are using an application that
allows setting the S3 hostname on the command-line, you might want to
use that instead.
Note that old version of python-boto require the port to be on s3_host
instead of s3_port.
Caveats
Certificate verification changed in Python version 2.7.9+, and you’ll need to
add ca_certificates_file and https_validate_certificates=True
entries in the [Boto] section of your .boto file. Otherwise python-boto
applications will keep trying to reconnect, and eventually fail with:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
The libres3_setup generated python-boto configuration files already
have the necessary entries since LibreS3 version 1.1.
4.4 s3fs-fuse
S3FS can be used to provide a FUSE-based file system backed by
LibreS3.
You must specify the SX username and access token in ~/.passwd-s3fs,
the URL for your LibreS3 server with -o url, and the certificate used by
LibreS3 in CURL_CA_BUNDLE:
$ cat >~/.passwd-s3fs <<EOF admin:0DPiKuNIrrVmD8IUCuw1hQxNqZfJ0hlBUgyckAolodd4C/4r4ecY3QAA EOF $ chmod 0600 ~/.passwd-s3fs $ mkdir ~/libres3-vol1 $ CURL_CA_BUNDLE=/etc/ssl/certs/libres3.pem s3fs -o url=https://libres3.example.com:8443 vol1 ~/libres3-vol1 -o uid=1000
If you want to access LibreS3 unencrypted for debugging purposes then:
$ s3fs -o url=http://libres3.example.com:8008 vol1 ~/libres3-vol1 -o uid=1000
Caveats
If s3fs fails to connect to LibreS3 it quits without an error message, and the
next time you access the mountpoint you get an error, including trying to
start s3fs again:
$ touch ~/libres3-vol1/x touch: cannot touch ’/home/USER/libres3-vol1/x’: Transport endpoint is not connected $ s3fs ... s3fs: unable to access MOUNTPOINT /home/USER/libres3-vol1: Transport endpoint is not connected $ fusermount -u ~/libres3-vol1 $ s3fs ...
You’ll have to manually unmount and it is useful to run s3fs in the
foreground to see the actual error message:
$ fusermount -u ~/libres3-vol1 $ CURL_CA_BUNDLE=/etc/ssl/certs/libres3.pem s3fs -o url=https://libres3.example.com:8443 vol1 ~/libres3-vol1 -o uid=1000 -f [...]
Common errors are using the wrong user/token in ~/.passwd-s3fs, or
s3fs reading the wrong passwd-s3fs (from path/etc for example), or that you
don’t have permissions to access the volume, etc.
4.5 DragonDisk
Add an account configured like this:
-
Provider
- Other S3 compatible service
-
Service endpoint
- libres3.example.com (s3_host from libres3.conf)
-
Access key
- your SX username
-
Secret key
- your SX secret token
-
HTTP port
- 8008 (s3_http_port from libres3.conf)
-
HTTPS port
- 8443 (s3_https_port from libres3.conf)
4.6 SDKs
The recommended way to pass the credentials to the AWS SDKs is to store
them in the file ~/.aws/credentials :
[default] aws_access_key_id=<your-sx-username> aws_secret_access_key=<your-sx-key>
Some examples on how to use LibreS3 with SDKs can be found in this
git repository:
$ git clone http://git.skylable.com/experimental && cd experimental $ git checkout ci $ cd testsuite $ ls aws-php-sdk s3testjava
PHP
After you install the SDK (with composer require aws/aws-sdk-php) you need
to set the endpoint and SSL certificate location to point to LibreS3:
require ’vendor/autoload.php’; use Aws\S3\S3Client; $endpoint=’libres3.example.com’; $s3Client = new Aws\S3\S3Client([ ’version’ => ’latest’, ’region’ => ’us-east-1’, // ’debug’ => true, ’endpoint’ => "https://$endpoint:8443", ’http’ => [’verify’ => ’/etc/ssl/certs/libres3.pem’] ]);
Java
Import LibreS3’s certificate into Java’s store (replace with actual) as root:
# export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 # keytool -storepass changeit -import -alias libres3 -keystore $JAVA_HOME/jre/lib/security/cacerts -file /etc/ssl/certs/libres3.pem
You can use a build.gradle file like this:
group ’com.example’ version ’1.0-SNAPSHOT’ buildscript { repositories { mavenCentral() } dependencies { classpath "io.spring.gradle:dependency-management-plugin:0.5.4.RELEASE" } } apply plugin: "io.spring.dependency-management" apply plugin: ’java’ sourceCompatibility = 1.5 repositories { mavenCentral() } dependencyManagement { imports { mavenBom ’com.amazonaws:aws-java-sdk-bom:1.10.49’ } } dependencies { compile ’com.amazonaws:aws-java-sdk-s3’, ’log4j:log4j:1.2.17’ testCompile group: ’junit’, name: ’junit’, version: ’4.11’ } jar { from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } } manifest { attributes ’Main-Class’: ’com.example.Main’ } }
Then you have to set the endpoint on the S3 client object:
AWSCredentials credentials = null; try { credentials = new ProfileCredentialsProvider().getCredentials(); } catch (Exception e) { // ... } System.setProperty(SDKGlobalConfiguration.ENFORCE_S3_SIGV4_SYSTEM_PROPERTY, "true"); AmazonS3 s3 = new AmazonS3Client(credentials); s3.setEndpoint("https://libres3.example.com:8443/");
Boto3
To connect to the HTTP endpoint (plain text – for testing purposes only!):
endpoint = "http://libres3.example.com:8008" self.session = botocore.session.get_session() self.region = ’us-east-1’ self.client = self.session.create_client(’s3’, region_name=self.region, endpoint_url=endpoint) self.client.meta.events.unregister(’before-sign.s3’, fix_s3_host)