From 73a8a4cde94ddfdea3b859c529777be3a0ede27c Mon Sep 17 00:00:00 2001 From: Delia Passalacqua <delia.passalacqua@garr.it> Date: Mon, 9 May 2022 10:56:35 +0200 Subject: [PATCH] add istructions to use s3 endpoint with rclone --- .../kb/objstore/rclone_quick_tutorial.rst | 98 ++++++++++++------- 1 file changed, 63 insertions(+), 35 deletions(-) diff --git a/web/support/kb/objstore/rclone_quick_tutorial.rst b/web/support/kb/objstore/rclone_quick_tutorial.rst index 19c88c13..36104f59 100644 --- a/web/support/kb/objstore/rclone_quick_tutorial.rst +++ b/web/support/kb/objstore/rclone_quick_tutorial.rst @@ -23,7 +23,7 @@ Install it and then download rclone from: Unzip the archive and open the bash in the rclone directory. From here, you can use rclone command as follow:: - ./rclone + ./rclone Download OpenStack credentials @@ -36,7 +36,7 @@ In order to use **Rclone**, you can either load the configuration from environme In both cases, you should edit the Rclone configuration file:: $ nano .rclone.conf - + Case 1: Take variables from environment """"""""""""""""""""""""""""""""""""""" @@ -49,7 +49,7 @@ Add the following text to *rclone.conf*:: Then execute the content of the file:: $ source app-credentials.sh - + Case 2: Write variables in the configuration file """"""""""""""""""""""""""""""""""""""""""""""""" Take note of these three variables in *app-credentials.sh*:: @@ -57,7 +57,7 @@ Take note of these three variables in *app-credentials.sh*:: OS_REGION_NAME OS_APPLICATION_CREDENTIAL_ID OS_APPLICATION_CREDENTIAL_SECRET - + Add the following text to *rclone.conf*:: [garr-cloud] @@ -69,7 +69,41 @@ Add the following text to *rclone.conf*:: application_credential_secret = <insert here the content of OS_APPLICATION_CREDENTIAL_SECRET> Mind that ``env_auth = true`` takes variables from environment, so you shouldn't insert it in this case. - + +Case 3: Use EC2 credentials +"""""""""""""""""""""""""" +First, you need to install the Openstack cli as described here in the `cli tutorial <https://cloud.garr.it/compute/install-cli/>`. + +Then execute the content of the file:: + + $ source app-credentials.sh + +And create the ec2 credentials:: + + $ openstack ec2 credentials create + +Take note of these two variables:: + + +------------+-----------------------+ + | Field | Value | + +------------+-----------------------+ + | access | <access_key> | + | secret | <secret_key> | + +------------+-----------------------+ + +Add the following text to *rclone.conf*:: + + [garr-cloud] + type = s3 + provider = AWS + access_key_id = <insert here the content of access_key> + secret_access_key = <insert here the content of secret_key> + endpoint = https://swift.cloud.garr.it + +.. note:: + + You can use EC2 credentials to access object storage with other tools. Check `S3 interface to object storage <https://cloud.garr.it/support/kb/objstore/s3_quick_tutorial/>`. + Check configuration ^^^^^^^^^^^^^^^^^^^ .. note:: @@ -116,7 +150,7 @@ Now, suppose you have these files on your local filesystem:: total 8 -rw-rw-r-- 1 ubuntu ubuntu 1103 Nov 13 15:31 file1.txt drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 13 15:32 subdir1 - + /tmp/test_dir/subdir1: total 4 -rw-rw-r-- 1 ubuntu ubuntu 459 Nov 13 15:32 file2.txt @@ -130,8 +164,8 @@ Execute the following command to synchronize it with the remote:: 459 sublevel/subdir1/file2.txt **Mind behaviour of sync!** It makes destination identical to source - - + + Copy files from remote to local ------------------------------- @@ -143,14 +177,14 @@ The following command copies files from remote to a local directory, create it i Checks: 0 / 0, - Transferred: 2 / 2, 100% Elapsed time: 300ms - + $ ls -lR checkDir/ checkDir/: total 12 -rw-rw-r-- 1 ubuntu ubuntu 1103 Nov 13 15:31 file1.txt -rw-rw-r-- 1 ubuntu ubuntu 1103 Nov 13 15:28 sample_file.txt drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 13 15:53 subdir1 - + checkDir/subdir1: total 4 -rw-rw-r-- 1 ubuntu ubuntu 459 Nov 13 15:32 file2.txt @@ -161,9 +195,9 @@ Mounting object storage on local filesystem **Linux:** First, you need to create a directory on which you will mount your filesystem:: - + $ mkdir ~/mnt-rclone - + Then you can simply mount your object storage with:: $ rclone -vv --vfs-cache-mode writes mount garr-cloud: ~/mnt-rclone @@ -172,15 +206,15 @@ Then you can simply mount your object storage with:: First you have to download **Winsfp**: - http://www.secfs.net/winfsp/rel/ + http://www.secfs.net/winfsp/rel/ WinFsp is an open source Windows File System Proxy which provides a FUSE emulation layer. Then you can simply mount your object storage with (no need to create the directory in advance):: - ./rclone -vv --vfs-cache-mode writes mount garr-cloud: C:/mnt-rclone + ./rclone -vv --vfs-cache-mode writes mount garr-cloud: C:/mnt-rclone -``vfs-cache-mode`` flag enable file caching, you can use either ``writes`` or ``full`` option. For further explanation you can see official documentation at the link: +``vfs-cache-mode`` flag enable file caching, you can use either ``writes`` or ``full`` option. For further explanation you can see official documentation at the link: https://rclone.org/commands/rclone_mount/#file-caching Now that your object storage is mounted, you can list, create and delete files in it. @@ -196,7 +230,7 @@ You can encrypt a directory in your remote container and decrypt it easily throu First, we create a new remote in the rclone configuration file. It will be a subdirectory of your working remote (i.e. ``garr-cloud`` remote and ``crypt-dir`` directory). So, everything inside ``garr-cloud:crypt-dir`` will be encrypted and anything outside won’t.:: $ nano .rclone.conf - + Copy and paste the following text at the end of the file:: [garr-cloud-crypt] @@ -204,21 +238,21 @@ Copy and paste the following text at the end of the file:: remote = garr-cloud:crypt_dir filename_encryption = standard directory_name_encryption = true - + Then set the passwords that will be saved obscured inside the config file:: - - $ rclone config password garr-cloud-crypt password <type_a_password> - $ rclone config password garr-cloud-crypt password2 <type_another_password> + + $ rclone config password garr-cloud-crypt password <type_a_password> + $ rclone config password garr-cloud-crypt password2 <type_another_password> Now we need to create ``crypt_dir`` directory inside ``garr-cloud``:: $ rclone mkdir garr-dev:crypt_dir - + or, if you have ``garr-cloud`` still mounted on ``~/mnt-rclone``:: - + $ mkdir ~/mnt-rclone/crypt_dir -Every file you will create inside ``garr-cloud-crypt`` container will be encrypted. +Every file you will create inside ``garr-cloud-crypt`` container will be encrypted. You can try it following these steps. First, we create a file to copy:: $ echo "Hello world" > test.txt @@ -230,8 +264,8 @@ Then, we copy it to the remote, in a new directory named ``test_dir``:: Now we can list the new created files through the mounted filesystem:: $ cd ~/mnt-rclone/crypt_dir/ - $ ls -R - + $ ls -R + You will get an output similar to this:: .: @@ -241,13 +275,13 @@ You will get an output similar to this:: b4tc2rcdasquuns71k9fa2uiss and if you cat the content of the file, you will see that it has been encrypted:: - + $ cat 0ruoo4gjnnuk01p4gok56li8ts/b4tc2rcdasquuns71k9fa2uiss - + To access and decrypt the file in a complete transparent way, you can copy it from the remote through:: $ rclone ls garr-cloud-crypt: - + 1048576 test_dir/test.txt $ rclone copy garr-cloud-crypt: new_dir @@ -259,11 +293,5 @@ To access and decrypt the file in a complete transparent way, you can copy it fr new_dir/test_dir: test.txt - $ cat new_dir/test_dir/test.txt + $ cat new_dir/test_dir/test.txt Hello world - - - - - - -- GitLab