From 916de7cbda3263c1de97c67324a9f7c46bfddd0b Mon Sep 17 00:00:00 2001
From: Fulvio Galeazzi <fulvio.galeazzi@garr.it>
Date: Thu, 28 Apr 2022 10:00:55 +0200
Subject: [PATCH] 2022-04-28: FG; Create separate ObjectStore section in FAQ,
added s3cmd.
---
web/support/kb/objstore/index.rst | 8 +
.../rclone_quick_tutorial.rst | 0
.../kb/objstore/s3fs_quick_tutorial.rst | 147 ++++++++++++++++++
.../s3fs_quick_tutorial.rst~} | 0
...ntials.rst => application_credentials.rst} | 0
5 files changed, 155 insertions(+)
create mode 100644 web/support/kb/objstore/index.rst
rename web/support/kb/{openstack => objstore}/rclone_quick_tutorial.rst (100%)
create mode 100644 web/support/kb/objstore/s3fs_quick_tutorial.rst
rename web/support/kb/{openstack/s3fs_quick_tutorial.rst => objstore/s3fs_quick_tutorial.rst~} (100%)
rename web/support/kb/openstack/{Application_credentials.rst => application_credentials.rst} (100%)
diff --git a/web/support/kb/objstore/index.rst b/web/support/kb/objstore/index.rst
new file mode 100644
index 00000000..81da97f2
--- /dev/null
+++ b/web/support/kb/objstore/index.rst
@@ -0,0 +1,8 @@
+Object Storage Use
+==================
+
+.. toctree::
+ :maxdepth: 1
+ :glob:
+
+ *
diff --git a/web/support/kb/openstack/rclone_quick_tutorial.rst b/web/support/kb/objstore/rclone_quick_tutorial.rst
similarity index 100%
rename from web/support/kb/openstack/rclone_quick_tutorial.rst
rename to web/support/kb/objstore/rclone_quick_tutorial.rst
diff --git a/web/support/kb/objstore/s3fs_quick_tutorial.rst b/web/support/kb/objstore/s3fs_quick_tutorial.rst
new file mode 100644
index 00000000..ad72ccb7
--- /dev/null
+++ b/web/support/kb/objstore/s3fs_quick_tutorial.rst
@@ -0,0 +1,147 @@
+S3 interface to object storage
+==============================
+
+Instead of `Rclone <https://cloud.garr.it/support/kb/openstack/rclone_quick_tutorial/>`_,
+we can use s3 APIs to connect to object storage.
+You may use:
+* `s3cmd`_ to manipulate buckets (create, get/put
+ objects) from command line
+* `s3fs <https://github.com/s3fs-fuse/s3fs-fuse>`_ to mount containers as filesystems
+
+
+S3cmd: manipulate object storage from command line
+--------------------------------------------------
+This command does not seem to like capital letters neither in bucket names
+nor in directory names.
+
+Install s3cmd
+*************
+On Ubuntu machine, you can install the packaged version with the following
+commands::
+
+ $ sudo apt update
+ $ sudo apt install s3cmd
+
+Configure environment
+*********************
+Create and download an `application credential <https://cloud.garr.it/compute/app-credential/>`_ from openstack dashboard, and take note of the *access_key* and
+the *secret_key*.
+
+Create a file ``~/.s3cfg`` with the following content::
+
+ [default]
+ access_key = <put_your_ACCESS-KEY_here>
+ secret_key = <put_your_SECRET-KEY_here>
+ host_base = swift.cloud.garr.it
+ host_bucket = %(bucket).swift.cloud.garr.it
+ use_https = True
+
+Check everything is correct by executing a simple command, which should return
+the list of buckets in your object storage area::
+
+ $ s3cmd ls
+ 2022-04-21 10:31 s3://fulvio
+ .....
+
+
+Use s3cmd
+*********
+
+Short summary of most common commands, for more information please visit
+the `s3cmd`_ page.
+
+List contents::
+ $ s3cmd ls
+ 2022-04-21 10:31 s3://fulvio
+
+Create new bucket::
+ $ s3cmd mb s3://mynewbucket/
+ Bucket 's3://mynewbucket/' created
+
+Put file::
+ $ s3cmd put testsmallfile s3://mynewbucket/
+ upload: 'testsmallfile' -> 's3://mynewbucket/testsmallfile' [1 of 1]
+ 10485760 of 10485760 100% in 0s 30.37 MB/s done
+
+Recursive copy, put whole directory (note missing trailing '/')::
+ $ s3cmd put -r testdir s3://mynewbucket/
+ upload: 'testdir/aRandomFile.png' -> 's3://mynewbucket/testdir/aRandomFile.png' [1 of 1]
+ 67819 of 67819 100% in 0s 1577.91 kB/s done
+
+Get a file (destination file name can be omitted, default to same name as remote)::
+ $ s3cmd get s3://mynewbucket/testdir/aRandomFile.png copyOfRandomFile.png
+ download: 's3://mynewbucket/testdir/aRandomFile.png' -> 'copyOfRandomFile.png' [1 of 1]
+ 67819 of 67819 100% in 0s 3.05 MB/s done
+
+Delete a file (enable recursion with '-r')::
+ $ s3cmd del s3://mynewbucket/testdir/aRandomFile.png
+ delete: 's3://mynewbucket/testdir/aRandomFile.png'
+
+Delete bucket (must be empty)::
+ $ s3cmd rb s3://mynewbucket
+ Bucket 's3://mynewbucket/' removed
+
+
+S3fs: mount a container as a filesystem
+---------------------------------------
+
+Install s3fs
+***************
+On Ubuntu machine, you can install the packaged version with the following commands::
+
+ $ sudo apt update
+ $ sudo apt install s3fs
+
+Check the version::
+
+ $ s3fs --version
+
+N.B. These instructions refer to version 1.86 available on Ubuntu 20.04. Different versions may require different configuration.
+
+Uncomment *user_allow_other* option by removing the *#*::
+
+ $ nano /etc/fuse.conf
+
+Create application credential
+*****************************
+
+Create and download an `application credential <https://cloud.garr.it/compute/app-credential/>`_ from openstack dashboard as *app-credentials.sh*.
+
+You need to install the Openstack cli as described here in the `cli tutorial <https://cloud.garr.it/compute/install-cli/>`.
+
+Then execute the content of the file::
+
+ $ source app-credentials.sh
+
+And create the ec2 credentials::
+
+ $ openstack ec2 credentials create -c access -c secret -f value | paste -sd: > ${HOME}/.passwd-s3fs
+ $ chmod 600 .passwd-s3fs
+
+Mount a container
+********************
+First, you need to create the container on your openstack project. Then you can mount your container on your local directory.
+Assume that you have a container named *test_container* and a local directory named *test_dir*::
+
+ $ s3fs test_container test_dir -o allow_other -o host=https://swift.cloud.garr.it -o use_path_request_style -o umask=000
+
+Now your container has been mounted on *test_dir* directory. You can access it and every change you make inside the directory is istantly made inside the container.
+
+Debug
+*****
+
+If you need to debug, add the following options at the mounting command::
+
+ -o dbglevel=info -f -o curldbg
+
+*Note that -f option cause the command to run in foreground, so CTRL+C will kill the command.*
+
+
+Unmount
+*******
+::
+
+ $ umount test_dir
+
+
+.. _s3cmd: https://s3tools.org/usage
diff --git a/web/support/kb/openstack/s3fs_quick_tutorial.rst b/web/support/kb/objstore/s3fs_quick_tutorial.rst~
similarity index 100%
rename from web/support/kb/openstack/s3fs_quick_tutorial.rst
rename to web/support/kb/objstore/s3fs_quick_tutorial.rst~
diff --git a/web/support/kb/openstack/Application_credentials.rst b/web/support/kb/openstack/application_credentials.rst
similarity index 100%
rename from web/support/kb/openstack/Application_credentials.rst
rename to web/support/kb/openstack/application_credentials.rst
--
GitLab