How to Access Amazon S3 Bucket like a Local File System

Mountpoint for Amazon S3 is now generally available and ready for production workloads. This open-source simple file client is optimized for read-heavy workloads that require high throughput and focused on high-performance access to large data sets.

In this post, we'll see how to effortlessly access an S3 bucket as a local file system by leveraging Mountpoint for Amazon S3.

Install Mountpoint for Amazon S3

Based on your Linux distributions and hardware-platform architecture, download prebuilt packages.

abhijit@AwsJunkie:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal
abhijit@AwsJunkie:~$ uname -ri
5.15.90.1-microsoft-standard-WSL2 x86_64

I am using the Ubuntu distro of Windows Subsystem for Linux (WSL2) running on x86_64 architecture. So, I'll download DEB-based distributions for x86_64.

RPM-based distributions (Amazon Linux, Fedora, CentOS, RHEL):

DEB-based distributions (Debian, Ubuntu):

Other distributions:

abhijit@AwsJunkie:~$ wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
--2023-08-21 15:30:14--  https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
Resolving s3.amazonaws.com (s3.amazonaws.com)... 16.182.66.96, 16.182.66.32, 52.216.208.152, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|16.182.66.96|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9398148 (9.0M) [binary/octet-stream]
Saving to: ‘mount-s3.deb’

mount-s3.deb          100%[=======================>]   8.96M  3.57MB/s    in 2.5s

2023-08-21 15:30:17 (3.57 MB/s) - ‘mount-s3.deb’ saved [9398148/9398148]
abhijit@AwsJunkie:~$ ls
mount-s3.deb

Install the downloaded Mountpoint for Amazon S3 package.

abhijit@AwsJunkie:~$ sudo apt install ./mount-s3.deb
Building dependency tree
Reading state information... Done
Note, selecting 'mount-s3' instead of './mount-s3.deb'
The following package was automatically installed and is no longer required:
  libxmlb1
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  mount-s3
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/9398 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 /home/abhijit/mount-s3.deb mount-s3 amd64 1.0.0 [9398 kB]
Selecting previously unselected package mount-s3.
(Reading database ... 42843 files and directories currently installed.)
Preparing to unpack /home/abhijit/mount-s3.deb ...
Unpacking mount-s3 (1.0.0) ...
Setting up mount-s3 (1.0.0) ...

To make sure the installation was successful, please check the version.

abhijit@AwsJunkie:~$ mount-s3 --version
mount-s3 1.0.0

Configure Mountpoint for Amazon S3

Mountpoint for Amazon S3 uses the same credentials configuration options as the AWS CLI. So, either use aws configure AWS CLI to create the credential file or create it manually.

abhijit@AwsJunkie:~$ aws configure
AWS Access Key ID [None]: AKIAUI2UW222A6I6R6O
AWS Secret Access Key [None]: 81926124b-8fcd-4239-9ab0-630f772222
Default region name [None]: us-east-1
Default output format [None]:

or,

Create ~/.aws/credentials file and add aws_access_key_id and aws_secret_access_key.

abhijit@AwsJunkie:~$ mkdir ~/.aws
abhijit@AwsJunkie:~$ sudo nano ~/.aws/credentials

Sample ~/.aws/credentials content

[default]
aws_access_key_id = AKIAUI2UW222A6I6R6O
aws_secret_access_key = 81926124b-8fcd-4239-9ab0-630f772222

Now create a local directory (e.g. ~/local-dir) and mount it with an S3 bucket (e.g. aj-mounts3-bucket).

abhijit@AwsJunkie:~$ mkdir local-dir
abhijit@AwsJunkie:~$ mount-s3 aj-mounts3-bucket local-dir
bucket aj-mounts3-bucket is mounted at local-dir

Demo - Mountpoint for Amazon S3

Now, we are good to access the S3 bucket as a local file system using different file operations.

Let's create a few files and a directory inside the mounted local directory.

abhijit@AwsJunkie:~$ echo 'file1' > file1.txt
abhijit@AwsJunkie:~$ echo 'file2' > file2.txt
abhijit@AwsJunkie:~$ mkdir dir1
abhijit@AwsJunkie:~$ echo 'file3' > dir1/file3.txt
abhijit@AwsJunkie:~/local-dir$ tree
.
├── dir1
│   └── file3.txt
├── file1.txt
└── file2.txt

1 directory, 3 files

Refresh S3 bucket. And we'll see the exact same files and directories in the S3 bucket as objects.

References

• Working with Mountpoint for Amazon S3
• Mountpoint for Amazon S3 - Github