Amazon S3 backup Tutorial -via Khaled

12 Feb 2010

Article by Khaled

To interact with amazon S3, we have many languages we can use as well as python. I chose python because it’s a scripting language. That means we don’t need to compile and we don’t need a virtual machine to run it, only an interpreter is needed.

There is some ready tools to interact with amazon S3 I found S3cmd it’s made with python and there is many methods within.

Installing S3cmd (nb: commands are in red color)

Its better to be root to have all permissions Laptop:~$ sudo -i

Take the zip file and copy it somewhere for example /home/Your_Directory/

1.To copy ———————————– Laptop:~# cp s3cmd-0.9.9.91 /home/Your_Directory

2. Decompress the file. —————- Laptop:~# unzip s3cmd.zip

3. Move to s3cmd folder .————— Laptop:~# cd s3cmd-0.9.9.91

4. Change setup.py mod to can execute it. Laptop:~/s3cmd-0.9.9.91# chmod +x setup.py

5. Run the setup file with this command. Laptop:~/s3cmd-0.9.9.91# ./setup.py

6. During the installation enter your access key and secret one, and choose secured connection.

Now s3cmd is installed and your connection with s3 account is set, so you can transfer files to and from s3 account there is many commands you can use to can use them you should be in s3cmd folder, I verified most of the code and it uses methods provided by amazon s3 developers.

The command that allows you to transfer from your server (linux) to amazon s3 is :

s3cmd-0.9.9.91#./s3cmd put Local_File s3://BUCKET_Name/Other_Files_In_Bucket

But this command will transfer “Local_File” to the bucket so if we have another new file within and we use the same command, the whole “Local_File” would be transferred too, so we loose in data transfer.

BUT if we use a sync command we can synchronize a local file with a remote one, it means that if we have a local folder and we add files in it step by step this command allows us to transfer only missing files to the remote file, this is how we minimize data transfer, so for example everyday we have a new tar.gz file added to our local file and our backup would be done everyday,

The command is :
3cmd-0.9.9.91# ./s3cmd sync /home/Your_Directory/local_file/ s3://BUCKET_Name/Remote_File

Every time we run this command it copies only missing files.

Automation of our transfer with sync command:

In linux OS we can automate execution of commands by adding them to a special file which has a specific (easy) syntax.
To access to this file we use this command ——— 3cmd-0.9.9.91# crontab -e

Our file is open now, we can insert any command to be executed at any time or day or month.

A sample line is so —————- * * * * * Command_to_execute

if we want to run any linux command for example at 3:59 PM we edit our line as follow
59 15 * * * Command_to_run

Suppose that we have our tar.gz files already in a Local_File and we want to transfer them to the Remote_File with sync command everyday at 3:00 am, the line would be:

0 3 * * * Path_Where_s3cmd_Folder_is/./s3cmd sync /home/Your-Directory/Local_File/ s3://BUCKET/Remote_File/