Python Minio Guide

142 May 30, 2023

Minio Python SDK for Amazon S3 Titan Cloud Storage

The Python SDK (Software Development Kit) for Amazon S3 (Simple Storage Service) Titan Cloud Storage is a collection of libraries and tools that developers can use to interact with Amazon S3 Titan Cloud Storage using the Python programming language.

Amazon S3 Titan Cloud Storage is a cloud-based storage service provided by Amazon Web Services (AWS) that enables users to store and retrieve data at scale. The Python SDK for Amazon S3 Titan Cloud Storage provides a convenient and easy-to-use interface for developers to interact with the service programmatically.

Overall, the Python SDK for Amazon S3 Titan Cloud Storage can simplify the process of integrating Amazon S3 Titan Cloud Storage into Python-based applications and workflows, making it easier for developers to build scalable, reliable, and high-performance applications.

Prerequisites

Prerequisites required are as follows:

  1. Python 3.7 or higher

Once you have met these prerequisites, you can start using the AWS SDK for Python (Boto3) to interact with Amazon S3 and other AWS services from your Python code.

Goals

After finishing this article, you'll be able to:

  1. Initialize Titan cloud instance.
  2. Check if the bucket exists
  3. Create a bucket.
  4. List buckets
  5. List bucket objects
  6. Remove an empty bucket
  7. Remove an object
  8. Remove objects
  9. Upload an object to the bucket

Steps to run Amazon S3 examples using Python (Boto3) SDK on Titan Cloud Storage

Here are the steps to get started with using the Python SDK for Amazon S3 Titan cloud storage:

Step 1. In

1. Install the Minio Python SDK

You can install the MinIO Python SDK using pip, a Python package manager. Open a terminal or command prompt and run the following command:

pip install minio

Step 2. Import the necessary modules

In your Python code, you will need to import the minio module and any other necessary modules for your application.
import minio

Step 3. Set up the Titan Cloud Storage instance

You will need to create an object that represents the client. You will need to provide the endpoint URL, access key, and secret key.
For example:
clientObject = minio.Minio(‘demo.s3.titancloudstorage.com’, access_key='ACCESS_KEY_HERE', secret_key='SECRET_KEY_HERE', secure=True)

Step 4. Use the Titan Cloud Storage

You can now use the client (Titan cloud stoarge) object to perform operations on your server deployment, such as creating buckets, uploading and downloading objects, and managing access policies. Here's an example of creating a bucket:


clientObject.make_bucket("pythonbucket", location="us-east-1")


Note: Replace 'my-bucket' with the name of the bucket you want to upload the file to.

That's it!

With these steps, you should now be able to create the Python SDK to interact with your Amazon S3 Titan cloud storage.

NOTE: The Titan Cloud Storage object is safe to use with multiple threads when the Python threading library is used. In particular, it is NOT safe to share it between different processes, like when you use multiprocessing.
The answer is to make a new object for each process and not use the same one for multiple processes.

Quick Start Examples

The SDK provides a set of Python classes and methods that allow developers to easily integrate Amazon S3 Titan with their Python applications. With the SDK, developers can perform common operations such as creating and managing buckets, uploading and downloading objects, and managing access policies.

To help developers get started quickly with the SDK, S3 provides a number of quick start examples. The quick start examples cover a range of scenarios, including uploading files to Amazon S3 Titan, downloading files from Amazon S3 Titan, and managing buckets and objects. The examples are well-documented and provide clear explanations of each step, making them easy to follow even for developers who are new to Amazon S3 Titan or the Python SDK.

  1. Initializing Titan cloud instance.
  2. Checking if the bucket exists
  3. Creating a bucket.
  4. Listing buckets
  5. Listing bucket objects
  6. Removing an empty bucket
  7. Removing an object
  8. Removing objects
  9. Uploading and downloading files to and from the bucket
  10. Copying and moving files between S3 buckets or within the same bucket

Bucket operations

1. Initializes a new Titan Cloud Connection Code:

The below code snippets Initializes a new Titan Cloud object storage client.
General Method:
Minio(endpoint,
access_key=None,
secret_key=None,
session_token=None,
secure=True,
region=None,
http_client=None,
credentials=None)

create_connection1.py


# Import MinIO library.
from minio import Minio
# Initialize titanCloudStorage with an endpoint and access/secret keys.
titanClient = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("You are connected successfully!!")


Run command:
>python create_connection1.py

2. Check if a bucket exists on Titan Cloud

The below code snippets checks if a bucket exists on the server. It returns True if the bucket exists.
General Method:
bucket_exists(bucket_name)

checkifbucketexists1.py


# Import MinIO library.
from minio import Minio
from minio.error import InvalidResponseError
# Initialize titanCloudStorage with an endpoint and access/secret keys.
titanClient = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("You are connected successfully!!")
try:
    print("Your bucket exists = ")
    print(titanClient.bucket_exists('Python-bucket1'))
except InvalidResponseError as err:
    print(err)


Run command:
>python checkifbucketexists1.py

3. Create a bucket on Titan Cloud

The below code snippets create a bucket on the Titan Cloud server with region and object lock.
General Method:
make_bucket(bucket_name, location=’us-east-1’, object_lock=False)

createbucket.py


# Import MinIO library.
from minio import Minio
from minio.error import InvalidResponseError
# Initialize titanCloudStorage with an endpoint and access/secret keys.
titanClient = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("You are connected successfully!!")
# Remove a bucket operation (only work if your bucket is empty.)
try:
    found = titanClient.bucket_exists("python-bucket-new")
    if found:
        titanClient.remove_bucket("python-bucket-new")
    else:
        print("No such Bucket exists")

except InvalidResponseError as err:
    print(err) 



Run command:
>python createbucket1.py

4. Remove an empty bucket with Titan Cloud Storage

The below code snippets removes a bucket, bucket should be empty to be successfully removed on a Titan Cloud.

General Method:
Remove_bucket(bucket_name)


Removebucket1.py

# Import MinIO library.
from minio import Minio

# Initialize titanCloudStorage with an endpoint and access/secret keys.
client = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("You are connected successfully!!")
# list all the buckets
buckets = client.list_buckets()
for bucket in buckets:
    print(bucket.name, bucket.creation_date)



Run command:
>python removebucket1.py

5. List buckets on Titan Cloud Storage

The below code snippets shows all the accessible buckets information on a Titan Cloud.

General Method:
list_buckets()

Listbuckets1.py 

# Import MinIO library.
from minio import Minio

# Initialize titanCloudStorage with an endpoint and access/secret keys.
client = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("You are connected successfully!!")
# list all the buckets
buckets = client.list_buckets()
for bucket in buckets:
    print(bucket.name, bucket.creation_date)



Run command:
>python listbuckets1.py

Object operations

Object operations refer to the various actions that can be performed on individual objects stored within a bucket in object storage systems.

1. Upload data to a bucket on Titan Cloud Storage

The code snippets below uploads data from a stream to an object in a bucket on a Titan Cloud.

General Method:
put_object(bucket_name, object_name, data, length, content_type=”application/octet-stream”, metadata=None, sse=None, progress=None, part_size=0, num_parallel_uploads=3, tags=None, retention=None, legal_hold=False)

Uploadobject1.py 

import os

from minio import Minio
from minio.error import InvalidResponseError

# Initialize titanCloudStorage with an endpoint and access/secret keys.
titanClient = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("You are connected successfully!!")
# Upload a file named myfile.txt 
try:
    with open('myfile.txt', 'rb') as file_data:
        fileobject = os.stat('myfile.txt')
        titanClient.put_object('python-bucket', 'object1',
        file_data, fileobject.st_size)

except InvalidResponseError as err:
    print(err)



Run command:
>python uploadobject1.py

2. List all the objects in the bucket on Titan Cloud

The code snippets below list the items' details, optionally including bucket versions created on a Titan Cloud. To list every object in our S3 bucket that is currently available, we'll use the listObjects() method:

General Method:
list_objects(bucket_name, prefix=None, recursive=False, start_after=None, include_user_meta=False, include_version=False, use_api_v1=False, use_url_encoding_type=True)

Listobjects1.py 

# Import MinIO library.
from minio import Minio

# Initialize titanCloudStorage with an endpoint and access/secret keys.
titanClient = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("\n You are connected successfully!!")

# List all object in python-bucket 
objects = titanClient.list_objects('python-bucket', recursive=True)
print("\nAll objects --- from python-bucket !!")
for obj in objects: 
    print(obj.bucket_name, obj.object_name.encode('utf-8'), obj.last_modified,
          obj.etag, obj.size, obj.content_type)

# List all object in my-bucket that begin with c 
objects = titanClient.list_objects('my-bucket', prefix='cat',
                                 recursive=True)
print("\nAll objects starts with cat ---from my-bucket !!")
for obj in objects:   
    print(obj.bucket_name, obj.object_name.encode('utf-8'), obj.last_modified,
          obj.etag, obj.content_type)
	

Run command:
>python listobjects1.py

3. Remove an object from bucket on Titan Cloud

The below code snippets removes an object from the bucket information on a Titan Cloud. We'll use the client instance's remove_object() method to delete an object, using the bucket name and object key as parameters:
General Method:
remove_object(bucket_name, object_name, version_id=None)

removeobject1.py 

# Import the Boto3 library
import boto3
# 1st Block of code
# Initialize the S3 client for Titan Cloud Storage
s3_titanobj = boto3.client('s3',
                  endpoint_url='https://{YOUR_INSTANCE}.s3.titancloudstorage.com',
                  aws_access_key_id=‘<your_access_key>‘,
                  aws_secret_access_key=‘<your_secret_key>‘)

# 2nd Block of code
# Specify the name of the bucket containing the object to remove
bucket_name = 'aws-new-bucket'

# Check if the bucket exists
try:
    response = s3_titanobj.head_bucket(Bucket=bucket_name)
    print(f"Bucket '{bucket_name}' exists!")
except Exception as e:
    print(f"Bucket '{bucket_name}' does not exist: {e}")
    exit()

# Specify the key (i.e., filename) of the object to remove
object_key = 'my-new-file.txt'

# Remove the object from the bucket
try:
    response = s3_titanobj.delete_object(Bucket=bucket_name, Key=object_key)
    print(f"Object '{object_key}' removed from bucket '{bucket_name}' successfully!")
except Exception as e:
    print(f"Error removing object '{object_key}' from bucket '{bucket_name}': {e}")# Import MinIO library.
from minio import Minio
from minio.error import InvalidResponseError
# Initialize titanCloudStorage with an endpoint and access/secret keys.
titanClient = Minio('demo.s3.titancloudstorage.com',
                    access_key='titanadmin',
                    secret_key='TitanDemo123',
                    secure=True)
print("You are connected successfully!!")

try:
    found = titanClient.bucket_exists("python-bucket")
    if found:
        titanClient.remove_object('python-bucket', 'object1')
    else:
        print("No such Bucket exists")

except InvalidResponseError as err:
    print(err)	

Run command:
>python removeobject1.py