Use a MinIO S3 bucket on Northflank

MinIO is an excellent option if you require high-performance object storage with an S3-compatible API. S3 buckets are a popular choice for storing and accessing all sorts of unstructured data, such as images and videos, log files, backups, and container images.

This guide will cover a few different scenarios you could use MinIO for, how to deploy MinIO, and how to access and use your addon via the MinIO Console in the browser or via the command line using the MinIO Client.

We will then look at using the MinIO API with the MinIO JavaScript Client to manage your MinIO instance and upload and download objects, with code examples.

Finally we'll touch upon some advanced features of MinIO: timed publicly-accessible URLs, high-entropy keys, and proxying MinIO with your own domain.

You can store content, including text, images, video, and audio for a website or application. These assets can be requested by your application or website with authentication, or you can use anonymous access policies, so your users can access content from a MinIO bucket directly.

Your MinIO instance can also be proxied, so you can use your own domain to serve requests.

You can use your MinIO deployment as storage for files, such as logs, database backups, container images, or other resources. You can keep the MinIO addon private so it is only accessible within your Northflank project, and locally via secure forwarding.

You can quickly and easily deploy MinIO on Northflank with this one-click deployment link.

Select the account and project you want to deploy MinIO in and create the addon. You can change any of the pre-configured settings, but for this guide ensure publicly accessible is enabled. You may want to increase the resources available to the addon, such as the compute plan and storage, if the addon will be handling a lot of simultaneous requests.

The name you give the addon will be used in Northflank-generated domains for internal and external use.

After creating the addon you will be taken to the overview page, where you can see the state of the deployment as well as the connection details. Wait a minute or two for the addon to finish provisioning, and it will be ready to access and use when it enters the running state.

Open the connection details page and find the EXTERNAL_MINIO_CONSOLE_ENDPOINT secret (you can filter secrets by external to see only publicly accessible endpoints). Copy the secret value, which is a code.run generated address, paste it into your browser’s address bar, and navigate to the site. You should see the MinIO console login screen.

Return to the connection details page on Northflank and to find the ADMIN_USERNAME and ADMIN_PASSWORD secret values to use to log in to MinIO Console (toggle the external filter off if it’s enabled).

To start with, select Buckets from the Console menu, and create a bucket. You can configure individual buckets to use versioning, enforce data retention, prevent deletion, and set quotas. For now, create a bucket with none of these settings enabled with the name minio-guide. You will be taken to the browse view, where you can see the paths and files stored in your new bucket.

Create a text file on your machine called file.txt, with the contents Hello MinIO!. In the MinIO Console, with your bucket selected, click upload, select upload file, and upload the file from your local machine. Click on your newly uploaded file to view the object details. You’ll also see the path to the file at the top of the bucket view.

Return to your MinIO addon’s connection details page in Northflank and copy the EXTERNAL_MINIO_ENDPOINT value. This is the endpoint where your files can be accessible, however your file will currently be inaccessible without authentication and you should be met with an access denied message when you try to access it.

Check this is the case by pasting the MinIO endpoint URL into your browser and copying the file path from your MinIO bucket. For example, our bucket is called minio-guide and the file is the image linked above (file.txt). This makes the URL for our stored object: <MINIO_EXTERNAL_ENDPOINT>/<BUCKET>/<OBJECT> or <MINIO_EXTERNAL_ENDPOINT>/minio-guide/file.txt. The port is included in the endpoint copied from Northflank, but it’s not necessary to include this when accessing resources. File paths can also be added, which would be prepended to the object name, e.g. <BUCKET>/<PATH>/<OBJECT>.

Return to MinIO Console and you will see that your bucket access is listed as private. Click 'configure bucket' (the gear icon in the top right) to change your bucket’s access policy. MinIO uses the same Identity and Access Management as Amazon AWS.

We don’t recommend that you set your bucket to public, as then anyone can modify objects in it. Instead, edit your access policy and select custom. Use the following IAM JSON to allow anonymous access to get all objects within the bucket minio-guide:

{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Action": ["s3:GetObject"],
          "Effect": "Allow",
          "Principal": {
            "AWS": ["*"]
          },
          "Resource": ["arn:aws:s3:::minio-guide/*"],
          "Sid": ""
        }
    ]
}

This policy allows anyone to access objects stored within the bucket without authentication, but not to list the contents of the bucket or modify objects. Set the policy and try to access the file again, it should now load. Using MinIO in this way would allow you to manually upload assets, for example for use on a website, and allow them to be accessed by anyone with the URL of the object.

You've now accessed your Northflank MinIO addon and configured your bucket so that the public can access any files you upload. There are lots more features and options to explore in the Console, which you can learn more about in MinIO’s documentation.

You can read next about how to manage MinIO using the CLI, or skip ahead to sections on the API and advanced use cases.

You can also manage MinIO using the command-line client. On some UNIX systems, the MinIO client mc may be aliased to mcli.

Open a terminal on your machine and copy the EXTERNAL_MINIO_CONNECT_COMMAND. Paste and run this command to add your MinIO deployment and credentials under the alias minio. You can run mc admin info minio to check you can access the deployment.

If you've followed the previous guide to use the MinIO console, either delete the minio-guide bucket (the contents must be deleted first), or create a bucket with a different name.

Enter the command mc mb minio/minio-guide to create a bucket called minio-guide.

Navigate to the directory where your file exists, or create one in your current directory with: echo “Hello MinIO!" >> file.txt.

Copy the file to your bucket with mc cp file.txt minio/minio-guide/file.txt, then run mc ls minio/minio-guide to check the contents of your bucket. It should show file.txt.

Return to your MinIO addon’s connection details page in Northflank and copy the EXTERNAL_MINIO_ENDPOINT value. This is the endpoint where your files can be accessible, however your file will currently be inaccessible without authentication and you should be met with an access denied message when you try to access it.

Check this is the case by pasting the MinIO endpoint URL into your browser and copying the file path from your MinIO bucket. For example, our bucket is called minio-guide and the file is the image linked above (file.txt). This makes the URL for our stored object: <MINIO_EXTERNAL_ENDPOINT>/<BUCKET>/<OBJECT> or <MINIO_EXTERNAL_ENDPOINT>/minio-guide/file.txt. The port is included in the endpoint copied from Northflank, but it’s not necessary to include this when accessing resources. File paths can also be added, which would be prepended to the object name, e.g. <BUCKET>/<PATH>/<OBJECT>.

Save the following as download-only-policy.json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Action": ["s3:GetObject"],
          "Effect": "Allow",
          "Principal": {
            "AWS": ["*"]
          },
          "Resource": ["arn:aws:s3:::minio-guide/*"],
          "Sid": ""
        }
    ]
}

From the same directory as the JSON file, or providing the full file path, run the command mc anonymous set-json download-only-policy.json minio/minio-guide.

This policy allows anyone to access objects stored within the minio-guide bucket, but not to list the contents of the bucket, while the anonymous command means the bucket contents can be accessed without authentication. Try to access the file again, it should now load. Using MinIO in this way would allow you to manually upload assets, for example for use on a website, and allow them to be accessed by anyone with the URL of the object.

You've now accessed your Northflank MinIO addon and configured your bucket so that the public can access any files you upload. You can learn more about managing MinIO using the Client in MinIO’s documentation.

You can read next about how to use MinIO via the API, or skip ahead to sections on advanced use cases.

Using MinIO to manually create buckets and manage assets isn’t scalable, so now we’ll take a look at integrating MinIO using the API.

This guide uses the MinIO JavaScript Client, but you can also use the S3 SDK to programmatically manage and use your MinIO instance. You will require node.js installed on your machine to run the examples in the Git repository locally.

Download the repository and open it in your IDE. See the readme for available commands, and take a look at the code in the `src` directory.

Run npm install in the terminal to install the required node modules.

If you haven’t already, create a MinIO addon. Create a .env file in your local copy of the repository with the values from HOST, ACCESS_KEY and SECRET_KEY from your Northflank addon's connection details page, as per the .env.example file.

The MinioClient module creates and returns a new MinIO JavaScript Client, which connects to the Northflank addon using the provided environment variables. All the method calls are handled asynchronously.

This module is imported and used to call methods to manage buckets and objects in other functions. You can learn more about the available methods and their arguments in the MinIO JavaScript Client API reference.

In a terminal in the repository directory run npm run list-buckets. This calls the MinIO Client’s listBuckets method, and will return a list of buckets on the instance.

Next run npm run create-bucket to create a bucket with the name minio-guide. This uses the makeBucket method. After the bucket has been created, setBucketPolicy is called to apply the download-only-policy.json to the bucket, to make uploaded objects publicly accessible. You can run list-buckets again to check the bucket has been created.

The repository includes a file called file-api.txt. Run npm run upload-object to read the file using the fs module, and upload it using the putObject method. This method is given three arguments: the name of the bucket to upload to, the name of the object to create, and the data the object should contain. It returns an object containing the etag (an MD5 hash of your object) and the versionId (if versioning is enabled on the bucket). Uploading an object with the same name will overwrite the existing object.

You don’t need to read data from the filesystem to create objects, you can include any data you want to store in the method. Visit the path <EXTERNAL_MINIO_ENDPOINT>/minio-guide/file-api.txt on your MinIO addon to view the uploaded file.

You can run npm run read-object to access the object you just created, which will call the getObject method, which is provided with the bucket name and object key as arguments. This method returns a stream, which is accessed using object.on(“data", …, provided with a callback function to write the object contents to the local file read-in-chunks.txt, which will be written to the data directory. As with uploading an object, it is not necessary to write object contents to a file. They could instead be stored as a variable in your application, transferred to a database, or rendered in a user’s browser.

This example repository contains only a few basic examples, the MinIO JavaScript Client has many more methods that will allow you to use your Northflank MinIO addon for a variety of use cases, see the MinIO JavaScript Client API reference for more information.

You can generate pre-signed URLs to allow temporary access to put or get objects in a bucket. This can be used when you don’t want to make the contents of a bucket public, but do want to allow temporary access for users.

For example, you can allow an authenticated user to request a pre-signed URL, which is generated on your backend using your access key and secret. You can then provide this URL to the user to allow them to upload or download a file in a private bucket.

You can run, from the example repository, npm run presigned-url. This creates a private bucket, secret-bucket, and uploads a file, private.txt, to it. If you try accessing this file via the normal path in your browser at your MinIO endpoint (<EXTERNAL_MINIO_ENDPOINT>/secret-bucket/private.txt), it should be inaccessible.

The presignedUrl method, with the HTTP method, the bucket name, object name, and expiry as arguments returns a presigned URL which is valid for one day to access the requested object.

Now open the presigned URL returned in your console in a browser and it should allow you access to the otherwise private object. You can also use methods such as presignedGetObject and presignedPutObject instead of providing the HTTP method as an argument, and the presignedPostPolicy method allows you to provide presigned URLs with specific policies for POST operations.

As an alternative to using publicly-accessible buckets or generating timed URLs, you can assign high-entropy names to your uploaded objects to reduce the risk of the contents of your buckets from being guessed or brute-forced.

Our example repository uses the uuid NPM package to upload objects with a universally unique identifier:

const objectName = uuidv4();
  const result = await minioClient
    .putObject(bucketName, objectName, objectContent)
    .catch((e) => {
      console.log("Error while creating object: ", e);
      throw e;
    });

You can try this with the command npm run high-entropy-key.

You can add a proxy server in front of your MinIO to serve requests using your own domain, rather than the MinIO domain, using Nginx.

You can proxy as many specific buckets as required by adding the bucket name as a location and setting proxy_pass as the MinIO endpoint. You can also proxy the entire MinIO instance by setting the root domain (location /) to proxy to the MinIO endpoint. However, if your MinIO instance is publicly accessible this will redirect the visitors to the root domain to the MinIO console, so this may not be desirable. You can also point the root domain at your application endpoint, a 404 page, etc.

To set up the Nginx proxy, first create a deployment service on Northflank with external image as the source, and use the default Docker Nginx image: nginx:latest. Expand the advanced configuration on the environment page of your Nginx service add the following file, with the mount path /etc/nginx/conf.d/default.conf:

server {
    listen 80;
    server_name <SERVER_NAME>;
    
    # Proxy requests to a specific bucket to MinIO server endpoint
    location /media/ {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass <MINIO_ENDPOINT>;
    }

    # Proxy any other request to the application server endpoint
    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_pass <APPLICATION_ENDPOINT>;
    }

}

Replace <SERVER_NAME> with your own custom domain (excluding the protocol and port, example.com not http://example.com:80). The domain must have been added to your Nginx service and assigned to port 80. Copy the endpoint value from your MinIO addon's connection details page to replace <MINIO_ENDPOINT>, and finally enter your application's endpoint for <APPLICATION_ENDPOINT> (both of these values should include the protocol and port, for example http://application:9000).

This configuration will redirect requests to the path at your domain to the MinIO bucket with the same name. For example, example.com/photos/picture.png would fetch the object with the name picture.png.

Alternatively you could replace the root domain block contents with return 404;, or direct visitors somewhere else:

# Return a 404 error page to users visiting the root domain
 location / {
    return 404;
 }

You can also use Kong gateway to proxy your MinIO addon. Follow this guide to deploy Kong and adapt this configuration to get started, saved as a secret file in your Kong service's environment variables with the path /var/lib/kong/kong.yml:

_format_version: "1.1"

services:
  - name: storage
    url: <MINIO_ENDPOINT>
    routes:
      - name: media
        strip_path: false
        paths:
          - /<BUCKET>
  - name: web
    url: <APPLICATION_ENDPOINT>
    routes:
      - name: web
        strip_path: false
        paths:
          - /

Copy the endpoint value from your MinIO addon's connection details page to replace <MINIO_ENDPOINT>, and finally enter your application's endpoint for <APPLICATION_ENDPOINT> (both of these values should include the protocol and port, for example http://application:9000).

This configuration will redirect requests to the path at your domain to the MinIO bucket with the same name. For example, example.com/photos/picture.png would fetch the object with the name picture.png. Add each bucket you want to proxy as a path, or alternatively proxy all requests (- /) to the MinIO addon.

You can also use Traefikto proxy your MinIO addon. Follow this guide to deploy Treafik . Ensure it is set to run with a custom command by going to the CMD override page, select custom command as the Docker runtime mode, and use the command: traefik --providers.file=true --providers.file.filename=/config/traefik.yml --entrypoints.main.address=:80. Then adapt this configuration, as a secret file in your Traefik service's environment variables with the path /config/traefik.yml, to get started:

http:
  services:
    storage:
      loadBalancer:
        servers:
          - url: <MINIO_ENDPOINT>
    web:
      loadBalancer:
        servers:
          - url: <APPLICATION_ENDPOINT>

  routers:
    storage:
      service: storage
      entryPoints:
        - main
      rule: "PathPrefix(`/<BUCKET>`)"
    web:
      service: web
      entryPoints:
        - main
      rule: "PathPrefix(`/`)"
      middlewares:
        - webHostHeaderSet

  middlewares:
    webHostHeaderSet:
      headers:
        customRequestHeaders:
          Host: "<APPLICATION_ENDPOINT_HOST>"

Copy the endpoint value from your MinIO addon's connection details page to replace <MINIO_ENDPOINT>, and enter your application's endpoint for <APPLICATION_ENDPOINT> (both of these values should include the protocol and port, for example http://application:9000). Enter your application service's name without the protocol and port for the host in the webHostHeaderSet middleware, which is applied to requests routed to web. For example if your application endpoint is set to the Northflank internal address http://application:9000, the host should be "application".

This configuration will redirect requests to the path at your domain to the MinIO bucket with the same name. For example, example.com/photos/picture.png would fetch the object with the name picture.png. Multiple paths can be added to the same rule with the or operator: rule: "PathPrefix(`/media`) || "PathPrefix(`/userdata`) || "PathPrefix(`/logs`)"

Add each bucket you want to proxy as a path, or alternatively proxy all requests (rule: "PathPrefix(`/`)") to the MinIO addon.