Transfer data to and from containers

You can transfer data to and from a container's ephemeral storage, or migrate data to and from a persistent volume.

If you want to transfer on a persistent volume and your service does not support the methods below, you can temporarily detach the volume from the existing service and attach it to one that does to transfer the data.

You can compress directories or groups of files to make downloading or uploading them quicker and simpler, especially if you have a large number of files to transfer. Compressing and uncompressing archives may require increased compute resources.

You can use curl or wget to download data from an external host to your containers and persistent volumes on Northflank. Similarly, you can transfer data from your containers or persistent volumes to external destinations.

You can execute a curl or wget command in a running container either via the shell in the Northflank application, using the CLI, API, or JavaScript Client, or in a template.

Download data using curl or wget

You can run a curl or wget command in your container to retrieve data from an external source:

curl -O https://example.com/path/data.zip

wget https://example.com/path/data.zip

You can also recursively download directories with wget: wget -r http://example.com/files/

You can upload files to S3 storage or other file-hosting services, or run a HTTP server locally to serve files from your machine (when combined with forwarding).

Similarly, you can serve files from your container using a HTTP server, and retrieve them using curl or wget. You can either configure your application to serve files over HTTP, or attach your volume to another service that does.

Send data using curl or wget

You can also send data from your container using curl to an endpoint that accepts file uploads.

curl -X POST -F "file=@/path/data.zip" https://example.com/upload`

wget --method=POST --body-file=/path/data.zip https://example.com/upload

Example simple HTTP server

Below is an example of a simple Python HTTP server and Dockerfile, which can be used to serve files from the provided directory.

import http.server
import socketserver
import os

directory = os.environ.get("DIRECTORY")
port = 8000

os.chdir(directory)

Handler = http.server.SimpleHTTPRequestHandler

with socketserver.TCPServer(("", port), Handler) as httpd:
    print(f"Serving files from '{directory}' at http://0.0.0.0:{port}")
    httpd.serve_forever()

FROM python:3.9-slim

ENV PYTHONUNBUFFERED=1
ENV DIRECTORY="/"

COPY /file_server.py .

EXPOSE 8000
CMD ["python3", "file_server.py"]

You can securely forward your service to your local machine. This will allow you to send and receive data without making it available on the internet. You will need to run a HTTP server locally to use wget and curl.

Securely transfer data with wget and curl

Both wget and curl support basic authentication and bearer tokens to secure transfers.

Basic authentication

curl -u username:password -O https://example.com/path/data.zip

wget --user=username --password=password https://example.com/path/data.zip

Bearer token

curl -H "Authorization: Bearer <your-token>" -O https://example.com/path/data.zip

wget --header="Authorization: Bearer <your-token>" https://example.com/path/data.zip