← Back to Guides
Header image for guide: Send logs to DigitalOcean Spaces from Northflank
Profile image for Dan Ambrose

By Dan Ambrose

Published 18th October 2022

Send logs to DigitalOcean Spaces from Northflank

S3 buckets are a great option for reliable, fast and cheap storage. This type of sink is a useful accompaniment to a feature-rich log aggregator and acts as a backup for all of your logs.

You can send logs to any S3 compatible bucket, but this guide will explain how to set up a log sink using a DigitalOcean Space.

Set up a Space

Create a new Space in DigitalOceans Spaces.

do-spaces-bucket.png

Create a Spaces access key

Create an access key for your new Space so Northflank can write files to the bucket. Navigate to the API section in the sidebar or follow this link.

do-spaces-access-keys.png

Once you’ve created the token, make sure to save the secret somewhere secure as it won’t be viewable once you leave the page.

With your new Space and Space access key created you can now connect Northflank to the bucket.

Create your S3 log sink for DigitalOcean

Open this link, or navigate to log sinks in your team or user account settings and select add a log sink.

create-DO-spaces-AWS-s3-sink.png

Copy your endpoint, which can be found under the name of your Space in DigitalOcean. You need to remove the bucket from the endpoint and enter them separately in the form.

Select US-East-1 as the region. All Spaces on DigitalOcean use this endpoint to connect, regardless of which region your Space is in.

Fill in your Space's access key name and provide the secret you generated earlier.

We recommend using Gzip compression to save on space, but you can also send logs as uncompressed text files.

Enable project targets if required. You can select specific projects to forward logs from, and only containers running in the selected projects will have their logs forwarded.

Click add log sink to create your integration.

View your logs

Before your log sink is created, Northflank tries to send a log line to your bucket to verify the credentials are correct. You should see an entry containing the log validating log sink credentials.

If your workloads are producing logs, they should start appearing straight away.

Logs sent to S3 buckets are automatically segmented into directories to help identify and find them, by project, resource type, and resource name. For example:

ProjectResource categoryResource
my-project/service/my-service -1/
my-project/service/my-service -2/
my-project/addon/mongo-db/
my-project/job/db-cleanup-cron/

Other things to note:

  • Northflank will create a folder to store logs each day, with the format date=YYYY-MM-DD
  • Each log file will be named with a Unix timestamp in seconds, followed by a UUID: [timestamp]-[UUID].[file-extension]
  • Logs will have the file extension .log if uncompressed, or .log.gzip for compressed logs
  • Logs will be sent in batches no greater than 32.7KB, or every second while logs are being generated

Update your log sink

If you want to modify your sink after creation, navigate to the log sink list and click on the sink to view its details.

You can update the project targets and sink data as well as pause/resume the sink.

Have questions or feedback? Contact us. We also have dedicated support plans for business customers available.

Share this article with your network