By Dan Ambrose
Published 17th April 2023
S3 buckets are a great option for reliable, fast and cheap storage. This type of sink is a useful accompaniment to a feature-rich log aggregator and acts as a backup for all of your logs.
You can send logs to any S3 compatible bucket, but this guide will explain how to set up a log sink using a DigitalOcean Space.
Create a new Space in DigitalOceans Spaces.
Create an access key for your new Space so Northflank can write files to the bucket. Navigate to the API section in the sidebar or follow this link.
Once you’ve created the token, make sure to save the secret somewhere secure as it won’t be viewable once you leave the page.
With your new Space and Space access key created you can now connect Northflank to the bucket.
Open this link, or navigate to log sinks in your team or user account settings and select add a log sink.
Copy your endpoint, which can be found under the name of your Space in DigitalOcean. You need to remove the bucket from the endpoint and enter them separately in the form.
Select US-East-1 as the region. All Spaces on DigitalOcean use this endpoint to connect, regardless of which region your Space is in.
Fill in your Space's access key name and provide the secret you generated earlier.
We recommend using Gzip compression to save on space, but you can also send logs as uncompressed text files.
By default, all logs from your account will be forwarded to your bucket. To target specific projects, open advanced options and specify which projects to forward logs from.
You can also enable custom label parsing if your application is configured to send additional keys in JSON-formatted log lines.
Click add log sink to create your integration.
Before your log sink is created, Northflank will send a log line to your bucket to verify the credentials are correct. You should see an entry containing the log validating log sink credentials.
If your workloads are producing logs, they should start appearing straight away.
Logs sent to S3 buckets are automatically segmented into directories to help identify and find them, by project, resource type, and resource name. For example:
| Project | Resource category | Resource |
|---|---|---|
| my-project/ | service/ | my-service -1/ |
| my-project/ | service/ | my-service -2/ |
| my-project/ | addon/ | mongo-db/ |
| my-project/ | job/ | db-cleanup-cron/ |
Other things to note:
- Northflank will create a folder to store logs each day, with the format
date=YYYY-MM-DD - Each log file will be named with a Unix timestamp in seconds, followed by a UUID:
[timestamp]-[UUID].[file-extension] - Logs will have the file extension .log if uncompressed, or .log.gzip for compressed logs
- Logs will be sent in batches no greater than 32.7KB, or every second while logs are being generated
If you want to modify your sink after creation, navigate to the log sink list and click on the sink to view its details.
You can update the project targets and sink data as well as pause/resume the sink.
Have questions or feedback? You can always reach out to our team.