v1

Observe /

Configure log sinks

You can integrate your preferred log aggregators and observability platforms with your Northflank account.

Log sinks allow you to forward logs from containers in specific projects, or your whole account, allowing you to unify and process logs from your entire operation. You can improve your observability with the features of your preferred log service; analyze and visualise metrics, perform searches, set up real-time alerts, and meet your log auditing requirements.

You can also integrate with your own logging solutions by sending logs to a custom HTTP endpoint or an Amazon S3 bucket.

Log sinks are in early preview and subject to change. Contact support@northflank.com to request early access to log sinks for your account.

Supported services

Northflank currently supports the following endpoints for log sinks:

Log encoding

By default, logs will be sent as plain text, some log sinks allow you to send logs as JSON instead.

Text

Text logs are sent as a single string containing only the log message itself, for example:

2022-09-12T09:21:40.817924799Z stdout F Mon Sep 12 09:21:40 UTC 2022: Log item 138051

JSON

JSON logs are sent as an array of objects, with each object representing a single log entry:

[{
  "entity":"<entity_id>",
  "host":"Northflank",
  "message":"<datetime with timezone> <log message>",
  "path":"/",
  "pod":"<service_name>-<pod_id>",
  "project":"<project_name>",
  "service":"<service_name>",
  "source":"Northflank",
  "source_type":"http",
  "timestamp":"<datetime with timezone>",
  "type":"<service|job|addon>"
}]

Create a log sink

Log sinks can be created on your user or team account in your account settings.

To add a new log sink, select the type of log sink you want and follow the specific instructions below for each type.

You can pause and resume your log sinks as required after they have been created, update the authentication details, and change the projects they target.

Project restrictions

You can enable make log sink target selected projects to select specific projects to forward logs from. Only containers running in the selected projects will have their logs forwarded.

You can leave this disabled to forward logs from all running containers in all projects on your account.

If you are creating a log sink on a team account and your role is restricted to specific projects you will only be able to create log sinks targeting the projects you have access to.

Validation log

When you create or update a log sink a validation log is sent to the endpoint to ensure the credentials entered are correct. This validation log will be stored in your sink and takes the following format:

{
  "app-name": "log-forwarder",
  "host": "Northflank",
  "message": "Validating log sink credentials",
  "severity": "info",
  "date": "<datetime with timezone>"
}

Create a Datadog log sink

  1. Select the Datadog sink type in your new Northflank log sink
  2. Select the region your Datadog account exists in by identifying it from the URL that you use to access it:
RegionSite URLSite parameter
EU1https://app.datadoghq.eudatadoghq.eu
US1https://app.datadoghq.comdatadoghq.com
US3https://us3.datadoghq.comus3.datadoghq.com
US5https://us5.datadoghq.comus5.datadoghq.com
  1. Navigate to organisation settings, open API keys, and create a new key. Make sure your account has at least the standard role, or a custom role with permissions to write logs .
  2. Copy the API key and paste it into the field in Northflank
  3. Select your project restrictions, if any, and add log sink
  4. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink

Create a Loki log sink

You can send your logs to a self-hosted instance of Loki, or to a Loki instance hosted on Grafana cloud .

Grafana cloud

  1. Select the Loki sink type in your new Northflank log sink
  2. Navigate to your Grafana cloud stack
  3. View the details for Loki
  4. Follow the instructions to add Loki as a data source in your Grafana instance, if it's not currently configured. Copy and/or save your API key, as it will not be shown again.
  5. Go to Northflank and enter the information from your Loki configuration:
    • URL
    • user
    • the API key you created when integrating Loki with Grafana
  6. Choose the encoding method for the logs: text (default) or JSON
  7. Select your project restrictions, if any, and add log sink
  8. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink

Self-hosted

  1. Select the Loki sink type in your new Northflank log sink
  2. Enter the URL of your Loki deployment
  3. Enter the username and password you set when configuring your Loki instance
  4. Choose the encoding method for the logs: text (default) or JSON
  5. Select your project restrictions, if any, and add log sink
  6. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink

Create a Papertrail log sink

  1. Select the Papertrail sink type in your new Northflank log sink
  2. Navigate to your Papertrail settings and open log destinations
  3. Create log destination, allow it to accept logs from unrecognised systems, and leave delete systems with no events enabled
  4. Choose to accept connections via port (recommended) or token (HTTPS). Leave TLS encryption enabled for TCP, and plain text enabled for UDP for port connections.
  5. On Northflank select the authentication method you chose when creating your log destination
    • For port enter the URL and port generated for your log destination, for example logs5.papertrailapp.com:12345
    • For token enter both the endpoint and token generated for your log destination
  6. Choose the encoding method for the logs: text (default) or JSON
  7. Select your project restrictions, if any, and add log sink
  8. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink

Create a Mezmo log sink

  1. Select the Mezmo (formerly LogDNA) sink type in your new Northflank log sink
  2. In Mezmo, open settings, navigate to the API keys page under organisation
  3. Generate a new ingestion key and copy the value
  4. Return to Northflank and paste the ingestion key into the API key field
  5. Select your project restrictions, if any, and add log sink
  6. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink

Create a Logtail log sink

  1. Select the Logtail sink type in your new Northflank log sink
  2. In your Logtail account navigate to sources and connect source
  3. Name your source, select HTTP as the platform, and create source
  4. Copy the source token, return to Northflank, and enter it into the API key field for your new log sink
  5. Select your project restrictions, if any, and add log sink
  6. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink

Create a Honeycomb log sink

  1. Select the Honeycomb sink type in your new Northflank log sink
  2. In Honeycomb navigate to environments and manage environments
  3. View API keys for the environment you want to use with Northflank and copy the API key value
  4. Return to Northflank and paste the API key value into the form
  5. Enter the name of your dataset, if it doesn't already exist in Honeycomb it will be created for you
  6. Select your project restrictions, if any, and add log sink
  7. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink
  • You must be a Honeycomb team owner to create and manage API keys
  • Your API key must have send events and create datasets permissions

Create an AWS S3 log sink

You can create an S3 sink hosted by Amazon Web Services, or another implementation of S3 storage. The following instructions are to integrate S3 hosted by AWS, for other implementations the exact details may differ.

  1. Select the AWS S3 sink type in your new Northflank log sink
  2. Go to your AWS console and select S3 from the storage section in the services menu
  3. Create a new bucket, or select an existing one, and enter the name of the bucket in Northflank
  4. Enter the service endpoint, for example s3.eu-west-1.amazonaws.com for a bucket in the EU-West-1 region
  5. Enter the region, even though it's included in the endpoint
  6. Open the IAM console and create a new IAM user with the necessary permissions, or a role containing the necessary permissions
  7. Copy the access key ID and access key secret to Northflank directly, and/or save them somewhere secure (you will not be able to view them again)
  8. Choose whether to send uncompressed plaintext logs, or logs compressed with gzip to your bucket
  9. Select your project restrictions, if any, and add log sink
  10. You should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink
  • Northflank will create a folder to store logs each day, with the format date=YYYY-MM-DD
  • Each log file will be named with a Unix timestamp in seconds, followed by a UUID: [timestamp]-[UUID].[file-extension]
  • Logs will have the file extension .log if uncompressed, or .log.gzip for compressed logs
  • Logs will be sent in batches no greater than 32.7KB, or every second while logs are being generated

Create a HTTP log sink

You can configure a custom HTTP endpoint to receive your logs from Northflank.

  1. Enter the URL of your log sink
  2. Select your authentication strategy: basic authentication or bearer token
    • Enter your username and password to authenticate using basic authentication
    • Enter your token to authenticate using bearer token
  3. Choose the encoding method for the logs: text (default) or JSON
  4. Select your project restrictions, if any, and add log sink
  5. Your HTTPS endpoint should receive a validation log entry to confirm your credentials are valid, and logs will now be sent to the sink

© 2022 Northflank Ltd. All rights reserved.