

Incident report: October 30 Security Incident
On October 30, 2026, an attacker exploited a previously unknown vulnerability in Northflank's API. It allowed the attacker to craft a payload which achieved RCE, effectively allowing the attacker to run commands inside one of our API components. They used that access to read and decrypt customer secrets and upload a subset of them to a server they controlled.
We patched the vulnerability 82 minutes after the first received security alert.
The status page was posted 35 minutes later and we began emailing every user, asking them to treat all secrets as exposed and rotate them. Since then, we have worked with Wiz’s incident response team and independent forensics firm FTI Consulting LLP to investigate the incident. We are extremely grateful to these two teams for the support to Northflank's team during the last two weeks.
To our customers, we are deeply sorry. The Northflank team has been working around the clock to investigate what happened, support affected customers and strengthen the security of the platform. We know that this incident has created extra work and worry.
We are deeply grateful for the patience, understanding and kindness customers have shown us while we work through it.
- An attacker achieved remote code execution (RCE) on a Northflank API component through a previously unknown vulnerability. That component held access to our database and the keys used to decrypt stored secrets.
- 1.5% of customer teams were affected.
- 1.8% of secret values stored on the platform were exfiltrated.
- At this point in time, the investigation has not identified evidence that Northflank user credentials or customer personal data were exfiltrated.
- There is no evidence that services, jobs, addons, templates or builds were accessed, and there is no evidence of access to customer workloads or workload data.
- Customer reports of misuse to date have been related to AI provider keys.
- The vulnerability was fixed on October 30 at 13:05 UTC by our engineers, closing the attacker’s route into the platform. We rotated the platform credentials held by the affected component, and existing secrets are being re-encrypted under new keys. From the moment we learned of the incident, we took precautionary steps assuming a worst-case scenario before the full scope was known, including asking every customer to rotate their secrets to reduce the potential impact of the incident.
- The platform remained online throughout the response, with brief service interruptions as we rotated credentials, deployed fixes and additional security measures.
All times are UTC.
| Time | Event |
|---|---|
| Oct 29 | First activity from the attacker probing our platform |
| Oct 30, 00:19 | The attacker signs up for Northflank and begins probing the API |
| 02:39 | The attacker achieves remote code execution on an API component |
| 02:41–03:08 | The attacker reads encryption keys, source code, queries the database and decrypts stored secrets inside the component and uploads the records that match their filters |
| 03:11 | Last activity excluding tool removal |
| 08:12 | The attacker removes their tools |
| 11:43 | Security alert reaches our team for unusual metadata requests by the attacker |
| 11:46 | Northflank team reviews alert |
| 12:17 | Full incident response begins as the investigation of the metadata service expands to unusual API activity and discovery of RCE |
| 13:05 | Vulnerability fixed |
| 13:40 | First status page post |
| 14:19–15:59 | Platform credentials rotated |
| 15:05 | Email to all users: treat every secret as exposed and rotate it |
| 19:25–22:54 | New detection rules and additional hardening of the affected feature and related API features |
| Oct 31, 09:00 | FTI Consulting LLP engaged |
| Oct 31, 12:56–23:21 | Self-service credential rotation released for database, cache, object storage and message queue addons |
| Oct 31, 23:06 | Audit logs made available through the API |
| Nov 1-Nov 5 | 10 additional security-hardening deployments to the core platform |
The attacker exploited a previously unknown vulnerability in one of our API components, allowing them to achieve remote code execution (RCE).
Because the component had access to both the database and the keys needed to decrypt stored secrets, the attacker was able to inspect its source code using standard command-line tools such as grep, determine how secrets were decrypted, and send the decrypted values to a server they controlled.
The attacker appears to have used AI code generation to modify their scripts in real time, allowing them to rapidly query, decrypt, process and exfiltrate data. Working from the component’s source code and database schema, they had a working decryption script within six minutes of their first search, at 02:53.
Our telemetry has been able to capture scripts used by the attacker. Their scripts decrypted each secret and secret group record inside the component and uploaded it only if it matched one of the following:
- Credential formats. Values in the format of a known credential:
- AI provider keys (OpenAI
sk-, Anthropicsk-ant-, GoogleAIza); - AWS access keys (
AKIA,ASIA); - private keys (
-----BEGIN … PRIVATE KEY); - Slack tokens (
xox…); - GitHub and GitLab tokens (
ghp_,gho_,glpat-and similar); - crypto wallet private keys and seed phrases.
- AI provider keys (OpenAI
- Record names. Secret groups and integrations whose names contained:
- AI providers;
- crypto (wallet, mnemonic, seed, Coinbase, Binance);
- cloud access (AWS, GCP, kubeconfig);
- general credential words (secret, token, API key, private key, credential, Stripe).
- Variable and file names. Secret variables and secret files whose names start with:
- an AI provider:
OPENAI,ANTHROPIC,CLAUDE,GEMINI,GOOGLE,VERTEX,BEDROCK,OPENROUTER,HUGGING,GROQ,MISTRAL,DEEPSEEK,XAI,PERPLEXITY,REPLICATE,ELEVEN; - a crypto term:
WALLET,MNEMONIC,SEED; - a cloud credential:
AWS,AWS_SECRET,AWS_ACCESS,AZURE; - a general key term:
API_KEY,SECRET_KEY,PRIVATE_KEY,ACCESS_KEY,PRIVATE,LLM,MODEL_KEY.
- an AI provider:
Based on our reproduction of the attack in an isolated environment using the tools and commands used by the attacker and recorded by our security vendor, 1.5% of customer teams had secret values extracted. The result has been verified against network flow logs and our external forensic auditor, FTI Consulting LLP, validated our verification process.
What was taken, by feature:
- Project secret groups. Decrypted environment variables and secret files. This accounts for almost all affected customers.
- Global secrets. Decrypted team-wide secrets.
- Notification integrations. Slack, Discord, Microsoft Teams and similar services.
- BYOC cloud provider integrations. Only integrations that used static credentials were affected. Integrations connected through cross-account or cross-service-account linking store no long-lived key with us, so no usable credential was taken from them, and the link could be rotated immediately. Customers who stored direct IAM credentials with us, had to rotate them manually.
There is no evidence that Git integrations have been used or abused, and no evidence of access to customer workloads or workload data.
We emailed every user on October 30, before we had established the scope, to ensure that every customer could act as quickly as possible.
Our security monitoring recorded the attacker's activity and alerted us at 11:43. Once the alert reached us, we fixed the vulnerability within 82 minutes.
We’ve been working with our security vendor to further help detect and alert on this new type of unprecedented automated attack.
If you have not already:
- Rotate every secret stored in Northflank, starting with AI provider keys, cloud keys, database passwords, payment keys and source-control tokens.
- Check usage and billing on your AI provider accounts for activity since October 30.
- We have revoked all Northflank API tokens. If you stored one as a secret, create a new one.
- Deleting a Northflank project or team does not invalidate keys for outside services. Rotate them at the provider.
If you have found unauthorized usage or charges:
- Contact your API provider’s account manager or support team to report the activity and request a billing review.
- Email us at support@northflank.com. Please include relevant audit logs, suspicious IP addresses, and correspondence with the API provider, including support case references.
- Connect cloud accounts with cross-account or cross-service-account linking, not long-lived IAM credentials.
- Where your provider supports it, use workload identity to inject short-lived credentials into your workloads instead of storing static keys. We are actively reviewing how to add workload identity support for a significant number of additional providers.
- Use egress gateways, and restrict third-party API tokens to your egress gateway IP addresses wherever the provider allows it. This applies to BYOC and PaaS customers alike.
- Subscribe to our status page: https://status.northflank.com
Completed:
- The vulnerability identified has been fixed, and the affected feature has been hardened.
- The platform credentials held by the affected component have been rotated, along with our database credentials and object storage credentials.
- All cross-account links, API keys, and VCS keys have been revoked and rotated.
- Added features for self-service credential rotation for all addons.
- Enabled audit logs for all customers (previously Enterprise-only).
- We've added audit logs via API available for Enterprise customers.
- Enhanced runtime detection for this kind of activity.
- Further automated intrusion response and blocking policies.
In progress:
- Continued hardening and control plane re-architecture.
- Continued deployment of AI-assisted red-teaming and pentesting against the platform.
- We are enhancing customer-facing features such as Workload Identity and Envelope Encryption.
- We will be releasing more security-related features for customer control and isolation in our changelog over the coming days and weeks.
| Type | Indicator | Role | First seen | Last seen |
|---|---|---|---|---|
| IPv4 | 3.212.63.55 (AWS EC2) | Attacker signup, command-and-control server and destination of the exfiltrated data | Oct 29, 22:57 | Oct 30, 08:12 |
| IPv4 | 209.50.164.61 | Source of the attacker's scripted Northflank API calls | Oct 30, 00:26 | Oct 30, 08:12 |
| Hostname | 3-212-63-55.sslip.io | Attacker-controlled server, configured as a self-hosted Git integration endpoint | Oct 30, 01:54 | |
| Ports | TCP 18700–18949 and TCP 8080 on 3.212.63.55 | Data upload over plain HTTP, and the attacker's webhook | Oct 30, 02:39 | Oct 30, 08:12 |
| User-Agent | Python-urllib/3.11, curl/7.88.1 | Attacker's API clients | Oct 30, 00:26 |
The attacker signed up with pt236432428@maxxspace[.]com and pt236718025@maxxspace[.]com
The attacker sent each command as a base64-encoded shell command, which the component’s own process decoded and ran. The attacker wrote their tools to /tmp and ran them in the component’s runtime.
| Type | Indicator |
|---|---|
| Files in /tmp | s.js, mshape.js, x.js, y.js |
| Files in /tmp | pg*.js, including pgc.js, pgcms.js, pgl.js, pggd.js, pgx.js, pgy.js, pgz.js, pgtok.js |
| Files in /tmp | rr.js, rr2.js, rr3.js |
| Files in /tmp | h.js, httpc.js, p.js, r.js |
| Files in /tmp | sw.sh, tg.txt, o |
| Cleanup | Files deleted from /tmp between 08:00 and 08:12 |
Recent attacks across developer platforms have shown how valuable stored credentials, particularly AI provider keys, have become to attackers. We need to account for that in how we build and operate Northflank, including limiting what a compromised component can access and how quickly suspicious activity reaches our engineers.
Security will remain a priority for our engineering team long after the immediate response is complete. We will share the work behind these changes, including what we learn and the decisions we make, so other teams can use those lessons to strengthen their own systems.
Thank you for your patience, your feedback and the kindness you have shown our team. We know this has been a difficult experience for customers, and we have work ahead of us to rebuild your trust.