Northflank breach: what to rotate first, and how to check whether your AI keys were abused

On 30 October 2026 Northflank told every customer to assume all environment variables were decrypted and to disable or rotate every key stored on the platform. Within hours, users reported that AI API keys held there had been used by someone else (Northflank status page). It is the ninth AI developer platform to disclose an incident this month, according to Wiz's "Black October" report, and the pattern is the same each time: the attacker wants the environment variables, and above all the ones that buy model inference.

This article is for engineers who deploy on Northflank, Railway or any platform that stores your secrets for you. It covers what is known about the Northflank incident, why stolen AI keys are worth so much, the order to rotate things in without taking production down, how to check AWS Bedrock for abuse with CloudTrail, and how to scope a key so the next leak costs less.

What happened in the Northflank breach?

A Wiz threat alert fired at 11:43 UTC on 30 October; by 13:58 Northflank had moved the incident to Identified and told customers to treat every environment variable as exposed. The root cause has not been published. Networking, services, builds, jobs, add-ons, certificates and logs were all marked as a major outage while the team patched (status page).

Two things in that timeline matter more than the rest. First, the instruction covers every secret — workload variables, secret groups and global secrets — not only the ones you think of as sensitive. Second, the abuse was visible within hours, which means the window between theft and use is short. Northflank has not said how the variables were decrypted, and the Hacker News thread has no statement from staff, so treat any root-cause theory you read elsewhere as unconfirmed.

Why do attackers target AI API keys on developer platforms?

Because a working inference key can be resold the same day. Wiz calls it LLMJacking: stealing access to large language models and reselling it. Its report lists nine platforms that disclosed incidents in October — Fireworks AI, Andromeda, Zeabur, Lightning AI, OpenCode, Railway, Warp, CommandCode and Northflank — and notes that every public report describes "targeted hunting of stored environment variables, especially those granting AI access" (Wiz).

Railway's own disclosure says the same from the victim's side: it detected anomalous activity in "systems that store customer environment variables" on 21 October, some decrypted values may have been taken, and it believes the attack is part of a series "targeting platform companies" to compromise AI inference providers (Railway).

Wiz followed two enterprise AWS keys that leaked through vendor breaches. In both, the first unauthorised InvokeModel call came within 30 minutes of the reported credential access, from a residential IP sweeping several organisations in one minute. For days afterwards each key saw 14–120 calls a day from one small Go client — a checker making sure the key still worked. Then the bulk traffic arrived (Wiz).

The restricted key in Case B still produced more than a million successful calls; most of its other traffic was throttled by regional quotas. Wiz estimates the two cases "likely ran up over $100k in usage" at list price, and the traffic looked like resold coding-agent sessions: heavy prompt caching and a long tail of large uncached inputs (Wiz). Scoping a key limits what it can reach. It does not stop a determined reseller from spending your quota.

What to rotate first after the Northflank breach, and in what order

Rotate what can be turned into money fastest, then what gives access to data, then everything else. Revoking at the issuer is what counts; changing the value in Northflank alone leaves the stolen copy working.

• AI provider and cloud keys first. OpenAI, Anthropic, Google, AWS and any inference reseller keys. Revoke them with the provider, create replacements, then update the secret in Northflank. Then check usage and billing for the last few days (see the next section).
• Database credentials second. Northflank now offers rotation for Postgres and MongoDB (since 30 October) and MySQL and Redis (since 31 October). If you enabled public access on an add-on, turn it off (status page).
• Payment, email and webhook secrets third. Stripe keys, SMTP passwords, webhook signing secrets, OAuth client secrets.
• Build and source credentials. Registry tokens, Git deploy keys and CI tokens stored as variables or secret groups.
• Everything else. Session-signing keys, encryption keys and internal service tokens. Rotating a session secret logs every user out, so schedule it, but do it.

What breaks when you rotate in a hurry is usually the order of operations. If you revoke a database password before the new one is in every service, connections fail as pools recycle. For anything that supports two live credentials — an IAM user can hold two access keys, and AI providers generally let you create several API keys — create the new key, deploy it, confirm traffic has moved, then revoke the old one. For a single-password resource, rotate during a short window and restart the services that hold connections. The one exception is a key you can see being abused: revoke that immediately and accept the outage.

Northflank switched on audit logs for every customer on 31 October. Use them to see what changed in your team since the incident: they record the event type, who or what triggered it, the source and, for configuration changes, the before-and-after template. Audit log sinks can forward events to S3, but new sinks do not export earlier events, so read the UI now rather than waiting for a sink.

How to check whether a stolen key was used on AWS Bedrock

Look in CloudTrail. Bedrock logs InvokeModel, InvokeModelWithResponseStream, Converse and ConverseStream as management events, which CloudTrail records by default, so you have a history even if you never set up data events (AWS docs). Filter by the access key that was stored on the platform.

# The access key ID that was stored in Northflank (not the secret)KEY=AKIAEXAMPLEEXAMPLE aws cloudtrail lookup-events \  --region us-east-1 \  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=$KEY \  --start-time 2026-10-29T00:00:00Z \  --query 'Events[?EventSource==`bedrock.amazonaws.com`].[EventTime,EventName]' \  --output table

lookup-events covers the last 90 days in one region at a time and returns pages of 50, so for a key used heavily it is quicker to count with a short script. This one groups calls by source IP and model, which is what you need to separate your own servers from someone else's:

import json, sysfrom collections import Counterfrom datetime import datetime, timezoneimport boto3 key_id, region = sys.argv[1], sys.argv[2]since = datetime(2026, 10, 29, tzinfo=timezone.utc) ct = boto3.client("cloudtrail", region_name=region)pages = ct.get_paginator("lookup_events").paginate(    LookupAttributes=[{"AttributeKey": "AccessKeyId", "AttributeValue": key_id}],    StartTime=since,) by_ip, by_model = Counter(), Counter()for page in pages:    for ev in page["Events"]:        if ev["EventSource"] != "bedrock.amazonaws.com":            continue        rec = json.loads(ev["CloudTrailEvent"])        if rec.get("eventType") != "AwsApiCall":  # skip stream-completion events            continue        by_ip[rec.get("sourceIPAddress")] += 1        model = (rec.get("requestParameters") or {}).get("modelId", "?")        by_model[model] += 1 print("calls by source IP:", by_ip.most_common(10))print("calls by model:", by_model.most_common(10))

Run it as python bedrock_key_usage.py AKIA... us-east-1 for each region you use. The eventType filter matters: streaming calls also write a stream-completion record of type AwsServiceEvent, which AWS says to exclude when you count invocations (AWS docs). Calls from addresses that are not yours, sudden use of models you never call, or a burst of quota look-ups just before heavy traffic all match what Wiz saw. Wiz published four addresses seen checking or using stolen keys between 18 and 30 October — 23.106.46.239, 38.246.114.118, 104.243.32.50 and 31.57.201.201 — while noting they are not directly linked to the Black October intrusions (Wiz).

For OpenAI, Anthropic and other direct providers, the equivalent check is the usage page of the organisation or project the key belongs to, plus your billing alerts. If you find abuse, report it to the provider as well as to Northflank support, as the status page asks; providers can sometimes see more about where the traffic went than you can.

How to scope a Bedrock key so the next leak costs less

Give each workload its own credentials, allow only the models and inference profiles it uses, and deny global routing unless you need it. Wiz notes that Case A's policy allowed global.* cross-region inference profiles, so only about 6% of the abusive calls were throttled; Case B's key allowed only us.* profiles, and thousands of reseller requests for global.* failed (Wiz).

Global inference profiles have IDs starting with global.. AWS documents a detail that catches people out: when Bedrock evaluates the global foundation model, aws:RequestedRegion is unspecified, so a deny on a named region does not match it (AWS docs). The service control policy AWS gives for turning global routing off is:

{  "Effect": "Deny",  "Action": "bedrock:*",  "Resource": "*",  "Condition": {    "StringEquals": {      "aws:RequestedRegion": "unspecified"    },    "ArnLike": {      "bedrock:InferenceProfileArn": "arn:aws:bedrock:*:*:inference-profile/global.*"    }  }}

That is a fragment: put it inside the Statement array of an SCP with "Version": "2012-10-17", and attach it to the accounts that run your workloads. On the identity side, replace bedrock:InvokeModel on * with the specific inference-profile and foundation-model ARNs each service needs. A key that can only call one model in one geography is worth less to a reseller, and a key that fails on the first catalogue probe is more likely to be dropped from the pool.

Should AI keys live in platform environment variables at all?

Long-lived keys in a platform's secret store are convenient, and October showed their cost: when the store is breached, every tenant's keys go at once. You cannot remove that risk entirely on a managed platform, but you can make what is stored there less valuable.

Where the provider supports it, prefer short-lived credentials over static keys — on AWS, a role assumed through web identity federation rather than an IAM user's access key — so a stolen value expires on its own. Where you must store a static key, issue one per service and per environment, give it a spending limit at the provider if one exists, and keep a written list of where each key is stored. Knowing which secrets live on which platform is what turns a rotation like this one from guesswork into a list.

Commenters in the Hacker News discussion split between moving back to hyperscalers and self-hosting. Neither is automatically safer: Wiz's two enterprise cases were AWS keys leaked through vendors, not through AWS. The useful lesson is narrower — any place that can decrypt your secrets is part of your attack surface, and it should hold as few high-value secrets as you can manage.

The one thing to remember

If a platform tells you to assume your environment variables were decrypted, the stolen copies are already being tested. Revoke at the issuer, starting with whatever turns into inference spend, then check the provider's logs for calls you did not make — and scope the replacement key so that next time it is worth less to whoever takes it.

Sources and further reading

• Northflank status page: security incident, 30–31 October 2026
• Wiz: Black October for AI developer platforms
• Railway: security disclosure, 21 October 2026
• AWS: Global cross-Region inference in Amazon Bedrock
• AWS: Logging Amazon Bedrock API calls with CloudTrail
• Northflank docs: audit logs
• Hacker News discussion of the Northflank breach