Postgres isn't slow, your storage is: how to find disk waits with pg_stat_io

A ClickHouse write-up of Sai Srirampur's POSETTE 2026 talk, "Postgres isn't slow, your storage is", is on the front page of Hacker News this week, and its headline number is hard to ignore: on the same instance type and the same 3.3-billion-row table, PostgreSQL on local NVMe ran a random-update workload at a median 16,030 transactions per second, against 1,734 on a baseline gp3 EBS volume (ClickHouse). Latency went from 36.9 ms to 4.0 ms. The CPU work per transaction was the same; the difference was time spent waiting for the disk.

That is one vendor's benchmark on one workload, and ClickHouse sells NVMe-backed Postgres. But the method behind it is useful whatever you run. This article shows how to tell whether your own Postgres is waiting on storage, using wait events, pg_stat_io and iostat; what to change on EBS before you move anything; and what local NVMe costs you in durability if you do move.

Where the 37 milliseconds went

The test ran on eight identical clusters, four on instance-store NVMe and four on gp3 provisioned at its 3,000 IOPS baseline, all on m6id.4xlarge (16 vCPUs, 64 GiB RAM) with shared_buffers at 16 GiB. The table, pgbench_accounts, was 482 GiB, about seven times the machine's memory. Sixty-four clients ran five minutes of single-row updates on random keys (ClickHouse):

\set aid random(1, 3300000000)UPDATE pgbench_accounts SET abalance = abalance + 1 WHERE aid = :aid;

The interesting part is the breakdown of one transaction. On EBS, almost all of it is waiting:

Two to three milliseconds of real work, and on EBS about thirty of waiting for a page to come off the volume or for the WAL to be flushed. The CPU profile says the same from the other side: over a 240-second window the NVMe host burned about 9.4 cores and the EBS host about one (ClickHouse). Maintenance followed suit. Vacuuming 10 GB of bloat took 366 s on NVMe and 964 s on EBS; draining a 10 GB logical replication backlog took 139 s and 238 s.

Why a working set larger than memory turns into disk waits

Postgres reads and writes 8 kB pages through shared_buffers, with the operating system's page cache behind it. While the hot data fits in those two, a random update is a memory operation plus one WAL flush at commit. When it does not, every miss is a synchronous read that the backend sits and waits for, and every commit still has to wait for the WAL to reach durable storage.

On a networked volume each of those round trips crosses the network. AWS describes gp3 as providing "single-digit millisecond latency" and recommends io2 Block Express for latency-sensitive workloads (AWS). A few milliseconds per read, several reads per transaction, and a flush on commit add up to the 37 ms above. The talk's abstract lists the symptoms this produces: low CPU, high latency, insert rates that fall as the data grows, and behaviour that is hard to predict on multi-terabyte datasets (POSETTE 2026).

Check whether your Postgres is waiting on storage

You do not need a benchmark to find out. Postgres will tell you what its backends are waiting on.

Turn on I/O timing

track_io_timing records how long block reads, writes and fsyncs take; track_wal_io_timing does the same for WAL (PostgreSQL docs). Both are off by default. They can be changed with a reload:

ALTER SYSTEM SET track_io_timing = on;ALTER SYSTEM SET track_wal_io_timing = on;SELECT pg_reload_conf();

Sample wait events during your busy hour

pg_stat_activity shows what each backend is waiting on right now. Sample it every second for a few minutes while the system is under real load:

SELECT wait_event_type, wait_event, count(*) AS backendsFROM pg_stat_activityWHERE state = 'active' AND backend_type = 'client backend'GROUP BY 1, 2ORDER BY backends DESC;\watch 1

A NULL wait event means the backend is on CPU. The ones that point at storage are IO:DataFileRead, waiting for a read from a relation data file; IO:WalSync, waiting for a WAL file to reach durable storage; and LWLock:WALWrite, waiting for WAL buffers to be written to disk (PostgreSQL docs). In the ClickHouse run, a mid-test snapshot had 29 of 64 backends on EBS in IO:DataFileRead, against 9 on NVMe (ClickHouse). If a large share of your active backends sit in those events at peak, storage is the first thing to look at, before query plans or connection counts.

Measure time per read with pg_stat_io

Wait events are snapshots. pg_stat_io, added in PostgreSQL 16, keeps cumulative counts and, with timing on, the time spent (PostgreSQL docs). Dividing one by the other gives an average latency per operation:

SELECT backend_type, object, context,       reads,       round((read_time  / nullif(reads, 0))::numeric, 3)  AS ms_per_read,       fsyncs,       round((fsync_time / nullif(fsyncs, 0))::numeric, 3) AS ms_per_fsyncFROM pg_stat_ioWHERE reads > 0 OR fsyncs > 0ORDER BY read_time DESC NULLS LASTLIMIT 15;

On PostgreSQL 18, WAL has its own rows here (object = 'wal'), and fsyncs/fsync_time on those rows are the commit flushes. PostgreSQL 18 removed wal_sync and wal_sync_time from pg_stat_wal and moved track_wal_io_timing to feed pg_stat_io instead (PostgreSQL 18 release notes), so on 16 and 17 read WAL sync time from pg_stat_wal instead.

Confirm at the volume

From the operating system, iostat from the sysstat package shows the device's view:

$ iostat -x 5 nvme1n1

Watch r_await and w_await (milliseconds per request) and r/s plus w/s. If reads and writes per second together sit flat at your provisioned IOPS while the wait times climb, the volume is throttling you.

Fix the volume before you move the database

The benchmark compared NVMe with gp3 at its baseline, and the post says a different EBS configuration would give different results (ClickHouse). Baseline is a low bar. gp3 includes 3,000 IOPS and 125 MiB/s in the storage price, and you can provision up to 80,000 IOPS (at 500 IOPS per GiB of volume) and 2,000 MiB/s for an extra charge (AWS). If your counters show you pinned at 3,000, raising the limit is a single online change:

$ aws ec2 modify-volume --volume-id vol-0123456789abcdef0 \    --iops 12000 --throughput 500$ aws ec2 describe-volumes-modifications --volume-ids vol-0123456789abcdef0

More IOPS raises the ceiling but does not shorten each round trip, so a workload limited by per-read latency rather than by the IOPS cap will improve less. That is the case the talk is about, and it is where local NVMe or more memory is the real fix.

ClickHouse ran a second, broader test in April with pgbench's TPC-B-like workload on 16-vCPU instances and default settings. At about 500 GB of data it reported these results (ClickHouse PostgresBench):

What local NVMe costs you

Instance-store NVMe is fast because it is a disk in the host, and that is also its weakness. AWS is explicit: the data persists across a reboot, but not when the instance is stopped, hibernated or terminated, or when the underlying disk fails, and on stop every block is cryptographically erased (AWS). A routine stop for maintenance wipes the database. The volumes cannot be snapshotted through the EBS API either (ClickHouse).

The design the talk recommends is the one durability then depends on: two standbys in other availability zones with quorum synchronous replication, so a commit returns once either standby has the WAL, plus continuous WAL archiving to object storage for point-in-time recovery (ClickHouse). With WAL-G, the primary's side of that looks like this:

synchronous_commit = onsynchronous_standby_names = 'ANY 1 (standby_a, standby_b)' archive_mode = onarchive_command = 'wal-g wal-push %p'

The names in synchronous_standby_names match each standby's application_name. Each replica needs its own NVMe and its own base backup, and you need to have actually restored from the archive at least once before you rely on it.

NVMe is also not a fix for everything. The talk's own abstract says wide scans and repeated aggregations over hundreds of millions of rows stay bound by CPU in the executor, because fast storage does not turn a row store into a column store (POSETTE 2026).

The one thing to remember: when Postgres is slow and the CPU is idle, measure the waits. The database will tell you whether it is the database.