Christopher Kujawa

On today's Chaos Day, we investigated a puzzling result of our daily load tests, reported in camunda/camunda#62783: without secondary storage, the REST load test is much slower than the same test with Elasticsearch, while for gRPC it is the other way around, as one would expect.

We run several load tests every day (gRPC and REST, each with and without Elasticsearch as secondary storage) against the current main. The "None" variants run without secondary storage to max out the engine and observe its limits. In the results above, None-gRPC completes about 491 process instances per second (PI/s), while None-REST completes only about 85 PI/s, even though its starter reports starting 435 PI/s. With Elasticsearch, the REST test completes about 170 PI/s and is at least in the same range as gRPC.

We had already seen that the starter, the application that creates process instances, was heavily CPU throttled. Today, we wanted to find out why.

TL;DR; The collapse was caused by the load tester itself, not by Camunda.

• The starter's HTTP client has a pool of 100 connections. At a configured rate of 500 PI/s, the starter scheduled more requests than the pool completed and queued the rest in memory, up to about 85,000 requests.
• The full heap made the JVM spend most of its CPU on garbage collection, which slowed the starter down and caused the CPU throttling we saw. Giving the starter more CPU made it worse: with two or more CPUs, the JVM runs the G1 garbage collector instead of the serial one, and the starter then has enough CPU to keep sending at the full rate until the heap is exhausted. The starter ran out of memory, and its scheduler died silently.
• Limiting the number of in-flight requests keeps the starter stable under the configured load, in an open workload model: when the system cannot keep up, the starter skips sending instead of queueing them. With this prototype, we reach about 375 PI/s out of the configured 500 PI/s. This is now limited by Camunda's CPU: all three Camunda pods run at their limit of 3 CPUs. The remaining difference to the 491 PI/s of None-gRPC is expected: the protocols differ, and there are known REST performance issues, for example camunda/camunda#35067.
• We also added HTTP client metrics (camunda/camunda#64477) that made the backlog visible. The investigation produced eight new issues and five draft fixes for the load tester, the Java client, and the load test configuration, listed at the end.

On today's Chaos Day, we investigated how Camunda clients (starters and workers) behave during cluster bootstrap authentication, prompted by camunda/camunda#58983, a report that load-test clients can be unable to authenticate for a long time right after a cluster is created. We wanted to walk through the whole startup sequence, Elasticsearch, Camunda, Management Identity, Keycloak, and clients, and find exactly where a client can get stuck.

TL;DR: During cluster bootstrap, a client can receive an OAuth 401 before Camunda has finished granting the permissions that make its token valid. The Camunda Java client's OAuthCredentialsProvider then latches into a non-retryable cooldown period that defaults to 5 minutes, so that a single early 401 can leave a client unable to authenticate for minutes, even though the rest of the cluster recovers within seconds. Lowering camunda.client.auth.token-fetch-non-retryable-cooldown to 30 seconds fixes this for our load tests (camunda/camunda#62862). Along the way, we also hit a false-positive ERROR log bug (camunda/camunda#62686, fixed) and confirmed two observability gaps: Management Identity exposes no metrics, and the client exposes none for its OAuth token-fetch/retry behavior (camunda/camunda#50684, camunda/camunda#51113).

Over the last week, we ran several endurance and stress tests against Camunda 8.7, 8.8, and 8.9 (using the latest patch versions). In this post, we want to summarize the results of our experiments, explain the differences between the versions, and share our findings with the community.

TL;DR; In stress tests, Camunda 8.7 was performing ~50% better than 8.8 and 8.9 in terms of processing throughput. That processing performance was traded for improved reliability and stability, achieved by applying queueing-theory-informed backpressure, a huge benefit for our users. Under stress, 8.7's importing was not able to keep up with the processing rate, so the importing backlog (and with it the data availability latency) grew unbounded, up to nine hours in our tests. In 8.8 and 8.9, data availability was instead kept under 40 seconds under stress (roughly an 810x difference). In the endurance tests, data availability was more than a factor of two smaller in 8.8+ as well. In terms of stability, we also observed the archiver failing in 8.7, which led to a backlog of completed process instances in the runtime indices; this component, known for its fragility in the past, has been significantly improved in later versions. In addition to the above-mentioned re-architecture changes and improvements, 8.8 and beyond come with even more features (which might even help in such cases, like dynamic scaling) and stability improvements, which we will look into in future posts.

On today's Chaos Day, we simulated an extended outage of a job worker in our realistic "bank customer complaint/dispute handling" load test to see how well the system recovers once the worker returns.

We picked this scenario because it had already happened to us once, by accident: a worker in one of our other load tests had gone down for a while, and when it came back, throughput looked fine while something underneath clearly was not. We never got to the bottom of it at the time. Today, we deliberately reproduced the same shape of failure, with the time to actually dig in.

TL;DR: We experimented with our realistic workload and took a job worker down for 80 minutes. Throughput recovered within seconds of its return, but the backlog behind it grew for another two hours, because the step we removed fans one process instance into 50 downstream jobs. Adding capacity and replicas shifted the throughput, but the backlog reduction always remained at the same rate. Two reasons for this. First, job push is prioritized, as new jobs are handed straight to a worker, so they never enter the queue the backlog lives in, leaving polling to drain it while competing for the same capacity. Second, the client's poll path has four defects of its own: two explain what we measured, and two more we found by reading the code. All observed behaviors are filed as issues: camunda/camunda#59631 for the engine side and camunda/camunda#59635 for the client.

In a previous Chaos Day and its variable-filtering follow-up, we measured Optimize's Elasticsearch overhead against Self-Managed load tests. Running the same kind of test against a Camunda SaaS cluster turned up something we didn't expect: Optimize's disk footprint there looks nothing like what we'd measured on Self-Managed. This Chaos Day traces that discovery to its root cause and confirms it through a controlled experiment.

TL;DR; A week-long test against a SaaS Advanced 4x cluster showed Optimize's indices taking up only ~7-10% of total Elasticsearch disk, versus ~59-100% on our Self-Managed weekly load test running the exact same workload. The cause: Optimize's includeObjectVariableValue flag (env CAMUNDA_OPTIMIZE_ZEEBE_INCLUDE_OBJECT_VARIABLE) defaults to true and flattens every JSON object variable into one stored variable per property, plus the raw serialized object itself. Camunda SaaS explicitly disables this; the public Self-Managed Helm chart does not, so any Self-Managed deployment that hasn't touched this setting silently pays for it. We confirmed this with an isolated A/B test that changes only this one flag: Optimize's ES disk share dropped from 62.8% to 7.6%, an 8.3x reduction, for the same workload. The number that matters most for capacity planning: total secondary storage per root process instance dropped from 6.34 MB to 2.97 MB, a 2.13x reduction. This ratio is specific to our payload's shape: flattening recurses through nested JSON with no depth limit, so a payload with deeper nesting or more object fields can cost considerably more than this.

In today's Chaos Day, we continued our experiments with Optimize and its behavior with the default configuration, when the underlying Elasticsearch cluster has nodes restarting.

This exercise is the continuation of the recent improvements and clarifications we added to our documentation regarding the configuration, impact and scaling of Camunda clusters running Optimize.

TL;DR;

1. The default Camunda configuration does not create replica shards for the zeebe-record-* indices, making them unavailable during any Elasticsearch node restart.
2. Due to how Zeebe routes records using partition IDs (low-cardinality values hashed with Murmur3), some shards end up completely empty while others hold all the data.
3. When one of the zeebe-record-* indices becomes unavailable, Optimize's single importer thread stalls all other imports — not just the affected resource type — and takes up to 45 minutes to fully recover.

In a previous Chaos Day, we measured what Optimize costs a cluster: at a realistic workload, it drove 3.4x higher Elasticsearch (ES) CPU and ~4x more ES disk than running without it. That post ended with an open question: Can variable handling be tuned to reduce the impact? This Chaos Day answers it.

We ran twelve load tests on Camunda 8.9.9, six variable-filtering configurations, each at both a realistic and a max workload, and compared throughput, CPU, memory, and ES disk usage across all configurations. All twelve started together and ran in parallel on identical infrastructure and the same Helm chart, each started fresh with an empty Elasticsearch, so their footprints are directly comparable.

TL;DR; Keeping variables out of Optimize is the big lever. Disabling variable export (index.variable=false) cuts total Elasticsearch storage ~60% and ES CPU ~65% at a realistic load, and recovers ~15-20% throughput at max load. That is enough to give back essentially all the throughput Optimize was shown to cost in the previous Chaos Day, while keeping Optimize running. The surprising part: Optimize stores a variable ~14x more expensively than the raw Zeebe export does (≈29x for high-cardinality string variables), so the cost lives almost entirely in Optimize's imported indices, not in the export. And because in Camunda 8.9 the Elasticsearch exporter feeds only Optimize, this costs you Optimize variable analytics only: Operate and Tasklist (served by the Camunda Exporter) keep their variables untouched.

In today's Chaos Day, we wanted to experiment with slow disks, as we have recently run into some incidents related to that. We want to understand and document how Camunda behaves in such scenarios.

We have two main experiments planned: one for primary storage and one for secondary storage (in this case, Elasticsearch) using slow disks.

TL;DR; Using HDDs instead of SSDs on Camunda's primary storage caused around 50% throughput degradation — not because of lower disk throughput, but because of higher latency, which directly stalls Raft replication and commit acknowledgement. Moving the slow disk to Elasticsearch (secondary storage) was even worse, dropping throughput by ~70% and accumulating a permanent export backlog of ~200k records, with memory growing from unexported in-flight data. Both experiments confirm that SSDs are essential for both storage layers, and our documentation for secondary storage needs to be updated to reflect this.

On today's Chaos Day, we wanted to experiment with disks filling up due to soft-paused exporters. We have recently encountered some incidents in which these disks filled up. We wanted to understand how Zeebe behaves in such scenarios. We had the following experiment planned: reproducing full disks because exporters are not confirming positions (due to soft-pausing).

TL;DR: We were able to reproduce the full-disk scenario with soft-pausing exporters. The node becomes unresponsive and rejects requests. After unpausing the exporters, we were able to free up disk space again, but it took a while because the exporters needed to re-export all unacknowledged records after the restart. Some interesting learnings from this experiment are that we should not keep exporters soft-paused for long. This can be especially problematic if nodes get restarted. When the disk is full, backpressure still reports zero, but all requests are rejected. Even REST requests are no longer successful.

On this Chaos Day, we measured the impact of Optimize on cluster performance and resource usage. We ran four 2-day load tests on Camunda 8.9.6 — two with Optimize enabled (max and realistic workloads) and two without — and compared throughput, latency, CPU, memory, and disk across all four.

TL;DR; Optimize has a measurable negative impact on throughput under high load (-22% completed PI/s), but the most striking finding is its additional CPU load on Elasticsearch and extra disk footprint at a realistic workload: Elasticsearch CPU was 3.4x higher with Optimize enabled. After two days at a realistic workload, the cluster with Optimize accumulated a maximum of 221.5 GiB of ES data, vs. 61.5 GiB without Optimize, a 3.6x difference. This is a critical finding for customers running Optimize at production scale, as it means that ES resources must be sized to account for Optimize's overhead, even at non-stress workloads.