20 posts tagged with "performance"

While writing our Sizing documentation for Camunda, we noticed that our previous load tests did not account for the default flow control settings. This impacted for example our experiments for Camunda 8.7 and beyond performance and reliability results. That means our results did not describe what we ship.

Today we ran several load tests with different flow control settings to understand how flow control affects throughput and data availability, and whether our earlier results need a correction.

Under stress, the data availability reported by our load tester ("how long after creation can a process instance be found via the search API") was at the 90-second limit for REST (p50), but about 52 seconds (p50) for gRPC. Today, we wanted to find out whether the REST API, the gateway, or our load tester is responsible.

TL;DR: The 90-second plateau is a client-side effect of the load tester, not a slow REST API.

• The search is the same REST endpoint in both setups, whether instances are created through REST or gRPC, and the server answers /v2/process-instances/search with a p50 of about 30 to 47 ms and a p90 of about 73 to 95 ms in all four tests (REST slightly higher).
• The starter (the load tester application that creates process instances) uses one Camunda client for creating instances (target: 300 process instances per second) and for the availability check. With REST, requests queue up in the client (all 100 pooled connections are leased, with about 2,900 requests waiting), and the availability search waits in the same queue. The starter measured about 15 seconds for a query that took the server 70 ms.
• The availability check starts the next check only after the previous one is completed, and returns at most 2,500 instances per check. With a 15-second cycle, it confirms at most about 167 instances per second, an upper bound that real checks stay below. Instances that are not confirmed in time are dropped after 90 seconds and recorded with the same metric as real latency, so timeouts appear as slow data availability.
• A second Camunda client only for the read requests brings the search time seen by the starter from about 15 seconds to about 0.12 to 0.15 seconds (camunda/camunda#65355, merged). The mean data availability of the REST test is about 57 to 68 seconds with this client (97 to 99 seconds without it). For gRPC, it is about 45 seconds.
• The remaining delay is related to the exporter backlog under the stress load, which can be estimated on the server side as backlog / export rate

On today's Chaos Day, we investigated a puzzling result of our daily load tests, reported in camunda/camunda#62783: without secondary storage, the REST load test is much slower than the same test with Elasticsearch, while for gRPC it is the other way around, as one would expect.

We run several load tests every day (gRPC and REST, each with and without Elasticsearch as secondary storage) against the current main. The "None" variants run without secondary storage to max out the engine and observe its limits. In the results above, None-gRPC completes about 491 process instances per second (PI/s), while None-REST completes only about 85 PI/s, even though its starter reports starting 435 PI/s. With Elasticsearch, the REST test completes about 170 PI/s and is at least in the same range as gRPC.

We had already seen that the starter, the application that creates process instances, was heavily CPU throttled. Today, we wanted to find out why.

TL;DR; The collapse was caused by the load tester itself, not by Camunda.

• The starter's HTTP client has a pool of 100 connections. At a configured rate of 500 PI/s, the starter scheduled more requests than the pool completed and queued the rest in memory, up to about 85,000 requests.
• The full heap made the JVM spend most of its CPU on garbage collection, which slowed the starter down and caused the CPU throttling we saw. Giving the starter more CPU made it worse: with two or more CPUs, the JVM runs the G1 garbage collector instead of the serial one, and the starter then has enough CPU to keep sending at the full rate until the heap is exhausted. The starter ran out of memory, and its scheduler died silently.
• Limiting the number of in-flight requests keeps the starter stable under the configured load, in an open workload model: when the system cannot keep up, the starter skips sending instead of queueing them. With this prototype, we reach about 375 PI/s out of the configured 500 PI/s. This is now limited by Camunda's CPU: all three Camunda pods run at their limit of 3 CPUs. The remaining difference to the 491 PI/s of None-gRPC is expected: the protocols differ, and there are known REST performance issues, for example camunda/camunda#35067.
• We also added HTTP client metrics (camunda/camunda#64477) that made the backlog visible. The investigation produced eight new issues and five draft fixes for the load tester, the Java client, and the load test configuration, listed at the end.

On today's Chaos Day, we simulated an extended outage of a job worker in our realistic "bank customer complaint/dispute handling" load test to see how well the system recovers once the worker returns.

We picked this scenario because it had already happened to us once, by accident: a worker in one of our other load tests had gone down for a while, and when it came back, throughput looked fine while something underneath clearly was not. We never got to the bottom of it at the time. Today, we deliberately reproduced the same shape of failure, with the time to actually dig in.

TL;DR: We experimented with our realistic workload and took a job worker down for 80 minutes. Throughput recovered within seconds of its return, but the backlog behind it grew for another two hours, because the step we removed fans one process instance into 50 downstream jobs. Adding capacity and replicas shifted the throughput, but the backlog reduction always remained at the same rate. Two reasons for this. First, job push is prioritized, as new jobs are handed straight to a worker, so they never enter the queue the backlog lives in, leaving polling to drain it while competing for the same capacity. Second, the client's poll path has four defects of its own: two explain what we measured, and two more we found by reading the code. All observed behaviors are filed as issues: camunda/camunda#59631 for the engine side and camunda/camunda#59635 for the client.

In a previous Chaos Day and its variable-filtering follow-up, we measured Optimize's Elasticsearch overhead against Self-Managed load tests. Running the same kind of test against a Camunda SaaS cluster turned up something we didn't expect: Optimize's disk footprint there looks nothing like what we'd measured on Self-Managed. This Chaos Day traces that discovery to its root cause and confirms it through a controlled experiment.

TL;DR; A week-long test against a SaaS Advanced 4x cluster showed Optimize's indices taking up only ~7-10% of total Elasticsearch disk, versus ~59-100% on our Self-Managed weekly load test running the exact same workload. The cause: Optimize's includeObjectVariableValue flag (env CAMUNDA_OPTIMIZE_ZEEBE_INCLUDE_OBJECT_VARIABLE) defaults to true and flattens every JSON object variable into one stored variable per property, plus the raw serialized object itself. Camunda SaaS explicitly disables this; the public Self-Managed Helm chart does not, so any Self-Managed deployment that hasn't touched this setting silently pays for it. We confirmed this with an isolated A/B test that changes only this one flag: Optimize's ES disk share dropped from 62.8% to 7.6%, an 8.3x reduction, for the same workload. The number that matters most for capacity planning: total secondary storage per root process instance dropped from 6.34 MB to 2.97 MB, a 2.13x reduction. This ratio is specific to our payload's shape: flattening recurses through nested JSON with no depth limit, so a payload with deeper nesting or more object fields can cost considerably more than this.

In a previous Chaos Day, we measured what Optimize costs a cluster: at a realistic workload, it drove 3.4x higher Elasticsearch (ES) CPU and ~4x more ES disk than running without it. That post ended with an open question: Can variable handling be tuned to reduce the impact? This Chaos Day answers it.

We ran twelve load tests on Camunda 8.9.9, six variable-filtering configurations, each at both a realistic and a max workload, and compared throughput, CPU, memory, and ES disk usage across all configurations. All twelve started together and ran in parallel on identical infrastructure and the same Helm chart, each started fresh with an empty Elasticsearch, so their footprints are directly comparable.

TL;DR; Keeping variables out of Optimize is the big lever. Disabling variable export (index.variable=false) cuts total Elasticsearch storage ~60% and ES CPU ~65% at a realistic load, and recovers ~15-20% throughput at max load. That is enough to give back essentially all the throughput Optimize was shown to cost in the previous Chaos Day, while keeping Optimize running. The surprising part: Optimize stores a variable ~14x more expensively than the raw Zeebe export does (≈29x for high-cardinality string variables), so the cost lives almost entirely in Optimize's imported indices, not in the export. And because in Camunda 8.9 the Elasticsearch exporter feeds only Optimize, this costs you Optimize variable analytics only: Operate and Tasklist (served by the Camunda Exporter) keep their variables untouched.

On this Chaos Day, we measured the impact of Optimize on cluster performance and resource usage. We ran four 2-day load tests on Camunda 8.9.6 — two with Optimize enabled (max and realistic workloads) and two without — and compared throughput, latency, CPU, memory, and disk across all four.

TL;DR; Optimize has a measurable negative impact on throughput under high load (-22% completed PI/s), but the most striking finding is its additional CPU load on Elasticsearch and extra disk footprint at a realistic workload: Elasticsearch CPU was 3.4x higher with Optimize enabled. After two days at a realistic workload, the cluster with Optimize accumulated a maximum of 221.5 GiB of ES data, vs. 61.5 GiB without Optimize, a 3.6x difference. This is a critical finding for customers running Optimize at production scale, as it means that ES resources must be sized to account for Optimize's overhead, even at non-stress workloads.

On this Chaos Day, we conducted an experiment to evaluate the performance of our platform without the use of the secondary storage. The goal was to understand how the system behaves under such conditions and whether and how performance would improve.

TL;DR; We observed that a cluster without secondary storage achieves significantly higher throughput, as it is not throttled by secondary storage and can reach up to 400 PI/s without issues. That is a factor of 1.7x higher than the cluster with secondary storage.

As businesses increasingly rely on process automation for their critical operations, the question of reliability becomes paramount. How can you trust that your automation platform will perform consistently under pressure, recover gracefully from failures, and maintain performance over time?

At Camunda, we've been asking ourselves these same questions for years, and today I want to share how our reliability testing practices have evolved to ensure our platform meets the demanding requirements of enterprise-scale deployments. I will also outline our plans to further invest in this crucial area.

In today's chaos experiment, we focused on stress-testing the Camunda 8 orchestration cluster under high-load conditions. We simulated a large number of concurrent process instances to evaluate the performance of processing and system reliability.

Due to our recent work in supporting load tests for different versions, we were able to compare how different Camunda versions handle stress.

TL;DR: Overall, we saw that all versions of the Camunda 8 orchestration cluster (with focus on the processing) are robust and can handle high loads effectively and reliably. In consideration of throughput and latency, with similar resource allocation among the brokers, 8.7.x outperforms other versions. If we consider our streamlined architecture (which now contains more components in a single application) and align the resources for 8.8.x, it can achieve similar throughput levels as 8.7.x, while maintaining significantly lower latency (a factor of 2). An overview of the results can be found in the Results section below.