52 posts tagged with "availability"

While writing our Sizing documentation for Camunda, we noticed that our previous load tests did not account for the default flow control settings. This impacted for example our experiments for Camunda 8.7 and beyond performance and reliability results. That means our results did not describe what we ship.

Today we ran several load tests with different flow control settings to understand how flow control affects throughput and data availability, and whether our earlier results need a correction.

Under stress, the data availability reported by our load tester ("how long after creation can a process instance be found via the search API") was at the 90-second limit for REST (p50), but about 52 seconds (p50) for gRPC. Today, we wanted to find out whether the REST API, the gateway, or our load tester is responsible.

TL;DR: The 90-second plateau is a client-side effect of the load tester, not a slow REST API.

• The search is the same REST endpoint in both setups, whether instances are created through REST or gRPC, and the server answers /v2/process-instances/search with a p50 of about 30 to 47 ms and a p90 of about 73 to 95 ms in all four tests (REST slightly higher).
• The starter (the load tester application that creates process instances) uses one Camunda client for creating instances (target: 300 process instances per second) and for the availability check. With REST, requests queue up in the client (all 100 pooled connections are leased, with about 2,900 requests waiting), and the availability search waits in the same queue. The starter measured about 15 seconds for a query that took the server 70 ms.
• The availability check starts the next check only after the previous one is completed, and returns at most 2,500 instances per check. With a 15-second cycle, it confirms at most about 167 instances per second, an upper bound that real checks stay below. Instances that are not confirmed in time are dropped after 90 seconds and recorded with the same metric as real latency, so timeouts appear as slow data availability.
• A second Camunda client only for the read requests brings the search time seen by the starter from about 15 seconds to about 0.12 to 0.15 seconds (camunda/camunda#65355, merged). The mean data availability of the REST test is about 57 to 68 seconds with this client (97 to 99 seconds without it). For gRPC, it is about 45 seconds.
• The remaining delay is related to the exporter backlog under the stress load, which can be estimated on the server side as backlog / export rate

On today's Chaos Day, we investigated how Camunda clients (starters and workers) behave during cluster bootstrap authentication, prompted by camunda/camunda#58983, a report that load-test clients can be unable to authenticate for a long time right after a cluster is created. We wanted to walk through the whole startup sequence, Elasticsearch, Camunda, Management Identity, Keycloak, and clients, and find exactly where a client can get stuck.

TL;DR: During cluster bootstrap, a client can receive an OAuth 401 before Camunda has finished granting the permissions that make its token valid. The Camunda Java client's OAuthCredentialsProvider then latches into a non-retryable cooldown period that defaults to 5 minutes, so that a single early 401 can leave a client unable to authenticate for minutes, even though the rest of the cluster recovers within seconds. Lowering camunda.client.auth.token-fetch-non-retryable-cooldown to 30 seconds fixes this for our load tests (camunda/camunda#62862). Along the way, we also hit a false-positive ERROR log bug (camunda/camunda#62686, fixed) and confirmed two observability gaps: Management Identity exposes no metrics, and the client exposes none for its OAuth token-fetch/retry behavior (camunda/camunda#50684, camunda/camunda#51113).

Over the last week, we ran several endurance and stress tests against Camunda 8.7, 8.8, and 8.9 (using the latest patch versions). In this post, we want to summarize the results of our experiments, explain the differences between the versions, and share our findings with the community.

TL;DR; In stress tests, Camunda 8.7 was performing ~50% better than 8.8 and 8.9 in terms of processing throughput. That processing performance was traded for improved reliability and stability, achieved by applying queueing-theory-informed backpressure, a huge benefit for our users. Under stress, 8.7's importing was not able to keep up with the processing rate, so the importing backlog (and with it the data availability latency) grew unbounded, up to nine hours in our tests. In 8.8 and 8.9, data availability was instead kept under 40 seconds under stress (roughly an 810x difference). In the endurance tests, data availability was more than a factor of two smaller in 8.8+ as well. In terms of stability, we also observed the archiver failing in 8.7, which led to a backlog of completed process instances in the runtime indices; this component, known for its fragility in the past, has been significantly improved in later versions. In addition to the above-mentioned re-architecture changes and improvements, 8.8 and beyond come with even more features (which might even help in such cases, like dynamic scaling) and stability improvements, which we will look into in future posts.

On today's Chaos Day, we simulated an extended outage of a job worker in our realistic "bank customer complaint/dispute handling" load test to see how well the system recovers once the worker returns.

We picked this scenario because it had already happened to us once, by accident: a worker in one of our other load tests had gone down for a while, and when it came back, throughput looked fine while something underneath clearly was not. We never got to the bottom of it at the time. Today, we deliberately reproduced the same shape of failure, with the time to actually dig in.

TL;DR: We experimented with our realistic workload and took a job worker down for 80 minutes. Throughput recovered within seconds of its return, but the backlog behind it grew for another two hours, because the step we removed fans one process instance into 50 downstream jobs. Adding capacity and replicas shifted the throughput, but the backlog reduction always remained at the same rate. Two reasons for this. First, job push is prioritized, as new jobs are handed straight to a worker, so they never enter the queue the backlog lives in, leaving polling to drain it while competing for the same capacity. Second, the client's poll path has four defects of its own: two explain what we measured, and two more we found by reading the code. All observed behaviors are filed as issues: camunda/camunda#59631 for the engine side and camunda/camunda#59635 for the client.

In today's Chaos Day, we continued our experiments with Optimize and its behavior with the default configuration, when the underlying Elasticsearch cluster has nodes restarting.

This exercise is the continuation of the recent improvements and clarifications we added to our documentation regarding the configuration, impact and scaling of Camunda clusters running Optimize.

TL;DR;

1. The default Camunda configuration does not create replica shards for the zeebe-record-* indices, making them unavailable during any Elasticsearch node restart.
2. Due to how Zeebe routes records using partition IDs (low-cardinality values hashed with Murmur3), some shards end up completely empty while others hold all the data.
3. When one of the zeebe-record-* indices becomes unavailable, Optimize's single importer thread stalls all other imports — not just the affected resource type — and takes up to 45 minutes to fully recover.

In today's Chaos Day, we wanted to experiment with slow disks, as we have recently run into some incidents related to that. We want to understand and document how Camunda behaves in such scenarios.

We have two main experiments planned: one for primary storage and one for secondary storage (in this case, Elasticsearch) using slow disks.

TL;DR; Using HDDs instead of SSDs on Camunda's primary storage caused around 50% throughput degradation — not because of lower disk throughput, but because of higher latency, which directly stalls Raft replication and commit acknowledgement. Moving the slow disk to Elasticsearch (secondary storage) was even worse, dropping throughput by ~70% and accumulating a permanent export backlog of ~200k records, with memory growing from unexported in-flight data. Both experiments confirm that SSDs are essential for both storage layers, and our documentation for secondary storage needs to be updated to reflect this.

On today's Chaos Day, we wanted to experiment with disks filling up due to soft-paused exporters. We have recently encountered some incidents in which these disks filled up. We wanted to understand how Zeebe behaves in such scenarios. We had the following experiment planned: reproducing full disks because exporters are not confirming positions (due to soft-pausing).

TL;DR: We were able to reproduce the full-disk scenario with soft-pausing exporters. The node becomes unresponsive and rejects requests. After unpausing the exporters, we were able to free up disk space again, but it took a while because the exporters needed to re-export all unacknowledged records after the restart. Some interesting learnings from this experiment are that we should not keep exporters soft-paused for long. This can be especially problematic if nodes get restarted. When the disk is full, backpressure still reports zero, but all requests are rejected. Even REST requests are no longer successful.

On this Chaos Day, we conducted an experiment to observe the impact on Elasticsearch when deploying a large number of process versions to a Camunda cluster, and how that pressure propagates through Optimize, the Zeebe Elasticsearch exporter, and ultimately back to the Camunda engine itself. During recent investigations, we identified a dependency between deployed process models and Elasticsearch shard usage, and wanted to experiment with it to understand what happens when we deploy X process models and where the actual limit lies.

TL;DR; We discovered a 1:1 relationship between Optimize indices and deployed processes, providing a clear, measurable limit on the number of process models that can coexist with Optimize on a given Elasticsearch cluster. Once the Elasticsearch cluster reaches its maximum normal shard limit (default 1000 per node, e.g., 3000 for a 3-node cluster), it stops creating new indices. The Zeebe engine remains unaffected initially, but the failure cascades the next day: once the Zeebe Elasticsearch exporter attempts to create its new dated index, the request is rejected, the exporter stalls, and the Camunda engine hits unrecoverable backpressure. Recovery requires manual intervention (raise cluster.max_shards_per_node, add nodes, or delete indices).

Over the past weeks, we have been spending more time improving our load testing and reliability testing coverage. One of the things we did was to enable REST API (by default, we tend to use gRPC).

While doing such, we were experiencing a weird load pattern. This seems to occur when enabling the REST API usage in our load tester clients, together with OIDC.

On today's Chaos day, we want to verify how the system behaves when using the REST API and OIDC together, and how this changes under different loads and versions. We were also validating whether this was related to the cluster configuration (testing with SaaS).

TL;DR; We were seeing recurring throughput drops, especially at higher load (300 PIs), but at lower load they were not visible. The issue was reproducible in 8.8 as well, so it was not related to the changes in 8.9. We couldn't reproduce the pattern in SaaS, as we weren't able to achieve the same load with the small clusters we used. While experimenting, we discovered several areas for improvement. The root cause turned out to be JWT tokens expiring while requests queued in the Apache HttpAsyncClient connection pool. Nic fixed this by moving token injection to after connection acquisition via #50124 🚀