12 posts tagged with "performance"

As businesses increasingly rely on process automation for their critical operations, the question of reliability becomes paramount. How can you trust that your automation platform will perform consistently under pressure, recover gracefully from failures, and maintain performance over time?

At Camunda, we've been asking ourselves these same questions for years, and today I want to share how our reliability testing practices have evolved to ensure our platform meets the demanding requirements of enterprise-scale deployments. I will also outline our plans to further invest in this crucial area.

From Humble Beginnings to Comprehensive Testing​

Our reliability testing journey began in early 2019 with what we then called "benchmarks" – simple load tests to validate basic performance of our Zeebe engine.

Over time, we recognized that running such benchmarks alone wasn't enough. We needed to ensure that Zeebe could handle real-world conditions, including failures and long-term operation. This realization led us to significantly expand our testing approach.

We introduced endurance tests that run for weeks, simulating sustained load to uncover memory leaks and performance degradation. These tests helped us validate that Zeebe could maintain its performance characteristics over extended periods of time. Investing in these endurance tests paid off, as we identified and resolved several critical issues that only manifested under prolonged load. Additionally, it allows us to build up experience on what a healthy system looks like and what we need to investigate faulty systems. With this, we were able to create Grafana dashboards that we can directly use to monitor our production systems and provide to our customers.

We embraced chaos engineering principles, developing a suite of chaos experiments to simulate failures in a controlled manner. We created zbchaos, an open-source fault injection tool tailored for Camunda, allowing us to automate and scale our chaos experiments. Automated chaos experiments now run daily against all supported versions of Camunda, covering a wide range of failure scenarios.

Additionally, we run semi-regular manual "chaos days" where we design and execute new chaos experiments, documenting our findings in our chaos engineering blog.

What started as a straightforward performance validation tool has evolved into a comprehensive framework that combines load testing, chaos engineering, and end-to-end testing. This evolution wasn't just about adding more tests. It reflected our growing understanding that reliability isn't a single metric but a multifaceted quality that emerges from systematic validation across different dimensions: performance under load, behavior during failures, and consistency over time.

We combine all of the above under the umbrella of what we now call "reliability testing." We define reliability testing as a type of software testing and practice that validates system performance and reliability. It can thus be done over time and with injection failure scenarios (injecting chaos).

If you are interested in more of the evolution of our reliability testing, I gave several Camunda Con Talks and wrote blog posts over the years that you might find interesting:

• Camunda Con 2020.2: Chaos Engineering Meets Zeebe
• Camunda Con 2024: Drinking our own Champagne: Chaos Experiments with Zeebe against Zeebe
• Drinking Our Champagne: Chaos Experiments with Zeebe against Zeebe
• Zbchaos — A new fault injection tool for Zeebe
• or see all the other Chaos days we have published

Why Reliability Testing Matters​

We prepare customers for enterprise-scale operations. For this, we need to be confident in building a product that is fault-tolerant, reliable, and that performs well even under turbulent conditions.

For our customers running mission-critical processes, reliability testing provides several crucial benefits:

• Proactive Issue Detection: We identify problems before they impact production environments. Memory leaks, performance degradation, and distributed system failures that only manifest under specific conditions are caught early in our testing cycles.
• Confidence in Long-Term Operation: Our endurance tests validate that Camunda can run fault-free over extended periods, ensuring your automated processes won't degrade over time.
• Graceful Failure Handling: Through chaos engineering, we verify that the platform handles failures elegantly, maintaining data consistency and recovering automatically when possible.
• Performance Assurance: Continuous load testing ensures that Camunda meets performance expectations (e.g., number of Process Instances / second), even as new features are added and the codebase evolves.

Our Current Testing Arsenal​

Today, our reliability testing encompasses two main pillars: load tests and chaos engineering.

Variations of Load Tests​

We run different variants of load tests continuously:

• Release Endurance Tests: Every supported version undergoes continuous endurance testing with artificial workloads, updated with each patch release
• Weekly Endurance Tests: Based on our main branch, these tests run for four weeks to detect newly introduced instabilities or performance regressions
• Daily Stress Tests: Shorter tests that validate the latest changes in our main branch under high load conditions

Our workload varies from artificial load (simple process definitions with minimal logic) to typical and realistic, complex processes that mimic real-world usage patterns.

Examples of such processes are:

Chaos Engineering​

Since late 2019, we've embraced chaos engineering principles to build confidence in our system's resilience. Our approach includes:

• Chaos Days: Regular events where we manually design and execute chaos experiments, documenting findings in our chaos engineering blog
• Game Days: Regular events where we simulate an incident in our production SaaS environment to validate our incident response processes
• Automated Chaos Experiments: Daily execution of 16 different chaos scenarios across all supported versions using our zbchaos tool. We drink our own champagne by using Camunda 8 to orchestrate our chaos experiments against Camunda.

Investing in the Future​

With the foundation we’ve established through years of focused reliability testing on the Zeebe engine and its distributed architecture, we’re now expanding that maturity across the entire Camunda product. Our goal is to develop an even more robust and trustworthy product overall. To achieve this, we are consolidating the reliability testing efforts that have historically existed across individual components into a centralized team. This unified approach enables us to scale our testing capabilities more efficiently, ensure consistent best practices, and share insights across teams, ultimately strengthening the reliability of every part of the product.

Some of our upcoming initiatives driven by this team include:

• Holistic Coverage: We're extending our reliability testing to cover all components of the Camunda 8 platform via a central reliability testing framework.
• Chaos Engineering: We're planning to introduce new chaos experiments that simulate more complex failure modes, including network partitions, data corruption, and cascading failures.
• Performance Optimization: Beyond maintaining performance, we utilize our testing infrastructure to identify optimization opportunities and validate improvements.
• Enhanced Observability: Building on our extensive Grafana dashboards, we continually improve our ability to detect and diagnose issues quickly.
• Establish Reliability Practices: We're formalizing reliability testing practices and guidelines that can be adopted across all engineering teams at Camunda.
• Enablement: With the resources we want to enable all of our more than 15 product teams at Camunda to understand, implement, and execute reliability testing principles in their work. Allowing them to build more reliable software from the start and scaling our efforts.

Building Trust Through Transparency​

Our commitment to reliability testing isn't just about internal quality assurance – it's about building trust with our customers and the broader community. That's why we:

• Publish our testing methodologies and results openly
• Share our learnings through blog posts and conference talks
• Provide tools like zbchaos as open source for the community

Conclusion​

Reliability testing at Camunda has evolved from simple benchmarks to a comprehensive practice that combines load testing, chaos engineering, and end-to-end validation. This evolution reflects our understanding that true reliability emerges from systematic testing across multiple dimensions.

For our customers, this means confidence that Camunda will perform reliably under their most demanding workloads. For engineers interested in joining our team, it represents an opportunity to work with cutting-edge testing practices at scale.

As we continue to invest in reliability testing, we remain committed to transparency and sharing our learnings with the community. After all, the reliability of process automation platforms isn't just a technical challenge – it's fundamental to the digital transformation of businesses worldwide.

---

Interested in learning more about our reliability testing practices? Check out our detailed documentation, explore our chaos engineering experiments, or reach out to discuss how Camunda's reliability testing ensures your critical processes run smoothly.

In today's chaos experiment, we focused on stress-testing the Camunda 8 orchestration cluster under high-load conditions. We simulated a large number of concurrent process instances to evaluate the performance of processing and system reliability.

Due to our recent work in supporting load tests for different versions, we were able to compare how different Camunda versions handle stress.

TL;DR: Overall, we saw that all versions of the Camunda 8 orchestration cluster (with focus on the processing) are robust and can handle high loads effectively and reliably. In consideration of throughput and latency, with similar resource allocation among the brokers, 8.7.x outperforms other versions. If we consider our streamlined architecture (which now contains more components in a single application) and align the resources for 8.8.x, it can achieve similar throughput levels as 8.7.x, while maintaining significantly lower latency (a factor of 2). An overview of the results can be found in the Results section below.

Hypothesis​

The objective of this chaos day is to estimate the scalability of Zeebe when brokers and partitions are scaled together: we expect to be able to see the system scaling linearly with the number of brokers/partition in terms of throughput and back pressure, while maintaining predictable latency.

General Experiment setup​

To test this, we ran a benchmark using the latest alpha version of Camunda 8.8.0-alpha6, with the old ElasticsearchExporter disabled, and the new CamundaExporter enabled. We also made sure Raft leadership was balanced before starting the test, meaning each broker is leader for exactly one partition, and we turned on partition scaling by adding the following environment variable:

• ZEEBE_BROKER_EXPERIMENTAL_FEATURES_ENABLEPARTITIONSCALING=true

Each broker also has a SSD-class volume with 32GB of disk space, limiting them to a few thousand IOPS. The processing load was 150 processes per second, with a large payload of 32KiB each. Each process instance has a single service task:

The processing load is generated by our own benchmarking application.

Initial cluster configuration​

To test this hypothesis, we will start with a standard configuration of the Camunda orchestration cluster:

• 3 nodes
• 3 partitions
• CPU limit: 2
• Memory limit: 2 GB

We will increase the load through a load generator in fixed increments until we start to see the nodes showing constant non zero backpressure, which is a sign that the system has hit its throughput limits.

Target cluster configuration​

Once that level of throughput is increased, we will scale broker & partitions while the cluster is under load to the new target value:

• 6 nodes
• 6 partitions
• CPU limit: 2
• Memory limit: 2 GB

Experiment​

We expect that during the scaling operation the backpressure/latencies might worsen, but only temporarily, as once the scaling operation has completed, the additional load it generate is not present anymore.

Then, we will execute the same procedure as above, until we hit 2x the critical throughput hit before.

Expectation​

If the system scales linearly, we expect to see similar level of performance metrics for similar values of the ratios PI (created/complete) per second / nr. of partition.

Steady state​

The system is started with a throughput of 150 Process instances created per second. As this is a standard benchmark configuration, nothing unexpected happens:

• The same number of process instances are completed as the ones created
• The expected number of jobs is completed per unit of time

At this point, we have the following topology:

First benchmark: 3 broker and 3 partitions​

Let's start increasing the load incrementally, by adding 30 Process instances/s for every step.

Time	Brokers	Partitions	Throughput	CPU Usage	Throttling (CPU)	Backpressure
09:30	3	3	150 PI/s, 150 jobs/s	1.28 / 1.44 / 1.02	12% / 7% / 1%	0
09:49	3	3	180 PI/s, 180 jobs/s	1.34 / 1.54 / 1.12	20% / 17% / 2%	0
10:00	3	3	210 PI/s, 210 jobs/s	1.79 / 1.62 / 1.33	28% / 42% / 4%	0
10:12	3	3	240 PI/s, 240 jobs/s	1.77 / 1.95 / 1.62	45% / 90% / 26%	0/0.5%

At 240 Process Instances spawned per second, the system starts to hit the limits:

And the backpressure is not zero anymore:

• The CPU throttling reaches almost 90% on one node (this is probably caused by only one node being selected as gateway as previously noted)
• Backpressure is now constantly above zero, even if it's just 0.5%, it's a sign that we are reaching the throughput limits.

Second part of the benchmark: scaling to 6 brokers and 6 partitions​

With 240 process instances per second being spawned, we send the commands to scale the cluster.

We first scale the zeebe statefulset to 6 brokers. As soon as the new brokers are running, even before they are healthy, we can send the command to include them in the cluster and to increase the number of partition to 6.

This can be done following the guide in the official docs.

Once the scaling has been completed, as can be seen from the Cluster operation section in the dashboard, we see the newly created partitions participate in the workload.

We now have the following topology:

As we did before, let's start increasing the load incrementally as we did with the other cluster configuration.

Time	Brokers	Partitions	Throughput	CPU Usage	Throttling (CPU)	Backpressure	Notes
10:27	6	6	240 PI/s	0.92/1.26/0.74/0.94/0.93/0.93	2.8/6.0/0.3/2.8/3.4/3.18	0	After scale up
11:05	6	6	300 PI/s	1.17/1.56/1.06/1.23/1.19/1.18	9%/29%/0.6%/9%/11%/10%	0	Stable
11:10	6	6	360 PI/s	1.39/1.76/1.26/1.43/1.37/1.42	19%/42%/2%/16%/21%/22%	0	Stable
11:10	6	6	420 PI/s	1.76/1.89/1.50/1.72/1.50/1.70	76%/84%/52%/71%/60%/65%	0 (spurts on 1 partition)	Pushing hard

However, at 11:32 one of the workers restarted, causing a spike in the processing due to jobs being yielded back to the engine, less jobs to be activated, and thus less to be completed. This caused a job backlog to build up in the engine. Once the worker restarted, the backlog was drained, leading to a spike in job completion requests: around 820 req/s, as opposed to the expected 420 req/s.

Because of this extra load, the cluster started to consume even more CPU, resulting in heavy CPU throttling from the cloud provider.

On top of this, eventually a broker restarted (most likely as we run on spot VMs). In order to continue with our test, we scaled the load down to 60 PI/s to give the cluster the time to heal.

Once the cluster was healthy again, we raised the throughput back to 480 PI/s to verify the scalability with twice as much throughput as the initial configuration.

The cluster was able to sustain 480 process instances per second with similar levels of backpressure of the initial configuration:

We can see below that CPU usage is high, and there is still some throttling, indicating we might be able to do more with a little bit of vertical scaling, or by scaling out and reducing the number of partitions per broker:

Conclusion​

We were able to verify that the cluster can scale almost linearly with new brokers and partitions, so long as the other components, like the secondary storage, workers, connectors, etc., are able to sustain a similar.

In particular, making sure that the secondary storage is able to keep up with the throughput turned out to be crucial to keep the cluster stable in order to avoid filling up the Zeebe disks, which would bring to a halt the cluster.

We encountered a similar issue when one worker restarts: initially it creates a backlog of unhandled jobs, which turns into a massive increase in requests per second when the worker comes back, as it starts activating jobs faster than the cluster can complete them.

Finally, with this specific test, it would be interesting to explore the limits of vertical scalability, as we often saw CPU throttling being a major blocker for processing. This would make for an interesting future experiment.

In our last Chaos day we experimented with the Camunda Exporter MVP. After our MVP we continued with Iteration 2, where we migrated the Archiver deployments and added a new Migration component (allows us to harmonize indices).

Additionally, some fixes and improvements have been done to the realistic benchmarks that should allow us to better compare the general performance with a realistic good performing benchmark.

Actually, this is what we want to explore and experiment with today.

• Does the Camunda Exporter (since the last benchmark) impact performance of the overall system?
  • If so how?
• How can we potentially mitigate this?

TL;DR; Today's, results showed that enabling the Camunda Exporter causes a 25% processing throughput drop. We identified the CPU as a bottleneck. It seems to be mitigated by either adjusting the CPU requests or removing the ES exporter. With these results, we are equipped to make further investigations and decisions.

After a long pause, I come back with an interesting topic to share and experiment with. Right now we are re-architecture Camunda 8. One important part (which I'm contributing to) is to get rid of Webapps Importer/Archivers and move data aggregation closer to the engine (inside a Zeebe Exporter).

Today, I want to experiment with the first increment/iteration of our so-called MVP. The MVP targets green field installations where you simply deploy Camunda (with a new Camunda Exporter enabled) without Importers.

TL;DR; All our experiments were successful. The MVP is a success, and we are looking forward to further improvements and additions. Next stop Iteration 2: Adding Archiving historic data and preparing for data migration (and polishing MVP).

Camunda Exporter​

The Camunda Exporter project deserves a complete own blog post, here is just a short summary.

Our current Camunda architecture looks something like this (simplified).

It has certain challenges, like:

• Space: duplication of data in ES
• Maintenance: duplication of importer and archiver logic
• Performance: Round trip (delay) of data visible to the user
• Complexity: installation and operational complexity (we need separate pods to deploy)
• Scalability: The Importer is not scalable in the same way as Zeebe or brokers (and workload) are.

These challenges we obviously wanted to overcome and the plan (as mentioned earlier) is to get rid of the need of separate importers and archivers (and in general to have separate application; but this is a different topic).

The plan for this project looks something like this:

We plan to:

1. Harmonize the existing indices stored in Elasticsearch/Opensearch
   • Space: Reduce the unnecessary data duplication
2. Move importer and archiver logic into a new Camunda exporter
   • Performance: This should allow us to reduce one additional hop (as we don't need to use ES/OS as a queue)
   • Maintenance: Indices and business logic is maintained in one place
   • Scalability: With this approach, we can scale with partitions, as Camunda Exporters are executed for each partition separately (soon partition scaling will be introduced)
   • Complexity: The Camunda Exporter will be built-in and shipped with Zeebe/Camunda 8. No additional pod/application is needed.

Note: Optimize is right now out of scope (due to time), but will later be part of this as well.

MVP​

After we know what we want to achieve what is the Minimum viable product (MVP)?

We have divided the Camunda Exporter in 3-4 iterations. You can see and read more about this here.

The first iteration contains the MVP (the first breakthrough). Providing the Camunda Exporter with the basic functionality ported from the Operate and Tasklist importers, writing into harmonized indices.

The MVP is targeting green field installations (clean installations) of Camunda 8 with Camunda Exporter without running the old Importer (no data migration yet),

Our first goal of this experiment is to use a benchmarks to derive new optimized cluster configuration that can handle at least 100 tasks per second, while maintaining low backpressure and low latency.

For our experiment, we use a newly defined realistic benchmark (with a more complex process model). More about this in a separate blog post.

The second goal is to scale out optimized cluster configuration resources linearly and see if the performance scales accordingly.

TL;DR;

We used a realistic benchmark to derive a new cluster configuration based on previous requirements.

When we scale this base configuration linearly we see that the performance increases almost linearly as well, while maintaining low backpressure and low latency.

Chaos Experiment​

Expected​

We expect that we can find a cluster configuration that can handle at 100 tasks second to be significantly reduced in resources in relation to our smaller clusters (G3-S HA Plan) since these can process significantly above our initial target.

We also expect that we can scale this base configuration linearly, and that the processing tasks rate to grow initially a bit faster than linearly due to the lower relative overhead, and if we keep expanding further to flatten due to the partition count being a bottleneck.

Actual​

Minimal Requirements for our Cluster​

Based on known customer usage, and our own previous experiments, we determined that the new cluster would need to create and complete a baseline of 100 tasks per second, or about 8.6 million tasks per day.

Other metrics that we want to preserve and keep track are the backpressure to preserve user experience, guarantee that exporting speed can keep up with the processing speed, write-to-import latency which tells us how long it takes for a record to be written to being imported by our other apps such as the operator.

Reverse Engineering the Cluster Configuration​

For our new configurations the only resources that we are going to change are the ones relevant to the factors described above. These are the resources allocated to our zeebe-brokers, gateway and elasticSearch.

Our starting point in resources was the configuration for our G3-S HA Plan as this already had the capability to significantly outperform the current goal of 100 tasks per second.

The next step was to deploy our realistic benchmark, with a payload of 5 customer disputes per instance and start 7 instances per second, this generated approximately 120 tasks per second (some buffer was added to guarantee performance).

After this we reduced the resources iteratively until we saw any increase in backpressure, given that no there was no backlog of records, and no significant increase in the write to import latency.

The results for our new cluster are specified bellow in the tables, where our starting cluster configuration is the G3-S HA Plan and the new configuration cluster is the G3 - BasePackage HA.

G3-S HA	CPU Limit	Memory Limit in GB
operate	2	2
operate.elasticsearch	6	6
optimize	2	2
tasklist	2	2
zeebe.broker	2.88	12
zeebe.gateway	0.9	0.8
TOTAL	15.78	24.8

G3 - BasePackage HA	CPU Limit	Memory Limit in GB
operate	1	1
operate.elasticsearch	3	4.5
optimize	1	1.6
tasklist	1	1
zeebe.broker	1.5	4.5
zeebe.gateway	0.6	1
TOTAL	8.1	13.6

Reduction in Resources for our Optimized Cluster​

	CPU Reduction (%)	Memory Reduction (%)
zeebe.broker	47.92	62.5
zeebe.gateway	33.33	-25.0
operate.elasticsearch	50.00	25.0

Total cluster reduction:

	G3-S HA	G3 - BasePackage HA	Reduction (%)
CPU Limits	15.78	8.1	49
Memory Limits	24.8	13.6	45

The process of reducing the hardware requirements was donne initially by scaling down the resources of the zeebe-broker, gateway and elasticSearch. The other components were left untouched, as they had no impact in our key metrics, and were scaled down later in separate experiences to maintain user experience.

Scaling out the Cluster​

Now for the scaling procedure we intend to see if we can linearly increase the allocated resources and having a corresponding performance increase, while keeping the backpressure low, low latency, and user experience.

For this we started with the G3 - BasePackage HA configuration and incremented the load again until we saw any increase in backpressure, capture our key metrics and repeated the process for the cluster configuration resources respectively multiplied by 2x, 3x, and 4x.

This means that the resources allocated for our clusters were:

	Base 1x	Base 2x	Base 3x	Base 4x
CPU Limits	8.7	17.4	26.1	34.8
Memory Limits	14.9	29.8	44.7	59.6

The results in the table bellow show the performance of our several cluster configurations:

	Base 1x	Base 2x	Base 3x	Base 4x
Process Instances/s	7	12	23	27
Tasks/s	125	217	414	486
Average Backpressure	2%	2%	3%	6%
Write-to-Import Latency	90s	120s	150s	390s
Write-to-Process Latency	140ms	89ms	200ms	160ms
Records Processed Rate	2500	4700	7800	11400
Records Exported Rate	2100	3900	6500	9200

This first observations is that the performance scales particularly well by just adding more resources to the cluster, particularly for a linear increase of the resources the performance as measured by tasks completed increases slightly less than linearly (comparing the 1x and 4x task/s we get 388% the initial rate).

This a very good result as it means that we can scale our system linearly (at least initially) to handle the expected increase in loads.

Importantly, the backpressure is kept low, and the write-to-import latency only increases significantly if we leave the cluster running at max rate for long periods of time. For slightly lower rates the write-to-import latency is kept in the single digits of seconds or lower tens. This might imply that a these sustained max rates, the amount records generated starts to be too much for either ElasticSearch or our web apps that import these records to handle. Some further investigation could be done here to investigate the bottleneck.

Another metric also relevant but not shown in this table is the backlog of records not exported, which kept at almost null through all the experiments conducted.

Bugs found​

During the initial tests, we had several OOM errors in the gateways pods. After some investigation, we found that this was exclusive to the Camunda 8. 6.0 version, which consumes more memory in the gateway than the previous versions. This explains why the gateway memory limits were the only resource that was increased in the new reduced cluster configuration.

In our last Chaos Day we experimented with Operate and different load (Zeebe throughput). We observed that a higher load caused a lower import latency in Operate. The conclusion was that it might be related to Zeebe's exporting configuration, which is affected by a higher load.

In today's chaos day we want to verify how different export and import configurations can affect the importing latency.

TL;DR; We were able to decrease the import latency by ~35% (from 5.7 to 3.7 seconds), by simply reducing the bulk.delay configuration. This worked on low load and even higher load, without significant issues.

🎉 Happy to announce that we are broadening the scope of our Chaos days, to look holistically at the whole Camunda Platform, starting today. In the past Chaos days we often had a close look (or concentrated mostly) at Zeebe performance and stability.

Today, we will look at the Operate import performance and how Zeebe processing throughput might affect (or not?) the throughput and latency of the Operate import. Is it decoupled as we thought?

The import time is an important metric, representing the time until data from Zeebe processing is visible to the User (excluding Elasticsearch's indexing). It is measured from when the record is written to the log, by the Zeebe processor, until Operate reads/imports it from Elasticsearch and converts it into its data model. We got much feedback (and experienced this on our own) that Operate is often lagging behind or is too slow, and of course we want to tackle and investigate this further.

The results from this Chaos day and related benchmarks should allow us to better understand how the current importing of Operate performs, and what its affects. Likely it will be a series of posts to investigate this further. In general, the data will give us some guidance and comparable numbers for the future to improve the importing time. See also related GitHub issue #16912 which targets to improve such.

TL;DR; We were not able to show that Zeebe throughput doesn't affect Operate importing time. We have seen that Operate can be positively affected by the throughput of Zeebe. Surprisingly, Operate was faster to import if Zeebe produced more data (with a higher throughput). One explanation of this might be that Operate was then less idle.

With Zeebe now supporting the addition and removal of brokers to a running cluster, we wanted to test three things:

1. Is there an impact on processing performance while scaling?
2. Is scaling resilient to high processing load?
3. Can scaling up improve processing performance?

TL;DR; Scaling up works even under high load and has low impact on processing performance. After scaling is complete, processing performance improves in both throughput and latency.

We experimented with the first version of dynamic scaling in Zeebe, adding or removing brokers for a running cluster.

Scaling up and down is a high-level operation that consists of many steps that need to be carried co-operatively by all brokers in the cluster. For example, adding new brokers first adds them to the replication group of the assigned partitions and then removes some of the older brokers from the replication group. Additionally, priorities need to be reconfigured to ensure that the cluster approaches balanced leadership eventually.

This orchestration over multiple steps ensures that all partitions are replicated by at least as many brokers as configured with the replicationFactor. As always, when it comes to orchestrating distributed systems, there are many edge cases and failure modes to consider.

The goal of this experiment was to verify that the operation is resilient to broker restarts. We can accept that operations take longer than usual to complete, but we need to make sure that the operation eventually succeeds with the expected cluster topology as result.

TL;DR; Both scaling up and down is resilient to broker restarts, with the only effect that the operation takes longer than usual to complete.