Kiwi.com's Migration from PostgreSQL to Cassandra to ScyllaDB

Kiwi.com  never stops innovating their product and their architecture. Over the past couple of years, they saw a significant rise in technology requirements both globally and internally and had already tried several database solutions. The transformation went from small applications to complex microservices architectures. They first migrated to Cassandra from a big PostgreSQL cluster to get better performance and scalability, but their demands never stopped growing. That is why they decided to go with ScyllaDB.

In this talk, Martin Strycek, engineering manager at Kiwi.com , covers how their team approached testing of ScyllaDB, the migration plan, how it impacts their business and how it influenced their high-level architecture of the application and infrastructure. 

Martin said that Kiwi.com first migrated to Cassandra from a big PostgreSQL cluster to get better performance and scalability, but their demands never stopped growing.

Martin covered the way his team approached testing of ScyllaDB, the migration plan, how it impacts the business and Kiwi.com’s high-level application and infrastructure architecture. In Martin’s view, ScyllaDB has had a significant impact on disaster recovery and availability of the overall system.

According to Martin, Kiwi.com quickly settled on ScyllaDB as a drop-in replacement for Cassandra, but they wanted to prove it out under real-life conditions before making the leap. With a healthy scepticism for vendor benchmarks, Kiwi.com set out to independently evaluate Cassandra versus ScyllaDB. To do so, the team defined equivalent configurations, traffic volumes, and workloads based on the Cassandra benchmark.

The goal was to test ScyllaDB raw speed and performance, along with ScyllaDB’s support for Kiwi.com’s specific workloads. They also wanted some insight into running on bare metal or on a cloud platform, testing GCP versus OVH, popular cloud provider in Europe. The final goal of the POC was to evaluate ScyllaDB’s cost relative to the Cassandra cluster they were running.

Overall, Martin used three approaches to testing:

• synthetic benchmarks
• shadowing production traffic
• internal benchmarking tool for reads

Kiwi.com worked closely with the ScyllaDB team to establish success criteria for the POC. Once the test bed of five nodes each was set up, Kiwi.com ran a set of synthetic benchmarks, shadowed production traffic, and used internal monitoring tools for reads.

Their tests demonstrated a stark difference between the two databases. With a replication factor of 4, Cassandra required 100 nodes to achieve 40K reads per second. With only 21 nodes, ScyllaDB was able to achieve 900K reads per second.

ScyllaDB vs. Cassandra Table

Best of all, Kiwi.com discovered that the running cost of ScyllaDB would be about 25% the cost of Cassandra. Martin provided a detailed breakdown of the hardware costs of running Cassandra versus ScyllaDB, on bare metal and Google Cloud Platform:

Comparison Table between Cassandra and ScyllaDB on cloud platform

A comparison of Kiwi.com’s hardware costs between Cassandra and ScyllaDB on cloud platforms

Having made the decision to go with ScyllaDB, the team undertook the migration to GCP and OVH instances running in multiple cities and geographical regions. In fact, Martin’s team installed the final server in the ScyllaDB cluster just before the presentation, displaying shadow traffic from the live system.

Martin pointed out that Kiwi.com is also excited about ScyllaDB’s roadmap. The ability to prioritize production traffic over analytics will be a huge advantage, since the many algorithms that Kiwi.com runs against the ScyllaDB clusters will have no discernable impact on the customer experience.

Martin wrapped up his ScyllaDB Summit talk by encouraging the audience to “never stop innovating”, stating, “This is the bottom line. If you are considering going from ScyllaDB to Cassanadra, I don’t know why you didn’t do that last week!”