How Fanatics Eliminated Cassandra Timeouts
“We not only reduced TCO, but also reduced the pain that the database engineering team was taking to actually maintain the cluster in a healthy state.” - Niraj Konathi, Director of Platform Engineering at Fanatics
About Fanatics
Fanatics is the leading online retailer of officially licensed sports merchandise. They power the ecommerce sites of all major professional sports leagues (NFL, MLB, NBA, NHL, NASCAR, PGA), major media brands (NBC Sports, CBS Sports, FOX Sports) and 200+ collegiate and professional team properties.
Fanatics’ Database Use Case
A range of functions associated with order capture, including orders that are created, carts that are modified, loyalty, promotional codes, order visibility, and rate limiting.
Fanatics’ Cassandra Challenge: Timeouts Leading to Abandoned Carts
When they moved their on-prem infrastructure to the cloud to support anticipated growth several years ago, Fanatics migrated from MySQL to Cassandra. But Cassandra led to node sprawl requiring a large cluster size, frequent garbage collection pauses, and CPU spikes during compactions — all of which led to timeouts. These issues were so severe that they were driving users to abandon shopping carts and exit their site. Attempting to avoid these issues, they continued scaling out Cassandra – ultimately creating a giant 55-node cluster. Running such a large cluster size increased EC2 costs, but more importantly, increased maintenance overhead. For example, rolling restarts took four or five hours on Cassandra.
Reflecting upon their Cassandra experiences, Niraj Konathi, Director of Platform Engineering at Fanatics, ruefully admitted, “We did not realize how bad it could get.”
Ultimately, these challenges led Fanatics to consider an alternate solution: ScyllaDB.
ScyllaDB is the #1 Apache Cassandra alternative. ScyllaDB provides the same CQL interface and queries, the same drivers, even the same on-disk SSTable format – but with a modern architecture designed to eliminate Cassandra performance issues, limitations, and operational barriers. ScyllaDB is built from the ground up in C++. No Java overhead. No garbage collection. And performance tuning? It’s automated.
Migrating from Cassandra to ScyllaDB
The Fanatics team selected and deployed ScyllaDB for their ordering use case. Their automated provisioning follows regular procedures, such as securing deployments with node-to-node and client-to-node encryption. Their data at rest is also encrypted. With ScyllaDB, their rolling restarts across the cluster now take less than an hour (compared to 5-8 hours with Cassandra). They have ScyllaDB Monitor connected to PagerDuty, and use ScyllaDB Manager for data repairs.
Fanatics’ Cassandra Migration Results
With ScyllaDB, Fanatics has reduced node count from 55 to 6 nodes, resulting in a drastic reduction in EC2 expenses, all while virtually eliminating the timeouts experienced by end users. During a recent peak minute, they saw nearly 280,000 IOPs for a solid minute –
and registered zero timeouts. This translates to happier customers as well as more productive application teams.