Using ScyllaDB for Order Capture at Fanatics

An online sports apparel powerhouse, Fanatics has to power an always-on scalable ecommerce site listing wares for over 1,000 vendors, and to track the shopping carts of millions of demanding sports fans. Find out how Fanatics uses ScyllaDB for order capture, how they handle abandonments, perform data mining and more. Learn their “practical engineering” lessons, and discover how Fanatics wrote their own management tool for cluster automation and autoscaling on AWS.

In the above video, Niraj Konathi, Fanatics’ Director of Platform Engineering,  presented how Fanatics uses ScyllaDB for order capture, to handle abandonments and perform data mining. Niraj also shared their “practical engineering” lessons and how Fanatics wrote their own management tool for cluster automation and autoscaling on AWS.

Fanatics sees 57% of its overall platform revenue from mobile. This number rises to almost 80% during hundreds of “hot market” events that occur each year.

The Move to the Cloud, and from MySQL to Cassandra

Fanatics’ journey to the cloud began several years ago when, to meet their scalability demands, they made the move from their on-premises infrastructure. At the same time they also made the move from MySQL to a new stack powered by a highly scalable NoSQL database — Apache Cassandra.

Cassandra served as the core of the primary store: order capture, order management, shopping carts and order mutations (order changes), accounts and users, loyalty and promo programs, order visibility and even rate limiting.

While this strategy worked for Fanatics, it was not without the typical Cassandra pain points: node sprawl requiring a large cluster size, frequent garbage collection (GC) pauses, CPU spikes during compactions — all of which led to timeouts. Niraj ruefully admitted, “We did not realize how bad it could get.”

For example, with cart mutations (changes to a shopping cart), Niraj painted the picture: “So large wide rows. Think about it. Every time a user comes online they change their carts. They add new items to that cart. And every time we actually stored the cart mutation because of analytics as well as for debugging.” They wanted to know why the user abandoned their cart and left the Fanatics.com site.

Unfortunately the underlying Java Virtual Machine (JVM) was causing frequent GC pauses. Which forced Fanatics to continue to expand multiple times; overprovisioning to try to surmount the problem. Their increased cluster size came with commensurate AWS EC2 costs, greater TCO and long periods of time spent maintaining the cluster.

Migrating from Cassandra to ScyllaDB in Production

The remedy to these pain points was ScyllaDB. “Just moving one use case [cart mutations] to ScyllaDB this year we got a huge benefit out of it.” Out of total cluster size of 55 Cassandra nodes, Fanatics were able to reduce 43 nodes of Cassandra to 6 nodes of ScyllaDB, dramatically reducing their EC2 bill. The workload on the remaining 12 nodes of Cassandra will be moved onto the six existing ScyllaDB nodes once Change Data Capture (CDC) reaches General Availability. (Note: CDC was released as an experimental feature in ScyllaDB Open Source 3.2.). This node reduction dramatically reduced their EC2 bill.

Performance was smooth and error-free even in their bursty traffic environment. “We had a hot market last week. During the peak minute we saw close to 180,000 IOPS… and we had zero timeouts.”

This reliable performance made both customers and the Fanatics application teams happier.