Is It Faster to Go with Redpanda Transactions than Without Them?!

We all know that distributed transactions are expensive, have higher latency and lower throughput compared to a non-transactional workload. It's just common sense that when we ask a system to maintain transactional guarantees it should spend more time on coordination and thus have poorer performance, right? Well, it's true that we can't get rid of this overhead. But at the same time, each transaction defines a unit of work, so the system stops dealing with individual requests and becomes more aware of the whole workload. Basically, it gets more information and may use it for new kinds of optimizations that compensate for the overhead. In this talk, I describe how Redpanda optimized the Kafka API and pushed throughput of distributed transactions up to eight times beyond an equivalent non-transactional workload while preserving sane latency.