Kafka or Sidekiq? Choosing the Right Async Tool for Your Ruby on Rails Backend

Every few months a client brings us a proposal to add Kafka to a Rails application that has maybe three background jobs. The reasoning is usually the same: "we might need to scale event processing later." Later rarely comes, and in the meantime the team now runs and monitors a distributed log they didn't need.

What Sidekiq actually gives you

Sidekiq solves a narrower problem than people think it does, and that's its strength. It takes a Ruby object, serializes its arguments, pushes them onto a Redis list, and guarantees a worker will eventually pick it up and run it, retrying with backoff if it fails. One dependency (Redis), one mental model (a job, at-least-once), and a UI you can hand to a support engineer on day one.

class ChargeInvoiceJob
  include Sidekiq::Job
  sidekiq_options retry: 10, queue: :billing

  def perform(invoice_id)
    invoice = Invoice.find(invoice_id)
    return if invoice.paid?

    PaymentGateway.charge(invoice)
  end
end

That's the whole contract. No topic to provision, no consumer group to rebalance, no schema registry to maintain.

What changes once Kafka is the right call

Kafka earns its complexity when a single business event needs to reach more than one system that don't know about each other, and each of them needs to process it independently, at its own pace, with the ability to replay history if a consumer was down or a bug shipped. Billing, fraud detection and a data warehouse all reacting to the same "PaymentCaptured" event is a legitimate Kafka use case. A single Rails app charging a card and sending a receipt is not.

• Ordering guarantees are per partition, not global, which changes how you key messages.
• Consumers own their offsets, so a slow consumer never blocks a fast one, but you now own rebalancing and lag monitoring.
• Replay is a feature, and also a liability: consumers must stay idempotent, forever.

The hybrid we ship most often

On most Rails codebases we work on, the answer isn't "Sidekiq or Kafka." It's Sidekiq for everything internal to the app, and a thin outbox that publishes only the handful of events other teams genuinely need onto a Kafka topic. That keeps the operational surface small while still giving the rest of the organization a reliable feed. We cover how to build that outbox safely, without losing events, in a separate article.

If you're unsure which side of that line your project sits on, the test is simple: count how many independent systems need the same event, today, not hypothetically. If the answer is one, you don't need a broker yet.