Why Your Payment Backend Needs a Dedicated Real-Time Layer (And How We Built One)
The thing that processes payments should not be the same thing that delivers notifications. Here's why splitting these into two systems — a payment backend and a dedicated push layer — makes everything better.
You know that moment when you order food on Zomato, you pay, and the delivery partner's phone instantly lights up with "Payment Received"? It feels seamless. Almost magical. But behind that two-second moment, there's a surprisingly complex system making it happen.
I recently built a payment processing prototype and ran into a problem that every team building real-time payment experiences eventually hits: the thing that processes payments should not be the same thing that delivers notifications. Let me walk you through why, and how splitting these into two separate systems made everything better.
The Problem With Doing Everything in One Place
The payment API is an HTTP server — it handles requests, talks to the database, and responds. WebSocket connections, on the other hand, are long-lived. They sit there, open, waiting. Every delivery partner with the app open is holding a persistent connection to your server. Now your payment API is doing two fundamentally different jobs: processing short-lived HTTP requests and babysitting thousands of long-lived socket connections.
The notification part was also naive. When a payment was confirmed, the server would broadcast the event to every connected WebSocket client. Every single one. It didn't care who the payment was for. The client-side app had to figure out "is this event meant for me?" That's wasteful, insecure, and just bad design.
So I decided to separate things.
Enter Propeller: A Dedicated Push Layer
How the Two Systems Work Together
Step 1 — The customer pays. The customer's app sends a request to the payment backend. The payment API creates an order in the database and talks to a payment gateway (Razorpay, Cashfree, or Juspay) to initiate the transaction. A smart router picks the best gateway based on availability and success rates.
Step 2 — The gateway confirms. After the customer completes the UPI payment, the gateway sends a webhook back to the payment backend. Instead of processing it immediately, the backend drops it into a Redis Stream — a durable queue. Webhooks can arrive in bursts and you can't afford to lose any.
Step 3 — The worker picks it up. A background worker reads from the queue. For each webhook, it first checks if the event was already processed (idempotency). Payment gateways sometimes send the same webhook multiple times — processing a payment twice would be catastrophic. If new, the worker updates the database: payment marked
SUCCESS, order status flips toPAID.Step 4 — The worker tells Propeller. Instead of pushing a notification over a basic WebSocket, it makes a single gRPC call to Propeller: "Send this event to
driver_456." One clean API call. That's it.Step 5 — Propeller delivers. Propeller already has the delivery partner's phone connected via a persistent gRPC stream or WebSocket. It looks up the channel for that client (
tenant:zomato:client:driver_456), publishes the event through Redis Pub/Sub, and the message flows to the exact device. Nobody else sees it. The screen shows "Payment Received" within milliseconds.
Why This Split Matters
Scale independently. On a Friday night, you might have 50,000 delivery partners connected (50,000 persistent connections) but only 5,000 payments per minute. With a monolithic approach, you'd scale the entire payment backend just to handle sockets. With the split, you scale Propeller horizontally without touching payment infrastructure.
Targeted delivery. Instead of broadcasting to everyone, Propeller routes events to specific users on specific devices. Each client subscribes to their own channel. Only they receive their events. A driver should never see payment details meant for another driver.
Multi-device awareness. A delivery partner might have two phones. Propeller tracks active devices per user via Redis Hashes, and can target a specific device or deliver to all active devices. The payment backend never has to think about this.
Resilience and fault isolation. If Propeller crashes, payments keep processing — the worker retries the gRPC call. If the payment backend goes down, connected clients stay connected to Propeller and don't notice. The two systems fail independently.
Multi-tenant by design. Channels are namespaced as
tenant:zomato:client:driver_456andtenant:blinkit:client:rider_789. The same Propeller instance serves multiple brands with zero cross-contamination.
The Connection Lifecycle
Authentication and Security
Clients can only listen
Backends can only publish
A compromised client token can never be used to inject fake payment confirmations.
The Reconciliation Safety Net
What This Looks Like from the User's Perspective
Wrapping Up
FAQ
What is a real-time push layer and why does a payment system need one?
A real-time push layer is a dedicated service whose sole responsibility is managing persistent client connections (WebSocket or gRPC streams) and routing events to the correct user and device. Payment backends should not handle this because they have fundamentally different scaling profiles — a payment API handles short-lived HTTP requests while a push layer manages thousands of long-lived connections. Separating them allows each to scale, fail, and evolve independently.
What is Propeller and what company built it?
Propeller is a dedicated push notification and real-time event routing service built internally by CRED, the Indian fintech company. It manages persistent client connections and routes events to specific users on specific devices, without knowing anything about the business logic (payments, orders, etc.) of the calling system.
How does idempotency work in webhook processing?
Payment gateways use at-least-once delivery, meaning the same webhook can arrive multiple times. An idempotency store (typically Redis) records every processed webhook event ID. Before processing any webhook, the worker checks this store. If the event ID already exists, it skips processing. This ensures a payment is recorded exactly once even if the webhook arrives five times.
Why use gRPC between the payment worker and Propeller instead of REST?
gRPC uses HTTP/2 and Protocol Buffers, giving it lower latency and smaller payload sizes compared to REST/JSON. For an internal service-to-service call that happens on every payment confirmation, this matters at scale. gRPC also provides strongly typed contracts between services, making the interface between the payment worker and Propeller explicit and versioned.