Skip to content
All posts
paymentspostgresqlarchitecturebackend

Handling money correctly: idempotency, append-only ledgers, and concurrency

June 2, 2026 4 min read· Mahmoud Makhlouf

Most bugs announce themselves. Money bugs don't. A double-charged card, a loyalty balance credited twice, a commission that settles before the goods ship — these pass every happy-path test and only surface weeks later in a furious support ticket or a reconciliation that doesn't add up.

Across the marketplaces and platforms I've shipped, "handle money correctly" comes down to three disciplines: make every money operation idempotent, store money as an append-only ledger, and serialize the operations that can race. None of them are exotic. All of them are skipped under deadline pressure, and all of them are expensive to retrofit.

Here's how I apply each, with concrete examples from production.

1. Idempotency: the network will retry, so assume it

Payment gateways retry webhooks. Mobile clients retry requests on flaky connections. Users double-tap. If "credit this order" or "charge this card" runs twice, you have a financial incident — unless the second run is a no-op.

On Stravo, a restaurant ordering platform with a points-based loyalty system, points are earned when an order is delivered. An order's status can flip more than once (a delivery gets marked, un-marked, re-marked), and each flip could naively re-credit points. The fix isn't application logic that "checks if we already credited" — that check itself races. The fix is a database-level uniqueness guarantee:

-- one earn event per order, enforced by the database, not the app
create unique index points_earn_once
  on points_ledger (customer_id, order_id, entry_type)
  where entry_type = 'EARN';

Now the second credit attempt throws a unique-violation that the application swallows as "already done." The guarantee lives where it can't be bypassed by a race or a buggy retry.

The same principle protects card payments. On Montra, an Egyptian B2B marketplace using Paymob, every incoming payment webhook is verified with an HMAC SHA-512 signature and carries an idempotency key. A replayed webhook — whether from Paymob's own retries or a malicious replay — is validated, recognized as already-processed, and ignored. The charge is applied exactly once inside a database transaction, never partially.

The rule I follow: any endpoint that moves money must be safe to call twice. If it isn't, it's a question of when it breaks, not if.

2. The ledger is append-only — never a mutable balance

The most common money-modelling mistake I see is a balance column that gets updated in place. It's simple, it's fast, and it destroys your ability to ever answer "why is this number what it is?"

Instead, I model money as an append-only ledger: a balance is the sum of its entries, never a stored figure you overwrite.

On Stravo, the loyalty system is exactly this — every earn, redeem, and refund is an immutable row. The customer's point balance is a SUM(delta) over the ledger. There is no balance to corrupt, and every point a customer holds is traceable to the order that produced it.

On Montra, the financial ledger goes a step further because it handles real money across customers, traders, and platform commissions. Every entry records:

FieldPurpose
balanceBefore / balanceAfterThe running balance at that entry, for fast reads and verification
checksum (SHA-256)Links each entry to the previous one — a hash chain
transactionRefA forensic handle for tracing any movement
entryTypePAYMENT_RECEIVED, DEPOSIT_PAID, COMMISSION_DEDUCTED, TRADER_PAYOUT, REFUND

Because each entry's checksum incorporates the previous entry, the ledger is tamper-evident: you cannot quietly edit an old row without breaking the chain from that point forward. Integrity is verifiable end-to-end, not assumed.

This append-only discipline also made Montra's commission system honest. Commission is accrued only on delivery confirmation — a formal DeliveryNote document (VENDOR_CLICK, CLIENT_ACK, or ADMIN_OVERRIDE) is what triggers the ledger entry. Money movements stay aligned with physical reality instead of optimistic order status.

3. Concurrency: serialize the operations that can race

Idempotency stops the same operation from running twice. It does nothing about two different operations racing over the same balance — for example, a customer placing two orders simultaneously, each trying to redeem the same points.

The naive read-modify-write ("read balance, check it's enough, subtract") is a textbook race: both transactions read the old balance, both pass the check, both redeem. The customer spends points they don't have.

On Stravo, redemption is server-authoritative and serialized per customer with a Postgres advisory lock:

-- inside the redemption transaction
select pg_advisory_xact_lock(hashtext('points:' || customer_id));
-- now this customer's redemptions are serialized;
-- re-read balance, validate, append the redeem entry

The lock is scoped to the customer and released automatically at transaction end (xact variant), so it never leaks. Two concurrent orders from the same customer queue behind each other; two orders from different customers don't block at all. And critically, pricing and balances are re-fetched server-side at commit time via Postgres RPCs — the client never controls the totals it submits.

There's a refund corner case most systems forget: what happens to redeemed points when an order is cancelled? On Stravo, a database trigger automatically refunds redeemed points on cancellation — as another ledger entry, of course, never a balance edit.

Why this matters to a client, not just an engineer

This isn't engineering for its own sake. A marketplace that double-charges loses customer trust permanently. A loyalty system that can be farmed leaks margin every day. A commission ledger that can't be audited turns every trader dispute into your word against theirs.

The patterns are cheap when designed in and brutal to retrofit, because retrofitting means re-modelling live financial data. Getting them right at the start is the difference between hoping the numbers are correct and being able to prove it.

If you're building something where money or live state has to be correct under load, that correctness is most of the work — and it's the part I care most about getting right.

// Let's build

Have a product in mind? Get a clear plan and a free 30-minute consultation.

Tell me what you're building. I'll come back with an approach, a rough timeline, and a ballpark — usually within 24 hours.

WhatsApp
WhatsApp