Distributed State Controls — Redis Redlocks

BACKEND SERIES

Day 28: Distributed State Controls — Redis Redlocks & Split-Brain Prevention

12 min read

Series: Logic & Legacy

Day 28 / 50

Level: Senior / Distributed Systems

⏳ Context: Yesterday, we implemented in-memory synchronization using asyncio.Lock and asyncio.Semaphore to protect single-process event loops. But when your backend scales out across 20 Kubernetes pods behind a load balancer, single-node locks offer zero protection. When Pod A and Pod B independently attempt to process the exact same payment or mutate the same database record simultaneously, your database becomes a battleground. To enforce multi-node mutual exclusion, we must step up to Distributed Locks using Redis and the Redlock algorithm.

1. The Naive Redis Lock Trap

The simplest way developers attempt distributed locking is via SET key value NX PX 30000. While this works in ideal network conditions, it introduces catastrophic failure modes when network pauses or GC freezes occur.

The Pitfall: The Unchecked TTL Expiry

If Client A acquires a 10-second lock, but a Stop-The-World Garbage Collection freeze or heavy I/O causes Client A to pause for 11 seconds, Redis automatically releases the lock. Client B acquires the lock. Client A wakes up, unaware it lost the lock, and writes corrupted state to your storage layer.

The Mental Model: The Hotel Keycard with Timer

Imagine a hotel keycard that automatically deactivates after 10 minutes. If you get delayed in the hallway and return at minute 11, the hotel maid has already been assigned your room and walked in. To prevent collisions, you need both a heartbeat lock renewer (keeping your key active while inside) and a strict Fencing Token at the door check.

Flawed Redis Lock (Lua Script Needed for Deletion)

import redis
import uuid
import time

r = redis.Redis(host="localhost", port=6379, db=0)

def unsafe_release_lock(lock_name, lock_token):
    # DANGER: Simply calling r.delete(lock_name) can delete someone else's lock!
    # Always use Atomic Lua scripts to verify ownership before deletion.
    lua_script = """
    if redis.call('get', KEYS[1]) == ARGV[1] then
        return redis.call('del', KEYS[1])
    else
        return 0
    end
    """
    return r.eval(lua_script, 1, lock_name, lock_token)

2. The Redlock Algorithm & Quorum Consensus

To avoid single-point-of-failure in single-instance Redis setups, Martin Kleppmann and Salvatore Sanfilippo (antirez) designed the Redlock algorithm. By deploying 5 independent Redis master nodes, a client acquires the lock if and only if it obtains the lock on a majority (at least 3 of 5) nodes within a strict timeout window.

Production Distributed Redlock in Python (aioredlock / redis-py)

import asyncio
from redis.asyncio import Redis

class DistributedLock:
    def __init__(self, redis_client: Redis, key: str, ttl_ms: int = 10000):
        self.redis = redis_client
        self.key = f"lock:{key}"
        self.ttl = ttl_ms
        self.token = None

    async def __aenter__(self):
        self.token = str(uuid.uuid4())
        acquired = await self.redis.set(self.key, self.token, nx=True, px=self.ttl)
        if not acquired:
            raise TimeoutError("Failed to acquire distributed lock.")
        return self

    async def __aexit__(self, exc_type, exc, tb):
        lua_release = """
        if redis.call('get', KEYS[1]) == ARGV[1] then
            return redis.call('del', KEYS[1])
        else
            return 0
        end
        """
        await self.redis.eval(lua_release, 1, self.key, self.token)

3. Fencing Tokens: The Ironclad Guard

No distributed lock is 100% immune to network delays or process pauses without a monotonically increasing Fencing Token. When a lock is issued, Redis or the lock manager increments a counter (e.g., token 34, 35, 36). When writing to PostgreSQL or S3, the database checks: UPDATE orders SET status='processed' WHERE id=123 AND fencing_token < 36.

The Shareable Quote: "Locks attempt to prevent concurrent action; Fencing Tokens guarantee that out-of-order execution is rejected by the underlying storage."

🛠️ Day 28 Project: Build a Resilient Distributed Mutex

Build a distributed task queue guard in Python using Redis and Lua atomic scripts.

• Create a Python context manager using redis-py or aioredlock.
• Implement an atomic Lua script for safe lock releasing.
• Simulate a network pause and verify that stale releases are safely ignored.

🔥 PRO UPGRADE / TEASER

Tomorrow, we move from locking mechanisms to event-driven architectures: mastering Kafka Partitioning, Consumer Groups, and At-Least-Once Delivery Guarantees in Python.

Architectural Consulting

If you are building high-concurrency Python backends or microservices and need senior architectural guidance, I am available for direct contracting.

Explore Enterprise Engagements →

← Previous

Day 27: The Event Loop Safeguard (Async Locks & Semaphores)

Next →

Day 29: Kafka Partitioning & Event Streams