<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Disruptive Studio Engineering]]></title><description><![CDATA[Disruptive Studio Engineering]]></description><link>https://disruptivestudio.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 21:01:24 GMT</lastBuildDate><atom:link href="https://disruptivestudio.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Durable Execution for AI Agents: Architecture and Failure Handling]]></title><description><![CDATA[An AI agent that works in a demo often breaks in production for a boring reason: the process running it dies halfway through. A deploy restarts the…]]></description><link>https://disruptivestudio.hashnode.dev/durable-execution-for-ai-agents-architecture-and-failure-handling</link><guid isPermaLink="true">https://disruptivestudio.hashnode.dev/durable-execution-for-ai-agents-architecture-and-failure-handling</guid><category><![CDATA[AI]]></category><category><![CDATA[architecture]]></category><category><![CDATA[Python]]></category><category><![CDATA[backend]]></category><dc:creator><![CDATA[Daniel Cruz]]></dc:creator><pubDate>Mon, 16 Nov 2026 17:00:30 GMT</pubDate><enclosure url="https://storage.googleapis.com/ariani-user-images-production/resources/organic-covers/en/generated/ae05d7d0-4de2-491c-807f-f68203249531.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>An AI agent that works in a demo often breaks in production for a boring reason: the process running it dies halfway through. A deploy restarts the container, a model API times out, a tool call hangs, or a person takes two days to approve a refund. If the agent's progress lives only in memory, you lose the run, or worse, you retry it and repeat side effects. Durable execution for AI agents is the engineering answer to that problem: every step of the agent loop is recorded so a run can resume exactly where it stopped. This article covers how to design it: the data model, the step loop, side effects, human approvals, and the trade-offs between building it yourself, using a framework checkpointer, or adopting a workflow engine.</p>
<p>If you are still deciding what agents should do in your company and how to roll them out, start with our <a href="https://disruptivestudio.com/blog/ai-agents-business-guide">7-step guide to AI agents for business</a>. This piece assumes that decision is made and focuses on the system underneath.</p>
<h2>Why agent runs fail differently from normal requests</h2>
<p>A typical web request is short, mostly deterministic and easy to retry. An agent run is none of those:</p>
<ul>
<li><strong>It is long.</strong> A run can involve several model calls and tool calls, and may pause for minutes or days waiting for a human.</li>
<li><strong>It is non-deterministic.</strong> Calling the model again with the same input can produce a different plan. You cannot simply rerun from the start and expect the same path.</li>
<li><strong>It has side effects.</strong> Tools send emails, create invoices, update a CRM record or issue refunds. Repeating them is a business problem, not just a technical one.</li>
<li><strong>It depends on flaky dependencies.</strong> Model providers rate-limit and time out, and internal APIs go down.</li>
</ul>
<p>Temporal's own framing is useful here: in announcing its OpenAI Agents SDK integration, it states plainly that <a href="https://temporal.io/blog/announcing-openai-agents-sdk-integration">"AI agents are distributed systems"</a>. Treat them like one: assume every step can fail, and design so that failure costs a retry instead of a lost run.</p>
<h2>What durable execution for AI agents actually means</h2>
<p>The core idea is to split the agent into two kinds of code:</p>
<ol>
<li><strong>Orchestration</strong>: the loop that decides what happens next (call the model, run a tool, wait for approval, finish). This code must be replayable.</li>
<li><strong>Steps</strong>: the I/O-heavy work, such as model requests, tool calls and external APIs. Each step's result is persisted once it completes.</li>
</ol>
<p>When the process crashes, a new worker loads the run's history, replays the orchestration using the <em>recorded</em> results instead of calling the model again, and continues from the first step that never finished. The Pydantic AI documentation describes the same split for its Temporal integration: <a href="https://ai.pydantic.dev/durable_execution/temporal/">model requests, tool calls that may require I/O, and MCP server communication are offloaded to Temporal activities</a>, while the agent's coordinating logic runs in the workflow.</p>
<p>That rule, "never redo a step whose result is already recorded", is what keeps a non-deterministic model from taking a different path after a restart.</p>
<h2>The data model if you build it yourself</h2>
<p>You do not need a workflow engine to get most of the benefit. A relational database and a strict step loop go a long way. A minimal schema:</p>
<pre><code class="language-sql">CREATE TABLE agent_runs (
  id            UUID PRIMARY KEY,
  agent_name    TEXT NOT NULL,
  status        TEXT NOT NULL,        -- running | waiting_approval | completed | failed | cancelled
  input         JSONB NOT NULL,
  lease_owner   TEXT,                 -- worker currently executing the run
  lease_expires TIMESTAMPTZ,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE agent_steps (
  run_id        UUID NOT NULL REFERENCES agent_runs(id),
  step_index    INT  NOT NULL,
  kind          TEXT NOT NULL,        -- model_call | tool_call | approval | final
  request       JSONB NOT NULL,       -- prompt or tool name + validated arguments
  result        JSONB,                -- null until the step completes
  status        TEXT NOT NULL,        -- pending | completed | failed
  attempts      INT  NOT NULL DEFAULT 0,
  idempotency_key TEXT NOT NULL UNIQUE,
  PRIMARY KEY (run_id, step_index)
);

CREATE TABLE agent_approvals (
  run_id        UUID NOT NULL,
  step_index    INT  NOT NULL,
  decision      TEXT,                 -- approved | rejected | edited
  edited_args   JSONB,
  decided_by    TEXT,
  decided_at    TIMESTAMPTZ,
  PRIMARY KEY (run_id, step_index)
);
</code></pre>
<p>Three design choices matter more than the exact columns:</p>
<ul>
<li><strong>Steps are append-only and ordered.</strong> The pair <code>(run_id, step_index)</code> is the step's identity. Replaying a run means walking these rows in order.</li>
<li><strong>The idempotency key is derived, not random.</strong> Use something like <code>run_id:step_index</code>. If a worker crashes after calling a tool but before saving the result, the retry sends the same key, and a well-behaved downstream API can deduplicate it.</li>
<li><strong>A lease prevents two workers from running the same run.</strong> A worker claims a run by setting <code>lease_owner</code> and <code>lease_expires</code> in a conditional update, and renews the lease while it works. If it dies, the lease expires and another worker picks the run up.</li>
</ul>
<h2>The step loop</h2>
<p>The orchestration loop is where correctness lives. In pseudo-code:</p>
<pre><code class="language-python">def execute_run(run_id, worker_id):
    if not claim_lease(run_id, worker_id):
        return  # another worker owns it

    history = load_completed_steps(run_id)  # ordered list
    messages = rebuild_messages(history)
    step_index = len(history)

    while True:
        # 1. Ask the model what to do next (a recorded step)
        decision = run_step(run_id, step_index, "model_call",
                            request={"messages": messages},
                            fn=lambda: call_model(messages, tools=ALLOWED_TOOLS))
        step_index += 1

        if decision.type == "final":
            mark_run(run_id, "completed", output=decision.content)
            return

        # 2. Validate tool arguments before anything runs
        tool = TOOLS[decision.tool_name]
        args = tool.schema.validate(decision.arguments)

        # 3. Pause for a human if the tool is sensitive
        if tool.requires_approval:
            approval = get_approval(run_id, step_index)
            if approval is None:
                request_approval(run_id, step_index, tool.name, args)
                mark_run(run_id, "waiting_approval")
                release_lease(run_id)
                return  # the process exits; nothing is held in memory
            if approval.decision == "rejected":
                messages.append(tool_rejected_message(tool.name))
                step_index += 1
                continue
            args = approval.edited_args or args

        # 4. Execute the tool exactly once per step (a recorded step)
        result = run_step(run_id, step_index, "tool_call",
                          request={"tool": tool.name, "args": args},
                          fn=lambda: tool.execute(args,
                                 idempotency_key=f"{run_id}:{step_index}"))
        messages.append(tool_result_message(tool.name, result))
        step_index += 1


def run_step(run_id, step_index, kind, request, fn):
    existing = get_step(run_id, step_index)
    if existing and existing.status == "completed":
        return existing.result           # replay: never call the model or tool again
    upsert_pending_step(run_id, step_index, kind, request)
    result = with_retries(fn, max_attempts=3, backoff="exponential")
    complete_step(run_id, step_index, result)
    return result
</code></pre>
<p>Notice what the loop never does: it never holds state that only exists in memory, and it never re-executes a completed step. Waiting for a human is not a sleeping thread; it is a status in the database. When the approval arrives, a webhook or queue message triggers <code>execute_run</code> again and the loop replays its way back to the same point.</p>
<h2>Side effects: the part that actually hurts</h2>
<p>Durable execution guarantees that your <em>orchestration</em> resumes correctly. It cannot guarantee that an external system did not receive a request whose response you never saved. That gap is where duplicate emails and double refunds come from. Close it with three patterns:</p>
<ul>
<li><strong>Idempotency keys on every write tool.</strong> Pass the derived key to downstream APIs that support it. For your own services, store processed keys and return the original result when a key repeats.</li>
<li><strong>Transactional outbox for internal effects.</strong> When a tool writes to your own database and publishes an event, write the record and the outgoing message in the same transaction, then let a separate process deliver the message.</li>
<li><strong>Compensation for things you cannot make idempotent.</strong> If a tool has no deduplication, design a reverse action (cancel the order, void the invoice) and record which steps need it if the run is abandoned.</li>
</ul>
<p>Classify every tool as read-only, idempotent write, or non-idempotent write before you give it to an agent. Read-only tools can retry freely. Non-idempotent writes should usually require approval or be wrapped in a service you control.</p>
<h2>Human approvals as durable waits</h2>
<p>Approvals are the clearest reason persistence is mandatory, not optional. The LangChain documentation for its human-in-the-loop middleware states that <a href="https://docs.langchain.com/oss/python/langchain/human-in-the-loop">you must configure a checkpointer to persist the graph state across interrupts</a>, and recommends a persistent checkpointer such as AsyncPostgresSaver or MongoDBSaver in production, keeping InMemorySaver for testing.</p>
<p>Whatever stack you use, design the approval record with care:</p>
<ul>
<li>Show the reviewer the exact validated arguments, not the model's prose about them.</li>
<li>Allow three outcomes: approve, reject with a reason the agent can read, or edit the arguments.</li>
<li>Record who decided and when. This is your audit trail.</li>
<li>Set an expiry. A run that waited too long should fail cleanly instead of executing on stale data.</li>
</ul>
<h2>Build, framework or workflow engine: the trade-offs</h2>
<p>There are three realistic approaches, and the right one depends on how long your runs are and how much infrastructure you want to own.</p>
<h3>1. Do it yourself on Postgres</h3>
<p><strong>Good for:</strong> a few agents, short-to-medium runs, teams that already operate Postgres and a queue. <strong>Pros:</strong> full control, no new infrastructure, the data model fits your domain. <strong>Cons:</strong> you own leases, retries, timers and replay logic, and subtle bugs in those are hard to find.</p>
<h3>2. Framework checkpointers (for example, LangGraph)</h3>
<p><strong>Good for:</strong> teams already building on that framework. The LangGraph docs explain that checkpointers save state so a person can <a href="https://docs.langchain.com/oss/python/langgraph/checkpointers">inspect, interrupt, and approve graph steps</a> and the graph can resume afterwards. They also expose durability modes; in the <code>"exit"</code> mode, changes are persisted only when graph execution exits, whether successfully, with an error or at an interrupt. That is cheaper but means a crash mid-run can lose the progress since the last exit, so choose the mode per workload. <strong>Cons:</strong> durability is scoped to the framework's graph; side effects inside tools are still your responsibility.</p>
<h3>3. A durable execution engine (for example, Temporal)</h3>
<p><strong>Good for:</strong> long-running, high-value or multi-agent workflows. Temporal's tutorial on durable agents notes that workflows preserve context through system interruptions and that <a href="https://learn.temporal.io/tutorials/ai/durable-ai-agent/">the duration of the Workflow Execution is irrelevant</a>. Temporal offers integrations with the <a href="https://temporal.io/blog/announcing-openai-agents-sdk-integration">OpenAI Agents SDK</a> (announced in public preview) and with the <a href="https://temporal.io/blog/building-durable-agents-with-temporal-and-ai-sdk-by-vercel">AI SDK by Vercel</a>. <strong>Cons:</strong> another system to run or pay for, and the workflow code must follow the engine's determinism rules, which takes some getting used to.</p>
<p>A pragmatic path: start with approach 1 or 2 for a single, well-scoped agent, and move to an engine when you have long waits, many concurrent runs or cross-service workflows.</p>
<h2>Security and cost controls that belong in the architecture</h2>
<p>Durability makes agents resilient; it also makes them persistent, which raises the stakes of a bad action. Build these into the loop, not into the prompt:</p>
<ul>
<li><strong>Allow-list tools per agent and per run.</strong> The model can only call what <code>ALLOWED_TOOLS</code> contains.</li>
<li><strong>Validate arguments with a schema</strong> before execution and before approval. Reject, do not coerce.</li>
<li><strong>Use scoped credentials.</strong> Each tool runs with the narrowest permissions it needs, never a shared admin token.</li>
<li><strong>Set budgets.</strong> Cap the number of steps, model tokens and wall-clock time per run, and fail the run when it exceeds them.</li>
<li><strong>Keep large payloads out of the step table.</strong> Store documents and long tool outputs in object storage and save a reference, so replay stays fast and checkpoints stay small.</li>
</ul>
<h2>Common mistakes</h2>
<ul>
<li>Keeping conversation state in process memory "for now".</li>
<li>Retrying the whole run from the start after a failure, which re-calls the model and can pick a different path.</li>
<li>Generating a random idempotency key on each attempt, which defeats the purpose.</li>
<li>Treating approval as a blocking call in a web request.</li>
<li>Letting tools decide their own permissions instead of enforcing them in the orchestrator.</li>
<li>Logging only the final answer, so you cannot explain how a run reached it.</li>
</ul>
<h2>Implementation checklist</h2>
<ul>
<li>Every run has a persistent ID and status.</li>
<li>Every model call and tool call is a recorded, ordered step.</li>
<li>Completed steps are never re-executed on replay.</li>
<li>Idempotency keys are derived from run and step, and passed downstream.</li>
<li>Each tool is classified as read-only, idempotent write or non-idempotent write.</li>
<li>Sensitive tools require approval, with expiry and an audit record.</li>
<li>A lease or engine guarantees one active worker per run.</li>
<li>Retries use backoff and a maximum attempt count.</li>
<li>Step, token and time budgets are enforced in code.</li>
<li>Large payloads live outside the checkpoint store.</li>
</ul>
<p>If you need this kind of backbone built into your own product, with your data model and your systems, that is the work our team does in <a href="https://disruptivestudio.com/custom-software">custom software development</a>.</p>
<h2>Frequently asked questions</h2>
<h3>Do I need Temporal to make an AI agent durable?</h3>
<p>No. A database with an ordered step table, derived idempotency keys and a lease per run covers many cases. An engine becomes worth it when runs are long, numerous or span several services.</p>
<h3>Is a framework checkpointer the same as durable execution?</h3>
<p>It covers part of it: saving and resuming agent state, which is what human approvals need. It does not make side effects inside your tools safe; idempotency and compensation are still your job.</p>
<h3>How do I stop a resumed agent from repeating an action?</h3>
<p>Never re-execute a completed step, and send a deterministic idempotency key with every write so the downstream system can ignore duplicates when the crash happened between the call and the save.</p>
<hr />
<p><em>Originally published on <a href="https://disruptivestudio.com/blog/durable-execution-for-ai-agents">Disruptive Studio</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[PostgreSQL Point-in-Time Recovery: A Production Architecture Guide]]></title><description><![CDATA[A nightly pg_dump can only take you back to last night. If someone runs a bad migration at 14:31, you want the database as it was at 14:30. That is what…]]></description><link>https://disruptivestudio.hashnode.dev/postgresql-point-in-time-recovery-a-production-architecture-guide</link><guid isPermaLink="true">https://disruptivestudio.hashnode.dev/postgresql-point-in-time-recovery-a-production-architecture-guide</guid><category><![CDATA[postgres]]></category><category><![CDATA[database]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Backup]]></category><dc:creator><![CDATA[Daniel Cruz]]></dc:creator><pubDate>Sat, 14 Nov 2026 17:00:49 GMT</pubDate><enclosure url="https://storage.googleapis.com/ariani-user-images-production/resources/organic-covers/en/generated/0a801d6a-1590-40ac-8450-dda34f310bdc.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A nightly <code>pg_dump</code> can only take you back to last night. If someone runs a bad migration at 14:31, you want the database as it was at 14:30. That is what <strong>PostgreSQL point-in-time recovery</strong> (PITR) does. This guide covers how to build it as a system: the architecture, archiving that reports failures honestly, a backup repository that is hard to destroy, a restore path you have practiced, and monitoring that warns you before you need a restore. For the business side (the 3-2-1 rule, RPO, RTO and who owns the plan), read our <a href="https://disruptivestudio.com/blog/small-business-data-backup-plan">small business data backup plan guide</a>. This article covers the engineering.</p>
<h2>How PostgreSQL point-in-time recovery works</h2>
<p>PITR combines two things: a physical <em>base backup</em> of the data directory, and a continuous archive of the write-ahead log (WAL) segments created after it. To recover, you restore the base backup and replay archived WAL until you reach a chosen target: a timestamp, a transaction ID, an LSN or a named restore point. The official documentation, <a href="https://www.postgresql.org/docs/16/continuous-archiving.html">Continuous Archiving and Point-in-Time Recovery</a>, explains how this works. Read it once from start to finish before you choose a tool.</p>
<p>Three facts from that documentation shape the whole design:</p>
<ul>
<li>To enable archiving, you set <code>wal_level</code> to <code>replica</code> or higher, set <code>archive_mode</code> to <code>on</code>, and configure an archive command.</li>
<li>The archive command only runs on <em>completed</em> WAL segments. A transaction committed a minute ago may still sit in a segment that has not been archived yet.</li>
<li>In rare cases, PostgreSQL may try to archive a WAL file it has already archived, so your archiver must handle duplicates safely.</li>
</ul>
<h2>Choosing an approach: the trade-offs</h2>
<ul>
<li><strong>Logical dumps (<code>pg_dump</code>)</strong>: portable across major versions and easy to restore one table at a time. They are only snapshots, though: your recovery point is the last dump, and on large databases both dumping and restoring are slow. Keep them in addition to PITR, not instead of it.</li>
<li><strong>Streaming replicas</strong>: they give you fast failover, but a replica is not a backup. A <code>DROP TABLE</code> reaches the replica within moments. A delayed replica (<code>recovery_min_apply_delay</code>) helps, but it does not replace an off-host archive.</li>
<li><strong>Managed database PITR</strong> (RDS, Cloud SQL, Azure, and others): the least work to run. Still check how long the provider keeps backups, whether backups survive if the instance or account is deleted, and whether you can export them somewhere else.</li>
<li><strong>Self-managed PITR with a dedicated tool</strong>: <a href="https://pgbackrest.org/">pgBackRest</a>, <a href="https://github.com/wal-g/wal-g">WAL-G</a> and <a href="https://pgbarman.org/">Barman</a> handle parallel transfer, compression, encryption, retention and verification. Unless you have an unusual constraint, pick one of these instead of writing your own <code>archive_command</code> script.</li>
</ul>
<p>The rest of this guide shows the plain mechanics, so you understand what those tools do and can judge them.</p>
<h2>Reference architecture</h2>
<ol>
<li><strong>Primary database</strong> with WAL archiving enabled.</li>
<li><strong>Archiver</strong>: the archive command or archive library, running on the database host.</li>
<li><strong>Repository</strong>: object storage in a <em>separate</em> account or project. The database host's credentials can write to it but cannot delete from it.</li>
<li><strong>Base backup job</strong>: a scheduled job, usually running from a backup host instead of the primary.</li>
<li><strong>Restore worker</strong>: a disposable machine or container that rebuilds the database from the repository on a schedule.</li>
<li><strong>Monitor</strong>: checks archive lag, archive failures, base backup age and the result of the last restore test.</li>
</ol>
<h2>Step 1: WAL archiving that reports failures honestly</h2>
<pre><code class="language-ini"># postgresql.conf
wal_level = replica
archive_mode = on
archive_command = '/usr/local/bin/archive_wal.sh %p %f'
archive_timeout = 60   # force a segment switch on quiet databases
</code></pre>
<p>PostgreSQL treats a zero exit code as "archived". If the script returns 0 before the data is durable in the repository, you have an unrecoverable gap and nothing tells you. The core logic, in pseudo-bash:</p>
<pre><code class="language-bash">#!/usr/bin/env bash
# archive_wal.sh %p %f  (pseudo-code: put_object/head_sha256 wrap your storage CLI)
set -euo pipefail
src="$1"; name="$2"
key="pg-main/wal/$name"
local_sum=$(sha256sum "$src" | cut -d' ' -f1)

if remote_sum=$(head_sha256 "$key" 2&gt;/dev/null); then
  # Already archived: success only if the content is identical
  [ "$remote_sum" = "$local_sum" ] &amp;&amp; exit 0
  echo "refusing to overwrite $key with different content" &gt;&amp;2
  exit 1
fi

encrypt "$src" | put_object "$key" --checksum-sha256 "$local_sum"
[ "$(head_sha256 "$key")" = "$local_sum" ] || exit 1
</code></pre>
<p>Design notes:</p>
<ul>
<li><strong>Duplicates are safe, overwrites are not.</strong> An identical file already in the archive counts as success. A different file under the same name is an incident, not something to overwrite silently.</li>
<li><strong>RPO is set by segment switching.</strong> Because only completed segments are archived, a quiet database can keep a partial segment open for a long time. <code>archive_timeout</code> limits that window, and you can force a switch with <code>pg_switch_wal()</code>. If you need an RPO close to zero, stream WAL continuously (<code>pg_receivewal</code> or your tool's async/streaming mode) instead of shortening the timeout.</li>
<li><strong>A failing archiver fills the disk.</strong> WAL segments that have not been archived stay in <code>pg_wal</code>. A broken credential can therefore turn into a full disk on the primary. Alert on this long before it happens.</li>
</ul>
<h2>Step 2: Base backups and retention</h2>
<pre><code class="language-bash">dir=/backups/base/$(date -u +%Y%m%dT%H%M%SZ)
pg_basebackup -h primary -U backup -D "$dir" -Fp -X stream -c fast
pg_verifybackup "$dir"          # checks files against the backup manifest
encrypt_and_upload "$dir"        # then remove the local copy
</code></pre>
<p>The documentation says to take file-system-level base backups for PITR, not <code>pg_dump</code>, and you do not need to stop the database to take them. The main trade-off is <strong>how often you take them versus how long a restore takes</strong>. Recovery replays every WAL segment between the base backup and the target. On a database with heavy writes, a week-old base backup can mean hours of replay. Take full backups more often, or use incremental backups (pgBackRest and WAL-G support them, and recent PostgreSQL releases include native incremental backup), until your <em>measured</em> restore time fits your RTO.</p>
<p>Retention rule: <strong>never delete WAL that is newer than your oldest kept base backup.</strong> If you keep base backups for 14 days, you must keep all WAL from the oldest one onward, or that backup cannot be recovered past its end point. Dedicated tools enforce this for you. Hand-written lifecycle rules usually don't.</p>
<h2>Step 3: Make the repository hard to destroy</h2>
<ul>
<li><strong>Separate the blast radius.</strong> Keep the repository in a different cloud account or project. If the database host is compromised, the attacker should get credentials that can only upload, not delete or change lifecycle rules.</li>
<li><strong>Immutability.</strong> Object-level write-once retention, such as <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html">Amazon S3 Object Lock</a> or the equivalents from other providers, stops deletion until the retention period ends. Set the retention period to at least your backup retention window.</li>
<li><strong>Client-side encryption.</strong> Encrypt before upload and store the key outside the repository account. Test that the restore worker can get the key without access to production.</li>
<li><strong>A second copy.</strong> Replicate the repository to another region or provider. This is the "1 off-site" part of 3-2-1, applied to WAL.</li>
</ul>
<h2>Step 4: The restore path</h2>
<p>Since PostgreSQL 12, recovery settings go in <code>postgresql.conf</code> (or <code>postgresql.auto.conf</code>), and you trigger recovery by creating a <code>recovery.signal</code> file in the data directory. See the <a href="https://www.postgresql.org/docs/15/runtime-config-wal.html">WAL configuration reference</a>.</p>
<pre><code class="language-ini"># after restoring the base backup into $PGDATA (owned by postgres, not root)
restore_command = '/usr/local/bin/fetch_wal.sh %f %p'
recovery_target_time = '2025-03-14 14:30:00+00'   # example target
recovery_target_action = 'pause'
</code></pre>
<ul>
<li>Set <strong>only one</strong> recovery target. Recent versions raise an error if you set more than one.</li>
<li>Use <code>pause</code> for incident restores. Run read-only queries to confirm the bad change is not there. If you need to go further, set a later target and restart; when the state is right, promote with <code>pg_promote()</code>.</li>
<li>Always write timestamps with an explicit time zone. Converting "14:30 local" in a hurry during an incident is a common source of mistakes.</li>
<li>Restore to a <strong>new</strong> instance. Don't overwrite the damaged primary, which you may still need for forensics or to copy rows the restore does not include.</li>
<li>Fix ownership and permissions after copying files. The docs point out that files restored as root are a common mistake.</li>
</ul>
<p>If you know a risky operation is coming, such as a big migration or a bulk import, create a named point first with <code>SELECT pg_create_restore_point('before_migration_42');</code> and target it with <code>recovery_target_name</code>.</p>
<h2>Step 5: Automate restore tests</h2>
<p>Until you have restored a backup, you don't know it works. Run this on a schedule on the restore worker:</p>
<pre><code class="language-bash">start=$(date +%s)
base=$(latest_base_backup)
fetch_and_decrypt "$base" /restore/data
chown -R postgres:postgres /restore/data &amp;&amp; chmod 700 /restore/data
touch /restore/data/recovery.signal
cat &gt;&gt; /restore/data/postgresql.auto.conf &lt;&lt;'EOF'
restore_command = '/usr/local/bin/fetch_wal.sh %f %p'
recovery_target_action = 'promote'
EOF
sudo -u postgres pg_ctl -D /restore/data -w -t 7200 start

# Business-level checks, not just "it started"
psql -U postgres -d app -Atc "SELECT max(created_at) FROM orders"   # freshness
psql -U postgres -d app -Atc "SELECT count(*) FROM customers"        # sanity
report_metric restore_seconds $(( $(date +%s) - start ))
report_metric restore_ok 1
</code></pre>
<p>Without a target, recovery replays to the end of the archive. The gap between <code>max(created_at)</code> and the start of the test is your RPO as it actually happened. The elapsed time is your actual RTO. Store both values over time. A test that has not run is a failure, so alert when the last successful run is too old, not only when a run fails.</p>
<h2>Monitoring that warns you before you need a restore</h2>
<pre><code class="language-sql">SELECT archived_count,
       failed_count,
       last_archived_wal,
       last_failed_wal,
       last_failed_time,
       now() - last_archived_time AS since_last_archive
FROM pg_stat_archiver;
</code></pre>
<p>Alert on these:</p>
<ul>
<li><code>since_last_archive</code> above a few multiples of <code>archive_timeout</code>.</li>
<li><code>last_failed_time</code> newer than <code>last_archived_time</code>, meaning the archiver is failing right now.</li>
<li>Disk usage of <code>pg_wal</code> growing steadily.</li>
<li>Age of the newest verified base backup.</li>
<li>Age and result of the last restore test.</li>
</ul>
<h2>Common mistakes</h2>
<ul>
<li>Treating a replica or a disk snapshot as the backup.</li>
<li>An archive command that returns 0 when the upload fails, for example a pipeline without <code>pipefail</code>.</li>
<li>Lifecycle rules that delete WAL still needed by the oldest base backup.</li>
<li>Database host credentials that can delete the repository.</li>
<li>Encryption keys stored only on the server you are trying to recover.</li>
<li>Never measuring replay time, so the RTO is a guess.</li>
<li>Restore runbooks written once and never run.</li>
</ul>
<h2>Checklist</h2>
<ul>
<li>Archiving on; archive command handles duplicates safely and checks uploads.</li>
<li><code>archive_timeout</code> or WAL streaming set to match your RPO.</li>
<li>Base backups verified with <code>pg_verifybackup</code> or your tool's verify command.</li>
<li>WAL retention covers the oldest base backup.</li>
<li>Repository in a separate account, with write-only credentials, object lock and a second region.</li>
<li>Client-side encryption; key stored and tested outside production.</li>
<li>Scheduled automated restore with business-level checks and recorded RTO/RPO.</li>
<li>Alerts on archive lag, archive failures, <code>pg_wal</code> growth, backup age and restore test age.</li>
</ul>
<p>If your product runs on PostgreSQL and nobody owns this pipeline, that is a gap worth closing. It is the kind of infrastructure we design and build as part of <a href="https://disruptivestudio.com/custom-software">custom software development</a>.</p>
<h2>Frequently asked questions</h2>
<h3>Is PITR possible on a managed database?</h3>
<p>Usually yes, and it is usually turned on with a retention setting. Check how far back you can restore, whether backups survive if the instance is deleted, and whether you can export to storage you control.</p>
<h3>Should I still keep pg_dump backups?</h3>
<p>Yes, as an extra layer. Logical dumps work across major versions and make it easy to restore a single table, which physical PITR cannot do directly.</p>
<h3>How often should I test restores?</h3>
<p>Often enough that a broken backup cannot go unnoticed longer than you could tolerate. Automate the test, and alert when it fails or has not run.</p>
<hr />
<p><em>Originally published on <a href="https://disruptivestudio.com/blog/postgresql-point-in-time-recovery">Disruptive Studio</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[Automated Dependency Updates: Designing a Safe Pipeline]]></title><description><![CDATA[Most teams treat automated dependency updates as a bot they switch on and forget, until the bot opens forty pull requests on a Monday, half of them red…]]></description><link>https://disruptivestudio.hashnode.dev/automated-dependency-updates-designing-a-safe-pipeline</link><guid isPermaLink="true">https://disruptivestudio.hashnode.dev/automated-dependency-updates-designing-a-safe-pipeline</guid><category><![CDATA[Devops]]></category><category><![CDATA[Security]]></category><category><![CDATA[ #githubactions]]></category><category><![CDATA[architecture]]></category><dc:creator><![CDATA[Daniel Cruz]]></dc:creator><pubDate>Thu, 12 Nov 2026 17:00:38 GMT</pubDate><enclosure url="https://storage.googleapis.com/ariani-user-images-production/resources/organic-covers/en/generated/48f32652-706e-446d-a529-cd9542d527c7.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most teams treat <strong>automated dependency updates</strong> as a bot they switch on and forget, until the bot opens forty pull requests on a Monday, half of them red, and someone disables it. Done properly, it works more like a small production pipeline: it detects new versions, classifies risk, waits, verifies, merges, deploys and can roll back. This guide covers how to design that pipeline: the policy model, tool trade-offs between Dependabot and Renovate, the CI gate, failure handling and the security details that keep the updater from becoming an attack path. If you need the business side (what a maintenance contract should cover, who owns what), start with our <a href="https://disruptivestudio.com/blog/software-maintenance-plan-guide">guide to building a software maintenance plan</a>. This article covers the engineering underneath it.</p>
<h2>Why dependency updates are a systems problem</h2>
<p>Every dependency bump carries one of four distinct risks, and each needs a different control:</p>
<ul>
<li><strong>Breaking change.</strong> The new version changes behavior or an API you rely on. Control: tests and a type checker that actually exercise the dependency.</li>
<li><strong>Bad or compromised release.</strong> A version is published broken or malicious and is pulled or patched shortly after. Control: a waiting period before adoption.</li>
<li><strong>Drift.</strong> You stop updating for months, so every later update is a multi-major jump that nobody wants to review. Control: a steady, scheduled cadence.</li>
<li><strong>Review fatigue.</strong> Too many small PRs, so people rubber-stamp or ignore them. Control: grouping, limits and auto-merge for low-risk classes.</li>
</ul>
<p>A bot on default settings handles none of these well. A designed pipeline handles all four.</p>
<h2>Architecture of an automated dependency updates pipeline</h2>
<p>Think of it as stages, each with a single job and a clear failure mode:</p>
<ol>
<li><strong>Detect.</strong> Scan manifests and lockfiles for each ecosystem (npm, pip, Docker images, GitHub Actions, Terraform providers).</li>
<li><strong>Classify.</strong> Label each candidate by update type (patch, minor, major, digest), dependency type (production or development) and whether it fixes a known advisory.</li>
<li><strong>Batch.</strong> Group related candidates so one PR holds one coherent change.</li>
<li><strong>Wait.</strong> Hold non-security updates until the release has existed for a minimum time.</li>
<li><strong>Verify.</strong> Run CI on a fresh install from the updated lockfile.</li>
<li><strong>Decide.</strong> Auto-merge, request review or hold, based on policy.</li>
<li><strong>Deploy and observe.</strong> Ship through the normal pipeline and watch error rates.</li>
<li><strong>Recover.</strong> Revert fast, pin with a reason and an owner, open a follow-up.</li>
</ol>
<p>The key design decision: <strong>classification and policy live in version-controlled config, not in people's heads.</strong> Reviewers should only see the PRs the policy says need a human.</p>
<h3>The policy model</h3>
<p>Before choosing a tool, write the policy down as a short list of update classes. It becomes your config and the "dependencies" section of your maintenance runbook.</p>
<ul>
<li><strong>Dev dependency, patch or minor:</strong> wait for the minimum release age, one weekly group, auto-merge on green CI.</li>
<li><strong>Production, patch:</strong> minimum release age, one group per ecosystem, auto-merge for trusted packages, otherwise review.</li>
<li><strong>Production, minor:</strong> minimum release age, one group per ecosystem, human review.</li>
<li><strong>Any major:</strong> minimum release age, one PR per package, review, and the PR only opens after explicit approval.</li>
<li><strong>Security fix:</strong> no wait (or a very short one), separate PR, fast-track review.</li>
</ul>
<p>Semantic versioning is what makes the classes meaningful: under the <a href="https://semver.org/">SemVer specification</a>, a major version signals incompatible API changes. Treat that as a hint, not a guarantee. Plenty of packages ship breaking changes in minors, which is why "auto-merge" always means "auto-merge if CI passes", never "merge blind".</p>
<p>In code, the decision step is a small pure function, easy to test and easy to explain in a review:</p>
<pre><code class="language-typescript">type UpdateType = "patch" | "minor" | "major" | "digest";

interface UpdateCandidate {
  name: string;
  updateType: UpdateType;
  depType: "prod" | "dev";
  releaseAgeHours: number;
  fixesAdvisory: boolean;
}

type Decision = "automerge" | "review" | "hold";

function decide(c: UpdateCandidate, ciGreen: boolean, policy: Policy): Decision {
  if (!ciGreen) return "hold";
  if (c.fixesAdvisory) return "review";                 // fast-track, but a human looks
  if (c.releaseAgeHours &lt; policy.minAgeHours) return "hold";
  if (c.updateType === "major") return "review";
  if (c.depType === "dev") return "automerge";
  if (c.updateType === "patch" &amp;&amp; policy.trusted.has(c.name)) return "automerge";
  return "review";
}
</code></pre>
<p>You rarely write this yourself, since Dependabot and Renovate implement versions of it, but keeping it explicit tells you which config options you actually need.</p>
<h2>Dependabot vs Renovate: the trade-offs</h2>
<p>Both tools can do the job. The difference is how much of the policy each one expresses natively.</p>
<h3>Dependabot</h3>
<p>Dependabot is built into GitHub, so there is nothing to host. Its <code>groups</code> setting lets you <a href="https://docs.github.com/en/code-security/tutorials/secure-your-dependencies/optimizing-pr-creation-version-updates">consolidate updates for multiple dependencies into a single pull request</a>, filtered by dependency type and update type. One constraint to know: <a href="https://docs.github.com/en/code-security/reference/supply-chain-security/troubleshoot-dependabot/dependabot-errors">groups must be configured per package ecosystem</a>. Security updates can be <a href="https://docs.github.com/en/code-security/concepts/supply-chain-security/dependabot-security-updates">grouped too</a>, which helps when one advisory touches several packages.</p>
<pre><code class="language-yaml"># .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 5
    groups:
      prod-minor-patch:
        dependency-type: "production"
        update-types: ["minor", "patch"]
      dev-minor-patch:
        dependency-type: "development"
        update-types: ["minor", "patch"]
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
</code></pre>
<p>Auto-merge with Dependabot usually means a separate workflow that reads the PR metadata and enables GitHub's auto-merge for allowed classes. That works, but your policy ends up split between two files.</p>
<h3>Renovate</h3>
<p>Renovate can be self-hosted or run as an app, and it supports several Git platforms, which matters if you are not only on GitHub. Its main advantage for this design is that the whole policy fits in one file. The <code>minimumReleaseAge</code> option <a href="https://docs.renovatebot.com/configuration-options/">requires Renovate to wait a set time before suggesting an update</a>, and when combined with automerge, Renovate creates the PR right away but only merges once the age has passed, adding a <code>renovate/stability-days</code> pending status check. Two caveats from the docs: the datasource <a href="https://docs.renovatebot.com/configuration-options/">must provide release timestamps</a> for this to work, and the maintainers <a href="https://docs.renovatebot.com/key-concepts/minimum-release-age/">recommend setting a minimum release age in both Renovate and your package manager</a> where it is supported.</p>
<pre><code class="language-json">{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "schedule": ["before 6am on monday"],
  "prConcurrentLimit": 5,
  "minimumReleaseAge": "3 days",
  "packageRules": [
    {
      "matchDepTypes": ["devDependencies"],
      "matchUpdateTypes": ["minor", "patch"],
      "groupName": "dev dependencies (non-major)",
      "automerge": true
    },
    {
      "matchUpdateTypes": ["major"],
      "dependencyDashboardApproval": true,
      "labels": ["major-update"]
    }
  ]
}
</code></pre>
<p>The "3 days" value is a starting point you should tune, not a standard. Longer waits catch more bad releases but leave you exposed for longer on normal updates. Security fixes should be exempt or get a much shorter wait.</p>
<p><strong>Rule of thumb:</strong> a single GitHub repo with a simple policy works well with Dependabot. Monorepos, several Git platforms, or a policy with many exceptions usually fit Renovate better.</p>
<h2>The verification gate: making green CI mean something</h2>
<p>Auto-merge is only as safe as the CI it depends on. For dependency PRs, the gate should:</p>
<ul>
<li><strong>Install from the lockfile, clean.</strong> Use <code>npm ci</code> (or <code>pip install --require-hashes</code>, <code>poetry install --sync</code> and similar) so CI tests exactly what will ship.</li>
<li><strong>Type-check and build.</strong> Many breaking changes in typed ecosystems show up at compile time, before any test runs.</li>
<li><strong>Exercise the integration boundary.</strong> Unit tests that mock the dependency prove nothing about the new version. Keep a few tests that hit the real library: the HTTP client, the ORM against a disposable database, the date library with time zones.</li>
<li><strong>Smoke-test the artifact.</strong> Start the built container and hit a health endpoint plus one or two critical routes.</li>
<li><strong>Fail on flakiness, don't retry around it.</strong> If flaky tests get auto-retried until green, auto-merge will eventually ship a real regression. Quarantine flaky tests explicitly and track them.</li>
</ul>
<pre><code class="language-yaml">name: ci
on: pull_request
permissions:
  contents: read
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@&lt;pinned-commit-sha&gt;
      - uses: actions/setup-node@&lt;pinned-commit-sha&gt;
        with:
          node-version-file: ".nvmrc"
      - run: npm ci
      - run: npm run typecheck
      - run: npm test -- --ci
      - run: npm run build
      - run: ./scripts/smoke-test.sh
</code></pre>
<h2>Failure handling: what happens when an update breaks production</h2>
<p>Plan for it, because it will happen. Your design needs three things.</p>
<p><strong>A one-step revert.</strong> Every dependency change lands as its own merge commit or squash, so reverting doesn't drag unrelated work with it. That is another reason not to mix dependency bumps into feature PRs.</p>
<p><strong>Pins with a reason and an owner.</strong> When you hold a package back, record why and what unblocks it, right next to the rule:</p>
<pre><code class="language-json">{
  "matchPackageNames": ["some-orm"],
  "allowedVersions": "&lt;5.0.0",
  "description": "Pinned: v5 changes the transaction API. Owner: backend team. Revisit when the migration ticket is closed."
}
</code></pre>
<p>Pins without an owner turn into the drift you were trying to avoid. Review them on the same schedule as the rest of the maintenance work.</p>
<p><strong>Deploy-time observation.</strong> Tag deploys that contain only dependency changes so that, when error rates move, the dashboard points at the cause right away.</p>
<h2>Security: don't let the updater become the attack path</h2>
<p>An automated pipeline that merges third-party code is, by definition, a supply-chain entry point. Harden it:</p>
<ul>
<li><strong>Least-privilege CI tokens.</strong> Declare <code>permissions:</code> explicitly and give dependency PR workflows read-only access unless a step truly needs more.</li>
<li><strong>No production secrets in PR builds.</strong> Install scripts from a new package version run inside your CI. Don't let them see deploy keys.</li>
<li><strong>Pin actions and base images by digest or commit SHA</strong>, then let the updater bump those pins. You get reproducibility and updates without floating tags.</li>
<li><strong>Commit lockfiles and fail CI if they are out of sync</strong> with the manifest.</li>
<li><strong>Require review for anything that changes the pipeline itself</strong>: CI workflows, the updater config, Dockerfiles. Use code owners so auto-merge never covers those paths.</li>
</ul>
<h2>Throughput: keeping the queue small</h2>
<p>A healthy pipeline has a short queue. Cap concurrent PRs (<code>open-pull-requests-limit</code> in Dependabot, <code>prConcurrentLimit</code> in Renovate), schedule non-security runs for a fixed window so they don't compete with feature work all week, and group aggressively for low-risk classes. If majors pile up, give them a recurring slot in the sprint. A major left alone for a year usually becomes a project.</p>
<h2>Checklist</h2>
<ul>
<li>Policy written and checked into the repo</li>
<li>Every ecosystem covered, including CI actions, container images and infrastructure providers</li>
<li>Low-risk classes grouped, majors separate</li>
<li>Minimum release age set for non-security updates, security updates exempt</li>
<li>CI installs from lockfile, type-checks, builds and smoke-tests</li>
<li>Integration tests use the real library, not mocks</li>
<li>Auto-merge limited to classes the policy allows, never covering CI or updater config</li>
<li>CI tokens least-privilege, no deploy secrets in PR builds</li>
<li>Actions and images pinned by SHA or digest</li>
<li>Every pin has a reason, an owner and a review date</li>
<li>Dependency-only deploys tagged in monitoring</li>
</ul>
<h2>Common mistakes</h2>
<ul>
<li><strong>Turning on the bot with defaults across a large repo.</strong> You get a flood of PRs and a team that learns to ignore them. Start with grouping and limits.</li>
<li><strong>Auto-merging on weak CI.</strong> If your tests don't touch the dependency, green means nothing.</li>
<li><strong>Ignoring non-application ecosystems.</strong> Outdated CI actions and base images are dependencies too.</li>
<li><strong>Waiting on security fixes the same as everything else.</strong> The release-age delay is for routine updates. Advisories need a fast lane.</li>
<li><strong>Permanent pins.</strong> A pin with no owner and no exit condition is debt that only grows.</li>
</ul>
<p>If your team inherited a codebase years behind on dependencies, that catch-up is usually a project of its own before any automation pays off. It's the kind of work we do in our <a href="https://disruptivestudio.com/custom-software">custom software development and maintenance</a> engagements.</p>
<h2>Frequently asked questions</h2>
<h3>Should I auto-merge production dependency updates?</h3>
<p>Only for classes your policy marks as low risk, typically patch updates of trusted packages, and only when CI exercises the real library. Minors and majors in production code should get human review.</p>
<h3>How long should the minimum release age be?</h3>
<p>There is no universal number. It's a trade-off between catching bad releases and staying current. Start with a few days for routine updates, exempt security fixes, and adjust based on what your team sees.</p>
<h3>Dependabot or Renovate?</h3>
<p>Dependabot if you're on GitHub with a simple policy and want nothing to host. Renovate if you need multiple Git platforms, monorepo support or a detailed policy expressed in one file.</p>
<hr />
<p><em>Originally published on <a href="https://disruptivestudio.com/blog/automated-dependency-updates-pipeline">Disruptive Studio</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[Load Testing and Graceful Degradation Before a Black Friday Sale]]></title><description><![CDATA[You cannot fully predict peak traffic for a sale, and adding servers does not protect you from a slow payment provider or a search service that falls over…]]></description><link>https://disruptivestudio.hashnode.dev/load-testing-and-graceful-degradation-before-a-black-friday-sale</link><guid isPermaLink="true">https://disruptivestudio.hashnode.dev/load-testing-and-graceful-degradation-before-a-black-friday-sale</guid><category><![CDATA[webdev]]></category><category><![CDATA[performance]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Testing]]></category><dc:creator><![CDATA[Daniel Cruz]]></dc:creator><pubDate>Tue, 10 Nov 2026 03:45:09 GMT</pubDate><enclosure url="https://storage.googleapis.com/ariani-user-images-production/resources/organic-covers/en/payments-ecommerce-01.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2>Plan for failure, not just for capacity</h2>
<p>You cannot fully predict peak traffic for a sale, and adding servers does not protect you from a slow payment provider or a search service that falls over. A better goal is a system that loses features in a planned order instead of failing all at once. That takes two kinds of work: load tests that reflect how real buyers behave, and degradation switches you have rehearsed before the sale.</p>
<h2>Model the traffic you will actually get</h2>
<p>Many load tests use a fixed number of virtual users that loop through a script. That is a <em>closed</em> model: when your system slows down, the test users wait, so they send fewer requests. Real shoppers do not wait for each other. When an email campaign goes out, new visitors keep arriving no matter how slow the site is.</p>
<p>An <em>open</em> model, which sets an arrival rate instead of a user count, shows this behavior. In k6 that is the <a href="https://grafana.com/docs/k6/latest/using-k6/scenarios/executors/ramping-arrival-rate/"><code>ramping-arrival-rate</code> executor</a>:</p>
<pre><code class="language-javascript">export const options = {
  scenarios: {
    campaign_spike: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 2000,
      stages: [
        { target: 10,  duration: '5m' },   // baseline
        { target: 300, duration: '1m' },   // email lands
        { target: 300, duration: '15m' },  // sustained
        { target: 50,  duration: '5m' },
      ],
    },
  },
  thresholds: {
    'http_req_duration{flow:checkout}': ['p(95)&lt;1500'],
    'http_req_failed{flow:checkout}': ['rate&lt;0.01'],
  },
};
</code></pre>
<p>The numbers above are placeholders. Base yours on your own analytics from past campaigns, then test at a multiple of that peak.</p>
<h2>Weight the journeys realistically</h2>
<p>A sale is not uniform traffic. Most sessions browse, fewer add to cart, and fewer still pay. Build your scenarios as weighted flows:</p>
<ul>
<li><strong>Browse:</strong> home page, category, product detail page, search.</li>
<li><strong>Cart:</strong> add item, update quantity, apply a coupon.</li>
<li><strong>Checkout:</strong> shipping quote, tax calculation, payment intent, confirmation.</li>
</ul>
<p>Tag each request by flow so thresholds apply to the paths that make money. A slow category page is annoying. A slow payment step costs you sales. Test against a staging environment with production-sized data. A catalog with fifty products will hide every slow query you have.</p>
<h2>Mock third parties, then test them separately</h2>
<p>Your checkout probably calls a payment gateway, a shipping rate API, a tax service, a fraud check, and an email provider. Do not load test those vendors' production systems without permission. Instead:</p>
<ol>
<li>Replace each one with a stub that can be configured for latency and error rate.</li>
<li>Run the main test with healthy stubs to find <em>your</em> limits.</li>
<li>Run it again with one stub set to slow (several seconds) and then failing. Watch what happens to your connection pools and worker threads.</li>
</ol>
<p>Step 3 is where most outages come from. One slow dependency with no timeout keeps every worker busy, and then the healthy pages fail too.</p>
<h2>Put a budget on every outbound call</h2>
<p>Every external call needs an explicit timeout, a limited retry policy, and a fallback. Write them down as a table the whole team can review:</p>
<pre><code class="language-yaml">dependencies:
  shipping_rates:
    timeout_ms: 1200
    retries: 1
    fallback: flat_rate_table
  tax_service:
    timeout_ms: 1000
    retries: 1
    fallback: precomputed_rates_by_region
  recommendations:
    timeout_ms: 300
    retries: 0
    fallback: hide_widget
  payment_gateway:
    timeout_ms: 10000
    retries: 0          # never blind-retry a charge
    fallback: show_retry_message
</code></pre>
<p>Like the load profile, these values are examples. The important part is that each dependency has a decision made in advance. Add a circuit breaker around non-critical calls so that after repeated failures you stop calling the service and go straight to the fallback for a cooldown period.</p>
<h2>Build kill switches and rank them</h2>
<p>Decide which features you are willing to turn off, and in what order, before you are under pressure:</p>
<table>
<thead>
<tr>
<th>Level</th>
<th>Turn off</th>
<th>Why it is safe</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Personalized recommendations, recently viewed</td>
<td>Decorative; heavy on reads</td>
</tr>
<tr>
<td>2</td>
<td>Live search suggestions, faceted filters</td>
<td>Fall back to cached category pages</td>
</tr>
<tr>
<td>3</td>
<td>Real-time stock counts on listing pages</td>
<td>Show "in stock" / "sold out" from cache</td>
</tr>
<tr>
<td>4</td>
<td>Account pages, order history</td>
<td>Not needed to buy</td>
</tr>
<tr>
<td>5</td>
<td>Waiting room at the entry point</td>
<td>Protects checkout for people already in</td>
</tr>
</tbody></table>
<p>Build each switch as a runtime flag that does not require a deploy:</p>
<pre><code class="language-python">def product_page(product_id):
    page = cache.get_or_render(product_id)
    if flags.enabled("recommendations"):
        page.recommendations = recs.fetch(product_id, timeout=0.3, default=[])
    if flags.enabled("live_stock"):
        page.stock = inventory.read(product_id)
    else:
        page.stock = cache.stock_badge(product_id)
    return page
</code></pre>
<p>Then rehearse. During a load test, flip each switch and confirm that latency drops and nothing breaks. A switch that has never been tested may not work when you need it.</p>
<h2>Cache browsing, protect checkout</h2>
<p>Browsing traffic is mostly the same pages read again and again. Serve product and category pages from a CDN or full-page cache with short expiry times. Use stale-while-revalidate behavior if your platform supports it, so an expired page still serves instantly while it refreshes in the background. Keep checkout, cart and anything tied to a session out of shared caches.</p>
<p>Pre-warm the cache for sale landing pages and doorbuster products before the campaign goes out. A cold cache plus a traffic spike can overload your origin servers in the first minute.</p>
<h2>Know your stop signals</h2>
<p>Agree on the metrics that trigger each degradation level, and who has authority to flip the switches:</p>
<ul>
<li>Checkout p95 latency above your threshold for several minutes.</li>
<li>Database connection pool saturation.</li>
<li>Payment error rate above its normal range.</li>
<li>Queue depth for order-confirmation emails growing without draining.</li>
</ul>
<p>Put these on a single dashboard and assign an on-call owner for the sale window.</p>
<p>This covers the engineering side of being ready for traffic. For the business-side list of payments, inventory and customer support, <a href="https://disruptivestudio.com/blog/prepare-website-black-friday-checklist">the full guide</a> from Disruptive Studio is a good companion. Run the tests, rank your switches and rehearse them, so the sale day itself goes quietly.</p>
]]></content:encoded></item></channel></rss>