Rehearsing the crash: why we built Scenario Driver

Every lending protocol makes the same promise: if a borrower's collateral falls too far, someone will step in, repay the debt and take the collateral before the protocol is left holding a loss. The on-chain program enforces the rules of that deal. But the program does not step in. A bot does.

That bot, a liquidator, is a piece of software somebody runs on a server. It watches thousands of positions, notices when one crosses the line, builds the right transaction, picks a fee, sends it, and retries when the network swallows it. Next to it sit its cousins: the keeper that pulls a vault's deposits out of a lending market when it smells trouble, the cranker that pushes fresh prices on-chain, the filler that settles orders, the solver that completes a cross-chain transfer. Together they are the off-chain half of a protocol, and for most of decentralised finance (DeFi) they are where risk stops being a formula and becomes a question: did the bot work?

It is a question that only matters on the worst day. In the February 2026 crash, Kamino's own monthly risk report counts 66,848 liquidation events in three days on a single protocol. By our count, more than 26,000 of them landed in one hour. That hour is the one the bot exists for, and it is the one hour nobody can book in advance on mainnet.

We built Scenario Driver to rehearse that hour on demand.

The half of the protocol nobody stress-tests

A smart contract today gets audited, fuzzed, sometimes formally verified. A liquidator gets deployed, watched on a quiet Tuesday, and trusted. The gap is not carelessness. It is that the conditions that break off-chain components are exactly the conditions you cannot create on purpose:

• Prices that move 20 percent in minutes, across every feed the protocol reads and every feed the bot reads, and not necessarily at the same moment.
• Oracles that stop. An oracle is the price feed a protocol trusts. It goes stale, or the on-chain price and the off-chain market price disagree. The protocol refuses to act on stale data. Does the bot? Does it notice when the feed comes back?
• A network that lies by omission. On Solana a transaction can be accepted by the endpoint, hand you back a signature, and never land. No error. Endpoints slow down, rate-limit, or answer "node unhealthy" at the exact moment everyone is sending at once.
• Crowds. Hundreds of positions crossing the line together, in sizes from dust to millions. Which first? At what fee? How many times?
• Time. Interest accrues, staleness windows expire, epochs roll over. Logic that depends on the clock cannot be tested without moving the clock.
• Drift. A bot compiled against last year's version of the protocol, or pointed at a price service that quietly shut down months ago.

Each of these has caused real losses somewhere, and the only way most teams have found to test for them is to wait.

Why the usual tests miss it

Teams are not short of testing tools. They are short of tools that test this.

• Unit tests, fuzzing and program-level simulators exercise the on-chain program with synthetic state and mocked prices. They are indispensable, and they never start the bot. There is no network endpoint for a real bot to connect to.
• Public test networks have an endpoint, but nothing real behind it: no real positions, no real liquidity, often a different version of the protocol. A liquidator's hardest job, finding the positions that matter among a hundred thousand, has nothing to find.
• Running the bot against mainnet in dry-run mode is the most common practice, and it does have real state. What it lacks is a crash. You wait for the market to produce one, you learn afterwards, and the one path you never exercise is the send path, because sending for real is the thing you were trying to avoid.
• Risk-modelling platforms simulate markets with hypothetical agents to recommend parameters. They answer "what should the liquidation bonus be?", not "does my bot land the transaction when the endpoint is dropping half of them?".
• Chaos-engineering tools from the web world inject faults very well, but they do not know what an oracle is, what a position is, or whether the liquidation that landed was actually correct.

Each existing approach covers part of the problem. None covers the combination.

What is missing is one place that combines five things: the real state of mainnet; control over prices, time and the network path; the customer's actual bot running unmodified; protocol-aware verification of what happened on-chain; and the ability to run the whole thing again tomorrow and get the same answer.

What Scenario Driver does

Scenario Driver runs a scripted market scenario on a private copy of mainnet while your off-chain component runs against that copy, and then checks what happened. A scenario is a short text file. In plain terms, a run looks like this:

1. Start from reality. A local copy of the chain, a fork, is started from the live network: the real protocol, the real markets, the real positions, running on a machine where nothing you do can touch mainnet.
2. Set the stage. Seed the prices you want, open positions of chosen sizes and leverage, fund the actors. If you want thousands of real positions in view so the bot's scans look like production, load them.
3. Start the bot. Your component starts unmodified, with its normal configuration, pointed at the fork. Between the two sits a proxy we control.
4. Play the timeline. A list of ticks: move a price (on every oracle the protocol reads and on the off-chain feeds the bot reads, in the right order), jump the clock forward, freeze a feed, drop or delay or fail requests for a window, then wait and check.
5. Check against the chain. "This position becomes liquidatable within fifteen seconds." "A liquidation of this position lands within two minutes, repays the right debt, receives collateral inside the bonus bounds, and leaves the position healthier." "The bot logged that it skipped the position before it logged that it sent." Assertions are answered by the protocol's own program and by the token movements on the fork, not by taking the bot's word for it.
6. Read the report. Every tick with its outcome, every decision the bot logged, every fault the proxy applied, the on-chain accounting of every liquidation, and a verdict. The report is written even when the scenario fails. Especially when it fails, since that is when you need to see what the bot was doing.

A scenario reads roughly like this (shortened):

name: crash-with-dropped-sends
chain: solana
protocol: { kind: kamino, market: main }
subject: { kind: my-liquidator }          # how to start and watch your bot
setup:
  seed_prices: { SOL: market, USDC: "1" }
  positions:
    - id: alice
      collateral: { token: SOL, amount: "100e9" }
      debt: { token: USDC, utilization: "0.70" }
ticks:
  - expect_position: { id: alice, healthy: true }
  - fault:                                  # on the endpoint the bot calls
      rpc: [ { match: { method: sendTransaction }, action: { drop: {} }, for: 20s } ]
  - set_price: { symbol: SOL, usd: "market * 0.80" }
  - expect_liquidation: { id: alice, within: 180s }

Read it top to bottom and you have the whole story: open a healthy position, make the network swallow every transaction for twenty seconds, crash the collateral by a fifth, and insist that the bot still gets the liquidation through within three minutes.

How we designed it

Our first version of this idea was a one-off test rig built for a customer on Ethereum: a forked chain, a controllable price oracle, their liquidator, and a list of timed steps. It found real bugs, and it was welded to one protocol. Scenario Driver is what we built when we decided the idea deserved to be a product. A few principles shaped it more than any feature.

The core knows nothing specific. The engine that runs timelines, moves prices, injects faults and writes reports never mentions a chain, a protocol, a token or a line from any bot's log. Everything specific lives at the edges. This is not tidiness for its own sake. It is what makes the second chain and the fifth protocol cheap, and it is why the first implementation being on Solana says nothing about where the product stops.

Four things vary, so there are four kinds of plug-in. How a chain is forked, how its clock moves, how state is written and observed: that is a chain driver (Solana today, built on the excellent open-source fork tooling each ecosystem already has; the Ethereum Virtual Machine next). How that chain's oracles, tokens and off-chain price feeds work: ecosystem adapters. What a position is, when it is liquidatable, how a liquidation is verified, which prices a market reads: a protocol plug-in. And how to start your bot, how to tell it is ready and how to read its logs: a subject manifest, a short description file, usually with no code at all.

Leave the bot alone. We do not patch customers' software to make it testable. We take control of the things the protocol trusts on the fork instead: the price accounts, the admin keys, the feeds. The bot talks to the fork through a proxy, so faults can be switched on and off in the middle of a run without restarting anything, and every fault that was applied is recorded next to the bot's reaction to it. When a component does need a change (a retired endpoint, say), the change is an explicit, reviewable patch file, never a silent edit.

The chain is the source of truth. Logs tell us what the bot thought; the chain tells us what happened. Position health is read from the protocol's own program, not recomputed by us, so we cannot disagree with the protocol about who is liquidatable. A liquidation counts only when the token movements on the fork show the debt repaid, the collateral received inside the protocol's bonus rules, the protocol's fee collected and the position healthier afterwards.

Every run is fresh, isolated and validated before it costs anything. A new fork and a new bot process per scenario; time only moves forward; runs get their own ports and directories, so several can run side by side. Scenario files are checked against every plug-in's rules before a fork is started, because a fork costs real minutes and real network calls, and a typo should not.

Runs must be repeatable. On Ethereum you can fork at a block number and get the same world every time. On Solana the standard node endpoints only serve the present, so two runs on different days start from different states. We solved this by recording everything a run fetched from the live network into a fixture, and replaying it later with no network at all. The actors' keys are derived from a seed, so every address matches, and two replays of the same fixture produce reports that compare equal. That is the difference between a demonstration and a check you can run on every commit of your bot.

A first Solana test case: Kamino

To test our recent extension of Scenario Driver to support Solana, we used Kamino, one of the largest lending protocols on Solana. To clarify: Kamino is not a Stateless Labs customer (yet) and was chosen because its programs are public and verified against mainnet and because it has published two off-chain components we could use as test subjects: a liquidator (a 2024 snapshot released under an open license) and the vault keeper that manages its Earn vaults. These are examples. Every one of them is a pattern that applies to any protocol with a bot in the loop. All tests and runs never touch mainnet and are conducted strictly on local forks.

The plain crash. Open a position, drop the collateral price by a fifth, expect the bot to liquidate it within a window, verify the accounting. This is the baseline. The published liquidator, once we had revived it (more on that below), handled it within one scan interval, two to three seconds after the crash.

Stale oracle, then recovery. The crash lands, then the price feed stops and the clock jumps ten minutes. A correct bot must not send: the protocol would reject the transaction, and a bot that keeps sending anyway is burning fees and attention. When the feed resumes, the bot must act. The scenario asserts the refusal and the recovery, in that order. Monthly risk reports discuss liquidations at length. We have yet to see one discuss how bots behaved while prices were stale.

Dropped transactions. For twenty seconds every send is accepted, returns a signature, and never lands, which is exactly what congestion looks like on Solana. Does the bot's resend loop carry the liquidation through once the window closes, or does it trust its own signature and move on?

A slow endpoint. Three hundred milliseconds added to every request. The bot gets slower; it must still land inside the window.

The crowded crash. Three positions of different sizes and leverage cross the line together. All must be liquidated, and the report shows the order the bot chose and how long each took. Scale this to thousands and you have a replay of a real crash.

A vault keeper under danger. The keeper's job is to move depositors' funds between lending reserves and to get them out when a reserve turns risky. We push a reserve's utilisation to 98 percent in one scenario and move the on-chain price 5 percent away from the off-chain market price in another. In both, the keeper must flag the danger, zero the reserve's allocation and pull the funds out. In a third, the endpoint answers "node unhealthy" to every send for thirty seconds around a deposit; the keeper must keep retrying and invest once the endpoint recovers. The keeper ran unmodified and passed all of them, which is the kind of sentence a protocol wants to be able to say about its own bots with evidence attached.

The negative control. A liquidator with sending switched off must fail the plain-crash scenario, with its "dry run" decisions visible in the report. A test rig that cannot fail is not a test rig.

What we found along the way. Out of the box, the published liquidator snapshot could not liquidate at all: the price service it called had been retired, and the copy of the protocol's logic compiled into it had fallen a year behind the chain, so a retired field read as zero and every position in the market looked unhealthy to it. That is no criticism of Kamino, whose production systems are private and whose maintained keeper passed everything. It is the point. Dead endpoints and compiled-in logic that drifts behind the chain are the two kinds of rot that creep into any bot nobody exercises, and a scenario run surfaces both within the first scan.

Where we are taking it

Scenario Driver today is a working system that supports both EVM chains and Solana. The roadmap is driven by what customers run, roughly in this order:

• More chains. Whatever chain the next customer's bots live on.
• More kinds of component. A second lending protocol with oracle circuit breakers of its own, then components that are not about prices at all: oracle crankers, order fillers, bridge relayers, and epoch-driven keepers such as restaking crankers, where the scenario is "the epoch rolled over while you were halfway through".
• Realistic swap legs. Most serious liquidators swap the seized collateral through an aggregator in the same transaction. We are building a quote service that prices against the fork's own pool state, so swap-then-liquidate bots run unmodified, plus the ability to move the pools themselves.
• Streaming data. Bots that consume validator data streams instead of polling an endpoint need those streams served from the fork, with the same fault injection.
• Replays of real crashes. Take the shape of the February 2026 crash, thousands of positions crossing in an hour, and play it back as a scenario against your bot, with the network conditions of that hour.
• Hosted runs. The same engine behind a queue, so every commit of your bot runs the scenario suite and the report lands in your pipeline, the way unit tests do today.

Who this is for

Protocols that run their own liquidators, keepers or crankers. Independent operators who liquidate for a living and want to know how their bot behaves before the next crash rather than after. Teams building vaults and strategies that depend on a keeper reacting in time. And audit customers who want the review to extend past the contract boundary to the software that actually protects the money.

If your honest answer to "how do you know your bot works in a crash?" is "it worked last time", we would like to show you a report. Scenario Driver is the engine behind our stress testing service. Get in touch and tell us what you run.