Rajandran R Creator of OpenAlgo Sketchmaker, RagZ. Building GenAI Applications. Telecom Engineer turned Full-time Derivative Trader. Mostly Trading Nifty, Banknifty, High Liquid Stock Derivatives. Trading the Markets Since 2006 onwards. Using Market Profile and Orderflow for more than a decade. Designed and published 100+ open source trading systems on various trading tools. Now the recent curiosity on building Trading AI Agents.

OpenEngine – Building Your First Event-Driven Backtesting Engine: A Step-by-Step Research Note

9 min read

An event driven backtesting engine replays historical data one bar (or tick) at a time and passes each update through the same chain a live system uses: market event, strategy signal, order, fill, portfolio update. Because the strategy only ever sees data up to the current event, look-ahead bias is much harder to introduce, and the same code can switch from simulated fills to a real broker API. This research note lays out how we plan to build OpenEngine on that design, with DuckDB for 1-minute data and the OpenAlgo PlaceOrder API for execution.

Algorithmic trading requires a realistic and robust simulation environment. This note describes how we are planning to build an open source, event‑driven backtesting engine optimized for 1‑minute data using DuckDB for fast historical data storage and querying. In addition, it details the integration of a broker API for order execution, using the OpenAlgo PlaceOrder API, as an example of how to enable live trading with minimal code changes. This modular, unified architecture helps ensure that your strategy’s behavior remains consistent from simulation to production.

OpenEngine - Building Your First Event-Driven Backtesting Engine: A Step-by-Step Research Note
Key takeaways
  • Event-driven engines process data bar by bar through market, signal, order and fill events, mirroring live trading.
  • Vectorised backtests are faster for research; event-driven ones are better when the same code must trade live.
  • DuckDB stores and queries years of 1-minute candles locally, fast enough for replay without a database server.
  • Swapping a simulated broker class for an OpenAlgo PlaceOrder client moves the engine from backtest to live.
  • A minimal working engine needs only five small classes and an event queue (example below).

1. Introduction

Before deploying trading strategies live, a high‑quality backtesting engine is essential for understanding performance under real market conditions. In this note, we explain why an event‑driven approach is advantageous compared to a purely vector‑based method, especially when using granular 1‑minute data. We also demonstrate how to integrate with a broker’s order placement API (OpenAlgo PlaceOrder) to seamlessly transition from simulation to live execution.

Python for traders, the quick way in

BEGINNER FRIENDLY · RATED 5.0

Python for traders, the quick way in

A crash course that gets you from zero to writing your own market data scripts.

  • Python basics with market examples
  • Fetch, clean and chart price data
  • Lifetime access
Start learning →by Marketcalls

2. Background and Motivation

2.1 Vector-Based vs. Event-Driven Backtesting

  • Vector-Based Backtesting:
    • Overview: Processes large data batches using vectorized operations (e.g., with pandas or NumPy).
    • Pros: High performance and simplicity for statistical operations.
    • Cons: Not ideal for simulating the sequential and dynamic aspects of live trading (e.g., latency, partial fills, order rejections).
  • Event-Driven Backtesting:
    • Overview: Simulates trading by processing discrete events (market data updates, signals, orders, fills) in order.
    • Pros: Closely mirrors live trading, supports stateful strategies, and unifies simulation and live execution within the same framework.
    • Cons: More architecturally complex and requires careful management of events, though this is usually manageable at a 1‑minute resolution.

Given the requirement for a unified codebase for both backtesting and live trading, the event‑driven approach is strongly recommended.

2.2 Why Use DuckDB?

DuckDB is an embeddable, high‑performance OLAP database that:

  • Provides Speed: Efficiently processes large volumes of columnar data such as 1‑minute historical market data.
  • Is Simple to Integrate: With its Python API, it allows you to store, query, and manage data with minimal setup.
  • Scales Well: Suitable for local testing and scalable enough for larger production deployments.

3. Key Components of the Backtesting Engine

A modular, event‑driven backtesting engine should consist of the following components:

  1. Data Handling and Storage:
    • Historical Data Loader: Import 1‑minute data into DuckDB.
    • DuckDB Integration: Use SQL queries to efficiently retrieve and process data during simulation.
    • Real-Time Data Feed: (For live trading) Ingest live data via APIs or WebSockets.
  2. Event System:
    • Event Types: Define events such as Market Data Events, Signal Events, Order Events, Fill Events, and Portfolio Update Events.
    • Event Queue and Dispatcher: Ensure events are processed sequentially and routed to the appropriate engine modules.
  3. Strategy Engine:
    • Signal Generation: Process incoming market events to generate buy/sell signals.
    • State Management: Maintain necessary context for sequential or stateful strategies.
  4. Order Management and Execution:
    • Order Manager: Translate signals into orders, simulate order fills in backtesting, and handle order status updates.
    • Broker Interface: Abstract the connection to broker APIs, ensuring that the same interface is used for both backtesting and live trading.
    • API Integration: In live trading, use endpoints like the OpenAlgo PlaceOrder API to send orders to the broker.
  5. Portfolio and Risk Management:
    • Position Sizing and Allocation: Determine trade sizes based on risk controls.
    • Risk Controls: Enforce stop-loss, take-profit, and other risk management rules.
    • Portfolio Valuation: Continuously update positions and track performance metrics (drawdowns, Sharpe ratio, etc.).
  6. Reporting and Analytics:
    • Metrics Calculation: Generate performance reports and visualizations.
    • Logging: Maintain detailed logs for debugging, auditing, and performance analysis.
  7. Configuration and Environment Management:
    • User Settings: Enable customization of strategy parameters, capital allocation, and risk limits.
    • Unified Environment: Ensure that settings work identically in both backtesting and live modes.

4. Integrating Broker API: OpenAlgo PlaceOrder Function

To execute orders in live trading (and even simulate them during backtesting), you need a robust API interface. Here’s an overview of the OpenAlgo PlaceOrder API function:

4.1 API Details

  • Endpoint URL:
    • Local Host: POST http://127.0.0.1:5000/api/v1/placeorder
    • Ngrok Domain: POST https://<your-ngrok-domain>.ngrok-free.app/api/v1/placeorder
    • Custom Domain: POST https://<your-custom-domain>/api/v1/placeorder
  • Request JSON Format:
{
    "apikey": "<your_app_apikey>",
    "strategy": "Test Strategy", 
    "symbol": "SAIL", 
    "action": "BUY", 
    "exchange": "NSE", 
    "pricetype": "MARKET", 
    "product": "MIS", 
    "quantity": "1",
    "price": "0",
    "trigger_price": "0",
    "disclosed_quantity": "0"
}

Parameter Descriptions:

ParameterDescriptionMandatory/OptionalDefault Value
apikeyApp API keyMandatory–
strategyStrategy nameMandatory–
exchangeExchange codeMandatory–
symbolTrading symbolMandatory–
actionBUY or SELLMandatory–
productProduct typeOptionalMIS
pricetypePrice typeOptionalMARKET
quantityOrder quantityMandatory–
pricePriceOptional0
trigger_priceTrigger priceOptional0
disclosed_quantityDisclosed quantityOptional0

Sample API Response:

{
    "orderid": "240307000614705",
    "status": "success"
}

Update (2026): the current OpenAlgo documentation lists the same eleven request fields, with pricetype accepting MARKET, LIMIT, SL and SL-M and product accepting MIS, CNC and NRML. The response can also carry a mode field, which is "analyze" when the order runs in OpenAlgo’s sandbox instead of going to the broker (OpenAlgo PlaceOrder docs). That sandbox is a useful middle step between backtest and live.

4.2 Integration in the Order Manager

When designing your Order Manager module, abstract the execution of orders so that in a live trading scenario, it calls the API. In backtesting, you might simulate the API call or use a similar interface for consistency. For example:

import requests

class BrokerInterface:
    def __init__(self, base_url, apikey):
        self.base_url = base_url
        self.apikey = apikey

    def place_order(self, strategy, symbol, action, exchange, product="MIS",
                    pricetype="MARKET", quantity="1", price="0", trigger_price="0", disclosed_quantity="0"):
        endpoint = f"{self.base_url}/api/v1/placeorder"
        payload = {
            "apikey": self.apikey,
            "strategy": strategy,
            "exchange": exchange,
            "symbol": symbol,
            "action": action,
            "product": product,
            "pricetype": pricetype,
            "quantity": quantity,
            "price": price,
            "trigger_price": trigger_price,
            "disclosed_quantity": disclosed_quantity
        }
        response = requests.post(endpoint, json=payload)
        if response.status_code == 200:
            return response.json()
        else:
            raise Exception("Order placement failed", response.text)

In your live trading engine, you would call this method when an order signal is generated. In backtesting, you can simulate this call to log the order details and update simulated portfolio values.

5. Step-by-Step Process for Building the Engine

Step 1: Data Acquisition and Storage

  • Collect Historical Data:
    Obtain your 1‑minute historical market data from Yahoo Finance or other providers.
  • Integrate DuckDB:
    Create a schema to store the data and load it using DuckDB’s Python API:
import duckdb
con = duckdb.connect('market_data.duckdb')
con.execute("""
  CREATE TABLE IF NOT EXISTS market_data (
      timestamp TIMESTAMP,
      open DOUBLE,
      high DOUBLE,
      low DOUBLE,
      close DOUBLE,
      volume BIGINT
  )
""")
con.execute("COPY market_data FROM '1min_data.csv' (FORMAT CSV, HEADER)")

If you need years of 1-minute data rather than a single CSV, OpenAlgo Historify downloads broker candles into a local DuckDB database and can export CSV, TXT, ZIP and Parquet (Historify docs). I walk through it in how to download 10 years of intraday data in CSV format, and DuckDB basics are covered in the DuckDB tutorial for traders.

Step 2: Design the Event‑Driven Architecture

  • Define Event Types:
    Create classes (or named tuples) for events such as MarketDataEvent, SignalEvent, OrderEvent, and FillEvent.
  • Implement the Event Queue and Dispatcher:
    Use Python’s built‑in libraries (e.g., queue.Queue) to manage event sequencing.

Step 3: Develop the Strategy Module

  • Base Strategy Interface:
    Define an abstract strategy class to generate signals.
  • Concrete Strategy Implementation:
    For instance, a moving average crossover strategy that processes each 1‑minute market event and outputs a signal.
  • State Management:
    Ensure your strategy maintains any necessary state (e.g., moving averages, current positions).

Step 4: Implement Order Management and Execution Simulation

  • Order Manager Module:
    Build a component to handle signal-to-order translation, using the BrokerInterface (as shown above) to place orders.
  • Broker Interface Integration:
    Integrate the OpenAlgo PlaceOrder API so that order execution is identical for live and simulated environments.

Step 5: Build Portfolio and Risk Management Components

  • Portfolio Manager:
    Track positions, cash balances, and overall portfolio value.
  • Risk Engine:
    Incorporate stop-loss, take-profit, and maximum exposure controls.

Step 6: Integrate the Simulation Loop

  • Data Replay:
    Develop a loop that reads 1‑minute data from DuckDB, converts each row into a market data event, and processes it through your event‑driven architecture.
  • Live Simulation:
    Ensure that the loop is capable of running in real time, processing live events and calling the order placement API when required.

Step 7: Reporting, Analytics, and Logging

  • Performance Metrics:
    Calculate cumulative returns, drawdowns, and other key indicators.
  • Visualization:
    Use libraries such as Matplotlib or Plotly to visualize equity curves and trade performance.
  • Logging:
    Implement detailed logging to trace events, order placements, and strategy decisions.

Step 8: Transitioning to Live Trading

  • Unified Engine Design:
    Ensure that your event‑processing loop and core modules remain identical whether processing historical data or live feeds.
  • API Integration:
    Replace simulated order fills with real API calls (e.g., using the BrokerInterface to call the OpenAlgo PlaceOrder endpoint).
  • Testing:
    Rigorously test in a paper trading or simulated environment before full live deployment.

A Minimal Event Driven Backtesting Engine in Python

To make the architecture concrete, here is a small working version of Steps 2 to 6. It reads the market_data table created in Step 1, runs a 20/50 moving average crossover, and fills orders on the next bar with slippage. The SimulatedBroker is the only class you replace to go live; its send() method would call the BrokerInterface.place_order() shown in section 4.2.

from collections import deque
from dataclasses import dataclass
import duckdb


@dataclass
class MarketEvent:
    ts: object
    close: float


@dataclass
class SignalEvent:
    ts: object
    action: str          # "BUY" or "SELL"


@dataclass
class OrderEvent:
    ts: object
    action: str
    quantity: int


@dataclass
class FillEvent:
    ts: object
    action: str
    quantity: int
    price: float


class SmaCross:
    """Strategy: reacts only to MarketEvents, emits SignalEvents."""
    def __init__(self, fast=20, slow=50):
        self.fast, self.slow, self.closes, self.pos = fast, slow, deque(maxlen=slow), 0

    def on_market(self, ev):
        self.closes.append(ev.close)
        if len(self.closes) < self.slow:
            return None
        f = sum(list(self.closes)[-self.fast:]) / self.fast
        s = sum(self.closes) / self.slow
        if f > s and self.pos <= 0:
            self.pos = 1
            return SignalEvent(ev.ts, "BUY")
        if f < s and self.pos > 0:
            self.pos = 0
            return SignalEvent(ev.ts, "SELL")
        return None


class SimulatedBroker:
    """Backtest execution: fills at the next bar's close plus slippage.
    Swap this class for one that calls OpenAlgo /api/v1/placeorder in live trading."""
    def __init__(self, slippage=0.0005):
        self.slippage, self.pending = slippage, []

    def send(self, order):
        self.pending.append(order)

    def on_market(self, ev):
        fills = []
        for o in self.pending:
            px = ev.close * (1 + self.slippage if o.action == "BUY" else 1 - self.slippage)
            fills.append(FillEvent(ev.ts, o.action, o.quantity, px))
        self.pending = []
        return fills


def run(db_path="market_data.duckdb", qty=1):
    con = duckdb.connect(db_path, read_only=True)
    rows = con.execute(
        "SELECT timestamp, close FROM market_data ORDER BY timestamp").fetchall()
    strategy, broker, events = SmaCross(), SimulatedBroker(), deque()
    cash, position, trades = 0.0, 0, []
    for ts, close in rows:
        events.append(MarketEvent(ts, close))
        while events:
            ev = events.popleft()
            if isinstance(ev, MarketEvent):
                events.extend(broker.on_market(ev))   # fill orders from the previous bar
                sig = strategy.on_market(ev)
                if sig:
                    events.append(sig)
            elif isinstance(ev, SignalEvent):
                broker.send(OrderEvent(ev.ts, ev.action, qty))
            elif isinstance(ev, FillEvent):
                sign = 1 if ev.action == "BUY" else -1
                position += sign * ev.quantity
                cash -= sign * ev.quantity * ev.price
                trades.append(ev)
    last = rows[-1][1] if rows else 0
    return {"trades": len(trades), "position": position, "pnl": round(cash + position * last, 2)}


if __name__ == "__main__":
    print(run())

Notice the order of operations inside the loop: pending orders are filled on the new bar before the strategy sees it, so a signal on bar N can only execute on bar N+1. That one rule removes the most common source of look-ahead bias in hand-written backtests. For a deeper look at how the queue works, see event driven backtesting and event loops for traders.

Event Driven vs Vectorised Backtesting at a Glance

VectorisedEvent driven
SpeedVery fast (whole arrays at once)Slower (one event at a time)
Look-ahead bias riskHigher; easy to use future rows by mistakeLower; strategy sees only past events
Order and fill realismApproximateCan model slippage, partial fills, order types
Same code for live tradingRarelyYes, by swapping the broker component
Best useParameter sweeps and idea screeningFinal validation and production

Common Mistakes When Building a Backtesting Engine

  • Filling on the signal bar’s close. If the signal uses the close, the fill must come later.
  • Ignoring costs. Brokerage, STT, exchange charges and slippage change the result for 1-minute strategies more than for daily ones.
  • Mixing adjusted and unadjusted data. Corporate actions in stocks and contract rolls in futures need consistent handling.
  • Session boundaries. Indicators and intraday exits should reset at 09:15 and square off before the broker’s cut-off.
  • No logging. Log every event so a backtest trade can be traced and compared with live fills.

For a checklist that applies to any backtest, read 7 things a trader should understand to do proper backtesting.

6. Discussion and Future Directions

By following this detailed, step‑by‑step process and leveraging an event‑driven design, you can build a backtesting engine that mirrors live trading conditions. Integration with DuckDB provides efficient data handling for granular 1‑minute data, and incorporating the OpenAlgo PlaceOrder API ensures a smooth transition to live execution.

Future Enhancements Might Include:

  • Support for sub‑minute or tick data.
  • Advanced order types and more realistic execution simulations.
  • Asynchronous event processing to further reduce latency.
  • Integration of machine learning models for dynamic strategy adjustments.

7. Summary and Next Steps

Developing a unified backtesting and live trading engine is challenging yet rewarding. By building an event‑driven system with robust data handling via DuckDB and seamless order execution through the OpenAlgo PlaceOrder API, you can create a framework that minimizes discrepancies between simulated and live performance. This research note has outlined the architectural considerations and practical steps needed to get started, providing a solid foundation for building and refining your trading engine.

Frequently Asked Questions

What is event driven backtesting?

It is a backtest that replays data as a sequence of events (new bar, signal, order, fill) and updates the strategy and portfolio after each one, the same way a live trading system runs.

Is event driven backtesting better than vectorised backtesting?

Not always. Vectorised tests are far faster for screening ideas. Event-driven tests are more realistic and let you reuse the code for live trading, so many traders use both.

Why use DuckDB for backtesting data?

DuckDB is an embedded columnar database: no server to run, fast SQL over millions of rows, and direct reading of CSV and Parquet files, which suits 1-minute market data.

Can an event driven backtesting engine trade live with OpenAlgo?

Yes. Replace the simulated broker with a class that calls the OpenAlgo PlaceOrder endpoint, and feed live bars or quotes into the same event queue.

Python Elite membership

MEMBERSHIP

Python Elite membership

15 Python, AI and algo trading programs in one plan, including QuantFlow and AI Bootcamp live mentorship.

  • Python from fundamentals to trading apps
  • Backtesting, broker APIs and OpenAlgo
  • Live QuantFlow and AI Bootcamp sessions
View Python Elite →by Marketcalls

Learn by watching: free videos every week

Market Profile, order flow, Python, AI and OpenAlgo tutorials on YouTube.

Rajandran R Creator of OpenAlgo Sketchmaker, RagZ. Building GenAI Applications. Telecom Engineer turned Full-time Derivative Trader. Mostly Trading Nifty, Banknifty, High Liquid Stock Derivatives. Trading the Markets Since 2006 onwards. Using Market Profile and Orderflow for more than a decade. Designed and published 100+ open source trading systems on various trading tools. Now the recent curiosity on building Trading AI Agents.

Leave a Reply

Get Notifications, Alerts on Market Updates, Trading Tools, Automation & More