Rajandran R Creator of OpenAlgo Sketchmaker, RagZ. Building GenAI Applications. Telecom Engineer turned Full-time Derivative Trader. Mostly Trading Nifty, Banknifty, High Liquid Stock Derivatives. Trading the Markets Since 2006 onwards. Using Market Profile and Orderflow for more than a decade. Designed and published 100+ open source trading systems on various trading tools. Now the recent curiosity on building Trading AI Agents.

Firecrawl Local: Fetch Stock Fundamental and Technical Data from Any Website with Python

20 min read

Watercolour sketch of an Indian trader's desk with a laptop chart, diya, chai and a globe sending web pages

Firecrawl Local is a single Rust binary that gives your scripts and AI agents a web fetch tool on your own machine: hand it a URL and it returns clean markdown, with no API key and no per-page credits. In this tutorial I install Firecrawl Local on Windows, pull fundamental data from Screener.in and technical data from Investing.com with Python, turn the pages into pandas tables and Plotly charts, and then connect the same binary to AI agents through MCP.

Every command, output and number below comes from a real run on my Windows 11 machine on 19 November 2026. That includes the parts that did not go smoothly, such as Investing.com blocking my fifth request, because that is the part most scraping tutorials leave out.

Key takeaways

  • Firecrawl Local is one open-source binary (AGPL-3.0) that scrapes, crawls and maps websites and serves the Firecrawl v2 API on localhost:3002. No database, no queue, no Docker.
  • cargo install firecrawl-local took about six minutes on my machine and produced a 23.6 MB executable.
  • A Screener.in company page came back as clean markdown in 245 ms. Five companies in one batch job took about 4 seconds.
  • Fetched pages live in RAM and are handed straight back to the caller. Nothing is cached on disk unless you save it. The one exception is a throwaway Chrome profile in the temp folder.
  • Investing.com served its technical page four times and then answered with an HTTP 403 bot check. Firecrawl Local flags that as "blocked": true. Treat it as a stop sign.
  • firecrawl-local mcp exposes the same engine to Claude Code, Cursor, Codex and other agents, so you can ask for fundamentals in plain English.

What is Firecrawl Local?

Firecrawl is a widely used web scraping API for AI applications. Until now you had two ways to use it: pay for the cloud API, or self-host the full stack with Docker, Redis, queues and workers. On 18 November 2026 the team announced a local edition called Firecrawl Local, a rebuild of the scraping core in Rust that ships as one executable.

Python for traders, the quick way in

BEGINNER FRIENDLY · RATED 5.0

Python for traders, the quick way in

A crash course that gets you from zero to writing your own market data scripts.

  • Python basics with market examples
  • Fetch, clean and chart price data
  • Lifetime access
Start learning →by Marketcalls

The announcement makes three claims: a scrape primitive 37 times faster than the self-hosted stack, 230 times less memory (11 MB idle) and a fully open-source codebase. On my machine a freshly started server sat at 13.5 MB, which is close to the claim for a Windows build.

That one binary gives you five things:

  • scrape: one or more URLs to markdown, HTML, raw HTML or a list of links.
  • crawl: follow links from a start URL and save one markdown file per page.
  • map: list a site’s URLs from its sitemaps.
  • serve: the Firecrawl v2 HTTP API on port 3002, so the existing Python and Node SDKs work unchanged.
  • mcp and init: an MCP server for AI agents and a one-command setup for them.

The official quickstart for Firecrawl Local documents every flag. This article is about putting it to work on market data.

How Firecrawl Local fetches a page: URL, local binary, plain fetch, local Chrome and clean markdown output

Why traders and investors should care

Most of the data we want as investors already sits on public web pages: ten years of profit and loss on Screener, shareholding patterns, peer tables, indicator summaries, concall links. Copying it into Excel by hand does not scale beyond a handful of stocks.

Classic scraping with requests and Beautiful Soup works, and I covered it in an older Beautiful Soup tutorial on Screener data. The catch is that you write and maintain a parser for every table on every site, and it breaks whenever the site changes its HTML.

Firecrawl Local changes the unit of work. The tool does the fetching, the JavaScript rendering and the cleaning, and returns markdown in which every HTML table is already a tidy pipe table. One small helper turns any of those tables into a DataFrame. The same output is what a language model reads best, which is why the tool doubles as the web fetch layer for AI agents.

  • Cost: no API key and no credits. Every local response reports creditsUsed: 0.
  • Privacy: your watchlist and the pages you research stay on your machine.
  • Speed: a static page takes a few hundred milliseconds.

What runs on your machine and what goes to the cloud

Firecrawl Local features that run on your machine versus features that need Firecrawl Cloud

The binary is called local, but it can hand some work to Firecrawl’s cloud. It helps to know the split before you start.

FeatureWhere it runsNotes
Scrape, crawl, map, batch scrapeYour machineNo API key needed
PDF and Office document parsingYour machineConverted to markdown locally
JavaScript renderingYour machineUses your installed Chrome or Chromium
Web searchFirecrawl CloudThere is no search index on a laptop
Pages that block your machineFirecrawl Cloud, optionalResult is tagged "source": "firecrawl-cloud"
JSON and summary formats, browser actionsCloud or the full self-hosted serverSkipped by the local engine

The FIRECRAWL_CLOUD environment variable controls the hand-off: auto (the default), search (only search queries go out) or off. I ran the server and all the Python tests in this article with FIRECRAWL_CLOUD=off and without a Firecrawl account, so what you see is the local engine alone.

Step 1: Install Firecrawl Local on Windows

The docs give install steps for macOS and Debian or Ubuntu. Windows is not mentioned, so I tried it: Firecrawl Local builds and runs fine on Windows 11 with the MSVC toolchain. You need:

  • Rust (stable, installed through rustup). I used Rust 1.99.1.
  • A C compiler: the Visual Studio C++ Build Tools on Windows, the Xcode command line tools on macOS, or build-essential on Ubuntu.
  • Chrome or Chromium (optional) for pages that need JavaScript.
  • Node.js 22 or newer (optional) for the MCP server.
  • Python 3 for the scripts in this article. I used Python 3.14.
cargo install firecrawl-local
firecrawl-local --version
firecrawl-local 0.1.0

The build compiled more than 350 crates and finished in 347 seconds on my machine. The result is a 23.6 MB firecrawl-local.exe in %USERPROFILE%\.cargo\bin, a folder rustup has already added to your PATH.

Step 2: Your first scrape from the command line

firecrawl-local scrape https://example.com
{
  "success": true,
  "data": {
    "markdown": "This domain is for use in documentation examples without needing permission. ...",
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com/",
      "statusCode": 200,
      "contentType": "text/html; charset=utf-8",
      "engine": "local-fetch",
      "creditsUsed": 0
    },
    "source": "local"
  }
}

Three fields are worth knowing. engine: local-fetch means a plain HTTP request was enough. source: local means the page never left your machine. creditsUsed is 0. Add --markdown to print only the content, or --format markdown,html,links to choose formats.

Now a real page. This is the consolidated Infosys page on Screener.in:

firecrawl-local scrape https://www.screener.in/company/INFY/consolidated/ --markdown |
  Out-File -Encoding utf8 infy.md

It returned 395 lines of markdown in 245 ms. An excerpt, with the quarterly table trimmed to four columns for width:

- Market Cap ₹ 4,49,983 Cr.
- Current Price ₹ 1,109
- High / Low ₹ 1,728 / 980
- Stock P/E 14.2
- Book Value ₹ 244
- Dividend Yield 4.33 %
- ROCE 40.0 %
- ROE 31.9 %

## Quarterly Results

|  | Dec 2025 | Mar 2026 | Jun 2026 | Sep 2026 |
| --- | --- | --- | --- | --- |
| Sales + | 45,479 | 46,402 | 48,211 | 49,244 |
| Operating Profit | 10,634 | 11,167 | 11,409 | 11,689 |
| OPM % | 23% | 24% | 24% | 24% |
| Net Profit + | 6,666 | 8,509 | 7,775 | 7,838 |
| EPS in Rs | 16.41 | 20.96 | 19.15 | 19.28 |

Profit and loss, balance sheet, cash flows, ratios and shareholding all arrive the same way, as markdown tables under their own ## headings.

Step 3: Start the Firecrawl Local API server

The command line is fine for one page. For Python work, run Firecrawl Local as a small API server and leave it open in its own terminal:

$env:FIRECRAWL_CLOUD = "off"   # keep every request on this machine
firecrawl-local serve
firecrawl-local listening on http://127.0.0.1:3002

Check it from a second terminal:

curl.exe http://localhost:3002/health
{"status":"ok","engine":"firecrawl-local","version":"0.1.0"}

The local routes are POST /v2/scrape, POST /v2/map, /v2/crawl, /v2/batch/scrape and GET /health. The server listens on 127.0.0.1 only and refuses private-network targets by default: when I asked it to scrape a page on 127.0.0.1 it answered BLOCKED_TARGET. To listen on another interface you must set a bearer token with --token.

Step 4: Connect the Python SDK to Firecrawl Local

The official Python SDK needs one change to talk to your own machine: point api_url at port 3002. No API key is required for local routes.

python -m venv .venv
.venv\Scripts\activate
pip install firecrawl-py pandas plotly kaleido
"""Step 1: talk to the Firecrawl Local server from Python.

Start the server first in another terminal:  firecrawl-local serve
"""
from firecrawl import Firecrawl

# No API key: the local routes on port 3002 do not need one.
app = Firecrawl(api_url="http://localhost:3002")

doc = app.scrape(
    "https://www.screener.in/company/INFY/consolidated/",
    formats=["markdown"],
)

print("Title   :", doc.metadata.title)
print("Status  :", doc.metadata.status_code)
print("Length  :", len(doc.markdown), "characters of markdown")
print("-" * 60)
print(doc.markdown[:400])
Title   : Infosys Ltd share price | About Infosys | Key Insights - Screener
Status  : 200
Length  : 28105 characters of markdown

If you already have code written for Firecrawl Cloud, set the environment variable FIRECRAWL_API_URL=http://127.0.0.1:3002 and the same code runs against Firecrawl Local.

Step 5: Fetch fundamental data from Screener.in

Markdown is easy to read but a trader wants DataFrames. Screener pages follow a fixed layout: a list of headline ratios at the top, then one section per statement. Three small functions cover all of it.

Turn the markdown into pandas DataFrames

Save this as fc_helpers.py. It splits a page by its ## headings, parses any pipe table into a DataFrame and cleans Indian number formats such as 4,49,983 Cr. into floats.

"""Small helpers that turn Firecrawl markdown into pandas objects."""
import re

import pandas as pd


def to_number(text):
    """'38,994' -> 38994.0, '+7%' -> 7.0, '4,49,983 Cr.' -> 449983.0, '' -> None."""
    cleaned = re.sub(r"[^0-9.\-]", "", str(text))
    try:
        return float(cleaned)
    except ValueError:
        return None


def md_sections(markdown):
    """Split a page into {'Quarterly Results': '...', 'Profit & Loss': '...'}."""
    sections, name, lines = {}, "top", []
    for line in markdown.splitlines():
        if line.startswith("## "):
            sections[name] = "\n".join(lines)
            # '## [Pivot Points](https://...)' -> 'Pivot Points'
            name = re.sub(r"\[(.*?)\]\(.*?\)", r"\1", line[3:]).strip()
            lines = []
        else:
            lines.append(line)
    sections[name] = "\n".join(lines)
    return sections


def md_tables(text):
    """Return every pipe table in a block of markdown as a DataFrame."""
    tables, block = [], []
    for line in text.splitlines() + [""]:
        if line.strip().startswith("|"):
            block.append(line.strip())
            continue
        if len(block) >= 3:
            rows = [[c.strip() for c in r.strip("|").split("|")] for r in block]
            header, body = rows[0], rows[2:]          # rows[1] is the |---| line
            width = len(header)
            body = [r[:width] + [""] * (width - len(r)) for r in body]
            tables.append(pd.DataFrame(body, columns=header))
        block = []
    return tables


def statement(markdown, heading, which=0):
    """One financial statement as a tidy DataFrame indexed by line item."""
    df = md_tables(md_sections(markdown)[heading])[which]
    df = df.rename(columns={df.columns[0]: "Item"})
    df["Item"] = df["Item"].str.replace("+", "", regex=False).str.strip()
    df = df.set_index("Item")
    return df.map(to_number)


def top_ratios(markdown):
    """The headline ratio list on a Screener company page as a dict."""
    wanted = ["Market Cap", "Current Price", "Stock P/E", "Book Value",
              "Dividend Yield", "ROCE", "ROE", "Face Value"]
    ratios = {}
    for line in md_sections(markdown)["top"].splitlines():
        for key in wanted:
            # keep the first hit only: '- Market Cap / Sales' appears later as a menu item
            if line.startswith(f"- {key} ") and key not in ratios:
                ratios[key] = to_number(line[len(key) + 3:])
    return ratios

Now a company page becomes four CSV files in a dozen lines:

"""Step 2: Screener.in company page -> pandas DataFrames -> CSV."""
from pathlib import Path

from firecrawl import Firecrawl

from fc_helpers import statement, top_ratios

SYMBOL = "INFY"
URL = f"https://www.screener.in/company/{SYMBOL}/consolidated/"

app = Firecrawl(api_url="http://localhost:3002")
md = app.scrape(URL, formats=["markdown"]).markdown

print("Headline ratios")
for name, value in top_ratios(md).items():
    print(f"  {name:<15} {value:>12,.2f}")

quarterly = statement(md, "Quarterly Results")
print("\nLast five quarters (Rs. crore)")
print(quarterly.loc[["Sales", "Operating Profit", "OPM %", "Net Profit", "EPS in Rs"]]
      .iloc[:, -5:].to_string())

annual = statement(md, "Profit & Loss")
print("\nAnnual profit and loss, last five columns (Rs. crore)")
print(annual.loc[["Sales", "Net Profit", "EPS in Rs", "Dividend Payout %"]]
      .iloc[:, -5:].to_string())

shareholding = statement(md, "Shareholding Pattern")
print("\nShareholding, last four quarters (%)")
print(shareholding.loc[["Promoters", "FIIs", "DIIs", "Public"]].iloc[:, -4:].to_string())

out = Path("out")
out.mkdir(exist_ok=True)
quarterly.to_csv(out / f"{SYMBOL}_quarterly.csv")
annual.to_csv(out / f"{SYMBOL}_annual.csv")
statement(md, "Balance Sheet").to_csv(out / f"{SYMBOL}_balance_sheet.csv")
statement(md, "Cash Flows").to_csv(out / f"{SYMBOL}_cash_flows.csv")
print("\nSaved 4 CSV files to", out.resolve())
Headline ratios
  Market Cap        449,983.00
  Current Price       1,109.00
  Stock P/E              14.20
  Book Value            244.00
  Dividend Yield          4.33
  ROCE                   40.00
  ROE                    31.90
  Face Value              5.00

Last five quarters (Rs. crore)
                  Sep 2025  Dec 2025  Mar 2026  Jun 2026  Sep 2026
Item
Sales             44490.00  45479.00  46402.00  48211.00  49244.00
Operating Profit  10535.00  10634.00  11167.00  11409.00  11689.00
OPM %                24.00     23.00     24.00     24.00     24.00
Net Profit         7375.00   6666.00   8509.00   7775.00   7838.00
EPS in Rs            17.73     16.41     20.96     19.15     19.28

Shareholding, last four quarters (%)
           Dec 2025  Mar 2026  Jun 2026  Sep 2026
Item
Promoters     14.52     14.38     13.82     13.90
FIIs          30.27     28.45     27.09     27.44
DIIs          41.08     43.19     42.78     42.66
Public        13.69     13.54     15.88     15.59

Saved 4 CSV files to D:\Firecrawl\Testing\out

The prices and ratios are what Screener showed after the close of 18 November 2026. If you chart in AmiBroker, the same numbers can also be viewed as Screener fundamentals inside AmiBroker.

Compare a watchlist with one batch scrape

One stock is a demo. A watchlist is the real use. batch_scrape sends all the URLs in one job and waits for the result:

"""Step 3: one batch job for a watchlist, one comparison table out."""
import time

import pandas as pd
from firecrawl import Firecrawl

from fc_helpers import statement, top_ratios

WATCHLIST = ["TCS", "INFY", "HCLTECH", "WIPRO", "TECHM"]
urls = [f"https://www.screener.in/company/{s}/consolidated/" for s in WATCHLIST]

app = Firecrawl(api_url="http://localhost:3002")

start = time.perf_counter()
job = app.batch_scrape(urls, formats=["markdown"])      # waits until the job completes
elapsed = time.perf_counter() - start
print(f"Batch status: {job.status}, pages: {job.completed}/{job.total}, "
      f"time: {elapsed:.2f} s")

rows = []
for doc in job.data:
    symbol = doc.metadata.source_url.split("/company/")[1].split("/")[0]
    ratios = top_ratios(doc.markdown)
    quarterly = statement(doc.markdown, "Quarterly Results")
    latest = quarterly.columns[-1]
    rows.append({
        "Symbol": symbol,
        "Mcap (Rs. cr)": ratios["Market Cap"],
        "Price": ratios["Current Price"],
        "P/E": ratios["Stock P/E"],
        "ROCE %": ratios["ROCE"],
        "ROE %": ratios["ROE"],
        "Div Yield %": ratios["Dividend Yield"],
        f"Sales YoY % ({latest})": quarterly.loc["YoY Sales growth", latest],
        f"OPM % ({latest})": quarterly.loc["OPM %", latest],
    })

table = pd.DataFrame(rows).set_index("Symbol").loc[WATCHLIST]
print(table.to_string())
table.to_csv("out/it_watchlist.csv")
Batch status: completed, pages: 5/5, time: 4.08 s
SymbolMcap (₹ cr)Price (₹)P/EROCE %ROE %Div yield %Sales YoY % (Sep 2026)OPM % (Sep 2026)
TCS8,60,9242,38015.863.051.82.691126
INFY4,49,9831,10914.240.031.94.331124
HCLTECH3,64,9071,34519.230.423.84.021221
WIPRO1,83,39518513.917.815.55.94920
TECHM1,71,5361,75030.023.117.52.911618

Five large-cap IT companies, nine fields each, in about four seconds and without opening a browser. Swap the list for your own holdings and the script does not change.

Chart the scraped data with Plotly

Once the numbers are in a CSV, Plotly turns them into a chart you can hover over in a notebook or export as a PNG. Each stock keeps one fixed colour, and the two measures get their own panels instead of sharing one chart with two y-axes.

"""Step 7: a Plotly chart from the watchlist CSV. Needs: pip install plotly kaleido"""
import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplots

wl = pd.read_csv("out/it_watchlist.csv", index_col=0).iloc[::-1]     # top row drawn on top
colours = {"TCS": "#2a78d6", "INFY": "#eb6834", "HCLTECH": "#1baf7a",
           "WIPRO": "#eda100", "TECHM": "#e87ba4"}                    # one colour per stock

fig = make_subplots(rows=1, cols=2, shared_yaxes=True,
                    subplot_titles=["Price to earnings (P/E)", "ROCE %"])
for col, measure in enumerate(["P/E", "ROCE %"], start=1):
    fig.add_trace(go.Bar(
        orientation="h", y=wl.index, x=wl[measure],
        marker_color=[colours[s] for s in wl.index], marker_cornerradius=6,
        text=[f"{v:.1f}" for v in wl[measure]], textposition="outside",
    ), row=1, col=col)
    fig.update_xaxes(visible=False, range=[0, wl[measure].max() * 1.2], row=1, col=col)

fig.update_layout(title="IT watchlist: valuation against return on capital",
                  template="simple_white", showlegend=False, width=1000, height=520)
fig.write_html("out/it_watchlist.html")      # interactive: hover, zoom, pan
fig.write_image("out/it_watchlist.png", scale=2)   # static PNG for a report or blog
IT stocks watchlist Plotly chart comparing P/E and ROCE for TCS, Infosys, HCL Tech, Wipro and Tech Mahindra

The same approach works for a single company. This chart is drawn from the quarterly CSV that the earlier script saved for Infosys:

Infosys quarterly sales and net profit Plotly chart built from Screener.in data scraped with Python

PNG export goes through Kaleido, which uses the Chrome already installed on your machine. fig.write_html() needs nothing extra and keeps the chart interactive.

Step 6: Fetch technical data from Investing.com

Investing.com publishes a technical page for every instrument with an indicator table, moving averages and pivot points. The same helpers parse it, because the page reaches Python as markdown tables under headings.

"""Step 4: technical snapshot from an Investing.com technical page."""
from firecrawl import Firecrawl

from fc_helpers import md_sections, md_tables, to_number

URL = "https://www.investing.com/equities/infosys-technical"

app = Firecrawl(api_url="http://localhost:3002")
doc = app.scrape(URL, formats=["markdown"])
if not doc.markdown:                       # the site answered with a bot check, not a page
    raise SystemExit(f"Blocked (HTTP {doc.metadata.status_code}). Stop and try again later.")
sections = md_sections(doc.markdown)

summary = next(name for name in sections if name.startswith("Summary:"))
print(doc.metadata.title)
print(summary.replace(":", ": "))

indicators = md_tables(sections["Technical Indicators"])[0]
indicators = indicators[["Name", "Value", "Action"]]
indicators["Value"] = indicators["Value"].map(to_number)
print("\nTechnical indicators")
print(indicators.to_string(index=False))

averages = md_tables(sections["Moving Averages"])[0]
averages.columns = ["Period", "_", "SMA", "SMA signal", "EMA", "EMA signal"]
averages = averages.drop(columns="_")
for column in ("SMA", "EMA"):
    averages[column] = averages[column].map(to_number)
print("\nMoving averages")
print(averages.to_string(index=False))

pivots = md_tables(sections["Pivot Points"])[0].set_index("Name")
print("\nPivot points")
print(pivots.loc[["Classic", "Fibonacci"]].to_string())

indicators.to_csv("out/INFY_indicators.csv", index=False)
averages.to_csv("out/INFY_moving_averages.csv", index=False)
INFY Technical Analysis, RSI and Moving Averages - Investing.com
Summary: Strong Sell

Technical indicators
               Name     Value          Action
            RSI(14)   29.5260            Sell
         STOCH(9,6)   11.7680        Oversold
       STOCHRSI(14)   19.5940        Oversold
        MACD(12,26)   -9.0500            Sell
            ADX(14)   22.6940         Neutral
        Williams %R  -95.0000        Oversold
            CCI(14) -102.6496            Sell
            ATR(14)    7.2950 Less Volatility

Moving averages
Period     SMA SMA signal     EMA EMA signal
   MA5 1113.00       Sell 1113.56       Sell
  MA20 1130.34       Sell 1126.72       Sell
  MA50 1141.43       Sell 1137.45       Sell
 MA200 1125.47       Sell 1126.94       Sell

Pivot points
                 S3        S2        S1 Pivot Points        R1        R2        R3
Classic    1,102.47  1,104.93  1,109.87     1,112.33  1,117.27  1,119.73  1,124.67
Fibonacci  1,104.93  1,107.76  1,109.51     1,112.33  1,115.16  1,116.91  1,119.73

The output is shortened by a few rows. The page stamps this snapshot as 18 November 2026, 10:05 AM GMT, which is 3:35 PM IST. It is the default timeframe the page serves, and the labels such as Strong Sell are Investing.com’s own readings, not mine. The plain fetch took 902 ms.

Rate limiting: why Investing.com blocked the fifth request

While testing I fetched Investing.com pages four times in under half an hour: one command-line scrape, the robots.txt file, the Python script above and one more scrape to measure page size. All four succeeded. The fifth request came back like this:

success: True
blocked: True | source: local
statusCode: 403 | engine: local-fetch | title: Just a moment...
markdown chars: 0

That is a bot check. The site stopped serving the page and returned a challenge screen with HTTP 403. Firecrawl Local reported it honestly: "blocked": true, an empty markdown field and no error. This is why the script above checks for empty markdown before it parses anything.

It was not the only one. A small PDF on w3.org, which I picked to test PDF parsing, answered the very first request with the same 403 challenge.

Websites rate limit for good reasons. Servers cost money, automated traffic can swamp real visitors, and portals like Investing.com license their prices and indicators from exchanges and data vendors on terms that do not cover bulk copying. Rate limiting usually shows up in one of four ways:

  • HTTP 429 Too Many Requests, often with a Retry-After header telling you how long to wait.
  • HTTP 403 with a challenge page such as the one above.
  • A CAPTCHA or a login wall where the content used to be.
  • A silent downgrade: empty tables, partial pages or stale data.

What to do when a site blocks you

I stopped requesting the site at that point. Rotating proxies, faking browser fingerprints or paying a CAPTCHA-solving service are ways of overriding a decision the site owner has made, and I do not recommend any of them. With an API key and FIRECRAWL_CLOUD=auto, Firecrawl Local can hand a blocked URL to Firecrawl Cloud. That changes who fetches the page. It does not change what the site’s terms allow.

For technical indicators there is a cleaner route anyway: pull OHLC candles from your broker API and compute RSI, MACD and moving averages yourself. You control the timeframe and the parameters, and nobody rate limits your own arithmetic.

JavaScript pages: when Firecrawl Local needs Chrome

Some tables are filled in by JavaScript after the page loads. On Screener the peer comparison is one of them. A plain fetch returns the placeholder text Loading peers table ... in its place. Tell Firecrawl Local to render the page in your local Chrome:

firecrawl-local scrape https://www.screener.in/company/INFY/consolidated/ --browser always --markdown
| S.No. | Company | CMP Rs. | P/E | Mar Cap Rs.Cr. | Div Yld % | NP Qtr Rs.Cr. | ROCE % |
| 1. | TCS | 2379.50 | 15.75 | 860923.92 | 2.69 | 13934.00 | 63.03 |
| 2. | Infosys | 1108.80 | 14.22 | 449983.43 | 4.33 | 7838.00 | 39.95 |
| 3. | HCL Technologies | 1344.70 | 19.18 | 364906.55 | 4.02 | 5132.00 | 30.40 |

The excerpt drops a few columns and the company links for width. The peers table is now in the markdown, at the cost of time: 2,808 ms against 245 ms for the plain fetch. Chrome runs headless with a throwaway profile. The --browser flag takes auto (the default), never or always, and the same choice is available through the FIRECRAWL_LOCAL_BROWSER environment variable.

Sections that need a login stay hidden. Screener’s Insights block came back as Please log in to see hidden values, which is exactly how it should be.

How fast is Firecrawl Local on a normal machine?

Firecrawl Local scrape timings for Screener.in and Investing.com pages on Windows 11

These are single runs from the command line, wall-clock time including process start-up:

PageEngineTimeRaw HTMLMarkdown
example.comPlain fetch115 msn/an/a
Screener.in company pagePlain fetch245 ms2,57,708 chars28,105 chars
Investing.com technical pagePlain fetch902 ms9,21,198 chars19,568 chars
Screener.in with peers tableLocal Chrome2,808 msn/an/a

The last two columns matter more than the milliseconds if you work with language models. The Screener page shrinks about 9 times and the Investing.com page about 47 times when converted from raw HTML to markdown. Fewer characters means fewer tokens, lower cost and less noise for the model to wade through.

Memory stayed small throughout. The server used 13.5 MB when freshly started, 18.4 MB after its first request and about 46 MB after the batch jobs in this article.

Where does Firecrawl Local keep the fetched data?

Short answer: in RAM, and only for as long as it takes to hand the page back to you. Firecrawl Local has no database and no cache folder. After dozens of scrapes I searched my user profile for anything it had stored and found no data directory at all. A page is downloaded into the process’s memory, converted to markdown there and returned on standard output or in the HTTP response. What happens next is up to your code.

ModeWhere the page livesHow long
scrape from the CLIRAM, then your terminal or the file you redirect toUntil the command exits
serve, single scrapeRAM, then the HTTP responseUntil the response is sent
serve, batch and crawl jobsRAM inside the server processOne hour from job start, or until the server stops
crawl --out folderOne markdown file per page in that folderUntil you delete it
Pages rendered in ChromeA throwaway Chrome profile in the temp folderRemoved when a CLI scrape exits
Your own scriptsWherever you save: CSV, Parquet, SQLiteYour choice

Because batch and crawl jobs sit in the server’s memory, collect the results before you close the server. That is also why its memory grew from 13.5 MB to about 46 MB during my tests.

The temporary Chrome profile and the agent’s context

The only thing written to disk without your asking is a browser profile. When the server needs Chrome it creates a folder named firecrawl-local-chrome-<pid>-<random> under %TEMP%. Mine grew to 63 MB. A one-off CLI scrape cleaned up after itself. When I stopped the server, both with a console break and with a forced kill, the folder and several headless Chrome processes stayed behind on Windows, so check %TEMP% after a long session.

One more point for agent users. When an AI agent fetches a page through Firecrawl Local, the markdown goes into the model’s context window. If the model runs in the cloud, the page text travels to that provider along with your prompt. The scraping is local. The model’s reading of it may not be.

Firecrawl Local for AI agents and MCP

Firecrawl Local MCP server flow connecting an AI agent to public web pages for stock research

Everything so far was Python calling a tool. The bigger idea behind Firecrawl Local is the other direction: an AI agent decides which pages it needs and fetches them itself. The bridge is MCP, the Model Context Protocol. If the term is new, read my plain-language guide to MCP for traders first.

An agent on its own cannot open a web page. It needs a tool, and the web fetch tools built into assistants are often limited: many return summaries, struggle with JavaScript pages or are metered. Firecrawl Local gives the agent a fetch tool that runs on your machine, costs nothing per page and returns compact markdown with the tables intact.

Connect Firecrawl Local to Claude Code, Cursor and Codex

One command registers the MCP server with every supported AI tool it finds: Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI and Codex. Preview it first with --dry-run. The block after the commands is an excerpt of what the dry run printed on my machine:

$env:HOME = $env:USERPROFILE     # Windows only: without this, init stops with "HOME is not set."
firecrawl-local init --dry-run   # show what would change
firecrawl-local init             # write the config
Firecrawl MCP server:
  Claude Code: would be added in C:\Users\<you>\.claude.json
  Windsurf: would be added in C:\Users\<you>\.codeium/windsurf/mcp_config.json
  Gemini CLI: would be added in C:\Users\<you>\.gemini/settings.json
  Codex: would be added in C:\Users\<you>\.codex/config.toml

What goes to Firecrawl cloud:
  - your search queries (there is no search index on a laptop)
  - the URL of a page that blocked this machine, so Firecrawl can fetch it for you
  - Alexandria data-provider calls, once you sign in
  - a random install ID and the name of your AI tool, to count installs
Pages that load fine here are never sent. FIRECRAWL_CLOUD=search keeps only search;
FIRECRAWL_CLOUD=off turns the cloud off.

The docs say init backs up each config file before it edits it. To add the server by hand, register firecrawl-local mcp as a stdio server with the full path to the binary:

{
  "mcpServers": {
    "firecrawl": {
      "command": "C:\\Users\\<you>\\.cargo\\bin\\firecrawl-local.exe",
      "args": ["mcp", "--public-only"],
      "env": { "FIRECRAWL_CLOUD": "off" }
    }
  }
}

--public-only makes the server refuse localhost and private-network addresses. Keep it on whenever an agent browses sites you do not control, so that a malicious page cannot trick the agent into reading your router or a local admin panel.

What the agent sees

To show there is no magic, this script plays the role of the agent. It starts firecrawl-local mcp, lists the tools and calls one, using nothing but JSON over standard input and output:

"""Step 5: speak MCP to firecrawl-local by hand, the way an AI agent does.

An agent starts `firecrawl-local mcp` as a child process and exchanges
newline-delimited JSON-RPC messages over stdin and stdout.
"""
import json
import os
import subprocess

env = dict(os.environ, FIRECRAWL_CLOUD="off")          # keep everything on this machine
proc = subprocess.Popen(
    ["firecrawl-local", "mcp", "--public-only"],
    stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
    text=True, encoding="utf-8", env=env,
)


def send(message):
    proc.stdin.write(json.dumps(message) + "\n")
    proc.stdin.flush()


def call(msg_id, method, params):
    send({"jsonrpc": "2.0", "id": msg_id, "method": method, "params": params})
    while True:                                          # skip notifications and logs
        reply = json.loads(proc.stdout.readline())
        if reply.get("id") == msg_id:
            return reply


hello = call(1, "initialize", {
    "protocolVersion": "2025-06-18",
    "capabilities": {},
    "clientInfo": {"name": "marketcalls-demo", "version": "1.0"},
})
print("Server:", hello["result"]["serverInfo"])
send({"jsonrpc": "2.0", "method": "notifications/initialized"})

tools = call(2, "tools/list", {})["result"]["tools"]
print(f"\n{len(tools)} tools exposed to the agent:")
for tool in tools:
    print(f"  {tool['name']:<32} {tool['description'].strip().splitlines()[0][:70]}")

answer = call(3, "tools/call", {
    "name": "firecrawl_scrape",
    "arguments": {
        "url": "https://www.screener.in/company/INFY/consolidated/",
        "formats": ["markdown"],
    },
})
text = answer["result"]["content"][0]["text"]
print("\nfirecrawl_scrape returned", len(text), "characters. First 300:")
print(text[:300])

proc.terminate()
Server: {'name': 'firecrawl-fastmcp', 'version': '3.40.0'}

30 tools exposed to the agent:
  firecrawl_scrape                 Scrape one URL and return its content: markdown by default, or HTML, l
  firecrawl_map                    Enumerate URLs indexed under one website through Firecrawl without fet
  firecrawl_search                 Use this when you need current or specific information from the web: r
  ...
  firecrawl_crawl                 Start a multi-page crawl at a website URL, poll it to a terminal state
  firecrawl_check_crawl_status     Retrieve the current status, progress, and available results for an ex
  ...

firecrawl_scrape returned 29596 characters.

The MCP server is the firecrawl-mcp npm package, started through npx, which is why Node.js 22 or newer is required. Of the 30 tools, the scrape, map and crawl tools run on your machine. The rest, such as search, the research tools and monitors, call Firecrawl Cloud. The tool list above is shortened where you see dots.

Agent use cases for traders and developers

With the server registered, you stop writing scrapers and start writing requests:

  • Ad hoc fundamentals: “Scrape the Screener pages for TCS, Infosys and HCL Tech and build a table of P/E, ROCE and the last four quarters of sales growth.”
  • Earnings season: “Read the latest concall transcript linked on this page and list management’s guidance changes.” PDFs are parsed on your machine: a 14-page public PDF took 3.2 seconds in my test.
  • Second opinion: “Open the technical page for Infosys and tell me whether RSI and MACD agree with the moving averages.”
  • Documentation for coding agents: crawl a broker’s API docs into markdown so the agent writes code against the current version, not its memory.
  • Research pipelines: a scheduled agent that refreshes a watchlist CSV after results and flags the stocks whose margins moved.

Developers building their own agents in Python do not need MCP at all. Point the Firecrawl SDK at port 3002 and call it from a tool function, the same way I wired NSE data into an Agno stock market agent. One caution applies to every setup: text from a web page is data, not instructions. Do not give the same agent the power to place orders without a human confirming them.

Crawl and map: whole sites with Firecrawl Local

Two more commands matter when you need a site, not a page. map lists URLs from the sitemaps, optionally filtered by text. crawl follows links from a start URL and writes one markdown file per page. I tried both on my own site:

firecrawl-local map https://www.marketcalls.in --search openalgo --limit 8
firecrawl-local crawl https://www.marketcalls.in/python/ --limit 3 --out pages
firecrawl-local : 3 pages in 3.34s (0.9 pages/s), 0 errors, 0 skipped by robots.txt, saved to pages

Note the phrase skipped by robots.txt in that summary. The crawler reads each site’s robots.txt and obeys it by default. There is an --ignore-robots flag. Leave it alone. A folder of markdown files is the ideal input for a retrieval system or for an agent that needs to read a product’s documentation.

Ethical scraping practices for financial websites

A local scraper removes the cost of fetching. It does not remove the responsibility. These are the rules I follow, and the block from Investing.com is a reminder of why they exist.

  • Read robots.txt and the terms before the first request. Screener’s robots.txt allows company pages and disallows user pages and search or sort queries. Investing.com’s robots.txt allows the technical pages, but its terms forbid storing or redistributing the data without written permission.
  • Keep it personal. Screener licenses its pages for personal, non-commercial viewing. Analysing your own watchlist is one thing. Republishing a dataset or selling a product built on it is another.
  • Fetch as rarely as the data changes. Financial statements change once a quarter. Fetching them every five minutes helps nobody.
  • Cache everything. If you fetched a page today, read it from disk for the rest of the day.
  • Go slowly. One request every few seconds per site. The CLI fetches 16 URLs at once by default, so pass --concurrency 1 or 2 when all the URLs are on one site.
  • Identify yourself. Firecrawl Local’s default user agent ends with FirecrawlLocal/0.1.0 (+https://firecrawl.dev/local). Do not disguise it as something else.
  • Stay out of logged-in and paid areas. Never hand your session cookies to a scraper.
  • Treat 403 and 429 as a stop sign. Back off for hours, not seconds, and never retry in a loop.
  • Prefer the official route when one exists. Exchange filings, company investor pages, Screener’s own Export to Excel button for logged-in users, your broker API and licensed data vendors are all better sources for bulk or commercial work.

A polite fetch helper in Python

The helper below builds the first few rules into code: a one-day disk cache, a pause between live requests and a hard stop when the site says no.

"""Step 8: polite fetching. Cache what you fetched, pause between requests, stop on a block."""
import hashlib
import time
from pathlib import Path

from firecrawl import Firecrawl

app = Firecrawl(api_url="http://localhost:3002")
CACHE = Path("cache")
CACHE.mkdir(exist_ok=True)
PAUSE = 5                 # seconds between live requests
MAX_AGE = 24 * 3600       # fundamentals change once a quarter: one fetch a day is plenty
_last_request = 0.0


def polite_scrape(url):
    global _last_request
    cached = CACHE / (hashlib.sha1(url.encode()).hexdigest() + ".md")
    if cached.exists() and time.time() - cached.stat().st_mtime < MAX_AGE:
        return cached.read_text(encoding="utf-8"), "cache"      # no request leaves the machine

    wait = PAUSE - (time.time() - _last_request)
    if wait > 0:
        time.sleep(wait)
    doc = app.scrape(url, formats=["markdown"])
    _last_request = time.time()

    status = doc.metadata.status_code
    if status in (403, 429) or not doc.markdown:
        # The site said no. Do not retry in a loop and do not switch tricks.
        raise RuntimeError(f"{url} answered HTTP {status}. Stopping.")
    cached.write_text(doc.markdown, encoding="utf-8")
    return doc.markdown, "live"


if __name__ == "__main__":
    for symbol in ["TCS", "INFY", "TCS"]:
        start = time.perf_counter()
        md, origin = polite_scrape(f"https://www.screener.in/company/{symbol}/consolidated/")
        print(f"{symbol:<6} {origin:<6} {len(md):>6,} chars  {time.perf_counter() - start:5.2f} s")
TCS    live   34,441 chars   2.21 s
INFY   live   28,105 chars   7.11 s
TCS    cache  34,441 chars   0.01 s

The second live request waited five seconds before it went out, and the repeat request for TCS never left the machine. Running the script again on the same day made no requests at all.

Common mistakes with Firecrawl Local

  • Forgetting to start the server. The Python SDK needs firecrawl-local serve running in another terminal. A connection error on port 3002 means it is not.
  • Expecting cloud-only features. Local scrapes skip the json and summary formats and ignore browser actions such as clicks. Ask for markdown and do the extraction in pandas or with your own model.
  • Closing the server before collecting a batch. Jobs live in memory and expire after an hour.
  • Parsing a blocked page. Check for empty markdown or a 403 or 429 status before you index into tables.
  • Using the plain fetch on JavaScript tables. If a section says Loading, rerun with --browser always.
  • Rupee sign errors on Windows. Set $env:PYTHONIOENCODING = "utf-8" before printing, and use Out-File -Encoding utf8 when you redirect to a file.
  • Running init on Windows without HOME. Version 0.1.0 stops with “HOME is not set.” Set it to your user profile first.
  • Leaving FIRECRAWL_CLOUD on auto without realising it. Set it to off if you want a strict guarantee that nothing leaves the machine.

Where Firecrawl Local fits in a trading workflow

Firecrawl Local is a research tool, not a data feed. It is very good at slow-moving public information and a poor fit for anything that needs ticks.

JobGood fit?Better tool when it is not
Weekly fundamentals refresh for a watchlistYes
Shareholding and peer comparison tablesYes
Reading concall transcripts and annual report PDFsYes
Feeding web pages to an AI agentYes
A daily indicator snapshot for a few swing ideasWith careCompute from broker OHLC data
Intraday quotes, option chains, order flowNoBroker API or a licensed datafeed
Backtesting on long price historyNoHistorical data from your broker or vendor

My own split is simple. Prices, orders and anything intraday go through the broker API. Fundamentals, filings and commentary come from the web through Firecrawl Local, cached on disk and refreshed when results come out. The two meet in pandas, and increasingly in an agent that can reach both.

If you want to try it this weekend, the order is: install the binary, scrape one Screener page from the command line, start the server, run the pandas script on your own watchlist and only then wire it into an agent. Each step works on its own, and each one is a few minutes.

Python Elite membership

MEMBERSHIP

Python Elite membership

15 Python, AI and algo trading programs in one plan, including QuantFlow and AI Bootcamp live mentorship.

  • Python from fundamentals to trading apps
  • Backtesting, broker APIs and OpenAlgo
  • Live QuantFlow and AI Bootcamp sessions
View Python Elite →by Marketcalls

Learn by watching: free videos every week

Market Profile, order flow, Python, AI and OpenAlgo tutorials on YouTube.

Rajandran R Creator of OpenAlgo Sketchmaker, RagZ. Building GenAI Applications. Telecom Engineer turned Full-time Derivative Trader. Mostly Trading Nifty, Banknifty, High Liquid Stock Derivatives. Trading the Markets Since 2006 onwards. Using Market Profile and Orderflow for more than a decade. Designed and published 100+ open source trading systems on various trading tools. Now the recent curiosity on building Trading AI Agents.

Leave a Reply

Get Notifications, Alerts on Market Updates, Trading Tools, Automation & More