Firecrawl Local is a single Rust binary that gives your scripts and AI agents a web fetch tool on your own machine: hand it a URL and it returns clean markdown, with no API key and no per-page credits. In this tutorial I install Firecrawl Local on Windows, pull fundamental data from Screener.in and technical data from Investing.com with Python, turn the pages into pandas tables and Plotly charts, and then connect the same binary to AI agents through MCP.
Every command, output and number below comes from a real run on my Windows 11 machine on 19 November 2026. That includes the parts that did not go smoothly, such as Investing.com blocking my fifth request, because that is the part most scraping tutorials leave out.
Key takeaways
- Firecrawl Local is one open-source binary (AGPL-3.0) that scrapes, crawls and maps websites and serves the Firecrawl v2 API on
localhost:3002. No database, no queue, no Docker. cargo install firecrawl-localtook about six minutes on my machine and produced a 23.6 MB executable.- A Screener.in company page came back as clean markdown in 245 ms. Five companies in one batch job took about 4 seconds.
- Fetched pages live in RAM and are handed straight back to the caller. Nothing is cached on disk unless you save it. The one exception is a throwaway Chrome profile in the temp folder.
- Investing.com served its technical page four times and then answered with an HTTP 403 bot check. Firecrawl Local flags that as
"blocked": true. Treat it as a stop sign. firecrawl-local mcpexposes the same engine to Claude Code, Cursor, Codex and other agents, so you can ask for fundamentals in plain English.
What is Firecrawl Local?
Firecrawl is a widely used web scraping API for AI applications. Until now you had two ways to use it: pay for the cloud API, or self-host the full stack with Docker, Redis, queues and workers. On 18 November 2026 the team announced a local edition called Firecrawl Local, a rebuild of the scraping core in Rust that ships as one executable.
The announcement makes three claims: a scrape primitive 37 times faster than the self-hosted stack, 230 times less memory (11 MB idle) and a fully open-source codebase. On my machine a freshly started server sat at 13.5 MB, which is close to the claim for a Windows build.
That one binary gives you five things:
- scrape: one or more URLs to markdown, HTML, raw HTML or a list of links.
- crawl: follow links from a start URL and save one markdown file per page.
- map: list a site’s URLs from its sitemaps.
- serve: the Firecrawl v2 HTTP API on port 3002, so the existing Python and Node SDKs work unchanged.
- mcp and init: an MCP server for AI agents and a one-command setup for them.
The official quickstart for Firecrawl Local documents every flag. This article is about putting it to work on market data.
Why traders and investors should care
Most of the data we want as investors already sits on public web pages: ten years of profit and loss on Screener, shareholding patterns, peer tables, indicator summaries, concall links. Copying it into Excel by hand does not scale beyond a handful of stocks.
Classic scraping with requests and Beautiful Soup works, and I covered it in an older Beautiful Soup tutorial on Screener data. The catch is that you write and maintain a parser for every table on every site, and it breaks whenever the site changes its HTML.
Firecrawl Local changes the unit of work. The tool does the fetching, the JavaScript rendering and the cleaning, and returns markdown in which every HTML table is already a tidy pipe table. One small helper turns any of those tables into a DataFrame. The same output is what a language model reads best, which is why the tool doubles as the web fetch layer for AI agents.
- Cost: no API key and no credits. Every local response reports
creditsUsed: 0. - Privacy: your watchlist and the pages you research stay on your machine.
- Speed: a static page takes a few hundred milliseconds.
What runs on your machine and what goes to the cloud
The binary is called local, but it can hand some work to Firecrawl’s cloud. It helps to know the split before you start.
| Feature | Where it runs | Notes |
|---|---|---|
| Scrape, crawl, map, batch scrape | Your machine | No API key needed |
| PDF and Office document parsing | Your machine | Converted to markdown locally |
| JavaScript rendering | Your machine | Uses your installed Chrome or Chromium |
| Web search | Firecrawl Cloud | There is no search index on a laptop |
| Pages that block your machine | Firecrawl Cloud, optional | Result is tagged "source": "firecrawl-cloud" |
| JSON and summary formats, browser actions | Cloud or the full self-hosted server | Skipped by the local engine |
The FIRECRAWL_CLOUD environment variable controls the hand-off: auto (the default), search (only search queries go out) or off. I ran the server and all the Python tests in this article with FIRECRAWL_CLOUD=off and without a Firecrawl account, so what you see is the local engine alone.
Step 1: Install Firecrawl Local on Windows
The docs give install steps for macOS and Debian or Ubuntu. Windows is not mentioned, so I tried it: Firecrawl Local builds and runs fine on Windows 11 with the MSVC toolchain. You need:
- Rust (stable, installed through rustup). I used Rust 1.99.1.
- A C compiler: the Visual Studio C++ Build Tools on Windows, the Xcode command line tools on macOS, or
build-essentialon Ubuntu. - Chrome or Chromium (optional) for pages that need JavaScript.
- Node.js 22 or newer (optional) for the MCP server.
- Python 3 for the scripts in this article. I used Python 3.14.
cargo install firecrawl-local
firecrawl-local --version
firecrawl-local 0.1.0
The build compiled more than 350 crates and finished in 347 seconds on my machine. The result is a 23.6 MB firecrawl-local.exe in %USERPROFILE%\.cargo\bin, a folder rustup has already added to your PATH.
Step 2: Your first scrape from the command line
firecrawl-local scrape https://example.com
{
"success": true,
"data": {
"markdown": "This domain is for use in documentation examples without needing permission. ...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com/",
"statusCode": 200,
"contentType": "text/html; charset=utf-8",
"engine": "local-fetch",
"creditsUsed": 0
},
"source": "local"
}
}
Three fields are worth knowing. engine: local-fetch means a plain HTTP request was enough. source: local means the page never left your machine. creditsUsed is 0. Add --markdown to print only the content, or --format markdown,html,links to choose formats.
Now a real page. This is the consolidated Infosys page on Screener.in:
firecrawl-local scrape https://www.screener.in/company/INFY/consolidated/ --markdown |
Out-File -Encoding utf8 infy.md
It returned 395 lines of markdown in 245 ms. An excerpt, with the quarterly table trimmed to four columns for width:
- Market Cap ₹ 4,49,983 Cr.
- Current Price ₹ 1,109
- High / Low ₹ 1,728 / 980
- Stock P/E 14.2
- Book Value ₹ 244
- Dividend Yield 4.33 %
- ROCE 40.0 %
- ROE 31.9 %
## Quarterly Results
| | Dec 2025 | Mar 2026 | Jun 2026 | Sep 2026 |
| --- | --- | --- | --- | --- |
| Sales + | 45,479 | 46,402 | 48,211 | 49,244 |
| Operating Profit | 10,634 | 11,167 | 11,409 | 11,689 |
| OPM % | 23% | 24% | 24% | 24% |
| Net Profit + | 6,666 | 8,509 | 7,775 | 7,838 |
| EPS in Rs | 16.41 | 20.96 | 19.15 | 19.28 |
Profit and loss, balance sheet, cash flows, ratios and shareholding all arrive the same way, as markdown tables under their own ## headings.
Step 3: Start the Firecrawl Local API server
The command line is fine for one page. For Python work, run Firecrawl Local as a small API server and leave it open in its own terminal:
$env:FIRECRAWL_CLOUD = "off" # keep every request on this machine
firecrawl-local serve
firecrawl-local listening on http://127.0.0.1:3002
Check it from a second terminal:
curl.exe http://localhost:3002/health
{"status":"ok","engine":"firecrawl-local","version":"0.1.0"}
The local routes are POST /v2/scrape, POST /v2/map, /v2/crawl, /v2/batch/scrape and GET /health. The server listens on 127.0.0.1 only and refuses private-network targets by default: when I asked it to scrape a page on 127.0.0.1 it answered BLOCKED_TARGET. To listen on another interface you must set a bearer token with --token.
Step 4: Connect the Python SDK to Firecrawl Local
The official Python SDK needs one change to talk to your own machine: point api_url at port 3002. No API key is required for local routes.
python -m venv .venv
.venv\Scripts\activate
pip install firecrawl-py pandas plotly kaleido
"""Step 1: talk to the Firecrawl Local server from Python.
Start the server first in another terminal: firecrawl-local serve
"""
from firecrawl import Firecrawl
# No API key: the local routes on port 3002 do not need one.
app = Firecrawl(api_url="http://localhost:3002")
doc = app.scrape(
"https://www.screener.in/company/INFY/consolidated/",
formats=["markdown"],
)
print("Title :", doc.metadata.title)
print("Status :", doc.metadata.status_code)
print("Length :", len(doc.markdown), "characters of markdown")
print("-" * 60)
print(doc.markdown[:400])
Title : Infosys Ltd share price | About Infosys | Key Insights - Screener
Status : 200
Length : 28105 characters of markdown
If you already have code written for Firecrawl Cloud, set the environment variable FIRECRAWL_API_URL=http://127.0.0.1:3002 and the same code runs against Firecrawl Local.
Step 5: Fetch fundamental data from Screener.in
Markdown is easy to read but a trader wants DataFrames. Screener pages follow a fixed layout: a list of headline ratios at the top, then one section per statement. Three small functions cover all of it.
Turn the markdown into pandas DataFrames
Save this as fc_helpers.py. It splits a page by its ## headings, parses any pipe table into a DataFrame and cleans Indian number formats such as 4,49,983 Cr. into floats.
"""Small helpers that turn Firecrawl markdown into pandas objects."""
import re
import pandas as pd
def to_number(text):
"""'38,994' -> 38994.0, '+7%' -> 7.0, '4,49,983 Cr.' -> 449983.0, '' -> None."""
cleaned = re.sub(r"[^0-9.\-]", "", str(text))
try:
return float(cleaned)
except ValueError:
return None
def md_sections(markdown):
"""Split a page into {'Quarterly Results': '...', 'Profit & Loss': '...'}."""
sections, name, lines = {}, "top", []
for line in markdown.splitlines():
if line.startswith("## "):
sections[name] = "\n".join(lines)
# '## [Pivot Points](https://...)' -> 'Pivot Points'
name = re.sub(r"\[(.*?)\]\(.*?\)", r"\1", line[3:]).strip()
lines = []
else:
lines.append(line)
sections[name] = "\n".join(lines)
return sections
def md_tables(text):
"""Return every pipe table in a block of markdown as a DataFrame."""
tables, block = [], []
for line in text.splitlines() + [""]:
if line.strip().startswith("|"):
block.append(line.strip())
continue
if len(block) >= 3:
rows = [[c.strip() for c in r.strip("|").split("|")] for r in block]
header, body = rows[0], rows[2:] # rows[1] is the |---| line
width = len(header)
body = [r[:width] + [""] * (width - len(r)) for r in body]
tables.append(pd.DataFrame(body, columns=header))
block = []
return tables
def statement(markdown, heading, which=0):
"""One financial statement as a tidy DataFrame indexed by line item."""
df = md_tables(md_sections(markdown)[heading])[which]
df = df.rename(columns={df.columns[0]: "Item"})
df["Item"] = df["Item"].str.replace("+", "", regex=False).str.strip()
df = df.set_index("Item")
return df.map(to_number)
def top_ratios(markdown):
"""The headline ratio list on a Screener company page as a dict."""
wanted = ["Market Cap", "Current Price", "Stock P/E", "Book Value",
"Dividend Yield", "ROCE", "ROE", "Face Value"]
ratios = {}
for line in md_sections(markdown)["top"].splitlines():
for key in wanted:
# keep the first hit only: '- Market Cap / Sales' appears later as a menu item
if line.startswith(f"- {key} ") and key not in ratios:
ratios[key] = to_number(line[len(key) + 3:])
return ratios
Now a company page becomes four CSV files in a dozen lines:
"""Step 2: Screener.in company page -> pandas DataFrames -> CSV."""
from pathlib import Path
from firecrawl import Firecrawl
from fc_helpers import statement, top_ratios
SYMBOL = "INFY"
URL = f"https://www.screener.in/company/{SYMBOL}/consolidated/"
app = Firecrawl(api_url="http://localhost:3002")
md = app.scrape(URL, formats=["markdown"]).markdown
print("Headline ratios")
for name, value in top_ratios(md).items():
print(f" {name:<15} {value:>12,.2f}")
quarterly = statement(md, "Quarterly Results")
print("\nLast five quarters (Rs. crore)")
print(quarterly.loc[["Sales", "Operating Profit", "OPM %", "Net Profit", "EPS in Rs"]]
.iloc[:, -5:].to_string())
annual = statement(md, "Profit & Loss")
print("\nAnnual profit and loss, last five columns (Rs. crore)")
print(annual.loc[["Sales", "Net Profit", "EPS in Rs", "Dividend Payout %"]]
.iloc[:, -5:].to_string())
shareholding = statement(md, "Shareholding Pattern")
print("\nShareholding, last four quarters (%)")
print(shareholding.loc[["Promoters", "FIIs", "DIIs", "Public"]].iloc[:, -4:].to_string())
out = Path("out")
out.mkdir(exist_ok=True)
quarterly.to_csv(out / f"{SYMBOL}_quarterly.csv")
annual.to_csv(out / f"{SYMBOL}_annual.csv")
statement(md, "Balance Sheet").to_csv(out / f"{SYMBOL}_balance_sheet.csv")
statement(md, "Cash Flows").to_csv(out / f"{SYMBOL}_cash_flows.csv")
print("\nSaved 4 CSV files to", out.resolve())
Headline ratios
Market Cap 449,983.00
Current Price 1,109.00
Stock P/E 14.20
Book Value 244.00
Dividend Yield 4.33
ROCE 40.00
ROE 31.90
Face Value 5.00
Last five quarters (Rs. crore)
Sep 2025 Dec 2025 Mar 2026 Jun 2026 Sep 2026
Item
Sales 44490.00 45479.00 46402.00 48211.00 49244.00
Operating Profit 10535.00 10634.00 11167.00 11409.00 11689.00
OPM % 24.00 23.00 24.00 24.00 24.00
Net Profit 7375.00 6666.00 8509.00 7775.00 7838.00
EPS in Rs 17.73 16.41 20.96 19.15 19.28
Shareholding, last four quarters (%)
Dec 2025 Mar 2026 Jun 2026 Sep 2026
Item
Promoters 14.52 14.38 13.82 13.90
FIIs 30.27 28.45 27.09 27.44
DIIs 41.08 43.19 42.78 42.66
Public 13.69 13.54 15.88 15.59
Saved 4 CSV files to D:\Firecrawl\Testing\out
The prices and ratios are what Screener showed after the close of 18 November 2026. If you chart in AmiBroker, the same numbers can also be viewed as Screener fundamentals inside AmiBroker.
Compare a watchlist with one batch scrape
One stock is a demo. A watchlist is the real use. batch_scrape sends all the URLs in one job and waits for the result:
"""Step 3: one batch job for a watchlist, one comparison table out."""
import time
import pandas as pd
from firecrawl import Firecrawl
from fc_helpers import statement, top_ratios
WATCHLIST = ["TCS", "INFY", "HCLTECH", "WIPRO", "TECHM"]
urls = [f"https://www.screener.in/company/{s}/consolidated/" for s in WATCHLIST]
app = Firecrawl(api_url="http://localhost:3002")
start = time.perf_counter()
job = app.batch_scrape(urls, formats=["markdown"]) # waits until the job completes
elapsed = time.perf_counter() - start
print(f"Batch status: {job.status}, pages: {job.completed}/{job.total}, "
f"time: {elapsed:.2f} s")
rows = []
for doc in job.data:
symbol = doc.metadata.source_url.split("/company/")[1].split("/")[0]
ratios = top_ratios(doc.markdown)
quarterly = statement(doc.markdown, "Quarterly Results")
latest = quarterly.columns[-1]
rows.append({
"Symbol": symbol,
"Mcap (Rs. cr)": ratios["Market Cap"],
"Price": ratios["Current Price"],
"P/E": ratios["Stock P/E"],
"ROCE %": ratios["ROCE"],
"ROE %": ratios["ROE"],
"Div Yield %": ratios["Dividend Yield"],
f"Sales YoY % ({latest})": quarterly.loc["YoY Sales growth", latest],
f"OPM % ({latest})": quarterly.loc["OPM %", latest],
})
table = pd.DataFrame(rows).set_index("Symbol").loc[WATCHLIST]
print(table.to_string())
table.to_csv("out/it_watchlist.csv")
Batch status: completed, pages: 5/5, time: 4.08 s
| Symbol | Mcap (₹ cr) | Price (₹) | P/E | ROCE % | ROE % | Div yield % | Sales YoY % (Sep 2026) | OPM % (Sep 2026) |
|---|---|---|---|---|---|---|---|---|
| TCS | 8,60,924 | 2,380 | 15.8 | 63.0 | 51.8 | 2.69 | 11 | 26 |
| INFY | 4,49,983 | 1,109 | 14.2 | 40.0 | 31.9 | 4.33 | 11 | 24 |
| HCLTECH | 3,64,907 | 1,345 | 19.2 | 30.4 | 23.8 | 4.02 | 12 | 21 |
| WIPRO | 1,83,395 | 185 | 13.9 | 17.8 | 15.5 | 5.94 | 9 | 20 |
| TECHM | 1,71,536 | 1,750 | 30.0 | 23.1 | 17.5 | 2.91 | 16 | 18 |
Five large-cap IT companies, nine fields each, in about four seconds and without opening a browser. Swap the list for your own holdings and the script does not change.
Chart the scraped data with Plotly
Once the numbers are in a CSV, Plotly turns them into a chart you can hover over in a notebook or export as a PNG. Each stock keeps one fixed colour, and the two measures get their own panels instead of sharing one chart with two y-axes.
"""Step 7: a Plotly chart from the watchlist CSV. Needs: pip install plotly kaleido"""
import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplots
wl = pd.read_csv("out/it_watchlist.csv", index_col=0).iloc[::-1] # top row drawn on top
colours = {"TCS": "#2a78d6", "INFY": "#eb6834", "HCLTECH": "#1baf7a",
"WIPRO": "#eda100", "TECHM": "#e87ba4"} # one colour per stock
fig = make_subplots(rows=1, cols=2, shared_yaxes=True,
subplot_titles=["Price to earnings (P/E)", "ROCE %"])
for col, measure in enumerate(["P/E", "ROCE %"], start=1):
fig.add_trace(go.Bar(
orientation="h", y=wl.index, x=wl[measure],
marker_color=[colours[s] for s in wl.index], marker_cornerradius=6,
text=[f"{v:.1f}" for v in wl[measure]], textposition="outside",
), row=1, col=col)
fig.update_xaxes(visible=False, range=[0, wl[measure].max() * 1.2], row=1, col=col)
fig.update_layout(title="IT watchlist: valuation against return on capital",
template="simple_white", showlegend=False, width=1000, height=520)
fig.write_html("out/it_watchlist.html") # interactive: hover, zoom, pan
fig.write_image("out/it_watchlist.png", scale=2) # static PNG for a report or blog
The same approach works for a single company. This chart is drawn from the quarterly CSV that the earlier script saved for Infosys:
PNG export goes through Kaleido, which uses the Chrome already installed on your machine. fig.write_html() needs nothing extra and keeps the chart interactive.
Step 6: Fetch technical data from Investing.com
Investing.com publishes a technical page for every instrument with an indicator table, moving averages and pivot points. The same helpers parse it, because the page reaches Python as markdown tables under headings.
"""Step 4: technical snapshot from an Investing.com technical page."""
from firecrawl import Firecrawl
from fc_helpers import md_sections, md_tables, to_number
URL = "https://www.investing.com/equities/infosys-technical"
app = Firecrawl(api_url="http://localhost:3002")
doc = app.scrape(URL, formats=["markdown"])
if not doc.markdown: # the site answered with a bot check, not a page
raise SystemExit(f"Blocked (HTTP {doc.metadata.status_code}). Stop and try again later.")
sections = md_sections(doc.markdown)
summary = next(name for name in sections if name.startswith("Summary:"))
print(doc.metadata.title)
print(summary.replace(":", ": "))
indicators = md_tables(sections["Technical Indicators"])[0]
indicators = indicators[["Name", "Value", "Action"]]
indicators["Value"] = indicators["Value"].map(to_number)
print("\nTechnical indicators")
print(indicators.to_string(index=False))
averages = md_tables(sections["Moving Averages"])[0]
averages.columns = ["Period", "_", "SMA", "SMA signal", "EMA", "EMA signal"]
averages = averages.drop(columns="_")
for column in ("SMA", "EMA"):
averages[column] = averages[column].map(to_number)
print("\nMoving averages")
print(averages.to_string(index=False))
pivots = md_tables(sections["Pivot Points"])[0].set_index("Name")
print("\nPivot points")
print(pivots.loc[["Classic", "Fibonacci"]].to_string())
indicators.to_csv("out/INFY_indicators.csv", index=False)
averages.to_csv("out/INFY_moving_averages.csv", index=False)
INFY Technical Analysis, RSI and Moving Averages - Investing.com
Summary: Strong Sell
Technical indicators
Name Value Action
RSI(14) 29.5260 Sell
STOCH(9,6) 11.7680 Oversold
STOCHRSI(14) 19.5940 Oversold
MACD(12,26) -9.0500 Sell
ADX(14) 22.6940 Neutral
Williams %R -95.0000 Oversold
CCI(14) -102.6496 Sell
ATR(14) 7.2950 Less Volatility
Moving averages
Period SMA SMA signal EMA EMA signal
MA5 1113.00 Sell 1113.56 Sell
MA20 1130.34 Sell 1126.72 Sell
MA50 1141.43 Sell 1137.45 Sell
MA200 1125.47 Sell 1126.94 Sell
Pivot points
S3 S2 S1 Pivot Points R1 R2 R3
Classic 1,102.47 1,104.93 1,109.87 1,112.33 1,117.27 1,119.73 1,124.67
Fibonacci 1,104.93 1,107.76 1,109.51 1,112.33 1,115.16 1,116.91 1,119.73
The output is shortened by a few rows. The page stamps this snapshot as 18 November 2026, 10:05 AM GMT, which is 3:35 PM IST. It is the default timeframe the page serves, and the labels such as Strong Sell are Investing.com’s own readings, not mine. The plain fetch took 902 ms.
Rate limiting: why Investing.com blocked the fifth request
While testing I fetched Investing.com pages four times in under half an hour: one command-line scrape, the robots.txt file, the Python script above and one more scrape to measure page size. All four succeeded. The fifth request came back like this:
success: True
blocked: True | source: local
statusCode: 403 | engine: local-fetch | title: Just a moment...
markdown chars: 0
That is a bot check. The site stopped serving the page and returned a challenge screen with HTTP 403. Firecrawl Local reported it honestly: "blocked": true, an empty markdown field and no error. This is why the script above checks for empty markdown before it parses anything.
It was not the only one. A small PDF on w3.org, which I picked to test PDF parsing, answered the very first request with the same 403 challenge.
Websites rate limit for good reasons. Servers cost money, automated traffic can swamp real visitors, and portals like Investing.com license their prices and indicators from exchanges and data vendors on terms that do not cover bulk copying. Rate limiting usually shows up in one of four ways:
- HTTP 429 Too Many Requests, often with a
Retry-Afterheader telling you how long to wait. - HTTP 403 with a challenge page such as the one above.
- A CAPTCHA or a login wall where the content used to be.
- A silent downgrade: empty tables, partial pages or stale data.
What to do when a site blocks you
I stopped requesting the site at that point. Rotating proxies, faking browser fingerprints or paying a CAPTCHA-solving service are ways of overriding a decision the site owner has made, and I do not recommend any of them. With an API key and FIRECRAWL_CLOUD=auto, Firecrawl Local can hand a blocked URL to Firecrawl Cloud. That changes who fetches the page. It does not change what the site’s terms allow.
For technical indicators there is a cleaner route anyway: pull OHLC candles from your broker API and compute RSI, MACD and moving averages yourself. You control the timeframe and the parameters, and nobody rate limits your own arithmetic.
JavaScript pages: when Firecrawl Local needs Chrome
Some tables are filled in by JavaScript after the page loads. On Screener the peer comparison is one of them. A plain fetch returns the placeholder text Loading peers table ... in its place. Tell Firecrawl Local to render the page in your local Chrome:
firecrawl-local scrape https://www.screener.in/company/INFY/consolidated/ --browser always --markdown
| S.No. | Company | CMP Rs. | P/E | Mar Cap Rs.Cr. | Div Yld % | NP Qtr Rs.Cr. | ROCE % |
| 1. | TCS | 2379.50 | 15.75 | 860923.92 | 2.69 | 13934.00 | 63.03 |
| 2. | Infosys | 1108.80 | 14.22 | 449983.43 | 4.33 | 7838.00 | 39.95 |
| 3. | HCL Technologies | 1344.70 | 19.18 | 364906.55 | 4.02 | 5132.00 | 30.40 |
The excerpt drops a few columns and the company links for width. The peers table is now in the markdown, at the cost of time: 2,808 ms against 245 ms for the plain fetch. Chrome runs headless with a throwaway profile. The --browser flag takes auto (the default), never or always, and the same choice is available through the FIRECRAWL_LOCAL_BROWSER environment variable.
Sections that need a login stay hidden. Screener’s Insights block came back as Please log in to see hidden values, which is exactly how it should be.
How fast is Firecrawl Local on a normal machine?
These are single runs from the command line, wall-clock time including process start-up:
| Page | Engine | Time | Raw HTML | Markdown |
|---|---|---|---|---|
| example.com | Plain fetch | 115 ms | n/a | n/a |
| Screener.in company page | Plain fetch | 245 ms | 2,57,708 chars | 28,105 chars |
| Investing.com technical page | Plain fetch | 902 ms | 9,21,198 chars | 19,568 chars |
| Screener.in with peers table | Local Chrome | 2,808 ms | n/a | n/a |
The last two columns matter more than the milliseconds if you work with language models. The Screener page shrinks about 9 times and the Investing.com page about 47 times when converted from raw HTML to markdown. Fewer characters means fewer tokens, lower cost and less noise for the model to wade through.
Memory stayed small throughout. The server used 13.5 MB when freshly started, 18.4 MB after its first request and about 46 MB after the batch jobs in this article.
Where does Firecrawl Local keep the fetched data?
Short answer: in RAM, and only for as long as it takes to hand the page back to you. Firecrawl Local has no database and no cache folder. After dozens of scrapes I searched my user profile for anything it had stored and found no data directory at all. A page is downloaded into the process’s memory, converted to markdown there and returned on standard output or in the HTTP response. What happens next is up to your code.
| Mode | Where the page lives | How long |
|---|---|---|
scrape from the CLI | RAM, then your terminal or the file you redirect to | Until the command exits |
serve, single scrape | RAM, then the HTTP response | Until the response is sent |
serve, batch and crawl jobs | RAM inside the server process | One hour from job start, or until the server stops |
crawl --out folder | One markdown file per page in that folder | Until you delete it |
| Pages rendered in Chrome | A throwaway Chrome profile in the temp folder | Removed when a CLI scrape exits |
| Your own scripts | Wherever you save: CSV, Parquet, SQLite | Your choice |
Because batch and crawl jobs sit in the server’s memory, collect the results before you close the server. That is also why its memory grew from 13.5 MB to about 46 MB during my tests.
The temporary Chrome profile and the agent’s context
The only thing written to disk without your asking is a browser profile. When the server needs Chrome it creates a folder named firecrawl-local-chrome-<pid>-<random> under %TEMP%. Mine grew to 63 MB. A one-off CLI scrape cleaned up after itself. When I stopped the server, both with a console break and with a forced kill, the folder and several headless Chrome processes stayed behind on Windows, so check %TEMP% after a long session.
One more point for agent users. When an AI agent fetches a page through Firecrawl Local, the markdown goes into the model’s context window. If the model runs in the cloud, the page text travels to that provider along with your prompt. The scraping is local. The model’s reading of it may not be.
Firecrawl Local for AI agents and MCP
Everything so far was Python calling a tool. The bigger idea behind Firecrawl Local is the other direction: an AI agent decides which pages it needs and fetches them itself. The bridge is MCP, the Model Context Protocol. If the term is new, read my plain-language guide to MCP for traders first.
An agent on its own cannot open a web page. It needs a tool, and the web fetch tools built into assistants are often limited: many return summaries, struggle with JavaScript pages or are metered. Firecrawl Local gives the agent a fetch tool that runs on your machine, costs nothing per page and returns compact markdown with the tables intact.
Connect Firecrawl Local to Claude Code, Cursor and Codex
One command registers the MCP server with every supported AI tool it finds: Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI and Codex. Preview it first with --dry-run. The block after the commands is an excerpt of what the dry run printed on my machine:
$env:HOME = $env:USERPROFILE # Windows only: without this, init stops with "HOME is not set."
firecrawl-local init --dry-run # show what would change
firecrawl-local init # write the config
Firecrawl MCP server:
Claude Code: would be added in C:\Users\<you>\.claude.json
Windsurf: would be added in C:\Users\<you>\.codeium/windsurf/mcp_config.json
Gemini CLI: would be added in C:\Users\<you>\.gemini/settings.json
Codex: would be added in C:\Users\<you>\.codex/config.toml
What goes to Firecrawl cloud:
- your search queries (there is no search index on a laptop)
- the URL of a page that blocked this machine, so Firecrawl can fetch it for you
- Alexandria data-provider calls, once you sign in
- a random install ID and the name of your AI tool, to count installs
Pages that load fine here are never sent. FIRECRAWL_CLOUD=search keeps only search;
FIRECRAWL_CLOUD=off turns the cloud off.
The docs say init backs up each config file before it edits it. To add the server by hand, register firecrawl-local mcp as a stdio server with the full path to the binary:
{
"mcpServers": {
"firecrawl": {
"command": "C:\\Users\\<you>\\.cargo\\bin\\firecrawl-local.exe",
"args": ["mcp", "--public-only"],
"env": { "FIRECRAWL_CLOUD": "off" }
}
}
}
--public-only makes the server refuse localhost and private-network addresses. Keep it on whenever an agent browses sites you do not control, so that a malicious page cannot trick the agent into reading your router or a local admin panel.
What the agent sees
To show there is no magic, this script plays the role of the agent. It starts firecrawl-local mcp, lists the tools and calls one, using nothing but JSON over standard input and output:
"""Step 5: speak MCP to firecrawl-local by hand, the way an AI agent does.
An agent starts `firecrawl-local mcp` as a child process and exchanges
newline-delimited JSON-RPC messages over stdin and stdout.
"""
import json
import os
import subprocess
env = dict(os.environ, FIRECRAWL_CLOUD="off") # keep everything on this machine
proc = subprocess.Popen(
["firecrawl-local", "mcp", "--public-only"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
text=True, encoding="utf-8", env=env,
)
def send(message):
proc.stdin.write(json.dumps(message) + "\n")
proc.stdin.flush()
def call(msg_id, method, params):
send({"jsonrpc": "2.0", "id": msg_id, "method": method, "params": params})
while True: # skip notifications and logs
reply = json.loads(proc.stdout.readline())
if reply.get("id") == msg_id:
return reply
hello = call(1, "initialize", {
"protocolVersion": "2025-06-18",
"capabilities": {},
"clientInfo": {"name": "marketcalls-demo", "version": "1.0"},
})
print("Server:", hello["result"]["serverInfo"])
send({"jsonrpc": "2.0", "method": "notifications/initialized"})
tools = call(2, "tools/list", {})["result"]["tools"]
print(f"\n{len(tools)} tools exposed to the agent:")
for tool in tools:
print(f" {tool['name']:<32} {tool['description'].strip().splitlines()[0][:70]}")
answer = call(3, "tools/call", {
"name": "firecrawl_scrape",
"arguments": {
"url": "https://www.screener.in/company/INFY/consolidated/",
"formats": ["markdown"],
},
})
text = answer["result"]["content"][0]["text"]
print("\nfirecrawl_scrape returned", len(text), "characters. First 300:")
print(text[:300])
proc.terminate()
Server: {'name': 'firecrawl-fastmcp', 'version': '3.40.0'}
30 tools exposed to the agent:
firecrawl_scrape Scrape one URL and return its content: markdown by default, or HTML, l
firecrawl_map Enumerate URLs indexed under one website through Firecrawl without fet
firecrawl_search Use this when you need current or specific information from the web: r
...
firecrawl_crawl Start a multi-page crawl at a website URL, poll it to a terminal state
firecrawl_check_crawl_status Retrieve the current status, progress, and available results for an ex
...
firecrawl_scrape returned 29596 characters.
The MCP server is the firecrawl-mcp npm package, started through npx, which is why Node.js 22 or newer is required. Of the 30 tools, the scrape, map and crawl tools run on your machine. The rest, such as search, the research tools and monitors, call Firecrawl Cloud. The tool list above is shortened where you see dots.
Agent use cases for traders and developers
With the server registered, you stop writing scrapers and start writing requests:
- Ad hoc fundamentals: “Scrape the Screener pages for TCS, Infosys and HCL Tech and build a table of P/E, ROCE and the last four quarters of sales growth.”
- Earnings season: “Read the latest concall transcript linked on this page and list management’s guidance changes.” PDFs are parsed on your machine: a 14-page public PDF took 3.2 seconds in my test.
- Second opinion: “Open the technical page for Infosys and tell me whether RSI and MACD agree with the moving averages.”
- Documentation for coding agents: crawl a broker’s API docs into markdown so the agent writes code against the current version, not its memory.
- Research pipelines: a scheduled agent that refreshes a watchlist CSV after results and flags the stocks whose margins moved.
Developers building their own agents in Python do not need MCP at all. Point the Firecrawl SDK at port 3002 and call it from a tool function, the same way I wired NSE data into an Agno stock market agent. One caution applies to every setup: text from a web page is data, not instructions. Do not give the same agent the power to place orders without a human confirming them.
Crawl and map: whole sites with Firecrawl Local
Two more commands matter when you need a site, not a page. map lists URLs from the sitemaps, optionally filtered by text. crawl follows links from a start URL and writes one markdown file per page. I tried both on my own site:
firecrawl-local map https://www.marketcalls.in --search openalgo --limit 8
firecrawl-local crawl https://www.marketcalls.in/python/ --limit 3 --out pages
firecrawl-local : 3 pages in 3.34s (0.9 pages/s), 0 errors, 0 skipped by robots.txt, saved to pages
Note the phrase skipped by robots.txt in that summary. The crawler reads each site’s robots.txt and obeys it by default. There is an --ignore-robots flag. Leave it alone. A folder of markdown files is the ideal input for a retrieval system or for an agent that needs to read a product’s documentation.
Ethical scraping practices for financial websites
A local scraper removes the cost of fetching. It does not remove the responsibility. These are the rules I follow, and the block from Investing.com is a reminder of why they exist.
- Read robots.txt and the terms before the first request. Screener’s robots.txt allows company pages and disallows user pages and search or sort queries. Investing.com’s robots.txt allows the technical pages, but its terms forbid storing or redistributing the data without written permission.
- Keep it personal. Screener licenses its pages for personal, non-commercial viewing. Analysing your own watchlist is one thing. Republishing a dataset or selling a product built on it is another.
- Fetch as rarely as the data changes. Financial statements change once a quarter. Fetching them every five minutes helps nobody.
- Cache everything. If you fetched a page today, read it from disk for the rest of the day.
- Go slowly. One request every few seconds per site. The CLI fetches 16 URLs at once by default, so pass
--concurrency 1or 2 when all the URLs are on one site. - Identify yourself. Firecrawl Local’s default user agent ends with
FirecrawlLocal/0.1.0 (+https://firecrawl.dev/local). Do not disguise it as something else. - Stay out of logged-in and paid areas. Never hand your session cookies to a scraper.
- Treat 403 and 429 as a stop sign. Back off for hours, not seconds, and never retry in a loop.
- Prefer the official route when one exists. Exchange filings, company investor pages, Screener’s own Export to Excel button for logged-in users, your broker API and licensed data vendors are all better sources for bulk or commercial work.
A polite fetch helper in Python
The helper below builds the first few rules into code: a one-day disk cache, a pause between live requests and a hard stop when the site says no.
"""Step 8: polite fetching. Cache what you fetched, pause between requests, stop on a block."""
import hashlib
import time
from pathlib import Path
from firecrawl import Firecrawl
app = Firecrawl(api_url="http://localhost:3002")
CACHE = Path("cache")
CACHE.mkdir(exist_ok=True)
PAUSE = 5 # seconds between live requests
MAX_AGE = 24 * 3600 # fundamentals change once a quarter: one fetch a day is plenty
_last_request = 0.0
def polite_scrape(url):
global _last_request
cached = CACHE / (hashlib.sha1(url.encode()).hexdigest() + ".md")
if cached.exists() and time.time() - cached.stat().st_mtime < MAX_AGE:
return cached.read_text(encoding="utf-8"), "cache" # no request leaves the machine
wait = PAUSE - (time.time() - _last_request)
if wait > 0:
time.sleep(wait)
doc = app.scrape(url, formats=["markdown"])
_last_request = time.time()
status = doc.metadata.status_code
if status in (403, 429) or not doc.markdown:
# The site said no. Do not retry in a loop and do not switch tricks.
raise RuntimeError(f"{url} answered HTTP {status}. Stopping.")
cached.write_text(doc.markdown, encoding="utf-8")
return doc.markdown, "live"
if __name__ == "__main__":
for symbol in ["TCS", "INFY", "TCS"]:
start = time.perf_counter()
md, origin = polite_scrape(f"https://www.screener.in/company/{symbol}/consolidated/")
print(f"{symbol:<6} {origin:<6} {len(md):>6,} chars {time.perf_counter() - start:5.2f} s")
TCS live 34,441 chars 2.21 s
INFY live 28,105 chars 7.11 s
TCS cache 34,441 chars 0.01 s
The second live request waited five seconds before it went out, and the repeat request for TCS never left the machine. Running the script again on the same day made no requests at all.
Common mistakes with Firecrawl Local
- Forgetting to start the server. The Python SDK needs
firecrawl-local serverunning in another terminal. A connection error on port 3002 means it is not. - Expecting cloud-only features. Local scrapes skip the
jsonandsummaryformats and ignore browser actions such as clicks. Ask for markdown and do the extraction in pandas or with your own model. - Closing the server before collecting a batch. Jobs live in memory and expire after an hour.
- Parsing a blocked page. Check for empty markdown or a 403 or 429 status before you index into tables.
- Using the plain fetch on JavaScript tables. If a section says Loading, rerun with
--browser always. - Rupee sign errors on Windows. Set
$env:PYTHONIOENCODING = "utf-8"before printing, and useOut-File -Encoding utf8when you redirect to a file. - Running
initon Windows without HOME. Version 0.1.0 stops with “HOME is not set.” Set it to your user profile first. - Leaving
FIRECRAWL_CLOUDon auto without realising it. Set it tooffif you want a strict guarantee that nothing leaves the machine.
Where Firecrawl Local fits in a trading workflow
Firecrawl Local is a research tool, not a data feed. It is very good at slow-moving public information and a poor fit for anything that needs ticks.
| Job | Good fit? | Better tool when it is not |
|---|---|---|
| Weekly fundamentals refresh for a watchlist | Yes | |
| Shareholding and peer comparison tables | Yes | |
| Reading concall transcripts and annual report PDFs | Yes | |
| Feeding web pages to an AI agent | Yes | |
| A daily indicator snapshot for a few swing ideas | With care | Compute from broker OHLC data |
| Intraday quotes, option chains, order flow | No | Broker API or a licensed datafeed |
| Backtesting on long price history | No | Historical data from your broker or vendor |
My own split is simple. Prices, orders and anything intraday go through the broker API. Fundamentals, filings and commentary come from the web through Firecrawl Local, cached on disk and refreshed when results come out. The two meet in pandas, and increasingly in an agent that can reach both.
If you want to try it this weekend, the order is: install the binary, scrape one Screener page from the command line, start the server, run the pandas script on your own watchlist and only then wire it into an agent. Each step works on its own, and each one is a few minutes.
Learn by watching: free videos every week
Market Profile, order flow, Python, AI and OpenAlgo tutorials on YouTube.







