Lesson 05 · Tools & data

Quant tooling: data, scraping, backtesting & broker APIs

By Robin CB · AI market analyst & investor

For a retail learner with a laptop, wifi, and AI tools — no capital needed to start. The ground rules: use official data paths, respect the terms that prohibit scraping, and treat every backtest as guilty until proven honest.

Last updated 31 Oct 2026 · Prices, limits, and features change — re-check the cited official page before spending money or placing orders

How evidence is labelled here

FACT verified against an official page or primary document. INTERPRETATION a reasonable reading of the facts. UNVERIFIED found only in secondary sources — a lead, not a fact.

Market data sources for India

NSE official data — the ground truth, mostly free at end of day

FACT. NSE publishes free daily "bhavcopy" files (end-of-day OHLCV for every traded security) through its All Reports page — e.g. the file pattern sec_bhavdata_full_DDMMYYYY.csv served from NSE's archives. The same page carries derivatives bhavcopies, open interest, delivery data, and corporate-action files. Nearly every Indian data product is built from these files.

FACT. NSE also sells deeper data — paid EOD/historical order and trade data via SFTP/online platform, tariff PDFs on its subscription page (updated 02/09/2026). If a vendor claims "official NSE tick data," this is the licence they would hold — ask for it.

FACT. NSE launched official MCP servers exposing 25+ tools, including bhavcopy and cash-market endpoints, with 5 years of bhavcopy history and cash-market live data delayed 1–3 minutes. Third-party MCP wrappers with 60+ tools exist but are NOT official — do not confuse them with NSE's own servers. The NSE website shows the live option chain in the browser; there is no documented public REST API for it.

BSE. BSE publishes its own daily bhavcopy and market reports from bseindia.com. The exact current BSE download URLs and any BSE equivalent of NSE's MCP servers are UNVERIFIED — check bseindia.com directly before building around BSE files.

Yahoo Finance / yfinance — free, unofficial, fragile

FACT. Yahoo Finance has no official public API (its old YQL API was retired in 2017). The Python library yfinance wraps Yahoo's undocumented internal endpoints. It is the fastest way to get adjusted daily history for NSE symbols (RELIANCE.NS, TCS.NS). The endpoints break whenever Yahoo changes authentication, and heavy use triggers HTTP 429 throttling or temporary IP blocks. Reported mitigations: sleeps between requests, aggressive caching, keeping the library updated. Rate numbers quoted in blogs (e.g. "~2,000 requests/hour") are UNVERIFIED. Prices are typically delayed 15–20 minutes; never use Yahoo as a live-trading feed.

TradingView, Alpha Vantage, TejHQ

TradingView covers NSE/BSE symbols with browser charts, screeners, and Pine Script; its free tier is genuinely usable. Exact free-tier limits change frequently — check tradingview.com/pricing. TradingView is a research terminal, not a data pipeline: bulk historical export from the free tier is restricted.

Alpha Vantage publishes a standard limit of 25 requests/day on its free tier, with 20+ years of equity history plus FX and commodities endpoints. Paid-tier figures quoted in secondary sources ($49.99–$249.99/month) are UNVERIFIED. A clean, documented fallback for global symbols and FX; for Indian EOD history it is rarely better than free bhavcopy + yfinance.

TejHQ (project's own site): starts from raw NSE/BSE bhavcopy and publishes cleaned Parquet datasets on HuggingFace, including split/bonus/dividend-adjusted closes keyed by ISIN and point-in-time liquid universes. Its pipeline code is open-source (MIT). Adjusted prices require a free key; universes sit behind a Retail plan quoted at ₹599/month on the site. The only source found that solves the two hardest EOD-data problems (adjustment and survivorship) in the open. Verify current pricing and coverage before depending on it.

Source	Cost	Latency	Best for	Restriction
NSE bhavcopy / reports	Free	EOD	Ground-truth daily OHLCV, OI, delivery	ToS bans automated scraping
NSE paid historical data	Paid (tariff PDFs)	EOD/tick	Licensed bulk history	Licence terms
NSE official MCP	Free (observed)	1–3 min delayed	Programmatic bhavcopy access	Official servers only
BSE bhavcopy	Free	EOD	BSE-only names	URLs UNVERIFIED
Yahoo/yfinance	Free	15–20 min delayed	Quick adjusted history, prototyping	Unofficial; 429s; no redistribution
TradingView	Free/paid	Near real-time charts	Charting, screening, Pine backtests	Export limits; not a pipeline
Alpha Vantage	Free 25/day; paid tiers	EOD/intraday	Documented API, FX/commodities	Small free tier
TejHQ	Free key / ₹599 mo (site quote)	T+1 EOD	Adjusted, survivorship-free history	EOD only

Web scraping — what is allowed, what is not

The hard rule: NSE's Terms of Use prohibit scraping

FACT. NSE's Terms of Use, clause 9, prohibit "any systematic or automated data collection activities (including without limitation scraping, data mining, data extraction and data harvesting)" on the NSE website/mobile app, and reserve NSE's right to investigate, take legal action, and terminate access. The same prohibition appears in Nifty Indices' terms. This is the single most important fact in this lesson for anyone building a scraper.

A tutorial claim found during research — that NSE "allows scraping for personal, non-commercial use only" — directly contradicts the Terms text above. Treat that claim as UNVERIFIED and unsafe to rely on. The practical reading: downloading the published bhavcopy files (which NSE provides as downloadable reports) is the intended access path; hammering the website's JSON endpoints with a bot is the prohibited one. NSE's web endpoints use an anti-bot cookie handshake — a technical control, not permission. Circumventing it at volume invites IP blocks, and the Terms give NSE legal recourse on top.

Ethical scraping practice (for sites that DO permit it): read robots.txt and the ToS first and obey the more restrictive; identify yourself with a real User-Agent and contact details; one request at a time with sleeps and exponential back-off on 429/503; cache everything (market data is immutable history — download once, store locally); respect the data licence (exchange data is licensed, not owned — do not redistribute or resell raw exchange data); prefer the official API whenever one exists.

Decision rule. Use an official/broker API when you need live prices, order placement, historical candles at scale, or anything you will show another person. Scrape (only where permitted) for one-off research, cached, at human pace. If your system breaks when a site changes its HTML, it was never a data pipeline — it was a fragile hack. Build on files and APIs.

Broker APIs in India

The regulatory frame you must know first

FACT. SEBI's circular "Safer participation of retail investors in Algorithmic trading" (4 February 2025, extended 30 September 2025) became fully mandatory for all brokers on 1 April 2026. Core requirements: the broker is principal and accountable for API orders; every algo order carries an exchange-issued Algo ID; secure API access (OAuth + two-factor authentication, registered static IP — one primary plus one backup); orders above 10 per second must be registered with the exchange; kill-switch capability; a 5-year audit trail; open/unrestricted APIs prohibited. For a personal learner: API order placement now requires a static IP registered with your broker; staying under 10 orders/second keeps you in the unregistered personal-algo lane; paper-trading systems that never touch the broker sit outside this framework entirely.

Broker comparison

Broker	API cost (verify)	Live + historical data	Sandbox / paper trading
Zerodha Kite Connect	Personal free (no data); Connect ₹500/mo/app	Connect only	None (official) — build your own mock
Upstox	Verify pricing; ₹10/order reported	Analytics Token reported free	Mock-only sandbox — no fills/positions
Angel One SmartAPI	Described free — verify	Yes (limits UNVERIFIED)	None confirmed — UNVERIFIED
Dhan DhanHQ	Trading free; Data ₹0/₹499 reported	Yes	Real sandbox (sandbox.dhan.co)
Fyers	Free + brokerage; Prime ₹499/mo reported	Yes (Prime raises limits)	None confirmed — UNVERIFIED

Zerodha Kite Connect (official support pages). Two tiers: Kite Connect Personal — free (orders, GTT, alerts, margin, portfolio; no live market data or historical data) and Kite Connect (Connect plan) — ₹500/month per app (adds WebSocket real-time data and historical candles). From October 2026, users generating ≥ ₹2,000 brokerage in a calendar month receive 500 Developer Credits (₹500) covering the next month's renewal. Third-party reviews still quoting ₹2,000/month conflict with the official page — treat the official page as primary and ₹2,000 as outdated/UNVERIFIED. Kite Connect data may not be displayed/redistributed externally. Rate limits quoted in secondary guides (~1 quote/sec, 3 historical requests/sec, 10 order requests/sec) are UNVERIFIED — verify against the Kite Connect docs.

Upstox (official rate-limit page). Combined order-API limits for regular algos: 10/sec, 500/min, 2,000/30 min. Its sandbox is mock-only: place/modify/cancel return mock success with no positions, holdings, fills, margin, or rejections and no live LTP; token valid 30 days; one sandbox app per user.

Dhan (official v2 docs). Rate limits: Order APIs 10/sec, 250/min, 1,000/hr, 7,000/day; Data APIs 5/sec, 100,000/day; Quote APIs 1/sec, unlimited. Dhan provides a sandbox at sandbox.dhan.co with the same API structure, no KYC/funds/data subscription needed; switching is a base-URL change. Pricing is UNVERIFIED — Trading APIs described as free, Data APIs free if 25 trades occurred in the last 30 days, else ₹499 + taxes (per a third-party review).

Fyers. API documentation at myapi.fyers.in/docsv3; the API itself is free with standard brokerage applying. Revised category-wise limits are UNVERIFIED (support article + secondary sources conflict in detail). No sandbox confirmed.

Interpretation. For a learner with zero capital, Dhan's sandbox and Upstox's mock sandbox are the only broker-provided safe environments found; Zerodha's ecosystem is the most documented but forces you to build your own paper-trading layer. In every case: paper-trade in your own code against live or delayed prices, and treat the broker API as the last mile for real orders only after months of clean paper results.

The Python quant stack

pandas + NumPy are the substrate of essentially all retail quant Python. If you can manipulate a DataFrame fluently — filtering, grouping, rolling windows, shifting — you can express almost any indicator or backtest; learn this before any specialised library.

Data-access wrappers (all unofficial — handle with care). yfinance — Yahoo data into pandas; cache results, expect breakage. nsepython — unofficial NSE wrapper; replicates the cookie handshake; not endorsed by NSE; its use sits under the ToS prohibition above — use official bhavcopy/MCP routes for anything durable. nselib, nsedt, nsefin — sibling unofficial wrappers, same caveats. Bhavcopy direct: read the CSVs with pandas — no wrapper needed, no ToS grey area, fully reproducible. This is the recommended default for EOD research.

Backtesting engines. vectorbt — vectorised backtesting on pandas/NumPy/Numba; best for scanning many parameter combinations fast; its speed is also its danger (see overfitting below). backtrader — event-driven engine, slower but closer to how live trading actually fills; secondary reports describe it as unmaintained since 2019 with Python 3.10+ issues — UNVERIFIED; test on your own Python version. TradingView Pine Script — fastest way to see an idea on a chart; not a Python pipeline.

Indicators: TA-Lib vs pandas-ta. TA-Lib: 200+ indicators in C, very fast; installation is the pain point (the C library must be installed first). pandas-ta: 130+ indicators, pure Python, much easier install — but the maintainer warned the library would be archived unless significant additional support arrived by 1 July 2026; forks exist (pandas-ta-openbb, pandas-ta-kozmoai). Whether archiving actually happened is UNVERIFIED — check the repository's current status. For a beginner, computing the 5–10 indicators you actually use directly in pandas (SMA, EMA, RSI, ATR, VWAP) is often more robust than depending on either library.

How to structure a backtest honestly

Most backtests lie. A backtest's job is not to prove a strategy works — it is to fail weak strategies cheaply.

Look-ahead bias. Using information unavailable at decision time — deciding on a bar's close and filling at that same close, or normalising over the whole series. Fix: signals from data strictly before the bar; fill at the next bar's open; all statistics on trailing windows only.
Survivorship bias. Testing only today's index members — pre-selected winners. One cited test put the distortion at ~18% vs ~11% realistic returns (illustrative only). Fix: point-in-time universes, keep delisted names, treat delisting as an exit.
Overfitting / multiple testing. Scanning hundreds of parameter combinations guarantees some combination looks brilliant by chance. Fix: cap combinations (one source suggests ~50), prefer few parameters and robust plateaus over sharp peaks.
Transaction costs. Brokerage, STT, exchange charges, GST, SEBI charges, stamp duty, spread, slippage. One source notes costs can erase 20–40% of gross returns in high-frequency strategies (illustrative). Fix: model all-in costs from your broker's actual charge sheet, plus a slippage assumption, in every backtest.
Validation discipline. Split in-sample (optimise) / out-of-sample (test once); walk-forward analysis; one locked holdout set touched exactly once; never shuffle time series.
Same code path. Backtest and live/paper systems must share signal code. Re-implementing the strategy for live trading is how "it worked in backtest" becomes "it lost live."

Screeners & alerts

Screener.in — fundamentals-first, ~5,000 NSE+BSE companies, ~50 free filters, custom formula queries (e.g. ROE > 20 AND Debt/Equity < 0.5), 10+ years of financials. Free core tier; premium quoted at ₹499/month in a secondary comparison — verify on screener.in. Best for long-term quality/value screens.
TradingView screener — technical + fundamental, NSE/BSE coverage, strong free tier; Pro plans quoted from ₹999/month in secondary sources — verify.
Chartink — India-focused real-time technical scans; free and premium tiers. Best for intraday/swing scans.
Tickertape, Moneycontrol, Investing.com — fundamental + technical screeners with free tiers; useful cross-checks. NSE website — basic screening free on nseindia.com.

Interpretation: a screener answers "which stocks deserve a look today." It is a filter, not a decision — every result still needs the fundamental/technical work in the other lessons.

Build your own screener (one evening, bhavcopy + pandas): download the day's bhavcopy plus ~250 trading days into one Parquet file keyed by symbol + date; compute per-symbol features (returns over 1/3/6/12 months, distance from 52-week high, volume vs 20-day average, ATR, price vs 50/200-day SMA); apply boolean filters with a liquidity floor (e.g. average daily value > ₹1 crore — avoids names you cannot exit); output a ranked CSV with a one-line reason per name; keep every day's output so you can later ask "did the screen predict anything?" (that question is a backtest). Alerts: notify only when the list changes.

Weekend starter project: paper-trading + journaling system

Goal: by Sunday night, a system that (a) pulls EOD data legally, (b) runs one simple, fully-specified strategy, (c) simulates fills honestly, and (d) forces a written reason for every trade. No broker account needed; no real money at risk.

Saturday — data + engine. Set up Python 3.11+, pip install pandas numpy yfinance. Download 3 years of daily data for Nifty 50 constituents (yfinance, cached to Parquet) AND the same period's NSE bhavcopy files; cross-check closes on 5 random dates; record source and download date in a DATA_README.md. Write one strategy, in markdown, before coding it — e.g. "Buy when close crosses above 50-day SMA with volume > 1.5× 20-day average; exit on close below 50-day SMA or −8% stop, whichever first; max 5 positions; equal weight; no leverage; no F&O." Simulator: fill at next open, deduct realistic costs plus 5 bps slippage assumption (documented), track cash, positions, and equity curve daily. Include delisted names if your universe had any.

Sunday — paper trading + journal. Each evening after close, recompute signals and append proposed orders to paper_orders.csv (date, symbol, side, intended price, rule that fired). Nothing executes. Record the next day's actual open as the fill. Journal: one row per paper trade — date, setup, entry reason in one sentence, planned stop, planned exit, outcome, and whether you followed the rule. Review weekly: win rate and average win/loss are secondary; rule-following rate is primary. Dashboard: one page showing equity curve, drawdown, trade count, rule-following rate, and the 3 worst trades with journal notes.

Backtesting checklist — run before trusting any result

Signals use only data available before the bar; fills at next open. Universe is point-in-time (delisted names included, or the bias stated). Parameter combinations counted and capped; no peeking at the holdout. All-in costs + slippage modelled. Out-of-sample and walk-forward tests passed; one locked holdout untouched. Results reported with drawdown and trade count, not just return. Same code path for backtest and paper trading. Rules written down before the test. Corporate actions handled — or the limitation written in the README. A losing outcome has a pre-committed response, decided in advance.

Data quality pitfalls

Adjusted vs unadjusted prices. FACT: NSE/BSE bhavcopy prices are not adjusted for splits and bonuses — a 2:1 split shows as a ~50% "crash" in raw bhavcopy. FACT (Zerodha support forum): Kite's historical charts are adjusted for bonus issues (equity; F&O charts are not adjusted). Yahoo's Adj Close is adjusted for splits and dividends. Never mix sources in one series.
Corporate actions generally. Splits, bonuses, rights, buybacks, mergers/demergers, dividends each change the price series, and NSE's PREV_CLOSE is not adjusted on split/bonus ex-dates. Keep a corporate-action calendar and apply adjustments from ex-dates, or use a pre-adjusted source.
Symbol changes, mergers, ISIN continuity. Key long histories by ISIN (present in bhavcopy files) and maintain a symbol-history map; TejHQ ships one.
Survivorship and delisting. Delisted names' final prices are real exits, not missing data — treat a delisting as a terminal event with stated recovery assumptions, not a row to drop.
Volume and timestamp mismatches across vendors. FACT (Upstox community investigation): daily candle volume and summed intraday volume differed by thousands of shares for RELIANCE/TCS/HDFCBANK on 1 Feb 2025, and daily closes sometimes reflect the last trade rather than the official exchange close. Cross-check any vendor's candles against bhavcopy on a sample of dates.
Timezone and session traps. NSE trades 09:15–15:30 IST; daily timestamps at 00:00 IST are a convention, not a trade time; intraday bars labelled 15:25 may exclude the closing auction. Document the timestamp convention of every dataset.
Stale/duplicate rows. Bhavcopy files occasionally contain duplicate or corrected rows. Dedupe on (symbol, date), keep file hashes, and re-download rather than patch when a file is revised.

Sources

Note: yfinance, nsepython/nselib/nsedt/nsefin, backtrader, and TradingView pricing are discussed in the text without official URLs because no official documentation URL was returned by research for those items; package names and PyPI are the lookup keys. BSE specifics, Angel One pricing/sandbox status, Fyers sandbox status, and several broker rate limits are marked UNVERIFIED in the text.