01753 Websites, Zero Standards
Nepal restructured into 753 local levels — metropolitan cities, municipalities, and rural municipalities — each with its own website and its own way of publishing tender notices. Add provincial ministries and federal departments on top.
For a contractor in, say, Pokhara who bids on road works, this means checking dozens of sites manually, every day, forever. Miss a day and you miss a tender you could have won.
What “no standards” actually means
- Some sites publish HTML tables; others publish a list of PDF links
- Dates appear in BS, AD, or both — sometimes in the same table
- Notice titles range from a full description to just “सूचना”
- Some sites have no HTTPS; some have expired certificates
- Uptime varies wildly — a site may be down for days
- Character encoding is inconsistent for Devanagari text
02System Architecture
SCHEDULER COLLECTORS PIPELINE ┌────────────┐ ┌───────────┐ ┌───────────┐ │ cron 3×/day │ → │ fetcher │ → │ normalize │ │ per-office │ │ + retry │ │ dedupe │ │ stagger │ │ + backoff │ │ classify │ └────────────┘ └───────────┘ └───────────┘ ↓ ┌───────────┐ │ SEARCH DB │ │ + filters │ └───────────┘
Each office is an independent collection job. One office being down never blocks the others — a lesson learned the hard way after a single unresponsive server stalled an entire early run.
03The Scraping Layer
Rather than 753 bespoke scrapers, the collectors are built around a small set of adapter patterns. Most government sites in Nepal run on a handful of common CMS templates, so one adapter often covers dozens of offices.
# Each office maps to an adapter + a config, not custom code { "office_id": "pokhara_metro", "name": "Pokhara Metropolitan City", "province": "Gandaki", "adapter": "table_listing_v2", "url": "https://.../notices", "selectors": { "rows": "table.notice-table tbody tr", "title": "td:nth-child(2)", "date": "td:nth-child(3)", "link": "td:nth-child(2) a@href" } }
Being a good citizen
# Rate limiting and politeness rules CONCURRENT_PER_HOST = 1 # never hammer one office DELAY_BETWEEN_REQS = 2s # breathing room TIMEOUT = 30s # slow servers are common RETRY_BACKOFF = 2x # exponential, max 3 tries USER_AGENT = "BidPalika/1.0 (+https://calenote.app/bidpalika)" # Conditional requests — don't re-download unchanged pages If-Modified-Since: <last successful fetch> If-None-Match: <stored ETag>
04Normalizing Chaos
The hardest problem isn't fetching — it's making 753 different output formats look like one thing.
Dates
A notice date might be २०८३/०६/१५, 2083-06-15, 15 Ashoj 2083, or 2026-10-01. The parser tries BS patterns first (including Devanagari numerals), falls back to AD, and stores both.
# Devanagari numerals → ASCII before anything else DEVANAGARI = "०१२३४५६७८९" def normalize_digits(text): for i, d in enumerate(DEVANAGARI): text = text.replace(d, str(i)) return text def parse_notice_date(raw): text = normalize_digits(raw.strip()) # BS years are 2000-2100; AD years are 1900-2100 # The leading number disambiguates most cases for pattern in BS_PATTERNS: if m := pattern.match(text): return bs_to_both(m) for pattern in AD_PATTERNS: if m := pattern.match(text): return ad_to_both(m) return None # flagged for manual review
Categories
Notices get classified into Works, Goods, Services, or Quotation using keyword matching over both Nepali and English terms — निर्माण and “construction” both signal Works, मालसामान and “supply” signal Goods.
05Deduplication
The same tender often appears multiple times — republished, amended, or listed under several categories. Dedup uses a composite fingerprint:
def fingerprint(notice): # Exact match on office + reference number is definitive if notice.reference_no: return hash((notice.office_id, notice.reference_no)) # Otherwise: office + normalized title + published date title = collapse_whitespace(strip_punctuation(notice.title)) return hash((notice.office_id, title, notice.published_ad))
Near-duplicates (an amended notice with a slightly different title) are detected with token-set similarity and linked as revisions rather than hidden — because an amendment is meaningful information for a bidder.
06Search & Filtering
Search has to work in both scripts. A contractor might type “बोलपत्र” or “bolpatra” or “tender” and expect the same results.
🔍 Bilingual index
Each notice is indexed under its Nepali text, a romanized transliteration, and mapped English keywords.
🗺 Geographic filters
Narrow by province → district → local level, or switch to federal and provincial notices only.
📋 Category filters
Works, goods, services, and quotations — so a road contractor never sees stationery supply notices.
⏱ Recency filters
Today, yesterday, 7 days, 30 days. Newest first, so a daily glance is enough.
07Doing This Responsibly
Scraping government sites at scale carries real responsibility. The rules I hold to:
1. Only public information.
Nothing behind a login, nothing requiring a bypass. These notices are published for public consumption by law.
2. Always link back to the source.
Every card opens the issuing office's own page. BidPalika is an index, not a replacement for the official notice.
3. Never republish terms.
Closing dates and conditions are set by the office and can be amended. Users are told, prominently, to confirm on the official notice.
4. Identify the crawler honestly.
A descriptive user agent with a contact URL. No pretending to be a browser.
08Lessons Learned
Government sites go down. Plan for it.
Every collector isolates failure. A dead office logs an error and the run continues. Coverage is reported honestly — a province shows “live” only when its offices actually publish.
Encoding will break your heart.
Devanagari served as Latin-1, double-encoded UTF-8, legacy Preeti font mappings — all of it in production. Normalize aggressively at ingest.
Config beats code.
Site redesigns are constant. When an office changes its layout, updating a selector in config takes minutes; editing a bespoke scraper takes an evening.
Honesty about limits builds trust.
Saying “we couldn't extract the deadline — check the source” is far better than showing a confident wrong date. Users notice.