Skip to content
اردو
PKT · UTC+5

Scheduled (published schedule data), not measured outages.

Portfolio · how Roshni is built

From a utility's PDF to this page, every day

Roshni is a small data platform. It collects published schedules and reports, checks and models them in a warehouse, matches names across sources and publishes this static site. Each step below is a command in one Python command-line tool, and GitHub Actions runs them in order every day.

The pipeline

The lamps light in the order the data moves.

  1. 01

    Public sources

    Load-shedding schedules from IESCO and K-Electric, IESCO's planned-shutdown press releases, the PBS Census 2023, NEPRA's performance reports and approved news feeds. Only approved sources, and robots.txt is checked for every host.

    • robots.txt
    • httpx
    • Beautiful Soup
    • RSS
  2. 02

    Polite snapshots

    A daily job fetches each source at most once a second, asks only for files that changed, and stores every new file once, with its SHA-256 hash and fetch time, on a separate branch.

    • GitHub Actions
    • SHA-256
    • ETag / If-Modified-Since
    • git
  3. 03

    Parse and check

    One parser per source turns PDFs and spreadsheets into typed rows saved as Parquet. Rows that cannot be read go to quarantine with a reason, and expectation suites check every table.

    • Python
    • Docker
    • poppler pdftotext
    • openpyxl
    • pydantic
    • Parquet
    • Great Expectations
  4. 04

    Warehouse

    SQL models build staging views, joins and marts in DuckDB, with tests on every model. The same models run unchanged on Databricks.

    • dbt
    • DuckDB
    • Databricks
  5. 05

    Match and analyse

    Feeder names are matched across sources by name similarity. Exact matches are accepted; the rest are decided by written rules, then by a language model choosing only among listed candidates, then by Claude, each with a stored confidence. Descriptive statistics and a weekly summary follow: the model writes prose only, and every number in it is checked.

    • RapidFuzz
    • OpenRouter
    • Claude
  6. 06

    Export and publish

    Tables are exported as versioned JSON, validated against a schema and typed into the site, which is built as static pages in English and Urdu and pushed to GitHub Pages.

    • JSON Schema
    • Next.js
    • React
    • TypeScript
    • Tailwind CSS
    • GitHub Pages

What runs it

GitHub Actions is production: a daily snapshot job, then the publish job (parse, validate, build, match, report, export, site), and news feeds every three hours. A failure opens an issue. An Airflow DAG in Docker runs the same commands locally as a showcase.

  • GitHub Actions
  • Apache Airflow
  • Docker Compose
  • Typer CLI
  • uv

What keeps it honest

Golden tests on every parser, expectations on parsed tables and tests on every model. The site has its own gates: every number on the Karachi page is traced back to the export, translations are checked, pages are checked for accessibility in light and dark, and the browser's scheduled-dark logic is compared with Python's on the full export.

  • pytest
  • ruff
  • pre-commit
  • dbt tests
  • Great Expectations
  • Vitest
  • axe-core
  • ESLint

In today's build

schedule files in use
8
feeders with a published schedule
1,719
planned shutdown windows
2,675
company-years from NEPRA
130
headlines on file
12

Counts from the published files. Last updated:

Look under the hood

Rules it follows

  • Schedules are not measurements: a scheduled window is never called an actual outage.
  • Nothing is invented: unreadable rows are quarantined with a reason, and known anomalies are flagged.
  • Times are shown in Pakistan time (UTC+5) and stored in UTC.
  • Approximate matches always show their confidence and who accepted them.

Source of the published site: github.com/azzammasood/roshni-site

Logos are trademarks of their owners, shown only to name the tools used (icons from Simple Icons, CC0). Tools without a published icon are shown as lettered tiles.