Section 04
Connector Framework
The interface every court plugin implements, and the health system that makes silent breakage impossible. This is the most important document here — the connector fleet is the product.
What the old system already got right
49 of the old connectors expose the same route — POST /api/find, taking court_id, case_type, case_number, case_year, and returning a matter payload. That is already a connector interface; it simply was never named, versioned, or enforced. Formalising it is less invention than recognition.
The interface
class Connector(Protocol):
court_id: str
capabilities: set[Capability] # SEARCH, CAUSELIST, ORDERS, DISPLAY_BOARD
async def search(self, q: CaseQuery) -> RawResult:
"""Hit the court site. Network only, no parsing."""
def parse(self, raw: RawResult) -> ParsedCase:
"""Markup to structured data. Pure, offline-testable."""
def normalise(self, parsed: ParsedCase) -> Matter:
"""Court-specific shapes to the shared domain model."""
async def causelist(self, d: date) -> list[CauselistEntry] | None:
"""None if the court does not publish one."""Why parse() is separated from search()
This split is the highest-value design decision in the framework. Because parse() is pure and takes bytes, every connector can be tested against saved HTML fixtures with no network and no court site. That makes ~70 connectors testable in CI in seconds, and it means a court changing its markup produces a failing test rather than a silently empty result.
The old system interleaves fetching and parsing inside Selenium page objects, which is precisely why it has no meaningful connector tests.
Court configuration
Everything that varies without code changes lives in the registry:
court:
id: patna-hc
name: "Patna High Court"
connector: patna_hc
base_url: "https://patnahighcourt.gov.in"
transport: http # http | browser
captcha: numeric_ocr # none | numeric_ocr | audio | manual
concurrency: 2 # simultaneous requests — be a good citizen
poll_window: "01:00-05:00 IST"
rate_limit: "30/min"
case_types: [CWJC, CRWJC, MJC]transport matters economically. Only 15 of the old repos still drive Selenium; 73 use plain requests. Headless Chrome costs roughly 50–100× the CPU and memory of an HTTP call, so transport: browser should require justification, and moving a connector from browser to http is always worth an engineer's week.
Poll execution
flowchart LR
A[Scheduler picks
due matters] --> B{Court
circuit open?}
B -->|yes| SKIP[Skip, alert already raised]
B -->|no| C[Enqueue by court
respecting concurrency]
C --> D[Worker: search]
D --> E{HTTP ok?}
E -->|no| R[Retry with backoff]
R --> D
E -->|yes| F[parse]
F --> G{Parse valid?}
G -->|no| H[Record failure
trip circuit at threshold]
G -->|yes| I[normalise]
I --> J{Differs from
last known?}
J -->|no| K[Record run, done]
J -->|yes| L[Write matter +
emit CHANGE_EVENT]
L --> M[Notification dispatcher]
Fig 4.1 — Single poll execution path
Connector health
This is the system that protects revenue, and it should be built before any bulk connector migration — it doubles as the regression suite for that migration.
Golden-file tests
Saved HTML per court, asserted parse output. Runs in CI on every commit, no network.
Live canary
Scheduled real fetch of a known-stable case per court. Asserts non-empty, well-formed output. Catches markup drift the fixtures cannot.
Circuit breaker
N consecutive failures for a court stops polling it, flags matters Stale, and pages. Prevents hammering a down site.
Anomaly detection
Success rate or change rate deviating sharply from baseline. Catches the worst case: a connector returning plausible but wrong data.
The last one deserves emphasis. A connector that returns nothing is easy to detect. A connector that returns a stale-but-valid-looking page after a site redesign is the failure that causes a missed hearing, and only baseline comparison catches it.
Being a good citizen
Court websites are public infrastructure, frequently underfunded and easy to overwhelm. Clerkify's access should be defensible if a registrar ever asks about it.
- Per-court concurrency limits, set conservatively and honoured by the queue rather than by convention.
- Overnight poll windows, so load lands away from the court's own working hours.
- Identifiable user agent with a contact address. Anonymous scraping invites blocking; identified, rate-limited access invites a conversation.
- Cache aggressively. A case with no listing and no hearing for six weeks does not need daily polling at the same cadence as one listed tomorrow. Adaptive poll frequency is both cheaper and more courteous.
- Back off on failure rather than retrying hard. A court site that is down should see less traffic from us, not more.
Adaptive polling is the main cost lever
Given the economic constraint, poll frequency is worth real engineering attention. A matter with a hearing tomorrow warrants a poll today; a dormant matter with no activity in months does not. Tiering matters by expected activity — driven by next_hearing and recent change history — plausibly cuts total poll volume by more than half without any user-visible loss.
Adding a court
Save real HTML for a search hit, a miss, and an error page.
Implement parse() against fixtures. Pure function, no network, TDD is natural here.
search() — HTTP if at all possible; browser only with justification.
Add the config row, including concurrency and poll window.
Nominate a known-stable case. The court goes live only once the canary is green.
Target: under a day per court for a familiar site. The old system's answer to this question was "create a repository."