AI-accelerated ingestion refactor: coding agents (with cross-repo architecture context) helped the team rapidly redesign crawl orchestration, parser boundaries, shared platform tiers, and Azure Data Factory (ADF) merge patterns—not just patch HTML drift.
Result: ingestion cost down ~90% and reliability up 4×—the combination that made multi-brand scale credible on a startup budget.
The problem
The client’s value proposition depends on aggregating BAC and listing data scattered across national brokerage websites—each with different HTML, Web Application Firewall (WAF) defenses, and regional site maps. Investors and early customers needed evidence the team could process millions of listings monthly without enterprise-scale spend, plus a public prototype to test product-market fit (PMF).
Traditional virtual machine (VM)-based crawl farms and per-brand Container Apps Environments (CAEs) would burn the budget before revenue. Parser changes and pipeline failures were continuous toil a three-person startup could not absorb manually.
Our approach
- Tiered platform (Terraform + Azure DevOps) — shared platform resource group (RG), Azure Container Registry (ACR), one shared CAE (7 brand CAEs consolidated to 1), headless fetch tier, and per-brand parser spokes
- Ingestion factory — Container App Jobs → parser Functions → JSON Lines (JSONL) → ADF → serverless SQL with idempotent merge stored procedures (SPs)
- Multi-brand onboarding — manifest-driven crawls ramped to 60,000 pages/segment; six national brokerages production-ready in dev (June 2026)
- Commercial surface in parallel — Blazor MVP, Chrome extension (address → BAC lookup), Stripe subscriptions, go-to-market (GTM) automation
- Ops Orchestrator — Azure DevOps (ADO) ingestion incidents → agent remediation → pull request (PR) review → guarded deploy
Outcomes
- ~90% lower ingestion cost and 4× higher reliability after AI-accelerated refactor of ingestion components
- Zero customer-managed VMs — Functions, Container App Jobs, ADF, serverless SQL, Platform as a Service (PaaS) App Service
- Six national brokerage brands production-ready in dev; hourly ADF pipelines into a listings database exceeding 6M rows
- Public MVP + Chrome extension + Stripe — searchable listings, founding-member program, GTM infrastructure complete
- Graphify knowledge graph (~11k files) — agents and engineers navigate 20+ repos without tribal context
“We had to look like a product company while running a data engineering shop. AI didn’t replace judgment—it let us refactor the ingestion factory fast enough to survive on startup economics.”
All case studies · Client testimonials · AI activation case study · GCCH platform case study · Low-code telehealth case study · Cloud & platform services
