Architecting Zero-Cost Data Applications: Leveraging Claude Code and Databricks AI Functions for Large-Scale Unstructured Data Extraction

In the current landscape of generative AI development, a significant gap exists between "demo-ware" and production-ready applications. Most tutorials culminate in a polished frontend displaying static JSON datasets running on localhost. While visually impressive, these demonstrations fail to address the fundamental complexities of real-world data engineering: unstructured data ingestion, schema enforcement, high-latency API orchestration, and scalable deployment.

This post explores a methodology for building a fully deployed, production-grade data application—tracking tech job market trends from Hacker News (2020–present)—with zero infrastructure overhead by integrating Claude Code with the Databricks Free Edition.

The Challenge of Unstructured Data at Scale

The project objective was to transform approximately 40,000 unstructured "Who is Hiring" threads from Hacker News into a structured, queryable SQL database. The raw data presents several high-entropy challenges:

1. Lack of Uniformity: Salary information ranges from explicit figures (e.g., "$150k") to vague descriptors ("competitive").
2. Entity Extraction Complexity: Identifying specific technical skills (e.g., "Rust" vs "Go"), seniority levels, and remote work policies requires semantic understanding rather than simple regex patterns.
3. Orchestration Overhead: Traditional ETL pipelines for this task typically require a Python-based loop that iterates through rows, calls an external LLM API (like OpenAI or Anthropic), manages rate limits, handles retries, and writes results back to a database—a process prone to failure and high token costs.

The Architecture: A Unified Data Platform

To bypass the "four-service problem" (separate bills for storage, compute, AI, and hosting), we utilized Databricks as a unified backend. Databristic provides four critical components in a single workspace:

• Storage: A managed catalog for SQL tables.
• Compute/Hosting: The ability to run FastAPI and React applications directly on the platform's compute resources.
• AI Gateway/Functions: Native integration of LLMs within the SQL engine.
• SQL Engine: High-performance querying for real-time analytics.

Phase 1: Agentic Planning with Claude Code

The development workflow began in Claude Code, utilizing its "Plan Mode." In agentic workflows, the most critical step is establishing a deterministic execution plan before allowing the model to enter "Auto Mode" (execution).

The prompt provided to Claude Code defined the tech stack (React and FastAPI) and the specific requirement for a one-time backfill of historical data. By forcing the model into Plan Mode, we could audit its proposed architecture—specifically ensuring that it accounted for the limitations of the Databricks Free Edition (e.g., manual data ingestion requirements).

Phase 2: Integrating Claude Code with Databricks CLI

To enable Claude Code to interact directly with our infrastructure, we implemented a specialized integration using the Databricks CLI. This involved two critical steps:

1. Authentication: Using databricks auth login --host <workspace-url> to establish a secure session via OAuth.
2. Agentic Skill Injection: Running databricks ai tools install --agents Claude Code.

This second step is technically significant; it installs specific "skills" (similar to Model Context Protocol/MCP servers) that provide the LLM with the necessary context and tool-calling capabilities to execute Databricks-specific commands, such as querying tables or managing compute.

Phase 3: The ETL Pipeline and AI Functions

The data ingestion phase involved hitting the Hacker News API to scrape top-level comments from all "Who is Hiring" threads since 2020. Once the raw text was ingested into a Databricks SQL table, we bypassed traditional Python-based LLM orchestration in favor of Databricks AI Functions.

Instead of writing an external script to loop through 40,000 rows, we utilized the ai_extract function directly within a SQL query. This allows the LLM to be invoked as a native SQL operator:

SELECT 
  ai_extract(post_content, 'JSON schema for skills, salary, seniority, and remote policy') 
FROM hn_jobs_table;

Technical Advantages of ai_extract:

• Zero Orchestration: No need to manage asynchronous loops or stateful retry logic in Python.
• Reduced Latency/Cost: The models are hosted within the Databricks workspace, minimizing data egress and eliminating the need for external API keys (OpenAI/Anthropic).
• Atomic Operations: The extraction is treated as a standard transformation step within the SQL engine, ensuring consistency across all 40,000 records.

During testing, we iterated on model selection to optimize for accuracy in extracting inconsistent salary formats and skill lists, eventually applying the most performant model to the entire dataset.

Phase 4: Deployment and Real-Time Analytics

The final stage involved deploying the React frontend and FastAPI backend onto Databricks compute. The resulting application provides a live interface for querying the processed data. We implemented a natural language interface using Databricks Genie, allowing users to ask questions like "What is the median salary for Go developers?" in plain English.

The system translates these natural language queries into optimized SQL, executes them against the structured hn_jobs table, and returns real-time visualizations (e.g., skill growth trends, salary inflation from 2020 to 2026).

Conclusion

By moving away from localized, static demonstrations and leveraging agentic coding tools like Claude Code alongside unified platforms like Databricks, we can build scalable, production-ready data applications at zero infrastructure cost. The key lies in utilizing AI Functions to move the intelligence to where the data lives, rather than attempting to move massive datasets to the intelligence.