Open Source Stack

The technology powering this platform — 100% open source, zero vendor lock-in.

Pipeline Flow

JDE SQL Server→Node.js Extractor→Bronze (PostgreSQL)→dbt Silver→Splink MDM→dbt Gold→Fastify API→Next.js Dashboard

Orchestrated nightly by Apache Airflow · Deployed on Railway Cloud

PostgreSQL

Database

The world's most advanced open source relational database. Powers all three layers of the medallion architecture — Bronze, Silver, and Gold.

"Primary data store for all extracted, transformed, and aggregated JDE data."

ACID compliantSchemas for layer isolationIndexing for query performanceFull SQL support

dbt Core

Transformation

Data build tool — transforms raw data in the warehouse using SQL and Jinja templating. The T in ELT.

"Builds all Silver and Gold models from Bronze raw data. Handles typing, deduplication, and business aggregations."

SQL-based transformationsDependency managementIncremental modelsBuilt-in testing

Apache Airflow

Orchestration

Platform to programmatically author, schedule, and monitor workflows. The brain of the pipeline.

"Orchestrates the full nightly pipeline — Bronze extraction → Silver dbt → Gold dbt — with dependency management and alerting."

DAG-based workflowsScheduled executionFailure alertingVisual monitoring UI

Node.js

Extraction

JavaScript runtime for building fast, scalable server-side applications. Powers the Bronze extraction layer.

"Extracts all 10 JDE tables from SQL Server and loads them into PostgreSQL Bronze schema."

Async extractionmssql + pg driversBatch processingProgress tracking

Fastify

API

Fast and low overhead web framework for Node.js. Powers the REST API that connects the Gold layer to the dashboard.

"Serves all dashboard data — customers, sales, AR aging, inventory, purchasing, statusboards, and RFQ submission."

REST APIPostgreSQL connection poolCORS enabledDeployed on Railway

Next.js

Frontend

The React framework for production. Powers the full dashboard with server-side rendering and TypeScript support.

"Renders the entire dashboard — statusboards, dashboards, master data, RFQ portal, architecture page, and this page."

React + TypeScriptApp RouterTailwind CSSDeployed on Railway

Splink

Record Linkage

Open-source Python library for probabilistic record linkage. Uses Fellegi-Sunter models to match records that refer to the same entity across different data sources.

"Matches customer records across 5 ERP systems — resolving 82 duplicates from 333 source records into 251 golden entities using Jaro-Winkler similarity, tax ID, and location."

Jaro-Winkler similarityEM parameter trainingBlocking rulesDuckDB backend

Integrated Services

Managed SaaS used for auth and payments — the only non-open-source pieces in the stack.

Clerk

Authentication

Drop-in authentication and user management for Next.js. Handles sign-in, sign-up, session management, and MFA out of the box.

"Gates the customer portal and shop — scoped sessions tie each user to their JDE customer_id."

Next.js middlewareOAuth-readySession managementDark-themed UI

Stripe

Payments

Hosted checkout and payments infrastructure. PCI-compliant card processing with webhooks for order state.

"Powers the e-commerce shop — Checkout Sessions land back in the app schema for order tracking and webhook replay."

Stripe CheckoutWebhook idempotencyapp.checkout_sessionTest mode sandbox

Built by JP Castro · johnpaulcastro@gmail.com