Portfolio · Data engineering · Full stack
Whisky prices, scraped and served.
DramValue is a price intelligence platform I built from scratch: automated ingestion from auction houses and retailers, normalization into a bottle catalog, and a public site at dramvalue.com for search, charts, and market trends.
What I built
End-to-end ownership: scrapers, database design, API, frontend, deployment, SEO, and public dataset publishing.
Data ingestion
Scrapy spiders on Celery beat, four-stage validation pipeline, deduplication, and PostgreSQL persistence.
Pipeline details →Analysis & intelligence
Bottle matching, cached price stats, market aggregates, trending views, and CSV exports to Hugging Face and Kaggle.
How data becomes insight →Platform & site
FastAPI + Jinja2 SSR, JWT auth, Chart.js price history, sitemaps, and production Docker on a VPS behind nginx.
Architecture & ops →Open datasets
CC0 exports with 1.3M+ prices — mirrored on Hugging Face and Kaggle for researchers and hobbyists.
Download the data →System at a glance
┌─────────────────┐ ┌──────────────┐ ┌─────────────────────────────────┐
│ Auction & retail│ │ Celery worker│ │ PostgreSQL │
│ sites (14 src) │────▶│ Scrapy │────▶│ bottles · prices · market_stats │
└─────────────────┘ │ 4-stage pipe │ └──────────────┬──────────────────┘
└──────────────┘ │
┌──────────────┐ ▼
│ Celery beat │ ┌─────────────────┐
│ (schedules) │ │ FastAPI + Jinja2 │
└──────────────┘ │ dramvalue.com │
└─────────────────┘
Timeline
Dec 2025 — Foundation
Architecture, Docker Compose stack, PostgreSQL schema, first scrapers online.
Dec 2025 — Data pipeline
Celery scheduling, normalization pipeline, Kaggle seed imports for brands and market stats.
Dec 2025 — Core platform
Service layer, search, price charts, trending — dramvalue.com goes live with SSL.
Aug 2026 — Scale & publish
1.35M prices, open dataset mirrors, SEO sitemaps, auth hardening, portfolio sunset to Vercel demo.
Stack
Backend
Python 3.11, FastAPI, SQLAlchemy async, Pydantic, Celery, Redis
Ingestion
Scrapy, custom pipelines, Tor proxy option, Shopify JSON + HTML spiders
Data
PostgreSQL 16, psql COPY exports, Hugging Face Hub, Kaggle datasets
Frontend
Jinja2, Tailwind CSS, Alpine.js, Chart.js — server-rendered pages