H
CV

Ahmedabad --:-- IST

01Extract

HarshPatel

The web is noise. I extract the signal.

Web Scraping Expert · Data Extraction · Automation

Dynamic, CAPTCHA-protected, anti-bot defended websites at production speed. I handle the projects others turn down.

years shipping
07+
engineers led
30+
tools mastered
50+
LLM APIs
10+
sites, one pipeline
1,500
Harsh S Patel, smiling, in a light jacket
fig. 01touch & hold to peek

// the human behind the pipelines

Hi, I'm Harsh. I build systems that read the web — and lead the teams that build them.

A Web Scraping & Data Extraction specialist with 7+ years of experience building production-scale scraping systems. I have successfully scraped highly dynamic, CAPTCHA-protected, and anti-bot defended websites that most developers can't handle — at impressive speed and reliability.

One of my most impactful projects was a large-scale ETL pipeline for a client building a domain-specific LLM — collecting structured data from nearly 1,500 websites, each with unique obstacles: CAPTCHAs, login walls, JavaScript rendering, and anti-scraping measures. My team of 5–6 designed a standardized extraction framework with configurable adapters per site, a centralized ETL layer normalizing all data into one unified schema, and a queue-based workflow with automatic retries. We completed all 1,500 sites within 2–3 months — running 3x faster than the client's own in-house team working in parallel. The output was clean and ready for LLM training with minimal rework.

I have also built ongoing price monitoring systems for e-commerce clients handling millions of product-match records with real-time alerting pipelines. My pipeline stack covers Scrapy, Playwright, Selenium, Puppeteer, httpx, aiohttp — combined with CAPTCHA bypass, proxy rotation, and fingerprint spoofing. Speed, consistency, and scalability of data extraction is my genuine competitive differentiator — I handle projects others turn down.

Based in
Ahmedabad, Gujarat
Local time
--:-- IST
Shipping since
2018
Coordinates
23.02°N 72.57°E
02Transform

Five transformations. From junior developer to leading 30+.

Starting as a junior PHP developer in 2018, I grew through senior engineering and module leadership — and now lead multiple technical teams.

stage_00 · source()

Education

  • 2014 — 2018B.E. Computer EngineeringLDRP-ITR, Gandhinagar · CGPA 8.5 / 10
  • 2014HSC ScienceGHSEB · 80%
  • 2012SSCGSEB · 89%

stage_01 · ingest()

Junior Developer

First role

Dynamic Web Solutions

Aug 2018 — Aug 2019 · 1 year

Built dynamic websites for multiple clients using PHP MVC architecture and Zend Framework. Delivered full-stack solutions with custom business logic and database design.

  • PHP
  • Zend
  • MVC
  • jQuery
  • MySQL
  • HTML/CSS

stage_02 · deepen()

Senior Developer

Senior dev

Large-scale e-commerce

Aug 2019 — Aug 2022 · 3 years

Core engineer on a major e-commerce platform. Deep expertise in Magento internals, performance optimisation, Solr/Elasticsearch search tuning, and containerised deployments.

  • Magento
  • PHP
  • MySQL
  • Solr
  • Docker
  • Git
  • Postman

stage_03 · orchestrate()

Group / Module Leader

Module lead

E-commerce platform

Aug 2022 — Aug 2024 · 2 years · Team of 12

Led a team of 10–12 developers on a high-traffic e-commerce platform. Owned critical modules: pricing strategy, shipping & tax logic, Google Ads, and competitor monitoring. Architected complex business logic driving direct revenue impact. Mentored mid-level developers and set engineering standards.

  • Magento
  • PHP
  • MySQL
  • GraphQL
  • Elasticsearch
  • Kafka
  • Docker
  • CI/CD
  • JIRA

stage_04 · infer()

Technical Team Lead & Tech Expert

Current

Company-wide AI/ML & automation

Aug 2024 — Present · Now · 20+ members

Leading company-wide AI/ML adoption and automation strategy. Built multiple end-to-end automation systems — real-time monitoring, LLM-driven processing, smart scheduling — all without paid tools. Integrated 10+ LLM APIs including local models via Ollama. Built enterprise-scale scraping pipelines handling CAPTCHAs and anti-bot systems. Managing 20+ members, providing internship training.

  • LangChain
  • RAG
  • Ollama
  • MCP
  • Scrapy
  • Playwright
  • n8n
  • Airflow
  • FastAPI
  • Vector DB
  • AWS
  • MLflow
  • HuggingFace
03Load

case_01 · 1500_sites.etl

Case file: 1,500 websites, one schema

0000 / 1500 sites extracted

  • ? CAPTCHAs
  • # Login walls
  • { JS rendering
  • % Anti-scraping

↓ 1 unified schema

01 · The brief

1,500 websites. Each one fighting back.

A client building a domain-specific LLM needed structured data from nearly 1,500 websites — every one with its own obstacles: CAPTCHAs, login walls, JavaScript rendering and anti-scraping measures.

02 · The framework

One framework. A configurable adapter per site.

My team of 5–6 designed a standardized extraction framework with configurable adapters per site — so every new site was configuration, not a rewrite.

03 · The run

Queue. Retry. Extract. Then normalize everything.

A queue-based workflow with automatic retries kept the crawl moving, while a centralized ETL layer normalized every source into one unified schema.

04 · The result

3× faster. All 1,500 sites in 2–3 months.

We finished running 3× faster than the client's own in-house team working in parallel. The output was clean and ready for LLM training with minimal rework.

More case files

  • case_02

    Price monitoring at e-commerce scale

    Ongoing price monitoring systems for e-commerce clients — millions of product-match records with real-time alerting pipelines.

    • Millions of records
    • Product matching
    • Real-time alerts
  • case_03

    AI automation systems, zero paid tools

    End-to-end automation systems — real-time monitoring, LLM-driven processing and smart scheduling — built without a single paid tool.

    • Real-time monitoring
    • LLM processing
    • Smart scheduling
  • case_04

    10+ LLM APIs, plus local models

    Integrated Claude, ChatGPT, Gemini, Grok, DeepSeek, Perplexity, Mistral, LLaMA and Qwen — and deployed local models via Ollama on free hosting alternatives to save costs.

    • 10+ APIs
    • Local models
    • Ollama
  • case_05

    Revenue-critical e-commerce modules

    On a high-traffic Magento platform: pricing strategy, shipping & tax logic, payment gateway integration, Google Ads management, competitor monitoring and Solr/Elasticsearch search tuning.

    • Magento
    • Pricing · shipping · tax
    • Search tuning
04Embed

78 tools, embedded. Related tools cluster together.

All skills by category

Languages

  • Python
  • PHP
  • JavaScript
  • HTML/CSS
  • jQuery
  • GraphQL

Frameworks

  • FastAPI
  • Django
  • Flask
  • Magento
  • Zend
  • Odoo

Databases & search

  • MySQL
  • PostgreSQL
  • BigQuery
  • ClickHouse
  • Elasticsearch
  • Solr

Orchestration & agents

  • LangChain
  • LangGraph
  • LangSmith
  • n8n
  • Airflow
  • Celery
  • AI Agents
  • MCP

ML libraries

  • PyTorch
  • TensorFlow
  • Keras
  • Scikit-learn
  • Transformers
  • HuggingFace
  • OpenCV
  • Pandas
  • NumPy
  • MLflow
  • Streamlit
  • Gradio

LLMs & prompting

  • Claude
  • ChatGPT
  • Gemini
  • Grok
  • DeepSeek
  • Perplexity
  • Ollama
  • Mistral
  • LLaMA
  • Qwen
  • Prompt Engineering

RAG & vector DBs

  • RAG
  • Embeddings
  • Chroma
  • Pinecone
  • Weaviate
  • Qdrant
  • FAISS
  • Milvus
  • Amazon S3 Vectors

Scraping & automation

  • Scrapy
  • Playwright
  • Selenium
  • Puppeteer
  • BeautifulSoup
  • httpx
  • aiohttp
  • Anti-CAPTCHA
  • Proxy Rotation
  • Redis

DevOps & tools

  • Docker
  • AWS
  • Git
  • GitHub Actions
  • Jenkins
  • CI/CD
  • Kafka
  • JIRA
  • Notion
  • Postman

Scroll to tour the clusters, or point at any label. ◆ marks what this lens cares about.

05Retrieve

Ask the index. It answers in milliseconds, for free.

A tiny retrieval system over everything on this page — the same idea behind the RAG pipelines I build, minus the API bill.

The answer, its sources and their scores show up here. Matching skills light up in the field behind.

Runs in your browser · BM25 over 41 hand-written notes · no LLM · $0 per query — or ask the human: harshpatel344930@gmail.com

06Optimize

“Every paid tool has a free alternative

— and I always find it.”

— core philosophy
Departures · paid stackMonthly bill: $$$$
  • LLM inferencePAID API CALLS replaced by LOCAL MODELS / OLLAMA
  • Vector searchMANAGED VECTOR DB replaced by CHROMA / FAISS / QDRANT
  • WorkflowsAUTOMATION SAAS replaced by N8N / AIRFLOW
  • Data extractionSCRAPING APIS replaced by SCRAPY / PLAYWRIGHT
  • Job queuesMANAGED QUEUES replaced by CELERY / REDIS
  • Model hostingPAID GPU HOSTING replaced by FREE ALTERNATIVES

Systems architected to save thousands every month across data, LLM and SaaS — without giving up production quality.

07Orchestrate

Lead by example. Hands-on technically, strong on delivery.

2018 — 2022

Individual contributor. Four years hands-on.

PHP MVC and Zend for clients, then Magento internals, performance and search tuning on a large-scale platform — before leading anyone.

2022

A team of 10–12. Revenue-critical modules.

As module leader on a high-traffic e-commerce platform: pricing strategy, shipping & tax logic, Google Ads and competitor monitoring. Mentored mid-level developers, set engineering standards.

2024

A 20+ member team. Automation & scraping.

Defining processes, setting quality standards and mentoring engineers at every level — while leading company-wide AI/ML adoption.

Always

Interns, trained. Freshers, made industry-ready.

Internship training in PHP, Python, AI/ML and web scraping — bridging academic knowledge and industry-ready skills.

engineers led in total
30+
Magento e-commerce team
10–12
automation & scraping team
20+
trained in PHP · Python · AI/ML
Interns

Key strengths

  • { }

    Web scraping at scale

    Dynamic sites, CAPTCHAs, anti-bot systems at production speed. Built & trained a 20+ member scraping team.

  • ~>

    LLM & RAG engineering

    Integrated 10+ LLM APIs and local models via Ollama. Expert in RAG, vector search, embeddings and agents.

  • </>

    Full stack engineering

    PHP, Python, Magento, FastAPI, Django. Complex business logic, e-commerce systems, and scalable APIs.

  • o-o

    Team leadership

    Led 30+ member teams across dev, automation and scraping. Mentored freshers, set standards, drove delivery.

  • <->

    AI automation systems

    End-to-end automation — real-time monitoring, LLM-driven processing, smart scheduling — zero paid tools.

  • ₹0

    Free over paid

    Every paid tool has a free alternative. Architect systems saving thousands monthly across data, LLM, and SaaS.

08Respond

Send a signal. I reply within 24 hours.

A freelance project, a full-time opportunity or a quick consultation — scraping, AI/ML, automation, backend or team training.

POST /harsh · content-type: human

harshpatel344930@gmail.com
Write to me →

// a good first message has

  1. 01What you're building, extracting or automating
  2. 02A rough timeline
  3. 03The best way to reach you

Direct channels