GitHub Trending Pipeline

An automated data pipeline that fetches the top 30 trending GitHub repositories daily, stores the raw data in PostgreSQL, and aggregates it by programming language. Orchestrated with Apache Airflow and containerised with Docker Compose.

Tech Stack

• Python — extract, transform, load logic
• GitHub REST API — data source
• pandas — data transformation
• PostgreSQL — data storage
• Apache Airflow — pipeline orchestration and scheduling
• Docker Compose — container management

Architecture

GitHub API → extract.py → transform.py → load.py → PostgreSQL
                                                         ↑
                                              Scheduled daily by Airflow

Project Structure

github-pipeline/
├── dags/
│   └── github_dag.py        # Airflow DAG — daily schedule
├── src/
│   ├── extract.py           # Calls GitHub Search API
│   ├── transform.py         # Cleans and shapes raw data
│   ├── load.py              # Inserts into PostgreSQL
│   └── pipeline.py          # Orchestrates extract → transform → load
├── sql/
│   └── create_tables.sql    # Table definitions
├── logs/                    # Airflow logs (auto-generated)
├── docker-compose.yml
├── requirements.txt
└── .env                     # Not committed — see setup below

Prerequisites

• Docker Desktop running
• PostgreSQL container pg-learn connected to de-network
• github_db and airflow_db databases created inside pg-learn
• A GitHub Personal Access Token (public_repo scope)

Setup

1. Clone the repo

git clone <repo-url>
cd github-pipeline

2. Create .env

GITHUB_TOKEN=your_github_token_here
DB_HOST=pg-learn
DB_PORT=5432
DB_NAME=github_db
DB_USER=admin
DB_PASSWORD=admin123

3. Create the tables

docker exec -i pg-learn psql -U admin -d github_db < sql/create_tables.sql

4. Start Airflow

docker-compose up -d

Wait about 30 seconds for the containers to initialise, then open the Airflow UI at http://localhost:8080. Login with admin / admin123.

5. Enable the DAG

In the Airflow UI, find github_trending_pipeline and toggle it on. It will run daily at midnight, or you can trigger it manually with the play button.

Concepts Learned

Apache Airflow

A workflow orchestration tool. You define a DAG (Directed Acyclic Graph) — a set of tasks with a defined order and schedule. Airflow's scheduler watches the clock and triggers the DAG automatically. Think of it like SQL Server Agent, but far more powerful and built for data pipelines.

• DAG — the overall workflow definition (name, schedule, start date)
• Task — an individual step inside the DAG
• PythonOperator — a task type that runs a Python function
• schedule_interval — when the DAG runs (@daily, @hourly, or a cron expression)
• catchup=False — tells Airflow not to backfill missed runs if it was offline; without this it would run once for every missed day since start_date
• depends_on — controls startup order between Docker Compose services; same idea as a job step that waits for the previous step to succeed

Docker Compose

A tool for defining and running multiple containers together using a single YAML file.

• x- extension fields — a YAML feature for shared config blocks; all services inherit from it using <<: *block-name, avoiding repetition
• env_file — tells Docker to read a .env file and inject its contents as environment variables into the container at startup
• _PIP_ADDITIONAL_REQUIREMENTS — an Airflow-specific env variable; when set, Airflow automatically pip-installs the listed packages when the container starts

GitHub REST API

The Search API endpoint (/search/repositories) accepts a query string to filter results — similar to a WHERE clause in SQL. Authentication is done via a Bearer token in the request header to avoid rate limiting.

Python Concepts

• timedelta — Python's way of doing date arithmetic; datetime.now() - timedelta(days=30) is equivalent to DATEADD(day, -30, GETDATE()) in SQL
• dict.get() — safely reads a key that may be absent or null, returning None instead of raising an error; used for nullable fields like language and description
• execute_batch — sends all rows to PostgreSQL in a single round trip instead of one execute() call per row; equivalent to a bulk insert
• pandas groupby / agg — equivalent to GROUP BY with aggregate functions in SQL; used to compute language_summary from the raw repositories data
• try / finally — guarantees cleanup code (like closing a DB connection) runs even if an exception is raised mid-execution
• sys.path.insert — tells Python where to look for importable modules; needed inside Docker because the container doesn't automatically know where the src/ folder is

Airflow Schedule and Docker Dependency

Airflow's scheduler runs inside a Docker container. This means:

• If Docker is stopped, the scheduler is stopped and no DAG runs will trigger
• catchup=False protects against a flood of backfill runs when the containers are restarted after being offline
• In production, Airflow runs on a cloud server that is always on (AWS MWAA, GCP Cloud Composer), so the schedule is always guaranteed

Troubleshooting: Airflow Login Failed on First Setup

Issue: The airflow-init container successfully initialised the Airflow database but failed to create the admin user. Each --flag in the airflow users create command was on its own line in the YAML command block, and Docker interpreted each flag as a separate shell command instead of arguments to a single command.

Error seen:

airflow users create command error: the following arguments are required: -e/--email ...
/bin/bash: line 3: --username: command not found

Fix: Put the entire airflow users create command on a single line:

command: bash -c "airflow db init && airflow users create --username admin --password admin123 --firstname Shivam --lastname Sharma --role Admin --email admin@example.com"

Lesson: Multi-line command blocks in Docker Compose YAML can behave unexpectedly. For complex bash commands, keeping everything on one line (or extracting to a shell script) is the safest approach.

Tables

repositories — one row per repo per day

Column	Type	Description
repo_id	BIGINT	GitHub's unique repo ID
full_name	VARCHAR	e.g. owner/repo-name
language	VARCHAR	Primary language (nullable)
stars	INTEGER	Star count at fetch time
fetched_at	TIMESTAMP	When the pipeline ran

language_summary — one row per language per day

Column	Type	Description
language	VARCHAR	Programming language
repo_count	INTEGER	Number of trending repos
total_stars	BIGINT	Combined star count
avg_stars	NUMERIC	Average stars per repo
summary_date	DATE	Date of the summary