|
| 1 | +# Self-Hosting LFX Community Data Platform (CDP) |
| 2 | + |
| 3 | +This guide provides technical instructions for self-hosting the Community edition of the LFX Community Data Platform (formerly crowd.dev) using Docker Compose. |
| 4 | + |
| 5 | +## Prerequisites |
| 6 | + |
| 7 | +Before you begin, ensure your system meets the following requirements: |
| 8 | + |
| 9 | +### Hardware Requirements |
| 10 | +- **CPU**: 4+ cores recommended. |
| 11 | +- **RAM**: 16GB+ recommended (the platform runs several services including OpenSearch, Kafka, and Postgres). |
| 12 | +- **Storage**: 50GB+ of free space (depends on the volume of data you plan to ingest). |
| 13 | + |
| 14 | +### Software Requirements |
| 15 | +- **Docker**: Engine 20.10+ and Docker Compose v2.0+. |
| 16 | +- **Node.js**: v16.16.0 (required for some local CLI utilities). |
| 17 | +- **Python**: 3.x (required for Tinybird CLI setup). |
| 18 | + |
| 19 | +## Getting Started |
| 20 | + |
| 21 | +1. **Clone the Repository**: |
| 22 | + ```bash |
| 23 | + git clone https://github.com/linuxfoundation/crowd.dev.git |
| 24 | + cd crowd.dev |
| 25 | + ``` |
| 26 | + |
| 27 | +2. **Install Dependencies**: |
| 28 | + ```bash |
| 29 | + pnpm install |
| 30 | + ``` |
| 31 | + |
| 32 | +3. **Configure Environment Variables**: |
| 33 | + Copy the distribution template to create your local override file: |
| 34 | + ```bash |
| 35 | + cp backend/.env.dist.local backend/.env.override.local |
| 36 | + ``` |
| 37 | + *Note: For a "composed" environment where services run in Docker, you may also want to reference `backend/.env.dist.composed` for the correct hostnames (e.g., `db` instead of `localhost`).* |
| 38 | + |
| 39 | +## Deployment Steps |
| 40 | + |
| 41 | +The platform uses a custom CLI tool located in `scripts/cli` to simplify management. |
| 42 | + |
| 43 | +### 1. Start Infrastructure (Scaffold) |
| 44 | +The "scaffold" includes essential services: PostgreSQL, Redis, OpenSearch, Kafka, and Tinybird. |
| 45 | + |
| 46 | +```bash |
| 47 | +cd scripts |
| 48 | +./cli scaffold up |
| 49 | +``` |
| 50 | + |
| 51 | +This command will: |
| 52 | +- Set up the `crowd-bridge` Docker network. |
| 53 | +- Start all infrastructure containers defined in `scripts/scaffold.yaml`. |
| 54 | +- Set up the Tinybird CLI and virtual environment. |
| 55 | + |
| 56 | +### 2. Run Database Migrations |
| 57 | +Once the infrastructure is up, apply the necessary schema migrations for Postgres and Tinybird: |
| 58 | + |
| 59 | +```bash |
| 60 | +./cli migrate-up |
| 61 | +``` |
| 62 | + |
| 63 | +### 3. Start Application Services |
| 64 | +You can start all application services (API, Frontend, Workers) at once: |
| 65 | + |
| 66 | +```bash |
| 67 | +./cli service up-all |
| 68 | +``` |
| 69 | + |
| 70 | +The application will be available at: |
| 71 | +- **Frontend**: http://localhost:8081 |
| 72 | +- **API**: http://localhost:4000/api (default) |
| 73 | + |
| 74 | +## Configuration Details |
| 75 | + |
| 76 | +Key environment variables to review in `backend/.env.override.local`: |
| 77 | + |
| 78 | +- `CROWD_API_URL`: The public-facing URL of your API. |
| 79 | +- `CROWD_API_JWT_SECRET`: A long, random string for securing sessions. |
| 80 | +- `CROWD_DB_*`: Database connection details (defaults work with `scaffold`). |
| 81 | +- `CROWD_REDIS_*`: Redis connection details. |
| 82 | +- `CROWD_TINYBIRD_BASE_URL`: URL for the local Tinybird instance (default: `http://localhost:7181/`). |
| 83 | + |
| 84 | +## Operations & Maintenance |
| 85 | + |
| 86 | +### Checking Logs |
| 87 | +To view logs for a specific service: |
| 88 | +```bash |
| 89 | +./cli service <service-name> logs |
| 90 | +``` |
| 91 | +*(Available service names can be found in `scripts/services/`)* |
| 92 | + |
| 93 | +### Backing up the Database |
| 94 | +```bash |
| 95 | +./cli db-backup <backup-name> |
| 96 | +``` |
| 97 | + |
| 98 | +### Restoring a Backup |
| 99 | +```bash |
| 100 | +./cli db-restore <backup-name> |
| 101 | +``` |
| 102 | + |
| 103 | +## Security & Production Notes |
| 104 | + |
| 105 | +- **Reverse Proxy**: It is highly recommended to use a reverse proxy (like Nginx) to handle SSL/TLS termination. A template is provided in `scripts/scaffold/nginx/`. |
| 106 | +- **Secrets**: Ensure `CROWD_API_JWT_SECRET` and all database passwords are changed from their default values in a production environment. |
| 107 | +- **Persistence**: All data is stored in Docker volumes (e.g., `pgdata-dev`, `opensearch-dev`). Ensure these volumes are backed up regularly. |
| 108 | + |
| 109 | +## Troubleshooting |
| 110 | + |
| 111 | +- **Memory Issues**: If services fail to start, check if Docker has enough memory allocated (especially on macOS/Windows). |
| 112 | +- **Network Conflicts**: Ensure the subnet `10.90.0.0/24` (default) does not conflict with your existing network. You can override this in `scripts/cli` or via environment variables. |
0 commit comments