diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index ddbf390..86a6c60 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -15,6 +15,8 @@ services: - STACK_DOMAIN_SUFFIX=${STACK_DOMAIN_SUFFIX:-ai.flexinit.nl} - STACK_IMAGE=${STACK_IMAGE:-git.app.flexinit.nl/flexinit/agent-stack:latest} - RESERVED_NAMES=${RESERVED_NAMES:-admin,api,www,root,system,test,demo,portal} + - SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} + - SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} env_file: - .env restart: unless-stopped diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 325c41a..765bf6e 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -15,6 +15,8 @@ services: - STACK_DOMAIN_SUFFIX=${STACK_DOMAIN_SUFFIX:-ai.flexinit.nl} - STACK_IMAGE=${STACK_IMAGE:-git.app.flexinit.nl/flexinit/agent-stack:latest} - RESERVED_NAMES=${RESERVED_NAMES:-admin,api,www,root,system,test,demo,portal} + - SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} + - SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} env_file: - .env restart: unless-stopped diff --git a/docker-compose.staging.yml b/docker-compose.staging.yml index 71552fe..7db9ccd 100644 --- a/docker-compose.staging.yml +++ b/docker-compose.staging.yml @@ -15,6 +15,8 @@ services: - STACK_DOMAIN_SUFFIX=${STACK_DOMAIN_SUFFIX:-ai.flexinit.nl} - STACK_IMAGE=${STACK_IMAGE:-git.app.flexinit.nl/flexinit/agent-stack:latest} - RESERVED_NAMES=${RESERVED_NAMES:-admin,api,www,root,system,test,demo,portal} + - SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} + - SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} env_file: - .env restart: unless-stopped diff --git a/docs/DOKPLOY_DEPLOYMENT.md b/docs/DOKPLOY_DEPLOYMENT.md index 45f9531..4029713 100644 --- a/docs/DOKPLOY_DEPLOYMENT.md +++ b/docs/DOKPLOY_DEPLOYMENT.md @@ -91,6 +91,75 @@ main branch → git.app.flexinit.nl/oussamadouhou/ai-stack-deployer:latest --- +## Shared Project Configuration (IMPORTANT) + +### What is Shared Project Deployment? + +The portal deploys **all user AI stacks as applications within a single shared Dokploy project**, instead of creating a new project for each user. This provides: + +- ✅ Better organization (all stacks in one place) +- ✅ Shared environment variables +- ✅ Centralized monitoring +- ✅ Easier management + +### How It Works + +``` +Dokploy Project: ai-stack-portal +├── Environment: deployments +│ ├── Application: john-dev +│ ├── Application: jane-prod +│ └── Application: alice-test +``` + +### Setting Up the Shared Project + +**Step 1: Create the Shared Project in Dokploy** + +1. In Dokploy UI, create a new project: + - Name: `ai-stack-portal` (or any name you prefer) + - Description: "Shared project for all user AI stacks" + +2. Note the **Project ID** (visible in URL or API response) + - Example: `2y2Glhz5Wy0dBNf6BOR_-` + +3. Get the **Environment ID**: + ```bash + curl -s "http://10.100.0.20:3000/api/project.one?projectId=2y2Glhz5Wy0dBNf6BOR_-" \ + -H "Authorization: Bearer $DOKPLOY_API_TOKEN" | jq -r '.environments[0].id' + ``` + - Example: `RqE9OFMdLwkzN7pif1xN8` + +**Step 2: Configure Project-Level Variables** + +In the shared project (`ai-stack-portal`), add these **project-level environment variables**: + +| Variable Name | Value | Purpose | +|---------------|-------|---------| +| `SHARED_PROJECT_ID` | `2y2Glhz5Wy0dBNf6BOR_-` | The project where user stacks deploy | +| `SHARED_ENVIRONMENT_ID` | `RqE9OFMdLwkzN7pif1xN8` | The environment within that project | + +**Step 3: Reference Variables in Portal Applications** + +The portal's docker-compose files use Dokploy's variable syntax to reference these: + +```yaml +environment: + - SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} + - SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} +``` + +**This syntax `${{project.VARIABLE}}` tells Dokploy**: "Get this value from the project-level environment variables" + +### Important Notes + +- ⚠️ **Both variables MUST be set** in the shared project for deployment to work +- ⚠️ If not set, portal will fall back to creating separate projects per user (legacy behavior) +- ✅ You can have different shared projects for dev/staging/prod environments +- ✅ All 3 portal deployments (dev/staging/prod) should point to their respective shared projects + +--- + ## Setting Up Dokploy ### Step 1: Create Dev Application @@ -107,12 +176,28 @@ main branch → git.app.flexinit.nl/oussamadouhou/ai-stack-deployer:latest - Enable SSL (via Traefik wildcard cert) 3. **Set Environment Variables**: + + **Important**: The portal application should be deployed **inside the shared project** (e.g., `ai-stack-portal-dev`). + + Then set these **project-level variables** in that shared project: + ```env + SHARED_PROJECT_ID= + SHARED_ENVIRONMENT_ID= + ``` + + And these **application-level variables** in the portal app: ```env DOKPLOY_URL=http://10.100.0.20:3000 DOKPLOY_API_TOKEN= STACK_DOMAIN_SUFFIX=ai.flexinit.nl STACK_IMAGE=git.app.flexinit.nl/flexinit/agent-stack:latest ``` + + The docker-compose file will automatically reference the project-level variables using: + ```yaml + SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} + SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} + ``` 4. **Configure Webhook**: - Event: **Push** diff --git a/docs/SHARED_PROJECT_DEPLOYMENT.md b/docs/SHARED_PROJECT_DEPLOYMENT.md new file mode 100644 index 0000000..8a88192 --- /dev/null +++ b/docs/SHARED_PROJECT_DEPLOYMENT.md @@ -0,0 +1,311 @@ +# Shared Project Deployment Architecture + +## Overview + +The AI Stack Deployer portal deploys **all user AI stacks to a single shared Dokploy project** instead of creating a new project for each user. + +--- + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Dokploy: ai-stack-portal (Shared Project) │ +│ ID: 2y2Glhz5Wy0dBNf6BOR_- │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ 📦 Portal Application: ai-stack-deployer-prod │ +│ ├─ Domain: portal.ai.flexinit.nl │ +│ ├─ Image: git.app.flexinit.nl/.../ai-stack-deployer:latest│ +│ └─ Env: SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} │ +│ │ +│ ───────────────────────────────────────────────────────────── │ +│ │ +│ 📦 User Stack: john-dev │ +│ ├─ Domain: john-dev.ai.flexinit.nl │ +│ ├─ Image: git.app.flexinit.nl/.../agent-stack:latest │ +│ └─ Deployed by: Portal │ +│ │ +│ 📦 User Stack: jane-prod │ +│ ├─ Domain: jane-prod.ai.flexinit.nl │ +│ ├─ Image: git.app.flexinit.nl/.../agent-stack:latest │ +│ └─ Deployed by: Portal │ +│ │ +│ 📦 User Stack: alice-test │ +│ ├─ Domain: alice-test.ai.flexinit.nl │ +│ ├─ Image: git.app.flexinit.nl/.../agent-stack:latest │ +│ └─ Deployed by: Portal │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## How It Works + +### Step 1: Portal Reads Configuration + +When a user submits a stack name (e.g., "john-dev"), the portal: + +1. **Reads environment variables**: + ```javascript + const sharedProjectId = process.env.SHARED_PROJECT_ID; + const sharedEnvironmentId = process.env.SHARED_ENVIRONMENT_ID; + ``` + +2. **These are set via Dokploy's project-level variables**: + ```yaml + environment: + - SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} + - SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} + ``` + +### Step 2: Portal Deploys to Shared Project + +Instead of creating a new project, the portal: + +```javascript +// OLD BEHAVIOR (legacy): +// createProject(`ai-stack-${username}`) ❌ Creates new project per user + +// NEW BEHAVIOR (current): +// Uses existing shared project ID ✅ +const projectId = sharedProjectId; // From environment variable +const environmentId = sharedEnvironmentId; + +// Creates application IN the shared project +createApplication({ + projectId: projectId, + environmentId: environmentId, + name: `${username}-stack`, + image: 'git.app.flexinit.nl/.../agent-stack:latest', + domain: `${username}.ai.flexinit.nl` +}); +``` + +### Step 3: User Accesses Their Stack + +User visits `https://john-dev.ai.flexinit.nl` → Traefik routes to their application inside the shared project. + +--- + +## Configuration Steps + +### 1. Create Shared Project in Dokploy + +1. In Dokploy UI, create project: + - **Name**: `ai-stack-portal` + - **Description**: "Shared project for all user AI stacks" + +2. Get the **Project ID**: + ```bash + # Via API + curl -s "http://10.100.0.20:3000/api/project.all" \ + -H "Authorization: Bearer $DOKPLOY_API_TOKEN" | \ + jq -r '.[] | select(.name=="ai-stack-portal") | .id' + + # Output: 2y2Glhz5Wy0dBNf6BOR_- + ``` + +3. Get the **Environment ID**: + ```bash + curl -s "http://10.100.0.20:3000/api/project.one?projectId=2y2Glhz5Wy0dBNf6BOR_-" \ + -H "Authorization: Bearer $DOKPLOY_API_TOKEN" | \ + jq -r '.environments[0].id' + + # Output: RqE9OFMdLwkzN7pif1xN8 + ``` + +### 2. Set Project-Level Variables + +In the shared project (`ai-stack-portal`), add these **project-level environment variables**: + +| Variable | Value | Example | +|----------|-------|---------| +| `SHARED_PROJECT_ID` | Your project ID | `2y2Glhz5Wy0dBNf6BOR_-` | +| `SHARED_ENVIRONMENT_ID` | Your environment ID | `RqE9OFMdLwkzN7pif1xN8` | + +**How to set in Dokploy UI**: +- Go to Project → Settings → Environment Variables +- Add variables at **project level** (not application level) + +### 3. Deploy Portal Application + +Deploy the portal **inside the same shared project**: + +1. **Application Details**: + - Name: `ai-stack-deployer-prod` + - Type: Docker Compose + - Compose File: `docker-compose.prod.yml` + - Branch: `main` + +2. **The docker-compose file automatically references project variables**: + ```yaml + environment: + - SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} # ← Magic happens here + - SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} + ``` + +3. **Dokploy resolves `${{project.VAR}}`** to the actual value from project-level variables. + +--- + +## Benefits + +### ✅ Centralized Management +All user stacks in one place: +- Easy to list all active stacks +- Shared monitoring dashboard +- Centralized logging + +### ✅ Resource Efficiency +- No overhead of separate projects per user +- Shared network and resources +- Easier to manage quotas + +### ✅ Simplified Configuration +- Project-level environment variables shared by all stacks +- Single source of truth for common configs +- Easy to update STACK_IMAGE for all users + +### ✅ Better Organization +``` +Projects in Dokploy: +├── ai-stack-portal (500 user applications) ✅ Clean +└── NOT: + ├── ai-stack-john + ├── ai-stack-jane + ├── ai-stack-alice + └── ... (500 separate projects) ❌ Messy +``` + +--- + +## Fallback Behavior + +If `SHARED_PROJECT_ID` and `SHARED_ENVIRONMENT_ID` are **not set**, the portal falls back to **legacy behavior**: + +```javascript +// Code in src/orchestrator/production-deployer.ts (lines 187-196) +const sharedProjectId = config.sharedProjectId || process.env.SHARED_PROJECT_ID; +const sharedEnvironmentId = config.sharedEnvironmentId || process.env.SHARED_ENVIRONMENT_ID; + +if (sharedProjectId && sharedEnvironmentId) { + // Use shared project ✅ + state.resources.projectId = sharedProjectId; + state.resources.environmentId = sharedEnvironmentId; + return; +} + +// Fallback: Create separate project per user ⚠️ +const projectName = `ai-stack-${config.stackName}`; +const existingProject = await this.client.findProjectByName(projectName); +// ... +``` + +**This ensures backwards compatibility** but is not recommended. + +--- + +## Troubleshooting + +### Portal Creates Separate Projects Instead of Using Shared Project + +**Cause**: `SHARED_PROJECT_ID` or `SHARED_ENVIRONMENT_ID` not set. + +**Solution**: +1. Check project-level variables in Dokploy: + ```bash + curl -s "http://10.100.0.20:3000/api/project.one?projectId=YOUR_PROJECT_ID" \ + -H "Authorization: Bearer $DOKPLOY_API_TOKEN" | \ + jq '.environmentVariables' + ``` + +2. Ensure the portal application's docker-compose references them: + ```yaml + environment: + - SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} + - SHARED_ENVIRONMENT_ID=${{project.SHARED_ENVIRONMENT_ID}} + ``` + +3. Redeploy the portal application. + +### Variable Reference Not Working + +**Symptom**: Portal logs show `undefined` for `SHARED_PROJECT_ID`. + +**Cause**: Using wrong syntax. + +**Correct syntax**: +```yaml +- SHARED_PROJECT_ID=${{project.SHARED_PROJECT_ID}} ✅ +``` + +**Wrong syntax**: +```yaml +- SHARED_PROJECT_ID=${SHARED_PROJECT_ID} ❌ (shell substitution, not Dokploy) +- SHARED_PROJECT_ID={{project.SHARED_PROJECT_ID}} ❌ (missing $) +``` + +### How to Verify Configuration + +Check portal container environment: +```bash +# SSH into Dokploy host +ssh user@10.100.0.20 + +# Inspect portal container +docker exec ai-stack-deployer env | grep SHARED + +# Should show: +SHARED_PROJECT_ID=2y2Glhz5Wy0dBNf6BOR_- +SHARED_ENVIRONMENT_ID=RqE9OFMdLwkzN7pif1xN8 +``` + +--- + +## Environment-Specific Shared Projects + +You can have **separate shared projects for dev/staging/prod**: + +| Portal Environment | Shared Project | Purpose | +|--------------------|----------------|---------| +| Dev | `ai-stack-portal-dev` | Development user stacks | +| Staging | `ai-stack-portal-staging` | Staging user stacks | +| Prod | `ai-stack-portal` | Production user stacks | + +Each portal deployment references its own shared project: +- `portal-dev.ai.flexinit.nl` → `ai-stack-portal-dev` +- `portal-staging.ai.flexinit.nl` → `ai-stack-portal-staging` +- `portal.ai.flexinit.nl` → `ai-stack-portal` + +--- + +## Migration from Legacy + +If you're currently using the legacy behavior (separate projects per user): + +### Option 1: Gradual Migration +- New deployments use shared project +- Old deployments remain in separate projects +- Migrate old stacks manually over time + +### Option 2: Full Migration +1. Create shared project +2. Set project-level variables +3. Redeploy all user stacks to shared project +4. Delete old separate projects + +**Note**: Migration requires downtime for each stack being moved. + +--- + +## Reference + +- **Environment Variable Syntax**: See Dokploy docs on project-level variables +- **Code Location**: `src/orchestrator/production-deployer.ts` (lines 178-200) +- **Example IDs**: `.env.example` (lines 25-27) + +--- + +**Questions?** Check the main deployment guide: `DOKPLOY_DEPLOYMENT.md`