clustered-fucks/docs/control-server-guide.md

# Control Server Operations Guide

**Host:** control (CT 127)  
**IP:** 192.168.1.127  
**Location:** pve2  
**User:** maddox  
**Last Updated:** January 24, 2026

---

## Quick Start

```bash
# Launch interactive menu
~/scripts/control-menu.sh

# Or jump straight to multi-host SSH
~/scripts/ssh-manager.sh

# Or run Ansible directly
cd ~/clustered-fucks
ansible all -m ping
```

---

## Directory Structure

```
/home/maddox/
├── .ssh/
│   ├── config              # SSH host definitions
│   ├── id_ed25519          # SSH private key
│   └── id_ed25519.pub      # Public key (add to new hosts)
│
├── clustered-fucks/        # Git repo (synced to Forgejo)
│   ├── ansible.cfg         # Ansible configuration
│   ├── inventory/
│   │   ├── hosts.yml       # Host inventory
│   │   └── group_vars/
│   │       └── all.yml     # Global variables
│   ├── playbooks/
│   │   ├── check-status.yml
│   │   ├── docker-prune.yml
│   │   ├── restart-utils.yml
│   │   ├── update-all.yml
│   │   ├── deploy-utils.yml
│   │   └── deploy-mealie.yml   # NEW
│   └── compose-files/      # Docker compose files for services
│       └── databases/
│           └── mealie/
│               ├── docker-compose.yml
│               └── .env    # NOT in git (secrets)
│
└── scripts/
    ├── ssh-manager.sh      # tmux multi-host launcher
    ├── control-menu.sh     # Interactive Ansible menu
    └── add-host.sh         # New host onboarding
```

---

## Managed Hosts

| Host | IP | User | Type | Ansible Group |
|------|-----|------|------|---------------|
| pve2 | .3 | root | Proxmox | proxmox_nodes |
| pve-dell | .4 | root | Proxmox | proxmox_nodes |
| replicant | .80 | maddox | VM | docker_hosts |
| databases | .81 | root | VM | docker_hosts |
| immich | .82 | root | VM | docker_hosts |
| media-transcode | .120 | root | LXC | docker_hosts |
| network-services | .121 | root | LXC | docker_hosts |
| download-stack | .122 | root | LXC | docker_hosts |
| docker666 | .123 | root | LXC | docker_hosts |
| tailscale-home | .124 | root | LXC | docker_hosts |
| dns-lxc | .125 | root | LXC | infrastructure |
| nas | .251 | maddox | NAS | legacy |
| alien | .252 | maddox | Docker | legacy |

---

## Ansible Playbooks

| Playbook | Target | Description |
|----------|--------|-------------|
| `check-status.yml` | all_managed | Disk, memory, container counts |
| `update-all.yml` | docker_hosts | apt upgrade all hosts |
| `docker-prune.yml` | docker_hosts | Clean unused Docker resources |
| `restart-utils.yml` | docker_hosts | Restart utils stack |
| `deploy-utils.yml` | docker_hosts | Deploy utils to new host |
| `deploy-mealie.yml` | databases | Deploy Mealie stack |

### Running Playbooks

```bash
cd ~/clustered-fucks

# Check all hosts
ansible-playbook playbooks/check-status.yml

# Update specific host
ansible-playbook playbooks/update-all.yml --limit databases

# Deploy service
ansible-playbook playbooks/deploy-mealie.yml
```

---

## Git Workflow

### Repository Info
- **Local:** `~/clustered-fucks/`
- **Remote:** `ssh://git@192.168.1.81:2222/maddox/clustered-fucks.git`
- **Web:** https://git.3ddbrewery.com/maddox/clustered-fucks

### Daily Workflow

```bash
cd ~/clustered-fucks

# 1. Before making changes, get latest
git pull

# 2. Make your changes (edit files, create playbooks, etc.)
vim playbooks/new-playbook.yml

# 3. See what changed
git status

# 4. Stage all changes
git add -A

# 5. Commit with a message
git commit -m "Add deployment playbook for service X"

# 6. Push to Forgejo
git push
```

### After Creating/Editing Files

**Always commit your changes!** Otherwise they only exist on the control server and could be lost.

```bash
cd ~/clustered-fucks
git add -A
git commit -m "Description of what you changed"
git push
```

### Quick Reference

| Command | What it does |
|---------|--------------|
| `git status` | Show what files have changed |
| `git pull` | Get latest changes from Forgejo |
| `git add -A` | Stage all changes for commit |
| `git commit -m "msg"` | Save changes with a message |
| `git push` | Upload commits to Forgejo |
| `git log --oneline -5` | Show last 5 commits |
| `git diff` | Show what changed (before staging) |

### What Goes in Git vs What Doesn't

**✅ Commit these:**
- Ansible playbooks (`playbooks/*.yml`)
- Inventory files (`inventory/hosts.yml`)
- Docker compose files (`compose-files/**/docker-compose.yml`)
- Documentation and scripts

**❌ Never commit these:**
- `.env` files (contain passwords/secrets)
- Private keys
- Temporary files

The `.gitignore` file already excludes `.env` files.

---

## Service Migration Workflow

### Standard Process

```bash
cd ~/clustered-fucks

# 1. Create compose file directory
mkdir -p compose-files/<target_host>/<service>

# 2. Create docker-compose.yml
vim compose-files/<target_host>/<service>/docker-compose.yml

# 3. Create .env for secrets (not committed to git)
vim compose-files/<target_host>/<service>/.env

# 4. Create Ansible playbook
vim playbooks/deploy-<service>.yml

# 5. Deploy via Ansible
ansible-playbook playbooks/deploy-<service>.yml

# 6. Rsync data from alien (if needed)
ssh alien "docker stop <container>"
rsync -avP maddox@alien:/path/to/data/ root@<target>:/home/docker/appdata/<service>/

# 7. Start the service
ansible <target> -m shell -a "cd /home/docker/appdata/<service> && docker compose up -d"

# 8. Update Traefik route (on Hetzner)

# 9. Test via domain

# 10. Stop old container on alien
ssh alien "docker stop <container>"

# 11. Commit playbook to git
git add -A
git commit -m "Add <service> deployment"
git push
```

### Playbook Template

```yaml
---
- name: Deploy <Service> to <host>
  hosts: <target_host>
  become: yes
  vars:
    service_name: <service>
    service_dir: /home/docker/appdata/{{ service_name }}

  tasks:
    - name: Create directories
      file:
        path: "{{ item }}"
        state: directory
        mode: '0755'
      loop:
        - "{{ service_dir }}"

    - name: Ensure proxy network exists
      community.docker.docker_network:
        name: proxy
        state: present

    - name: Copy docker-compose.yml
      copy:
        src: ../compose-files/<host>/{{ service_name }}/docker-compose.yml
        dest: "{{ service_dir }}/docker-compose.yml"
        mode: '0644'

    - name: Copy .env file
      copy:
        src: ../compose-files/<host>/{{ service_name }}/.env
        dest: "{{ service_dir }}/.env"
        mode: '0600'

    - name: Start stack
      community.docker.docker_compose_v2:
        project_src: "{{ service_dir }}"
        state: present
        recreate: always
```

---

## Common Operations

### SSH to Specific Host
```bash
ssh databases
ssh alien
ssh nas  # Uses port 44822 automatically
```

### Run Command on All Docker Hosts
```bash
ansible docker_hosts -m shell -a "docker ps -q | wc -l"
```

### Check Container Logs
```bash
ansible databases -m shell -a "docker logs mealie 2>&1 | tail -30"
```

### Copy File to Host
```bash
ansible databases -m copy -a "src=./file.txt dest=/tmp/file.txt"
```

---

## Troubleshooting

### SSH Connection Refused
```bash
# Access via Proxmox console:
# For LXC: pct enter <CT_ID>
# For VM: qm terminal <VM_ID>

# Then inside:
apt install openssh-server
systemctl enable ssh
systemctl start ssh
```

### SSH Permission Denied
```bash
# Copy your key to the host
ssh-copy-id <hostname>

# Or fix permissions on target:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
```

### Git Push Fails
```bash
# Check remote is correct
git remote -v

# Should show:
# origin  ssh://git@192.168.1.81:2222/maddox/clustered-fucks.git

# If wrong, fix it:
git remote set-url origin ssh://git@192.168.1.81:2222/maddox/clustered-fucks.git
```

---

## Changelog

| Date | Change |
|------|--------|
| 2026-01-23 | Initial deployment |
| 2026-01-24 | Added deploy-mealie.yml, enhanced git workflow docs |