silverbullet-notes/docs/servers/hetzner/README.md

# Hetzner Server (192.168.12.3) Documentation

_Last updated: 2026-01-05_

---

## Server Overview

This server is a secondary server running Docker-based infrastructure, primarily focused on financial automation, identity management, and supporting services. It operates at **192.168.12.3** (hostname: **im**) and serves as a companion to the primary server at 192.168.1.251.

**Primary Role:** Financial automation server with MariaDB replication, identity provider, and custom applications

**Key Functions:**
- **Financial Automation:** Node-RED flows for automated transaction processing
- **Identity Provider:** Authentik SSO for centralized authentication
- **Database Replication:** MariaDB secondary for disaster recovery
- **Traefik Management:** Web-based configuration manager with database backend
- **VPN Connectivity:** Tailscale mesh VPN and ProtonVPN privacy layer

---

## Server Specifications

- **IP Address:** 192.168.12.3
- **Hostname:** im
- **Local Filesystem:** /volume1/docker
- **Operating System:** Linux
- **Container Runtime:** Docker with Docker Compose
- **Timezone:** America/New_York

---

## Key Services

### Authentication & Identity

- **Authentik** (id.3ddbrewery.com, id.fails.me)
  - Modern identity provider with SSO, OAuth2, SAML support
  - PostgreSQL 16 backend
  - Redis session storage
  - Gmail SMTP integration
  - 4 containers: postgres, redis, server, worker

- **Authelia** (CONFIGURED BUT NOT RUNNING)
  - Authentication proxy with Redis backend
  - Configured domains: auth.fails.me, auth.3ddbrewery.com

### Infrastructure Services

- **MariaDB Secondary** (192.168.12.3:3306)
  - Read-only replica of primary server (192.168.1.251)
  - Contains `node-staging` database (does NOT replicate - isolated for testing)
  - Contains `traefik_config` database for Traefik configuration
  - 60M memory reservation, 0.4 CPU limit

- **Gluetun VPN** (38888:HTTP proxy, 38388:Shadowsocks, 38000:Control)
  - ProtonVPN client (US servers: Secaucus, Chicago, New York)
  - HTTP proxy on port 38888
  - Ad, malware, and surveillance blocking enabled
  - Port forwarding enabled

- **Tailscale** (im-ts)
  - Mesh VPN for remote access
  - Advertises route: 192.168.12.3/32
  - Accepts routes from other nodes
  - Acts as exit node
  - Host network mode

- **Docker Socket Proxy** (192.168.12.3:2376)
  - Secure Docker API access for Portainer
  - Limited permissions (containers, images, networks, volumes)
  - Bound to private IP only for security
  - Read-only Docker socket access

- **Traefik Configuration Manager** (tm.3ddbrewery.com, tm.fails.me)
  - Custom Flask web application
  - Database-backed Traefik configuration
  - Git version control (local repository)
  - Automatic YAML generation from database
  - Complete audit trail and change history
  - MariaDB backend (traefik_config database)

### Application Services

- **Firefly III** (f.3ddbrewery.com, port 6182)
  - Personal finance manager
  - Version 6.2.21 (pinned - known working with automated transactions)
  - MariaDB 11.3 database
  - Redis cache
  - Homepage widget integration
  - 3 containers: firefly, db, redis

- **Node-RED** (node-het.3ddbrewery.com, port 1880)
  - Financial automation platform
  - Runs automated transaction flows
  - Integrates with `node-staging` database
  - Integrates with Firefly III API
  - Custom healthcheck with ntfy notification
  - Access to /home/maddox for file operations

### Utility Services

- **Autoheal**
  - Monitors container health
  - Automatically restarts unhealthy containers (labeled with `autoheal=true`)
  - Check interval: 5 seconds
  - Webhook notifications to ntfy (https://ntfy.3ddbrewery.com/autoheal-IM)

- **Watchtower**
  - Automatic container updates
  - Updates containers labeled with `com.centurylinklabs.watchtower.enable=true`
  - Poll interval: 1 hour
  - Email notifications (xoppaw@gmail.com → brian.w.maddox@gmail.com)
  - Cleanup old images after update

### Matrix/Synapse Stack

This server also hosts a complete Matrix/Synapse installation (23 containers) managed separately. These are listed in the documentation but not detailed per instructions.

**Matrix containers include:**
- Synapse homeserver
- Element web client
- PostgreSQL database with automated backups
- Multiple bridges (WhatsApp, Telegram, Signal, Google Messages)
- Bots (Maubot, reminder-bot)
- Monitoring (Prometheus, Grafana, node-exporter)
- Support services (Coturn, ntfy, Exim relay, Heisenbridge)
- Traefik reverse proxy with certificate dumper

---

## Documentation Index

### [00-service-inventory.md](./00-service-inventory.md)
Complete inventory of all Docker services running on this server, including:
- Detailed container specifications
- Port mappings and volumes
- Dependencies and relationships
- Resource limits and healthchecks
- Traefik routing configuration
- Homepage integration details

**Sections:**
- Authentication & Identity Services (Authentik, Authelia)
- Infrastructure Services (MariaDB, Gluetun, Tailscale, Socket Proxy, Traefik-mod)
- Application Services (Firefly III, Node-RED)
- Utility Services (Autoheal, Watchtower)
- Matrix/Synapse Containers (23 containers listed)

---

### [01-databases.md](./01-databases.md)
Comprehensive documentation of all database systems, including:
- MariaDB instances (mariadb-secondary, Firefly-DB)
- PostgreSQL instances (Authentik, Matrix)
- Redis instances (Authentik, Firefly, Authelia)
- Access methods and connection details
- Backup procedures and recovery
- Database-specific configurations

**Key Databases:**
- `node-staging` - Financial bot testing (does NOT replicate)
- `traefik_config` - Traefik configuration storage
- `firefly` - Personal finance data
- `authentik` - Identity/SSO data

**Access Preference:** phpMyAdmin for MariaDB management (alternative: command-line)

---

### [02-network-architecture.md](./02-network-architecture.md)
Detailed network architecture documentation, including:
- Docker networks (18 total)
- Traefik reverse proxy configuration
- VPN setup (Tailscale + ProtonVPN)
- Port mappings and security
- Network isolation strategies
- Inter-server connectivity

**Network Highlights:**
- External `traefik` network for reverse proxy
- Service-specific isolated networks (authentik, firefly, gluetun, etc.)
- Matrix/Synapse networks (8 networks)
- Host network mode for Tailscale
- Network security and isolation

---

### [03-custom-applications.md](./03-custom-applications.md)
Documentation of custom applications and specialized configurations:

**Traefik Configuration Manager (traefik-mod):**
- Flask web application for Traefik management
- Database-backed configuration (MariaDB)
- Git version control (local repository)
- Web interface for routers, services, middlewares
- Automatic YAML generation and validation
- Complete audit trail and change history

**Node-RED Financial Automation:**
- Dedicated financial automation instance
- Custom flows for transaction processing
- `node-staging` database integration
- Firefly III API integration
- Custom healthcheck with ntfy alerts
- NPM package management

---

## Important Notes

### Node-staging Database

The `node-staging` database on mariadb-secondary is **intentionally isolated**:
- **Does NOT replicate** from primary server
- Used exclusively for financial bot testing
- Provides safe testing environment without affecting production data
- Accessed by Node-RED financial automation flows

### Synapse/Matrix Containers

This server runs a complete Matrix/Synapse installation with 23 containers. These are managed separately (not in `/volume1/docker/`) and are documented by name only per instructions. For detailed Matrix documentation, refer to Matrix-specific documentation (not included here).

### Firefly III Version

Firefly III is **pinned to version 6.2.21** because this version is known to work reliably with automated transaction flows. Do not update without testing automation compatibility.

### Traefik Configuration

Traefik configuration is managed via the **traefik-mod** web interface. Direct YAML editing is discouraged - use the web UI at tm.3ddbrewery.com or tm.fails.me instead. All changes are version-controlled via Git and stored in the database.

### MariaDB Management

User prefers **phpMyAdmin** for MariaDB database management. Access phpMyAdmin on the primary server and connect to 192.168.12.3:3306 for this server's MariaDB instance.

---

## Quick Reference

### Common Commands

**Service Management:**
```bash
# Navigate to service directory
cd /volume1/docker/<service-name>

# Start service
docker compose up -d

# Stop service
docker compose down

# Restart service
docker compose restart

# View logs
docker compose logs -f
```

**Container Management:**
```bash
# List running containers
docker ps

# Check container health
docker ps --filter "label=autoheal=true"

# View container logs
docker logs <container-name> -f

# Access container shell
docker exec -it <container-name> /bin/bash
```

**Database Access:**
```bash
# MariaDB secondary
docker exec -it mariadb-secondary mysql -u root -p

# Firefly database
docker exec -it Firefly-DB mysql -u fireflyuser -p firefly

# Authentik PostgreSQL
docker exec -it authentik-postgres psql -U authentik -d authentik
```

**Network Troubleshooting:**
```bash
# List networks
docker network ls

# Inspect network
docker network inspect traefik

# Check connectivity
docker exec <container> ping <target>
docker exec <container> nc -zv <target> <port>
```

**VPN Status:**
```bash
# Tailscale status
docker exec tailscale tailscale status

# Gluetun status
curl http://192.168.12.3:38000/v1/openvpn/status
```

---

## Service URLs

| Service | URL | Authentication |
|---------|-----|----------------|
| Authentik | https://id.3ddbrewery.com<br>https://id.fails.me | Authentik SSO |
| Traefik Manager | https://tm.3ddbrewery.com<br>https://tm.fails.me | Authentik SSO |
| Node-RED | https://node-het.3ddbrewery.com | Username/Password |
| Firefly III | https://f.3ddbrewery.com<br>http://192.168.12.3:6182 | Firefly Login |
| Gluetun Control | http://192.168.12.3:38000 | None |

---

## Port Reference

| Port | Service | Purpose | Access |
|------|---------|---------|--------|
| 80 | Traefik | HTTP (→ HTTPS) | Public |
| 443 | Traefik | HTTPS | Public |
| 8448 | Traefik | Matrix Federation | Public |
| 3306 | MariaDB | Database | Public (use with caution) |
| 1880 | Node-RED | Automation Platform | Public (via Traefik) |
| 6182 | Firefly III | Finance Manager | Public |
| 2376 | Socket Proxy | Docker API | 192.168.12.3 only |
| 38888 | Gluetun | HTTP Proxy | Public |
| 38388 | Gluetun | Shadowsocks | Public |
| 38000 | Gluetun | Control API | Public |

---

## Container Statistics

**Total Containers:** 38
- **/volume1/docker services:** 15 containers (9 services)
- **Matrix/Synapse stack:** 23 containers (managed separately)

**Services in /volume1/docker:**
1. authentik (4 containers)
2. firefly (3 containers)
3. mariadb (1 container)
4. gluetun (1 container)
5. tailscale (1 container)
6. socket-proxy (1 container)
7. node-red (1 container)
8. utils (2 containers: autoheal, watchtower)
9. traefik-mod (1 container)

**Configured but Not Running:**
- authelia (2 containers: authelia, authelia_redis)

**Active Databases:** 5
- mariadb-secondary (MariaDB latest)
- Firefly-DB (MariaDB 11.3)
- authentik-postgres (PostgreSQL 16)
- matrix-postgres (PostgreSQL 17.7 - Matrix stack)
- Redis instances: 3 active (authentik, firefly, matrix)

**Docker Networks:** 18
- 1 external (traefik)
- 9 service-specific (/volume1/docker services)
- 8 Matrix/Synapse networks

---

## Automation & Monitoring

**Automatic Updates:**
- **Watchtower:** Checks hourly for image updates
- **Label:** `com.centurylinklabs.watchtower.enable=true`
- **Notifications:** Email to brian.w.maddox@gmail.com
- **Cleanup:** Removes old images after update

**Health Monitoring:**
- **Autoheal:** Checks every 5 seconds
- **Label:** `autoheal=true`
- **Action:** Automatic restart of unhealthy containers
- **Notifications:** Webhook to ntfy (autoheal-IM topic)

**Node-RED Custom Healthcheck:**
- **Interval:** 120 seconds
- **Failure Action:** Sends notification to ntfy
- **Topic:** hetzner_alerts
- **Priority:** High

---

## Backup & Recovery

### Database Backups

**MariaDB Secondary:**
- **Replication:** Live replica from 192.168.1.251 (disaster recovery)
- **Manual Backup:** `docker exec mariadb-secondary mysqldump ...`
- **Backup Directory:** `/volume1/docker/backup`

**Firefly Database:**
```bash
docker exec Firefly-DB mysqldump -u fireflyuser -p firefly > /volume1/docker/backup/firefly_$(date +%Y%m%d).sql
```

**Authentik PostgreSQL:**
```bash
docker exec authentik-postgres pg_dump -U authentik authentik > /volume1/docker/backup/authentik_$(date +%Y%m%d).sql
```

**Matrix PostgreSQL:**
- Automated backups via matrix-postgres-backup container
- Image: prodrigestivill/postgres-backup-local:18-alpine

### Configuration Backups

**Traefik Configuration:**
- Automatic backups before every change
- Stored in `/volume1/docker/traefik-mod/backups/`
- Retention: 30 days
- Git version control (local repository)

**Node-RED Flows:**
```bash
cp /volume1/docker/node-red/flows.json /volume1/docker/backup/node-red-flows-$(date +%Y%m%d).json
```

**Docker Compose Files:**
```bash
tar -czf /volume1/docker/backup/docker-compose-files-$(date +%Y%m%d).tar.gz /volume1/docker/*/docker-compose.yml
```

---

## Security Overview

### Authentication

- **Authentik SSO:** Centralized authentication for selected services
- **Traefik Middlewares:** authentik@file for SSO-protected services
- **Node-RED:** Username/password authentication
- **Firefly III:** Application-level authentication

### Network Security

- **Service Isolation:** Each service stack on isolated Docker network
- **Database Isolation:** PostgreSQL and Redis on internal networks only
- **Socket Proxy:** Limited permissions, bound to private IP only
- **TLS/SSL:** All web services use HTTPS via Traefik
- **Certificate Management:** Automatic Let's Encrypt certificates

### Container Security

- **Security Options:** no-new-privileges enabled on most containers
- **Read-only Filesystems:** Where applicable (e.g., Redis)
- **User Restrictions:** Non-root users (1000:1000) where possible
- **Capability Dropping:** Minimal capabilities granted
- **Resource Limits:** CPU and memory limits prevent resource exhaustion

### VPN Security

- **Tailscale:** Encrypted mesh VPN for remote access
- **ProtonVPN:** Privacy VPN layer via Gluetun
- **Route Advertisement:** Only 192.168.12.3/32 advertised
- **Firewall:** Docker iptables rules + host firewall

---

## Troubleshooting

### Service Won't Start

```bash
# Check logs
docker logs <container-name>

# Check dependencies
docker compose ps

# Check network
docker network inspect <network-name>

# Rebuild container
cd /volume1/docker/<service-name>
docker compose down
docker compose build
docker compose up -d
```

### Database Connection Issues

```bash
# Check database is running
docker ps | grep -E "mariadb|postgres"

# Test connection
docker exec -it <container> ping <database-host>
docker exec -it <container> nc -zv <database-host> <port>

# Check database logs
docker logs mariadb-secondary
docker logs authentik-postgres
```

### Traefik Routing Issues

```bash
# Check Traefik logs
docker logs matrix-traefik

# Verify service is on traefik network
docker network inspect traefik

# Check Traefik configuration
cat /matrix/traefik/config/dyno.yml

# Use traefik-mod web interface
# https://tm.3ddbrewery.com
```

### Network Connectivity Issues

```bash
# Check Docker networks
docker network ls

# Inspect network
docker network inspect <network-name>

# Test connectivity
docker exec <container> ping <target>
docker exec <container> curl -I <url>
```

### VPN Issues

```bash
# Tailscale status
docker exec tailscale tailscale status
docker exec tailscale tailscale ping <node>

# Gluetun status
curl http://192.168.12.3:38000/v1/openvpn/status
docker logs gluetun
```

---

## Additional Resources

### Documentation Files

- **maria.md** (in traefik-mod): phpMyAdmin database setup instructions
- **MIGRATION_GUIDE.md** (in traefik-mod/docs): Traefik config migration guide
- **database-schema.md** (in traefik-mod/docs): Database schema documentation
- **IMPLEMENTATION_COMPLETE.md** (in traefik-mod/docs): Testing checklist

### External Documentation

- **Authentik:** https://docs.goauthentik.io/
- **Firefly III:** https://docs.firefly-iii.org/
- **Node-RED:** https://nodered.org/docs/
- **Traefik:** https://doc.traefik.io/traefik/
- **Gluetun:** https://github.com/qdm12/gluetun
- **Tailscale:** https://tailscale.com/kb/

---

## Contact & Support

For issues or questions:
- Check service-specific logs: `docker logs <container-name>`
- Review documentation in this directory
- Check autoheal logs for restart events: `docker logs autoheal`
- Review watchtower logs for update issues: `docker logs watchtower`

**Notification Channels:**
- **Autoheal:** https://ntfy.3ddbrewery.com/autoheal-IM
- **Watchtower:** Email to brian.w.maddox@gmail.com
- **Node-RED Health:** http://192.168.1.70:6741/hetzner_alerts

---

## Conclusion

This Hetzner server provides critical infrastructure for financial automation, identity management, and disaster recovery. The documentation in this directory provides comprehensive coverage of all services, configurations, and operational procedures.

**Key Takeaways:**
- All services containerized with Docker Compose
- Automatic health monitoring and updates
- Database replication for disaster recovery
- Custom applications for Traefik management and financial automation
- Dual VPN setup for access and privacy
- Comprehensive documentation for all components

**When transferring to Silverbullet:**
All markdown files in this directory (`/home/maddox/hetzner-docs/`) are ready to be imported into the main server's Silverbullet documentation system for centralized documentation management.