silverbullet-notes/docs/servers/hetzner.md

# Hetzner Server (192.168.12.3) - Custom Applications

_Last updated: 2026-01-05_

This document provides detailed information about custom applications and specialized configurations running on the Hetzner server.

---

## Table of Contents

- hetzner/)](#traefik-configuration-manager-traefik-mod)
- [Node-RED Financial Automation](#node-red-financial-automation)
- [Common Operations](#common-operations)

---

## Traefik Configuration Manager (traefik-mod)

### Overview

A custom Flask web application that provides a database-backed configuration management system for Traefik reverse proxy. Replaces manual YAML file editing with a web interface while maintaining Git version control.

**Container:** traefik-mod
**Image:** traefik-mod-traefik-mod (custom build)
**Source Location:** `/volume1/docker/traefik-mod/`
**Access URLs:**
- https://tm.3ddbrewery.com
- https://tm.fails.me

---

### Architecture

**Technology Stack:**
- **Backend:** Python Flask
- **Database:** MariaDB (mariadb-secondary container)
- **Version Control:** Git (local repository)
- **Config Format:** YAML (auto-generated from database)
- **Authentication:** Authentik SSO via Traefik middleware

**Data Flow:**
```
User → Web Interface (Flask)
           ↓
      MariaDB Database
           ↓
      YAML Generator
           ↓
      /matrix/traefik/config/dyno.yml
           ↓
      Git Commit (local)
           ↓
      Change History (database)
```

---

### Database Schema

**Database:** traefik_config (on mariadb-secondary)
**User:** traefik_user

**Tables:**
- **routers:** HTTP/TCP router configurations
- **services:** Backend service definitions
- **middlewares:** Middleware configurations (auth, headers, rate limiting, etc.)
- **config_history:** Audit trail of all configuration changes

**Key Features:**
- Primary keys: Auto-incrementing IDs
- Unique constraints: Router/service/middleware names
- JSON fields: Complex configurations (servers, headers, etc.)
- Timestamps: created_at, updated_at for all entries
- Audit trail: Full history of changes with user and timestamp

---

### File Structure

```
/volume1/docker/traefik-mod/
├── docker-compose.yml          # Container definition
├── Dockerfile                   # Custom image build
├── app.py                       # Flask application
├── requirements.txt             # Python dependencies
├── .env                         # Environment variables
├── maria.md                     # phpMyAdmin setup instructions
├── backups/                     # Configuration backups (30-day retention)
├── templates/                   # HTML templates
├── static/                      # CSS, JavaScript
├── utils/
│   ├── db.py                    # Database connection
│   ├── yaml_generator.py        # YAML generation from DB
│   └── git_manager.py           # Git operations
├── scripts/
│   └── migrate_yaml_to_db.py   # Migration script (YAML → DB)
└── docs/
    ├── MIGRATION_GUIDE.md       # Complete migration documentation
    ├── database-schema.md       # Database schema details
    └── IMPLEMENTATION_COMPLETE.md  # Testing checklist
```

---

### Configuration

**Environment Variables:**
```yaml
TZ: America/New_York
FLASK_ENV: production
FLASK_SECRET_KEY: <secret>
DYNO_FILE_PATH: /config/dyno.yml
BACKUP_DIR: /backups
BACKUP_RETENTION_DAYS: 30
BACKUP_SCHEDULE: 0 0 * * *
TRAEFIK_CONTAINER_NAME: traefik
UID: 1000
GID: 1000
DB_HOST: mariadb-secondary
DB_PORT: 3306
DB_NAME: traefik_config
DB_USER: traefik_user
DB_PASSWORD: <password>
GIT_ENABLED: true
SQL_ECHO: false
```

**Volume Mounts:**
- `/matrix/traefik/config/dyno.yml:/config/dyno.yml:rw` - Generated Traefik config
- `./backups:/backups:rw` - Backup storage
- `/var/run/docker.sock:/var/run/docker.sock:ro` - Docker API access (read-only)

**Resource Limits:**
- CPU: 0.5
- Memory Limit: 256M
- Memory Reservation: 64M

---

### Web Interface Features

**Configuration Management:**
- Create/Edit/Delete routers, services, middlewares
- Form view for guided configuration
- YAML edit mode for advanced users
- Real-time YAML validation
- Search across all configuration items

**Router Management:**
- HTTP and TCP router support
- Domain-based routing rules
- TLS configuration with cert resolvers
- Middleware assignment
- Service assignment
- Priority configuration

**Service Management:**
- Load balancer configuration
- Server definitions (host:port)
- Health check configuration
- Sticky sessions
- Pass host header settings

**Middleware Management:**
- Authentication (BasicAuth, Authentik forward auth)
- Headers (secure headers, CORS, custom headers)
- Rate limiting
- IP whitelisting
- Redirects (scheme, regex)
- Strip prefix
- Retry configuration

**Version Control:**
- Automatic Git commits on every change
- Commit messages with change details
- Browse Git history
- View diffs between versions
- Rollback capability (via Git)

**Audit Trail:**
- Complete history in `config_history` table
- User identification
- Timestamp tracking
- Change type (create, update, delete)
- Before/after values

**Operations:**
- Restart Traefik button (with countdown timer)
- Backup before every change
- Export configuration as YAML
- Import from existing YAML (migration)

---

### Database Setup

**Manual Setup via phpMyAdmin:**

User prefers phpMyAdmin for database management. Complete setup instructions are in `/volume1/docker/traefik-mod/maria.md`.

**Steps:**
1. Access phpMyAdmin on primary server
2. Connect to mariadb-secondary (192.168.12.3:3306)
3. Create database: `traefik_config`
4. Create user: `traefik_user`
5. Grant privileges: ALL on `traefik_config.*`
6. Import schema (if provided) or let application create tables

**Important:**
- Uses `--skip-db-creation` flag in initialization
- No MariaDB root password required in .env
- Database/user created manually by user via phpMyAdmin

---

### API Endpoints

**Web Interface:**
- `GET /` - Dashboard/home page
- `GET /routers` - List all routers
- `GET /routers/new` - Create new router
- `GET /routers/<id>/edit` - Edit router
- `POST /routers/<id>/delete` - Delete router
- Similar patterns for `/services` and `/middlewares`

**System Operations:**
- `GET /health` - Healthcheck endpoint
- `POST /restart-traefik` - Restart Traefik container
- `GET /search` - Global search
- `GET /history` - View change history
- `GET /export` - Export configuration as YAML

**YAML Operations:**
- `POST /generate-yaml` - Regenerate YAML from database
- `GET /view-yaml` - View current generated YAML
- `GET /backup` - Manual backup

---

### Common Operations

**Access Web Interface:**
```bash
# Browser access (requires Authentik login)
https://tm.3ddbrewery.com
https://tm.fails.me
```

**View Logs:**
```bash
cd /volume1/docker/traefik-mod
docker compose logs -f
```

**Rebuild Container:**
```bash
cd /volume1/docker/traefik-mod
docker compose down
docker compose build
docker compose up -d
```

**Database Access (via phpMyAdmin):**
1. Open phpMyAdmin on primary server
2. Connect to 192.168.12.3:3306
3. Select `traefik_config` database
4. Browse tables: routers, services, middlewares, config_history

**Manual YAML Generation:**
```bash
docker exec -it traefik-mod python -c "
from utils.yaml_generator import YAMLGenerator
generator = YAMLGenerator()
generator.write_yaml_file(backup=True)
"
```

**View Git History:**
```bash
# Last 20 commits
docker exec -it traefik-mod git -C /config log --oneline -20

# Detailed commit
docker exec -it traefik-mod git -C /config show <commit-hash>

# View all commits
docker exec -it traefik-mod git -C /config log --oneline
```

**Migration (YAML → Database):**
```bash
# Dry run (preview changes)
docker exec -it traefik-mod python scripts/migrate_yaml_to_db.py --dry-run

# Execute migration
docker exec -it traefik-mod python scripts/migrate_yaml_to_db.py
```

**Restart Traefik:**
```bash
# Via web interface (recommended)
# Click "Restart Traefik" button on dashboard

# Manual restart
docker restart matrix-traefik

# Check Traefik status
docker ps | grep traefik
docker logs matrix-traefik --tail 50
```

**Backup Configuration:**
```bash
# Backups stored in ./backups/
ls -lh /volume1/docker/traefik-mod/backups/

# Manual backup
docker exec -it traefik-mod python -c "
from utils.yaml_generator import YAMLGenerator
generator = YAMLGenerator()
generator.write_yaml_file(backup=True)
"
```

---

### Troubleshooting

**Check Application Health:**
```bash
# Healthcheck endpoint
curl http://localhost:5000/health

# Container status
docker ps | grep traefik-mod
docker inspect traefik-mod | grep -A 5 Health
```

**Database Connection Issues:**
```bash
# Test database connection
docker exec -it traefik-mod python -c "
from utils.db import get_db
db = get_db()
print('Database connected successfully')
"

# Check MariaDB is accessible
docker exec -it traefik-mod ping mariadb-secondary
docker exec -it traefik-mod nc -zv mariadb-secondary 3306
```

**YAML Generation Issues:**
```bash
# Check YAML file exists and is writable
docker exec -it traefik-mod ls -lh /config/dyno.yml

# Validate YAML syntax
docker exec -it traefik-mod python -c "
import yaml
with open('/config/dyno.yml', 'r') as f:
    config = yaml.safe_load(f)
    print('YAML is valid')
"
```

**Git Issues:**
```bash
# Check Git status
docker exec -it traefik-mod git -C /config status

# View Git config
docker exec -it traefik-mod git -C /config config --list
```

---

### Development Setup

**Build Custom Image:**
```bash
cd /volume1/docker/traefik-mod
docker compose build
```

**Run in Development Mode:**
```bash
# Edit .env
FLASK_ENV=development
SQL_ECHO=true

# Restart container
docker compose down && docker compose up -d

# View detailed logs
docker compose logs -f
```

**Database Schema Updates:**
1. Modify database schema via phpMyAdmin
2. Update `docs/database-schema.md`
3. Update migration scripts if needed
4. Test YAML generation
5. Rebuild container

---

### Security

**Authentication:**
- Protected by Authentik forward authentication
- Traefik middleware: `authentik@file`
- Requires valid Authentik login

**Authorization:**
- All authenticated users have full access
- No role-based access control (all-or-nothing)
- Trust model: users with Authentik access are trusted

**Database Security:**
- Dedicated database user (traefik_user)
- Limited to traefik_config database only
- Credentials in .env (not committed to Git)

**Docker Socket:**
- Read-only access to Docker socket
- Used only for restarting Traefik container
- No other Docker operations permitted

**Audit Trail:**
- All changes logged to config_history table
- Git commits provide secondary audit trail
- Timestamps and user identification

---

### Backup and Recovery

**Automatic Backups:**
- Triggered before every configuration change
- Stored in `/volume1/docker/traefik-mod/backups/`
- Retention: 30 days
- File format: `dyno.yml.backup-YYYYMMDD-HHMMSS`

**Git History:**
- Local Git repository in `/config/` (inside container)
- Every change committed automatically
- Commit messages include change type and details
- Full history preserved indefinitely

**Manual Backup:**
```bash
# Copy current YAML
cp /matrix/traefik/config/dyno.yml /volume1/docker/backup/dyno.yml.$(date +%Y%m%d)

# Export database
docker exec mariadb-secondary mysqldump -u traefik_user -p traefik_config > /volume1/docker/backup/traefik_config_$(date +%Y%m%d).sql
```

**Recovery:**
```bash
# Restore from backup file
cp /volume1/docker/traefik-mod/backups/dyno.yml.backup-YYYYMMDD-HHMMSS /matrix/traefik/config/dyno.yml
docker restart matrix-traefik

# Restore from Git history
docker exec -it traefik-mod git -C /config log --oneline
docker exec -it traefik-mod git -C /config checkout <commit-hash> dyno.yml
docker restart matrix-traefik

# Restore database from dump
docker exec -i mariadb-secondary mysql -u traefik_user -p traefik_config < backup_file.sql
```

---

## Node-RED Financial Automation

### Overview

Node-RED instance dedicated to financial transaction automation, including automated imports, categorization, and synchronization with Firefly III.

**Container:** node-red
**Image:** nodered/node-red:latest
**Source Location:** `/volume1/docker/node-red/`
**Data Location:** `/volume1/docker/node-red/` (flows, settings, node_modules)
**Access URL:** https://node-het.3ddbrewery.com

---

### Architecture

**Technology Stack:**
- **Platform:** Node-RED (Node.js-based flow programming)
- **Runtime:** Node.js
- **Authentication:** Username/password (NODE_RED_USERNAME/PASSWORD)
- **Data Storage:** JSON files in /data
- **Credential Encryption:** NODE_RED_CREDENTIAL_SECRET

**Integration Points:**
- **Firefly III:** API integration for transaction management
- **node-staging Database:** MariaDB database for transaction staging
- **ntfy:** Notification service for alerts
- **External Services:** Banks, financial APIs (via flows)

---

### Configuration

**Environment Variables:**
```yaml
TZ: America/New_York
NODE_RED_CREDENTIAL_SECRET: 396572538122375581201991
NODE_RED_USERNAME: maddox
NODE_RED_PASSWORD: ./sk8nh8
NPM_CONFIG_CACHE: /data/.npm
NPM_CONFIG_PREFIX: /data/.npm
NODE_PATH: /data/node_modules
```

**Volume Mounts:**
- `/volume1/docker/node-red:/data:rw` - Node-RED data (flows, settings, modules)
- `/home/maddox:/media:rw` - User directory access for file operations

**Network:**
- Connected to `traefik` network for external access
- No isolated network (direct access to other services)

**Resource Limits:**
- No explicit CPU/memory limits configured
- Runs as user 1000:1000

---

### Custom Healthcheck

**Healthcheck Configuration:**
```yaml
test: ["CMD-SHELL", "node /healthcheck.js || (curl -H 'Title: Node-RED Down' -H 'Priority: high' -H 'Tags: alert,node-red,down' -d 'Node-RED health check failed on Hetzner-fin' http://192.168.1.70:6741/hetzner_alerts && exit 1)"]
interval: 120s
timeout: 60s
retries: 4
start_period: 120s
```

**Features:**
- Primary check: Custom Node.js healthcheck script
- Fallback: Sends notification to ntfy if health check fails
- Notification server: http://192.168.1.70:6741/hetzner_alerts
- Alert topic: hetzner_alerts
- Tags: alert, node-red, down
- High priority notification

**Notification Details:**
- Title: "Node-RED Down"
- Message: "Node-RED health check failed on Hetzner-fin"
- Ensures administrators are alerted to Node-RED failures

---

### Flows and Functionality

**Purpose:**
Node-RED on this server runs automated financial transaction flows:
- Transaction imports from external sources
- Data transformation and categorization
- Staging in `node-staging` database
- Synchronization with Firefly III
- Error handling and notifications

**Flow Structure:**
Flows are stored in `/volume1/docker/node-red/flows.json` (accessible in /data inside container)

**Common Flow Patterns:**
1. **Scheduled Triggers:** Cron-based automation
2. **API Polling:** Check external sources for new transactions
3. **Data Transformation:** Parse and normalize transaction data
4. **Database Operations:** Stage transactions in `node-staging`
5. **Firefly API Calls:** Create/update transactions in Firefly III
6. **Error Handling:** Catch errors, log, send notifications

---

### Database Integration

**node-staging Database:**
- **Host:** mariadb-secondary (192.168.12.3:3306)
- **Database:** node-staging
- **Purpose:** Staging area for financial transactions
- **Replication:** Does NOT replicate (isolated for testing)
- **Access:** Node-RED flows connect directly to MariaDB

**Connection Details:**
Node-RED flows include MySQL nodes configured to connect to:
- Host: 192.168.12.3 (or mariadb-secondary if on same Docker network)
- Port: 3306
- Database: node-staging
- User: (configured in flow credentials)

---

### Installed NPM Packages

**Package Management:**
- NPM cache: `/data/.npm`
- NPM prefix: `/data/.npm`
- Node modules: `/data/node_modules`

**Common Packages (likely installed):**
- `node-red-node-mysql` - MySQL database integration
- `node-red-contrib-cron-plus` - Advanced scheduling
- HTTP request nodes (built-in) - API calls to Firefly III
- Dashboard nodes - UI for monitoring (if used)

**Install Additional Packages:**
```bash
docker exec -it node-red npm install <package-name>
```

**View Installed Packages:**
```bash
docker exec -it node-red npm list --depth=0
```

---

### Common Operations

**Access Web Interface:**
```bash
# Browser access
https://node-het.3ddbrewery.com

# Login credentials
Username: maddox
Password: ./sk8nh8
```

**View Logs:**
```bash
docker logs node-red -f
```

**Restart Node-RED:**
```bash
docker restart node-red
```

**Access Container Shell:**
```bash
docker exec -it node-red /bin/bash
```

**Install NPM Packages:**
```bash
docker exec -it node-red npm install <package-name>

# Example: Install MySQL node
docker exec -it node-red npm install node-red-node-mysql
```

**Backup Flows:**
```bash
# Flows stored in /volume1/docker/node-red/flows.json
cp /volume1/docker/node-red/flows.json /volume1/docker/backup/node-red-flows-$(date +%Y%m%d).json

# Backup entire Node-RED data directory
tar -czf /volume1/docker/backup/node-red-data-$(date +%Y%m%d).tar.gz /volume1/docker/node-red/
```

**Restore Flows:**
```bash
# Copy backup to flows.json
cp /volume1/docker/backup/node-red-flows-YYYYMMDD.json /volume1/docker/node-red/flows.json

# Restart Node-RED
docker restart node-red
```

**Check Database Connection:**
```bash
# Test connection from Node-RED container
docker exec -it node-red sh -c "nc -zv 192.168.12.3 3306"

# Or test MySQL connection
docker exec -it mariadb-secondary mysql -u <user> -p node-staging -e "SHOW TABLES;"
```

---

### Troubleshooting

**Node-RED Won't Start:**
```bash
# Check logs
docker logs node-red

# Check file permissions
ls -lh /volume1/docker/node-red/

# Ensure ownership is 1000:1000
sudo chown -R 1000:1000 /volume1/docker/node-red/
```

**Flow Execution Issues:**
```bash
# Check Node-RED debug panel (in web interface)
# Enable debug nodes in flows
# View logs in real-time
docker logs node-red -f
```

**Database Connection Failures:**
```bash
# Verify MariaDB is running
docker ps | grep mariadb

# Test network connectivity
docker exec -it node-red ping mariadb-secondary

# Check database exists
docker exec -it mariadb-secondary mysql -u root -p -e "SHOW DATABASES;"
```

**Credential Decryption Issues:**
```bash
# Verify NODE_RED_CREDENTIAL_SECRET hasn't changed
docker inspect node-red | grep CREDENTIAL_SECRET

# If secret changed, flows_cred.json needs re-encryption
# (This requires manual intervention)
```

---

### Security

**Authentication:**
- Username/password authentication enabled
- Credentials: maddox / ./sk8nh8
- Protected by Traefik reverse proxy
- No additional Authentik protection (direct auth only)

**Credential Storage:**
- Flow credentials encrypted with NODE_RED_CREDENTIAL_SECRET
- Stored in `/data/flows_cred.json`
- Secret must remain constant for credential decryption

**File Access:**
- Has access to `/home/maddox` directory (mounted as /media)
- Can read/write files on host filesystem
- Runs as user 1000:1000 (limited privileges)

**Network Access:**
- Can access any service on traefik network
- Can access external internet (for API calls)
- Direct access to MariaDB on port 3306

---

### Monitoring and Alerts

**Healthcheck:**
- Custom healthcheck script (/healthcheck.js)
- Runs every 120 seconds
- Timeout: 60 seconds
- Retries: 4 before considered unhealthy

**Failure Notifications:**
- Automatic ntfy notification on health check failure
- Notification URL: http://192.168.1.70:6741/hetzner_alerts
- Topic: hetzner_alerts
- Priority: high

**Autoheal Integration:**
- Container labeled with `autoheal=true`
- Autoheal monitors health status
- Automatic restart if unhealthy

**Watchtower Integration:**
- Container labeled with watchtower enable
- Automatic image updates
- Rolling restart on update

---

### Development and Debugging

**Enable Debug Logging:**
```bash
# In Node-RED web interface:
# 1. Add Debug nodes to flows
# 2. View Debug panel (right sidebar)
# 3. Enable verbose logging in settings

# Container logs show Node-RED startup and errors
docker logs node-red -f
```

**Edit Settings:**
```bash
# Settings file
nano /volume1/docker/node-red/settings.js

# Restart after changes
docker restart node-red
```

**Install Development Tools:**
```bash
# Access container
docker exec -it node-red /bin/bash

# Install tools
npm install -g <package>

# View environment
env | grep NODE
```

---

## Common Operations

### Service Restart

**Restart All Services:**
```bash
# Restart everything in /volume1/docker/
cd /volume1/docker
for dir in */; do
    if [ -f "$dir/docker-compose.yml" ]; then
        echo "Restarting $dir"
        cd "$dir"
        docker compose restart
        cd ..
    fi
done
```

**Restart Specific Service:**
```bash
cd /volume1/docker/<service-name>
docker compose restart
```

### Log Viewing

**View Logs for Service:**
```bash
cd /volume1/docker/<service-name>
docker compose logs -f
```

**View Logs for Specific Container:**
```bash
docker logs <container-name> -f

# Examples:
docker logs traefik-mod -f
docker logs node-red -f
docker logs mariadb-secondary -f
```

### Container Rebuilding

**Rebuild Custom Images:**
```bash
# Traefik-mod
cd /volume1/docker/traefik-mod
docker compose down
docker compose build
docker compose up -d

# Check status
docker ps | grep traefik-mod
```

### Health Monitoring

**Check All Container Health:**
```bash
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.State}}"

# Filter only unhealthy containers
docker ps --filter health=unhealthy
```

**Check Autoheal Logs:**
```bash
docker logs autoheal -f
```

### Upgrade matrix implementation
```bash
just update
 - or -
git pull

 - then - 
just roles

 - then -
just install-all
 - or -
just setup-all

# If you receive an error about a missing role
just roles

# If needed, add files to commit and stash
git add -A
git stash
git pull

# Check services
just run-tags self-check
 - or -
ansible-playbook -i inventory/hosts setup.yml --tags=self-check
```

---

## Summary

**Custom Applications:**
1. **traefik-mod:** Database-backed Traefik configuration manager with web UI
2. **node-red:** Financial automation platform with custom flows and database integration

**Key Features:**
- Both applications use web interfaces for management
- Database integration with mariadb-secondary
- Automatic backups and version control (traefik-mod)
- Health monitoring and notifications (node-red)
- Authentication via Authentik (traefik-mod) or built-in (node-red)

**Management:**
- Web-based administration preferred
- phpMyAdmin for database management
- Git version control for configuration tracking
- Automated monitoring via autoheal and watchtower