maddox/silverbullet-notes
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
silverbullet-notes/docs/servers/hetzner
History
maddox 3dc0f4dacf Initial SilverBullet sync
2026-01-25 00:20:24 +00:00
..
.md Initial SilverBullet sync 2026-01-25 00:20:24 +00:00
00-service-inventory.md Initial SilverBullet sync 2026-01-25 00:20:24 +00:00
01-databases.md Initial SilverBullet sync 2026-01-25 00:20:24 +00:00
README.md Initial SilverBullet sync 2026-01-25 00:20:24 +00:00
summary.txt Initial SilverBullet sync 2026-01-25 00:20:24 +00:00

README.md

Hetzner Server (192.168.12.3) Documentation

Last updated: 2026-01-05

Server Overview

This server is a secondary server running Docker-based infrastructure, primarily focused on financial automation, identity management, and supporting services. It operates at 192.168.12.3 (hostname: im) and serves as a companion to the primary server at 192.168.1.251.

Primary Role: Financial automation server with MariaDB replication, identity provider, and custom applications

Key Functions:

  • Financial Automation: Node-RED flows for automated transaction processing
  • Identity Provider: Authentik SSO for centralized authentication
  • Database Replication: MariaDB secondary for disaster recovery
  • Traefik Management: Web-based configuration manager with database backend
  • VPN Connectivity: Tailscale mesh VPN and ProtonVPN privacy layer

Server Specifications

  • IP Address: 192.168.12.3
  • Hostname: im
  • Local Filesystem: /volume1/docker
  • Operating System: Linux
  • Container Runtime: Docker with Docker Compose
  • Timezone: America/New_York

Key Services

Authentication & Identity

  • Authentik (id.3ddbrewery.com, id.fails.me)

    • Modern identity provider with SSO, OAuth2, SAML support
    • PostgreSQL 16 backend
    • Redis session storage
    • Gmail SMTP integration
    • 4 containers: postgres, redis, server, worker

  • Authelia (CONFIGURED BUT NOT RUNNING)

    • Authentication proxy with Redis backend
    • Configured domains: auth.fails.me, auth.3ddbrewery.com

Infrastructure Services

  • MariaDB Secondary (192.168.12.3:3306)

    • Read-only replica of primary server (192.168.1.251)
    • Contains node-staging database (does NOT replicate - isolated for testing)
    • Contains traefik_config database for Traefik configuration
    • 60M memory reservation, 0.4 CPU limit

  • Gluetun VPN (38888:HTTP proxy, 38388:Shadowsocks, 38000:Control)

    • ProtonVPN client (US servers: Secaucus, Chicago, New York)
    • HTTP proxy on port 38888
    • Ad, malware, and surveillance blocking enabled
    • Port forwarding enabled

  • Tailscale (im-ts)

    • Mesh VPN for remote access
    • Advertises route: 192.168.12.3/32
    • Accepts routes from other nodes
    • Acts as exit node
    • Host network mode

  • Docker Socket Proxy (192.168.12.3:2376)

    • Secure Docker API access for Portainer
    • Limited permissions (containers, images, networks, volumes)
    • Bound to private IP only for security
    • Read-only Docker socket access

  • Traefik Configuration Manager (tm.3ddbrewery.com, tm.fails.me)

    • Custom Flask web application
    • Database-backed Traefik configuration
    • Git version control (local repository)
    • Automatic YAML generation from database
    • Complete audit trail and change history
    • MariaDB backend (traefik_config database)

Application Services

  • Firefly III (f.3ddbrewery.com, port 6182)

    • Personal finance manager
    • Version 6.2.21 (pinned - known working with automated transactions)
    • MariaDB 11.3 database
    • Redis cache
    • Homepage widget integration
    • 3 containers: firefly, db, redis

  • Node-RED (node-het.3ddbrewery.com, port 1880)

    • Financial automation platform
    • Runs automated transaction flows
    • Integrates with node-staging database
    • Integrates with Firefly III API
    • Custom healthcheck with ntfy notification
    • Access to /home/maddox for file operations

Utility Services

  • Autoheal

    • Monitors container health
    • Automatically restarts unhealthy containers (labeled with autoheal=true)
    • Check interval: 5 seconds
    • Webhook notifications to ntfy (https://ntfy.3ddbrewery.com/autoheal-IM)

  • Watchtower

    • Automatic container updates
    • Updates containers labeled with com.centurylinklabs.watchtower.enable=true
    • Poll interval: 1 hour
    • Email notifications (xoppaw@gmail.com → brian.w.maddox@gmail.com)
    • Cleanup old images after update

Matrix/Synapse Stack

This server also hosts a complete Matrix/Synapse installation (23 containers) managed separately. These are listed in the documentation but not detailed per instructions.

Matrix containers include:

  • Synapse homeserver
  • Element web client
  • PostgreSQL database with automated backups
  • Multiple bridges (WhatsApp, Telegram, Signal, Google Messages)
  • Bots (Maubot, reminder-bot)
  • Monitoring (Prometheus, Grafana, node-exporter)
  • Support services (Coturn, ntfy, Exim relay, Heisenbridge)
  • Traefik reverse proxy with certificate dumper

Documentation Index

00-service-inventory.md

Complete inventory of all Docker services running on this server, including:

  • Detailed container specifications
  • Port mappings and volumes
  • Dependencies and relationships
  • Resource limits and healthchecks
  • Traefik routing configuration
  • Homepage integration details

Sections:

  • Authentication & Identity Services (Authentik, Authelia)
  • Infrastructure Services (MariaDB, Gluetun, Tailscale, Socket Proxy, Traefik-mod)
  • Application Services (Firefly III, Node-RED)
  • Utility Services (Autoheal, Watchtower)
  • Matrix/Synapse Containers (23 containers listed)

01-databases.md

Comprehensive documentation of all database systems, including:

  • MariaDB instances (mariadb-secondary, Firefly-DB)
  • PostgreSQL instances (Authentik, Matrix)
  • Redis instances (Authentik, Firefly, Authelia)
  • Access methods and connection details
  • Backup procedures and recovery
  • Database-specific configurations

Key Databases:

  • node-staging - Financial bot testing (does NOT replicate)
  • traefik_config - Traefik configuration storage
  • firefly - Personal finance data
  • authentik - Identity/SSO data

Access Preference: phpMyAdmin for MariaDB management (alternative: command-line)

02-network-architecture.md

Detailed network architecture documentation, including:

  • Docker networks (18 total)
  • Traefik reverse proxy configuration
  • VPN setup (Tailscale + ProtonVPN)
  • Port mappings and security
  • Network isolation strategies
  • Inter-server connectivity

Network Highlights:

  • External traefik network for reverse proxy
  • Service-specific isolated networks (authentik, firefly, gluetun, etc.)
  • Matrix/Synapse networks (8 networks)
  • Host network mode for Tailscale
  • Network security and isolation

03-custom-applications.md

Documentation of custom applications and specialized configurations:

Traefik Configuration Manager (traefik-mod):

  • Flask web application for Traefik management
  • Database-backed configuration (MariaDB)
  • Git version control (local repository)
  • Web interface for routers, services, middlewares
  • Automatic YAML generation and validation
  • Complete audit trail and change history

Node-RED Financial Automation:

  • Dedicated financial automation instance
  • Custom flows for transaction processing
  • node-staging database integration
  • Firefly III API integration
  • Custom healthcheck with ntfy alerts
  • NPM package management

Important Notes

Node-staging Database

The node-staging database on mariadb-secondary is intentionally isolated:

  • Does NOT replicate from primary server
  • Used exclusively for financial bot testing
  • Provides safe testing environment without affecting production data
  • Accessed by Node-RED financial automation flows

Synapse/Matrix Containers

This server runs a complete Matrix/Synapse installation with 23 containers. These are managed separately (not in /volume1/docker/) and are documented by name only per instructions. For detailed Matrix documentation, refer to Matrix-specific documentation (not included here).

Firefly III Version

Firefly III is pinned to version 6.2.21 because this version is known to work reliably with automated transaction flows. Do not update without testing automation compatibility.

Traefik Configuration

Traefik configuration is managed via the traefik-mod web interface. Direct YAML editing is discouraged - use the web UI at tm.3ddbrewery.com or tm.fails.me instead. All changes are version-controlled via Git and stored in the database.

MariaDB Management

User prefers phpMyAdmin for MariaDB database management. Access phpMyAdmin on the primary server and connect to 192.168.12.3:3306 for this server's MariaDB instance.

Quick Reference

Common Commands

Service Management:

# Navigate to service directory
cd /volume1/docker/<service-name>

# Start service
docker compose up -d

# Stop service
docker compose down

# Restart service
docker compose restart

# View logs
docker compose logs -f

Container Management:

# List running containers
docker ps

# Check container health
docker ps --filter "label=autoheal=true"

# View container logs
docker logs <container-name> -f

# Access container shell
docker exec -it <container-name> /bin/bash

Database Access:

# MariaDB secondary
docker exec -it mariadb-secondary mysql -u root -p

# Firefly database
docker exec -it Firefly-DB mysql -u fireflyuser -p firefly

# Authentik PostgreSQL
docker exec -it authentik-postgres psql -U authentik -d authentik

Network Troubleshooting:

# List networks
docker network ls

# Inspect network
docker network inspect traefik

# Check connectivity
docker exec <container> ping <target>
docker exec <container> nc -zv <target> <port>

VPN Status:

# Tailscale status
docker exec tailscale tailscale status

# Gluetun status
curl http://192.168.12.3:38000/v1/openvpn/status

Service URLs

Service URL Authentication
Authentik https://id.3ddbrewery.com
https://id.fails.me		 Authentik SSO
Traefik Manager https://tm.3ddbrewery.com
https://tm.fails.me		 Authentik SSO
Node-RED https://node-het.3ddbrewery.com Username/Password
Firefly III https://f.3ddbrewery.com
http://192.168.12.3:6182		 Firefly Login
Gluetun Control http://192.168.12.3:38000 None

Port Reference

Port Service Purpose Access
80 Traefik HTTP (→ HTTPS) Public
443 Traefik HTTPS Public
8448 Traefik Matrix Federation Public
3306 MariaDB Database Public (use with caution)
1880 Node-RED Automation Platform Public (via Traefik)
6182 Firefly III Finance Manager Public
2376 Socket Proxy Docker API 192.168.12.3 only
38888 Gluetun HTTP Proxy Public
38388 Gluetun Shadowsocks Public
38000 Gluetun Control API Public

Container Statistics

Total Containers: 38

  • /volume1/docker services: 15 containers (9 services)
  • Matrix/Synapse stack: 23 containers (managed separately)

Services in /volume1/docker:

  1. authentik (4 containers)
  2. firefly (3 containers)
  3. mariadb (1 container)
  4. gluetun (1 container)
  5. tailscale (1 container)
  6. socket-proxy (1 container)
  7. node-red (1 container)
  8. utils (2 containers: autoheal, watchtower)
  9. traefik-mod (1 container)

Configured but Not Running:

  • authelia (2 containers: authelia, authelia_redis)

Active Databases: 5

  • mariadb-secondary (MariaDB latest)
  • Firefly-DB (MariaDB 11.3)
  • authentik-postgres (PostgreSQL 16)
  • matrix-postgres (PostgreSQL 17.7 - Matrix stack)
  • Redis instances: 3 active (authentik, firefly, matrix)

Docker Networks: 18

  • 1 external (traefik)
  • 9 service-specific (/volume1/docker services)
  • 8 Matrix/Synapse networks

Automation & Monitoring

Automatic Updates:

  • Watchtower: Checks hourly for image updates
  • Label: com.centurylinklabs.watchtower.enable=true
  • Notifications: Email to brian.w.maddox@gmail.com
  • Cleanup: Removes old images after update

Health Monitoring:

  • Autoheal: Checks every 5 seconds
  • Label: autoheal=true
  • Action: Automatic restart of unhealthy containers
  • Notifications: Webhook to ntfy (autoheal-IM topic)

Node-RED Custom Healthcheck:

  • Interval: 120 seconds
  • Failure Action: Sends notification to ntfy
  • Topic: hetzner_alerts
  • Priority: High

Backup & Recovery

Database Backups

MariaDB Secondary:

  • Replication: Live replica from 192.168.1.251 (disaster recovery)
  • Manual Backup: docker exec mariadb-secondary mysqldump ...
  • Backup Directory: /volume1/docker/backup

Firefly Database:

docker exec Firefly-DB mysqldump -u fireflyuser -p firefly > /volume1/docker/backup/firefly_$(date +%Y%m%d).sql

Authentik PostgreSQL:

docker exec authentik-postgres pg_dump -U authentik authentik > /volume1/docker/backup/authentik_$(date +%Y%m%d).sql

Matrix PostgreSQL:

  • Automated backups via matrix-postgres-backup container
  • Image: prodrigestivill/postgres-backup-local:18-alpine

Configuration Backups

Traefik Configuration:

  • Automatic backups before every change
  • Stored in /volume1/docker/traefik-mod/backups/
  • Retention: 30 days
  • Git version control (local repository)

Node-RED Flows:

cp /volume1/docker/node-red/flows.json /volume1/docker/backup/node-red-flows-$(date +%Y%m%d).json

Docker Compose Files:

tar -czf /volume1/docker/backup/docker-compose-files-$(date +%Y%m%d).tar.gz /volume1/docker/*/docker-compose.yml

Security Overview

Authentication

  • Authentik SSO: Centralized authentication for selected services
  • Traefik Middlewares: authentik@file for SSO-protected services
  • Node-RED: Username/password authentication
  • Firefly III: Application-level authentication

Network Security

  • Service Isolation: Each service stack on isolated Docker network
  • Database Isolation: PostgreSQL and Redis on internal networks only
  • Socket Proxy: Limited permissions, bound to private IP only
  • TLS/SSL: All web services use HTTPS via Traefik
  • Certificate Management: Automatic Let's Encrypt certificates

Container Security

  • Security Options: no-new-privileges enabled on most containers
  • Read-only Filesystems: Where applicable (e.g., Redis)
  • User Restrictions: Non-root users (1000:1000) where possible
  • Capability Dropping: Minimal capabilities granted
  • Resource Limits: CPU and memory limits prevent resource exhaustion

VPN Security

  • Tailscale: Encrypted mesh VPN for remote access
  • ProtonVPN: Privacy VPN layer via Gluetun
  • Route Advertisement: Only 192.168.12.3/32 advertised
  • Firewall: Docker iptables rules + host firewall

Troubleshooting

Service Won't Start

# Check logs
docker logs <container-name>

# Check dependencies
docker compose ps

# Check network
docker network inspect <network-name>

# Rebuild container
cd /volume1/docker/<service-name>
docker compose down
docker compose build
docker compose up -d

Database Connection Issues

# Check database is running
docker ps | grep -E "mariadb|postgres"

# Test connection
docker exec -it <container> ping <database-host>
docker exec -it <container> nc -zv <database-host> <port>

# Check database logs
docker logs mariadb-secondary
docker logs authentik-postgres

Traefik Routing Issues

# Check Traefik logs
docker logs matrix-traefik

# Verify service is on traefik network
docker network inspect traefik

# Check Traefik configuration
cat /matrix/traefik/config/dyno.yml

# Use traefik-mod web interface
# https://tm.3ddbrewery.com

Network Connectivity Issues

# Check Docker networks
docker network ls

# Inspect network
docker network inspect <network-name>

# Test connectivity
docker exec <container> ping <target>
docker exec <container> curl -I <url>

VPN Issues

# Tailscale status
docker exec tailscale tailscale status
docker exec tailscale tailscale ping <node>

# Gluetun status
curl http://192.168.12.3:38000/v1/openvpn/status
docker logs gluetun

Additional Resources

Documentation Files

  • maria.md (in traefik-mod): phpMyAdmin database setup instructions
  • MIGRATION_GUIDE.md (in traefik-mod/docs): Traefik config migration guide
  • database-schema.md (in traefik-mod/docs): Database schema documentation
  • IMPLEMENTATION_COMPLETE.md (in traefik-mod/docs): Testing checklist

External Documentation

Contact & Support

For issues or questions:

  • Check service-specific logs: docker logs <container-name>
  • Review documentation in this directory
  • Check autoheal logs for restart events: docker logs autoheal
  • Review watchtower logs for update issues: docker logs watchtower

Notification Channels:

Conclusion

This Hetzner server provides critical infrastructure for financial automation, identity management, and disaster recovery. The documentation in this directory provides comprehensive coverage of all services, configurations, and operational procedures.

Key Takeaways:

  • All services containerized with Docker Compose
  • Automatic health monitoring and updates
  • Database replication for disaster recovery
  • Custom applications for Traefik management and financial automation
  • Dual VPN setup for access and privacy
  • Comprehensive documentation for all components

When transferring to Silverbullet: All markdown files in this directory (/home/maddox/hetzner-docs/) are ready to be imported into the main server's Silverbullet documentation system for centralized documentation management.