Local hosting behind a VPS
I want to replicate the architecture used by autistici/inventati, which hosts services on machines that are not directly accessible from the internet, and external traffic is routed to them from a VPS using a VPN.
What I want to achieve:
- keep all data on a computer running in my home
- run multiple apps on one or more home servers
- keep apps isolated, they must not be able to communicate via the VPN
- expose more than just web apps, e.g. a mail server
- maybe, one day, have multiple “public entrypoints” for some of the services
- maybe, one day, have multiple replicas of some of the services, on multiple home servers
The architecture includes:
- a wireguard VPN container on the VPS
- a caddy container on the VPS, that shares the network with the wireguard container
- a wireguard VPN container on the home server, that connects to the wireguard container on the VPS
- a web app running on the home server inside a container, sharing the network with the wireguard container on the home server
This is the example I will use:
- the web app will be hosted on
website.example.com - the VPS has the IP address
1.2.3.4 - the VPN will use the
10.0.0.0/24network - the VPN server running on the VPS will have IP
10.0.0.1 - the VPN server will be reachable via the
51820port - the web app running on the home server will have IP
10.0.0.2
Wireguard basics
The private keys for the two parties have to be generated with:
1wg genkey
The public keys can then be generated with:
1echo PRIVATE_KEY | wg pubkey
VPS configuration
The wireguard container on the VPS looks like:
1services:
2 vpn:
3 image: linuxserver/wireguard
4 cap_add:
5 - NET_ADMIN
6 - SYS_MODULE
7 restart: unless-stopped
8 sysctls:
9 net.ipv4.ip_forward: 1
10 environment:
11 PUID: 1000
12 PGID: 1000
13 ports:
14 - "80:80"
15 - "443:443"
16 - "51820:51820/udp"
17 volumes:
18 - ./wg-config/wg0.conf:/config/wg_confs/wg0.conf:ro
The wg0.conf file needs the following contents:
1[Interface]
2Address = 10.0.0.1/24
3PrivateKey = SERVER_PRIVATE_KEY
4ListenPort = 51820
5
6PostUp = iptables -A FORWARD -i wg0 -j ACCEPT
7PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
8PostDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
The reverse proxy on the VPS looks like this:
1services:
2 reverse-proxy:
3 image: caddy:latest
4 restart: unless-stopped
5 network_mode: "service:vpn"
6 volumes:
7 - ./caddy/Caddyfile:/etc/caddy/Caddyfile
8 - ./caddy/data:/data
9 - ./caddy/config:/config
The Caddyfile configuration should contain an entry for each app that you want to expose:
1website.example.com {
2 reverse_proxy 10.0.0.2:80
3}
Home server configuration
On the home server, the VPN container looks like this:
1 services:
2 vpn:
3 image: linuxserver/wireguard
4 cap_add:
5 - NET_ADMIN
6 restart: unless-stopped
7 sysctls:
8 net.ipv4.conf.all.src_valid_mark: 1
9 volumes:
10 - ./wg-config/wg0.conf:/config/wg_confs/wg0.conf
11 healthcheck:
12 test: ["CMD-SHELL", "ping -c 1 -w 5 10.0.0.1 || exit 1"]
13 interval: 30s
The health check will report the VPN container as healthy as long as it’s connected to the VPN.
The wg0.conf file has the following contents:
1[Interface]
2Address = 10.0.0.2/24
3PrivateKey = APP_PRIVATE_KEY
4
5[Peer]
6PublicKey = SERVER_PUBLIC_KEY
7Endpoint = 1.2.3.4:51820
8AllowedIPs = 0.0.0.0/0
9PersistentKeepalive = 25
You will need to give a different IP address to each server that you add.
The container for the app running on the home server needs to be configured with network_mode: "service:vpn".
Adding apps to the VPS
Each app must be added to the configuration of wireguard container running on the VPS:
1[Peer]
2PublicKey = APP_PUBLIC_KEY
3AllowedIPs = 10.0.0.2/32
To add a new peer to the VPS without restarting the wireguard container you can use:
1docker compose exec vpn wg set wg0 peer APP_PUBLIC_KEY allowed-ips 10.0.0.2/32
Bonus things
Maintenance mode for caddy
Use this post for reference.
Exposing more ports than just HTTP/S
If your app needs to receive traffic on ports other than 80 and 443, you need to:
- expose the port on the wireguard container running on the VPS
- route traffic directly from the wireguard container to the app container by updating the wireguard configuration
1PostUp = iptables -t nat -A PREROUTING -p tcp --dport 22 -j DNAT --to-destination 10.0.0.2:22
2PostDown = iptables -t nat -D PREROUTING -p tcp --dport 22 -j DNAT --to-destination 10.0.0.2:22
Showing the right IP in nginx
Caddy adds an X-Forwarded-For header to all requests,
but if you’re using nginx you will see that all requests come from 10.0.0.1.
To tell nginx to trust the IP sent by the VPN server:
1set_real_ip_from 10.0.0.1;
2real_ip_header X-Forwarded-For;
3real_ip_recursive on;
This can be mounted directly on the official nginx container:
1services:
2 server:
3 image: nginx:stable-alpine
4 volumes:
5 - ./nginx.conf:/etc/nginx/conf.d/proxy.conf:ro
Stuff I want to find out
Can I simplify this configuration to avoid repetitions in the VPS and home server?
How safe is this?