Advanced Selfhosting with Nebula & Traefik

Posted on Views:

This article is a continuation of my previous article on Traefik, which you can find here. Previously, I went through some examples of how I used Traefik so this article would assume you have the basic knowledge of Traefik, if you are not comfortable with it, feel free to go back to my article or read the official documentation from traefik.

Motivation

I've been hosting some apps in my local home network using Raspberry Pi for a while now, and I've also hosted some apps on Google Cloud.

Whenever I want to access things on Raspberry Pi while not at home, I would have to use Nebula which is a tool for overlay network. I would have to connect to the VPN, and then access the applications.

This proves to be inconvenient, and not easy to use. One day, I started to play around with the idea of putting things together with Traefik, and it worked so well and drives me to write this article.

Why hosting things separately?

Some may wonder, why I would host things in both Raspberry Pi and GCP. Well, the main reason for this is cost efficiency and data privacy/security.

You can use whatever CPU and storage you want within your local network without incurring monthly costs, and you would have the added benefits of having data on premise.

Nebula setup

As mentioned previously, Nebula is a mesh network tool. Here are some instuctrtion on setting it up.

Prerequisites

  • A server with public IP (this is a MUST, doesn't need a domain), so you can use the free Google Cloud VPS.
  • A client that's connected to internet

Setup

  • Download and extract the nebula binary

    1
    wget -O - https://github.com/slackhq/nebula/releases/latest/download/nebula-linux-amd64.tar.gz | tar xzvf -

    This will download the tar.gz file and extract the content to the current working directory.

  • Download and extract the Nebula sample config file

    1
    wget -O https://github.com/slackhq/nebula/raw/master/examples/config.yml

  • You will get two binaries and a config file after the commands above:

    1
    2
    3
    ├── nebula <-- this is needed for both the client and server
    ├── nebula-cert <-- this is the program used to genreate both root CA and client
    ├── config.yaml <-- this is a Nebula sample config file

  • Create a Root CA authority using command:

    1
    nebula-cert ca 'Awesome Inc.'

  • Create a client using command:

    1
    2
    3
    nebula-cert sign -name 'client-name' \
        -ip "ip address" \
        -groups "group-name"

  • Setup the server (this is the machine with public IP address, it will become so called Light House in Nebula concept)

    • Download or transfer the nebula binary to the server with public IP.
    • Copy the config file and rename it to config-lighthouse.yaml
    • Follow what's in the config file and modify the file so that it's suitable for the lighthouse.
    • Create a config folder at /etc/nebula, copy ca.crt to the folder.
    • Create a host key and certificate using 
      1
      2
      3
      ./nebula-cert sign \
                  -name "lighthouse1" \
                  -ip "192.168.100.1/24"
    • The command above will create lighthouse1.crt and lighthouse1.key, copy both of them to /etc/nebula, name them as host.crt and host.key
    • Start the service using 
      1
      ./nebula -conifg /etc/nebula/config.yaml

  • Setup the client

    • Create a host key and certificate using 
      1
      2
      3
      ./nebula-cert sign \
                  -name "device1" \
                  -ip "192.168.100.101/24"
    • The command above will create device1.crt and device1.key, copy both of them to /etc/nebula, name them as host.crt and host.key
    • Use the same command used in setup server to connect to the client: 
      1
      ./nebula -conifg /etc/nebula/config.yaml

  • Tun setup
    For both client and server-side, you will need to enable tun.
    As described by Wikipedia:

    In computer networking, TUN and TAP are kernel virtual network devices. Being network devices supported entirely in software, they differ from ordinary network devices which are backed by physical network adapters.
    In short, Tun is like a virtual network card for Linux.
    To enable tun in Nebula, you need to set the tun.disable to false which is the default. You can check the Nebula documentation for more information on this.

Docker setup

The setup on Docker will also be two parts, the client and server.

Client (I'm using Raspberry Pi running ArchLinux)

Here is a updated Docker Compose file based on the one used in previous article:

Client Traefik Docker Compose file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
version: '3'

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.10
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun

    # Enables the web UI and tells Traefik to listen to docker
    command:
      - "--log.level=DEBUG"
      - "--api.dashboard=true"
      - "--accesslog"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12"
      - "--providers.docker=true"
      - "--providers.docker.exposedByDefault=false"
      - "--providers.docker.network=traefik-proxy"
    networks:
      - "traefik-proxy"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`example.domain.com`)"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth"
      # Check
      - "traefik.http.middlewares.auth.basicauth.users=YOUR USERNAME AND PASSWORD"
      - "traefik.port=8080"
      - "traefik.ping=true"
    ports:
      # The HTTP port
      - "80:80"
      - "443:443"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
#Docker Networks
networks:
  traefik-proxy:
    external: true

The part to note is the following:

1
2
3
4
cap_add:
  - NET_ADMIN
devices:
  - /dev/net/tun
  • cap_add is used to specify additional container capabilities for Docker container. It is part of Linux, here, I specify the CAP_NET_ADMIN capability which is used to specify rights related to network
  • devices is used to specify the device the Docker container will have access to. Here, we are specifying /dev/net/tun, this is the dev/net/tun

Server (GCP server that's setup as Nebula lighthouse)

The Traefik config for the server side will be similar to what the client looks like, we will reuse the Docker file for client with some modification.

Server Traefik Docker Compose file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
version: '3'

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.10
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun

    # Enables the web UI and tells Traefik to listen to docker
    command:
      - "--log.level=DEBUG"
      - "--api.dashboard=true"
      - "--accesslog"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12"
      - "--providers.docker=true"
      - "--providers.docker.exposedByDefault=false"
      - "--providers.docker.network=traefik-proxy"
      - "--providers.file.directory=./confs.d"
    networks:
      - "traefik-proxy"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`example.domain.com`)"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth"
      # Check
      - "traefik.http.middlewares.auth.basicauth.users=YOUR USERNAME AND PASSWORD"
      - "traefik.port=8080"
      - "traefik.ping=true"
    ports:
      # The HTTP port
      - "80:80"
      - "443:443"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./confs.d:/confs.d:ro
#Docker Networks
networks:
  traefik-proxy:
    external: true

I've made several modifications.

  • Added the --providers.file.directory=./confs.d part to the command, this is so that we can put the configuration related to raspberry pi in a separate file.
  • Added one more mountpoint to the Docker container as in ./confs.d:/confs.d:ro

Now, we will go through what's in the confs.d folder. At the moment, there's only one file in it, I named it reverse-to-pi.toml.

reverse_to_pi_toml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[http]
    [http.routers]
       # Define a connection between requests and services
       [http.routers.to-pi]
        rule = "Host(`somedomain.example.com`)"
        tls = true
        # If the rule matches, applies the middleware
        middlewares = ["test-user"]
        # If the rule matches, forward to the pi service (declared below)
        service = "pi"

    [http.middlewares]
      # Define an authentication mechanism
      [http.middlewares.test-user.basicAuth]
        users = ["USERNAME AND PASSWORD HASH"]

    [http.services]
      # Define how to reach an existing service on our infrastructure
      [http.services.pi.loadBalancer]
           passHostHeader = true
           serversTransport = "transport"
      [[http.services.pi.loadBalancer.servers]]
           url = "your_nebula_client_ip"
    [http.serversTransports.transport]
      disableHTTP2 = true

The above configuration would essentially treat the Nebula client as a loadbalancer back-end, and the current Traefik instance would essentially act as a reverse proxy to raspberry pi when the specific domain matches. Because the loadbalancer at this level is set to forward host header, the traffic would be forwarded to that instance and that Traefik instance will route the traffic to the container that matches the given domain.

Bonus: Cloudflare CDN

Cloudflare

As a bonus, one can also setup the CloudFlare to act as a CDN and DNS server to provide some additional security.
Here are some ideas on how to utilize Cloudflare in this case:

  • The DNS infrastructure (i.e. domain name) can be managed completely by Terraform, so that you don't have to manually manage the domains of all self-hosted app.
  • Cloudflare can provide SSL certificates, so you don't need to worry about managing those stuff yourself.
  • Cloudflare can provide some basic WAF, so unwanted traffic can be blocked.

Future Improvements: GitHub Actions, Forward Auth and more?

While this is a complete solution, and something I've been running for months without any issue, it has room for improvements.

GitHub Actions

As an experimenting exercise, GitHub Action could be used to manage the infrastructure -- whenever there's a new self-hosted app added, the infrastructure would be automatically populated and deployed through Terraform.

Forward Auth

Currently, the authentication is only HTTP basic auth, which is not secure at all. The idea to improve this would be implementing forward auth so the infrastructure is more secure. Some candidates to choose from, include:

  • Authelia, Apache-2.0, Go, 18.1k stars as of writing
  • Authentik, MIT, Python, 5.3k stars as of writing
  • KeyCloak, Apache-2.0, Java, 18.4k stars as of writing.
    I'm leaning towards Authelia at the moment, but haven't got spare time to try it out.

Conclusion

By combining the power of Nebula and Traefik, we are essentially opening another possibility of self hosting. Doing it this way would provide several benefits:

  • Cost Saving: The public facing server (or the Nebula lighthouse) can essentially be a free tier instance that does nothing but routing traffic between the two Traefik instance.
  • Data Retention: You don't need to worry about your data.
  • Secure & Fast: By using Cloudflare, the whole infrastructure has relatively good security as well as speed.

References

https://docs.docker.com/compose/compose-file/05-services/#cap_add
https://doc.traefik.io/traefik/middlewares/http/forwardauth/
https://goauthentik.io/docs/providers/proxy/server_traefik
https://www.authelia.com/integration/proxies/traefik/
https://en.wikipedia.org/wiki/TUN/TAP