OpenFlow setup behind a reverse proxy and ssl offloader

Guides
1

Hi all, I’m new to OpenRPA/OpenFlow and have just managed to get OpenFlow running in my infrastructure following the documentation and using the docker-compose files provided in the docker repository. It was a journey that spanned trough a few days so I think it’s worth sharing back to the community. To serve as help for others and to validate the process is valid. My setup runs in in VM on a Proxmox hypervisor in Hetzner (a German hosting provider).

I broke down the process to:

  • Set up the DNS
    • An A record, like: openflow.yourdomain.com pointing to the public IP address
    • A CNAME record, like: *.openflow.yourdomain.com to point to the A name above
  • A Let’s Encrypt SSL Certificate for that domain - also with wildcard
  • A Reverse Proxy (haproxy) that listens in 80 and 443 of the public IP address and redirects every domain name request to the port 80 of the traefik container
  • Finally the docker-compose.yml has to be configured to work using ssl offloader and the domain name.

I hope this is usefull for others.

DNS Part

In my Hosting Provider (Hetzner) I created the A and CNAME records

Type Name Value TTL
A openflow.<yourdomain.com> <your.ip.add.ress> 30
CNAME *.openflow openflow 30

SSL part (haproxy)

We use haproxy as the load balancer/reverse proxy/frontend for other web applications and services in our infrastructure, so we added

New Wildcard Certificate

Requested a new certificate for the new domain name using Let’s Encrypt certbot.

Let’s Encrypt certificates can work with wildcard domains using the DNS challenge. For this I found this article in the Hetzner comunity DNS Validated Let's Encrypt Certificates that provides shell script to setup the DNS callenge usinh the provider API.

Install Dependencies

apt update
apt install curl jq certbot

Download Scripts

curl https://raw.githubusercontent.com/dschoeffm/hetzner-dns-certbot/master/certbot-hetzner-auth.sh > /usr/local/bin/certbot-hetzner-auth.sh
curl https://raw.githubusercontent.com/dschoeffm/hetzner-dns-certbot/master/certbot-hetzner-cleanup.sh > /usr/local/bin/certbot-hetzner-cleanup.sh
chmod +x /usr/local/bin/certbot-hetzner-auth.sh
chmod +x /usr/local/bin/certbot-hetzner-cleanup.sh

Acquire API Token

Login to Hetzner DNS Console and create a token.

Save the token to /etc/hetzner-dns-token:

echo <token> /etc/hetzner-dns-token

Get the Certificate

certbot certonly --manual --preferred-challenges=dns --manual-auth-hook /usr/local/bin/certbot-hetzner-auth.sh --manual-cleanup-hook /usr/local/bin/certbot-hetzner-cleanup.sh -d openflow.yourdomain.com -d *.openflow.yourdomain.com

Concatenate certificate files for haproxy

sudo cat /etc/letsencrypt/live/openflow.yourdomain.com/fullchain.pem /etc/letsencrypt/live/openflow.yourdomain.com/privkey.pem > /etc/ssl/openflow.pem

Reverse Proxy (haproxy.conf)

Add this to the HAPROXY config file

# /etc/haproxy/haproxy.cfg
frontend myfrontend
        bind :80
        bind *:443 ssl crt /etc/ssl/openflow.pem 
        http-request redirect scheme https unless { ssl_fc }        

        # letsencryp validation path for cert request
        acl ACL_letsencrypt path_beg /.well-known/acme-challenge/
        use_backend be_letsencrypt if ACL_letsencrypt

        # ACL for openflow.<yourdomain>
        acl ACL_openflow hdr_end(host) -i openflow.yourdomain.com
        use_backend be_openflow if { ssl_fc_sni openflow.yourdomain.com }
        use_backend be_openflow if ACL_openflow

backend be_letsencrypt
        server certbot 127.0.0.1:8899

backend be_openflow
        # New implementation of OpenFlow
        server server2 192.168.115.130:80

OpenFlow Server

To run OpenFlow I use a VM with a minimal Debian 11 installation on a Proxmox hypervisor.

Configure the CPU in Proxmox

In the proxmox console, configure the CPU Type to: host

This avoids the error WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that while starting mongodb

Configure max_map_count

In the VM running docker (host)

sysctl -w vm.max_map_count=262144

This avoids the warning vm.max_map_count is too low while starting mongodb

Create a folder

mkdir /opt/openflow

Download the docker repo

git clone https://github.com/open-rpa/docker.git
cd docker

Edit docker-compose.yml

After some troubleshooting I ended up using the following docker-compose.yaml file:

version: "3.3"
services:
  mongodb:
    image: mongo
    restart: always
    command: "--bind_ip_all --replSet rs0"
    environment:
      - MONGO_REPLICA_SET_NAME=rs0
    volumes:
      - mongodb_data:/data/db
  mongosetup:
    image: mongo
    depends_on:
      - mongodb
    restart: "no"
    command: >
      mongosh --host mongodb:27017 --eval
      '
      db = (new Mongo("mongodb:27017")).getDB("openflow");
      config = {
      "_id" : "rs0",
      "members" : [
        {
          "_id" : 0,
          "host" : "mongodb:27017"
        }
      ]
      };
      rs.initiate(config);
      '
  traefik:
    image: traefik
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    restart: always
    volumes:
      - "//var/run/docker.sock:/var/run/docker.sock:ro"
  rabbitmq:
    image: rabbitmq
    restart: always
  api:
    labels:
      - traefik.enable=true
      - traefik.frontend.passHostHeader=true
      - traefik.http.routers.http-router.entrypoints=web
      - traefik.http.routers.http-router.rule=Host(`openflow.yourdomain.com`)
      - traefik.http.routers.http-router.service=http-service
      - traefik.http.services.http-service.loadbalancer.server.port=3000
      - traefik.http.routers.grpc-router.rule=Host(`grpc.openflow.yourdomain.com`)
      - traefik.http.routers.grpc-router.service=grpc-service
      - traefik.http.routers.grpc-router.entrypoints=web
      - traefik.http.services.grpc-service.loadbalancer.server.port=50051
      - traefik.http.services.grpc-service.loadbalancer.server.scheme=h2c
    image: openiap/openflow
    ports:
      - "5858:5858"
    deploy:
      replicas: 1
    pull_policy: always
    restart: always
    volumes:
      - "//var/run/docker.sock:/var/run/docker.sock"
    depends_on:
      - rabbitmq
      - mongodb
    environment:
      - auto_create_users=true
      - auto_create_domains=
      - websocket_package_size=25000
      - websocket_max_package_count=1048576
      - protocol=https
      - port=3000
      - domain=openflow.yourdomain.com
      - log_with_colors=false
      # uncomment below 2 lines, if you have set replicas above 1
      # - enable_openflow_amqp=true
      # - amqp_prefetch=25
      # uncomment to add agents to the same docker compose project ( will breake running docker compose up -d if any agents running )
      # - agent_docker_use_project=true
      - agent_oidc_userinfo_endpoint=http://api:3000/oidc/me
      - agent_oidc_issuer=https://openflow.yourdomain.com/oidc
      - agent_oidc_authorization_endpoint=https://openflow.yourdomain.com/oidc/auth
      - agent_oidc_token_endpoint=http://api:3000/oidc/token
      - amqp_url=amqp://guest:guest@rabbitmq
      - mongodb_url=mongodb://mongodb:27017/?replicaSet=rs0
      - mongodb_db=openflow
      - aes_secret=O1itlrmA47WzxPj95YHD2sZs7IchYaQI25mQ
volumes:
  mongodb_data:
    driver: local

Start Script

/opt/openflow/docker/normal-up.sh

Stop Script

/opt/openflow/docker/normal-down.sh

Update

/opt/openflow/docker/normal-pull.sh

Check Docker Status

docker ps
2 Likes
2

Nice and detailed guide. Thank you :hugs::+1:

traefik is an teverse proxy, so why are you add one more?
Also you can let traefic handle lets encrypt too using this file

3

Hi @Allan_Zimmermann,
I decided to go with what I know and what I already use to frontend other applications in my development environment. Also, when deploying applications to customers I most often have to deploy behind hardware Load Balancers. Having a separate proxy VM helps resembling those to develop in a similar environments.

Thanks for your feedback!