Port Forwarding with HAProxy

I recently acquired a beefy bare-metal server and wanted to run a bunch of services within VMs based on KVM managed by libvirt. Only ports 80 and 443 for these VMs would be exposed and the rest of the ports (say SSH) visible only from the internal network.

Initially, I just thought of going with Nginx because I was familiar with it. But I had heard a lot about HAProxy and wanted to give it a shot.

I had libvirt assign a static IPv4 address for each VM and then used that to correctly forward the terminated TLS connection at HAProxy.

Here's the libvirt network configuration:

<network>
  <name>default</name>
  <uuid>{uuid}</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:34:12:10'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:a9:2c:0b' name='vm1' ip='192.168.122.2'/>
      <host mac='52:54:00:a9:2c:0c' name='vm2' ip='192.168.122.3'/>
      ...
    </dhcp>
  </ip>
</network>

I then created VMs with the above host mac addresses and the corresponding IP address got attached.

The HAProxy config was as below:

frontend http
     bind :::80 v4v6
     bind :::443 v4v6 ssl crt-list /home/fedora/ssl-list.txt
     mode http
     option forwardfor
     option http-server-close
     redirect scheme https if !{ ssl_fc }

     use_backend vm1_example if { req.ssl_sni -i vm1.example.com }
     use_backend vm1_example if { hdr(Host) -i vm1.example.com }

     use_backend vm2_example if { req.ssl_sni -i vm2.example.com www.vm2.example.com }
     use_backend vm2_example if { hdr(Host) -i vm2.example.com www.vm2.example.com }

backend vm1_example
     server vm1 192.168.122.2:80 maxconn 32

backend vm2_example
     server vm2 192.168.122.3:80 maxconn 32

HAProxy forwards the HTTP connections either based on the SNI (if supported by your SSL certificate provider) or the HTTP Header.

The ssl-list.txt file held the location of the SSL certificates (thanks LetsEncrypt):

/etc/letsencrypt/live/vm1.example.com/full_cert.pem vm1.example.com
/etc/letsencrypt/live/vm2.example.com/full_cert.pem vm2.example.com www.vm2.example.com

Later on, I ran an SMTP server within it's own VM and port-forwarded to it as well. The HAProxy configuration file had the following lines appended to it:

frontend smtpd
  bind :::25 v4v6
  mode tcp
  no option http-server-close
  timeout client 1m
  log global
  option tcplog
  default_backend smtp

backend smtp
  mode tcp
  no option http-server-close
  log global
  option tcplog
  timeout server 1m
  timeout connect 5s
  server postfix 192.168.122.4:25 send-proxy