UI Drafter Blog

Jails Networking

November 2, 2020
Eric Fortis

UI Drafter runs on two servers, each one has three jails. The jails for the databases and application servers communicate privately (dotted lines), while the reverse proxy ones listen on the internet.

Server A Server B Rev. Proxy App Server Database Rev. Proxy App Server Database

Each server has:

xnic xnic Server A Server B inic inic xbridge ibridge ibridge nginx_j node_j pg_j .2.20 ngx_b .2.30 node_b .2.40 pg_b .3.20 ngx_node_a .3.30 ngx_node_b .4.30 node_pg_a .4.40 node_pg_b inode_b ipg_b inode_b ipg_b

VNIC Labels

Also, each server has four encrypted tunnels (spiped). The arrows point to the pipe's server-end.

spiped tunnels .56.31 .56.41 .56.30 .56.40 :5432 :5433 :5434 :5435 :5432 :5433 :5434 :5435

Load Balancing

In the DNS provider, I added an "A" record for each server. Therefore, they get round-robin load balanced free of charge.

Cloudflare Multiple A Records for Load Balancing with DNS

Cost and Hardware

Each server is rented for about $1,400/year to Hivelocity®. Both are SuperMicro®:

  • 6 Cores 3.3/4.5GHz E-2136
  • 32GB ECC DDR4
  • 2 × 480GB SSD
  • 20TB/mo at 1Gbps. 5 IPs. DDoS FENS
  • IPMI over VPN


The configuration files (*.conf) are exactly the same in both servers. For that, they read server-specific settings from other files. For example, Server A's /etc/inic would have

The hardware and company-specific parts will be highlighted in green.


This rc.conf specifies the bridges and epairs to create at the hosts' boot time. Also, how to connect those epair's VNICs to the bridges.

About rc.conf's ifconfig syntax

Each of the next two lines rename an interface (bridge0). The first one shows how it's typed it in the shell, which is non-persistent across reboots. While the second one is for the rc.conf (persistent).

ifconfig bridge0 name xbridge

# Rename and configure the NICs 
ifconfig_xnic="`/bin/cat /etc/xnic`"
ifconfig_inic="`/bin/cat /etc/inic` vlanhwtag 2222"

defaultrouter=`/bin/cat /etc/gateway`

# Create the virtual devices. e.g., ifconfig bridge0 create
# Rename the bridges and VNICs ifconfig_bridge0_name=xbridge ifconfig_bridge1_name=ibridge ifconfig_epair0a_name=ngx_a ifconfig_epair0b_name=ngx_b ifconfig_epair1a_name=node_a ifconfig_epair1b_name=node_b ifconfig_epair2a_name=pg_a ifconfig_epair2b_name=pg_b ifconfig_epair3a_name=ngx_node_a ifconfig_epair3b_name=ngx_node_b ifconfig_epair4a_name=node_pg_a ifconfig_epair4b_name=node_pg_b ifconfig_epair5a_name=inode_a ifconfig_epair5b_name=inode_b ifconfig_epair6a_name=ipg_a ifconfig_epair6b_name=ipg_b # Enable the VNICs that we don't assign IPs to ifconfig_ngx_a=up ifconfig_node_a=up ifconfig_pg_a=up ifconfig_ngx_node_a=up ifconfig_node_pg_a=up ifconfig_inode_a=up ifconfig_ipg_a=up # Connect the NICs to the bridges. e.g., ifconfig ibridge addm inic ifconfig_ibridge="
addm inic
addm inode_a
addm ipg_a
addm ngx_a
addm node_a
addm pg_a
# Services jail_enable=YES jail_reverse_stop=YES gateway_enable=YES pf_enable=YES


The jails boot up in the order they appear in this file. Also, it specifies which VNICs the host will hand-over to the jails.

As PostgreSQL requires System V's shared memory, sysvshm provides and namespaces it to the database jail.

path = "/jails/$name";
host.hostname = "$name.uidrafter.lan";

devfs_ruleset = 4;

exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";

pg_j {
  sysvshm = "new";
  vnet.interface = pg_b, node_pg_b, ipg_b;

node_j {
  vnet.interface = node_b, ngx_node_b, inode_b, node_pg_a;

nginx_j {
  vnet.interface = ngx_b, ngx_node_a;

Jails rc.conf's

This section has the jail-boot time configurations. By the time these files run, the host has already delegated the network interfaces to the jails. Therefore, we use these rc.conf files to configure the VNICs, in addition to the gateway, tunnels, and services.






ifconfig_inode_b=`/bin/cat /etc/ip_inode_b`/24


# Tunnel to peer database

spiped_pipe_N2P_source="[`/bin/cat /etc/ip_inode_b`]:5432"
spiped_pipe_N2P_target="[`/bin/cat /etc/ip_peer_ipg_b`]:5433"



ifconfig_ipg_b=`/bin/cat /etc/ip_ipg_b`/24


# Tunnels
spiped_pipes="N2P PGS PGC"

spiped_pipe_N2P_source="[`/bin/cat /etc/ip_ipg_b`]:5433"
spiped_pipe_N2P_target="[`/bin/cat /etc/ip_ipg_b`]:5432"

spiped_pipe_PGS_source="[`/bin/cat /etc/ip_ipg_b`]:5434"
spiped_pipe_PGS_target="[`/bin/cat /etc/ip_ipg_b`]:5432"

spiped_pipe_PGC_source="[`/bin/cat /etc/ip_ipg_b`]:5435"
spiped_pipe_PGC_target="[`/bin/cat /etc/ip_peer_ipg_b`]:5434"


Firewall (pf)

This firewall configuration denies all traffic by default, and then allows what's needed. It assumes xnic is on a public IP, if not, remove its subnet from the <martians> table.

Allowed Incoming Traffic

Only the Nginx jail is open to the internet. SSHing to the host or jails is only possible from IPs listed in the <xpeers> table.

Rate Limits

If an IP connects 100 times within 10 seconds it'll be blocked. Not only from making new connections, but also from using established ones, as flush global terminates them.

Effectively, that IP gets added to the <ratelimit> table. Therefore, you can use a cron to reallow those IPs. For example, to remove IPs that are at least three minutes old from that table:

/sbin/pfctl -t ratelimit -T expire 180

How to exclude a company from rate-limits? Create a table with their IPs, and add a pass in quick rule before the block in quick one; think of quick as an early return. For instance, if Cloudflare® is intercepting your traffic, populate a <bypass> table with their IPs and:

pass in quick on xnic proto tcp from <bypass> to port 443
block in quick …

Allowed Outgoing Traffic

The Node jail can connect to Stripe® API and Fastmail®. As they need DNS services, Cloudflare® and Google® resolvers are allowed too.

NTP is allowed via inic to private NTP servers.

How to deploy to these servers? Copy over to them with rsync. The orchestration server IP must be in the <xpeers> table.

How to generate TLS certificates? Like deploying, see my previous post: Isolated Let's Encrypt TLS Certificate Creation.

How to extract backups? rsync from the backup/orchestration server too.

The general answer to those questions is: initiate the connection from the orchestration server.


xbr_net = "10.0.2/24"
nginx_j = ""
node_j  = ""
pg_j    = ""

port_nginx = "{ 443 80 }"
port_email = "465"

resolvers = "{ }"
priv_ntp = "{ }"

table <ratelimit>
table <blocklist> file "/etc/ips_blocklist"  # DShield Daily
table <martians>  file "/etc/ips_martians"   # Bogon →
table <payments>  file "/etc/ips_stripe"     # Stripe API IPs →
table <email>     file "/etc/ips_fastmail"   # 66.111.4/24
table <xpeers>    file "/etc/ips_xnic_peers"
table <ipg_j>     file "/etc/ip_ipg_b"
table <inode_j>   file "/etc/ip_inode_b"
table <pgpeer>    file "/etc/ip_peer_ipg_b"
table <nodepeer>  file "/etc/ip_peer_inode_b"

# Normalization. Prevents IP Fragmentation Attacks and Inspection Evasion
scrub in all fragment reassemble no-df

# Translation/Redirects
nat on xnic from $xbr_net -> (xnic:0)
rdr on xnic proto tcp from any      to port $port_nginx -> $nginx_j
rdr on xnic proto tcp from <xpeers> to port 2220 -> $nginx_j port 22
rdr on xnic proto tcp from <xpeers> to port 2230 -> $node_j  port 22
rdr on xnic proto tcp from <xpeers> to port 2240 -> $pg_j    port 22

# Blockers
antispoof quick for xnic
block in quick on xnic \
      from { <blocklist> <ratelimit> <martians> no-route urpf-failed }
block all

# Nginx Incoming Traffic
pass in quick on xnic proto tcp from any \
     to port $port_nginx keep state \
     (max-src-conn-rate 100/10, overload <ratelimit> flush global)
pass out quick on xbridge proto tcp to $nginx_j port $port_nginx

# Tunnels
pass in  quick on ipg_a   proto tcp from <ipg_j>    to <pgpeer> port 5434
pass in  quick on inic    proto tcp from <pgpeer>   to <ipg_j>  port 5434
pass in  quick on inode_a proto tcp from <inode_j>  to <pgpeer> port 5433
pass in  quick on inic    proto tcp from <nodepeer> to <ipg_j>  port 5433
pass out quick on ibridge proto tcp from any        to any      port { 5434 5433 }

# Node.js Outgoing Traffic
pass in  on xbridge proto udp from $node_j to $resolvers port 53
pass in  on xbridge proto tcp from $node_j to <payments> port 443
pass in  on xbridge proto tcp from $node_j to <email>    port $port_email
pass out on xnic    proto udp from any     to $resolvers port 53
pass out on xnic    proto tcp from any     to <payments> port 443
pass out on xnic    proto tcp from any     to <email>    port $port_email

pass in  on xnic    proto tcp from <xpeers> to any      port 22
pass out on xbridge proto tcp from <xpeers> to $xbr_net port 22

pass out on inic proto udp to $priv_ntp port 123

# Uncomment for updating FreeBSD and packages
# pass in  on xbridge from $xbr_net
# pass out on xnic