IPSec VPN Between PFsense and Raspberry Pi for my Carputer

Introduction

So having recently got my Carputer up and running I decided I wanted to connect it to my LAN at home and for my business. I’ve already got a pfsense installation for my border router so can easily add a new IPSec VPN node connecting in to it – with just a couple of quick changes to my existing setup.

Things I wanted to achieve:

1. Change my car’s network over to the 10.2.0.0/24 network (10.0.0.0/16 is for my house network, 10.1.0.0/16 is for AWS services and 10.3.0.0/16 is where I allocate my /30s from). I’m not sure my car is going to need a /24, but I’m not exactly short on address space.
2. Install a dynamic DNS client on the Pi to keep the IP updated whenever the vehicle is online
3. Setup VPN connection from my car’s router to my main network border router – any 10.0.0.0/16 should flow here

Network Change

For some reason I setup my car as a 192.168.1.0/24 block. This made very little sense at the time and even less now – so time to change it. The first step is to change the address range that udhcpd offers. In my case all I need to change are the opt router and start/end addresses. This gives me a /etc/udhcpd.conf that looks like this:

start           10.2.0.100
end             10.2.0.254
interface       wlan0
remaining       yes
opt dns 8.8.8.8 4.2.2.2
opt subnet 255.255.255.0
opt router 10.2.0.1
opt lease 864000
option  domain  guytp.org

Next it was to modify the IP of the Pi itself in /etc/network/interfaces to match the gateway above:

auto lo

iface lo inet loopback
iface eth0 inet dhcp

iface wlan0 inet static
  address 10.2.0.1
  netmask 255.255.255.0
up iptables-restore < /etc/iptables.ipv4.nat

allow hotplug wwan0
iface wwan0 inet dhcp

Finally I just need to change the static IP on my OSMC connection to be 10.2.0.2. If your version of OSMC is newer you may be able to actually do this through the user interface. If that is the case then note that any changes you made manually will be wiped out. First check the My OSMC menu for a network option and if it isn’t there you need to manually edit files. OSMC uses connman to configure it’s IPs you need to select “Exit” from the power off menu in OSMC and keep pressing down escape. Eventually you’ll be at a command prompt and login as osmc for both the username and password. Then type in:

sudo bash
cd /var/lib/connman
ls

There will be a folder starting “wifi” and in there a file called settings. Edit this file and change the router and IP address and then you’re done. My file ended up looking like this:

[wifi_XXXXXXXXXXXX_XXXXXXXXXXXXXX_managed_psk]
Name=XXXXXXX
SSID=XXXXXXXXXXXXXX
Frequency=2437
Favorite=true
AutoConnect=true
Modified=2015-04-26T00:18:35.264754Z
Passphrase=XXXXXXXXXX
IPv4.method=manual
IPv6.method=auto
IPv6.privacy=disabled
IPv4.netmask_prefixlen=16
IPv4.local_address=10.2.0.2
IPv4.gateway=10.2.0.1
Nameservers=8.8.8.8;4.2.2.2;



I thought I was done but soon found my web traffic blocked.  If you installed squid don’t forget to edit /etc/squd3/squid.conf to change your local IP range as well.

 
<h1>Dynamic DNS Client</h1>

<p>I use no-ip for my dynamic DNS and then just have a CNAME from my guytp.org domain to the appropriate no-ip hosts with a 15 second TTL.  So for me I wanted to download the no-ip source for linux.  You need to download it, extract it and then compile/install by following the steps below.  When do you "make install" you’ll be guided through your config settings.</p>


guytp@GTP-RRSC-RTB-01:~$ wget http://www.no-ip.com/client/linux/noip-duc-linux.tar.gz

… snipped …

100%[======================================>] 134,188      174K/s   in 0.8s    

2015-04-26 21:00:17 (174 KB/s) - `noip-duc-linux.tar.gz' saved [134188/134188]

guytp@GTP-RRSC-RTB-01:~$ tar -xzf noip-duc-linux.tar.gz 
guytp@GTP-RRSC-RTB-01:~$ cd noip-2.1.9-1/
guytp@GTP-RRSC-RTB-01:~/noip-2.1.9-1$ make
gcc -Wall -g -Dlinux -DPREFIX=\"/usr/local\" noip2.c -o noip2

… snipped …

guytp@GTP-RRSC-RTB-01:~/noip-2.1.9-1$ sudo make install

… snipped …

Auto configuration for Linux client of no-ip.com.

Multiple network devices have been detected.

Please select the Internet interface from this list.

By typing the number associated with it.
0	wlan0
1	wwan0
1
Please enter the login/email string for no-ip.com  XXXXXXX@guytp.org  
Please enter the password for user 'guy@guytp.org'  ********

2 hosts are registered to this account.
Do you wish to have them all updated?[N] (y/N)  n
Do you wish to have host [XXXXXXX.ddns.net] updated?[N] (y/N)  n
Do you wish to have host [XXXXXXX.ddns.net] updated?[N] (y/N)  y
Please enter an update interval:[30]  15
Do you wish to run something at successful update?[N] (y/N)  n

New configuration file '/tmp/no-ip2.conf' created.

mv /tmp/no-ip2.conf /usr/local/etc/no-ip2.conf
guytp@GTP-RRSC-RTB-01:~/noip-2.1.9-1$

To get this to run as a service on the Pi I created a file /etc/init.d/noip with the following contents:

#! /bin/sh
# /etc/init.d/noip 

### BEGIN INIT INFO
# Provides:          noip
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Simple script to start a program at boot
# Description:       A simple script from www.stuffaboutcode.com which will start / stop a program a boot / shutdown.
### END INIT INFO

# If you want a command to always run, put it here

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting noip"
    # run application you want to start
    /usr/local/bin/noip2
    ;;
  stop)
    echo "Stopping noip"
    # kill application you want to stop
    killall noip2
    ;;
  *)
    echo "Usage: /etc/init.d/noip {start|stop}"
    exit 1
    ;;
esac

exit 0

I then made this executable, tested it worked and finally registered it to be included in startup/shutdown:

sudo chmod 755 /etc/init.d/noip
sudo /etc/init.d/noip start
sudo /etc/init.d/noip staop
sudo update-rc.d noip defaults

And with that we’ve got dynamic DNS up and working. I can immediately see that my guytp.org alias is pointing at my 4G connection and I’ve got a way to determine my real public IP.

VPN Connection

This proved to be more than a little bit of a challenge. I ended up spending several evenings trying various attempts to get this working. At one end of my VPN (in my main network) I have a PFSense box running as border router and site-to-site VPN connections. To connect from the Raspberry Pi with full subnet routing rather than just host-to-host I decided to use IPSec. This required the use of the tool openswan.

There are several "gotchas" when you’re doing this in the scenario I describe (also acting as wireless LAN router and using 3G/4G as your connection).

• You’re more than likely behind NAT in your mobile connection as I discovered with dynamic DNS above
• You may even have something more weird than that depending on your provider
• There are a couple of steps that are missed out of several guides

I’ll go through my working configuration. To get my VPN connection up I first installed the requisite packages:

sudo apt-get install openswan uml-utilities chkconfig lsof

You now want to edit /etc/ipsec.secrets to setup your shared key. Since I’m only managing one tunnel I just use it to match any of my VPN connections. If you had more than one the first %any is the "left" side of the connection and the second is the "right" side’s identifier.

%any %any: PSK "Blah Blah Blah"

Next you need to setup the profile for the connection itself by editing /etc/ipsec.conf:

version 2.0
config setup
 protostack=netkey
 nat_traversal=yes
 keep_alive=30

conn Guytp
 type=tunnel
 authby=secret
 left=%defaultroute
 right=wan0.guytp.org
 leftsubnet=10.2.0.0/24
 leftid=10.2.0.1
 rightsubnet=10.0.0.0/16
 rightid=@GTP-BSK-RTB-01
 ike=aes128-sha1;modp1536!
 phase2alg=aes128-sha1;modp1536
 keyexchange= ike
 pfs= no
 auto= start
 lifetime=1h

The important things in this config are:

• Guytp - the name of the connection
• leftsubnet - This is the subnet you are providing over WiFi from the Raspberry Pi in my scenario. If you’re not setup exactly like me this is fundamentally the subnet local to the Raspberry Pi.
• leftid - This is how the RaspberryPi identifies itself to the remote end (PFSense). You can leave this blank and a default is picked based on IP, or you can enter a hostname or even an arbitrary string prefixed with @. I could only get the use of an IP working in PFSense or I got negotiation failures
• rightsubnet - This is the subnet that you are connecting to at the remote end (PFSense). It’s important to know you can only route one subnet per tunnel and you cannot just add a route manually to a gateway at the other end of the connection. Since I haven’t got anything overlapping with 10.0.0.0/16 this wasn’t an issue but it may be depending on your mobile provider’s NAT
• rightid - This is the name that the remote end (PFSense) identifies with. In this instance I did manage to use a friendly name

The rest of these are values that finally worked after lots of playing around. Very important is the nat_traversal at the top.

Before we can bring the Openswan IPSec link up to the PFSense box from the Raspberry Pi we also need to configure a few other things. You need to disable redirects on all network cards for openswan to function. To do this add the following lines at the bottom of /etc sysctl.conf:

net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.wlan0.send_redirects = 0
net.ipv4.conf.wlan0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

To reload these values run:

sudo sysctl -p /etc/sysctl.conf

At this point you need to configure the remote end (PFSense). The only issues I had here were:

• To route to more than just my default /24 on 10.0.1.0 I needed to explicitly include 10.0.0.0/16 in the local network of Phase 2.
• Due to the dynamic IP and NAT nature of the mobile connection you need to enter the "Remote gateway" as your dynamic DNS hostname (i.e. non-NAT address). If your external IP happens to change before your dynamic DNS you won’t be able to bring the connection up
• Due to the identifier being a defined IP address the PFSense distribution doesn’t find the Pre-Shared key so I explicitly have to set the key in Pre-Shared Keys. This is really unacceptable but more about that later - this does kind of ruin it with NAT.

Here are some screenshots of my PFSense configuration and firewall.

Finally to make sure everything is looking good for Openswan run these two commands:

sudo unlink /bin/sh
sudo ln -s /bin/bash /bin/sh

You can now try to bring the IPSec connection up.

sudo service ipsec restart

Check that your Raspberry Pi’s OS configuration is valid by running the below - everything should be OK, N/A or Disabled:

guytp@GTP-RRSC-RTB-01:~$ sudo ipsec verify
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                             	[OK]
Linux Openswan U2.6.37/K3.18.7-v7+ (netkey)
Checking for IPsec support in kernel                        [OK]
 SAref kernel support                                       [N/A]
 NETKEY:  Testing XFRM related proc values                  [OK]
Checking that pluto is running                              [OK]
 Pluto listening for IKE on udp 500                         [OK]
 Pluto listening for NAT-T on udp 4500                      [OK]
Two or more interfaces found, checking IP forwarding        [OK]
Checking NAT and MASQUERADEing                              [OK]
Checking for 'ip' command                                   [OK]
Checking /bin/sh is not /bin/dash                           [OK]
Checking for 'iptables' command                             [OK]
Opportunistic Encryption Support                            [DISABLED]
guytp@GTP-RRSC-RTB-01:~$ 

Assuming your other router is set up you should see your connection come online. If you don’t then you can check out /var/log/auth.log and /var/log/syslog for more information on errors. You can also run:

sudo ipsec auto —status

Hopefully if everything is good you should see something like this at the bottom:

000 #1: "Guytp":4500 STATE_MAIN_I4 (ISAKMP SA established); EVENT_SA_REPLACE in 1468s; newest ISAKMP; lastdpd=10s(seq in:0 out:0); idle; import:admin initiate

Your remote end should also show that the tunnel is up. If you see anything other than STATE_MAIN_i4 with ISAKMP SA established something went wrong. Double, triple and quadruple check your configs then start poking around in the logs above and on the remote end. This can be a real pain.

Now to test some traffic. There are a few other important things here:

• The route to your remote network (on PFSense) will not show in the RPi routing table directly
• There are no additional interfaces such as tun0 or ipsec0 on the Raspberry Pi side
• Due to dynamic routing you cannot do route add -net <something> gw <IP on remote side> - this will fail
• If you do want to see the route you’ll need to try ip xfrm policy or ip xfrm state and take a read through there
• You cannot ping directly (by default) from either the Raspberry Pi or PFsense side without specifying a network interface

Try to ping from the remote end to your Raspberry Pi once the tunnel is up. If you’re pinging from the router itself you’ll need to specify the interface - i.e. from FreeBSD with the IP 10.0.1.1 as the internal part of the PFSense VPN it would be:

ping -I 10.0.1.1 10.2.0.1

From any other machine on the PFSense side of the network pings should work normally.

At the moment traffic will not be flowing from the Pi outwards though if the Pi initiates it. This took me days to solve but with one simple command to sort out the SNAT issues. This is listed as being a slight quirk of the way the NETKEY (the Linux IP stack in the Raspberry Pi kernel) works. This simple command should fix it though and store the updated ip-tables for later use.

sudo iptables -t nat -I POSTROUTING -s 10.2.0.0/24 -d 10.0.1.0/24 -j ACCEPT
sudo iptables-save > /etc/iptables.ipv4.nat

You again cannot ping directly from the Pi without specifying an outgoing interface so in this case:

ping -I wlan0 10.0.1.1

And with that your Raspberry Pi / Wireless Router should now be connected to your VPN and capable of routing the traffic between subnets. At this point any device connected by WiFi in my car can directly connect to anywhere in my corporate or home networks. Pretty cool. I can get 10MBps over it as well which is about as much as I can get over 4G itself in my house.

My biggest issue here is that due to the PSK identification being based on a public-facing IP in PFSense. It seems even with hostname it stores the IP in Racoon (Openswan equivalent in PFSense) configuration. My solution rather than spending days more working this out has been to order a SIM card with no NAT and a static IP address which should be here shortly. I ordered mine from Andrews and Arnold (who can even do the termination with L2TP at mobile network level if required - but less fun) http://www.aa.net.uk/telecoms-mobile-data.html.

I didn’t bother routing traffic to AWS via the link. I could have done but this would have involved extra tunnels and since currently everything in AWS isn’t going to be touched by the car I’ll leave it. The easiest way to do this though is just duplicate your entries above to bring a second tunnel up between the Pi and router but changing the right subnet.

If that didn’t work for you prepare for many, many sleepless nights but I’ll happily help if I can. For now I’m just glad to have finally got this setup and working.