My Home Lab ESX Host is Online

All the bits for my home VMWare lab have arrived and I’ve been building out the ESX host over the weekend. The hardware build went reasonably well apart from a few cables I needed to pop down and get from Maplin.

The Build

This was my first time installing VMWare ESXi 5.5 which should have just gone swimmingly. The Asus KCMA-D8 motherboard, AMD 4332 CPUs and all peripherals were on the VMWare hardware compatibility list. I decided that rather than using a memory stick to install VMWare (even though there is an internal USB socket on the KCMA-D8) I was going to go for an SSD that I’d also use for installing ISOs. This should have been nice and easy.

There appears to be some slightly incompatibility between the Asus board and VMWare though. During the install there is an apparent hang whilst the module is loaded for the IP KVM. I actually attempted two reboots it hung for long before doing some googling and this appears to be a known issue. If you leave it about ten minutes the rest of the install process carries on fine. 

Once ESXi is installed the same hang happens with every single boot. For now I’ve just disabled the module in ESXi’s configuration for the host as shown below. Since I will be using this lab for non-commercial use at the moment I won’t be licensing VSphere. This limits some of the functionality available and is why, for now, I’ll be just using the Windows thick-client to manage my individual host. I may well change this and licence the full VMWare in a few months.

So once everything is installed if you’re using the KCMA-D8 you will need to head into your VMWare client and make a change to the setting as shown below.

Picture of setting to change here

The other thing to note is the giant Noctura heatsink/fans I bought only just fit into my 4U case by literally a single milimetre.  Anything smaller and they won’t fit.  They’re excellent coolers – at full load the CPUs sit at about 25 degrees almost silently and that is with only 2 of the 4 fans connected and set to low speed.  I’d probably recommend anybody else going for the 10cm versions of them instead as it’s going to be a lot less cramped inside the server case.  The good news is that the two spare fans have given me extra cooling for the disks.

Internal picture of fans

Other than that – simple and I’ve got my first decent VMWare infrastructure online for well under £1,000 sporting 12 cores and 32GB of RAM.

The one thing I haven’t yet sorted is storage space or any kind of redundancy. I’m using a single 500GB disk for the operating systems for now and have also added several old external disks to pass through to my file server host. I’ll be looking more at SAN storage options in a future post.

Pictures of host

Network Configuration

I decided to set up quite a few virtual machines over the course of the weekend for my business infrastructure within the home and some capability for the items in my house that connect in to the network. I’ve also now received the two VLAN-enabled gigabit switches so these have been deployed in my office and comms rack within the house.

I decided at the high level to segregate my network in to several VLANs each mapped to it’s own /24 subnet for simplicity.

VLAN Subnet Description
N/A Router-to-router /28s
1 Core (ESX host, switch interfaces, etc)
2 Servers
3 VOIP Handsets
4 Workstations over Ethernet
5 WiFi
6 Network Peripherals (printers, household devices)
7 VPN clients with full access

I’ve not used most of this yet so very much subject to change but it feels like a good starting point.

I’ve virtualised literally everything – even my routers. I’m using PFSense for routing. I’ve created a border PFSense router that has its own dedicated network port from VMWare. This directly connects to the PPPoE ADSL modem and sits as my main network gateway and external firewall. In addition to this I also have a dedicated VLAN router whose sole job is to route between the different VLANs. This is again PFSense but with one virtual network connection to each of the VLANs.

The ESX itself is tagged to receive frames for all VLANs on both the switch and as VLAN 4095 in the ESX configuration.

I’ve not yet got WiFi capabilities that support VLAN tagging so for now the WAPs themselves are hooked up to VLAN 5 along with their clients. This certainly isn’t ideal.

I’ve certainly seen some problems so far with this setup where parts of the house touch it. A lot of consumer devices just aren’t designed with network segregation in mind as it turns out. For example having the Sonos on a different subnet to the WiFi clients rendered it useless. I did a bit of protocol analysis to see what requests were being made and it seems to be restricted to local subnet broadcasts and Google informs me that UPNP is used as well. I’m currently looking at putting a simple bit of software onto the VLAN router to proxy these requests – although my first quick attempt failed. More on this when I manage to get it working as it feels it should be really easy. Sadly for now the Sonos is also in the wrong VLAN and is masquerading as a WiFi client.

My Hosts

With the network design sketched on the back of a fag packet it was time to start building hosts. I decided to build with a naming convention that gives me scope to build out to cloud-based VMs integrated into the same network so all hosts are named Client-Location-MachineType-Number. In this instance everything is prefixed with GTP-BSK for Guytp Limited in Basingstoke.

Host Type Functionality
GTP-BSK-RTB-01 PFSense Border router and VPN
GTP-BSK-RTV-01 PFSense Inter-VLAN router and DHCP relay
GTP-BSK-DC-01 Windows Server 2012 R2 Domain controller and primary DNS
GTP-BSK-INF-01 Windows Server 2012 R2 DHCP, secondary DNS, Windows Update services and any other ancillary functionality the is needed by LAN
GTP-BSK-FS-01 Windows Server 2012 R2 File server (SMB, NFS, FTP). All external disks connect to this host directly. I’ve also configured it as a print and scan server for the network.
GTP-BSK-OSX-01 Mac OS X To do everything for Mac clients in my network. The name is a little generic but the Mac clients will still overwhelmingly use Windows infrastructure.
GTP-BSK-UTM-01 Unified Thread Management Advanced firewall capability. I’m currently testing Sophos UTM. This isn’t being proactively used yet. Looking at use as VPN concentrator.

I’m also looking at building (but haven’t yet) an Asterisk host for my VOIP server to replace a massively underpowered Mini ITX rack mount Asterisk box I built six years ago.

So far all has gone well and the machines are incredibly responsive. Most have been giving 60GB of disk space to start with, 2GB of RAM and 2 CPU cores.  I’ve also played around for various test builds to get to this point and ran through a few versions of router software and tested client builds. The great part about virtualisation in such an environment is it easily lets you test everything you may want to use until you’re happy.

I haven’t yet managed to build the OSX host – there are issues with that which I’ll post about separately when I’ve worked through them in a couple of days.

Screenshot of all VMs

That’s That…

So that’s all there is too it really. Not the most exciting VMWare setup in the world but hopefully I’ll be able to add to it over the next few months. I’m looking at adding dedicated SAN storage, sorting out the Mac OS X virtualisation and connecting my infrastructure to both a public cloud environment and that of my colleagues’ networks.

Whilst this has certainly been good fun for very little money the biggest benefit is for the software development work I undertake. I am now able to spin up environments for different client projects as and when needed whilst keeping them completely separate. It’s also great to get an idea of load capabilities for projects much more easily prior to go live and I think this is a very worthwhile investment for anybody working as a freelance developer.

I’ll try and tie up some of the loose ends with the ESX project over the next couple of weeks – my SAN, the issues of UPNP and Mac OSX virtualisation. As and when I go through future builds I’ll post anything of particular interest but since these were all pretty cookie-cutter I thought I’d leave them for now.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s