28 Core Xeon Workstation Build for Developers

I’m a Mac!

I’ve been using Apple products since 2005.  A friend came over and bought me his Power PC iBook and I fell in love with the thing.  It had a real terminal window, I could barely crash the thing no matter what I tried, the user interface was beautiful yet I felt I had all the power of a Linux workstation.  Why had I not used one of these before?  Unfortunately the lack of x86 processor limited some of the options I wanted to do.  About a month later Apple switched to Intel and I didn’t look back.  I got rid of the Dell PowerEdge server that was sat under my desk with a 24″ CRT monitor and switched to the first generation of MacBook Pro.

For the last decade I’ve used Apple products, I’ve owned half a dozen MacBook Pros, a MacBook Air, an iMac Retina, a Mac Pro (before they turned them in to rubbish bins) and even an XServe and XServe RAID.  I’ve had pretty much every iPhone since 2010, three iPads and an Apple TV.  Suffice to say I’ve been part of the Apple ecosystem.  I’ve not loved Apple in a while though.  As each generation of product comes out I’ve been less wowed and more frustrated for the last five years.

The top specced MacBook Pro I purchased this year have made me decide to leave the Apple world for now.  Running VMs (in Parallels or VMware) I found my fans almost always running.  Switching to bootcamp for the first time this year I found it more bearable but still found the laptop just not coping as well with the demands I was throwing at it as I would expect for over £2,000.  After a few months using Windows as my primary operating system I realised that I didn’t miss Mac OS at all and actually – shock horror – Windows 10 feels like a much more modern operating system than Mac OS X does these days.

With the upcoming changes to my life and need to have a good developer machine I’ve decided that it is time for me to part ways with Apple.  Goodbye shiny aluminium, it’s time for me to get my hands dirty once more and build a PC.

Now I’m a PC!

I used to always build computers (in fact I recently found a build log of my first ever PC – check out Project 386 and these stunning 1990s photos:

First PC Build - 2   First PC Build - 1

I’ve built servers over the last few years but the last time I built a computer for my own use was probably in 2002.  I used to always build them somewhere around 2003 it became cheaper to buy the ludicrously reduced Dell workstations or servers and then I switched to Apple.  This is the first opportunity I’ve had to spec out a machine I really want for myself in a very long time and the last change to buy something that’s well specified before becoming an impoverished student once more.

I had around £1,600 of budget to play with from selling my iPad and MacBook Pro.

Most builds on the internet and most current advice from friends ended up coming along gaming routes.  Hugely overclockable 4 core processors that I could get up to 5GHz per core over with some creativity but that didn’t feel right for me.  I like cores.  I love cores.  I love to run virtual servers.  I like code that uses as much CPU time as it can – I’ve recently been working on some foreign exchange trading projects and I need to run an awful lot of operations in parallel – ideally whilst running some servers and watching some video.

My workload is unusual.  My gaming PCs are designed to max out a few cores and run them as fast as possible.  I feel that I benefit better from more cores and that’s just not where consumer chips are at.

Enter the Supercomputer!

Initially I wanted a super quick-to-boot system like the Skylake 6700K that a friend of mine recently built for under £1,000.  But with just four cores (let’s forget hyperthreading for now) that’s just not going to cut it.  This led me to the Broadwell-E range which ranged from the Broadwell 6800K to the 6950K.  The six cores of the 6800K felt it may be a bare minimum but the and a better bet than the 6700K for pretty much the same price, but the 6950X felt much more like what I wanted but at £1,500 for the CPU alone I couldn’t justify it.

A bit of research later and many days of tinkering with specs and I decided to go with the Xeon E5 2620 V4 with 8 2.1GHz cores for the same price as an i7 6800K.  This gave me an extra two cores with the potential to add a second chip taking it to 16 cores.  Now that’s more like what I wanted.  I did some sums and decided I could just about justify going for the dual CPU version so 16 cores at 2.1GHz burstable to 3GHz.  Now that feels like the first major CPU improvement I’ve had since buying an 8 core Mac Pro many years ago.

Some more spec tinkering later and I had stumbled across some Xeon E5 2660V4 with a slightly different model number on eBay for the same price.  These offered 14 cores at 2.1GHz.  It took me a while to track them down but it turns out these were samples for testing or review.  With no guarantee that they were going to work I decided to use these in my build giving me a total of 28 cores and 56 threads.  Now that’s nuts.

The final build I came up with can be found here but is primarily:

  • Dual Xeon 2660 V4 with a total of 28 cores, 56 threads and 70MB L3 cache
  • 32GB ECC RAM (maximum of 1TB supported)
  • Asus Z10PE-D16 WS motherboard
  • Samsung 950 Pro M2 SSD
  • A pretty Enthoo Luxe case

Conclusion

The build went incredibly smoothly (check out the video) and I’m over the moon.  This thing may take a year to boot (thank you server motherboards) but once I’m booted I have more CPUs than I can shake a stick at and I’ve not managed to max it out once.  My SSD has over 2GB/s transfer rates and the setup is almost completely silent even when running at 100% CPU utilisation for hours at a time.

The Asus board is going to be great for future expansion with up to 1TB of RAM, more PCIx16 slots than I know what to do with and support for SLI (which is nice on a Xeon board) but I do just wish it didn’t take a year to boot.  Expensive but with very few options in the Dual Xeon workstation space I’m happy with the choice I made.  I was frustrated the M.2 socket on the motherboard is only x2 speed rather than x4 which meant I also needed to buy a StarTech PCIe x4 card but that was only an extra £20 at the end of the day and I do have an additional M.2 socket spare at least if I want a less-insane SSD in the future.

I’m glad to be back in the PC world and in many ways wish I’d done this sooner although the time feels right now.  It’s hard to imagine me outgrowing this machine any time soon (with the exception of graphics if I start doing GPU-intensive development or RAM which I’ve got plenty of capacity to upgrade).

The total build for the PC itself (excluding peripherals) came in at £1,926.50 delivered.  Whilst I definitely went a little over budget I’m more than happy with the results and think it was worth the extra.  It’s certainly not a laptop but it is crazy that I can build something this powerful for less than the cost of my last laptop.

One happy geek here – now I just need to get the rest of my office as shiny as my new PC.

Advertisements

20 thoughts on “28 Core Xeon Workstation Build for Developers

  1. Guy, My system is same MoBo with two E5-2683v3 installed with 32GB of ECC 2133 Ram. I am able to use AITweaker and bump the BCLK to 104 without stability issues within Chief Architect 3D Raytracing. C Bench CPU test 3301. No other BIOS adjustments done as I have no idea what all these settings mean are do to the system. I would be most interested in what you have done in the BIOS to improve benchmarks. Thanks, too, for the great youtube video. Fantastic job! Very informative and actually entertaining. Guess I’m a geek at heart.

    Cheers!

    Like

    1. Hey Scott, thanks for the feedback and great to hear you liked the video. I tried a variety of tweaks (including the build-in overclocking) and found majority of them just prevented it from booting at all. Xeons aren’t really designed for OC and the motherboard we’ve got certainly doesn’t support it well. I believe the ES ones I have may be OCable in a consumer board but not any of the dual ones. I’ve just stuck with the stock settings eventually for stability – any boost I get in CPU speed I figure is going to be fairly minimal. I rarely find a bottleneck that a few percent would impact. Give me a year when I’m bored and I might try all the possible settings but I got fairly bored of having totally reset the BIOS and start all over each time I tweaked too far. If you have success let me know your settings and I’ll give them a go too 🙂 The second post in this thread has a link to some Linus Tech Tips videos where OCing was tried – results weren’t phenomenal. https://linustechtips.com/main/topic/390575-xeon-v3s-that-can-overclock/

      Like

      1. Guy,

        Thanks for taking time to reply to me. The only thing I have done so far is bump the BCLK to 104. My biggest issue is not knowing what all the BIOS settings mean and or are used for. I wish AMI would publish a document explaining all the various settings and how they interact with one another. My other issue is all the postings of benchmarking results. I have yet to achieve the numbers posted by so many it leads me to believe there is some fabrication of figures going on…whatever happen to “Honesty is the best policy”?

        Anyway, thanks for the feedback. I will surely keep you posted.

        Cheers Mate!

        Like

  2. It depends which benchmarks you have achieved. I’d say that’s a really good Cinebench score – I’ve achieved around 3,100 in my highest run (immediately after Windows was installed with noting else) and with your chip OCd we should be around the same turbo boost so I’d say they are comparable. Feels like a good score to me – how much more were you hoping for?

    Xeons are pretty much fancy i7s so you can follow guides from http://www.masterslair.com/determining-your-maximum-bclk-base-clock-frequency-i7-i5-i3 and http://www.pcgamer.com/how-to-overclock-your-cpu-processor/ which should go through the other settings. Specific thread here may help as well: http://www.overclock.net/t/1591590/xeon-hacking-and-overclocking-x79-x99-beyond-x58 – where you are feels pretty good though I’d say 🙂

    Guy

    Like

    1. Guy,

      Thanks for the info. I wanted to share my recent Z10PE-D8 WS BIOS tweaks with you as you seem to be the only user of same hardware that has commented back with any level of data. I have pics if you can provide me a means to upload them to you but until then, Here is what I have done to achieve positive results.

      Baseline:
      Win 10 Pro
      Z10PE-D8 WS BIOS 3304
      (2) E5-2683v3 Production Release (NOT ES)
      32GB ECC 2133 Ram

      Cenibench Results with all BIOS at DEFAULT Settings
      CPU’s 2653

      BIOS Tweaks that work: (Defaults same unless noted below)
      AITweaker | BCLK 105.0
      Spread Spectrum | Disabled
      External Digi+ Power Control:
      Load Line Calibration | Level 4
      DRAM-AB, CD, EF, GH | 110%

      Reports:
      HWmonitor: CLOCKS Base 2625Mhz – Max 3151Mhz
      Cenibench: 3455

      Let me know if you have any questions and or suggestions on what you are doing to your board. BTW, I am not a gamer. I use my system for Architectural Design and Raytracing.

      Thanks,

      MrScotte

      Liked by 1 person

      1. Nice! I want to try this as well. I’ll post my results as well. Hey Scott Have you tried using the Ai Overclocker at level 3?

        Like

      2. Guy,

        Sorry for the delay in responding. No, I have not tried level 3. I would like to find a definition to all the levels listed in the bios and what they do. So many options with so little explanations.

        Staying at it!

        Like

      3. Hey guys, quick update tried tonight and the best I got was 103BClk. Any of the AI OC level 1 would randomly freeze, levels 2/3 wouldn’t even turn on. With BClk 104 I did boot but got a freeze after a few minutes (could get >3200 in Cinebench though). Currently at 103 and can get a 3150 in Cinebench 3150 from 2900 and at 2.17 from 2.1GHz. I suspect 104 would be attainable with some more tweaking but haven’t got any more time to look at this for now. I am going to stress test @ 103 and see how well it works. If either of you fancy testing against something try and follow this through for tweaks 🙂 http://www.tweaktown.com/image.php?image=imagescdn.tweaktown.com/content/7/4/7481_881_tweaktowns-ultimate-intel-skylake-overclocking-guide_full.jpg (but obviously different targets as you’re not going to hit 5GHz with one of these).

        Like

      4. I just tried this and I hit 3436cb. I was getting 3200cb easy on stock stettings. My CPUs are the e5 2683v3 es. The speed of my CPUs are 2.10 ghz.

        I also noticed a setting that says cpu lock. I didnt notice a difference whether i had it enabled or disabled. Have you guys seen this or messed with it?

        Like

      5. Just to add my stock CPU speeds are 2.00 ghz. I am also using 4 DIMMs x 8gb of RAM for each CPU.

        Another issue that I noticed is that my FPS are quiet low compared to my i7 6700k. I tried two different graphics cards. Here are my results: (All tests were on cinebench open gl test.)

        With 6700k –
        980Ti about 175 fps

        With Dual E5-2683v3 –
        980Ti about 95 fps
        1060 about 95 fps

        I also did a test with Unigine Valley (with 980Ti) and with the 6700k I was getting much higher fps. With the Dual E5 2683v3 CPUs the 980 Ti wasn’t even reaching 40C.With the 6700K the temp would reach a little over 50C. btw im using an EVGA 980 Ti Hybrid.

        It seems like there is a huge bottleneck because of the lower single core clock on the E5 CPUs. You guys have any more info on this?

        Like

      6. Frankly I would not compare the two based on FPS. Send a serious Raytrace demand to both and watch the Xeon’s walk away from the i7. These are two different CPU’s. Designed for entirely different purposes.

        BTW, are you running the Z10P Mobo? I’m still looking for BIOS SETTING help.

        Like

      7. Hi Scott,

        Yes I am. I have z10pe-d16 ws. I used those specs you mentioned.

        AITweaker | BCLK 105.0
        Spread Spectrum | Disabled
        External Digi+ Power Control:
        Load Line Calibration | Level 4
        DRAM-AB, CD, EF, GH | 110%

        I went from 2.00 to 2.10 Ghz
        3200 to 3400 cb on cinebench

        I am also looking for BIOS Setting help lol.

        Like

  3. For what it’s worth – I did try some more OCing and even with above settings and feedback I ended up with a very unstable situation when it came to rendering even if Prime 95 seemed to run stable for a day. I have a suspicion in my case it’s an issue with OCing the ECC 2400MHz RAM and that alternate RAM or disabling ECC may help. Not going to spend much longer investigating though as to be honest for a 3-4% gain I don’t want to risk the stability I have at stock settings.

    Like

  4. OT: Enjoyed your video “Being a Software Developer Sucks!” in which you questioned the wisdom of using ORMs where simple SQL scales better. I agree but isn’t it more often the case that you don’t have the choice unless you’re working on your own project? I mean, ASP.net, Spring MVC/Boot, Rails, Django, you name it all come with ORMs and its idiomatic to use them and probably required. How have you managed to avoid using ORMs when working for someone else?

    Like

    1. All of those may have ERM options available – but doesn’t mean you have to use them. For ASP.Net an ORM like EF is ultimately going to be based upon Ado.Net. There are many times I’ve had to use an ORM, but there are other times it hasn’t been the correct solution. I’ve been high enough in organisations to influence those decisions. If you have a reason to push for a particular approach then suggest it internally – just have a good reason for it. Sadly a valid response is “staff leave really quickly, have low talent and we need to go with a technology people can service for 5 years. The framework XYZ may be rubbish but it means we can at least support it” – so you need significant enough justification to go against that. An example of reason why – in one place we’d use both NHibernate and EF in various projects but couldn’t customise them sufficiently to bring some datasets back efficiently. Switching just 2 segments of the application from NHibernate/EF to ADO.Net directly reduced bandwidth from a SQL Server from 20MBps to 4MBps. We didn’t do entire application though – only targeted parts to meet specific goals. Supportability as an argument will usually win out.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s