When people often discuss their home labs, they often discuss old desktop computers cluttering their abode. When people say that they “run a game server”, they typically mean that they pay someone like aushost.gg to run the server while they run the community and do some config tweaking.
Me? Well I purchased my own hardware and colo’d it.
I’ve been asked the question “how do you learn new technologies” a few times in the past week and I have answered the same way each time; I have a DL360 sitting at Equinix BR1 which has enough capacity for anything that I need to test and more. Based on the reactions, I’m going to guess it’s not a common response.
Today I figured I would take some time out to discuss this ‘little’ project. While it’s nothing overly impressive, it’s a somewhat interesting setup
It was one hell of a purchasing decision. Normally with businesses, I look for the best hardware to meet their needs and spec up accordingly; using reasonable budgets to achieve my goal. It’s not that crazy for a company to dump $30,000 on a small compute cluster, but you’d never do that for a test lab. So min-maxing was a key component of things.
Initially I was considering a whitebox solution using something like a 1ru supwermicro case and ebay-sourced parts to cram as much compute into it as I could, but the pricing was crazy. Disused enterprise servers are normally found cheap on ebay, and sometimes they come with a decent provision of RAM.
Older enterprise servers also run DDR3 RAM which is a little more affordable these days, and I don’t need 10Gb because my uplink is only 1Gb/1Gb. Typically they have built-in RAID controllers, IPMI for when that comes in handy, and are rock-solid.
The R630 was my top contender due to my familiarity with them. Back when I was deploying R630’s as audio servers for major radio stations, we saw quite a lot of “kind-of-DOA” units reach our office, but Dells support was always fantastic. Typically speaking, you’re not going to have any issues with a used R630 because the company would have instantly RMA’d them, so they’re a pretty safe bet.
Another contender was a Cisco UCS C220 M3. These are are the same class as R630’s however it’s possible to find them super cheap because people just don’t like using them. Unfortunately, their lack of popularity makes aftermarket parts difficult to source; so rails and caddies would be a pain.
Finally, the DL360 G8 which I landed on. They are super popular in Australia so greys-resellers list heaps of them on ebay, bringing the prices down and leaving a lot of spare parts available on eBay. HP’s are pretty good units however their extended support is absolutely ridiculous. That said; they are rock solid units that don’t complain too heavily about non-HP parts being attached, and you can find them with iDrac Enterprise physically licensed to the machine.
Essentially, my contenders were the R630 and DL360, with the UCS C220 M3 being a contender if I found one for under $300 with 128GB RAM.
eBay is fantastic for some things. I found a diskless DL360 with 128GB RAM and dual E5-2630’s for only $500. Fun fact; it was a mislisting and the seller changed the listing to $550 for the same unit with 96GB of RAM, and the unit that was shipped to me had mismatched ram.
Essentially, it just came down to price. The DL360 and R630 are both popular units and R630’s can be found cheap because companies use them as OEM units and you can find new-old OEM stock from DR sites.
Anyway, this leads me into another thing:
At 240v, DL360 G8’s run 0.4A at idle. This was an issue because everyone offered 0.2A with 1RU colo. I ideally wanted 3RU so that I could run a second server and a firewall, however I didn’t have the money to do that so I opted to stick to 1RU.
Unfortunately, 3RU typically gives something like 3A of power but 1RU only gave 0.2A. ColoAu were extremely unresponsive but Servers Australia were extremely proactive in getting me onboarded, so I ended up opting for them. They have also increased their permitted power usage to 1A per RU up to 5RU; proving that they understand their customers usage.
Given that I’m Queensland based, I didn’t want to catch a flight to SY3 just to swap disks out at night, so I had to go to NextDC or Equinix in Brisbane.
Word of advice; BR1 Equinix sucks. It is absolutely adequate for hosting something up however it’s not staffed 24/7 meaning you’ll run into issues with after hours access. I was left standing around Brisbane streets for hours waiting for my promised access to my server to be granted, all because BR1 is only staffed during office hours.
On the plus side, Servers Australia went above and beyond making sure that I got access to my server, sending a tech down on a lime scooter to give me access. I was also signed up with a 24/7 access pass to the provision, so that I don’t have to pre-register for visits anymore.
All up, SAU have been fantastic and despite the average runnings on BR1 (friendly staff though, when they’re there) compared to somewhere like SY3, I do wish that they had a NextDC offering.
For $500+postage I had my DL360 G8 delivered to me from Sydney. It came with dual E5-2630’s and 128GB of RAM; it had 6x16GB sticks of RAM (96GB) with 4x8GB installed to bring it to 128GB of unbalanced memory.
It has eight slots for disks, but didn’t come with any caddies or rails, so I had to spend $200 on that. Once the ebay caddies and rails were in, I purchased eight 500GB SSDs from Umart for around $1000 (annoyingly, they’re $30/unit cheaper at the moment). This gives me 3TB in RAID6 with double parity, so the server will keep running even if I drop two of the disks.
Unfortunately, you do not want to run RAID6 behind an enterprise RAID card. I ran it on a R630 and it would constantly nuke itself; you also don’t want to run the operating system on the ZFS array or bad juju may occur, so it just really wasn’t an option. That said, I purchased a $20 128GB SD card and installed Proxmox on that; if there’s ever an issue, I can just DD another SD card and swap it out.
I’m just reviewing some of my older posts and I thought that I would add an update to this.
ServersAustralia have had fantastic support and the account managers have seemed quite nice, however their pricing is just insane. They were obviously pushing to lock in pricing before a price hike, however I was already planning to leave them before this.
I think I pay ~$150 inc GST PCM for the colo. Comparatively, Fluccs is around $90 inc GST PCM and it also comes with a /29.
ServersAustralia tried to negotiate by throwing in added transfer but it’s not something that I really need. I do need the /29, so it was disappointing that they couldn’t come to the table on that considering their pricing has nearly doubled since I signed up three years ago.
On this though, the DL360 is running 256GB of DDR3 RAM now. I did purchase some 2667v3’s for the server but I forgot to take thermal paste to the DC when I went to upgrade them. The Ark servers are suffering HORRIBLE lag because of the absolutely pathetic CPUs that I’m running (E5-2630 v1 @ 2.3GHz).
I’ve since purchased an R630 which has dual 2697v4’s if memory serves me correctly. It also has 256GB of RAM and I’m just waiting for a good deal to come up for another 256GB. This will be deployed to Fluccs as soon as I can, and the DL360 will be brought home for upgrades before also being deployed to fluccs.