«608 »
  • Post
  • Reply
Don Dongington
Sep 27, 2005

#ideasboom

College Slice

Paul MaudDib posted:

I'm rebuilding my storage around a HP Z400 CAD workstation with a Xeon W3565 (Bloomfield) and 16 GB of ECC. You can get similar machines on NewEgg or eBay or other surplus channels pretty cheaply, they go for about $250 nowadays I think. Triple-channel RAM, not sure about PCIe lanes. Big downside, only PCIe 2.0. Also, it uses what looks like a standard 24-pin ATX connector but it actually has a few pins swapped and won't boot with an off-the-shelf PSU, although IIRC you can find adapters. How big a pool can you actually get on 16 GB before performance tails off?


I actually do have one of these, (W3550 though) but I recently gave it to my neice with my old GPU in to replace her old pentium which died. It only had 4gb of ECC anyway, and they only have two 3.5mm drive bays so at the minimum I'd have to:

- Put the i5 in a case, get a bigger PSU for it to support the GPU, and buy her a new copy of Win10 home because I used the free upgrade to get it onto there
- Get another 4-12gb of ECC DDR3
- Get 5.25-3.5" drive bay converters
- Buy a PCI-E sled instead of just using the M.2>Sata converters I already have
- Never have room for more than 4 drives

So I feel like i'd be better off just going with the Sandy Bridge until more mainstream denverton offerings appear. The value of ECC seems limited for my use case.

Also yeah the idea would be to stripe the two m.2 drives and partition them off to L2ARC/ZIL.

Ziploc
Sep 19, 2006
MX-5

Why does FreeNAS default compression on by default now? And even once it's turned off, the storage manager is reporting a compression ratio.

Wut?



SamDabbers
May 26, 2003



Fallen Rib

Compression is on by default because it has very little CPU overhead. If the data being written is not compressible the algorithm will bail out early and write it uncompressed. Turning off compression doesn't decompress existing compressed data on the disk, which is why the compression ratio is still greater than 1.00x

Desuwa
Jun 2, 2011

I'm telling my mommy. That pubbie doesn't do video games right!

Don Dongington posted:

The biggest issue as I see it, is that my budget will stretch to maybe one 4TB Red but not two of them right now, so redundancy is definitely out of the question - but none of the data in question is stuff that I can't just download again, so I'm not really bothered. I was thinking the best approach would be to buy a new 4TB, create a single disk vdev and and a new zpool, copy the contents of the other 4tb over to that, create another single disk vdev and add it to the zpool, etc. until both of my current disks and data are on there. Based on my understanding, this way if a single disk fails I lose that vdev, but not the whole zpool, and I take a performance hit for my particular use case, as bigger vdevs are apparently better for HD media, as I understand it. As I will only be streaming to maybe 2 clients at a time, this shouldn't matter.

I can't stress enough that I REALLY don't need redundancy for this stuff, but this setup seems better than JBOD or a single vdev without parity for what I'm doing. I can use crashplan or something if I decide I really don't wanna lose a chunk of it to disk failure. I may consider migrating it all to a denverton setup when that comes about, to gain the advantages of more cores and ECC, but I'm mostly just serving media to Kodi with this thing. I do want the ability to add more disks as I can afford them, and would prefer to avoid the risk of losing the whole lot if just one carks it.

Let me know if there's a better way to do this in budget, I guess. Or if this is a really stupid idea even for linux isos.

Bolded part is not possible. In ZFS there's no such thing as just losing one vdev but not the entire pool because data is always striped across all vdevs in the zpool.

You'll want separate zpools consisting of a single vdev made up of one disk each. It's fine to do multiple pools though, there's nothing wrong with that; you still get checksumming and the knowledge that if a drive starts to fail but doesn't instantly die you won't be reading garbage data without knowing.

apropos man
Sep 5, 2016

You get a hundred and forty one thousand years and you're out in eight!

The fragmentation on my zpool mirror volume is up to 26% already. Does data become fragged in the traditional sense? Do I need to do anything? What's the biggest frag anyone's had?

Don Dongington
Sep 27, 2005

#ideasboom

College Slice

Desuwa posted:

Bolded part is not possible. In ZFS there's no such thing as just losing one vdev but not the entire pool because data is always striped across all vdevs in the zpool.

You'll want separate zpools consisting of a single vdev made up of one disk each. It's fine to do multiple pools though, there's nothing wrong with that; you still get checksumming and the knowledge that if a drive starts to fail but doesn't instantly die you won't be reading garbage data without knowing.

Hrm ok, that seems logical. It seems like I'm negating like 2/3rds of the functionality of ZFS here vs just using something LVM though, right?

apropos man
Sep 5, 2016

You get a hundred and forty one thousand years and you're out in eight!

Maybe, but in order to feel at home in this thread, you must've been convinced to use ZFS.

Don Dongington
Sep 27, 2005

#ideasboom

College Slice

apropos man posted:

Maybe, but in order to feel at home in this thread, you must've been convinced to use ZFS.

Okay anyone wanna buy a baby? Only 9 months on the clock. Also will consider trades for 4TiB NAS drives of reputable make,

Desuwa
Jun 2, 2011

I'm telling my mommy. That pubbie doesn't do video games right!

Don Dongington posted:

Hrm ok, that seems logical. It seems like I'm negating like 2/3rds of the functionality of ZFS here vs just using something LVM though, right?

I'm honestly not familiar with the more advanced usages of LVM but I don't think LVM offers the functionality you want either, where you can somehow make a big volume that can survive individual disks failing without losing data on other drives, since LVM works on the level of volumes and knows nothing about the data on top of it. That's a selling point for things like unraid, snapraid, or other exotic file systems that don't really offer the same guarantees that ZFS offers.

I'd say you're getting the most important advantages of ZFS (checksumming, scrubs, stability, etc) and only losing the ability to recover from partial or complete failures without restoring from backups or having to re-torrent all your linux ISOs.

Don Dongington
Sep 27, 2005

#ideasboom

College Slice

Desuwa posted:

I'm honestly not familiar with the more advanced usages of LVM but I don't think LVM offers the functionality you want either, where you can somehow make a big volume that can survive individual disks failing without losing data on other drives, since LVM works on the level of volumes and knows nothing about the data on top of it. That's a selling point for things like unraid, snapraid, or other exotic file systems that don't really offer the same guarantees that ZFS offers.

I'd say you're getting the most important advantages of ZFS (checksumming, scrubs, stability, etc) and only losing the ability to recover from partial or complete failures without restoring from backups or having to re-torrent all your linux ISOs.

I think I'm just gonna hold off until I have enough disks to go RaidZ1 tbh.

Thanks for the sounding board, goons

Paul MaudDib
May 3, 2006

"Tell me of your home world, Usul"


You can replicate that somewhat with rsync. Obviously rsync doesn't know filesystem-level stuff but you can run a command that'll synchronize two drives, or two folders (one on a big spanned volume, the other on a dedicated backup?) But you don't really need LVM in the mix at all unless you want a spanned volume or RAID.

The advantage of ZFS is of course that it does understand filesystem-level stuff, so it can handle use-cases like "moving files around" in efficient ways rather than deleting and re-copying.

The advantage of either of these over RAID is they allow you an "oops" button, if you delete something it should still be on the backup drive until you run a sync command. ZFS can turn that into a full incremental snapshot if you want, too.

Paul MaudDib fucked around with this message at 07:11 on Jul 25, 2017

evol262
Nov 30, 2010
#!/usr/bin/perl

Paul MaudDib posted:

You can replicate that somewhat with rsync. Obviously rsync doesn't know filesystem-level stuff but you can run a command that'll synchronize two drives, or two folders (one on a big spanned volume, the other on a dedicated backup?) But you don't really need LVM in the mix at all unless you want a spanned volume or RAID.

The advantage of ZFS is of course that it does understand filesystem-level stuff, so it can handle use-cases like "moving files around" in efficient ways rather than deleting and re-copying.

The advantage of either of these over RAID is they allow you an "oops" button, if you delete something it should still be on the backup drive until you run a sync command. ZFS can turn that into a full incremental snapshot if you want, too.
You need to use a regular filesystem on top of LVM, which obviously understands "filesystem-level stuff", and it can take incremental snapshots.

LVM offers flexibility for Linux instead of shuffling partitions, which is why it's used even in single-disk configs. ZFS obviously wins, but LVM isn't quite as useless as presented here.

devilmouse
Mar 26, 2004

It's just like real life.


Is there a good "setting up freenas for the first time" guide out there? I'm currently burning in all the disks and am comfortable with ZFS, but is there an accepted set of best practices (set up weekly scrubs, set up email notifs, here are the alerts and tunables you care about, etc) for freenas specifically?

D. Ebdrup
Mar 13, 2009



apropos man posted:

The fragmentation on my zpool mirror volume is up to 26% already. Does data become fragged in the traditional sense? Do I need to do anything? What's the biggest frag anyone's had?
It's copy-on-write, so of course it'll have more fragmentation than traditional filesystems - but ZFS uses write caching to mitigate this which means ZFS will only write data from the ZIL to disk every 5 seconds (by default, or x number of megabytes) in order to avoid fragmentation - with the caveat that FSYNC and OSYNC calls (such as from databases, NFS, iSCSI and VMs) will always be written to disk immediately, unless you have a SLOG. So whatever fragmentation the pool has is probably due to some really weird ZIL tuning or because whatever you're running is issuing synchronous writes of some sort.

Also, it's worth noting that assuming your working dataset is smaller than the total amount of memory in your system, you'll never really encounter fragmented data and the access time penalty associated with it, because everything will be kept in ARC.

apropos man
Sep 5, 2016

You get a hundred and forty one thousand years and you're out in eight!

I've got 16GB RAM and it's a 1TB pool that's been up to about 600GB full and now is about 400GB full. I'll have a look at some of those abbreviations you mentioned later. Thanks.

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

apropos man posted:

The fragmentation on my zpool mirror volume is up to 26% already. Does data become fragged in the traditional sense? Do I need to do anything? What's the biggest frag anyone's had?
That's the nature of copy-on-write. If you change a single byte in a 10MB file, the block the byte gets changed it will be rewritten elsewhere on disk. So if the 10MB was formerly sequential for whatever reason, it isn't anymore. Now imagine this over tons of files.

ZFS mitigates this a bit by readahead and prefetching, escalator sorting* of IOs and extensive caching.

(*: At least it did on OpenSolaris, not sure what happens on Linux, which has its own means in the kernel I think, or FreeBSD/NAS.)

apropos man
Sep 5, 2016

You get a hundred and forty one thousand years and you're out in eight!

Now I understand why caching has an important part to play in ZFS. Cheers.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell



IIRC, I think somewhere up there someone was asking about ram for ZFS vs amount of storage.

I've got something like 40-50TB on a system with 12GB (well, I just upgraded to 32GB for non-zfs reasons) and I never have trouble saturating my gigabit ethernet which is all that matters for me. Not using deduplication.

eightysixed
Sep 23, 2004

I always tell the truth. Even when I lie.


G-Prime posted:

In case anybody missed it, Best Buy has the extremely shuckable 8TB WD EasyStore drives on sale right now for an absurdly low price ($159, lowest I've ever seen for them). They contain a single Red, and people claim they can be RMA'd without the case. I snagged 8, and am currently firing off SMART tests on them. Holy shit they're loud in the external cases.

So should I buy these? No matter what, I'm migrating off of Xpenology this weekend and going to run unRAID and was going to buy 4TB Reds from Amazon - Which are currently $132. 8TB for $159, seems too good to be true. They're actually Red's in the externals? And it's literally juts as easy as opening the case and taking the drive out?... Something seems strange.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

eightysixed posted:

So should I buy these? No matter what, I'm migrating off of Xpenology this weekend and going to run unRAID and was going to buy 4TB Reds from Amazon - Which are currently $132. 8TB for $159, seems too good to be true. They're actually Red's in the externals? And it's literally juts as easy as opening the case and taking the drive out?... Something seems strange.

It's not the first time a manufacturer has used NAS or other higher-end drives in a specific run of externals. Why they do is anyone's guess--maybe they had extra stock they wanted to get rid of, maybe these specific drives are slightly less reliable than their normal brethren, maybe there's no real production cost difference between a Red and a Green so they just threw in whatever they had the most of sitting on the shelves, who knows.

The one thing that is worth noting is that, if they all come up as externals in WD's systems like G-Prime's did, the warranty will be less than the normal desktop version (2yrs vs 3yrs), and there's the possibility (but kinda doubtful) that they might deny warranty support if they think you cracked it open.

Even with that, though, this looks like a fantastic deal.

eightysixed
Sep 23, 2004

I always tell the truth. Even when I lie.


But just to be certain, before I throw down $1,000 - You literally just open the case, take out the drive, and pop it in to your server, plug and play style?

D. Ebdrup
Mar 13, 2009



Combat Pretzel posted:

(*: At least it did on OpenSolaris, not sure what happens on Linux, which has its own means in the kernel I think, or FreeBSD/NAS.)
FreeBSDs implementation, aside from making use of the VFS, is as close to the upstream maintained by OpenZFS/Illumos (though there's a tiny bit of argument of which implementation is the reference one, since ZFS on FreeBSD is used far more than on Illumos). Linux, as I recall having been told, has a far bigger diff, because of various things that Linux can't do (although they're good about upstreaming the fixes for the things).

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

eightysixed posted:

But just to be certain, before I throw down $1,000 - You literally just open the case, take out the drive, and pop it in to your server, plug and play style?

Yes, they're just normal drives attached to a small USB adapter card. If you felt so inclined, you could shove a different drive back into the card/case and it would work just fine as an external.

Backblaze made something of a name for themselves a few years back by "shucking" drives like this on a near-industrial scale, because doing so was cheaper than trying to buy normal desktop drives thanks to the fallout from the typhoon.

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

D. Ebdrup posted:

FreeBSDs implementation, aside from making use of the VFS, is as close to the upstream maintained by OpenZFS/Illumos (though there's a tiny bit of argument of which implementation is the reference one, since ZFS on FreeBSD is used far more than on Illumos). Linux, as I recall having been told, has a far bigger diff, because of various things that Linux can't do (although they're good about upstreaming the fixes for the things).
I've researched this some more, even on Linux it still uses its own IO scheduler and actually forces the Linux one to noop.

--edit: This applies to disks used as whole. If you stick partitions into the pool, it doesn't do that. That'll probably include pools created on FreeNAS, which does some partitioning bullshit due to GEOM I think? If that's the case, importing an old FreeNAS pool into Linux, you should force the noop scheduler for best performance.

Combat Pretzel fucked around with this message at 18:55 on Jul 25, 2017

PraxxisParadoX
Jan 24, 2004
bittah.com

Pillbug

Combat Pretzel posted:

I've researched this some more, even on Linux it still uses its own IO scheduler and actually forces the Linux one to noop.

--edit: This applies to disks used as whole. If you stick partitions into the pool, it doesn't do that. That'll probably include pools created on FreeNAS, which does some partitioning bullshit due to GEOM I think? If that's the case, importing an old FreeNAS pool into Linux, you should force the noop scheduler for best performance.

Mind sharing said research? I'm doing a migration from FreeNAS to Linux ZFS over the coming weeks and am interested in knowing about the differences.

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

https://bugs.launchpad.net/ubuntu/+...ux/+bug/1550301

Relevant quote:

quote:

It does not set the scheduler for a disk if a partition is used in a pool out of respect for the possibility that there are non-ZFS partitions on the same disk. https://github.com/zfsonlinux/zfs/issues/152

There was another page where it explained that using anything else than noop can have an performance impact, but I can't find it in the Chrome history.

Combat Pretzel fucked around with this message at 19:11 on Jul 25, 2017

EL BROMANCE
Jun 10, 2006

COWABUNGA DUDES!



eightysixed posted:

But just to be certain, before I throw down $1,000 - You literally just open the case, take out the drive, and pop it in to your server, plug and play style?

Picture guide in the attachment of this first post:

https://hardforum.com/threads/180-8...inside.1930595/

eightysixed
Sep 23, 2004

I always tell the truth. Even when I lie.


EL BROMANCE posted:

Picture guide in the attachment of this first post:

https://hardforum.com/threads/180-8...inside.1930595/

Highly helpful. I don't think I'll destroy a credit card, but it seems like a few guitar picks would do the trick

Paul MaudDib
May 3, 2006

"Tell me of your home world, Usul"


With a NAS case like a U-NAS NSC-810a, would I want fans pushing in or pulling out? Normally in gaming cases you want positive pressure, and it would probably help reduce dust if I could do that. Would I want static-pressure fans pushing in, or high-volume?

How much SLC do I need for a ZIL, realistically, and how fast? The two throughput scenarios here would be gigabit ethernet, and Infiniband QDR or FDR x8 (64 or 128 gbit). I see you can get little modules that attach to internal USB 3.0 headers, would that be totally inadequate for these use-cases? I'm not intending to hammer writes across InfiniBand very frequently (it would mostly just be for reads), just trying to design around leaving myself upgrade room.

That also means it'll be running across the PCH. Does it hurt performance (or increase risk of data loss) to have ZIL or L2ARC on the PCH?

Paul MaudDib
May 3, 2006

"Tell me of your home world, Usul"


The QVL for my motherboard of choice is ancient, last updated 2015.12, and there are literally six choices in total. How much of a risk are you taking by departing from the QVL for RAM? As long as I use the same type (registered ECC DDR4) it should pretty much just work, right? Let alone if I'm using a different-capacity version of the same kit?

I'm considering a processor with 1866 max, and my high-end option would be 2133, so 2400+ on the RAM isn't really necessary anyway. I would think running the board/memory way below its design speeds should make it even less prone to trouble, correct?

Paul MaudDib fucked around with this message at 01:14 on Jul 26, 2017

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

NVMe or SATA SSDs? The PCH should have enough bandwidth for multiple SATA ones when pegged. NVMe probably just for pegging one. In the case of SATA, if it's a Marvell controller, forget it.

As far as cooling goes, I went with positive pressure, partly because of this awkward PC-Q25B case from Lian-Li, but it dropped the chip temperature of my ConnectX3 by 5°C.

Paul MaudDib
May 3, 2006

"Tell me of your home world, Usul"


Combat Pretzel posted:

NVMe or SATA SSDs? The PCH should have enough bandwidth for multiple SATA ones when pegged. NVMe probably just for pegging one. In the case of SATA, if it's a Marvell controller, forget it.

As far as cooling goes, I went with positive pressure, partly because of this awkward PC-Q25B case from Lian-Li, but it dropped the chip temperature of my ConnectX3 by 5°C.

NVMe but to start with it'd be a cheapo probably (for L2ARC).

It appears the onboard slot is limited to 2.0x2 (10 gbps theoretical) though, so I guess it's probably more of a boot drive than a L2ARC slot. It'd probably make sense to go with a PCIe sled right away for L2ARC then. The onboard M.2 might still be OK for boot, or that might be a place to put a USB 3.1 controller (which also has a 10 gbps theoretical limit anyway).

But, is it a problem to have the ZIL on the PCH? The HDDs (up to 8x) also have to run over the PCH as well since they'd be using the onboard SATA.

Also, is 2 GB of ZIL enough? There is a huge price difference between the little 2 GB thing that goes into your USB 3.0 header and a ZeusRAM 8 GB. My use-cases aren't too intense overall so a $1K SSD is totally overkill, but on the other hand I don't know enough to say if it'll seriously hurt performance either.

I suspect positive pressure is the correct answer too.

Paul MaudDib fucked around with this message at 01:31 on Jul 26, 2017

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

Paul MaudDib posted:

...NAS building stuff...
For RAM QVL, you can pretty much ignore it. It's fairly unlikely for you to manage to get RAM that it won't play nice with, unless your motherboard happens to be particularly finicky. What board is it, anyhow?

Running below-spec isn't likely to help you much; it'll either work or it won't, for the most part.

For fans, I'd leave them in whatever direction U-NAS has them, but I also doubt that it'll make a huge difference. Dust mitigation is probably a bigger concern for something you plan on shoving in a closet and ignoring. It's hard to see exactly how the vents all line up there, but it looks like there's only one small side-vent for the HDD bay, which would make me think that you might be better off with high static-pressure fans over high-volume. But again, even stuffed full of HDDs, you're unlikely to have too many thermal issues as long as you have some airflow at all.

As for "how much SLC do I need for a ZIL" the answer is "probably zero." ZIL only really makes sense if you need to bump your IOPS, as it generally doesn't impact bulk throughput, and typically doesn't work on asynch transfers (so SMB, unless you tweak it). It's mostly a benefit for database applications taking a hammering and shitting out a fuck ton of tiny writes that the ZIL can more or less cache up and then serve out to the HDDs when the drives are ready to accept them. Even if you did want one, you wouldn't need it to be large--a few GB is generally considered sufficient, and you'd probably want to mirror it. For home use, it's completely unnecessary.

In that the PCH is more or less just a glorified switch these days, I can't see why it would be more likely to lose data than using PCIe lanes straight to the CPU. Performance also shouldn't be impacted unless you are trying to transfer truly monumental amounts of data: assuming you're basing your server off a Haswell C222-based motherboard, the PCH supports 8x PCIe 2.0 lanes, for 4GB/s bandwidth (more like 3GB/s in the real world). Infiniband QDR tops out at ~1GB/s. Sooooo, yeah. Probably fine.

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

I have no opinion on the ZIL. For the home user case it seems useless.

In regards to L2ARC goes, I guess the faster the better. If you run ZVOLs with small volblocksizes, I wouldn't make it too large, since each L2ARC block pointer is like 170 bytes. I have my L2ARC limited to 128GB, all my ZVOLs have 16KB volblocksize, so in the absolute worst case, a full L2ARC only steals ~1.4GB of ARC (system only has a total of 16GB RAM). And I suppose you'd need to disable the l2arc_noprefetch parameter, so it caches everything that gets evicted from ARC (--edit: well, almost everything, there's a feed rate, that defaults to 8MB/s), otherwise it just keeps the first few blocks of common IO entrypoints (how I understand it anyway, i.e. supply the first few blocks from L2ARC while the disks catch up).

Paul MaudDib
May 3, 2006

"Tell me of your home world, Usul"


DrDork posted:

For RAM QVL, you can pretty much ignore it. It's fairly unlikely for you to manage to get RAM that it won't play nice with, unless your motherboard happens to be particularly finicky. What board is it, anyhow?

This is what I'm currently looking at, with a used processor (~$250), and probably only half the RAM to start with and see how it goes (the RAM is actually $63 a stick from Directron). Someday, a quad M.2 sled for L2ARC or database indexes (just like a 960 Evo or something, nothing fancy) and a dual-port InfiniBand QDR adapter.

The proper answer is "buy a rack" of course, but my apartment is laid out like shit (cut into too many small rooms) and there's just no place for it. I'm trying to pare down obvious computers into stuff that can be tucked away reasonably well, where possible. By splitting up storage and processing I go from a huge-ass loud/hot turbo-nerd case, to one normal computer and one normal NAS but really good homelab performance when used together. I like the whole Synology DS1817 form-factor, but this gives me more expandability via NVMe SSDs and InfiniBand. Imagine having an RDMA connection from a workstation to an enormous and fast data store, that would be pretty fun for homelab shit. Right now I'd just be looking for something to throw my drives into, but the cost of the basic build isn't tremendously different from a Synology or something. For the same price I get a much beefier overall device, I get ECC right off the bat, and I keep my options open for future expansion. Since the SATA ports all run on the PCH you have a whole 40 lanes free on the CPU (x16x16x8 - although you only get two 1-slot ATX expansion bulkheads) and a M.2 on the PCH (2.0x2 speed, not great, but it's there), and you can move up to quad-channel RAM if you need. X99 platform, baby.

Case: U-NAS NSC-810A Server Chassis ($219.99)
Motherboard: Asus - X99-M WS Micro ATX LGA2011-3 Motherboard ($270.13 @ Newegg)
CPU: Intel - Xeon E5-2620 V3 2.4GHz 6-Core Processor ($250 on eBay)
CPU Cooler: ID-COOLING - IS-VC45 44.3 CFM CPU Cooler ($48.63 @ Newegg Marketplace)
CPU Fan: Noctua - NF-A9x14 29.7 CFM 92mm Fan ($16.59 @ Newegg)
Memory: Kingston - 8GB (1 x 8GB) Registered DDR4-2133 Memory ($102.83 @ Amazon)
Memory: Kingston - 8GB (1 x 8GB) Registered DDR4-2133 Memory ($102.83 @ Amazon)
Memory: Kingston - 8GB (1 x 8GB) Registered DDR4-2133 Memory ($102.83 @ Amazon)
Memory: Kingston - 8GB (1 x 8GB) Registered DDR4-2133 Memory ($102.83 @ Amazon)
Power Supply: SeaSonic - 300W 80+ Gold Certified Fully-Modular Flex ATX Power Supply ($44.99 @ SuperBiiz)
Case Fan: Noctua - NF-F12 PWM 55.0 CFM 120mm Fan ($19.50 @ Newegg)
Case Fan: Noctua - NF-F12 PWM 55.0 CFM 120mm Fan ($19.50 @ Newegg)

There are a few plausible Xeon E5 models (retail, non-ES/QS) available used at the very bottom end of the used market. The 4-core chips clock higher but heat is going to be a problem in this case. The 2620s clock pretty well on a few cores but really drop under high parallel load, which seems to me to actually be the desirable behavior in a thermal-limited situation. Should do fine regardless, even if that cooler is questionable as hell. There's very little that fits a 2011 in a case with a CPU clearance of 52mm max, this is the less bad of the two that aren't actual server blowers. The Noctua might help a little. Positive pressure will hopefully force the heat out rather than it building up in some spots.



Threadripper would be cool but I think it would mean giving up high clocks to fit in a workable TDP, and the cooler clearance is going to be even more of a pain. Also the build really needs at least 8 SATA ports onboard, or a an onboard secondary controller (Asrock has C2550 and C2750 boards like this, sadly the controller is Marvell), or else you lose one of your two slots to a controller. This is one of just a handful of suitable boards I've dug up. It might take a while before one shows up for TR. If I actually have a legitimate need for it, I can build a TR workstation in a normal case and RDMA to my huge data store.

DrDork posted:

As for "how much SLC do I need for a ZIL" the answer is "probably zero." ZIL only really makes sense if you need to bump your IOPS, as it generally doesn't impact bulk throughput, and typically doesn't work on asynch transfers (so SMB, unless you tweak it). It's mostly a benefit for database applications taking a hammering and shitting out a fuck ton of tiny writes that the ZIL can more or less cache up and then serve out to the HDDs when the drives are ready to accept them. Even if you did want one, you wouldn't need it to be large--a few GB is generally considered sufficient, and you'd probably want to mirror it. For home use, it's completely unnecessary.

In that the PCH is more or less just a glorified switch these days, I can't see why it would be more likely to lose data than using PCIe lanes straight to the CPU. Performance also shouldn't be impacted unless you are trying to transfer truly monumental amounts of data: assuming you're basing your server off a Haswell C222-based motherboard, the PCH supports 8x PCIe 2.0 lanes, for 4GB/s bandwidth (more like 3GB/s in the real world). Infiniband QDR tops out at ~1GB/s. Sooooo, yeah. Probably fine.

I actually just ran into our infrastructure guy on the way out of work and he said the same thing, they just let ZFS handle the ZIL on all their production systems. Of course we just had a SAN disaster a year or so ago, so... (one of the drive controllers in our dev environment started getting flaky over the course of a year or so)

I just wasn't sure if the PCH did any buffering that might lead to trouble with the ZIL. Should be fine then. (I'm mostly just trying to probe as to any future needs for scalability here)

Infiniband QDR tops out at 1 GB/s per link, a standard adapter is either a single or dual 4x port, and you can gang them into an 8x link. Still probably total overkill for home use anyway (I'm really more after the RDMA) and I'm not planning on RAID to start with. But as much as possible I'm trying to leave my options open for scaling upwards if I ever wanted.

Paul MaudDib fucked around with this message at 05:53 on Jul 26, 2017

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

Uh, about RDMA, what cards, what operating systems on the client(s) and on the NAS?

Without RDMA, I still can shovel 28GBit over my 40GBit link with iperf (four threads).

Paul MaudDib
May 3, 2006

"Tell me of your home world, Usul"


Combat Pretzel posted:

Uh, about RDMA, what cards, what operating systems on the client(s) and on the NAS?

Oh, I guess I should have realized that might be a problem.

Either FreeBSD, Solaris, or Illumos on the NAS, and I'd figured on some flavor of linux server OS on the clients, like Ubuntu or CentOS.

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

In that case, have fun, SRP or iSER should do fine. If you were wanting to use RDMA with Windows, you'd probably have to rely on outdated drivers or something else. Microsoft coerced Mellanox to drop SRP support on their driver stack, which left me a little salty. Too much fiddling to get outdated drivers to work. While Samba has a development branch with SMB Direct support somewhere, it was last updated Dec 2016, and I can't find any progress reports. On the other hand, it'd be nice for Microsoft to implement iSER in their initiator, but hell will freeze over before that happens, due to aforementioned coercion.

Paul MaudDib
May 3, 2006

"Tell me of your home world, Usul"


My thought with InfiniBand was that IB RDMA should provide a good mechanism for database-as-a-service or a storage backend for self-hosted databases on other machines. That's part of my goal here with leaving myself shitloads of RAM channels/capacity and PCIe lanes for NVMe expandability. Is that a difficult/bad use-case?

Someone mentioned that Shadow Copy Service can use ZFS as a snapshot mechanism, that seems pretty cool for backups too. Right now I manage my backups manually and I'd really like to get things set up better. Would that be impacted by the Windows driver problems you mentioned? I suppose backups can always run over Ethernet if it comes down to it, but it would be nice to get IB from Windows if I could.

Paul MaudDib fucked around with this message at 02:15 on Jul 26, 2017

Combat Pretzel
Jun 23, 2004

No, seriously... what kurds?!

RDMA is really just a different transport. Not sure how exactly SRP works, but when iSCSI or SMB3 do negotiate RDMA via iSER and SMB Direct respectively, they yank out the TCP/IP transport from under the connection and replace it with RDMA (loosely speaking). On the outside, their operation is still the same to the application/OS.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«608 »