«608 »
  • Post
  • Reply
Red_Fred
Oct 21, 2010



Fallen Rib

Thanks this is all super helpful. Are there any recommended guides out there for the full setup?

D. Ebdrup
Mar 13, 2009



Red_Fred posted:

Thanks this is all super helpful. Are there any recommended guides out there for the full setup?
FreeNAS has a whole user-guide (probably inspired, at least in part, by the FreeBSD Handbook).

Matt Zerella
Oct 7, 2002


If you're at all interested in UnRAID, check out spaceinvaderone on YouTube.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

MagusDraco posted:

FreeNAS does have some hardware requirements/heavily suggests you use ECC Ram or at least did at the time I was considering using FreeNAS

Yeah, that's mostly because ZFS assumes that what it's getting from RAM is legit, and in theory it can quietly corrupt things if your RAM takes a shit. In reality it's more likely that a faulty RAM stick will just take the entire system down, though, and if you're just storing it's not that big of a risk anyhow, and RAID Is Not Backup so you should have other copies, anyhow.

Really it's a sticker warning saying "If you use non-ECC RAM in a production enterprise environment and something goes wrong, don't blame us."

Zorak of Michigan
Jun 10, 2006

Waiting for his chance

I think it's also an internet reaction to ZFS' strong data integrity guarantees. People see that and assert "this ZFS configuration means your data is safer from corruption than your home in Montana is from tsunamis," and then someone else feels the need to chip in and say, "yes, that's true on disk, but without ECC, your data is only five nines safe, since it can still be corrupted in RAM," and then by the next forum page, it's been hyped into "if you do not use ECC with ZFS, you may as well keep your data on 88k floppies stored in the microwave."

H110Hawk
Dec 28, 2006


Zorak of Michigan posted:

I think it's also an internet reaction to ZFS' strong data integrity guarantees. People see that and assert "this ZFS configuration means your data is safer from corruption than your home in Montana is from tsunamis," and then someone else feels the need to chip in and say, "yes, that's true on disk, but without ECC, your data is only five nines safe, since it can still be corrupted in RAM," and then by the next forum page, it's been hyped into "if you do not use ECC with ZFS, you may as well keep your data on 88k floppies stored in the microwave."

This one time, one bit in a 12megapixel picture got flipped and

G-Prime
Apr 30, 2003

Baby, when it's love,
if it's not rough it isn't fun.


This one time a bit in a 4KB binary got flipped and

IOwnCalculus
Apr 2, 2003





Just found out I can finagle another 4U of space.

Also found out it's cheaper to buy a Netapp DS4243 and swap the SAS controller in it for a different Xyratex unit, than it is to buy a SAS expander and cables for the 20-bay I already have.

Anyone want to buy a Norco 4020 setup?

Violator
May 15, 2003



How important is it to let spinning drives to go to sleep? I have two externals (one for macOS Time Machine and another as a temp file repo) and them waking up periodically throughout the day stalls everything for a few seconds while I'm working.

H110Hawk
Dec 28, 2006


Violator posted:

How important is it to let spinning drives to go to sleep? I have two externals (one for macOS Time Machine and another as a temp file repo) and them waking up periodically throughout the day stalls everything for a few seconds while I'm working.

It's not.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll

Nap Ghost

Eh, one bit flip is corrected with ECC memory and I found scores of them in my BIOS logs before. If you don’t correct it your OS should normally reboot / halt unless you have a crazy setting to ignore ECC correction errors. Parity bits are sufficient for detecting 2+ bit errors and correcting 1 bit errors. The bigger problem in practice is faulty RAM causing random reboots on servers rather than bit flips from cosmic rays unless you’re at NSA’s datacenter in Utah in the goddamn mountains.

Violator posted:

How important is it to let spinning drives to go to sleep? I have two externals (one for macOS Time Machine and another as a temp file repo) and them waking up periodically throughout the day stalls everything for a few seconds while I'm working.
Spin-up and spin down frequently is where mechanical hard drives can have faults. Professionally, I’ve had the most drives fail when they’ve run continuously for 6+ years and finally shut down only to realize they will never spin up again. It’s fine to have spin-up and down a few times a day for home machines. People really only have problems when it happens hundreds of times a day. Drives are rated for many thousands of spin-ups and downs.

H110Hawk
Dec 28, 2006


necrobobsledder posted:

Eh, one bit flip is corrected with ECC memory and I found scores of them in my BIOS logs before. If you don’t correct it your OS should normally reboot / halt unless you have a crazy setting to ignore ECC correction errors. Parity bits are sufficient for detecting 2+ bit errors and correcting 1 bit errors. The bigger problem in practice is faulty RAM causing random reboots on servers rather than bit flips from cosmic rays unless you’re at NSA’s datacenter in Utah in the goddamn mountains.

Non-ECC ram doesn't report errors because it cannot know about them. My point is the odds of this happening and causing data corruption to the point of you even noticing is incredibly slim. This may change as full disk encryption becomes the norm.

D. Ebdrup
Mar 13, 2009



All this talk about ECC is missing one important point: Where ZFS was designed, which is to say the early-2000s Sun machines, non-ECC memory wasn't an option. The reason it wasn't an option is not because ECC is magic, but because Sun used to do Big Iron where reliability, availability and servicability guides are so prevalent that not only do you have ECC, you also have RAIM (either memory mirroring, or distributed parity with lockstepping). ECC is as much about ensuring the system availability, as much as it is about data reliability.
Besides, ZFS has a debug option that enables checksumming even in memory - and while I'm not going to say that nobody uses it, I certainly never heard of anyone who's that paranoid.

At least with ZFS, you know which file(s) exhibit corruption and can restore from the backup that you ought to have no matter what. A lot of other solutions, including some very expensive hardware ones, can't do that.

H110Hawk posted:

Non-ECC ram doesn't report errors because it cannot know about them. My point is the odds of this happening and causing data corruption to the point of you even noticing is incredibly slim. This may change as full disk encryption becomes the norm.
On systems where transparent per-dataset or full-disk encryption is the norm, I would strongly hope that they're using encryption which contains a HMAC, as well as checksumming that doesn't collide. Currently, this means SHA256-only, and unfortunately it's not even SHA512/t so it'd better be offloaded - preferably in a way that can also handle AES-GCM which indludes encryption and HMAC in one step.

D. Ebdrup fucked around with this message at 20:59 on Jan 11, 2019

H110Hawk
Dec 28, 2006


D. Ebdrup posted:

On systems where transparent per-dataset or full-disk encryption is the norm, I would strongly hope that they're using encryption which contains a HMAC, as well as checksumming that doesn't ever collide (*). Currently, this means SHA256-only, and unfortunately it's not even SHA512/t so it'd better be offloaded - preferably in a way that can also handle AES-GCM which indludes encryption and HMAC in one step.

You're going to notice if you get a 1-bit flip post HMAC computation upon read as you are likely to get an AES block that is entirely illegible to the system or fails HMAC. It's going to be a lot worse than just losing a bit in a jpeg.

D. Ebdrup
Mar 13, 2009



H110Hawk posted:

You're going to notice if you get a 1-bit flip post HMAC computation upon read as you are likely to get an AES block that is entirely illegible to the system or fails HMAC. It's going to be a lot worse than just losing a bit in a jpeg.
It's nowhere near that simple. Even assuming someone is foolish enough to be dealing with asyncronous data on mission critical data, the data stays in memory for a maximum of 5 seconds (which is when the default length of a transaction group expires, but it can be much sooner if a data threshold is met before that). Once the ZIL is full or the 5 seconds have passed, the data gets checsummed and written on-disk (where it's either some form of n-way mirroring or distributed parity up to three levels deep, with the often-used option of ditto-blocks - at least when it comes to mission-critical data).

The (*) from before was meant to be a postscript about how SHA256 isn't guarenteed to be cryptographically secure forever, but

D. Ebdrup fucked around with this message at 21:10 on Jan 11, 2019

H110Hawk
Dec 28, 2006


D. Ebdrup posted:

It's nowhere near that simple. Even assuming someone is foolish enough to be dealing with asyncronous data on mission critical data, the data stays in memory for a maximum of 5 seconds (which is when the default length of a transaction group expires, but it can be much sooner if a data threshold is met before that). Once the ZIL is full or the 5 seconds have passed, the data gets checsummed and written on-disk (where it's either some form of n-way mirroring or distributed parity up to three levels deep, with the often-used option of ditto-blocks - at least when it comes to mission-critical data).

The (*) from before was meant to be a postscript about how SHA256 isn't guarenteed to be cryptographically secure forever, but

Oops I wasn't thinking about ZFS just general purpose. Interesting to learn though! Just in general for home use ECC is typically going to be overkill. Even for your most precious pictures.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll

Nap Ghost

Cryptographically secure hashes are a completely different set of algorithms than ones built for error correction and detection such as Hamming and Solomon-Reed and not just from a performance standpoint. If someone is attacking your ZFS caches there are probably better attack vectors to choose. And if we’re talking RAM that’s insecure and trying to corrupt data by exfilatrating it in strange ways you’ll need some serious crazy engineering work to get through the various VFS and ZFS layers and to be resilient across different versions of OSes and library versions, too.

General purpose memory transactions on a heterogeneous workload system mean a lot of other things would go wrong before anything related to ZFS or software RAID in general. But with a system entirely dedicated to high throughput I/O just plain statistics imply you’re going to have a higher chance of data corruption from a bad memory transaction. This is part of why Matt Ahrens has gone on record to state that ZFS is no more susceptible to memory problems that normal software RAID would be.

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast


H110Hawk posted:

Oops I wasn't thinking about ZFS just general purpose. Interesting to learn though! Just in general for home use ECC is typically going to be overkill. Even for your most precious pictures.

I don't see why it's overkill.. I have a couple of denverton based NASes, 4GB of ECC RAM wasn't exactly a great expense. I guess it depends on the hardware and the amount you need, though

IOwnCalculus
Apr 2, 2003





It's not when you're buying new (or used) gear, but compared to trying to repurpose existing gear for $0 it's a different story.

bobfather
Sep 20, 2001

I will analyze your nervous system for beer money

Man, Stablebit DrivePool is really good. And so are CloudDrive and Scanner.

I just retired my FreeNAS install and don’t miss it a bit. So much nicer to manage everything on WS2016.

redeyes
Sep 14, 2002
I LOVE THE WHITE STRIPES!

bobfather posted:

Man, Stablebit DrivePool is really good. And so are CloudDrive and Scanner.

I just retired my FreeNAS install and don’t miss it a bit. So much nicer to manage everything on WS2016.

I completely agree. For windows NAS stuff it is the best for a home user. So easy to use and manage and everything. Never had a problem or lost data and it's been like 8 years.

eames
May 9, 2009



Is it normal for Synology to have such long delays between product announcement and availability?
They announced the DS619slim in June and it's not even listed for sale yet.

MagusDraco
Nov 11, 2011

even speedwagon was trolled


bobfather posted:

Man, Stablebit DrivePool is really good. And so are CloudDrive and Scanner.

I just retired my FreeNAS install and don’t miss it a bit. So much nicer to manage everything on WS2016.

How does CloudDrive work. I only just realized I purchased a license of it with drivepool and scanner two years ago but never really set it up and forgot all about it

bobfather
Sep 20, 2001

I will analyze your nervous system for beer money

MagusDraco posted:

How does CloudDrive work. I only just realized I purchased a license of it with drivepool and scanner two years ago but never really set it up and forgot all about it

You can set up a cloud storage provider as a local mount. You can cache whatever you want locally for quick access. You can also encrypt the remote files.

Basically it’s a real-time cloud backup.

FCKGW
May 21, 2006

aaaaaaaaaa
AAAAAAAAAAA
HHHHHHHHH!!!!!!!!




If you pair it with DrivePool you can also pool multiple cloud drives into a single pool or make the cloud drives part of your local pool and specify which folders are backed up local and in the cloud. It's pretty neat.

slidebite
Nov 6, 2005

Good egg


I'm fairly new in to the NAS thing and recently picked up a Netgear ReadyNAS 2120 that I have hooked up and chugging along on my home network.

What is the best way to access the files remotely, off my network? Like say I'm in a hotel or something? Netgear's ReadyCloud?

H110Hawk
Dec 28, 2006


slidebite posted:

I'm fairly new in to the NAS thing and recently picked up a Netgear ReadyNAS 2120 that I have hooked up and chugging along on my home network.

What is the best way to access the files remotely, off my network? Like say I'm in a hotel or something? Netgear's ReadyCloud?

Remember them before you leave. Sync them to Google drive. Literally anything other than "Companyname Cloud Access" or poking a hole in your firewall (short of VPN).

Schadenboner
Aug 15, 2011

I MEAN, TURN OFF YOURE MONITOR, MIGTH EXPLAIN YOUR BAD POSTS, HOPE THIS HELPS?!

So I’ve never really used Synologies much beyond set it-and-forget it for some customers with very basic storage needs.

How much access does the user have to the underlying hardware? A quad-core intel with 8 RAM could be a decent SOHO DNS/PiHole box if you can use it (especially if you can run off the SSD rather than the storage drives)?

Schadenboner fucked around with this message at 21:31 on Jan 12, 2019

ChubbyPitbull
Dec 10, 2005
Awww....look how OHMYGODMYHAND!

So following the recommendations from earlier in the thread I'm now all set up with a Synology DS218play with 2x 8TB drives ripped out of some EasyStores. And.....I love it. I was originally just looking for some large centralized storage I could back up and I'm just now finding out all the new toys that are now on the table.

I got the shares set up and automatically backing up via HyperBackup to Google Drive. Everyone in the household is now able to store and backup their files. After everyone pooled their music and video together, I found out about Plex and now have that running as well. It's been awesome! We had an older smart TV that still had the Plex app so I got a Plex pass so people at the house can watch or listen on whatever device they want.

What I've been trying to do now is see about getting Gitea set up on the DS218play for easy project tracking tied into the automatic cloud backup. I found this project here: https://github.com/jboxberger/synology-gitea-jboxberger , but when I went to manually install the SPK it complained about not having Docker available. Googling around, it appears that the DS218play is one of a handful of Synologies that doesn't support Docker? The DS218+ appears to. I believe Docker supports the Armv8 processor on the DS218play, but Synology only provides Docker support for their Intel CPUs. Is there anyway to manually get a version of Docker running on the DS218play or am I SOL?

H110Hawk
Dec 28, 2006


Schadenboner posted:

So I’ve never really used Synologies much beyond set it-and-forget it for some customers with very basic storage needs.

How much access does the user have to the underlying hardware? A quad-core intel with 8 RAM could be a decent SOHO DNS/PiHole box if you can use it (especially if you can run off the SSD rather than the storage drives)?

Yes you can run docker and stuff. Just make sure you know which arch you are getting (arm, x86_64?). Why do you care which disk your dns/pihole runs on? Neither are going to utilize any I/O worth measuring.

Schadenboner
Aug 15, 2011

I MEAN, TURN OFF YOURE MONITOR, MIGTH EXPLAIN YOUR BAD POSTS, HOPE THIS HELPS?!

H110Hawk posted:

Yes you can run docker and stuff. Just make sure you know which arch you are getting (arm, x86_64?). Why do you care which disk your dns/pihole runs on? Neither are going to utilize any I/O worth measuring.

I dunno, storage is storage and application is application?



Good to know though.

The 619 slim looks like it's gonna be sexy af if it ever releases?

H110Hawk
Dec 28, 2006


Schadenboner posted:

I dunno, storage is storage and application is application?



Good to know though.

The 619 slim looks like it's gonna be sexy af if it ever releases?

Segregation is fine and all but for something meant to run from a sdcard don't worry about it. Overall you're likely constrained by disk slots so don't waste em.

D. Ebdrup
Mar 13, 2009



I think there's a potential for Seagate to be onto something with ~4TB SATA3 SSDs:

Let's hope they can actually push the prices down enough for them to be worth buying.

Hed
Mar 31, 2004



Fun Shoe

That’s great. I don’t know enough about the designs to guess but will be interesting to see how design decisions of ZFS and btrfs will be limiting / open up new possibilities as access latency and throughout reach levels perhaps not considered during their inception.

uhhhhahhhhohahhh
Oct 9, 2012


D. Ebdrup posted:


Let's hope they can actually push the prices down enough for them to be worth buying.


Don't count on it

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness


While I have no doubt that they'll be pricey, I wouldn't bother even looking at a UK pre-release price, especially when Seagate hasn't officially stated prices or availability.

That said, that price is still 100-200 less than what Amazon.uk has the 860 Evo 4TB listed at.

kazmeyer
Jul 26, 2001

'Cause we're the good guys.



Finally had a WD black fail on me. First time I've ever had WD fail on me before I got rid of the drive.

At least it failed on me in such a way I was able to get off a few pieces of irreplaceable data before it completely ate shit.

And at least it ate shit a good six months before that wonderful five-year warranty was up.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll

Nap Ghost

It's going to be the 10th birthday of my 1 TB WD Black I got for just a tad over $110. That's kind of impressive how poorly drives advanced in the past 10 years price-capacity wise.

Avenging Dentist
Oct 1, 2005

oh my god is that a circular saw that does not go in my mouth aaaaagh

At the risk of asking a silly question, does the following look sane? My use cases are to serve media to our Kodi box (no need for transcoding), backing up 2 PCs, and possibly serving media to VLC on mobile devices. I don't have anything I'd consider truly irreplaceable aside from like my password database, but obviously minimizing the amount of stuff I lose would be nice. The plan is to use RAID 6 and whatever I need to get file-level healing (btrfs I think?). I just want something reasonably plug-and-play, since I already have more than enough technology to janitor.

Synology DS918+
WD Red Pro 6TB WD6003FFBX x4
APC 600VA UPS

Some followup questions:
  • Windows 10 File History vs the old Win 7-style Backup and Restore? Is one particularly better than the other? (e: both PCs are on Win10 Pro, so I should be able to do network backups as I understand it.)
  • Any tips on doing cloud backups from Synology? The stuff I'd actually want to backup is probably 10 GB for "essentials", 100 GB for stuff I'd really prefer to keep, and 3 TB if I backed up all my media (I'm not much of a packrat by this thread's standards).
  • How easy/feasible is it to keep the Synology locked down from people trying to break into it? I saw there were a number of vulnerabilities, but I'm not sure if they'd only be relevant if I were letting people log into the Synology remotely.

Avenging Dentist fucked around with this message at 06:35 on Jan 15, 2019

astral
Apr 26, 2004



Rather than 4x6TB red pros, what about shucking 4x 10TB easystores @ 180ish each from Best Buy?

https://www.bestbuy.com/site/wd-eas...p?skuId=6290669

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«608 »