|
Thanks this is all super helpful. Are there any recommended guides out there for the full setup?
|
![]() |
|
Red_Fred posted:Thanks this is all super helpful. Are there any recommended guides out there for the full setup?
|
![]() |
|
If you're at all interested in UnRAID, check out spaceinvaderone on YouTube.
|
![]() |
|
MagusDraco posted:FreeNAS does have some hardware requirements/heavily suggests you use ECC Ram or at least did at the time I was considering using FreeNAS Yeah, that's mostly because ZFS assumes that what it's getting from RAM is legit, and in theory it can quietly corrupt things if your RAM takes a shit. In reality it's more likely that a faulty RAM stick will just take the entire system down, though, and if you're just storing ![]() Really it's a sticker warning saying "If you use non-ECC RAM in a production enterprise environment and something goes wrong, don't blame us."
|
![]() |
|
I think it's also an internet reaction to ZFS' strong data integrity guarantees. People see that and assert "this ZFS configuration means your data is safer from corruption than your home in Montana is from tsunamis," and then someone else feels the need to chip in and say, "yes, that's true on disk, but without ECC, your data is only five nines safe, since it can still be corrupted in RAM," and then by the next forum page, it's been hyped into "if you do not use ECC with ZFS, you may as well keep your data on 88k floppies stored in the microwave."
|
![]() |
|
Zorak of Michigan posted:I think it's also an internet reaction to ZFS' strong data integrity guarantees. People see that and assert "this ZFS configuration means your data is safer from corruption than your home in Montana is from tsunamis," and then someone else feels the need to chip in and say, "yes, that's true on disk, but without ECC, your data is only five nines safe, since it can still be corrupted in RAM," and then by the next forum page, it's been hyped into "if you do not use ECC with ZFS, you may as well keep your data on 88k floppies stored in the microwave." This one time, one bit in a 12megapixel picture got flipped and
|
![]() |
|
This one time a bit in a 4KB binary got flipped and
|
![]() |
|
Just found out I can finagle another 4U of space. Also found out it's cheaper to buy a Netapp DS4243 and swap the SAS controller in it for a different Xyratex unit, than it is to buy a SAS expander and cables for the 20-bay I already have. Anyone want to buy a Norco 4020 setup? ![]()
|
![]() |
|
How important is it to let spinning drives to go to sleep? I have two externals (one for macOS Time Machine and another as a temp file repo) and them waking up periodically throughout the day stalls everything for a few seconds while I'm working.
|
![]() |
|
Violator posted:How important is it to let spinning drives to go to sleep? I have two externals (one for macOS Time Machine and another as a temp file repo) and them waking up periodically throughout the day stalls everything for a few seconds while I'm working. It's not.
|
![]() |
|
Eh, one bit flip is corrected with ECC memory and I found scores of them in my BIOS logs before. If you dont correct it your OS should normally reboot / halt unless you have a crazy setting to ignore ECC correction errors. Parity bits are sufficient for detecting 2+ bit errors and correcting 1 bit errors. The bigger problem in practice is faulty RAM causing random reboots on servers rather than bit flips from cosmic rays unless youre at NSAs datacenter in Utah in the goddamn mountains.Violator posted:How important is it to let spinning drives to go to sleep? I have two externals (one for macOS Time Machine and another as a temp file repo) and them waking up periodically throughout the day stalls everything for a few seconds while I'm working.
|
![]() |
|
necrobobsledder posted:Eh, one bit flip is corrected with ECC memory and I found scores of them in my BIOS logs before. If you dont correct it your OS should normally reboot / halt unless you have a crazy setting to ignore ECC correction errors. Parity bits are sufficient for detecting 2+ bit errors and correcting 1 bit errors. The bigger problem in practice is faulty RAM causing random reboots on servers rather than bit flips from cosmic rays unless youre at NSAs datacenter in Utah in the goddamn mountains. Non-ECC ram doesn't report errors because it cannot know about them. My point is the odds of this happening and causing data corruption to the point of you even noticing is incredibly slim. This may change as full disk encryption becomes the norm.
|
![]() |
|
All this talk about ECC is missing one important point: Where ZFS was designed, which is to say the early-2000s Sun machines, non-ECC memory wasn't an option. The reason it wasn't an option is not because ECC is magic, but because Sun used to do Big Iron where reliability, availability and servicability guides are so prevalent that not only do you have ECC, you also have RAIM (either memory mirroring, or distributed parity with lockstepping). ECC is as much about ensuring the system availability, as much as it is about data reliability. Besides, ZFS has a debug option that enables checksumming even in memory - and while I'm not going to say that nobody uses it, I certainly never heard of anyone who's that paranoid. At least with ZFS, you know which file(s) exhibit corruption and can restore from the backup that you ought to have no matter what. A lot of other solutions, including some very expensive hardware ones, can't do that. H110Hawk posted:Non-ECC ram doesn't report errors because it cannot know about them. My point is the odds of this happening and causing data corruption to the point of you even noticing is incredibly slim. This may change as full disk encryption becomes the norm. D. Ebdrup fucked around with this message at 20:59 on Jan 11, 2019 |
![]() |
|
D. Ebdrup posted:On systems where transparent per-dataset or full-disk encryption is the norm, I would strongly hope that they're using encryption which contains a HMAC, as well as checksumming that doesn't ever collide (*). Currently, this means SHA256-only, and unfortunately it's not even SHA512/t so it'd better be offloaded - preferably in a way that can also handle AES-GCM which indludes encryption and HMAC in one step. You're going to notice if you get a 1-bit flip post HMAC computation upon read as you are likely to get an AES block that is entirely illegible to the system or fails HMAC. It's going to be a lot worse than just losing a bit in a jpeg.
|
![]() |
|
H110Hawk posted:You're going to notice if you get a 1-bit flip post HMAC computation upon read as you are likely to get an AES block that is entirely illegible to the system or fails HMAC. It's going to be a lot worse than just losing a bit in a jpeg. The (*) from before was meant to be a postscript about how SHA256 isn't guarenteed to be cryptographically secure forever, but ![]() D. Ebdrup fucked around with this message at 21:10 on Jan 11, 2019 |
![]() |
|
D. Ebdrup posted:It's nowhere near that simple. Even assuming someone is foolish enough to be dealing with asyncronous data on mission critical data, the data stays in memory for a maximum of 5 seconds (which is when the default length of a transaction group expires, but it can be much sooner if a data threshold is met before that). Once the ZIL is full or the 5 seconds have passed, the data gets checsummed and written on-disk (where it's either some form of n-way mirroring or distributed parity up to three levels deep, with the often-used option of ditto-blocks - at least when it comes to mission-critical data). Oops I wasn't thinking about ZFS just general purpose. Interesting to learn though! Just in general for home use ECC is typically going to be overkill. Even for your most precious pictures.
|
![]() |
|
Cryptographically secure hashes are a completely different set of algorithms than ones built for error correction and detection such as Hamming and Solomon-Reed and not just from a performance standpoint. If someone is attacking your ZFS caches there are probably better attack vectors to choose. And if were talking RAM thats insecure and trying to corrupt data by exfilatrating it in strange ways youll need some serious crazy engineering work to get through the various VFS and ZFS layers and to be resilient across different versions of OSes and library versions, too. General purpose memory transactions on a heterogeneous workload system mean a lot of other things would go wrong before anything related to ZFS or software RAID in general. But with a system entirely dedicated to high throughput I/O just plain statistics imply youre going to have a higher chance of data corruption from a bad memory transaction. This is part of why Matt Ahrens has gone on record to state that ZFS is no more susceptible to memory problems that normal software RAID would be.
|
![]() |
|
H110Hawk posted:Oops I wasn't thinking about ZFS just general purpose. Interesting to learn though! Just in general for home use ECC is typically going to be overkill. Even for your most precious pictures. I don't see why it's overkill.. I have a couple of denverton based NASes, 4GB of ECC RAM wasn't exactly a great expense. I guess it depends on the hardware and the amount you need, though
|
![]() |
|
It's not when you're buying new (or used) gear, but compared to trying to repurpose existing gear for $0 it's a different story.
|
![]() |
|
Man, Stablebit DrivePool is really good. And so are CloudDrive and Scanner. I just retired my FreeNAS install and dont miss it a bit. So much nicer to manage everything on WS2016.
|
![]() |
|
bobfather posted:Man, Stablebit DrivePool is really good. And so are CloudDrive and Scanner. I completely agree. For windows NAS stuff it is the best for a home user. So easy to use and manage and everything. Never had a problem or lost data and it's been like 8 years.
|
![]() |
|
Is it normal for Synology to have such long delays between product announcement and availability? They announced the DS619slim in June and it's not even listed for sale yet.
|
![]() |
|
bobfather posted:Man, Stablebit DrivePool is really good. And so are CloudDrive and Scanner. How does CloudDrive work. I only just realized I purchased a license of it with drivepool and scanner two years ago but never really set it up and forgot all about it
|
![]() |
|
MagusDraco posted:How does CloudDrive work. I only just realized I purchased a license of it with drivepool and scanner two years ago but never really set it up and forgot all about it You can set up a cloud storage provider as a local mount. You can cache whatever you want locally for quick access. You can also encrypt the remote files. Basically its a real-time cloud backup.
|
![]() |
|
If you pair it with DrivePool you can also pool multiple cloud drives into a single pool or make the cloud drives part of your local pool and specify which folders are backed up local and in the cloud. It's pretty neat.
|
![]() |
|
I'm fairly new in to the NAS thing and recently picked up a Netgear ReadyNAS 2120 that I have hooked up and chugging along on my home network. What is the best way to access the files remotely, off my network? Like say I'm in a hotel or something? Netgear's ReadyCloud?
|
![]() |
|
slidebite posted:I'm fairly new in to the NAS thing and recently picked up a Netgear ReadyNAS 2120 that I have hooked up and chugging along on my home network. Remember them before you leave. Sync them to Google drive. Literally anything other than "Companyname Cloud Access" or poking a hole in your firewall (short of VPN).
|
![]() |
|
So Ive never really used Synologies much beyond set it-and-forget it for some customers with very basic storage needs. How much access does the user have to the underlying hardware? A quad-core intel with 8 RAM could be a decent SOHO DNS/PiHole box if you can use it (especially if you can run off the SSD rather than the storage drives)? Schadenboner fucked around with this message at 21:31 on Jan 12, 2019 |
![]() |
|
So following the recommendations from earlier in the thread I'm now all set up with a Synology DS218play with 2x 8TB drives ripped out of some EasyStores. And.....I love it. I was originally just looking for some large centralized storage I could back up and I'm just now finding out all the new toys that are now on the table. I got the shares set up and automatically backing up via HyperBackup to Google Drive. Everyone in the household is now able to store and backup their files. After everyone pooled their music and video together, I found out about Plex and now have that running as well. It's been awesome! We had an older smart TV that still had the Plex app so I got a Plex pass so people at the house can watch or listen on whatever device they want. What I've been trying to do now is see about getting Gitea set up on the DS218play for easy project tracking tied into the automatic cloud backup. I found this project here: https://github.com/jboxberger/synology-gitea-jboxberger , but when I went to manually install the SPK it complained about not having Docker available. Googling around, it appears that the DS218play is one of a handful of Synologies that doesn't support Docker? The DS218+ appears to. I believe Docker supports the Armv8 processor on the DS218play, but Synology only provides Docker support for their Intel CPUs. Is there anyway to manually get a version of Docker running on the DS218play or am I SOL?
|
![]() |
|
Schadenboner posted:So I’ve never really used Synologies much beyond set it-and-forget it for some customers with very basic storage needs. Yes you can run docker and stuff. Just make sure you know which arch you are getting (arm, x86_64?). Why do you care which disk your dns/pihole runs on? Neither are going to utilize any I/O worth measuring.
|
![]() |
|
H110Hawk posted:Yes you can run docker and stuff. Just make sure you know which arch you are getting (arm, x86_64?). Why do you care which disk your dns/pihole runs on? Neither are going to utilize any I/O worth measuring. I dunno, storage is storage and application is application? ![]() Good to know though. The 619 slim looks like it's gonna be sexy af if it ever releases?
|
![]() |
|
Schadenboner posted:I dunno, storage is storage and application is application? Segregation is fine and all but for something meant to run from a sdcard don't worry about it. Overall you're likely constrained by disk slots so don't waste em.
|
![]() |
|
I think there's a potential for Seagate to be onto something with ~4TB SATA3 SSDs:![]() Let's hope they can actually push the prices down enough for them to be worth buying.
|
![]() |
|
Thats great. I dont know enough about the designs to guess but will be interesting to see how design decisions of ZFS and btrfs will be limiting / open up new possibilities as access latency and throughout reach levels perhaps not considered during their inception.
|
![]() |
|
D. Ebdrup posted:
Don't count on it
|
![]() |
|
While I have no doubt that they'll be pricey, I wouldn't bother even looking at a UK pre-release price, especially when Seagate hasn't officially stated prices or availability. That said, that price is still 100-200 less than what Amazon.uk has the 860 Evo 4TB listed at.
|
![]() |
|
Finally had a WD black fail on me. First time I've ever had WD fail on me before I got rid of the drive. At least it failed on me in such a way I was able to get off a few pieces of irreplaceable data before it completely ate shit. And at least it ate shit a good six months before that wonderful five-year warranty was up.
|
![]() |
|
It's going to be the 10th birthday of my 1 TB WD Black I got for just a tad over $110. That's kind of impressive how poorly drives advanced in the past 10 years price-capacity wise.
|
![]() |
|
At the risk of asking a silly question, does the following look sane? My use cases are to serve media to our Kodi box (no need for transcoding), backing up 2 PCs, and possibly serving media to VLC on mobile devices. I don't have anything I'd consider truly irreplaceable aside from like my password database, but obviously minimizing the amount of stuff I lose would be nice. The plan is to use RAID 6 and whatever I need to get file-level healing (btrfs I think?). I just want something reasonably plug-and-play, since I already have more than enough technology to janitor. Synology DS918+ WD Red Pro 6TB WD6003FFBX x4 APC 600VA UPS Some followup questions:
Avenging Dentist fucked around with this message at 06:35 on Jan 15, 2019 |
![]() |
|
Rather than 4x6TB red pros, what about shucking 4x 10TB easystores @ 180ish each from Best Buy? https://www.bestbuy.com/site/wd-eas...p?skuId=6290669
|
![]() |