«608 »
  • Post
  • Reply
Mister Speaker
May 8, 2007



Fallen Rib

What should I do for a backup solution for my Mac?

I've got four 6TB HDDs and four 1TB SSDs in my 5,1 tower, so yeah, I'm looking for something sizeable. A friend recommended I get a USB3 PCI card and some sort of RAID enclosure, but I'm sure there are other options.

H110Hawk
Dec 28, 2006


Mister Speaker posted:

What should I do for a backup solution for my Mac?

I've got four 6TB HDDs and four 1TB SSDs in my 5,1 tower, so yeah, I'm looking for something sizeable. A friend recommended I get a USB3 PCI card and some sort of RAID enclosure, but I'm sure there are other options.

Not backup your anime/ . Sign up for backblaze with the rest. (Unless you're doing professional video/photo in which case I rescind my snarky remark.)

Mister Speaker
May 8, 2007



Fallen Rib

I am doing audio work mostly, but I'm getting into some photo and video editing.

H110Hawk
Dec 28, 2006


Mister Speaker posted:

I am doing audio work mostly, but I'm getting into some photo and video editing.

How much new content are you taking in daily? You're at 25 TB right now, and really those 4x6T disks would be really well served by a raid array just to give you a homogeneous chunk of space. If you want to backup locally then I would get a NAS, move my live work to that, then backup to the individual disks for accidental deletion protection.

For remote backup (fire, theft protection) Assuming you don't take in more than a few gigs a day as an annualized average something like Backblaze personal on your desktop or B2 on your NAS would get you backed up. This is mostly dependent on your internet connection. You can set things like lifecycle rules if that's important to you, for example: Do not delete for 90 days, keep versions for 30 days. (This protects you from a cryptolocker style attack, as the remote end refuses to delete the previous versions of the file. Otherwise a Google Drive Unlimited subscription could also work.

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

ESOS isn't working out. Not really sure what the issue is, I've setup iSCSI before, but not with ESOS.

Gave it a Target, gave it a LUN/Disk, assigned it to the Target, gave it initiators, and Xen keeps coming back with 'No Portal Found', and I can see from the logs on the iSCSI that its successfully connecting, almost like ESOS is failing to share the LUN with Xen for mapping.

D. Ebdrup
Mar 13, 2009



CommieGIR posted:

ESOS isn't working out. Not really sure what the issue is, I've setup iSCSI before, but not with ESOS.

Gave it a Target, gave it a LUN/Disk, assigned it to the Target, gave it initiators, and Xen keeps coming back with 'No Portal Found', and I can see from the logs on the iSCSI that its successfully connecting, almost like ESOS is failing to share the LUN with Xen for mapping.
Now try it with FreeBSD

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

D. Ebdrup posted:

Now try it with FreeBSD

The HP P800 doesnt support JBOD, so I'm worried about using ZFS on top of hardware RAID.

So what I did was RAID-0 pairs of disks together, so that ZFS can treat all the individual pairs as a single disk, and ZFS can handle parity, so when a single disk in a pair fails, that vdisk fails, and avoids nested parity redundancy.

Easier than getting a JBOD controller for now

Setting up FreeNAS now.

CommieGIR fucked around with this message at 21:46 on Jun 8, 2019

H2SO4
Sep 11, 2001

put your money in a log cabin




Buglord

man y'all really hell bent on using zfs with raid controllers despite the big shiny warnings telling you not to, huh

D. Ebdrup
Mar 13, 2009



I waited (for way too long, it feels like), to get a LSI SAS 9211-8i - but I had no choice, because I refuse to put ZFS on RAID controllers, and I couldn't afford the used ones I could find.
The one I finally found that was cheap enough, also came pre-flashed with the SAS2008 IT-mode firmware.

So now I have a buildserver. It's running poudriere configured to use tmpfs for all of the builder data (including the builders themselves), 10GB tmpfs for ccache (which I may up to 20GB), and in case there's any data that's not cached by ccache, so it gets caught by ZFS' ARC.
Poudriere is also set to use as many builders as it needs to per jail (normally, it uses only one builder per jail), so sometimes the load spikes to over 50.
The NUC has a SSD whereas this machine has a pair of mirrored 5400RPM 2.5" spinning rust - so disk I/O isn't really a factor, even with +16 instances of compilers which need to get data from disks at times.

Have an image:


EDIT: And since I forgot to press ctrl+t, here are some of the stats copied from poudrieres status output
code:
[12amd64-local-htpc] [2019-06-10_14h32m36s] [parallel_build:] Queued: 349 Built: 315 Failed: 0   Skipped: 0   Ignored: 0   Tobuild: 34   Time: 01:27:20
        [01]: devel/gobject-introspection | gobject-introspection-1.56.1,1 build           (00:00:25 / 00:00:48)
        [02]: devel/tex-web2c           | tex-web2c-20150521_3      configure       (00:00:28 / 00:00:48)
        [03]: graphics/poppler          | poppler-0.76.1            build           (00:00:18 / 00:00:48)
        [04]: multimedia/ffmpeg         | ffmpeg-4.1.3_6,1          pkg-depends     (00:00:00 / 00:00:20)
        [05]: graphics/glew             | glew-2.1.0                build           (00:02:10 / 00:02:17)
[01:27:24] Logs: /data/logs/bulk/12amd64-local-htpc/2019-06-10_14h32m36s
For comparisons sake, the NUC built the same number of packages in 8-9 hours

D. Ebdrup fucked around with this message at 13:25 on Jun 10, 2019

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

H2SO4 posted:

man y'all really hell bent on using zfs with raid controllers despite the big shiny warnings telling you not to, huh

I dont have much of a choice: Until I replace the HP P800, my options for presenting the drives to the OS are:

RAID 0, 1+0, 5, or 6E. Thats it.

Which RAID level allows the least amount of interference while ensuring the most presentation for ZFS purposes?

RAID 0. Because with two drives in RAID 0, it can appear as a single, unbuffered and un-paritied drive versus all other RAID levels.

Now, the other option was of course just presenting a large drive to FreeNAS, which I did with the RAID-5, in which case FreeNAS doesnt even do ZFS RAID to it.

Doesnt matter. All the 76gb SAS will be replaced with 900gb SAS this weekend.

H2SO4
Sep 11, 2001

put your money in a log cabin




Buglord

CommieGIR posted:

I dont have much of a choice: Until I replace the HP P800, my options for presenting the drives to the OS are:

RAID 0, 1+0, 5, or 6E. Thats it.

Which RAID level allows the least amount of interference while ensuring the most presentation for ZFS purposes?

RAID 0. Because with two drives in RAID 0, it can appear as a single, unbuffered and un-paritied drive versus all other RAID levels.



The problem with that is you said this in your previous post:

quote:

So what I did was RAID-0 pairs of disks together, so that ZFS can treat all the individual pairs as a single disk, and ZFS can handle parity, so when a single disk in a pair fails, that vdisk fails, and avoids nested parity redundancy.

The least risky way of using a RAID controller to present disks to ZFS is to put em in individual RAID 0 logical disks, one drive per VD. Does the P800 not let you do that? I've only used the P410.

D. Ebdrup
Mar 13, 2009



H2SO4 posted:

The least risky way of using a RAID controller to present disks to ZFS is to put em in individual RAID 0 logical disks, one drive per VD. Does the P800 not let you do that? I've only used the P410.
Assuming that the RAID controller isn't fucking with cache control, which isn't a safe assumption.
If ZFS can't control the caches, how can it flush them when it needs to? ZFS Intent Log uses that cache control, as it stores 5 seconds/X number of bytes (I can't remember the amount, but it's not big) worth of syncronous writes per TXG to flush them to disk in an atomic operation.

H2SO4
Sep 11, 2001

put your money in a log cabin




Buglord

Of course... that's why I'm saying using RAID controllers at all is a bad idea.

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

H2SO4 posted:

The least risky way of using a RAID controller to present disks to ZFS is to put em in individual RAID 0 logical disks, one drive per VD. Does the P800 not let you do that? I've only used the P410.

I'll try that when the 900GB SAS gets here.

Bob Morales
Aug 18, 2006

I love the succulent taste of cop boots

Any Sacramento goons going to grab a Storage Pod 2.0 or 3.0 from BackBlaze?

https://www.backblaze.com/blog/come...e-pods-that-is/

quote:

Pods that have reached their end of service in our data centers.

Here’s the catch. You need to come and get them! If you’re in the Sacramento area (or interested in traveling to it), we’re going to have a pickup day on Friday, June 28th, 2019.

We’re unfortunately NOT, under any circumstances, able to ship. First come, first served.

Specs: These are a mix of Storage Pod 2.0 and Storage Pod 3.0 chassis with most of the components included (except for hard drives, including boot drives). We can’t guarantee that the components aren’t nearing the end of their life, but these pods should be in decent working order (and if they aren’t what a fun tinkering project!).

Things to Know About The Storage Pods:

Storage Pod 2.0 and 3.0 chassis only.
We are NOT supporting these in ANY way – if they do not function, you will need to tinker!
No boot drives. You will need to procure one before these become bootable.
We are not able to ship these pods. If you cannot come in person on the date and time below, please make arrangements for someone to come and pick them up on your behalf.

IOwnCalculus
Apr 2, 2003





Neat, but also I can see that becoming a bloodbath.

Volguus
Mar 3, 2009


IOwnCalculus posted:

Neat, but also I can see that becoming a bloodbath.

I can see that becoming an eBay sale.

Heners_UK
Jun 1, 2002


UnRAID's trial keys don't actually stop the system after they expire, you just cannot reboot it or stop the array as it won't start again. With that in mind, I managed to extend my 60 day trial for almost 5 months! Until today my luck ran out and something caused the server to reboot after it's last extension had expired.

However, it had convinced me that the minimal amount of computer janitoring needed to keep it going was worth parting with USD$89/CAD$121 one time for a Plus (12 device) licence. At my rate of hard drive purchases, it'll be years before I even desire more than that, and I only loose out on $10 if I ever upgrade. I also find myself using the VMs more than I thought I would.

My hand was a little forced though as the absence of Plex tonight would be noticed!

D. Ebdrup
Mar 13, 2009



Heners_UK posted:

My hand was a little forced though as the absence of Plex tonight would be noticed!
SOAF is a powerful motivator.

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

Got my drives, they are from a Netapp so I'm in the middle of formatting them all to 512 versus 520 sectors. Tested the first batch with the P800, and it can see them after they are formatted.

And thanks H2SO4, I can indeed set the RAID-0 for a single drive, so I can pass a single disk through that way.

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

Got all the drives formatted, they all showed up under the P800, and RAID-0'es them individually and they all showed up in FreeNAS, got them ZFS'ed and zvol'ed out for my iSCSI

CommieGIR fucked around with this message at 12:47 on Jun 14, 2019

cyberia
Jun 24, 2011

Do not call me that!
Snuffles was my slave name.
You shall now call me Snowball; because my fur is pretty and white.

So I recently bought a Synology DS418Play NAS with 4x 4tb drives. When I set it up and connected to it it seems to have automatically created a single volume using all 4 drives. I've already transferred a bunch of data to it and set up a Plex server but ideally I would like to have it set up with two volumes, one for media and one to backup my computers. Is there any way to create a second volume while keeping the data that's already on the NAS or do I have to wipe it and start again?

If I have to wipe it at what point in the install process do I create the second volume?

Secondary question, what is the easiest way to backup video files from my Android phone? The DS File app has a photo backup function that works great but doesn't pick up video files. The DS Cloud app seems to allow syncing of just video files but I'm wondering if that's the easiest way to do it?

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

Reporting back: ZFS and iSCSI are not friends. At all. Going to have to move to NFS.

Gay Retard
Jun 7, 2003



AT&T is coming on Monday to install U-Verse 1000 mbps Fiber. I see many shucked 10 TB drives in my future.

AgentCow007
May 20, 2004
TITLE TEXT

cyberia posted:

So I recently bought a Synology DS418Play NAS with 4x 4tb drives....

Secondary question, what is the easiest way to backup video files from my Android phone? The DS File app has a photo backup function that works great but doesn't pick up video files. The DS Cloud app seems to allow syncing of just video files but I'm wondering if that's the easiest way to do it?

The first part of your question sounds misguided and you should probably leave it alone and just make a different share folder.

As for apps, you should check out Drive and Moments instead of DS File/DS Cloud. Synology has decided to recreate the entire G-suite (as well as Evernote) for self-hosting and its fantastic.

D. Ebdrup
Mar 13, 2009



CommieGIR posted:

Reporting back: ZFS and iSCSI are not friends. At all. Going to have to move to NFS.
What's wrong with them? I've running iSCSI device extends on top of zvols with ctld since it was added in FreeBSD 10.0-RELEASE.
Everything that does syncronous writes (iSCSI, NFS v2+, databases, etc.) benefits form having at least a pair of mirrored SLOG devices, but unless you have IOPS goals that have to be met, you don't need them
Remember, too, that since you're using ZFS on top of single-disk RAID0 arrays, ZFS has no control over caching or anything else, which may hurt performance too when the ZIL gets full.

Gay Retard posted:

AT&T is coming on Monday to install U-Verse 1000 mbps Fiber. I see many shucked 10 TB drives in my future.
The only thing 1/1Gbps FTTH did for me was show me that no cloud provider that I use for backup will let me upload at anything approaching those speeds.

H2SO4
Sep 11, 2001

put your money in a log cabin




Buglord

D. Ebdrup posted:

The only thing 1/1Gbps FTTH did for me was show me that no cloud provider that I use for backup will let me upload at anything approaching those speeds.

This is very much true. The video game CDNs like Steam/Epic/Ubisoft tend to be able to fire data at warp speed (assuming you've got a POP close to you) but outside of that you generally don't see a huge improvement beyond a certain point. The use case gradually starts to shift away from raw speed per user to capacity for multiple users.

H110Hawk
Dec 28, 2006


cyberia posted:

So I recently bought a Synology DS418Play NAS with 4x 4tb drives. When I set it up and connected to it it seems to have automatically created a single volume using all 4 drives. I've already transferred a bunch of data to it and set up a Plex server but ideally I would like to have it set up with two volumes, one for media and one to backup my computers. Is there any way to create a second volume while keeping the data that's already on the NAS or do I have to wipe it and start again?

If I have to wipe it at what point in the install process do I create the second volume?

Secondary question, what is the easiest way to backup video files from my Android phone? The DS File app has a photo backup function that works great but doesn't pick up video files. The DS Cloud app seems to allow syncing of just video files but I'm wondering if that's the easiest way to do it?

Just add a second share.

DSfile should be picking up video files. Do you have it set to just "everything, full auto" ?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

If you want a synology accessible from anywhere, how are you guys typically doing that?

My friend has the Synology web interface exposed to the internet on a non-standard port to accomplish this. Is the Synology security trusted enough to do this? I would have stuck it behind a VPN.

Moey
Oct 22, 2010

I LIKE TO MOVE IT


fletcher posted:

If you want a synology accessible from anywhere, how are you guys typically doing that?

My friend has the Synology web interface exposed to the internet on a non-standard port to accomplish this. Is the Synology security trusted enough to do this? I would have stuck it behind a VPN.

That is a horrible move. Port scanners will still find that.

VPN is what you want.

Pardot
Jul 25, 2001


Moey posted:

That is a horrible move. Port scanners will still find that.

VPN is what you want.

Another alternative is ssh (make sure password auth is off, key only) and ssh tunnel the ports you want when you need to.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Moey posted:

That is a horrible move. Port scanners will still find that.

VPN is what you want.

Ok, figured that would be the response

Looks like the Synology can act as VPN. That doesn't seem ideal either though. I'm thinking a separate hardware VPN would be more appropriate, since their modem/router/firewall doesn't seem to have VPN capabilities.

KOTEX GOD OF BLOOD
Jul 7, 2012



Fallen Rib

It does have a VPN server and I think you should be ok as long as you’re using SSH keypair or at least a really strong password?

Wizard of the Deep
Sep 25, 2005


I run PiVPN on the same Raspberry Pi that runs PiHole for me, and access my pornfiles on the go that way. That means I can get PiHole adblocking away from home, too.

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

Or setup OpenVPN with Google MFA.

Gay Retard
Jun 7, 2003



D. Ebdrup posted:

The only thing 1/1Gbps FTTH did for me was show me that no cloud provider that I use for backup will let me upload at anything approaching those speeds.

Are there any particular services that can max out a 1 gbps connection? I'm not currently doing any cloud-based back-ups at the moment. I thought about signing up for a month's of Mega.

It will be nice VPNing into my NAS whenever I'm not home and not having to worry about bogging down my home network.

D. Ebdrup
Mar 13, 2009



Gay Retard posted:

Are there any particular services that can max out a 1 gbps connection? I'm not currently doing any cloud-based back-ups at the moment. I thought about signing up for a month's of Mega.

It will be nice VPNing into my NAS whenever I'm not home and not having to worry about bogging down my home network.
You mean other than the 10G anycast servers used by speedtest.tele2.net and local Steam servers? Not really, not in a consistent manner.
Not even well-seeded torrents can manage it, despite that being in theory more likely.

Heners_UK
Jun 1, 2002


Gay Retard posted:

It will be nice VPNing into my NAS whenever I'm not home and not having to worry about bogging down my home network.

You won't. With public WiFi contention, you'd be lucky to even on a typical home connection.

Crunchy Black
Oct 24, 2017

CASTOR: Uh, it was all fine and you don't remember?
VINDMAN: No, it was bad and I do remember.




CommieGIR posted:

Reporting back: ZFS and iSCSI are not friends. At all. Going to have to move to NFS.

Yeah I have had a similar experience personally and professionally.

D. Ebdrup
Mar 13, 2009



Anything that does sync writes on ZFS needs a SLC flash SSD (which itself needs capasitors enough to do a full cache dump before shutting down in case the system loses power).

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«608 »