«608 »
  • Post
  • Reply
frumpsnake
Jan 30, 2001

The sad part is, he wasn't always evil.

quote:

So I figured that instead of putting the jail on my hard drives, I could just stick it on the other 30 GB or so of the USB key I'm using for the embedded version of NAS4Free.
Or you could decide that ~1-3GB of software isn't going to make a dent in your many terabytes of HDD storage.

Honestly, if you want to run your OS from USB and install additional software in a FreeBSD jail, FreeNAS is a much better option than NAS4Free Embedded. The embedded version is definitely a second class citizen compared to the full HDD install (at which point, you might prefer a full FreeBSD or OpenIndiana), and Nas4Free's jail process is much more of a pain in the ass than the two click process for FreeNAS, not to mention the availablity of one-click PBIs for Transmission/Sickbeard/etc if you don't want to jump into your jail and install from ports or packages.

frumpsnake fucked around with this message at 02:16 on Jan 2, 2013

tarepanda
Mar 26, 2011

Living the Dream

Yeah, I figured that in the end and went with FreeNAS. The jail was being finicky and actually ended up taking me something like two hours to figure out -- it wasn't actually seeing the internet for some reason, so none of the PBIs would work.

Sickbeard STILL isn't working and claims that Cheetah isn't installed...

D. Ebdrup
Mar 13, 2009



You need to add a default gateway.

D. Ebdrup fucked around with this message at 08:44 on Jan 2, 2013

tarepanda
Mar 26, 2011

Living the Dream

I'd already figured out the problem when I posted. It wasn't the gateway though, it was some other stupid mistake.

I've given up on sickbeard for now.

Couchpotato is great with SABnzbd, but it's funky with transmission. I came home yesterday to find it downloading 27 copies of the SAME THING.

yomisei
Mar 18, 2011


So I just accidentally put the ethernet plug into the other than usual of my two connections on my desktop mainboard after getting a new silent case with a sweet array of 140mm fans and a fan control.

The transfer speed to my NAS jumped from 60MB/s to 95MB/s. The reason: One was a Realtek, the other one was an Intel one.

Steakandchips
Apr 30, 2009



Intel adaptors are far far better for sustained througput.

yomisei
Mar 18, 2011


At first I thought this was a bit weird, as my NAS has a RTL8111E chip onboard, which had to be on par with my Intel one. Then I checked again and noticed my particular MB version had a RTL8110SC instead of a RTL8111E, which is connected via PCI instead of PCI-E. The limited bandwidth of the RTL8110SC seems to be in line with tests found elsewhere.

xevz
Jul 16, 2005
Mjew.

yomisei posted:

At first I thought this was a bit weird, as my NAS has a RTL8111E chip onboard, which had to be on par with my Intel one. Then I checked again and noticed my particular MB version had a RTL8110SC instead of a RTL8111E, which is connected via PCI instead of PCI-E. The limited bandwidth of the RTL8110SC seems to be in line with tests found elsewhere.

Realtek performance is seldom (never) as good as Intel. At least not in relation to CPU load (and during high CPU load).

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll

Nap Ghost

Intel NICs are one of many reasons why I opted for Xeons and server motherboards instead of cobbling together consumer-grade desktop parts for my prosumer needs. Unfortunately, VMware ESXi 5 doesn't support one of the Intel NICs I've got on my board despite the board being on the HCL from VMware Oh well, VT-d and ECC RAM is worth the other BS, and I've got a PCI Intel NIC ready if I need it.

yomisei
Mar 18, 2011


Finally got around to measure the power consumption and I was shocked how much it actually needs. 20W if the server is completely off (10W with the PSU switch to off), 50W without 4x3TB WD Reds and 70W with them. Unplugging one of the ram sticks and the two 120mm fans didn't make a huge difference (roughly 1W). 4x 4.4W with 80%+ efficiency seems to fit here, so the meter is working correctly.

I don't mind the disks using 20W if they run, but the 20W standby and 50W idle is quite high for a system that's supposed to have a 35W TDP cpu in it. Judging from a quick "N40L power" googling those seem to run at 3W off and 40W idle. My older modular 550W PSU is in the same ballpark as my new server PSU with a worse efficiency.

Will I get a better result switching to a PicoPSU? Additionally, are there any ways to make FreeNAS use S1-S3 power saving states?

Krakkles
May 5, 2003

like and subscribe for more passive-aggressive roadway bullshit adventure in Chigcao

So I have a potentially dumb question. I set up alerts from my FreeNAS box, and I get two emails every day. One is pretty obviously looking at account security, and that one I think I understand. The other, though, is where I think warnings about hard disks (which is what I'm really worried about) would show up, and I'm not positive on how to interpret it.

Here's an example:

code:
Removing stale files from /var/preserve:

Cleaning out old system announcements:

Backup passwd and group files:

Verifying group file syntax:
/etc/group is fine

Backing up package db directory:

Disk status:
Filesystem                Size    Used   Avail Capacity  Mounted on
/dev/ufs/FreeNASs1a       926M    378M    473M    44%    /
devfs                     1.0k    1.0k      0B   100%    /dev
/dev/md0                  4.6M    3.2M    1.0M    76%    /etc
/dev/md1                  823k    2.0k    756k     0%    /mnt
/dev/md2                  149M     23M    113M    17%    /var
/dev/ufs/FreeNASs4         19M    1.3M     16M     7%    /data
brick                     3.0T    915G    2.1T    29%    /mnt/brick
brick/backup_user1        256G     37G    219G    14%    /mnt/brick/backup_user1
brick/backup_windowsPC    500G    389G    110G    78%    /mnt/brick/backup_windowsPC

Last dump(s) done (Dump '>' file systems):

Checking status of zfs pools:
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
brick  7.25T  2.71T  4.54T    37%  1.00x  ONLINE  /mnt

all pools are healthy

Checking status of ATA raid partitions:

Checking status of gmirror(8) devices:

Checking status of graid3(8) devices:

Checking status of gstripe(8) devices:

Network interface status:
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
bge0   1500 <Link#1>      a0:b3:cc:e8:94:18 1408623029     0     0 1642043758     0     0
bge0   1500 192.168.5.0   192.168.5.105     1408228091     -     - 1931924285     -     -
usbus     0 <Link#2>                               0     0     0        0     0     0
usbus     0 <Link#3>                               0     0     0        0     0     0
usbus     0 <Link#4>                               0     0     0        0     0     0
usbus     0 <Link#5>                               0     0     0        0     0     0
usbus     0 <Link#6>                               0     0     0        0     0     0
usbus     0 <Link#7>                               0     0     0        0     0     0
lo0   16384 <Link#8>                          321213     0     0   321213     0     0
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -
lo0   16384 localhost     ::1                      2     -     -        2     -     -
lo0   16384 your-net      localhost           321149     -     -   321149     -     -

Security check:
    (output mailed separately)

Checking status of 3ware RAID controllers:
Alarms (most recent first):
  No new alarms.

-- End of daily output --
Where would a failing drive show up in this?

UndyingShadow
May 15, 2006
You're looking ESPECIALLY shadowy this evening, Sir

Krakkles posted:

Checking status of zfs pools:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
brick 7.25T 2.71T 4.54T 37% 1.00x ONLINE /mnt

all pools are healthy



Probably right there. That should show up as degraded and it shouldn't say "All pools are healthy"

Krakkles
May 5, 2003

like and subscribe for more passive-aggressive roadway bullshit adventure in Chigcao

Awesome. Thank you. That's what I had assumed, but I realized assuming with something like this isn't a great idea.

tarepanda
Mar 26, 2011

Living the Dream

How do you measure how much power your system uses?

I'm starting to think I should grab an Intel NIC today -- my system has only been doing around 10mbit, a little more.

Though maybe my router is bottlenecking things, too...

Nam Taf
Jun 25, 2005

I am Fat Man, hear me roar!



tarepanda posted:

I'm starting to think I should grab an Intel NIC today -- my system has only been doing around 10mbit, a little more.

Though maybe my router is bottlenecking things, too...

Your cable could also be the problem. Check if they're Cat 5E/Cat 6 or just plain old Cat 5. The latter will do 100Mbit, whereas the former will do 1GBit.

That said, how the hell are you coping on 10Mbit? Are you sure you meant that?

tarepanda
Mar 26, 2011

Living the Dream

Yeah, my bad. My WIRELESS is operating around 10 mbit, the NAS is operating at around 30 mbit.

My router may be the issue. It's a Buffalo with Tomato (Shibby) on it. Not gigabit, though.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

tarepanda posted:

Yeah, my bad. My WIRELESS is operating around 10 mbit, the NAS is operating at around 30 mbit.

My router may be the issue. It's a Buffalo with Tomato (Shibby) on it. Not gigabit, though.
Well, first let's be sure we're using our units correctly. Do you mean really mean 10 and 30 Mbps? Because that's equivalent to 1.25 and 3.75 MB/s, respectively, which is embarrassingly slow, even for WiFi. That said, if you're still slaving away with a 802.11g 54 Mbps connection, yeah, you max out around 6MB/s.

You not using GigE equipment is going to seriously limit your speeds no matter what, though. 100 Mbps equipment means you're limited to ~10MB/s in practice. You can get cheap GigE routers for like $30, and Cat5E/6 cables aren't exactly expensive, either. $50 of upgrades should see you join the 50+ MB/s crowd (for wired performance, anyhow--wireless will still be limited to something much lower).

tarepanda
Mar 26, 2011

Living the Dream

DrDork posted:

Well, first let's be sure we're using our units correctly. Do you mean really mean 10 and 30 Mbps? Because that's equivalent to 1.25 and 3.75 MB/s, respectively, which is embarrassingly slow, even for WiFi. That said, if you're still slaving away with a 802.11g 54 Mbps connection, yeah, you max out around 6MB/s.

Yep, that's what I mean.

DrDork posted:

You not using GigE equipment is going to seriously limit your speeds no matter what, though. 100 Mbps equipment means you're limited to ~10MB/s in practice. You can get cheap GigE routers for like $30, and Cat5E/6 cables aren't exactly expensive, either. $50 of upgrades should see you join the 50+ MB/s crowd (for wired performance, anyhow--wireless will still be limited to something much lower).

The cables are easy to find, but my main problem is that it's much harder to find Linksys etc. routers here in Japan and the gear tends to be much more expensive, too. Buffalo and IOData are the only brands sold in stores and pretty much the only ones available -- my router is probably four years old or so (WHR-HP-G54) and I've kept it because it's compatible with Tomato. The original Japanese firmwares tend to be absolute shit.

I'm using cat 5e/6 cables, but it doesn't make much of a difference when the router is probably the limiting point here.

The cherry on the cake is that I have a 160 mbit cable connection. :-/

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

Can't order one online or have a friend mail you one? Being limited to 100Mb lines is really sad

Nam Taf
Jun 25, 2005

I am Fat Man, hear me roar!



tarepanda posted:

Yep, that's what I mean.


The cables are easy to find, but my main problem is that it's much harder to find Linksys etc. routers here in Japan and the gear tends to be much more expensive, too. Buffalo and IOData are the only brands sold in stores and pretty much the only ones available -- my router is probably four years old or so (WHR-HP-G54) and I've kept it because it's compatible with Tomato. The original Japanese firmwares tend to be absolute shit.

I'm using cat 5e/6 cables, but it doesn't make much of a difference when the router is probably the limiting point here.

The cherry on the cake is that I have a 160 mbit cable connection. :-/

You can't walk in to a shop and just buy a 5 port gigabit switch for AUD$10 or equivalent? Plug all your internal devices in to that andp lug that in to your router. That way at least your internal network is gigabit. Then if you have a 100Mbit connection to the router, at least you're most of the way there for your internet too.

Alternatively, jsut drop the couple of hundred on a new router and enjoy full speed everything.

Gism0
Mar 20, 2003

huuuh?

tarepanda: the easiest way to find out if it's a problem with the router is to connect a computer directly to your internet connection (where your fibre terminates) with a small ethernet cable. This will bypass the router entirely. You may need to set up PPPoE if your ISP uses that to authenticate.

Nam Taf posted:

5 port gigabit switch for AUD$10

That's a 10/100 switch.. and isn't he in Japan?

yomisei
Mar 18, 2011


Turns out my power meter doesn't like the range below 20W, as the measurement of a lamp (45W) with the NAS together wielded no 10-20W off/standby consumption whatsoever. The NAS itself still runs at 45-50W w/o hdds though.

I enabled powerd in the advanced tab and confirmed it working by looking up the frequency via sysctl dev.cpu, it shaved a good 5-7W off. FreeNAS uses C1 states by default, but I'd like it to use C3/C6 (ACPI C2/C3) too. I found this guide giving a detailed approach. Does anyone have tried this out yet? Particularly the C2/C3 and Intel GPU bits I'd like to get on my NAS, but I'm currently failing trying to manipulate the /boot/loader.conf file due to it being mounted read-only from the usb stick

edit: using 'mount -uw /' makes the /boot/loader.conf and /conf/base/etc/rc.conf (use this one instead of /etc/rc.conf) writable, but it hasn't shown a great improvement in power usage with C3, 100hz interrupts and reduced timers, maybe 3-4W less. It'll run with all disks spinning at 60W in idle and spun down at 45W. Next step would be to get a better PSU, maybe a powerful PicoPSU.

yomisei fucked around with this message at 18:57 on Jan 7, 2013

yomisei
Mar 18, 2011


What are you custom-built NAS people using as a system disk? I'm trying to get suspend-to-ram (S3) working in my FreeNAS setup and basically the USB stick shits itself after resume due to the usb controller being fully powered down, so I have to switch to a disk. I thought of using a SSD of maybe 64GB size. Wake-on-LAN and STR would be a powersaving dream.

Any SSD recommendations?

KS
Jun 10, 2003


Outrageous Lumpwad

Just a thought, but I can't imagine you're ever going to make back the cost of that SSD in electricity savings if you're just talking about a few watts.

yomisei
Mar 18, 2011


I know the SSD doesn't save any power per se, but I'd like to have the NAS in suspense and able to wake up on lan in a few seconds, which I can't do via USB stick. Compared to it running 24/7 (at 45W) it would make up for its cost in about half a year.

Fatal
Jul 29, 2004

I'm gunna kill you BITCH!!!


yomisei posted:

I know the SSD doesn't save any power per se, but I'd like to have the NAS in suspense and able to wake up on lan in a few seconds, which I can't do via USB stick. Compared to it running 24/7 (at 45W) it would make up for its cost in about half a year.

Have you actually done the calculations for this? Lets say you use your NAS for 6 hours a day vs 24hrs, that's a savings of 810Wh or 0.81kWh . Power in my area is like $0.085 per kWh so it costs about $0.068 per day more to operate your NAS for 24 hours vs 6 hours. Over the course of a year that's about $25 difference. Where does a 60GB SSD cost $12.50? Hell, each additional hour you keep your NAS on is only $1.40, so 1 hour a day costs you $1.40 per year, 24 hours a day costs $33.50. Adjust accordingly for your local power prices but I honestly don't see the point.

Maths:

45wh = 0.045kWh
0.045kWh*0.085$/kWh=0.003825$/hour (cost in $ per hour to operate)
0.003825$/h*24h/day=0.0918$/day (cost to operate continuously for 1 day)
0.0918$/day*365day/year=33.507$/year (cost to operate continuously for 1 year)

Fatal fucked around with this message at 20:54 on Jan 7, 2013

yomisei
Mar 18, 2011


Math works differently in Europe, where you have to pay ~0.25€/kWh

Fatal
Jul 29, 2004

I'm gunna kill you BITCH!!!


yomisei posted:

Math works differently in Europe, where you have to pay ~0.25€/kWh

Ouch, that's all I have to say on that. At least you have socialized healthcare*!

*maybe

Waffle Zone
Mar 17, 2004

by Y Kant Ozma Post


I'm planning to run a media server in the back room of my house with 3-4 1TB drives making 2-3TB of media storage. I want to run plex media center to serve my media to 2 Rokus and a PS3, which rules out FreeNAS unfortunately. I also need to run SABnzbd, couchpotato, sickbeard, and headphones which are all python so they should run on pretty much anything. To top it off, I need a torrent client and that should be easy to find on any OS. Does anybody have recommendations?

Ninja Rope
Oct 22, 2005

Wee.


My N40L has 5 drives in RAID-Z2. I can get sustained writes at about 77 megs/second over gige. Is that comparable to what others are getting? I'm not using jumbo frames, but I was thinking of installing an Intel NIC into the server to see if I can squeeze a little more performance out.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

Waffle Zone posted:

I'm planning to run a media server in the back room of my house with 3-4 1TB drives making 2-3TB of media storage. I want to run plex media center to serve my media to 2 Rokus and a PS3, which rules out FreeNAS unfortunately. I also need to run SABnzbd, couchpotato, sickbeard, and headphones which are all python so they should run on pretty much anything. To top it off, I need a torrent client and that should be easy to find on any OS. Does anybody have recommendations?
If you are totally sold on Plex, you're restricted to the supported OSes, which pretty much means you're either going with Windows, or Ubuntu/Fedora/CentOS/etc depending on what you're more comfortable with doing. If you're not a Linux guy, the Windows route would obviously be a bit faster and easier for you, but it'd also be a good excuse to play around with Linux a bit and learn wtf. Of the above three, Ubuntu is usually regarded as the most "newbie friendly," last I checked.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

Ninja Rope posted:

My N40L has 5 drives in RAID-Z2. I can get sustained writes at about 77 megs/second over gige. Is that comparable to what others are getting? I'm not using jumbo frames, but I was thinking of installing an Intel NIC into the server to see if I can squeeze a little more performance out.
You are at least average, and are above what a lot of people get. An Intel PRO NIC might bump you into the 80's, but with your current speeds I'd imagine mostly it would just give you added reliability and less frequent disconnects and other generally infrequent (but annoying) connection shenanigans.

Fatal
Jul 29, 2004

I'm gunna kill you BITCH!!!


Waffle Zone posted:

I'm planning to run a media server in the back room of my house with 3-4 1TB drives making 2-3TB of media storage. I want to run plex media center to serve my media to 2 Rokus and a PS3, which rules out FreeNAS unfortunately. I also need to run SABnzbd, couchpotato, sickbeard, and headphones which are all python so they should run on pretty much anything. To top it off, I need a torrent client and that should be easy to find on any OS. Does anybody have recommendations?

You could always virtualize. That's what I did, ESXi host with VMs for days.

OppyDoppyDopp
Feb 17, 2012


Would an N40L make a good HTPC for watching videos and nothing more? A NAS+HTPC solution seems hideously expensive in comparison.

Crackbone
May 23, 2003
Probation
Can't post for 30 hours!


Nap Ghost

HTJ posted:

Would an N40L make a good HTPC for watching videos and nothing more? A NAS+HTPC solution seems hideously expensive in comparison.

If all you're doing is watching videos, there are dozens of media boxes out there. Boxee, NeoTV, Popcorn Hour, Plex clients on various boxes, etc. You should be able to get something for ~$150 that will handle most any video file. Makes way more sense than trying to use a NAS box for HTPC work.

OppyDoppyDopp
Feb 17, 2012


The only difficulty with one of those boxes is that I have several TBs of local media and don't think streaming it from my computer to the box over wireless would work very well, but I may be wrong.

mik
Oct 16, 2003
oh

Waffle Zone posted:

I'm planning to run a media server in the back room of my house with 3-4 1TB drives making 2-3TB of media storage. I want to run plex media center to serve my media to 2 Rokus and a PS3, which rules out FreeNAS unfortunately. I also need to run SABnzbd, couchpotato, sickbeard, and headphones which are all python so they should run on pretty much anything. To top it off, I need a torrent client and that should be easy to find on any OS. Does anybody have recommendations?

Obviously depends on your budget, or if you're a dyed-in-the-wool whiteboxer, but you can get a Synology unit to do pretty much all of that fairly easily.

edit: I see you said you have three things you want to serve Plex to, I guess you'd want to look into how well the different models would perform streaming to multiple devices simultaneously.

mik fucked around with this message at 23:06 on Jan 8, 2013

Don Lapre
Mar 28, 2001

If you're having problems you're either holding the phone wrong or you have tiny girl hands.


Streaming should be no problem, transcoding is a problem.

chizad
Jul 9, 2001

'Cus we find ourselves in the same old mess
Singin' drunken lullabies

Ninja Rope posted:

My N40L has 5 drives in RAID-Z2. I can get sustained writes at about 77 megs/second over gige. Is that comparable to what others are getting? I'm not using jumbo frames, but I was thinking of installing an Intel NIC into the server to see if I can squeeze a little more performance out.

I've got a very similar setup with an Intel NIC, and while I haven't crunched the actual numbers, 77MB/sec sounds about right for the average throughput I'm seeing. One thing I've noticed is at least with NAS4Free 9.1 and CIFS shares, throughput seems to be inconsistent. One transfer I'll be up around 90-100MB/sec and the next will struggle to break 40MB/sec, even though it's coming from the same machine and the data being copied in both instances is similar.

IOwnCalculus
Apr 2, 2003





IOwnCalculus posted:

So, has anyone ever run across a file in ZFS that can't be deleted?

Background info on the setup: Nexenta 3.1.3.5 running on top of ESXi 5.0, with drive controllers passed directly to Nexenta. tank is a raidz2 on a stack of 1.5TB SATA drives. Some useless .AppleDouble file apparently got mangled, setting the following error:

code:
Volume: tank
=============================================================================
  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: [url]http://www.sun.com/msg/ZFS-8000-8A[/url]
 scan: scrub in progress since Sun Dec  9 03:00:05 2012
    687G scanned out of 6.45T at 9.93M/s, 169h32m to go
    0 repaired, 10.40% done
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     2
          raidz2-0  ONLINE       0     0     2
            c2t0d0  ONLINE       0     0     0
            c2t3d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t4d0  ONLINE       0     0     0
            c2t5d0  ONLINE       0     0     0
        cache
          c3t3d0    ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /volumes/tank/lol/fake/path/.AppleDouble/dickbutt.png
It's the only file that has any corruption, and being an .AppleDouble piece of shit, I couldn't care less about its existence. However, if I ever try to delete it (even from the console of the Nexenta box itself), it spins for a few seconds and then the Nexenta box hardlocks / reboots. I haven't yet been able to get a good enough look at the console before it reboots to figure out what it's spitting up but I think it's some form of segfault / kernel panic / whatever the shit goes for a BSoD in Solaris-land.

Quoting myself for a somewhat interesting resolution to this, at least as far as zpool status is confirmed; a drive popped and now that it's resilvering, it no longer shows that error:

code:
$ zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Mon Jan  7 18:35:48 2013
    1.69T scanned out of 6.77T at 20.1M/s, 73h34m to go
    288G resilvered, 24.95% done
config:

        NAME             STATE     READ WRITE CKSUM
        tank             DEGRADED     0     0     0
          raidz2-0       DEGRADED     0     0     0
            c2t0d0       ONLINE       0     0     0
            c2t3d0       ONLINE       0     0     0
            c2t2d0       ONLINE       0     0     0
            c2t1d0       ONLINE       0     0     0
            replacing-4  FAULTED      0     0     0
              c2t4d0     FAULTED      0     0     0  too many errors
              c3t2d0     ONLINE       0     0     0  (resilvering)
            c2t5d0       ONLINE       0     0     0
        cache
          c3t3d0         ONLINE       0     0     0

errors: No known data errors
Also the rate at which the resilver goes seems to be increasing. When it first started it was estimating 200-240 hours

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«608 »