You're probably better off using traditional RAID-5 arrays with the smallest drives you can buy (1-2TB) and traditional 4:th gen filessystems such as ext4, reiserfs, XFS.
If you can't help it and need to have a go, this is the gist of it: BTRFS might be a little more bleeding edge but has very nice raid handling, growing, shrinking etc. Don't use check --repair, it's destructive (like repair should be, right?)
ZFS is more mature, but it's also more stiff. If you want to grow, shrink, and such, you're in a world of pain. add drives to a pool without being able to remove them, even though they are unused. Intuitive ? :D
Long version
I gave up BTRFS on
the desktop, but I'd like to share my efforts of trying to fix my
problems, in hope I can help some poor noob like me.
I decided to use BTRFS after reading the ArsTechnica article about the
next-gen filesystems, and BTRFS seemed like the natural choice, open
source, built into linux, etc. I even bought a HP microserver to have
everything on because none of the commercial NAS-es supported BTRFS.
What a mistake, I wasted weeks in total managing something that could
have taken a day to set up, and I'd have MUCH more functionality now
(if I wasn't hit by some ransomware, that is).
I had three 1TB drives, chose to use raid, and all was good for a
while, until started fiddling with Motion, the image capturing
software. When you kill that process (my take on it) a file can be
written but it ends up with question marks instead of attributes, and
it's impossible to remove. BTRFS check --repair is not recommended, it
crashes , doesn't fix all problems, and I later found out that my
lost+found dir had about 39G of lost files and dirs.
I spent about two days trying to fix everything, removing a disk,
adding it again, checking , you name it. I ended up removing one disk,
reformatting it, and moving the data there. Now I removed BTRFS
entirely and replaced it with a OpenZFS mirror array, to which I'll
add the third disk later when I transferred everything over.
Please have a look at the console logs. I've been running linux on the
desktop for the past 15 years, so I'm not a noob, but for running
BTRFS you better be involved in the development of it. In my humble
opinion, it's not for us "users" just yet. Not even for power users.
For those of you considering building a NAS without special purposes,
don't. Buy a synology, pop in a couple of drives, and enjoy the ride.
Unfortunately, the other alternative is ZFS. ZFS is another typical example of software designed from the ground up, and implemented without any consideration to normal people, users, that is.
ZFS is amazing in so many ways. So feature rich, so smart, yet so bad when it comes to user experience. Let me explain: most Apple users don't know what a filesystem is, and that's how it's supposed to be.
You buy a computer, flip it open, do your work, close the lid, open it again, etc. It JUST WORKS. ZFS has the same problem as Linux on the desktop, it doesn't just work. It might look so, but it's full of pitfalls, and just like in that Amiga game "Another World", you need to start over from the beginning if you made a so called mistake, and it's not clear what is a mistake , and what's not. You just have to be an expert from the beginning, and know exactly what you want, and what you're supposed to do, and how, in order to not fail multiple times before you achieve it.
Let me give you a couple of examples:
1. you have 3 drives, one with data, two free. You want to RAID them together in a RAID-5 like configuration. According to the documentation, it's not possible. Actually it is, by creating a sparse file, and adding it to the obligatory 3-drive array together with the two real drives, offline the sparse drive, and copy the data over to the two remaining online drives. All good, right ? Not by a long shot. Most people won't be able to create a large enough sparse file to match a 1/3/ or 4TB drive, which means your array will be limited to the size of the sparse file, AND YOU CAN'T grow it unless you replace the sparse file with a larger drive, which you of course can't because your large drive is full of data you can't copy over. By the way, BTRFS is really good at this growing shrinking stuff, too bad it's unstable.
2. OK , screw that 3-drive parity array stuff that people anyway trash talk because its slow, takes ages to resilver(replace) drives, and so on. I'll go with the preferred solution, mirrored drives, which gives me a whopping 50% usage of my drives. Cool, 1-2-3 and I have a two-drive array. Now I copy over the files, everything looks good, and I have a spare drive, which I want to use, so I don't get 33% drive usage by having it as a spare. So I format it with zfs, connect it to the pool, and boom. it's there. BUT, the zpool grows onto the extra drive, and I'm thinking, how is everything going to react if some files are copied across the mirror and the extra drive, and one of them fails ? Everything is going to die probably. So I want to remove the extra drive from the pool, but I cant. I CAN'T. Why ? WHY? I have to move the data to somewhere else, delete the pool, the mirror, and everything, and start over again. It's so fucking stupid I'm amazed how people this smart, able to craft such amazing technologies I barely understand after 10 years of being a programmer, how these insanely smart people make so awkward software. Because there's no Steve Jobs (read: a regular guy, that demands something simple) to keep them on their toes.
This type of software is for themselves alone, not for you and me (even though me means over 10 years as a software developer, and over 20 years of working with software)
Here's a walk through the catch 22 that is fixing problems in BTRFS:
root /home/storage/motion/2017-05-24 1 ls -al
ls: cannot access '36-20170524201346-02.jpg': No such file or directory
ls: cannot access '36-20170524201346-02.jpg': No such file or directory
total 4
drwxrwxrwx 1 motion motion 114 Sep 14 12:48 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
-rwxr-xr-x 1 adyhasch adyhasch 62 Sep 14 12:43 remove.py
root /home/storage/motion/2017-05-24 1 touch test.raw
root /home/storage/motion/2017-05-24 cat /dev/random > test.raw
^C
root /home/storage/motion/2017-05-24 ls -al
ls: cannot access '36-20170524201346-02.jpg': No such file or directory
ls: cannot access '36-20170524201346-02.jpg': No such file or directory
total 8
drwxrwxrwx 1 motion motion 130 Sep 14 13:12 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
-rwxr-xr-x 1 adyhasch adyhasch 62 Sep 14 12:43 remove.py
-rwxrwxrwx 1 root root 338 Sep 14 13:12 test.raw
root /home/storage/motion/2017-05-24 1 cp test.raw 36-20170524201346-02.jpg
'test.raw' -> '36-20170524201346-02.jpg'
root /home/storage/motion/2017-05-24 ls -al
total 20
drwxrwxrwx 1 motion motion 178 Sep 14 13:13 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-rwxr-xr-x 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxr-xr-x 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxr-xr-x 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxr-xr-x 1 adyhasch adyhasch 62 Sep 14 12:43 remove.py
-rwxrwxrwx 1 root root 338 Sep 14 13:12 test.raw
root /home/storage/motion/2017-05-24 chmod 777 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 ls -al
total 20
drwxrwxrwx 1 motion motion 178 Sep 14 13:13 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxr-xr-x 1 adyhasch adyhasch 62 Sep 14 12:43 remove.py
-rwxrwxrwx 1 root root 338 Sep 14 13:12 test.raw
root /home/storage/motion/2017-05-24 unlink 36-20170524201346-02.jpg
unlink: cannot unlink '36-20170524201346-02.jpg': No such file or directory
root /home/storage/motion/2017-05-24 1 ls -al
total 20
drwxrwxrwx 1 motion motion 178 Sep 14 13:13 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxr-xr-x 1 adyhasch adyhasch 62 Sep 14 12:43 remove.py
-rwxrwxrwx 1 root root 338 Sep 14 13:12 test.raw
root /home/storage/motion/2017-05-24 journalctl -k | grep BTRFS
Sep 14 09:41:58 jenna kernel: BTRFS: device label BTRFS Redundant storage devid 4 transid 348450 /dev/sdd
Sep 14 09:41:58 jenna kernel: BTRFS: device label BTRFS Redundant storage devid 2 transid 348450 /dev/sdb
Sep 14 09:41:58 jenna kernel: BTRFS: device label BTRFS Redundant storage devid 3 transid 348450 /dev/sdc
Sep 14 09:41:58 jenna kernel: BTRFS info (device sdc): enabling auto defrag
Sep 14 09:41:58 jenna kernel: BTRFS info (device sdc): disabling disk space caching
Sep 14 12:52:36 jenna kernel: BTRFS: Transaction aborted (error -2)
Sep 14 12:52:36 jenna kernel: BTRFS: error (device sdc) in btrfs_rename:9943: errno=-2 No such entry
Sep 14 12:52:36 jenna kernel: BTRFS info (device sdc): forced readonly
Sep 14 13:02:26 jenna kernel: BTRFS error (device sdc): cleaner transaction attach returned -30
Sep 14 13:03:41 jenna kernel: BTRFS info (device sdc): disk space caching is enabled
root /home/storage/motion/2017-05-24
root ~ btrfs scrub status /home/storage/
scrub status for 72ea6622-5098-4a0f-bea1-9a5e5a325735
scrub started at Fri Sep 14 13:06:46 2018 and finished after 00:56:35
total bytes scrubbed: 1.16TiB with 0 errors
root /home/storage/motion/2017-05-24 stat 36-20170524201346-02.jpg
File: 36-20170524201346-02.jpg
Size: 338 Blocks: 8 IO Block: 4096 regular file
Device: 29h/41d Inode: 12616879 Links: 1
Access: (0777/-rwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-14 13:13:35.477264025 +0200
Modify: 2018-09-14 13:13:35.477264025 +0200
Change: 2018-09-14 13:14:02.025170343 +0200
Birth: -
root /home/storage/motion/2017-05-24 1 find . -inum 12616879 -exec rm -i {} \;
rm: remove regular file './36-20170524201346-02.jpg'? y
rm: cannot remove './36-20170524201346-02.jpg': No such file or directory
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 ls -al
total 20
drwxrwxrwx 1 motion motion 178 Sep 14 13:13 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxrwxrwx 1 root root 338 Sep 14 13:13 36-20170524201346-02.jpg
-rwxr-xr-x 1 adyhasch adyhasch 62 Sep 14 12:43 remove.py
-rwxrwxrwx 1 root root 338 Sep 14 13:12 test.raw
root /home/storage/motion/2017-05-24 rm 36-20170524201346-02.jpg
rm: cannot remove '36-20170524201346-02.jpg': No such file or directory
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24 rm -f 36-20170524201346-02.jpg
root /home/storage/motion/2017-05-24
... more of the same
root /home/storage/motion rm -rf 2017-05-24/
rm: cannot remove '2017-05-24/': Directory not empty
root /home/storage/motion 1 ls -al 2017-05-24/
ls: cannot access '2017-05-24/36-20170524201346-02.jpg': No such file or directory
ls: cannot access '2017-05-24/36-20170524201346-02.jpg': No such file or directory
ls: cannot access '2017-05-24/36-20170524201346-02.jpg': No such file or directory
total 0
drwxrwxrwx 1 motion motion 144 Sep 14 14:25 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
root ~ btrfs check /dev/sdb
warning, device 3 is missing
warning, device 3 is missing
Checking filesystem on /dev/sdb
UUID: 72ea6622-5098-4a0f-bea1-9a5e5a325735
checking extents
checking free space cache
failed to load free space cache for block group 9998483259392
failed to load free space cache for block group 10388251541504
failed to load free space cache for block group 10483848118272
checking fs roots
root 5 inode 11189411 errors 200, dir isize wrong
unresolved ref dir 11189411 index 0 namelen 0 name filetype 0 errors 6, no dir index, no inode ref
unresolved ref dir 11189411 index 9477 namelen 24 name 36-20170524201346-02.jpg filetype 1 errors 1, no dir item
root 5 inode 12616877 errors 2000, link count wrong
unresolved ref dir 11189411 index 9482 namelen 24 name 36-20170524201346-02.jpg filetype 1 errors 1, no dir item
root 5 inode 12616879 errors 2000, link count wrong
unresolved ref dir 11189411 index 9484 namelen 24 name 36-20170524201346-02.jpg filetype 1 errors 1, no dir item
found 639613362176 bytes used err is 1
total csum bytes: 605048928
total tree bytes: 828735488
total fs tree bytes: 182419456
total extent tree bytes: 18399232
btree space waste bytes: 47806043
file data blocks allocated: 969656111104
referenced 634590535680
root ~ 1 btrfs check --repair /dev/sdb
enabling repair mode
warning, device 3 is missing
warning, device 3 is missing
Checking filesystem on /dev/sdb
UUID: 72ea6622-5098-4a0f-bea1-9a5e5a325735
checking extents
Unable to find block group for 0
extent-tree.c:289: find_search_start: Assertion `1` failed.
btrfs[0x43e418]
btrfs(btrfs_reserve_extent+0x5c9)[0x4425df]
btrfs(btrfs_alloc_free_block+0x63)[0x44297c]
btrfs(__btrfs_cow_block+0xfc)[0x436636]
btrfs(btrfs_cow_block+0x8b)[0x436bd8]
btrfs[0x43ad82]
btrfs(btrfs_commit_transaction+0xb8)[0x43c5dc]
btrfs[0x4268b4]
btrfs(cmd_check+0x1111)[0x427d6d]
btrfs(main+0x12f)[0x40a341]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fd7a78002e1]
btrfs(_start+0x2a)[0x40a37a]
root ~ 1 btrfs check --repair /dev/sdc
enabling repair mode
warning, device 2 is missing
Checking filesystem on /dev/sdc
UUID: 72ea6622-5098-4a0f-bea1-9a5e5a325735
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
reset isize for dir 11189411 root 5
unresolved ref dir 11189411 index 0 namelen 0 name filetype 0 errors 6, no dir index, no inode ref
unresolved ref dir 11189411 index 9477 namelen 24 name 36-20170524201346-02.jpg filetype 1 errors 1, no dir item
invalid dir item size
Moving file '36-20170524201346-02.jpg' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 12616877
invalid dir item size
Moving file '36-20170524201346-02.jpg.12616879' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 12616879
unresolved ref dir 11189411 index 0 namelen 0 name filetype 0 errors 6, no dir index, no inode ref
unresolved ref dir 11189411 index 9477 namelen 24 name 36-20170524201346-02.jpg filetype 1 errors 1, no dir item
checking csums
checking root refs
found 639613362176 bytes used err is 0
total csum bytes: 605048928
total tree bytes: 828735488
total fs tree bytes: 182419456
total extent tree bytes: 18399232
btree space waste bytes: 47806043
file data blocks allocated: 969656111104
referenced 634590535680
root ~ 251 btrfs check /dev/sdb
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 9998522662912 wanted 348736 found 348741
parent transid verify failed on 9998522662912 wanted 348736 found 348741
Ignoring transid failure
Couldn't setup extent tree
Couldn't open file system
root ~ 251 mount /home/storage/
root ~ watch btrfs scrub status /home/storage/
root ~ ls /home/storage/motion/2017-05-24/
ls: cannot access '/home/storage/motion/2017-05-24/36-20170524201346-02.jpg': No such file or directory
36-20170524201346-02.jpg
total 0
drwxrwxrwx 1 motion motion 24 Sep 14 14:25 .
drwxrwxr-x 1 motion adyhasch 60 Sep 14 09:42 ..
-????????? ? ? ? ? ? 36-20170524201346-02.jpg
Back to square one
[12031.946724] BTRFS error (device sdc): cleaner transaction attach returned -30
[19272.100407] BTRFS error (device sdc): bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 0, gen 1
[19272.104100] BTRFS error (device sdc): bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 0, gen 2
[19272.120344] BTRFS error (device sdc): bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 0, gen 3
