Storage Solutions, Virtual Me

ZFS zvols for VM usage

Leave a comment

These experiments were performed to determine the best setup for creating a handful of VMs on Xen, utilizing a minimum of disk space, with the realization that much of the filesystem across the multiple VMs will be identical. I may, however, want the size of each VM’s filesystem to be different, depending on the purpose.

Setup

# dd if=/dev/zero of=test_pool bs=1G count=10
# for i in 1 2 3 4 5 6; do dd if=/dev/urandom of=tiny_file_$i bs=1M count=200; done
# for i in 1 2 3 4; do dd if=/dev/urandom of=small_file_$i bs=1G count=1; done
# for i in 1 2; do dd if=/dev/urandom of=big_file_$i bs=1G count=2; done
# zpool create test test_pool

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          9.8G     0  9.8G   0% /test

# zfs list -r test
NAME   USED  AVAIL  REFER  MOUNTPOINT
test   106K  9.78G    30K  /test

So, I have a ZFS pool with about 9.8G usable space in it.

A note on usage of `sync`: I tend to execute the `sync` command after each copy operation because I have found that taking a snapshot of a zvol that has cached writes results in an incomplete or corrupt snapshot. Yet another thing to consider when using ZFS with zvols.

Cloning

# zfs create -V 3G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp small_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# cp small_file_2 /test/disk1
# sync

At this point, here’s what my pool looks like:

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.6G     0  4.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        5.23G  4.55G    31K  /test
test/disk1  5.23G  7.65G  2.13G  -

Already, size-related numbers are starting to get a bit hazy. According to df, my pool is approximately 7.5G in size (4.6 + 2.9). According to zfs list, it still looks like about 9.8 total (5.23 + 4.55). Now, if I clone the first snapshot:

# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.6G     0  4.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1
/dev/zd112                    2.9G  1.1G  1.8G  37% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        5.23G  4.55G    32K  /test
test/disk1  5.23G  7.65G  2.13G  -
test/disk2   146K  4.55G  1.12G  -

The numbers get even murkier. The used and available for the two disks look correct in df, and the available in the pool hasn’t changed, as I would have expected, but it’s getting hard to figure out exactly how much space is used and available where. Total space across all three items in df: 10.4G. So now let’s copy a new file into the clone:

# cp small_file_3 /test/disk2
# sync
# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          3.6G     0  3.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1
/dev/zd112                    2.9G  2.1G  738M  74% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        6.24G  3.54G    32K  /test
test/disk1  5.23G  6.64G  2.13G  -
test/disk2  1.01G  3.54G  2.13G  -

Okay. I think I see where these values are going. The available in the pool has decreased by 1G, and the used in disk2 has increased by 1G. Makes sense. Total size reported by df across all three items: 9.4G. Now what if I resize the zvols?

# zfs set volsize=5G test/disk1
# resize2fs /dev/zvol/test/disk1
# zfs set volsize=4G test/disk2
# resize2fs /dev/zvol/test/disk2

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          1.5G  128K  1.5G   1% /test
/dev/zd96                     4.9G  2.1G  2.7G  44% /test/disk1
/dev/zd112                    3.9G  2.1G  1.7G  55% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.31G  1.48G    32K  /test
test/disk1  7.29G  6.63G  2.13G  -
test/disk2  1.01G  1.48G  2.13G  -

Hmmm… What happened there? I increased total zvol size by 3G (+2G disk1, +1G disk2). The available space in the pool went down by about 2G, and the used space in disk1 went up by about 2G, but the used space in disk 2 stayed pretty much the same. Total space across all three items in df: 10.3G. Let’s fill them up and see what we get.

# cp big_file_1 /test/disk1
# cp tiny_file_1 /test/disk1
# cp tiny_file_2 /test/disk1
# cp tiny_file_3 /test/disk1
# cp small_file_4 /test/disk2
# cp tiny_file_4 /test/disk2
# cp tiny_file_5 /test/disk2
# cp tiny_file_6 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
/dev/zd96                     4.9G  4.6G   23M 100% /test/disk1
/dev/zd112                    3.9G  3.6G   74M  99% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.78G      0    32K  /test
test/disk1  7.29G  2.53G  4.76G  -
test/disk2  2.49G      0  3.61G  -

Umm…. Ouch. Where did my pool go? The df command doesn’t even list it anymore, and according to zfs list, it’s completely full. Also, I have this mess in dmesg:

[385389.346555] end_request: critical space allocation error, dev zd112, sector 8067072
[385389.346562] end_request: critical space allocation error, dev zd112, sector 8077312
[385389.346567] end_request: critical space allocation error, dev zd112, sector 8097792
[385389.346573] end_request: critical space allocation error, dev zd112, sector 8081408
[385389.346578] end_request: critical space allocation error, dev zd112, sector 8068096
[385389.346583] end_request: critical space allocation error, dev zd112, sector 8058880
[385389.346588] end_request: critical space allocation error, dev zd112, sector 8050688
[385389.346594] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 142606336 size 4194304 starting block 1009792)
[385389.346603] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 142606336 size 8388608 starting block 1012352)
[385389.346611] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 142606336 size 4194304 starting block 1010304)
[385389.346619] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 134217728 size 8388608 starting block 1008640)
[385389.346625] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 138412032 size 4194304 starting block 1007488)
[385389.346637] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 134217728 size 4194304 starting block 1006464)
[385389.346639] buffer_io_error: 170 callbacks suppressed
[385389.346645] Buffer I/O error on device zd112, logical block 1009664
[385389.346647] Buffer I/O error on device zd112, logical block 1010189
[385389.346650] Buffer I/O error on device zd112, logical block 1012247
[385389.346653] Buffer I/O error on device zd112, logical block 1008521
[385389.346656] Buffer I/O error on device zd112, logical block 1007368
[385389.346661] Buffer I/O error on device zd112, logical block 1006337
[385389.346663] Buffer I/O error on device zd112, logical block 1009665
[385389.346665] Buffer I/O error on device zd112, logical block 1010190
[385389.346668] Buffer I/O error on device zd112, logical block 1012248
[385389.346670] Buffer I/O error on device zd112, logical block 1008522
[385389.346726] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 134217728 size 8388608 starting block 1008512)
[385389.346946] end_request: critical space allocation error, dev zd112, sector 8056832
[385389.346959] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 138412032 size 4194304 starting block 1007232)
[385389.346979] end_request: critical space allocation error, dev zd112, sector 8099840
[385389.346995] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 142606336 size 8388608 starting block 1012608)
[385389.346998] end_request: critical space allocation error, dev zd112, sector 8100864
[385389.347006] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 142606336 size 8388608 starting block 1012864)
[385389.699816] Aborting journal on device zd112-8.
[385389.699839] EXT4-fs error (device zd112) in ext4_free_blocks:4858: Journal has aborted
[385389.711017] EXT4-fs (zd112): Delayed block allocation failed for inode 15 at logical offset 46080 with max blocks 1024 with error 30
[385389.711025] EXT4-fs (zd112): This should not happen!! Data will be lost
[385389.711025] 
[385389.711046] EXT4-fs error (device zd112) in ext4_writepages:2536: Journal has aborted
[385389.712520] EXT4-fs error (device zd112): ext4_journal_check_start:56: Detected aborted journal
[385389.712530] EXT4-fs (zd112): Remounting filesystem read-only
[385389.712644] EXT4-fs (zd112): ext4_writepages: jbd2_start: 1024 pages, ino 16; err -30
[385389.783776] Buffer I/O error on device zd112, logical block 786433
[385389.783784] lost page write due to I/O error on zd112
[385389.783806] Buffer I/O error on device zd112, logical block 786434
[385389.783809] lost page write due to I/O error on zd112

Indicating that disk2 is now mounted read-only:

# touch /test/disk2/foo
touch: cannot touch \u2018/test/disk2/foo\u2019: Read-only file system

Yup, read-only. Let’s try this again, but with a much-larger overlap between the two disks:

# umount /test/disk2
# umount /test/disk1
# zpool destroy test
# zpool create test test_pool
# zfs create -V 3.5G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp big_file_1 /test/disk1
# cp small_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2/
# zfs set volsize=5G test/disk1
# zfs set volsize=4G test/disk2
# resize2fs /dev/zvol/test/disk1 
# resize2fs /dev/zvol/test/disk2
# cp small_file_2 /test/disk1
# cp tiny_file_1 /test/disk1
# cp tiny_file_2 /test/disk1
# cp tiny_file_3 /test/disk1
# cp tiny_file_4 /test/disk2
# cp tiny_file_5 /test/disk2
# cp tiny_file_6 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          900M     0  900M   0% /test
/dev/zd96                     4.9G  4.6G   16M 100% /test/disk1
/dev/zd112                    3.9G  3.6G   74M  99% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.90G   899M    32K  /test
test/disk1  8.31G  4.43G  4.76G  -
test/disk2   607M   899M  3.75G  -

Well, at least this time I didn’t run out of disk space. No errors in dmesg either. I don’t see a huge space savings, though: 4.6G on one disk, 3.6G on the second, 3G “shared” in the snapshot… total disk usage: 8.9G??? In fact, it seems to take up more total space than if I were to just store those same files in the pool directly. Let’s try that. In my two zvols, I have stored two big files, three small files, and six tiny files:

# umount /test/disk1
# umount /test/disk2
# zpool destroy test
# zpool create test /storage/tmp/test_pool
# cp big_file_1 /test
# cp big_file_2 /test
# cp small_file_1 /test
# cp small_file_2 /test
# cp small_file_3 /test
# cp tiny_file_1 /test
# cp tiny_file_2 /test
# cp tiny_file_3 /test
# cp tiny_file_4 /test
# cp tiny_file_5 /test
# cp tiny_file_6 /test
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          9.8G  8.2G  1.7G  84% /test

# zfs list -r test
NAME   USED  AVAIL  REFER  MOUNTPOINT
test  8.18G  1.60G  8.18G  /test

Perhaps resizing the zvols caused some kind of disconnect in the data sharing? Good theory. Let’s try it without resizing:

# umount /test/disk1
# umount /test/disk2
# zpool destroy test
# zpool create test test_pool
# zfs create -V 4.5G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp big_file_1 /test/disk1/
# cp small_file_1 /test/disk1
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# zfs destroy test/disk2
# zfs destroy test/disk1@s1
# sync
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2
# df -h | grep "test\|Size"
# cp small_file_2 /test/disk1
# cp small_file_3 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          893M     0  893M   0% /test
/dev/zd96                     4.4G  4.1G   57M  99% /test/disk1
/dev/zd112                    4.4G  4.1G   57M  99% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.91G   892M    32K  /test
test/disk1  7.90G  4.50G  4.27G  -
test/disk2  1.01G   892M  4.27G  -

Again, around 8G of data, with 3G shared, takes about 8.9G of storage space. I’m not impressed with cloning zvols as a means to save on storage space.

Dedup

What does it mean to dedup data between zvols? When data is written to disk2 that is a duplicate of data already in disk1, where does the extra space saved by the dedup logic go to? Let’s experiment:

# zfs create -V 5G test/disk1
# zfs set dedup=on test
# zfs create -V 4G test/disk2
# mkdir /test/disk1
# mkdir /test/disk2
# mkfs.ext4 /dev/zvol/test/disk1 
# mkfs.ext4 /dev/zvol/test/disk2
# mount /dev/zvol/test/disk1 /test/disk1
# mount /dev/zvol/test/disk2 /test/disk2
# cp big_file_1 /test/disk1
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          790M  128K  790M   1% /test
/dev/zd96                     4.8G  2.1G  2.6G  45% /test/disk1
/dev/zd112                    3.9G  8.0M  3.6G   1% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.35G   798M    32K  /test
test/disk1  5.16G  3.75G  2.19G  -
test/disk2  4.13G  4.72G   194M  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.09G  7.84G    21%  1.17x  ONLINE  -

# cp big_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          2.8G  128K  2.8G   1% /test
/dev/zd96                     4.8G  2.1G  2.6G  45% /test/disk1
/dev/zd112                    3.9G  2.1G  1.6G  56% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.38G  2.76G    32K  /test
test/disk1  5.16G  5.69G  2.23G  -
test/disk2  4.13G  4.67G  2.21G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.15G  7.79G    21%  2.19x  ONLINE  -

# cp small_file_1 /test/disk1
# cp small_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          3.8G     0  3.8G   0% /test
/dev/zd96                     4.8G  3.1G  1.6G  67% /test/disk1
/dev/zd112                    3.9G  3.1G  603M  84% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.42G  3.70G    32K  /test
test/disk1  5.16G  5.62G  3.24G  -
test/disk2  4.13G  4.60G  3.22G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.26G  6.68G    32%  2.12x  ONLINE  -

# cp small_file_2 /test/disk1
# cp tiny_file_1 /test/disk1
# cp tiny_file_2 /test/disk1
# cp tiny_file_3 /test/disk1
# cp tiny_file_4 /test/disk2
# cp tiny_file_5 /test/disk2
# cp tiny_file_6 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          3.7G     0  3.7G   0% /test
/dev/zd96                     4.8G  4.6G     0 100% /test/disk1
/dev/zd112                    3.9G  3.6G  2.9M 100% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.46G  3.66G    32K  /test
test/disk1  5.16G  3.97G  4.85G  -
test/disk2  4.13G  3.96G  3.82G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  5.64G  4.30G    56%  1.65x  ONLINE  -

Now this shows some promise! This is what I was hoping to see with my cloning tests!

Compression

Is compression going to work the same as dedup? Will I see the space savings as a return of storage to the pool? Let’s see.

# zfs create -V 9G test/disk1
# zfs set compression=on test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# dd if=/dev/zero of=/test/disk1/file1 bs=1G count=1
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M     0  511M   0% /test
/dev/zd96                     8.8G  1.1G  7.3G  13% /test/disk1

# dd if=/dev/zero of=/test/disk1/file2 bs=1G count=5
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M  128K  511M   1% /test
/dev/zd96                     8.8G  6.1G  2.3G  73% /test/disk1

# dd if=/dev/zero of=/test/disk1/file3 bs=1G count=2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M     0  511M   0% /test
/dev/zd96                     8.8G  8.1G  255M  97% /test/disk1

# dd if=/dev/zero of=/test/disk1/file4 bs=1M count=250
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M     0  511M   0% /test
/dev/zd96                     8.8G  8.3G  4.4M 100% /test/disk1

# zfs get compressratio test/disk1
NAME        PROPERTY       VALUE  SOURCE
test/disk1  compressratio  11.65x  -

# resize2fs /dev/zvol/test/disk1 
resize2fs 1.42.9 (4-Feb-2014)
The filesystem is already 2359296 blocks long.  Nothing to do!

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.28G   510M    31K  /test
test/disk1  9.28G  9.78G   334K  -

As far as I can tell, the data really is compressed on disk. The compressratio is large (though significantly smaller than what I would expect). The dd commands each completed very quickly, indicating that little data was actually written to disk. But I can’t seem to figure out how to take advantage of the space savings that the compression provided. I’m somewhat disappointed.

Dedup combined with cloning

Well, dedup will give me the space savings that I was hoping to achieve by cloning zvols. But can I turn on dedup and use cloning to create multiple copies of a VM that all start in a common state? Let’s try that.

# zfs set dedup=on test
# zfs create -V 3G test/disk1
# mkdir /test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mount /dev/zvol/test/disk1 /test/disk1/
# cp big_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.6G     0  4.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1
/dev/zd112                    2.9G  2.1G  738M  74% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        5.29G  4.60G    32K  /test
test/disk1  5.23G  7.69G  2.13G  -
test/disk2   185K  4.60G  2.13G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.09G  7.85G    21%  1.05x  ONLINE  -

# zfs set volsize=5G test/disk1
# zfs set volsize=4G test/disk2
# resize2fs /dev/zvol/test/disk1 
# resize2fs /dev/zvol/test/disk2
# cp small_file_1 /test/disk1
# cp small_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          2.5G     0  2.5G   0% /test
/dev/zd96                     4.9G  3.1G  1.7G  66% /test/disk1
/dev/zd112                    3.9G  3.1G  680M  82% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.42G  2.46G    32K  /test
test/disk1  7.29G  6.60G  3.15G  -
test/disk2  1.02G  2.46G  3.15G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.17G  6.77G    31%  1.37x  ONLINE  -

Remember, in my dedup tests above, when I had two filesystems with both big_file_1 and small_file_1 on each, I had 3.8G free space, and zpool list reported a 2.12x dedup ratio. Seeding my new filesystem with a clone of a snapshot isn’t going too well. Maybe if I overwrite the cloned copy of big_file_1? Perhaps it will notice then that it is a duplicate?

# cp big_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          2.4G  128K  2.4G   1% /test
/dev/zd96                     4.9G  3.1G  1.7G  66% /test/disk1
/dev/zd112                    3.9G  3.1G  680M  82% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        10.5G  2.34G    32K  /test
test/disk1  7.29G  6.48G  3.15G  -
test/disk2  3.04G  2.34G  3.76G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.25G  6.69G    32%  2.03x  ONLINE  -

Hmmm… The dedup ratio went up (though not quite to 2.12x) but the available space actually went down! What’s with that? I’m going to try to replace the copy of big_file_1 on disk 1.

# cp big_file_1 /test/disk1
# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.1G  128K  4.1G   1% /test
/dev/zd96                     4.9G  3.1G  1.7G  66% /test/disk1
/dev/zd112                    3.9G  3.1G  680M  82% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        10.5G  4.25G    32K  /test
test/disk1  7.29G  6.42G  4.65G  -
test/disk2  3.04G  4.25G  3.76G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.40G  6.53G    34%  2.70x  ONLINE  -

There we go. Now I’m seeing similar results to what I saw in the original dedup tests. My conclusion: deduplication doesn’t do anything when cloning zvols.

Dedup with Send / Receive

I expect this should work, but I’ve been surprised with some results so far, so it’s worth experimenting just to be sure.

# zpool create test test_pool
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            9.8G     0  9.8G   0% /test

# zfs list -r test
NAME   USED  AVAIL  REFER  MOUNTPOINT
test   108K  9.78G    30K  /test

# zfs set dedup="on" test
# zfs create -V 3G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp small_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# cp small_file_2 /test/disk1
# sync
# zfs snap test/disk1@s2
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            5.7G     0  5.7G   0% /test
/dev/zd0        2.9G  2.1G  738M  74% /test/disk1

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        4.28G  5.61G    31K  /test
test/disk1  4.22G  7.69G  2.13G  -

# zfs create -V 3G test/disk2
# zfs send test/disk1@s1 | zfs receive -F test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            2.5G     0  2.5G   0% /test
/dev/zd0        2.9G  2.1G  738M  74% /test/disk1
/dev/zd16       2.9G  1.1G  1.8G  37% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.51G  2.47G    32K  /test
test/disk1  4.22G  4.55G  2.13G  -
test/disk2  4.22G  5.56G  1.12G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.13G  7.80G    21%  1.60x  ONLINE  -

# umount /test/disk2 
# zfs send -i s1 test/disk1@s2 | zfs receive -F test/disk2
# mount /dev/zvol/test/disk2 /test/disk2
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            1.5G  128K  1.5G   1% /test
/dev/zd16       2.9G  2.1G  738M  74% /test/disk1
/dev/zd0        2.9G  2.1G  738M  74% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        10.6G  1.41G    32K  /test
test/disk1  5.23G  4.50G  2.13G  -
test/disk2  5.23G  4.50G  2.13G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.15G  7.79G    21%  2.10x  ONLINE  -

# zfs create -V 3G test/disk3
cannot create 'test/disk3': out of space

I don’t really get this one. The `df` and `zfs list` results indicate that it’s not working. Available space stays at about 1.5G, regardless of the amount of overlap. However, the `zpool list` shows about 7.8G free, and it does indicate that the dedup ratio is increasing as I increase the amount of overlap between the zvols.

In the end, though, the last statement is telling. Even though the two zvols are 66% overlapping (2G out of 3G identical) there isn’t enough room in the pool to create a third zvol. I would expect this to work. First zvol uses 3G, right? Second zvol uses 1G (3G – 2G dup data). That means out of my 10G pool, only 4G is in use. I should have plenty of space available for another 3G zvol. If I can’t use the space savings, what’s the point?

Conclusion

When working with VMs on ZFS zvols, if space savings is a major concern for you, I recommend the following:

  • Create a ZFS pool to hold all of your VM root filesystems
  • Turn on deduplication across the entire pool
  • Do not use compression
  • Do not clone a new machine from a snapshot of an old machine
  • Do not use `zfs send` / `zfs receive` to create a new machine from a snapshot of an old machine

Or maybe better yet, don’t try to use dedup across zvols. The results are pretty confusing, and I’m not sure it’s worth it.

Health

On losing weight (Part 1)

8 Comments

Many people have noticed that I have lost a lot of weight (almost 90 lbs. as of this writing) and I am frequently asked: “How did you do it?” Unfortunately, there is no simple answer to this question. There is no single “just do this.” No magic bullet. The weight loss that I have experienced is the culmination of multiple small changes made based on a long process of trial-and-error. It is my intention to catalog the steps that I have taken, via multiple blog posts over the next few months, in the off chance that it might be useful (or at least entertaining) to someone. Here’s a sneak preview of topics I intend to cover:

  1. Dan’s Weight History – Everyone has their own reasons for their health, be it good health, or bad health. Every person is a complex combination of body, soul, and spirit, and these pieces affect each other in interesting ways. In order to understand my journey to health, I think it will be helpful to understand the thoughts and feelings that led me to gain weight in the first place, and what motivated me to lose it.
  2. Noom Weight Loss App – The spark that started me on this journey, and one of my main sources of inspiration during the majority of it, was an application that I downloaded for my smartphone.
  3. Calorie Counting – Yes, I count calories. But just counting is not enough. You need to have a plan. Budget your food consumption. Know what you’re eating, and when you eat it. I’ll let you know how I go about it.
  4. Volumetrics – The decisions that I have made in changing my diet have been based mostly around the simple concept of “volumetrics”. I intend to give a brief overview of what volumetrics means, and how I follow it. Hopefully this will illuminate some of the decisions I have made.
  5. Dan’s Eating Habits – Up until this point, it’s all been background information: history and theory. This is the nuts-and-bolts. What specific changes have I made to achieve the results that I got? This section might get a bit long (winded) and may take up several posts. We’ll see how motivated I am.
  6. Other Weight Loss Changes – Researchers seem pretty unanimous in their recommendation that lasting weight loss must come from changing eating habits. Just Google “diet vs exercise” and you will come up with a whole host of articles from various sources that mostly back up this claim. However, I don’t think that diet should be the sole component of an overall weight loss strategy. In my final entry, I will outline some other changes that I have made that I believe have had a beneficial effect on my health.

So if you’re curious, stick around. Who knows. Perhaps some piece of my ramblings could spark something in your own life that starts you on a similar journey to better health. That is my hope, and prayer. I have discovered something that has been elusive to me for years. It would be selfish of me to not at least attempt to share that discovery with others.

– danBhentschel

All Things Dad

Play it again, Dad

Leave a comment

One of my goals as father is to help my children to find joy in their everyday, mundane tasks. I have always been a huge fan of music, and so it seems natural to me to pass on that joy of music to my young charges. With the advent of connected, do-anything, smart phones, these two objectives (enjoying daily tasks and learning music) coincide quite nicely. Let me elaborate.

I have my phone on my person at almost all times of the day. Outside doing yard work, in the bathroom brushing the kids’ teeth, cooking breakfast in the morning… At any time, I can switch on the tunes. My favorite time to crank it up is when getting ready for bed at night. Kids are going every-which-way, bedrooms, bathrooms, up and down stairs. Everyone is tired. Often emotions are high. A little bit of music just helps to smooth things over a bit.

Granted, my little phone doesn’t have the greatest audio quality, but for most situations, it is perfectly suited for some fun, unobtrusive, background accompaniment. I even have a playlist with kid-friendly music setup on my phone for just this purpose. It’s currently running around 120 songs, and I’m constantly on the lookout for new material to add to the mix.

Good sources of new music include:

  • Kids’ movies (especially the end credits)
  • Sports events (including things like Monster Jam)
  • Live shows (musicals, Disney on Ice, circus, etc.)
  • Televised events (parades, halftime shows, etc.)
  • School events (talent show, assemblies, etc.)

I have an application on my phone that I keep handy for just such times. If I hear a song that sounds exciting, or it is associated with a memorable event, then I launch “Sound Hound” on my phone. It listens to the music for a few seconds, and then (usually) is able to tell me the name of the song, and the artist. It even provides a handy link for me to buy the track from Amazon. I normally will do so (only $.99) and instantly add it to the kids’ playlist. Then later on, when they are brushing their teeth or taking a bath, the song will come on, and they will be all excited. “That’s the song from the movie we saw today, Daddy!”

I also get the inverse fairly frequently: we’ll be out and about, shopping at Wegmans or sitting at dinner at a restaurant, and one of the kids (usually Jasmin) will hear a song come on the radio and inform me, “That’s the song from on your phone, Daddy!”

It’s not a big thing. It doesn’t cost all that much in either time or money, and yet it makes quite an impression. I love it when Timmy is all excited to hear the Kidz Bop version of “Thrift Shop” come on, or when Jasmin starts hopping around (often in her undies while dressing) to the sound of One Direction, or when Jacob starts shadow boxing to “Eye of the Tiger”.

It all boils down to one simple fact: Making them happy makes me happy!

– danBhentschel