Category Archives: Musings

Hopefully a little more thought-out than the ramblings.

I’ve conducted my share of interviews over the years. Early on, I didn’t have much of a strategy for interviewing candidates. Interviewing without a strategy didn’t work out too well for me, though. On a few occasions I recommended a candidate only to find out after working with them for a little while that they were not really as good a fit for the position as I had originally anticipated.

So, based on my past failures I have devised an interview strategy that I think works pretty well for me today. Apologies ahead-of-time for those of you non-computer peoples who may have inadvertently stumbled into this post. I am a computer software developer, and a lot of my strategy revolves around interviewing candidates for software development positions.

High level strategy

My interviews all use the following basic outline:

Introduce myself – This includes a brief overview of my own employment history, followed by a short description of my current duties and responsibilities. (about 5 minutes)
Ask questions of the candidate – This is the real meat of the interview, and where I plan to spend the most time. (as much time as I have available)
Allow the candidate to ask me questions – I try to be as open and candid as possible. If I tell them something that scares them away, then perhaps they weren’t right for the job to begin with. (allow about 10 minutes at the end of the interview)

Goal-oriented questions

I’m very interested in a candidate’s goals. If my organization is going to invest time in training a new hire, I don’t want it to be someone who sees the job as a temporary stepping-stone to a different position.

Where do you see yourself in 5 years?

No, nothing ground-breaking here. It’s a pretty standard interview question, and yet it’s funny how many people are unprepared to answer this one. I may follow up with:

What will you be doing in 5 years?
What will be different from today?
How about 10 years?

What kind of developer are you?

This is a multiple-choice question. I once heard (or read, I don’t remember which) that all developers fall into one (or some combination) of four categories. The four categories are:

Clock puncher – This person is a competent, professional developer. They show up every day, sit at their computer, produce code, and go home at the end of the day. They receive a paycheck that adequately compensates them for their contribution, and they look forward to the day when they retire.
Ladder climber – This person is also very competent and professional. They perform their duties diligently and to the best of their ability, frequently going above-and-beyond expectations. They have a constant eye to the next step. What’s after software development? Team lead? MBA? Program manager? Development manger? Higher?
Artisan – This person sees software development as an art form. They look at an empty screen as a blank canvas on which to craft their masterpiece. Every function and every class must conform to some internal aesthetic that they have developed over years of study and practice. They will repeatedly refactor the same section of code until they are satisfied. They will post up their favorite creations to forums of like-minded individuals for others to “ooh” and “ahh” over.
Scientist – This person is motivated by the problem. Each feature they work on is a puzzle to be solved, and this is their driving factor. The software is simply a means to a solution.

There really isn’t a wrong answer to this question. All roles are actually useful parts of a well-rounded development team. In fact, they are roughly analogous to the more detailed Belbin team roles.

Theory-oriented questions

I consider a firm grounding in theoretical software development practices to be very important. Good software development principles and practices are universal, and apply to almost every software language in existence. A strong theoretical understanding is what allows the good developers to learn new languages and technologies.

Design patterns

To get a general idea of a candidate’s knowledge of design patterns, I will usually open up the discussion with:

What design patterns have you used?

The answer to this question quickly and efficiently addresses a whole host of other questions:

Are you familiar with the concept of design patterns?
What patterns do you know about?
Do you know how and when to apply them?

Assuming that their answer is acceptable, I will usually follow up on one or more of their responses to get more details:

When did you use this pattern?
Why did you select this pattern?
What other patterns did you consider?

What, in your opinion, is the most important purpose for design patterns?

This is sort of a trick question. I ask them “in your opinion,” but I already have the “correct” answer to the question in mind. The most important purpose for design patterns is to assist in communication. I don’t even consider this debatable.

According to Wikipedia:

The usefulness of speaking of patterns is to have a common terminology for discussing the situations designers already see over and over.

The venerable Gang of Four book states in the introduction:

The pattern name is a handle we can use to describe a design problem, its solutions, and consequences in a word or two. Naming a pattern immediately increases our design vocabulary.

To date, no candidate has answered this question 100% correctly to my satisfaction, and I’ve asked it many times. Is it a useless question then? I don’t think so. How the candidate tackles the question is useful.

Bonus question: What’s the difference between applying design patterns vs. applying the DRY principle?

Language concepts

These questions focus on the concepts behind how languages work.

What is your favorite programming language?

I want to drive this next section of the interview towards a language that the candidate feels comfortable with. Followups include:

What do you like about this language?
What don’t you like?
Compare / contrast with <some other language from their resume>

Design principles

I usually skip this part of the interview, or at least breeze through it. Most candidates don’t know much in this section.

Are you familiar with SOLID design principles?

Do any of the following sound familiar to you:

Single Responsibility Principle (SRP)
Open Closed Principle (OCP)
Liskov Substitution Principle (LSP)
Interface Segregation Principle (ISP)
Dependency Inversion Principle (DIP)

Have you heard of the saying “Prefer composition over inheritance?”

Can you explain that recommendation?

Practice-oriented questions

These questions are geared more towards actual development practices.

Refactoring

What’s the purpose of refactoring code?

Some points I’m looking for here:

Remove duplication (DRY)
Improve readability / ease of comprehension
Enhance testability
Isolate bugs or problem areas
Improve performance
Prepare for the addition of new functionality

What are some of the risks of refactoring? How can they be mitigated?

Refactoring means changing the structure of code without changing the behavior. The risk is that in the process of changing the structure, you inadvertently change the behavior. Mitigation strategies that I am hoping to hear include:

Testing (unit tests)
Utilize refactoring tools
Refactor code in small steps
Know the common refactorings and how and when to use them

Testing

Can you describe the differences between unit tests, component tests, and feature tests?

This one is somewhat controversial in the development community. Is the candidate aware of the different camps in the debate? Can they describe the various points of view? It opens up the possibility of follow-on questions:

What is the purpose for each type of test?
Have you ever used any of these testing techniques?
What are some methods of implementing each type of test?

Language-oriented questions

If I am familiar with the candidate’s language of choice (and I almost always am) then I like to ask them some specific questions about that language. I have a standard list of questions that I pick from. I don’t usually ask all the questions under a given language, but I like to have several prepared.

Java questions

What’s the difference between the `Integer` object and the `int` primitive?
What is an anonymous class?
What is a nested class? Why would you use nested instead of anonymous?
What’s the difference between checked and unchecked exceptions?
What are generics?
What are annotations? What are some things they can be used for?

C++ questions

What is a virtual method?
What is the C++ equivalent of an interface in other languages?
What is a pure virtual class?
What is multiple inheritance? What is the danger of using multiple inheritance?
Are you familiar with the boost libraries? How about the new C++11 standard (previously called Technical Report 1)?
What is the difference between a pointer and a reference?
What are templates?

C# questions

Can you explain the concept of a delegate?
What is a partial class?
Are you familiar with LINQ expressions?
What are generics?
What is a nullable primitive type? How is it represented in code?
What is a property? How is it different from a member variable?

– danBhentschel

Storage Solutions, Virtual Me

ZFS zvols for VM usage

June 27, 2014 danBhentschel Leave a comment

These experiments were performed to determine the best setup for creating a handful of VMs on Xen, utilizing a minimum of disk space, with the realization that much of the filesystem across the multiple VMs will be identical. I may, however, want the size of each VM’s filesystem to be different, depending on the purpose.

Setup

# dd if=/dev/zero of=test_pool bs=1G count=10
# for i in 1 2 3 4 5 6; do dd if=/dev/urandom of=tiny_file_$i bs=1M count=200; done
# for i in 1 2 3 4; do dd if=/dev/urandom of=small_file_$i bs=1G count=1; done
# for i in 1 2; do dd if=/dev/urandom of=big_file_$i bs=1G count=2; done
# zpool create test test_pool

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          9.8G     0  9.8G   0% /test

# zfs list -r test
NAME   USED  AVAIL  REFER  MOUNTPOINT
test   106K  9.78G    30K  /test

So, I have a ZFS pool with about 9.8G usable space in it.

A note on usage of `sync`: I tend to execute the `sync` command after each copy operation because I have found that taking a snapshot of a zvol that has cached writes results in an incomplete or corrupt snapshot. Yet another thing to consider when using ZFS with zvols.

Cloning

# zfs create -V 3G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp small_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# cp small_file_2 /test/disk1
# sync

At this point, here’s what my pool looks like:

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.6G     0  4.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        5.23G  4.55G    31K  /test
test/disk1  5.23G  7.65G  2.13G  -

Already, size-related numbers are starting to get a bit hazy. According to df, my pool is approximately 7.5G in size (4.6 + 2.9). According to zfs list, it still looks like about 9.8 total (5.23 + 4.55). Now, if I clone the first snapshot:

# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.6G     0  4.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1
/dev/zd112                    2.9G  1.1G  1.8G  37% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        5.23G  4.55G    32K  /test
test/disk1  5.23G  7.65G  2.13G  -
test/disk2   146K  4.55G  1.12G  -

The numbers get even murkier. The used and available for the two disks look correct in df, and the available in the pool hasn’t changed, as I would have expected, but it’s getting hard to figure out exactly how much space is used and available where. Total space across all three items in df: 10.4G. So now let’s copy a new file into the clone:

# cp small_file_3 /test/disk2
# sync
# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          3.6G     0  3.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1
/dev/zd112                    2.9G  2.1G  738M  74% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        6.24G  3.54G    32K  /test
test/disk1  5.23G  6.64G  2.13G  -
test/disk2  1.01G  3.54G  2.13G  -

Okay. I think I see where these values are going. The available in the pool has decreased by 1G, and the used in disk2 has increased by 1G. Makes sense. Total size reported by df across all three items: 9.4G. Now what if I resize the zvols?

# zfs set volsize=5G test/disk1
# resize2fs /dev/zvol/test/disk1
# zfs set volsize=4G test/disk2
# resize2fs /dev/zvol/test/disk2

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          1.5G  128K  1.5G   1% /test
/dev/zd96                     4.9G  2.1G  2.7G  44% /test/disk1
/dev/zd112                    3.9G  2.1G  1.7G  55% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.31G  1.48G    32K  /test
test/disk1  7.29G  6.63G  2.13G  -
test/disk2  1.01G  1.48G  2.13G  -

Hmmm… What happened there? I increased total zvol size by 3G (+2G disk1, +1G disk2). The available space in the pool went down by about 2G, and the used space in disk1 went up by about 2G, but the used space in disk 2 stayed pretty much the same. Total space across all three items in df: 10.3G. Let’s fill them up and see what we get.

# cp big_file_1 /test/disk1
# cp tiny_file_1 /test/disk1
# cp tiny_file_2 /test/disk1
# cp tiny_file_3 /test/disk1
# cp small_file_4 /test/disk2
# cp tiny_file_4 /test/disk2
# cp tiny_file_5 /test/disk2
# cp tiny_file_6 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
/dev/zd96                     4.9G  4.6G   23M 100% /test/disk1
/dev/zd112                    3.9G  3.6G   74M  99% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.78G      0    32K  /test
test/disk1  7.29G  2.53G  4.76G  -
test/disk2  2.49G      0  3.61G  -

Umm…. Ouch. Where did my pool go? The df command doesn’t even list it anymore, and according to zfs list, it’s completely full. Also, I have this mess in dmesg:

[385389.346555] end_request: critical space allocation error, dev zd112, sector 8067072
[385389.346562] end_request: critical space allocation error, dev zd112, sector 8077312
[385389.346567] end_request: critical space allocation error, dev zd112, sector 8097792
[385389.346573] end_request: critical space allocation error, dev zd112, sector 8081408
[385389.346578] end_request: critical space allocation error, dev zd112, sector 8068096
[385389.346583] end_request: critical space allocation error, dev zd112, sector 8058880
[385389.346588] end_request: critical space allocation error, dev zd112, sector 8050688
[385389.346594] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 142606336 size 4194304 starting block 1009792)
[385389.346603] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 142606336 size 8388608 starting block 1012352)
[385389.346611] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 142606336 size 4194304 starting block 1010304)
[385389.346619] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 134217728 size 8388608 starting block 1008640)
[385389.346625] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 138412032 size 4194304 starting block 1007488)
[385389.346637] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 134217728 size 4194304 starting block 1006464)
[385389.346639] buffer_io_error: 170 callbacks suppressed
[385389.346645] Buffer I/O error on device zd112, logical block 1009664
[385389.346647] Buffer I/O error on device zd112, logical block 1010189
[385389.346650] Buffer I/O error on device zd112, logical block 1012247
[385389.346653] Buffer I/O error on device zd112, logical block 1008521
[385389.346656] Buffer I/O error on device zd112, logical block 1007368
[385389.346661] Buffer I/O error on device zd112, logical block 1006337
[385389.346663] Buffer I/O error on device zd112, logical block 1009665
[385389.346665] Buffer I/O error on device zd112, logical block 1010190
[385389.346668] Buffer I/O error on device zd112, logical block 1012248
[385389.346670] Buffer I/O error on device zd112, logical block 1008522
[385389.346726] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 134217728 size 8388608 starting block 1008512)
[385389.346946] end_request: critical space allocation error, dev zd112, sector 8056832
[385389.346959] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 16 (offset 138412032 size 4194304 starting block 1007232)
[385389.346979] end_request: critical space allocation error, dev zd112, sector 8099840
[385389.346995] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 142606336 size 8388608 starting block 1012608)
[385389.346998] end_request: critical space allocation error, dev zd112, sector 8100864
[385389.347006] EXT4-fs warning (device zd112): ext4_end_bio:317: I/O error writing to inode 15 (offset 142606336 size 8388608 starting block 1012864)
[385389.699816] Aborting journal on device zd112-8.
[385389.699839] EXT4-fs error (device zd112) in ext4_free_blocks:4858: Journal has aborted
[385389.711017] EXT4-fs (zd112): Delayed block allocation failed for inode 15 at logical offset 46080 with max blocks 1024 with error 30
[385389.711025] EXT4-fs (zd112): This should not happen!! Data will be lost
[385389.711025] 
[385389.711046] EXT4-fs error (device zd112) in ext4_writepages:2536: Journal has aborted
[385389.712520] EXT4-fs error (device zd112): ext4_journal_check_start:56: Detected aborted journal
[385389.712530] EXT4-fs (zd112): Remounting filesystem read-only
[385389.712644] EXT4-fs (zd112): ext4_writepages: jbd2_start: 1024 pages, ino 16; err -30
[385389.783776] Buffer I/O error on device zd112, logical block 786433
[385389.783784] lost page write due to I/O error on zd112
[385389.783806] Buffer I/O error on device zd112, logical block 786434
[385389.783809] lost page write due to I/O error on zd112

Indicating that disk2 is now mounted read-only:

# touch /test/disk2/foo
touch: cannot touch \u2018/test/disk2/foo\u2019: Read-only file system

Yup, read-only. Let’s try this again, but with a much-larger overlap between the two disks:

# umount /test/disk2
# umount /test/disk1
# zpool destroy test
# zpool create test test_pool
# zfs create -V 3.5G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp big_file_1 /test/disk1
# cp small_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2/
# zfs set volsize=5G test/disk1
# zfs set volsize=4G test/disk2
# resize2fs /dev/zvol/test/disk1 
# resize2fs /dev/zvol/test/disk2
# cp small_file_2 /test/disk1
# cp tiny_file_1 /test/disk1
# cp tiny_file_2 /test/disk1
# cp tiny_file_3 /test/disk1
# cp tiny_file_4 /test/disk2
# cp tiny_file_5 /test/disk2
# cp tiny_file_6 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          900M     0  900M   0% /test
/dev/zd96                     4.9G  4.6G   16M 100% /test/disk1
/dev/zd112                    3.9G  3.6G   74M  99% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.90G   899M    32K  /test
test/disk1  8.31G  4.43G  4.76G  -
test/disk2   607M   899M  3.75G  -

Well, at least this time I didn’t run out of disk space. No errors in dmesg either. I don’t see a huge space savings, though: 4.6G on one disk, 3.6G on the second, 3G “shared” in the snapshot… total disk usage: 8.9G??? In fact, it seems to take up more total space than if I were to just store those same files in the pool directly. Let’s try that. In my two zvols, I have stored two big files, three small files, and six tiny files:

# umount /test/disk1
# umount /test/disk2
# zpool destroy test
# zpool create test /storage/tmp/test_pool
# cp big_file_1 /test
# cp big_file_2 /test
# cp small_file_1 /test
# cp small_file_2 /test
# cp small_file_3 /test
# cp tiny_file_1 /test
# cp tiny_file_2 /test
# cp tiny_file_3 /test
# cp tiny_file_4 /test
# cp tiny_file_5 /test
# cp tiny_file_6 /test
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          9.8G  8.2G  1.7G  84% /test

# zfs list -r test
NAME   USED  AVAIL  REFER  MOUNTPOINT
test  8.18G  1.60G  8.18G  /test

Perhaps resizing the zvols caused some kind of disconnect in the data sharing? Good theory. Let’s try it without resizing:

# umount /test/disk1
# umount /test/disk2
# zpool destroy test
# zpool create test test_pool
# zfs create -V 4.5G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp big_file_1 /test/disk1/
# cp small_file_1 /test/disk1
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# zfs destroy test/disk2
# zfs destroy test/disk1@s1
# sync
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2
# df -h | grep "test\|Size"
# cp small_file_2 /test/disk1
# cp small_file_3 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          893M     0  893M   0% /test
/dev/zd96                     4.4G  4.1G   57M  99% /test/disk1
/dev/zd112                    4.4G  4.1G   57M  99% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.91G   892M    32K  /test
test/disk1  7.90G  4.50G  4.27G  -
test/disk2  1.01G   892M  4.27G  -

Again, around 8G of data, with 3G shared, takes about 8.9G of storage space. I’m not impressed with cloning zvols as a means to save on storage space.

Dedup

What does it mean to dedup data between zvols? When data is written to disk2 that is a duplicate of data already in disk1, where does the extra space saved by the dedup logic go to? Let’s experiment:

# zfs create -V 5G test/disk1
# zfs set dedup=on test
# zfs create -V 4G test/disk2
# mkdir /test/disk1
# mkdir /test/disk2
# mkfs.ext4 /dev/zvol/test/disk1 
# mkfs.ext4 /dev/zvol/test/disk2
# mount /dev/zvol/test/disk1 /test/disk1
# mount /dev/zvol/test/disk2 /test/disk2
# cp big_file_1 /test/disk1
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          790M  128K  790M   1% /test
/dev/zd96                     4.8G  2.1G  2.6G  45% /test/disk1
/dev/zd112                    3.9G  8.0M  3.6G   1% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.35G   798M    32K  /test
test/disk1  5.16G  3.75G  2.19G  -
test/disk2  4.13G  4.72G   194M  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.09G  7.84G    21%  1.17x  ONLINE  -

# cp big_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          2.8G  128K  2.8G   1% /test
/dev/zd96                     4.8G  2.1G  2.6G  45% /test/disk1
/dev/zd112                    3.9G  2.1G  1.6G  56% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.38G  2.76G    32K  /test
test/disk1  5.16G  5.69G  2.23G  -
test/disk2  4.13G  4.67G  2.21G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.15G  7.79G    21%  2.19x  ONLINE  -

# cp small_file_1 /test/disk1
# cp small_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          3.8G     0  3.8G   0% /test
/dev/zd96                     4.8G  3.1G  1.6G  67% /test/disk1
/dev/zd112                    3.9G  3.1G  603M  84% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.42G  3.70G    32K  /test
test/disk1  5.16G  5.62G  3.24G  -
test/disk2  4.13G  4.60G  3.22G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.26G  6.68G    32%  2.12x  ONLINE  -

# cp small_file_2 /test/disk1
# cp tiny_file_1 /test/disk1
# cp tiny_file_2 /test/disk1
# cp tiny_file_3 /test/disk1
# cp tiny_file_4 /test/disk2
# cp tiny_file_5 /test/disk2
# cp tiny_file_6 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          3.7G     0  3.7G   0% /test
/dev/zd96                     4.8G  4.6G     0 100% /test/disk1
/dev/zd112                    3.9G  3.6G  2.9M 100% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.46G  3.66G    32K  /test
test/disk1  5.16G  3.97G  4.85G  -
test/disk2  4.13G  3.96G  3.82G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  5.64G  4.30G    56%  1.65x  ONLINE  -

Now this shows some promise! This is what I was hoping to see with my cloning tests!

Compression

Is compression going to work the same as dedup? Will I see the space savings as a return of storage to the pool? Let’s see.

# zfs create -V 9G test/disk1
# zfs set compression=on test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# dd if=/dev/zero of=/test/disk1/file1 bs=1G count=1
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M     0  511M   0% /test
/dev/zd96                     8.8G  1.1G  7.3G  13% /test/disk1

# dd if=/dev/zero of=/test/disk1/file2 bs=1G count=5
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M  128K  511M   1% /test
/dev/zd96                     8.8G  6.1G  2.3G  73% /test/disk1

# dd if=/dev/zero of=/test/disk1/file3 bs=1G count=2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M     0  511M   0% /test
/dev/zd96                     8.8G  8.1G  255M  97% /test/disk1

# dd if=/dev/zero of=/test/disk1/file4 bs=1M count=250
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          511M     0  511M   0% /test
/dev/zd96                     8.8G  8.3G  4.4M 100% /test/disk1

# zfs get compressratio test/disk1
NAME        PROPERTY       VALUE  SOURCE
test/disk1  compressratio  11.65x  -

# resize2fs /dev/zvol/test/disk1 
resize2fs 1.42.9 (4-Feb-2014)
The filesystem is already 2359296 blocks long.  Nothing to do!

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        9.28G   510M    31K  /test
test/disk1  9.28G  9.78G   334K  -

As far as I can tell, the data really is compressed on disk. The compressratio is large (though significantly smaller than what I would expect). The dd commands each completed very quickly, indicating that little data was actually written to disk. But I can’t seem to figure out how to take advantage of the space savings that the compression provided. I’m somewhat disappointed.

Dedup combined with cloning

Well, dedup will give me the space savings that I was hoping to achieve by cloning zvols. But can I turn on dedup and use cloning to create multiple copies of a VM that all start in a common state? Let’s try that.

# zfs set dedup=on test
# zfs create -V 3G test/disk1
# mkdir /test/disk1
# mkfs.ext4 /dev/zvol/test/disk1 
# mount /dev/zvol/test/disk1 /test/disk1/
# cp big_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# zfs clone test/disk1@s1 test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.6G     0  4.6G   0% /test
/dev/zd96                     2.9G  2.1G  738M  74% /test/disk1
/dev/zd112                    2.9G  2.1G  738M  74% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        5.29G  4.60G    32K  /test
test/disk1  5.23G  7.69G  2.13G  -
test/disk2   185K  4.60G  2.13G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.09G  7.85G    21%  1.05x  ONLINE  -

# zfs set volsize=5G test/disk1
# zfs set volsize=4G test/disk2
# resize2fs /dev/zvol/test/disk1 
# resize2fs /dev/zvol/test/disk2
# cp small_file_1 /test/disk1
# cp small_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          2.5G     0  2.5G   0% /test
/dev/zd96                     4.9G  3.1G  1.7G  66% /test/disk1
/dev/zd112                    3.9G  3.1G  680M  82% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.42G  2.46G    32K  /test
test/disk1  7.29G  6.60G  3.15G  -
test/disk2  1.02G  2.46G  3.15G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.17G  6.77G    31%  1.37x  ONLINE  -

Remember, in my dedup tests above, when I had two filesystems with both big_file_1 and small_file_1 on each, I had 3.8G free space, and zpool list reported a 2.12x dedup ratio. Seeding my new filesystem with a clone of a snapshot isn’t going too well. Maybe if I overwrite the cloned copy of big_file_1? Perhaps it will notice then that it is a duplicate?

# cp big_file_1 /test/disk2
# sync

# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          2.4G  128K  2.4G   1% /test
/dev/zd96                     4.9G  3.1G  1.7G  66% /test/disk1
/dev/zd112                    3.9G  3.1G  680M  82% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        10.5G  2.34G    32K  /test
test/disk1  7.29G  6.48G  3.15G  -
test/disk2  3.04G  2.34G  3.76G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.25G  6.69G    32%  2.03x  ONLINE  -

Hmmm… The dedup ratio went up (though not quite to 2.12x) but the available space actually went down! What’s with that? I’m going to try to replace the copy of big_file_1 on disk 1.

# cp big_file_1 /test/disk1
# df -h | grep "test\|Size"
Filesystem                    Size  Used Avail Use% Mounted on
test                          4.1G  128K  4.1G   1% /test
/dev/zd96                     4.9G  3.1G  1.7G  66% /test/disk1
/dev/zd112                    3.9G  3.1G  680M  82% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        10.5G  4.25G    32K  /test
test/disk1  7.29G  6.42G  4.65G  -
test/disk2  3.04G  4.25G  3.76G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  3.40G  6.53G    34%  2.70x  ONLINE  -

There we go. Now I’m seeing similar results to what I saw in the original dedup tests. My conclusion: deduplication doesn’t do anything when cloning zvols.

Dedup with Send / Receive

I expect this should work, but I’ve been surprised with some results so far, so it’s worth experimenting just to be sure.

# zpool create test test_pool
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            9.8G     0  9.8G   0% /test

# zfs list -r test
NAME   USED  AVAIL  REFER  MOUNTPOINT
test   108K  9.78G    30K  /test

# zfs set dedup="on" test
# zfs create -V 3G test/disk1
# mkfs.ext4 /dev/zvol/test/disk1
# mkdir /test/disk1
# mount /dev/zvol/test/disk1 /test/disk1
# cp small_file_1 /test/disk1
# sync
# zfs snap test/disk1@s1
# cp small_file_2 /test/disk1
# sync
# zfs snap test/disk1@s2
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            5.7G     0  5.7G   0% /test
/dev/zd0        2.9G  2.1G  738M  74% /test/disk1

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        4.28G  5.61G    31K  /test
test/disk1  4.22G  7.69G  2.13G  -

# zfs create -V 3G test/disk2
# zfs send test/disk1@s1 | zfs receive -F test/disk2
# mkdir /test/disk2
# mount /dev/zvol/test/disk2 /test/disk2
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            2.5G     0  2.5G   0% /test
/dev/zd0        2.9G  2.1G  738M  74% /test/disk1
/dev/zd16       2.9G  1.1G  1.8G  37% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        8.51G  2.47G    32K  /test
test/disk1  4.22G  4.55G  2.13G  -
test/disk2  4.22G  5.56G  1.12G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.13G  7.80G    21%  1.60x  ONLINE  -

# umount /test/disk2 
# zfs send -i s1 test/disk1@s2 | zfs receive -F test/disk2
# mount /dev/zvol/test/disk2 /test/disk2
# df -h | grep "test\|Size"
Filesystem      Size  Used Avail Use% Mounted on
test            1.5G  128K  1.5G   1% /test
/dev/zd16       2.9G  2.1G  738M  74% /test/disk1
/dev/zd0        2.9G  2.1G  738M  74% /test/disk2

# zfs list -r test
NAME         USED  AVAIL  REFER  MOUNTPOINT
test        10.6G  1.41G    32K  /test
test/disk1  5.23G  4.50G  2.13G  -
test/disk2  5.23G  4.50G  2.13G  -

# zpool list test
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
test  9.94G  2.15G  7.79G    21%  2.10x  ONLINE  -

# zfs create -V 3G test/disk3
cannot create 'test/disk3': out of space

I don’t really get this one. The `df` and `zfs list` results indicate that it’s not working. Available space stays at about 1.5G, regardless of the amount of overlap. However, the `zpool list` shows about 7.8G free, and it does indicate that the dedup ratio is increasing as I increase the amount of overlap between the zvols.

In the end, though, the last statement is telling. Even though the two zvols are 66% overlapping (2G out of 3G identical) there isn’t enough room in the pool to create a third zvol. I would expect this to work. First zvol uses 3G, right? Second zvol uses 1G (3G – 2G dup data). That means out of my 10G pool, only 4G is in use. I should have plenty of space available for another 3G zvol. If I can’t use the space savings, what’s the point?

Conclusion

When working with VMs on ZFS zvols, if space savings is a major concern for you, I recommend the following:

Create a ZFS pool to hold all of your VM root filesystems
Turn on deduplication across the entire pool
Do not use compression
Do not clone a new machine from a snapshot of an old machine
Do not use `zfs send` / `zfs receive` to create a new machine from a snapshot of an old machine

Or maybe better yet, don’t try to use dedup across zvols. The results are pretty confusing, and I’m not sure it’s worth it.

All Things Dad

Play it again, Dad

June 22, 2014 danBhentschel Leave a comment

One of my goals as father is to help my children to find joy in their everyday, mundane tasks. I have always been a huge fan of music, and so it seems natural to me to pass on that joy of music to my young charges. With the advent of connected, do-anything, smart phones, these two objectives (enjoying daily tasks and learning music) coincide quite nicely. Let me elaborate.

I have my phone on my person at almost all times of the day. Outside doing yard work, in the bathroom brushing the kids’ teeth, cooking breakfast in the morning… At any time, I can switch on the tunes. My favorite time to crank it up is when getting ready for bed at night. Kids are going every-which-way, bedrooms, bathrooms, up and down stairs. Everyone is tired. Often emotions are high. A little bit of music just helps to smooth things over a bit.

Granted, my little phone doesn’t have the greatest audio quality, but for most situations, it is perfectly suited for some fun, unobtrusive, background accompaniment. I even have a playlist with kid-friendly music setup on my phone for just this purpose. It’s currently running around 120 songs, and I’m constantly on the lookout for new material to add to the mix.

Good sources of new music include:

Kids’ movies (especially the end credits)
Sports events (including things like Monster Jam)
Live shows (musicals, Disney on Ice, circus, etc.)
Televised events (parades, halftime shows, etc.)
School events (talent show, assemblies, etc.)

I have an application on my phone that I keep handy for just such times. If I hear a song that sounds exciting, or it is associated with a memorable event, then I launch “Sound Hound” on my phone. It listens to the music for a few seconds, and then (usually) is able to tell me the name of the song, and the artist. It even provides a handy link for me to buy the track from Amazon. I normally will do so (only $.99) and instantly add it to the kids’ playlist. Then later on, when they are brushing their teeth or taking a bath, the song will come on, and they will be all excited. “That’s the song from the movie we saw today, Daddy!”

I also get the inverse fairly frequently: we’ll be out and about, shopping at Wegmans or sitting at dinner at a restaurant, and one of the kids (usually Jasmin) will hear a song come on the radio and inform me, “That’s the song from on your phone, Daddy!”

It’s not a big thing. It doesn’t cost all that much in either time or money, and yet it makes quite an impression. I love it when Timmy is all excited to hear the Kidz Bop version of “Thrift Shop” come on, or when Jasmin starts hopping around (often in her undies while dressing) to the sound of One Direction, or when Jacob starts shadow boxing to “Eye of the Tiger”.

It all boils down to one simple fact: Making them happy makes me happy!