VMware and encfs, Part 2

Perhaps before declaring victory on storing a VMware virtual machine on an encfs encrypted filesystem, I should have tried powering on the VM. When I tried that, I found that it refused to start. Viewing the log revealed the following message:

Oct 28 23:38:59: vcpu-0| Could not mmap paging file : No such device
Oct 28 23:38:59: vcpu-0| Failed to allocate page for guest RAM!

Searching the VMware community forums turned up someone having a similar problem when storing the VM on an NTFS filesystem accessed via a FUSE driver.

Apparently VMware creates a disk file containing the guest’s physical memory and then reads the file into a memory map. This doesn’t work on a FUSE filesystem like encfs, and so the guest fails. So the solution is to avoid creating the memory-mapped file by adding the following line to the vmx file for the virtual machine:

mainMem.useNamedFile = "false"

With that small tweak, the guest started normally. Really.

VMware and encfs

Posted as a public service for the next person who runs into this.

I was trying to move a VMware virtual machine to a new host, storing the files on a filesystem encrypted with encfs. Previously, when I’ve moved a virtual machine, the process went like this:

• Power down virtual machine.
• Copy virtual machine directory to new host.
• Start VMware console on new host.
• Browse to new virtual machine directory, open virtual machine.
• Power machine on, answer question about UUID’s, and move on with life.

But when I tried to do this in the encrypted directory, I got an the error message “unable to add virtual machine to inventory … No permission to perform this operation.” Googling that resulted in some clues about permissions on the /etc/vmware/vm-list and /etc/vmware/vm-list-private files, which control the virtual machine inventory. But the VMware setup on the host with the encrypted disk was not different from the original host, where the files are writable only by root.  I was convinced it had to do with the encryption. So I moved the VM files off of the encrypted filesystem, and I was able to start it.

With that in mind, I did some research on encfs, thinking that it was having trouble with the large (>4 GB) disk files. I found out that encfs could theoretically handle disks of at least 128GB in size, so that was ruled out as a possibility. But then I remembered that the files containing the virtual machine inventory were only writable by root – which means that some root process adds any virtual machine you open to the inventory. And since by default encfs only allows the owner to see the filesystem (not even root can see it), the attempt to add the encrypted VM to the inventory was failing because VMware couldn’t find the files.

Fortunately, you can tell encfs to allow root to see the encrypted filesystem when you mount it, and once I did that, problem solved.