If you've read my last post, you'll already have some background on the program that I'm going to use to demonstrate an awesome side effect of that particular program(mhddfs). If you haven't, read it now and then come back to this post.
This is an easy 4 step process.
This is the (hypothetical, I don't need it myself) situation: I've got 20GB of data that I need to burn to DVD(4.7GB single layer discs) and I can't find a no-nonsense tool to split up this data logically so that I can burn all of the data using as much space as possible.
Step 1: Create enough 4.7GB disk images to fit 20GB of data(around 5) and mount them. Note the paths where they're mounted at.
Step 2: Bind them all together using mhddfs.
Step 3: Copy all of the data to burn into the mhddfs mounted volume.
Step 4: Unmount all of the volumes/images and burn the contents of the images onto DVDs.
(optional) Step 5: Profit!
Problem solved.
It actually works really well for DVDs because of the way mhddfs decides where to place files. Its default behaviour is to use the next lowest priority folder when the current priority's size has reached approximately 4.2GB. Eg. You drop 1,000 equally sized files totalling 4.4GB onto the mhddfs volume(2 images, 4.7GB each. A has priority 1, B has 2), around 950 make their way to A and the rest get pushed over to B. Note that this value can be changed in the mount options so if you want to use more of the image, consult the manual page/readme to find the option(specified in bytes if I remember correctly).
Friday, July 24, 2009
Sunday, July 19, 2009
File Encryption
I've found an easy way to have an encrypted disk image that can expand it's size arbitrarily and span several discs. The really cool part: you can read each disc on it's own. Also works for any set of folders across any set of drives or even across NFS/SSHFS/etc.
A few requirements: you need to be on a platform that supports Fuse, that platform must support encypted disk images of arbitrary size.
First, grab mhddfs @ this link (an excellent google translation) and compile it. Second, grab a tool such as TrueCrypt or if you use OSX, fire up Disk Utility.
Third, create at least 2 encrypted images in the tool of your choice and using any size you want. If this is data you wish to archive to optical media, make the images to fit the media(that's what I do). Mount each image and note the path they're each mounted at.
Suppose you have 2 images, A and B, and they're mounted at /Volumes/A and /Volumes/B. You wish to mount the combined volume as AB. Open up a terminal and execute
Now /Volumes/AB contains the union of A and B and if the same file(only file, folders are merged) exists in both, mhddfs takes the version in A(the first folder passed to it, B is the second and therefore second priority). You can do this for an arbitrary amount of disc images(actually, mhddfs works on folders so if you have a number of related files spread across several folders/drives you can bind them all together into one folder).
There's only one caveat: mhddfs won't split files larger than the largest amount of free space in any one folder across all of the folders. So if all your disc images are 2GB and you drop a 2.1GB file in, it won't work. This means that any file has to be able to fit in exactly one disc image/folder. Eg: you have 2 disc images of 2GB, A has 500MB free space and B the same. Therefore, mhddfs reports 1GB free on the volume so you figure you can drop a 1GB file onto it. Wrong! This file has to be able to fit into A or B and neither have the required free space so you get an out of space error. mhddfs won't give up if the first folder doesn't have the free space. It also won't move a file to another of the source folders if that would free up enough space. In other words, it won't mess with the structure as it is. If you move a file in AB that's stored on A into a folder that's stored on B, it'll create the folder on A and move it into that.
The best part: go ahead and make another disc image, adjust the command line for mhddfs(add another comma+folder after the last one), and mount! Eg. I want to add a volume C to the union. The mhddfs command line would be:
A few more notes: the order you put the folders on the command line matters, mhddfs uses that to determine where to place a file. The order=priority with the first folder being 1st priority, second being 2nd priority, etc.
I've found several uses beyond encrypted disc images. I've got torrents spread across several drives because there's not enough room on any one drive(all of 'em are small drives). I've bound all the folders together into one superfolder where I can even set my client to download to. mhddfs works out the details about which drive to write to!
Props to the guy who wrote this!
Just realized this post was all over the place. I'm tired and don't wanna go back and fix it so tough.
A few requirements: you need to be on a platform that supports Fuse, that platform must support encypted disk images of arbitrary size.
First, grab mhddfs @ this link (an excellent google translation) and compile it. Second, grab a tool such as TrueCrypt or if you use OSX, fire up Disk Utility.
Third, create at least 2 encrypted images in the tool of your choice and using any size you want. If this is data you wish to archive to optical media, make the images to fit the media(that's what I do). Mount each image and note the path they're each mounted at.
Suppose you have 2 images, A and B, and they're mounted at /Volumes/A and /Volumes/B. You wish to mount the combined volume as AB. Open up a terminal and execute
shell# mkdir /Volumes/AB
shell# mhddfs /Volumes/A,/Volumes/B /Volumes/AB -o local,volname="AB",auto_xattr
Now /Volumes/AB contains the union of A and B and if the same file(only file, folders are merged) exists in both, mhddfs takes the version in A(the first folder passed to it, B is the second and therefore second priority). You can do this for an arbitrary amount of disc images(actually, mhddfs works on folders so if you have a number of related files spread across several folders/drives you can bind them all together into one folder).
There's only one caveat: mhddfs won't split files larger than the largest amount of free space in any one folder across all of the folders. So if all your disc images are 2GB and you drop a 2.1GB file in, it won't work. This means that any file has to be able to fit in exactly one disc image/folder. Eg: you have 2 disc images of 2GB, A has 500MB free space and B the same. Therefore, mhddfs reports 1GB free on the volume so you figure you can drop a 1GB file onto it. Wrong! This file has to be able to fit into A or B and neither have the required free space so you get an out of space error. mhddfs won't give up if the first folder doesn't have the free space. It also won't move a file to another of the source folders if that would free up enough space. In other words, it won't mess with the structure as it is. If you move a file in AB that's stored on A into a folder that's stored on B, it'll create the folder on A and move it into that.
The best part: go ahead and make another disc image, adjust the command line for mhddfs(add another comma+folder after the last one), and mount! Eg. I want to add a volume C to the union. The mhddfs command line would be:
shell# mhddfs /Volumes/A,/Volumes/B,/Volumes/C /Volumes/ABC -o local,volname="ABC",auto_xattr
A few more notes: the order you put the folders on the command line matters, mhddfs uses that to determine where to place a file. The order=priority with the first folder being 1st priority, second being 2nd priority, etc.
I've found several uses beyond encrypted disc images. I've got torrents spread across several drives because there's not enough room on any one drive(all of 'em are small drives). I've bound all the folders together into one superfolder where I can even set my client to download to. mhddfs works out the details about which drive to write to!
Props to the guy who wrote this!
Just realized this post was all over the place. I'm tired and don't wanna go back and fix it so tough.
Subscribe to:
Posts (Atom)
