Sigil.us

Storage Management :: Global File Systems and ZFS

One enterprise architecture feature that every admin dreams of is simple storage management. Having one storage network and a global filesystem that all nodes can access, and being able to easily have space “appear” on the file system by adding LUNs to the global file system is a huge time saver.

Redhat Linux

GFS(2)
Redhat has done well with GFS(2) from a systems management perspective (I have read that it doesn’t scale as well as lustre). The cluster file system technology is pretty simple comparatively speaking even though there are several layers involved. You don’t need metadata controllers or independent systems acting as lock managers. You do need to either have fencing hardware, or a manual fencing process in place to account for failed nodes.

Adding storage involves using LVM to add the LUN to the volume, expand the volume, then using gfs2_grow to expand the filesystem.

At its simplest a configuration could look like the following:

Sun Solaris/OpenSolaris

SAM-QFS
Sun has had SAM-QFS for quite some time. It is more complex than GFS(2). To be fair it also has more features. It requires dedicated metadata controllers. This solution doesn’t interest me much due to the fact that I use Solaris/OpenSolaris for ZFS to simplify storage management.

Adding storage to the shared QFS file system appears to involve a similar process to GFS(2).

At its simplest a configuration could look like the following:

Lustre, BTRFS/CRFS, ZFS and the Future

Lustre
Lustre seems to be more complex than both GFS(2) and SAM-QFS. It also seems to be more scalable. It requires 2 types of controllers: MDS (metadata) and OSS (storage). So from a management perspective it requires quite a bit more overhead.

It looks like ZFS will be added as a backend storage format in the future versions 2.x+ this might simplify LUN management a little for the solution.

BTRFS/CRFS
BTRFS is a Linux ZFS workalike. It is still in development but the final version is shaping up to have ZFS feature parity (minus easy device/raid management, zvols, and other sweet things like zfs send recv). What interests me however is CRFS.

CRFS also in development, appears to be a global network filesystem that basically exports BTRFS file systems.

ZFS
What I would like to see is ZFS to become cluster aware. Being able to use OpenSolaris as a storage host and export either a zfs or a zpool to nodes, and perhaps run a lockmanager process similar to GFS(2) would be pretty slick.

For now with ZFS

Creating ZFS file systems on disk boxes connected to a SAN fabric really isn’t that difficult. It is however a bit time consuming, and you really should configure LUN masking for each LUN and node. Other wise you run the risk of accidentally corrupting data by using the same LUN at the same time on different nodes.

Exporting zvols with Comstar/iSCSI also doesn’t seem like the best idea, because again you should configure target portal groups and host access, and I don’t think that running ZFS on top of a ZFS based virtual block device (zvol) would be great for performance. Jeff Bonwick had stated something about improving the zvol performance (block pass through?) in his last ZFS talk. I am not sure wether it has been done yet or not. There also are no best practice notes about ZFS on iSCSI zvols.

I would think using UFS on a zvol would be better performance wise, but the whole purpose of the exercise for me is to provide snapshot functionality to the host consuming the filesystem, and filesystem delegation to the zones on the host.

I would really like to be able to use zvols as opposed to SAN fabric LUNs because then the zones could use snapshots, I could mirror the zvols from a Comstar storage host to a remote system for disaster recovery, and only have to deal with one zpool.

So how about it? Is anyone else using Comstar, ZFS, and OpenSolaris in the fashion I have described What is your strategy for consolidating storage management with OpenSolaris?

Read More

Life :: Todo

Well the weather has turned which means (in Michigan anyway) productivity reigns. In addition to my normal work I would like to accomplish the following by June:

• Write one webscale application in Grails
• Write one webscale application in Ruby on Rails
• Write some horribly convoluted shell app that demonstrates all features of bash shell scripting as an exercise
• Apply for a design patent on the idea backing the Grails application
• Complete RHCE RedHat Certified Engineer certification
• Complete SCSA Sun Certified System Administrator certification

I think the only really irritating item on the list is the SCSA cert. I will need to buy a crappy sparc box to get used to openboot again. Wish I could afford a T1000 at least I could use that for something when I was done. Also not a huge fan of Solaris 10 but it will pay to be certified because I would imagine most contract work would involve Solaris 10 -> 11 migrations or Solaris 10 -> RedHat (bummer).

Read More

Cloud Computing :: Companies to watch

Cloud Computing the latest tech buzzword. When discussing CC there are, for the most part, two types of companies. Those that allow you to build your own scalable web platforms on their ummm platform, and those that have their own scalable platform on which you can deploy your application.

Joyent
Joyent would fall into the first category. You can use their accelerator technology as building blocks for your own custom cloud based deployment.

What they are doing right (brief):

• NO VIRTUAL DISKS. Unlike amazon you are NOT going to have IO issues at Joyent. virtual disks are not used, all file system access is native.
• NO CONTRACTS. Turn up. Turn down yay!
• PERFORMANCE. Since the Joyent platform is based on OpenSolaris/Solaris Zones performance is quite a bit better than other solutions. Zones outperform Xen/xVM and Vmware by quite a large factor.

Rackspace
Rackspace has several interesting offerings: Cloud Sites, Cloud Servers, and Cloud Files. All of these services are accessible from the same control panel. Which is really quite nice. Cloud Sites is the scalable application hosting service (although that doesn’t really do it justice). Cloud Servers is a Xen based VM host. Cloud Files is similar to amazon S3.

What they are doing right (brief):.

• NO CONTRACTS. Turn up. Turn down yay! (Cloud Servers)
• AFFORDABLE. Host all the sites you want on Cloud Sites starts at $100 a month (Usage is metered upwards after a certain point)
• GREAT MANAGEABILITY. The Rackpace control panels are excellent and all services are accessible from one interface

Google
Google’s appengine product falls into the application hosting category. Pretty much the only cloud python host. They also have a Java environment. There are some limitations to deploying on Google however as time goes on we are seeing many of these limitations disappearing.

What they are doing right (brief):

• NO NEED TO MANAGE THE PLATFORM. This is a boon to developers that just are not platform engineers. Write, then deploy. Done.
• APIs. Google provides a lot of valuable APIs for integration with their services, and utility right out of the box.

Engine Yard
Engine Yard falls into the application hosting category. They would be the premier Ruby on Rails app host. Everything about this company impresses me.

What they are doing right (brief):

• NO NEED TO MANAGE THE PLATFORM. This is a boon to developers that just are not platform engineers. Write, then deploy. Done.
• SLICK MANAGEMENT INTERFACE. I have never seen such a great management interface for a hosting platform. Great job!

Read More

Sun Java System Web Server on OpenSolaris :: Yay!

I am quite happy to see that Update 4 of Java System Web Server 7 was certified to run on OpenSolaris! Can’t wait to try this out for a couple of reasons:

(Keep in mind our current web hosting platform is Ubuntu Server + mods + Apache + mods + PHP + mods)

1. No ZFS support on Linux. Being able to provide snapshot support for individual hosting clients on a shared system would be too sweet! Also being able to stream those snapshots to our SAN for longterm storage (easily) would be even better!
2. The ability to host JSP without screwing around with the associated Apache modules.
3. Having a decent interface for provisioning shared deployments without having to maintain the mess of code that is required to bolt such functionality on to Apache.
4. ZFS (yeah I know sheesh)

GRIN: http://pkg.opensolaris.org/webstack/info/0/sun-webserver7%407.0%2C5.11%3A20081202T021336Z

Read More

OpenSolaris :: Long term deployments

I have no problems using OpenSolaris for my storage systems. Upgrading them to a new release every 6 months is no big deal no matter what sort of pain is involved.

However I am thinking about a deploying OpenSolaris for mail use. My questions are:

1. There is now a release repository, is upgrading from one release to another using IPS painful? I have yet to attempt it.
2. What sort of issues will I run into attempting this?
3. I see some people have been successful at running com suite on OpenSolaris, should I attempt this? Or would this be an exercise in futility.
4. Will we ever see the Suns Enterprise software in the /extra repository?

Thanks in advance for any responses.

Read More

Solaris :: Less Suck Now

I am a huge fan of OpenSolaris, I believe this project will fix the nightmare that is currently the enterprise Solaris distribution. The question is when.

I believe there is still debate on how the OpenSolaris package management changes and installer will make it back into Solaris proper. Does anyone know what the story is?

I wouldn’t care however it doesn’t appear that Sun is going to support its Java Communications Suite on OpenSolaris proper. What are Sun’s plans for this? Being able to install any of the comm suite or enterprise suite components with IPS would be a huge win.

If anyone could point me in the right direction that would be great.

In the mean time I am going to work on a minimal JumpStart configuration that will install a minimal Solaris package set with the packages needed to run comm suite, and have a decent working shell environment.

Read More

ZFS :: File System Awesomeness

I have been building NAS/SAN systems lately for our company’s remote backup product. We started out using Nexenta’s storage appliance software which utilizes ZFS. After using it for quite some time I think I am going to migrate to OpenSolaris as soon as the November release is available. It turns out that after learning about and using ZFS I really don’t need the web interface to accomplish my objectives.

Some of the great things about ZFS:

1. Not having to have RAID cards to use RAID
2. Being able to add storage to a storage pool with out screwing around with a volume manager
3. The ability to import existing storage pools
4. Infinite snapshots that occupy minimal amounts of storage space
5. The ability to use ZFS send and recv to incrementally replicate file systems over SSH links
6. Online error correction, better than RAID data protection
7. Simple pool and file system management
8. Online file system compression and encryption
9. Virtual block device support

Couple this with OpenSolaris’s native in kernel CIFS and iSCSI server and you have the most inexspensive enterprise storage solution ever.

Truly an admins dream. Unfortunately Linux folks most likely will never see ZFS integrated with the Linux kernel. Hope is not lost however, in 3-4 years Oracle’s BTRFS will be complete and seems to have a similar feature set.

In the BSD camp, it seems the FreeBSD community has done a pretty decent job integrating ZFS. There are only a couple of features missing at present and it appears to be stabilizing.

Dragonfly BSD of course has Matt Dillon’s HAMMER file system which I have been following quite closely. I will probably play with it some more in a couple of releases.

Of course the reason I do not typically use the BSD OS’s in production environments is due to the lack of a binary update system for the base distribution. I like to be able to update quickly and update often, and patching and compiling updates from source is far too time consuming. (OpenBSD being the exception).

With Snow Leopard we should see ZFS make its official debut on the Mac platform, I guess time will tell.

While I would like to see this file system integrated with more open source OS’s I am quite content to stick with OpenSolaris for my storage systems.

UPDATE

Oh and Sun, great work on the 7000 series storage appliances, the implementation is completely sublime. I just wish that I could afford one

Read More