Sigil.us

1

Storage Management :: Global File Systems and ZFS

One enterprise architecture feature that every admin dreams of is simple storage management. Having one storage network and a global filesystem that all nodes can access, and being able to easily have space “appear” on the file system by adding LUNs to the global file system is a huge time saver.

Redhat Linux

GFS(2)
Redhat has done well with GFS(2) from a systems management perspective (I have read that it doesn’t scale as well as lustre). The cluster file system technology is pretty simple comparatively speaking even though there are several layers involved. You don’t need metadata controllers or independent systems acting as lock managers. You do need to either have fencing hardware, or a manual fencing process in place to account for failed nodes.

Adding storage involves using LVM to add the LUN to the volume, expand the volume, then using gfs2_grow to expand the filesystem.

At its simplest a configuration could look like the following:

Sun Solaris/OpenSolaris

SAM-QFS
Sun has had SAM-QFS for quite some time. It is more complex than GFS(2). To be fair it also has more features. It requires dedicated metadata controllers. This solution doesn’t interest me much due to the fact that I use Solaris/OpenSolaris for ZFS to simplify storage management.

Adding storage to the shared QFS file system appears to involve a similar process to GFS(2).

At its simplest a configuration could look like the following:

Lustre, BTRFS/CRFS, ZFS and the Future

Lustre
Lustre seems to be more complex than both GFS(2) and SAM-QFS. It also seems to be more scalable. It requires 2 types of controllers: MDS (metadata) and OSS (storage). So from a management perspective it requires quite a bit more overhead.

It looks like ZFS will be added as a backend storage format in the future versions 2.x+ this might simplify LUN management a little for the solution.

BTRFS/CRFS
BTRFS is a Linux ZFS workalike. It is still in development but the final version is shaping up to have ZFS feature parity (minus easy device/raid management, zvols, and other sweet things like zfs send recv). What interests me however is CRFS.

CRFS also in development, appears to be a global network filesystem that basically exports BTRFS file systems.

ZFS
What I would like to see is ZFS to become cluster aware. Being able to use OpenSolaris as a storage host and export either a zfs or a zpool to nodes, and perhaps run a lockmanager process similar to GFS(2) would be pretty slick.

For now with ZFS

Creating ZFS file systems on disk boxes connected to a SAN fabric really isn’t that difficult. It is however a bit time consuming, and you really should configure LUN masking for each LUN and node. Other wise you run the risk of accidentally corrupting data by using the same LUN at the same time on different nodes.

Exporting zvols with Comstar/iSCSI also doesn’t seem like the best idea, because again you should configure target portal groups and host access, and I don’t think that running ZFS on top of a ZFS based virtual block device (zvol) would be great for performance. Jeff Bonwick had stated something about improving the zvol performance (block pass through?) in his last ZFS talk. I am not sure wether it has been done yet or not. There also are no best practice notes about ZFS on iSCSI zvols.

I would think using UFS on a zvol would be better performance wise, but the whole purpose of the exercise for me is to provide snapshot functionality to the host consuming the filesystem, and filesystem delegation to the zones on the host.

I would really like to be able to use zvols as opposed to SAN fabric LUNs because then the zones could use snapshots, I could mirror the zvols from a Comstar storage host to a remote system for disaster recovery, and only have to deal with one zpool.

So how about it? Is anyone else using Comstar, ZFS, and OpenSolaris in the fashion I have described What is your strategy for consolidating storage management with OpenSolaris?

Read More

0

RedHat EL5 :: Surprisingly Better

So I have been putting RHEL5.4 through its paces in preparation for the RHCE exam. It has come a long way since version 4. Yum is actually pretty ok. The distro itself is snappy even on garbage hardware. All in all though not as bad as I remember RHEL being.

Things that still bother me vs Ubuntu LTS:

• Comparatively small distro package set.
• File system layout in respect to etc and var: stuff is all over the place. Seriously each configurable software package having its own directory in etc and var sub directories (log etc) is nice. It is uniform and easy to guess where things might be.
• $300/yr

Read More

0

Life :: Todo

Well the weather has turned which means (in Michigan anyway) productivity reigns. In addition to my normal work I would like to accomplish the following by June:

• Write one webscale application in Grails
• Write one webscale application in Ruby on Rails
• Write some horribly convoluted shell app that demonstrates all features of bash shell scripting as an exercise
• Apply for a design patent on the idea backing the Grails application
• Complete RHCE RedHat Certified Engineer certification
• Complete SCSA Sun Certified System Administrator certification

I think the only really irritating item on the list is the SCSA cert. I will need to buy a crappy sparc box to get used to openboot again. Wish I could afford a T1000 at least I could use that for something when I was done. Also not a huge fan of Solaris 10 but it will pay to be certified because I would imagine most contract work would involve Solaris 10 -> 11 migrations or Solaris 10 -> RedHat (bummer).

Read More

4

SX:CE EOL: THANK GOD

I have been following this discussion closely and most perspectives voiced have been those of long time Solaris users or those that have invested in SX:CE for production deployments (why you would choose to do so is beyond me entirely, considering that the intent to discontinue SX:CE has been well known for quite some time, and the original purpose of the release was for early access/testing). So I would like to relate my perspective as a Linux convert and general server OS/Management critic. I use Linux/OpenSolaris on the server as a server OS, I use OS X on the desktop.

From Linux to OpenSolaris

When we first started building our hosting platform in 2000 I already had pretty strong background using RedHat Linux and OpenBSD for a significantly sized deployment at a fulfillment house that hosted some rather large web applications. Back then (1997) RedHat didn’t have yum so package management was a very manual affair. Although it was better than just compiling and installing software. RedHat decided to make the shift to the Enterprise distro and for $300 a year per machine you could get paid updates using up2date. Neat, we ran on RedHat 7.3 until they announced EOL on the updates. The fact is RPM package management was not good enough to pay for. Even with yum, installation and removal of dependencies are regularly mangled. If you are building and deploying your own packages this is even more pronounced. You can FORCE it all to work but I really am not into that. Good bye RedHat.

Next stop Debian. Debian was pretty great. Apt was pretty amazing. I really wish I had started using Debian from the start. Nearly flawless dependency addition and removal on package install uninstall and upgrading to a new release was very easy if I decided to do so. Nearly every open source package I used was already built for Debian and in the release repository. Everything was 2 commands away. The problem with Debian: no regular release schedule. I could never plan on having x feature at x date and I had no idea when the security/errata updates would cease for a given release. This was a huge problem for platform development and management. Good bye Debian.

Next stop Ubuntu LTS. Great! I now had a release schedule, and I knew exactly how long security/errata updates would be supplied for a given release.

The Linux draw: a summary

• Not having to build packages due to a huge software repository and 3rd party trusted repositories: TIME SAVED
• Ease of setting up internal repository:  TIME SAVED
• No need to manually resolve package dependencies: TIME SAVED
• Keep all systems up to date with latest security updates, two commands: TIME SAVED
• Deploy a new system install with just a required package list: TIME SAVED
• Not having to setup and maintain a custom system deployment architecture: TIME SAVED
• Regular release schedule and errata/security update time line: FUTURE ASSURED

On Solaris: I had never really been interested in Solaris until sun decided to open source the OS. Prior to that I had to maintain a pile of Solaris 8 machines for a client that was heavily invested in Coldfusion. This experience had pretty much soured my view of the OS. The package management was terrible. Installing patches was painful. Installing software was painful. What good is instant deployment if managing the system after install is so damn time consuming? At any rate managing Solaris 8 and later 9 was far more work than managing RedHat pre version 7.0 and that was only involving keeping the security patches on the systems current.

The OpenSolaris project pulled me into Solaris land again. Why? IPS, ZFS, Zones, SMF, DTrace, and all the goodies that make OpenSolaris a compelling storage platform. I started using OpenSolaris for storage only. I really liked the uniform nature of the admin utilities. I decided to also deploy Sun Communications Suite on OpenSolaris. Although I had to play a little bit with the installer and hack up patchadd to allow patches to be applied to comms I am happy with it.

OpenSolaris over Ubuntu: a summary

• IPS/PKG is like a nextgen APT/DEB with ZFS integration. beadm, is great, being able to apply updates to a new boot environment and activate that environment, then rollback if there is an issue is priceless. I also like the IPS repo system better than APT’s
• ZFS
• Zones, sorry KVM doesn’t cut it.
• Crossbow (can I dump OpenBSD for firewall/VLAN management/VPN concentration/routing? Time will tell)
• SMF
• DTrace
• OpenSolaris management utilities are far more uniform than the Linux equivalents
• OpenSolaris seems to perform quite a bit better for certain streaming applications
• Sun’s JDK/JRE is available in the release repo
• I haven’t benchmarked this but I swear that Java server daemons (in this instance a custom SMTP server) process requests faster on OpenSolaris than on Linux

As you can see with OpenSolaris I can have my cake and eat it too. With OpenSolaris I see the future perfect server OS. I could run ONE OS on all my server hardware and not have to deal with the issues of managing Ubuntu LTS, and OpenBSD based on their independent strengths. Now obviously there are issues that need to be resolved and the developers at Sun have said these will be resolved.

Issues: a summary

• GNU userland vs Solaris userland shell path: Please stop with this, I really don’t care either way the matter is so trivial so simple to change it just isn’t something that should justify all the attention that it receives
• Automated Install: So far this seems like a slick setup. I haven’t evaluated it much due to the fact that a simple package list install from a repo is fast enough to satisfy my needs. In any event the developers have stated they are working on polishing the feature
• IPS needs polishing: So far all the issues I have with IPS have been acknowledged as such, or there is an existing RFE
• No Text installer: Next release will have a text installer
• Sparse Zones: This is an issue for me and I would like to see it resolved. However for some reason inherit pkg-dir umm seems to work for me for my Java server apps??? I even see the loopback mounts and everything runs? Haven’t really looked into it much past that…
• Updating Zones: The process of updating a zone’s packages via image-update is too convoluted this needs to be much easier.
• Sun enterprise repo. I would like to see the comms components packaged for OpenSolaris and updates enabled via IPS instead of SVR4 patches (yuk)
• Stable release schedule and at least security updates and severe errata updates in the release repo. I am assuming this is a resource related issue and that it will be resolved after the next release.

Going Forward

EOLing SX:CE is the best possible choice going forward. Obviously Sun’s developer resources are limited. If those resources can be devoted to resolving the issues with OpenSolaris we can move forward out of the SVR4 nightmare that inhibits the adoption of Solaris by users THAT WILL NEVER GIVE UP SANE PACKAGE AND UPDATE MANAGEMENT.

I am sick of looking over the listings on career builder and dice and seeing all the jobs involving Solaris to RedHat migrations. More of the same will not curtail the mass exodus from Solaris to RedHat. That is what SX:CE is; more of the same.

Read More