Sigil.us

Storage Management :: Global File Systems and ZFS

One enterprise architecture feature that every admin dreams of is simple storage management. Having one storage network and a global filesystem that all nodes can access, and being able to easily have space “appear” on the file system by adding LUNs to the global file system is a huge time saver.

Redhat Linux

GFS(2)
Redhat has done well with GFS(2) from a systems management perspective (I have read that it doesn’t scale as well as lustre). The cluster file system technology is pretty simple comparatively speaking even though there are several layers involved. You don’t need metadata controllers or independent systems acting as lock managers. You do need to either have fencing hardware, or a manual fencing process in place to account for failed nodes.

Adding storage involves using LVM to add the LUN to the volume, expand the volume, then using gfs2_grow to expand the filesystem.

At its simplest a configuration could look like the following:

Sun Solaris/OpenSolaris

SAM-QFS
Sun has had SAM-QFS for quite some time. It is more complex than GFS(2). To be fair it also has more features. It requires dedicated metadata controllers. This solution doesn’t interest me much due to the fact that I use Solaris/OpenSolaris for ZFS to simplify storage management.

Adding storage to the shared QFS file system appears to involve a similar process to GFS(2).

At its simplest a configuration could look like the following:

Lustre, BTRFS/CRFS, ZFS and the Future

Lustre
Lustre seems to be more complex than both GFS(2) and SAM-QFS. It also seems to be more scalable. It requires 2 types of controllers: MDS (metadata) and OSS (storage). So from a management perspective it requires quite a bit more overhead.

It looks like ZFS will be added as a backend storage format in the future versions 2.x+ this might simplify LUN management a little for the solution.

BTRFS/CRFS
BTRFS is a Linux ZFS workalike. It is still in development but the final version is shaping up to have ZFS feature parity (minus easy device/raid management, zvols, and other sweet things like zfs send recv). What interests me however is CRFS.

CRFS also in development, appears to be a global network filesystem that basically exports BTRFS file systems.

ZFS
What I would like to see is ZFS to become cluster aware. Being able to use OpenSolaris as a storage host and export either a zfs or a zpool to nodes, and perhaps run a lockmanager process similar to GFS(2) would be pretty slick.

For now with ZFS

Creating ZFS file systems on disk boxes connected to a SAN fabric really isn’t that difficult. It is however a bit time consuming, and you really should configure LUN masking for each LUN and node. Other wise you run the risk of accidentally corrupting data by using the same LUN at the same time on different nodes.

Exporting zvols with Comstar/iSCSI also doesn’t seem like the best idea, because again you should configure target portal groups and host access, and I don’t think that running ZFS on top of a ZFS based virtual block device (zvol) would be great for performance. Jeff Bonwick had stated something about improving the zvol performance (block pass through?) in his last ZFS talk. I am not sure wether it has been done yet or not. There also are no best practice notes about ZFS on iSCSI zvols.

I would think using UFS on a zvol would be better performance wise, but the whole purpose of the exercise for me is to provide snapshot functionality to the host consuming the filesystem, and filesystem delegation to the zones on the host.

I would really like to be able to use zvols as opposed to SAN fabric LUNs because then the zones could use snapshots, I could mirror the zvols from a Comstar storage host to a remote system for disaster recovery, and only have to deal with one zpool.

So how about it? Is anyone else using Comstar, ZFS, and OpenSolaris in the fashion I have described What is your strategy for consolidating storage management with OpenSolaris?

Read More

Life :: Todo

Well the weather has turned which means (in Michigan anyway) productivity reigns. In addition to my normal work I would like to accomplish the following by June:

• Write one webscale application in Grails
• Write one webscale application in Ruby on Rails
• Write some horribly convoluted shell app that demonstrates all features of bash shell scripting as an exercise
• Apply for a design patent on the idea backing the Grails application
• Complete RHCE RedHat Certified Engineer certification
• Complete SCSA Sun Certified System Administrator certification

I think the only really irritating item on the list is the SCSA cert. I will need to buy a crappy sparc box to get used to openboot again. Wish I could afford a T1000 at least I could use that for something when I was done. Also not a huge fan of Solaris 10 but it will pay to be certified because I would imagine most contract work would involve Solaris 10 -> 11 migrations or Solaris 10 -> RedHat (bummer).

Read More

Cloud Computing :: Companies to watch

Cloud Computing the latest tech buzzword. When discussing CC there are, for the most part, two types of companies. Those that allow you to build your own scalable web platforms on their ummm platform, and those that have their own scalable platform on which you can deploy your application.

Joyent
Joyent would fall into the first category. You can use their accelerator technology as building blocks for your own custom cloud based deployment.

What they are doing right (brief):

• NO VIRTUAL DISKS. Unlike amazon you are NOT going to have IO issues at Joyent. virtual disks are not used, all file system access is native.
• NO CONTRACTS. Turn up. Turn down yay!
• PERFORMANCE. Since the Joyent platform is based on OpenSolaris/Solaris Zones performance is quite a bit better than other solutions. Zones outperform Xen/xVM and Vmware by quite a large factor.

Rackspace
Rackspace has several interesting offerings: Cloud Sites, Cloud Servers, and Cloud Files. All of these services are accessible from the same control panel. Which is really quite nice. Cloud Sites is the scalable application hosting service (although that doesn’t really do it justice). Cloud Servers is a Xen based VM host. Cloud Files is similar to amazon S3.

What they are doing right (brief):.

• NO CONTRACTS. Turn up. Turn down yay! (Cloud Servers)
• AFFORDABLE. Host all the sites you want on Cloud Sites starts at $100 a month (Usage is metered upwards after a certain point)
• GREAT MANAGEABILITY. The Rackpace control panels are excellent and all services are accessible from one interface

Google
Google’s appengine product falls into the application hosting category. Pretty much the only cloud python host. They also have a Java environment. There are some limitations to deploying on Google however as time goes on we are seeing many of these limitations disappearing.

What they are doing right (brief):

• NO NEED TO MANAGE THE PLATFORM. This is a boon to developers that just are not platform engineers. Write, then deploy. Done.
• APIs. Google provides a lot of valuable APIs for integration with their services, and utility right out of the box.

Engine Yard
Engine Yard falls into the application hosting category. They would be the premier Ruby on Rails app host. Everything about this company impresses me.

What they are doing right (brief):

• NO NEED TO MANAGE THE PLATFORM. This is a boon to developers that just are not platform engineers. Write, then deploy. Done.
• SLICK MANAGEMENT INTERFACE. I have never seen such a great management interface for a hosting platform. Great job!

Read More

SX:CE EOL: THANK GOD

I have been following this discussion closely and most perspectives voiced have been those of long time Solaris users or those that have invested in SX:CE for production deployments (why you would choose to do so is beyond me entirely, considering that the intent to discontinue SX:CE has been well known for quite some time, and the original purpose of the release was for early access/testing). So I would like to relate my perspective as a Linux convert and general server OS/Management critic. I use Linux/OpenSolaris on the server as a server OS, I use OS X on the desktop.

From Linux to OpenSolaris

When we first started building our hosting platform in 2000 I already had pretty strong background using RedHat Linux and OpenBSD for a significantly sized deployment at a fulfillment house that hosted some rather large web applications. Back then (1997) RedHat didn’t have yum so package management was a very manual affair. Although it was better than just compiling and installing software. RedHat decided to make the shift to the Enterprise distro and for $300 a year per machine you could get paid updates using up2date. Neat, we ran on RedHat 7.3 until they announced EOL on the updates. The fact is RPM package management was not good enough to pay for. Even with yum, installation and removal of dependencies are regularly mangled. If you are building and deploying your own packages this is even more pronounced. You can FORCE it all to work but I really am not into that. Good bye RedHat.

Next stop Debian. Debian was pretty great. Apt was pretty amazing. I really wish I had started using Debian from the start. Nearly flawless dependency addition and removal on package install uninstall and upgrading to a new release was very easy if I decided to do so. Nearly every open source package I used was already built for Debian and in the release repository. Everything was 2 commands away. The problem with Debian: no regular release schedule. I could never plan on having x feature at x date and I had no idea when the security/errata updates would cease for a given release. This was a huge problem for platform development and management. Good bye Debian.

Next stop Ubuntu LTS. Great! I now had a release schedule, and I knew exactly how long security/errata updates would be supplied for a given release.

The Linux draw: a summary

• Not having to build packages due to a huge software repository and 3rd party trusted repositories: TIME SAVED
• Ease of setting up internal repository:  TIME SAVED
• No need to manually resolve package dependencies: TIME SAVED
• Keep all systems up to date with latest security updates, two commands: TIME SAVED
• Deploy a new system install with just a required package list: TIME SAVED
• Not having to setup and maintain a custom system deployment architecture: TIME SAVED
• Regular release schedule and errata/security update time line: FUTURE ASSURED

On Solaris: I had never really been interested in Solaris until sun decided to open source the OS. Prior to that I had to maintain a pile of Solaris 8 machines for a client that was heavily invested in Coldfusion. This experience had pretty much soured my view of the OS. The package management was terrible. Installing patches was painful. Installing software was painful. What good is instant deployment if managing the system after install is so damn time consuming? At any rate managing Solaris 8 and later 9 was far more work than managing RedHat pre version 7.0 and that was only involving keeping the security patches on the systems current.

The OpenSolaris project pulled me into Solaris land again. Why? IPS, ZFS, Zones, SMF, DTrace, and all the goodies that make OpenSolaris a compelling storage platform. I started using OpenSolaris for storage only. I really liked the uniform nature of the admin utilities. I decided to also deploy Sun Communications Suite on OpenSolaris. Although I had to play a little bit with the installer and hack up patchadd to allow patches to be applied to comms I am happy with it.

OpenSolaris over Ubuntu: a summary

• IPS/PKG is like a nextgen APT/DEB with ZFS integration. beadm, is great, being able to apply updates to a new boot environment and activate that environment, then rollback if there is an issue is priceless. I also like the IPS repo system better than APT’s
• ZFS
• Zones, sorry KVM doesn’t cut it.
• Crossbow (can I dump OpenBSD for firewall/VLAN management/VPN concentration/routing? Time will tell)
• SMF
• DTrace
• OpenSolaris management utilities are far more uniform than the Linux equivalents
• OpenSolaris seems to perform quite a bit better for certain streaming applications
• Sun’s JDK/JRE is available in the release repo
• I haven’t benchmarked this but I swear that Java server daemons (in this instance a custom SMTP server) process requests faster on OpenSolaris than on Linux

As you can see with OpenSolaris I can have my cake and eat it too. With OpenSolaris I see the future perfect server OS. I could run ONE OS on all my server hardware and not have to deal with the issues of managing Ubuntu LTS, and OpenBSD based on their independent strengths. Now obviously there are issues that need to be resolved and the developers at Sun have said these will be resolved.

Issues: a summary

• GNU userland vs Solaris userland shell path: Please stop with this, I really don’t care either way the matter is so trivial so simple to change it just isn’t something that should justify all the attention that it receives
• Automated Install: So far this seems like a slick setup. I haven’t evaluated it much due to the fact that a simple package list install from a repo is fast enough to satisfy my needs. In any event the developers have stated they are working on polishing the feature
• IPS needs polishing: So far all the issues I have with IPS have been acknowledged as such, or there is an existing RFE
• No Text installer: Next release will have a text installer
• Sparse Zones: This is an issue for me and I would like to see it resolved. However for some reason inherit pkg-dir umm seems to work for me for my Java server apps??? I even see the loopback mounts and everything runs? Haven’t really looked into it much past that…
• Updating Zones: The process of updating a zone’s packages via image-update is too convoluted this needs to be much easier.
• Sun enterprise repo. I would like to see the comms components packaged for OpenSolaris and updates enabled via IPS instead of SVR4 patches (yuk)
• Stable release schedule and at least security updates and severe errata updates in the release repo. I am assuming this is a resource related issue and that it will be resolved after the next release.

Going Forward

EOLing SX:CE is the best possible choice going forward. Obviously Sun’s developer resources are limited. If those resources can be devoted to resolving the issues with OpenSolaris we can move forward out of the SVR4 nightmare that inhibits the adoption of Solaris by users THAT WILL NEVER GIVE UP SANE PACKAGE AND UPDATE MANAGEMENT.

I am sick of looking over the listings on career builder and dice and seeing all the jobs involving Solaris to RedHat migrations. More of the same will not curtail the mass exodus from Solaris to RedHat. That is what SX:CE is; more of the same.

Read More

OpenSolaris :: 2009.6 (111a)

After setting up several new machines with intention of migrating them to the official release in June, I have to say this is the best OSOL release ever!

• IPS is snappier
• CIFS sharing seems to work a LOT better
• I am not sure if it is better driver support or what but disk and network IO seem to be that much better (Areca, intel)
• The whole OS just seems more polished

Anyway I will write more as I continue to work on my platform, but if this is an indication of what we can expect of future OSOL releases, I predict a bright future indeed! Great job to everyone working on the project!

Read More

OpenSolaris + Sun Communications Suite

This post is about my experiences and thoughts relating to installing and running sun comms suite on OpenSolaris, and it’s suitability as an email platform in general.

INSTALLATION

The following components installed happily on OpenSolaris with minimal tweaking (mostly the creation of symlinks to NSS and TLS libs).

• Sun Directory Server Enterprise Edition.
• Sun Messaging Server
• Sun Calendar Server

These components I probably could have gotten to run but it was such a battle dealing with the Identity suite that I opted to just load the App Server and all the components that required it on a RedHat system (yeah I know yuck right?).

• Sun Java Application Server Enterprise Edition
• Sun Identity Suite – Access Manager
• Delegated Administrator
• Convergence
• Sun Instant Messaging Server

It looks like the developers are eliminating the requirement to have access manager installed for the version 7 release.

Overall I am pleased with the platform, some notes:

Messaging Server

I love it. It is very fast, and has all the features I have ever wanted and more. I don’t particularly care for the configuration file syntax but it is tolerable and to be truthful the default install is so decent that you really don’t have to do too much outside of say UBE and virus filtering/policy. I like the way channels and the dispatcher work. I like being able to create seperate process pools for various channels.

Convergence

Is really awesome, there are some small ui issues but it is easily the best webmail/webcal client out there. If it had message threading it would be perfect.

Indentity Suite Access Manager

This thing is a beast. Easily the most difficult component to get working. I am really not sure how I feel about SSO for my type of deployment yet. I think given the option I would rather not deal with this component, and would rather just see it eliminated and setup ldap query caching or something.

Delegated Administrator

I am satisfied with the command line environment. IM provisioning is about halfway finished. The web interface is decent but not great. Over all it is not as customizeable as I would like. There is also very little documentation about writing plugins or modifying the JSP and XML templates to facilitate more advanced needs. I really wish that the commcli servlet interface was documented. I have never worked with servlets in Java before, and I have only written a couple small apps in the language, so perhaps I am missing something here. I suppose I could load the servlet class files up in an IDE and see if I can extract the method names and parameters. It would be nice if I could build my own flex based interface and just not use the web based DA.

Application Server

This is an irritation for me, there is a bug that causes the admin user to stay logged in for some reason. Going through sun’s site there is an update to go from 9.1 to the new glass fish EE server. Guess what the Solaris version of the patch is non pay, the redhat version is pay. I guess I don’t understand this. Is this standard practice with sun and their products that run on 3rd party OS’s? Had I known this I would have just dealt with the pain of a standard Solaris install.

SSL

SSL was a major PAIN IN THE ASS to get working on all services. I understand that all the components are setup to be distributed accross multiple systems (my deployment uses 6) but having seperate key stores for each component, and much worse different keystore formats, is a pain to deal with. JKS just plain sux. I had to use a 3rd party tool to import a key that was not generated in the current key store because the standard tool set doesn’t import keys into JKS, only certs.

Future Plans

I like where sun is going with the comms platform. Reworking the calendar server, implementing caldav and carddav is HUGE.

My Future Plans With Comms

I really want my mail platform to be as open and self supportive as possible. This is only acheivable with fully open components. I also want the entire thing running on OpenSolaris with as many of the stock packages as possible. After version 7 comes out I want to eliminate the access manager from my deployment, and provision directly from the LDAP system. I will then attempt to switch from the Java System LDAP server to OpenDS. This actually doesn’t look to be that difficult. With that finished I would like to switch to the OpenSolaris packaged version of GlassFish if possible. As far as Delegated administrator I would like to replace the web interface with a flex based web app and either directly provision in the LDAP directory or interface with the commcli server.

I would really like it if native OpenSolaris packages could be provided for comms, but if they never are I think I can engineer a satisfactory solution on my own and still take advantage of the toolset.

Read More

Sun Xvm Server :: First Impressions

Pros

1. Installer is great, very quick very easy
2. Web Interface is great
3. Overall functionality is on par with VMware ESXi

Cons

1. No support for VLANs on virtual interfaces (yet)
2. dom0 uses a boatload of RAM, 2GB. My test system only has 2GB at present so I was unable to test guest creation, and as a result the system was a bit sluggish. For enterprise deployments this is a non-issue, however I am trying to put together a solution for SMB that would utilize Xvm server and I think the high RAM requirement might make this cost prohibitive.

Questions

1. Will dom0 require 2GB of ram in the production release?
2. Since I am unable to create a guest until I get more RAM, will I be able send VM snapshots to a remote system easily? Say like over the internet as I do with ZFS send, recv?

Read More

Why Mirroring is not a backup :: WOW

http://hardware.slashdot.org/article.pl?sid=09/01/02/1546214

All I can say is wow. Data accidents happen. Claiming that a bug in the software could not have overwritten the data, is not a valid excuse for only maintaining a mirror. That is why ZFS makes so many administrators happy. An on system snapshot could have saved the data.

Currently I maintain a mirror of our main data store with 10 days of snapshots. The mirror’s snapshots are also replicated to a remote system using ZFS send, recv. This has saved numerous hours when clients or staff make errors involving accidental file modification or removal. No restoring from tape, no digging through tar files. Just rollback to a snap shot or cd into the snap shot directory and copy out the lost data.

Read More

Sun Java System Web Server on OpenSolaris :: Yay!

I am quite happy to see that Update 4 of Java System Web Server 7 was certified to run on OpenSolaris! Can’t wait to try this out for a couple of reasons:

(Keep in mind our current web hosting platform is Ubuntu Server + mods + Apache + mods + PHP + mods)

1. No ZFS support on Linux. Being able to provide snapshot support for individual hosting clients on a shared system would be too sweet! Also being able to stream those snapshots to our SAN for longterm storage (easily) would be even better!
2. The ability to host JSP without screwing around with the associated Apache modules.
3. Having a decent interface for provisioning shared deployments without having to maintain the mess of code that is required to bolt such functionality on to Apache.
4. ZFS (yeah I know sheesh)

GRIN: http://pkg.opensolaris.org/webstack/info/0/sun-webserver7%407.0%2C5.11%3A20081202T021336Z

Read More

IT & Policy :: Why Technological Solutions are not Always the Answer

As a manager for many companies’ networks and IT strategies as a whole,  I am continuously confronted with problems in that the client is requesting a technological solution where an education and policy solution is far less expensive to implement and maintain.

One of the most frustrating; is that of employee web usage prohibition/enablement. All of the technology solutions to this problem that I have reviewed (I have implemented and reviewed many) end up costing the client far more in management man hours than the client would save by restricting the web browsing habits of misbehaving employees. Now from a consultants stand point this may be fine, because well they are billing more, but this is not the way I choose to operate.

Let us examine the root of the problem. Employees may be using work time to engage in activities online that are not work related. Isn’t this an HR problem? As such shouldn’t HR be responsible for maintaining and enforcing policies as they relate to web usage in the work place? The answer is yes they should be.

As an employer it is a fairly easy task to identify ineffective employees. Let’s face it, do you really want to have individuals working for you that you have to force to work by taking away a potentially abusable resource? I know I do not.

After this has been established the solution to the problem becomes a lot more simple. We simply need to enable HR to be able to review a specific employee’s online habits should the need arise.

So we setup a less invasive non prohibitive logging technology that could be in the form of a transparent proxy or an authenticating proxy that logs and creates reports relating to employee web usage habits. Now we have the records that HR can use to validate employee compliance to their established web usage policy should the need arise, or at performance review time. No expensive hardware or configuration is required, no expensive provider subscriptions are required and most importantly no management overhead is required to continuously maintain lists of sites that employees can and cannot access.

This is the best example I can think of that illustrates that good policy and a simple IT solution goes much farther then the latest and greatest in web usage management from any of the firms that are out there pimping their garbage.

(realize that I am referring to positions that require some amount of web access, it is very easy to remove web access completely from a specific group of employees, such as those involved in manual labor or data entry positions)

Read More