Sigil.us

Shell :: Simple Apache Pig Identification

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

#!/bin/bash
#apachepig.sh
#simple script for identifying shared hosting pigs on a small server
#only definitive for a forking server, threading would need modification
#should be enhanced for polling, but useful for processes that are hung
#which is what we are looking for right?
 
ARGS=("$@")
 
#GLOBALS
WEBROOT=/home
DAEMONNAME=apache2
 
#ARGS
ACTION=${ARGS[0]}
#add single PID functionality later
PID=${ARGS[1]}
 
case $ACTION in
	mem)
		PROCESSES=$(top -b -n1 | grep $DAEMONNAME | sort -r -k 6)
	;;
	cpu)
		PROCESSES=$(top -b -n1 | grep $DAEMONNAME | sort -r -k 5)
	;;
	time)
		PROCESSES=$(top -b -n1 | grep $DAEMONNAME | sort -r -k 7)
	;;
	*)
		echo "operation not supported"
	;;
esac
 
IFS=$'\n'
for x in $PROCESSES; do
	echo "$x"
	PID=$(echo $x | awk '{print $1}')
	OUTPUT=$(lsof -w -p $PID | grep "$WEBROOT")
	if [ $OUTPUT ]; then
		echo $OUTPUT
	else
		echo 'NO PIGS'
	fi	
	echo " "
done

Read More

Shell :: simple MySQL DB provisioning

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

#!/bin/bash
#provmysqldb.sh
 
ARGS=("$@")
 
#GLOBALS
MYSQLHOST=your.server.name
MYSQLPASS=yourpass
 
#ARGS
ACTION=${ARGS[0]}
USER=${ARGS[1]}
#maybe we want to split this out later
DB=${ARGS[1]}
 
if [[ $ACTION && $USER && $DB ]]; then
	case $ACTION in
		create)
			#check to see if db exists
			echo "use ${DB}" | mysql -s -h $MYSQLHOST -u root -p$MYSQLPASS 2> /dev/null
 
			if [ $? -ne 0 ]; then
				echo -n 'enter password: '
				read PASS
 
				OP="
				create database ${DB};
				create user '${USER}'@'%' identified by '${PASS}';
				grant all ON ${DB}.* to '${USER}'@'%';
				flush privileges;"
			else
				echo "The database/user: ${DB} already exists"
			fi
			;;
		delete)
			#check to see if db exists
			echo "use ${DB}" | mysql -h $MYSQLHOST -u root -p$MYSQLPASS 2> /dev/null
 
			if [ $? -eq 0 ]; then
				OP="
				drop database ${DB};
				drop user '${USER}'@'%';
				flush privileges;"
			else
				echo "The database/user: ${DB} doesn't exist"
			fi
			;;
		*)
			echo 'Operation not supported'
			;;
	esac
 
	if [[ $OP ]]; then
		echo $OP | mysql -h $MYSQLHOST -u root -p$MYSQLPASS 2> /dev/null
		if [ $? -eq 0 ]; then
			echo 'Operation Succeeded!'
		else
			echo 'Operation Failed!'
		fi
	fi
 
 
else
	echo 'USAGE: provmysqldb.sh <action:create|delete> <dbname>'
fi

Read More

Love the Cloud :: More new companies

Linode
A Xen VPS hosting company.
According to these benchmarks they have the best Xen VPS performance so far.

Heroku
A ruby app hosting company with a slick API and a fully managed stack. I love their web design too!

Read More

IPS/pkg :: A business friendly software distribution and packaging system

I just noticed this change to pkg in OpenSolaris. These functionality additions allow for the conditional display and acceptance of software licenses on package install for the IPS system.

Why is this significant? Because it opens up the package system for commercial software distributors. If you couple this with the IPS repository functionality and require keys for the repo you have subscription updates, and license management covered!

Your software purchase and install path looks like:

Go to web site -> do they support opensolaris -> yes -> buy -> add repo -> pkg install software

Update your software from now until forever:

pkg install software or

pkg image-update

I really don’t like having to run custom installers from vendors that run scripts that do all kinds of stuff to the system and also install package sets. If the software packaging system is opened up to vendors we can manage our open source and commercial applications from one place.

I would also like to see open source projects build their own package repositories for all the major distributions. This would allow us to stay current with some software packages who’s development out paces the release cycles of the enterprise distros (RHEL, Ubuntu LTS).

Read More

Storage Management :: Global File Systems and ZFS

One enterprise architecture feature that every admin dreams of is simple storage management. Having one storage network and a global filesystem that all nodes can access, and being able to easily have space “appear” on the file system by adding LUNs to the global file system is a huge time saver.

Redhat Linux

GFS(2)
Redhat has done well with GFS(2) from a systems management perspective (I have read that it doesn’t scale as well as lustre). The cluster file system technology is pretty simple comparatively speaking even though there are several layers involved. You don’t need metadata controllers or independent systems acting as lock managers. You do need to either have fencing hardware, or a manual fencing process in place to account for failed nodes.

Adding storage involves using LVM to add the LUN to the volume, expand the volume, then using gfs2_grow to expand the filesystem.

At its simplest a configuration could look like the following:

Sun Solaris/OpenSolaris

SAM-QFS
Sun has had SAM-QFS for quite some time. It is more complex than GFS(2). To be fair it also has more features. It requires dedicated metadata controllers. This solution doesn’t interest me much due to the fact that I use Solaris/OpenSolaris for ZFS to simplify storage management.

Adding storage to the shared QFS file system appears to involve a similar process to GFS(2).

At its simplest a configuration could look like the following:

Lustre, BTRFS/CRFS, ZFS and the Future

Lustre
Lustre seems to be more complex than both GFS(2) and SAM-QFS. It also seems to be more scalable. It requires 2 types of controllers: MDS (metadata) and OSS (storage). So from a management perspective it requires quite a bit more overhead.

It looks like ZFS will be added as a backend storage format in the future versions 2.x+ this might simplify LUN management a little for the solution.

BTRFS/CRFS
BTRFS is a Linux ZFS workalike. It is still in development but the final version is shaping up to have ZFS feature parity (minus easy device/raid management, zvols, and other sweet things like zfs send recv). What interests me however is CRFS.

CRFS also in development, appears to be a global network filesystem that basically exports BTRFS file systems.

ZFS
What I would like to see is ZFS to become cluster aware. Being able to use OpenSolaris as a storage host and export either a zfs or a zpool to nodes, and perhaps run a lockmanager process similar to GFS(2) would be pretty slick.

For now with ZFS

Creating ZFS file systems on disk boxes connected to a SAN fabric really isn’t that difficult. It is however a bit time consuming, and you really should configure LUN masking for each LUN and node. Other wise you run the risk of accidentally corrupting data by using the same LUN at the same time on different nodes.

Exporting zvols with Comstar/iSCSI also doesn’t seem like the best idea, because again you should configure target portal groups and host access, and I don’t think that running ZFS on top of a ZFS based virtual block device (zvol) would be great for performance. Jeff Bonwick had stated something about improving the zvol performance (block pass through?) in his last ZFS talk. I am not sure wether it has been done yet or not. There also are no best practice notes about ZFS on iSCSI zvols.

I would think using UFS on a zvol would be better performance wise, but the whole purpose of the exercise for me is to provide snapshot functionality to the host consuming the filesystem, and filesystem delegation to the zones on the host.

I would really like to be able to use zvols as opposed to SAN fabric LUNs because then the zones could use snapshots, I could mirror the zvols from a Comstar storage host to a remote system for disaster recovery, and only have to deal with one zpool.

So how about it? Is anyone else using Comstar, ZFS, and OpenSolaris in the fashion I have described What is your strategy for consolidating storage management with OpenSolaris?

Read More

Bash :: Extension change script

A simple script to recursively change filename extensions (case insensitive)

#!/bin/bash
#chext.sh
 
ARGS=("$@")
 
if [[ -d ${ARGS[0]} && ${ARGS[1]} && ${ARGS[2]} ]]; then
        for x in $(find ${ARGS[0]} -name "*.${ARGS[1]}"); do
                y=$(echo $x | sed "s/\.${ARGS[1]}\$/\.${ARGS[2]}/i")
                mv $x $y
                echo "$x -> $y"
        done
else
        echo 'USAGE: chext.sh'
fi

Read More

Grails :: Broken Mail Template Rendering

Ugg. So one thing all web applications HAVE to do is send mail. In order to keep your code tidy you should be using templates to handle your mail formatting. In PHP + Zend Framework I have accomplished this cleanly via the following:

1
2
3
4
5
6
7
8
9
10
11
12

$view = new Zend_View();
$view->addScriptPath(BASE_PATH.'/tpl');
$view->somevar = $somevar;
$html = $view->render('someTemplate.phtml');
 
$mail = new Zend_Mail();
$mail->setBodyHtml($html);
$mail->setFrom($config->site->serviceAddress, $config->site->appName);
$mail->addTo($email);
 
$mail->setSubject($config->site->appName.' - Message Subject');
$mail->send();

Zend_View handles the output buffering and rendering of the template. I have never really had a problem with it.

In Grails this is supposed to work:

1
2
3
4
5
6

sendMail {
	to user.email
        from 'email@emails.com'
        subject 'Subject'
        body (view:'/mail/myTemplate', model[user:user])
}

sendMail is a mail service plugin that accepts a closure as an argument, it also has a shortcut to the gsp template engine to render the template. Seems more simplistic, however there is a lot of code under the hood driving it. It seems that the output buffering within the Grails framework is very tied to the servlet response.

Anyway within a Grails Service (haven’t tried it within a controller yet) it munges the output buffer. Probably due to the output buffer juggling that is going on within various layers of the response and template rendering wrappers. The result is a failed remoting call.

The fix (figured this out after profiling and digging around the web) is to wrap sendMail in a thread:

1
2
3
4
5
6
7
8
9

def t = new Thread() {
	sendMail {
		to user.email
                from 'email@emails.com'
                subject 'Subject'
                body (view:'/mail/myTemplate', model[user:user])
	}
}
t.start()

This leads me to believe that somewhere within GrailsPageResponseWrapper the original output buffer is being overwritten or lost which is killing the response writer. Spawning a new thread for the operation preserves the original output buffer.

I am surprised that no one else is using this functionality outside of their controllers, and only 2 people have encountered the issue. I don’t know enough about the grails stack to debug it fully myself yet.

Here is a link to the bug report:
http://jira.codehaus.org/browse/GRAILSPLUGINS-1548

Please fix?

Read More

Shell Web Server :: Netcat is neat

A very basic bash web server. Netcat is pretty epic. Referenced Paul Buchheit

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

#!/bin/bash
 
HOMEDIR=./
RESP=/tmp/webresp
[ -p $RESP ] || mkfifo $RESP
 
while true ; do
( cat $RESP ) | nc -l 8000 | (
REQ=`while read L && [ " " "<" "$L" ] ; do echo "$L" ; done`
URL=`echo $REQ | head -1 | awk -F" " '{print $2}'`
echo "[`date '+%Y-%m-%d %H:%M:%S'`] $REQ" | head -1
 
index='index.html'
if [ "$URL" == "/" ]; then	
	file=index.html
else
	file=$URL
fi
 
if [ ! -f $HOMEDIR$file ]; then
	CODE=400
	CONTENT='Not Found'
else
	CODE=200
	CONTENT=`cat $HOMEDIR$file`
fi
 
cat >$RESP <<EOF
HTTP/1.1 $CODE OK
Cache-Control: private
Content-Type: text/plain
Server: bash/3.2.48
Connection: Close
Content-Length: ${#CONTENT}
 
$CONTENT
EOF
)
done

Read More

RedHat EL5 :: Surprisingly Better

So I have been putting RHEL5.4 through its paces in preparation for the RHCE exam. It has come a long way since version 4. Yum is actually pretty ok. The distro itself is snappy even on garbage hardware. All in all though not as bad as I remember RHEL being.

Things that still bother me vs Ubuntu LTS:

• Comparatively small distro package set.
• File system layout in respect to etc and var: stuff is all over the place. Seriously each configurable software package having its own directory in etc and var sub directories (log etc) is nice. It is uniform and easy to guess where things might be.
• $300/yr

Read More

Elementary Algorithms :: Array Difference

UPDATED

In an interview I was posed a problem: given 2 arrays a and b write a program that will print the elements of a that are not in b.

The easiest (incorrect solution) in PHP:

1
2
3
4
5
6
7
8
9
10
11
12
13

#!/usr/bin/php
<?
$a = array(1, 2, 3, 4, 9, 20, 30, 82, 57);
$b = array(1, 2, 3, 82, 57, 100, 32);
 
$c = array();
foreach($a as $aval) {
	if(!in_array($aval, $b)) $c[] = $aval;
}
 
print_r($c);
 
?>

Now as I had stated in the interview this is NOT the best way to accomplish this. Of course that would all depend on how in_array is implemented in PHP. If in_array is simply a short cut to a true false foreach loop then we have a problem. If n and m represent the number of elements of a and b respectively iterations = n * m. Or O(mn). Not efficient. Since this is PHP I am guessing this is the case (a simple sequential search).

So what is a better solution (best?) to allow for better performance as n and m increase? This is the best I could come up with after a quick look at a couple of algorithms.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

#!/usr/bin/php
<?
$a = array(3, 1, 2, 4, 9, 20, 30, 82, 57);
$b = array(1, 2, 3, 82, 57, 100, 32);
 
//binary search function
function binSearch($ar, $n, $value) {
	$l = 0;
	$r = $n;
	while($r >= $l) {
		$mid = ($l + $r)/2;
		if($ar[$mid] == $value) {
			return true;
		} else if ($ar[$mid]< $value) {
			$l = $mid + 1;
		} else {
			$r = $mid - 1;
		}
	}
 
	return false;
}
 
//sort the first array used in the bin search
sort($b);
$n = count($b);
$m = count($a);
$c = array();
for($i=0;$i<$m;$i++) {
	if(!binSearch($b, $n, $a[$i])) {
		$c[] = $a[$i];
	} 
}
 
print_r($c);
 
?>

Removing the negation operator prior to the function call will change the program to array intersection vs array diff. The math for this binary search solution is O(m log2 n).

So consider the following:
m = 8
n = 8

Solution 1:
(mn) = 64

Solution 2:
16*log2(8) = 48

Basically due to the first array being sorted and chunking we are able to do fewer comparisons instead of a comparison for each element. As the size of the chunk decreases the worst case is that the value doesn’t exist.

It also occurs to me that you will want to count each array and sort the shortest and use that as the first argument and the longer array[val] as the second to further reduce the number of iterations. Of course the order of the arguments matters for this problem (in a not in b). I suppose you could flip flop logic/operator in/on the binary search call/function though to account for that as well.

Anyway this is basic CS, everyone probably knows this but I couldn’t come up with it off the top of my head, and the problem was driving me nuts cycling in the back of my mind so I had to write it out.

UPDATE
I still kept thinking about this so here is another based on array merge and comparing next array elements. Since this is a diff and not an intersection we need to merge the first array twice. Any value that shows up twice in the merged array is in a and not in b.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/usr/bin/php
<?
$a = array(3, 1, 2, 4, 9, 20, 30, 82, 57);
$b = array(1, 2, 3, 82, 57, 100, 32);
 
$c = array_merge($a, $b);
$c = array_merge($a, $c);
$d = array();
sort($c);
 
 
$i = 0;
 
while($i <= count($c)) {
 
	if($c[$i+1] == $c[$i] && $c[$i+2] == $c[$i]) { 
		$i+=3;
	} else if ($c[$i+1] == $c[$i]) {
		$d[] = $c[$i];
		$i+=2;
	} else {
		$i++;
	}
}
 
print_r($d);
?>

The math ends up being something like ~O(m + m + n).

Read More