Run level, run!

17 October 2008

ec2 linux runlevels xen

Trying to debug why my init scripts weren't initing on EC2, I checked all the normal places.

The chkconfig tool reported that my services should be on at runlevel 3:

jboss   0:off   1:off   2:off   3:on    4:off   5:on    6:off

My /etc/inittab says I was booting to runlevel 3:


Yet the console did not show my things to be spinning. A little googling around, I found the runlevel command (I'm not a linux expert...) which explained to me I was in runlevel 4.

N 4

4? Really? 4? I wasn't expecting that.

Apparently this is just how EC2/Xen work. They force you up into runlevel 4.

What does 4 mean? Asking the Wikipedia, you'll see a pattern:

  • Typical linux: unused
  • Debian/Ubuntu: same as 2,3 and 5, weirdos
  • RHEL/Fedora: unused/user-definable
  • SUSE: unused/user-definable
  • Slackware/Gentoo: Same as 3, which implies graphics, even
  • Sys-V: seems to imply multi-user with graphics

I had no idea that runlevels were all over the map. I also had no idea that EC2 would disregard my inittab and make up their own ideas.

Thanks, guys. Another hour-long round of RPMing, kickstarting, and image-creation ahead of me...

Boot up JBoss

16 October 2008

java jboss rails thincrust virtualization vmware

On my path towards clustering a Rails app on JBoss on EC2, I stumbled across Bryan Kearney and the other Thincrust guys. With their help, I now have a JBoss AS5 + jboss-rails "appliance" ready to roll.

Grab the raw image or the VMWare image, and play along at home.

Fire up the image in your favorite virtualization environment. I give my virtual machine at least a gig of RAM. Marvel at the pretty Grub splash screen, courtesy of James Cobb ( designer).

Let it boot on up, and you'll notice a handful of things:

  • Very few things roll across during boot. The image is an assembly of "just enough" OS bits.
  • Before JBoss boots, "Installing Appliance" or "Updating Appliance" will scroll by. This is the Thincrust magic that allows adjustment of the appliance in a consistent and controlled manner (via Puppet).
  • JBoss AS 5 (with rails deployer) starts at boot.

You can login with root password of thincrust.

The login prompt will tell you the IP address of the appliance, since it probably booted off DHCP. JBoss will be up and running at http://<IP_ADDRESS>:8080/.

You can su jboss, whose $HOME is /opt/jboss/jboss-as5, which coincidentally is $JBOSS_HOME. The default configuration is used to start the AS. Logs are under /opt/jboss/jboss-as5/server/default/log/. And to deploy, just drop something into /opt/jboss/jboss-as5/server/default/deploy/ and it'll hot-deploy.

To control the service, as root:

  • service jboss stop
  • service jboss start
  • service jboss status

I'll make the RPMs used to build this available sometime soon. Until then, you can poke around the bits I use to create the RPMs ultimately used by Thincrust to build the appliance image. They are packaged in a way that makes my Red Hat brethren throw up in their mouths a little bit.

Also, once I test'em on EC2, I'll throw out public AMIs for testing.

By no means is this complete. This just marked a nice spike of a milestone along the way. There's still plenty of things that'll poke you in the eye if you're not careful. Always wear your safety harness. Drink plenty of fluids. Keep away from children.

Stumbling through technology

11 October 2008

ec2 java jboss puppet thincrust virtualization

It can sometimes be funny, the paths we take.

Once I'd verified that a Rails app deployed on JBoss would indeed cluster, I sat in my farmhouse, looking at my lone little Mac. Not much of a cluster to play with.

So I started looking at Amazon EC2, which is truly very nice, particularly when paired with Elasticfox.

Of course, firing up a cluster on EC2 requires a nicely-produced, ready-to-boot machine image or a lot of manual configuration on each node.

Sacha pointed me to another group within Red Hat: Thincrust.

Thincrust configures a Fedora disk image with "just enough" OS bits, and provides a way to add/update applications on it. It's a happy mixture of Kickstart, Puppet and Yum.

The goal is to make it easy to produce "appliance" images that could be flung onto machines, real or virtual. Need a Git server? Fire up the Git appliance on the cloud. Need a Drupal server? Fire one of those up. Need a JBoss server or cluster? We're working on that..

Thincrust is built upon Fedora. My Mac is not. But VMWare Fusion lets me run Fedora. So I can create Fedora-based Thincrust images. Which I'll deploy on EC2. Which is running Xen, probably on RHEL. And ultimately allowing you to run your Ruby-on-Rails applications on a Java stack.

There are times I can't remember exactly which OS, language, or virtualization environment I should be thinking in.

Elasticfox, OSX and iTerm

01 October 2008

applescript ec2 elasticfox osa osx

Elasticfox is a nice extension for managing your relationship with EC2 from within Firefox. It's XUL-based, somewhat homely and odd. But it's so much nicer than dealing with the ec2-* scripts directly and copying Amazon's wacky hash identifiers hither and thither.

But on OSX, the "ssh to this instance" button is wired up to I can't stand I much prefer iTerm.

So, a little AppleScript hacking, and Elasticfox now uses iTerm to connect to instances through ssh.

Instead of jamming it all into a dozen -e options on osascript, I just put it in a file, and changed my configuration.

Elasticfox configuration

And the script itself:

on run argv
  tell application "iTerm"
    set cmd to "ssh -i " & item 1 of argv & " " & item 2 of argv
    set myterm to the last terminal
    tell myterm
      launch session "Default Session"
      tell the last session
        write text the cmd
      end tell
    end tell
  end tell
end run

Reminder: RaleighRubyCamp

30 September 2008

barcamp java jruby raleigh ruby

Don't forget, in a few weeks at the Red Hat offices in Raleigh, there will be an unconference-style Ruby BarCamp on October 18th.

My plan is to put together a few clumps of slides, prepare a cluster on EC2, and see if anyone wants to hear about JBoss Rails. I'll of course put them online sometimes before/during/after the camp, since that's part of the rules.

Multiple instances of AS on OSX

29 September 2008

java jboss osx

Rysiek asked me to elucidate my clustering on OSX...

By default, it seems that OSX provides exactly 1 localhost address, unlike RHEL. I normally make use of when I'm deploying on Linux, and figured I'd do the same on OSX.

To accomplish this, you need to create some new localhost IPs to play with.

sudo ifconfig lo0 alias up
sudo ifconfig lo0 alias up
sudo ifconfig lo0 alias up

Now, when you run your AS with ./, just pass -b or -b etc, and the entire stack will bind to that IP address.

At this point, I'm allowing each AS instance to share a work directory and such, which is probably not exactly the safest thing to be doing. Normally each cluster node would be a discrete machine with its own JBOSS_HOME.

For the apache config, in /private/etc/apache2/httpd.conf, I uncommented the line to allow vhost file loading:

# Virtual hosts
Include /private/etc/apache2/extra/httpd-vhosts.conf

Then, in the httpd-vhosts.conf file I fixed it up to include my virtual host for the head-end load-balancing stuff.

The <VirtualHost> is only barely conformant, and even the DocumentRoot is gratuitous, really. It looks like this:

<VirtualHost *:80>
  ServerName app.local.ballast
  DocumentRoot "/Users/bob/public_html"

  RewriteEngine on

  RewriteMap    lb      'prg:/Users/bob/bin/load_balancer local.ballast 1 3'
  RewriteRule   ^/(.*)$ ${lb:$1}           [P,L]

What that is doing is passing each requested path to the load_balancer script I've borrowed from I-don't-know-'where. It returns an augmented URL pointing to one of my localhost-bound nodes.

#!/usr/bin/env perl
## -- load balancing script

$| = 1;

$name   = $ARGV[0];         # the hostname base
$first  = $ARGV[1];         # the first server (not 0 here, because 0 is myself)
$last   = $ARGV[2];         # the last server in the round-robin

$cnt = 0;
while (<STDIN>) {
    $server = sprintf("node%d.%s", $cnt+$first, $name);
    $cnt = (($cnt+1) % ($last+1-$first));
    print "http://$server:8080/$_";

If a path of /foo/bar is handed to it on STDIN, it'll return something in the format of http://node1.local.ballast:8080/foo/bar or http://node2.local.ballast:8080/foo/bar.

Through the magic of /etc/hosts those friendly names point to the localhost bound AS cluster nodes, along with app.local.ballast pointing to good old traditional, where httpd is normally listening. That's my head-end from the httpd-vhosts.conf.       localhost app.local.ballast      node1.local.ballast      node2.local.ballast      node3.local.ballast

Fire up a browser and I surf to http://app.local.ballast/, apache answers, and immediately and invisibly proxies the request to port 8080 of one of my localhost-bound JBoss AS instances.


JBoR: Will it cluster?

29 September 2008

clustering java jboss jbosscache jruby rails

JBoss on Rails will indeed cluster!

After modifying and dropping my jboss-rails.deployer into an 'all' configured server of JBoss AS 5, and firing up 3 instances on my localhost (non-trivial on OSX...):

10:43:28,409 INFO  [RPCManagerImpl] Received new cluster view: [|2] [,,]
10:43:28,435 INFO  [RPCManagerImpl] Cache local address is
10:43:28,469 INFO  [ComponentRegistry] JBoss Cache version: JBossCache 'Poblano' 2.2.0.GA

And I've got 3 nodes running the same Rails app, all sharing a cookie and a JBossCache cache. Nick Sieger's JRuby-Rack handles binding the Rails session to the actual servlet session, and JBossCache takes care of the rest.

A little 8-line perl round-robinning load-balancer is wired up through mod_rewrite in my Apache httpd.conf to throw requests to each of the nodes. Anything set in the session is immediately available at the next request which lands at a different node.

Further down the line, we can look at a clustered cache for caching AR models and view fragments. Not too shabby.

It should be fairly easy to create a nice Amazon EC2 AMI with Fedora+AS5+jboss-rails, plus some better Rake/capistrano tasks, and make for quick cluster deployment. Any EC2 experts wanting to jump in?

Print like you mean it

23 September 2008

eclipse enscript java printing

When digging through code, I find it's good to print it out, wander off away from the machine, and do some reading.

But if you print any reasonable-sized chunk of code from Eclipse, you'll be carrying around several reams of paper, since Eclipse apparently thinks we're all blind illiterates.

For the past decade, my friend & mentor Jim Crossley and I have been passing the same encscript alias back and forth. I recently had to request it from him again. (Mental note: put it in SCM somewhere...)

  alias print='enscript --color=blackwhite -E -T4 -M Letterdj -C -2 -fCourier@5/5 -r $*'

It'll spew out your code in landscape mode, syntax-highlighted, two columns of 90 lines each (180 lines total per page), with line-numbers and a nice-looking header.

Many times, this allows you to fit an entire class on a single sheet without any gratuitous wrapping due to Eclipse's 24pt font usage.

JBoss Amputation

23 September 2008

java jboss jbossas rails

Since JBoss AS 5 is built on top of Microcontainer, it's effectively a network of beans just doing their respective jobs. You have already probably noticed that it ships with 3 included configurations: minimal, default, and all.

Unfortunately, they're awefully far apart along the spectrum of configuration options. The minimal configuration barely gets the container running, while the all configuration includes, well, everything. For most apps, it's safe to start with the default configuration. But default to who? It's the 80% case. But the 80% of the world who gets by with default, it still probably includes way more than they need. Of course, each user needs a different subset of that 80%.

Given that the MC is managing a graph, it seems like we should be able to actually perform a solve to determine what is or is not required. Or at least get a good idea. Bonus points if someone can then twiddle conf/, deploy/ and deployers/ to tidy things up.

Anyhow, for the jboss-as-rails project, I'm kicking it old-school, and going with default. I'm jamming it into git, and hopefully with some help, we can get it pared down to what we need.

The jboss-as-rails project is simply a configuration of AS that includes only what we need, along with the jboss-rails.deployer from the jboss-rails project (notice the subtle naming difference...).

Here's a picture to see how it'll all ultimately fit together.

The plugin will offer rake tasks for managing the included AS, deploying your app. I'll also produce an AS-free version of the plugin, assuming you want to manage your own AS separately.

And poke around jboss-as-rails, see what we can rip out.

JBoss on Rails

22 September 2008

java jboss jbossas rails ruby

Tomorrow is my first real status update call with my boss, Sacha Labourey. I've been anxious to deliver something, to prove I hadn't gone completely pudding-brained during my tenure as management.

This morning, it all finally came together in a pleasing fashion, causing me to hoot and holler loud enough to scare the cats and probably some cows.

I've just pushed an ugly-but-working deployer targeting JBoss-AS 5.0.0.CR2 (the latest and greatest!)

It's not very consumable at this point, as it's just a deployer, not a nice Rails plugin with a set of Rake tasks. Heck, it doesn't even undeploy yet.

But adding the deployer to your server's deployers/ directory allows you symlink live RAILS_ROOTs into your deploy/ directory, and be running on JBoss.

Live. In-situ. Edit your controllers or views as you like, and your changes are immediately reflected in the running instance. Just like with ./script/server. It does not even have to redeploy your app. The rails framework is handling the magic reloading.

It's taken me some time to dig through the innards of JBoss-Microcontainer, and a few false starts, but I finally figured out a super simple deployment process.

I'd previously been trying to manipulate a RAILS_ROOT into a synthetic Java WAR archive, and shoe-horn things around that. But I have the freedom to go lower than that, so the jboss-rails deployer just sets up a Catalina context appropriately, without regard to WEB-INF or other non-Rails stuff. There's no need for that cruft. Likewise, I can directly control and manipulate the classpath, so the RAILS_ROOT does not even have to have any JRuby bits in it.

The example application (src/test/ballast) is a virgin rails app with ActiveRecord disabled so I don't have to deal with database-driver gems just yet.

Once deployed, a Rails app looks like pretty much any other web-app. The jboss.rails.deployment domain contains deployment objects for each rails app. And jboss.web contains all the webby bits floating around.

I need to go back and remove the dead-end code I've left in my wake, and update the tests I'd disabled while in a coding flury (bad Bob!) I plan to put together an easy-to-consume plugin gem which contains an nicely-configured AS along with the jboss-rails deployer pre-installed, along with rake tasks to start/stop AS, and deploy your app. I'd also like to give clustering a whirl, and see what we can do.

It's been an excellent 3 weeks back as an engineer.