DIY Information Radiator

 

 

Information Radiators are used to provide people with feedback on the current status of code builds and automated tests in Continuous Integration and Agile development environments.

The basic idea is that when developers commit a code change, they can easily and quickly see that it has been picked up by the automated build process, and then (ideally within 10 minutes) see the result of their change; did the build succeeded and did the automated tests pass?

Martin Fowler’s description goes in to more detail on the ideas behind this approach and the function that Information Radiators serve.

The normal convention for these is to use colour coded blocks per build, using:

  • Green for good/passing jobs
  • Amber for either currently building/running jobs (or sometimes for unstable ones)
  • Red for failed jobs

Generally you want to keep things as clean, simple and uncluttered as possible, but sometimes it is helpful to add in a bit more info.

Details I have occasionally found worth adding include things like;

  • name and/or picture of the user who triggered (or broke!) the build
  • commit message from the code change that triggered the build
  • build number
  • history – number of recent fails or passes
  • date/time last failed and last checked

if you use amber for “unstable” rather than “build in progress”, you may want to add text to say if the job is currently building – I often use the “spinning wheel” icon thingy from jenkins itself :

spinner

Why build your own?

There are tons of readily available plugins that allow you to quickly and easily produce a Radiator or Wall panel from Jenkins, so why go to the bother of making your own?

Plugins are usually linked to one Jenkins instance (the one they are running on) and I have often found that alone to be too restrictive – having too many different radiators all over the place makes things too cluttered and uncertain, and people can easily start to “switch off” from them all – having one screen that people can understand at a glance usually works best.

Changing requirements – developers are constantly wanting/looking to improve processes and often come up with ideas and requests for things to try that may help them do their jobs – adding a bit of information from another source, for example, or changing the colours used to a different shade, or adding curved borders etc etc…

What I have found often works best, is to get all of the data I am interested in inside a database then write my own simple but flexible presentation layer from scratch – this gives me all the flexibility I could want (or may find myself wanting or needing later…) and importantly, it also allows me to leverage additional benefits by combining data from Jenkins with data from elsewhere – this means I can easily produce reports, charts, metrics etc that present a view comprised from multiple data sources throughout an organisation – for example, you can then easily create reports that combine:

  • jenkins – live information on the current state of builds and tests
  • defects – data on bugs and changes pulled from bug tracking apps (usually via jenkins jobs)
  • version control – the actual code change can be extracted from version control and linked to both the developer and bug details – the “svn log” command is useful for this
  • environment monitors – state and health of environments; database and app server health, deployed patch and code levels etc (again, these are usually other Jenkins jobs!)

and you can add in anything else you can get your hands on 🙂

This allows you to track the flow of a change right through the development life cycle – from the initial change/requirement/defect to the change itself at the code/file level, then the testing and building of it and the eventual release. This is far more than you need for a simple Information Radiator, but using this approach means that you can easily reuse much of the work in different ways.

Part II covers the technical approach I use for Extracting data from Jenkins

Jenkins Agent Nodes – using the Swarm Plugin

NOTE: This post and approach is quite old now; a better alternative for dynamically provisioning and scaling Jenkins Agents and running CI/CD Pipelines is demonstrated here.


I’ve been trying out a new (to me at least) way to add a Jenkins Agent Node – using UDP auto discovery via the Jenkins Swarm Plugin

This is a very easy and nice way to do it, with minimal configuration/hassle required so you can quickly and easily add new Jenkins Agent Nodes to your Master Jenkins instance as and when they are required.

Here are my notes from setting this up – it’s pretty simple to do and worked well for me out of the box…

Set up a new instance of Jenkins:

  • Make & cd to a working directory

mkdir jenkinsswarm; cd jenkinsswarm

  • Fetch jenkins.war

curl -O http://mirrors.karan.org/jenkins/war/1.506/jenkins.war

  • Start Jenkins

{/path/to/java/bin/}java -jar jenkins.war

After that, you should get console output along these lines…

Running from: /root/jenkinsswarm/jenkins.war
webroot: $user.home/.jenkins
18-Mar-2013 15:19:26 winstone.Logger logInternal
INFO: Beginning extraction from war file
Jenkins home directory: /root/.jenkins found at: $user.home/.jenkins
18-Mar-2013 15:19:33 winstone.Logger logInternal
INFO: HTTP Listener started: port=8080
18-Mar-2013 15:19:33 winstone.Logger logInternal
INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled
18-Mar-2013 15:19:34 jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
18-Mar-2013 15:19:41 jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
18-Mar-2013 15:19:41 jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
18-Mar-2013 15:19:44 org.jenkinsci.main.modules.sshd.SSHD start
INFO: Started SSHD at port 25133
18-Mar-2013 15:19:44 jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
18-Mar-2013 15:19:44 hudson.TcpAgentListener <init>
INFO: JNLP agent listener started on TCP port 41790
18-Mar-2013 15:19:44 hudson.WebAppMain$2 run
INFO: Jenkins is fully up and running

– that looks happy enough, and as you can see from the line “HTTP Listener started: port=8080” it’s running on the default port, so connect to http://yourhost:8080 and you should see something like this…

the next step is to install the Swarm Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin) on this Jenkins Master instance so that Swarm Clients can connect to it.

Do this by going to “Manage Jenkins > Manage Plugins > Available” then selecting to install the “Swarm Plugin“.

Once that’s done you should see that the plugin has been installed…


Now that your new Jenkins server is set up and ready, hop over to your other Jenkins Agent/Client host and do the following…

mkdir for the swarm client

mkdir swarmclient; cd swarmclient/

Get the Swarm Client jar file from the ‘net

curl -O http://maven.jenkins-ci.org/content/repositories/releases/org/jenkins-ci/plugins/swarm-client/1.8/swarm-client-1.8-jar-with-dependencies.jar

Start up the Client

java -jar swarm-client-1.8-jar-with-dependencies.jar
Found 1 eligible Jenkins.
Connecting to http://mydomain.com:8080/
Attempting to connect to http://mydomain.com:8080/ a2721b16-04e4-0d962
18-Mar-2013 15:33:22 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Required credentials not available for BASIC <any realm>@mydomain.com:8080
18-Mar-2013 15:33:22 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Preemptive authentication requested but no default credentials available
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Hudson agent is running in headless mode.
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://mydomain.com:8080/]
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to myhost.mydomain.com:43932
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

Now take a look at your browser and you should see a new Node automatically added to the Master Jenkins instance…

A very handy and flexible approach to adding/managing Nodes and workload – many thanks to the developers behind this!

Cheers,

Don

New Home/lab ESXi 5 Server – Part 1

I wrote a while ago about my plans to set up a home VM Ware ESXi 5 server… and although you can’t tell the difference, this site is now happily running on it 🙂

Spec:

The server I went for is an HP ProLiant ML110 G6. It’s got a single socket Xeon Quad Core X3430 processor, 16GB of RAM and at the moment 2TB of SATA3 disk – I will add more when I finish migrating data off the old servers.

There are some limits on the spec you can use with the free version of ESX – I think it’s currently one physical processor and 32G RAM, which means this server is fine.

Install and setup:

The ML110 is a compact and well-made server and is very quiet when up and running – it sounds like a hovercraft for the first few seconds, but quickly calms down to run not much louder than a normal desktop PC. The chassis isn’t large but it’s well laid out and there’s room for 4 HDD’s in there, maybe more if required if the DVD drive was removed (it’s not on the ESX compatibility list anyway so I can’t use it for this).

I installed ESXi 5 from a 2GB USB drive which is attached to the USB socket directly on the motherboard – the BIOS boots from this no problem, and gave me the option to install ESXi 5 on to the USB drive, leaving my HDD(‘s) free to use as 100% dedicated ESX Datastores, and also meaning I can exchange them when I need to without having to reinstall or worry about the VM Ware OS.

The server-side installation of ESXi 5 is a breeze – I rehearsed and wrote about it on my ESXi 5 on an iMac under Fusion post  and it was no different here – it took about 10 minutes or so and there were no tricky questions. Specify the user name and password, tell it to use all of the available HDD and there’s not much more to it. Once you have set up networking the way you want it (the DHCP setup should be fine for most installs) you don’t really need to go near the server again – it’s all managed remotely via the client applications and SSH (more on that in Part 2) from then on.

Creating VMs:

Once you have the vSphere client install done (it’s far more trouble than the server install – plus it requires .Not and J# runtimes – argh! – so it requires Windows – double-argh! – so had to be done initially on a Windows VM running on my iMac in my case… triple-argh!) you can connect to your ESXi 5 server using the IP address, User Name and Password that you set up and start creating some VMs.

To get the first VM created (in my case this had to be a Windows one that I could then use to run the vSphere client on and RDP over to instead of having to run a VM on my Mac all the time), I uploaded an ISO image to the datastore that I had created, then added a new VM in the vSphere Client and set that ISO as the CD image it should load at boot time. You just specify the OS type, RAM, CPU, Networking and disk(s) you want and power it on – all very easy and quick.

Converting a VM Ware Workstation VM in to a VM Ware ESX Guest

I also wanted to convert the Ubuntu VM Ware Workstation image that this web site runs on, so I could move it off the old server and have it running on ESX as a “proper” ESX guest/host. This was really easy too; the VM Ware Converter allows you to specify a local VM of pretty much any type and supply the details of the target ESX server, and it then converts and loads it all for you – it took quite a while to complete but it worked without issues, and I was then able to power on my Ubuntu website VM under ESXi 5, where it’s happily running right now. No need to reinstall WordPress, Postfix, PHP, Java, Jenkins, MySQL etc etc – happy days.

Here are some Pics of the ESXi5 console shortly after set-up…

 

1. General info on the reported spec and current overall resource usage of the HP ProLiant ML110 G6:

ESXi 5 console general information

 

2. Some of the Health Monitoring and General Configuration options:

Inventory and general settings and diagnostics pic

3. Overview of Guest and Host resources:

Guest and Host resources

 

Summary:

I did a fair bit of research beforehand to make sure the install and hardware would be ok, which meant the actual set up was trivial – once it’s done all you need to do is create VMs and allocate resources; there is very little work or maintenance required – especially compared to what would be needed to run multiple physical servers all with their own hardware. Creating new VMs is very easy, and the performance is good so far – the processor is not stressed at all, and the ESX memory management does a good job – I’ve had up to 6 VMs running at the same time and still have about half the memory free!

Next plans:

One of my main reasons for doing this was to provide a test platform for automating, creating and managing Linux VMs using Jenkins as a front end and DNS records to control what is deployed where and when – I want to be able to select a few options, then click a button and have my new host created in minutes and to the right spec, similar to the Amazon EC2 set up but code deployment linked in too, and I will write more on this when I’ve done it.

Plans include a mixture of: VM Ware Templates, Perl, Jenkins jobs, Jenkins Nodes, Puppet, Tomcat, etc

Next Post:

There are a few other things I have already done that I’d like to document too including…

Accessing ESXi 5 via SSH – how to and a summary of useful commands etc
More detail on Remote desktop via ssh tunnels etc
VM Ware command line tools
DNS and AD/LDAP servers

VM Ware ESXi under VM Ware Fusion

Hardware

I’m looking at getting a new home lab/server set up, and am seriously considering ordering one of these:

HP ProLiant ML110 G6 Quad Core X3430

It will need filled with memory and a load of disk space, but I reckon that’s still a whole lot of server for the money.

The Plan

The plan is to install VM Ware ESXi 5 on to it (using the USB Drive), and manage the server through the remote VM Ware vSphere Client app. This will allow me to create new Virtual Machines and migrate my existing appliances over to this server, then I can retire the old servers these have been running on. I’m also wanting to develop some automation processes for managing VM’s – creating new ones, bringing them up and down etc using Jenkins to orchestrate the processes, so this will allow me to work on that too.

Before ordering the ML110 I wanted to take a better look at the installation of ESXi – it doesn’t sound difficult, but while I’ve used it often I’d never set it up before so wanted to see what was involved.

VM Ware ESXi 5 Installation under Fusion on an iMac

It’s a little bit crazy and recursive, but, I realised I may be able to do this on my iMac under VM Ware Fusion, which it turns out does allow you to install ESX as a Virtual Machine itself…. which you can then use to manage and create new (Virtual?) Virtual Machines – a bit of a brain-ouch, and it’s clearly not going to be fast, but it’s good enough for my testing.

Installing ESXi is very straightforward – I selected the obvious option of “VM Ware ESX” in Fusion on my iMac, told it I had an image I wanted to use and pointed it at the VM Ware ESXi 5 ISO image I had downloaded from VM Ware (you need to register then fumble about their site for a while to find and download the free version – that’s the way I did it anyway). Keep a note of the serial number they give you, as that will remove the 60 day trial restriction later.

Fusion suggested a 40G file system for this instance and allocated 2GB RAM and 2 cores, which I was quite happy to run with for my test. Speaking of RAM – there was a restriction on the amount of physical RAM you could use with the free version of ESXi 5 – it was 8GB a while ago but this has now been increased to something more sensible – 32GB I think?

There are no surprises or major decision needed during the install of ESXi 5 itself – it took all of 5 minutes to run through and reboot, and that was with it running as a Virtual Machine on an overloaded iMac – on proper hardware like the HP ProLiant ML110 G6 Quad Core X3430 that I’m looking at, it would be loads faster.

When the install is done, there’s nothing more to see on the ESXi (VM), apart from the HTTP address it gives you to connect to the ESXi Host and access a simple web page it serves with links to download the VM Ware vSphere Client application to another host and start managing your server. This address was given by DHCP in my case – I think you can specify or change this easily if you want to.

Client Installation

Downloading the VM Ware vSphere Client took a while (longer than installing ESXi did!) as it came from the VM Ware site rather than being directly served by the ESXi host. Now for the bad news… for clients, you have a choice of running either Windows or Linux. No mention of an OSX vSphere client, so I had to fire up a Windows VM just to set up the client app on… not what I had been hoping for – there’s a petition asking VM Ware to sort this out here:

http://communities.vmware.com/thread/128538?start=525&tstart=0

When I get things running I could create a VM on my ESXi host which I can RDP on to, but that’s still a pretty ugly solution – if the Linux client is ok I’d go with that over the Windows one, and I think there’s also a Web Interface. But, part of the reason for me doing this in the first place is so that I can look in to the SDK and API’s for automating the creation of VM’s with VM Ware using Jenkins though, so I’ll grin and bear the Windows yuckness and see how things go.

Installing the vSphere client gets worse and worse though – my VM needed an update to its Microsoft .Not Framework (something I tend to avoid) which churned away for quite some time, and the console looks to be written in J# too (yes seriously – J# – what the… ?), which meant another “Framework upgrade” which took another while and a half – so the client set up ended up taking about 10 times longer than the server, and I had to run a Windows VM just for it… not too cool.

Once done, I could point my vSphere client at the IP address of the ESXi (VM) using its advertised IP address, the default user name (which I’d forgotten to take a note of – it’s “root”) and the password I’d specified during the install. This gets me to the Hypervisor where I can start creating and managing my own VM’s.

Cool stuff, despite the client letting things down.

Setting up OpenNebula on Ubuntu

Some very rough notes on installing and configuring OpenNebula on an Ubuntu host.

On the Server host:

apt-get install opennebula

Adding system user `oneadmin’ (UID nnn) …
Adding new user `oneadmin’ (UID nnn) with group `nogroup’ …
The home directory `/var/lib/one’ already exists.  Not copying from `/etc/skel’.
adduser: Warning: The home directory `/var/lib/one’ does not belong to the user you are currently creating.
Generating public/private rsa key pair.
Your identification has been saved in /var/lib/one/.ssh/id_rsa.
Your public key has been saved in /var/lib/one/.ssh/id_rsa.pub.
The key fingerprint is:
(key fingerprint)oneadmin@linux
The key’s randomart image is:
+–[ RSA 2048]—-+
|     .    o..    |
(more random artwork)

 

test the installation by running the new “onehost” command:

 

Usage:
onehost [<options>] <command> [<parameters>]

Options:
-l, –list x,y,z                 Selects columns to display with list
command
–list-columns               Information about the columns available
to display, order or filter
-o, –order x,y,z                Order by these columns, column starting
with – means decreasing order
-f, –filter x,y,z               Filter data. An array is specified
with column=value pairs.
-d, –delay seconds              Sets the delay in seconds for top
command
-v, –verbose                    Tells more information if the command
is successful
-h, –help                       Shows this help message
–version                    Shows version and copyright information

Commands:

* create (Adds a new machine to the pool)
onehost create <hostname> <im_mad> <vmm_mad> <tm_mad>

* show (Gets info from a host)
onehost show <host_id>

* delete (Removes a machine from the pool)
onehost delete <host_id>

* list (Lists machines in the pool)
onehost list

* enable (Enables host)
onehost enable <host_id>

* disable (Disables host)
onehost disable <host_id>

* top (Lists hosts continuously)
onehost top

 

So far so simple, so it’s time to set up a new host and install the client…

 

This is done by installing the node package like so:

sudo apt-get install opennebula-node

then starting defining and adding the node to the Master instance via the onehost command.

After that’s done, you can move on to set up a private network for your cloud, create you own KVM images, and start firing up VM’s in your own personal cloud.

More detail coming soon…

Pin It on Pinterest