Extracting data from Jenkins

 

 

In Part I,  Information Radiators, I covered what they are, what the main benefits are, and the approach I usually use to set them up. This post goes in to more technical detail on how I extract this data from Jenkins.

My usual setup/architecture for Jenkins Information Radiators goes something along these lines:

  • TV screens running Mozilla Firefox or Google Chrome in Kiosk Mode, and Tab Mix Plus set up to rotate tabs (if required)
  • JSP Pages served via Tomcat on Linux server (which also runs the data extracting script described below)
  • MySQL database on Linux server – contains tables with data pulled from Jenkins and other sources, and the config data too (which URL’s to monitor)

And you’ll need some Jenkins instances/jobs to monitor too, obviously 🙂

The Jenkins XML API is very useful for automating tasks like this – if you simply append “/api/xml” to a
Jenkins job URL, it will serve up an XML version – note there is also a JSON API and a CLI and plenty of other options, but I’m using what suits me.

The Jenkins XML API

For example, if you go to one of your Jenkins jobs and add /api/xml like this:

“http://yourjenkinsserver:8080/job/yourjobname/api/xml

you should get back some XML, possibly roughly like this example:

<?xml version="1.0"?>
<freeStyleBuild>
 <action>
 <parameter>
 <name>LOWER_ENV</name>
 <value>dev</value>
 </parameter>
 </action>
 <action>
 <cause>
 <shortDescription>Started by timer</shortDescription>
 </cause>
 </action>
 <building>false</building>
 <duration>61886</duration>
 <fullDisplayName>MyJob #580</fullDisplayName>
 <id>2014-04-01_10-01-50</id>
 <keepLog>false</keepLog>
 <number>580</number>
 <result>SUCCESS</result>
 <timestamp>1396342910088</timestamp>
 <url>http://jenkinsserver:8080/view/MyView/job/MyJob/580/</url>
 <builtOn/>
 <changeSet/>
</freeStyleBuild>

That XML contains loads of very useful information inside handy XML tag descriptions – you just need a way to get at that data and then you can present it as you like…

XPAth queries and the Jenkins XML API

so to automate that, I used to extend that approach a to query Jenkins via the XML API using XPAth queries to bring back just the data I actually wanted, quite like querying a database.

For example, wget’ing this URL would return just the current value of the <building> tag in the above XML:

http://yourjenkinsserver:8080/job/yourjobname/api/xml?xpath=//building/text()

e.g. “true” or “false” – this was very useful and easy to do, but the functionality was removed/disabled in recent versions of Jenkins for security reasons, meaning that my processes that used it needed rewritten 🙁

Extracting the data – Plan B…

So, here’s the new solution I went for – the real scripts/methods do some error handling and cleaning up etc but I’m just highlighting the main functions and the high level logic behind each of them here;

get_url’s method:

query a table in MySQL that contains a list of the job names and URL’s to monitor
for each $JOB_NAME found, it calls the get_file method, passing that the URL as a parameter.

get_file method:

this takes a URL param, and uses curl to fetch and save the XML data from that URL to a temporary file (“xmlfile”):

curl -sL "$1" | xmllint --format - > xmlfile

Note I’m using “xmllint –format” there to nicely format the XML data, which makes processing it later much easier.

get_data method:

this first calls “get_if_building” (see below) to see if the job is currently running or not, then it does:

TRUE_VAR="true"
 if [[ "$IS_BUILDING" == "$TRUE_VAR" ]]; then
 RESULT_TEXT="building..."
 else
 RESULT_TEXT=`grep "result>" xmlfile | awk -F\> '{print $2}' | awk -F\< '{print $1}'`
 fi

get_if_building method:

this simply checks and sets the IS_BUILDING var like so:

IS_BUILDING=`grep building xmlfile | awk -F\> '{print $2}' | awk -F\< '{print $1}'`

Putting it all together

My script then updates the MySQL database with the results from each check: success/failure, date, build number, user, change details etc

I then have JSP pages that read data from that table, and translate things like true/false in to HTML that sets the background colours (Red, Amber, Green), and shows the appropriate blocks and details per job.

If you have a few browsers/TV’s or Monitors showing these strategically placed around the office, developers get rapid feedback on the result of their code changes which speeds up development, increases quality and reduces development time and costs – and they can be fun to watch and set up too 🙂

Cheers,

Don

DIY Information Radiator

 

 

Information Radiators are used to provide people with feedback on the current status of code builds and automated tests in Continuous Integration and Agile development environments.

The basic idea is that when developers commit a code change, they can easily and quickly see that it has been picked up by the automated build process, and then (ideally within 10 minutes) see the result of their change; did the build succeeded and did the automated tests pass?

Martin Fowler’s description goes in to more detail on the ideas behind this approach and the function that Information Radiators serve.

The normal convention for these is to use colour coded blocks per build, using:

  • Green for good/passing jobs
  • Amber for either currently building/running jobs (or sometimes for unstable ones)
  • Red for failed jobs

Generally you want to keep things as clean, simple and uncluttered as possible, but sometimes it is helpful to add in a bit more info.

Details I have occasionally found worth adding include things like;

  • name and/or picture of the user who triggered (or broke!) the build
  • commit message from the code change that triggered the build
  • build number
  • history – number of recent fails or passes
  • date/time last failed and last checked

if you use amber for “unstable” rather than “build in progress”, you may want to add text to say if the job is currently building – I often use the “spinning wheel” icon thingy from jenkins itself :

spinner

Why build your own?

There are tons of readily available plugins that allow you to quickly and easily produce a Radiator or Wall panel from Jenkins, so why go to the bother of making your own?

Plugins are usually linked to one Jenkins instance (the one they are running on) and I have often found that alone to be too restrictive – having too many different radiators all over the place makes things too cluttered and uncertain, and people can easily start to “switch off” from them all – having one screen that people can understand at a glance usually works best.

Changing requirements – developers are constantly wanting/looking to improve processes and often come up with ideas and requests for things to try that may help them do their jobs – adding a bit of information from another source, for example, or changing the colours used to a different shade, or adding curved borders etc etc…

What I have found often works best, is to get all of the data I am interested in inside a database then write my own simple but flexible presentation layer from scratch – this gives me all the flexibility I could want (or may find myself wanting or needing later…) and importantly, it also allows me to leverage additional benefits by combining data from Jenkins with data from elsewhere – this means I can easily produce reports, charts, metrics etc that present a view comprised from multiple data sources throughout an organisation – for example, you can then easily create reports that combine:

  • jenkins – live information on the current state of builds and tests
  • defects – data on bugs and changes pulled from bug tracking apps (usually via jenkins jobs)
  • version control – the actual code change can be extracted from version control and linked to both the developer and bug details – the “svn log” command is useful for this
  • environment monitors – state and health of environments; database and app server health, deployed patch and code levels etc (again, these are usually other Jenkins jobs!)

and you can add in anything else you can get your hands on 🙂

This allows you to track the flow of a change right through the development life cycle – from the initial change/requirement/defect to the change itself at the code/file level, then the testing and building of it and the eventual release. This is far more than you need for a simple Information Radiator, but using this approach means that you can easily reuse much of the work in different ways.

Part II covers the technical approach I use for Extracting data from Jenkins

Solaris 10 on ESXi5, with SunRay Server and Clients

About 10 years ago, I bought a SunRay 1g Client box with the plan of setting it up as a thin client to the (physical and very noisy) Sun Solaris Server I was running at the time. The old Sun server sadly didn’t last much longer, and I replaced it with a FreeBSD one before I managed to get the SunRay up and running… so it sat in a cupboard until recently.

The new plan is to use the SunRay 1g as a thin client to a Solaris VM running on my ESXi5 server. This seems like a good idea as it gives me a way to access and manage the ESX server without having to fire up another machine – plus, I think SunRay clients are awesome bits of kit – if I didn’t have an iMac I would use these, as they are silent, boot instantly, use hardly any power and make no noise… it seems like the ideal compliment to an ESX server.

Here’s a pic of my SunRay client…

 

So, here are some notes, links and pics on installing an Oracle Solaris 10 Virtual Machine on ESXi5, setting up the SunRay Server Software, and getting the SunRay client (finally) working…

From the Oracle site, Sun Ray Clients are:

“… simple, low-cost devices that are ideal for displaying server-hosted virtual desktops. With no moving parts and no local operating system to manage, Sun Ray Clients provide a cost-effective, highly functional thin client alternative to desktop and laptop computers and reduce many of the problems associated with traditional desktop deployments.”

http://www.oracle.com/us/technologies/virtualization/061984.html

 

They need a Server running the SunRay Server Software to connect to – I think you can use Linux these days but I wanted to set up a Solaris host anyway, so I went with the traditional approach – albeit running on VM Ware.

None of this was very difficult, and the only slight issue I had was getting DHCP set up correctly; as I understand it, Sun Ray clients need to be sent additional information when they get an IP address, and that means it’s easier to set up the SunRay Server as your DHCP server – once I’d done that (and configured my Router to act as a DHCP Relay, handing DHCP requests over to the Solaris host), it all worked well.

 

Install Solaris 10 on VMWare ESXi5:

I downloaded the DVD image from Oracle, and then uploaded it to the data store on the ESX Server, created a new VM and set it to mount the image as a DVD drive at boot time. The spec of my Solaris 10 VM is 8GB RAM, 2x vCPU and 500G disk for now – I will change this later when I see how it performs.

Install VM Ware tools:

mkdir / vmwaretools/

cp /cdrom/vmwaretools/vmware-solaris-tools.tar.gz .

# tar zxvf vmware-solaris-tools.tar.gz – that didn’t work – no GNU tar 🙁

gunzip vmware-solaris-tools.tar.gz

tar xvf vmware-solaris-tools.tar

cd vmware-tools-distrib/

./vmware-install.pl

All done, much better 🙂

 

Setup  DHCP & NIS on Solaris 10

This command starts up the (GUI) DHCP Manager wizardy thing, which will prompt you for all required info to set up DHCP:
/usr/sadm/admin/bin/dhcpmgr &

As as I understand it, your SunRay clients will look for a DHCP server to (also) give them the details of the SunRay Server to connect to. I coudln’t see a way to use my existing DHCP server (my broadband router) and still be able to tell the clients how to connect to the Solaris Server – that would have been preferable for me.

cp /etc/nsswitch.dns /etc/nsswitch.conf

add the google DNS Server and the internal IP of my router to /etc/resolv.conf….

echo “nameserver 8.8.8.8” >> /etc/resolv.conf
echo “nameserver 192.168.0.1” >> /etc/resolv.conf

 

Install SunRay Server Software (SRSS)

This was all pretty simple, my notes are brief but there’s a decent guide here:

http://docs.oracle.com/cd/E22662_01/E22659/html/Installing-Howto-Install.html

I downloaded the installer and extracted it to a tar file on another host beforehand, which was handy as I hadn’t sorted out GNU tar yet…

mkdir /sunrayinstall

cd /sunrayinstall

scp don@myubuntuserver:/tmp/V32065-01.zip .

unzip V32065-01.zip

cd srs_5.3.0.0-SunOS.i386/

./utinstall

export SRSS_INST=/opt/SUNWut

# Enable LAN Access:

$SRSS_INST/sbin/utadm -L on

$SRSS_INST/sbin/utrestart

Things to look at later:

connector for vmware: http://docs.oracle.com/cd/E21907_01/PDF/EN/SRVMWC1dot2InstallGuide.pdf

 

Here are some of the other commands and checks I looked at while getting it running…

/opt/SUNWut/sbin/utdesktop

/opt/SUNWut/sbin/utgstatus

/opt/SUNWut/sbin/utquery -d 192.168.0.58

/opt/SUNWut/sbin/utsetup

/opt/SUNWut/sbin/utconfig

tail -1000f /var/opt/SUNWut/log/messages

 

install Apache Tomcat for SunRay Server Software

I’m not sure if this is necessary?

cd /opt/apache-tomcat

cd bin

export JRE_HOME=/usr

./catalina.sh start

 

Setup NTP on Solaris 10

ls -al /etc/inet/ntp.*

touch /var/ntp/ntp.drift

cp /etc/inet/ntp.client /etc/inet/ntp.conf

svcadm enable svc:/network/ntp

date

Here’s a picture of the new Admin Tool – I only have 2 SunRay clients to manage (and even fewer users!), but I may still have to search eBay for some Smart Cards…

http://docs.oracle.com/cd/E19214-01/820-0412/new_admin_tool.html

 

Once that was all done and my Router was configured to relay DHCP requests to the DHCP Server running on the Solaris 10 VM, I was able to switch on my SunRay 1g client and be presented with a logon screen – yay!

And finally, here’s a pic of the SunRay client sitting on top of the HP ProLiant ML110 G6 ESXi5 home lab server, which is running the Solaris 10 Virtual Machine it’s connecting to…

 

Persisting file permissions in a tar.gz file with Ant & tar

Discovered an interesting issue recently where file permissions were not being preserved when Taring up a build with Ant.

The existing approach was to chmod the files as desired then simply tar them all up and hope for the best:

<tar destfile=”dist/${hudson.build.tag}.tar.gz” basedir=”dist/” compression=”gzip” />

This doesn’t work,  but if you use tarfileset and set the filemode you can explicitly set things as required like this:

<tar destfile=”dist/${hudson.build.tag}.tar.gz” longfile=”gnu” compression=”gzip”>

<tarfileset dir=”dist/” filemode=”755″>

<include name=”**/*scripts/*.sh” />

<include name=”**/somescript.ext” />

</tarfileset>

<tarfileset dir=”dist/”>

<include name=”**/*” />

<exclude name=”**/*scripts/*.sh” />

<exclude name=”**/somescript.ext” />

</tarfileset>

</tar>

Here I am adding the first two scripts with chmod 755, then adding everything else which will get the default/umask permissions. I exclude the previous files – not sure if that’s required or not but I don’t want to risk overwriting them.

Now when you gunzip and tar xvf the resulting build, you get the required permissions.

There’s more info and further examples in the Apache Ant Manual.

Cheers,

Don

Serving WordPress as the default page

Here’s a note of what I needed to do in order to get WordPress serving as the default site on my domain – it was originally at www.donaldsimpson.co.uk/wordpress/ and I wanted it to just be www.donaldsimpson.co.uk

A bit of a Google shows there are many ways to do this, but here’s how I did it:

vi /opt/bitnami/apache2/conf/httpd.conf

then comment the current entry and add a new one pointing to the htdocs dir for WordPress:

#DocumentRoot “/opt/bitnami/apache2/htdocs”
DocumentRoot “/opt/bitnami/apps/wordpress/htdocs”

Then restart Apache (/opt/bitnami/apache2/bin/apachectl restart or similar) after which you just need to go to the WordPress Admin General Settings page and change these values to point to the root of your site/domain:

WodPress address (URL) www.donaldsimpson.co.uk

Site address (URL) www.donaldsimpson.co.uk

And that should be that – you can now delete that backup you made at the start…

 

Update:

It’s may be a good idea to define your WP_HOME and WP_SITEURL in your wp-config.php file too, like so:

define(‘WP_HOME’, ‘http://www.donaldsimpson.co.uk’);
define(‘WP_SITEURL’, ‘http://www.donaldsimpson.co.uk’);

This avoids a database lookup to get these details, which should speed things up fractionally too 🙂