SCons on Ubuntu for C++ and Java builds

Hi,

Here are my notes from playing around with SCons – I’m using it to build both C++ and Java projects, and thought I’d share my notes using a simple example of the setup and approach used for anyone else new to SCons.

This files for this project can be cloned from GitHub:
https://github.com/DonaldSimpson/java_and_c_with_scons

It’s very easy to install scons on Ubuntu via apt-get, just:

sudo apt-get install scons

You can then check your scons install & version info:

scons --version
SCons by Steven Knight et al.:
 script: v2.2.0.issue-2856:2676:d23b7a2f45e8[MODIFIED], 2012/08/05 15:38:28, by garyo on oberbrunner-dev
 engine: v2.2.0.issue-2856:2676:d23b7a2f45e8[MODIFIED], 2012/08/05 15:38:28, by garyo on oberbrunner-dev
 engine path: ['/usr/lib/scons/SCons']
Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 The SCons Foundation

Creating a new project is trivial – it’s all relative to this directory:

mkdir newscons
cd newscons

and for the simplest example I could think of , let’s use Hello World in C++:

vim hello.c

#include <stdio.h>
int
main()
{
 printf("Hello, from C world!n");
}

that’s it – :wq from vim, then create a new SConstruct file and add in this one line:

Program('hello.c')

– that’s that done, :wq again.

so now we have just two files; the source for a tiny c program, and the one-line SConstruct file defining it

don@ubuntuserver:~/newscons$ ls -al
total 16
drwxrwxr-x 2 don don 4096 Oct 1 11:33 .
drwxr-xr-x 42 don don 4096 Oct 1 11:26 ..
-rw-rw-r-- 1 don don 51 Oct 1 11:31 hello.c
-rw-rw-r-- 1 don don 40 Oct 1 11:32 SConstruct

you can now kick off the scons build process by simply typing “scons”…

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o hello.o -c hello.c
gcc -o hello hello.o
scons: done building targets.

and then test the compiled C program…

don@ubuntuserver:~/newscons$ ./hello
Hello, from C world!

So, scons has just created the binary “hello”, a “hello.o” Object file and “.sconsign.dblite”, which is what scons uses to store file signatures so that if you now do “scons” again (without making any changes), you will see something like the following:

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.

as there have been no changes, but if you

don@ubuntuserver:~/newscons$ rm .sconsign.dblite

then try again…

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o hello.o -c hello.c
gcc -o hello hello.o
scons: done building targets.

that will make scons recompile regardless.

Now for the Java SCons example, let’s create a src/ dir and add in a simple Java example…

mkdir src

vim src/HelloWorld.java

class HelloWorld
{
 public static void main(String args[])
 {
 System.out.println("Hello, Java World!");
 }
}

Then edit the SConstruct to tell scons about the Java example – using “build” as the output destination for the built Java class file, and “src” as the source dir – I’m also leaving in the C++ example here too…

don@ubuntuserver:~/newscons$ cat SConstruct
Program('hello.c')
Java('build','src')
don@ubuntuserver:~/newscons$

run that, and you get:

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src/HelloWorls.java
scons: done building targets.

so we now have the following files in our scons project:

don@ubuntuserver:~/newscons$ find .
.
./hello
./build
./build/HelloWorld.class
./hello.c
./hello.o
./.sconsign.dblite
./SConstruct
./src
./src/HelloWorls.java

and the new Java class can be tested like so:

don@ubuntuserver:~/newscons$ java -classpath build HelloWorld
Hello, Java World!

– that’s it for this simple example – there’s tons more that SCons can do, and here’s a good place to look: http://www.scons.org/

Cheers,

Don

Monitoring Jenkins Slave Nodes with Groovy

This script is now available on GitHub:

https://github.com/DonaldSimpson/groovy

Here is a simple little Groovy script to monitor the health of slaves in Jenkins and let you know when they fail.

I use a lot of Jenkins slaves on many different Jenkins master instances, and I often have something like this running in a Jenkins job that lets me know if there are issues anywhere – which it will do if any of the Jenkins Slave Nodes are offline or dead etc.

To set this up, you first need to install the Groovy Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Groovy+plugin).

Do this in Jenkins by going to “Manage Jenkins -> Manage Plugins -> Available”, then search for “Groovy Plugin”.

Once that’s done, create a new “Freestyle” Jenkins job and add a Build Step to “Execute System Groovy Script”.

Here’s the code:

int exitcode = 0
for (slave in hudson.model.Hudson.instance.slaves) {
 if (slave.getComputer().isOffline().toString() == "true"){
 println('Slave ' + slave.name + " is offline!"); 
 exitcode++;
 }
}

if (exitcode > 0){
 println("We have a Slave down, failing the build!");
 return 1;
}

You can see what’s going on here, but to be clear the main processing logic is:

for each Slave;
-  if it's offline, increment the exit code

if the exit code is greater than zero;
 - return 1, which will fail the job and trigger an email alert.

Obviously this is deliberately simple, and you could extend it to go off and do any number of things – including trying to bring the slave back online again with something like this: https://wiki.jenkins-ci.org/display/JENKINS/Monitor+and+Restart+Offline+Slaves.

I then configure the job run periodically through the Jenkins cron (check “build periodically” in the build triggers section of your jobs config, then put in a cron schedule – “* * * * *” or “@midnight” for example), and to email me if it fails so I know there’s something up.

A word of warning: don’t try changing the “return 1” to something like System.exit(1)… I did that initially, and it killed the running Jenkins instance… doh! 🙂

So when the Groovy script detects an Offline Jenkins Slave Node, the console output should look something like this:


Building on master in workspace /apps/jenkins/jobs/MonitorSlaves/workspace
Slave <YOURSLAVENAME> is offline!
We have a Slave down, failing the build!
Script returned: 1
Build step 'Execute system Groovy script' marked build as failure
Finished: FAILURE
<send an email or whatever here...>

Cheers,

Don

Jenkins Agent Nodes – using the Swarm Plugin

NOTE: This post and approach is quite old now; a better alternative for dynamically provisioning and scaling Jenkins Agents and running CI/CD Pipelines is demonstrated here.


I’ve been trying out a new (to me at least) way to add a Jenkins Agent Node – using UDP auto discovery via the Jenkins Swarm Plugin

This is a very easy and nice way to do it, with minimal configuration/hassle required so you can quickly and easily add new Jenkins Agent Nodes to your Master Jenkins instance as and when they are required.

Here are my notes from setting this up – it’s pretty simple to do and worked well for me out of the box…

Set up a new instance of Jenkins:

  • Make & cd to a working directory

mkdir jenkinsswarm; cd jenkinsswarm

  • Fetch jenkins.war

curl -O http://mirrors.karan.org/jenkins/war/1.506/jenkins.war

  • Start Jenkins

{/path/to/java/bin/}java -jar jenkins.war

After that, you should get console output along these lines…

Running from: /root/jenkinsswarm/jenkins.war
webroot: $user.home/.jenkins
18-Mar-2013 15:19:26 winstone.Logger logInternal
INFO: Beginning extraction from war file
Jenkins home directory: /root/.jenkins found at: $user.home/.jenkins
18-Mar-2013 15:19:33 winstone.Logger logInternal
INFO: HTTP Listener started: port=8080
18-Mar-2013 15:19:33 winstone.Logger logInternal
INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled
18-Mar-2013 15:19:34 jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
18-Mar-2013 15:19:41 jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
18-Mar-2013 15:19:41 jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
18-Mar-2013 15:19:44 org.jenkinsci.main.modules.sshd.SSHD start
INFO: Started SSHD at port 25133
18-Mar-2013 15:19:44 jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
18-Mar-2013 15:19:44 hudson.TcpAgentListener <init>
INFO: JNLP agent listener started on TCP port 41790
18-Mar-2013 15:19:44 hudson.WebAppMain$2 run
INFO: Jenkins is fully up and running

– that looks happy enough, and as you can see from the line “HTTP Listener started: port=8080” it’s running on the default port, so connect to http://yourhost:8080 and you should see something like this…

the next step is to install the Swarm Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin) on this Jenkins Master instance so that Swarm Clients can connect to it.

Do this by going to “Manage Jenkins > Manage Plugins > Available” then selecting to install the “Swarm Plugin“.

Once that’s done you should see that the plugin has been installed…


Now that your new Jenkins server is set up and ready, hop over to your other Jenkins Agent/Client host and do the following…

mkdir for the swarm client

mkdir swarmclient; cd swarmclient/

Get the Swarm Client jar file from the ‘net

curl -O http://maven.jenkins-ci.org/content/repositories/releases/org/jenkins-ci/plugins/swarm-client/1.8/swarm-client-1.8-jar-with-dependencies.jar

Start up the Client

java -jar swarm-client-1.8-jar-with-dependencies.jar
Found 1 eligible Jenkins.
Connecting to http://mydomain.com:8080/
Attempting to connect to http://mydomain.com:8080/ a2721b16-04e4-0d962
18-Mar-2013 15:33:22 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Required credentials not available for BASIC <any realm>@mydomain.com:8080
18-Mar-2013 15:33:22 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Preemptive authentication requested but no default credentials available
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Hudson agent is running in headless mode.
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://mydomain.com:8080/]
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to myhost.mydomain.com:43932
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

Now take a look at your browser and you should see a new Node automatically added to the Master Jenkins instance…

A very handy and flexible approach to adding/managing Nodes and workload – many thanks to the developers behind this!

Cheers,

Don

Getting the current user in Jenkins

Updated! Yet Again!

There are several ideas and comments below on ways to get the logged in user in Jenkins – the person/user that triggered the current build.

This has changed/broken/altered quite a few times since I wrote the initial post (about 8 years ago) as changes have been made to Jenkins – not least being the introduction of Pipelines and Groovy! For quite some time the best solution was, probably, to use a plugin for this (see “build user vars” below).

If you’re writing a Jenkins Pipeline and you’re happy adding in the Build User Vars plugin, then you can simply do this to access the BUILD_USER variable in your Jenkins groovy Pipeline job:

wrap([$class: 'BuildUser']) {
  echo "BUILD_USER that started this Pipeline: ${BUILD_USER}"
}

A few alternative and older ideas follow…

I recently wanted (again) to get the logged-in user that triggered the build in Jenkins and came across my own post when searching… doh. I didn’t really feel like adding a plugin just for this one – supposedly simple – task so had another look at ways to achieve this, and this latest approach seems to work well…

It’s not pretty or robust, and I’m doing it in two goes (to demonstrate getting the correct line then refining things with sed to just the name) which you could simplify, but this is hopefully enough to illustrate the general plan. You’d need to validate it for builds that are started by timer and things like that too.

Here is the plain text version:

# Just show the output:
set -
# change dir to where this builds files are kept:
cd /home/don/jenkins/home/jobs/GetJenkinsJobUserName/builds/${BUILD_NUMBER}
# To start, just get the whole string from the 'log' file:
export STARTED_BY="`grep -i Started log`"
# Output it to the console:
echo "STARTED BY USER = ${STARTED_BY}"
# refine this to just the user name:
export JUST_NAME="`echo "${STARTED_BY}" | sed "s@Started by user@@"`"
echo "Jenkins User Name is ${JUST_NAME}"

This produces the following output when run:

That seems to do the job!

All the original thoughts and comments are below….

Here is a bit of code for getting the details of the logged-in user in Jenkins or Hudson – useful for auditing which user started a given build. I often add this to an audit database, that over time builds up a log of who has done what, where and when, as well as indicating the current state – I then query that through a JSP page that generates a dynamic table that is included via an iFrame in the Description part of the corresponding job… but anyway, back on topic…

Update: the Build User Vars plugin can now provide this functionality. Just install that then check the “Set jenkins user build variables” checkbox under the “Build Environment” section in the configuration page for the job you want to use it, then you should be able to access the following variables in your job:

  • BUILD_USER — full name of user started build,
  • BUILD_USER_FIRST_NAME — first name of user started build,
  • BUILD_USER_LAST_NAME — last name of user started build,
  • BUILD_USER_ID — id of user started build.

As far as I know, there’s no built-in way to do this (apart from the plugin now mentioned above). e.g. through a Jenkins environment variable like $CURRENT_USER or $USER_ID, but using wget, the Jenkins XML API and a little bit of XPath, you can easily query for many things at runtime like so:

THIS_USER=”$(wget -nv –no-proxy “$BUILD_URL/api/xml”‘?xpath=//userId/text()’ -O -)”

That line fetches the logged-on user id by requesting (via wget) the XML for the current BUILD_URL, filtering for the userId text and assigning that value to the variable $THIS_USER, so that it can be used elsewhere in the job.

You can change this around to get the username if you prefer:

THIS_USER=”$(wget -nv –no-proxy “$BUILD_URL/api/xml”‘?xpath=//userName/text()’ -O -)”

Or, try adding “/api/xml” to the end of a Jenkins URL, and you can see the numerous other elements you could request via the Jenkins XML API and XPath.

I have often used this approach to query and report on the status of multiple downstream jobs, for example. Sometimes it’s easier to wget the XML then parse it locally at runtime (with sed/awk/whatever you like) for the data you are interested in – the number of passed, failed or unstable builds can be counted by searching for the number of corresponding “blue”, “red” or “yellow” elements, for example.

UPDATE: It looks like the current version of Jenkins has disabled or broken this feature/ability, but there are always alternative solutions…

For example, if you have curl and xml_grep to hand, you could do it like this…

THIS_USER=`curl –silent $BUILD_URL | xml_grep –text_only userName`

most systems will have curl or alternatively wget, and if you don’t have xml_grep some plain old grep or awk will do the job.

Cheers,

Don

New Home/lab ESXi 5 Server – Part 1

I wrote a while ago about my plans to set up a home VM Ware ESXi 5 server… and although you can’t tell the difference, this site is now happily running on it 🙂

Spec:

The server I went for is an HP ProLiant ML110 G6. It’s got a single socket Xeon Quad Core X3430 processor, 16GB of RAM and at the moment 2TB of SATA3 disk – I will add more when I finish migrating data off the old servers.

There are some limits on the spec you can use with the free version of ESX – I think it’s currently one physical processor and 32G RAM, which means this server is fine.

Install and setup:

The ML110 is a compact and well-made server and is very quiet when up and running – it sounds like a hovercraft for the first few seconds, but quickly calms down to run not much louder than a normal desktop PC. The chassis isn’t large but it’s well laid out and there’s room for 4 HDD’s in there, maybe more if required if the DVD drive was removed (it’s not on the ESX compatibility list anyway so I can’t use it for this).

I installed ESXi 5 from a 2GB USB drive which is attached to the USB socket directly on the motherboard – the BIOS boots from this no problem, and gave me the option to install ESXi 5 on to the USB drive, leaving my HDD(‘s) free to use as 100% dedicated ESX Datastores, and also meaning I can exchange them when I need to without having to reinstall or worry about the VM Ware OS.

The server-side installation of ESXi 5 is a breeze – I rehearsed and wrote about it on my ESXi 5 on an iMac under Fusion post  and it was no different here – it took about 10 minutes or so and there were no tricky questions. Specify the user name and password, tell it to use all of the available HDD and there’s not much more to it. Once you have set up networking the way you want it (the DHCP setup should be fine for most installs) you don’t really need to go near the server again – it’s all managed remotely via the client applications and SSH (more on that in Part 2) from then on.

Creating VMs:

Once you have the vSphere client install done (it’s far more trouble than the server install – plus it requires .Not and J# runtimes – argh! – so it requires Windows – double-argh! – so had to be done initially on a Windows VM running on my iMac in my case… triple-argh!) you can connect to your ESXi 5 server using the IP address, User Name and Password that you set up and start creating some VMs.

To get the first VM created (in my case this had to be a Windows one that I could then use to run the vSphere client on and RDP over to instead of having to run a VM on my Mac all the time), I uploaded an ISO image to the datastore that I had created, then added a new VM in the vSphere Client and set that ISO as the CD image it should load at boot time. You just specify the OS type, RAM, CPU, Networking and disk(s) you want and power it on – all very easy and quick.

Converting a VM Ware Workstation VM in to a VM Ware ESX Guest

I also wanted to convert the Ubuntu VM Ware Workstation image that this web site runs on, so I could move it off the old server and have it running on ESX as a “proper” ESX guest/host. This was really easy too; the VM Ware Converter allows you to specify a local VM of pretty much any type and supply the details of the target ESX server, and it then converts and loads it all for you – it took quite a while to complete but it worked without issues, and I was then able to power on my Ubuntu website VM under ESXi 5, where it’s happily running right now. No need to reinstall WordPress, Postfix, PHP, Java, Jenkins, MySQL etc etc – happy days.

Here are some Pics of the ESXi5 console shortly after set-up…

 

1. General info on the reported spec and current overall resource usage of the HP ProLiant ML110 G6:

ESXi 5 console general information

 

2. Some of the Health Monitoring and General Configuration options:

Inventory and general settings and diagnostics pic

3. Overview of Guest and Host resources:

Guest and Host resources

 

Summary:

I did a fair bit of research beforehand to make sure the install and hardware would be ok, which meant the actual set up was trivial – once it’s done all you need to do is create VMs and allocate resources; there is very little work or maintenance required – especially compared to what would be needed to run multiple physical servers all with their own hardware. Creating new VMs is very easy, and the performance is good so far – the processor is not stressed at all, and the ESX memory management does a good job – I’ve had up to 6 VMs running at the same time and still have about half the memory free!

Next plans:

One of my main reasons for doing this was to provide a test platform for automating, creating and managing Linux VMs using Jenkins as a front end and DNS records to control what is deployed where and when – I want to be able to select a few options, then click a button and have my new host created in minutes and to the right spec, similar to the Amazon EC2 set up but code deployment linked in too, and I will write more on this when I’ve done it.

Plans include a mixture of: VM Ware Templates, Perl, Jenkins jobs, Jenkins Nodes, Puppet, Tomcat, etc

Next Post:

There are a few other things I have already done that I’d like to document too including…

Accessing ESXi 5 via SSH – how to and a summary of useful commands etc
More detail on Remote desktop via ssh tunnels etc
VM Ware command line tools
DNS and AD/LDAP servers

Managing Jenkins as a service and starting at boot time

Linux Services

I have an Ubuntu Linux VM that runs Jenkins, and to make life simpler I wanted to set up Jenkins to run as a service. I also wanted it to start up automatically at boot time, as that’s the sole function of the VM it’s running on.

So, here are my notes on setting up Jenkins as a service on Ubuntu Linux, which includes a script to manage (start, stop, check, restart) the Jenkins process too.

On Ubuntu (and most othe rLinux versions) you can check the current services with the command “service –status-all” – this should give you a list of all services, and their current status. You can (as root) also do “service <name> start” (or stop or restart) for each one.

Jenkins as a Service

To create a new service for Jenkins, take a look at the existing scripts in /etc/init.d/ for some examples.

By convention there are three main methods a basic service implements, they are; start, stop and restart. About the most basic structure of a service script is therfore something like this:

#!/bin/bash
start() {
echo “Starting Service”
# Do start things here.
}stop() {
echo “Stopping Service”
# Do stop things here.
}restart() {
echo “Restarting Service…”
stop
start
}

case “$1″ in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)

obviously you still need to implement the start and stop functions though 🙂

To see a far more detailed example, take a look at the file /etc/init.d/skeleton, which you could copy and update to suit your needs if you prefer.

I decided on something halfway between the very basic example above and the far more complex example in “skeleton”, and did something like this (saved as “/etc/init.d/jenkins”):

# Example Jenkins auto-start service script
#
# description: manages Jenkins as a service
# processname: jenkins
# pidfile: /var/run/jenkins.pid
# author: www.DonaldSimpson.co.uk# The user and the home dir that Jenkins runs under
jenkins=/usr/local/jenkins
# Your startup and stop scripts (see below)
startup=$jenkins/bin/startup.sh
shutdwn=$jenkins/bin/shutdown.shstart_jenkins() {
echo “Starting Jenkins services…”
su – jenkins -c “sh $startup”
}

stop_jenkins() {
echo “Stopping Jenkins services…”
su – jenkins -c “sh $shutdwn”
}

status_jenkins() {
# Check for any other process containing jenkins.war
# This could be improved upon (see script below)
numproc=`ps -ef | grep [j]enkins.war | wc -l`
if [ $numproc -ne 0 ]; then
echo “Jenkins is running…”
else
echo “Jenkins is NOT running…”
fi
}

case “$1” in
start)
start_jenkins
;;
stop)
stop_jenkins
;;
status)
status_jenkins
;;
restart)
stop_jenkins
start_jenkins
;;
*)
echo “Usage: $0 {start|stop|status|restart}”
exit 1
esac
exit 0

Update that to suit then save and change the permissions to make it executable:

chmod +x /etc/init.d/jenkins

then you can check (as root) that you can call the methods in the script:

service jenkins status

service jenkins stop

service jenkins start

These should all now run as the Jenkins user via sudo and say something when called, even if it is just “I can’t find the scripts you told me to call” 🙂

Jenkins Startup and Stop scripts

So, you now need to create the missing startup and shutdown scripts, in my example they were:

/usr/local/jenkins/bin/startup.sh

and

/usr/local/jenkins/bin/shutdown.sh

An example of the Jenkins start-up and management script I normally use is at the end of this post – the main idea though, is that it sets what you want to set then starts Jenkins via nohup and outputs to a log file

I have also added and included some basic tests to my scripts and some (very) rudimentary error handling/checking, but you shoudl get the idea and all you really need is this line (with the variables set correctly):

${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &

Note: I often run multiple instances of Jenkins, so I explicitly specify the HTTP_PORT, and I use the -D${MARKER} to allow me to safely and easily find the correct PID for each project.
The httpListenAddress can normally go unset, but it’s something I’ve had to set before on multi-home’d hosts.

The stop part of my Jenkins management script finds the correct PID like this (and you could use a filter for the correct -D${MARKER} if you want that too):

PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`

and simply kills it.

For some of my scripts I also filter for process ID’s that were started in the current directory by checking “pwdx” against the PID, but only where I’m sure the corresponding start-up process is correct/reliable.

So, tweak to your taste and that should be the Jenkins service created and working too now; for start|stop|restart you can create individual scripts or wrappers that call something like the one script below with parameters, or break the script in to separate files if you prefer.

Setting Jenkins to start at boot time

If you want to start Jenkins at boot time/startup automatically then you still need to do one more small step.  There are many different ways to do this depending on personal preference, your requirements and your version of Linux, but on Ubuntu it can be done easily with:

update-rc.d jenkins defaults

“update-rc.d” simply installs and removes System-V style init script links – read the man page for full details, but the same idea applies to most versions of Linux.

update-rc.d -f  jenkins remove

will undo this if you no longer want it.

An example Jenkins server/process management script

As an example, here is my script to manage Jenkins processes – it could use some improvements but the basic start, stop, restart and check should give you enough to sort out something that works and suits your needs.

Better formatted version available here: https://github.com/DonaldSimpson/scripts/blob/master/JenkinsProcessManager.sh

#! /bin/bash -p
# www.donaldsimpson.co.uk
# Script to start|stop|restart|check an instance of Jenkins
# For each new instance, the PROJECT and HTTP_PORT need to be updated:
export PROJECT=jenkins
export HTTP_PORT=9000
EXIT_STRING=””
JENKINS_ROOT=/opt/apps/jenkins
export JENKINS_HOME=${JENKINS_ROOT}/${PROJECT}
JENKINS_WAR=${JENKINS_HOME}/jenkins.warWAIT_TIME=5
START_WAIT_TIME=15
JAVA_HOME=/usr
PATH=$JAVA_HOME/bin:$PATH
JAVA=${JAVA_HOME}/bin/java
NOHUP=/usr/bin/nohup
LOG_FILE=${JENKINS_HOME}/debug_${PROJECT}.log
MARKER=”JenkinsProcFor_${PROJECT}”NC=/bin/nc
WGET=/usr/bin/wget
LSOF=/usr/bin/lsof
AWK=/usr/bin/awk
GREP=/bin/grep
FUSER=/bin/fuser
#################################################################################
### Functions Start #############################################################
#################################################################################

cleanup(){
# Perform any clean-up activities here…
[ “${EXIT_STRING}” != “0” ] && echo `date “+%d/%m/%y %H:%M:%S”` ${EXIT_STRING}
exit 0
}

trap ” QUIT

_error() {
EXIT_STRING=”${1}”
[ “${1}” != “0” ] && EXIT_STRING=”ERROR: ${1}, please investigate, terminating.”
cleanup  # Never returns from this call…
}

say(){
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
}

saybold(){
tput bold 2>/dev/null
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
tput sgr0 2>/dev/null
}

check_folders(){
say “”
saybold “Checking all required folders exist…”
for REQUIRED_DIR in ${JENKINS_ROOT} ${JENKINS_HOME} ${JAVA_HOME}
do
[ ! -d “${REQUIRED_DIR}” ] && _error “Necessary directory: ${REQUIRED_DIR} does not exist”
say “Found required directory: ${REQUIRED_DIR}”
done
saybold “Done.”
say “”
}

check_files(){
say “”
saybold “Checking all required files exist…”
for REQUIRED_FILE in ${JENKINS_WAR} ${NC} ${WGET} ${JAVA} ${LSOF} ${FUSER} ${GREP} ${AWK}
do
[ ! -f “${REQUIRED_FILE}” ] && _error “Necessary file: ${REQUIRED_file} does not exist”
say “Found required file ${REQUIRED_FILE}”
done
saybold “Done.”
say “”
}

check_port_closed(){
say “Checking that port ${HTTP_PORT} is closed…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
_error “Required Jenkins port ${HTTP_PORT} is already in use”
else
say “Ok, required port ${HTTP_PORT} is available, continuing…”
fi
}

check_port_open(){
say “Checking that port ${HTTP_PORT} is open…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
say “Ok, a process is listening on port ${HTTP_PORT}, continuing…”
else
_error “Required Jenkins port ${HTTP_PORT} has not been opened.”
fi
}

start_process(){
cd ${JENKINS_HOME}
saybold “Starting Process now…”
${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &
say “Process initiated.”
}

check_log(){
say “Checking the log file ${LOG_FILE} for an HTTP listener…”
STARTED=`${GREP} -c “HTTP Listener started” ${LOG_FILE}`
if [ ${STARTED} -eq “0” ]; then
_error “An HTTP Listener has not reported as started in the log file ${LOG_FILE}”
else
saybold “An HTTP Listener is reported as started in the log file ${LOG_FILE}”
fi
}

check_html(){
# These checks need error handling, but you get the general idea.
say “Checking that localhost:${HTTP_PORT} is serving Jenkins pages…”
TEMP_WGETDIR=tempwgetdir$$
mkdir ${TEMP_WGETDIR}
cd ${TEMP_WGETDIR}
${WGET} -q http://localhost:${HTTP_PORT}
GOT_HTML=`${GREP} -c Dashboard index.html`
cd ${JENKINS_HOME}
rm -rf ${TEMP_WGETDIR}
if [ ${GOT_HTML} -eq “0” ]; then
_error “Unable to get an HTML page from the Server.”
fi
saybold “Recieved valid HTML from the server, all looks ok.”
}

check_process(){
check_port_open
check_log
check_html
say “A Jenkins instance is listening on port ${HTTP_PORT} for project ${PROJECT}.”
say “The process is logging debug info to the log file: ${LOG_FILE}”
}

stop_proc(){
check_port_open
saybold “Looking for the Process ID attached to port ${HTTP_PORT}”
PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`
if [ “${PID}” == “” ]; then
saybold “Unable to detect the Process ID that is listening on port ${HTTP_PORT}!”
PID=`${FUSER} ${LOG_FILE}`
if [ “${PID}” == “” ]; then
_error “Unable to find the PID that has the log file open too!”
else
say “Ok, found PID ${PID}”
fi
else
saybold “Found a PID of $PID, killing it now…”
fi
kill -9 ${PID}
say “Waiting ${WAIT_TIME} seconds for the process to die…”
sleep ${WAIT_TIME}
saybold “Done, checking port is closed…”
check_port_closed
saybold “All done.”
}

start(){
check_folders
check_files
check_port_closed
start_process
saybold “Waiting ${START_WAIT_TIME} seconds for the process to start up…”
sleep ${START_WAIT_TIME}
check_process
}

restart(){
stop_proc
start
}

#################################################################################
### Script Start ################################################################
#################################################################################

case “$1” in
start)
start
;;
stop)
stop_proc
;;
restart)
restart
;;
check)
check_process
;;
*)
echo “Usage: $0 {start|stop|restart|check}”
esac
# Exit cleanly
_error “0”

Hope that helps! Any constructive comments, requests or suggestions for improvement are very welcome 🙂

Cheers, and sorry about the indenting,

Don

VM Ware ESXi under VM Ware Fusion

Hardware

I’m looking at getting a new home lab/server set up, and am seriously considering ordering one of these:

HP ProLiant ML110 G6 Quad Core X3430

It will need filled with memory and a load of disk space, but I reckon that’s still a whole lot of server for the money.

The Plan

The plan is to install VM Ware ESXi 5 on to it (using the USB Drive), and manage the server through the remote VM Ware vSphere Client app. This will allow me to create new Virtual Machines and migrate my existing appliances over to this server, then I can retire the old servers these have been running on. I’m also wanting to develop some automation processes for managing VM’s – creating new ones, bringing them up and down etc using Jenkins to orchestrate the processes, so this will allow me to work on that too.

Before ordering the ML110 I wanted to take a better look at the installation of ESXi – it doesn’t sound difficult, but while I’ve used it often I’d never set it up before so wanted to see what was involved.

VM Ware ESXi 5 Installation under Fusion on an iMac

It’s a little bit crazy and recursive, but, I realised I may be able to do this on my iMac under VM Ware Fusion, which it turns out does allow you to install ESX as a Virtual Machine itself…. which you can then use to manage and create new (Virtual?) Virtual Machines – a bit of a brain-ouch, and it’s clearly not going to be fast, but it’s good enough for my testing.

Installing ESXi is very straightforward – I selected the obvious option of “VM Ware ESX” in Fusion on my iMac, told it I had an image I wanted to use and pointed it at the VM Ware ESXi 5 ISO image I had downloaded from VM Ware (you need to register then fumble about their site for a while to find and download the free version – that’s the way I did it anyway). Keep a note of the serial number they give you, as that will remove the 60 day trial restriction later.

Fusion suggested a 40G file system for this instance and allocated 2GB RAM and 2 cores, which I was quite happy to run with for my test. Speaking of RAM – there was a restriction on the amount of physical RAM you could use with the free version of ESXi 5 – it was 8GB a while ago but this has now been increased to something more sensible – 32GB I think?

There are no surprises or major decision needed during the install of ESXi 5 itself – it took all of 5 minutes to run through and reboot, and that was with it running as a Virtual Machine on an overloaded iMac – on proper hardware like the HP ProLiant ML110 G6 Quad Core X3430 that I’m looking at, it would be loads faster.

When the install is done, there’s nothing more to see on the ESXi (VM), apart from the HTTP address it gives you to connect to the ESXi Host and access a simple web page it serves with links to download the VM Ware vSphere Client application to another host and start managing your server. This address was given by DHCP in my case – I think you can specify or change this easily if you want to.

Client Installation

Downloading the VM Ware vSphere Client took a while (longer than installing ESXi did!) as it came from the VM Ware site rather than being directly served by the ESXi host. Now for the bad news… for clients, you have a choice of running either Windows or Linux. No mention of an OSX vSphere client, so I had to fire up a Windows VM just to set up the client app on… not what I had been hoping for – there’s a petition asking VM Ware to sort this out here:

http://communities.vmware.com/thread/128538?start=525&tstart=0

When I get things running I could create a VM on my ESXi host which I can RDP on to, but that’s still a pretty ugly solution – if the Linux client is ok I’d go with that over the Windows one, and I think there’s also a Web Interface. But, part of the reason for me doing this in the first place is so that I can look in to the SDK and API’s for automating the creation of VM’s with VM Ware using Jenkins though, so I’ll grin and bear the Windows yuckness and see how things go.

Installing the vSphere client gets worse and worse though – my VM needed an update to its Microsoft .Not Framework (something I tend to avoid) which churned away for quite some time, and the console looks to be written in J# too (yes seriously – J# – what the… ?), which meant another “Framework upgrade” which took another while and a half – so the client set up ended up taking about 10 times longer than the server, and I had to run a Windows VM just for it… not too cool.

Once done, I could point my vSphere client at the IP address of the ESXi (VM) using its advertised IP address, the default user name (which I’d forgotten to take a note of – it’s “root”) and the password I’d specified during the install. This gets me to the Hypervisor where I can start creating and managing my own VM’s.

Cool stuff, despite the client letting things down.

Easy method to tunnel VNC over an SSH connection

This has been done many times in many different ways, but I’ve finally found a workable solution that suits my needs and my setup, and (perhaps most importantly when it comes to ports and tunneling…) that I can get my head around 🙂

 

So… at home, I have a Windows machine with remote desktop enabled, but I don’t want to open up yet another port on my router or firewall (or to expose a Windows machine to the world) for the few times that I want to get a remote desktop connection to it.

But, I have a web-facing Ubuntu Linux host which I use frequently to access the non-Windows parts of my Home Network – another Linux host, an occasionally used FreeBSD VM, Solaris 10 server, a couple of iMac’s and my ATV2.

I can see the Windows file system ok by ssh’ing in to the Linux box (it’s file system is shared and mounted) but I can’t see or access the Windows desktop when I’m outside of my home network. Doing so is handy for access to the VM Ware vClient to manage my ESX Server, etc.

A solution for this is to tunnel the VNC protocol from my remote host to the Internal Windows server via the (already exposed) SSH connection on my web-facing Linux host. To do this from outside my home, I kick off an SSH session like this:

ssh -L 5900:192.168.0.111:5900 myuser@www.donaldsimpson.co.uk

that command breaks down like this:

ssh using the -L arg ([-L [bind_address:]port:host:hostport]), which forwards the local port 5900 to 192.168.0.111, and attaches the tunnel on its port 5900 (the port that RDP or VNC is running on).

Note that the IP address 192.168.0.111 is my “internal” Windows server with a non-routable IP address – it’s accessible to/from my Linux Host (www.donaldsimpson.co.uk), but not from anywhere outside my network.

After that, I specify the SSH logon details for my Linux host and when that connection is made I’m prompted for the password for “myuser@www.donaldsimpson.co.uk“. When that is accepted I am logged on to my Linux box at the console (Cygwin with SSH installed in this case) as per usual, and I’ll need to leave this terminal open. The SSH Tunnel has been created and will end when this session does.

The next and final step is to launch “mstsc” (or VNC or another RDP Client) on my local host and create a connection – not connecting to www.donaldsimpson.co.uk, but to 127.0.0.1:5566 – this is the local end of the tunneled connection; at the other end, via my SSH Host, is the listening port on the Windows host. After supplying the requested credentials, it makes the connection, then finally shows me my Windows Desktop – woohoo, job done.

Setting up Perforce on a Linux Server and a Windows Client

As described on WikiPedia, Perforce “is a commercial, proprietary, centralized revision control system developed by Perforce Software, Inc.”

Like Subversion, it’s a client/server system where the server manages a database of revisioned files, and clients connect to checkout, modify and send back changes for others to pick up.

I wanted to check out the latest version so thought I’d install it on this server and set up a client… and that I may as well capture the steps and put them here.

In my case, the server is an Ubuntu Linux  host, and my client machine is a Windows XP workstation.

There’s not a lot to do when installing Perforce, so getting a straightforward instance up and running is a breeze. Basically, get the binary, export or pass it a few settings if you don’t want the defaults, then kick it off – here’s the detail from my notes:

Download p4d binary (for this platform and architecture):
wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4d

You can put this wherever you want, and set up a P4ROOT to specify the Perforce root directory – but don’t use that location for anything else (including client workspaces).

mkdir /apps/perforce
mv p4d /apps/perforce; cd /apps/perforce
chmod +x p4d

 

Most  Perforce options can either be exported or passed as a command-line arg, so you can choose

export P4ROOT=/apps/perforce
– or –
-r /apps/perforce

The default port is 1666, and remember that if you change this on the Server you will need to change it on your Perforce client(s) too.  In my example I’m using 9002:

export P4PORT=9002
– or –
-p 9002

So I ended up with a command line that looked like this:

nohup ./p4d -r /apps/perforce -J /var/log/journal -L /var/log/p4err -p 9002 &

Which I will probably put in to a simple startPerforce.sh script, and a probably a stopPerforce.sh script too that contains this and maybe the port number and full path to the binary location:

p4 admin stop

That’s it for the Server side at the moment, there’s a process up and running (you can check the output in nohup.out) so it’s time to set up and connect a client…

 

I’m going for a Windows client installation, which means downloading the correct version from the Perforce site then running p4vinst.exe. There’s nothing really to report here; select the usual options like directories and let it install.

Oh, I also needed to create a user, so back on the Linux Server I also downloaded the p4 binary:

wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4

Exported the P4PORT (since I was using a custom one to get through my firewalls), then add a user:

export P4PORT=9002
chmod +x p4
./p4 user -f Donald
User Donald saved.

Now you can test connecting to your Perforce Server with the  P4Admin GUI and the P4V GUI on the Windows client host by specifying the correct port (if you changed the default) and a valid user name. Once that’s done, you can admin your Depot’s and add/change/commit files, see revision and history information and all that good stuff. There are also command line and web interfaces too which can be useful for temporary use and for scripting but the Windows GUI’s are nice to use and quite self explanatory – if you’ve used a similar revision control system like Subversion and an Eclipse-like IDE before there’s not much of a learning curve here.

The Perforce Help and Documentation is all very good, and their Perforce 2010.2: System Administrator’s Guide covers all of the above in more detail, and touches on more advanced topics too – Perforce Performance Tuning, Backup and Recovery, Replication, and the Perforce Broker (P4Broker) etc.

There’s also a Perforce plugin for Jenkins, which once installed allows you to choose Perforce as one of the SCM option in your Jenkins jobs, but the above hopefully covers the initial setup of both the Perforce Client and the Perforce Server on Windows and Linux respectively.

 

Jenkins Agent Nodes

This Jenkins Agent Nodes post covers:

  • What are they?
  • Why may I want one?
  • How do you create one?
    • tasks on On the Master/Server
    • tasks on On the Agent/Client
  • Other ways of creating Agent Nodes
  • Related posts and links

What are they?

Jenkins Agents  are small Java “Client” processes that connect back to a “Master” Jenkins instance over the Java Network Launch Protocol (JNLP).

Why may I want one?

Once it’s up and running, an Agent instance can be used to run tasks from a Master Jenkins instance on one or more remote machines providing an easy to use and flexible distributed system which can lend itself to a wide variety of tasks.

As these are Java processes, you are not restricted by architecture and can mix and match the OS’s of your agent nodes as required – Windows, Linux, UNIX, iSeries, OVMS etc – anything capable of running a modern version of Java (I think JNLP was introduced in 1.5?) and you can also group and categorize subsets of different types (both logical and physical) of Agents; intended use, availability, location, available resources, Cloud or VM versus Physical tin – anything that helps you decide when you want to use which host.

There are many different ways you can choose to utilize these nodes – they can be used to spread the load of an intensive build process over many machines when they are available, you can delegate specific tasks to specific machines only, or you can use labels to group different classes or types of Nodes that are available for certain tasks, making the most use of your available resources. You can also have Jenkins create Cloud server instances – Amazon EC2 for example – when certain thresholds are reached, and stop them when they are no longer required.

This post focuses on a pretty manual approach to the creation of Jenkins Agent Nodes with the intention of explaining them well enough to allow you to create them on any platform that can run a modern version of Java – there are probably simpler solutions depending on your needs and setup. A later post will touch on a few of the many possible uses for these nodes.

So, how do you create one?

There are several different ways to go about setting up an Agent, and the “best” approach depends on your situation, needs and environment(s) – for a simple Linux setup letting Jenkins do all the work for you makes life really easy, you can just select that option in your new Jenkins Agent Node and complete this screen to have Jenkins set it up for you:

Where the Username and Password are the credentials you want Jenkins to use to connect and start the Agent process on the remote server. This simple approach also allows the Master instance to initiate the JNLP connection and bring your agents online when required, avoiding any need to manually visit each agent node.

This keeps things nice and simple and reduces the admin overhead too,  but sometimes this type of approach can’t be used (on different OS’s like OVMS, iSeries, Windows etc) and I’m going to go on to outline what I think is the most “versatile” method – defining the Node on the Master instance, and manually setting up and starting the corresponding Agent/Client Node on the remote host – going through these steps should provide enough detail on how Agent Nodes work and connect to get one up and running on anything that can run a JVM.

1. On the Master/Server

Define the host: navigate to Jenkins > Manage Jenkins > Manage Nodes > New Node
Enter a suitable Node Name (I’d recommend something descriptive, and usually including the host name or part of it) then either select to create a “Dumb Agent” or copy an existing Node if you have one, then complete the configuration page similar to this:


where you specify the requested properties – path, labels, usage, executors etc. These are explained in more detail in the “?” for each item if required.

Here you can also state if you want to keep your Jenkins Agent for tied jobs only, or if it is to be utilized as much as possible – this obviously depends on your requirements. You can also specify the Launch method that best applies to your needs & requirements.

2. On the Agent/Client host

You don’t need to do very much to create a new agent node – typically if I’m setting up a few *NIX and Windows hosts I would archive a simple shell/DOS script that starts and manages the process along with the slave.jar file from the Master Jenkins instance. There are alternative methods that may suit your needs – you can start agents via SSH from the Master server for example and there’s a comparable method for Windows – but this simple approach should help you understand the underlying idea that applies to them all.

You can “wget” (or use a Browser on Windows) the slave.jar file directly from the Master Jenkins instance using the URL

http://[your jenkins host]:[port number]/jnlpJars/slave.jar

If you let JNLP initiate the process, the slave.jar will be downloaded from Jenkins automatically.

Note that Jenkins will inherit the effective permissions of the user that starts the process – this is to be expected, but it’s often worth having a think about the security aspects of this, along with the access requirements for the types of things you want your agent to be able to do… or not do.

On Windows hosts, you can use the jenkins-agent.exe to easily install Jenkins as a Windows Service, which can then be started at boot time and run under whatever user/permissions you wish set via the Services panel.

My *NIX “startagent.sh” script does a few environment/sanity checks, then kicks of the agent process something like so:

${NOHUP} ${JAVA} -jar slave.jar -jnlpUrl http://SERVERNAME:PORT/computer/USER__NODENAME/slave-agent.jnlp &

The HTTP URL there should match the one provided by the Jenkins page when you were defining the Node. If all goes well you should see the node state changed to Connected on the Master Hudson instance, and if not, then nohup.out should provide some pretty obvious pointers on the problem.

Some common causes are:

Jenkins host, port or node name wrong
Java version not found/wrong
Lack of write permissions to the file system
Lack of space (check /tmp too)
Port already in use
Errors in the jenkins-slave.xml file if you’ve tweaked it
Firewalls…

Jenkins also provides some health monitoring of the connected Node which you can see in the Jenkins > Nodes page:
Disk Space, Free Temp Space, Clock time/sync, Response Time and Free Swap are monitored
and you can have your Node taken off line if any of these passes a set threshold.

This should hopefully be enough info to provide an overview of what Jenkins Agents are, and enough to get one up and running on your chosen platform. Where possible it’s best to keep things simple – use SSH and let the Master instance manage things for you if you can – but when that’s not possible there are alternatives.

When I get the chance I will add some information on the next steps – creating and delegating jobs on Jenkins Agent Nodes, and some thoughts and suggestions for just a few of the many uses for this sort of distributed Build and Deployment system.

Related Posts and Links:

Monitoring Jenkins Slave Nodes with Groovy
– how to keep an  eye on your Jenkins Slaves

Jenkins Slave Nodes – using the Swarm Plugin
– automatically connect new Slave Nodes to create a “Swarm”

Getting the current user in Jenkins
– several approaches

Managing Jenkins as a service and starting at boot time
– on Linux & Windows

Jenkins plugins
– details on some of my most frequently used plugins

Jenkins DIY Information Radiators
– what they are for, and how to make your own

The Jenkins Wiki has more details information on Distributed Builds and different slave-launching strategies.

Feedback, questions and constructive comments are very welcome!

Pin It on Pinterest