SCons on Ubuntu for C++ and Java builds

Hi,

Here are my notes from playing around with SCons – I’m using it to build both C++ and Java projects, and thought I’d share my notes using a simple example of the setup and approach used for anyone else new to SCons.

This files for this project can be cloned from GitHub:
https://github.com/DonaldSimpson/java_and_c_with_scons

It’s very easy to install scons on Ubuntu via apt-get, just:

sudo apt-get install scons

You can then check your scons install & version info:

scons --version
SCons by Steven Knight et al.:
 script: v2.2.0.issue-2856:2676:d23b7a2f45e8[MODIFIED], 2012/08/05 15:38:28, by garyo on oberbrunner-dev
 engine: v2.2.0.issue-2856:2676:d23b7a2f45e8[MODIFIED], 2012/08/05 15:38:28, by garyo on oberbrunner-dev
 engine path: ['/usr/lib/scons/SCons']
Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 The SCons Foundation

Creating a new project is trivial – it’s all relative to this directory:

mkdir newscons
cd newscons

and for the simplest example I could think of , let’s use Hello World in C++:

vim hello.c

#include <stdio.h>
int
main()
{
 printf("Hello, from C world!n");
}

that’s it – :wq from vim, then create a new SConstruct file and add in this one line:

Program('hello.c')

– that’s that done, :wq again.

so now we have just two files; the source for a tiny c program, and the one-line SConstruct file defining it

don@ubuntuserver:~/newscons$ ls -al
total 16
drwxrwxr-x 2 don don 4096 Oct 1 11:33 .
drwxr-xr-x 42 don don 4096 Oct 1 11:26 ..
-rw-rw-r-- 1 don don 51 Oct 1 11:31 hello.c
-rw-rw-r-- 1 don don 40 Oct 1 11:32 SConstruct

you can now kick off the scons build process by simply typing “scons”…

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o hello.o -c hello.c
gcc -o hello hello.o
scons: done building targets.

and then test the compiled C program…

don@ubuntuserver:~/newscons$ ./hello
Hello, from C world!

So, scons has just created the binary “hello”, a “hello.o” Object file and “.sconsign.dblite”, which is what scons uses to store file signatures so that if you now do “scons” again (without making any changes), you will see something like the following:

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.

as there have been no changes, but if you

don@ubuntuserver:~/newscons$ rm .sconsign.dblite

then try again…

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o hello.o -c hello.c
gcc -o hello hello.o
scons: done building targets.

that will make scons recompile regardless.

Now for the Java SCons example, let’s create a src/ dir and add in a simple Java example…

mkdir src

vim src/HelloWorld.java

class HelloWorld
{
 public static void main(String args[])
 {
 System.out.println("Hello, Java World!");
 }
}

Then edit the SConstruct to tell scons about the Java example – using “build” as the output destination for the built Java class file, and “src” as the source dir – I’m also leaving in the C++ example here too…

don@ubuntuserver:~/newscons$ cat SConstruct
Program('hello.c')
Java('build','src')
don@ubuntuserver:~/newscons$

run that, and you get:

don@ubuntuserver:~/newscons$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src/HelloWorls.java
scons: done building targets.

so we now have the following files in our scons project:

don@ubuntuserver:~/newscons$ find .
.
./hello
./build
./build/HelloWorld.class
./hello.c
./hello.o
./.sconsign.dblite
./SConstruct
./src
./src/HelloWorls.java

and the new Java class can be tested like so:

don@ubuntuserver:~/newscons$ java -classpath build HelloWorld
Hello, Java World!

– that’s it for this simple example – there’s tons more that SCons can do, and here’s a good place to look: http://www.scons.org/

Cheers,

Don

Monitoring Jenkins Slave Nodes with Groovy

This script is now available on GitHub:

https://github.com/DonaldSimpson/groovy

Here is a simple little Groovy script to monitor the health of slaves in Jenkins and let you know when they fail.

I use a lot of Jenkins slaves on many different Jenkins master instances, and I often have something like this running in a Jenkins job that lets me know if there are issues anywhere – which it will do if any of the Jenkins Slave Nodes are offline or dead etc.

To set this up, you first need to install the Groovy Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Groovy+plugin).

Do this in Jenkins by going to “Manage Jenkins -> Manage Plugins -> Available”, then search for “Groovy Plugin”.

Once that’s done, create a new “Freestyle” Jenkins job and add a Build Step to “Execute System Groovy Script”.

Here’s the code:

int exitcode = 0
for (slave in hudson.model.Hudson.instance.slaves) {
 if (slave.getComputer().isOffline().toString() == "true"){
 println('Slave ' + slave.name + " is offline!"); 
 exitcode++;
 }
}

if (exitcode > 0){
 println("We have a Slave down, failing the build!");
 return 1;
}

You can see what’s going on here, but to be clear the main processing logic is:

for each Slave;
-  if it's offline, increment the exit code

if the exit code is greater than zero;
 - return 1, which will fail the job and trigger an email alert.

Obviously this is deliberately simple, and you could extend it to go off and do any number of things – including trying to bring the slave back online again with something like this: https://wiki.jenkins-ci.org/display/JENKINS/Monitor+and+Restart+Offline+Slaves.

I then configure the job run periodically through the Jenkins cron (check “build periodically” in the build triggers section of your jobs config, then put in a cron schedule – “* * * * *” or “@midnight” for example), and to email me if it fails so I know there’s something up.

A word of warning: don’t try changing the “return 1” to something like System.exit(1)… I did that initially, and it killed the running Jenkins instance… doh! 🙂

So when the Groovy script detects an Offline Jenkins Slave Node, the console output should look something like this:


Building on master in workspace /apps/jenkins/jobs/MonitorSlaves/workspace
Slave <YOURSLAVENAME> is offline!
We have a Slave down, failing the build!
Script returned: 1
Build step 'Execute system Groovy script' marked build as failure
Finished: FAILURE
<send an email or whatever here...>

Cheers,

Don

Oracle admin tasks

Oracle admin tasks – here are some basic queries and script examples I have gathered and adapted from various sources – the Internet, colleagues etc.

GitHub repo: https://github.com/DonaldSimpson/oracle_scripts

My main interest in this is in doing both day to day maintenance tasks to support environments, and in scripting monitoring and preventative Jenkins jobs that report on various aspects of Oracle Database servers – these automated database monitors have proved very worthwhile, and often identify upcoming issues before they cause problems (e.g. expiring users, table spaces filling up, disabled constraints and triggers, etc etc).

 

Find and kill sessions:

Connect:
sqlplus / as sysdba

Set the line size so things look better:
set linesize 999

Then run a query to show active users:
SELECT s.osuser, s.status, s.process, s.machine, s.inst_id, s.sid, s.serial#, p.spid, s.username, s.program FROM gv$session s JOIN gv$process p ON p.addr = s.paddr AND p.inst_id = s.inst_id WHERE s.type != ‘BACKGROUND’;

If you want to kill one, use the SID and SERIAL from the above:

ALTER SYSTEM KILL SESSION ‘{SID},{SERIAL}’;

———————–

Tablespaces; finding, resizing and autoextending:

sqlplus / as sysdba
set linesize 999

List Oracle Database tablespace files:

SELECT FILE_NAME as FNAME, TABLESPACE_NAME as TSPACE,BYTES, AUTOEXTENSIBLE as AUTOEX, MAXBYTES as MAXB,INCREMENT_BY as INC FROM DBA_DATA_FILES;

From the above, get the file name for the Table Space that needs altered, and do something like this:

ALTER DATABASE DATAFILE ‘{/path to above TS file, eg /ora/path/undotbs_0001.dbf}’ AUTOEXTEND ON NEXT 64m MAXSIZE 2G;

———————–

Start and Stop things:

Listeners…

lsnrctl start listener_Name
lsnrctl stop listener_Name

databases…
sqlplus / as sysdba
startup
shutdown immediate

———————–

Find invalid objects:

Optionally filtered by owner(s) and without synonyms…

select owner || ‘.’ || object_name || ‘[‘ || object_type || ‘]’
from dba_objects
where status = ‘INVALID’
and object_type != ‘SYNONYM’
and owner in (‘SYSTEM’,’SYS’,’TOOLS’,’DEVUSER’);

———————–

Check Constraints and Triggers:

SELECT * FROM all_constraints WHERE status <> ‘ENABLED’;

or filter by users:

SELECT * FROM all_constraints WHERE owner = ‘ARBOR’ and status <> ‘ENABLED’;

Triggers are similar:

select * from all_triggers where status <> ‘ENABLED’;

———————–

Check for locked/locking users:

those already locked:

select * from dba_users where username in (‘SYSTEM’,’SYS’,’TOOLS’,’DEVUSER’) and lock_date is not null;

or those about to be locked (I add this to my Jenkins Database monitoring jobs so you get some warning…):

select * from dba_users where expiry_date < (trunc(SYSDATE) +7) and lock_date = null;

———————–

Check the Oracle Wallet:

Check to see if encrytion is present:

select * from dba_encrypted_columns;

if that brings something back, then you can check the state of the Oracle Wallet:

SELECT status from v$encryption_wallet where status not like ‘OPEN’;

———————–

Running SQL scripts from Shell scripts:

This can be done in various ways, but I tend to either use this approach to simply run a file and exit:

echo “About to run ${SCRIPT_NAME} on ${SERVER}…”
echo exit | sqlplus ${DB_USER}/${DB_PASSWORD}@${SERVER} @/path/to/sql/scripts/${SCRIPT_NAME}.sql
echo “Script ${SCRIPT_NAME} complete.” # now check the return code etc…

or sometimes a HEREDOC is more suitable, something like this example for checking database links work:

echo “Checking ${DBLINK} link for user ${DB_USER}…”
DBLINK_CHECK=$(sqlplus -s -l ${DB_USER}/${DB_PASS}@${ADM_DBASE}<<EOF
set echo off heading off feedback off
SELECT ‘Link works’ from dual@${DBLINK};
exit;
EOF)
if [ $? -ne 0 ]
then
echo “ERROR: Checking link ${DBLINK} as ${DB_USER} FAILED”
fi

———————–

If you find any of these useful or would like to suggest additions or changes please let me know.

Cheers,

Don

Getting the current user in Jenkins

Updated! Yet Again!

There are several ideas and comments below on ways to get the logged in user in Jenkins – the person/user that triggered the current build.

This has changed/broken/altered quite a few times since I wrote the initial post (about 8 years ago) as changes have been made to Jenkins – not least being the introduction of Pipelines and Groovy! For quite some time the best solution was, probably, to use a plugin for this (see “build user vars” below).

If you’re writing a Jenkins Pipeline and you’re happy adding in the Build User Vars plugin, then you can simply do this to access the BUILD_USER variable in your Jenkins groovy Pipeline job:

wrap([$class: 'BuildUser']) {
  echo "BUILD_USER that started this Pipeline: ${BUILD_USER}"
}

A few alternative and older ideas follow…

I recently wanted (again) to get the logged-in user that triggered the build in Jenkins and came across my own post when searching… doh. I didn’t really feel like adding a plugin just for this one – supposedly simple – task so had another look at ways to achieve this, and this latest approach seems to work well…

It’s not pretty or robust, and I’m doing it in two goes (to demonstrate getting the correct line then refining things with sed to just the name) which you could simplify, but this is hopefully enough to illustrate the general plan. You’d need to validate it for builds that are started by timer and things like that too.

Here is the plain text version:

# Just show the output:
set -
# change dir to where this builds files are kept:
cd /home/don/jenkins/home/jobs/GetJenkinsJobUserName/builds/${BUILD_NUMBER}
# To start, just get the whole string from the 'log' file:
export STARTED_BY="`grep -i Started log`"
# Output it to the console:
echo "STARTED BY USER = ${STARTED_BY}"
# refine this to just the user name:
export JUST_NAME="`echo "${STARTED_BY}" | sed "s@Started by user@@"`"
echo "Jenkins User Name is ${JUST_NAME}"

This produces the following output when run:

That seems to do the job!

All the original thoughts and comments are below….

Here is a bit of code for getting the details of the logged-in user in Jenkins or Hudson – useful for auditing which user started a given build. I often add this to an audit database, that over time builds up a log of who has done what, where and when, as well as indicating the current state – I then query that through a JSP page that generates a dynamic table that is included via an iFrame in the Description part of the corresponding job… but anyway, back on topic…

Update: the Build User Vars plugin can now provide this functionality. Just install that then check the “Set jenkins user build variables” checkbox under the “Build Environment” section in the configuration page for the job you want to use it, then you should be able to access the following variables in your job:

  • BUILD_USER — full name of user started build,
  • BUILD_USER_FIRST_NAME — first name of user started build,
  • BUILD_USER_LAST_NAME — last name of user started build,
  • BUILD_USER_ID — id of user started build.

As far as I know, there’s no built-in way to do this (apart from the plugin now mentioned above). e.g. through a Jenkins environment variable like $CURRENT_USER or $USER_ID, but using wget, the Jenkins XML API and a little bit of XPath, you can easily query for many things at runtime like so:

THIS_USER=”$(wget -nv –no-proxy “$BUILD_URL/api/xml”‘?xpath=//userId/text()’ -O -)”

That line fetches the logged-on user id by requesting (via wget) the XML for the current BUILD_URL, filtering for the userId text and assigning that value to the variable $THIS_USER, so that it can be used elsewhere in the job.

You can change this around to get the username if you prefer:

THIS_USER=”$(wget -nv –no-proxy “$BUILD_URL/api/xml”‘?xpath=//userName/text()’ -O -)”

Or, try adding “/api/xml” to the end of a Jenkins URL, and you can see the numerous other elements you could request via the Jenkins XML API and XPath.

I have often used this approach to query and report on the status of multiple downstream jobs, for example. Sometimes it’s easier to wget the XML then parse it locally at runtime (with sed/awk/whatever you like) for the data you are interested in – the number of passed, failed or unstable builds can be counted by searching for the number of corresponding “blue”, “red” or “yellow” elements, for example.

UPDATE: It looks like the current version of Jenkins has disabled or broken this feature/ability, but there are always alternative solutions…

For example, if you have curl and xml_grep to hand, you could do it like this…

THIS_USER=`curl –silent $BUILD_URL | xml_grep –text_only userName`

most systems will have curl or alternatively wget, and if you don’t have xml_grep some plain old grep or awk will do the job.

Cheers,

Don

Solaris 10 on ESXi5, with SunRay Server and Clients

About 10 years ago, I bought a SunRay 1g Client box with the plan of setting it up as a thin client to the (physical and very noisy) Sun Solaris Server I was running at the time. The old Sun server sadly didn’t last much longer, and I replaced it with a FreeBSD one before I managed to get the SunRay up and running… so it sat in a cupboard until recently.

The new plan is to use the SunRay 1g as a thin client to a Solaris VM running on my ESXi5 server. This seems like a good idea as it gives me a way to access and manage the ESX server without having to fire up another machine – plus, I think SunRay clients are awesome bits of kit – if I didn’t have an iMac I would use these, as they are silent, boot instantly, use hardly any power and make no noise… it seems like the ideal compliment to an ESX server.

Here’s a pic of my SunRay client…

So, here are some notes, links and pics on installing an Oracle Solaris 10 Virtual Machine on ESXi5, setting up the SunRay Server Software, and getting the SunRay client (finally) working…

From the Oracle site, Sun Ray Clients are:

“… simple, low-cost devices that are ideal for displaying server-hosted virtual desktops. With no moving parts and no local operating system to manage, Sun Ray Clients provide a cost-effective, highly functional thin client alternative to desktop and laptop computers and reduce many of the problems associated with traditional desktop deployments.”

http://www.oracle.com/us/technologies/virtualization/061984.html

They need a Server running the SunRay Server Software to connect to – I think you can use Linux these days but I wanted to set up a Solaris host anyway, so I went with the traditional approach – albeit running on VM Ware.

None of this was very difficult, and the only slight issue I had was getting DHCP set up correctly; as I understand it, Sun Ray clients need to be sent additional information when they get an IP address, and that means it’s easier to set up the SunRay Server as your DHCP server – once I’d done that (and configured my Router to act as a DHCP Relay, handing DHCP requests over to the Solaris host), it all worked well.

Install Solaris 10 on VMWare ESXi5:

I downloaded the DVD image from Oracle, and then uploaded it to the data store on the ESX Server, created a new VM and set it to mount the image as a DVD drive at boot time. The spec of my Solaris 10 VM is 8GB RAM, 2x vCPU and 500G disk for now – I will change this later when I see how it performs.

Install VM Ware tools:
mkdir / vmwaretools/

cp /cdrom/vmwaretools/vmware-solaris-tools.tar.gz .

# tar zxvf vmware-solaris-tools.tar.gz – that didn’t work – no GNU tar :-(

gunzip vmware-solaris-tools.tar.gz

tar xvf vmware-solaris-tools.tar

cd vmware-tools-distrib/

./vmware-install.pl

All done, much better 🙂

Setup  DHCP & NIS on Solaris 10

This command starts up the (GUI) DHCP Manager wizardy thing, which will prompt you for all required info to set up DHCP:
/usr/sadm/admin/bin/dhcpmgr &

As as I understand it, your SunRay clients will look for a DHCP server to (also) give them the details of the SunRay Server to connect to. I coudln’t see a way to use my existing DHCP server (my broadband router) and still be able to tell the clients how to connect to the Solaris Server – that would have been preferable for me.

cp /etc/nsswitch.dns /etc/nsswitch.conf

add the google DNS Server and the internal IP of my router to /etc/resolv.conf….

echo "nameserver 8.8.8.8" >> /etc/resolv.conf
echo "nameserver 192.168.0.1" >> /etc/resolv.conf

Install SunRay Server Software (SRSS)

This was all pretty simple, my notes are brief but there’s a decent guide here:

http://docs.oracle.com/cd/E22662_01/E22659/html/Installing-Howto-Install.html

I downloaded the installer and extracted it to a tar file on another host beforehand, which was handy as I hadn’t sorted out GNU tar yet…

mkdir /sunrayinstall

cd /sunrayinstall

scp don@myubuntuserver:/tmp/V32065-01.zip .

unzip V32065-01.zip

cd srs_5.3.0.0-SunOS.i386/

./utinstall

export SRSS_INST=/opt/SUNWut

# Enable LAN Access:

$SRSS_INST/sbin/utadm -L on

$SRSS_INST/sbin/utrestart

Things to look at later:

connector for vmware: http://docs.oracle.com/cd/E21907_01/PDF/EN/SRVMWC1dot2InstallGuide.pdf

Here are some of the other commands and checks I looked at while getting it running…

/opt/SUNWut/sbin/utdesktop

/opt/SUNWut/sbin/utgstatus

/opt/SUNWut/sbin/utquery -d 192.168.0.58

/opt/SUNWut/sbin/utsetup

/opt/SUNWut/sbin/utconfig

tail -1000f /var/opt/SUNWut/log/messages
Install Apache Tomcat for SunRay Server Software

I’m not sure if this is necessary?

cd /opt/apache-tomcat

cd bin

export JRE_HOME=/usr

./catalina.sh start
Setup NTP on Solaris 10
ls -al /etc/inet/ntp.*

touch /var/ntp/ntp.drift

cp /etc/inet/ntp.client /etc/inet/ntp.conf

svcadm enable svc:/network/ntp

date

Here’s a picture of the new Admin Tool – I only have 2 SunRay clients to manage (and even fewer users!), but I may still have to search eBay for some Smart Cards…

Update: I now have an unopened pack of smart cards (50 I think?) – a gift from a Solaris admin chum!

http://docs.oracle.com/cd/E19214-01/820-0412/new_admin_tool.html

Once that was all done and my Router was configured to relay DHCP requests to the DHCP Server running on the Solaris 10 VM, I was able to switch on my SunRay 1g client and be presented with a logon screen – yay!

And finally, here’s a pic of the SunRay client sitting on top of the HP ProLiant ML110 G6 ESXi5 home lab server, which is running the Solaris 10 Virtual Machine it’s connecting to…

New Home/lab ESXi 5 Server – Part 1

I wrote a while ago about my plans to set up a home VM Ware ESXi 5 server… and although you can’t tell the difference, this site is now happily running on it 🙂

Spec:

The server I went for is an HP ProLiant ML110 G6. It’s got a single socket Xeon Quad Core X3430 processor, 16GB of RAM and at the moment 2TB of SATA3 disk – I will add more when I finish migrating data off the old servers.

There are some limits on the spec you can use with the free version of ESX – I think it’s currently one physical processor and 32G RAM, which means this server is fine.

Install and setup:

The ML110 is a compact and well-made server and is very quiet when up and running – it sounds like a hovercraft for the first few seconds, but quickly calms down to run not much louder than a normal desktop PC. The chassis isn’t large but it’s well laid out and there’s room for 4 HDD’s in there, maybe more if required if the DVD drive was removed (it’s not on the ESX compatibility list anyway so I can’t use it for this).

I installed ESXi 5 from a 2GB USB drive which is attached to the USB socket directly on the motherboard – the BIOS boots from this no problem, and gave me the option to install ESXi 5 on to the USB drive, leaving my HDD(‘s) free to use as 100% dedicated ESX Datastores, and also meaning I can exchange them when I need to without having to reinstall or worry about the VM Ware OS.

The server-side installation of ESXi 5 is a breeze – I rehearsed and wrote about it on my ESXi 5 on an iMac under Fusion post  and it was no different here – it took about 10 minutes or so and there were no tricky questions. Specify the user name and password, tell it to use all of the available HDD and there’s not much more to it. Once you have set up networking the way you want it (the DHCP setup should be fine for most installs) you don’t really need to go near the server again – it’s all managed remotely via the client applications and SSH (more on that in Part 2) from then on.

Creating VMs:

Once you have the vSphere client install done (it’s far more trouble than the server install – plus it requires .Not and J# runtimes – argh! – so it requires Windows – double-argh! – so had to be done initially on a Windows VM running on my iMac in my case… triple-argh!) you can connect to your ESXi 5 server using the IP address, User Name and Password that you set up and start creating some VMs.

To get the first VM created (in my case this had to be a Windows one that I could then use to run the vSphere client on and RDP over to instead of having to run a VM on my Mac all the time), I uploaded an ISO image to the datastore that I had created, then added a new VM in the vSphere Client and set that ISO as the CD image it should load at boot time. You just specify the OS type, RAM, CPU, Networking and disk(s) you want and power it on – all very easy and quick.

Converting a VM Ware Workstation VM in to a VM Ware ESX Guest

I also wanted to convert the Ubuntu VM Ware Workstation image that this web site runs on, so I could move it off the old server and have it running on ESX as a “proper” ESX guest/host. This was really easy too; the VM Ware Converter allows you to specify a local VM of pretty much any type and supply the details of the target ESX server, and it then converts and loads it all for you – it took quite a while to complete but it worked without issues, and I was then able to power on my Ubuntu website VM under ESXi 5, where it’s happily running right now. No need to reinstall WordPress, Postfix, PHP, Java, Jenkins, MySQL etc etc – happy days.

Here are some Pics of the ESXi5 console shortly after set-up…

 

1. General info on the reported spec and current overall resource usage of the HP ProLiant ML110 G6:

ESXi 5 console general information

 

2. Some of the Health Monitoring and General Configuration options:

Inventory and general settings and diagnostics pic

3. Overview of Guest and Host resources:

Guest and Host resources

 

Summary:

I did a fair bit of research beforehand to make sure the install and hardware would be ok, which meant the actual set up was trivial – once it’s done all you need to do is create VMs and allocate resources; there is very little work or maintenance required – especially compared to what would be needed to run multiple physical servers all with their own hardware. Creating new VMs is very easy, and the performance is good so far – the processor is not stressed at all, and the ESX memory management does a good job – I’ve had up to 6 VMs running at the same time and still have about half the memory free!

Next plans:

One of my main reasons for doing this was to provide a test platform for automating, creating and managing Linux VMs using Jenkins as a front end and DNS records to control what is deployed where and when – I want to be able to select a few options, then click a button and have my new host created in minutes and to the right spec, similar to the Amazon EC2 set up but code deployment linked in too, and I will write more on this when I’ve done it.

Plans include a mixture of: VM Ware Templates, Perl, Jenkins jobs, Jenkins Nodes, Puppet, Tomcat, etc

Next Post:

There are a few other things I have already done that I’d like to document too including…

Accessing ESXi 5 via SSH – how to and a summary of useful commands etc
More detail on Remote desktop via ssh tunnels etc
VM Ware command line tools
DNS and AD/LDAP servers

Managing Jenkins as a service and starting at boot time

Linux Services

I have an Ubuntu Linux VM that runs Jenkins, and to make life simpler I wanted to set up Jenkins to run as a service. I also wanted it to start up automatically at boot time, as that’s the sole function of the VM it’s running on.

So, here are my notes on setting up Jenkins as a service on Ubuntu Linux, which includes a script to manage (start, stop, check, restart) the Jenkins process too.

On Ubuntu (and most othe rLinux versions) you can check the current services with the command “service –status-all” – this should give you a list of all services, and their current status. You can (as root) also do “service <name> start” (or stop or restart) for each one.

Jenkins as a Service

To create a new service for Jenkins, take a look at the existing scripts in /etc/init.d/ for some examples.

By convention there are three main methods a basic service implements, they are; start, stop and restart. About the most basic structure of a service script is therfore something like this:

#!/bin/bash
start() {
echo “Starting Service”
# Do start things here.
}stop() {
echo “Stopping Service”
# Do stop things here.
}restart() {
echo “Restarting Service…”
stop
start
}

case “$1″ in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)

obviously you still need to implement the start and stop functions though 🙂

To see a far more detailed example, take a look at the file /etc/init.d/skeleton, which you could copy and update to suit your needs if you prefer.

I decided on something halfway between the very basic example above and the far more complex example in “skeleton”, and did something like this (saved as “/etc/init.d/jenkins”):

# Example Jenkins auto-start service script
#
# description: manages Jenkins as a service
# processname: jenkins
# pidfile: /var/run/jenkins.pid
# author: www.DonaldSimpson.co.uk# The user and the home dir that Jenkins runs under
jenkins=/usr/local/jenkins
# Your startup and stop scripts (see below)
startup=$jenkins/bin/startup.sh
shutdwn=$jenkins/bin/shutdown.shstart_jenkins() {
echo “Starting Jenkins services…”
su – jenkins -c “sh $startup”
}

stop_jenkins() {
echo “Stopping Jenkins services…”
su – jenkins -c “sh $shutdwn”
}

status_jenkins() {
# Check for any other process containing jenkins.war
# This could be improved upon (see script below)
numproc=`ps -ef | grep [j]enkins.war | wc -l`
if [ $numproc -ne 0 ]; then
echo “Jenkins is running…”
else
echo “Jenkins is NOT running…”
fi
}

case “$1” in
start)
start_jenkins
;;
stop)
stop_jenkins
;;
status)
status_jenkins
;;
restart)
stop_jenkins
start_jenkins
;;
*)
echo “Usage: $0 {start|stop|status|restart}”
exit 1
esac
exit 0

Update that to suit then save and change the permissions to make it executable:

chmod +x /etc/init.d/jenkins

then you can check (as root) that you can call the methods in the script:

service jenkins status

service jenkins stop

service jenkins start

These should all now run as the Jenkins user via sudo and say something when called, even if it is just “I can’t find the scripts you told me to call” 🙂

Jenkins Startup and Stop scripts

So, you now need to create the missing startup and shutdown scripts, in my example they were:

/usr/local/jenkins/bin/startup.sh

and

/usr/local/jenkins/bin/shutdown.sh

An example of the Jenkins start-up and management script I normally use is at the end of this post – the main idea though, is that it sets what you want to set then starts Jenkins via nohup and outputs to a log file

I have also added and included some basic tests to my scripts and some (very) rudimentary error handling/checking, but you shoudl get the idea and all you really need is this line (with the variables set correctly):

${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &

Note: I often run multiple instances of Jenkins, so I explicitly specify the HTTP_PORT, and I use the -D${MARKER} to allow me to safely and easily find the correct PID for each project.
The httpListenAddress can normally go unset, but it’s something I’ve had to set before on multi-home’d hosts.

The stop part of my Jenkins management script finds the correct PID like this (and you could use a filter for the correct -D${MARKER} if you want that too):

PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`

and simply kills it.

For some of my scripts I also filter for process ID’s that were started in the current directory by checking “pwdx” against the PID, but only where I’m sure the corresponding start-up process is correct/reliable.

So, tweak to your taste and that should be the Jenkins service created and working too now; for start|stop|restart you can create individual scripts or wrappers that call something like the one script below with parameters, or break the script in to separate files if you prefer.

Setting Jenkins to start at boot time

If you want to start Jenkins at boot time/startup automatically then you still need to do one more small step.  There are many different ways to do this depending on personal preference, your requirements and your version of Linux, but on Ubuntu it can be done easily with:

update-rc.d jenkins defaults

“update-rc.d” simply installs and removes System-V style init script links – read the man page for full details, but the same idea applies to most versions of Linux.

update-rc.d -f  jenkins remove

will undo this if you no longer want it.

An example Jenkins server/process management script

As an example, here is my script to manage Jenkins processes – it could use some improvements but the basic start, stop, restart and check should give you enough to sort out something that works and suits your needs.

Better formatted version available here: https://github.com/DonaldSimpson/scripts/blob/master/JenkinsProcessManager.sh

#! /bin/bash -p
# www.donaldsimpson.co.uk
# Script to start|stop|restart|check an instance of Jenkins
# For each new instance, the PROJECT and HTTP_PORT need to be updated:
export PROJECT=jenkins
export HTTP_PORT=9000
EXIT_STRING=””
JENKINS_ROOT=/opt/apps/jenkins
export JENKINS_HOME=${JENKINS_ROOT}/${PROJECT}
JENKINS_WAR=${JENKINS_HOME}/jenkins.warWAIT_TIME=5
START_WAIT_TIME=15
JAVA_HOME=/usr
PATH=$JAVA_HOME/bin:$PATH
JAVA=${JAVA_HOME}/bin/java
NOHUP=/usr/bin/nohup
LOG_FILE=${JENKINS_HOME}/debug_${PROJECT}.log
MARKER=”JenkinsProcFor_${PROJECT}”NC=/bin/nc
WGET=/usr/bin/wget
LSOF=/usr/bin/lsof
AWK=/usr/bin/awk
GREP=/bin/grep
FUSER=/bin/fuser
#################################################################################
### Functions Start #############################################################
#################################################################################

cleanup(){
# Perform any clean-up activities here…
[ “${EXIT_STRING}” != “0” ] && echo `date “+%d/%m/%y %H:%M:%S”` ${EXIT_STRING}
exit 0
}

trap ” QUIT

_error() {
EXIT_STRING=”${1}”
[ “${1}” != “0” ] && EXIT_STRING=”ERROR: ${1}, please investigate, terminating.”
cleanup  # Never returns from this call…
}

say(){
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
}

saybold(){
tput bold 2>/dev/null
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
tput sgr0 2>/dev/null
}

check_folders(){
say “”
saybold “Checking all required folders exist…”
for REQUIRED_DIR in ${JENKINS_ROOT} ${JENKINS_HOME} ${JAVA_HOME}
do
[ ! -d “${REQUIRED_DIR}” ] && _error “Necessary directory: ${REQUIRED_DIR} does not exist”
say “Found required directory: ${REQUIRED_DIR}”
done
saybold “Done.”
say “”
}

check_files(){
say “”
saybold “Checking all required files exist…”
for REQUIRED_FILE in ${JENKINS_WAR} ${NC} ${WGET} ${JAVA} ${LSOF} ${FUSER} ${GREP} ${AWK}
do
[ ! -f “${REQUIRED_FILE}” ] && _error “Necessary file: ${REQUIRED_file} does not exist”
say “Found required file ${REQUIRED_FILE}”
done
saybold “Done.”
say “”
}

check_port_closed(){
say “Checking that port ${HTTP_PORT} is closed…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
_error “Required Jenkins port ${HTTP_PORT} is already in use”
else
say “Ok, required port ${HTTP_PORT} is available, continuing…”
fi
}

check_port_open(){
say “Checking that port ${HTTP_PORT} is open…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
say “Ok, a process is listening on port ${HTTP_PORT}, continuing…”
else
_error “Required Jenkins port ${HTTP_PORT} has not been opened.”
fi
}

start_process(){
cd ${JENKINS_HOME}
saybold “Starting Process now…”
${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &
say “Process initiated.”
}

check_log(){
say “Checking the log file ${LOG_FILE} for an HTTP listener…”
STARTED=`${GREP} -c “HTTP Listener started” ${LOG_FILE}`
if [ ${STARTED} -eq “0” ]; then
_error “An HTTP Listener has not reported as started in the log file ${LOG_FILE}”
else
saybold “An HTTP Listener is reported as started in the log file ${LOG_FILE}”
fi
}

check_html(){
# These checks need error handling, but you get the general idea.
say “Checking that localhost:${HTTP_PORT} is serving Jenkins pages…”
TEMP_WGETDIR=tempwgetdir$$
mkdir ${TEMP_WGETDIR}
cd ${TEMP_WGETDIR}
${WGET} -q http://localhost:${HTTP_PORT}
GOT_HTML=`${GREP} -c Dashboard index.html`
cd ${JENKINS_HOME}
rm -rf ${TEMP_WGETDIR}
if [ ${GOT_HTML} -eq “0” ]; then
_error “Unable to get an HTML page from the Server.”
fi
saybold “Recieved valid HTML from the server, all looks ok.”
}

check_process(){
check_port_open
check_log
check_html
say “A Jenkins instance is listening on port ${HTTP_PORT} for project ${PROJECT}.”
say “The process is logging debug info to the log file: ${LOG_FILE}”
}

stop_proc(){
check_port_open
saybold “Looking for the Process ID attached to port ${HTTP_PORT}”
PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`
if [ “${PID}” == “” ]; then
saybold “Unable to detect the Process ID that is listening on port ${HTTP_PORT}!”
PID=`${FUSER} ${LOG_FILE}`
if [ “${PID}” == “” ]; then
_error “Unable to find the PID that has the log file open too!”
else
say “Ok, found PID ${PID}”
fi
else
saybold “Found a PID of $PID, killing it now…”
fi
kill -9 ${PID}
say “Waiting ${WAIT_TIME} seconds for the process to die…”
sleep ${WAIT_TIME}
saybold “Done, checking port is closed…”
check_port_closed
saybold “All done.”
}

start(){
check_folders
check_files
check_port_closed
start_process
saybold “Waiting ${START_WAIT_TIME} seconds for the process to start up…”
sleep ${START_WAIT_TIME}
check_process
}

restart(){
stop_proc
start
}

#################################################################################
### Script Start ################################################################
#################################################################################

case “$1” in
start)
start
;;
stop)
stop_proc
;;
restart)
restart
;;
check)
check_process
;;
*)
echo “Usage: $0 {start|stop|restart|check}”
esac
# Exit cleanly
_error “0”

Hope that helps! Any constructive comments, requests or suggestions for improvement are very welcome 🙂

Cheers, and sorry about the indenting,

Don

VM Ware ESXi under VM Ware Fusion

Hardware

I’m looking at getting a new home lab/server set up, and am seriously considering ordering one of these:

HP ProLiant ML110 G6 Quad Core X3430

It will need filled with memory and a load of disk space, but I reckon that’s still a whole lot of server for the money.

The Plan

The plan is to install VM Ware ESXi 5 on to it (using the USB Drive), and manage the server through the remote VM Ware vSphere Client app. This will allow me to create new Virtual Machines and migrate my existing appliances over to this server, then I can retire the old servers these have been running on. I’m also wanting to develop some automation processes for managing VM’s – creating new ones, bringing them up and down etc using Jenkins to orchestrate the processes, so this will allow me to work on that too.

Before ordering the ML110 I wanted to take a better look at the installation of ESXi – it doesn’t sound difficult, but while I’ve used it often I’d never set it up before so wanted to see what was involved.

VM Ware ESXi 5 Installation under Fusion on an iMac

It’s a little bit crazy and recursive, but, I realised I may be able to do this on my iMac under VM Ware Fusion, which it turns out does allow you to install ESX as a Virtual Machine itself…. which you can then use to manage and create new (Virtual?) Virtual Machines – a bit of a brain-ouch, and it’s clearly not going to be fast, but it’s good enough for my testing.

Installing ESXi is very straightforward – I selected the obvious option of “VM Ware ESX” in Fusion on my iMac, told it I had an image I wanted to use and pointed it at the VM Ware ESXi 5 ISO image I had downloaded from VM Ware (you need to register then fumble about their site for a while to find and download the free version – that’s the way I did it anyway). Keep a note of the serial number they give you, as that will remove the 60 day trial restriction later.

Fusion suggested a 40G file system for this instance and allocated 2GB RAM and 2 cores, which I was quite happy to run with for my test. Speaking of RAM – there was a restriction on the amount of physical RAM you could use with the free version of ESXi 5 – it was 8GB a while ago but this has now been increased to something more sensible – 32GB I think?

There are no surprises or major decision needed during the install of ESXi 5 itself – it took all of 5 minutes to run through and reboot, and that was with it running as a Virtual Machine on an overloaded iMac – on proper hardware like the HP ProLiant ML110 G6 Quad Core X3430 that I’m looking at, it would be loads faster.

When the install is done, there’s nothing more to see on the ESXi (VM), apart from the HTTP address it gives you to connect to the ESXi Host and access a simple web page it serves with links to download the VM Ware vSphere Client application to another host and start managing your server. This address was given by DHCP in my case – I think you can specify or change this easily if you want to.

Client Installation

Downloading the VM Ware vSphere Client took a while (longer than installing ESXi did!) as it came from the VM Ware site rather than being directly served by the ESXi host. Now for the bad news… for clients, you have a choice of running either Windows or Linux. No mention of an OSX vSphere client, so I had to fire up a Windows VM just to set up the client app on… not what I had been hoping for – there’s a petition asking VM Ware to sort this out here:

http://communities.vmware.com/thread/128538?start=525&tstart=0

When I get things running I could create a VM on my ESXi host which I can RDP on to, but that’s still a pretty ugly solution – if the Linux client is ok I’d go with that over the Windows one, and I think there’s also a Web Interface. But, part of the reason for me doing this in the first place is so that I can look in to the SDK and API’s for automating the creation of VM’s with VM Ware using Jenkins though, so I’ll grin and bear the Windows yuckness and see how things go.

Installing the vSphere client gets worse and worse though – my VM needed an update to its Microsoft .Not Framework (something I tend to avoid) which churned away for quite some time, and the console looks to be written in J# too (yes seriously – J# – what the… ?), which meant another “Framework upgrade” which took another while and a half – so the client set up ended up taking about 10 times longer than the server, and I had to run a Windows VM just for it… not too cool.

Once done, I could point my vSphere client at the IP address of the ESXi (VM) using its advertised IP address, the default user name (which I’d forgotten to take a note of – it’s “root”) and the password I’d specified during the install. This gets me to the Hypervisor where I can start creating and managing my own VM’s.

Cool stuff, despite the client letting things down.

Easy method to tunnel VNC over an SSH connection

This has been done many times in many different ways, but I’ve finally found a workable solution that suits my needs and my setup, and (perhaps most importantly when it comes to ports and tunneling…) that I can get my head around 🙂

 

So… at home, I have a Windows machine with remote desktop enabled, but I don’t want to open up yet another port on my router or firewall (or to expose a Windows machine to the world) for the few times that I want to get a remote desktop connection to it.

But, I have a web-facing Ubuntu Linux host which I use frequently to access the non-Windows parts of my Home Network – another Linux host, an occasionally used FreeBSD VM, Solaris 10 server, a couple of iMac’s and my ATV2.

I can see the Windows file system ok by ssh’ing in to the Linux box (it’s file system is shared and mounted) but I can’t see or access the Windows desktop when I’m outside of my home network. Doing so is handy for access to the VM Ware vClient to manage my ESX Server, etc.

A solution for this is to tunnel the VNC protocol from my remote host to the Internal Windows server via the (already exposed) SSH connection on my web-facing Linux host. To do this from outside my home, I kick off an SSH session like this:

ssh -L 5900:192.168.0.111:5900 myuser@www.donaldsimpson.co.uk

that command breaks down like this:

ssh using the -L arg ([-L [bind_address:]port:host:hostport]), which forwards the local port 5900 to 192.168.0.111, and attaches the tunnel on its port 5900 (the port that RDP or VNC is running on).

Note that the IP address 192.168.0.111 is my “internal” Windows server with a non-routable IP address – it’s accessible to/from my Linux Host (www.donaldsimpson.co.uk), but not from anywhere outside my network.

After that, I specify the SSH logon details for my Linux host and when that connection is made I’m prompted for the password for “myuser@www.donaldsimpson.co.uk“. When that is accepted I am logged on to my Linux box at the console (Cygwin with SSH installed in this case) as per usual, and I’ll need to leave this terminal open. The SSH Tunnel has been created and will end when this session does.

The next and final step is to launch “mstsc” (or VNC or another RDP Client) on my local host and create a connection – not connecting to www.donaldsimpson.co.uk, but to 127.0.0.1:5566 – this is the local end of the tunneled connection; at the other end, via my SSH Host, is the listening port on the Windows host. After supplying the requested credentials, it makes the connection, then finally shows me my Windows Desktop – woohoo, job done.

Setting up Perforce on a Linux Server and a Windows Client

As described on WikiPedia, Perforce “is a commercial, proprietary, centralized revision control system developed by Perforce Software, Inc.”

Like Subversion, it’s a client/server system where the server manages a database of revisioned files, and clients connect to checkout, modify and send back changes for others to pick up.

I wanted to check out the latest version so thought I’d install it on this server and set up a client… and that I may as well capture the steps and put them here.

In my case, the server is an Ubuntu Linux  host, and my client machine is a Windows XP workstation.

There’s not a lot to do when installing Perforce, so getting a straightforward instance up and running is a breeze. Basically, get the binary, export or pass it a few settings if you don’t want the defaults, then kick it off – here’s the detail from my notes:

Download p4d binary (for this platform and architecture):
wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4d

You can put this wherever you want, and set up a P4ROOT to specify the Perforce root directory – but don’t use that location for anything else (including client workspaces).

mkdir /apps/perforce
mv p4d /apps/perforce; cd /apps/perforce
chmod +x p4d

 

Most  Perforce options can either be exported or passed as a command-line arg, so you can choose

export P4ROOT=/apps/perforce
– or –
-r /apps/perforce

The default port is 1666, and remember that if you change this on the Server you will need to change it on your Perforce client(s) too.  In my example I’m using 9002:

export P4PORT=9002
– or –
-p 9002

So I ended up with a command line that looked like this:

nohup ./p4d -r /apps/perforce -J /var/log/journal -L /var/log/p4err -p 9002 &

Which I will probably put in to a simple startPerforce.sh script, and a probably a stopPerforce.sh script too that contains this and maybe the port number and full path to the binary location:

p4 admin stop

That’s it for the Server side at the moment, there’s a process up and running (you can check the output in nohup.out) so it’s time to set up and connect a client…

 

I’m going for a Windows client installation, which means downloading the correct version from the Perforce site then running p4vinst.exe. There’s nothing really to report here; select the usual options like directories and let it install.

Oh, I also needed to create a user, so back on the Linux Server I also downloaded the p4 binary:

wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4

Exported the P4PORT (since I was using a custom one to get through my firewalls), then add a user:

export P4PORT=9002
chmod +x p4
./p4 user -f Donald
User Donald saved.

Now you can test connecting to your Perforce Server with the  P4Admin GUI and the P4V GUI on the Windows client host by specifying the correct port (if you changed the default) and a valid user name. Once that’s done, you can admin your Depot’s and add/change/commit files, see revision and history information and all that good stuff. There are also command line and web interfaces too which can be useful for temporary use and for scripting but the Windows GUI’s are nice to use and quite self explanatory – if you’ve used a similar revision control system like Subversion and an Eclipse-like IDE before there’s not much of a learning curve here.

The Perforce Help and Documentation is all very good, and their Perforce 2010.2: System Administrator’s Guide covers all of the above in more detail, and touches on more advanced topics too – Perforce Performance Tuning, Backup and Recovery, Replication, and the Perforce Broker (P4Broker) etc.

There’s also a Perforce plugin for Jenkins, which once installed allows you to choose Perforce as one of the SCM option in your Jenkins jobs, but the above hopefully covers the initial setup of both the Perforce Client and the Perforce Server on Windows and Linux respectively.

 

Pin It on Pinterest

%d bloggers like this: