New Home/lab ESXi 5 Server – Part 1

I wrote a while ago about my plans to set up a home VM Ware ESXi 5 server… and although you can’t tell the difference, this site is now happily running on it 🙂

Spec:

The server I went for is an HP ProLiant ML110 G6. It’s got a single socket Xeon Quad Core X3430 processor, 16GB of RAM and at the moment 2TB of SATA3 disk – I will add more when I finish migrating data off the old servers.

There are some limits on the spec you can use with the free version of ESX – I think it’s currently one physical processor and 32G RAM, which means this server is fine.

Install and setup:

The ML110 is a compact and well-made server and is very quiet when up and running – it sounds like a hovercraft for the first few seconds, but quickly calms down to run not much louder than a normal desktop PC. The chassis isn’t large but it’s well laid out and there’s room for 4 HDD’s in there, maybe more if required if the DVD drive was removed (it’s not on the ESX compatibility list anyway so I can’t use it for this).

I installed ESXi 5 from a 2GB USB drive which is attached to the USB socket directly on the motherboard – the BIOS boots from this no problem, and gave me the option to install ESXi 5 on to the USB drive, leaving my HDD(‘s) free to use as 100% dedicated ESX Datastores, and also meaning I can exchange them when I need to without having to reinstall or worry about the VM Ware OS.

The server-side installation of ESXi 5 is a breeze – I rehearsed and wrote about it on my ESXi 5 on an iMac under Fusion post  and it was no different here – it took about 10 minutes or so and there were no tricky questions. Specify the user name and password, tell it to use all of the available HDD and there’s not much more to it. Once you have set up networking the way you want it (the DHCP setup should be fine for most installs) you don’t really need to go near the server again – it’s all managed remotely via the client applications and SSH (more on that in Part 2) from then on.

Creating VMs:

Once you have the vSphere client install done (it’s far more trouble than the server install – plus it requires .Not and J# runtimes – argh! – so it requires Windows – double-argh! – so had to be done initially on a Windows VM running on my iMac in my case… triple-argh!) you can connect to your ESXi 5 server using the IP address, User Name and Password that you set up and start creating some VMs.

To get the first VM created (in my case this had to be a Windows one that I could then use to run the vSphere client on and RDP over to instead of having to run a VM on my Mac all the time), I uploaded an ISO image to the datastore that I had created, then added a new VM in the vSphere Client and set that ISO as the CD image it should load at boot time. You just specify the OS type, RAM, CPU, Networking and disk(s) you want and power it on – all very easy and quick.

Converting a VM Ware Workstation VM in to a VM Ware ESX Guest

I also wanted to convert the Ubuntu VM Ware Workstation image that this web site runs on, so I could move it off the old server and have it running on ESX as a “proper” ESX guest/host. This was really easy too; the VM Ware Converter allows you to specify a local VM of pretty much any type and supply the details of the target ESX server, and it then converts and loads it all for you – it took quite a while to complete but it worked without issues, and I was then able to power on my Ubuntu website VM under ESXi 5, where it’s happily running right now. No need to reinstall WordPress, Postfix, PHP, Java, Jenkins, MySQL etc etc – happy days.

Here are some Pics of the ESXi5 console shortly after set-up…

 

1. General info on the reported spec and current overall resource usage of the HP ProLiant ML110 G6:

ESXi 5 console general information

 

2. Some of the Health Monitoring and General Configuration options:

Inventory and general settings and diagnostics pic

3. Overview of Guest and Host resources:

Guest and Host resources

 

Summary:

I did a fair bit of research beforehand to make sure the install and hardware would be ok, which meant the actual set up was trivial – once it’s done all you need to do is create VMs and allocate resources; there is very little work or maintenance required – especially compared to what would be needed to run multiple physical servers all with their own hardware. Creating new VMs is very easy, and the performance is good so far – the processor is not stressed at all, and the ESX memory management does a good job – I’ve had up to 6 VMs running at the same time and still have about half the memory free!

Next plans:

One of my main reasons for doing this was to provide a test platform for automating, creating and managing Linux VMs using Jenkins as a front end and DNS records to control what is deployed where and when – I want to be able to select a few options, then click a button and have my new host created in minutes and to the right spec, similar to the Amazon EC2 set up but code deployment linked in too, and I will write more on this when I’ve done it.

Plans include a mixture of: VM Ware Templates, Perl, Jenkins jobs, Jenkins Nodes, Puppet, Tomcat, etc

Next Post:

There are a few other things I have already done that I’d like to document too including…

Accessing ESXi 5 via SSH – how to and a summary of useful commands etc
More detail on Remote desktop via ssh tunnels etc
VM Ware command line tools
DNS and AD/LDAP servers

Managing Jenkins as a service and starting at boot time

Linux Services

I have an Ubuntu Linux VM that runs Jenkins, and to make life simpler I wanted to set up Jenkins to run as a service. I also wanted it to start up automatically at boot time, as that’s the sole function of the VM it’s running on.

So, here are my notes on setting up Jenkins as a service on Ubuntu Linux, which includes a script to manage (start, stop, check, restart) the Jenkins process too.

On Ubuntu (and most othe rLinux versions) you can check the current services with the command “service –status-all” – this should give you a list of all services, and their current status. You can (as root) also do “service <name> start” (or stop or restart) for each one.

Jenkins as a Service

To create a new service for Jenkins, take a look at the existing scripts in /etc/init.d/ for some examples.

By convention there are three main methods a basic service implements, they are; start, stop and restart. About the most basic structure of a service script is therfore something like this:

#!/bin/bash
start() {
echo “Starting Service”
# Do start things here.
}stop() {
echo “Stopping Service”
# Do stop things here.
}restart() {
echo “Restarting Service…”
stop
start
}

case “$1″ in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)

obviously you still need to implement the start and stop functions though 🙂

To see a far more detailed example, take a look at the file /etc/init.d/skeleton, which you could copy and update to suit your needs if you prefer.

I decided on something halfway between the very basic example above and the far more complex example in “skeleton”, and did something like this (saved as “/etc/init.d/jenkins”):

# Example Jenkins auto-start service script
#
# description: manages Jenkins as a service
# processname: jenkins
# pidfile: /var/run/jenkins.pid
# author: www.DonaldSimpson.co.uk# The user and the home dir that Jenkins runs under
jenkins=/usr/local/jenkins
# Your startup and stop scripts (see below)
startup=$jenkins/bin/startup.sh
shutdwn=$jenkins/bin/shutdown.shstart_jenkins() {
echo “Starting Jenkins services…”
su – jenkins -c “sh $startup”
}

stop_jenkins() {
echo “Stopping Jenkins services…”
su – jenkins -c “sh $shutdwn”
}

status_jenkins() {
# Check for any other process containing jenkins.war
# This could be improved upon (see script below)
numproc=`ps -ef | grep [j]enkins.war | wc -l`
if [ $numproc -ne 0 ]; then
echo “Jenkins is running…”
else
echo “Jenkins is NOT running…”
fi
}

case “$1” in
start)
start_jenkins
;;
stop)
stop_jenkins
;;
status)
status_jenkins
;;
restart)
stop_jenkins
start_jenkins
;;
*)
echo “Usage: $0 {start|stop|status|restart}”
exit 1
esac
exit 0

Update that to suit then save and change the permissions to make it executable:

chmod +x /etc/init.d/jenkins

then you can check (as root) that you can call the methods in the script:

service jenkins status

service jenkins stop

service jenkins start

These should all now run as the Jenkins user via sudo and say something when called, even if it is just “I can’t find the scripts you told me to call” 🙂

Jenkins Startup and Stop scripts

So, you now need to create the missing startup and shutdown scripts, in my example they were:

/usr/local/jenkins/bin/startup.sh

and

/usr/local/jenkins/bin/shutdown.sh

An example of the Jenkins start-up and management script I normally use is at the end of this post – the main idea though, is that it sets what you want to set then starts Jenkins via nohup and outputs to a log file

I have also added and included some basic tests to my scripts and some (very) rudimentary error handling/checking, but you shoudl get the idea and all you really need is this line (with the variables set correctly):

${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &

Note: I often run multiple instances of Jenkins, so I explicitly specify the HTTP_PORT, and I use the -D${MARKER} to allow me to safely and easily find the correct PID for each project.
The httpListenAddress can normally go unset, but it’s something I’ve had to set before on multi-home’d hosts.

The stop part of my Jenkins management script finds the correct PID like this (and you could use a filter for the correct -D${MARKER} if you want that too):

PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`

and simply kills it.

For some of my scripts I also filter for process ID’s that were started in the current directory by checking “pwdx” against the PID, but only where I’m sure the corresponding start-up process is correct/reliable.

So, tweak to your taste and that should be the Jenkins service created and working too now; for start|stop|restart you can create individual scripts or wrappers that call something like the one script below with parameters, or break the script in to separate files if you prefer.

Setting Jenkins to start at boot time

If you want to start Jenkins at boot time/startup automatically then you still need to do one more small step.  There are many different ways to do this depending on personal preference, your requirements and your version of Linux, but on Ubuntu it can be done easily with:

update-rc.d jenkins defaults

“update-rc.d” simply installs and removes System-V style init script links – read the man page for full details, but the same idea applies to most versions of Linux.

update-rc.d -f  jenkins remove

will undo this if you no longer want it.

An example Jenkins server/process management script

As an example, here is my script to manage Jenkins processes – it could use some improvements but the basic start, stop, restart and check should give you enough to sort out something that works and suits your needs.

Better formatted version available here: https://github.com/DonaldSimpson/scripts/blob/master/JenkinsProcessManager.sh

#! /bin/bash -p
# www.donaldsimpson.co.uk
# Script to start|stop|restart|check an instance of Jenkins
# For each new instance, the PROJECT and HTTP_PORT need to be updated:
export PROJECT=jenkins
export HTTP_PORT=9000
EXIT_STRING=””
JENKINS_ROOT=/opt/apps/jenkins
export JENKINS_HOME=${JENKINS_ROOT}/${PROJECT}
JENKINS_WAR=${JENKINS_HOME}/jenkins.warWAIT_TIME=5
START_WAIT_TIME=15
JAVA_HOME=/usr
PATH=$JAVA_HOME/bin:$PATH
JAVA=${JAVA_HOME}/bin/java
NOHUP=/usr/bin/nohup
LOG_FILE=${JENKINS_HOME}/debug_${PROJECT}.log
MARKER=”JenkinsProcFor_${PROJECT}”NC=/bin/nc
WGET=/usr/bin/wget
LSOF=/usr/bin/lsof
AWK=/usr/bin/awk
GREP=/bin/grep
FUSER=/bin/fuser
#################################################################################
### Functions Start #############################################################
#################################################################################

cleanup(){
# Perform any clean-up activities here…
[ “${EXIT_STRING}” != “0” ] && echo `date “+%d/%m/%y %H:%M:%S”` ${EXIT_STRING}
exit 0
}

trap ” QUIT

_error() {
EXIT_STRING=”${1}”
[ “${1}” != “0” ] && EXIT_STRING=”ERROR: ${1}, please investigate, terminating.”
cleanup  # Never returns from this call…
}

say(){
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
}

saybold(){
tput bold 2>/dev/null
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
tput sgr0 2>/dev/null
}

check_folders(){
say “”
saybold “Checking all required folders exist…”
for REQUIRED_DIR in ${JENKINS_ROOT} ${JENKINS_HOME} ${JAVA_HOME}
do
[ ! -d “${REQUIRED_DIR}” ] && _error “Necessary directory: ${REQUIRED_DIR} does not exist”
say “Found required directory: ${REQUIRED_DIR}”
done
saybold “Done.”
say “”
}

check_files(){
say “”
saybold “Checking all required files exist…”
for REQUIRED_FILE in ${JENKINS_WAR} ${NC} ${WGET} ${JAVA} ${LSOF} ${FUSER} ${GREP} ${AWK}
do
[ ! -f “${REQUIRED_FILE}” ] && _error “Necessary file: ${REQUIRED_file} does not exist”
say “Found required file ${REQUIRED_FILE}”
done
saybold “Done.”
say “”
}

check_port_closed(){
say “Checking that port ${HTTP_PORT} is closed…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
_error “Required Jenkins port ${HTTP_PORT} is already in use”
else
say “Ok, required port ${HTTP_PORT} is available, continuing…”
fi
}

check_port_open(){
say “Checking that port ${HTTP_PORT} is open…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
say “Ok, a process is listening on port ${HTTP_PORT}, continuing…”
else
_error “Required Jenkins port ${HTTP_PORT} has not been opened.”
fi
}

start_process(){
cd ${JENKINS_HOME}
saybold “Starting Process now…”
${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &
say “Process initiated.”
}

check_log(){
say “Checking the log file ${LOG_FILE} for an HTTP listener…”
STARTED=`${GREP} -c “HTTP Listener started” ${LOG_FILE}`
if [ ${STARTED} -eq “0” ]; then
_error “An HTTP Listener has not reported as started in the log file ${LOG_FILE}”
else
saybold “An HTTP Listener is reported as started in the log file ${LOG_FILE}”
fi
}

check_html(){
# These checks need error handling, but you get the general idea.
say “Checking that localhost:${HTTP_PORT} is serving Jenkins pages…”
TEMP_WGETDIR=tempwgetdir$$
mkdir ${TEMP_WGETDIR}
cd ${TEMP_WGETDIR}
${WGET} -q http://localhost:${HTTP_PORT}
GOT_HTML=`${GREP} -c Dashboard index.html`
cd ${JENKINS_HOME}
rm -rf ${TEMP_WGETDIR}
if [ ${GOT_HTML} -eq “0” ]; then
_error “Unable to get an HTML page from the Server.”
fi
saybold “Recieved valid HTML from the server, all looks ok.”
}

check_process(){
check_port_open
check_log
check_html
say “A Jenkins instance is listening on port ${HTTP_PORT} for project ${PROJECT}.”
say “The process is logging debug info to the log file: ${LOG_FILE}”
}

stop_proc(){
check_port_open
saybold “Looking for the Process ID attached to port ${HTTP_PORT}”
PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`
if [ “${PID}” == “” ]; then
saybold “Unable to detect the Process ID that is listening on port ${HTTP_PORT}!”
PID=`${FUSER} ${LOG_FILE}`
if [ “${PID}” == “” ]; then
_error “Unable to find the PID that has the log file open too!”
else
say “Ok, found PID ${PID}”
fi
else
saybold “Found a PID of $PID, killing it now…”
fi
kill -9 ${PID}
say “Waiting ${WAIT_TIME} seconds for the process to die…”
sleep ${WAIT_TIME}
saybold “Done, checking port is closed…”
check_port_closed
saybold “All done.”
}

start(){
check_folders
check_files
check_port_closed
start_process
saybold “Waiting ${START_WAIT_TIME} seconds for the process to start up…”
sleep ${START_WAIT_TIME}
check_process
}

restart(){
stop_proc
start
}

#################################################################################
### Script Start ################################################################
#################################################################################

case “$1” in
start)
start
;;
stop)
stop_proc
;;
restart)
restart
;;
check)
check_process
;;
*)
echo “Usage: $0 {start|stop|restart|check}”
esac
# Exit cleanly
_error “0”

Hope that helps! Any constructive comments, requests or suggestions for improvement are very welcome 🙂

Cheers, and sorry about the indenting,

Don

VM Ware ESXi under VM Ware Fusion

Hardware

I’m looking at getting a new home lab/server set up, and am seriously considering ordering one of these:

HP ProLiant ML110 G6 Quad Core X3430

It will need filled with memory and a load of disk space, but I reckon that’s still a whole lot of server for the money.

The Plan

The plan is to install VM Ware ESXi 5 on to it (using the USB Drive), and manage the server through the remote VM Ware vSphere Client app. This will allow me to create new Virtual Machines and migrate my existing appliances over to this server, then I can retire the old servers these have been running on. I’m also wanting to develop some automation processes for managing VM’s – creating new ones, bringing them up and down etc using Jenkins to orchestrate the processes, so this will allow me to work on that too.

Before ordering the ML110 I wanted to take a better look at the installation of ESXi – it doesn’t sound difficult, but while I’ve used it often I’d never set it up before so wanted to see what was involved.

VM Ware ESXi 5 Installation under Fusion on an iMac

It’s a little bit crazy and recursive, but, I realised I may be able to do this on my iMac under VM Ware Fusion, which it turns out does allow you to install ESX as a Virtual Machine itself…. which you can then use to manage and create new (Virtual?) Virtual Machines – a bit of a brain-ouch, and it’s clearly not going to be fast, but it’s good enough for my testing.

Installing ESXi is very straightforward – I selected the obvious option of “VM Ware ESX” in Fusion on my iMac, told it I had an image I wanted to use and pointed it at the VM Ware ESXi 5 ISO image I had downloaded from VM Ware (you need to register then fumble about their site for a while to find and download the free version – that’s the way I did it anyway). Keep a note of the serial number they give you, as that will remove the 60 day trial restriction later.

Fusion suggested a 40G file system for this instance and allocated 2GB RAM and 2 cores, which I was quite happy to run with for my test. Speaking of RAM – there was a restriction on the amount of physical RAM you could use with the free version of ESXi 5 – it was 8GB a while ago but this has now been increased to something more sensible – 32GB I think?

There are no surprises or major decision needed during the install of ESXi 5 itself – it took all of 5 minutes to run through and reboot, and that was with it running as a Virtual Machine on an overloaded iMac – on proper hardware like the HP ProLiant ML110 G6 Quad Core X3430 that I’m looking at, it would be loads faster.

When the install is done, there’s nothing more to see on the ESXi (VM), apart from the HTTP address it gives you to connect to the ESXi Host and access a simple web page it serves with links to download the VM Ware vSphere Client application to another host and start managing your server. This address was given by DHCP in my case – I think you can specify or change this easily if you want to.

Client Installation

Downloading the VM Ware vSphere Client took a while (longer than installing ESXi did!) as it came from the VM Ware site rather than being directly served by the ESXi host. Now for the bad news… for clients, you have a choice of running either Windows or Linux. No mention of an OSX vSphere client, so I had to fire up a Windows VM just to set up the client app on… not what I had been hoping for – there’s a petition asking VM Ware to sort this out here:

http://communities.vmware.com/thread/128538?start=525&tstart=0

When I get things running I could create a VM on my ESXi host which I can RDP on to, but that’s still a pretty ugly solution – if the Linux client is ok I’d go with that over the Windows one, and I think there’s also a Web Interface. But, part of the reason for me doing this in the first place is so that I can look in to the SDK and API’s for automating the creation of VM’s with VM Ware using Jenkins though, so I’ll grin and bear the Windows yuckness and see how things go.

Installing the vSphere client gets worse and worse though – my VM needed an update to its Microsoft .Not Framework (something I tend to avoid) which churned away for quite some time, and the console looks to be written in J# too (yes seriously – J# – what the… ?), which meant another “Framework upgrade” which took another while and a half – so the client set up ended up taking about 10 times longer than the server, and I had to run a Windows VM just for it… not too cool.

Once done, I could point my vSphere client at the IP address of the ESXi (VM) using its advertised IP address, the default user name (which I’d forgotten to take a note of – it’s “root”) and the password I’d specified during the install. This gets me to the Hypervisor where I can start creating and managing my own VM’s.

Cool stuff, despite the client letting things down.

Easy method to tunnel VNC over an SSH connection

This has been done many times in many different ways, but I’ve finally found a workable solution that suits my needs and my setup, and (perhaps most importantly when it comes to ports and tunneling…) that I can get my head around 🙂

 

So… at home, I have a Windows machine with remote desktop enabled, but I don’t want to open up yet another port on my router or firewall (or to expose a Windows machine to the world) for the few times that I want to get a remote desktop connection to it.

But, I have a web-facing Ubuntu Linux host which I use frequently to access the non-Windows parts of my Home Network – another Linux host, an occasionally used FreeBSD VM, Solaris 10 server, a couple of iMac’s and my ATV2.

I can see the Windows file system ok by ssh’ing in to the Linux box (it’s file system is shared and mounted) but I can’t see or access the Windows desktop when I’m outside of my home network. Doing so is handy for access to the VM Ware vClient to manage my ESX Server, etc.

A solution for this is to tunnel the VNC protocol from my remote host to the Internal Windows server via the (already exposed) SSH connection on my web-facing Linux host. To do this from outside my home, I kick off an SSH session like this:

ssh -L 5900:192.168.0.111:5900 myuser@www.donaldsimpson.co.uk

that command breaks down like this:

ssh using the -L arg ([-L [bind_address:]port:host:hostport]), which forwards the local port 5900 to 192.168.0.111, and attaches the tunnel on its port 5900 (the port that RDP or VNC is running on).

Note that the IP address 192.168.0.111 is my “internal” Windows server with a non-routable IP address – it’s accessible to/from my Linux Host (www.donaldsimpson.co.uk), but not from anywhere outside my network.

After that, I specify the SSH logon details for my Linux host and when that connection is made I’m prompted for the password for “myuser@www.donaldsimpson.co.uk“. When that is accepted I am logged on to my Linux box at the console (Cygwin with SSH installed in this case) as per usual, and I’ll need to leave this terminal open. The SSH Tunnel has been created and will end when this session does.

The next and final step is to launch “mstsc” (or VNC or another RDP Client) on my local host and create a connection – not connecting to www.donaldsimpson.co.uk, but to 127.0.0.1:5566 – this is the local end of the tunneled connection; at the other end, via my SSH Host, is the listening port on the Windows host. After supplying the requested credentials, it makes the connection, then finally shows me my Windows Desktop – woohoo, job done.

Setting up Perforce on a Linux Server and a Windows Client

As described on WikiPedia, Perforce “is a commercial, proprietary, centralized revision control system developed by Perforce Software, Inc.”

Like Subversion, it’s a client/server system where the server manages a database of revisioned files, and clients connect to checkout, modify and send back changes for others to pick up.

I wanted to check out the latest version so thought I’d install it on this server and set up a client… and that I may as well capture the steps and put them here.

In my case, the server is an Ubuntu Linux  host, and my client machine is a Windows XP workstation.

There’s not a lot to do when installing Perforce, so getting a straightforward instance up and running is a breeze. Basically, get the binary, export or pass it a few settings if you don’t want the defaults, then kick it off – here’s the detail from my notes:

Download p4d binary (for this platform and architecture):
wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4d

You can put this wherever you want, and set up a P4ROOT to specify the Perforce root directory – but don’t use that location for anything else (including client workspaces).

mkdir /apps/perforce
mv p4d /apps/perforce; cd /apps/perforce
chmod +x p4d

 

Most  Perforce options can either be exported or passed as a command-line arg, so you can choose

export P4ROOT=/apps/perforce
– or –
-r /apps/perforce

The default port is 1666, and remember that if you change this on the Server you will need to change it on your Perforce client(s) too.  In my example I’m using 9002:

export P4PORT=9002
– or –
-p 9002

So I ended up with a command line that looked like this:

nohup ./p4d -r /apps/perforce -J /var/log/journal -L /var/log/p4err -p 9002 &

Which I will probably put in to a simple startPerforce.sh script, and a probably a stopPerforce.sh script too that contains this and maybe the port number and full path to the binary location:

p4 admin stop

That’s it for the Server side at the moment, there’s a process up and running (you can check the output in nohup.out) so it’s time to set up and connect a client…

 

I’m going for a Windows client installation, which means downloading the correct version from the Perforce site then running p4vinst.exe. There’s nothing really to report here; select the usual options like directories and let it install.

Oh, I also needed to create a user, so back on the Linux Server I also downloaded the p4 binary:

wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4

Exported the P4PORT (since I was using a custom one to get through my firewalls), then add a user:

export P4PORT=9002
chmod +x p4
./p4 user -f Donald
User Donald saved.

Now you can test connecting to your Perforce Server with the  P4Admin GUI and the P4V GUI on the Windows client host by specifying the correct port (if you changed the default) and a valid user name. Once that’s done, you can admin your Depot’s and add/change/commit files, see revision and history information and all that good stuff. There are also command line and web interfaces too which can be useful for temporary use and for scripting but the Windows GUI’s are nice to use and quite self explanatory – if you’ve used a similar revision control system like Subversion and an Eclipse-like IDE before there’s not much of a learning curve here.

The Perforce Help and Documentation is all very good, and their Perforce 2010.2: System Administrator’s Guide covers all of the above in more detail, and touches on more advanced topics too – Perforce Performance Tuning, Backup and Recovery, Replication, and the Perforce Broker (P4Broker) etc.

There’s also a Perforce plugin for Jenkins, which once installed allows you to choose Perforce as one of the SCM option in your Jenkins jobs, but the above hopefully covers the initial setup of both the Perforce Client and the Perforce Server on Windows and Linux respectively.

 

Jenkins Agent Nodes

This Jenkins Agent Nodes post covers:

  • What are they?
  • Why may I want one?
  • How do you create one?
    • tasks on On the Master/Server
    • tasks on On the Agent/Client
  • Other ways of creating Agent Nodes
  • Related posts and links

What are they?

Jenkins Agents  are small Java “Client” processes that connect back to a “Master” Jenkins instance over the Java Network Launch Protocol (JNLP).

Why may I want one?

Once it’s up and running, an Agent instance can be used to run tasks from a Master Jenkins instance on one or more remote machines providing an easy to use and flexible distributed system which can lend itself to a wide variety of tasks.

As these are Java processes, you are not restricted by architecture and can mix and match the OS’s of your agent nodes as required – Windows, Linux, UNIX, iSeries, OVMS etc – anything capable of running a modern version of Java (I think JNLP was introduced in 1.5?) and you can also group and categorize subsets of different types (both logical and physical) of Agents; intended use, availability, location, available resources, Cloud or VM versus Physical tin – anything that helps you decide when you want to use which host.

There are many different ways you can choose to utilize these nodes – they can be used to spread the load of an intensive build process over many machines when they are available, you can delegate specific tasks to specific machines only, or you can use labels to group different classes or types of Nodes that are available for certain tasks, making the most use of your available resources. You can also have Jenkins create Cloud server instances – Amazon EC2 for example – when certain thresholds are reached, and stop them when they are no longer required.

This post focuses on a pretty manual approach to the creation of Jenkins Agent Nodes with the intention of explaining them well enough to allow you to create them on any platform that can run a modern version of Java – there are probably simpler solutions depending on your needs and setup. A later post will touch on a few of the many possible uses for these nodes.

So, how do you create one?

There are several different ways to go about setting up an Agent, and the “best” approach depends on your situation, needs and environment(s) – for a simple Linux setup letting Jenkins do all the work for you makes life really easy, you can just select that option in your new Jenkins Agent Node and complete this screen to have Jenkins set it up for you:

Where the Username and Password are the credentials you want Jenkins to use to connect and start the Agent process on the remote server. This simple approach also allows the Master instance to initiate the JNLP connection and bring your agents online when required, avoiding any need to manually visit each agent node.

This keeps things nice and simple and reduces the admin overhead too,  but sometimes this type of approach can’t be used (on different OS’s like OVMS, iSeries, Windows etc) and I’m going to go on to outline what I think is the most “versatile” method – defining the Node on the Master instance, and manually setting up and starting the corresponding Agent/Client Node on the remote host – going through these steps should provide enough detail on how Agent Nodes work and connect to get one up and running on anything that can run a JVM.

1. On the Master/Server

Define the host: navigate to Jenkins > Manage Jenkins > Manage Nodes > New Node
Enter a suitable Node Name (I’d recommend something descriptive, and usually including the host name or part of it) then either select to create a “Dumb Agent” or copy an existing Node if you have one, then complete the configuration page similar to this:


where you specify the requested properties – path, labels, usage, executors etc. These are explained in more detail in the “?” for each item if required.

Here you can also state if you want to keep your Jenkins Agent for tied jobs only, or if it is to be utilized as much as possible – this obviously depends on your requirements. You can also specify the Launch method that best applies to your needs & requirements.

2. On the Agent/Client host

You don’t need to do very much to create a new agent node – typically if I’m setting up a few *NIX and Windows hosts I would archive a simple shell/DOS script that starts and manages the process along with the slave.jar file from the Master Jenkins instance. There are alternative methods that may suit your needs – you can start agents via SSH from the Master server for example and there’s a comparable method for Windows – but this simple approach should help you understand the underlying idea that applies to them all.

You can “wget” (or use a Browser on Windows) the slave.jar file directly from the Master Jenkins instance using the URL

http://[your jenkins host]:[port number]/jnlpJars/slave.jar

If you let JNLP initiate the process, the slave.jar will be downloaded from Jenkins automatically.

Note that Jenkins will inherit the effective permissions of the user that starts the process – this is to be expected, but it’s often worth having a think about the security aspects of this, along with the access requirements for the types of things you want your agent to be able to do… or not do.

On Windows hosts, you can use the jenkins-agent.exe to easily install Jenkins as a Windows Service, which can then be started at boot time and run under whatever user/permissions you wish set via the Services panel.

My *NIX “startagent.sh” script does a few environment/sanity checks, then kicks of the agent process something like so:

${NOHUP} ${JAVA} -jar slave.jar -jnlpUrl http://SERVERNAME:PORT/computer/USER__NODENAME/slave-agent.jnlp &

The HTTP URL there should match the one provided by the Jenkins page when you were defining the Node. If all goes well you should see the node state changed to Connected on the Master Hudson instance, and if not, then nohup.out should provide some pretty obvious pointers on the problem.

Some common causes are:

Jenkins host, port or node name wrong
Java version not found/wrong
Lack of write permissions to the file system
Lack of space (check /tmp too)
Port already in use
Errors in the jenkins-slave.xml file if you’ve tweaked it
Firewalls…

Jenkins also provides some health monitoring of the connected Node which you can see in the Jenkins > Nodes page:
Disk Space, Free Temp Space, Clock time/sync, Response Time and Free Swap are monitored
and you can have your Node taken off line if any of these passes a set threshold.

This should hopefully be enough info to provide an overview of what Jenkins Agents are, and enough to get one up and running on your chosen platform. Where possible it’s best to keep things simple – use SSH and let the Master instance manage things for you if you can – but when that’s not possible there are alternatives.

When I get the chance I will add some information on the next steps – creating and delegating jobs on Jenkins Agent Nodes, and some thoughts and suggestions for just a few of the many uses for this sort of distributed Build and Deployment system.

Related Posts and Links:

Monitoring Jenkins Slave Nodes with Groovy
– how to keep an  eye on your Jenkins Slaves

Jenkins Slave Nodes – using the Swarm Plugin
– automatically connect new Slave Nodes to create a “Swarm”

Getting the current user in Jenkins
– several approaches

Managing Jenkins as a service and starting at boot time
– on Linux & Windows

Jenkins plugins
– details on some of my most frequently used plugins

Jenkins DIY Information Radiators
– what they are for, and how to make your own

The Jenkins Wiki has more details information on Distributed Builds and different slave-launching strategies.

Feedback, questions and constructive comments are very welcome!

Setting up OpenNebula on Ubuntu

Some very rough notes on installing and configuring OpenNebula on an Ubuntu host.

On the Server host:

apt-get install opennebula

Adding system user `oneadmin’ (UID nnn) …
Adding new user `oneadmin’ (UID nnn) with group `nogroup’ …
The home directory `/var/lib/one’ already exists.  Not copying from `/etc/skel’.
adduser: Warning: The home directory `/var/lib/one’ does not belong to the user you are currently creating.
Generating public/private rsa key pair.
Your identification has been saved in /var/lib/one/.ssh/id_rsa.
Your public key has been saved in /var/lib/one/.ssh/id_rsa.pub.
The key fingerprint is:
(key fingerprint)oneadmin@linux
The key’s randomart image is:
+–[ RSA 2048]—-+
|     .    o..    |
(more random artwork)

 

test the installation by running the new “onehost” command:

 

Usage:
onehost [<options>] <command> [<parameters>]

Options:
-l, –list x,y,z                 Selects columns to display with list
command
–list-columns               Information about the columns available
to display, order or filter
-o, –order x,y,z                Order by these columns, column starting
with – means decreasing order
-f, –filter x,y,z               Filter data. An array is specified
with column=value pairs.
-d, –delay seconds              Sets the delay in seconds for top
command
-v, –verbose                    Tells more information if the command
is successful
-h, –help                       Shows this help message
–version                    Shows version and copyright information

Commands:

* create (Adds a new machine to the pool)
onehost create <hostname> <im_mad> <vmm_mad> <tm_mad>

* show (Gets info from a host)
onehost show <host_id>

* delete (Removes a machine from the pool)
onehost delete <host_id>

* list (Lists machines in the pool)
onehost list

* enable (Enables host)
onehost enable <host_id>

* disable (Disables host)
onehost disable <host_id>

* top (Lists hosts continuously)
onehost top

 

So far so simple, so it’s time to set up a new host and install the client…

 

This is done by installing the node package like so:

sudo apt-get install opennebula-node

then starting defining and adding the node to the Master instance via the onehost command.

After that’s done, you can move on to set up a private network for your cloud, create you own KVM images, and start firing up VM’s in your own personal cloud.

More detail coming soon…

Mount Windows share on Ubuntu

Some notes on setting up and auto mounting a Windows Share on an Ubuntu host.

I’ve had to Google the details for this more than once, so thought I’d write up the steps here for next time…

First, if it’s not there already, add the Windows IP address and HostName  to the Ubuntu /etc/hosts file:

vim /etc/hosts, then add something like:
192.168.0.123 MyWindowsHostName

 

Now install the smbfs packages if you don’t already have them:

sudo apt-get install smbfs
sudo apt-get install smbclient

 

Once that’s complete, and assuming the windows shares are set up ok (and there are no firewall issues, and you can ping the host etc), check that we can view the host and its shares with smbclient:

root@linux:/mnt# smbclient -L MyWindowsHostName
Enter root’s password:
Domain=[LIMBO] OS=[Windows 5.1] Server=[Windows 2000 LAN Manager]

Sharename       Type      Comment
———                —-         ——-
Video                 Disk
IPC$                   IPC       Remote IPC
Music                 Disk

Domain=[LIMBO] OS=[Windows 5.1] Server=[Windows 2000 LAN Manager]

 

If that gives you back something that looks like your Windows hosts and its share(s), things are looking good 🙂

Further information on using smbclient can be found here if you have any problems.

 

Now it’s time to make a local mount point, so as root or via sudo;
mkdir /mnt/Video

Then as root create a password file /etc/cifspw with the login credentials for your Windows account.

username=WINDOWSUSERNAME
password=WINDOWSPASSWORD

it would be good practice to secure that file so that only the owner (root) has read/write access to it:
$ sudo chmod 600 /etc/cifspw

Then vim /etc/fstab and add a line for the mount:
//MyWindowsHostName/Video                                  /mnt/Video      cifs exec,credentials=/etc/cifspw 0 0

If all goes well, the Windows share should now automatially mount to /mnt/Video at boot time.

If you can’t wait to test it, you can do:

sudo mount -a

and check /mnt/Video to see your data… hopefully!

Cheers,

Don

 

Some Useful Solaris Commands

Here are a few (mostly) Solaris tips and tricks I have found useful and wanted to keep a note of.

 

prstat

This provides similar info to top on Linux boxes – you can run it as plain old prstat, or give it some options. I like prstat -a as it reports on both processes and users. As with all of these commands, the man pages have further details.

 

xargs

Not just a Solaris command, but this is very useful on any *NIX box – I frequently use it to translate the output from the previous command in to something that can be undertood by the next one, for example:

find . -type f -name *.txt | xargs ls -alrt

Will translate and pass the output of the “find” command to ls in a way that ls understands.

 

pargs

I use the pargs command when I need to get more information on a running process than the Solaris ps utility will give (there’s no -v option) if they have a lot of arguments.

Call pargs with the PID of your choice, and it will display a nice list of each argument that the process was started with, for example:

> pargs 16446
16446:  /usr/jdk/jdk1.6.0/jre/bin/java com.MyJavaProgram
argv[0]: /usr/jdk/jdk1.6/jre/bin/java
argv[1]: com.MyJavaProgram
argv[2]: MyFirstArgument.ini
argv[3]: SomeOtherArg.txt
argv[4]: AndAnotherArg

pargs can also display all of this info on one line with the -l option (useful for scripting), and if you call it with -e it also displays all of the Environment variables too.

 

pwdx

Simply pass it a PID and it will tell you the current working directory for that process.

 

[g]rep

When writing a shell script that queries running processes, I often find my own script showing up in the results – for instance a script that does a “ps -eaf | grep MyProcessName” may pick up the java process I’m after (the running instance of “./MyProcessName“) and the grep process-check itself (as in the “ps -eaf | grep MyProcessname“).

A handy way to avoid this is by changing your search criteria to “grep [M]yProcessName” instead. Grep interprets (and effectively ignores) the square brackets, with the result that your grep query no longer matches its own search 🙂

 

I will add more when I think of them, if you have any good ones then please post them!

Persisting file permissions in a tar.gz file with Ant & tar

Discovered an interesting issue recently where file permissions were not being preserved when Taring up a build with Ant.

The existing approach was to chmod the files as desired then simply tar them all up and hope for the best:

<tar destfile=”dist/${hudson.build.tag}.tar.gz” basedir=”dist/” compression=”gzip” />

This doesn’t work,  but if you use tarfileset and set the filemode you can explicitly set things as required like this:

<tar destfile=”dist/${hudson.build.tag}.tar.gz” longfile=”gnu” compression=”gzip”>

<tarfileset dir=”dist/” filemode=”755″>

<include name=”**/*scripts/*.sh” />

<include name=”**/somescript.ext” />

</tarfileset>

<tarfileset dir=”dist/”>

<include name=”**/*” />

<exclude name=”**/*scripts/*.sh” />

<exclude name=”**/somescript.ext” />

</tarfileset>

</tar>

Here I am adding the first two scripts with chmod 755, then adding everything else which will get the default/umask permissions. I exclude the previous files – not sure if that’s required or not but I don’t want to risk overwriting them.

Now when you gunzip and tar xvf the resulting build, you get the required permissions.

There’s more info and further examples in the Apache Ant Manual.

Cheers,

Don

Pin It on Pinterest

%d bloggers like this: