Kubernetes – with Minikube and Helm – part 2

This is the second half of the Kubernetes with Minikube and Helm presentation, the first half explains all of the steps we went through to get to this point, and is available here:

In this section we cover the following:

  • Helm and Tiller – what they are, when & why you’d maybe use them
  • Helm and Tiller – prep, install and Helm Charts
  • Deploying Jenkins via Helm Charts
  • and WordPress w/MariaDB too
  • Wrap up

The below are mostly my technical notes from this session, with some added blurb/explanation.

Helm and Tiller – what they are, when & why you’d maybe use them

From the Helm site:

“Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Charts are easy to create, version, share, and publish — so start using Helm and stop the copy-and-paste.”

https://helm.sh/

Helm is basically a package manager for Kubernetes applications. You can choose from a large list of Stable (or not so!) ready made packages and use the Helm Charts to quickly and easily deploy them to your own Kubernetes Cluster.

This makes light work of some very complex deployment tasks, and it’s also possible to extend these ready-made charts to suit your needs, and to write your own Charts from scratch, or pass your own values to override default ones, or… many other interesting options!

For this session we are looking at installing Helm, reviewing some example Helm Charts and deploying a few “vanilla” ones to the cluster we created in the first half of the session. We also touch upon the life-cycle of Helm Charts – it’s similar to dockers – and point out some of the ways this could be extended and customised to suit your needs – more on this at a later date hopefully.

Helm and Tiller – prep, install and Helm Charts

First, installing Helm – it’s as easy as this, run on your laptop/host that’s running the Minikube k8s we setup earlier:

Get & chmod the get_helm script, then run it:

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh

chmod 700 get_helm.sh

./get_helm.sh

Tiller is the client part of Helm and is deployed inside your k8s cluster. It’s set to be removed with the release of Helm 3, but the basic functionality wont really change. More details here https://helm.sh/blog/helm-3-preview-pt1/

Next we do the Tiller prep & install – add RBAC for tiller, deploy via helm and take a look at the running pods:

kubectl create serviceaccount -n kube-system tiller

kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

helm init --service-account tiller

kubectl --namespace kube-system get pods

Helm Charts – look at the list of available stable Charts, then deploy a couple. The github repo is here

https://github.com/helm/charts

Update the local helm repo info:

helm repo update

then, for example, install Redis from its Helm Chart to the k8s cluster as easily as this:

helm install stable/redis

or helm install stable/mysql and check the console output that explains how to access the newly deployed app.

keep an eye on the pods to see what’s going on: watch kubectl get pods -o wide

Deploying Jenkins via Helm Charts

helm ls

helm delete <things you don't want any more to free up resources>

helm install --set serviceType=NodePort --name jenki stable/jenkins

again, watch kubectl get pods -o wide

now get the URL for the Jenkins service from Minikube:

minikube service --url=true jenki-jenkins

Hit that URL in your browser, and grab the password in UI from Pods > Jenki and log in to Jenkins with the user “admin”:

That’s a Jenkins instance deployed via Helm and Tiller and a Helm Chart to our Kubernetes Cluster running inside Minikube via a VirtualBox VM… all done in a few minutes. And it’s all customisable, repeatable, highly scaleable and awesome.

and WordPress w/MariaDB too

This was the “bonus demo” if my laptop wasn’t on fire – and thanks to some rapid cleaning up it managed fine – showing how quickly we could deploy a functional WordPress with MariaDB backend to our k8s cluster using the Helm Chart.

To prepare for this I did a helm ls to see all the things I had running. then helm delete --purge jenki, gave it a while to recover then had to do

kubectl delete pods <jenkinpod>

before starting the WordPress Chart deployment with

helm install --set serviceType=NodePort --name wp-k8s stable/wordpress

watch kubectl get pods -o wide for a while – note the chart is configured with the mariadb pod as a pre requisite of the wordpress instance:

Once it’s started we requested the service URL from Minikube again, making ingress nice and easy:

minikube service --url=true wp-k8s-wordpress

Hit that in the browser, using https and accepting the cert warning…

then logged in as `user` and qureied for the password in the k8s secret…

echo Password: $(kubectl get secret wp-k8s-wordpress -o
jsonpath="{.data.wordpress-password}" | base64 --decode)

and logged in to WordPress:

Wrap up

That’s it – we covered a lot in this session, and plan to use this as a platform to explore Helm in more detail later, writing our own Helm Charts and providing our own customisations to them.

minikube delete; rm -rf ~/.minikube

Cleans up everything we’d done:

Leaving just the local tools to remove if you want to – see the first half for a reminder.

Cheers,

Don

Jenkins Global Pipeline Libraries – a v.quick start guide

This post runs through the steps required to start using Global Pipeline Libraries in your Jenkins Pipelines.

There are many posts about these all over the ‘net, but they mostly seemed overly complex and not too helpful to me – I just wanted to know how to get the most basic example possible working quickly on my dev Jenkins instance, so I could see how they work in practice and take it from there.

That’s what this post covers – getting a simple “Hello World” type example library published and made available in Jenkins, then calling it very easily from within a Pipeline job with the expected results. More detail and advanced usage to come later… these are a very powerful addition to Jenkins pipelines.

This is done in three simple and logical steps:

Create a Library and Publish it

Tell Jenkins about this nice new library

Calling the Global Library from my Jenkins Pipeline


The first step is to…

Create your Library and publish it somewhere.

I have reused one of my existing GitHub repos: https://github.com/DonaldSimpson/groovy.git for this example, but most version control systems should do just as well.

That’s all that’s needed for this most-basic example – here is the code in plain text, as taken from the guide here:

#!/usr/bin/env groovy
def call(String name = 'human') {
    // Any valid steps can be called from this code, just like in other
    // Scripted Pipeline
    echo "Hello, ${name}."
}

It is important to note that the file is in a “vars” directory, this is the naming convention Jenkins expects to find your groovy libraries within, and is best followed.

A. Note

Next step is to:

Tell Jenkins about this nice new library

This is done by going to Manage Jenkins then Configure System, then scrolling down to Global Pipeline Libraries and defining a new instance of one, just like this:

The settings used here are:

Name: mycommonlibs // any name you’d like to reference these libraries by

Default version: master // or use a branch or version number if you prefer

I then checked the three tick boxes, especially the Load implicitly which removes the need to load Libraries explicitly in your Jenkinsfile (you can do this and it may be very useful depending on your needs, but I want simple and easy for now).

The final section tells Jenkins where this Library is:

https://github.com/DonaldSimpson/groovy.git

and I provide a user to access GitHub with.

That is all that is needed to set up a Library and tell Jenkins all about it.

Note that anyone with write access to the location of your defined Libraries will effectively have full access to your Jenkins instance

W. Arning

And finally, it’s time time for a test drive…

Calling the Global Library from my Jenkins Pipeline:

    sayHello ()
    sayHello 'Donald'

To end up with a mega-basic Pipeline that looks like this:

When this Jenkins Pipeline job is run, it generates the following output:

Summary

Which as you can see means that Jenkins has pulled in the Shared Library from GitHub, resolved and called the sayHello() method from the remote common library, called it again with a passed parameter (‘Donald’) and produced the expected results. Yay. How neat and how easy was that?

There’s a whole lot more you can do with Global Pipeline Libraries in Jenkins. From this point you can easily add complexity and functionality to build up a library of powerful and useful utilities that will greatly improve the quality and manageability of your Pipelines.

I plan to expand on some of these points in a later post, but hopefully this shows how to quickly and easily start using them.

Cheers,

Don

Meetup – Deploying Openshift to AWS with HashiCorp Terraform and Ansible

 

Automated IT Solutions presented a talk on “Deploying Openshift to AWS with HashiCorp Terraform and Ansible”, by Liam Lavelle on 16th October 2018.

 

We would like to thank

 

  • Liam Lavelle for an interesting, informative and fun session
  • Everyone that came along to make it such a good event, with some great questions, helpful answers and interesting discussions
  • Hays for the beer, pizza, venue and help with everything

Hope to see you all at the next one soon!

The slides and all materials used in this session are available on our GitHub repo here:

 

Deploying Openshift to AWS with HashiCorp Terraform and Ansible

Tuesday, Oct 16, 2018, 6:15 PM

HAYS
7 Castle St, Edinburgh EH2 3AH Edinburgh, GB

30 Members Went

In this session we look at Infrastructure as Code and Configuration as Code, as we demonstrate how to use these approaches to deploy RedHat OpenShift to AWS with HashiCorp Terraform and Ansible. We start off with configuring AWS credentials, then use HashiCorp Terraform to create the AWS infrastructure needed to deploy and run our own RedHat OpenSh…

Check out this Meetup →

 

Here are the details:
When:
Tuesday, October 16th, 2018
6:15 PM to 9:00 PM

 

Where:
Hays office on the 2nd floor
7 Castle St, Edinburgh EH2 3AH · Edinburgh

 

What:
Deploying Openshift to AWS with HashiCorp Terraform and Ansible

 

Agenda:

In this session we look at Infrastructure as Code and Configuration as Code, as we demonstrate how to use these approaches to deploy RedHat OpenShift to AWS with HashiCorp Terraform and Ansible.

We start off with configuring AWS credentials, then use HashiCorp Terraform to create the AWS infrastructure needed to deploy and run our own RedHat OpenShift cluster.

We then go through using Ansible to deploy OpenShift to AWS, followed by a review of the Cluster, then take a quick look at troubleshooting any issues you may encounter.

There will be a break in the middle for beer & pizza courtesy of Hays, and we will wrap things up with a quick Q&A and feedback session.

If you would like to bring your own laptop and follow along, please do!

Who:
Intermediate Linux and some AWS knowledge is useful but not essential.

New Meetup – Vagrant from scratch to LAMP stack

Automated IT Solutions are running a new Meetup in Edinburgh on Friday 18th May, check out the details and register for this free session here – beer, pizza and free HashiCorp stickers included!:

Vagrant from scratch to LAMP stack

Friday, May 18, 2018, 6:15 PM

HAYS
7 Castle St, Edinburgh EH2 3AH Edinburgh, GB

18 Members Attending

Automated IT Solutions are presenting a session on HashiCorp Vagrant: “from scratch to LAMP stack” by Adam Cheney. In this session you will learn: – Vagrant basics, introduction and usage – How to install and configure Vagrant – Provisioning VMs with Vagrant and Ansible followed by a live demonstration/workshop of building a LAMP stack within Vagra…

Check out this Meetup →

Jenkins and Docker – Part 1 of 3

This post is the first in a series of 3 introducing the combined power of Jenkins, Docker, and the Jenkins DSL.

They should hopefully provide enough information to get to grips with both Docker and Jenkins – what they both do and how to use them – by showing some practical examples of them working together.

The first step, if you haven’t already, is to download and install Docker on your platform – the Docker website covers this in good detail for most platforms…

Docker for Mac

Docker for Windows

Docker for Linux

Once that’s done, you can try it out with the customary “Hello World” example…

I’m running Docker on an Ubuntu VM, but the commands and the results are the same regardless of platform – that’s one of the main Docker concepts.

You can then check which processes (docker containers) are running using the “docker ps” command – in my example you can see that there’s one Jenkins image running. If you run “docker ps -a” you will see all containers (including stopped ones, of which I have a few on this host):

and you can check your Docker version with:

root@ubuntud:~# docker --version
Docker version 1.13.0, build 49bf474

Now that the basic setup is done, we can move on to something a little more interesting – downloading and running a “Dockerised” Jenkins container.

I’m going to use my own Dockerised Jenkins Image, and there will be more detail on that in the next post – you’re welcome to try it out too, just run this command in your terminal:

docker run -d -p 8080:8080 donaldsimpson/dockerjenkins

if you don’t happen to have my docker image cached locally (like I do) then docker will automatically download it for you from Docker Hub then run it:

That command did a quite few important things, here’s a quick explanation of them all:

docker run -d

tells docker that we want to run the container in the background so that we can carry on and do other things while it runs. The alternative is -it, for an interactive/foreground session.

docker run -d -p 8080:8080

The -p 8080:8080 tells docker to map port 8080 on the local host to port 8080 in the running container. This means that when we visit localhost:8080 the request will be passed through to the container.

docker run -d -p 8080:8080 donaldsimpson/dockerjenkins

and finally, we have the namespace and name of the Docker image we want to run – my “donaldsimpson/dockerjenkins” one – more on this later!

You can now visit port 8080 on your Docker host and see that Jenkins is up and running….

 

That’s Jenkins up and running and being happily served from the Docker container that was just pulled from Docker Hub – how easy was that?!

And the best thing is, it’s entirely and reliably repeatable, it’s guaranteed to work the same on all platforms that can run Docker, and you can quickly and easily update, delete, replace, change or share it with others! Ok, that’s more than one thing, but the point is that there’s a lot to like here 🙂

That’s it for this post – in the next one we will look in to the various elements that came together to make this work – the code and configuration files in my Git repo, the automated build process on Docker Hub that builds and updates the Docker Image, and how the two are related.

Extending Jenkins book

My new book, Extending Jenkins by Donald Simpson, has been published!

Here’s a free sample: Chapter 8 – Testing and Debugging Jenkins Plugins

You can buy the full book in either electronic or paperback format direct from the publishers or through Amazon here in the UK or Amazon in the US

About This Book

  • Find out how to interact with Jenkins from within Eclipse, NetBeans, and IntelliJ IDEA
  • Develop custom solutions that act upon Jenkins information in real time
  • A step-by-step, practical guide to help you learn about extension points in existing plugins and how to build your own plugin

Who This Book Is For

This book is aimed primarily at developers and administrators who are interested in taking their interaction and usage of Jenkins to the next level.

The book assumes you have a working knowledge of Jenkins and programming in general, and an interest in learning about the different approaches to customizing and extending Jenkins so it fits your requirements and your environment perfectly.

Table of Contents

1: Preparatory Steps
2: Automating the Jenkins UI
3: Jenkins and the IDE
4: The API and the CLI
5: Extension Points
6: Developing Your Own Jenkins Plugin
7: Extending Jenkins Plugins
8: Testing and Debugging Jenkins Plugins
9: Putting Things Together

What You Will Learn

  • Retrieve and act upon Jenkins information in real time
  • Find out how to interact with Jenkins through a variety of IDEs
  • Develop your own Form and Input validation and customization
  • Explore how Extension points work, and develop your own Jenkins plugin
  • See how to use the Jenkins API and command-line interface
  • Get to know how to remotely update your Jenkins configuration
  • Design and develop your own Information Radiator
  • Discover how Jenkins customization can help improve quality and reduce costs

In Detail

Jenkins CI is the leading open source continuous integration server. It is written in Java and has a wealth of plugins to support the building and testing of virtually any project. Jenkins supports multiple Software Configuration Management tools such as Git, Subversion, and Mercurial.

This book explores and explains the many extension points and customizations that Jenkins offers its users, and teaches you how to develop your own Jenkins extensions and plugins.

First, you will learn how to adapt Jenkins and leverage its abilities to empower DevOps, Continuous Integration, Continuous Deployment, and Agile projects. Next, you will find out how to reduce the cost of modern software development, increase the quality of deliveries, and thereby reduce the time to market. We will also teach you how to create your own custom plugins using Extension points.

Finally, we will show you how to combine everything you learned over the course of the book into one real-world scenario.

Extracting data from Jenkins

 

 

In Part I,  Information Radiators, I covered what they are, what the main benefits are, and the approach I usually use to set them up. This post goes in to more technical detail on how I extract this data from Jenkins.

My usual setup/architecture for Jenkins Information Radiators goes something along these lines:

  • TV screens running Mozilla Firefox or Google Chrome in Kiosk Mode, and Tab Mix Plus set up to rotate tabs (if required)
  • JSP Pages served via Tomcat on Linux server (which also runs the data extracting script described below)
  • MySQL database on Linux server – contains tables with data pulled from Jenkins and other sources, and the config data too (which URL’s to monitor)

And you’ll need some Jenkins instances/jobs to monitor too, obviously 🙂

The Jenkins XML API is very useful for automating tasks like this – if you simply append “/api/xml” to a
Jenkins job URL, it will serve up an XML version – note there is also a JSON API and a CLI and plenty of other options, but I’m using what suits me.

The Jenkins XML API

For example, if you go to one of your Jenkins jobs and add /api/xml like this:

“http://yourjenkinsserver:8080/job/yourjobname/api/xml

you should get back some XML, possibly roughly like this example:

<?xml version="1.0"?>
<freeStyleBuild>
 <action>
 <parameter>
 <name>LOWER_ENV</name>
 <value>dev</value>
 </parameter>
 </action>
 <action>
 <cause>
 <shortDescription>Started by timer</shortDescription>
 </cause>
 </action>
 <building>false</building>
 <duration>61886</duration>
 <fullDisplayName>MyJob #580</fullDisplayName>
 <id>2014-04-01_10-01-50</id>
 <keepLog>false</keepLog>
 <number>580</number>
 <result>SUCCESS</result>
 <timestamp>1396342910088</timestamp>
 <url>http://jenkinsserver:8080/view/MyView/job/MyJob/580/</url>
 <builtOn/>
 <changeSet/>
</freeStyleBuild>

That XML contains loads of very useful information inside handy XML tag descriptions – you just need a way to get at that data and then you can present it as you like…

XPAth queries and the Jenkins XML API

so to automate that, I used to extend that approach a to query Jenkins via the XML API using XPAth queries to bring back just the data I actually wanted, quite like querying a database.

For example, wget’ing this URL would return just the current value of the <building> tag in the above XML:

http://yourjenkinsserver:8080/job/yourjobname/api/xml?xpath=//building/text()

e.g. “true” or “false” – this was very useful and easy to do, but the functionality was removed/disabled in recent versions of Jenkins for security reasons, meaning that my processes that used it needed rewritten 🙁

Extracting the data – Plan B…

So, here’s the new solution I went for – the real scripts/methods do some error handling and cleaning up etc but I’m just highlighting the main functions and the high level logic behind each of them here;

get_url’s method:

query a table in MySQL that contains a list of the job names and URL’s to monitor
for each $JOB_NAME found, it calls the get_file method, passing that the URL as a parameter.

get_file method:

this takes a URL param, and uses curl to fetch and save the XML data from that URL to a temporary file (“xmlfile”):

curl -sL "$1" | xmllint --format - > xmlfile

Note I’m using “xmllint –format” there to nicely format the XML data, which makes processing it later much easier.

get_data method:

this first calls “get_if_building” (see below) to see if the job is currently running or not, then it does:

TRUE_VAR="true"
 if [[ "$IS_BUILDING" == "$TRUE_VAR" ]]; then
 RESULT_TEXT="building..."
 else
 RESULT_TEXT=`grep "result>" xmlfile | awk -F\> '{print $2}' | awk -F\< '{print $1}'`
 fi

get_if_building method:

this simply checks and sets the IS_BUILDING var like so:

IS_BUILDING=`grep building xmlfile | awk -F\> '{print $2}' | awk -F\< '{print $1}'`

Putting it all together

My script then updates the MySQL database with the results from each check: success/failure, date, build number, user, change details etc

I then have JSP pages that read data from that table, and translate things like true/false in to HTML that sets the background colours (Red, Amber, Green), and shows the appropriate blocks and details per job.

If you have a few browsers/TV’s or Monitors showing these strategically placed around the office, developers get rapid feedback on the result of their code changes which speeds up development, increases quality and reduces development time and costs – and they can be fun to watch and set up too 🙂

Cheers,

Don

DIY Information Radiator

 

 

Information Radiators are used to provide people with feedback on the current status of code builds and automated tests in Continuous Integration and Agile development environments.

The basic idea is that when developers commit a code change, they can easily and quickly see that it has been picked up by the automated build process, and then (ideally within 10 minutes) see the result of their change; did the build succeeded and did the automated tests pass?

Martin Fowler’s description goes in to more detail on the ideas behind this approach and the function that Information Radiators serve.

The normal convention for these is to use colour coded blocks per build, using:

  • Green for good/passing jobs
  • Amber for either currently building/running jobs (or sometimes for unstable ones)
  • Red for failed jobs

Generally you want to keep things as clean, simple and uncluttered as possible, but sometimes it is helpful to add in a bit more info.

Details I have occasionally found worth adding include things like;

  • name and/or picture of the user who triggered (or broke!) the build
  • commit message from the code change that triggered the build
  • build number
  • history – number of recent fails or passes
  • date/time last failed and last checked

if you use amber for “unstable” rather than “build in progress”, you may want to add text to say if the job is currently building – I often use the “spinning wheel” icon thingy from jenkins itself :

spinner

Why build your own?

There are tons of readily available plugins that allow you to quickly and easily produce a Radiator or Wall panel from Jenkins, so why go to the bother of making your own?

Plugins are usually linked to one Jenkins instance (the one they are running on) and I have often found that alone to be too restrictive – having too many different radiators all over the place makes things too cluttered and uncertain, and people can easily start to “switch off” from them all – having one screen that people can understand at a glance usually works best.

Changing requirements – developers are constantly wanting/looking to improve processes and often come up with ideas and requests for things to try that may help them do their jobs – adding a bit of information from another source, for example, or changing the colours used to a different shade, or adding curved borders etc etc…

What I have found often works best, is to get all of the data I am interested in inside a database then write my own simple but flexible presentation layer from scratch – this gives me all the flexibility I could want (or may find myself wanting or needing later…) and importantly, it also allows me to leverage additional benefits by combining data from Jenkins with data from elsewhere – this means I can easily produce reports, charts, metrics etc that present a view comprised from multiple data sources throughout an organisation – for example, you can then easily create reports that combine:

  • jenkins – live information on the current state of builds and tests
  • defects – data on bugs and changes pulled from bug tracking apps (usually via jenkins jobs)
  • version control – the actual code change can be extracted from version control and linked to both the developer and bug details – the “svn log” command is useful for this
  • environment monitors – state and health of environments; database and app server health, deployed patch and code levels etc (again, these are usually other Jenkins jobs!)

and you can add in anything else you can get your hands on 🙂

This allows you to track the flow of a change right through the development life cycle – from the initial change/requirement/defect to the change itself at the code/file level, then the testing and building of it and the eventual release. This is far more than you need for a simple Information Radiator, but using this approach means that you can easily reuse much of the work in different ways.

Part II covers the technical approach I use for Extracting data from Jenkins

Jenkins Slave Nodes – using the Swarm Plugin

 

 

I’ve been trying out a new (to me at least) way to add a Jenkins Slave Node – using UDP auto discovery via the Jenkins Swarm Plugin

This is a very easy and nice way to do it, with minimal configuration/hassle required so you can quickly and easily add new Jenkins Slave Nodes to your Master Jenkins instance as and when they are required.

Here are my notes from setting this up – it’s pretty simple to do and worked well for me out of the box…

Set up a new instance of Jenkins:

  • Make & cd to a working directory

mkdir jenkinsswarm; cd jenkinsswarm

  • Fetch jenkins.war

curl -O http://mirrors.karan.org/jenkins/war/1.506/jenkins.war

  • Start Jenkins

{/path/to/java/bin/}java -jar jenkins.war

After that, you should get console output along these lines…

Running from: /root/jenkinsswarm/jenkins.war
webroot: $user.home/.jenkins
18-Mar-2013 15:19:26 winstone.Logger logInternal
INFO: Beginning extraction from war file
Jenkins home directory: /root/.jenkins found at: $user.home/.jenkins
18-Mar-2013 15:19:33 winstone.Logger logInternal
INFO: HTTP Listener started: port=8080
18-Mar-2013 15:19:33 winstone.Logger logInternal
INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled
18-Mar-2013 15:19:34 jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
18-Mar-2013 15:19:35 jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
18-Mar-2013 15:19:41 jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
18-Mar-2013 15:19:41 jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
18-Mar-2013 15:19:44 org.jenkinsci.main.modules.sshd.SSHD start
INFO: Started SSHD at port 25133
18-Mar-2013 15:19:44 jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
18-Mar-2013 15:19:44 hudson.TcpSlaveAgentListener <init>
INFO: JNLP slave agent listener started on TCP port 41790
18-Mar-2013 15:19:44 hudson.WebAppMain$2 run
INFO: Jenkins is fully up and running

– that looks happy enough, and as you can see from the line “HTTP Listener started: port=8080” it’s running on the default port, so connect to http://yourhost:8080 and you should see something like this…

1_newJenkins

the next step is to install the Swarm Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin) on this Jenkins Master instance so that Swarm Clients can connect to it.

Do this by going to “Manage Jenkins > Manage Plugins > Available” then selecting to install the “Swarm Plugin“.

Once that’s done you should see that the plugin has been installed…

2_installPlugin
Now that your new Jenkins server is set up and ready, hop over to your other Jenkins Slave/Client host and do the following…

mkdir for the swarm client

mkdir swarmclient; cd swarmclient/

Get the Swarm Client jar file from the ‘net

curl -O http://maven.jenkins-ci.org/content/repositories/releases/org/jenkins-ci/plugins/swarm-client/1.8/swarm-client-1.8-jar-with-dependencies.jar

Start up the Client

java -jar swarm-client-1.8-jar-with-dependencies.jar
Found 1 eligible Jenkins.
Connecting to http://mydomain.com:8080/
Attempting to connect to http://mydomain.com:8080/ a2721b16-04e4-0d962
18-Mar-2013 15:33:22 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Required credentials not available for BASIC <any realm>@mydomain.com:8080
18-Mar-2013 15:33:22 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Preemptive authentication requested but no default credentials available
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Hudson agent is running in headless mode.
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://mydomain.com:8080/]
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to myhost.mydomain.com:43932
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
18-Mar-2013 15:33:23 hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

Now take a look at your browser and you should see a new Node automatically added to the Master Jenkins instance…
3_slaveConnected

 

A very handy and flexible approach to adding/managing Nodes and workload – many thanks to the developers behind this!

Cheers,

Don

 

Managing Jenkins as a service and starting at boot time

Linux Services

I have an Ubuntu Linux VM that runs Jenkins, and to make life simpler I wanted to set up Jenkins to run as a service. I also wanted it to start up automatically at boot time, as that’s the sole function of the VM it’s running on.

So, here are my notes on setting up Jenkins as a service on Ubuntu Linux, which includes a script to manage (start, stop, check, restart) the Jenkins process too.

On Ubuntu (and most othe rLinux versions) you can check the current services with the command “service –status-all” – this should give you a list of all services, and their current status. You can (as root) also do “service <name> start” (or stop or restart) for each one.

Jenkins as a Service

To create a new service for Jenkins, take a look at the existing scripts in /etc/init.d/ for some examples.

By convention there are three main methods a basic service implements, they are; start, stop and restart. About the most basic structure of a service script is therfore something like this:

#!/bin/bash
start() {
echo “Starting Service”
# Do start things here.
}stop() {
echo “Stopping Service”
# Do stop things here.
}restart() {
echo “Restarting Service…”
stop
start
}

case “$1″ in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)

obviously you still need to implement the start and stop functions though 🙂

To see a far more detailed example, take a look at the file /etc/init.d/skeleton, which you could copy and update to suit your needs if you prefer.

I decided on something halfway between the very basic example above and the far more complex example in “skeleton”, and did something like this (saved as “/etc/init.d/jenkins”):

# Example Jenkins auto-start service script
#
# description: manages Jenkins as a service
# processname: jenkins
# pidfile: /var/run/jenkins.pid
# author: www.DonaldSimpson.co.uk# The user and the home dir that Jenkins runs under
jenkins=/usr/local/jenkins
# Your startup and stop scripts (see below)
startup=$jenkins/bin/startup.sh
shutdwn=$jenkins/bin/shutdown.shstart_jenkins() {
echo “Starting Jenkins services…”
su – jenkins -c “sh $startup”
}

stop_jenkins() {
echo “Stopping Jenkins services…”
su – jenkins -c “sh $shutdwn”
}

status_jenkins() {
# Check for any other process containing jenkins.war
# This could be improved upon (see script below)
numproc=`ps -ef | grep [j]enkins.war | wc -l`
if [ $numproc -ne 0 ]; then
echo “Jenkins is running…”
else
echo “Jenkins is NOT running…”
fi
}

case “$1” in
start)
start_jenkins
;;
stop)
stop_jenkins
;;
status)
status_jenkins
;;
restart)
stop_jenkins
start_jenkins
;;
*)
echo “Usage: $0 {start|stop|status|restart}”
exit 1
esac
exit 0

Update that to suit then save and change the permissions to make it executable:

chmod +x /etc/init.d/jenkins

then you can check (as root) that you can call the methods in the script:

service jenkins status

service jenkins stop

service jenkins start

These should all now run as the Jenkins user via sudo and say something when called, even if it is just “I can’t find the scripts you told me to call” 🙂

Jenkins Startup and Stop scripts

So, you now need to create the missing startup and shutdown scripts, in my example they were:

/usr/local/jenkins/bin/startup.sh

and

/usr/local/jenkins/bin/shutdown.sh

 

An example of the Jenkins start-up and management script I normally use is at the end of this post – the main idea though, is that it sets what you want to set then starts Jenkins via nohup and outputs to a log file

I have also added and included some basic tests to my scripts and some (very) rudimentary error handling/checking, but you shoudl get the idea and all you really need is this line (with the variables set correctly):

${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &

Note: I often run multiple instances of Jenkins, so I explicitly specify the HTTP_PORT, and I use the -D${MARKER} to allow me to safely and easily find the correct PID for each project.
The httpListenAddress can normally go unset, but it’s something I’ve had to set before on multi-home’d hosts.

The stop part of my Jenkins management script finds the correct PID like this (and you could use a filter for the correct -D${MARKER} if you want that too):

PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`

and simply kills it.

For some of my scripts I also filter for process ID’s that were started in the current directory by checking “pwdx” against the PID, but only where I’m sure the corresponding start-up process is correct/reliable.

So, tweak to your taste and that should be the Jenkins service created and working too now; for start|stop|restart you can create individual scripts or wrappers that call something like the one script below with parameters, or break the script in to separate files if you prefer.

Setting Jenkins to start at boot time

If you want to start Jenkins at boot time/startup automatically then you still need to do one more small step.  There are many different ways to do this depending on personal preference, your requirements and your version of Linux, but on Ubuntu it can be done easily with:

 

update-rc.d jenkins defaults

 

“update-rc.d” simply installs and removes System-V style init script links – read the man page for full details, but the same idea applies to most versions of Linux.

 

update-rc.d -f  jenkins remove

 

will undo this if you no longer want it.

An example Jenkins server/process management script

As an example, here is my script to manage Jenkins processes – it could use some improvements but the basic start, stop, restart and check should give you enough to sort out something that works and suits your needs.

#! /bin/bash -p
# www.donaldsimpson.co.uk
# Script to start|stop|restart|check an instance of Jenkins
# For each new instance, the PROJECT and HTTP_PORT need to be updated:
export PROJECT=jenkins
export HTTP_PORT=9000
EXIT_STRING=””
JENKINS_ROOT=/opt/apps/jenkins
export JENKINS_HOME=${JENKINS_ROOT}/${PROJECT}
JENKINS_WAR=${JENKINS_HOME}/jenkins.warWAIT_TIME=5
START_WAIT_TIME=15
JAVA_HOME=/usr
PATH=$JAVA_HOME/bin:$PATH
JAVA=${JAVA_HOME}/bin/java
NOHUP=/usr/bin/nohup
LOG_FILE=${JENKINS_HOME}/debug_${PROJECT}.log
MARKER=”JenkinsProcFor_${PROJECT}”NC=/bin/nc
WGET=/usr/bin/wget
LSOF=/usr/bin/lsof
AWK=/usr/bin/awk
GREP=/bin/grep
FUSER=/bin/fuser
#################################################################################
### Functions Start #############################################################
#################################################################################

cleanup(){
# Perform any clean-up activities here…
[ “${EXIT_STRING}” != “0” ] && echo `date “+%d/%m/%y %H:%M:%S”` ${EXIT_STRING}
exit 0
}

trap ” QUIT

_error() {
EXIT_STRING=”${1}”
[ “${1}” != “0” ] && EXIT_STRING=”ERROR: ${1}, please investigate, terminating.”
cleanup  # Never returns from this call…
}

say(){
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
}

saybold(){
tput bold 2>/dev/null
echo `date “+%d/%m/%y %H:%M:%S”` “${1}”
tput sgr0 2>/dev/null
}

check_folders(){
say “”
saybold “Checking all required folders exist…”
for REQUIRED_DIR in ${JENKINS_ROOT} ${JENKINS_HOME} ${JAVA_HOME}
do
[ ! -d “${REQUIRED_DIR}” ] && _error “Necessary directory: ${REQUIRED_DIR} does not exist”
say “Found required directory: ${REQUIRED_DIR}”
done
saybold “Done.”
say “”
}

check_files(){
say “”
saybold “Checking all required files exist…”
for REQUIRED_FILE in ${JENKINS_WAR} ${NC} ${WGET} ${JAVA} ${LSOF} ${FUSER} ${GREP} ${AWK}
do
[ ! -f “${REQUIRED_FILE}” ] && _error “Necessary file: ${REQUIRED_file} does not exist”
say “Found required file ${REQUIRED_FILE}”
done
saybold “Done.”
say “”
}

check_port_closed(){
say “Checking that port ${HTTP_PORT} is closed…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
_error “Required Jenkins port ${HTTP_PORT} is already in use”
else
say “Ok, required port ${HTTP_PORT} is available, continuing…”
fi
}

check_port_open(){
say “Checking that port ${HTTP_PORT} is open…”
${NC} -w 1 localhost ${HTTP_PORT}
if [ $? -eq 0 ]; then
say “Ok, a process is listening on port ${HTTP_PORT}, continuing…”
else
_error “Required Jenkins port ${HTTP_PORT} has not been opened.”
fi
}

start_process(){
cd ${JENKINS_HOME}
saybold “Starting Process now…”
${NOHUP} ${JAVA} -jar ${JENKINS_WAR} -D${MARKER} –httpListenAddress=0.0.0.0 –httpPort=${HTTP_PORT} > ${LOG_FILE} &
say “Process initiated.”
}

check_log(){
say “Checking the log file ${LOG_FILE} for an HTTP listener…”
STARTED=`${GREP} -c “HTTP Listener started” ${LOG_FILE}`
if [ ${STARTED} -eq “0” ]; then
_error “An HTTP Listener has not reported as started in the log file ${LOG_FILE}”
else
saybold “An HTTP Listener is reported as started in the log file ${LOG_FILE}”
fi
}

check_html(){
# These checks need error handling, but you get the general idea.
say “Checking that localhost:${HTTP_PORT} is serving Jenkins pages…”
TEMP_WGETDIR=tempwgetdir$$
mkdir ${TEMP_WGETDIR}
cd ${TEMP_WGETDIR}
${WGET} -q http://localhost:${HTTP_PORT}
GOT_HTML=`${GREP} -c Dashboard index.html`
cd ${JENKINS_HOME}
rm -rf ${TEMP_WGETDIR}
if [ ${GOT_HTML} -eq “0” ]; then
_error “Unable to get an HTML page from the Server.”
fi
saybold “Recieved valid HTML from the server, all looks ok.”
}

check_process(){
check_port_open
check_log
check_html
say “A Jenkins instance is listening on port ${HTTP_PORT} for project ${PROJECT}.”
say “The process is logging debug info to the log file: ${LOG_FILE}”
}

stop_proc(){
check_port_open
saybold “Looking for the Process ID attached to port ${HTTP_PORT}”
PID=`${LSOF} -w -n -i tcp:${HTTP_PORT} | ${GREP} -v COMMAND | ${AWK} {‘print $2’}`
if [ “${PID}” == “” ]; then
saybold “Unable to detect the Process ID that is listening on port ${HTTP_PORT}!”
PID=`${FUSER} ${LOG_FILE}`
if [ “${PID}” == “” ]; then
_error “Unable to find the PID that has the log file open too!”
else
say “Ok, found PID ${PID}”
fi
else
saybold “Found a PID of $PID, killing it now…”
fi
kill -9 ${PID}
say “Waiting ${WAIT_TIME} seconds for the process to die…”
sleep ${WAIT_TIME}
saybold “Done, checking port is closed…”
check_port_closed
saybold “All done.”
}

start(){
check_folders
check_files
check_port_closed
start_process
saybold “Waiting ${START_WAIT_TIME} seconds for the process to start up…”
sleep ${START_WAIT_TIME}
check_process
}

restart(){
stop_proc
start
}

#################################################################################
### Script Start ################################################################
#################################################################################

case “$1” in
start)
start
;;
stop)
stop_proc
;;
restart)
restart
;;
check)
check_process
;;
*)
echo “Usage: $0 {start|stop|restart|check}”
esac
# Exit cleanly
_error “0”

Hope that helps! Any constructive comments, requests or suggestions for improvement are very welcome 🙂

Cheers, and sorry about the indenting,

Don