Setting up Perforce on a Linux Server and a Windows Client

As described on WikiPedia, Perforce “is a commercial, proprietary, centralized revision control system developed by Perforce Software, Inc.”

Like Subversion, it’s a client/server system where the server manages a database of revisioned files, and clients connect to checkout, modify and send back changes for others to pick up.

I wanted to check out the latest version so thought I’d install it on this server and set up a client… and that I may as well capture the steps and put them here.

In my case, the server is an Ubuntu Linux  host, and my client machine is a Windows XP workstation.

There’s not a lot to do when installing Perforce, so getting a straightforward instance up and running is a breeze. Basically, get the binary, export or pass it a few settings if you don’t want the defaults, then kick it off – here’s the detail from my notes:

Download p4d binary (for this platform and architecture):
wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4d

You can put this wherever you want, and set up a P4ROOT to specify the Perforce root directory – but don’t use that location for anything else (including client workspaces).

mkdir /apps/perforce
mv p4d /apps/perforce; cd /apps/perforce
chmod +x p4d

 

Most  Perforce options can either be exported or passed as a command-line arg, so you can choose

export P4ROOT=/apps/perforce
– or –
-r /apps/perforce

The default port is 1666, and remember that if you change this on the Server you will need to change it on your Perforce client(s) too.  In my example I’m using 9002:

export P4PORT=9002
– or –
-p 9002

So I ended up with a command line that looked like this:

nohup ./p4d -r /apps/perforce -J /var/log/journal -L /var/log/p4err -p 9002 &

Which I will probably put in to a simple startPerforce.sh script, and a probably a stopPerforce.sh script too that contains this and maybe the port number and full path to the binary location:

p4 admin stop

That’s it for the Server side at the moment, there’s a process up and running (you can check the output in nohup.out) so it’s time to set up and connect a client…

 

I’m going for a Windows client installation, which means downloading the correct version from the Perforce site then running p4vinst.exe. There’s nothing really to report here; select the usual options like directories and let it install.

Oh, I also needed to create a user, so back on the Linux Server I also downloaded the p4 binary:

wget http://www.perforce.com/downloads/perforce/r10.2/bin.linux26x86/p4

Exported the P4PORT (since I was using a custom one to get through my firewalls), then add a user:

export P4PORT=9002
chmod +x p4
./p4 user -f Donald
User Donald saved.

Now you can test connecting to your Perforce Server with the  P4Admin GUI and the P4V GUI on the Windows client host by specifying the correct port (if you changed the default) and a valid user name. Once that’s done, you can admin your Depot’s and add/change/commit files, see revision and history information and all that good stuff. There are also command line and web interfaces too which can be useful for temporary use and for scripting but the Windows GUI’s are nice to use and quite self explanatory – if you’ve used a similar revision control system like Subversion and an Eclipse-like IDE before there’s not much of a learning curve here.

The Perforce Help and Documentation is all very good, and their Perforce 2010.2: System Administrator’s Guide covers all of the above in more detail, and touches on more advanced topics too – Perforce Performance Tuning, Backup and Recovery, Replication, and the Perforce Broker (P4Broker) etc.

There’s also a Perforce plugin for Jenkins, which once installed allows you to choose Perforce as one of the SCM option in your Jenkins jobs, but the above hopefully covers the initial setup of both the Perforce Client and the Perforce Server on Windows and Linux respectively.

 

Some PHP examples

I recently wrote a couple of PHP Pages for my site:

UK Area Code Search which searches my database for a specified full or partial area code or town

and

Crossword Solver  which searches for possible matches to a partial word.

It’s been a while since I’d done any PHP (all of the recent web-dev stuff I’ve written has been either JSP, Python or CGI) so I thought I’d keep some notes on my own ‘refresher course’ and do a brief write up of the main steps involved.

Both of these apps are basically quite similar; they take some user input, search in a database, then display the results on a web page.

For tasks that need done repeatedly, like sanitising user inputs, it’s worth creating a simple function:

function cleanvar($input){
if (strlen($input) > 1){
$input = ‘_’;
}
return $input;
}

this allows you to quickly create, populate and sanitise a variable in one go like so:

$mynewvar = cleanvar($_POST[‘userselection’]);

When the page loads, you can check if there is anything to process or not by looking at the “submit” element:

if(isset($_POST[‘submit’]))
{
# do posty type things…
}

iterate through and clean up all passed parameters:

$myquery = “”;
foreach($_POST as $vblname => $value)
$myquery = $myquery . $value;

with some text replacements:

$myquery = str_replace(“Unknown”,”_”,$myquery);
$myquery = str_replace(“Search”,””,$myquery);

alternatively you could use the Request object to get each passed var explicitly, e.g. $_REQUEST[‘myparam’]

Connecting to a database is very nice and easy in PHP:

$con = mysql_connect(“myservername”,”myusername”,”mypassword”);
if (!$con)
{
die(‘Could not connect: ‘ . mysql_error());
}
mysql_select_db(“myschema”, $con);

Once connected, execute a query – I use a hard LIMIT to avoid returning all data:

$result = mysql_query(“SELECT lcase(word) as word FROM mytable where word like ‘$myquery’ LIMIT 0, 200”);

you could change the LIMIT parameters to create “paging” for your results, so the next page would show

LIMIT 200, 400

and so on.

Check for results and iterate through them:

while($row = mysql_fetch_array($result))
{
$counter++;
echo “Found ” . $counter . ” records: ” . $row[‘word’] . “”;
# etc etc
}

remember to close the MySQL connection when done:

mysql_close($con);

And that’s about it – some sanity checking and error handling is needed, plus outputting the HTML part, but for a quick and simple PHP page that takes user input, queries a database and shows results, the above steps should do the job.

As I’m using WordPress I wanted to get my PHP pages looking like they “belong” (getting my custom PHP pages to use the current WordPress Theme and CSS etc); there are several solutions for this like WordPress plugins for custom PHP pages and creating custom WordPress Templates. For now I have just included my PHP examples in an iFrame and explicitly use the site’s CSS to make them fit in, but I’d like to investigate what works best for me and sort this out “properly” at some point.

 

 

Jenkins Slave Nodes

 

 

This Jenkins Slave Nodes post covers:

  • What are they?
  • Why may I want one?
  • How do you create one?
    • tasks on On the Master/Server
    • tasks on On the Slave/Client
  • Other ways of creating Slave Nodes
  • Related posts and links

What are they?

Jenkins Slave Nodes are small Java “Client” processes that connect back to a “Master” Jenkins instance over the Java Network Launch Protocol (JNLP).

Why may I want one?

Once it’s up and running, a Slave instance can be used to run tasks from a Master Jenkins instance on one or more remote machines providing an easy to use and flexible distributed system which can lend itself to a wide variety of tasks.

As these are Java processes, you are not restricted by architecture and can mix and match the OS’s of your slave nodes as required – Windows, Linux, UNIX, iSeries, OVMS etc – anything capable of running a modern version of Java (I think JNLP was introduced in 1.5?) and you can also group and categorize subsets of different types (both logical and physical) of Slaves; intended use, availability, location, available resources, Cloud or VM versus Physical tin – anything that helps you decide when you want to use which host.

There are many different ways you can choose to utilize these nodes – they can be used to spread the load of an intensive build process over many machines when they are available, you can delegate specific tasks to specific machines only, or you can use labels to group different classes or types of Nodes that are available for certain tasks, making the most use of your available resources. You can also have Jenkins create Cloud server instances – Amazon EC2 for example – when certain thresholds are reached, and stop them when they are no longer required.

This post focuses on a pretty manual approach to the creation of Jenkins Slave Nodes with the intention of explaining them well enough to allow you to create them on any platform that can run a modern version of Java – there are probably simpler solutions depending on your needs and setup. A later post will touch on a few of the many possible uses for these nodes.

So, how do you create one?

There are several different ways to go about setting up a Slave, and the “best” approach depends on your situation, needs and environment(s) – for a simple Linux setup letting Jenkins do all the work for you makes life really easy, you can just select that option in your new Jenkins Slave Node and complete this screen to have Jenkins set it up for you:

Where the Username and Password are the credentials you want Jenkins to use to connect and start the Slave process on the remote server. This simple approach also allows the Master instance to initiate the JNLP connection and bring your slaves online when required, avoiding any need to manually visit each slave node.

This keeps things nice and simple and reduces teh admin overhead too,  but sometimes this type of approach can’t be used (on different OS’s like OVMS, iSeries, Windows etc) and I’m going to go on to outline what I think is the most “versatile” method – defining the Node on the Master instance, and manually setting up and starting the corresponding Slave/Client Node on the remote host – going through these steps should provide enough detail on how Slave Nodes work and connect to get one up and running on anything that can run a JVM.

1. On the Master/Server

Define the host: navigate to Jenkins > Manage Jenkins > Manage Nodes > New Node
Enter a suitable Node Name (I’d recommend something descriptive, and usually including the host name or part of it) then either select to create a “Dumb Slave” or copy an existing Node if you have one, then complete the configuration page similar to this:


where you specify the requested properties – path, labels, usage, executors etc. These are explained in more detail in the “?” for each item if required.

Here you can also state if you want to keep your Jenkins Slave for tied jobs only, or if it is to be utilized as much as possible – this obviously depends on your requirements. You can also specify the Launch method that best applies to your needs & requirements.

2. On the Slave/Client

You don’t need to do very much to create a new slave node – typically if I’m setting up a few *NIX and Windows hosts I would archive a simple shell/DOS script that starts and manages the process along with the slave.jar file from the Master Jenkins instance. There are alternative methods that may suit your needs – you can start slaves via SSH from the Master server for example and there’s a comparable method for Windows – but this simple approach should help you understand the underlying idea that applies to them all.

You can “wget” (or use a Browser on Windows) the slave.jar file directly from the Master Jenkins instance using the URL

http://[your jenkins host]:[port number]/jnlpJars/slave.jar

If you let JNLP initiate the process, the slave.jar will be downloaded from Jenkins automatically.

Note that Jenkins will inherit the effective permissions of the user that starts the process – this is to be expected, but it’s often worth having a think about the security aspects of this, along with the access requirements for the types of things you want your slave to be able to do… or not do.

On Windows hosts, you can use the jenkins-slave.exe to easily install Jenkins as a Windows Service, which can then be started at boot time and run under whatever user/permissions you wish set via the Services panel.

My *NIX “startslave.sh” script does a few environment/sanity checks, then kicks of the slave process something like so:

${NOHUP} ${JAVA} -jar slave.jar -jnlpUrl http://SERVERNAME:PORT/computer/USER__NODENAME/slave-agent.jnlp &

The HTTP URL there should match the one provided by the Jenkins page when you were defining the Node. If all goes well you should see the node state changed to Connected on the Master Hudson instance, and if not, then nohup.out should provide some pretty obvious pointers on the problem.

Some common causes are:

Jenkins host, port or node name wrong
Java version not found/wrong
Lack of write permissions to the file system
Lack of space (check /tmp too)
Port already in use
Errors in the jenkins-slave.xml file if you’ve tweaked it
Firewalls…

Jenkins also provides some health monitoring of the connected Node which you can see in the Jenkins > Nodes page:
Disk Space, Free Temp Space, Clock time/sync, Response Time and Free Swap are monitored
and you can have your Node taken off line if any of these passes a set threshold.

This should hopefully be enough info to provide an overview of what Jenkins Slaves are, and enough to get one up and running on your chosen platform. Where possible it’s best to keep things simple – use SSH and let the Master instance manage things for you if you can – but when that’s not possible there are alternatives.

When I get the chance I will add some information on the next steps – creating and delegating jobs on Jenkins Slave Nodes, and some thoughts and suggestions for just a few of the many uses for this sort of distributed Build and Deployment system.

Related Posts and Links:

Monitoring Jenkins Slave Nodes with Groovy
– how to keep an  eye on your Jenkins Slaves

Jenkins Slave Nodes – using the Swarm Plugin
– automatically connect new Slave Nodes to create a “Swarm”

Getting the current user in Jenkins
– several approaches

Managing Jenkins as a service and starting at boot time
– on Linux & Windows

Jenkins plugins
– details on some of my most frequently used plugins

Jenkins DIY Information Radiators
– what they are for, and how to make your own

The Jenkins Wiki has more details information on Distributed Builds and different slave-launching strategies.

Feedback, questions and constructive comments are very welcome!