Category: Linux

CCTV with Tenvis cameras and ZoneMinder

This post details the processes I went through to create my own DIY home CCTV system.

Topics covered include:
1. Hardware – some cheap but impressive Tenvis TH692 720p IP cameras, and some Power over Ethernet (PoE) injectors and extractors to go with them
2. Camera setup – how to set them up, connect to them, and a quick summary of basic functions
3. Clients – info on a few different ways to attach to and use the cameras – VLC, Kodi/XBMC, RTSP and the built-in app and web interfaces
4. Jenkins – using Jenkins jobs to capture and record from Tenvis cameras
5. ZoneMinder – installation, OVA and manual install, settings used
6. Summary, links and general info

1. Hardware
On recommendation from a friend, the cameras I went for are these:
Tenvis TH692’s
“720P HD Outdoor Network Wireless CCTV IP Camera with 15M Night Vision”

these cameras are currently available on Amazon for only £27 each!

The cameras can happily run over WiFi but as they will still need a power connection, I have opted to run them over Ethernet and to send the power over the CAT6 cable too – this way there’s still only 1 cable required, and I get a faster network connection too.

To do this I have used these:
AKORD® POE Passive Power Over Ethernet Adapter Injector Extractor Kit

These clever little beasties work with the power adapter that comes with the Tenvis TH692’s, and come complete with both a PoE Injector and Extractor, for only £3.99 – another mega-bargain! I haven’t tested them for outdoor use in bad weather yet, but suspect they may require some protection from the elements, which is fair enough.

2. Camera Setup
Connecting the cameras to your home network and getting them up and running is pretty easy. You need to connect them wired initially and use DHCP to assign an address. With that done, you can then use the supplied software to find, connect to and configure the cameras. After that’s complete, you can connect them to your WiFi, change the name/label for the camera, set up users and passwords, set up Email and FTP alerts and settings and so on.

3. Clients
I found the supplied software sufficient for the initial setup, and the phone app (search for “NEW Tenvis” in the App store) works very well, allowing you to monitor your camera(s) from anywhere in the world assuming you’ve got an internet connection at both ends. Here’s a picture from my iPhone:

iphone_tenvis

 

The web interface relies on browser plugins and didn’t work on my Mac under Chrome, Firefox or Safari – it wanted an out dated QuickTime plugin which I couldn’t get working, though I confess I didn’t try too hard. It worked ok on my Windows VM, but I don’t want to use that interface or that OS. Luckily there are plenty of alternative options though, as these cameras use RTSP…

“The Real Time Streaming Protocol (RTSP) is a network control protocol designed for use in entertainment and communications systems to control streaming media servers. The protocol is used for establishing and controlling media sessions between end points.”

[ Source: Real Time Streaming Protocol – Wikipedia, the free encyclopedia ]

This opens up several options for connecting to the cameras, and means that you don’t need to rely on the supplied software and interfaces. For me, this is what makes these cameras so good.

Here are the solutions I use, though there are many more available…

VLC – VideoLAN – as you’ll probably know, this great free and open source cross-platform multimedia player plays pretty much anything, and on pretty much every platform.  Not surprisingly, I found I could point this player at the cameras RTSP feed, enabling me to view the video content from all devices that VLC runs on.

I use this approach on my Mac laptop mostly, and it’s as easy as creating a small config file for each camera feed then clicking on it to open the live feed. The files can be saved with “.m3u” extensions, as long as you’ve set that file type to be handled by VLC.

For example, here are the contents of the “cctv_driveway.m3u” file I currently have on my OSX Desktop, and that I click to connect to that feed:

#EXTM3U
#EXTINF:0, Driveway CCTV
rtsp://USERNAME:YOURPASSWORD@192.168.0.151:554/1

that’s it – just 3 lines.

Line 1, “#EXTM3U” is the file header which must be the first line of the file – like a Bash “shebang”.

Line 2, “#EXTINF:0, Driveway CCTV” contains the track information (just a zero here) and the title of the feed. This is displayed as “Driveway CCTV” in the VLC Window title, which is a handy feature.

Line 3, ” rtsp://USERNAME:YOURPASSWORD@192.168.0.151:554/1″ is simply the RTSP URL for the camera feed you want to stream from.

The RTSP URL contains the protocol (rtsp://), then user and password details, then the address of the camera (192.168.0.151 in this case), which is followed by the port the feed is served on: 554. This port can be seen in the camera config during the initial setup, but if you are unsure you can run a simple nmap scan against your camera like this:

nmap

Here we can see port 80 and 8080 are open for the web interfaces (viewing and configuration respectively), and 554 which is the standard RTSP port.

This useful web page can also generate the correct RTSP URL for many popular cameras:
Tenvis IP camera URL

The final part of the URL is the endpoint to connect to on the remote camera/host – you can see in the config above that I am connecting to “/1” at the very end of the third line in my M3U file; this is the location for the full 720 HD feed for these particualr cameras. There are also lower resolution feeds available which can also be useful to know about, especially when monitoring multiple cameras or connecting remotely (e.g. with lower bandwidth).

For these Tenvis cameras, changing to the “/12” endpoint will fetch the lower quality feed, and there are other options inbetween that you can use to suit your requirements. These end points can also be modified further through the Tenvis settings app (which is running on port 8080).

Kodi (formerly XBMC) – from a quick google it looks like there are several ways in which Kodi can be set up to consume and view RTSP feeds. The simple option I’ve gone for is, again, to create a tiny config file containing the settings for each camera, and to place these files on my NAS storage. This means that watching a camera live on my TV is as simple as selecting the corresponding file in Kodi, and it will launch the stream just like you had clicked on a movie.

The files I use have the “.strm” extension and simply contain the full URL for the RTSP stream:

rtsp://user:password@192.168.0.156:554/1

Using this simple approach, I can click on files like “cctv_driveway.strm” in Kodi to launch the various streams. Because I only ever use this on my TV or Projector, I go for the full 720 HD feed in these files via the “/1” end points.

4. Jenkins

Disclaimer: I have a tendency to use Jenkins to automate everything. 
Sometimes this extends to things that don't really need it, just to see if/how it can be done. 
This section and idea is driven from that personal tendency/obsession.
The ZoneMinder solution (described below) is by far the more sensible option for most cases :-)

After setting up some cameras and connecting to them, I then wanted to record and archive the footage. The provided software enables you to set up FTP archiving and email alerts, but I wanted to do something more flexible, that would allow me to easily change & tune the retention, housekeeping and archiving. The approach I used is slightly unusual, but it’s very simple, effective and flexible, allowing me to easily tweak things to suit my requirements.

To use Jenkins for recording and managing my CCTV Camera feeds, I went through the following high-level steps:

1. Create a new ‘Freestyle’ Jenkins job, set to run on my Ubuntu host

2. Add an ‘Execute shell’ step. To this I added the following shell commands:

export MY_DATE=`date +"%Y%m%d%H%M%S"`
rm -f *.ts
/usr/bin/vlc -vvv rtsp://USER:PASSWORD@192.168.0.151:554/12 --sout=file/ts:/home/don/cctv/recording-${MY_DATE}.ts -I dummy --stop-time=1800 vlc://quit
mv /home/don/cctv/recording-${MY_DATE}.ts .

This is cleaning up any previous/old files then capturing 30 minutes of output from the camera via VLC, writing the data stream to a file. After 30 minites VLC quits, and I move the newly captured file to the current working directory with a timestamp in the filename.

3. Archive files
After the shell command above is complete, I have configured the Jenkins job to archive the captured file along with this job run. This makes it nice and easy to browse through previous (date & timestamped) jobs and simply click to view the corresponding video capture from that time.

4. Create a Jenkins job loop
At the end of every 30 minute run, I set the “Build other projects” option for this build to trigger another run of this same job, creating an infinite loop of 30 minute runs. There’s a tiny pause between the job ending and the next build starting, but it’s only a second or two at most, which I can live with.

Once I was happy that the data was being captured and archived ok, I was then able to configure and tune the retention through Jenkins – there are loads of Jenkins built-in options that enable you to do things like ‘keep the last x builds’, or ‘keep builds for n days’, or whatever you would like. You can also mark certain builds as ‘keep forever’ if you wanted to preserve anything interesting.

This process works well for me, and the CPU and memory usage created from having 3 of these jobs running constantly is, to my surprise, next to nothing; thanks to the impressive efficiency of VLC.

The disk usage is the main issue here; with this approach I’m constantly recording, and you can fill up a LOT of disk by writing several HD video streams to disk! One plan I was considering is to reencode video footage at a lesser bitrate (to reduce the file size) as they get older (using another Jenkins job), but I think that may be over-kill: for me, 2 weeks retention with the ability to archive/keep anything I want to quite easily is more than enough really.

5. ZoneMinder
Nearly every search I did when looking for software to manage my new CCTV cameras led me to the same place – https://zoneminder.com/

Like VLC, Kodi and Jenkins, ZoneMinder is a fantastic bit of software; it’s free, there’s loads of documentation, and it’s extremely configurable. For managing CCTV video recordings I’ve not yet seen anything that compares to it, even if you are willing to spend serious money.

Initially I tried installing everything in a ready-made VM Template – an OVA file – I think it was this one:
http://blog.waldrondigital.com/2012/09/23/zoneminder-virtual-machine-appliance-for-vmware-esxi-workstation-fusion/

This is a great solution and can be a real timesaver to get you up and running, especially if you don’t have a VM with Ubuntu and a LAMP stack to hand. It took something like 2 minutes to deploy this to my ESX server, and it was working a few minutes after that. The software was out of date with the VM I downloaded and deployed, but there are clear and easy instructions on that page explaining how to update to the latest versions.

I decided I didn’t want the overhead of running another VM just for this one function, and as I already have a few running I looked in to installing ZoneMinder from scratch on an existing Ubuntu VM, which is actually pretty easy as detailed here:

http://zoneminder.readthedocs.io/en/stable/installationguide/ubuntu.html

This went quite smoothly, I had to do a couple of MySQL tweaks but it took about 20 minutes from start to finish, and I ended up with ZoneMinder running on an existing Ubuntu host which will mean less update and maintenance grief for me (as oppposed to running a separate and dedicated VM just for ZoneMinder).

It took a little experimenting to get the Tenvis TH692 cameras working in ZoneMinder, but nothing complex – here’s what I used for the “General” settings with the Tenvis TH692 cameras:

ZoneMinderGeneral

and here are the “Source” settings for the RTSP Stream, using the same basic details we’ve used to set up VLC, Kodi etc previously:

ZoneMinderSource

Once that’s done, you can tweak the settings to your liking. You can have ZoneMinder record events as they happen and archive them, and/or use it to act as a nicer web interface to your cameras. You get the option to cycle through your different cameras automatically, or you can watch several feeds on one page – the options and possibilities are great.

One of the main points of using ZoneMinder for me is that it serves the camera feeds to the browser without the need for plugins like QuickTime, and it works well on all operating systems I’ve tried – and all devices.

Note that it’s advisable to set up a ZoneMinder Filter to archive your old footage – preferably before your disks get full!

This link explains how to do this in a variety of ways:

http://zoneminder.readthedocs.io/en/latest/faq.html

After some inital experimenting I have gone for both a “Purge after x days” filter and a “Purge when disk over 50% full” – both types of Filter are detailed in that FAQ.

Summary

I can now connect to all of my cameras from all of my devices – my Nexus tablet, mobile phone, Mac and Linux computers, television, projector – quickly and easily, and from anywhere. I can also monitor, record, replay and generate alerts whenever they are required, and tune each camera to suit my needs. I think these cameras are a total bargain, the HD picture quality is excellent, and the night time IR is good too. If you are happy to set up your own connectivity and monitoring solutions like ZoneMinder (or Jenkins) you can quite easily create a sophisicated system for very little cost, and it’s good fun too!

 

Extending Jenkins book

My new book, Extending Jenkins by Donald Simpson, has been published!

Extending Jenkins

There is a free sample chapter available here:
Chapter 8 – Testing and Debugging Jenkins Plugins

You can buy the full book in either electronic or paperback format direct from the publishers or through Amazon here in the UK or Amazon in the US

About This Book

  • Find out how to interact with Jenkins from within Eclipse, NetBeans, and IntelliJ IDEA
  • Develop custom solutions that act upon Jenkins information in real time
  • A step-by-step, practical guide to help you learn about extension points in existing plugins and how to build your own plugin

Who This Book Is For

This book is aimed primarily at developers and administrators who are interested in taking their interaction and usage of Jenkins to the next level.

The book assumes you have a working knowledge of Jenkins and programming in general, and an interest in learning about the different approaches to customizing and extending Jenkins so it fits your requirements and your environment perfectly.

Table of Contents

1: Preparatory Steps
2: Automating the Jenkins UI
3: Jenkins and the IDE
4: The API and the CLI
5: Extension Points
6: Developing Your Own Jenkins Plugin
7: Extending Jenkins Plugins
8: Testing and Debugging Jenkins Plugins
9: Putting Things Together

What You Will Learn

  • Retrieve and act upon Jenkins information in real time
  • Find out how to interact with Jenkins through a variety of IDEs
  • Develop your own Form and Input validation and customization
  • Explore how Extension points work, and develop your own Jenkins plugin
  • See how to use the Jenkins API and command-line interface
  • Get to know how to remotely update your Jenkins configuration
  • Design and develop your own Information Radiator
  • Discover how Jenkins customization can help improve quality and reduce costs

In Detail

Jenkins CI is the leading open source continuous integration server. It is written in Java and has a wealth of plugins to support the building and testing of virtually any project. Jenkins supports multiple Software Configuration Management tools such as Git, Subversion, and Mercurial.

This book explores and explains the many extension points and customizations that Jenkins offers its users, and teaches you how to develop your own Jenkins extensions and plugins.

First, you will learn how to adapt Jenkins and leverage its abilities to empower DevOps, Continuous Integration, Continuous Deployment, and Agile projects. Next, you will find out how to reduce the cost of modern software development, increase the quality of deliveries, and thereby reduce the time to market. We will also teach you how to create your own custom plugins using Extension points.

Finally, we will show you how to combine everything you learned over the course of the book into one real-world scenario.

Beginning Docker video course

Blog updates have been scarce recently as I have been busy working on a couple of publications… the first of which has just been released…

https://www.linuxjournal.com/node/1338951

This is a hands-on video course packed with practical examples to get you started with Docker.

Here is the course overview video:

And here is a free sample video from Section 2, “Docker Basics” where we take a look at running containers and the 3 different types of “containerized” commands:

and this final sample video is taken from Section 5 – “Running a Web Application with Docker“.

In this clip we build our own web application using Python, pip and Redis, which we will then “dockerize” and ship to “production”:

About This Video

  • Master Docker commands by creating and publishing a sample web application
  • Build and manage your own custom Docker Containers to set up data sources, filesystems, and networking
  • Build your own personal Heroku PaaS with Dokku

Who This Video Is For

If you’re a developer who wants to learn about Docker, a powerful tool to manage your applications effectively on various platforms, this course is perfect for you! It assumes basic knowledge of Linux but supplies everything you need to know to get your own Docker environment up and running.

What You Will Learn

  • Build new Docker containers and find and manage existing ones
  • Use the Docker Index, and create your own private one by using containers
  • Discover ways to automate Docker, and harness the power of containers!
  • Build your own Docker powered mini-Heroku Paas with Dokku
  • Set up Docker on your environment based on your application’s custom requirements
  • Master Docker patterns and enhancements using the Ambassador and Minimal containers

In Detail

One of the major challenges while creating an application is adapting your application to run smoothly on all of the plethora of operating systems available. Docker is an extremely efficient technology that allows you to wrap all your code along with its supporting files into a single bundle; it also guarantees that your application will behave in the same way on any host powered by Docker. You can also easily reuse existing Docker containers or create and publish your own. Unlike Virtual Machines, Docker containers are lightweight and more efficient.

Beginning Docker starts with the fundamentals of Docker—explaining how it works, how to set it up, and how to get started on leveraging the benefits of this technology. The course goes on to cover more advanced features and shows you how to create and share your own Docker images.

You will learn how to install Docker on your own machine, then how to manage it effectively, and then progress to creating and publishing your very own application. You will then learn a bit more about Docker Containers; built-in features and commands such as volumes, mounts, ports, and linking and constraining containers; before diving into running a web application.

Docker has functionality such as the Docker web API to handle complex automation processes which will be explained in detail. You will also learn how to use the Docker Hub to fetch and share containers, before running through the creation of your own Docker powered mini-Heroku

Beginning Docker covers everything required to get you up and running with Docker, with detailed real-world examples and helpful tips to make sure you get the most from it.

Style and Approach

An easy-to-follow and structured video tutorial with practical examples of Docker to help you get to grips with each and every aspect.

The course will take you on a journey from the basics to the advanced application of Docker containers, and includes several real-world scenarios to learn from.

Cheers,

Don

Jenkins and Docker – Part One

This post lists the steps taken to get started with Docker – the basic install and getting a Docker container up and running on Ubuntu.

There’s nothing new or unusual in this section, but it forms the background of the next post(s) where I plan to go in to detail on different approaches to using Docker with Jenkins (and vice versa).

If you’re unfamiliar with Docker here is a good introduction:

and you can try Docker out easily on the docker site: https://www.docker.com/tryit/

 

 

In my case the Ubuntu is a pretty ordinary “ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-35-generic x86_64)” running as a guest VM on my Development VM Ware ESXi Server.

Installing docker on Ubuntu is trivial – here are the commands I found via a quick google, there were no issues…

apt-get -y install docker.io

(FYI it’s called docker.io as there’s a previous/existing package with the name docker)

fix path issues:

ln -sf /usr/bin/docker.io /usr/local/bin/docker
sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker.io

have docker start up at boot:

update-rc.d docker.io defaults

and that’s that done – all very simple, quick and straightforward, now it’s time to pull down an Ubuntu image…

docker pull ubuntu

once that’s done, you are ready to run a command (bash in the case of this quick test I found suggested elsewhere) in a docker container like so:

docker run -i -t ubuntu /bin/bash

(-i attaches stdin & stdout and -t allocates a terminal)

That’s it for this post – the next will look at dynamic Jenkins Slave provisioning using Docker Containers, Jenkins plugins for Docker and ways to use Docker for a variety of Jenkins build, deployment and test tasks for Continuous Integration, Continuous Build and DevOps purposes.

Cheers,

Don

 

 

 

 

Extracting data from Jenkins

In Part I,  Information Radiators, I covered what they are, what the main benefits are, and the approach I usually use to set them up. This post goes in to more technical detail on how I extract this data from Jenkins.

My usual setup/architecture for Jenkins Information Radiators goes something along these lines:

  • TV screens running Mozilla Firefox or Google Chrome in Kiosk Mode, and Tab Mix Plus set up to rotate tabs (if required)
  • JSP Pages served via Tomcat on Linux server (which also runs the data extracting script described below)
  • MySQL database on Linux server – contains tables with data pulled from Jenkins and other sources, and the config data too (which URL’s to monitor)

And you’ll need some Jenkins instances/jobs to monitor too, obviously 🙂

The Jenkins XML API is very useful for automating tasks like this – if you simply append “/api/xml” to a
Jenkins job URL, it will serve up an XML version – note there is also a JSON API and a CLI and plenty of other options, but I’m using what suits me.

The Jenkins XML API

For example, if you go to one of your Jenkins jobs and add /api/xml like this:

“http://yourjenkinsserver:8080/job/yourjobname/api/xml”

you should get back some XML, possibly roughly like this example:

<?xml version="1.0"?>
<freeStyleBuild>
 <action>
 <parameter>
 <name>LOWER_ENV</name>
 <value>dev</value>
 </parameter>
 </action>
 <action>
 <cause>
 <shortDescription>Started by timer</shortDescription>
 </cause>
 </action>
 <building>false</building>
 <duration>61886</duration>
 <fullDisplayName>MyJob #580</fullDisplayName>
 <id>2014-04-01_10-01-50</id>
 <keepLog>false</keepLog>
 <number>580</number>
 <result>SUCCESS</result>
 <timestamp>1396342910088</timestamp>
 <url>http://jenkinsserver:8080/view/MyView/job/MyJob/580/</url>
 <builtOn/>
 <changeSet/>
</freeStyleBuild>

That XML contains loads of very useful information inside handy XML tag descriptions – you just need a way to get at that data and then you can present it as you like…

XPAth queries and the Jenkins XML API

so to automate that, I used to extend that approach a to query Jenkins via the XML API using XPAth queries to bring back just the data I actually wanted, quite like querying a database.

For example, wget’ing this URL would return just the current value of the <building> tag in the above XML:

http://yourjenkinsserver:8080/job/yourjobname/api/xml?xpath=//building/text()

e.g. “true” or “false” – this was very useful and easy to do, but the functionality was removed/disabled in recent versions of Jenkins for security reasons, meaning that my processes that used it needed rewritten 🙁

Extracting the data – Plan B…

So, here’s the new solution I went for – the real scripts/methods do some error handling and cleaning up etc but I’m just highlighting the main functions and the high level logic behind each of them here;

get_url’s method:

query a table in MySQL that contains a list of the job names and URL’s to monitor
for each $JOB_NAME found, it calls the get_file method, passing that the URL as a parameter.

get_file method:

this takes a URL param, and uses curl to fetch and save the XML data from that URL to a temporary file (“xmlfile”):

curl -sL "$1" | xmllint --format - > xmlfile

Note I’m using “xmllint –format” there to nicely format the XML data, which makes processing it later much easier.

get_data method:

this first calls “get_if_building” (see below) to see if the job is currently running or not, then it does:

TRUE_VAR="true"
 if [[ "$IS_BUILDING" == "$TRUE_VAR" ]]; then
 RESULT_TEXT="building..."
 else
 RESULT_TEXT=`grep "result>" xmlfile | awk -F\> '{print $2}' | awk -F\< '{print $1}'`
 fi

get_if_building method:

this simply checks and sets the IS_BUILDING var like so:

IS_BUILDING=`grep building xmlfile | awk -F\> '{print $2}' | awk -F\< '{print $1}'`

Putting it all together

My script then updates the MySQL database with the results from each check: success/failure, date, build number, user, change details etc

I then have JSP pages that read data from that table, and translate things like true/false in to HTML that sets the background colours (Red, Amber, Green), and shows the appropriate blocks and details per job.

If you have a few browsers/TV’s or Monitors showing these strategically placed around the office, developers get rapid feedback on the result of their code changes which speeds up development, increases quality and reduces development time and costs – and they can be fun to watch and set up too 🙂

Cheers,

Don

DIY Information Radiator

 

 

Information Radiators are used to provide people with feedback on the current status of code builds and automated tests in Continuous Integration and Agile development environments.

The basic idea is that when developers commit a code change, they can easily and quickly see that it has been picked up by the automated build process, and then (ideally within 10 minutes) see the result of their change; did the build succeeded and did the automated tests pass?

Martin Fowler’s description goes in to more detail on the ideas behind this approach and the function that Information Radiators serve.

The normal convention for these is to use colour coded blocks per build, using:

  • Green for good/passing jobs
  • Amber for either currently building/running jobs (or sometimes for unstable ones)
  • Red for failed jobs

Generally you want to keep things as clean, simple and uncluttered as possible, but sometimes it is helpful to add in a bit more info.

Details I have occasionally found worth adding include things like;

  • name and/or picture of the user who triggered (or broke!) the build
  • commit message from the code change that triggered the build
  • build number
  • history – number of recent fails or passes
  • date/time last failed and last checked

if you use amber for “unstable” rather than “build in progress”, you may want to add text to say if the job is currently building – I often use the “spinning wheel” icon thingy from jenkins itself :

spinner

Why build your own?

There are tons of readily available plugins that allow you to quickly and easily produce a Radiator or Wall panel from Jenkins, so why go to the bother of making your own?

Plugins are usually linked to one Jenkins instance (the one they are running on) and I have often found that alone to be too restrictive – having too many different radiators all over the place makes things too cluttered and uncertain, and people can easily start to “switch off” from them all – having one screen that people can understand at a glance usually works best.

Changing requirements – developers are constantly wanting/looking to improve processes and often come up with ideas and requests for things to try that may help them do their jobs – adding a bit of information from another source, for example, or changing the colours used to a different shade, or adding curved borders etc etc…

What I have found often works best, is to get all of the data I am interested in inside a database then write my own simple but flexible presentation layer from scratch – this gives me all the flexibility I could want (or may find myself wanting or needing later…) and importantly, it also allows me to leverage additional benefits by combining data from Jenkins with data from elsewhere – this means I can easily produce reports, charts, metrics etc that present a view comprised from multiple data sources throughout an organisation – for example, you can then easily create reports that combine:

  • jenkins – live information on the current state of builds and tests
  • defects – data on bugs and changes pulled from bug tracking apps (usually via jenkins jobs)
  • version control – the actual code change can be extracted from version control and linked to both the developer and bug details – the “svn log” command is useful for this
  • environment monitors – state and health of environments; database and app server health, deployed patch and code levels etc (again, these are usually other Jenkins jobs!)

and you can add in anything else you can get your hands on 🙂

This allows you to track the flow of a change right through the development life cycle – from the initial change/requirement/defect to the change itself at the code/file level, then the testing and building of it and the eventual release. This is far more than you need for a simple Information Radiator, but using this approach means that you can easily reuse much of the work in different ways.

Part II covers the technical approach I use for Extracting data from Jenkins

Minecraft Server Admin/Op commands

Minecraft Server Admin/Op commands are entered into the chat window and are preceded by a “/” when playing in-game, but when entering commands at the server console you need to miss out the preceding “/”.

(This table is copied from http://wiki.multiplay.co.uk/Minecraft/Operators and is here purely for my convenience and laziness)

Command Description Usage
help or ? Shows a list of server commands in the console or in-game. help
kick Removes player from the server with the message “Kicked by admin” kick username
ban Bans player from the server with the message “Banned by admin”, banned players who attempt to connect are presented with the message “You are banned from this server!” ban username
pardon Removes the specified player from banned-players.txt, allowing them to connect to the server again pardon username
ban-ip Bans an IP address from the server, the full IP address must be specified; wildcards are not valid. Players who are banned through this method will see “Your IP address is banned from this server!” when attempting to connect ban-ip ipaddress
pardon-ip Removes the specified IP from banned-ips.txt, allowing players with that IP to connect to the server again pardon-ip ipaddress
op Writes the players name to ops.txt, giving them access to the op commands op player
deop Removes a player from ops.txt, revoking their access to the op commands deop player
tp Moves the first player specified to the location of the second player specified tp player1 player2
give Gives a specified amount of a block, the ids are known as data values give player id amount
stop Gracefully stops the server stop
save-all Forces a server-wide level save save-all
save-off Disables terrain saving (useful for backup scripts) save-off
save-on Re-enables terrain saving. save-on
list Lists all currently connected players list
say Broadcasts a message to all players without a player name being shown and putting message in pink text say message
whitelist Enable or disable whitelisting (i.e. only listed players may join) whitelist on/off
whitelist Add or remove player from the whitelist whitelist add/remove player
whitelist list Lists all currently whitelisted players whitelist list
whitelist reload Reload the whitelist from white-list.txt whitelist reload
time Add to or set the world time. Amount may be a number between 0 and 24000, inclusive, where 0 is dawn and 12000 is dusk time add/set amount

Minecraft Server Admin tips – using the Linux screen command

 

 

Getting started with the Linux “screen” command:

This tip follows on from the the earlier post Minecraft and Minecraft Pocket Edition Servers on Ubuntu Linux

The Linux “screen” command is a very useful tool for managing one or many Minecraft and Minecraft Android/Pocket Edition processes.

First off, check you have screen installed on your Linux server by doing;

don@ubuntuserver:~$ which screen

that  should return something like:

/usr/bin/screen

if it doesn’t, you may need to install it like so:

sudo apt-get install screen

(or whatever works for your platform)

then you can start kicking off your Minecraft processes like this

 screen ./start.sh

when that’s up and running you can disconnect from the screen by doing

Ctrl-a d

This will detach you from the screen/console but leave the server running – meaning you can log off or go and do something else.

To reattach to the running screen (assuming there’s only one) and get back to the console at a later time, you just do this:

screen –r

If there is more than one screen sessions available you’ll be given a list of them, something like this:

There are several suitable screens on:
15772.pts-1.ubuntuserver(06/01/14 22:05:25)     (Detached)
2088.pts-4.ubuntuserver (06/01/14 15:20:59)     (Attached)
2013.pts-3.ubuntuserver (06/01/14 15:20:13)     (Detached)

from which you can then do:

screen -r [pid.]tty.host

using the pid and tty host values from that list, e.g. “screen –r 2013.pts-3.ubuntuserver” or “screen –r 15772.pts-1.ubuntuserver” in my example.

There are loads of other useful screen commands – check out the man page for further info – but the above allows me to manage multiple Minecraft servers easily.

Loading CSV data in to MySQL – random quotations app Part 1

Time for a new PHP and MySQL app – “Who said that?” – a search tool for famous quotations.

I created these two web applications a while back:

UK post code search
crossword solver

and then wrote this page:
Some PHP examples
detailing roughly how they were put together, but this time I wanted to create a searchable database of famous quotations, and focus on the MySQL side of things a bit more too (so that next time I will have a note of how I did it!).

I found a very nice CSV data file on http://thewebminer.com/download for free – I don’t really do Facebook much and don’t have a Twitter account so I thought/hoped they’d settle for a blog post in exchange…

After installing and setting up MySQL, connect to your database…

mysql –user=myuser –password=myusualpassword dev

— or connect without specifying a database/schema and do “show databases;”

mysql> show tables;
 +---------------+
 | Tables_in_dev |
 +---------------+
 | areacodes |
 | dictionary |
 +---------------+
 2 rows in set (0.02 sec)

For this little app, I want to create a new table with fields for each row in the CSV file
plus I’d like an auto_increment field to make fetching random numbers easier

CREATE TABLE quotes (
 id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
 quote varchar(800),
 author varchar(100),
 genre varchar(100)
 );
mysql> show tables;
 +---------------+
 | Tables_in_dev |
 +---------------+
 | areacodes |
 | dictionary |
 | quotes |
 +---------------+
 3 rows in set (0.00 sec)
mysql> describe quotes;
 +--------+--------------+------+-----+---------+----------------+
 | Field | Type | Null | Key | Default | Extra |
 +--------+--------------+------+-----+---------+----------------+
 | id | int(11) | NO | PRI | NULL | auto_increment |
 | quote | varchar(800) | YES | | NULL | |
 | author | varchar(100) | YES | | NULL | |
 | genre | varchar(100) | YES | | NULL | |
 +--------+--------------+------+-----+---------+----------------+
 4 rows in set (0.06 sec)

ok, the table looks good, so I can load the CSV data file – note that I’ve got the “quotes35000.csv” file I downloaded from
http://thewebminer.com/download sitting in the current directory:

mysql> load data local infile 'quotes35000.csv' into table quotes
 -> fields terminated by ';'
 -> lines terminated by 'n'
 -> (quote, author, genre);
 Query OK, 35002 rows affected (1.54 sec)
 Records: 35002 Deleted: 0 Skipped: 0 Warnings: 0

that looks like it went well (“35002 rows affected”), time to check it:

mysql> select count(*) from quotes;
 +----------+
 | count(*) |
 +----------+
 | 35002 |
 +----------+
 1 row in set (0.04 sec)
mysql> select * from quotes where author like '%Einstein' and genre like 'attitude%';
 +------+-----------------------------------------------------+-----------------+-----------+
 | id | quote | author | genre |
 +------+-----------------------------------------------------+-----------------+-----------+
 |4647 | Weakness of attitude becomes weakness of character. | Albert Einstein | attitude
 +------+-----------------------------------------------------+-----------------+-----------+
 1 row in set (0.02 sec)

All looking good, the row count and the returned query match what I’d expect having looked at the contents of the CSV file.

I also want to do a “random quote of the day thing”, so looked in to ways to do this in MySQL – my initial thought was to use something basic like “ORDER BY RAND() LIMIT 0,1;” to bring back one random row, but I guessed there may be better ways.

Google led me to this site which has some good examples and some performance/comparison details too:
http://akinas.com/pages/en/blog/mysql_random_row/

so I tried this…

mysql> SELECT * FROM quotes WHERE id >= (SELECT FLOOR( MAX(id) * RAND()) FROM quotes ) ORDER BY id LIMIT 1;
 +-----+--------------------------------------------------------------------------+-----------------+-------+
 | id | quote | author | genre |
 +-----+--------------------------------------------------------------------------+-----------------+-------+
 |84 | Men are like wine - some turn to vinegar, but the best improve with age. | Pope John XXIII | age
 +-----+--------------------------------------------------------------------------+-----------------+-------+
 1 row in set (1.54 sec)

then this…

mysql> SELECT * FROM quotes ORDER BY RAND() LIMIT 0,1;
 +-------+-------------------------------------------------------------------------------------+----------------+-------+
 | id | quote | author | genre |
 +-------+-------------------------------------------------------------------------------------+----------------+-------+
 |29470 | The good die young, because they see it's no use living if you have got to be good. | John Barrymore | good
 +-------+-------------------------------------------------------------------------------------+----------------+-------+
 1 row in set (0.73 sec)

and found to my surprise that in this case, looking at the timings, the simple approach looks to be faster – probably because of the relatively small table and its simple structure?

Anyway, that’s the database side of things sorted, the next part is to put together some PHP code to allow searching for quotes based on author, partial quote or genre, and to write a simple “random quote” generator kind of thing.

Cheers,

Don

Minecraft and Minecraft Pocket Edition Servers on Ubuntu Linux

I was asked to set up some Home LAN Minecraft and Pocketmine Servers for my children, so they can play at home and online with their friends in a safe/private environment – I was also interested in the geekier side of setting these servers up and seeing what all the Minecraft fuss is about 🙂

This post covers:

  • setting up a “normal” Minecraft Server on Linux
  • setting up a Minecraft Pocket Edition Server on Linux
  • getting started with the Linux “screen” command to manage multiple Minecraft Server processes

Setting up a normal Minecraft server on Linux (and I’d imagine any other OS) is very easy – you just need Java and the minecraft server jar file which you can get here:

https://minecraft.net/download

And as that page says, just sort out Java then kick off the process like this:

 java -Xmx1024M -Xms1024M -jar minecraft_server.jar nogui

Making sure you change the minecraft_server.jar  to match the name of the file you downloaded (e.g. minecraft_server.1.7.4.jar or whatever).

I originally put that command in a script and started it with nohup, but realised that you then lose the ability to interact with the process, so I have changed my approach and use the screen command which is much better – see the notes below on how that can be done, it’s easy and very useful.

Also, I think it’s safer to stop your Minecraft server/world cleanly by doing “stop” in the console, rather than Ctrl-c or killing the PID – after doing this I stopped getting these error messages in the console output:

 [ERROR] A level 8 error happened: "Uninitialized string offset: 8192" in...

 

Setting up a Minecraft Pocket Edition server… this was initially quite a contrast to the above, but it got better…

I started off by following the Pocketmine Server setup and installation instructions I found which said to simply download the install script (PocketMine-MP_Installer_XXXXX.sh) and run it:

https://github.com/PocketMine/PocketMine-MP/wiki/Setting-up-a-Server#wiki-linux

but this really didn’t work for me and led to a catalogue of vague errors and a whole load of googling which took me back to the olden times of fighting with make, configure, libraries, conflicts, missing tools and config files and installing all sorts of things based on vague hints from obscure error messages found in cryptic log files… you get the general idea I guess.

It has a simple and happy ending (below), but my experience started off something like this…

don@ubuntuserver:~/pe_minecraft$ ./start.sh
 09:43:39 [ERROR] Unable to find the pthreads extension.
 09:43:39 [ERROR] Unable to find the cURL extension.
 09:43:39 [ERROR] Unable to find the SQLite3 extension.
 09:43:39 [ERROR] Please use the installer provided on the homepage.

And that was after running the installer provided on the homepage and checking the dependencies were all there… so I debugged the steps in that script and was then on to the log files… where one thing led to another for quite a while… some of the more memorable ones are…

 “Compiler error reporting is too harsh for ./configure (perhaps remove -Werror).”
“configure: error: C compiler cannot create executables”

tried “apt-get install build-essential” to fix that, got a bit further

/usr/include/stdc-predef.h:30:26: fatal error: bits/predefs.h: No such file or directory

tried “apt-get install libc6-dev-i386” to fix that, got a bit further

There were several other issues and no end in sight, and I realised it really shouldn’t be this frustrating, so I looked for another approach.

The Minecraft MP Server code is all hosted on github here:

https://github.com/shoghicp/PocketMine-MP?1358555786480

So I simply downloaded everything from there (you could do “git clone” or “svn export”, but I went for wget’ing the latest zip file):

 wget https://github.com/PocketMine/PocketMine-MP/archive/master.zip

extracted that then ran compile :

./src/build/compile.sh

waited a while then ran start:

./start.sh

and this time there were no problems, my Minecraft Pocket Edition MP server was up and running on the home LAN and local devices were able to connect no problem.

 

The Linux “screen” command is very useful tool for Minecraft Server Admins – please see this post for details on getting started with it:

Minecraft Server Admin tips – using the Linux screen command

it allows you to manage multiple Minecraft Server Processes and consoles easily, without having to keep multiple sessions open.

Cheers,

Don

Pin It on Pinterest