Using jenkins to build my opensource projects

I've been playing around lately with jenkins. Jenkins is a continuous integration server which has quite a good list of plugins. I've now got most of my open source projects building everytime a change is made. An in the cases of projects like AtomicParsley Shild Fork, I've gotten it to build on 4 different platforms every time a change is made to the source code.

Jenkins makes this pretty easy by allowing a master controll server and then salve servers. So I run the slaves on my different platforms. There are also plugins that allow VM's to be started for needed platforms. Thats something I want to explore for building on different Linux distributions.

The real advantage for me is when the build finished and Jenkins publishes the builds. I have it setup to uploaded AtomicParsley builds the public area of my drop box account. These are then linked to from the project website. In the case of MediaManager, it triggers another build which pushes the current build to the OpenSuSE build server and creates nighly RPM's of the project. These can be used by people who want to live on the bleeding edge.

The other thing I really like is that it collects all the test reports and generated documentation onto the jenkins server. This allows me to quickly review the latest build. I can see how this would be very usefull for teams of develops. However now find the building side of stuff is taken care of automatically leaving me much more time for development. Creating and testing relases is much less of a chore now.

Anyhow Media Manager is getting much closer to a release. Just testing things like the installers at the moment. Once I'm happy with them, I'll be pushing out the release :-)

Initial Release of Atomic Parsley Shield Fork

A while back, I created a fork of atomic parsley so that I could use it in Media Manager. Atomic Parsley is used to display and edit metdata in a mp4 file. It was not quite up to the task of setting some of the new iTunes atoms, so I created a fork with the new features.

Well things have progressed pretty well and I've just uploaded the first release (Version 0.9.5) of my fork of atomic parsley. Here is a list of the features added in this release:

• CMake Build scripts.
• Option to output MP4 atoms as XML.
• Added support for 1080p iTunes atoms (--hdvideo option also takes a number)
• Added --flavor option for setting flvr atom.
• Allow sort order fields to be set in the format of --sortOrder name=value 
• Added missing sort albumn option
• Added option for setting plID atom

I realise that lot's of applications use atomic parsley under the hood for handling mp4 file atoms, so I'm hoping that the --outputXML option will make reading the atom contents a lot easier. 

I also took the optunity to setup a private jenkins continuous integration build server. This builds on 4 different platforms every time their is a commit to the bitbucket build repository. When their is a successful build, it uploads them to the public area of my drop box account. The end result is I now have nightly builds if atomic parsley as well as stable builds. To see the links to the prebuilt binaries, visit the project website.

If anyone finds any issues or features they would like to see, please raise a issue on the project website.

Initial release of itunes-remote-control-server

I've just published the inital release of my project itunes-remote-control-server. This is a project to make it possible to control iTunes on a headless server without a GUI. It provides a TCP server which can be connected to locally or remote by other applications to control iTunes. Alternatively it also provides command line commands for controlling iTunes.

Currently the server and scripts are able to perform these tasks:

• Add new tracks/videos to iTunes Library.
• Remove tracks/videos from iTunes Library.
• Tell iTunes that the metadata needs to be refreshed from a file of the track/video.
• List tracks that have missing files (dead tracks).
• Remove tracks that have missing files (dead tracks).
• Display information about tracks/videos in the iTunes library.

These operations can be performed by command line scripts, but also by using a TCP server. The next release of MediaManager will use the server to keep iTunes in sync with managed media libraries. In fact I've been running the latest svn version of MediaManager using the control server for a little while now. Seems to be working very well.

The server will work on windows or Mac OS X as long as the requirements are installed.

• iTunes (latest)
• Mac OSX or Windows
• Mac Ruby ( On Mac OSX)
• Ruby Gems: escape, sqlite3

See the install guides on the wiki for details on how to install it. If you find any issues, then let me know my the issue part of the website.

I'm hoping that work can start on the next version now. Currently have ideas for some features such as:

• Different database backends
• Play, pause, skip tracks etc...
• Create a GUI to control iTunes remotely.

Adding more meta data capabilities to MediaManager

I'm please with the progress that have been make towards the next release of Media Manager. The biggest difference in this release will be much improved meta data stored within iTunes files and the ability to keep iTunes in sync. These achievements have cause be to create other projects. The first of these I already mention was a iTunes remote control server written in ruby. This is nearing it's first release, it just needs it's windows support finishing off.

I wanted this release of media manager to start a lot more meta data in m4v files so that they look a lot more like iTunes meta data. This means when you rip a DVD and want to iTunes to see it, it will look as if it can from the iTunes store. In turn this should mean it is viewable on iPads and apple TV's etc...

This release of Media Manager move to using the AtomicParsley command line tool to encode and view mp4 metadata. This tool has been around a while, but the orginal site has not been updated in a long time. Since then their have been new atom types that have been added to iTunes. So I came across  a fork of atomic parsley. This is pretty good and has had a number of features and bug fixes. This still does not do everything I would like, so in turn I've forked this project. I'm hoping that my fork's changes can be push upstream once they are ready. My fork can be found here https://bitbucket.org/shield007/atomicparsley.

So I will be packing up my fork of AtomicParsley with Media Manager. The new features I've add so far are the ability to set the flavour atom (flvr) and a output option to list iTunes atoms in a XML format.

The flvr atom can be seen in files downloaded from the iTunes store. So that I've found that it contains these values:

• 1:128
• 2:256
• 4:640x480LC-128
• 6:640x480LC-256
• 7:720p

It took me a while to work out what all of these mean, by I think I have it now. 7:720p is for videos that are 720p. These also don't have to be 16:9 aspect ratio.  1:128 is for audio files encoded in 128Kbs lossy. Simlary 2:256 is audio files encoded in 2:256 lossy. 4 and 6 were the tricky ones. It turns out that 640x480 is the 480p resolution, LC is the format profile of the audio. and 128/256 is the Kbs of the audio.

Usually 4 is used by low definition films and TV, 6 is used by music videos and 7 is used by High Def content. Their are probably other entries for this atom. If people know of any let me know. I can cater for them as well.

Media Manager is now making use of a tool call mediainfo to query the video and audio information of the file.   This enables media manager to know thinks like the flavour, aspect ration, type of audio etc...

New Project: itunes-remote-control-server

The other day I create a new opensource project on Google code called itunes-remote-control-server. The idea behind the project is to create a TCP server that can be used to control iTunes. This will allow easy headless control of a iTunes machine.

I'm aim to use the server with my other project Media Manager. The idea being that once media manager has made changes to media files it can tell iTunes about the changes. This will keep iTunes in sync with the media directories been managed by media manager. Their are two parts to this, a new store within Media Manager which will hopefully be in the next release and a TCP server on the iTunes machine.

So I've created a basic server that works on Mac OSX. It's written in ruby and uses mac ruby script bridge to talk to iTunes. I'm hoping to add windows iTunes support before a official release, but this is still at investigation stage at the moment.

Currently the server will allow you to add or remote tracks from the iTunes library as well as listing/removing tracks whose files can't be found any more (Dead tracks). Their is simple authentication support and it's configuration is done via a XML file.

Their are also some command line utilities that can be used instead of the server if people want to drive iTunes from the CLI without the server. So far these are for finding tracks not in the iTunes library and dead tracks.

So currently I've not released it, though I hope this will not take to long. Just need to look into the windows side of things. If people want to try it out, then they will need to do the following:

git clone https://jp.stanford@code.google.com/p/itunes-remote-control-server
cd itunes-remote-control-server
gem build ./itunes-controller.gemspec
gem install itunes-controller-0.1.0.gem



This will install the build gem and provide the command line tools:

itunesController.rb   <- The TCP Server
llistDeadTracks.rb
listNewTracks.rb
removeDeadTracks.rb

Anyway I'll post a update again when It's time for a official release. At that point the ruby gem will be downloadable from the website.

Media Manager v2.0 Released!

Media Manager v2.0 has just been released. The project has had a name changed since the last updated (previously known as MediaInfoFetcher). This is a major update that focus on managing media (Currently TV Shows and Films) and much greater reliability when fetching meta data.

Some of the change high lights are as follows:

• Better searching for film and TV information using XBMC scrapers
• Tool to download and stay uptodate with the latest XBMC scrapers
• Much improved film title and show name lookup by variours searching strategies
• New actions, such as ability to execute system commands on media files and rename the media files
• Enhanced renaming patterns which new token types and optional parts to patterns.
• Can lookup film information from .NFO files.
• Improved configuration options
• New XML store for caching media information
• Improved tag chimp source, that will use the sites API instead of scraping the HTML.
• Maybe new configuration options (and a new configuration format).
• New installer for all platforms.

This has been quite a while coming, but it's finally here. Please use the project website to let me know of bugs or feature requests. Also if you find the application useful then let me know on my blog.

So click here for more information.

MP4 Libraries in Media Manager

MediaManager 2.0 is pretty much ready to release apart from a problem with the MP4ItunesStore. This store is used to save metadata into mp4/m4v files so that they appear correctly in iTunes and can be used with AppleTV, iPad's etc....

In version 1.x this store was based around a command line application AtomicParsely. However this always anoyed me as it was not very easy to use and was slot to drive it from java.

In version 2.x I've been working towards a pure java solution. I first of starting to use jaudiotagger. I found this while working on my podcaster application. This is more about audio files than video though. It works well for podcaster, but not very well for MediaManager. I had to hack it quite a bit before it would work, then I started to find problems with parsing some MP4 files.

I stumbled across mp4parser next. This is pretty good and does everything I need or so it seemed at first. I found a number of bugs in the library. I fixed a number of these and sent patches to the project (Still not been applied :-( ), I then found a case where a DVD I had turned into a MP4 file would get corrupted.  raised a issue on this also. Not only did it courrpt the file, but it gave me no way of detecting a error. Due to these problems I had to stop using the library. The project has a hight amount of activity so I'm still hoping these things might get fixed.

I've looked around for other Java solutions, but not found any. I could possibly write my own or fork one of the others, but that's too much work for now. Instead I've started to look at using a native library called MP4v2 which is used by a number of other C/C++ projects. To create the bindings I'm using JNA. This as not been without problems though. Currently on Linux I'm seeing segfaults unless I use a 32Bit sun JVM. Strangely this happens on a 64Bit sun JVM also. I'm currently hopeing to find out why and fix the problem, but will see what happens.......... It might mean we only have iTunes support for curetain Arch/JVM's.

Anyways once these problems are solved, MediaManager will be ready for release