Petter Reinholdtsen

Back in oktober last year, when I started looking at the LinuxCNC system, I proposed to change the documentation build system make life easier for translators. The original system consisted of independently written documentation files for each language, with no automated way to track changes done in other translations and no help for the translators to know how much was left to translated. By using the po4a system to generate POT and PO files from the English documentation, this can be improved. A small team of LinuxCNC contributors got together and today our labour finally payed off. Since a few hours ago, it is now possible to translate the LinuxCNC documentation on Weblate, alongside the program itself.

The effort to migrate the documentation to use po4a has been both slow and frustrating. I am very happy we finally made it.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Recently I wanted to upgrade the firmware of my thinkpad, and located the firmware download page from Lenovo (which annoyingly do not allow access via Tor, forcing me to hand them more personal information that I would like). The download from Lenovo is a bootable ISO image, which is a bit of a problem when all I got available is a USB memory stick. I tried booting the ISO as a USB stick, but this did not work. But genisoimage came to the rescue.

The geteltorito program in the genisoimage binary package is able to convert the bootable ISO image to a bootable USB stick using a simple command line recipe, which I then can write to the most recently inserted USB stick:

geteltorito -o usbstick.img lenovo-firmware.iso
sudo dd bs=10M if=usbstick.img of=$(ls -tr /dev/sd?|tail -1)

This USB stick booted the firmware upgrader just fine, and in a few minutes my machine had the latest and greatest BIOS firmware in place.

After many months of hard work by the good people involved in LinuxCNC, the system was accepted Sunday into Debian. Once it was available from Debian, I was surprised to discover from its popularity-contest numbers that people have been reporting its use since 2012. Its project site might be a good place to check out, but sadly is not working when visiting via Tor.

But what is LinuxCNC, you are probably wondering? Perhaps a Wikipedia quote is in place?

"LinuxCNC is a software system for numerical control of machines such as milling machines, lathes, plasma cutters, routers, cutting machines, robots and hexapods. It can control up to 9 axes or joints of a CNC machine using G-code (RS-274NGC) as input. It has several GUIs suited to specific kinds of usage (touch screen, interactive development)."

It can even control 3D printers. And even though the Wikipedia page indicate that it can only work with hard real time kernel features, it can also work with the user space soft real time features provided by the Debian kernel. The source code is available from Github. The last few months I've been involved in the translation setup for the program and documentation. Translators are most welcome to join the effort using Weblate.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

The Debian Lego team saw a lot of activity the last few weeks. All the packages under the team umbrella has been updated to fix packaging, lintian issues and BTS reports. In addition, a new and inspiring team member appeared on both the debian-lego-team Team mailing list and IRC channel #debian-lego. If you are interested in Lego CAD design and LEGO Mindstorms programming, check out the team wiki page to see what Debian can offer the Lego enthusiast.

Patches has been sent upstream, causing new upstream releases, one even the first one in more than ten years, and old upstreams was released with new ones. There are still a lot of work left, and the team welcome more members to help us make sure Debian is the Linux distribution of choice for Lego builders. If you want to contribute, join us in the IRC channel and become part of the team on Salsa.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

I am happy observe that the The Debian Administrator's Handbook is available in six languages now. I am not sure which one of these are completely proof read, but the complete book is available in these languages:

• English
• Norwegian Bokmål
• German
• Indonesian
• Brazil Portuguese
• Spanish

This is the list of languages more than 70% complete, in other words with not too much left to do:

• Chinese (Simplified) - 90%
• French - 79%
• Italian - 79%
• Japanese - 77%
• Arabic (Morocco) - 75%
• Persian - 71%

I wonder how long it will take to bring these to 100%.

Then there is the list of languages about halfway done:

• Russian - 63%
• Swedish - 53%
• Chinese (Traditional) - 46%
• Catalan - 45%

Several are on to a good start:

• Dutch - 26%
• Vietnamese - 25%
• Polish - 23%
• Czech - 22%
• Turkish - 18%

Finally, there are the ones just getting started:

• Korean - 4%
• Croatian - 2%
• Greek - 2%
• Danish - 1%
• Romanian - 1%

If you want to help provide a Debian instruction book in your own language, visit Weblate to contribute to the translations.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

After a lot of hard work by its maintainer Alexandre Viau and others, the decentralized communication platform Jami (earlier known as Ring), managed to get its latest version into Debian Testing. Several of its dependencies has caused build and propagation problems, which all seem to be solved now.

In addition to the fact that Jami is decentralized, similar to how bittorrent is decentralized, I first of all like how it is not connected to external IDs like phone numbers. This allow me to set up computers to send me notifications using Jami without having to find get a phone number for each computer. Automatic notification via Jami is also made trivial thanks to the provided client side API (as a DBus service). Here is my bourne shell script demonstrating how to let any system send a message to any Jami address. It will create a new identity before sending the message, if no Jami identity exist already:

#!/bin/sh
#
# Usage: $0  
#
# Send  to , create local jami account if
# missing.
#
# License: GPL v2 or later at your choice
# Author: Petter Reinholdtsen


if [ -z "$HOME" ] ; then
    echo "error: missing \$HOME, required for dbus to work"
    exit 1
fi

# First, get dbus running if not already running
DBUSLAUNCH=/usr/bin/dbus-launch
PIDFILE=/run/asterisk/dbus-session.pid
if [ -e $PIDFILE ] ; then
    . $PIDFILE
    if ! kill -0 $DBUS_SESSION_BUS_PID 2>/dev/null ; then
        unset DBUS_SESSION_BUS_ADDRESS
    fi
fi
if [ -z "$DBUS_SESSION_BUS_ADDRESS" ] && [ -x "$DBUSLAUNCH" ]; then
    DBUS_SESSION_BUS_ADDRESS="unix:path=$HOME/.dbus"
    dbus-daemon --session --address="$DBUS_SESSION_BUS_ADDRESS" --nofork --nopidfile --syslog-only < /dev/null > /dev/null 2>&1 3>&1 &
    DBUS_SESSION_BUS_PID=$!
    (
        echo DBUS_SESSION_BUS_PID=$DBUS_SESSION_BUS_PID
        echo DBUS_SESSION_BUS_ADDRESS=\""$DBUS_SESSION_BUS_ADDRESS"\"
        echo export DBUS_SESSION_BUS_ADDRESS
    ) > $PIDFILE
    . $PIDFILE
fi &

dringop() {
    part="$1"; shift
    op="$1"; shift
    dbus-send --session \
        --dest="cx.ring.Ring" /cx/ring/Ring/$part cx.ring.Ring.$part.$op $*
}

dringopreply() {
    part="$1"; shift
    op="$1"; shift
    dbus-send --session --print-reply \
        --dest="cx.ring.Ring" /cx/ring/Ring/$part cx.ring.Ring.$part.$op $*
}

firstaccount() {
    dringopreply ConfigurationManager getAccountList | \
      grep string | awk -F'"' '{print $2}' | head -n 1
}

account=$(firstaccount)

if [ -z "$account" ] ; then
    echo "Missing local account, trying to create it"
    dringop ConfigurationManager addAccount \
      dict:string:string:"Account.type","RING","Account.videoEnabled","false"
    account=$(firstaccount)
    if [ -z "$account" ] ; then
        echo "unable to create local account"
        exit 1
    fi
fi

# Not using dringopreply to ensure $2 can contain spaces
dbus-send --print-reply --session \
  --dest=cx.ring.Ring \
  /cx/ring/Ring/ConfigurationManager \
  cx.ring.Ring.ConfigurationManager.sendTextMessage \
  string:"$account" string:"$1" \
  dict:string:string:"text/plain","$2" 

If you want to check it out yourself, visit the the Jami system project page to learn more, and install the latest Jami client from Debian Unstable or Testing.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

I am happy to report that we finally made it! Norwegian Bokmål became the first translation published on paper of the new Buster based edition of "The Debian Administrator's Handbook". The print proof reading copy arrived some days ago, and it looked good, so now the book is approved for general distribution. This updated paperback edition is available from lulu.com. The book is also available for download in electronic form as PDF, EPUB and Mobipocket, and can also be read online.

I am very happy to wrap up this Creative Common licensed project, which concludes several months of work by several volunteers. The number of Linux related books published in Norwegian are few, and I really hope this one will gain many readers, as it is packed with deep knowledge on Linux and the Debian ecosystem. The book will be available for various Internet book stores like Amazon and Barnes & Noble soon, but I recommend buying "Håndbok for Debian-administratoren" directly from the source at Lulu.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Thanks to the good work of several volunteers, the updated edition of the Norwegian translation for "The Debian Administrator's Handbook" is now almost completed. After many months of proof reading, I consider the proof reading complete enough for us to move to the next step, and have asked for the print version to be prepared and sent of to the print on demand service lulu.com. While it is still not to late if you find any incorrect translations on the hosted Weblate service, but it will be soon. :) You can check out the Buster edition on the web until the print edition is ready.

The book will be for sale on lulu.com and various web book stores, with links available from the web site for the book linked to above. I hope a lot of readers find it useful.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Three years ago, the first Norwegian Bokmål edition of "The Debian Administrator's Handbook" was published. This was based on Debian Jessie. Now a new and updated version based on Buster is getting ready. Work on the updated Norwegian Bokmål edition has been going on for a few months now, and yesterday, we reached the first mile stone, with 100% of the texts being translated. A lot of proof reading remains, of course, but a major step towards a new edition has been taken.

The book is translated by volunteers, and we would love to get some help with the proof reading. The translation uses the hosted Weblate service, and we welcome everyone to have a look and submit improvements and suggestions. There is also a proof readers PDF available on request, get in touch if you want to help out that way.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

As a member of the Norwegian Unix User Group, I have the pleasure of receiving the USENIX magazine ;login: several times a year. I rarely have time to read all the articles, but try to at least skim through them all as there is a lot of nice knowledge passed on there. I even carry the latest issue with me most of the time to try to get through all the articles when I have a few spare minutes.

The other day I came across a nice article titled "The Secure Socket API: TLS as an Operating System Service" with a marvellous idea I hope can make it all the way into the POSIX standard. The idea is as simple as it is powerful. By introducing a new socket() option IPPROTO_TLS to use TLS, and a system wide service to handle setting up TLS connections, one both make it trivial to add TLS support to any program currently using the POSIX socket API, and gain system wide control over certificates, TLS versions and encryption systems used. Instead of doing this:

int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);

the program code would be doing this:

int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TLS);

According to the ;login: article, converting a C program to use TLS would normally modify only 5-10 lines in the code, which is amazing when compared to using for example the OpenSSL API.

The project has set up the https://securesocketapi.org/ web site to spread the idea, and the code for a kernel module and the associated system daemon is available from two github repositories: ssa and ssa-daemon. Unfortunately there is no explicit license information with the code, so its copyright status is unclear. A request to solve this about it has been unsolved since 2018-08-17.

I love the idea of extending socket() to gain TLS support, and understand why it is an advantage to implement this as a kernel module and system wide service daemon, but can not help to think that it would be a lot easier to get projects to move to this way of setting up TLS if it was done with a user space approach where programs wanting to use this API approach could just link with a wrapper library.

I recommend you check out this simple and powerful approach to more secure network connections. :)

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Half a year ago, I wrote about the Jami communication client, capable of peer-to-peer encrypted communication. It handle both messages, audio and video. It uses distributed hash tables instead of central infrastructure to connect its users to each other, which in my book is a plus. I mentioned briefly that it could also work as a SIP client, which came in handy when the higher educational sector in Norway started to promote Zoom as its video conferencing solution. I am reluctant to use the official Zoom client software, due to their copyright license clauses prohibiting users to reverse engineer (for example to check the security) and benchmark it, and thus prefer to connect to Zoom meetings with free software clients.

Jami worked OK as a SIP client to Zoom as long as there was no password set on the room. The Jami daemon leak memory like crazy (approximately 1 GiB a minute) when I am connected to the video conference, so I had to restart the client every 7-10 minutes, which is not great. I tried to get other SIP Linux clients to work without success, so I decided I would have to live with this wart until someone managed to fix the leak in the dring code base. But another problem showed up once the rooms were password protected. I could not get my dial tone signaling through from Jami to Zoom, and dial tone signaling is used to enter the password when connecting to Zoom. I tried a lot of different permutations with my Jami and Asterisk setup to try to figure out why the signaling did not get through, only to finally discover that the fundamental problem seem to be that Zoom is simply not able to receive dial tone signaling when connecting via SIP. There seem to be nothing wrong with the Jami and Asterisk end, it is simply broken in the Zoom end. I got help from a very skilled VoIP engineer figuring out this last part. And being a very skilled engineer, he was also able to locate a solution for me. Or to be exact, a workaround that solve my initial problem of connecting to password protected Zoom rooms using Jami.

So, how do you do this, I am sure you are wondering by now. The trick is already documented from Zoom, and it is to modify the SIP address to include the room password. What is most surprising about this is that the automatically generated email from Zoom with instructions on how to connect via SIP do not mention this. The SIP address to use normally consist of the room ID (a number), an @ character and the IP address of the Zoom SIP gateway. But Zoom understand a lot more than just the room ID in front of the at sign. The format is "[Meeting ID].[Password].[Layout].[Host Key]", and you can here see how you can both enter password, control the layout (full screen, active presence and gallery) and specify the host key to start the meeting. The full SIP address entered into Jami to provide the password will then look like this (all using made up numbers):

sip:657837644.522827@192.168.169.170

Now if only jami would reduce its memory usage, I could even recommend this setup to others. :)

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

The curiosity got the better of me when Slashdot reported that New Jersey was desperately looking for COBOL programmers, and a few days later it was reported that IBM tried to locate COBOL programmers.

I thus decided to have a look at free software alternatives to learn COBOL, and had the pleasure to find GnuCOBOL was already in Debian. It used to be called Open Cobol, and is a "compiler" transforming COBOL code to C or C++ before giving it to GCC or Visual Studio to build binaries.

I managed to get in touch with upstream, and was impressed with the quick response, and also was happy to see a new Debian maintainer taking over when the original one recently asked to be replaced. A new Debian upload was done as recently as yesterday.

Using the Debian package, I was able to follow a simple COBOL introduction and make and run simple COBOL programs. It was fun to learn a new programming language. If you want to test for yourself, the GnuCOBOL Wikipedia page have a few simple examples to get you startet.

As I do not have much experience with COBOL, I do not know how standard compliant it is, but it claim to pass most tests from COBOL test suite, which sound good to me. It is nice to know it is possible to learn COBOL using software without any usage restrictions, and I am very happy such nice free software project as this is available. If you as me is curious about COBOL, check it out.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Some years ago, in 2016, I wrote for the first time about the Ring peer to peer messaging system. It would provide messaging without any central server coordinating the system and without requiring all users to register a phone number or own a mobile phone. Back then, I could not get it to work, and put it aside until it had seen more development. A few days ago I decided to give it another try, and am happy to report that this time I am able to not only send and receive messages, but also place audio and video calls. But only if UDP is not blocked into your network.

The Ring system changed name earlier this year to Jami. I tried doing web search for 'ring' when I discovered it for the first time, and can only applaud this change as it is impossible to find something called Ring among the noise of other uses of that word. Now you can search for 'jami' and this client and the Jami system is the first hit at least on duckduckgo.

Jami will by default encrypt messages as well as audio and video calls, and try to send them directly between the communicating parties if possible. If this proves impossible (for example if both ends are behind NAT), it will use a central SIP TURN server maintained by the Jami project. Jami can also be a normal SIP client. If the SIP server is unencrypted, the audio and video calls will also be unencrypted. This is as far as I know the only case where Jami will do anything without encryption.

Jami is available for several platforms: Linux, Windows, MacOSX, Android, iOS, and Android TV. It is included in Debian already. Jami also work for those using F-Droid without any Google connections, while Signal do not. The protocol is described in the Ring project wiki. The system uses a distributed hash table (DHT) system (similar to BitTorrent) running over UDP. On one of the networks I use, I discovered Jami failed to work. I tracked this down to the fact that incoming UDP packages going to ports 1-49999 were blocked, and the DHT would pick a random port and end up in the low range most of the time. After talking to the developers, I solved this by enabling the dhtproxy in the settings, thus using TCP to talk to a central DHT proxy instead of peering directly with others. I've been told the developers are working on allowing DHT to use TCP to avoid this problem. I also ran into a problem when trying to talk to the version of Ring included in Debian Stable (Stretch). Apparently the protocol changed between beta2 and the current version, making these clients incompatible. Hopefully the protocol will not be made incompatible in the future.

It is worth noting that while looking at Jami and its features, I came across another communication platform I have not tested yet. The Tox protocol and family of Tox clients. It might become the topic of a future blog post.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

I høst ble jeg inspirert til å bidra til oversettelsen av strategispillet Unknown Horizons, og oversatte de nesten 200 strengene i prosjektet til bokmål. Deretter har jeg gått å ventet på at det kom en ny utgave som inneholdt disse oversettelsene. Nå er endelig ventetiden over. Den nye versjonen kom på nyåret, og ble lastet opp i Debian for noen få dager siden. I går kveld fikk jeg testet det ut, og må innrømme at oversettelsene fungerer fint. Fant noen få tekster som måtte justeres, men ikke noe alvorlig. Har oppdatert oversettelsen på Weblate, slik at neste utgave vil være enda bedre. :)

Spillet er et ressursstyringsspill ala Civilization, og er morsomt å spille for oss som liker slikt. :)

Som vanlig, hvis du bruker Bitcoin og ønsker å vise din støtte til det jeg driver med, setter jeg pris på om du sender Bitcoin-donasjoner til min adresse 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b. Merk, betaling med bitcoin er ikke anonymt. :)

I am amazed and very pleased to discover that since a few days ago, everything you need to program the BBC micro:bit is available from the Debian archive. All this is thanks to the hard work of Nick Morrott and the Debian python packaging team. The micro:bit project recommend the mu-editor to program the microcomputer, as this editor will take care of all the machinery required to injekt/flash micropython alongside the program into the micro:bit, as long as the pieces are available.

There are three main pieces involved. The first to enter Debian was python-uflash, which was accepted into the archive 2019-01-12. The next one was mu-editor, which showed up 2019-01-13. The final and hardest part to to into the archive was firmware-microbit-micropython, which needed to get its build system and dependencies into Debian before it was accepted 2019-01-20. The last one is already in Debian Unstable and should enter Debian Testing / Buster in three days. This all allow any user of the micro:bit to get going by simply running 'apt install mu-editor' when using Testing or Unstable, and once Buster is released as stable, all the users of Debian stable will be catered for.

As a minor final touch, I added rules to the isenkram package for recognizing micro:bit and recommend the mu-editor package. This make sure any user of the isenkram desktop daemon will get a popup suggesting to install mu-editor then the USB cable from the micro:bit is inserted for the first time.

This should make it easier to have fun.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

A fun way to learn how to program Python is to follow the instructions in the book "Learn to program with Minecraft", which introduces programming in Python to people who like to play with Minecraft. The book uses a Python library to talk to a TCP/IP socket with an API accepting build instructions and providing information about the current players in a Minecraft world. The TCP/IP API was first created for the Minecraft implementation for Raspberry Pi, and has since been ported to some server versions of Minecraft. The book contain recipes for those using Windows, MacOSX and Raspian. But a little known fact is that you can follow the same recipes using the free software construction game Minetest.

There is a Minetest module implementing the same API, making it possible to use the Python programs coded to talk to Minecraft with Minetest too. I uploaded this module to Debian two weeks ago, and as soon as it clears the FTP masters NEW queue, learning to program Python with Minetest on Debian will be a simple 'apt install' away. The Debian package is maintained as part of the Debian Games team, and the packaging rules are currently located under 'unfinished' on Salsa.

You will most likely need to install several of the Minetest modules in Debian for the examples included with the library to work well, as there are several blocks used by the example scripts that are provided via modules in Minetest. Without the required blocks, a simple stone block is used instead. My initial testing with a analog clock did not get gold arms as instructed in the python library, but instead used stone arms.

I tried to find a way to add the API to the desktop version of Minecraft, but were unable to find any working recipes. The recipes I found are only working with a standalone Minecraft server setup. Are there any options to use with the normal desktop version?

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

As part of my involvement in the Nikita archive API project, I've been importing a fairly large lump of emails into a test instance of the archive to see how well this would go. I picked a subset of my notmuch email database, all public emails sent to me via @lists.debian.org, giving me a set of around 216 000 emails to import. In the process, I had a look at the various attachments included in these emails, to figure out what to do with attachments, and noticed that one of the most common attachment formats do not have an official MIME type registered with IANA/IETF. The output from diff, ie the input for patch, is on the top 10 list of formats included in these emails. At the moment people seem to use either text/x-patch or text/x-diff, but neither is officially registered. It would be better if one official MIME type were registered and used everywhere.

To try to get one official MIME type for these files, I've brought up the topic on the media-types mailing list. If you are interested in discussion which MIME type to use as the official for patch files, or involved in making software using a MIME type for patches, perhaps you would like to join the discussion?

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

A few days, I rescued a Windows victim over to Debian. To try to rescue the remains, I helped set up automatic sync with Google Drive. I did not find any sensible Debian package handling this automatically, so I rebuild the grive2 source from the Ubuntu UPD8 PPA to do the task and added a autostart desktop entry and a small shell script to run in the background while the user is logged in to do the sync. Here is a sketch of the setup for future reference.

I first created ~/googledrive, entered the directory and ran 'grive -a' to authenticate the machine/user. Next, I created a autostart hook in ~/.config/autostart/grive.desktop to start the sync when the user log in:

[Desktop Entry]
Name=Google drive autosync
Type=Application
Exec=/home/user/bin/grive-sync

Finally, I wrote the ~/bin/grive-sync script to sync ~/googledrive/ with the files in Google Drive.

#!/bin/sh
set -e
cd ~/
cleanup() {
    if [ "$syncpid" ] ; then
        kill $syncpid
    fi
}
trap cleanup EXIT INT QUIT
/usr/lib/grive/grive-sync.sh listen googledrive 2>&1 | sed "s%^%$0:%" &
syncpdi=$!
while true; do
    if ! xhost >/dev/null 2>&1 ; then
        echo "no DISPLAY, exiting as the user probably logged out"
        exit 1
    fi
    if [ ! -e /run/user/1000/grive-sync.sh_googledrive ] ; then
        /usr/lib/grive/grive-sync.sh sync googledrive
    fi
    sleep 300
done 2>&1 | sed "s%^%$0:%"

Feel free to use the setup if you want. It can be assumed to be GNU GPL v2 licensed (or any later version, at your leisure), but I doubt this code is possible to claim copyright on.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

I continue to explore my Kodi installation, and today I wanted to tell it to play a youtube URL I received in a chat, without having to insert search terms using the on-screen keyboard. After searching the web for API access to the Youtube plugin and testing a bit, I managed to find a recipe that worked. If you got a kodi instance with its API available from http://kodihost/jsonrpc, you can try the following to have check out a nice cover band.

curl --silent --header 'Content-Type: application/json' \
  --data-binary '{ "id": 1, "jsonrpc": "2.0", "method": "Player.Open",
  "params": {"item": { "file":
  "plugin://plugin.video.youtube/play/?video_id=LuRGVM9O0qg" } } }' \
  http://projector.local/jsonrpc

I've extended kodi-stream program to take a video source as its first argument. It can now handle direct video links, youtube links and 'desktop' to stream my desktop to Kodi. It is almost like a Chromecast. :)

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

For a while now, I have looked for a sensible way to share images with my family using a self hosted solution, as it is unacceptable to place images from my personal life under the control of strangers working for data hoarders like Google or Dropbox. The last few days I have drafted an approach that might work out, and I would like to share it with you. I would like to publish images on a server under my control, and point some Internet connected display units using some free and open standard to the images I published. As my primary language is not limited to ASCII, I need to store metadata using UTF-8. Many years ago, I hoped to find a digital photo frame capable of reading a RSS feed with image references (aka using the <enclosure> RSS tag), but was unable to find a current supplier of such frames. In the end I gave up that approach.

Some months ago, I discovered that XScreensaver is able to read images from a RSS feed, and used it to set up a screen saver on my home info screen, showing images from the Daily images feed from NASA. This proved to work well. More recently I discovered that Kodi (both using OpenELEC and LibreELEC) provide the Feedreader screen saver capable of reading a RSS feed with images and news. For fun, I used it this summer to test Kodi on my parents TV by hooking up a Raspberry PI unit with LibreELEC, and wanted to provide them with a screen saver showing selected pictures from my selection.

Armed with motivation and a test photo frame, I set out to generate a RSS feed for the Kodi instance. I adjusted my Freedombox instance, created /var/www/html/privatepictures/, wrote a small Perl script to extract title and description metadata from the photo files and generate the RSS file. I ended up using Perl instead of python, as the libimage-exiftool-perl Debian package seemed to handle the EXIF/XMP tags I ended up using, while python3-exif did not. The relevant EXIF tags only support ASCII, so I had to find better alternatives. XMP seem to have the support I need.

I am a bit unsure which EXIF/XMP tags to use, as I would like to use tags that can be easily added/updated using normal free software photo managing software. I ended up using the tags set using this exiftool command, as these tags can also be set using digiKam:

exiftool -headline='The RSS image title' \
  -description='The RSS image description.' \
  -subject+=for-family photo.jpeg

I initially tried the "-title" and "keyword" tags, but they were invisible in digiKam, so I changed to "-headline" and "-subject". I use the keyword/subject 'for-family' to flag that the photo should be shared with my family. Images with this keyword set are located and copied into my Freedombox for the RSS generating script to find.

Are there better ways to do this? Get in touch if you have better suggestions.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Last night, I wrote a recipe to stream a Linux desktop using VLC to a instance of Kodi. During the day I received valuable feedback, and thanks to the suggestions I have been able to rewrite the recipe into a much simpler approach requiring no setup at all. It is a single script that take care of it all.

This new script uses GStreamer instead of VLC to capture the desktop and stream it to Kodi. This fixed the video quality issue I saw initially. It further removes the need to add a m3u file on the Kodi machine, as it instead connects to the JSON-RPC API in Kodi and simply ask Kodi to play from the stream created using GStreamer. Streaming the desktop to Kodi now become trivial. Copy the script below, run it with the DNS name or IP address of the kodi server to stream to as the only argument, and watch your screen show up on the Kodi screen. Note, it depend on multicast on the local network, so if you need to stream outside the local network, the script must be modified. Also note, I have no idea if audio work, as I only care about the picture part.

#!/bin/sh
#
# Stream the Linux desktop view to Kodi.  See
# https://people.skolelinux.org/pere/blog/Streaming_the_Linux_desktop_to_Kodi_using_VLC_and_RTSP.html
# for backgorund information.

# Make sure the stream is stopped in Kodi and the gstreamer process is
# killed if something go wrong (for example if curl is unable to find the
# kodi server).  Do the same when interrupting this script.
kodicmd() {
    host="$1"
    cmd="$2"
    params="$3"
    curl --silent --header 'Content-Type: application/json' \
	 --data-binary "{ \"id\": 1, \"jsonrpc\": \"2.0\", \"method\": \"$cmd\", \"params\": $params }" \
	 "http://$host/jsonrpc"
}
cleanup() {
    if [ -n "$kodihost" ] ; then
	# Stop the playing when we end
	playerid=$(kodicmd "$kodihost" Player.GetActivePlayers "{}" |
			    jq .result[].playerid)
	kodicmd "$kodihost" Player.Stop "{ \"playerid\" : $playerid }" > /dev/null
    fi
    if [ "$gstpid" ] && kill -0 "$gstpid" >/dev/null 2>&1; then
	kill "$gstpid"
    fi
}
trap cleanup EXIT INT

if [ -n "$1" ]; then
    kodihost=$1
    shift
else
    kodihost=kodi.local
fi

mcast=239.255.0.1
mcastport=1234
mcastttl=1

pasrc=$(pactl list | grep -A2 'Source #' | grep 'Name: .*\.monitor$' | \
  cut -d" " -f2|head -1)
gst-launch-1.0 ximagesrc use-damage=0 ! video/x-raw,framerate=30/1 ! \
  videoconvert ! queue2 ! \
  x264enc bitrate=8000 speed-preset=superfast tune=zerolatency qp-min=30 \
  key-int-max=15 bframes=2 ! video/x-h264,profile=high ! queue2 ! \
  mpegtsmux alignment=7 name=mux ! rndbuffersize max=1316 min=1316 ! \
  udpsink host=$mcast port=$mcastport ttl-mc=$mcastttl auto-multicast=1 sync=0 \
  pulsesrc device=$pasrc ! audioconvert ! queue2 ! avenc_aac ! queue2 ! mux. \
  > /dev/null 2>&1 &
gstpid=$!

# Give stream a second to get going
sleep 1

# Ask kodi to start streaming using its JSON-RPC API
kodicmd "$kodihost" Player.Open \
	"{\"item\": { \"file\": \"udp://@$mcast:$mcastport\" } }" > /dev/null

# wait for gst to end
wait "$gstpid"

I hope you find the approach useful. I know I do.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

PS: See the followup post for a even better approach.

A while back, I was asked by a friend how to stream the desktop to my projector connected to Kodi. I sadly had to admit that I had no idea, as it was a task I never had tried. Since then, I have been looking for a way to do so, preferable without much extra software to install on either side. Today I found a way that seem to kind of work. Not great, but it is a start.

I had a look at several approaches, for example using uPnP DLNA as described in 2011, but it required a uPnP server, fuse and local storage enough to store the stream locally. This is not going to work well for me, lacking enough free space, and it would impossible for my friend to get working.

Next, it occurred to me that perhaps I could use VLC to create a video stream that Kodi could play. Preferably using broadcast/multicast, to avoid having to change any setup on the Kodi side when starting such stream. Unfortunately, the only recipe I could find using multicast used the rtp protocol, and this protocol seem to not be supported by Kodi.

On the other hand, the rtsp protocol is working! Unfortunately I have to specify the IP address of the streaming machine in both the sending command and the file on the Kodi server. But it is showing my desktop, and thus allow us to have a shared look on the big screen at the programs I work on.

I did not spend much time investigating codeces. I combined the rtp and rtsp recipes from the VLC Streaming HowTo/Command Line Examples, and was able to get this working on the desktop/streaming end.

vlc screen:// --sout \
  '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{dst=projector.local,port=1234,sdp=rtsp://192.168.11.4:8080/test.sdp}'

I ssh-ed into my Kodi box and created a file like this with the same IP address:

echo rtsp://192.168.11.4:8080/test.sdp \
  > /storage/videos/screenstream.m3u

Note the 192.168.11.4 IP address is my desktops IP address. As far as I can tell the IP must be hardcoded for this to work. In other words, if someone elses machine is going to do the steaming, you have to update screenstream.m3u on the Kodi machine and adjust the vlc recipe. To get started, locate the file in Kodi and select the m3u file while the VLC stream is running. The desktop then show up in my big screen. :)

When using the same technique to stream a video file with audio, the audio quality is really bad. No idea if the problem is package loss or bad parameters for the transcode. I do not know VLC nor Kodi enough to tell.

Update 2018-07-12: Johannes Schauer send me a few succestions and reminded me about an important step. The "screen:" input source is only available once the vlc-plugin-access-extra package is installed on Debian. Without it, you will see this error message: "VLC is unable to open the MRL 'screen://'. Check the log for details." He further found that it is possible to drop some parts of the VLC command line to reduce the amount of hardcoded information. It is also useful to consider using cvlc to avoid having the VLC window in the desktop view. In sum, this give us this command line on the source end

cvlc screen:// --sout \
  '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{sdp=rtsp://:8080/}'

and this on the Kodi end

echo rtsp://192.168.11.4:8080/ \
  > /storage/videos/screenstream.m3u

Still bad image quality, though. But I did discover that streaming a DVD using dvdsimple:///dev/dvd as the source had excellent video and audio quality, so I guess the issue is in the input or transcoding parts, not the rtsp part. I've tried to change the vb and ab parameters to use more bandwidth, but it did not make a difference.

I further received a suggestion from Einar Haraldseid to try using gstreamer instead of VLC, and this proved to work great! He also provided me with the trick to get Kodi to use a multicast stream as its source. By using this monstrous oneliner, I can stream my desktop with good video quality in reasonable framerate to the 239.255.0.1 multicast address on port 1234:

gst-launch-1.0 ximagesrc use-damage=0 ! video/x-raw,framerate=30/1 ! \
  videoconvert ! queue2 ! \
  x264enc bitrate=8000 speed-preset=superfast tune=zerolatency qp-min=30 \
  key-int-max=15 bframes=2 ! video/x-h264,profile=high ! queue2 ! \
  mpegtsmux alignment=7 name=mux ! rndbuffersize max=1316 min=1316 ! \
  udpsink host=239.255.0.1 port=1234 ttl-mc=1 auto-multicast=1 sync=0 \
  pulsesrc device=$(pactl list | grep -A2 'Source #' | \
    grep 'Name: .*\.monitor$' |  cut -d" " -f2|head -1) ! \
  audioconvert ! queue2 ! avenc_aac ! queue2 ! mux.

and this on the Kodi end

echo udp://@239.255.0.1:1234 \
  > /storage/videos/screenstream.m3u

Note the trick to pick a valid pulseaudio source. It might not pick the one you need. This approach will of course lead to trouble if more than one source uses the same multicast port and address. Note the ttl-mc=1 setting, which limit the multicast packages to the local network. If the value is increased, your screen will be broadcasted further, one network "hop" for each increase (read up on multicast to learn more. :)!

Having cracked how to get Kodi to receive multicast streams, I could use this VLC command to stream to the same multicast address. The image quality is way better than the rtsp approach, but gstreamer seem to be doing a better job.

cvlc screen:// --sout '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{mux=ts,dst=239.255.0.1,port=1234,sdp=sap}'

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Five years ago, I measured what the most supported MIME type in Debian was, by analysing the desktop files in all packages in the archive. Since then, the DEP-11 AppStream system has been put into production, making the task a lot easier. This made me want to repeat the measurement, to see how much things changed. Here are the new numbers, for unstable only this time:

Debian Unstable:

  count MIME type
  ----- -----------------------
     56 image/jpeg
     55 image/png
     49 image/tiff
     48 image/gif
     39 image/bmp
     38 text/plain
     37 audio/mpeg
     34 application/ogg
     33 audio/x-flac
     32 audio/x-mp3
     30 audio/x-wav
     30 audio/x-vorbis+ogg
     29 image/x-portable-pixmap
     27 inode/directory
     27 image/x-portable-bitmap
     27 audio/x-mpeg
     26 application/x-ogg
     25 audio/x-mpegurl
     25 audio/ogg
     24 text/html

The list was created like this using a sid chroot: "cat /var/lib/apt/lists/*sid*_dep11_Components-amd64.yml.gz| zcat | awk '/^ - \S+\/\S+$/ {print $2 }' | sort | uniq -c | sort -nr | head -20"

It is interesting to see how image formats have passed text/plain as the most announced supported MIME type. These days, thanks to the AppStream system, if you run into a file format you do not know, and want to figure out which packages support the format, you can find the MIME type of the file using "file --mime <filename>", and then look up all packages announcing support for this format in their AppStream metadata (XML or .desktop file) using "appstreamcli what-provides mimetype <mime-type>. For example if you, like me, want to know which packages support inode/directory, you can get a list like this:

% appstreamcli what-provides mimetype inode/directory | grep Package: | sort
Package: anjuta
Package: audacious
Package: baobab
Package: cervisia
Package: chirp
Package: dolphin
Package: doublecmd-common
Package: easytag
Package: enlightenment
Package: ephoto
Package: filelight
Package: gwenview
Package: k4dirstat
Package: kaffeine
Package: kdesvn
Package: kid3
Package: kid3-qt
Package: nautilus
Package: nemo
Package: pcmanfm
Package: pcmanfm-qt
Package: qweborf
Package: ranger
Package: sirikali
Package: spacefm
Package: spacefm
Package: vifm
%

Using the same method, I can quickly discover that the Sketchup file format is not yet supported by any package in Debian:

% appstreamcli what-provides mimetype  application/vnd.sketchup.skp
Could not find component providing 'mimetype::application/vnd.sketchup.skp'.
%

Yesterday I used it to figure out which packages support the STL 3D format:

% appstreamcli what-provides mimetype  application/sla|grep Package
Package: cura
Package: meshlab
Package: printrun
%

PS: A new version of Cura was uploaded to Debian yesterday.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Quite regularly, I let my Debian Sid/Unstable chroot stay untouch for a while, and when I need to update it there is not enough free space on the disk for apt to do a normal 'apt upgrade'. I normally would resolve the issue by doing 'apt install <somepackages>' to upgrade only some of the packages in one batch, until the amount of packages to download fall below the amount of free space available. Today, I had about 500 packages to upgrade, and after a while I got tired of trying to install chunks of packages manually. I concluded that I did not have the spare hours required to complete the task, and decided to see if I could automate it. I came up with this small script which I call 'apt-in-chunks':

#!/bin/sh
#
# Upgrade packages when the disk is too full to upgrade every
# upgradable package in one lump.  Fetching packages to upgrade using
# apt, and then installing using dpkg, to avoid changing the package
# flag for manual/automatic.

set -e

ignore() {
    if [ "$1" ]; then
	grep -v "$1"
    else
	cat
    fi
}

for p in $(apt list --upgradable | ignore "$@" |cut -d/ -f1 | grep -v '^Listing...'); do
    echo "Upgrading $p"
    apt clean
    apt install --download-only -y $p
    for f in /var/cache/apt/archives/*.deb; do
	if [ -e "$f" ]; then
	    dpkg -i /var/cache/apt/archives/*.deb
	    break
	fi
    done
done

The script will extract the list of packages to upgrade, try to download the packages needed to upgrade one package, install the downloaded packages using dpkg. The idea is to upgrade packages without changing the APT mark for the package (ie the one recording of the package was manually requested or pulled in as a dependency). To use it, simply run it as root from the command line. If it fail, try 'apt install -f' to clean up the mess and run the script again. This might happen if the new packages conflict with one of the old packages. dpkg is unable to remove, while apt can do this.

It take one option, a package to ignore in the list of packages to upgrade. The option to ignore a package is there to be able to skip the packages that are simply too large to unpack. Today this was 'ghc', but I have run into other large packages causing similar problems earlier (like TeX).

Update 2018-07-08: Thanks to Paul Wise, I am aware of two alternative ways to handle this. The "unattended-upgrades --minimal-upgrade-steps" option will try to calculate upgrade sets for each package to upgrade, and then upgrade them in order, smallest set first. It might be a better option than my above mentioned script. Also, "aptutude upgrade" can upgrade single packages, thus avoiding the need for using "dpkg -i" in the script above.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

A new version of the 3D printer slicer software Cura, version 3.1.0, is now available in Debian Testing (aka Buster) and Debian Unstable (aka Sid). I hope you find it useful. It was uploaded the last few days, and the last update will enter testing tomorrow. See the release notes for the list of bug fixes and new features. Version 3.2 was announced 6 days ago. We will try to get it into Debian as well.

More information related to 3D printing is available on the 3D printing and 3D printer wiki pages in Debian.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

After several months of working and waiting, I am happy to report that the nice and user friendly 3D printer slicer software Cura just entered Debian Unstable. It consist of five packages, cura, cura-engine, libarcus, fdm-materials, libsavitar and uranium. The last two, uranium and cura, entered Unstable yesterday. This should make it easier for Debian users to print on at least the Ultimaker class of 3D printers. My nearest 3D printer is an Ultimaker 2+, so it will make life easier for at least me. :)

The work to make this happen was done by Gregor Riepl, and I was happy to assist him in sponsoring the packages. With the introduction of Cura, Debian is up to three 3D printer slicers at your service, Cura, Slic3r and Slic3r Prusa. If you own or have access to a 3D printer, give it a go. :)

The 3D printer software is maintained by the 3D printer Debian team, flocking together on the 3dprinter-general mailing list and the #debian-3dprinting IRC channel.

The next step for Cura in Debian is to update the cura package to version 3.0.3 and then update the entire set of packages to version 3.1.0 which showed up the last few days.

At my nearby maker space, Sonen, I heard the story that it was easier to generate gcode files for theyr 3D printers (Ultimake 2+) on Windows and MacOS X than Linux, because the software involved had to be manually compiled and set up on Linux while premade packages worked out of the box on Windows and MacOS X. I found this annoying, as the software involved, Cura, is free software and should be trivial to get up and running on Linux if someone took the time to package it for the relevant distributions. I even found a request for adding into Debian from 2013, which had seem some activity over the years but never resulted in the software showing up in Debian. So a few days ago I offered my help to try to improve the situation.

Now I am very happy to see that all the packages required by a working Cura in Debian are uploaded into Debian and waiting in the NEW queue for the ftpmasters to have a look. You can track the progress on the status page for the 3D printer team.

The uploaded packages are a bit behind upstream, and was uploaded now to get slots in the NEW queue while we work up updating the packages to the latest upstream version.

On a related note, two competitors for Cura, which I found harder to use and was unable to configure correctly for Ultimaker 2+ in the short time I spent on it, are already in Debian. If you are looking for 3D printer "slicers" and want something already available in Debian, check out slic3r and slic3r-prusa. The latter is a fork of the former.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Every mobile phone announce its existence over radio to the nearby mobile cell towers. And this radio chatter is available for anyone with a radio receiver capable of receiving them. Details about the mobile phones with very good accuracy is of course collected by the phone companies, but this is not the topic of this blog post. The mobile phone radio chatter make it possible to figure out when a cell phone is nearby, as it include the SIM card ID (IMSI). By paying attention over time, one can see when a phone arrive and when it leave an area. I believe it would be nice to make this information more available to the general public, to make more people aware of how their phones are announcing their whereabouts to anyone that care to listen.

I am very happy to report that we managed to get something visualizing this information up and running for Oslo Skaperfestival 2017 (Oslo Makers Festival) taking place today and tomorrow at Deichmanske library. The solution is based on the simple recipe for listening to GSM chatter I posted a few days ago, and will show up at the stand of Åpen Sone from the Computer Science department of the University of Oslo. The presentation will show the nearby mobile phones (aka IMSIs) as dots in a web browser graph, with lines to the dot representing mobile base station it is talking to. It was working in the lab yesterday, and was moved into place this morning.

We set up a fairly powerful desktop machine using Debian Buster/Testing with several (five, I believe) RTL2838 DVB-T receivers connected and visualize the visible cell phone towers using an English version of Hopglass. A fairly powerfull machine is needed as the grgsm_livemon_headless processes from gr-gsm converting the radio signal to data packages is quite CPU intensive.

The frequencies to listen to, are identified using a slightly patched scan-and-livemon (to set the --args values for each receiver), and the Hopglass data is generated using the patches in my meshviewer-output branch. For some reason we could not get more than four SDRs working. There is also a geographical map trying to show the location of the base stations, but I believe their coordinates are hardcoded to some random location in Germany, I believe. The code should be replaced with code to look up location in a text file, a sqlite database or one of the online databases mentioned in the github issue for the topic.

If this sound interesting, visit the stand at the festival!

A little more than a month ago I wrote how to observe the SIM card ID (aka IMSI number) of mobile phones talking to nearby mobile phone base stations using Debian GNU/Linux and a cheap USB software defined radio, and thus being able to pinpoint the location of people and equipment (like cars and trains) with an accuracy of a few kilometer. Since then we have worked to make the procedure even simpler, and it is now possible to do this without any manual frequency tuning and without building your own packages.

The gr-gsm package is now included in Debian testing and unstable, and the IMSI-catcher code no longer require root access to fetch and decode the GSM data collected using gr-gsm.

Here is an updated recipe, using packages built by Debian and a git clone of two python scripts:

1. Start with a Debian machine running the Buster version (aka testing).
2. Run 'apt install gr-gsm python-numpy python-scipy python-scapy' as root to install required packages.
3. Fetch the code decoding GSM packages using 'git clone github.com/Oros42/IMSI-catcher.git'.
4. Insert USB software defined radio supported by GNU Radio.
5. Enter the IMSI-catcher directory and run 'python scan-and-livemon' to locate the frequency of nearby base stations and start listening for GSM packages on one of them.
6. Enter the IMSI-catcher directory and run 'python simple_IMSI-catcher.py' to display the collected information.

Note, due to a bug somewhere the scan-and-livemon program (actually its underlying program grgsm_scanner) do not work with the HackRF radio. It does work with RTL 8232 and other similar USB radio receivers you can get very cheaply (for example from ebay), so for now the solution is to scan using the RTL radio and only use HackRF for fetching GSM data.

As far as I can tell, a cell phone only show up on one of the frequencies at the time, so if you are going to track and count every cell phone around you, you need to listen to all the frequencies used. To listen to several frequencies, use the --numrecv argument to scan-and-livemon to use several receivers. Further, I am not sure if phones using 3G or 4G will show as talking GSM to base stations, so this approach might not see all phones around you. I typically see 0-400 IMSI numbers an hour when looking around where I live.

I've tried to run the scanner on a Raspberry Pi 2 and 3 running Debian Buster, but the grgsm_livemon_headless process seem to be too CPU intensive to keep up. When GNU Radio print 'O' to stdout, I am told there it is caused by a buffer overflow between the radio and GNU Radio, caused by the program being unable to read the GSM data fast enough. If you see a stream of 'O's from the terminal where you started scan-and-livemon, you need a give the process more CPU power. Perhaps someone are able to optimize the code to a point where it become possible to set up RPi3 based GSM sniffers? I tried using Raspbian instead of Debian, but there seem to be something wrong with GNU Radio on raspbian, causing glibc to abort().

On friday, I came across an interesting article in the Norwegian web based ICT news magazine digi.no on how to collect the IMSI numbers of nearby cell phones using the cheap DVB-T software defined radios. The article refered to instructions and a recipe by Keld Norman on Youtube on how to make a simple $7 IMSI Catcher, and I decided to test them out.

The instructions said to use Ubuntu, install pip using apt (to bypass apt), use pip to install pybombs (to bypass both apt and pip), and the ask pybombs to fetch and build everything you need from scratch. I wanted to see if I could do the same on the most recent Debian packages, but this did not work because pybombs tried to build stuff that no longer build with the most recent openssl library or some other version skew problem. While trying to get this recipe working, I learned that the apt->pip->pybombs route was a long detour, and the only piece of software dependency missing in Debian was the gr-gsm package. I also found out that the lead upstream developer of gr-gsm (the name stand for GNU Radio GSM) project already had a set of Debian packages provided in an Ubuntu PPA repository. All I needed to do was to dget the Debian source package and built it.

The IMSI collector is a python script listening for packages on the loopback network device and printing to the terminal some specific GSM packages with IMSI numbers in them. The code is fairly short and easy to understand. The reason this work is because gr-gsm include a tool to read GSM data from a software defined radio like a DVB-T USB stick and other software defined radios, decode them and inject them into a network device on your Linux machine (using the loopback device by default). This proved to work just fine, and I've been testing the collector for a few days now.

The updated and simpler recipe is thus to

1. start with a Debian machine running Stretch or newer,
2. build and install the gr-gsm package available from http://ppa.launchpad.net/ptrkrysik/gr-gsm/ubuntu/pool/main/g/gr-gsm/,
3. clone the git repostory from https://github.com/Oros42/IMSI-catcher,
4. run grgsm_livemon and adjust the frequency until the terminal where it was started is filled with a stream of text (meaning you found a GSM station).
5. go into the IMSI-catcher directory and run 'sudo python simple_IMSI-catcher.py' to extract the IMSI numbers.

To make it even easier in the future to get this sniffer up and running, I decided to package the gr-gsm project for Debian (WNPP #871055), and the package was uploaded into the NEW queue today. Luckily the gnuradio maintainer has promised to help me, as I do not know much about gnuradio stuff yet.

I doubt this "IMSI cacher" is anywhere near as powerfull as commercial tools like The Spy Phone Portable IMSI / IMEI Catcher or the Harris Stingray, but I hope the existance of cheap alternatives can make more people realise how their whereabouts when carrying a cell phone is easily tracked. Seeing the data flow on the screen, realizing that I live close to a police station and knowing that the police is also wearing cell phones, I wonder how hard it would be for criminals to track the position of the police officers to discover when there are police near by, or for foreign military forces to track the location of the Norwegian military forces, or for anyone to track the location of government officials...

It is worth noting that the data reported by the IMSI-catcher script mentioned above is only a fraction of the data broadcasted on the GSM network. It will only collect one frequency at the time, while a typical phone will be using several frequencies, and not all phones will be using the frequencies tracked by the grgsm_livemod program. Also, there is a lot of radio chatter being ignored by the simple_IMSI-catcher script, which would be collected by extending the parser code. I wonder if gr-gsm can be set up to listen to more than one frequency?

I finally received a copy of the Norwegian Bokmål edition of "The Debian Administrator's Handbook". This test copy arrived in the mail a few days ago, and I am very happy to hold the result in my hand. We spent around one and a half year translating it. This paperbook edition is available from lulu.com. If you buy it quickly, you save 25% on the list price. The book is also available for download in electronic form as PDF, EPUB and Mobipocket, as can be read online as a web page.

This is the second book I publish (the first was the book "Free Culture" by Lawrence Lessig in English, French and Norwegian Bokmål), and I am very excited to finally wrap up this project. I hope "Håndbok for Debian-administratoren" will be well received.

Aftenposten melder i dag om feil i eksamensoppgavene for eksamen i politikk og menneskerettigheter, der teksten i bokmåls og nynorskutgaven ikke var like. Oppgaveteksten er gjengitt i artikkelen, og jeg ble nysgjerring på om den fri oversetterløsningen Apertium ville gjort en bedre jobb enn Utdanningsdirektoratet. Det kan se slik ut.

Her er bokmålsoppgaven fra eksamenen:

Drøft utfordringene knyttet til nasjonalstatenes og andre aktørers rolle og muligheter til å håndtere internasjonale utfordringer, som for eksempel flykningekrisen.

Vedlegge er eksempler på tekster som kan gi relevante perspektiver på temaet:

1. Flykningeregnskapet 2016, UNHCR og IDMC
2. «Grenseløst Europa for fall» A-Magasinet, 26. november 2015

Dette oversetter Apertium slik:

Drøft utfordringane knytte til nasjonalstatane sine og rolla til andre aktørar og høve til å handtera internasjonale utfordringar, som til dømes *flykningekrisen.

Vedleggja er døme på tekster som kan gje relevante perspektiv på temaet:

1. *Flykningeregnskapet 2016, *UNHCR og *IDMC
2. «*Grenseløst Europa for fall» A-Magasinet, 26. november 2015

Ord som ikke ble forstått er markert med stjerne (*), og trenger ekstra språksjekk. Men ingen ord er forsvunnet, slik det var i oppgaven elevene fikk presentert på eksamen. Jeg mistenker dog at "andre aktørers rolle og muligheter til ..." burde vært oversatt til "rolla til andre aktørar og deira høve til ..." eller noe slikt, men det er kanskje flisespikking. Det understreker vel bare at det alltid trengs korrekturlesning etter automatisk oversettelse.

Over the years, administrating thousand of NFS mounting linux computers at the time, I often needed a way to detect if the machine was experiencing NFS hang. If you try to use df or look at a file or directory affected by the hang, the process (and possibly the shell) will hang too. So you want to be able to detect this without risking the detection process getting stuck too. It has not been obvious how to do this. When the hang has lasted a while, it is possible to find messages like these in dmesg:

nfs: server nfsserver not responding, still trying
nfs: server nfsserver OK

It is hard to know if the hang is still going on, and it is hard to be sure looking in dmesg is going to work. If there are lots of other messages in dmesg the lines might have rotated out of site before they are noticed.

While reading through the nfs client implementation in linux kernel code, I came across some statistics that seem to give a way to detect it. The om_timeouts sunrpc value in the kernel will increase every time the above log entry is inserted into dmesg. And after digging a bit further, I discovered that this value show up in /proc/self/mountstats on Linux.

The mountstats content seem to be shared between files using the same file system context, so it is enough to check one of the mountstats files to get the state of the mount point for the machine. I assume this will not show lazy umounted NFS points, nor NFS mount points in a different process context (ie with a different filesystem view), but that does not worry me.

The content for a NFS mount point look similar to this:

[...]
device /dev/mapper/Debian-var mounted on /var with fstype ext3
device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
        opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
        age:    7863311
        caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
        sec:    flavor=1,pseudoflavor=1
        events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0 
        bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809 
        RPC iostats version: 1.0  p/v: 100003/3 (nfs)
        xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
        per-op statistics
                NULL: 0 0 0 0 0 0 0 0
             GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
             SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
              LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
              ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
            READLINK: 125 125 0 20472 18620 0 1112 1118
                READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
               WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
              CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
               MKDIR: 3680 3680 0 773980 993920 26 23990 24245
             SYMLINK: 903 903 0 233428 245488 6 5865 5917
               MKNOD: 80 80 0 20148 21760 0 299 304
              REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
               RMDIR: 3367 3367 0 645112 484848 22 5782 6002
              RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
                LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
             READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
         READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
              FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
              FSINFO: 2 2 0 232 328 0 1 1
            PATHCONF: 1 1 0 116 140 0 0 0
              COMMIT: 0 0 0 0 0 0 0 0

device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
[...]

The key number to look at is the third number in the per-op list. It is the number of NFS timeouts experiences per file system operation. Here 22 write timeouts and 5 access timeouts. If these numbers are increasing, I believe the machine is experiencing NFS hang. Unfortunately the timeout value do not start to increase right away. The NFS operations need to time out first, and this can take a while. The exact timeout value depend on the setup. For example the defaults for TCP and UDP mount points are quite different, and the timeout value is affected by the soft, hard, timeo and retrans NFS mount options.

The only way I have been able to get working on Debian and RedHat Enterprise Linux for getting the timeout count is to peek in /proc/. But according to Solaris 10 System Administration Guide: Network Services, the 'nfsstat -c' command can be used to get these timeout values. But this do not work on Linux, as far as I can tell. I asked Debian about this, but have not seen any replies yet.

Is there a better way to figure out if a Linux NFS client is experiencing NFS hangs? Is there a way to detect which processes are affected? Is there a way to get the NFS mount going quickly once the network problem causing the NFS hang has been cleared? I would very much welcome some clues, as we regularly run into NFS hangs.

For almost a year now, we have been working on making a Norwegian Bokmål edition of The Debian Administrator's Handbook. Now, thanks to the tireless effort of Ole-Erik, Ingrid and Andreas, the initial translation is complete, and we are working on the proof reading to ensure consistent language and use of correct computer science terms. The plan is to make the book available on paper, as well as in electronic form. For that to happen, the proof reading must be completed and all the figures need to be translated. If you want to help out, get in touch.

A fresh PDF edition in A4 format (the final book will have smaller pages) of the book created every morning is available for proofreading. If you find any errors, please visit Weblate and correct the error. The state of the translation including figures is a useful source for those provide Norwegian bokmål screen shots and figures.

A few days ago I ordered a small batch of the ChaosKey, a small USB dongle for generating entropy created by Bdale Garbee and Keith Packard. Yesterday it arrived, and I am very happy to report that it work great! According to its designers, to get it to work out of the box, you need the Linux kernel version 4.1 or later. I tested on a Debian Stretch machine (kernel version 4.9), and there it worked just fine, increasing the available entropy very quickly. I wrote a small test oneliner to test. It first print the current entropy level, drain /dev/random, and then print the entropy level for five seconds. Here is the situation without the ChaosKey inserted:

% cat /proc/sys/kernel/random/entropy_avail; \
  dd bs=1M if=/dev/random of=/dev/null count=1; \
  for n in $(seq 1 5); do \
     cat /proc/sys/kernel/random/entropy_avail; \
     sleep 1; \
  done
300
0+1 oppføringer inn
0+1 oppføringer ut
28 byte kopiert, 0,000264565 s, 106 kB/s
4
8
12
17
21
%

The entropy level increases by 3-4 every second. In such case any application requiring random bits (like a HTTPS enabled web server) will halt and wait for more entrpy. And here is the situation with the ChaosKey inserted:

% cat /proc/sys/kernel/random/entropy_avail; \
  dd bs=1M if=/dev/random of=/dev/null count=1; \
  for n in $(seq 1 5); do \
     cat /proc/sys/kernel/random/entropy_avail; \
     sleep 1; \
  done
1079
0+1 oppføringer inn
0+1 oppføringer ut
104 byte kopiert, 0,000487647 s, 213 kB/s
433
1028
1031
1035
1038
%

Quite the difference. :) I bought a few more than I need, in case someone want to buy one here in Norway. :)

Update: The dongle was presented at Debconf last year. You might find the talk recording illuminating. It explains exactly what the source of randomness is, if you are unable to spot it from the schema drawing available from the ChaosKey web site linked at the start of this blog post.

Did you ever wonder where the web trafic really flow to reach the web servers, and who own the network equipment it is flowing through? It is possible to get a glimpse of this from using traceroute, but it is hard to find all the details. Many years ago, I wrote a system to map the Norwegian Internet (trying to figure out if our plans for a network game service would get low enough latency, and who we needed to talk to about setting up game servers close to the users. Back then I used traceroute output from many locations (I asked my friends to run a script and send me their traceroute output) to create the graph and the map. The output from traceroute typically look like this:

traceroute to www.stortinget.no (85.88.67.10), 30 hops max, 60 byte packets
 1  uio-gw10.uio.no (129.240.202.1)  0.447 ms  0.486 ms  0.621 ms
 2  uio-gw8.uio.no (129.240.24.229)  0.467 ms  0.578 ms  0.675 ms
 3  oslo-gw1.uninett.no (128.39.65.17)  0.385 ms  0.373 ms  0.358 ms
 4  te3-1-2.br1.fn3.as2116.net (193.156.90.3)  1.174 ms  1.172 ms  1.153 ms
 5  he16-1-1.cr1.san110.as2116.net (195.0.244.234)  2.627 ms he16-1-1.cr2.oslosda310.as2116.net (195.0.244.48)  3.172 ms he16-1-1.cr1.san110.as2116.net (195.0.244.234)  2.857 ms
 6  ae1.ar8.oslosda310.as2116.net (195.0.242.39)  0.662 ms  0.637 ms ae0.ar8.oslosda310.as2116.net (195.0.242.23)  0.622 ms
 7  89.191.10.146 (89.191.10.146)  0.931 ms  0.917 ms  0.955 ms
 8  * * *
 9  * * *
[...]

This show the DNS names and IP addresses of (at least some of the) network equipment involved in getting the data traffic from me to the www.stortinget.no server, and how long it took in milliseconds for a package to reach the equipment and return to me. Three packages are sent, and some times the packages do not follow the same path. This is shown for hop 5, where three different IP addresses replied to the traceroute request.

There are many ways to measure trace routes. Other good traceroute implementations I use are traceroute (using ICMP packages) mtr (can do both ICMP, UDP and TCP) and scapy (python library with ICMP, UDP, TCP traceroute and a lot of other capabilities). All of them are easily available in Debian.

This time around, I wanted to know the geographic location of different route points, to visualize how visiting a web page spread information about the visit to a lot of servers around the globe. The background is that a web site today often will ask the browser to get from many servers the parts (for example HTML, JSON, fonts, JavaScript, CSS, video) required to display the content. This will leak information about the visit to those controlling these servers and anyone able to peek at the data traffic passing by (like your ISP, the ISPs backbone provider, FRA, GCHQ, NSA and others).

Lets pick an example, the Norwegian parliament web site www.stortinget.no. It is read daily by all members of parliament and their staff, as well as political journalists, activits and many other citizens of Norway. A visit to the www.stortinget.no web site will ask your browser to contact 8 other servers: ajax.googleapis.com, insights.hotjar.com, script.hotjar.com, static.hotjar.com, stats.g.doubleclick.net, www.google-analytics.com, www.googletagmanager.com and www.netigate.se. I extracted this by asking PhantomJS to visit the Stortinget web page and tell me all the URLs PhantomJS downloaded to render the page (in HAR format using their netsniff example. I am very grateful to Gorm for showing me how to do this). My goal is to visualize network traces to all IP addresses behind these DNS names, do show where visitors personal information is spread when visiting the page.

When I had a look around for options, I could not find any good free software tools to do this, and decided I needed my own traceroute wrapper outputting KML based on locations looked up using GeoIP. KML is easy to work with and easy to generate, and understood by several of the GIS tools I have available. I got good help from by NUUG colleague Anders Einar with this, and the result can be seen in my kmltraceroute git repository. Unfortunately, the quality of the free GeoIP databases I could find (and the for-pay databases my friends had access to) is not up to the task. The IP addresses of central Internet infrastructure would typically be placed near the controlling companies main office, and not where the router is really located, as you can see from the KML file I created using the GeoLite City dataset from MaxMind.

I also had a look at the visual traceroute graph created by the scrapy project, showing IP network ownership (aka AS owner) for the IP address in question. The graph display a lot of useful information about the traceroute in SVG format, and give a good indication on who control the network equipment involved, but it do not include geolocation. This graph make it possible to see the information is made available at least for UNINETT, Catchcom, Stortinget, Nordunet, Google, Amazon, Telia, Level 3 Communications and NetDNA.

In the process, I came across the web service GeoTraceroute by Salim Gasmi. Its methology of combining guesses based on DNS names, various location databases and finally use latecy times to rule out candidate locations seemed to do a very good job of guessing correct geolocation. But it could only do one trace at the time, did not have a sensor in Norway and did not make the geolocations easily available for postprocessing. So I contacted the developer and asked if he would be willing to share the code (he refused until he had time to clean it up), but he was interested in providing the geolocations in a machine readable format, and willing to set up a sensor in Norway. So since yesterday, it is possible to run traces from Norway in this service thanks to a sensor node set up by the NUUG assosiation, and get the trace in KML format for further processing.

Here we can see a lot of trafic passes Sweden on its way to Denmark, Germany, Holland and Ireland. Plenty of places where the Snowden confirmations verified the traffic is read by various actors without your best interest as their top priority.

Combining KML files is trivial using a text editor, so I could loop over all the hosts behind the urls imported by www.stortinget.no and ask for the KML file from GeoTraceroute, and create a combined KML file with all the traces (unfortunately only one of the IP addresses behind the DNS name is traced this time. To get them all, one would have to request traces using IP number instead of DNS names from GeoTraceroute). That might be the next step in this project.

Armed with these tools, I find it a lot easier to figure out where the IP traffic moves and who control the boxes involved in moving it. And every time the link crosses for example the Swedish border, we can be sure Swedish Signal Intelligence (FRA) is listening, as GCHQ do in Britain and NSA in USA and cables around the globe. (Hm, what should we tell them? :) Keep that in mind if you ever send anything unencrypted over the Internet.

PS: KML files are drawn using the KML viewer from Ivan Rublev, as it was less cluttered than the local Linux application Marble. There are heaps of other options too.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

I received a very nice Christmas present today. As my regular readers probably know, I have been working on the the Isenkram system for many years. The goal of the Isenkram system is to make it easier for users to figure out what to install to get a given piece of hardware to work in Debian, and a key part of this system is a way to map hardware to packages. Isenkram have its own mapping database, and also uses data provided by each package using the AppStream metadata format. And today, AppStream in Debian learned to look up hardware the same way Isenkram is doing it, ie using fnmatch():

% appstreamcli what-provides modalias \
  usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00
Identifier: pymissile [generic]
Name: pymissile
Summary: Control original Striker USB Missile Launcher
Package: pymissile
% appstreamcli what-provides modalias usb:v0694p0002d0000
Identifier: libnxt [generic]
Name: libnxt
Summary: utility library for talking to the LEGO Mindstorms NXT brick
Package: libnxt
---
Identifier: t2n [generic]
Name: t2n
Summary: Simple command-line tool for Lego NXT
Package: t2n
---
Identifier: python-nxt [generic]
Name: python-nxt
Summary: Python driver/interface/wrapper for the Lego Mindstorms NXT robot
Package: python-nxt
---
Identifier: nbc [generic]
Name: nbc
Summary: C compiler for LEGO Mindstorms NXT bricks
Package: nbc
%

A similar query can be done using the combined AppStream and Isenkram databases using the isenkram-lookup tool:

% isenkram-lookup usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00
pymissile
% isenkram-lookup usb:v0694p0002d0000
libnxt
nbc
python-nxt
t2n
%

You can find modalias values relevant for your machine using cat $(find /sys/devices/ -name modalias).

If you want to make this system a success and help Debian users make the most of the hardware they have, please help add AppStream metadata for your package following the guidelines documented in the wiki. So far only 11 packages provide such information, among the several hundred hardware specific packages in Debian. The Isenkram database on the other hand contain 101 packages, mostly related to USB dongles. Most of the packages with hardware mapping in AppStream are LEGO Mindstorms related, because I have, as part of my involvement in the Debian LEGO team given priority to making sure LEGO users get proposed the complete set of packages in Debian for that particular hardware. The team also got a nice Christmas present today. The nxt-firmware package made it into Debian. With this package in place, it is now possible to use the LEGO Mindstorms NXT unit with only free software, as the nxt-firmware package contain the source and firmware binaries for the NXT brick.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

The Isenkram system I wrote two years ago to make it easier in Debian to find and install packages to get your hardware dongles to work, is still going strong. It is a system to look up the hardware present on or connected to the current system, and map the hardware to Debian packages. It can either be done using the tools in isenkram-cli or using the user space daemon in the isenkram package. The latter will notify you, when inserting new hardware, about what packages to install to get the dongle working. It will even provide a button to click on to ask packagekit to install the packages.

Here is an command line example from my Thinkpad laptop:

% isenkram-lookup  
bluez
cheese
ethtool
fprintd
fprintd-demo
gkrellm-thinkbat
hdapsd
libpam-fprintd
pidgin-blinklight
thinkfan
tlp
tp-smapi-dkms
tp-smapi-source
tpb
%

It can also list the firware package providing firmware requested by the load kernel modules, which in my case is an empty list because I have all the firmware my machine need:

% /usr/sbin/isenkram-autoinstall-firmware -l
info: did not find any firmware files requested by loaded kernel modules.  exiting
%

The last few days I had a look at several of the around 250 packages in Debian with udev rules. These seem like good candidates to install when a given hardware dongle is inserted, and I found several that should be proposed by isenkram. I have not had time to check all of them, but am happy to report that now there are 97 packages packages mapped to hardware by Isenkram. 11 of these packages provide hardware mapping using AppStream, while the rest are listed in the modaliases file provided in isenkram.

These are the packages with hardware mappings at the moment. The marked packages are also announcing their hardware support using AppStream, for everyone to use:

air-quality-sensor, alsa-firmware-loaders, argyll, array-info, avarice, avrdude, b43-fwcutter, bit-babbler, bluez, bluez-firmware, brltty, broadcom-sta-dkms, calibre, cgminer, cheese, colord, colorhug-client, dahdi-firmware-nonfree, dahdi-linux, dfu-util, dolphin-emu, ekeyd, ethtool, firmware-ipw2x00, fprintd, fprintd-demo, galileo, gkrellm-thinkbat, gphoto2, gpsbabel, gpsbabel-gui, gpsman, gpstrans, gqrx-sdr, gr-fcdproplus, gr-osmosdr, gtkpod, hackrf, hdapsd, hdmi2usb-udev, hpijs-ppds, hplip, ipw3945-source, ipw3945d, kde-config-tablet, kinect-audio-setup, libnxt, libpam-fprintd, lomoco, madwimax, minidisc-utils, mkgmap, msi-keyboard, mtkbabel, nbc, nqc, nut-hal-drivers, ola, open-vm-toolbox, open-vm-tools, openambit, pcgminer, pcmciautils, pcscd, pidgin-blinklight, printer-driver-splix, pymissile, python-nxt, qlandkartegt, qlandkartegt-garmin, rosegarden, rt2x00-source, sispmctl, soapysdr-module-hackrf, solaar, squeak-plugins-scratch, sunxi-tools, t2n, thinkfan, thinkfinger-tools, tlp, tp-smapi-dkms, tp-smapi-source, tpb, tucnak, uhd-host, usbmuxd, viking, virtualbox-ose-guest-x11, w1retap, xawtv, xserver-xorg-input-vmmouse, xserver-xorg-input-wacom, xserver-xorg-video-qxl, xserver-xorg-video-vmware, yubikey-personalization and zd1211-firmware

If you know of other packages, please let me know with a wishlist bug report against the isenkram-cli package, and ask the package maintainer to add AppStream metadata according to the guidelines to provide the information for everyone. In time, I hope to get rid of the isenkram specific hardware mapping and depend exclusively on AppStream.

Note, the AppStream metadata for broadcom-sta-dkms is matching too much hardware, and suggest that the package with with any ethernet card. See bug #838735 for the details. I hope the maintainer find time to address it soon. In the mean time I provide an override in isenkram.

In my early years, I played the epic game Elite on my PC. I spent many months trading and fighting in space, and reached the 'elite' fighting status before I moved on. The original Elite game was available on Commodore 64 and the IBM PC edition I played had a 64 KB executable. I am still impressed today that the authors managed to squeeze both a 3D engine and details about more than 2000 planet systems across 7 galaxies into a binary so small.

I have known about the free software game Oolite inspired by Elite for a while, but did not really have time to test it properly until a few days ago. It was great to discover that my old knowledge about trading routes were still valid. But my fighting and flying abilities were gone, so I had to retrain to be able to dock on a space station. And I am still not able to make much resistance when I am attacked by pirates, so I bougth and mounted the most powerful laser in the rear to be able to put up at least some resistance while fleeing for my life. :)

When playing Elite in the late eighties, I had to discover everything on my own, and I had long lists of prices seen on different planets to be able to decide where to trade what. This time I had the advantages of the Elite wiki, where information about each planet is easily available with common price ranges and suggested trading routes. This improved my ability to earn money and I have been able to earn enough to buy a lot of useful equipent in a few days. I believe I originally played for months before I could get a docking computer, while now I could get it after less then a week.

If you like science fiction and dreamed of a life as a vagabond in space, you should try out Oolite. It is available for Linux, MacOSX and Windows, and is included in Debian and derivatives since 2011.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Two years ago, I did some experiments with eatmydata and the Debian installation system, observing how using eatmydata could speed up the installation quite a bit. My testing measured speedup around 20-40 percent for Debian Edu, where we install around 1000 packages from within the installer. The eatmydata package provide a way to disable/delay file system flushing. This is a bit risky in the general case, as files that should be stored on disk will stay only in memory a bit longer than expected, causing problems if a machine crashes at an inconvenient time. But for an installation, if the machine crashes during installation the process is normally restarted, and avoiding disk operations as much as possible to speed up the process make perfect sense.

I added code in the Debian Edu specific installation code to enable eatmydata, but did not have time to push it any further. But a few months ago I picked it up again and worked with the libeatmydata package maintainer Mattia Rizzolo to make it easier for everyone to get this installation speedup in Debian. Thanks to our cooperation There is now an eatmydata-udeb package in Debian testing and unstable, and simply enabling/installing it in debian-installer (d-i) is enough to get the quicker installations. It can be enabled using preseeding. The following untested kernel argument should do the trick:

preseed/early_command="anna-install eatmydata-udeb"

This should ask d-i to install the package inside the d-i environment early in the installation sequence. Having it installed in d-i in turn will make sure the relevant scripts are called just after debootstrap filled /target/ with the freshly installed Debian system to configure apt to run dpkg with eatmydata. This is enough to speed up the installation process. There is a proposal to extend the idea a bit further by using /etc/ld.so.preload instead of apt.conf, but I have not tested its impact.

I Norge er det mange som trenger å skrive både bokmål og nynorsk. Eksamensoppgaver, offentlige brev og nyheter er eksempler på tekster der det er krav om skriftspråk. I tillegg til alle skoleoppgavene som elever over det ganske land skal levere inn hvert år. Det mange ikke vet er at selv om de kommersielle alternativene Google Translate og Bing Translator ikke kan bidra med å oversette mellom bokmål og nynorsk, så finnes det et utmerket fri programvarealternativ som kan. Oversetterverktøyet Apertium har støtte for en rekke språkkombinasjoner, og takket være den utrettelige innsatsen til blant annet Kevin Brubeck Unhammer, kan en bruke webtjenesten til å fylle inn en tekst på bokmål eller nynorsk, og få den automatoversatt til det andre skriftspråket. Resultatet er ikke perfekt, men et svært godt utgangspunkt. Av og til er resultatet så bra at det kan benyttes uten endringer. Jeg vet f.eks. at store deler av Joomla ble oversatt til nynorsk ved hjelp Apertium. Høres det ut som noe du kan ha bruk for? Besøk i så fall Apertium.org og fyll inn teksten din i webskjemaet der.

Hvis du trenger maskinell tilgang til den bakenforliggende teknologien kan du enten installere pakken apertium-nno-nob på en Debian-maskin eller bruke web-API-et tilgjengelig fra api.apertium.org. Se API-dokumentasjonen for detaljer om web-API-et. Her kan du se hvordan resultatet blir for denne teksten som ble skrevet på bokmål over maskinoversatt til nynorsk.

---

I Noreg er det mange som treng å skriva både bokmål og nynorsk. Eksamensoppgåver, offentlege brev og nyhende er døme på tekster der det er krav om skriftspråk. I tillegg til alle skuleoppgåvene som elevar over det ganske land skal levera inn kvart år. Det mange ikkje veit er at sjølv om dei kommersielle alternativa Google *Translate og Bing *Translator ikkje kan bidra med å omsetja mellom bokmål og nynorsk, så finst det eit utmerka fri programvarealternativ som kan. Omsetjarverktøyet *Apertium har støtte for ei rekkje språkkombinasjonar, og takka vera den utrøyttelege innsatsen til blant anna Kevin Brubeck Unhammer, kan ein bruka *webtjenesten til å fylla inn ei tekst på bokmål eller nynorsk, og få den *automatoversatt til det andre skriftspråket. Resultatet er ikkje perfekt, men eit svært godt utgangspunkt. Av og til er resultatet så bra at det kan nyttast utan endringar. Eg veit t.d. at store delar av *Joomla vart omsett til nynorsk ved hjelp *Apertium. Høyrast det ut som noko du kan ha bruk for? Besøk i så fall *Apertium.org og fyll inn teksta di i *webskjemaet der.

Viss du treng *maskinell tilgjenge til den *bakenforliggende teknologien kan du anten installera pakken *apertium-*nno-*nob på ein *Debian-maskin eller bruka *web-*API-eit tilgjengeleg frå *api.*apertium.org. Sjå *API-dokumentasjonen for detaljar om *web-*API-eit. Her kan du sjå korleis resultatet vert for denne teksta som vart skreva på bokmål over *maskinoversatt til nynorsk.

The Coz profiler, a nice profiler able to run benchmarking experiments on the instrumented multi-threaded program, finally made it into Debian unstable yesterday. Lluís Vilanova and I have spent many months since I blogged about the coz tool in August working with upstream to make it suitable for Debian. There are still issues with clang compatibility, inline assembly only working x86 and minimized JavaScript libraries.

To test it, install 'coz-profiler' using apt and run it like this:

coz run --- /path/to/binary-with-debug-info

This will produce a profile.coz file in the current working directory with the profiling information. This is then given to a JavaScript application provided in the package and available from a project web page. To start the local copy, invoke it in a browser like this:

sensible-browser /usr/share/coz-profiler/viewer/index.htm

See the project home page and the USENIX ;login: article on Coz for more information on how it is working.

A while back I received a Gyro sensor for the NXT Mindstorms controller as a birthday present. It had been on my wishlist for a while, because I wanted to build a Segway like balancing lego robot. I had already built a simple balancing robot with the kids, using the light/color sensor included in the NXT kit as the balance sensor, but it was not working very well. It could balance for a while, but was very sensitive to the light condition in the room and the reflective properties of the surface and would fall over after a short while. I wanted something more robust, and had the gyro sensor from HiTechnic I believed would solve it on my wishlist for some years before it suddenly showed up as a gift from my loved ones. :)

Unfortunately I have not had time to sit down and play with it since then. But that changed some days ago, when I was searching for lego segway information and came across a recipe from HiTechnic for building the HTWay, a segway like balancing robot. Build instructions and source code was included, so it was just a question of putting it all together. And thanks to the great work of many Debian developers, the compiler needed to build the source for the NXT is already included in Debian, so I was read to go in less than an hour. The resulting robot do not look very impressive in its simplicity:

Because I lack the infrared sensor used to control the robot in the design from HiTechnic, I had to comment out the last task (taskControl). I simply placed /* and */ around it get the program working without that sensor present. Now it balances just fine until the battery status run low:

Now we would like to teach it how to follow a line and take remote control instructions using the included Bluetooth receiver in the NXT.

If you, like me, love LEGO and want to make sure we find the tools they need to work with LEGO in Debian and all our derivative distributions like Ubuntu, check out the LEGO designers project page and join the Debian LEGO team. Personally I own a RCX and NXT controller (no EV3), and would like to make sure the Debian tools needed to program the systems I own work as they should.

In July I wrote how to get the Signal Chrome/Chromium app working without the ability to receive SMS messages (aka without a cell phone). It is time to share some experiences and provide an updated setup.

The Signal app have worked fine for several months now, and I use it regularly to chat with my loved ones. I had a major snag at the end of my summer vacation, when the the app completely forgot my setup, identity and keys. The reason behind this major mess was running out of disk space. To avoid that ever happening again I have started storing everything in userdata/ in git, to be able to roll back to an earlier version if the files are wiped by mistake. I had to use it once after introducing the git backup. When rolling back to an earlier version, one need to use the 'reset session' option in Signal to get going, and notify the people you talk with about the problem. I assume there is some sequence number tracking in the protocol to detect rollback attacks. The git repository is rather big (674 MiB so far), but I have not tried to figure out if some of the content can be added to a .gitignore file due to lack of spare time.

I've also hit the 90 days timeout blocking, and noticed that this make it impossible to send messages using Signal. I could still receive them, but had to patch the code with a new timestamp to send. I believe the timeout is added by the developers to force people to upgrade to the latest version of the app, even when there is no protocol changes, to reduce the version skew among the user base and thus try to keep the number of support requests down.

Since my original recipe, the Signal source code changed slightly, making the old patch fail to apply cleanly. Below is an updated patch, including the shell wrapper I use to start Signal. The original version required a new user to locate the JavaScript console and call a function from there. I got help from a friend with more JavaScript knowledge than me to modify the code to provide a GUI button instead. This mean that to get started you just need to run the wrapper and click the 'Register without mobile phone' to get going now. I've also modified the timeout code to always set it to 90 days in the future, to avoid having to patch the code regularly.

So, the updated recipe for Debian Jessie:

1. First, install required packages to get the source code and the browser you need. Signal only work with Chrome/Chromium, as far as I know, so you need to install it.

   apt install git tor chromium
   git clone https://github.com/WhisperSystems/Signal-Desktop.git
   

2. Modify the source code using command listed in the the patch block below.
3. Start Signal using the run-signal-app wrapper (for example using `pwd`/run-signal-app).
4. Click on the 'Register without mobile phone', will in a phone number you can receive calls to the next minute, receive the verification code and enter it into the form field and press 'Register'. Note, the phone number you use will be user Signal username, ie the way others can find you on Signal.
5. You can now use Signal to contact others. Note, new contacts do not show up in the contact list until you restart Signal, and there is no way to assign names to Contacts. There is also no way to create or update chat groups. I suspect this is because the web app do not have a associated contact database.

I am still a bit uneasy about using Signal, because of the way its main author moxie0 reject federation and accept dependencies to major corporations like Google (part of the code is fetched from Google) and Amazon (the central coordination point is owned by Amazon). See for example the LibreSignal issue tracker for a thread documenting the authors view on these issues. But the network effect is strong in this case, and several of the people I want to communicate with already use Signal. Perhaps we can all move to Ring once it work on my laptop? It already work on Windows and Android, and is included in Debian and Ubuntu, but not working on Debian Stable.

Anyway, this is the patch I apply to the Signal code to get it working. It switch to the production servers, disable to timeout, make registration easier and add the shell wrapper:

cd Signal-Desktop; cat <<EOF | patch -p1
diff --git a/js/background.js b/js/background.js
index 24b4c1d..579345f 100644
--- a/js/background.js
+++ b/js/background.js
@@ -33,9 +33,9 @@
         });
     });
 
-    var SERVER_URL = 'https://textsecure-service-staging.whispersystems.org';
+    var SERVER_URL = 'https://textsecure-service-ca.whispersystems.org';
     var SERVER_PORTS = [80, 4433, 8443];
-    var ATTACHMENT_SERVER_URL = 'https://whispersystems-textsecure-attachments-staging.s3.amazonaws.com';
+    var ATTACHMENT_SERVER_URL = 'https://whispersystems-textsecure-attachments.s3.amazonaws.com';
     var messageReceiver;
     window.getSocketStatus = function() {
         if (messageReceiver) {
diff --git a/js/expire.js b/js/expire.js
index 639aeae..beb91c3 100644
--- a/js/expire.js
+++ b/js/expire.js
@@ -1,6 +1,6 @@
 ;(function() {
     'use strict';
-    var BUILD_EXPIRATION = 0;
+    var BUILD_EXPIRATION = Date.now() + (90 * 24 * 60 * 60 * 1000);
 
     window.extension = window.extension || {};
 
diff --git a/js/views/install_view.js b/js/views/install_view.js
index 7816f4f..1d6233b 100644
--- a/js/views/install_view.js
+++ b/js/views/install_view.js
@@ -38,7 +38,8 @@
             return {
                 'click .step1': this.selectStep.bind(this, 1),
                 'click .step2': this.selectStep.bind(this, 2),
-                'click .step3': this.selectStep.bind(this, 3)
+                'click .step3': this.selectStep.bind(this, 3),
+                'click .callreg': function() { extension.install('standalone') },
             };
         },
         clearQR: function() {
diff --git a/options.html b/options.html
index dc0f28e..8d709f6 100644
--- a/options.html
+++ b/options.html
@@ -14,7 +14,10 @@
         <div class='nav'>
           <h1>{{ installWelcome }}</h1>
           <p>{{ installTagline }}</p>
-          <div> <a class='button step2'>{{ installGetStartedButton }}</a> </div>
+          <div> <a class='button step2'>{{ installGetStartedButton }}</a>
+	    <br> <a class="button callreg">Register without mobile phone</a>
+
+	  </div>
           <span class='dot step1 selected'></span>
           <span class='dot step2'></span>
           <span class='dot step3'></span>
--- /dev/null   2016-10-07 09:55:13.730181472 +0200
+++ b/run-signal-app   2016-10-10 08:54:09.434172391 +0200
@@ -0,0 +1,12 @@
+#!/bin/sh
+set -e
+cd $(dirname $0)
+mkdir -p userdata
+userdata="`pwd`/userdata"
+if [ -d "$userdata" ] && [ ! -d "$userdata/.git" ] ; then
+    (cd $userdata && git init)
+fi
+(cd $userdata && git add . && git commit -m "Current status." || true)
+exec chromium \
+  --proxy-server="socks://localhost:9050" \
+  --user-data-dir=$userdata --load-and-launch-app=`pwd`
EOF
chmod a+rx run-signal-app

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

The Isenkram system provide a practical and easy way to figure out which packages support the hardware in a given machine. The command line tool isenkram-lookup and the tasksel options provide a convenient way to list and install packages relevant for the current hardware during system installation, both user space packages and firmware packages. The GUI background daemon on the other hand provide a pop-up proposing to install packages when a new dongle is inserted while using the computer. For example, if you plug in a smart card reader, the system will ask if you want to install pcscd if that package isn't already installed, and if you plug in a USB video camera the system will ask if you want to install cheese if cheese is currently missing. This already work just fine.

But Isenkram depend on a database mapping from hardware IDs to package names. When I started no such database existed in Debian, so I made my own data set and included it with the isenkram package and made isenkram fetch the latest version of this database from git using http. This way the isenkram users would get updated package proposals as soon as I learned more about hardware related packages.

The hardware is identified using modalias strings. The modalias design is from the Linux kernel where most hardware descriptors are made available as a strings that can be matched using filename style globbing. It handle USB, PCI, DMI and a lot of other hardware related identifiers.

The downside to the Isenkram specific database is that there is no information about relevant distribution / Debian version, making isenkram propose obsolete packages too. But along came AppStream, a cross distribution mechanism to store and collect metadata about software packages. When I heard about the proposal, I contacted the people involved and suggested to add a hardware matching rule using modalias strings in the specification, to be able to use AppStream for mapping hardware to packages. This idea was accepted and AppStream is now a great way for a package to announce the hardware it support in a distribution neutral way. I wrote a recipe on how to add such meta-information in a blog post last December. If you have a hardware related package in Debian, please announce the relevant hardware IDs using AppStream.

In Debian, almost all packages that can talk to a LEGO Mindestorms RCX or NXT unit, announce this support using AppStream. The effect is that when you insert such LEGO robot controller into your Debian machine, Isenkram will propose to install the packages needed to get it working. The intention is that this should allow the local user to start programming his robot controller right away without having to guess what packages to use or which permissions to fix.

But when I sat down with my son the other day to program our NXT unit using his Debian Stretch computer, I discovered something annoying. The local console user (ie my son) did not get access to the USB device for programming the unit. This used to work, but no longer in Jessie and Stretch. After some investigation and asking around on #debian-devel, I discovered that this was because udev had changed the mechanism used to grant access to local devices. The ConsoleKit mechanism from /lib/udev/rules.d/70-udev-acl.rules no longer applied, because LDAP users no longer was added to the plugdev group during login. Michael Biebl told me that this method was obsolete and the new method used ACLs instead. This was good news, as the plugdev mechanism is a mess when using a remote user directory like LDAP. Using ACLs would make sure a user lost device access when she logged out, even if the user left behind a background process which would retain the plugdev membership with the ConsoleKit setup. Armed with this knowledge I moved on to fix the access problem for the LEGO Mindstorms related packages.

The new system uses a udev tag, 'uaccess'. It can either be applied directly for a device, or is applied in /lib/udev/rules.d/70-uaccess.rules for classes of devices. As the LEGO Mindstorms udev rules did not have a class, I decided to add the tag directly in the udev rules files included in the packages. Here is one example. For the nqc C compiler for the RCX, the /lib/udev/rules.d/60-nqc.rules file now look like this:

SUBSYSTEM=="usb", ACTION=="add", ATTR{idVendor}=="0694", ATTR{idProduct}=="0001", \
    SYMLINK+="rcx-%k", TAG+="uaccess"

The key part is the 'TAG+="uaccess"' at the end. I suspect all packages using plugdev in their /lib/udev/rules.d/ files should be changed to use this tag (either directly or indirectly via 70-uaccess.rules). Perhaps a lintian check should be created to detect this?

I've been unable to find good documentation on the uaccess feature. It is unclear to me if the uaccess tag is an internal implementation detail like the udev-acl tag used by /lib/udev/rules.d/70-udev-acl.rules. If it is, I guess the indirect method is the preferred way. Michael asked for more documentation from the systemd project and I hope it will make this clearer. For now I use the generic classes when they exist and is already handled by 70-uaccess.rules, and add the tag directly if no such class exist.

To learn more about the isenkram system, please check out my blog posts tagged isenkram.

To help out making life for LEGO constructors in Debian easier, please join us on our IRC channel #debian-lego and join the Debian LEGO team in the Alioth project we created yesterday. A mailing list is not yet created, but we are working on it. :)

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

In April we started to work on a Norwegian Bokmål edition of the "open access" book on how to set up and administrate a Debian system. Today I am happy to report that the first draft is now publicly available. You can find it on get the Debian Administrator's Handbook page (under Other languages). The first eight chapters have a first draft translation, and we are working on proofreading the content. If you want to help out, please start contributing using the hosted weblate project page, and get in touch using the translators mailing list. Please also check out the instructions for contributors. A good way to contribute is to proofread the text and update weblate if you find errors.

Our goal is still to make the Norwegian book available on paper as well as electronic form.

This summer, I read a great article "coz: This Is the Profiler You're Looking For" in USENIX ;login: about how to profile multi-threaded programs. It presented a system for profiling software by running experiences in the running program, testing how run time performance is affected by "speeding up" parts of the code to various degrees compared to a normal run. It does this by slowing down parallel threads while the "faster up" code is running and measure how this affect processing time. The processing time is measured using probes inserted into the code, either using progress counters (COZ_PROGRESS) or as latency meters (COZ_BEGIN/COZ_END). It can also measure unmodified code by measuring complete the program runtime and running the program several times instead.

The project and presentation was so inspiring that I would like to get the system into Debian. I created a WNPP request for it and contacted upstream to try to make the system ready for Debian by sending patches. The build process need to be changed a bit to avoid running 'git clone' to get dependencies, and to include the JavaScript web page used to visualize the collected profiling information included in the source package. But I expect that should work out fairly soon.

The way the system work is fairly simple. To run an coz experiment on a binary with debug symbols available, start the program like this:

coz run --- program-to-run

This will create a text file profile.coz with the instrumentation information. To show what part of the code affect the performance most, use a web browser and either point it to http://plasma-umass.github.io/coz/ or use the copy from git (in the gh-pages branch). Check out this web site to have a look at several example profiling runs and get an idea what the end result from the profile runs look like. To make the profiling more useful you include <coz.h> and insert the COZ_PROGRESS or COZ_BEGIN and COZ_END at appropriate places in the code, rebuild and run the profiler. This allow coz to do more targeted experiments.

A video published by ACM presenting the Coz profiler is available from Youtube. There is also a paper from the 25th Symposium on Operating Systems Principles available titled Coz: finding code that counts with causal profiling.

The source code for Coz is available from github. It will only build with clang because it uses a C++ feature missing in GCC, but I've submitted a patch to solve it and hope it will be included in the upstream source soon.

Please get in touch if you, like me, would like to see this piece of software in Debian. I would very much like some help with the packaging effort, as I lack the in depth knowledge on how to package C++ libraries.

Yesterday, I tried to unlock a HTC Desire HD phone, and it proved to be a slight challenge. Here is the recipe if I ever need to do it again. It all started by me wanting to try the recipe to set up an hardened Android installation from the Tor project blog on a device I had access to. It is a old mobile phone with a broken microphone The initial idea had been to just install CyanogenMod on it, but did not quite find time to start on it until a few days ago.

The unlock process is supposed to be simple: (1) Boot into the boot loader (press volume down and power at the same time), (2) select 'fastboot' before (3) connecting the device via USB to a Linux machine, (4) request the device identifier token by running 'fastboot oem get_identifier_token', (5) request the device unlocking key using the HTC developer web site and unlock the phone using the key file emailed to you.

Unfortunately, this only work fi you have hboot version 2.00.0029 or newer, and the device I was working on had 2.00.0027. This apparently can be easily fixed by downloading a Windows program and running it on your Windows machine, if you accept the terms Microsoft require you to accept to use Windows - which I do not. So I had to come up with a different approach. I got a lot of help from AndyCap on #nuug, and would not have been able to get this working without him.

First I needed to extract the hboot firmware from the windows binary for HTC Desire HD downloaded as 'the RUU' from HTC. For this there is is a github project named unruu using libunshield. The unshield tool did not recognise the file format, but unruu worked and extracted rom.zip, containing the new hboot firmware and a text file describing which devices it would work for.

Next, I needed to get the new firmware into the device. For this I followed some instructions available from HTC1Guru.com, and ran these commands as root on a Linux machine with Debian testing:

adb reboot-bootloader
fastboot oem rebootRUU
fastboot flash zip rom.zip
fastboot flash zip rom.zip
fastboot reboot

The flash command apparently need to be done twice to take effect, as the first is just preparations and the second one do the flashing. The adb command is just to get to the boot loader menu, so turning the device on while holding volume down and the power button should work too.

With the new hboot version in place I could start following the instructions on the HTC developer web site. I got the device token like this:

fastboot oem get_identifier_token 2>&1 | sed 's/(bootloader) //'

And once I got the unlock code via email, I could use it like this:

fastboot flash unlocktoken Unlock_code.bin

And with that final step in place, the phone was unlocked and I could start stuffing the software of my own choosing into the device. So far I only inserted a replacement recovery image to wipe the phone before I start. We will see what happen next. Perhaps I should install Debian on it. :)

For a while now, I have wanted to test the Signal app, as it is said to provide end to end encrypted communication and several of my friends and family are already using it. As I by choice do not own a mobile phone, this proved to be harder than expected. And I wanted to have the source of the client and know that it was the code used on my machine. But yesterday I managed to get it working. I used the Github source, compared it to the source in the Signal Chrome app available from the Chrome web store, applied patches to use the production Signal servers, started the app and asked for the hidden "register without a smart phone" form. Here is the recipe how I did it.

First, I fetched the Signal desktop source from Github, using

git clone https://github.com/WhisperSystems/Signal-Desktop.git

Next, I patched the source to use the production servers, to be able to talk to other Signal users:

cat <<EOF | patch -p0
diff -ur ./js/background.js userdata/Default/Extensions/bikioccmkafdpakkkcpdbppfkghcmihk/0.15.0_0/js/background.js
--- ./js/background.js  2016-06-29 13:43:15.630344628 +0200
+++ userdata/Default/Extensions/bikioccmkafdpakkkcpdbppfkghcmihk/0.15.0_0/js/background.js    2016-06-29 14:06:29.530300934 +0200
@@ -47,8 +47,8 @@
         });
     });
 
-    var SERVER_URL = 'https://textsecure-service-staging.whispersystems.org';
-    var ATTACHMENT_SERVER_URL = 'https://whispersystems-textsecure-attachments-staging.s3.amazonaws.com';
+    var SERVER_URL = 'https://textsecure-service-ca.whispersystems.org:4433';
+    var ATTACHMENT_SERVER_URL = 'https://whispersystems-textsecure-attachments.s3.amazonaws.com';
     var messageReceiver;
     window.getSocketStatus = function() {
         if (messageReceiver) {
diff -ur ./js/expire.js userdata/Default/Extensions/bikioccmkafdpakkkcpdbppfkghcmihk/0.15.0_0/js/expire.js
--- ./js/expire.js      2016-06-29 13:43:15.630344628 +0200
+++ userdata/Default/Extensions/bikioccmkafdpakkkcpdbppfkghcmihk/0.15.0_0/js/expire.js2016-06-29 14:06:29.530300934 +0200
@@ -1,6 +1,6 @@
 ;(function() {
     'use strict';
-    var BUILD_EXPIRATION = 0;
+    var BUILD_EXPIRATION = 1474492690000;
 
     window.extension = window.extension || {};
 
EOF

The first part is changing the servers, and the second is updating an expiration timestamp. This timestamp need to be updated regularly. It is set 90 days in the future by the build process (Gruntfile.js). The value is seconds since 1970 times 1000, as far as I can tell.

Based on a tip and good help from the #nuug IRC channel, I wrote a script to launch Signal in Chromium.

#!/bin/sh
cd $(dirname $0)
mkdir -p userdata
exec chromium \
  --proxy-server="socks://localhost:9050" \
  --user-data-dir=`pwd`/userdata --load-and-launch-app=`pwd`

The script start the app and configure Chromium to use the Tor SOCKS5 proxy to make sure those controlling the Signal servers (today Amazon and Whisper Systems) as well as those listening on the lines will have a harder time location my laptop based on the Signal connections if they use source IP address.

When the script starts, one need to follow the instructions under "Standalone Registration" in the CONTRIBUTING.md file in the git repository. I right clicked on the Signal window to get up the Chromium debugging tool, visited the 'Console' tab and wrote 'extension.install("standalone")' on the console prompt to get the registration form. Then I entered by land line phone number and pressed 'Call'. 5 seconds later the phone rang and a robot voice repeated the verification code three times. After entering the number into the verification code field in the form, I could start using Signal from my laptop.

As far as I can tell, The Signal app will leak who is talking to whom and thus who know who to those controlling the central server, but such leakage is hard to avoid with a centrally controlled server setup. It is something to keep in mind when using Signal - the content of your chats are harder to intercept, but the meta data exposing your contact network is available to people you do not know. So better than many options, but not great. And sadly the usage is connected to my land line, thus allowing those controlling the server to associate it to my home and person. I would prefer it if only those I knew could tell who I was on Signal. There are options avoiding such information leakage, but most of my friends are not using them, so I am stuck with Signal for now.

Update 2017-01-10: There is an updated blog post on this topic in Experience and updated recipe for using the Signal app without a mobile phone.

When I set out a few weeks ago to figure out which multimedia player in Debian claimed to support most file formats / MIME types, I was a bit surprised how varied the sets of MIME types the various players claimed support for. The range was from 55 to 130 MIME types. I suspect most media formats are supported by all players, but this is not really reflected in the MimeTypes values in their desktop files. There are probably also some bogus MIME types listed, but it is hard to identify which one this is.

Anyway, in the mean time I got in touch with upstream for some of the players suggesting to add more MIME types to their desktop files, and decided to spend some time myself improving the situation for my favorite media player VLC. The fixes for VLC entered Debian unstable yesterday. The complete list of MIME types can be seen on the Multimedia player MIME type support status Debian wiki page.

The new "best" multimedia player in Debian? It is VLC, followed by totem, parole, kplayer, gnome-mpv, mpv, smplayer, mplayer-gui and kmplayer. I am sure some of the other players desktop files support several of the formats currently listed as working only with vlc, toten and parole.

A sad observation is that only 14 MIME types are listed as supported by all the tested multimedia players in Debian in their desktop files: audio/mpeg, audio/vnd.rn-realaudio, audio/x-mpegurl, audio/x-ms-wma, audio/x-scpls, audio/x-wav, video/mp4, video/mpeg, video/quicktime, video/vnd.rn-realvideo, video/x-matroska, video/x-ms-asf, video/x-ms-wmv and video/x-msvideo. Personally I find it sad that video/ogg and video/webm is not supported by all the media players in Debian. As far as I can tell, all of them can handle both formats.

Many years ago, when koffice was fresh and with few users, I decided to test its presentation tool when making the slides for a talk I was giving for NUUG on Japhar, a free Java virtual machine. I wrote the first draft of the slides, saved the result and went to bed the day before I would give the talk. The next day I took a plane to the location where the meeting should take place, and on the plane I started up koffice again to polish the talk a bit, only to discover that kpresenter refused to load its own data file. I cursed a bit and started making the slides again from memory, to have something to present when I arrived. I tested that the saved files could be loaded, and the day seemed to be rescued. I continued to polish the slides until I suddenly discovered that the saved file could no longer be loaded into kpresenter. In the end I had to rewrite the slides three times, condensing the content until the talk became shorter and shorter. After the talk I was able to pinpoint the problem – kpresenter wrote inline images in a way itself could not understand. Eventually that bug was fixed and kpresenter ended up being a great program to make slides. The point I'm trying to make is that we expect a program to be able to load its own data files, and it is embarrassing to its developers if it can't.

Did you ever experience a program failing to load its own data files from the desktop file browser? It is not a uncommon problem. A while back I discovered that the screencast recorder gtk-recordmydesktop would save an Ogg Theora video file the KDE file browser would refuse to open. No video player claimed to understand such file. I tracked down the cause being file --mime-type returning the application/ogg MIME type, which no video player I had installed listed as a MIME type they would understand. I asked for file to change its behavour and use the MIME type video/ogg instead. I also asked several video players to add video/ogg to their desktop files, to give the file browser an idea what to do about Ogg Theora files. After a while, the desktop file browsers in Debian started to handle the output from gtk-recordmydesktop properly.

But history repeats itself. A few days ago I tested the music system Rosegarden again, and I discovered that the KDE and xfce file browsers did not know what to do with the Rosegarden project files (*.rg). I've reported the rosegarden problem to BTS and a fix is commited to git and will be included in the next upload. To increase the chance of me remembering how to fix the problem next time some program fail to load its files from the file browser, here are some notes on how to fix it.

The file browsers in Debian in general operates on MIME types. There are two sources for the MIME type of a given file. The output from file --mime-type mentioned above, and the content of the shared MIME type registry (under /usr/share/mime/). The file MIME type is mapped to programs supporting the MIME type, and this information is collected from the desktop files available in /usr/share/applications/. If there is one desktop file claiming support for the MIME type of the file, it is activated when asking to open a given file. If there are more, one can normally select which one to use by right-clicking on the file and selecting the wanted one using 'Open with' or similar. In general this work well. But it depend on each program picking a good MIME type (preferably a MIME type registered with IANA), file and/or the shared MIME registry recognizing the file and the desktop file to list the MIME type in its list of supported MIME types.

The /usr/share/mime/packages/rosegarden.xml entry for the Shared MIME database look like this:

<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
  <mime-type type="audio/x-rosegarden">
    <sub-class-of type="application/x-gzip"/>
    <comment>Rosegarden project file</comment>
    <glob pattern="*.rg"/>
  </mime-type>
</mime-info>

This states that audio/x-rosegarden is a kind of application/x-gzip (it is a gzipped XML file). Note, it is much better to use an official MIME type registered with IANA than it is to make up ones own unofficial ones like the x-rosegarden type used by rosegarden.

The desktop file of the rosegarden program failed to list audio/x-rosegarden in its list of supported MIME types, causing the file browsers to have no idea what to do with *.rg files:

% grep Mime /usr/share/applications/rosegarden.desktop
MimeType=audio/x-rosegarden-composition;audio/x-rosegarden-device;audio/x-rosegarden-project;audio/x-rosegarden-template;audio/midi;
X-KDE-NativeMimeType=audio/x-rosegarden-composition
%

The fix was to add "audio/x-rosegarden;" at the end of the MimeType= line.

If you run into a file which fail to open the correct program when selected from the file browser, please check out the output from file --mime-type for the file, ensure the file ending and MIME type is registered somewhere under /usr/share/mime/ and check that some desktop file under /usr/share/applications/ is claiming support for this MIME type. If not, please report a bug to have it fixed. :)

The isenkram system is a user-focused solution in Debian for handling hardware related packages. The idea is to have a database of mappings between hardware and packages, and pop up a dialog suggesting for the user to install the packages to use a given hardware dongle. Some use cases are when you insert a Yubikey, it proposes to install the software needed to control it; when you insert a braille reader list it proposes to install the packages needed to send text to the reader; and when you insert a ColorHug screen calibrator it suggests to install the driver for it. The system work well, and even have a few command line tools to install firmware packages and packages for the hardware already in the machine (as opposed to hotpluggable hardware).

The system was initially written using aptdaemon, because I found good documentation and example code on how to use it. But aptdaemon is going away and is generally being replaced by PackageKit, so Isenkram needed a rewrite. And today, thanks to the great patch from my college Sunil Mohan Adapa in the FreedomBox project, the rewrite finally took place. I've just uploaded a new version of Isenkram into Debian Unstable with the patch included, and the default for the background daemon is now to use PackageKit. To check it out, install the isenkram package and insert some hardware dongle and see if it is recognised.

If you want to know what kind of packages isenkram would propose for the machine it is running on, you can check out the isenkram-lookup program. This is what it look like on a Thinkpad X230:

% isenkram-lookup 
bluez
cheese
fprintd
fprintd-demo
gkrellm-thinkbat
hdapsd
libpam-fprintd
pidgin-blinklight
thinkfan
tleds
tp-smapi-dkms
tp-smapi-source
tpb
%p

The hardware mappings come from several places. The preferred way is for packages to announce their hardware support using the cross distribution appstream system. See previous blog posts about isenkram to learn how to do that.

Yesterday I updated the battery-stats package in Debian with a few patches sent to me by skilled and enterprising users. There were some nice user and visible changes. First of all, both desktop menu entries now work. A design flaw in one of the script made the history graph fail to show up (its PNG was dumped in ~/.xsession-errors) if no controlling TTY was available. The script worked when called from the command line, but not when called from the desktop menu. I changed this to look for a DISPLAY variable or a TTY before deciding where to draw the graph, and now the graph window pop up as expected.

The next new feature is a discharge rate estimator in one of the graphs (the one showing the last few hours). New is also the user of colours showing charging in blue and discharge in red. The percentages of this graph is relative to last full charge, not battery design capacity.

The other graph show the entire history of the collected battery statistics, comparing it to the design capacity of the battery to visualise how the battery life time get shorter over time. The red line in this graph is what the previous graph considers 100 percent:

In this graph you can see that I only charge the battery to 80 percent of last full capacity, and how the capacity of the battery is shrinking. :(

The last new feature is in the collector, which now will handle more hardware models. On some hardware, Linux power supply information is stored in /sys/class/power_supply/ACAD/, while the collector previously only looked in /sys/class/power_supply/AC/. Now both are checked to figure if there is power connected to the machine.

If you are interested in how your laptop battery is doing, please check out the battery-stats in Debian unstable, or rebuild it on Jessie to get it working on Debian stable. :) The upstream source is available from github. Patches are very welcome.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Today, after many years of hard work from many people, ZFS for Linux finally entered Debian. The package status can be seen on the package tracker for zfs-linux. and the team status page. If you want to help out, please join us. The source code is available via git on Alioth. It would also be great if you could help out with the dkms package, as it is an important piece of the puzzle to get ZFS working.

Where I set out to figure out which multimedia player in Debian claim support for most file formats.

A few years ago, I had a look at the media support for Browser plugins in Debian, to get an idea which plugins to include in Debian Edu. I created a script to extract the set of supported MIME types for each plugin, and used this to find out which multimedia browser plugin supported most file formats / media types. The result can still be seen on the Debian wiki, even though it have not been updated for a while. But browser plugins are less relevant these days, so I thought it was time to look at standalone players.

A few days ago I was tired of VLC not being listed as a viable player when I wanted to play videos from the Norwegian National Broadcasting Company, and decided to investigate why. The cause is a missing MIME type in the VLC desktop file. In the process I wrote a script to compare the set of MIME types announced in the desktop file and the browser plugin, only to discover that there is quite a large difference between the two for VLC. This discovery made me dig up the script I used to compare browser plugins, and adjust it to compare desktop files instead, to try to figure out which multimedia player in Debian support most file formats.

The result can be seen on the Debian Wiki, as a table listing all MIME types supported by one of the packages included in the table, with the package supporting most MIME types being listed first in the table.

The best multimedia player in Debian? It is totem, followed by parole, kplayer, mpv, vlc, smplayer mplayer-gui gnome-mpv and kmplayer. Time for the other players to update their announced MIME support?

A friend of mine made me aware of The Pyra, a handheld computer which will be delivered with Debian preinstalled. I would love to get one of those for my birthday. :)

The machine is a complete ARM-based PC with micro HDMI, SATA, USB plugs and many others connectors, and include a full keyboard and a 5" LCD touch screen. The 6000mAh battery is claimed to provide a whole day of battery life time, but I have not seen any independent tests confirming this. The vendor is still collecting preorders, and the last I heard last night was that 22 more orders were needed before production started.

As far as I know, this is the first handheld preinstalled with Debian. Please let me know if you know of any others. Is it the first computer being sold with Debian preinstalled?

During this weekends bug squashing party and developer gathering, we decided to do our part to make sure there are good books about Debian available in Norwegian Bokmål, and got in touch with the people behind the Debian Administrator's Handbook project to get started. If you want to help out, please start contributing using the hosted weblate project page, and get in touch using the translators mailing list. Please also check out the instructions for contributors.

The book is already available on paper in English, French and Japanese, and our goal is to get it available on paper in Norwegian Bokmål too. In addition to the paper edition, there are also EPUB and Mobi versions available. And there are incomplete translations available for many more languages.

Just for fun I had a look at the popcon number of ZFS related packages in Debian, and was quite surprised with what I found. I use ZFS myself at home, but did not really expect many others to do so. But I might be wrong.

According to the popcon results for spl-linux, there are 1019 Debian installations, or 0.53% of the population, with the package installed. As far as I know the only use of the spl-linux package is as a support library for ZFS on Linux, so I use it here as proxy for measuring the number of ZFS installation on Linux in Debian. In the kFreeBSD variant of Debian the ZFS feature is already available, and there the popcon results for zfsutils show 1625 Debian installations or 0.84% of the population. So I guess I am not alone in using ZFS on Debian.

But even though the Debian project leader Lucas Nussbaum announced in April 2015 that the legal obstacles blocking ZFS on Debian were cleared, the package is still not in Debian. The package is again in the NEW queue. Several uploads have been rejected so far because the debian/copyright file was incomplete or wrong, but there is no reason to give up. The current status can be seen on the team status page, and the source code is available on Alioth.

As I want ZFS to be included in next version of Debian to make sure my home server can function in the future using only official Debian packages, and the current blocker is to get the debian/copyright file accepted by the FTP masters in Debian, I decided a while back to try to help out the team. This was the background for my blog post about creating, updating and checking debian/copyright semi-automatically, and I used the techniques I explored there to try to find any errors in the copyright file. It is not very easy to check every one of the around 2000 files in the source package, but I hope we this time got it right. If you want to help out, check out the git source and try to find missing entries in the debian/copyright file.

Since this morning, the battery-stats package in Debian include an extended collector that will collect the complete battery history for later processing and graphing. The original collector store the battery level as percentage of last full level, while the new collector also record battery vendor, model, serial number, design full level, last full level and current battery level. This make it possible to predict the lifetime of the battery as well as visualise the energy flow when the battery is charging or discharging.

The new tools are available in /usr/share/battery-stats/ in the version 0.5.1 package in unstable. Get the new battery level graph and lifetime prediction by running:

/usr/share/battery-stats/battery-stats-graph /var/log/battery-stats.csv

Or select the 'Battery Level Graph' from your application menu.

The flow in/out of the battery can be seen by running (no menu entry yet):

/usr/share/battery-stats/battery-stats-graph-flow

I'm not quite happy with the way the data is visualised, at least when there are few data points. The graphs look a bit better with a few years of data.

A while back one important feature I use in the battery stats collector broke in Debian. The scripts in /usr/lib/pm-utils/power.d/ were no longer executed. I suspect it happened when Jessie started using systemd, but I do not know. The issue is reported as bug #818649 against pm-utils. I managed to work around it by adding an udev rule to call the collector script every time the power connector is connected and disconnected. With this fix in place it was finally time to make a new release of the package, and get it into Debian.

If you are interested in how your laptop battery is doing, please check out the battery-stats in Debian unstable, or rebuild it on Jessie to get it working on Debian stable. :) The upstream source is available from github. As always, patches are very welcome.

Back in September, I blogged about the system I wrote to collect statistics about my laptop battery, and how it showed the decay and death of this battery (now replaced). I created a simple deb package to handle the collection and graphing, but did not want to upload it to Debian as there were already a battery-stats package in Debian that should do the same thing, and I did not see a point of uploading a competing package when battery-stats could be fixed instead. I reported a few bugs about its non-function, and hoped someone would step in and fix it. But no-one did.

I got tired of waiting a few days ago, and took matters in my own hands. The end result is that I am now the new upstream developer of battery stats (available from github) and part of the team maintaining battery-stats in Debian, and the package in Debian unstable is finally able to collect battery status using the /sys/class/power_supply/ information provided by the Linux kernel. If you install the battery-stats package from unstable now, you will be able to get a graph of the current battery fill level, to get some idea about the status of the battery. The source package build and work just fine in Debian testing and stable (and probably oldstable too, but I have not tested). The default graph you get for that system look like this:

My plans for the future is to merge my old scripts into the battery-stats package, as my old scripts collected a lot more details about the battery. The scripts are merged into the upstream battery-stats git repository already, but I am not convinced they work yet, as I changed a lot of paths along the way. Will have to test a bit more before I make a new release.

I will also consider changing the file format slightly, as I suspect the way I combine several values into one field might make it impossible to know the type of the value when using it for processing and graphing.

If you would like I would like to keep an close eye on your laptop battery, check out the battery-stats package in Debian and on github. I would love some help to improve the system further.

Making packages for Debian requires quite a lot of attention to details. And one of the details is the content of the debian/copyright file, which should list all relevant licenses used by the code in the package in question, preferably in machine readable DEP5 format.

For large packages with lots of contributors it is hard to write and update this file manually, and if you get some detail wrong, the package is normally rejected by the ftpmasters. So getting it right the first time around get the package into Debian faster, and save both you and the ftpmasters some work.. Today, while trying to figure out what was wrong with the zfsonlinux copyright file, I decided to spend some time on figuring out the options for doing this job automatically, or at least semi-automatically.

Lucikly, there are at least two tools available for generating the file based on the code in the source package, debmake and cme. I'm not sure which one of them came first, but both seem to be able to create a sensible draft file. As far as I can tell, none of them can be trusted to get the result just right, so the content need to be polished a bit before the file is OK to upload. I found the debmake option in a blog posts from 2014.

To generate using debmake, use the -cc option:

debmake -cc > debian/copyright

Note there are some problems with python and non-ASCII names, so this might not be the best option.

The cme option is based on a config parsing library, and I found this approach in a blog post from 2015. To generate using cme, use the 'update dpkg-copyright' option:

cme update dpkg-copyright

This will create or update debian/copyright. The cme tool seem to handle UTF-8 names better than debmake.

When the copyright file is created, I would also like some help to check if the file is correct. For this I found two good options, debmake -k and license-reconcile. The former seem to focus on license types and file matching, and is able to detect ineffective blocks in the copyright file. The latter reports missing copyright holders and years, but was confused by inconsistent license names (like CDDL vs. CDDL-1.0). I suspect it is good to use both and fix all issues reported by them before uploading. But I do not know if the tools and the ftpmasters agree on what is important to fix in a copyright file, so the package might still be rejected.

The devscripts tool licensecheck deserve mentioning. It will read through the source and try to find all copyright statements. It is not comparing the result to the content of debian/copyright, but can be useful when verifying the content of the copyright file.

Are you aware of better tools in Debian to create and update debian/copyright file. Please let me know, or blog about it on planet.debian.org.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Update 2016-02-20: I got a tip from Mike Gabriel on how to use licensecheck and cdbs to create a draft copyright file

licensecheck --copyright -r `find * -type f` | \
  /usr/lib/cdbs/licensecheck2dep5 > debian/copyright.auto

He mentioned that he normally check the generated file into the version control system to make it easier to discover license and copyright changes in the upstream source. I will try to do the same with my packages in the future.

Update 2016-02-21: The cme author recommended against using -quiet for new users, so I removed it from the proposed command line.

The appstream system is taking shape in Debian, and one provided feature is a very convenient way to tell you which package to install to make a given firmware file available when the kernel is looking for it. This can be done using apt-file too, but that is for someone else to blog about. :)

Here is a small recipe to find the package with a given firmware file, in this example I am looking for ctfw-3.2.3.0.bin, randomly picked from the set of firmware announced using appstream in Debian unstable. In general you would be looking for the firmware requested by the kernel during kernel module loading. To find the package providing the example file, do like this:

% apt install appstream
[...]
% apt update
[...]
% appstreamcli what-provides firmware:runtime ctfw-3.2.3.0.bin | \
  awk '/Package:/ {print $2}'
firmware-qlogic
%

See the appstream wiki page to learn how to embed the package metadata in a way appstream can use.

This same approach can be used to find any package supporting a given MIME type. This is very useful when you get a file you do not know how to handle. First find the mime type using file --mime-type, and next look up the package providing support for it. Lets say you got an SVG file. Its MIME type is image/svg+xml, and you can find all packages handling this type like this:

% apt install appstream
[...]
% apt update
[...]
% appstreamcli what-provides mimetype image/svg+xml | \
  awk '/Package:/ {print $2}'
bkchem
phototonic
inkscape
shutter
tetzle
geeqie
xia
pinta
gthumb
karbon
comix
mirage
viewnior
postr
ristretto
kolourpaint4
eog
eom
gimagereader
midori
%

I believe the MIME types are fetched from the desktop file for packages providing appstream metadata.

Most people seem not to realise that every time they walk around with the computerised radio beacon known as a mobile phone their position is tracked by the phone company and often stored for a long time (like every time a SMS is received or sent). And if their computerised radio beacon is capable of running programs (often called mobile apps) downloaded from the Internet, these programs are often also capable of tracking their location (if the app requested access during installation). And when these programs send out information to central collection points, the location is often included, unless extra care is taken to not send the location. The provided information is used by several entities, for good and bad (what is good and bad, depend on your point of view). What is certain, is that the private sphere and the right to free movement is challenged and perhaps even eradicated for those announcing their location this way, when they share their whereabouts with private and public entities.

The phone company logs provide a register of locations to check out when one want to figure out what the tracked person was doing. It is unavailable for most of us, but provided to selected government officials, company staff, those illegally buying information from unfaithful servants and crackers stealing the information. But the public information can be collected and analysed, and a free software tool to do so is called Creepy or Cree.py. I discovered it when I read an article about Creepy in the Norwegian newspaper Aftenposten i November 2014, and decided to check if it was available in Debian. The python program was in Debian, but the version in Debian was completely broken and practically unmaintained. I uploaded a new version which did not work quite right, but did not have time to fix it then. This Christmas I decided to finally try to get Creepy operational in Debian. Now a fixed version is available in Debian unstable and testing, and almost all Debian specific patches are now included upstream.

The Creepy program visualises geolocation information fetched from Twitter, Instagram, Flickr and Google+, and allow one to get a complete picture of every social media message posted recently in a given area, or track the movement of a given individual across all these services. Earlier it was possible to use the search API of at least some of these services without identifying oneself, but these days it is impossible. This mean that to use Creepy, you need to configure it to log in as yourself on these services, and provide information to them about your search interests. This should be taken into account when using Creepy, as it will also share information about yourself with the services.

The picture above show the twitter messages sent from (or at least geotagged with a position from) the city centre of Oslo, the capital of Norway. One useful way to use Creepy is to first look at information tagged with an area of interest, and next look at all the information provided by one or more individuals who was in the area. I tested it by checking out which celebrity provide their location in twitter messages by checkout out who sent twitter messages near a Norwegian TV station, and next could track their position over time, making it possible to locate their home and work place, among other things. A similar technique have been used to locate Russian soldiers in Ukraine, and it is both a powerful tool to discover lying governments, and a useful tool to help people understand the value of the private information they provide to the public.

The package is not trivial to backport to Debian Stable/Jessie, as it depend on several python modules currently missing in Jessie (at least python-instagram, python-flickrapi and python-requests-toolbelt).

(I have uploaded the image to screenshots.debian.net and licensed it under the same terms as the Creepy program in Debian.)

During his DebConf15 keynote, Jacob Appelbaum observed that those listening on the Internet lines would have good reason to believe a computer have a given security hole if it download a security fix from a Debian mirror. This is a good reason to always use encrypted connections to the Debian mirror, to make sure those listening do not know which IP address to attack. In August, Richard Hartmann observed that encryption was not enough, when it was possible to interfere download size to security patches or the fact that download took place shortly after a security fix was released, and proposed to always use Tor to download packages from the Debian mirror. He was not the first to propose this, as the apt-transport-tor package by Tim Retout already existed to make it easy to convince apt to use Tor, but I was not aware of that package when I read the blog post from Richard.

Richard discussed the idea with Peter Palfrader, one of the Debian sysadmins, and he set up a Tor hidden service on one of the central Debian mirrors using the address vwakviie2ienjx6t.onion, thus making it possible to download packages directly between two tor nodes, making sure the network traffic always were encrypted.

Here is a short recipe for enabling this on your machine, by installing apt-transport-tor and replacing http and https urls with tor+http and tor+https, and using the hidden service instead of the official Debian mirror site. I recommend installing etckeeper before you start to have a history of the changes done in /etc/.

apt install apt-transport-tor
sed -i 's% http://ftp.debian.org/% tor+http://vwakviie2ienjx6t.onion/%' /etc/apt/sources.list
sed -i 's% http% tor+http%' /etc/apt/sources.list

If you have more sources listed in /etc/apt/sources.list.d/, run the sed commands for these too. The sed command is assuming your are using the ftp.debian.org Debian mirror. Adjust the command (or just edit the file manually) to match your mirror.

This work in Debian Jessie and later. Note that tools like apt-file only recently started using the apt transport system, and do not work with these tor+http URLs. For apt-file you need the version currently in experimental, which need a recent apt version currently only in unstable. So if you need a working apt-file, this is not for you.

Another advantage from this change is that your machine will start using Tor regularly and at fairly random intervals (every time you update the package lists or upgrade or install a new package), thus masking other Tor traffic done from the same machine. Using Tor will become normal for the machine in question.

On Freedombox, APT is set up by default to use apt-transport-tor when Tor is enabled. It would be great if it was the default on any Debian system.

When I was a kid, we used to collect "car numbers", as we used to call the car license plate numbers in those days. I would write the numbers down in my little book and compare notes with the other kids to see how many region codes we had seen and if we had seen some exotic or special region codes and numbers. It was a fun game to pass time, as we kids have plenty of it.

A few days I came across the OpenALPR project, a free software project to automatically discover and report license plates in images and video streams, and provide the "car numbers" in a machine readable format. I've been looking for such system for a while now, because I believe it is a bad idea that the automatic number plate recognition tool only is available in the hands of the powerful, and want it to be available also for the powerless to even the score when it comes to surveillance and sousveillance. I discovered the developer wanted to get the tool into Debian, and as I too wanted it to be in Debian, I volunteered to help him get it into shape to get the package uploaded into the Debian archive.

Today we finally managed to get the package into shape and uploaded it into Debian, where it currently waits in the NEW queue for review by the Debian ftpmasters.

I guess you are wondering why on earth such tool would be useful for the common folks, ie those not running a large government surveillance system? Well, I plan to put it in a computer on my bike and in my car, tracking the cars nearby and allowing me to be notified when number plates on my watch list are discovered. Another use case was suggested by a friend of mine, who wanted to set it up at his home to open the car port automatically when it discovered the plate on his car. When I mentioned it perhaps was a bit foolhardy to allow anyone capable of placing his license plate number of a piece of cardboard to open his car port, men replied that it was always unlocked anyway. I guess for such use case it make sense. I am sure there are other use cases too, for those with imagination and a vision.

If you want to build your own version of the Debian package, check out the upstream git source and symlink ./distros/debian to ./debian/ before running "debuild" to build the source. Or wait a bit until the package show up in unstable.

Around three years ago, I created the isenkram system to get a more practical solution in Debian for handing hardware related packages. A GUI system in the isenkram package will present a pop-up dialog when some hardware dongle supported by relevant packages in Debian is inserted into the machine. The same lookup mechanism to detect packages is available as command line tools in the isenkram-cli package. In addition to mapping hardware, it will also map kernel firmware files to packages and make it easy to install needed firmware packages automatically. The key for this system to work is a good way to map hardware to packages, in other words, allow packages to announce what hardware they will work with.

I started by providing data files in the isenkram source, and adding code to download the latest version of these data files at run time, to ensure every user had the most up to date mapping available. I also added support for storing the mapping in the Packages file in the apt repositories, but did not push this approach because while I was trying to figure out how to best store hardware/package mappings, the appstream system was announced. I got in touch and suggested to add the hardware mapping into that data set to be able to use appstream as a data source, and this was accepted at least for the Debian version of appstream.

A few days ago using appstream in Debian for this became possible, and today I uploaded a new version 0.20 of isenkram adding support for appstream as a data source for mapping hardware to packages. The only package so far using appstream to announce its hardware support is my pymissile package. I got help from Matthias Klumpp with figuring out how do add the required metadata in pymissile. I added a file debian/pymissile.metainfo.xml with this content:

<?xml version="1.0" encoding="UTF-8"?>
<component>
  <id>pymissile</id>
  <metadata_license>MIT</metadata_license>
  <name>pymissile</name>
  <summary>Control original Striker USB Missile Launcher</summary>
  <description>
    <p>
      Pymissile provides a curses interface to control an original
      Marks and Spencer / Striker USB Missile Launcher, as well as a
      motion control script to allow a webcamera to control the
      launcher.
    </p>
  </description>
  <provides>
    <modalias>usb:v1130p0202d*</modalias>
  </provides>
</component>

The key for isenkram is the component/provides/modalias value, which is a glob style match rule for hardware specific strings (modalias strings) provided by the Linux kernel. In this case, it will map to all USB devices with vendor code 1130 and product code 0202.

Note, it is important that the license of all the metadata files are compatible to have permissions to aggregate them into archive wide appstream files. Matthias suggested to use MIT or BSD licenses for these files. A challenge is figuring out a good id for the data, as it is supposed to be globally unique and shared across distributions (in other words, best to coordinate with upstream what to use). But it can be changed later or, so we went with the package name as upstream for this project is dormant.

To get the metadata file installed in the correct location for the mirror update scripts to pick it up and include its content the appstream data source, the file must be installed in the binary package under /usr/share/appdata/. I did this by adding the following line to debian/pymissile.install:

debian/pymissile.metainfo.xml usr/share/appdata

With that in place, the command line tool isenkram-lookup will list all packages useful on the current computer automatically, and the GUI pop-up handler will propose to install the package not already installed if a hardware dongle is inserted into the machine in question.

Details of the modalias field in appstream is available from the DEP-11 proposal.

To locate the modalias values of all hardware present in a machine, try running this command on the command line:

cat $(find /sys/devices/|grep modalias)

To learn more about the isenkram system, please check out my blog posts tagged isenkram.

A blog post from my fellow Debian developer Paul Wise titled "The GPL is not magic pixie dust" explain the importance of making sure the GPL is enforced. I quote the blog post from Paul in full here with his permission:

The GPL is not magic pixie dust. It does not work by itself.
The first step is to choose a copyleft license for your code.
The next step is, when someone fails to follow that copyleft license, it must be enforced
and its a simple fact of our modern society that such type of work
is incredibly expensive to do and incredibly difficult to do.

-- Bradley Kuhn, in FaiF episode 0x57

As the Debian Website used to imply, public domain and permissively licensed software can lead to the production of more proprietary software as people discover useful software, extend it and or incorporate it into their hardware or software products. Copyleft licenses such as the GNU GPL were created to close off this avenue to the production of proprietary software but such licenses are not enough. With the ongoing adoption of Free Software by individuals and groups, inevitably the community's expectations of license compliance are violated, usually out of ignorance of the way Free Software works, but not always. As Karen and Bradley explained in FaiF episode 0x57, copyleft is nothing if no-one is willing and able to stand up in court to protect it. The reality of today's world is that legal representation is expensive, difficult and time consuming. With gpl-violations.org in hiatus until some time in 2016, the Software Freedom Conservancy (a tax-exempt charity) is the major defender of the Linux project, Debian and other groups against GPL violations. In March the SFC supported a lawsuit by Christoph Hellwig against VMware for refusing to comply with the GPL in relation to their use of parts of the Linux kernel. Since then two of their sponsors pulled corporate funding and conferences blocked or cancelled their talks. As a result they have decided to rely less on corporate funding and more on the broad community of individuals who support Free Software and copyleft. So the SFC has launched a campaign to create a community of folks who stand up for copyleft and the GPL by supporting their work on promoting and supporting copyleft and Free Software.

If you support Free Software, like what the SFC do, agree with their compliance principles, are happy about their successes in 2015, work on a project that is an SFC member and or just want to stand up for copyleft, please join Christopher Allan Webber, Carol Smith, Jono Bacon, myself and others in becoming a supporter. For the next week your donation will be matched by an anonymous donor. Please also consider asking your employer to match your donation or become a sponsor of SFC. Don't forget to spread the word about your support for SFC via email, your blog and or social media accounts.

I agree with Paul on this topic and just signed up as a Supporter of Software Freedom Conservancy myself. Perhaps you should be a supporter too?

I've needed a new OpenPGP key for a while, but have not had time to set it up properly. I wanted to generate it offline and have it available on a OpenPGP smart card for daily use, and learning how to do it and finding time to sit down with an offline machine almost took forever. But finally I've been able to complete the process, and have now moved from my old GPG key to a new GPG key. See the full transition statement, signed with both my old and new key for the details. This is my new key:

pub   3936R/111D6B29EE4E02F9 2015-11-03 [expires: 2019-11-14]
      Key fingerprint = 3AC7 B2E3 ACA5 DF87 78F1  D827 111D 6B29 EE4E 02F9
uid                  Petter Reinholdtsen <pere@hungry.com>
uid                  Petter Reinholdtsen <pere@debian.org>
sub   4096R/87BAFB0E 2015-11-03 [expires: 2019-11-02]
sub   4096R/F91E6DE9 2015-11-03 [expires: 2019-11-02]
sub   4096R/A0439BAB 2015-11-03 [expires: 2019-11-02]

The key can be downloaded from the OpenPGP key servers, signed by my old key.

If you signed my old key (DB4CCC4B2A30D729), I'd very much appreciate a signature on my new key, details and instructions in the transition statement. I m happy to reciprocate if you have a similarly signed transition statement to present.

When I get a new laptop, the battery life time at the start is OK. But this do not last. The last few laptops gave me a feeling that within a year, the life time is just a fraction of what it used to be, and it slowly become painful to use the laptop without power connected all the time. Because of this, when I got a new Thinkpad X230 laptop about two years ago, I decided to monitor its battery state to have more hard facts when the battery started to fail.

First I tried to find a sensible Debian package to record the battery status, assuming that this must be a problem already handled by someone else. I found battery-stats, which collects statistics from the battery, but it was completely broken. I sent a few suggestions to the maintainer, but decided to write my own collector as a shell script while I waited for feedback from him. Via a blog post about the battery development on a MacBook Air I also discovered batlog, not available in Debian.

I started my collector 2013-07-15, and it has been collecting battery stats ever since. Now my /var/log/hjemmenett-battery-status.log file contain around 115,000 measurements, from the time the battery was working great until now, when it is unable to charge above 7% of original capacity. My collector shell script is quite simple and look like this:

#!/bin/sh
# Inspired by
# http://www.ifweassume.com/2013/08/the-de-evolution-of-my-laptop-battery.html
# See also
# http://blog.sleeplessbeastie.eu/2013/01/02/debian-how-to-monitor-battery-capacity/
logfile=/var/log/hjemmenett-battery-status.log

files="manufacturer model_name technology serial_number \
    energy_full energy_full_design energy_now cycle_count status"

if [ ! -e "$logfile" ] ; then
    (
	printf "timestamp,"
	for f in $files; do
	    printf "%s," $f
	done
	echo
    ) > "$logfile"
fi

log_battery() {
    # Print complete message in one echo call, to avoid race condition
    # when several log processes run in parallel.
    msg=$(printf "%s," $(date +%s); \
	for f in $files; do \
	    printf "%s," $(cat $f); \
	done)
    echo "$msg"
}

cd /sys/class/power_supply

for bat in BAT*; do
    (cd $bat && log_battery >> "$logfile")
done

The script is called when the power management system detect a change in the power status (power plug in or out), and when going into and out of hibernation and suspend. In addition, it collect a value every 10 minutes. This make it possible for me know when the battery is discharging, charging and how the maximum charge change over time. The code for the Debian package is now available on github.

The collected log file look like this:

timestamp,manufacturer,model_name,technology,serial_number,energy_full,energy_full_design,energy_now,cycle_count,status,
1376591133,LGC,45N1025,Li-ion,974,62800000,62160000,39050000,0,Discharging,
[...]
1443090528,LGC,45N1025,Li-ion,974,4900000,62160000,4900000,0,Full,
1443090601,LGC,45N1025,Li-ion,974,4900000,62160000,4900000,0,Full,

I wrote a small script to create a graph of the charge development over time. This graph depicted above show the slow death of my laptop battery.

But why is this happening? Why are my laptop batteries always dying in a year or two, while the batteries of space probes and satellites keep working year after year. If we are to believe Battery University, the cause is me charging the battery whenever I have a chance, and the fix is to not charge the Lithium-ion batteries to 100% all the time, but to stay below 90% of full charge most of the time. I've been told that the Tesla electric cars limit the charge of their batteries to 80%, with the option to charge to 100% when preparing for a longer trip (not that I would want a car like Tesla where rights to privacy is abandoned, but that is another story), which I guess is the option we should have for laptops on Linux too.

Is there a good and generic way with Linux to tell the battery to stop charging at 80%, unless requested to charge to 100% once in preparation for a longer trip? I found one recipe on askubuntu for Ubuntu to limit charging on Thinkpad to 80%, but could not get it to work (kernel module refused to load).

I wonder why the battery capacity was reported to be more than 100% at the start. I also wonder why the "full capacity" increases some times, and if it is possible to repeat the process to get the battery back to design capacity. And I wonder if the discharge and charge speed change over time, or if this stay the same. I did not yet try to write a tool to calculate the derivative values of the battery level, but suspect some interesting insights might be learned from those.

Update 2015-09-24: I got a tip to install the packages acpi-call-dkms and tlp (unfortunately missing in Debian stable) packages instead of the tp-smapi-dkms package I had tried to use initially, and use 'tlp setcharge 40 80' to change when charging start and stop. I've done so now, but expect my existing battery is toast and need to be replaced. The proposal is unfortunately Thinkpad specific.

Several people contacted me after my previous blog post about my need for a new laptop, and provided very useful feedback. I wish to thank every one of these. Several pointed me to the possibility of fixing my X230, and I am already in the process of getting Lenovo to do so thanks to the on site, next day support contract covering the machine. But the battery is almost useless (I expect to replace it with a non-official battery) and I do not expect the machine to live for many more years, so it is time to plan its replacement. If I did not have a support contract, it was suggested to find replacement parts using FrancEcrans, but it might present a language barrier as I do not understand French.

One tip I got was to use the Skinflint web service to compare laptop models. It seem to have more models available than prisjakt.no. Another tip I got from someone I know have similar keyboard preferences was that the HP EliteBook 840 keyboard is not very good, and this matches my experience with earlier EliteBook keyboards I tested. Because of this, I will not consider it any further.

When I wrote my blog post, I was not aware of Thinkpad X250, the newest Thinkpad X model. The keyboard reintroduces mouse buttons (which is missing from the X240), and is working fairly well with Debian Sid/Unstable according to Corsac.net. The reports I got on the keyboard quality are not consistent. Some say the keyboard is good, others say it is ok, while others say it is not very good. Those with experience from X41 and and X60 agree that the X250 keyboard is not as good as those trusty old laptops, and suggest I keep and fix my X230 instead of upgrading, or get a used X230 to replace it. I'm also told that the X250 lack leds for caps lock, disk activity and battery status, which is very convenient on my X230. I'm also told that the CPU fan is running very often, making it a bit noisy. In any case, the X250 do not work out of the box with Debian Stable/Jessie, one of my requirements.

I have also gotten a few vendor proposals, one was Pro-Star, another was Libreboot. The latter look very attractive to me.

Again, thank you all for the very useful feedback. It help a lot as I keep looking for a replacement.

Update 2015-07-06: I was recommended to check out the lapstore.de web shop for used laptops. They got several different old thinkpad X models, and provide one year warranty.

My primary work horse laptop is failing, and will need a replacement soon. The left 5 cm of the screen on my Thinkpad X230 started flickering yesterday, and I suspect the cause is a broken cable, as changing the angle of the screen some times get rid of the flickering.

My requirements have not really changed since I bought it, and is still as I described them in 2013. The last time I bought a laptop, I had good help from prisjakt.no where I could select at least a few of the requirements (mouse pin, wifi, weight) and go through the rest manually. Three button mouse and a good keyboard is not available as an option, and all the three laptop models proposed today (Thinkpad X240, HP EliteBook 820 G1 and G2) lack three mouse buttons). It is also unclear to me how good the keyboard on the HP EliteBooks are. I hope Lenovo have not messed up the keyboard, even if the quality and robustness in the X series have deteriorated since X41.

I wonder how I can find a sensible laptop when none of the options seem sensible to me? Are there better services around to search the set of available laptops for features? Please send me an email if you have suggestions.

Update 2015-07-23: I got a suggestion to check out the FSF list of endorsed hardware, which is useful background information.

By now, it is well known that Debian Jessie will not be using sysvinit as its boot system by default. But how can one keep using sysvinit in Jessie? It is fairly easy, and here are a few recipes, courtesy of Erich Schubert and Simon McVittie.

If you already are using Wheezy and want to upgrade to Jessie and keep sysvinit as your boot system, create a file /etc/apt/preferences.d/use-sysvinit with this content before you upgrade:

Package: systemd-sysv
Pin: release o=Debian
Pin-Priority: -1

This file content will tell apt and aptitude to not consider installing systemd-sysv as part of any installation and upgrade solution when resolving dependencies, and thus tell it to avoid systemd as a default boot system. The end result should be that the upgraded system keep using sysvinit.

If you are installing Jessie for the first time, there is no way to get sysvinit installed by default (debootstrap used by debian-installer have no option for this), but one can tell the installer to switch to sysvinit before the first boot. Either by using a kernel argument to the installer, or by adding a line to the preseed file used. First, the kernel command line argument:

preseed/late_command="in-target apt-get install --purge -y sysvinit-core"

Next, the line to use in a preseed file:

d-i preseed/late_command string in-target apt-get install -y sysvinit-core

One can of course also do this after the first boot by installing the sysvinit-core package.

I recommend only using sysvinit if you really need it, as the sysvinit boot sequence in Debian have several hardware specific bugs on Linux caused by the fact that it is unpredictable when hardware devices show up during boot. But on the other hand, the new default boot system still have a few rough edges I hope will be fixed before Jessie is released.

Update 2014-11-26: Inspired by a blog post by Torsten Glaser, added --purge to the preseed line.

The right to communicate with your friends and family in private, without anyone snooping, is a right every citicen have in a liberal democracy. But this right is under serious attack these days.

A while back it occurred to me that one way to make the dragnet surveillance conducted by NSA, GCHQ, FRA and others (and confirmed by the whisleblower Snowden) more expensive for Internet email, is to deliver all email using SMTP via Tor. Such SMTP option would be a nice addition to the FreedomBox project if we could send email between FreedomBox machines without leaking metadata about the emails to the people peeking on the wire. I proposed this on the FreedomBox project mailing list in October and got a lot of useful feedback and suggestions. It also became obvious to me that this was not a novel idea, as the same idea was tested and documented by Johannes Berg as early as 2006, and both the Mailpile and the Cables systems propose a similar method / protocol to pass emails between users.

To implement such system one need to set up a Tor hidden service providing the SMTP protocol on port 25, and use email addresses looking like username@hidden-service-name.onion. With such addresses the connections to port 25 on hidden-service-name.onion using Tor will go to the correct SMTP server. To do this, one need to configure the Tor daemon to provide the hidden service and the mail server to accept emails for this .onion domain. To learn more about Exim configuration in Debian and test the design provided by Johannes Berg in his FAQ, I set out yesterday to create a Debian package for making it trivial to set up such SMTP over Tor service based on Debian. Getting it to work were fairly easy, and the source code for the Debian package is available from github. I plan to move it into Debian if further testing prove this to be a useful approach.

If you want to test this, set up a blank Debian machine without any mail system installed (or run apt-get purge exim4-config to get rid of exim4). Install tor, clone the git repository mentioned above, build the deb and install it on the machine. Next, run /usr/lib/exim4-smtorp/setup-exim-hidden-service and follow the instructions to get the service up and running. Restart tor and exim when it is done, and test mail delivery using swaks like this:

torsocks swaks --server dutlqrrmjhtfa3vp.onion \
  --to fbx@dutlqrrmjhtfa3vp.onion

This will test the SMTP delivery using tor. Replace the email address with your own address to test your server. :)

The setup procedure is still to complex, and I hope it can be made easier and more automatic. Especially the tor setup need more work. Also, the package include a tor-smtp tool written in C, but its task should probably be rewritten in some script language to make the deb architecture independent. It would probably also make the code easier to review. The tor-smtp tool currently need to listen on a socket for exim to talk to it and is started using xinetd. It would be better if no daemon and no socket is needed. I suspect it is possible to get exim to run a command line tool for delivery instead of talking to a socket, and hope to figure out how in a future version of this system.

Until I wipe my test machine, I can be reached using the fbx@dutlqrrmjhtfa3vp.onion mail address, deliverable over SMTorP. :)

If you ever had to moderate a mailman list, like the ones on alioth.debian.org, you know the web interface is fairly slow to operate. First you visit one web page, enter the moderation password and get a new page shown with a list of all the messages to moderate and various options for each email address. This take a while for every list you moderate, and you need to do it regularly to do a good job as a list moderator. But there is a quick alternative, the listadmin program. It allow you to check lists for new messages to moderate in a fraction of a second. Here is a test run on two lists I recently took over:

% time listadmin xiph
fetching data for pkg-xiph-commits@lists.alioth.debian.org ... nothing in queue
fetching data for pkg-xiph-maint@lists.alioth.debian.org ... nothing in queue

real    0m1.709s
user    0m0.232s
sys     0m0.012s
%

In 1.7 seconds I had checked two mailing lists and confirmed that there are no message in the moderation queue. Every morning I currently moderate 68 mailman lists, and it normally take around two minutes. When I took over the two pkg-xiph lists above a few days ago, there were 400 emails waiting in the moderator queue. It took me less than 15 minutes to process them all using the listadmin program.

If you install the listadmin package from Debian and create a file ~/.listadmin.ini with content like this, the moderation task is a breeze:

username username@example.org
spamlevel 23
default discard
discard_if_reason "Posting restricted to members only. Remove us from your mail list."

password secret
adminurl https://{domain}/mailman/admindb/{list}
mailman-list@lists.example.com

password hidden
other-list@otherserver.example.org

There are other options to set as well. Check the manual page to learn the details.

If you are forced to moderate lists on a mailman installation where the SSL certificate is self signed or not properly signed by a generally accepted signing authority, you can set a environment variable when calling listadmin to disable SSL verification:

PERL_LWP_SSL_VERIFY_HOSTNAME=0 listadmin

If you want to moderate a subset of the lists you take care of, you can provide an argument to the listadmin script like I do in the initial screen dump (the xiph argument). Using an argument, only lists matching the argument string will be processed. This make it quick to accept messages if you notice the moderation request in your email.

Without the listadmin program, I would never be the moderator of 68 mailing lists, as I simply do not have time to spend on that if the process was any slower. The listadmin program have saved me hours of time I could spend elsewhere over the years. It truly is nice free software.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Update 2014-10-27: Added missing 'username' statement in configuration example. Also, I've been told that the PERL_LWP_SSL_VERIFY_HOSTNAME=0 setting do not work for everyone. Not sure why.

When PXE installing laptops with Debian, I often run into the problem that the WiFi card require some firmware to work properly. And it has been a pain to fix this using preseeding in Debian. Normally something more is needed. But thanks to my isenkram package and its recent tasksel extension, it has now become easy to do this using simple preseeding.

The isenkram-cli package provide tasksel tasks which will install firmware for the hardware found in the machine (actually, requested by the kernel modules for the hardware). (It can also install user space programs supporting the hardware detected, but that is not the focus of this story.)

To get this working in the default installation, two preeseding values are needed. First, the isenkram-cli package must be installed into the target chroot (aka the hard drive) before tasksel is executed in the pkgsel step of the debian-installer system. This is done by preseeding the base-installer/includes debconf value to include the isenkram-cli package. The package name is next passed to debootstrap for installation. With the isenkram-cli package in place, tasksel will automatically use the isenkram tasks to detect hardware specific packages for the machine being installed and install them, because isenkram-cli contain tasksel tasks.

Second, one need to enable the non-free APT repository, because most firmware unfortunately is non-free. This is done by preseeding the apt-mirror-setup step. This is unfortunate, but for a lot of hardware it is the only option in Debian.

The end result is two lines needed in your preseeding file to get firmware installed automatically by the installer:

base-installer base-installer/includes string isenkram-cli
apt-mirror-setup apt-setup/non-free boolean true

The current version of isenkram-cli in testing/jessie will install both firmware and user space packages when using this method. It also do not work well, so use version 0.15 or later. Installing both firmware and user space packages might give you a bit more than you want, so I decided to split the tasksel task in two, one for firmware and one for user space programs. The firmware task is enabled by default, while the one for user space programs is not. This split is implemented in the package currently in unstable.

If you decide to give this a go, please let me know (via email) how this recipe work for you. :)

So, I bet you are wondering, how can this work. First and foremost, it work because tasksel is modular, and driven by whatever files it find in /usr/lib/tasksel/ and /usr/share/tasksel/. So the isenkram-cli package place two files for tasksel to find. First there is the task description file (/usr/share/tasksel/descs/isenkram.desc):

Task: isenkram-packages
Section: hardware
Description: Hardware specific packages (autodetected by isenkram)
 Based on the detected hardware various hardware specific packages are
 proposed.
Test-new-install: show show
Relevance: 8
Packages: for-current-hardware

Task: isenkram-firmware
Section: hardware
Description: Hardware specific firmware packages (autodetected by isenkram)
 Based on the detected hardware various hardware specific firmware
 packages are proposed.
Test-new-install: mark show
Relevance: 8
Packages: for-current-hardware-firmware

The key parts are Test-new-install which indicate how the task should be handled and the Packages line referencing to a script in /usr/lib/tasksel/packages/. The scripts use other scripts to get a list of packages to install. The for-current-hardware-firmware script look like this to list relevant firmware for the machine:

#!/bin/sh
#
PATH=/usr/sbin:$PATH
export PATH
isenkram-autoinstall-firmware -l

With those two pieces in place, the firmware is installed by tasksel during the normal d-i run. :)

If you want to test what tasksel will install when isenkram-cli is installed, run DEBIAN_PRIORITY=critical tasksel --test --new-install to get the list of packages that tasksel would install.

Debian Edu will be pilots in testing this feature, as isenkram is used there now to install firmware, replacing the earlier scripts.

Today I came across an unexpected Ubuntu boot screen. Above the bread shelf on the ICA shop at Storo in Oslo, the grub menu of Ubuntu with Linux kernel 3.2.0-23 (ie probably version 12.04 LTS) was stuck on a screen normally showing the bread types and prizes:

If it had booted as it was supposed to, I would never had known about this hidden Linux installation. It is interesting what errors can reveal.

The lsdvd project got a new set of developers a few weeks ago, after the original developer decided to step down and pass the project to fresh blood. This project is now maintained by Petter Reinholdtsen and Steve Dibb.

I just wrapped up a new lsdvd release, available in git or from the download page. This is the changelog dated 2014-10-03 for version 0.17.

• Ignore 'phantom' audio, subtitle tracks
• Check for garbage in the program chains, which indicate that a track is non-existant, to work around additional copy protection
• Fix displaying content type for audio tracks, subtitles
• Fix pallete display of first entry
• Fix include orders
• Ignore read errors in titles that would not be displayed anyway
• Fix the chapter count
• Make sure the array size and the array limit used when initialising the palette size is the same.
• Fix array printing.
• Correct subsecond calculations.
• Add sector information to the output format.
• Clean up code to be closer to ANSI C and compile without warnings with more GCC compiler warnings.

This change bring together patches for lsdvd in use in various Linux and Unix distributions, as well as patches submitted to the project the last nine years. Please check it out. :)

The Debian Edu / Skolelinux project provide a Linux solution for schools, including a powerful desktop with education software, a central server providing web pages, user database, user home directories, central login and PXE boot of both clients without disk and the installation to install Debian Edu on machines with disk (and a few other services perhaps to small to mention here). We in the Debian Edu team are currently working on the Jessie based version, trying to get everything in shape before the freeze, to avoid having to maintain our own package repository in the future. The current status can be seen on the Debian wiki, and there is still heaps of work left. Some fatal problems block testing, breaking the installer, but it is possible to work around these to get anyway. Here is a recipe on how to get the installation limping along.

First, download the test ISO via ftp, http or rsync (use ftp.skolelinux.org::cd-edu-testing-nolocal-netinst/debian-edu-amd64-i386-NETINST-1.iso). The ISO build was broken on Tuesday, so we do not get a new ISO every 12 hours or so, but thankfully the ISO we already got we are able to install with some tweaking.

When you get to the Debian Edu profile question, go to tty2 (use Alt-Ctrl-F2), run

nano /usr/bin/edu-eatmydata-install

and add 'exit 0' as the second line, disabling the eatmydata optimization. Return to the installation, select the profile you want and continue. Without this change, exim4-config will fail to install due to a known bug in eatmydata.

When you get the grub question at the end, answer /dev/sda (or if this do not work, figure out what your correct value would be. All my test machines need /dev/sda, so I have no advice if it do not fit your need.

If you installed a profile including a graphical desktop, log in as root after the initial boot from hard drive, and install the education-desktop-XXX metapackage. XXX can be kde, gnome, lxde, xfce or mate. If you want several desktop options, install more than one metapackage. Once this is done, reboot and you should have a working graphical login screen. This workaround should no longer be needed once the education-tasks package version 1.801 enter testing in two days.

I believe the ISO build will start working on two days when the new tasksel package enter testing and Steve McIntyre get a chance to update the debian-cd git repository. The eatmydata, grub and desktop issues are already fixed in unstable and testing, and should show up on the ISO as soon as the ISO build start working again. Well the eatmydata optimization is really just disabled. The proper fix require an upload by the eatmydata maintainer applying the patch provided in bug #702711. The rest have proper fixes in unstable.

I hope this get you going with the installation testing, as we are quickly running out of time trying to get our Jessie based installation ready before the distribution freeze in a month.

I use the lsdvd tool to handle my fairly large DVD collection. It is a nice command line tool to get details about a DVD, like title, tracks, track length, etc, in XML, Perl or human readable format. But lsdvd have not seen any new development since 2006 and had a few irritating bugs affecting its use with some DVDs. Upstream seemed to be dead, and in January I sent a small probe asking for a version control repository for the project, without any reply. But I use it regularly and would like to get an updated version into Debian. So two weeks ago I tried harder to get in touch with the project admin, and after getting a reply from him explaining that he was no longer interested in the project, I asked if I could take over. And yesterday, I became project admin.

I've been in touch with a Gentoo developer and the Debian maintainer interested in joining forces to maintain the upstream project, and I hope we can get a new release out fairly quickly, collecting the patches spread around on the internet into on place. I've added the relevant Debian patches to the freshly created git repository, and expect the Gentoo patches to make it too. If you got a DVD collection and care about command line tools, check out the git source and join the project mailing list. :)

The Debian installer could be a lot quicker. When we install more than 2000 packages in Skolelinux / Debian Edu using tasksel in the installer, unpacking the binary packages take forever. A part of the slow I/O issue was discussed in bug #613428 about too much file system sync-ing done by dpkg, which is the package responsible for unpacking the binary packages. Other parts (like code executed by postinst scripts) might also sync to disk during installation. All this sync-ing to disk do not really make sense to me. If the machine crash half-way through, I start over, I do not try to salvage the half installed system. So the failure sync-ing is supposed to protect against, hardware or system crash, is not really relevant while the installer is running.

A few days ago, I thought of a way to get rid of all the file system sync()-ing in a fairly non-intrusive way, without the need to change the code in several packages. The idea is not new, but I have not heard anyone propose the approach using dpkg-divert before. It depend on the small and clever package eatmydata, which uses LD_PRELOAD to replace the system functions for syncing data to disk with functions doing nothing, thus allowing programs to live dangerous while speeding up disk I/O significantly. Instead of modifying the implementation of dpkg, apt and tasksel (which are the packages responsible for selecting, fetching and installing packages), it occurred to me that we could just divert the programs away, replace them with a simple shell wrapper calling "eatmydata $program $@", to get the same effect. Two days ago I decided to test the idea, and wrapped up a simple implementation for the Debian Edu udeb.

The effect was stunning. In my first test it reduced the running time of the pkgsel step (installing tasks) from 64 to less than 44 minutes (20 minutes shaved off the installation) on an old Dell Latitude D505 machine. I am not quite sure what the optimised time would have been, as I messed up the testing a bit, causing the debconf priority to get low enough for two questions to pop up during installation. As soon as I saw the questions I moved the installation along, but do not know how long the question were holding up the installation. I did some more measurements using Debian Edu Jessie, and got these results. The time measured is the time stamp in /var/log/syslog between the "pkgsel: starting tasksel" and the "pkgsel: finishing up" lines, if you want to do the same measurement yourself. In Debian Edu, the tasksel dialog do not show up, and the timing thus do not depend on how quickly the user handle the tasksel dialog.

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

The test is done using a netinst ISO on a USB stick, so some of the time is spent downloading packages. The connection to the Internet was 100Mbit/s during testing, so downloading should not be a significant factor in the measurement. Download typically took a few seconds to a few minutes, depending on the amount of packages being installed.

The speedup is implemented by using two hooks in Debian Installer, the pre-pkgsel.d hook to set up the diverts, and the finish-install.d hook to remove the divert at the end of the installation. I picked the pre-pkgsel.d hook instead of the post-base-installer.d hook because I test using an ISO without the eatmydata package included, and the post-base-installer.d hook in Debian Edu can only operate on packages included in the ISO. The negative effect of this is that I am unable to activate this optimization for the kernel installation step in d-i. If the code is moved to the post-base-installer.d hook, the speedup would be larger for the entire installation.

I've implemented this in the debian-edu-install git repository, and plan to provide the optimization as part of the Debian Edu installation. If you want to test this yourself, you can create two files in the installer (or in an udeb). One shell script need do go into /usr/lib/pre-pkgsel.d/, with content like this:

#!/bin/sh
set -e
. /usr/share/debconf/confmodule
info() {
    logger -t my-pkgsel "info: $*"
}
error() {
    logger -t my-pkgsel "error: $*"
}
override_install() {
    apt-install eatmydata || true
    if [ -x /target/usr/bin/eatmydata ] ; then
        for bin in dpkg apt-get aptitude tasksel ; do
            file=/usr/bin/$bin
            # Test that the file exist and have not been diverted already.
            if [ -f /target$file ] ; then
                info "diverting $file using eatmydata"
                printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
                    > /target$file.edu
                chmod 755 /target$file.edu
                in-target dpkg-divert --package debian-edu-config \
                    --rename --quiet --add $file
                ln -sf ./$bin.edu /target$file
            else
                error "unable to divert $file, as it is missing."
            fi
        done
    else
        error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
    fi
}

override_install

To clean up, another shell script should go into /usr/lib/finish-install.d/ with code like this:

#! /bin/sh -e
. /usr/share/debconf/confmodule
error() {
    logger -t my-finish-install "error: $@"
}
remove_install_override() {
    for bin in dpkg apt-get aptitude tasksel ; do
        file=/usr/bin/$bin
        if [ -x /target$file.edu ] ; then
            rm /target$file
            in-target dpkg-divert --package debian-edu-config \
                --rename --quiet --remove $file
            rm /target$file.edu
        else
            error "Missing divert for $file."
        fi
    done
    sync # Flush file buffers before continuing
}

remove_install_override

In Debian Edu, I placed both code fragments in a separate script edu-eatmydata-install and call it from the pre-pkgsel.d and finish-install.d scripts.

By now you might ask if this change should get into the normal Debian installer too? I suspect it should, but am not sure the current debian-installer coordinators find it useful enough. It also depend on the side effects of the change. I'm not aware of any, but I guess we will see if the change is safe after some more testing. Perhaps there is some package in Debian depending on sync() and fsync() having effect? Perhaps it should go into its own udeb, to allow those of us wanting to enable it to do so without affecting everyone.

Update 2014-09-24: Since a few days ago, enabling this optimization will break installation of all programs using gnutls because of bug #702711. An updated eatmydata package in Debian will solve it.

Update 2014-10-17: The bug mentioned above is fixed in testing and the optimization work again. And I have discovered that the dpkg-divert trick is not really needed and implemented a slightly simpler approach as part of the debian-edu-install package. See tools/edu-eatmydata-install in the source package.

Update 2014-11-11: Unfortunately, a new bug #765738 in eatmydata only triggering on i386 made it into testing, and broke this installation optimization again. If unblock request 768893 is accepted, it should be working again.

Yesterday, I had the pleasure of attending a talk with the Norwegian Unix User Group about the OpenPGP keyserver pool sks-keyservers.net, and was very happy to learn that there is a large set of publicly available key servers to use when looking for peoples public key. So far I have used subkeys.pgp.net, and some times wwwkeys.nl.pgp.net when the former were misbehaving, but those days are ended. The servers I have used up until yesterday have been slow and some times unavailable. I hope those problems are gone now.

Behind the round robin DNS entry of the sks-keyservers.net service there is a pool of more than 100 keyservers which are checked every day to ensure they are well connected and up to date. It must be better than what I have used so far. :)

Yesterdays speaker told me that the service is the default keyserver provided by the default configuration in GnuPG, but this do not seem to be used in Debian. Perhaps it should?

Anyway, I've updated my ~/.gnupg/options file to now include this line:

keyserver pool.sks-keyservers.net

With GnuPG version 2 one can also locate the keyserver using SRV entries in DNS. Just for fun, I did just that at work, so now every user of GnuPG at the University of Oslo should find a OpenGPG keyserver automatically should their need it:

% host -t srv _pgpkey-http._tcp.uio.no
_pgpkey-http._tcp.uio.no has SRV record 0 100 11371 pool.sks-keyservers.net.
%

Now if only the HKP lookup protocol supported finding signature paths, I would be very happy. It can look up a given key or search for a user ID, but I normally do not want that, but to find a trust path from my key to another key. Given a user ID or key ID, I would like to find (and download) the keys representing a signature path from my key to the key in question, to be able to get a trust path between the two keys. This is as far as I can tell not possible today. Perhaps something for a future version of the protocol?

The Debian Edu / Skolelinux project provide an instruction manual for teachers, system administrators and other users that contain useful tips for setting up and maintaining a Debian Edu installation. This text is about how the text processing of this manual is handled in the project.

One goal of the project is to provide information in the native language of its users, and for this we need to handle translations. But we also want to make sure each language contain the same information, so for this we need a good way to keep the translations in sync. And we want it to be easy for our users to improve the documentation, avoiding the need to learn special formats or tools to contribute, and the obvious way to do this is to make it possible to edit the documentation using a web browser. We also want it to be easy for translators to keep the translation up to date, and give them help in figuring out what need to be translated. Here is the list of tools and the process we have found trying to reach all these goals.

We maintain the authoritative source of our manual in the Debian wiki, as several wiki pages written in English. It consist of one front page with references to the different chapters, several pages for each chapter, and finally one "collection page" gluing all the chapters together into one large web page (aka the AllInOne page). The AllInOne page is the one used for further processing and translations. Thanks to the fact that the MoinMoin installation on wiki.debian.org support exporting pages in the Docbook format, we can fetch the list of pages to export using the raw version of the AllInOne page, loop over each of them to generate a Docbook XML version of the manual. This process also download images and transform image references to use the locally downloaded images. The generated Docbook XML files are slightly broken, so some post-processing is done using the documentation/scripts/get_manual program, and the result is a nice Docbook XML file (debian-edu-wheezy-manual.xml) and a handfull of images. The XML file can now be used to generate PDF, HTML and epub versions of the English manual. This is the basic step of our process, making PDF (using dblatex), HTML (using xsltproc) and epub (using dbtoepub) version from Docbook XML, and the resulting files are placed in the debian-edu-doc-en binary package.

But English documentation is not enough for us. We want translated documentation too, and we want to make it easy for translators to track the English original. For this we use the poxml package, which allow us to transform the English Docbook XML file into a translation file (a .pot file), usable with the normal gettext based translation tools used by those translating free software. The pot file is used to create and maintain translation files (several .po files), which the translations update with the native language translations of all titles, paragraphs and blocks of text in the original. The next step is combining the original English Docbook XML and the translation file (say debian-edu-wheezy-manual.nb.po), to create a translated Docbook XML file (in this case debian-edu-wheezy-manual.nb.xml). This translated (or partly translated, if the translation is not complete) Docbook XML file can then be used like the original to create a PDF, HTML and epub version of the documentation.

The translators use different tools to edit the .po files. We recommend using lokalize, while some use emacs and vi, others can use web based editors like Poodle or Transifex. All we care about is where the .po file end up, in our git repository. Updated translations can either be committed directly to git, or submitted as bug reports against the debian-edu-doc package.

One challenge is images, which both might need to be translated (if they show translated user applications), and are needed in different formats when creating PDF and HTML versions (epub is a HTML version in this regard). For this we transform the original PNG images to the needed density and format during build, and have a way to provide translated images by storing translated versions in images/$LANGUAGECODE/. I am a bit unsure about the details here. The package maintainers know more.

If you wonder what the result look like, we provide the content of the documentation packages on the web. See for example the Italian PDF version or the German HTML version. We do not yet build the epub version by default, but perhaps it will be done in the future.

To learn more, check out the debian-edu-doc package, the manual on the wiki and the translation instructions in the manual.

It would be nice if it was easier in Debian to get all the hardware related packages relevant for the computer installed automatically. So I implemented one, using my Isenkram package. To use it, install the tasksel and isenkram packages and run tasksel as user root. You should be presented with a new option, "Hardware specific packages (autodetected by isenkram)". When you select it, tasksel will install the packages isenkram claim is fit for the current hardware, hot pluggable or not.

The implementation is in two files, one is the tasksel menu entry description, and the other is the script used to extract the list of packages to install. The first part is in /usr/share/tasksel/descs/isenkram.desc and look like this:

Task: isenkram
Section: hardware
Description: Hardware specific packages (autodetected by isenkram)
 Based on the detected hardware various hardware specific packages are
 proposed.
Test-new-install: mark show
Relevance: 8
Packages: for-current-hardware

The second part is in /usr/lib/tasksel/packages/for-current-hardware and look like this:

#!/bin/sh
#
(
    isenkram-lookup
    isenkram-autoinstall-firmware -l
) | sort -u

All in all, a very short and simple implementation making it trivial to install the hardware dependent package we all may want to have installed on our machines. I've not been able to find a way to get tasksel to tell you exactly which packages it plan to install before doing the installation. So if you are curious or careful, check the output from the isenkram-* command line tools first.

The information about which packages are handling which hardware is fetched either from the isenkram package itself in /usr/share/isenkram/, from git.debian.org or from the APT package database (using the Modaliases header). The APT package database parsing have caused a nasty resource leak in the isenkram daemon (bugs #719837 and #730704). The cause is in the python-apt code (bug #745487), but using a workaround I was able to get rid of the file descriptor leak and reduce the memory leak from ~30 MiB per hardware detection down to around 2 MiB per hardware detection. It should make the desktop daemon a lot more useful. The fix is in version 0.7 uploaded to unstable today.

I believe the current way of mapping hardware to packages in Isenkram is is a good draft, but in the future I expect isenkram to use the AppStream data source for this. A proposal for getting proper AppStream support into Debian is floating around as DEP-11, and GSoC project will take place this summer to improve the situation. I look forward to seeing the result, and welcome patches for isenkram to start using the information when it is ready.

If you want your package to map to some specific hardware, either add a "Xb-Modaliases" header to your control file like I did in the pymissile package or submit a bug report with the details to the isenkram package. See also all my blog posts tagged isenkram for details on the notation. I expect the information will be migrated to AppStream eventually, but for the moment I got no better place to store it.

The Freedombox project is working on providing the software and hardware to make it easy for non-technical people to host their data and communication at home, and being able to communicate with their friends and family encrypted and away from prying eyes. It is still going strong, and today a major mile stone was reached.

Today, the last of the packages currently used by the project to created the system images were accepted into Debian Unstable. It was the freedombox-setup package, which is used to configure the images during build and on the first boot. Now all one need to get going is the build code from the freedom-maker git repository and packages from Debian. And once the freedombox-setup package enter testing, we can build everything directly from Debian. :)

Some key packages used by Freedombox are freedombox-setup, plinth, pagekite, tor, privoxy, owncloud and dnsmasq. There are plans to integrate more packages into the setup. User documentation is maintained on the Debian wiki. Please check out the manual and help us improve it.

To test for yourself and create boot images with the FreedomBox setup, run this on a Debian machine using a user with sudo rights to become root:

sudo apt-get install git vmdebootstrap mercurial python-docutils \
  mktorrent extlinux virtualbox qemu-user-static binfmt-support \
  u-boot-tools
git clone http://anonscm.debian.org/git/freedombox/freedom-maker.git \
  freedom-maker
make -C freedom-maker dreamplug-image raspberry-image virtualbox-image

Root access is needed to run debootstrap and mount loopback devices. See the README in the freedom-maker git repo for more details on the build. If you do not want all three images, trim the make line. Note that the virtualbox-image target is not really virtualbox specific. It create a x86 image usable in kvm, qemu, vmware and any other x86 virtual machine environment. You might need the version of vmdebootstrap in Jessie to get the build working, as it include fixes for a race condition with kpartx.

If you instead want to install using a Debian CD and the preseed method, boot a Debian Wheezy ISO and use this boot argument to load the preseed values:

url=http://www.reinholdtsen.name/freedombox/preseed-jessie.dat

I have not tested it myself the last few weeks, so I do not know if it still work.

If you wonder how to help, one task you could look at is using systemd as the boot system. It will become the default for Linux in Jessie, so we need to make sure it is usable on the Freedombox. I did a simple test a few weeks ago, and noticed dnsmasq failed to start during boot when using systemd. I suspect there are other problems too. :) To detect problems, there is a test suite included, which can be run from the plinth web interface.

Give it a go and let us know how it goes on the mailing list, and help us get the new release published. :) Please join us on IRC (#freedombox on irc.debian.org) and the mailing list if you want to help make this vision come true.

For a while now, I have been looking for a sensible offsite backup solution for use at home. My requirements are simple, it must be cheap and locally encrypted (in other words, I keep the encryption keys, the storage provider do not have access to my private files). One idea me and my friends had many years ago, before the cloud storage providers showed up, was to use Google mail as storage, writing a Linux block device storing blocks as emails in the mail service provided by Google, and thus get heaps of free space. On top of this one can add encryption, RAID and volume management to have lots of (fairly slow, I admit that) cheap and encrypted storage. But I never found time to implement such system. But the last few weeks I have looked at a system called S3QL, a locally mounted network backed file system with the features I need.

S3QL is a fuse file system with a local cache and cloud storage, handling several different storage providers, any with Amazon S3, Google Drive or OpenStack API. There are heaps of such storage providers. S3QL can also use a local directory as storage, which combined with sshfs allow for file storage on any ssh server. S3QL include support for encryption, compression, de-duplication, snapshots and immutable file systems, allowing me to mount the remote storage as a local mount point, look at and use the files as if they were local, while the content is stored in the cloud as well. This allow me to have a backup that should survive fire. The file system can not be shared between several machines at the same time, as only one can mount it at the time, but any machine with the encryption key and access to the storage service can mount it if it is unmounted.

It is simple to use. I'm using it on Debian Wheezy, where the package is included already. So to get started, run apt-get install s3ql. Next, pick a storage provider. I ended up picking Greenqloud, after reading their nice recipe on how to use S3QL with their Amazon S3 service, because I trust the laws in Iceland more than those in USA when it come to keeping my personal data safe and private, and thus would rather spend money on a company in Iceland. Another nice recipe is available from the article S3QL Filesystem for HPC Storage by Jeff Layton in the HPC section of Admin magazine. When the provider is picked, figure out how to get the API key needed to connect to the storage API. With Greencloud, the key did not show up until I had added payment details to my account.

Armed with the API access details, it is time to create the file system. First, create a new bucket in the cloud. This bucket is the file system storage area. I picked a bucket name reflecting the machine that was going to store data there, but any name will do. I'll refer to it as bucket-name below. In addition, one need the API login and password, and a locally created password. Store it all in ~root/.s3ql/authinfo2 like this:

[s3c]
storage-url: s3c://s.greenqloud.com:443/bucket-name
backend-login: API-login
backend-password: API-password
fs-passphrase: local-password

I create my local passphrase using pwget 50 or similar, but any sensible way to create a fairly random password should do it. Armed with these details, it is now time to run mkfs, entering the API details and password to create it:

# mkdir -m 700 /var/lib/s3ql-cache
# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
  --ssl s3c://s.greenqloud.com:443/bucket-name
Enter backend login: 
Enter backend password: 
Before using S3QL, make sure to read the user's guide, especially
the 'Important Rules to Avoid Loosing Data' section.
Enter encryption password: 
Confirm encryption password: 
Generating random encryption key...
Creating metadata tables...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 0.00 MB of compressed metadata.
# 

The next step is mounting the file system to make the storage available.

# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
Using 4 upload threads.
Downloading and decompressing metadata...
Reading metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Mounting filesystem...
# df -h /s3ql
Filesystem                              Size  Used Avail Use% Mounted on
s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
#

The file system is now ready for use. I use rsync to store my backups in it, and as the metadata used by rsync is downloaded at mount time, no network traffic (and storage cost) is triggered by running rsync. To unmount, one should not use the normal umount command, as this will not flush the cache to the cloud storage, but instead running the umount.s3ql command like this:

# umount.s3ql /s3ql
# 

There is a fsck command available to check the file system and correct any problems detected. This can be used if the local server crashes while the file system is mounted, to reset the "already mounted" flag. This is what it look like when processing a working file system:

# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
Using cached metadata.
File system seems clean, checking anyway.
Checking DB integrity...
Creating temporary extra indices...
Checking lost+found...
Checking cached objects...
Checking names (refcounts)...
Checking contents (names)...
Checking contents (inodes)...
Checking contents (parent inodes)...
Checking objects (reference counts)...
Checking objects (backend)...
..processed 5000 objects so far..
..processed 10000 objects so far..
..processed 15000 objects so far..
Checking objects (sizes)...
Checking blocks (referenced objects)...
Checking blocks (refcounts)...
Checking inode-block mapping (blocks)...
Checking inode-block mapping (inodes)...
Checking inodes (refcounts)...
Checking inodes (sizes)...
Checking extended attributes (names)...
Checking extended attributes (inodes)...
Checking symlinks (inodes)...
Checking directory reachability...
Checking unix conventions...
Checking referential integrity...
Dropping temporary indices...
Backing up old metadata...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 0.89 MB of compressed metadata.
# 

Thanks to the cache, working on files that fit in the cache is very quick, about the same speed as local file access. Uploading large amount of data is to me limited by the bandwidth out of and into my house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, which is very close to my upload speed, and downloading the same Debian installation ISO gave me 610 kiB/s, close to my download speed. Both were measured using dd. So for me, the bottleneck is my network, not the file system code. I do not know what a good cache size would be, but suspect that the cache should e larger than your working set.

I mentioned that only one machine can mount the file system at the time. If another machine try, it is told that the file system is busy:

# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
Using 8 upload threads.
Backend reports that fs is still mounted elsewhere, aborting.
#

The file content is uploaded when the cache is full, while the metadata is uploaded once every 24 hour by default. To ensure the file system content is flushed to the cloud, one can either umount the file system, or ask S3QL to flush the cache and metadata using s3qlctrl:

# s3qlctrl upload-meta /s3ql
# s3qlctrl flushcache /s3ql
# 

If you are curious about how much space your data uses in the cloud, and how much compression and deduplication cut down on the storage usage, you can use s3qlstat on the mounted file system to get a report:

# s3qlstat /s3ql
Directory entries:    9141
Inodes:               9143
Data blocks:          8851
Total data size:      22049.38 MB
After de-duplication: 21955.46 MB (99.57% of total)
After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
Database size:        2.39 MB (uncompressed)
(some values do not take into account not-yet-uploaded dirty blocks in cache)
#

I mentioned earlier that there are several possible suppliers of storage. I did not try to locate them all, but am aware of at least Greenqloud, Google Drive, Amazon S3 web serivces, Rackspace and Crowncloud. The latter even accept payment in Bitcoin. Pick one that suit your need. Some of them provide several GiB of free storage, but the prize models are quite different and you will have to figure out what suits you best.

While researching this blog post, I had a look at research papers and posters discussing the S3QL file system. There are several, which told me that the file system is getting a critical check by the science community and increased my confidence in using it. One nice poster is titled "An Innovative Parallel Cloud Storage System using OpenStack’s SwiftObject Store and Transformative Parallel I/O Approach" by Hsing-Bung Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields and Pamela Smith. Please have a look.

Given my problems with different file systems earlier, I decided to check out the mounted S3QL file system to see if it would be usable as a home directory (in other word, that it provided POSIX semantics when it come to locking and umask handling etc). Running my test code to check file system semantics, I was happy to discover that no error was found. So the file system can be used for home directories, if one chooses to do so.

If you do not want a locally file system, and want something that work without the Linux fuse file system, I would like to mention the Tarsnap service, which also provide locally encrypted backup using a command line client. It have a nicer access control system, where one can split out read and write access, allowing some systems to write to the backup and others to only read from it.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

The Freedombox project is working on providing the software and hardware for making it easy for non-technical people to host their data and communication at home, and being able to communicate with their friends and family encrypted and away from prying eyes. It has been going on for a while, and is slowly progressing towards a new test release (0.2).

And what day could be better than the Pi day to announce that the new version will provide "hard drive" / SD card / USB stick images for Dreamplug, Raspberry Pi and VirtualBox (or any other virtualization system), and can also be installed using a Debian installer preseed file. The Debian based Freedombox is now based on Debian Jessie, where most of the needed packages used are already present. Only one, the freedombox-setup package, is missing. To try to build your own boot image to test the current status, fetch the freedom-maker scripts and build using vmdebootstrap with a user with sudo access to become root:

git clone http://anonscm.debian.org/git/freedombox/freedom-maker.git \
  freedom-maker
sudo apt-get install git vmdebootstrap mercurial python-docutils \
  mktorrent extlinux virtualbox qemu-user-static binfmt-support \
  u-boot-tools
make -C freedom-maker dreamplug-image raspberry-image virtualbox-image

Root access is needed to run debootstrap and mount loopback devices. See the README for more details on the build. If you do not want all three images, trim the make line. But note that thanks to a race condition in vmdebootstrap, the build might fail without the patch to the kpartx call.

If you instead want to install using a Debian CD and the preseed method, boot a Debian Wheezy ISO and use this boot argument to load the preseed values:

url=http://www.reinholdtsen.name/freedombox/preseed-jessie.dat

But note that due to a recently introduced bug in apt in Jessie, the installer will currently hang while setting up APT sources. Killing the 'apt-cdrom ident' process when it hang a few times during the installation will get the installation going. This affect all installations in Jessie, and I expect it will be fixed soon.

Give it a go and let us know how it goes on the mailing list, and help us get the new release published. :) Please join us on IRC (#freedombox on irc.debian.org) and the mailing list if you want to help make this vision come true.

Many years ago, I wrote a GPL licensed version of the netgroup and innetgr tools, because I needed them in Skolelinux. I called the project ng-utils, and it has served me well. I placed the project under the Hungry Programmer umbrella, and it was maintained in our CVS repository. But many years ago, the CVS repository was dropped (lost, not migrated to new hardware, not sure), and the project have lacked a proper home since then.

Last summer, I had a look at the package and made a new release fixing a irritating crash bug, but was unable to store the changes in a proper source control system. I applied for a project on Alioth, but did not have time to follow up on it. Until today. :)

After many hours of cleaning and migration, the ng-utils project now have a new home, and a git repository with the highlight of the history of the project. I published all release tarballs and imported them into the git repository. As the project is really stable and not expected to gain new features any time soon, I decided to make a new release and call it 1.0. Visit the new project home on https://alioth.debian.org/projects/ng-utils/ if you want to check it out. The new version is also uploaded into Debian Unstable.

A few days ago I decided to try to help the Hurd people to get their changes into sysvinit, to allow them to use the normal sysvinit boot system instead of their old one. This follow up on the great Google Summer of Code work done last summer by Justus Winter to get Debian on Hurd working more like Debian on Linux. To get started, I downloaded a prebuilt hard disk image from http://ftp.debian-ports.org/debian-cd/hurd-i386/current/debian-hurd.img.tar.gz, and started it using virt-manager.

The first think I had to do after logging in (root without any password) was to get the network operational. I followed the instructions on the Debian GNU/Hurd ports page and ran these commands as root to get the machine to accept a IP address from the kvm internal DHCP server:

settrans -fgap /dev/netdde /hurd/netdde
kill $(ps -ef|awk '/[p]finet/ { print $2}')
kill $(ps -ef|awk '/[d]evnode/ { print $2}')
dhclient /dev/eth0

After this, the machine had internet connectivity, and I could upgrade it and install the sysvinit packages from experimental and enable it as the default boot system in Hurd.

But before I did that, I set a password on the root user, as ssh is running on the machine it for ssh login to work a password need to be set. Also, note that a bug somewhere in openssh on Hurd block compression from working. Remember to turn that off on the client side.

Run these commands as root to upgrade and test the new sysvinit stuff:

cat > /etc/apt/sources.list.d/experimental.list <<EOF
deb http://http.debian.net/debian/ experimental main
EOF
apt-get update
apt-get dist-upgrade
apt-get install -t experimental initscripts sysv-rc sysvinit \
    sysvinit-core sysvinit-utils
update-alternatives --config runsystem

To reboot after switching boot system, you have to use reboot-hurd instead of just reboot, as there is not yet a sysvinit process able to receive the signals from the normal 'reboot' command. After switching to sysvinit as the boot system, upgrading every package and rebooting, the network come up with DHCP after boot as it should, and the settrans/pkill hack mentioned at the start is no longer needed. But for some strange reason, there are no longer any login prompt in the virtual console, so I logged in using ssh instead.

Note that there are some race conditions in Hurd making the boot fail some times. No idea what the cause is, but hope the Hurd porters figure it out. At least Justus said on IRC (#debian-hurd on irc.debian.org) that they are aware of the problem. A way to reduce the impact is to upgrade to the Hurd packages built by Justus by adding this repository to the machine:

cat > /etc/apt/sources.list.d/hurd-ci.list <<EOF
deb http://darnassus.sceen.net/~teythoon/hurd-ci/ sid main
EOF

At the moment the prebuilt virtual machine get some packages from http://ftp.debian-ports.org/debian, because some of the packages in unstable do not yet include the required patches that are lingering in BTS. This is the completely list of "unofficial" packages installed:

# aptitude search '?narrow(?version(CURRENT),?origin(Debian Ports))'
i   emacs                   - GNU Emacs editor (metapackage)
i   gdb                     - GNU Debugger
i   hurd-recommended        - Miscellaneous translators
i   isc-dhcp-client         - ISC DHCP client
i   isc-dhcp-common         - common files used by all the isc-dhcp* packages
i   libc-bin                - Embedded GNU C Library: Binaries
i   libc-dev-bin            - Embedded GNU C Library: Development binaries
i   libc0.3                 - Embedded GNU C Library: Shared libraries
i A libc0.3-dbg             - Embedded GNU C Library: detached debugging symbols
i   libc0.3-dev             - Embedded GNU C Library: Development Libraries and Hea
i   multiarch-support       - Transitional package to ensure multiarch compatibilit
i A x11-common              - X Window System (X.Org) infrastructure
i   xorg                    - X.Org X Window System
i A xserver-xorg            - X.Org X server
i A xserver-xorg-input-all  - X.Org X server -- input driver metapackage
#

All in all, testing hurd has been an interesting experience. :) X.org did not work out of the box and I never took the time to follow the porters instructions to fix it. This time I was interested in the command line stuff.

Coverity is a nice tool to find problems in C, C++ and Java code using static source code analysis. It can detect a lot of different problems, and is very useful to find memory and locking bugs in the error handling part of the source. The company behind it provide check of free software projects as a community service, and many hundred free software projects are already checked. A few days ago I decided to have a closer look at the Coverity system, and discovered that the gnash and ipmitool projects I am involved with was already registered. But these are fairly big, and I would also like to have a small and easy project to check, and decided to request checking of the chrpath project. It was added to the checker and discovered seven potential defects. Six of these were real, mostly resource "leak" when the program detected an error. Nothing serious, as the resources would be released a fraction of a second later when the program exited because of the error, but it is nice to do it right in case the source of the program some time in the future end up in a library. Having fixed all defects and added a mailing list for the chrpath developers, I decided it was time to publish a new release. These are the release notes:

New in 0.16 released 2014-01-14:

• Fixed all minor bugs discovered by Coverity.
• Updated config.sub and config.guess from the GNU project.
• Mention new project mailing list in the documentation.

You can download the new version 0.16 from alioth. Please let us know via the Alioth project if something is wrong with the new release. The test suite did not discover any old errors, so if you find a new one, please also include a test suite check.

After many years break from the package and a vain hope that development would be continued by someone else, I finally pulled my acts together this morning and wrapped up a new release of chrpath, the command line tool to modify the rpath and runpath of already compiled ELF programs. The update was triggered by the persistence of Isha Vishnoi at IBM, which needed a new config.guess file to get support for the ppc64le architecture (powerpc 64-bit Little Endian) he is working on. I checked the Debian, Ubuntu and Fedora packages for interesting patches (failed to find the source from OpenSUSE and Mandriva packages), and found quite a few nice fixes. These are the release notes:

New in 0.15 released 2013-11-24:

• Updated config.sub and config.guess from the GNU project to work with newer architectures. Thanks to isha vishnoi for the heads up.
• Updated README with current URLs.
• Added byteswap fix found in Ubuntu, credited Jeremy Kerr and Matthias Klose.
• Added missing help for -k|--keepgoing option, using patch by Petr Machata found in Fedora.
• Rewrite removal of RPATH/RUNPATH to make sure the entry in .dynamic is a NULL terminated string. Based on patch found in Fedora credited Axel Thimm and Christian Krause.

You can download the new version 0.15 from alioth. Please let us know via the Alioth project if something is wrong with the new release. The test suite did not discover any old errors, so if you find a new one, please also include a testsuite check.

If one of the points of switching to a new init system in Debian is to get rid of huge init.d scripts, I doubt we need to switch away from sysvinit and init.d scripts at all. Here is an example init.d script, ie a rewrite of /etc/init.d/rsyslog:

#!/lib/init/init-d-script
### BEGIN INIT INFO
# Provides:          rsyslog
# Required-Start:    $remote_fs $time
# Required-Stop:     umountnfs $time
# X-Stop-After:      sendsigs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: enhanced syslogd
# Description:       Rsyslog is an enhanced multi-threaded syslogd.
#                    It is quite compatible to stock sysklogd and can be 
#                    used as a drop-in replacement.
### END INIT INFO
DESC="enhanced syslogd"
DAEMON=/usr/sbin/rsyslogd

Pretty minimalistic to me... For the record, the original sysv-rc script was 137 lines, and the above is just 15 lines, most of it meta info/comments.

How to do this, you ask? Well, one create a new script /lib/init/init-d-script looking something like this:

#!/bin/sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service

#
do_start()
{
	# Return
	#   0 if daemon has been started
	#   1 if daemon was already running
	#   2 if daemon could not be started
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
		|| return 1
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
		$DAEMON_ARGS \
		|| return 2
	# Add code here, if necessary, that waits for the process to be ready
	# to handle requests from services started subsequently which depend
	# on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
	# Return
	#   0 if daemon has been stopped
	#   1 if daemon was already stopped
	#   2 if daemon could not be stopped
	#   other if a failure occurred
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
	RETVAL="$?"
	[ "$RETVAL" = 2 ] && return 2
	# Wait for children to finish too if this is a daemon that forks
	# and if the daemon is only ever run from this initscript.
	# If the above conditions are not satisfied then add some other code
	# that waits for the process to drop all resources that could be
	# needed by services started subsequently.  A last resort is to
	# sleep for some time.
	start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
	[ "$?" = 2 ] && return 2
	# Many daemons don't delete their pidfiles when they exit.
	rm -f $PIDFILE
	return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
	#
	# If the daemon can reload its configuration without
	# restarting (for example, when it is sent a SIGHUP),
	# then implement that here.
	#
	start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
	return 0
}

SCRIPTNAME=$1
scriptbasename="$(basename $1)"
echo "SN: $scriptbasename"
if [ "$scriptbasename" != "init-d-library" ] ; then
    script="$1"
    shift
    . $script
else
    exit 0
fi

NAME=$(basename $DAEMON)
PIDFILE=/var/run/$NAME.pid

# Exit if the package is not installed
#[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

case "$1" in
  start)
	[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  stop)
	[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  status)
	status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
	;;
  #reload|force-reload)
	#
	# If do_reload() is not implemented then leave this commented out
	# and leave 'force-reload' as an alias for 'restart'.
	#
	#log_daemon_msg "Reloading $DESC" "$NAME"
	#do_reload
	#log_end_msg $?
	#;;
  restart|force-reload)
	#
	# If the "reload" option is implemented then remove the
	# 'force-reload' alias
	#
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case "$?" in
	  0|1)
		do_start
		case "$?" in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
		;;
	  *)
		# Failed to stop
		log_end_msg 1
		;;
	esac
	;;
  *)
	echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
	exit 3
	;;
esac

:

It is based on /etc/init.d/skeleton, and could be improved quite a lot. I did not really polish the approach, so it might not always work out of the box, but you get the idea. I did not try very hard to optimize it nor make it more robust either.

A better argument for switching init system in Debian than reducing the size of init scripts (which is a good thing to do anyway), is to get boot system that is able to handle the kernel events sensibly and robustly, and do not depend on the boot to run sequentially. The boot and the kernel have not behaved sequentially in years.

The SPICE protocol for remote display access is the preferred solution with oVirt and RedHat Enterprise Virtualization, and I was sad to discover the other day that the browser plugin needed to use these systems seamlessly was missing in Debian. The request for a package was from 2012-04-10 with no progress since 2013-04-01, so I decided to wrap up a package based on the great work from Cajus Pollmeier and put it in a collab-maint maintained git repository to get a package I could use. I would very much like others to help me maintain the package (or just take over, I do not mind), but as no-one had volunteered so far, I just uploaded it to NEW. I hope it will be available in Debian in a few days.

The source is now available from http://anonscm.debian.org/gitweb/?p=collab-maint/spice-xpi.git;a=summary.

The vmdebootstrap program is a a very nice system to create virtual machine images. It create a image file, add a partition table, mount it and run debootstrap in the mounted directory to create a Debian system on a stick. Yesterday, I decided to try to teach it how to make images for Raspberry Pi, as part of a plan to simplify the build system for the FreedomBox project. The FreedomBox project already uses vmdebootstrap for the virtualbox images, but its current build system made multistrap based system for Dreamplug images, and it is lacking support for Raspberry Pi.

Armed with the knowledge on how to build "foreign" (aka non-native architecture) chroots for Raspberry Pi, I dived into the vmdebootstrap code and adjusted it to be able to build armel images on my amd64 Debian laptop. I ended up giving vmdebootstrap five new options, allowing me to replicate the image creation process I use to make Debian Jessie based mesh node images for the Raspberry Pi. First, the --foreign /path/to/binfm_handler option tell vmdebootstrap to call debootstrap with --foreign and to copy the handler into the generated chroot before running the second stage. This allow vmdebootstrap to create armel images on an amd64 host. Next I added two new options --bootsize size and --boottype fstype to teach it to create a separate /boot/ partition with the given file system type, allowing me to create an image with a vfat partition for the /boot/ stuff. I also added a --variant variant option to allow me to create smaller images without the Debian base system packages installed. Finally, I added an option --no-extlinux to tell vmdebootstrap to not install extlinux as a boot loader. It is not needed on the Raspberry Pi and probably most other non-x86 architectures. The changes were accepted by the upstream author of vmdebootstrap yesterday and today, and is now available from the upstream project page.

To use it to build a Raspberry Pi image using Debian Jessie, first create a small script (the customize script) to add the non-free binary blob needed to boot the Raspberry Pi and the APT source list:

#!/bin/sh
set -e # Exit on first error
rootdir="$1"
cd "$rootdir"
cat <<EOF > etc/apt/sources.list
deb http://http.debian.net/debian/ jessie main contrib non-free
EOF
# Install non-free binary blob needed to boot Raspberry Pi.  This
# install a kernel somewhere too.
wget https://raw.github.com/Hexxeh/rpi-update/master/rpi-update \
    -O $rootdir/usr/bin/rpi-update
chmod a+x $rootdir/usr/bin/rpi-update
mkdir -p $rootdir/lib/modules
touch $rootdir/boot/start.elf
chroot $rootdir rpi-update

Next, fetch the latest vmdebootstrap script and call it like this to build the image:

sudo ./vmdebootstrap \
    --variant minbase \
    --arch armel \
    --distribution jessie \
    --mirror http://http.debian.net/debian \
    --image test.img \
    --size 600M \
    --bootsize 64M \
    --boottype vfat \
    --log-level debug \
    --verbose \
    --no-kernel \
    --no-extlinux \
    --root-password raspberry \
    --hostname raspberrypi \
    --foreign /usr/bin/qemu-arm-static \
    --customize `pwd`/customize \
    --package netbase \
    --package git-core \
    --package binutils \
    --package ca-certificates \
    --package wget \
    --package kmod

The list of packages being installed are the ones needed by rpi-update to make the image bootable on the Raspberry Pi, with the exception of netbase, which is needed by debootstrap to find /etc/hosts with the minbase variant. I really wish there was a way to set up an Raspberry Pi using only packages in the Debian archive, but that is not possible as far as I know, because it boots from the GPU using a non-free binary blob.

The build host need debootstrap, kpartx and qemu-user-static and probably a few others installed. I have not checked the complete build dependency list.

The resulting image will not use the hardware floating point unit on the Raspberry PI, because the armel architecture in Debian is not optimized for that use. So the images created will be a bit slower than Raspbian based images.

The last few days I came across a few good causes that should get wider attention. I recommend signing and donating to each one of these. :)

Via Debian Project News for 2013-10-14 I came across the Outreach Program for Women program which is a Google Summer of Code like initiative to get more women involved in free software. One debian sponsor has offered to match any donation done to Debian earmarked for this initiative. I donated a few minutes ago, and hope you will to. :)

And the Electronic Frontier Foundation just announced plans to create video documentaries about the excessive spying on every Internet user that take place these days, and their need to fund the work. I've already donated. Are you next?

For my Norwegian audience, the organisation Studentenes og Akademikernes Internasjonale Hjelpefond is collecting signatures for a statement under the heading Bloggers United for Open Access for those of us asking for more focus on open access in the Norwegian government. So far 499 signatures. I hope you will sign it too.

The Freedombox project have been going on for a while, and have presented the vision, ideas and solution several places. Here is a little collection of videos of talks and presentation of the project.

• FreedomBox - 2,5 minute marketing film (Youtube)
• Eben Moglen discusses the Freedombox on CBS news 2011 (Youtube)
• Eben Moglen - Freedom in the Cloud - Software Freedom, Privacy and and Security for Web 2.0 and Cloud computing at ISOC-NY Public Meeting 2010 (Youtube)
• Fosdem 2011 Keynote by Eben Moglen presenting the Freedombox (Youtube)
• Presentation of the Freedombox by James Vasile at Elevate in Gratz 2011 (Youtube)
• Freedombox - Discovery, Identity, and Trust by Nick Daly at Freedombox Hackfest New York City in 2012 (Youtube)
• Introduction to the Freedombox at Freedombox Hackfest New York City in 2012 (Youtube)
• Freedom, Out of the Box! by Bdale Garbee at linux.conf.au Ballarat, 2012 (Youtube)
• Freedombox 1.0 by Eben Moglen and Bdale Garbee at Fosdem 2013 (FOSDEM)
• What is the FreedomBox today by Bdale Garbee at Debconf13 in Vaumarcus 2013 (Youtube)

A larger list is available from the Freedombox Wiki.

On other news, I am happy to report that Freedombox based on Debian Jessie is coming along quite well, and soon both Owncloud and using Tor should be available for testers of the Freedombox solution. :) In a few weeks I hope everything needed to test it is included in Debian. The withsqlite package is already in Debian, and the plinth package is pending in NEW. The third and vital part of that puzzle is the metapackage/setup framework, which is still pending an upload. Join us on IRC (#freedombox on irc.debian.org) and the mailing list if you want to help make this vision come true.

I was introduced to the Freedombox project in 2010, when Eben Moglen presented his vision about serving the need of non-technical people to keep their personal information private and within the legal protection of their own homes. The idea is to give people back the power over their network and machines, and return Internet back to its intended peer-to-peer architecture. Instead of depending on a central service, the Freedombox will give everyone control over their own basic infrastructure.

I've intended to join the effort since then, but other tasks have taken priority. But this summers nasty news about the misuse of trust and privilege exercised by the "western" intelligence gathering communities increased my eagerness to contribute to a point where I actually started working on the project a while back.

The initial Debian initiative based on the vision from Eben Moglen, is to create a simple and cheap Debian based appliance that anyone can hook up in their home and get access to secure and private services and communication. The initial deployment platform have been the Dreamplug, which is a piece of hardware I do not own. So to be able to test what the current Freedombox setup look like, I had to come up with a way to install it on some hardware I do have access to. I have rewritten the freedom-maker image build framework to use .deb packages instead of only copying setup into the boot images, and thanks to this rewrite I am able to set up any machine supported by Debian Wheezy as a Freedombox, using the previously mentioned deb (and a few support debs for packages missing in Debian).

The current Freedombox setup consist of a set of bootstrapping scripts (freedombox-setup), and a administrative web interface (plinth + exmachina + withsqlite), as well as a privacy enhancing proxy based on privoxy (freedombox-privoxy). There is also a web/javascript based XMPP client (jwchat) trying (unsuccessfully so far) to talk to the XMPP server (ejabberd). The web interface is pluggable, and the goal is to use it to enable OpenID services, mesh network connectivity, use of TOR, etc, etc. Not much of this is really working yet, see the project TODO for links to GIT repositories. Most of the code is on github at the moment. The HTTP proxy is operational out of the box, and the admin web interface can be used to add/remove plinth users. I've not been able to do anything else with it so far, but know there are several branches spread around github and other places with lots of half baked features.

Anyway, if you want to have a look at the current state, the following recipes should work to give you a test machine to poke at.

Debian Wheezy amd64

1. Fetch normal Debian Wheezy installation ISO.
2. Boot from it, either as CD or USB stick.

3. Press [tab] on the boot prompt and add this as a boot argument to the Debian installer:

   url=http://www.reinholdtsen.name/freedombox/preseed-wheezy.dat

4. Answer the few language/region/password questions and pick disk to install on.
5. When the installation is finished and the machine have rebooted a few times, your Freedombox is ready for testing.

Raspberry Pi Raspbian

1. Fetch a Raspbian SD card image, create SD card.
2. Boot from SD card, extend file system to fill the card completely.

3. Log in and add this to /etc/sources.list:

   deb http://www.reinholdtsen.name/freedombox wheezy main
   

4. Run this as root:

   wget -O - http://www.reinholdtsen.name/freedombox/BE1A583D.asc | \
      apt-key add -
   apt-get update
   apt-get install freedombox-setup
   /usr/lib/freedombox/setup
   

5. Reboot into your freshly created Freedombox.

You can test it on other architectures too, but because the freedombox-privoxy package is binary, it will only work as intended on the architectures where I have had time to build the binary and put it in my APT repository. But do not let this stop you. It is only a short "apt-get source -b freedombox-privoxy" away. :)

Note that by default Freedombox is a DHCP server on the 192.168.1.0/24 subnet, so if this is your subnet be careful and turn off the DHCP server by running "update-rc.d isc-dhcp-server disable" as root.

Please let me know if this works for you, or if you have any problems. We gather on the IRC channel #freedombox on irc.debian.org and the project mailing list.

Once you get your freedombox operational, you can visit http://your-host-name:8001/ to see the state of the plint welcome screen (dead end - do not be surprised if you are unable to get past it), and next visit http://your-host-name:8001/help/ to look at the rest of plinth. The default user is 'admin' and the default password is 'secret'.

Earlier, I reported about my problems using an Intel SSD 520 Series 180 GB disk. Friday I was told by IBM that the original disk should be thrown away. And as there no longer was a problem if I bricked the firmware, I decided today to try to install Intel firmware to replace the Lenovo firmware currently on the disk.

I searched the Intel site for firmware, and found issdfut_2.0.4.iso (aka Intel SATA Solid-State Drive Firmware Update Tool) which according to the site should contain the latest firmware for SSD disks. I inserted the broken disk in one of my spare laptops and booted the ISO from a USB stick. The disk was recognized, but the program claimed the newest firmware already were installed and refused to insert any Intel firmware. So no change, and the disk is still unable to handle write load. :( I guess the only way to get them working would be if Lenovo releases new firmware. No idea how likely that is. Anyway, just blogging about this test for completeness. I got a working Samsung disk, and see no point in spending more time on the broken disks.

Today I switched to my new laptop. I've previously written about the problems I had with my new Thinkpad X230, which was delivered with an 180 GB Intel SSD disk with Lenovo firmware that did not handle sustained writes. My hardware supplier have been very forthcoming in trying to find a solution, and after first trying with another identical 180 GB disks they decided to send me a 256 GB Samsung SSD disk instead to fix it once and for all. The Samsung disk survived the installation of Debian with encrypted disks (filling the disk with random data during installation killed the first two), and I thus decided to trust it with my data. I have installed it as a Debian Edu Wheezy roaming workstation hooked up with my Debian Edu Squeeze main server at home using Kerberos and LDAP, and will use it as my work station from now on.

As this is a solid state disk with no moving parts, I believe the Debian Wheezy default installation need to be tuned a bit to increase performance and increase life time of the disk. The Linux kernel and user space applications do not yet adjust automatically to such environment. To make it easier for my self, I created a draft Debian package ssd-setup to handle this tuning. The source for the ssd-setup package is available from collab-maint, and it is set up to adjust the setup of the machine by just installing the package. If there is any non-SSD disk in the machine, the package will refuse to install, as I did not try to write any logic to sort file systems in SSD and non-SSD file systems.

I consider the package a draft, as I am a bit unsure how to best set up Debian Wheezy with an SSD. It is adjusted to my use case, where I set up the machine with one large encrypted partition (in addition to /boot), put LVM on top of this and set up partitions on top of this again. See the README file in the package source for the references I used to pick the settings. At the moment these parameters are tuned:

• Set up cryptsetup to pass TRIM commands to the physical disk (adding discard to /etc/crypttab)
• Set up LVM to pass on TRIM commands to the underlying device (in this case a cryptsetup partition) by changing issue_discards from 0 to 1 in /etc/lvm/lvm.conf.
• Set relatime as a file system option for ext3 and ext4 file systems.
• Tell swap to use TRIM commands by adding 'discard' to /etc/fstab.
• Change I/O scheduler from cfq to deadline using a udev rule.
• Run fstrim on every ext3 and ext4 file system every night (from cron.daily).
• Adjust sysctl values vm.swappiness to 1 and vm.vfs_cache_pressure to 50 to reduce the kernel eagerness to swap out processes.

During installation, I cancelled the part where the installer fill the disk with random data, as this would kill the SSD performance for little gain. My goal with the encrypted file system is to ensure those stealing my laptop end up with a brick and not a working computer. I have no hope in keeping the really resourceful people from getting the data on the disk (see XKCD #538 for an explanation why). Thus I concluded that adding the discard option to crypttab is the right thing to do.

I considered using the noop I/O scheduler, as several recommended it for SSD, but others recommended deadline and a benchmark I found indicated that deadline might be better for interactive use.

I also considered using the 'discard' file system option for ext3 and ext4, but read that it would give a performance hit ever time a file is removed, and thought it best to that that slowdown once a day instead of during my work.

My package do not set up tmpfs on /var/run, /var/lock and /tmp, as this is already done by Debian Edu.

I have not yet started on the user space tuning. I expect iceweasel need some tuning, and perhaps other applications too, but have not yet had time to investigate those parts.

The package should work on Ubuntu too, but I have not yet tested it there.

As for the answer to the question in the title of this blog post, as far as I know, the only solution I know about is to replace the disk. It might be possible to flash it with Intel firmware instead of the Lenovo firmware. But I have not tried and did not want to do so without approval from Lenovo as I wanted to keep the warranty on the disk until a solution was found and they wanted the broken disks back.

A few days ago, I wrote about the problems I experienced with my new X230 and its SSD disk, which was dying during installation because it is unable to cope with sustained write. My supplier is in contact with Lenovo, and they wanted to send a replacement disk to try to fix the problem. They decided to send an identical model, so my hopes for a permanent fix was slim.

Anyway, today I got the replacement disk and tried to install Debian Edu Wheezy with encrypted disk on it. The new disk have the same firmware version as the original. This time my hope raised slightly as the installation progressed, as the original disk used to die after 4-7% of the disk was written to, while this time it kept going past 10%, 20%, 40% and even past 50%. But around 60%, the disk died again and I was back on square one. I still do not have a new laptop with a disk I can trust. I can not live with a disk that might lock up when I download a new Debian Edu / Skolelinux ISO or other large files. I look forward to hearing from my supplier with the next proposal from Lenovo.

The original disk is marked Intel SSD 520 Series 180 GB, 11S0C38722Z1ZNME35X1TR, ISN: CVCV321407HB180EGN, SA: G57560302, FW: LF1i, 29MAY2013, PBA: G39779-300, LBA 351,651,888, LI P/N: 0C38722, Pb-free 2LI, LC P/N: 16-200366, WWN: 55CD2E40002756C4, Model: SSDSC2BW180A3L 2.5" 6Gb/s SATA SSD 180G 5V 1A, ASM P/N 0C38732, FRU P/N 45N8295, P0C38732.

The replacement disk is marked Intel SSD 520 Series 180 GB, 11S0C38722Z1ZNDE34N0L0, ISN: CVCV315306RK180EGN, SA: G57560-302, FW: LF1i, 22APR2013, PBA: G39779-300, LBA 351,651,888, LI P/N: 0C38722, Pb-free 2LI, LC P/N: 16-200366, WWN: 55CD2E40000AB69E, Model: SSDSC2BW180A3L 2.5" 6Gb/s SATA SSD 180G 5V 1A, ASM P/N 0C38732, FRU P/N 45N8295, P0C38732.

The only difference is in the first number (serial number?), ISN, SA, date and WNPP values. Mentioning all the details here in case someone is able to use the information to find a way to identify the failing disk among working ones (if any such working disk actually exist).

The upcoming Saturday, 2013-07-13, we are organising a combined Debian Edu developer gathering and Debian and Ubuntu bug squashing party in Oslo. It is organised by the member assosiation NUUG and the Debian Edu / Skolelinux project together with the hack space Bitraf.

It starts 10:00 and continue until late evening. Everyone is welcome, and there is no fee to participate. There is on the other hand limited space, and only room for 30 people. Please put your name on the event wiki page if you plan to join us.

Half a year ago, I reported that I had to find a replacement for my trusty old Thinkpad X41. Unfortunately I did not have much time to spend on it, and it took a while to find a model I believe will do the job, but two days ago the replacement finally arrived. I ended up picking a Thinkpad X230 with SSD disk (NZDAJMN). I first test installed Debian Edu Wheezy as a roaming workstation, and it seemed to work flawlessly. But my second installation with encrypted disk was not as successful. More on that below.

I had a hard time trying to track down a good laptop, as my most important requirements (robust and with a good keyboard) are never listed in the feature list. But I did get good help from the search feature at Prisjakt, which allowed me to limit the list of interesting laptops based on my other requirements. A bit surprising that SSD disk are not disks according to that search interface, so I had to drop specifying the number of disks from my search parameters. I also asked around among friends to get their impression on keyboards and robustness.

So the new laptop arrived, and it is quite a lot wider than the X41. I am not quite convinced about the keyboard, as it is significantly wider than my old keyboard, and I have to stretch my hand a lot more to reach the edges. But the key response is fairly good and the individual key shape is fairly easy to handle, so I hope I will get used to it. My old X40 was starting to fail, and I really needed a new laptop now. :)

Turning off the touch pad was simple. All it took was a quick visit to the BIOS during boot it disable it.

But there is a fatal problem with the laptop. The 180 GB SSD disk lock up during load. And this happen when installing Debian Wheezy with encrypted disk, while the disk is being filled with random data. I also tested to install Ubuntu Raring, and it happen there too if I reenable the code to fill the disk with random data (it is disabled by default in Ubuntu). And the bug with is already known. It was reported to Debian as BTS report #691427 2012-10-25 (journal commit I/O error on brand-new Thinkpad T430s ext4 on lvm on SSD). It is also reported to the Linux kernel developers as Kernel bugzilla report #51861 2012-12-20 (Intel SSD 520 stops working under load (SSDSC2BW180A3L in Lenovo ThinkPad T430s)). It is also reported on the Lenovo forums, both for T430 2012-11-10 and for X230 03-20-2013. The problem do not only affect installation. The reports state that the disk lock up during use if many writes are done on the disk, so it is much no use to work around the installation problem and end up with a computer that can lock up at any moment. There is even a small C program available that will lock up the hard drive after running a few minutes by writing to a file.

I've contacted my supplier and asked how to handle this, and after contacting PCHELP Norway (request 01D1FDP) which handle support requests for Lenovo, his first suggestion was to upgrade the disk firmware. Unfortunately there is no newer firmware available from Lenovo, as my disk already have the most recent one (version LF1i). I hope to hear more from him today and hope the problem can be fixed. :)

Half a year ago, I reported that I had to find a replacement for my trusty old Thinkpad X41. Unfortunately I did not have much time to spend on it, but today the replacement finally arrived. I ended up picking a Thinkpad X230 with SSD disk (NZDAJMN). I first test installed Debian Edu Wheezy as a roaming workstation, and it worked flawlessly. As I write this, it is installing what I hope will be a more final installation, with a encrypted hard drive to ensure any dope head stealing it end up with an expencive door stop.

I had a hard time trying to track down a good laptop, as my most important requirements (robust and with a good keyboard) are never listed in the feature list. But I did get good help from the search feature at Prisjakt, which allowed me to limit the list of interesting laptops based on my other requirements. A bit surprising that SSD disk are not disks, so I had to drop number of disks from my search parameters.

I am not quite convinced about the keyboard, as it is significantly wider than my old keyboard, and I have to stretch my hand a lot more to reach the edges. But the key response is fairly good and the individual key shape is fairly easy to handle, so I hope I will get used to it. My old X40 was starting to fail, and I really needed a new laptop now. :)

I look forward to figuring out how to turn off the touch pad.

It annoys me when the computer fail to do automatically what it is perfectly capable of, and I have to do it manually to get things working. One such task is to find out what firmware packages are needed to get the hardware on my computer working. Most often this affect the wifi card, but some times it even affect the RAID controller or the ethernet card. Today I pushed version 0.4 of the Isenkram package including a new script isenkram-autoinstall-firmware handling the process of asking all the loaded kernel modules what firmware files they want, find debian packages providing these files and install the debian packages. Here is a test run on my laptop:

# isenkram-autoinstall-firmware 
info: kernel drivers requested extra firmware: ipw2200-bss.fw ipw2200-ibss.fw ipw2200-sniffer.fw
info: fetching http://http.debian.net/debian/dists/squeeze/Contents-i386.gz
info: locating packages with the requested firmware files
info: Updating APT sources after adding non-free APT source
info: trying to install firmware-ipw2x00
firmware-ipw2x00
firmware-ipw2x00
Preconfiguring packages ...
Selecting previously deselected package firmware-ipw2x00.
(Reading database ... 259727 files and directories currently installed.)
Unpacking firmware-ipw2x00 (from .../firmware-ipw2x00_0.28+squeeze1_all.deb) ...
Setting up firmware-ipw2x00 (0.28+squeeze1) ...
# 

When all the requested firmware is present, a simple message is printed instead:

# isenkram-autoinstall-firmware 
info: did not find any firmware files requested by loaded kernel modules.  exiting
# 

It could use some polish, but it is already working well and saving me some time when setting up new machines. :)

So, how does it work? It look at the set of currently loaded kernel modules, and look up each one of them using modinfo, to find the firmware files listed in the module meta-information. Next, it download the Contents file from a nearby APT mirror, and search for the firmware files in this file to locate the package with the requested firmware file. If the package is in the non-free section, a non-free APT source is added and the package is installed using apt-get install. The end result is a slightly better working machine.

I hope someone find time to implement a more polished version of this script as part of the hw-detect debian-installer module, to finally fix BTS report #655507. There really is no need to insert USB sticks with firmware during a PXE install when the packages already are available from the nearby Debian mirror.

When installing RedHat, Fedora, Debian and Ubuntu on some machines, the screen just turn black when Linux boot, either during installation or on first boot from the hard disk. I've seen it once in a while the last few years, but only recently understood the cause. I've seen it on HP laptops, and on my latest acquaintance the Packard Bell laptop. The reason seem to be in the wiring of some laptops. The system to control the screen background light is inverted, so when Linux try to turn the brightness fully on, it end up turning it off instead. I do not know which Linux drivers are affected, but this post is about the i915 driver used by the Packard Bell EasyNote LV, Thinkpad X40 and many other laptops.

The problem can be worked around two ways. Either by adding i915.invert_brightness=1 as a kernel option, or by adding a file in /etc/modprobe.d/ to tell modprobe to add the invert_brightness=1 option when it load the i915 kernel module. On Debian and Ubuntu, it can be done by running these commands as root:

echo options i915 invert_brightness=1 | tee /etc/modprobe.d/i915.conf
update-initramfs -u -k all

Since March 2012 there is a mechanism in the Linux kernel to tell the i915 driver which hardware have this problem, and get the driver to invert the brightness setting automatically. To use it, one need to add a row in the intel_quirks array in the driver source drivers/gpu/drm/i915/intel_display.c (look for "static struct intel_quirk intel_quirks"), specifying the PCI device number (vendor number 8086 is assumed) and subdevice vendor and device number.

My Packard Bell EasyNote LV got this output from lspci -vvnn for the video card in question:

00:02.0 VGA compatible controller [0300]: Intel Corporation \
    3rd Gen Core processor Graphics Controller [8086:0156] \
    (rev 09) (prog-if 00 [VGA controller])
 Subsystem: Acer Incorporated [ALI] Device [1025:0688]
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- \
    ParErr- Stepping- SE RR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- \
    SERR-  [disabled]
 Capabilities: 
 Kernel driver in use: i915

The resulting intel_quirks entry would then look like this:

struct intel_quirk intel_quirks[] = {
       ...
        /* Packard Bell EasyNote LV11HC needs invert brightness quirk */
	{ 0x0156, 0x1025, 0x0688, quirk_invert_brightness },
       ...
}

According to the kernel module instructions (as seen using modinfo i915), information about hardware needing the invert_brightness flag should be sent to the dri-devel (at) lists.freedesktop.org mailing list to reach the kernel developers. But my email about the laptop sent 2013-06-03 have not yet shown up in the web archive for the mailing list, so I suspect they do not accept emails from non-subscribers. Because of this, I sent my patch also to the Debian bug tracking system instead as BTS report #710938, to make sure the patch is not lost.

Unfortunately, it is not enough to fix the kernel to get Laptops with this problem working properly with Linux. If you use Gnome, your worries should be over at this point. But if you use KDE, there is something in KDE ignoring the invert_brightness setting and turning on the screen during login. I've reported it to Debian as BTS report #711237, and have no idea yet how to figure out exactly what subsystem is doing this. Perhaps you can help? Perhaps you know what the Gnome developers did to handle this, and this can give a clue to the KDE developers? Or you know where in KDE the screen brightness is changed during login? If so, please update the BTS report (or get in touch if you do not know how to update BTS).

Update 2013-07-19: The correct fix for this machine seem to be acpi_backlight=vendor, to disable ACPI backlight support completely, as the ACPI information on the machine is trash and it is better to leave it to the intel video driver to control the screen backlight.

Two days ago, I asked how I could install Linux on a Packard Bell EasyNote LV computer preinstalled with Windows 8. I found a solution, but am horrified with the obstacles put in the way of Linux users on a laptop with UEFI and Windows 8.

I never found out if the cause of my problems were the use of UEFI secure booting or fast boot. I suspect fast boot was the problem, causing the firmware to boot directly from HD without considering any key presses and alternative devices, but do not know UEFI settings enough to tell.

There is no way to install Linux on the machine in question without opening the box and disconnecting the hard drive! This is as far as I can tell, the only way to get access to the firmware setup menu without accepting the Windows 8 license agreement. I am told (and found description on how to) that it is possible to configure the firmware setup once booted into Windows 8. But as I believe the terms of that agreement are completely unacceptable, accepting the license was never an alternative. I do not enter agreements I do not intend to follow.

I feared I had to return the laptops and ask for a refund, and waste many hours on this, but luckily there was a way to get it to work. But I would not recommend it to anyone planning to run Linux on it, and I have become sceptical to Windows 8 certified laptops. Is this the way Linux will be forced out of the market place, by making it close to impossible for "normal" users to install Linux without accepting the Microsoft Windows license terms? Or at least not without risking to loose the warranty?

I've updated the Linux Laptop wiki page for Packard Bell EasyNote LV, to ensure the next person do not have to struggle as much as I did to get Linux into the machine.

Thanks to Bob Rosbag, Florian Weimer, Philipp Kern, Ben Hutching, Michael Tokarev and others for feedback and ideas.

I've run into quite a problem the last few days. I bought three new laptops for my parents and a few others. I bought Packard Bell Easynote LV to run Kubuntu on and use as their home computer. But I am completely unable to figure out how to install Linux on it. The computer is preinstalled with Windows 8, and I suspect it uses UEFI instead of a BIOS to boot.

The problem is that I am unable to get it to PXE boot, and unable to get it to boot the Linux installer from my USB stick. I have yet to try the DVD install, and still hope it will work. when I turn on the computer, there is no information on what buttons to press to get the normal boot menu. I expect to get some boot menu to select PXE or USB stick booting. When booting, it first ask for the language to use, then for some regional settings, and finally if I will accept the Windows 8 terms of use. As these terms are completely unacceptable to me, I have no other choice but to turn off the computer and try again to get it to boot the Linux installer.

I have gathered my findings so far on a Linlap page about the Packard Bell EasyNote LV model. If you have any idea how to get Linux installed on this machine, please get in touch or update that wiki page. If I can't find a way to install Linux, I will have to return the laptop to the seller and find another machine for my parents.

I wonder, is this the way Linux will be forced out of the market using UEFI and "secure boot" by making it impossible to install Linux on new Laptops?

Debian Edu / Skolelinux is an operating system based on Debian intended for use in schools. It contain a turn-key solution for the computer network provided to pupils in the primary schools. It provide both the central server, network boot servers and desktop environments with heaps of educational software. The project was founded almost 12 years ago, 2001-07-02. If you want to support the project, which is in need for cash to fund developer gatherings and other project related activity, please donate some money.

A topic that come up again and again on the Debian Edu mailing lists and elsewhere, is the question on how to transform a Debian or Ubuntu installation into a Debian Edu installation. It isn't very hard, and last week I wrote a script to replicate the steps done by the Debian Edu installer.

The script, debian-edu-bless in the debian-edu-config package, will go through these six steps and transform an existing Debian Wheezy or Ubuntu (untested) installation into a Debian Edu Workstation:

1. Add skolelinux related APT sources.
2. Create /etc/debian-edu/config with the wanted configuration.
3. Install debian-edu-install to load preseeding values and pull in our configuration.
4. Preseed debconf database with profile setup in /etc/debian-edu/config, and run tasksel to install packages according to the profile specified in the config above, overriding some of the Debian automation machinery.
5. Run debian-edu-cfengine-D installation to configure everything that could not be done using preseeding.
6. Ask for a reboot to enable all the configuration changes.

There are some steps in the Debian Edu installation that can not be replicated like this. Disk partitioning and LVM setup, for example. So this script just assume there is enough disk space to install all the needed packages.

The script was created to help a Debian Edu student working on setting up Raspberry Pi as a Debian Edu client, and using it he can take the existing Raspbian installation and transform it into a fully functioning Debian Edu Workstation (or Roaming Workstation, or whatever :).

The default setting in the script is to create a KDE Workstation. If a LXDE based Roaming workstation is wanted instead, modify the PROFILE and DESKTOP values at the top to look like this instead:

PROFILE="Roaming-Workstation"
DESKTOP="lxde"

The script could even become useful to set up Debian Edu servers in the cloud, by starting with a virtual Debian installation at some virtual hosting service and setting up all the services on first boot.

In January, I announced a new IRC channel #debian-lego, for those of us in the Debian and Linux community interested in LEGO, the marvellous construction system from Denmark. We also created a wiki page to have a place to take notes and write down our plans and hopes. And several people showed up to help. I was very happy to see the effect of my call. Since the small start, we have a debtags tag hardware::hobby:lego tag for LEGO related packages, and now count 10 packages related to LEGO and Mindstorms:

brickos	alternative OS for LEGO Mindstorms RCX. Supports development in C/C++
leocad	virtual brick CAD software
libnxt	utility library for talking to the LEGO Mindstorms NX
lnpd	daemon for LNP communication with BrickOS
nbc	compiler for LEGO Mindstorms NXT bricks
nqc	Not Quite C compiler for LEGO Mindstorms RCX
python-nxt	python driver/interface/wrapper for the Lego Mindstorms NXT robot
python-nxt-filer	simple GUI to manage files on a LEGO Mindstorms NXT
scratch	easy to use programming environment for ages 8 and up
t2n	simple command-line tool for Lego NXT

Some of these are available in Wheezy, and all but one are currently available in Jessie/testing. leocad is so far only available in experimental.

If you care about LEGO in Debian, please join us on IRC and help adding the rest of the great free software tools available on Linux for LEGO designers.

When I woke up this morning, I was very happy to see that the release announcement for Debian Wheezy was waiting in my mail box. This is a great Debian release, and I expect to move my machines at home over to it fairly soon.

The new debian release contain heaps of new stuff, and one program in particular make me very happy to see included. The Scratch program, made famous by the Teach kids code movement, is included for the first time. Alongside similar programs like kturtle and turtleart, it allow for visual programming where syntax errors can not happen, and a friendly programming environment for learning to control the computer. Scratch will also be included in the next release of Debian Edu.

And now that Wheezy is wrapped up, we can wrap up the next Debian Edu/Skolelinux release too. The first alpha release went out last week, and the next should soon follow.

Today the Isenkram package finally made it into the archive, after lingering in NEW for many months. I uploaded it to the Debian experimental suite 2013-01-27, and today it was accepted into the archive.

Isenkram is a system for suggesting to users what packages to install to work with a pluggable hardware device. The suggestion pop up when the device is plugged in. For example if a Lego Mindstorm NXT is inserted, it will suggest to install the program needed to program the NXT controller. Give it a go, and report bugs and suggestions to BTS. :)

My last bitcoin related blog post mentioned that the new bitcoin package for Debian was waiting in NEW. It was accepted by the Debian ftp-masters 2013-01-19, and have been available in unstable since then. It was automatically copied to Ubuntu, and is available in their Raring version too.

But there is a strange problem with the build that block this new version from being available on the i386 and kfreebsd-i386 architectures. For some strange reason, the autobuilders in Debian for these architectures fail to run the test suite on these architectures (BTS #672524). We are so far unable to reproduce it when building it manually, and no-one have been able to propose a fix. If you got an idea what is failing, please let us know via the BTS.

One feature that is annoying me with of the bitcoin client, because I often run low on disk space, is the fact that the client will exit if it run short on space (BTS #696715). So make sure you have enough disk space when you run it. :)

As usual, if you use bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Yesterday, I asked for testers for my prototype for making Debian better at handling pluggable hardware devices, which I set out to create earlier this month. Several valuable testers showed up, and caused me to really want to to open up the development to more people. But before I did this, I want to come up with a sensible name for this project. Today I finally decided on a new name, and I have renamed the project from hw-support-handler to this new name. In the process, I moved the source to git and made it available as a collab-maint repository in Debian. The new name? It is Isenkram. To fetch and build the latest version of the source, use

git clone http://anonscm.debian.org/git/collab-maint/isenkram.git
cd isenkram && git-buildpackage -us -uc

I have not yet adjusted all files to use the new name yet. If you want to hack on the source or improve the package, please go ahead. But please talk to me first on IRC or via email before you do major changes, to make sure we do not step on each others toes. :)

If you wonder what 'isenkram' is, it is a Norwegian word for iron stuff, typically meaning tools, nails, screws, etc. Typical hardware stuff, in other words. I've been told it is the Norwegian variant of the German word eisenkram, for those that are familiar with that word.

Update 2013-01-26: Added -us -us to build instructions, to avoid confusing people with an error from the signing process.

Update 2013-01-27: Switch to HTTP URL for the git clone argument to avoid the need for authentication.

Early this month I set out to try to improve the Debian support for pluggable hardware devices. Now my prototype is working, and it is ready for a larger audience. To test it, fetch the source from the Debian Edu subversion repository, build and install the package. You might have to log out and in again activate the autostart script.

The design is simple:

• Add desktop entry in /usr/share/autostart/ causing a program hw-support-handlerd to start when the user log in.
• This program listen for kernel events about new hardware (directly from the kernel like udev does), not using HAL dbus events as I initially did.
• When new hardware is inserted, look up the hardware modalias in the APT database, a database available via HTTP and a database available as part of the package.
• If a package is mapped to the hardware in question, the package isn't installed yet and this is the first time the hardware was plugged in, show a desktop notification suggesting to install the package or packages.
• If the user click on the 'install package now' button, ask aptdaemon via the PackageKit API to install the requrired package.
• aptdaemon ask for root password or sudo password, and install the package while showing progress information in a window.

I still need to come up with a better name for the system. Here are some screen shots showing the prototype in action. First the notification, then the password request, and finally the request to approve all the dependencies. Sorry for the Norwegian Bokmål GUI.

The prototype still need to be improved with longer timeouts, but is already useful. The database of hardware to package mappings also need more work. It is currently compatible with the Ubuntu way of storing such information in the package control file, but could be changed to use other formats instead or in addition to the current method. I've dropped the use of discover for this mapping, as the modalias approach is more flexible and easier to use on Linux as long as the Linux kernel expose its modalias strings directly.

Update 2013-01-21 16:50: Due to popular demand, here is the command required to check out and build the source: Use 'svn checkout svn://svn.debian.org/debian-edu/trunk/src/hw-support-handler/; cd hw-support-handler; debuild'. If you lack debuild, install the devscripts package.

Update 2013-01-23 12:00: The project is now renamed to Isenkram and the source moved from the Debian Edu subversion repository to a Debian collab-maint git repository. See build instructions for details.

This Christmas my trusty old laptop died. It died quietly and suddenly in bed. With a quiet whimper, it went completely quiet and black. The power button was no longer able to turn it on. It was a IBM Thinkpad X41, and the best laptop I ever had. Better than both Thinkpads X30, X31, X40, X60, X61 and X61S. Far better than the Compaq I had before that. Now I need to find a replacement. To keep going during Christmas, I moved the one year old SSD disk to my old X40 where it fitted (only one I had left that could use it), but it is not a durable solution.

My laptop needs are fairly modest. This is my wishlist from when I got a new one more than 10 years ago. It still holds true.:)

• Lightweight (around 1 kg) and small volume (preferably smaller than A4).
• Robust, it will be in my backpack every day.
• Three button mouse and a mouse pin instead of touch pad.
• Long battery life time. Preferable a week.
• Internal WIFI network card.
• Internal Twisted Pair network card.
• Some USB slots (2-3 is plenty)
• Good keyboard - similar to the Thinkpad.
• Video resolution at least 1024x768, with size around 12" (A4 paper size).
• Hardware supported by Debian Stable, ie the default kernel and X.org packages.
• Quiet, preferably fan free (or at least not using the fan most of the time).

You will notice that there are no RAM and CPU requirements in the list. The reason is simply that the specifications on laptops the last 10-15 years have been sufficient for my needs, and I have to look at other features to choose my laptop. But are there still made as robust laptops as my X41? The Thinkpad X60/X61 proved to be less robust, and Thinkpads seem to be heading in the wrong direction since Lenovo took over. But I've been told that X220 and X1 Carbon might still be useful.

Perhaps I should rethink my needs, and look for a pad with an external keyboard? I'll have to check the Linux Laptops site for well-supported laptops, or perhaps just buy one preinstalled from one of the vendors listed on the Linux Pre-loaded site.

Some times I try to figure out which Iceweasel browser plugin to install to get support for a given MIME type. Thanks to specifications done by Ubuntu and Mozilla, it is possible to do this in Debian. Unfortunately, not very many packages provide the needed meta information, Anyway, here is a small script to look up all browser plugin packages announcing ther MIME support using this specification:

#!/usr/bin/python
import sys
import apt
def pkgs_handling_mimetype(mimetype):
    cache = apt.Cache()
    cache.open(None)
    thepkgs = []
    for pkg in cache:
        version = pkg.candidate
        if version is None:
            version = pkg.installed
        if version is None:
            continue
        record = version.record
        if not record.has_key('Npp-MimeType'):
            continue
        mime_types = record['Npp-MimeType'].split(',')
        for t in mime_types:
            t = t.rstrip().strip()
            if t == mimetype:
                thepkgs.append(pkg.name)
    return thepkgs
mimetype = "audio/ogg"
if 1 < len(sys.argv):
    mimetype = sys.argv[1]
print "Browser plugin packages supporting %s:" % mimetype
for pkg in pkgs_handling_mimetype(mimetype):
    print "  %s" %pkg

It can be used like this to look up a given MIME type:

% ./apt-find-browserplug-for-mimetype 
Browser plugin packages supporting audio/ogg:
  gecko-mediaplayer
% ./apt-find-browserplug-for-mimetype application/x-shockwave-flash
Browser plugin packages supporting application/x-shockwave-flash:
  browser-plugin-gnash
%

In Ubuntu this mechanism is combined with support in the browser itself to query for plugins and propose to install the needed packages. It would be great if Debian supported such feature too. Is anyone working on adding it?

Update 2013-01-18 14:20: The Debian BTS request for icweasel support for this feature is #484010 from 2008 (and #698426 from today). Lack of manpower and wish for a different design is the reason thus feature is not yet in iceweasel from Debian.

The DEP-11 proposal to add AppStream information to the Debian archive, is a proposal to make it possible for a Desktop application to propose to the user some package to install to gain support for a given MIME type, font, library etc. that is currently missing. With such mechanism in place, it would be possible for the desktop to automatically propose and install leocad if some LDraw file is downloaded by the browser.

To get some idea about the current content of the archive, I decided to write a simple program to extract all .desktop files from the Debian archive and look up the claimed MIME support there. The result can be found on the Skolelinux FTP site. Using the collected information, it become possible to answer the question in the title. Here are the 20 most supported MIME types in Debian stable (Squeeze), testing (Wheezy) and unstable (Sid). The complete list is available from the link above.

Debian Stable:

  count MIME type
  ----- -----------------------
     32 text/plain
     30 audio/mpeg
     29 image/png
     28 image/jpeg
     27 application/ogg
     26 audio/x-mp3
     25 image/tiff
     25 image/gif
     22 image/bmp
     22 audio/x-wav
     20 audio/x-flac
     19 audio/x-mpegurl
     18 video/x-ms-asf
     18 audio/x-musepack
     18 audio/x-mpeg
     18 application/x-ogg
     17 video/mpeg
     17 audio/x-scpls
     17 audio/ogg
     16 video/x-ms-wmv

Debian Testing:

  count MIME type
  ----- -----------------------
     33 text/plain
     32 image/png
     32 image/jpeg
     29 audio/mpeg
     27 image/gif
     26 image/tiff
     26 application/ogg
     25 audio/x-mp3
     22 image/bmp
     21 audio/x-wav
     19 audio/x-mpegurl
     19 audio/x-mpeg
     18 video/mpeg
     18 audio/x-scpls
     18 audio/x-flac
     18 application/x-ogg
     17 video/x-ms-asf
     17 text/html
     17 audio/x-musepack
     16 image/x-xbitmap

Debian Unstable:

  count MIME type
  ----- -----------------------
     31 text/plain
     31 image/png
     31 image/jpeg
     29 audio/mpeg
     28 application/ogg
     27 image/gif
     26 image/tiff
     26 audio/x-mp3
     23 audio/x-wav
     22 image/bmp
     21 audio/x-flac
     20 audio/x-mpegurl
     19 audio/x-mpeg
     18 video/x-ms-asf
     18 video/mpeg
     18 audio/x-scpls
     18 application/x-ogg
     17 audio/x-musepack
     16 video/x-ms-wmv
     16 video/x-msvideo

I am told that PackageKit can provide an API to access the kind of information mentioned in DEP-11. I have not yet had time to look at it, but hope the PackageKit people in Debian are on top of these issues.

Update 2013-01-16 13:35: Updated numbers after discovering a typo in my script.

Yesterday, I wrote about the modalias values provided by the Linux kernel following my hope for better dongle support in Debian. Using this knowledge, I have tested how modalias values attached to package names can be used to map packages to hardware. This allow the system to look up and suggest relevant packages when I plug in some new hardware into my machine, and replace discover and discover-data as the database used to map hardware to packages.

I create a modaliases file with entries like the following, containing package name, kernel module name (if relevant, otherwise the package name) and globs matching the relevant hardware modalias.

Package: package-name
Modaliases: module(modaliasglob, modaliasglob, modaliasglob)

It is fairly trivial to write code to find the relevant packages for a given modalias value using this file.

An entry like this would suggest the video and picture application cheese for many USB web cameras (interface bus class 0E01):

Package: cheese
Modaliases: cheese(usb:v*p*d*dc*dsc*dp*ic0Eisc01ip*)

An entry like this would suggest the pcmciautils package when a CardBus bridge (bus class 0607) PCI device is present:

Package: pcmciautils
Modaliases: pcmciautils(pci:v*d*sv*sd*bc06sc07i*)

An entry like this would suggest the package colorhug-client when plugging in a ColorHug with USB IDs 04D8:F8DA:

Package: colorhug-client
Modaliases: colorhug-client(usb:v04D8pF8DAd*)

I believe the format is compatible with the format of the Packages file in the Debian archive. Ubuntu already uses their Packages file to store their mappings from packages to hardware.

By adding a XB-Modaliases: header in debian/control, any .deb can announce the hardware it support in a way my prototype understand. This allow those publishing packages in an APT source outside the Debian archive as well as those backporting packages to make sure the hardware mapping are included in the package meta information. I've tested such header in the pymissile package, and its modalias mapping is working as it should with my prototype. It even made it to Ubuntu Raring.

To test if it was possible to look up supported hardware using only the shell tools available in the Debian installer, I wrote a shell implementation of the lookup code. The idea is to create files for each modalias and let the shell do the matching. Please check out and try the hw-support-lookup shell script. It run without any extra dependencies and fetch the hardware mappings from the Debian archive and the subversion repository where I currently work on my prototype.

When I use it on a machine with a yubikey inserted, it suggest to install yubikey-personalization:

% ./hw-support-lookup
yubikey-personalization
%

When I run it on my Thinkpad X40 with a PCMCIA/CardBus slot, it propose to install the pcmciautils package:

% ./hw-support-lookup
pcmciautils
%

If you know of any hardware-package mapping that should be added to my database, please tell me about it.

It could be possible to generate several of the mappings between packages and hardware. One source would be to look at packages with kernel modules, ie packages with *.ko files in /lib/modules/, and extract their modalias information. Another would be to look at packages with udev rules, ie packages with files in /lib/udev/rules.d/, and extract their vendor/model information to generate a modalias matching rule. I have not tested any of these to see if it work.

If you want to help implementing a system to let us propose what packages to install when new hardware is plugged into a Debian machine, please send me an email or talk to me on #debian-devel.

While looking into how to look up Debian packages based on hardware information, to find the packages that support a given piece of hardware, I refreshed my memory regarding modalias values, and decided to document the details. Here are my findings so far, also available in the Debian Edu subversion repository:

Modalias decoded

This document try to explain what the different types of modalias values stands for. It is in part based on information from <URL: https://wiki.archlinux.org/index.php/Modalias >, <URL: http://unix.stackexchange.com/questions/26132/how-to-assign-usb-driver-to-device >, <URL: http://code.metager.de/source/history/linux/stable/scripts/mod/file2alias.c > and <URL: http://cvs.savannah.gnu.org/viewvc/dmidecode/dmidecode.c?root=dmidecode&view=markup >.

The modalias entries for a given Linux machine can be found using this shell script:

find /sys -name modalias -print0 | xargs -0 cat | sort -u

The supported modalias globs for a given kernel module can be found using modinfo:

% /sbin/modinfo psmouse | grep alias:
alias:          serio:ty05pr*id*ex*
alias:          serio:ty01pr*id*ex*
%

PCI subtype

A typical PCI entry can look like this. This is an Intel Host Bridge memory controller:

pci:v00008086d00002770sv00001028sd000001ADbc06sc00i00

This represent these values:

 v   00008086  (vendor)
 d   00002770  (device)
 sv  00001028  (subvendor)
 sd  000001AD  (subdevice)
 bc  06        (bus class)
 sc  00        (bus subclass)
 i   00        (interface)

The vendor/device values are the same values outputted from 'lspci -n' as 8086:2770. The bus class/subclass is also shown by lspci as 0600. The 0600 class is a host bridge. Other useful bus values are 0300 (VGA compatible card) and 0200 (Ethernet controller).

Not sure how to figure out the interface value, nor what it means.

USB subtype

Some typical USB entries can look like this. This is an internal USB hub in a laptop:

usb:v1D6Bp0001d0206dc09dsc00dp00ic09isc00ip00

Here is the values included in this alias:

 v    1D6B  (device vendor)
 p    0001  (device product)
 d    0206  (bcddevice)
 dc     09  (device class)
 dsc    00  (device subclass)
 dp     00  (device protocol)
 ic     09  (interface class)
 isc    00  (interface subclass)
 ip     00  (interface protocol)

The 0900 device class/subclass means hub. Some times the relevant class is in the interface class section. For a simple USB web camera, these alias entries show up:

usb:v0AC8p3420d5000dcEFdsc02dp01ic01isc01ip00
usb:v0AC8p3420d5000dcEFdsc02dp01ic01isc02ip00
usb:v0AC8p3420d5000dcEFdsc02dp01ic0Eisc01ip00
usb:v0AC8p3420d5000dcEFdsc02dp01ic0Eisc02ip00

Interface class 0E01 is video control, 0E02 is video streaming (aka camera), 0101 is audio control device and 0102 is audio streaming (aka microphone). Thus this is a camera with microphone included.

ACPI subtype

The ACPI type is used for several non-PCI/USB stuff. This is an IR receiver in a Thinkpad X40:

acpi:IBM0071:PNP0511:

The values between the colons are IDs.

DMI subtype

The DMI table contain lots of information about the computer case and model. This is an entry for a IBM Thinkpad X40, fetched from /sys/devices/virtual/dmi/id/modalias:

dmi:bvnIBM:bvr1UETB6WW(1.66):bd06/15/2005:svnIBM:pn2371H4G:pvrThinkPadX40:rvnIBM:rn2371H4G:rvrNotAvailable:cvnIBM:ct10:cvrNotAvailable:

The values present are

 bvn  IBM            (BIOS vendor)
 bvr  1UETB6WW(1.66) (BIOS version)
 bd   06/15/2005     (BIOS date)
 svn  IBM            (system vendor)
 pn   2371H4G        (product name)
 pvr  ThinkPadX40    (product version)
 rvn  IBM            (board vendor)
 rn   2371H4G        (board name)
 rvr  NotAvailable   (board version)
 cvn  IBM            (chassis vendor)
 ct   10             (chassis type)
 cvr  NotAvailable   (chassis version)

The chassis type 10 is Notebook. Other interesting values can be found in the dmidecode source:

  3 Desktop
  4 Low Profile Desktop
  5 Pizza Box
  6 Mini Tower
  7 Tower
  8 Portable
  9 Laptop
 10 Notebook
 11 Hand Held
 12 Docking Station
 13 All In One
 14 Sub Notebook
 15 Space-saving
 16 Lunch Box
 17 Main Server Chassis
 18 Expansion Chassis
 19 Sub Chassis
 20 Bus Expansion Chassis
 21 Peripheral Chassis
 22 RAID Chassis
 23 Rack Mount Chassis
 24 Sealed-case PC
 25 Multi-system
 26 CompactPCI
 27 AdvancedTCA
 28 Blade
 29 Blade Enclosing

The chassis type values are not always accurately set in the DMI table. For example my home server is a tower, but the DMI modalias claim it is a desktop.

SerIO subtype

This type is used for PS/2 mouse plugs. One example is from my test machine:

serio:ty01pr00id00ex00

The values present are

  ty  01  (type)
  pr  00  (prototype)
  id  00  (id)
  ex  00  (extra)

This type is supported by the psmouse driver. I am not sure what the valid values are.

Other subtypes

There are heaps of other modalias subtypes according to file2alias.c. There is the rest of the list from that source: amba, ap, bcma, ccw, css, eisa, hid, i2c, ieee1394, input, ipack, isapnp, mdio, of, parisc, pcmcia, platform, scsi, sdio, spi, ssb, vio, virtio, vmbus, x86cpu and zorro. I did not spend time documenting all of these, as they do not seem relevant for my intended use with mapping hardware to packages when new stuff is inserted during run time.

Looking up kernel modules using modalias values

To check which kernel modules provide support for a given modalias, one can use the following shell script:

  for id in $(find /sys -name modalias -print0 | xargs -0 cat | sort -u); do \
    echo "$id" ; \
    /sbin/modprobe --show-depends "$id"|sed 's/^/  /' ; \
  done

The output can look like this (only the first few entries as the list is very long on my test machine):

  acpi:ACPI0003:
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/acpi/ac.ko 
  acpi:device:
  FATAL: Module acpi:device: not found.
  acpi:IBM0068:
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/char/nvram.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/leds/led-class.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/net/rfkill/rfkill.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/platform/x86/thinkpad_acpi.ko 
  acpi:IBM0071:PNP0511:
    insmod /lib/modules/2.6.32-5-686/kernel/lib/crc-ccitt.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/net/irda/irda.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/net/irda/nsc-ircc.ko 
  [...]

If you want to help implementing a system to let us propose what packages to install when new hardware is plugged into a Debian machine, please send me an email or talk to me on #debian-devel.

Update 2013-01-15: Rewrite "cat $(find ...)" to "find ... -print0 | xargs -0 cat" to make sure it handle directories in /sys/ with space in them.

As part of my investigation on how to improve the support in Debian for hardware dongles, I dug up my old Mark and Spencer USB Rocket Launcher and updated the Debian package pymissile to make sure udev will fix the device permissions when it is plugged in. I also added a "Modaliases" header to test it in the Debian archive and hopefully make the package be proposed by jockey in Ubuntu when a user plug in his rocket launcher. In the process I moved the source to a git repository under collab-maint, to make it easier for any DD to contribute. Upstream is not very active, but the software still work for me even after five years of relative silence. The new git repository is not listed in the uploaded package yet, because I want to test the other changes a bit more before I upload the new version. If you want to check out the new version with a .desktop file included, visit the gitweb view or use "git clone git://anonscm.debian.org/collab-maint/pymissile.git".

One thing that annoys me with Debian and Linux distributions in general, is that there is a great package management system with the ability to automatically install software packages by downloading them from the distribution mirrors, but no way to get it to automatically install the packages I need to use the hardware I plug into my machine. Even if the package to use it is easily available from the Linux distribution. When I plug in a LEGO Mindstorms NXT, it could suggest to automatically install the python-nxt, nbc and t2n packages I need to talk to it. When I plug in a Yubikey, it could propose the yubikey-personalization package. The information required to do this is available, but no-one have pulled all the pieces together.

Some years ago, I proposed to use the discover subsystem to implement this. The idea is fairly simple:

• Add a desktop entry in /usr/share/autostart/ pointing to a program starting when a user log in.
• Set this program up to listen for kernel events emitted when new hardware is inserted into the computer.
• When new hardware is inserted, look up the hardware ID in a database mapping to packages, and take note of any non-installed packages.
• Show a message to the user proposing to install the discovered package, and make it easy to install it.

I am not sure what the best way to implement this is, but my initial idea was to use dbus events to discover new hardware, the discover database to find packages and PackageKit to install packages.

Yesterday, I found time to try to implement this idea, and the draft package is now checked into the Debian Edu subversion repository. In the process, I updated the discover-data package to map the USB ids of LEGO Mindstorms and Yubikey devices to the relevant packages in Debian, and uploaded a new version 2.2013.01.09 to unstable. I also discovered that the current discover package in Debian no longer discovered any USB devices, because /proc/bus/usb/devices is no longer present. I ported it to use libusb as a fall back option to get it working. The fixed package version 2.1.2-6 is now in experimental (didn't upload it to unstable because of the freeze).

With this prototype in place, I can insert my Yubikey, and get this desktop notification to show up (only once, the first time it is inserted):

For this prototype to be really useful, some way to automatically install the proposed packages by pressing the "Please install program(s)" button should to be implemented.

If this idea seem useful to you, and you want to help make it happen, please help me update the discover-data database with mappings from hardware to Debian packages. Check if 'discover-pkginstall -l' list the package you would like to have installed when a given hardware device is inserted into your computer, and report bugs using reportbug if it isn't. Or, if you know of a better way to provide such mapping, please let me know.

This prototype need more work, and there are several questions that should be considered before it is ready for production use. Is dbus the correct way to detect new hardware? At the moment I look for HAL dbus events on the system bus, because that is the events I could see on my Debian Squeeze KDE desktop. Are there better events to use? How should the user be notified? Is the desktop notification mechanism the best option, or should the background daemon raise a popup instead? How should packages be installed? When should they not be installed?

If you want to help getting such feature implemented in Debian, please send me an email. :)

During Christmas, I have worked a bit on the Debian support for LEGO Mindstorm NXT. My son and I have played a bit with my NXT set, and I discovered I had to build all the tools myself because none were already in Debian Squeeze. If Debian support for LEGO is something you care about, please join me on the IRC channel #debian-lego (server irc.debian.org). There is a lot that could be done to improve the Debian support for LEGO designers. For example both CAD software and Mindstorm compilers are missing. :)

Update 2012-01-03: A project page including links to Lego related packages is now available.

Let me start by wishing you all marry Christmas and a happy new year! I hope next year will prove to be a good year.

Bitcoin, the digital decentralised "currency" that allow people to transfer bitcoins between each other with minimal overhead, is a very interesting experiment. And as I wrote a few days ago, the bitcoin situation in Debian is about to improve a bit. The new debian source package (version 0.7.2-2) was uploaded yesterday, and is waiting in the NEW queue for one of the ftpmasters to approve the new bitcoin-qt package name.

And thanks to the great work of Jonas and the rest of the bitcoin team in Debian, you can easily test the package in Debian Squeeze using the following steps to get a set of working packages:

git clone git://git.debian.org/git/collab-maint/bitcoin
cd bitcoin
DEB_MAINTAINER_MODE=1 DEB_BUILD_OPTIONS=noupnp fakeroot debian/rules clean
DEB_BUILD_OPTIONS=noupnp git-buildpackage --git-ignore-new

You might have to install some build dependencies as well. The list of commands should give you two packages, bitcoind and bitcoin-qt, ready for use in a Squeeze environment. Note that the client will download the complete set of bitcoin "blocks", which need around 5.6 GiB of data on my machine at the moment. Make sure your ~/.bitcoin/ directory have lots of spare room if you want to download all the blocks. The client will warn if the disk is getting full, so there is not really a problem if you got too little room, but you will not be able to get all the features out of the client.

As usual, if you use bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

It has been a while since I wrote about bitcoin, the decentralised peer-to-peer based crypto-currency, and the reason is simply that I have been busy elsewhere. But two days ago, I started looking at the state of bitcoin in Debian again to try to recover my old bitcoin wallet. The package is now maintained by a team of people, and the grunt work had already been done by this team. We owe a huge thank you to all these team members. :) But I was sad to discover that the bitcoin client is missing in Wheezy. It is only available in Sid (and an outdated client from backports). The client had several RC bugs registered in BTS blocking it from entering testing. To try to help the team and improve the situation, I spent some time providing patches and triaging the bug reports. I also had a look at the bitcoin package available from Matt Corallo in a PPA for Ubuntu, and moved the useful pieces from that version into the Debian package.

After checking with the main package maintainer Jonas Smedegaard on IRC, I pushed several patches into the collab-maint git repository to improve the package. It now contains fixes for the RC issues (not from me, but fixed by Scott Howard), build rules for a Qt GUI client package, konqueror support for the bitcoin: URI and bash completion setup. As I work on Debian Squeeze, I also created a patch to backport the latest version. Jonas is going to look at it and try to integrate it into the git repository before uploading a new version to unstable.

I would very much like bitcoin to succeed, to get rid of the centralized control currently exercised in the monetary system. I find it completely unacceptable that the USA government is collecting transaction data for almost all international money transfers (most are done in USD and transaction logs shipped to the spooks), and that the major credit card companies can block legal money transactions to Wikileaks. But for bitcoin to succeed, more people need to use bitcoins, and more people need to accept bitcoins when they sell products and services. Improving the bitcoin support in Debian is a small step in the right direction, but not enough. Unfortunately the user experience when browsing the web and wanting to pay with bitcoin is still not very good. The bitcoin: URI is a step in the right direction, but need to work in most or every browser in use. Also the bitcoin-qt client is too heavy to fire up to do a quick transaction. I believe there are other clients available, but have not tested them.

My experiment with bitcoins showed that at least some of my readers use bitcoin. I received 20.15 BTC so far on the address I provided in my blog two years ago, as can be seen on the blockexplorer service. Thank you everyone for your donation. The blockexplorer service demonstrates quite well that bitcoin is not quite anonymous and untracked. :) I wonder if the number of users have gone up since then. If you use bitcoin and want to show your support of my activity, please send Bitcoin donations to the same address as last time, 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

As I mentioned this summer, I have created a Computer Science song book a few years ago, and today I finally found time to create a public Gitorious repository for the project.

If you want to help out, please clone the source and submit patches to the HTML version. To generate the PDF and PostScript version, please use prince XML, or let me know about a useful free software processor capable of creating a good looking PDF from the HTML.

Want to sing? You can still find the song book in HTML, PDF and PostScript formats at Petter's Computer Science Songbook.

I dag fyller Debian-prosjektet 19 år. Jeg har fulgt det de siste 12 årene, og er veldig glad for å kunne si gratulerer med dagen, Debian!

Many years ago, while studying Computer Science at the University of Tromsø, I started collecting computer related songs for use at parties. The original version was written in LaTeX, but a few years ago I got help from Håkon W. Lie, one of the inventors of W3C CSS, to convert it to HTML while keeping the ability to create a nice book in PDF format. I have not had time to maintain the book for a while now, and guess I should put it up on some public version control repository where others can help me extend and update the book. If anyone is volunteering to help me with this, send me an email. Also let me know if there are songs missing in my book.

I have not mentioned the book on my blog so far, and it occured to me today that I really should let all my readers share the joys of singing out load about programming, computers and computer networks. Especially now that Debconf 12 is about to start (and I am not going). Want to sing? Check out Petter's Computer Science Songbook.

At work we have heaps of servers. I believe the total count is around 1000 at the moment. To be able to get help from the vendors when something go wrong, we want to keep the firmware on the servers up to date. If the firmware isn't the latest and greatest, the vendors typically refuse to start debugging any problems until the firmware is upgraded. So before every reboot, we want to upgrade the firmware, and we would really like everyone handling servers at the university to do this themselves when they plan to reboot a machine. For that to happen we at the unix server admin group need to provide the tools to do so.

To make firmware upgrading easier, I am working on a script to fetch and install the latest firmware for the servers we got. Most of our hardware are from Dell and HP, so I have focused on these servers so far. This blog post is about the Dell part.

On the Dell FTP site I was lucky enough to find an XML file with firmware information for all 11th generation servers, listing which firmware should be used on a given model and where on the FTP site I can find it. Using a simple perl XML parser I can then download the shell scripts Dell provides to do firmware upgrades from within Linux and reboot when all the firmware is primed and ready to be activated on the first reboot.

This is the Dell related fragment of the perl code I am working on. Are there anyone working on similar tools for firmware upgrading all servers at a site? Please get in touch and lets share resources.

#!/usr/bin/perl
use strict;
use warnings;
use File::Temp qw(tempdir);
BEGIN {
    # Install needed RHEL packages if missing
    my %rhelmodules = (
        'XML::Simple' => 'perl-XML-Simple',
        );
    for my $module (keys %rhelmodules) {
        eval "use $module;";
        if ($@) {
            my $pkg = $rhelmodules{$module};
            system("yum install -y $pkg");
            eval "use $module;";
        }
    }
}
my $errorsto = 'pere@hungry.com';

upgrade_dell();

exit 0;

sub run_firmware_script {
    my ($opts, $script) = @_;
    unless ($script) {
        print STDERR "fail: missing script name\n";
        exit 1
    }
    print STDERR "Running $script\n\n";

    if (0 == system("sh $script $opts")) { # FIXME correct exit code handling
        print STDERR "success: firmware script ran succcessfully\n";
    } else {
        print STDERR "fail: firmware script returned error\n";
    }
}

sub run_firmware_scripts {
    my ($opts, @dirs) = @_;
    # Run firmware packages
    for my $dir (@dirs) {
        print STDERR "info: Running scripts in $dir\n";
        opendir(my $dh, $dir) or die "Unable to open directory $dir: $!";
        while (my $s = readdir $dh) {
            next if $s =~ m/^\.\.?/;
            run_firmware_script($opts, "$dir/$s");
        }
        closedir $dh;
    }
}

sub download {
    my $url = shift;
    print STDERR "info: Downloading $url\n";
    system("wget --quiet \"$url\"");
}

sub upgrade_dell {
    my @dirs;
    my $product = `dmidecode -s system-product-name`;
    chomp $product;

    if ($product =~ m/PowerEdge/) {

        # on RHEL, these pacakges are needed by the firwmare upgrade scripts
        system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail');

        my $tmpdir = tempdir(
            CLEANUP => 1
            );
        chdir($tmpdir);
        fetch_dell_fw('catalog/Catalog.xml.gz');
        system('gunzip Catalog.xml.gz');
        my @paths = fetch_dell_fw_list('Catalog.xml');
        # -q is quiet, disabling interactivity and reducing console output
        my $fwopts = "-q";
        if (@paths) {
            for my $url (@paths) {
                fetch_dell_fw($url);
            }
            run_firmware_scripts($fwopts, $tmpdir);
        } else {
            print STDERR "error: Unsupported Dell model '$product'.\n";
            print STDERR "error: Please report to $errorsto.\n";
        }
        chdir('/');
    } else {
        print STDERR "error: Unsupported Dell model '$product'.\n";
        print STDERR "error: Please report to $errorsto.\n";
    }
}

sub fetch_dell_fw {
    my $path = shift;
    my $url = "ftp://ftp.us.dell.com/$path";
    download($url);
}

# Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which
# firmware packages to download from Dell.  Only work for Linux
# machines and 11th generation Dell servers.
sub fetch_dell_fw_list {
    my $filename = shift;

    my $product = `dmidecode -s system-product-name`;
    chomp $product;
    my ($mybrand, $mymodel) = split(/\s+/, $product);

    print STDERR "Finding firmware bundles for $mybrand $mymodel\n";

    my $xml = XMLin($filename);
    my @paths;
    for my $bundle (@{$xml->{SoftwareBundle}}) {
        my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content};
        my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content};
        my $oscode;
        if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) {
            $oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode};
        } else {
            $oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode};
        }
        if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode)
        {
            @paths = map { $_->{path} } @{$bundle->{Contents}->{Package}};
        }
    }
    for my $component (@{$xml->{SoftwareComponent}}) {
        my $componenttype = $component->{ComponentType}->{value};

        # Drop application packages, only firmware and BIOS
        next if 'APAC' eq $componenttype;

        my $cpath = $component->{path};
        for my $path (@paths) {
            if ($cpath =~ m%/$path$%) {
                push(@paths, $cpath);
            }
        }
    }
    return @paths;
}

The code is only tested on RedHat Enterprise Linux, but I suspect it could work on other platforms with some tweaking. Anyone know a index like Catalog.xml is available from HP for HP servers? At the moment I maintain a similar list manually and it is quickly getting outdated.

Wouter Verhelst have some interesting comments and opinions on my blog post on the need to clean up /etc/rcS.d/ in Debian and my blog post about the default KDE desktop in Debian. I only have time to address one small piece of his comment now, and though it best to address the misunderstanding he bring forward:

Currently, a system admin has four options: [...] boot to a single-user system (by adding 'single' to the kernel command line; this runs rcS and rc1 scripts)

This make me believe Wouter believe booting into single user mode and booting into runlevel 1 is the same. I am not surprised he believe this, because it would make sense and is a quite sensible thing to believe. But because the boot in Debian is slightly broken, runlevel 1 do not work properly and it isn't the same as single user mode. I'll try to explain what is actually happing, but it is a bit hard to explain.

Single user mode is defined like this in /etc/inittab: "~~:S:wait:/sbin/sulogin". This means the only thing that is executed in single user mode is sulogin. Single user mode is a boot state "between" the runlevels, and when booting into single user mode, only the scripts in /etc/rcS.d/ are executed before the init process enters the single user state. When switching to runlevel 1, the state is in fact not ending in runlevel 1, but it passes through runlevel 1 and end up in the single user mode (see /etc/rc1.d/S03single, which runs "init -t1 S" to switch to single user mode at the end of runlevel 1. It is confusing that the 'S' (single user) init mode is not the mode enabled by /etc/rcS.d/ (which is more like the initial boot mode).

This summary might make it clearer. When booting for the first time into single user mode, the following commands are executed: "/etc/init.d/rc S; /sbin/sulogin". When booting into runlevel 1, the following commands are executed: "/etc/init.d/rc S; /etc/init.d/rc 1; /sbin/sulogin". A problem show up when trying to continue after visiting single user mode. Not all services are started again as they should, causing the machine to end up in an unpredicatble state. This is why Debian admins recommend rebooting after visiting single user mode.

A similar problem with runlevel 1 is caused by the amount of scripts executed from /etc/rcS.d/. When switching from say runlevel 2 to runlevel 1, the services started from /etc/rcS.d/ are not properly stopped when passing through the scripts in /etc/rc1.d/, and not started again when switching away from runlevel 1 to the runlevels 2-5. I believe the problem is best fixed by moving all the scripts out of /etc/rcS.d/ that are not required to get a functioning single user mode during boot.

I have spent several years investigating the Debian boot system, and discovered this problem a few years ago. I suspect it originates from when sysvinit was introduced into Debian, a long time ago.

In the Debian boot system, several packages include scripts that are started from /etc/rcS.d/. In fact, there is a bite more of them than make sense, and this causes a few problems. What kind of problems, you might ask. There are at least two problems. The first is that it is not possible to recover a machine after switching to runlevel 1. One need to actually reboot to get the machine back to the expected state. The other is that single user boot will sometimes run into problems because some of the subsystems are activated before the root login is presented, causing problems when trying to recover a machine from a problem in that subsystem. A minor additional point is that moving more scripts out of rcS.d/ and into the other rc#.d/ directories will increase the amount of scripts that can run in parallel during boot, and thus decrease the boot time.

So, which scripts should start from rcS.d/. In short, only the scripts that _have_ to execute before the root login prompt is presented during a single user boot should go there. Everything else should go into the numeric runlevels. This means things like lm-sensors, fuse and x11-common should not run from rcS.d, but from the numeric runlevels. Today in Debian, there are around 115 init.d scripts that are started from rcS.d/, and most of them should be moved out. Do your package have one of them? Please help us make single user and runlevel 1 better by moving it.

Scripts setting up the screen, keyboard, system partitions etc. should still be started from rcS.d/, but there is for example no need to have the network enabled before the single user login prompt is presented.

As always, things are not so easy to fix as they sound. To keep Debian systems working while scripts migrate and during upgrades, the scripts need to be moved from rcS.d/ to rc2.d/ in reverse dependency order, ie the scripts that nothing in rcS.d/ depend on can be moved, and the next ones can only be moved when their dependencies have been moved first. This migration must be done sequentially while we ensure that the package system upgrade packages in the right order to keep the system state correct. This will require some coordination when it comes to network related packages, but most of the packages with scripts that should migrate do not have anything in rcS.d/ depending on them. Some packages have already been updated, like the sudo package, while others are still left to do. I wish I had time to work on this myself, but real live constrains make it unlikely that I will find time to push this forward.

While at Debconf11, I have several times during discussions mentioned the issues I believe should be improved in Debian for its desktop to be useful for more people. The use case for this is my parents, which are currently running Kubuntu which solve the issues.

I suspect these four missing features are not very hard to implement. After all, they are present in Ubuntu, so if we wanted to do this in Debian we would have a source.

1. Simple GUI based upgrade of packages. When there are new packages available for upgrades, a icon in the KDE status bar indicate this, and clicking on it will activate the simple upgrade tool to handle it. I have no problem guiding both of my parents through the process over the phone. If a kernel reboot is required, this too is indicated by the status bars and the upgrade tool. Last time I checked, nothing with the same features was working in KDE in Debian.
2. Simple handling of missing Firefox browser plugins. When the browser encounter a MIME type it do not currently have a handler for, it will ask the user if the system should search for a package that would add support for this MIME type, and if the user say yes, the APT sources will be searched for packages advertising the MIME type in their control file (visible in the Packages file in the APT archive). If one or more packages are found, it is a simple click of the mouse to add support for the missing mime type. If the package require the user to accept some non-free license, this is explained to the user. The entire process make it more clear to the user why something do not work in the browser, and make the chances higher for the user to blame the web page authors and not the browser for any missing features.
3. Simple handling of missing multimedia codec/format handlers. When the media players encounter a format or codec it is not supporting, a dialog pop up asking the user if the system should search for a package that would add support for it. This happen with things like MP3, Windows Media or H.264. The selection and installation procedure is very similar to the Firefox browser plugin handling. This is as far as I know implemented using a gstreamer hook. The end result is that the user easily get access to the codecs that are present from the APT archives available, while explaining more on why a given format is unsupported by Ubuntu.
4. Better browser handling of some MIME types. When displaying a text/plain file in my Debian browser, it will propose to start emacs to show it. If I remember correctly, when doing the same in Kunbutu it show the file as a text file in the browser. At least I know Opera will show text files within the browser. I much prefer the latter behaviour.

There are other nice features as well, like the simplified suite upgrader, but given that I am the one mostly doing the dist-upgrade, it do not matter much.

I really hope we could get these features in place for the next Debian release. It would require the coordinated effort of several maintainers, but would make the end user experience a lot better.

The Norwegian FiksGataMi site is build on Debian/Squeeze, and this platform was chosen because I am most familiar with Debian (being a Debian Developer for around 10 years) because it is the latest stable Debian release which should get security support for a few years.

The web service is written in Perl, and depend on some perl modules that are missing in Debian at the moment. It would be great if these modules were added to the Debian archive, allowing anyone to set up their own FixMyStreet clone in their own country using only Debian packages. The list of modules missing in Debian/Squeeze isn't very long, and I hope the perl group will find time to package the 12 modules Catalyst::Plugin::SmartURI, Catalyst::Plugin::Unicode::Encoding, Catalyst::View::TT, Devel::Hide, Sort::Key, Statistics::Distributions, Template::Plugin::Comma, Template::Plugin::DateTime::Format, Term::Size::Any, Term::Size::Perl, URI::SmartURI and Web::Scraper to make the maintenance of FixMyStreet easier in the future.

Thanks to the great tools in Debian, getting the missing modules installed on my server was a simple call to 'cpan2deb Module::Name' and 'dpkg -i' to install the resulting package. But this leave me with the responsibility of tracking security problems, which I really do not have time for.

Here is a small update for my English readers. Most of my blog posts have been in Norwegian the last few weeks, so here is a short update in English.

The kids still keep me too busy to get much free software work done, but I did manage to organise a project to get a Norwegian port of the British service FixMyStreet up and running, and it has been running for a month now. The entire project has been organised by me and two others. Around Christmas we gathered sponsors to fund the development work. In January I drafted a contract with mySociety on what to develop, and in February the development took place. Most of it involved converting the source to use GPS coordinates instead of British easting/northing, and the resulting code should be a lot easier to get running in any country by now. The Norwegian FiksGataMi is using OpenStreetmap as the map source and the source for administrative borders in Norway, and support for this had to be added/fixed.

The Norwegian version went live March 3th, and we spent the weekend polishing the system before we announced it March 7th. The system is running on a KVM instance of Debian/Squeeze, and has seen almost 3000 problem reports in a few weeks. Soon we hope to announce the Android and iPhone versions making it even easier to report problems with the public infrastructure.

Perhaps something to consider for those of you in countries without such service?

The last few days I have looked at ways to track open security issues here at my work with the University of Oslo. My idea is that it should be possible to use the information about security issues available on the Internet, and check our locally maintained/distributed software against this information. It should allow us to verify that no known security issues are forgotten. The CVE database listing vulnerabilities seem like a great central point, and by using the package lists from Debian mapped to CVEs provided by the testing security team, I believed it should be possible to figure out which security holes were present in our free software collection.

After reading up on the topic, it became obvious that the first building block is to be able to name software packages in a unique and consistent way across data sources. I considered several ways to do this, for example coming up with my own naming scheme like using URLs to project home pages or URLs to the Freshmeat entries, or using some existing naming scheme. And it seem like I am not the first one to come across this problem, as MITRE already proposed and implemented a solution. Enter the Common Platform Enumeration dictionary, a vocabulary for referring to software, hardware and other platform components. The CPE ids are mapped to CVEs in the National Vulnerability Database, allowing me to look up know security issues for any CPE name. With this in place, all I need to do is to locate the CPE id for the software packages we use at the university. This is fairly trivial (I google for 'cve cpe $package' and check the NVD entry if a CVE for the package exist).

To give you an example. The GNU gzip source package have the CPE name cpe:/a:gnu:gzip. If the old version 1.3.3 was the package to check out, one could look up cpe:/a:gnu:gzip:1.3.3 in NVD and get a list of 6 security holes with public CVE entries. The most recent one is CVE-2010-0001, and at the bottom of the NVD page for this vulnerability the complete list of affected versions is provided.

The NVD database of CVEs is also available as a XML dump, allowing for offline processing of issues. Using this dump, I've written a small script taking a list of CPEs as input and list all CVEs affecting the packages represented by these CPEs. One give it CPEs with version numbers as specified above and get a list of open security issues out.

Of course for this approach to be useful, the quality of the NVD information need to be high. For that to happen, I believe as many as possible need to use and contribute to the NVD database. I notice RHEL is providing a map from CVE to CPE, indicating that they are using the CPE information. I'm not aware of Debian and Ubuntu doing the same.

To get an idea about the quality for free software, I spent some time making it possible to compare the CVE database from Debian with the CVE database in NVD. The result look fairly good, but there are some inconsistencies in NVD (same software package having several CPEs), and some inaccuracies (NVD not mentioning buggy packages that Debian believe are affected by a CVE). Hope to find time to improve the quality of NVD, but that require being able to get in touch with someone maintaining it. So far my three emails with questions and corrections have not seen any reply, but I hope contact can be established soon.

An interesting application for CPEs is cross platform package mapping. It would be useful to know which packages in for example RHEL, OpenSuSe and Mandriva are missing from Debian and Ubuntu, and this would be trivial if all linux distributions provided CPE entries for their packages.

In the discover-data package in Debian, there is a script to report useful information about the running hardware for use when people report missing information. One part of this script that I find very useful when debugging hardware problems, is the part mapping loaded kernel module to the PCI device it claims. It allow me to quickly see if the kernel module I expect is driving the hardware I am struggling with. To see the output, make sure discover-data is installed and run /usr/share/bug/discover-data 3>&1. The relevant output on one of my machines like this:

loaded modules:
10de:03eb i2c_nforce2
10de:03f1 ohci_hcd
10de:03f2 ehci_hcd
10de:03f0 snd_hda_intel
10de:03ec pata_amd
10de:03f6 sata_nv
1022:1103 k8temp
109e:036e bttv
109e:0878 snd_bt87x
11ab:4364 sky2

The code in question look like this, slightly modified for readability and to drop the output to file descriptor 3:

if [ -d /sys/bus/pci/devices/ ] ; then
    echo loaded pci modules:
    (
        cd /sys/bus/pci/devices/
        for address in * ; do
            if [ -d "$address/driver/module" ] ; then
                module=`cd $address/driver/module ; pwd -P | xargs basename`
                if grep -q "^$module " /proc/modules ; then
                    address=$(echo $address |sed s/0000://)
                    id=`lspci -n -s $address | tail -n 1 | awk '{print $3}'`
                    echo "$id $module"
                fi
            fi
        done
    )
    echo
fi

Similar code could be used to extract USB device module mappings:

if [ -d /sys/bus/usb/devices/ ] ; then
    echo loaded usb modules:
    (
        cd /sys/bus/usb/devices/
        for address in * ; do
            if [ -d "$address/driver/module" ] ; then
                module=`cd $address/driver/module ; pwd -P | xargs basename`
                if grep -q "^$module " /proc/modules ; then
                    address=$(echo $address |sed s/0000://)
                    id=$(lsusb -s $address | tail -n 1 | awk '{print $6}')
                    if [ "$id" ] ; then
                        echo "$id $module"
                    fi
                fi
            fi
        done
    )
    echo
fi

This might perhaps be something to include in other tools as well.

The last few days I have spent at work here at the University of Oslo testing if the new batch of computers will work with Linux. Every year for the last few years the university have organised shared bid of a few thousand computers, and this year HP won the bid. Two different desktops and five different laptops are on the list this year. We in the UNIX group want to know which one of these computers work well with RHEL and Ubuntu, the two Linux distributions we currently handle at the university.

My test method is simple, and I share it here to get feedback and perhaps inspire others to test hardware as well. To test, I PXE install the OS version of choice, and log in as my normal user and run a few applications and plug in selected pieces of hardware. When something fail, I make a note about this in the test matrix and move on. If I have some spare time I try to report the bug to the OS vendor, but as I only have the machines for a short time, I rarely have the time to do this for all the problems I find.

Anyway, to get to the point of this post. Here is the simple tests I perform on a new model.

• Is PXE installation working? I'm testing with RHEL6, Ubuntu Lucid and Ubuntu Maverik at the moment. If I feel like it, I also test with RHEL5 and Debian Edu/Squeeze.
• Is X.org working? If the graphical login screen show up after installation, X.org is working.
• Is hardware accelerated OpenGL working? Running glxgears (in package mesa-utils on Ubuntu) and writing down the frames per second reported by the program.
• Is sound working? With Gnome and KDE, a sound is played when logging in, and if I can hear this the test is successful. If there are several audio exits on the machine, I try them all and check if the Gnome/KDE audio mixer can control where to send the sound. I normally test this by playing a HTML5 video in Firefox/Iceweasel.
• Is the USB subsystem working? I test this by plugging in a USB memory stick and see if Gnome/KDE notices this.
• Is the CD/DVD player working? I test this by inserting any CD/DVD I have lying around, and see if Gnome/KDE notices this.
• Is any built in camera working? Test using cheese, and see if a picture from the v4l device show up.
• Is bluetooth working? Use the Gnome/KDE browsing tool to see if any bluetooth devices are discovered. In my office, I normally see a few.
• For laptops, is the SD or Compaq Flash reader working. I have memory modules lying around, and stick them in and see if Gnome/KDE notice this.
• For laptops, is suspend/hibernate working? I'm testing if the special button work, and if the laptop continue to work after resume.
• For laptops, is the extra buttons working, like audio level, adjusting background light, switching on/off external video output, switching on/off wifi, bluetooth, etc? The set of buttons differ from laptop to laptop, so I just write down which are working and which are not.
• Some laptops have smart card readers, finger print readers, acceleration sensors etc. I rarely test these, as I do not know how to quickly test if they are working or not, so I only document their existence.

By now I suspect you are really curious what the test results are for the HP machines I am testing. I'm not done yet, so I will report the test results later. For now I can report that HP 8100 Elite work fine, and hibernation fail with HP EliteBook 8440p on Ubuntu Lucid, and audio fail on RHEL6. Ubuntu Maverik worked with 8440p. As you can see, I have most machines left to test. One interesting observation is that Ubuntu Lucid has almost twice the frame rate than RHEL6 with glxgears. No idea why.

As I continue to explore BitCoin, I've starting to wonder what properties the system have, and how it will be affected by laws and regulations here in Norway. Here are some random notes.

One interesting thing to note is that since the transactions are verified using a peer to peer network, all details about a transaction is known to everyone. This means that if a BitCoin address has been published like I did with mine in my initial post about BitCoin, it is possible for everyone to see how many BitCoins have been transfered to that address. There is even a web service to look at the details for all transactions. There I can see that my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b have received 16.06 Bitcoin, the 1LfdGnGuWkpSJgbQySxxCWhv8MHqvwst3 address of Simon Phipps have received 181.97 BitCoin and the address 1MCwBbhNGp5hRm5rC1Aims2YFRe2SXPYKt of EFF have received 2447.38 BitCoins so far. Thank you to each and every one of you that donated bitcoins to support my activity. The fact that anyone can see how much money was transfered to a given address make it more obvious why the BitCoin community recommend to generate and hand out a new address for each transaction. I'm told there is no way to track which addresses belong to a given person or organisation without the person or organisation revealing it themselves, as Simon, EFF and I have done.

In Norway, and in most other countries, there are laws and regulations limiting how much money one can transfer across the border without declaring it. There are money laundering, tax and accounting laws and regulations I would expect to apply to the use of BitCoin. If the Skolelinux foundation (SLX Debian Labs) were to accept donations in BitCoin in addition to normal bank transfers like EFF is doing, how should this be accounted? Given that it is impossible to know if money can cross the border or not, should everything or nothing be declared? What exchange rate should be used when calculating taxes? Would receivers have to pay income tax if the foundation were to pay Skolelinux contributors in BitCoin? I have no idea, but it would be interesting to know.

For a currency to be useful and successful, it must be trusted and accepted by a lot of users. It must be possible to get easy access to the currency (as a wage or using currency exchanges), and it must be easy to spend it. At the moment BitCoin seem fairly easy to get access to, but there are very few places to spend it. I am not really a regular user of any of the vendor types currently accepting BitCoin, so I wonder when my kind of shop would start accepting BitCoins. I would like to buy electronics, travels and subway tickets, not herbs and books. :) The currency is young, and this will improve over time if it become popular, but I suspect regular banks will start to lobby to get BitCoin declared illegal if it become popular. I'm sure they will claim it is helping fund terrorism and money laundering (which probably would be true, as is any currency in existence), but I believe the problems should be solved elsewhere and not by blaming currencies.

The process of creating new BitCoins is called mining, and it is CPU intensive process that depend on a bit of luck as well (as one is competing against all the other miners currently spending CPU cycles to see which one get the next lump of cash). The "winner" get 50 BitCoin when this happen. Yesterday I came across the obvious way to join forces to increase ones changes of getting at least some coins, by coordinating the work on mining BitCoins across several machines and people, and sharing the result if one is lucky and get the 50 BitCoins. Check out BitCoin Pool if this sounds interesting. I have not had time to try to set up a machine to participate there yet, but have seen that running on ones own for a few days have not yield any BitCoins througth mining yet.

Update 2010-12-15: Found an interesting criticism of bitcoin. Not quite sure how valid it is, but thought it was interesting to read. The arguments presented seem to be equally valid for gold, which was used as a currency for many years.

With this weeks lawless governmental attacks on Wikileak and free speech, it has become obvious that PayPal, visa and mastercard can not be trusted to handle money transactions. A blog post from Simon Phipps on bitcoin reminded me about a project that a friend of mine mentioned earlier. I decided to follow Simon's example, and get involved with BitCoin. I got some help from my friend to get it all running, and he even handed me some bitcoins to get started. I even donated a few bitcoins to Simon for helping me remember BitCoin.

So, what is bitcoins, you probably wonder? It is a digital crypto-currency, decentralised and handled using peer-to-peer networks. It allows anonymous transactions and prohibits central control over the transactions, making it impossible for governments and companies alike to block donations and other transactions. The source is free software, and while the key dependency wxWidgets 2.9 for the graphical user interface is missing in Debian, the command line client builds just fine. Hopefully Jonas will get the package into Debian soon.

Bitcoins can be converted to other currencies, like USD and EUR. There are companies accepting bitcoins when selling services and goods, and there are even currency "stock" markets where the exchange rate is decided. There are not many users so far, but the concept seems promising. If you want to get started and lack a friend with any bitcoins to spare, you can even get some for free (0.05 bitcoin at the time of writing). Use BitcoinWatch to keep an eye on the current exchange rates.

As an experiment, I have decided to set up bitcoind on one of my machines. If you want to support my activity, please send Bitcoin donations to the address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b. Thank you!

In the latest issue of Linux Journal, the readers choices were presented, and the winner among the multimedia player were VLC. Personally, I like VLC, and it is my player of choice when I first try to play a video file or stream. Only if VLC fail will I drag out gmplayer to see if it can do better. The reason is mostly the failure model and trust. When VLC fail, it normally pop up a error message reporting the problem. When mplayer fail, it normally segfault or just hangs. The latter failure mode drain my trust in the program.

But even if VLC is my player of choice, we have choosen to use mplayer in Debian Edu/Skolelinux. The reason is simple. We need a good browser plugin to play web videos seamlessly, and the VLC browser plugin is not very good. For example, it lack in-line control buttons, so there is no way for the user to pause the video. Also, when I last tested the browser plugins available in Debian, the VLC plugin failed on several video pages where mplayer based plugins worked. If the browser plugin for VLC was as good as the gecko-mediaplayer package (which uses mplayer), we would switch.

While VLC is a good player, its user interface is slightly annoying. The most annoying feature is its inconsistent use of keyboard shortcuts. When the player is in full screen mode, its shortcuts are different from when it is playing the video in a window. For example, space only work as pause when in full screen mode. I wish it had consisten shortcuts and that space also would work when in window mode. Another nice shortcut in gmplayer is [enter] to restart the current video. It is very nice when playing short videos from the web and want to restart it when new people arrive to have a look at what is going on.

Michael Biebl suggested to me on IRC, that I changed my automated upgrade testing of the Lenny Gnome and KDE Desktop to do apt-get autoremove when using apt-get. This seem like a very good idea, so I adjusted by test scripts and can now present the updated result from today:

This is for Gnome:

Installed using apt-get, missing with aptitude

apache2.2-bin aptdaemon baobab binfmt-support browser-plugin-gnash cheese-common cli-common cups-pk-helper dmz-cursor-theme empathy empathy-common freedesktop-sound-theme freeglut3 gconf-defaults-service gdm-themes gedit-plugins geoclue geoclue-hostip geoclue-localnet geoclue-manual geoclue-yahoo gnash gnash-common gnome gnome-backgrounds gnome-cards-data gnome-codec-install gnome-core gnome-desktop-environment gnome-disk-utility gnome-screenshot gnome-search-tool gnome-session-canberra gnome-system-log gnome-themes-extras gnome-themes-more gnome-user-share gstreamer0.10-fluendo-mp3 gstreamer0.10-tools gtk2-engines gtk2-engines-pixbuf gtk2-engines-smooth hamster-applet libapache2-mod-dnssd libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libart2.0-cil libboost-date-time1.42.0 libboost-python1.42.0 libboost-thread1.42.0 libchamplain-0.4-0 libchamplain-gtk-0.4-0 libcheese-gtk18 libclutter-gtk-0.10-0 libcryptui0 libdiscid0 libelf1 libepc-1.0-2 libepc-common libepc-ui-1.0-2 libfreerdp-plugins-standard libfreerdp0 libgconf2.0-cil libgdata-common libgdata7 libgdu-gtk0 libgee2 libgeoclue0 libgexiv2-0 libgif4 libglade2.0-cil libglib2.0-cil libgmime2.4-cil libgnome-vfs2.0-cil libgnome2.24-cil libgnomepanel2.24-cil libgpod-common libgpod4 libgtk2.0-cil libgtkglext1 libgtksourceview2.0-common libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-cairo2.0-cil libmono-corlib2.0-cil libmono-i18n-west2.0-cil libmono-posix2.0-cil libmono-security2.0-cil libmono-sharpzip2.84-cil libmono-system2.0-cil libmtp8 libmusicbrainz3-6 libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libopal3.6.8 libpolkit-gtk-1-0 libpt2.6.7 libpython2.6 librpm1 librpmio1 libsdl1.2debian libsrtp0 libssh-4 libtelepathy-farsight0 libtelepathy-glib0 libtidy-0.99-0 media-player-info mesa-utils mono-2.0-gac mono-gac mono-runtime nautilus-sendto nautilus-sendto-empathy p7zip-full pkg-config python-aptdaemon python-aptdaemon-gtk python-axiom python-beautifulsoup python-bugbuddy python-clientform python-coherence python-configobj python-crypto python-cupshelpers python-elementtree python-epsilon python-evolution python-feedparser python-gdata python-gdbm python-gst0.10 python-gtkglext1 python-gtksourceview2 python-httplib2 python-louie python-mako python-markupsafe python-mechanize python-nevow python-notify python-opengl python-openssl python-pam python-pkg-resources python-pyasn1 python-pysqlite2 python-rdflib python-serial python-tagpy python-twisted-bin python-twisted-conch python-twisted-core python-twisted-web python-utidylib python-webkit python-xdg python-zope.interface remmina remmina-plugin-data remmina-plugin-rdp remmina-plugin-vnc rhythmbox-plugin-cdrecorder rhythmbox-plugins rpm-common rpm2cpio seahorse-plugins shotwell software-center system-config-printer-udev telepathy-gabble telepathy-mission-control-5 telepathy-salut tomboy totem totem-coherence totem-mozilla totem-plugins transmission-common xdg-user-dirs xdg-user-dirs-gtk xserver-xephyr

Installed using apt-get, removed with aptitude

cheese ekiga eog epiphany-extensions evolution-exchange fast-user-switch-applet file-roller gcalctool gconf-editor gdm gedit gedit-common gnome-games gnome-games-data gnome-nettool gnome-system-tools gnome-themes gnuchess gucharmap guile-1.8-libs libavahi-ui0 libdmx1 libgalago3 libgtk-vnc-1.0-0 libgtksourceview2.0-0 liblircclient0 libsdl1.2debian-alsa libspeexdsp1 libsvga1 rhythmbox seahorse sound-juicer system-config-printer totem-common transmission-gtk vinagre vino

Installed using aptitude, missing with apt-get

gstreamer0.10-gnomevfs

Installed using aptitude, removed with apt-get

[nothing]

This is for KDE:

Installed using apt-get, missing with aptitude

ksmserver

Installed using apt-get, removed with aptitude

kwin network-manager-kde

Installed using aptitude, missing with apt-get

arts dolphin freespacenotifier google-gadgets-gst google-gadgets-xul kappfinder kcalc kcharselect kde-core kde-plasma-desktop kde-standard kde-window-manager kdeartwork kdeartwork-emoticons kdeartwork-style kdeartwork-theme-icon kdebase kdebase-apps kdebase-workspace kdebase-workspace-bin kdebase-workspace-data kdeeject kdelibs kdeplasma-addons kdeutils kdewallpapers kdf kfloppy kgpg khelpcenter4 kinfocenter konq-plugins-l10n konqueror-nsplugins kscreensaver kscreensaver-xsavers ktimer kwrite libgle3 libkde4-ruby1.8 libkonq5 libkonq5-templates libnetpbm10 libplasma-ruby libplasma-ruby1.8 libqt4-ruby1.8 marble-data marble-plugins netpbm nuvola-icon-theme plasma-dataengines-workspace plasma-desktop plasma-desktopthemes-artwork plasma-runners-addons plasma-scriptengine-googlegadgets plasma-scriptengine-python plasma-scriptengine-qedje plasma-scriptengine-ruby plasma-scriptengine-webkit plasma-scriptengines plasma-wallpapers-addons plasma-widget-folderview plasma-widget-networkmanagement ruby sweeper update-notifier-kde xscreensaver-data-extra xscreensaver-gl xscreensaver-gl-extra xscreensaver-screensaver-bsod

Installed using aptitude, removed with apt-get

ark google-gadgets-common google-gadgets-qt htdig kate kdebase-bin kdebase-data kdepasswd kfind klipper konq-plugins konqueror ksysguard ksysguardd libarchive1 libcln6 libeet1 libeina-svn-06 libggadget-1.0-0b libggadget-qt-1.0-0b libgps19 libkdecorations4 libkephal4 libkonq4 libkonqsidebarplugin4a libkscreensaver5 libksgrd4 libksignalplotter4 libkunitconversion4 libkwineffects1a libmarblewidget4 libntrack-qt4-1 libntrack0 libplasma-geolocation-interface4 libplasmaclock4a libplasmagenericshell4 libprocesscore4a libprocessui4a libqalculate5 libqedje0a libqtruby4shared2 libqzion0a libruby1.8 libscim8c2a libsmokekdecore4-3 libsmokekdeui4-3 libsmokekfile3 libsmokekhtml3 libsmokekio3 libsmokeknewstuff2-3 libsmokeknewstuff3-3 libsmokekparts3 libsmokektexteditor3 libsmokekutils3 libsmokenepomuk3 libsmokephonon3 libsmokeplasma3 libsmokeqtcore4-3 libsmokeqtdbus4-3 libsmokeqtgui4-3 libsmokeqtnetwork4-3 libsmokeqtopengl4-3 libsmokeqtscript4-3 libsmokeqtsql4-3 libsmokeqtsvg4-3 libsmokeqttest4-3 libsmokeqtuitools4-3 libsmokeqtwebkit4-3 libsmokeqtxml4-3 libsmokesolid3 libsmokesoprano3 libtaskmanager4a libtidy-0.99-0 libweather-ion4a libxklavier16 libxxf86misc1 okteta oxygencursors plasma-dataengines-addons plasma-scriptengine-superkaramba plasma-widget-lancelot plasma-widgets-addons plasma-widgets-workspace polkit-kde-1 ruby1.8 systemsettings update-notifier-common

Running apt-get autoremove made the results using apt-get and aptitude a bit more similar, but there are still quite a lott of differences. I have no idea what packages should be installed after the upgrade, but hope those that do can have a look.

Most of the computers in use by the Debian Edu/Skolelinux project are virtual machines. And they have been Xen machines running on a fairly old IBM eserver xseries 345 machine, and we wanted to migrate them to KVM on a newer Dell PowerEdge 2950 host machine. This was a bit harder that it could have been, because we set up the Xen virtual machines to get the virtual partitions from LVM, which as far as I know is not supported by KVM. So to migrate, we had to convert several LVM logical volumes to partitions on a virtual disk file.

I found a nice recipe to do this, and wrote the following script to do the migration. It uses qemu-img from the qemu package to make the disk image, parted to partition it, losetup and kpartx to present the disk image partions as devices, and dd to copy the data. I NFS mounted the new servers storage area on the old server to do the migration.

#!/bin/sh

# Based on
# http://searchnetworking.techtarget.com.au/articles/35011-Six-steps-for-migrating-Xen-virtual-machines-to-KVM

set -e
set -x

if [ -z "$1" ] ; then
    echo "Usage: $0 <hostname>"
    exit 1
else
    host="$1"
fi

if [ ! -e /dev/vg_data/$host-disk ] ; then
    echo "error: unable to find LVM volume for $host"
    exit 1
fi

# Partitions need to be a bit bigger than the LVM LVs.  not sure why.
disksize=$( lvs --units m | grep $host-disk | awk '{sum = sum + $4} END { print int(sum * 1.05) }')
swapsize=$( lvs --units m | grep $host-swap | awk '{sum = sum + $4} END { print int(sum * 1.05) }')
totalsize=$(( ( $disksize + $swapsize ) ))

img=$host.img
#dd if=/dev/zero of=$img bs=1M count=$(( $disksize + $swapsize ))
qemu-img create $img ${totalsize}MMaking room on the Debian Edu/Sqeeze DVD

parted $img mklabel msdos
parted $img mkpart primary linux-swap 0 $disksize
parted $img mkpart primary ext2 $disksize $totalsize
parted $img set 1 boot on

modprobe dm-mod
losetup /dev/loop0 $img
kpartx -a /dev/loop0

dd if=/dev/vg_data/$host-disk of=/dev/mapper/loop0p1 bs=1M
fsck.ext3 -f /dev/mapper/loop0p1 || true
mkswap /dev/mapper/loop0p2

kpartx -d /dev/loop0
losetup -d /dev/loop0

The script is perhaps so simple that it is not copyrightable, but if it is, it is licenced using GPL v2 or later at your discretion.

After doing this, I booted a Debian CD in rescue mode in KVM with the new disk image attached, installed grub-pc and linux-image-686 and set up grub to boot from the disk image. After this, the KVM machines seem to work just fine.

I'm still running upgrade testing of the Lenny Gnome and KDE Desktop, but have not had time to spend on reporting the status. Here is a short update based on a test I ran 20101118.

I still do not know what a correct migration should look like, so I report any differences between apt and aptitude and hope someone else can see if anything should be changed.

This is for Gnome:

Installed using apt-get, missing with aptitude

apache2.2-bin aptdaemon at-spi baobab binfmt-support browser-plugin-gnash cheese-common cli-common cpp-4.3 cups-pk-helper dmz-cursor-theme empathy empathy-common finger freedesktop-sound-theme freeglut3 gconf-defaults-service gdm-themes gedit-plugins geoclue geoclue-hostip geoclue-localnet geoclue-manual geoclue-yahoo gnash gnash-common gnome gnome-backgrounds gnome-cards-data gnome-codec-install gnome-core gnome-desktop-environment gnome-disk-utility gnome-screenshot gnome-search-tool gnome-session-canberra gnome-spell gnome-system-log gnome-themes-extras gnome-themes-more gnome-user-share gs-common gstreamer0.10-fluendo-mp3 gstreamer0.10-tools gtk2-engines gtk2-engines-pixbuf gtk2-engines-smooth hal-info hamster-applet libapache2-mod-dnssd libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libart2.0-cil libatspi1.0-0 libboost-date-time1.42.0 libboost-python1.42.0 libboost-thread1.42.0 libchamplain-0.4-0 libchamplain-gtk-0.4-0 libcheese-gtk18 libclutter-gtk-0.10-0 libcryptui0 libcupsys2 libdiscid0 libeel2-data libelf1 libepc-1.0-2 libepc-common libepc-ui-1.0-2 libfreerdp-plugins-standard libfreerdp0 libgail-common libgconf2.0-cil libgdata-common libgdata7 libgdl-1-common libgdu-gtk0 libgee2 libgeoclue0 libgexiv2-0 libgif4 libglade2.0-cil libglib2.0-cil libgmime2.4-cil libgnome-vfs2.0-cil libgnome2.24-cil libgnomepanel2.24-cil libgnomeprint2.2-data libgnomeprintui2.2-common libgnomevfs2-bin libgpod-common libgpod4 libgtk2.0-cil libgtkglext1 libgtksourceview-common libgtksourceview2.0-common libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-cairo2.0-cil libmono-corlib2.0-cil libmono-i18n-west2.0-cil libmono-posix2.0-cil libmono-security2.0-cil libmono-sharpzip2.84-cil libmono-system2.0-cil libmtp8 libmusicbrainz3-6 libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libopal3.6.8 libpolkit-gtk-1-0 libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libpt2.6.7 libpython2.6 librpm1 librpmio1 libsdl1.2debian libservlet2.4-java libsrtp0 libssh-4 libtelepathy-farsight0 libtelepathy-glib0 libtidy-0.99-0 libxalan2-java libxerces2-java media-player-info mesa-utils mono-2.0-gac mono-gac mono-runtime nautilus-sendto nautilus-sendto-empathy openoffice.org-writer2latex openssl-blacklist p7zip p7zip-full pkg-config python-4suite-xml python-aptdaemon python-aptdaemon-gtk python-axiom python-beautifulsoup python-bugbuddy python-clientform python-coherence python-configobj python-crypto python-cupshelpers python-cupsutils python-eggtrayicon python-elementtree python-epsilon python-evolution python-feedparser python-gdata python-gdbm python-gst0.10 python-gtkglext1 python-gtkmozembed python-gtksourceview2 python-httplib2 python-louie python-mako python-markupsafe python-mechanize python-nevow python-notify python-opengl python-openssl python-pam python-pkg-resources python-pyasn1 python-pysqlite2 python-rdflib python-serial python-tagpy python-twisted-bin python-twisted-conch python-twisted-core python-twisted-web python-utidylib python-webkit python-xdg python-zope.interface remmina remmina-plugin-data remmina-plugin-rdp remmina-plugin-vnc rhythmbox-plugin-cdrecorder rhythmbox-plugins rpm-common rpm2cpio seahorse-plugins shotwell software-center svgalibg1 system-config-printer-udev telepathy-gabble telepathy-mission-control-5 telepathy-salut tomboy totem totem-coherence totem-mozilla totem-plugins transmission-common xdg-user-dirs xdg-user-dirs-gtk xserver-xephyr zip

Installed using apt-get, removed with aptitude

arj bluez-utils cheese dhcdbd djvulibre-desktop ekiga eog epiphany-extensions epiphany-gecko evolution-exchange fast-user-switch-applet file-roller gcalctool gconf-editor gdm gedit gedit-common gnome-app-install gnome-games gnome-games-data gnome-nettool gnome-system-tools gnome-themes gnome-utils gnome-vfs-obexftp gnome-volume-manager gnuchess gucharmap guile-1.8-libs hal libavahi-compat-libdnssd1 libavahi-core5 libavahi-ui0 libbind9-50 libbluetooth2 libcamel1.2-11 libcdio7 libcucul0 libcurl3 libdirectfb-1.0-0 libdmx1 libdvdread3 libedata-cal1.2-6 libedataserver1.2-9 libeel2-2.20 libepc-1.0-1 libepc-ui-1.0-1 libexchange-storage1.2-3 libfaad0 libgadu3 libgalago3 libgd2-noxpm libgda3-3 libgda3-common libggz2 libggzcore9 libggzmod4 libgksu1.2-0 libgksuui1.0-1 libgmyth0 libgnome-desktop-2 libgnome-pilot2 libgnomecups1.0-1 libgnomeprint2.2-0 libgnomeprintui2.2-0 libgpod3 libgraphviz4 libgtk-vnc-1.0-0 libgtkhtml2-0 libgtksourceview1.0-0 libgtksourceview2.0-0 libgucharmap6 libhesiod0 libicu38 libisccc50 libisccfg50 libiw29 libjaxp1.3-java-gcj libkpathsea4 liblircclient0 libltdl3 liblwres50 libmagick++10 libmagick10 libmalaga7 libmozjs1d libmpfr1ldbl libmtp7 libmysqlclient15off libnautilus-burn4 libneon27 libnm-glib0 libnm-util0 libopal-2.2 libosp5 libparted1.8-10 libpisock9 libpisync1 libpoppler-glib3 libpoppler3 libpt-1.10.10 libraw1394-8 libsdl1.2debian-alsa libsensors3 libsexy2 libsmbios2 libsoup2.2-8 libspeexdsp1 libssh2-1 libsuitesparse-3.1.0 libsvga1 libswfdec-0.6-90 libtalloc1 libtotem-plparser10 libtrackerclient0 libvoikko1 libxalan2-java-gcj libxerces2-java-gcj libxklavier12 libxtrap6 libxxf86misc1 libzephyr3 mysql-common rhythmbox seahorse sound-juicer swfdec-gnome system-config-printer totem-common totem-gstreamer transmission-gtk vinagre vino w3c-dtd-xhtml wodim

Installed using aptitude, missing with apt-get

gstreamer0.10-gnomevfs

Installed using aptitude, removed with apt-get

[nothing]

This is for KDE:

Installed using apt-get, missing with aptitude

autopoint bomber bovo cantor cantor-backend-kalgebra cpp-4.3 dcoprss edict espeak espeak-data eyesapplet fifteenapplet finger gettext ghostscript-x git gnome-audio gnugo granatier gs-common gstreamer0.10-pulseaudio indi kaddressbook-plugins kalgebra kalzium-data kanjidic kapman kate-plugins kblocks kbreakout kbstate kde-icons-mono kdeaccessibility kdeaddons-kfile-plugins kdeadmin-kfile-plugins kdeartwork-misc kdeartwork-theme-window kdeedu kdeedu-data kdeedu-kvtml-data kdegames kdegames-card-data kdegames-mahjongg-data kdegraphics-kfile-plugins kdelirc kdemultimedia-kfile-plugins kdenetwork-kfile-plugins kdepim-kfile-plugins kdepim-kio-plugins kdessh kdetoys kdewebdev kdiamond kdnssd kfilereplace kfourinline kgeography-data kigo killbots kiriki klettres-data kmoon kmrml knewsticker-scripts kollision kpf krosspython ksirk ksmserver ksquares kstars-data ksudoku kubrick kweather libasound2-plugins libboost-python1.42.0 libcfitsio3 libconvert-binhex-perl libcrypt-ssleay-perl libdb4.6++ libdjvulibre-text libdotconf1.0 liberror-perl libespeak1 libfinance-quote-perl libgail-common libgsl0ldbl libhtml-parser-perl libhtml-tableextract-perl libhtml-tagset-perl libhtml-tree-perl libio-stringy-perl libkdeedu4 libkdegames5 libkiten4 libkpathsea5 libkrossui4 libmailtools-perl libmime-tools-perl libnews-nntpclient-perl libopenbabel3 libportaudio2 libpulse-browse0 libservlet2.4-java libspeechd2 libtiff-tools libtimedate-perl libunistring0 liburi-perl libwww-perl libxalan2-java libxerces2-java lirc luatex marble networkstatus noatun-plugins openoffice.org-writer2latex palapeli palapeli-data parley parley-data poster psutils pulseaudio pulseaudio-esound-compat pulseaudio-module-x11 pulseaudio-utils quanta-data rocs rsync speech-dispatcher step svgalibg1 texlive-binaries texlive-luatex ttf-sazanami-gothic

Installed using apt-get, removed with aptitude

amor artsbuilder atlantik atlantikdesigner blinken bluez-utils cvs dhcdbd djvulibre-desktop imlib-base imlib11 kalzium kanagram kandy kasteroids katomic kbackgammon kbattleship kblackbox kbounce kbruch kcron kdat kdemultimedia-kappfinder-data kdeprint kdict kdvi kedit keduca kenolaba kfax kfaxview kfouleggs kgeography kghostview kgoldrunner khangman khexedit kiconedit kig kimagemapeditor kitchensync kiten kjumpingcube klatin klettres klickety klines klinkstatus kmag kmahjongg kmailcvt kmenuedit kmid kmilo kmines kmousetool kmouth kmplot knetwalk kodo kolf kommander konquest kooka kpager kpat kpdf kpercentage kpilot kpoker kpovmodeler krec kregexpeditor kreversi ksame ksayit kshisen ksig ksim ksirc ksirtet ksmiletris ksnake ksokoban kspaceduel kstars ksvg ksysv kteatime ktip ktnef ktouch ktron kttsd ktuberling kturtle ktux kuickshow kverbos kview kviewshell kvoctrain kwifimanager kwin kwin4 kwordquiz kworldclock kxsldbg libakode2 libarts1-akode libarts1-audiofile libarts1-mpeglib libarts1-xine libavahi-compat-libdnssd1 libavahi-core5 libavc1394-0 libbind9-50 libbluetooth2 libboost-python1.34.1 libcucul0 libcurl3 libcvsservice0 libdirectfb-1.0-0 libdjvulibre21 libdvdread3 libfaad0 libfreebob0 libgd2-noxpm libgraphviz4 libgsmme1c2a libgtkhtml2-0 libicu38 libiec61883-0 libindex0 libisccc50 libisccfg50 libiw29 libjaxp1.3-java-gcj libk3b3 libkcal2b libkcddb1 libkdeedu3 libkdegames1 libkdepim1a libkgantt0 libkleopatra1 libkmime2 libkpathsea4 libkpimexchange1 libkpimidentities1 libkscan1 libksieve0 libktnef1 liblockdev1 libltdl3 liblwres50 libmagick10 libmimelib1c2a libmodplug0c2 libmozjs1d libmpcdec3 libmpfr1ldbl libneon27 libnm-util0 libopensync0 libpisock9 libpoppler-glib3 libpoppler-qt2 libpoppler3 libraw1394-8 librss1 libsensors3 libsmbios2 libssh2-1 libsuitesparse-3.1.0 libswfdec-0.6-90 libtalloc1 libxalan2-java-gcj libxerces2-java-gcj libxtrap6 lskat mpeglib network-manager-kde noatun pmount tex-common texlive-base texlive-common texlive-doc-base texlive-fonts-recommended tidy ttf-dustin ttf-kochi-gothic ttf-sjfonts

Installed using aptitude, missing with apt-get

dolphin kde-core kde-plasma-desktop kde-standard kde-window-manager kdeartwork kdebase kdebase-apps kdebase-workspace kdebase-workspace-bin kdebase-workspace-data kdeutils kscreensaver kscreensaver-xsavers libgle3 libkonq5 libkonq5-templates libnetpbm10 netpbm plasma-widget-folderview plasma-widget-networkmanagement xscreensaver-data-extra xscreensaver-gl xscreensaver-gl-extra xscreensaver-screensaver-bsod

Installed using aptitude, removed with apt-get

kdebase-bin konq-plugins konqueror

Answering the call from the Gnash project for buildbot slaves to test the current source, I have set up a virtual KVM machine on the Debian Edu/Skolelinux virtualization host to test the git source on Debian/Squeeze. I hope this can help the developers in getting new releases out more often.

As the developers want less main-stream build platforms tested to, I have considered setting up a Debian/kfreebsd machine as well. I have also considered using the kfreebsd architecture in Debian as a file server in NUUG to get access to the 5 TB zfs volume we currently use to store DV video. Because of this, I finally got around to do a test installation of Debian/Squeeze with kfreebsd. Installation went fairly smooth, thought I noticed some visual glitches in the cdebconf dialogs (black cursor left on the screen at random locations). Have not gotten very far with the testing. Noticed cfdisk did not work, but fdisk did so it was not a fatal problem. Have to spend some more time on it to see if it is useful as a file server for NUUG. Will try to find time to set up a gnash buildbot slave on the Debian Edu/Skolelinux this weekend.

3D printing is just great. I just came across this Debian logo in 3D linked in from the thingiverse blog.

Some updates.

My gnash pledge to raise money for the project is going well. The lower limit of 10 signers was reached in 24 hours, and so far 13 people have signed it. More signers and more funding is most welcome, and I am really curious how far we can get before the time limit of December 24 is reached. :)

On the #gnash IRC channel on irc.freenode.net, I was just tipped about what appear to be a great code coverage tool capable of generating code coverage stats without any changes to the source code. It is called kcov, and can be used using kcov <directory> <binary>. It is missing in Debian, but the git source built just fine in Squeeze after I installed libelf-dev, libdwarf-dev, pkg-config and libglib2.0-dev. Failed to build in Lenny, but suspect that is solvable. I hope kcov make it into Debian soon.

Finally found time to wrap up the release notes for a new alpha release of Debian Edu, and just published the second alpha test release of the Squeeze based Debian Edu / Skolelinux release. Give it a try if you need a complete linux solution for your school, including central infrastructure server, workstations, thin client servers and diskless workstations. A nice touch added yesterday is RDP support on the thin client servers, for windows clients to get a Linux desktop on request.

In the Debian popularity-contest numbers, the adobe-flashplugin package the second most popular used package that is missing in Debian. The sixth most popular is flashplayer-mozilla. This is a clear indication that working flash is important for Debian users. Around 10 percent of the users submitting data to popcon.debian.org have this package installed.

In the report written by Lars Risan in August 2008 («Skolelinux i bruk – Rapport for Hurum kommune, Universitetet i Agder og stiftelsen SLX Debian Labs»), one of the most important problems schools experienced with Debian Edu/Skolelinux was the lack of working Flash. A lot of educational web sites require Flash to work, and lacking working Flash support in the web browser and the problems with installing it was perceived as a good reason to stay with Windows.

I once saw a funny and sad comment in a web forum, where Linux was said to be the retarded cousin that did not really understand everything you told him but could work fairly well. This was a comment regarding the problems Linux have with proprietary formats and non-standard web pages, and is sad because it exposes a fairly common understanding of whose fault it is if web pages that only work in for example Internet Explorer 6 fail to work on Firefox, and funny because it explain very well how annoying it is for users when Linux distributions do not work with the documents they receive or the web pages they want to visit.

This is part of the reason why I believe it is important for Debian and Debian Edu to have a well working Flash implementation in the distribution, to get at least popular sites as Youtube and Google Video to working out of the box. For Squeeze, Debian have the chance to include the latest version of Gnash that will make this happen, as the new release 0.8.8 was published a few weeks ago and is resting in unstable. The new version work with more sites that version 0.8.7. The Gnash maintainers have asked for a freeze exception, but the release team have not had time to reply to it yet. I hope they agree with me that Flash is important for the Debian desktop users, and thus accept the new package into Squeeze.

I discovered this while doing automated testing of upgrades from Debian Lenny to Squeeze. A few packages in Debian still got circular dependencies, and it is often claimed that apt and aptitude should be able to handle this just fine, but some times these dependency loops causes apt to fail.

An example is from todays upgrade of KDE using aptitude. In it, a bug in kdebase-workspace-data causes perl-modules to fail to upgrade. The cause is simple. If a package fail to unpack, then only part of packages with the circular dependency might end up being unpacked when unpacking aborts, and the ones already unpacked will fail to configure in the recovery phase because its dependencies are unavailable.

In this log, the problem manifest itself with this error:

dpkg: dependency problems prevent configuration of perl-modules:
 perl-modules depends on perl (>= 5.10.1-1); however:
  Version of perl on system is 5.10.0-19lenny2.
dpkg: error processing perl-modules (--configure):
 dependency problems - leaving unconfigured

The perl/perl-modules circular dependency is already reported as a bug, and will hopefully be solved as soon as possible, but it is not the only one, and each one of these loops in the dependency tree can cause similar failures. Of course, they only occur when there are bugs in other packages causing the unpacking to fail, but it is rather nasty when the failure of one package causes the problem to become worse because of dependency loops.

Thanks to the tireless effort by Bill Allombert, the number of circular dependencies left in Debian is dropping, and perhaps it will reach zero one day. :)

Todays testing also exposed a bug in update-notifier and different behaviour between apt-get and aptitude, the latter possibly caused by some circular dependency. Reported both to BTS to try to get someone to look at it.

This is a followup on my previous work on merging all the computer related LDAP objects in Debian Edu.

As a step to try to see if it possible to merge the DNS and DHCP LDAP objects, I have had a look at how the packages pdns-backend-ldap and dhcp3-server-ldap in Debian use the LDAP server. The two implementations are quite different in how they use LDAP.

To get this information, I started slapd with debugging enabled and dumped the debug output to a file to get the LDAP searches performed on a Debian Edu main-server. Here is a summary.

powerdns

Clues on how to set up PowerDNS to use a LDAP backend is available on the web.

PowerDNS have two modes of operation using LDAP as its backend. One "strict" mode where the forward and reverse DNS lookups are done using the same LDAP objects, and a "tree" mode where the forward and reverse entries are in two different subtrees in LDAP with a structure based on the DNS names, as in tjener.intern and 2.2.0.10.in-addr.arpa.

In tree mode, the server is set up to use a LDAP subtree as its base, and uses a "base" scoped search for the DNS name by adding "dc=tjener,dc=intern," to the base with a filter for "(associateddomain=tjener.intern)" for the forward entry and "dc=2,dc=2,dc=0,dc=10,dc=in-addr,dc=arpa," with a filter for "(associateddomain=2.2.0.10.in-addr.arpa)" for the reverse entry. For forward entries, it is looking for attributes named dnsttl, arecord, nsrecord, cnamerecord, soarecord, ptrrecord, hinforecord, mxrecord, txtrecord, rprecord, afsdbrecord, keyrecord, aaaarecord, locrecord, srvrecord, naptrrecord, kxrecord, certrecord, dsrecord, sshfprecord, ipseckeyrecord, rrsigrecord, nsecrecord, dnskeyrecord, dhcidrecord, spfrecord and modifytimestamp. For reverse entries it is looking for the attributes dnsttl, arecord, nsrecord, cnamerecord, soarecord, ptrrecord, hinforecord, mxrecord, txtrecord, rprecord, aaaarecord, locrecord, srvrecord, naptrrecord and modifytimestamp. The equivalent ldapsearch commands could look like this:

ldapsearch -h ldap \
  -b dc=tjener,dc=intern,ou=hosts,dc=skole,dc=skolelinux,dc=no \
  -s base -x '(associateddomain=tjener.intern)' dNSTTL aRecord nSRecord \
  cNAMERecord sOARecord pTRRecord hInfoRecord mXRecord tXTRecord \
  rPRecord aFSDBRecord KeyRecord aAAARecord lOCRecord sRVRecord \
  nAPTRRecord kXRecord certRecord dSRecord sSHFPRecord iPSecKeyRecord \
  rRSIGRecord nSECRecord dNSKeyRecord dHCIDRecord sPFRecord modifyTimestamp

ldapsearch -h ldap \
  -b dc=2,dc=2,dc=0,dc=10,dc=in-addr,dc=arpa,ou=hosts,dc=skole,dc=skolelinux,dc=no \
  -s base -x '(associateddomain=2.2.0.10.in-addr.arpa)'
  dnsttl, arecord, nsrecord, cnamerecord soarecord ptrrecord \
  hinforecord mxrecord txtrecord rprecord aaaarecord locrecord \
  srvrecord naptrrecord modifytimestamp

In Debian Edu/Lenny, the PowerDNS tree mode is used with ou=hosts,dc=skole,dc=skolelinux,dc=no as the base, and these are two example LDAP objects used there. In addition to these objects, the parent objects all th way up to ou=hosts,dc=skole,dc=skolelinux,dc=no also exist.

dn: dc=tjener,dc=intern,ou=hosts,dc=skole,dc=skolelinux,dc=no
objectclass: top
objectclass: dnsdomain
objectclass: domainrelatedobject
dc: tjener
arecord: 10.0.2.2
associateddomain: tjener.intern

dn: dc=2,dc=2,dc=0,dc=10,dc=in-addr,dc=arpa,ou=hosts,dc=skole,dc=skolelinux,dc=no
objectclass: top
objectclass: dnsdomain2
objectclass: domainrelatedobject
dc: 2
ptrrecord: tjener.intern
associateddomain: 2.2.0.10.in-addr.arpa

In strict mode, the server behaves differently. When looking for forward DNS entries, it is doing a "subtree" scoped search with the same base as in the tree mode for a object with filter "(associateddomain=tjener.intern)" and requests the attributes dnsttl, arecord, nsrecord, cnamerecord, soarecord, ptrrecord, hinforecord, mxrecord, txtrecord, rprecord, aaaarecord, locrecord, srvrecord, naptrrecord and modifytimestamp. For reverse entires it also do a subtree scoped search but this time the filter is "(arecord=10.0.2.2)" and the requested attributes are associateddomain, dnsttl and modifytimestamp. In short, in strict mode the objects with ptrrecord go away, and the arecord attribute in the forward object is used instead.

The forward and reverse searches can be simulated using ldapsearch like this:

ldapsearch -h ldap -b ou=hosts,dc=skole,dc=skolelinux,dc=no -s sub -x \
  '(associateddomain=tjener.intern)' dNSTTL aRecord nSRecord \
  cNAMERecord sOARecord pTRRecord hInfoRecord mXRecord tXTRecord \
  rPRecord aFSDBRecord KeyRecord aAAARecord lOCRecord sRVRecord \
  nAPTRRecord kXRecord certRecord dSRecord sSHFPRecord iPSecKeyRecord \
  rRSIGRecord nSECRecord dNSKeyRecord dHCIDRecord sPFRecord modifyTimestamp

ldapsearch -h ldap -b ou=hosts,dc=skole,dc=skolelinux,dc=no -s sub -x \
  '(arecord=10.0.2.2)' associateddomain dnsttl modifytimestamp

In addition to the forward and reverse searches , there is also a search for SOA records, which behave similar to the forward and reverse lookups.

A thing to note with the PowerDNS behaviour is that it do not specify any objectclass names, and instead look for the attributes it need to generate a DNS reply. This make it able to work with any objectclass that provide the needed attributes.

The attributes are normally provided in the cosine (RFC 1274) and dnsdomain2 schemas. The latter is used for reverse entries like ptrrecord and recent DNS additions like aaaarecord and srvrecord.

In Debian Edu, we have created DNS objects using the object classes dcobject (for dc), dnsdomain or dnsdomain2 (structural, for the DNS attributes) and domainrelatedobject (for associatedDomain). The use of structural object classes make it impossible to combine these classes with the object classes used by DHCP.

There are other schemas that could be used too, for example the dnszone structural object class used by Gosa and bind-sdb for the DNS attributes combined with the domainrelatedobject object class, but in this case some unused attributes would have to be included as well (zonename and relativedomainname).

My proposal for Debian Edu would be to switch PowerDNS to strict mode and not use any of the existing objectclasses (dnsdomain, dnsdomain2 and dnszone) when one want to combine the DNS information with DHCP information, and instead create a auxiliary object class defined something like this (using the attributes defined for dnsdomain and dnsdomain2 or dnszone):

objectclass ( some-oid NAME 'dnsDomainAux'
    SUP top
    AUXILIARY
    MAY ( ARecord $ MDRecord $ MXRecord $ NSRecord $ SOARecord $ CNAMERecord $
          DNSTTL $ DNSClass $ PTRRecord $ HINFORecord $ MINFORecord $
          TXTRecord $ SIGRecord $ KEYRecord $ AAAARecord $ LOCRecord $
          NXTRecord $ SRVRecord $ NAPTRRecord $ KXRecord $ CERTRecord $
          A6Record $ DNAMERecord
    ))

This will allow any object to become a DNS entry when combined with the domainrelatedobject object class, and allow any entity to include all the attributes PowerDNS wants. I've sent an email to the PowerDNS developers asking for their view on this schema and if they are interested in providing such schema with PowerDNS, and I hope my message will be accepted into their mailing list soon.

ISC dhcp

The DHCP server searches for specific objectclass and requests all the object attributes, and then uses the attributes it want. This make it harder to figure out exactly what attributes are used, but thanks to the working example in Debian Edu I can at least get an idea what is needed without having to read the source code.

In the DHCP server configuration, the LDAP base to use and the search filter to use to locate the correct dhcpServer entity is stored. These are the relevant entries from /etc/dhcp3/dhcpd.conf:

ldap-base-dn "dc=skole,dc=skolelinux,dc=no";
ldap-dhcp-server-cn "dhcp";

The DHCP server uses this information to nest all the DHCP configuration it need. The cn "dhcp" is located using the given LDAP base and the filter "(&(objectClass=dhcpServer)(cn=dhcp))". The search result is this entry:

dn: cn=dhcp,dc=skole,dc=skolelinux,dc=no
cn: dhcp
objectClass: top
objectClass: dhcpServer
dhcpServiceDN: cn=DHCP Config,dc=skole,dc=skolelinux,dc=no

The content of the dhcpServiceDN attribute is next used to locate the subtree with DHCP configuration. The DHCP configuration subtree base is located using a base scope search with base "cn=DHCP Config,dc=skole,dc=skolelinux,dc=no" and filter "(&(objectClass=dhcpService)(|(dhcpPrimaryDN=cn=dhcp,dc=skole,dc=skolelinux,dc=no)(dhcpSecondaryDN=cn=dhcp,dc=skole,dc=skolelinux,dc=no)))". The search result is this entry:

dn: cn=DHCP Config,dc=skole,dc=skolelinux,dc=no
cn: DHCP Config
objectClass: top
objectClass: dhcpService
objectClass: dhcpOptions
dhcpPrimaryDN: cn=dhcp, dc=skole,dc=skolelinux,dc=no
dhcpStatements: ddns-update-style none
dhcpStatements: authoritative
dhcpOption: smtp-server code 69 = array of ip-address
dhcpOption: www-server code 72 = array of ip-address
dhcpOption: wpad-url code 252 = text

Next, the entire subtree is processed, one level at the time. When all the DHCP configuration is loaded, it is ready to receive requests. The subtree in Debian Edu contain objects with object classes top/dhcpService/dhcpOptions, top/dhcpSharedNetwork/dhcpOptions, top/dhcpSubnet, top/dhcpGroup and top/dhcpHost. These provide options and information about netmasks, dynamic range etc. Leaving out the details here because it is not relevant for the focus of my investigation, which is to see if it is possible to merge dns and dhcp related computer objects.

When a DHCP request come in, LDAP is searched for the MAC address of the client (00:00:00:00:00:00 in this example), using a subtree scoped search with "cn=DHCP Config,dc=skole,dc=skolelinux,dc=no" as the base and "(&(objectClass=dhcpHost)(dhcpHWAddress=ethernet 00:00:00:00:00:00))" as the filter. This is what a host object look like:

dn: cn=hostname,cn=group1,cn=THINCLIENTS,cn=DHCP Config,dc=skole,dc=skolelinux,dc=no
cn: hostname
objectClass: top
objectClass: dhcpHost
dhcpHWAddress: ethernet 00:00:00:00:00:00
dhcpStatements: fixed-address hostname

There is less flexiblity in the way LDAP searches are done here. The object classes need to have fixed names, and the configuration need to be stored in a fairly specific LDAP structure. On the positive side, the invidiual dhcpHost entires can be anywhere without the DN pointed to by the dhcpServer entries. The latter should make it possible to group all host entries in a subtree next to the configuration entries, and this subtree can also be shared with the DNS server if the schema proposed above is combined with the dhcpHost structural object class.

Conclusion

The PowerDNS implementation seem to be very flexible when it come to which LDAP schemas to use. While its "tree" mode is rigid when it come to the the LDAP structure, the "strict" mode is very flexible, allowing DNS objects to be stored anywhere under the base cn specified in the configuration.

The DHCP implementation on the other hand is very inflexible, both regarding which LDAP schemas to use and which LDAP structure to use. I guess one could implement ones own schema, as long as the objectclasses and attributes have the names used, but this do not really help when the DHCP subtree need to have a fairly fixed structure.

Based on the observed behaviour, I suspect a LDAP structure like this might work for Debian Edu:

ou=services
  cn=machine-info (dhcpService) - dhcpServiceDN points here
    cn=dhcp (dhcpServer)
    cn=dhcp-internal (dhcpSharedNetwork/dhcpOptions)
      cn=10.0.2.0 (dhcpSubnet)
        cn=group1 (dhcpGroup/dhcpOptions)
    cn=dhcp-thinclients (dhcpSharedNetwork/dhcpOptions)
      cn=192.168.0.0 (dhcpSubnet)
        cn=group1 (dhcpGroup/dhcpOptions)
    ou=machines - PowerDNS base points here
      cn=hostname (dhcpHost/domainrelatedobject/dnsDomainAux)

This is not tested yet. If the DHCP server require the dhcpHost entries to be in the dhcpGroup subtrees, the entries can be stored there instead of a common machines subtree, and the PowerDNS base would have to be moved one level up to the machine-info subtree.

The combined object under the machines subtree would look something like this:

dn: dc=hostname,ou=machines,cn=machine-info,dc=skole,dc=skolelinux,dc=no
dc: hostname
objectClass: top
objectClass: dhcpHost
objectclass: domainrelatedobject
objectclass: dnsDomainAux
associateddomain: hostname.intern
arecord: 10.11.12.13
dhcpHWAddress: ethernet 00:00:00:00:00:00
dhcpStatements: fixed-address hostname.intern

One could even add the LTSP configuration associated with a given machine, as long as the required attributes are available in a auxiliary object class.

For a while now, I have wanted to find a way to change the DNS and DHCP services in Debian Edu to use the same LDAP objects for a given computer, to avoid the possibility of having a inconsistent state for a computer in LDAP (as in DHCP but no DNS entry or the other way around) and make it easier to add computers to LDAP.

I've looked at how powerdns and dhcpd is using LDAP, and using this information finally found a solution that seem to work.

The old setup required three LDAP objects for a given computer. One forward DNS entry, one reverse DNS entry and one DHCP entry. If we switch powerdns to use its strict LDAP method (ldap-method=strict in pdns-debian-edu.conf), the forward and reverse DNS entries are merged into one while making it impossible to transfer the reverse map to a slave DNS server.

If we also replace the object class used to get the DNS related attributes to one allowing these attributes to be combined with the dhcphost object class, we can merge the DNS and DHCP entries into one. I've written such object class in the dnsdomainaux.schema file (need proper OIDs, but that is a minor issue), and tested the setup. It seem to work.

With this test setup in place, we can get away with one LDAP object for both DNS and DHCP, and even the LTSP configuration I suggested in an earlier email. The combined LDAP object will look something like this:

  dn: cn=hostname,cn=group1,cn=THINCLIENTS,cn=DHCP Config,dc=skole,dc=skolelinux,dc=no
  cn: hostname
  objectClass: dhcphost
  objectclass: domainrelatedobject
  objectclass: dnsdomainaux
  associateddomain: hostname.intern
  arecord: 10.11.12.13
  dhcphwaddress: ethernet 00:00:00:00:00:00
  dhcpstatements: fixed-address hostname
  ldapconfigsound: Y

The DNS server uses the associateddomain and arecord entries, while the DHCP server uses the dhcphwaddress and dhcpstatements entries before asking DNS to resolve the fixed-adddress. LTSP will use dhcphwaddress or associateddomain and the ldapconfig* attributes.

I am not yet sure if I can get the DHCP server to look for its dhcphost in a different location, to allow us to put the objects outside the "DHCP Config" subtree, but hope to figure out a way to do that. If I can't figure out a way to do that, we can still get rid of the hosts subtree and move all its content into the DHCP Config tree (which probably should be renamed to be more related to the new content. I suspect cn=dnsdhcp,ou=services or something like that might be a good place to put it.

If you want to help out with implementing this for Debian Edu, please contact us on debian-edu@lists.debian.org.

Vagrant mentioned on IRC today that ltsp_config now support sourcing files from /usr/share/ltsp/ltsp_config.d/ on the thin clients, and that this can be used to fetch configuration from LDAP if Debian Edu choose to store configuration there.

Armed with this information, I got inspired and wrote a test module to get configuration from LDAP. The idea is to look up the MAC address of the client in LDAP, and look for attributes on the form ltspconfigsetting=value, and use this to export SETTING=value to the LTSP clients.

The goal is to be able to store the LTSP configuration attributes in a "computer" LDAP object used by both DNS and DHCP, and thus allowing us to store all information about a computer in one place.

This is a untested draft implementation, and I welcome feedback on this approach. A real LDAP schema for the ltspClientAux objectclass need to be written. Comments, suggestions, etc?

# Store in /opt/ltsp/$arch/usr/share/ltsp/ltsp_config.d/ldap-config
#
# Fetch LTSP client settings from LDAP based on MAC address
#
# Uses ethernet address as stored in the dhcpHost objectclass using
# the dhcpHWAddress attribute or ethernet address stored in the
# ieee802Device objectclass with the macAddress attribute.
#
# This module is written to be schema agnostic, and only depend on the
# existence of attribute names.
#
# The LTSP configuration variables are saved directly using a
# ltspConfig prefix and uppercasing the rest of the attribute name.
# To set the SERVER variable, set the ltspConfigServer attribute.
#
# Some LDAP schema should be created with all the relevant
# configuration settings.  Something like this should work:
# 
# objectclass ( 1.1.2.2 NAME 'ltspClientAux'
#     SUP top
#     AUXILIARY
#     MAY ( ltspConfigServer $ ltsConfigSound $ ... )

LDAPSERVER=$(debian-edu-ldapserver)
if [ "$LDAPSERVER" ] ; then
    LDAPBASE=$(debian-edu-ldapserver -b)
    for MAC in $(LANG=C ifconfig |grep -i hwaddr| awk '{print $5}'|sort -u) ; do
	filter="(|(dhcpHWAddress=ethernet $MAC)(macAddress=$MAC))"
	ldapsearch -h "$LDAPSERVER" -b "$LDAPBASE" -v -x "$filter" | \
	    grep '^ltspConfig' | while read attr value ; do
	    # Remove prefix and convert to upper case
	    attr=$(echo $attr | sed 's/^ltspConfig//i' | tr a-z A-Z)
	    # bass value on to clients
	    eval "$attr=$value; export $attr"
	done
    done
fi

I'm not sure this shell construction will work, because I suspect the while block might end up in a subshell causing the variables set there to not show up in ltsp-config, but if that is the case I am sure the code can be restructured to make sure the variables are passed on. I expect that can be solved with some testing. :)

If you want to help out with implementing this for Debian Edu, please contact us on debian-edu@lists.debian.org.

Update 2010-07-17: I am aware of another effort to store LTSP configuration in LDAP that was created around year 2000 by PC Xperience, Inc., 2000. I found its files on a personal home page over at redhat.com.

Since my last post about available LDAP tools in Debian, I was told about a LDAP GUI that is even better than luma. The java application jXplorer is claimed to be capable of moving LDAP objects and subtrees using drag-and-drop, and can authenticate using Kerberos. I have only tested the Kerberos authentication, but do not have a LDAP setup allowing me to rewrite LDAP with my test user yet. It is available in Debian testing and unstable at the moment. The only problem I have with it is how it handle errors. If something go wrong, its non-intuitive behaviour require me to go through some query work list and remove the failing query. Nothing big, but very annoying.

Here is a short update on my my Debian Lenny->Squeeze upgrade testing. Here is a summary of the difference for Gnome when it is upgraded by apt-get and aptitude. I'm not reporting the status for KDE, because the upgrade crashes when aptitude try because of missing conflicts (#584861 and #585716).

At the end of the upgrade test script, dpkg -l is executed to get a complete list of the installed packages. Based on this I see these differences when I did a test run today. As usual, I do not really know what the correct set of packages would be, but thought it best to publish the difference.

Installed using apt-get, missing with aptitude

at-spi cpp-4.3 finger gnome-spell gstreamer0.10-gnomevfs libatspi1.0-0 libcupsys2 libeel2-data libgail-common libgdl-1-common libgnomeprint2.2-data libgnomeprintui2.2-common libgnomevfs2-bin libgtksourceview-common libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libservlet2.4-java libxalan2-java libxerces2-java openoffice.org-writer2latex openssl-blacklist p7zip python-4suite-xml python-eggtrayicon python-gtkhtml2 python-gtkmozembed svgalibg1 xserver-xephyr zip

Installed using apt-get, removed with aptitude

bluez-utils dhcdbd djvulibre-desktop epiphany-gecko gnome-app-install gnome-mount gnome-vfs-obexftp gnome-volume-manager libao2 libavahi-compat-libdnssd1 libavahi-core5 libbind9-50 libbluetooth2 libcamel1.2-11 libcdio7 libcucul0 libcurl3 libdirectfb-1.0-0 libdvdread3 libedata-cal1.2-6 libedataserver1.2-9 libeel2-2.20 libepc-1.0-1 libepc-ui-1.0-1 libexchange-storage1.2-3 libfaad0 libgd2-noxpm libgda3-3 libgda3-common libggz2 libggzcore9 libggzmod4 libgksu1.2-0 libgksuui1.0-1 libgmyth0 libgnome-desktop-2 libgnome-pilot2 libgnomecups1.0-1 libgnomeprint2.2-0 libgnomeprintui2.2-0 libgpod3 libgraphviz4 libgtkhtml2-0 libgtksourceview1.0-0 libgucharmap6 libhesiod0 libicu38 libisccc50 libisccfg50 libiw29 libkpathsea4 libltdl3 liblwres50 libmagick++10 libmagick10 libmalaga7 libmtp7 libmysqlclient15off libnautilus-burn4 libneon27 libnm-glib0 libnm-util0 libopal-2.2 libosp5 libparted1.8-10 libpisock9 libpisync1 libpoppler-glib3 libpoppler3 libpt-1.10.10 libraw1394-8 libsensors3 libsmbios2 libsoup2.2-8 libssh2-1 libsuitesparse-3.1.0 libswfdec-0.6-90 libtalloc1 libtotem-plparser10 libtrackerclient0 libvoikko1 libxalan2-java-gcj libxerces2-java-gcj libxklavier12 libxtrap6 libxxf86misc1 libzephyr3 mysql-common swfdec-gnome totem-gstreamer wodim

Installed using aptitude, missing with apt-get

gnome gnome-desktop-environment hamster-applet python-gnomeapplet python-gnomekeyring python-wnck rhythmbox-plugins xorg xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-kbd xserver-xorg-input-mouse xserver-xorg-input-synaptics xserver-xorg-video-all xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-dummy xserver-xorg-video-fbdev xserver-xorg-video-glint xserver-xorg-video-i128 xserver-xorg-video-i740 xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic xserver-xorg-video-nouveau xserver-xorg-video-nv xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-radeonhd xserver-xorg-video-rendition xserver-xorg-video-s3 xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb xserver-xorg-video-tdfx xserver-xorg-video-tga xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-vesa xserver-xorg-video-vmware xserver-xorg-video-voodoo

Installed using aptitude, removed with apt-get

deskbar-applet xserver-xorg xserver-xorg-core xserver-xorg-input-wacom xserver-xorg-video-intel xserver-xorg-video-openchrome

I was told on IRC that the xorg-xserver package was changed in git today to try to get apt-get to not remove xorg completely. No idea when it hits Squeeze, but when it does I hope it will reduce the difference somewhat.

The last few days I have been looking into the status of the LDAP directory in Debian Edu, and in the process I started to miss a GUI tool to browse the LDAP tree. The only one I was able to find in Debian/Squeeze and Lenny is LUMA, which has proved to be a great tool to get a overview of the current LDAP directory populated by default in Skolelinux. Thanks to it, I have been able to find empty and obsolete subtrees, misplaced objects and duplicate objects. It will be installed by default in Debian/Squeeze. If you are working with LDAP, give it a go. :)

I did notice one problem with it I have not had time to report to the BTS yet. There is no .desktop file in the package, so the tool do not show up in the Gnome and KDE menus, but only deep down in in the Debian submenu in KDE. I hope that can be fixed before Squeeze is released.

I have not yet been able to get it to modify the tree yet. I would like to move objects and remove subtrees directly in the GUI, but have not found a way to do that with LUMA yet. So in the mean time, I use ldapvi for that.

If you have tips on other GUI tools for LDAP that might be useful in Debian Edu, please contact us on debian-edu@lists.debian.org.

Update 2010-06-29: Ross Reedstrom tipped us about the gq package as a useful GUI alternative. It seem like a good tool, but is unmaintained in Debian and got a RC bug keeping it out of Squeeze. Unless that changes, it will not be an option for Debian Edu based on Squeeze.

A while back, I complained about the fact that it is not possible with the provided schemas for storing DNS and DHCP information in LDAP to combine the two sets of information into one LDAP object representing a computer.

In the mean time, I discovered that a simple fix would be to make the dhcpHost object class auxiliary, to allow it to be combined with the dNSDomain object class, and thus forming one object for one computer when storing both DHCP and DNS information in LDAP.

If I understand this correctly, it is not safe to do this change without also changing the assigned number for the object class, and I do not know enough about LDAP schema design to do that properly for Debian Edu.

Anyway, for future reference, this is how I believe we could change the DHCP schema to solve at least part of the problem with the LDAP schemas available today from IETF.

--- dhcp.schema    (revision 65192)
+++ dhcp.schema    (working copy)
@@ -376,7 +376,7 @@
 objectclass ( 2.16.840.1.113719.1.203.6.6
        NAME 'dhcpHost'
        DESC 'This represents information about a particular client'
-       SUP top
+       SUP top AUXILIARY
        MUST cn
        MAY  (dhcpLeaseDN $ dhcpHWAddress $ dhcpOptionsDN $ dhcpStatements $ dhcpComments $ dhcpOption)
        X-NDS_CONTAINMENT ('dhcpService' 'dhcpSubnet' 'dhcpGroup') )

I very much welcome clues on how to do this properly for Debian Edu/Squeeze. We provide the DHCP schema in our debian-edu-config package, and should thus be free to rewrite it as we see fit.

If you want to help out with implementing this for Debian Edu, please contact us on debian-edu@lists.debian.org.

A few times I have had the need to simulate the way tasksel installs packages during the normal debian-installer run. Until now, I have ended up letting tasksel do the work, with the annoying problem of not getting any feedback at all when something fails (like a conffile question from dpkg or a download that fails), using code like this:

export DEBIAN_FRONTEND=noninteractive
tasksel --new-install

This would invoke tasksel, let its automatic task selection pick the tasks to install, and continue to install the requested tasks without any output what so ever. Recently I revisited this problem while working on the automatic package upgrade testing, because tasksel would some times hang without any useful feedback, and I want to see what is going on when it happen. Then it occured to me, I can parse the output from tasksel when asked to run in test mode, and use that aptitude command line printed by tasksel then to simulate the tasksel run. I ended up using code like this:

export DEBIAN_FRONTEND=noninteractive
cmd="$(in_target tasksel -t --new-install | sed 's/debconf-apt-progress -- //')"
$cmd

The content of $cmd is typically something like "aptitude -q --without-recommends -o APT::Install-Recommends=no -y install ~t^desktop$ ~t^gnome-desktop$ ~t^laptop$ ~pstandard ~prequired ~pimportant", which will install the gnome desktop task, the laptop task and all packages with priority standard , required and important, just like tasksel would have done it during installation.

A better approach is probably to extend tasksel to be able to install packages without using debconf-apt-progress, for use cases like this.

My testing of Debian upgrades from Lenny to Squeeze continues, and I've finally made the upgrade logs available from https://people.skolelinux.org/pere/debian-upgrade-testing/. I am now testing dist-upgrade of Gnome and KDE in a chroot using both apt and aptitude, and found their differences interesting. This time I will only focus on their removal plans.

After installing a Gnome desktop and the laptop task, apt-get wants to remove 72 packages when dist-upgrading from Lenny to Squeeze. The surprising part is that it want to remove xorg and all xserver-xorg-video* drivers. Clearly not a good choice, but I am not sure why. When asking aptitude to do the same, it want to remove 129 packages, but most of them are library packages I suspect are no longer needed. Both of them want to remove bluetooth packages, which I do not know. Perhaps these bluetooth packages are obsolete?

For KDE, apt-get want to remove 82 packages, among them kdebase which seem like a bad idea and xorg the same way as with Gnome. Asking aptitude for the same, it wants to remove 192 packages, none which are too surprising.

I guess the removal of xorg during upgrades should be investigated and avoided, and perhaps others as well. Here are the complete list of planned removals. The complete logs is available from the URL above. Note if you want to repeat these tests, that the upgrade test for kde+apt-get hung in the tasksel setup because of dpkg asking conffile questions. No idea why. I worked around it by using 'echo >> /proc/pidofdpkg/fd/0' to tell dpkg to continue.

apt-get gnome 72
bluez-gnome cupsddk-drivers deskbar-applet gnome gnome-desktop-environment gnome-network-admin gtkhtml3.14 iceweasel-gnome-support libavcodec51 libdatrie0 libgdl-1-0 libgnomekbd2 libgnomekbdui2 libmetacity0 libslab0 libxcb-xlib0 nautilus-cd-burner python-gnome2-desktop python-gnome2-extras serpentine swfdec-mozilla update-manager xorg xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-kbd xserver-xorg-input-mouse xserver-xorg-input-synaptics xserver-xorg-input-wacom xserver-xorg-video-all xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-cyrix xserver-xorg-video-dummy xserver-xorg-video-fbdev xserver-xorg-video-glint xserver-xorg-video-i128 xserver-xorg-video-i740 xserver-xorg-video-imstt xserver-xorg-video-intel xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic xserver-xorg-video-nsc xserver-xorg-video-nv xserver-xorg-video-openchrome xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-radeonhd xserver-xorg-video-rendition xserver-xorg-video-s3 xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb xserver-xorg-video-tdfx xserver-xorg-video-tga xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-v4l xserver-xorg-video-vesa xserver-xorg-video-vga xserver-xorg-video-vmware xserver-xorg-video-voodoo xulrunner-1.9 xulrunner-1.9-gnome-support

aptitude gnome 129
bluez-gnome bluez-utils cpp-4.3 cupsddk-drivers dhcdbd djvulibre-desktop finger gnome-app-install gnome-mount gnome-network-admin gnome-spell gnome-vfs-obexftp gnome-volume-manager gstreamer0.10-gnomevfs gtkhtml3.14 libao2 libavahi-compat-libdnssd1 libavahi-core5 libavcodec51 libbluetooth2 libcamel1.2-11 libcdio7 libcucul0 libcupsys2 libcurl3 libdatrie0 libdirectfb-1.0-0 libdvdread3 libedataserver1.2-9 libeel2-2.20 libeel2-data libepc-1.0-1 libepc-ui-1.0-1 libfaad0 libgail-common libgd2-noxpm libgda3-3 libgda3-common libgdl-1-0 libgdl-1-common libggz2 libggzcore9 libggzmod4 libgksu1.2-0 libgksuui1.0-1 libgmyth0 libgnomecups1.0-1 libgnomekbd2 libgnomekbdui2 libgnomeprint2.2-0 libgnomeprint2.2-data libgnomeprintui2.2-0 libgnomeprintui2.2-common libgnomevfs2-bin libgpod3 libgraphviz4 libgtkhtml2-0 libgtksourceview-common libgtksourceview1.0-0 libgucharmap6 libhesiod0 libicu38 libiw29 libkpathsea4 libltdl3 libmagick++10 libmagick10 libmalaga7 libmetacity0 libmtp7 libmysqlclient15off libnautilus-burn4 libneon27 libnm-glib0 libnm-util0 libopal-2.2 libosp5 libparted1.8-10 libpoppler-glib3 libpoppler3 libpt-1.10.10 libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libraw1394-8 libsensors3 libslab0 libsmbios2 libsoup2.2-8 libssh2-1 libsuitesparse-3.1.0 libswfdec-0.6-90 libtalloc1 libtotem-plparser10 libtrackerclient0 libxalan2-java libxalan2-java-gcj libxcb-xlib0 libxerces2-java libxerces2-java-gcj libxklavier12 libxtrap6 libxxf86misc1 libzephyr3 mysql-common nautilus-cd-burner openoffice.org-writer2latex openssl-blacklist p7zip python-4suite-xml python-eggtrayicon python-gnome2-desktop python-gnome2-extras python-gtkhtml2 python-gtkmozembed python-numeric python-sexy serpentine svgalibg1 swfdec-gnome swfdec-mozilla totem-gstreamer update-manager wodim xserver-xorg-video-cyrix xserver-xorg-video-imstt xserver-xorg-video-nsc xserver-xorg-video-v4l xserver-xorg-video-vga zip

apt-get kde 82
cupsddk-drivers karm kaudiocreator kcoloredit kcontrol kde kde-core kdeaddons kdeartwork kdebase kdebase-bin kdebase-bin-kde3 kdebase-kio-plugins kdesktop kdeutils khelpcenter kicker kicker-applets knewsticker kolourpaint konq-plugins konqueror korn kpersonalizer kscreensaver ksplash libavcodec51 libdatrie0 libkiten1 libxcb-xlib0 quanta superkaramba texlive-base-bin xorg xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-kbd xserver-xorg-input-mouse xserver-xorg-input-synaptics xserver-xorg-input-wacom xserver-xorg-video-all xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-cyrix xserver-xorg-video-dummy xserver-xorg-video-fbdev xserver-xorg-video-glint xserver-xorg-video-i128 xserver-xorg-video-i740 xserver-xorg-video-imstt xserver-xorg-video-intel xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic xserver-xorg-video-nsc xserver-xorg-video-nv xserver-xorg-video-openchrome xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-radeonhd xserver-xorg-video-rendition xserver-xorg-video-s3 xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb xserver-xorg-video-tdfx xserver-xorg-video-tga xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-v4l xserver-xorg-video-vesa xserver-xorg-video-vga xserver-xorg-video-vmware xserver-xorg-video-voodoo xulrunner-1.9

aptitude kde 192
bluez-utils cpp-4.3 cupsddk-drivers cvs dcoprss dhcdbd djvulibre-desktop dosfstools eyesapplet fifteenapplet finger gettext ghostscript-x imlib-base imlib11 indi kandy karm kasteroids kaudiocreator kbackgammon kbstate kcoloredit kcontrol kcron kdat kdeadmin-kfile-plugins kdeartwork-misc kdeartwork-theme-window kdebase-bin-kde3 kdebase-kio-plugins kdeedu-data kdegraphics-kfile-plugins kdelirc kdemultimedia-kappfinder-data kdemultimedia-kfile-plugins kdenetwork-kfile-plugins kdepim-kfile-plugins kdepim-kio-plugins kdeprint kdesktop kdessh kdict kdnssd kdvi kedit keduca kenolaba kfax kfaxview kfouleggs kghostview khelpcenter khexedit kiconedit kitchensync klatin klickety kmailcvt kmenuedit kmid kmilo kmoon kmrml kodo kolourpaint kooka korn kpager kpdf kpercentage kpf kpilot kpoker kpovmodeler krec kregexpeditor ksayit ksim ksirc ksirtet ksmiletris ksmserver ksnake ksokoban ksplash ksvg ksysv ktip ktnef kuickshow kverbos kview kviewshell kvoctrain kwifimanager kwin kwin4 kworldclock kxsldbg libakode2 libao2 libarts1-akode libarts1-audiofile libarts1-mpeglib libarts1-xine libavahi-compat-libdnssd1 libavahi-core5 libavc1394-0 libavcodec51 libbluetooth2 libboost-python1.34.1 libcucul0 libcurl3 libcvsservice0 libdatrie0 libdirectfb-1.0-0 libdjvulibre21 libdvdread3 libfaad0 libfreebob0 libgail-common libgd2-noxpm libgraphviz4 libgsmme1c2a libgtkhtml2-0 libicu38 libiec61883-0 libindex0 libiw29 libk3b3 libkcal2b libkcddb1 libkdeedu3 libkdepim1a libkgantt0 libkiten1 libkleopatra1 libkmime2 libkpathsea4 libkpimexchange1 libkpimidentities1 libkscan1 libksieve0 libktnef1 liblockdev1 libltdl3 libmagick10 libmimelib1c2a libmozjs1d libmpcdec3 libneon27 libnm-util0 libopensync0 libpisock9 libpoppler-glib3 libpoppler-qt2 libpoppler3 libraw1394-8 libsmbios2 libssh2-1 libsuitesparse-3.1.0 libtalloc1 libtiff-tools libxalan2-java libxalan2-java-gcj libxcb-xlib0 libxerces2-java libxerces2-java-gcj libxtrap6 mpeglib networkstatus openoffice.org-writer2latex pmount poster psutils quanta quanta-data superkaramba svgalibg1 tex-common texlive-base texlive-base-bin texlive-common texlive-doc-base texlive-fonts-recommended xserver-xorg-video-cyrix xserver-xorg-video-imstt xserver-xorg-video-nsc xserver-xorg-video-v4l xserver-xorg-video-vga xulrunner-1.9

The last few days I have done some upgrade testing in Debian, to see if the upgrade from Lenny to Squeeze will go smoothly. A few bugs have been discovered and reported in the process (#585410 in nagios3-cgi, #584879 already fixed in enscript and #584861 in kdebase-workspace-data), and to get a more regular testing going on, I am working on a script to automate the test.

The idea is to create a Lenny chroot and use tasksel to install a Gnome or KDE desktop installation inside the chroot before upgrading it. To ensure no services are started in the chroot, a policy-rc.d script is inserted. To make sure tasksel believe it is to install a desktop on a laptop, the tasksel tests are replaced in the chroot (only acceptable because this is a throw-away chroot).

A naive upgrade from Lenny to Squeeze using aptitude dist-upgrade currently always fail because udev refuses to upgrade with the kernel in Lenny, so to avoid that problem the file /etc/udev/kernel-upgrade is created. The bug report #566000 make me suspect this problem do not trigger in a chroot, but I touch the file anyway to make sure the upgrade go well. Testing on virtual and real hardware have failed me because of udev so far, and creating this file do the trick in such settings anyway. This is a known issue and the current udev behaviour is intended by the udev maintainer because he lack the resources to rewrite udev to keep working with old kernels or something like that. I really wish the udev upstream would keep udev backwards compatible, to avoid such upgrade problem, but given that they fail to do so, I guess documenting the way out of this mess is the best option we got for Debian Squeeze.

Anyway, back to the task at hand, testing upgrades. This test script, which I call upgrade-test for now, is doing the trick:

#!/bin/sh
set -ex

if [ "$1" ] ; then
    desktop=$1
else
    desktop=gnome
fi

from=lenny
to=squeeze

exec < /dev/null
unset LANG
mirror=http://ftp.skolelinux.org/debian
tmpdir=chroot-$from-upgrade-$to-$desktop
fuser -mv .
debootstrap $from $tmpdir $mirror
chroot $tmpdir aptitude update
cat > $tmpdir/usr/sbin/policy-rc.d <<EOF
#!/bin/sh
exit 101
EOF
chmod a+rx $tmpdir/usr/sbin/policy-rc.d
exit_cleanup() {
    umount $tmpdir/proc
}
mount -t proc proc $tmpdir/proc
# Make sure proc is unmounted also on failure
trap exit_cleanup EXIT INT

chroot $tmpdir aptitude -y install debconf-utils

# Make sure tasksel autoselection trigger.  It need the test scripts
# to return the correct answers.
echo tasksel tasksel/desktop multiselect $desktop | \
    chroot $tmpdir debconf-set-selections

# Include the desktop and laptop task
for test in desktop laptop ; do
    echo > $tmpdir/usr/lib/tasksel/tests/$test <<EOF
#!/bin/sh
exit 2
EOF
    chmod a+rx $tmpdir/usr/lib/tasksel/tests/$test
done

DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical
export DEBIAN_FRONTEND DEBIAN_PRIORITY
chroot $tmpdir tasksel --new-install

echo deb $mirror $to main > $tmpdir/etc/apt/sources.list
chroot $tmpdir aptitude update
touch $tmpdir/etc/udev/kernel-upgrade
chroot $tmpdir aptitude -y dist-upgrade
fuser -mv

I suspect it would be useful to test upgrades with both apt-get and with aptitude, but I have not had time to look at how they behave differently so far. I hope to get a cron job running to do the test regularly and post the result on the web. The Gnome upgrade currently work, while the KDE upgrade fail because of the bug in kdebase-workspace-data

I am not quite sure what kind of extract from the huge upgrade logs (KDE 167 KiB, Gnome 516 KiB) it make sense to include in this blog post, so I will refrain from trying. I can report that for Gnome, aptitude report 760 packages upgraded, 448 newly installed, 129 to remove and 1 not upgraded and 1024MB need to be downloaded while for KDE the same numbers are 702 packages upgraded, 507 newly installed, 193 to remove and 0 not upgraded and 1117MB need to be downloaded

I am very happy to notice that the Gnome desktop + laptop upgrade is able to migrate to dependency based boot sequencing and parallel booting without a hitch. Was unsure if there were still bugs with packages failing to clean up their obsolete init.d script during upgrades, and no such problem seem to affect the Gnome desktop+laptop packages.

If Debian is to migrate to upstart on Linux, I expect some init.d scripts to migrate (some of) their operations to upstart job while keeping the init.d for hurd and kfreebsd. The packages with such needs will need a way to get their init.d scripts to behave differently when used with sysvinit and with upstart. Because of this, I had a look at the environment variables set when a init.d script is running under upstart, and when it is not.

With upstart, I notice these environment variables are set when a script is started from rcS.d/ (ignoring some irrelevant ones like COLUMNS):

DEFAULT_RUNLEVEL=2
previous=N
PREVLEVEL=
RUNLEVEL=
runlevel=S
UPSTART_EVENTS=startup
UPSTART_INSTANCE=
UPSTART_JOB=rc-sysinit

With sysvinit, these environment variables are set for the same script.

INIT_VERSION=sysvinit-2.88
previous=N
PREVLEVEL=N
RUNLEVEL=S
runlevel=S

The RUNLEVEL and PREVLEVEL environment variables passed on from sysvinit are not set by upstart. Not sure if it is intentional or not to not be compatible with sysvinit in this regard.

For scripts needing to behave differently when upstart is used, looking for the UPSTART_JOB environment variable seem to be a good choice.

Via the blog of Rob Weir I came across the very interesting essay named The Art of Standards Wars (PDF 25 pages). I recommend it for everyone following the standards wars of today.

When using sitesummary at a site to track machines, it is possible to get a list of the machine types in use thanks to the DMI information extracted from each machine. The script to do so is included in the sitesummary package, and here is example output from the Skolelinux build servers:

maintainer:~# /usr/lib/sitesummary/hardware-model-summary
  vendor                    count
  Dell Computer Corporation     1
    PowerEdge 1750              1
  IBM                           1
    eserver xSeries 345 -[8670M1X]-     1
  Intel                         2
  [no-dmi-info]                 3
maintainer:~#

The quality of the report depend on the quality of the DMI tables provided in each machine. Here there are Intel machines without model information listed with Intel as vendor and no model, and virtual Xen machines listed as [no-dmi-info]. One can add -l as a command line option to list the individual machines.

A larger list is available from the the city of Narvik, which uses Skolelinux on all their shools and also provide the basic sitesummary report publicly. In their report there are ~1400 machines. I know they use both Ubuntu and Skolelinux on their machines, and as sitesummary is available in both distributions, it is trivial to get all of them to report to the same central collector.

It is strange to watch how a bug in Debian causing KDM to fail to start at boot when an NVidia video card is used is handled. The problem seem to be that the nvidia X.org driver uses a long time to initialize, and this duration is longer than kdm is configured to wait.

I came across two bugs related to this issue, #583312 initially filed against initscripts and passed on to nvidia-glx when it became obvious that the nvidia drivers were involved, and #524751 initially filed against kdm and passed on to src:nvidia-graphics-drivers for unknown reasons.

To me, it seem that no-one is interested in actually solving the problem nvidia video card owners experience and make sure the Debian distribution work out of the box for these users. The nvidia driver maintainers expect kdm to be set up to wait longer, while kdm expect the nvidia driver maintainers to fix the driver to start faster, and while they wait for each other I guess the users end up switching to a distribution that work for them. I have no idea what the solution is, but I am pretty sure that waiting for each other is not it.

I wonder why we end up handling bugs this way.

A few days ago, parallel booting was enabled in Debian/testing. The feature seem to hold up pretty well, but three fairly serious issues are known and should be solved:

• The wicd package seen to break NFS mounting and network setup when parallel booting is enabled. No idea why, but the wicd maintainer seem to be on the case.
• The nvidia X driver seem to have a race condition triggered more easily when parallel booting is in effect. The maintainer is on the case.
• The sysv-rc package fail to properly enable dependency based boot sequencing (the shutdown is broken) when old file-rc users try to switch back to sysv-rc. One way to solve it would be for file-rc to create /etc/init.d/.legacy-bootordering, and another is to try to make sysv-rc more robust. Will investigate some more and probably upload a workaround in sysv-rc to help those trying to move from file-rc to sysv-rc get a working shutdown.

All in all not many surprising issues, and all of them seem solvable before Squeeze is released. In addition to these there are some packages with bugs in their dependencies and run level settings, which I expect will be fixed in a reasonable time span.

If you report any problems with dependencies in init.d scripts to the BTS, please usertag the report to get it to show up at the list of usertagged bugs related to this.

Update: Correct bug number to file-rc issue.

After a long break from debian-installer development, I finally found time today to return to the project. Having to spend less time working dependency based boot in debian, as it is almost complete now, definitely helped freeing some time.

A while back, I ran into a problem while working on Debian Edu. We include some firmware packages on the Debian Edu CDs, those needed to get disk and network controllers working. Without having these firmware packages available during installation, it is impossible to install Debian Edu on the given machine, and because our target group are non-technical people, asking them to provide firmware packages on an external medium is a support pain. Initially, I expected it to be enough to include the firmware packages on the CD to get debian-installer to find and use them. This proved to be wrong. Next, I hoped it was enough to symlink the relevant firmware packages to some useful location on the CD (tried /cdrom/ and /cdrom/firmware/). This also proved to not work, and at this point I found time to look at the debian-installer code to figure out what was going to work.

The firmware loading code is in the hw-detect package, and a closer look revealed that it would only look for firmware packages outside the installation media, so the CD was never checked for firmware packages. It would only check USB sticks, floppies and other "external" media devices. Today I changed it to also look in the /cdrom/firmware/ directory on the mounted CD or DVD, which should solve the problem I ran into with Debian edu. I also changed it to look in /firmware/, to make sure the installer also find firmware provided in the initrd when booting the installer via PXE, to allow us to provide the same feature in the PXE setup included in Debian Edu.

To make sure firmware deb packages with a license questions are not activated without asking if the license is accepted, I extended hw-detect to look for preinst scripts in the firmware packages, and run these before activating the firmware during installation. The license question is asked using debconf in the preinst, so this should solve the issue for the firmware packages I have looked at so far.

If you want to discuss the details of these features, please contact us on debian-boot@lists.debian.org.

Since this evening, parallel booting is the default in Debian/unstable for machines using dependency based boot sequencing. Apparently the testing of concurrent booting has been wider than expected, if I am to believe the input on debian-devel@, and I concluded a few days ago to move forward with the feature this weekend, to give us some time to detect any remaining problems before Squeeze is frozen. If serious problems are detected, it is simple to change the default back to sequential boot. The upload of the new sysvinit package also activate a new upstream version.

More information about dependency based boot sequencing is available from the Debian wiki. It is currently possible to disable parallel booting when one run into problems caused by it, by adding this line to /etc/default/rcS:

CONCURRENCY=none

If you report any problems with dependencies in init.d scripts to the BTS, please usertag the report to get it to show up at the list of usertagged bugs related to this.

In the recent Debian Edu versions, the sitesummary system is used to keep track of the machines in the school network. Each machine will automatically report its status to the central server after boot and once per night. The network setup is also reported, and using this information it is possible to get the MAC address of all network interfaces in the machines. This is useful to update the DHCP configuration.

To give some idea how to use sitesummary, here is a one-liner to ist all MAC addresses of all machines reporting to sitesummary. Run this on the collector host:

perl -MSiteSummary -e 'for_all_hosts(sub { print join(" ", get_macaddresses(shift)), "\n"; });'

This will list all MAC addresses assosiated with all machine, one line per machine and with space between the MAC addresses.

To allow system administrators easier job at adding static DHCP addresses for hosts, it would be possible to extend this to fetch machine information from sitesummary and update the DHCP and DNS tables in LDAP using this information. Such tool is unfortunately not written yet.

The last few days a new boot system called systemd has been introduced to the free software world. I have not yet had time to play around with it, but it seem to be a very interesting alternative to upstart, and might prove to be a good alternative for Debian when we are able to switch to an event based boot system. Tollef is in the process of getting systemd into Debian, and I look forward to seeing how well it work. I like the fact that systemd handles init.d scripts with dependency information natively, allowing them to run in parallel where upstart at the moment do not.

Unfortunately do systemd have the same problem as upstart regarding platform support. It only work on recent Linux kernels, and also need some new kernel features enabled to function properly. This means kFreeBSD and Hurd ports of Debian will need a port or a different boot system. Not sure how that will be handled if systemd proves to be the way forward.

In the mean time, based on the input on debian-devel@ regarding parallel booting in Debian, I have decided to enable full parallel booting as the default in Debian as soon as possible (probably this weekend or early next week), to see if there are any remaining serious bugs in the init.d dependencies. A new version of the sysvinit package implementing this change is already in experimental. If all go well, Squeeze will be released with parallel booting enabled by default.

These days, the init.d script dependencies in Squeeze are quite complete, so complete that it is actually possible to run all the init.d scripts in parallell based on these dependencies. If you want to test your Squeeze system, make sure dependency based boot sequencing is enabled, and add this line to /etc/default/rcS:

CONCURRENCY=makefile

That is it. It will cause sysv-rc to use the startpar tool to run scripts in parallel using the dependency information stored in /etc/init.d/.depend.boot, /etc/init.d/.depend.start and /etc/init.d/.depend.stop to order the scripts. Startpar is configured to try to start the kdm and gdm scripts as early as possible, and will start the facilities required by kdm or gdm as early as possible to make this happen.

Give it a try, and see if you like the result. If some services fail to start properly, it is most likely because they have incomplete init.d script dependencies in their startup script (or some of their dependent scripts have incomplete dependencies). Report bugs and get the package maintainers to fix it. :)

Running scripts in parallel could be the default in Debian when we manage to get the init.d script dependencies complete and correct. I expect we will get there in Squeeze+1, if we get manage to test and fix the remaining issues.

If you report any problems with dependencies in init.d scripts to the BTS, please usertag the report to get it to show up at the list of usertagged bugs related to this.

Since this evening, with the upload of sysvinit version 2.87dsf-2, and the upload of insserv version 1.12.0-10 yesterday, Debian unstable have been migrated to using dependency based boot sequencing. This conclude work me and others have been doing for the last three days. It feels great to see this finally part of the default Debian installation. Now we just need to weed out the last few problems that are bound to show up, to get everything ready for Squeeze.

The next step is migrating /sbin/init from sysvinit to upstart, and fixing the more fundamental problem of handing the event based non-predictable kernel in the early boot.

After several years of frustration with the lack of activity from the existing sysvinit upstream developer, I decided a few weeks ago to take over the package and become the new upstream. The number of patches to track for the Debian package was becoming a burden, and the lack of synchronization between the distribution made it hard to keep the package up to date.

On the new sysvinit team is the SuSe maintainer Dr. Werner Fink, and my Debian co-maintainer Kel Modderman. About 10 days ago, I made a new upstream tarball with version number 2.87dsf (for Debian, SuSe and Fedora), based on the patches currently in use in these distributions. We Debian maintainers plan to move to this tarball as the new upstream as soon as we find time to do the merge. Since the new tarball was created, we agreed with Werner at SuSe to make a new upstream project at Savannah, and continue development there. The project is registered and currently waiting for approval by the Savannah administrators, and as soon as it is approved, we will import the old versions from svn and continue working on the future release.

It is a bit ironic that this is done now, when some of the involved distributions are moving to upstart as a syvinit replacement.

I spent Monday and tuesday this week in London with a lot of the people involved in the boot system on Debian and Ubuntu, to see if we could find more ways to speed up the boot system. This was an Ubuntu funded developer gathering. It was quite productive. We also discussed the future of boot systems, and ways to handle the increasing number of boot issues introduced by the Linux kernel becoming more and more asynchronous and event base. The Ubuntu approach using udev and upstart might be a good way forward. Time will show.

Anyway, there are a few ways at the moment to speed up the boot process in Debian. All of these should be applied to get a quick boot:

• Use dash as /bin/sh.
• Disable the init.d/hwclock*.sh scripts and make sure the hardware clock is in UTC.
• Install and activate the insserv package to enable dependency based boot sequencing, and enable concurrent booting.

These points are based on the Google summer of code work done by Carlos Villegas.

Support for makefile-style concurrency during boot was uploaded to unstable yesterday. When we tested it, we were able to cut 6 seconds from the boot sequence. It depend on very correct dependency declaration in all init.d scripts, so I expect us to find edge cases where the dependences in some scripts are slightly wrong when we start using this.

On our IRC channel for this effort, #pkg-sysvinit, a new idea was introduced by Raphael Geissert today, one that could affect the startup speed as well. Instead of starting some scripts concurrently from rcS.d/ and another set of scripts from rc2.d/, it would be possible to run a of them in the same process. A quick way to test this would be to enable insserv and run 'mv /etc/rc2.d/S* /etc/rcS.d/; insserv'. Will need to test if that work. :)

Hvert år de siste årene har BSA, lobbyfronten til de store programvareselskapene som Microsoft og Apple, publisert en rapport der de gjetter på hvor mye piratkopiering påfører i tapte inntekter i ulike land rundt om i verden. Resultatene er tendensiøse. For noen dager siden kom siste rapport, og det er flere kritiske kommentarer publisert de siste dagene. Et spesielt interessant kommentar fra Sverige, BSA höftade Sverigesiffror, oppsummeres slik:

I sin senaste rapport slår BSA fast att 25 procent av all mjukvara i Sverige är piratkopierad. Det utan att ha pratat med ett enda svenskt företag. "Man bör nog kanske inte se de här siffrorna som helt exakta", säger BSAs Sverigechef John Hugosson.

Mon tro om de er like metodiske når de gjetter på andelen piratkopiering i Norge? To andre kommentarer er BSA piracy figures need a shot of reality og Does The WIPO Copyright Treaty Work?

Fant lenkene via oppslag på Slashdot.

Kom over interessante tall fra IDG om utviklingen av linuxservermarkedet. Fikk meg til å tenke på antall tjenermaskiner ved Universitetet i Oslo der jeg jobber til daglig. En rask opptelling forteller meg at vi har 490 (61%) fysiske unix-tjener (mest linux men også noen solaris) og 196 (25%) windowstjenere, samt 112 (14%) virtuelle unix-tjenere. Med den bakgrunnskunnskapen kan jeg godt tro at IDG er inne på noe.

Dagens IT melder at Intel hevder at det er dyrt å miste en datamaskin, når en tar tap av arbeidstid, fortrolige dokumenter, personopplysninger og alt annet det innebærer. Det er ingen tvil om at det er en kostbar affære å miste sin datamaskin, og det er årsaken til at jeg har kryptert harddisken på både kontormaskinen og min bærbare. Begge inneholder personopplysninger jeg ikke ønsker skal komme på avveie, den første informasjon relatert til jobben min ved Universitetet i Oslo, og den andre relatert til blant annet foreningsarbeide. Kryptering av diskene gjør at det er lite sannsynlig at dophoder som kan finne på å rappe maskinene får noe ut av dem. Maskinene låses automatisk etter noen minutter uten bruk, og en reboot vil gjøre at de ber om passord før de vil starte opp. Jeg bruker Debian på begge maskinene, og installasjonssystemet der gjør det trivielt å sette opp krypterte disker. Jeg har LVM på toppen av krypterte partisjoner, slik at alt av datapartisjoner er kryptert. Jeg anbefaler alle å kryptere diskene på sine bærbare. Kostnaden når det er gjort slik jeg gjør det er minimale, og gevinstene er betydelige. En bør dog passe på passordet. Hvis det går tapt, må maskinen reinstalleres og alt er tapt.

Krypteringen vil ikke stoppe kompetente angripere som f.eks. kjøler ned minnebrikkene før maskinen rebootes med programvare for å hente ut krypteringsnøklene. Kostnaden med å forsvare seg mot slike angripere er for min del høyere enn gevinsten. Jeg tror oddsene for at f.eks. etteretningsorganisasjoner har glede av å titte på mine maskiner er minimale, og ulempene jeg ville oppnå ved å forsøke å gjøre det vanskeligere for angripere med kompetanse og ressurser er betydelige.

There are two software projects that have had huge influence on the quality of free software, and I wanted to mention both in case someone do not yet know them.

The first one is valgrind, a tool to detect and expose errors in the memory handling of programs. It is easy to use, all one need to do is to run 'valgrind program', and it will report any problems on stdout. It is even better if the program include debug information. With debug information, it is able to report the source file name and line number where the problem occurs. It can report things like 'reading past memory block in file X line N, the memory block was allocated in file Y, line M', and 'using uninitialised value in control logic'. This tool has made it trivial to investigate reproducible crash bugs in programs, and have reduced the number of this kind of bugs in free software a lot.

The second one is Coverity which is a source code checker. It is able to process the source of a program and find problems in the logic without running the program. It started out as the Stanford Checker and became well known when it was used to find bugs in the Linux kernel. It is now a commercial tool and the company behind it is running a community service for the free software community, where a lot of free software projects get their source checked for free. Several thousand defects have been found and fixed so far. It can find errors like 'lock L taken in file X line N is never released if exiting in line M', or 'the code in file Y lines O to P can never be executed'. The projects included in the community service project have managed to get rid of a lot of reliability problems thanks to Coverity.

I believe tools like this, that are able to automatically find errors in the source, are vital to improve the quality of software and make sure we can get rid of the crashing and failing software we are surrounded by today.

Julien Blache claim that no patch is better than a useless patch. I completely disagree, as a patch allow one to discuss a concrete and proposed solution, and also prove that the issue at hand is important enough for someone to spent time on fixing it. No patch do not provide any of these positive properties.

Where I work at the University of Oslo, one decision stand out as a very good one to form a long lived computer infrastructure. It is the simple one, lost by many in todays computer industry: Standardize on open network protocols and open exchange/storage formats, not applications. Applications come and go, while protocols and files tend to stay, and thus one want to make it easy to change application and vendor, while avoiding conversion costs and locking users to a specific platform or application.

This approach make it possible to replace the client applications independently of the server applications. One can even allow users to use several different applications as long as they handle the selected protocol and format. In the normal case, only one client application is recommended and users only get help if they choose to use this application, but those that want to deviate from the easy path are not blocked from doing so.

It also allow us to replace the server side without forcing the users to replace their applications, and thus allow us to select the best server implementation at any moment, when scale and resouce requirements change.

I strongly recommend standardizing - on open network protocols and open formats, but I would never recommend standardizing on a single application that do not use open network protocol or open formats.

I'm sitting on the train going home from this weekends Debian Edu/Skolelinux development gathering. I got a bit done tuning the desktop, and looked into the dynamic service location protocol implementation avahi. It look like it could be useful for us. Almost 30 people participated, and I believe it was a great environment to get to know the Skolelinux system. Walter Bender, involved in the development of the Sugar educational platform, presented his stuff and also helped me improve my OLPC installation. He also showed me that his Turtle Art application can be used in standalone mode, and we agreed that I would help getting it packaged for Debian. As a standalone application it would be great for Debian Edu. We also tried to get the video conferencing working with two OLPCs, but that proved to be too hard for us. The application seem to need more work before it is ready for me. I look forward to getting home and relax now. :)

The state of standardized LDAP schemas on Linux is far from optimal. There is RFC 2307 documenting one way to store NIS maps in LDAP, and a modified version of this normally called RFC 2307bis, with some modifications to be compatible with Active Directory. The RFC specification handle the content of a lot of system databases, but do not handle DNS zones and DHCP configuration.

In Debian Edu/Skolelinux, we would like to store information about users, SMB clients/hosts, filegroups, netgroups (users and hosts), DHCP and DNS configuration, and LTSP configuration in LDAP. These objects have a lot in common, but with the current LDAP schemas it is not possible to have one object per entity. For example, one need to have at least three LDAP objects for a given computer, one with the SMB related stuff, one with DNS information and another with DHCP information. The schemas provided for DNS and DHCP are impossible to combine into one LDAP object. In addition, it is impossible to implement quick queries for netgroup membership, because of the way NIS triples are implemented. It just do not scale. I believe it is time for a few RFC specifications to cleam up this mess.

I would like to have one LDAP object representing each computer in the network, and this object can then keep the SMB (ie host key), DHCP (mac address/name) and DNS (name/IP address) settings in one place. It need to be efficently stored to make sure it scale well.

I would also like to have a quick way to map from a user or computer and to the net group this user or computer is a member.

Active Directory have done a better job than unix heads like myself in this regard, and the unix side need to catch up. Time to start a new IETF work group?

Endelig er Debian Lenny gitt ut. Et langt steg videre for Debian-prosjektet, og en rekke nye programpakker blir nå tilgjengelig for de av oss som bruker den stabile utgaven av Debian. Neste steg er nå å få Skolelinux / Debian Edu ferdig oppdatert for den nye utgaven, slik at en oppdatert versjon kan slippes løs på skolene. Takk til alle debian-utviklerne som har gjort dette mulig. Endelig er f.eks. fungerende avhengighetsstyrt bootsekvens tilgjengelig i stabil utgave, vha pakken insserv.

This weekend we had a small developer gathering for Debian Edu in Oslo. Most of Saturday was used for the general assemly for the member organization, but the rest of the weekend I used to tune the LTSP installation. LTSP now work out of the box on the 10-network. Acer Aspire One proved to be a very nice thin client, with both screen, mouse and keybard in a small box. Was working on getting the diskless workstation setup configured out of the box, but did not finish it before the weekend was up.

Did not find time to look at the 4 VGA cards in one box we got from the Brazilian group, so that will have to wait for the next development gathering. Would love to have the Debian Edu installer automatically detect and configure a multiseat setup when it find one of these cards.

Recently I have spent some time evaluating the multimedia browser plugins available in Debian Lenny, to see which one we should use by default in Debian Edu. We need an embedded video playing plugin with control buttons to pause or stop the video, and capable of streaming all the multimedia content available on the web. The test results and notes are available on the Debian wiki. I was surprised how few of the plugins are able to fill this need. My personal video player favorite, VLC, has a really bad plugin which fail on a lot of the test pages. A lot of the MIME types I would expect to work with any free software player (like video/ogg), just do not work. And simple formats like the audio/x-mplegurl format (m3u playlists), just isn't supported by the totem and vlc plugins. I hope the situation will improve soon. No wonder sites use the proprietary Adobe flash to play video.

For Lenny, we seem to end up with the mplayer plugin. It seem to be the only one fitting our needs. :/