Blog entries published in 2009
Feeds: RSS | Atom

Deleting Amazon S3 buckets using Python

Published: 2009-08-09 10:38 UTC. Tags: software misctools

For a while, I used Duplicity to make backups to an Amazon S3 bucket. That kind of worked, but I had to do a lot of scripting myself to get it working automatically, so after finding out about Jungledisk, I switched to that. Jungledisk has a nice little desktop applet that keeps track of doing my backups while my computer is on, etc. That's convenient.

Anyway, the Duplicity/S3 experiments left me with an Amazon S3 bucket with about 9000 objects. Getting rid of that proved to be something of a challenge - you have to delete all objects inside the bucket before you can delete the bucket itself, and there's no API call for doing that. I also tried the web application for managing buckets, S3FM but that didn't cope too well with that many objects - my web browser just hung.

I have to admit I could have put more effort into googling before solving it by writing my own script - but writing my own script was more fun :-).

My script managed to delete all 9000 objects without trouble, although it did take quite a while to complete - I let it run overnight.

If you need to do the same thing, it's available here:

http://github.com/forsberg/misctools/tree/0d6cbd189b80e52501d94742fa04bba36b7f34a6/S3

StackOverflow has several other solutions: http://stackoverflow.com/questions/27267/delete-amazon-s3-buckets

0 comments.

Playing with a SkyTraq Venus GPS logger under Linux

Published: 2009-07-28 21:40 UTC. Tags: linux gis

I got a new toy today - a GPS logger. Got it as part of a magazine subscription deal - two issues + GPS logger for about $12. That magazine was not my kind of magazine, but getting a GPS logger delivered home for $12 was worth having to throw away two magazine issues :-). For other people trying to get this to work under Linux, the magazine in question was Aktiv Träning.

After some fiddling, I came to the conclusion that the device is based on the SkyTraq Venus chipset:

skytraq: Venus device found: Kernel version = 1.3.3, ODM version = 1.4.5, revision (Y/M/D) = 07/12/11

After plugging the device into an USB port, it appears as a serial port, in my case /dev/ttyUSB0. The data can be retrieved using GPSBabel, but it has to be the development version - the stable version does not feature the skytraq driver required. The development version is available via CVS from sourceforge, see http://sourceforge.net/projects/gpsbabel/develop for details on how to access CVS.

Once you have the development version of gpsbabel, getting the data is as easy as:

./gpsbabel -D 9 -i skytraq,initbaud=38400,baud=38400,erase -f /dev/ttyUSB0 -o gpx -F out.gpx

This will write the GPS coordinate log to out.gpx in the GPS Exchange format - a lightweight XML format.

The above command will also erase the log from the device, so the next download contains just the latest log.

Since I like playing with Google Earth, I also learned that the GPX can be converted into KML - the format Google Earth can read, by running:

./gpsbabel -i gpx -f out.gpx -o kml -F out.kml

Just open out.kml with Google Earth, and you'll see where the GPS logger has been.

On popular request, here's a few images of the device.

Skytraq Venus device Inside the battery compartment of a Skytraq Venus device

The label inside the battery compartment has the following text:

GT-750F/L-Lite
GPS Receiver(Data Logger)
RoHS CE FCC ID:VHP-750F
Made in Taiwan CANMORE
0 comments.

Calculating the Distance of a GPS log

Published: 2009-07-28 21:20 UTC. Tags: linux gis

Playing with my GPS logger, I wanted to know the distance travelled between all the points in a GPS log. For example, if I went for a bicycle trip and brought the logger, it would be nice to know how many kilometers I have travelled.

Trying to find a tool that did this without being cumbersome to use proved more difficult than I thought, so I did a quick hack with the help of the python bindings for gpsd. It's a very simple script:

$ ./distance.py promenad2.gpx
427.077378454

Given a GPX file, it will print out the number of meters travelled.

Fetch it in the fgpstools repository on github

0 comments.

Simple Parallelization with xargs

Published: 2009-05-30 20:46 UTC. Tags: parallelization

If you run a modern computer, chances are high that you have a multi-core processor. That forces us into paralell programming to get more performance, which is much harder than doing serial programming.

The operating system and existing shell commands can do much for us, though. Let's say you have a bunch of files where each should be processed by a command. Of course, with multiple cores, this could be sped up by running multiple instances of the command simultaneously. And there's a very easy way to do that - xargs. An example for converting raw files into jpeg:

find . -type f -name '\*.CR2' \|xargs -n 1 -P 3 ~/bin/raw2jpeg

This will find all files named *.CR2 and feed that list to xargs, which will run up to three (-P 3) processes simultaneously, giving each command one filename (-n 1) on its commandline.

0 comments.

Norwegian Wifi

Published: 2009-04-16 19:14 UTC. Tags: network

The hotel where I'm staying in Oslo has a very special kind of Wireless Internet. It's the kind where you bring your laptop out in the corridor, connect to the network, and then bring the network with you into your room.

Connecting directly from the room doesn't work, even though there's plenty of signal. Well, at least sometimes, there's plenty of signal. Tonight there is, yesterday night I couldn't see the net.

Must be some norweigan special setup.

0 comments.

Noisiest fridge in norther europe?

Published: 2009-04-14 18:29 UTC. Tags: travel
A norwegian refrigerator

The fridge on the picture is probably the most noisy fridge in northern Europe. You'll find it in one of the rooms at Cochs Pensjonat, Oslo. It spends its days, and especially nights, starting with a loud bang about once every 15 minutes, which very effectively wakes people up if they happen to sleep in the room, something people tend to do in hotel rooms.

Oh, how I love to fill the internet with worthless bullshit!

0 comments.

Restart WWW

Published: 2009-04-13 16:31 UTC. Tags: world wide web

I'm on a bus4you with destination Oslo for a visit to Opera Software's main offices. The bus has wireless internet. That's good, I've been reading webpages on map-reduce, distributed filesystems and other interesting stuff the whole trip.

The bus driver mentioned that the network sometimes stops working, and requires a restart when that happens. It happened, and after confirming with one of my co-passenges that they too had trouble, I went to tell the driver.

No problem, he said, I'll restart it. And to restart it, he pressed a button marked "WWW".

So, if the WWW doesn't work for you, now you know where to find the button with which it can be restarted - just find the nearest bus4you bus! :-)

1 comments.

I'm not a worm broadcaster!

Published: 2009-02-28 19:24 UTC. Tags: photography
Bear at Kolmårdens Djurpark
You talkin' to me? A Blog Challenge? Eh? (Bear, Kolmården Zoo, 2005)
I have received a "challenge". Apparently, that's something you do in the blog world. At least, that's something you do on less nerdy and more frequently updated blogs than this one.. ;-)

I'm supposed to take the sixth image in the sixth folder in my photo collection, post it here with a comment, and then challenge six other people to do the same thing.

But.. that's an Internet Worm. Naturally, being a responsible computer scientist, I can't help spreading a virus! This blog post is an antivirus.

Still, I'm more than happy to show you, dear reader, some of my pictures. I'll even write a comment for each and every one of them, following the rules of the challenge to some point..

Boats, Nessebar, Bulgaria
The owner of these boats obviously found a theme. Nessebar Bulgaria. Don't go there, it's a horrible touristish place
Sunroofs, Nessebar, Bulgaria
Sunshades in the morning. Again, this is from Nessebar, Bulgaria, where you should not go since it's full of tourists and restaurants that all have EVERYTHING as their speciality. But, if you sneak up at 0600 in the morning and spend an hour or so taking photos, you can find some nice angles..
Student with makeup, Student
       Orchestra Festival, Linköping, 2005
SOF Linköping, 2005 (Student Orchestra Festival).
Horse
This horse was having lot's of fun, rolling around on the ground. But now it's decided it's time for some serious.. running around all over the place!

Bah.. getting tired of finding/scaling images. I could keep on for a while, with over 11000 images in my image database..

This will have to do. Not spreading this virus further by challenging any other bloggers, at the same time maintaing a high level of nerdyness!

1 comments.

Are happy programmers dangerous?

Published: 2009-02-26 21:18 UTC. Tags: software

I went to a seminar about Scrum today at ENEA. It was one of those "let's have a seminar and then give people free food and beer so they buy more consultants from us" type of events.

Even more interesting was the questions after the seminar. Lot's of people from different tech companies in Linköping. Someone said that Scrum kept the programmers happy which would produce better code. That's probably true. Here comes the fun part - another person in the audience were worried that happy programmers would code things they thought were fun instead of the things they were supposed to do.

Hmm.. yeah, right. That's the way it works. Or maybe not! I'd say that the risk is much bigger that bored programmers spend their time working at things they shouldn't do.

I would really like to know where this person works, so I can avoid working there.

1 comments.

ATI RS690+dualhead+resume from disk

Published: 2009-02-12 17:23 UTC. Tags: hardware linux

The ATI fglrx driver I run against my ATI RS690 chip in my dualhead setup definitely has its quirks. When resuming from disk, both monitors must be turned on when I press the computer's power button, or the X server will behave very odd.

Hard to describe what happens, but when moving the mouse, the whole desktop image is moved instead of only the mouse pointer.

Hardware! I hate hardware! I want all of it to be emulated in software.

1 comments.

Why I left Plone - Part 1, the Learning Curve

Published: 2009-02-07 11:44 UTC. Tags: plone noplone

For a long time, I was a Plone user and developer. I tried hard to keep myself convinced that Plone was the solution to many web-related problems, not only strictly Content Management-related ones, but all sorts of web applications.

A while ago, I realized that I was, just that - trying hard to keep myself convinced. That is really not a good sign when it comes to frameworks you use in your programming - if you're to work with them and get productive, you have to like them because they actually are good, and fun, to work with. This is not the case with Plone, and I'll try to explain why.

After starting to write this, I realized that I have so many reasons to leave Plone that this will have to be the first in a series of posts. Stay tuned!

The Learning Curve

I started using Plone while looking for a good solution for the external website of the company I was then working for. After some evaluation of different alternatives, I decided on Plone. Plone was then at version 2.0.6, if I recall correctly.

We hired a summer intern with some previous Plone experience with the task of doing different kinds of integration with our existing systems, including some LDAP integration. I remember that he was spending lots of time cursing especially the LDAP integration and the user management code.

About the time when the integration were finished and we began thinking about deplying, Plone 2.1 was released.

Explosion

KABOOM - an explosion of new things to learn!

Plone 2.1 was like an entirely new piece of software. Lot's and lot's and lot's of things had changed. At least that was the impression you got if you were not a Plone Guru.

I made the decision of going for Plone 2.1, since I didn't want us to be left behind with incompatible Products (that's the Plone term for more or less useful addons that can be downloaded and installed in a Plone instance - I'll probably write something about them later..). So, lot's of time had to be spent again trying to fixup the site into a usable state. The LDAP integration we hoped for had to be half-ditched because of time constraints, and we also had to lower the ambitions on the creation of a nice-looking graphic appearance, instead reusing some old CSS we alread had. Even then, fighting the Plone styles system was a pain, and it didn't really get better in later versions.

We finally got the site deployed. It didn't get updated as often as I had hoped for, and frankly, it didn't work that well, with hickups and performance trouble.

After a few years, Plone 2.5 arrived.

Explosion

KABOOM - an explosion of new things to learn!

This was where the magical mystery tour of Zope3 technology began to show its ugly face in the code. You had to learn stuff like adapters, interfaces, etc. And although these are well known concepts in the programming world, the key is to neither overuse them nor overcomplicate!

I don't remember, but I think I managed to lift the site up to Plone 2.5. I was really starting to get tired of Plone at this point.

The fun however didn't end there, because some time later, Plone 3 arrived. And again.. KABOOM - lot's of new things to learn. To be just, some of the new things were already familiar if you had learnt them in Plone 2.5, but there were still plenty of new weirdness.

Explosion

For me, not being a full time web site developer, it was near to impossible to keep track of all changes and concepts. It was especially hard to know which concepts to use for good compatibility with future versions of Plone.

New things arrived all the time - one example being the way you were supposed to make skins (the graphic appearance). It was completely revamped several times, each time introducing new layers of abstraction and complexity. Stuff like the stylesheet registry, where you are supposed to write XML code that registers your stylesheets in a magic registry that's accessible only via the magic web interface (the ZMI). Argl!

Trying to manage the way installations are made became more and more complicated, being a python script to begin with, then growing into countless XML files per product, making it very hard to keep track of what's going into what file. Argl, again!

No, I'm all happy I left Plone, and the learning curve is just one of the reasons.

(Beautiful explosion images from PD Photo.org)

0 comments.

Finally, Suspend to Disk is working!

Published: 2009-02-06 22:05 UTC. Tags: linux

I've never had any luck with the suspend to disk functionality on Linux, but now, after upgrade to Ubuntu 8.10, it's working, even at my home workstation with mirrored disk setup.

Some people had it working years ago, so I guess I've been having bad luck with my hardware combos or something. Still, this is an area where Linux has definitely been behind Windows - where this stuff just works. That is on the other hand probably mostly because the hardware manufacturers only test on Windows.

Regardless of who's to blame for the previous trouble, I'm happy with the fact that it's now working - I enjoy being able to shut down the computer and then start it again in the state I left it. Leaving it on just for that purpose is not something I want to do, trying as I am to be a responsible energy consumer.

0 comments.

Dualhead at home

Published: 2009-02-04 19:27 UTC. Tags: open source hardware linux syncmaster_204b
Dualhead setup

I upgraded my home workstation to Ubuntu 8.10 the other day. I was kind of worried that this would cause trouble with my X Window System setup. I've had major trouble with my Samsung Syncmaster 204b before, with its faulty DVI implementation being the major source of trouble.

My worries were well justified - no image on DVI after reboot. Tried connecting it via VGA, and that kind of worked - a slight fuzziness, which almost went away after pressing the auto-adjust button on the monitor.

I got tired of the monitor and decided to go for a new one - so I ordered a Benq FP241W 24" WIDE TFT. It's using the less common MVA display technology instead of the more common TN. This is supposed to mean that it's colours and viewing angle are superior to standard panel.

Thinking about it, I came to the conclusion that I might be able to stand a slight fuzziness on my 20" samsung if using it as a secondary panel in a dualhead setup. So, when the new panel arrived, I connected the 20" Samsung via VGA, and the new monitor via DVI.

With my usual luck, I got the same problems with the new panel as with the old one - problems syncing the DVI signal, with the result being that the panel went black.

However, after switching from the open source ATI driver to the proprietary fglrx driver, things started working, and using amdcccle, the ATI driver control program, setting up dualhead was really easy.

The result - a resolution of 3840x1200 at home. That's even slightly better than at work, where Opera Software have equipped me with a 19"+24" combo.

I'm not looking forward to the next Ubuntu upgrade, though. Sometimes, I just hate the X window system.

1 comments.

Calculating the automatically assigned IPv6 adress given prefix and MAC

Published: 2009-02-04 18:36 UTC. Tags: network ipv6

Today I had the need to calculate the automatically assigned IPv6 adress I knew a host probably had since the network has a router advertisment daemon.

I knew the network prefix and the hardware address (MAC/EUI-48). I suspected ipv6calc could probably do the job, but I had great trouble finding out how. I'm sure the commandline syntax for ipv6calc makes sense once you get used to it..

Anyway, here's how, for my own and your future reference:

ipv6calc --in prefix+mac --action prefixmac2ipv6 2001:aaaa:bb:cccc::/64 AA:BB:CC:DD:EE:FF --out ipv6addr

IPv6 prefix as well as MAC obfuscated to protect the innocent.

Relevant links:

0 comments.

Ripping 300 CDs..

Published: 2009-01-19 19:52 UTC. Tags: linux squeezebox

I'm currently in the process of transferring my CD collection into digital format for use with my Squeezebox. This is a major project when you own around 300 CDs..

Stack of CDs

Ripping CDs, but to what format?

I did some thinking and evaluation before choosing format for the storage of my CDs in digital format;

Ogg Vorbis

Ogg Vorbis was one of the alternatives. It is patent free, which would fit my free software background. There are a whole bunch of devices supporting it, but with the SO buying a new mp3 player that didn't support Ogg, and the Squeezebox only half-supporting it, Ogg was unfortunately not an alternative.

FLAC

FLAC were another alternative. Free, and also lossless - the thought of actually storing the CDs without loss of data was tempting. Again, it fell on compatibility. Finding devices such as mp3 players and car audio equipment.

One option would have been to store as FLAC and then recode everything to mp3, possibly on demand, but I demeed that out as too much work with little gain.

MP3

So, as you might have guessed, for compatibility, I choose to rip my CDs to mp3. I'm using the LAME encoder with the best quality variable bitrate (VBR) encoding. According to the Hydrogen Audio Wiki, LAME has the best VBR encoder out there.

The resulting MP3's play directly on the Squeezebox hardware without problems, and works as intended in the SO's mp3 player as well.

1 comments.