Premature migration

Ok, so I'm not ready just yet to go over to jottit.com, cool and python based as the service is, the way it works is in too much flux to run a serious site on it for now. They've just completely changed the way the navigation works, so that my blog entries are no longer sorted, and show meaningless ids instead of titles in the text of the links. Also they now have rss/atom feeds, but they aren't very usable for the public. They show changes to pages, but for a blog, you mostly just want a feed of new entries, and a changes/deletions feed is more of an added feature. So happily back to blogger it is ;)

The Musical Gardener's Tools #3: The Kitchen Sync

One of the potential downsides of obtaining your music from a large number of mixed quality sources is that your collection will be overrun by crap if you don't aggressively cull the crap. Since I listen to music on at least 4 machines (my laptop, my work desktop, my home desktop and my iAudio M5 hard drive player) synchronisation could become nightmarish: If I delete something from my laptop and I sync with any of the other machines, I don't want the deleted crap to reappear, but I do want new stuff I downloaded to get transferred. The way I solved this is with a few scripts using the wonderful rsync and a bit of self-discipline:

syncing between computers

I have two scripts on my work desktop called hello.sh and goodbye.sh. The former I run every day when I come into the office in the morning and this synchronizes all music from my laptop onto my desktop, including new, changed or deleted files:

#! /bin/sh
rsync -avz --delete laptop:~/ogg/ ~/ogg
~/ogg/mp3blogs/update
./rm_empty
rsync -avz --delete ~/ogg/ laptop:~/ogg

where 'laptop' is the hostname of the laptop, and 'update' and 'rm_empty' are the names of the scripts mentioned in a previous post. So, the script does the following, in order:

1. synchronize files from laptop to desktop
2. download new files from selected mp3blogs to the desktop
3. remove any empty directories under the ogg directory on the desktop
4. synchronize files from desktop to laptop

That last step is actually redundant when I don't forget to use the accompanying 'goodbye.sh' script when I leave at night, but sometimes I do, when I have to run for a train. The 'goodbye.sh' script is even simpler:

#! /bin/sh
./rm_empty
rsync -avz --delete ~/ogg/ laptop:~/ogg

and does the following:

1. remove any empty directories under the ogg directory on the desktop
2. synchronize files from desktop to laptop

syncing between a computer and a music player

For this I wrote a little Python script, mostly because I like Python syntax much better than whatever shell script syntax (yeah, I'm new school), but it could be easily solved differently. The use case here is: all the music on any one of my computers will never fit on the puny 20GB my music player sports. That's ok, because this is only meant to hold the music I *know* I like, and to which I like to relax on the train to and from home. So the problem is we want to synchronize a subset of the music on (for instance) my desktop. I made a script that does this:

#!/usr/bin/env python
from os.path import isdir
from os import listdir, system

local = '/home/eric/ogg'
iaudio = '/media/IAUDIO/MUSIC'

localdirs = listdir(local)
iaudiodirs = listdir(iaudio)

for entry in iaudiodirs:
    iaudio_path = iaudio + '/' + entry
    local_path = local + '/' + entry
    if isdir(iaudio_path):
        if entry not in localdirs:
            print "synching %s from iaudo to local" % iaudio_path
            system('rsync --size-only --delete --delete-excluded \
            --exclude-from= /home/eric/.rsync/exclude -avz \
            --no-group %s/ %s' % (iaudio_path, local_path))
        else:
            print "synching %s from local to iaudo" % entry
            system('rsync --size-only --delete --delete-excluded \
            --exclude-from= /home/eric/.rsync/exclude -avz \
            --no-group %s/ %s' % (local_path, iaudio_path))

With small modifications, this can be made to work with any music player that behaves like an external HD under Linux (obviously paths and directory names need to be changed, I did not try to make this script generic). What it does is run through all the artist directories on my player. If an artist directory exists there that is not on my desktop, it copies it, under the assumption that it is a new artist that I like and picked up somewhere or other. If the artist directory *does* exist, it does the exact reverse: it syncs from the computer *to* the player, under the assumption that I only delete or add single files on the desktop, since it's too much of a hassle to do it on the music player directly. If either of these assumptions are not valid in your case, obviously the script wouldn't work for you without some serious modification.

Metropolis 2007

Sunday july 1st is the annual free Metropolis festival in Rotterdam. I've never actually been, but heard great things about it, and it looks like this year has another great line-up, so I'm definitely planning on going this time. I've tagged and bagged another radio station:

I've started tagging the confirmed artists for Lowlands 2007 (as lowlands 2007, surprisingly,) so watch that space for and ever updating radio station of all this year's edition goodness. I'll add the artists as they are officially confirmed. (There are some unconfirmed artists there, obviously tagged by people who know more. Looks like fun though, Cansei de Ser Sexy, yay!)

And here's a radio player widget thingy, for that station:

New last.fm features rock

There are a few new features last.fm introduced last week (I guess it was last week, it may have actually been earlier, they're sneaky that way.) that make it an even more useful service than it already was.

Along with the introduction of some shiny new widgets that are sure to appeal to the myspace crowd, they threw in some great new functionality:

RSS/Ical feeds for the recommended events.

YAY! Now I can see what artists that I listen to are playing venues near me. This is sure to make my schedule even more hectic and my wallet more empty. upcoming.org has been doing the same thing for a long time, but there are two reasons why I expect last.fm to work better for me: It does only music, whereas upcoming is starting to include a lot of business/tech conferences, which I have other channels for. It filters on my taste or where my friends are going, in addition to location, (upcoming only does locations), and it does that in a smarter way too: it allows you to set a geographic radius, instead of saying, I want to include these and these cities. I'm not sure how much difference this will make in practice, but I do like that I will be notified of stuff happening near me, even if it's happening just outside one of the cities in my area.

One thing I like about upcoming that the last.fm don't yet seem to have is the buttons to send an event directly to a calendar. The ical feed can be used to send all events to a separate calendar, I guess, but I like a little more manual control. The 'Add this to your [foo] calendar' buttons in upcoming are a great thing. (For me foo==google calendar, especially now that it's fully integratable with thunderbird + lightning through this nice plugin. Installation instructions here.)

Configure your overall top artist and top tracks lists to include the last 12, 6 or 3 months only.

This is great if you have been using last.fm for a while, and would like your evolving interests to show up in your profile. For instance, I have listened to a *lot* of Joni Mitchell, and I suppose I will continue to do so, which means that she and other long time favorites tend to keep newer dicoveries out of my charts. Now that I've set the charts to show 12 months, she'll still be in the top 10, but not on 1st place, and other older artists which I've not been listening to quite so much anymore actually have a chance to drop out of the top 50.

The Musical Gardener's Tools #2: More Sources

My second biggest source for new music is the web, where, with a little work, a lot of high quality free and legal stuff is to be had. Here are some of my tips:

www.last.fm

Easily my favorite website/service of the last years. For anyone still unfamiliar with it, what it does, in a nutshell, is keep track of all music you listen to on your computer (or even on your portable music player,) and generate weekly and lifelong personal and global charts from that.

While people with charts fetishes may feel that's quite exciting already, where last.fm positively shines is what it does with those charts; After a few hundred songs, it starts to compute your musical neighbours, and recommended artists you may or may not have heard of. It lets you listen to a personal 'recommended radio' station, which is in my opinion last.fm's greatest feature. It will play the artists last.fm thinks you might like based on your neighbours, in addition to personal recommendations from other users, and recommendations sent to groups you belong to.

What I usually do is have 'recommendation fridays' where instead of starting my regular music player, I listen to recommendation radio all day. If stuff comes by that I really like, I check whether it's available as a download on last.fm, (there are loads of free downloads,) or see if it's available elsewhere.

See the sidebar on the right for my weekly artist chart, and a link to 'thisfred radio' which you can listen to from any flash enabled browser, or from last.fm's own standalone music player.

www.daytrotter.com

The Daytrotter Sessions are a great and consistently high quality source of unique mp3s. The idea is that bands touring the area stop by at daytrotter, exclusively record three or four songs, which are then put up as free mp3s on the site. The bands are usually on the indie side of the fence, and on the verge of breaking through, although there are some bigger names in the list.

To consistently make available a new interesting session at least every week for a good while now, is a pretty amazing achievement. The new edition is a welcome surprise in my bag o' RSS each week.

A few of my personal favorites:

• Casiotone for the Painfully Alone
• About. This one just in, and maybe a bit chauvinistic, since they're from the Netherlands. I gather they'll be playing South by Southwest (see below) in Austin this month, so do check them out if you're there (if you are: I'm green with envy,) and in the mood for some high energy melodic bleepcore laptop pop.

South By Southwest Showcase Torrents

I've never been to SxSW, but every year it looks like I'm missing a lot, and I definitely plan to save up and go there one year. That year won't be 2007 unfortunately. *Fortunately*, for us Atlantically challenged Erpians, SxSW makes available a torrent of mp3s from artists that will be playing the festival each year. Apparently, not all of the music industry is clueless. The torrents go back to 2005, and are pretty large. it's some 8GB of music, a *lot* of it very good.

amiestreet.com

Just discovered this today: Amie Street is an mp3 web store with several twists: First of all: DRM-free, which is a sine qua non for me, but not terribly earth shattering. What is interesting is their business model: All mp3s start out as free, as in beer, downloads, but rise in price as they get more recommendations. People recommending the mp3s that get popular get a little kickback, if I understand correctly, which they can use to buy other mp3s. So it literally pays to check out new and unknown stuff, and the less adventurous/miserly users have a pretty good indication of popularity in the price of individual mp3s.

After sifting through some of the free mp3s, I must say the quality is varied to say the least, but that's to be expected. What I think I'll do is shell out some money, and jump in after the first round of sifting through is done, and look for the gems in the 1-10¢ price range. Watch this space for my recommendations.

I do think this might work as a business model, where you let users with little money pay with their time, and vice versa. It does feel right. And they don't just have completely unknown bands on there either. I already saw Barenaked Ladies and Au Revoir Simone advertised.

Your favorite mp3blogs and wget

A slightly more geeky way to get your mp3s, which I originally found here and then slightly adapted to suit my particular needs better.

As noted by Jeffrey in his post, using wget for this in the wrong way can cause bandwidth problems for the sites you are hitting, so use caution: presumably you are targeting those sites because you like the music they make available, causing them problems is probably not the best way to ensure they continue to do so.

The way I call wget is:

wget -U"Mozilla/5.0" -r -l1 -H -t1 -x -nc -np -P ~/mp3blogs/ -A.mp3,.ogg -erobots=off -i ~/mp3blogs/urls.txt

(That should be all on one line.)

My wget call differs from Jeffrey's in the following ways:

• I added -nc which stands for 'no clobber', it means it won't re-download files that are already there, which I'm sure makes the site owners happier. I think the original does a checksum check on the files, so it won't reload them, *unless* they have changed. Since I use mp3gain on the files, and almost always correct some tags, that means they would always be downloaded again in my case, losing the changes I made...
• I removed -nd and added -x, which forces directories for the entire url path, because I like having the directories over a single directory with all the files: It shows me where the files came from, so I can give kudos for those I like, and if I end up getting a lot of crappy ones from a particular site, I can remove its url from urls.txt. This can mean a lot of empty directories after a while, but I have a script for that too, see below.
• I added .ogg to the file mask, just on the off chance that someone out there is providing oggs rather than mp3s.

Some more nice wget tips can be found here on linux.com

After the update, I run the following bash script to remove any empty directory trees that are created by using wget in this way:

#!/bin/bash
LS="$(find ~/mp3blogs -type d -empty)"
echo $LS
while [ -n "$LS" ]; do
    find ~/mp3blogs -type d -empty -print0 | xargs -0 rm -rf
    LS="$(find ~/mp3blogs -type d -empty)"
done

[Edit 2007-08-23:] One thing that script doesn't take into account is album covers: my excellent music player lets me directly delete songs from the hard drive if I decide I don't like them, but when jpegs or playlist files remain in a directory when all the songs have gone, it won't ever get cleaned up. So I wrote a new version, that also takes an argument for the path:

set -u
find $1 -depth -type d | while read dir
do
    songList=`find "$dir" \( -iname '*.ogg' -o -iname '*.mp3' \)`
    if [[ -z "$songList" ]]
    then
        rm -rf "$dir"
    fi
done

Then all that remains is to run the recursive mp3gain and vorbisgain commands I described in my previous post.

Of course I call these 3 commands (and then some I will talk about in an upcoming post) from a single master script, called 'hello', which I run about once a day while I get morning coffee.

The Musical Gardener's Tools #1: Ripping your CDs

My number one source for music files is still plain old fashioned CDs, and I suspect it will be a while before anything changes that. I like browsing through the new stuff and the bargain bins, and I'll buy pretty much anything if it's cheap enough on the off chance that there's something worthwhile on there. Usually there isn't, but I did find a few gems over the years.

For ripping CDs to ogg vorbis files (my favorite, but mp3s work just the same) I use grip, a great little ripper for Linux, that does everything I must have:

• rip to wav, usually even if the CD is 'protected' by some hare-brained DRM, although I make it a point not to consciously buy CDs with DRM, so I haven't tried with very many.
• look up metadata over the net, so I don't have to type over the liner notes.
• convert to ogg or mp3 with the quality settings of my choice. (Basically it calls the appropriate command line utilities, and you can specify the parameters.)
• save the files in a place and with a filename that I can fully specify. (I use oggs/artist/album/artist-album-tracknumber-title, everything lower cased, spaces replaced by underscores. I realize there's some redundancy in there, and the filenames are a tad long, but it does make identifying individual files by their filename easy.)

For those still left on windows, I recommend CDex which I've used in the past and which has a nearly identical feature set. It's also open source.

The next step is volume gain: when you listen to large playlist as opposed to single CDs, large differences in volume quickly become annoying. What I do is normalize the (perceived) volume of all my files, to prevent nasty headphone surprises and constant twiddling of volume knobs.

On Linux there's the command line utilities vorbisgain (for oggs) and mp3gain, probably in your distribution already, I know they are in Ubuntu. I use the following two commands, depending on whether I'm dealing with oggs or mp3s:

find . -name "*.mp3" -print0 | xargs -0 mp3gain -k -r -f #recmp3gain
find . -name "*.ogg" -print0 | xargs -0 vorbisgain -fr #recvorbisgain

The find/xargs combo means that I process all files of the relevant type from the current directory and below. This works for huge numbers of files, where just calling find and the mp3gain or vorbisgain command would not.

The comments at the end, #recmp3gain and #recvorbisgain, are of course not necessary. I just use them so that I can use CTRL-R on the command line, type '#recv' or '#recm' and have the entire command. You can do the same with an alias of course, but somehow I'm always hesitant to pollute the global namespace, and this is a nice alternative.

The options mean find everything, skip the files that have already been adjusted, and check and adjust the rest. For large numbers of files, this can take a while and a lot of CPU. I use global gain rather than album gain, (which would keep the relative loudness of tracks from one album intact,) because then I would have to run the command on each album folder separately, and also it only really matters for classical albums, where the differences in loudness between tracks can be huge. I don't really listen to classical music much, but if you do, you probably want to use album gain.

The last thing I usually do, is check the tags against musicbrainz, with the picard tagger. The nice thing about that tagger is that it will recognize individual files, not just CDs, and the quality of the tag data is usually higher than that of freedb.org, which grip, and a lot of other CD rippers use. Also, it adds its own musicbrainz tags, which can be used by players or other software to look up the most current version of the tags in the musicbrainz database. For instance, the last.fm plugin for my favorite music player[1] uses that to submit information.

[1] More about that in another post.

The Musical Gardener's Tools: Introduction

A large part of my time is spent either actively or passively listening to music. As a rule, I don't really like radio, because of the repetitive and predictable music on it, and carrying CDs around is impractical, so what I usually do is listen to my collection of oggs and mp3s.

To keep things interesting, I check a lot of sites periodically for new songs to download, and I tend to throw a lot of them away again after listening, the theory being that my collection is forever getting better and better through constantly keeping an eye on what grows there, and culling the crap, sort of like gardening. Or so I imagine.

I'm also a big fan of last.fm, where everything I play is registered, (guilty secrets and all,) and which regularly recommends me some great new bands in return. Because last.fm only works when the id3 tags in your music files are correct, (*and*, I suspect, because I'm anal about categorizing stuff,) I also tend to do a lot of editing of same tags.

Taking the above as a given, I'm constantly searching for the best tools to make my life easier, and I thought I'd share some of my favorites....