thisblog

2010-03-31T14:02:00.000-07:00

After a recent snafu where google disabled my blog for a few weeks without even so much as sending me a notification, I've decided to move to posterous. It took a while for their import utility to fully grok my blogger feed, but it just started working, so from now on, my infrequent ramblings can be found here: The new and hardly improved thisblog.

Generating Band Names and Song Titles

2010-02-25T19:16:00.000-08:00

Yesterday my distinguished colleague Stuart Langridge said he could use some fake band names for functional tests in the Ubuntu One Music Store he's working on. Since I'd done a similar thing once before (for realistic sounding Dutch city/village names.) I figured I could hash an algorhithm out pretty quickly, and so I did, last night.

I started with a simple Markov chain class that can analyse a body of data and then generate text that is like the source data, but does not actually occur in it. At first I used the same algorithm for artist names and song titles, but I soon decided generating songs word by word, and artist names character by character made more sense, since proper names (and a lot of band names) don't adhere to spelling rules anyway.

The song titles came out pretty great (as in, in every batch I generated there were at least a few funny ones,) but the artist names remained problematic, so the next thing I tried was splitting the artists into groups and people. This seems to be generating even better results, but splitting the list of artists (which I generated with a throwaway plugin for quodlibet, see below) turns out to be a lot of work. I did a few hundred manually, and the results below are quite cool, but there's a lot of partial duplication. Perhaps I shall finally download the musicbrainz data set, and see if I can generate separate lists of people and groups/bands from that easily. (HINT: if someone has such lists lying about, I would be immensely grateful if you could mail them to thisfred at gmail. That would be me.)

Anyway, here's a single unedited run of the script:

Jack Planter - First true love will die
The Islandry The Mountals - And We Wake Up (live acoustic)
Eef Bunyan - 08-Welcome to Rock Remix
Luna - The Deacon (Duke Dumont Remix)
Jolie Newsom - What A Christmas Duel
Dj Funky Banhardo Villah Priest - King Of The Enemy
The Ian Mouse - Carol Of The Goober Woobers
Robby Brokend - The Same Machine
Princeformeroon) - Long Dark Blues
Joha - What The Fuck Out
Cartripes Young Lips - How Blue You Can Live Without You
Rude Stooges - I Called Out Your Window?
El Perrown - And a She Wolf (Moto Blanco Radio Edit)
Jana Talmann - When My Broken Shield
Flip Kowlie Newsom - Lookin' For A Propulsion Device Based On Heim's Quantum Theory)
Kelle Shock - Used to Hate Us)
Billiams - Forever On The Verge
Iggy Polly - Extraball ft. Amanda Blank
Marton - We Are Decided
...And thers of Leon & Kypski - One Of These Days (Clifton Chenier cover)
Misse Dayton - Home On The Edge
Whisperdrag - Son of Rio Mix (Single Version)
Raftwer - Song A Day Another Day
Stard, Run Run - Begin to See What I Meant to Be Glad
Asobius Pip - Music Sounds Better with You
Ill Girl Seed - Never an Easy Way to the Center of the South
The Weeper Girls - Mind How You Feel It?
Flip Kowlie Newsom - Sondre Lerche - To Plant A Seed.mp3 [Unknown]
Alexand - Far Cry (live in the Jungle
Rocker - Veins To The City
Sean Lionhearthan - put it on the Dancefloor (John B Remix)
Trail Riot - Islands In The Gale/Josephine
Williott - Make It Home
Death - Section 7 (Hanging Around the Christmas Tree On Fire (Holy Ghost! Remix)
C-Monobotix - Never Make a Noise
MC Ricard Coxon - Hell - Part Four
Williams Jebeniana Nastarr - Drop Some Silver In The Dead
Williott - Le Le
Alexandt - Pop song for our City
Beth Lakemann - A Friend (That I've Never Understood
Corns of Happies - The Only Healer - Featuring Caroline Schutz Of Folksongs For The Winter
The Naturday - Awoken By a Horse
The Plaza Cent - Like a Mama
Emma Pop - Mogwai and Summer Walks
Ra Ricord Citi 80 - I.C. Y'All (feat. Busta Rhymes, Raekwon & Lil Wayne)
Wints Sected The Tegenwoording Cooks - To Save You
Madow - I Sold My Hands Are Made
Read - 94b Christmas in July
Franco et and - Can't Turn You Into the Pit
The Do Roots - Remember When I Was a Lover
Killalobos - How We Do Is Wrong
David Krauss - Remedy (A1 Bassline Remix) - L2
Eringfielson - Rock The Beach (Neil Young Cover Live 8/15/2003)
Mitch Harcourtis Pilar - Bathroom Gurgle (Duke Dumont Ode To Todd Mix)
Shape - Lost in the game (pt 1)
Tweakes - If You've Got Hopes
Steve Elliams - Dis policeman keeps on kicking me to the Mardi Gras In New Orleans
Billaloner - Motown Never Sounded So Good They Named it Thrice
Van Lidbo - Take me Down
Elastin Trainfully Bessy Bean Moby Grape. - How's The World Can Stop Me Worryin' Bout That Girl
Jural - Long Live The Fallen Aristocracy.mp3 [Unknown]
The Machiefs - Hunt Like the Real Thing
Case - Back In Your Window
Bennie Hollalobos - Shoes (A Bang Gang Remixxx)
Foung Afteras - It Aint Me Babe.mp3 [Unknown]
Dar Willalobotnicks - You Still Believe In Christmas
Anthony Robinson - The Pink Wig To My World Fell Down (Single Version)
erlights - Fall From Your Bed
Eddie Kennor - Last Kiss (Originally recorded by J. Bryson & 1st draft by Zaki Ibrahim)
Chrissy Elliams - Sunday Kind of Chill
Erlendrick Ense - Get Up I Feel For You
Mic Spareck Plan - Walking With a Mixx
Jay Bird - I'm In Your Area
Neko Catra - More Like It
Сергей Шнургей Шнургей Шнургей Шнургей Шнуров - Back of the Dead
Doctors - What Once Was Will Be Free
Emmy Cliff - We Are Golden (Jokers of the seasons
Sean Lionheart - CrowdedHouse - Something Special
Lesbian Cobra - If I Got 5 On It (Clean Edit)
Del Maar - ...Has A Way
Brooks Stra - The Hazards of Love
Chiness Candy - Brahms: Studies, Anh 1A/1 - Presto - Allegro con spirito
Hermanna Nadle - The Other Version (ft. Kid Cudi)
Tokyo Police - Got It and Grab It
A Silents - Your Ex-Lover Is Dead (Remaster)
Tigerince - Now I'm Here You're There (Mexicans with Guns Remix)
Page Fays - I Won't Be That Way
J Dieneman - Got To Make You Strong
The Walkmena Vistener - Someone to Love You Until My Veins Again
Digable Strung - Standing At the speed of life
Willalobos - Sinatra - It Was
Wayne Staalendricks - Steven McCauley for President (Exclusive)
Bonna - Devil Made Us Do It Again
Palaxy - small town (live)
Territsen - We Got The Money I've Got It Bad and Young Jeezy: National Anthem
Shears - A Lonely Construction Worker
Garvie - Walking On A Cloud Of Smoke and Sassafras
Broobinski - Lords of The World, Jonah
Williams - Lake Shore Drive (Todd Terje Edit)
Two Bassibles - Madmen's Discotheque (Disconet Casey Jones (On The Road

As you can see there's a lot of names that are too close to real names to be interesting, and quite a few common patterns. Also broken parentheses, due to (my implementation of) Markov chains being only mildly context sensitive. (See how I used that term in an actual sentence? Totally worth it, that education.) Also, I can't guarantee that none of these titles or artist names aren't actually real, because of trivial lower case/upper case and or white space and punctuation differences, or, due to the artist not being in my source data set.

And here's the code that generated it:


import random

class Markov(object):

    def __init__(self, words=False):
        self.db = {}
        self.lines = set([''])
        self.words = words
        if words:
            self.prevs = 2
        else:
            self.prevs = 3

    def process_file(self, filename):
        with open(filename, 'r') as file:
            for line in file:
                self.process_line(line)

    def process_line(self, line):
        self.lines.add(line.strip())
        prevs = []
        for i in range(self.prevs):
            prevs.append(None)
        if self.words:
            line = line.split()
            line.append('\n')
        for character in line:
            self.db.setdefault(
                tuple(prevs), []).append(character)
            prevs.append(character)
            prevs = prevs[1:]

    def generate_line(self):
        line = ''
        tries = 0
        while line.strip() in self.lines and tries < 100:
            tries += 1
            prevs = []
            for i in range(self.prevs):
                prevs.append(None)
            line = ''
            while True:
                char = random.choice(self.db[tuple(prevs)])
                if char == '\n':
                    break
                prevs.append(char)
                prevs = prevs[1:]
                line += char
                if self.words:
                    line += ' '
        return line.strip()

n = Markov()
n.process_file('names.txt')

g = Markov()
g.process_file('groups.txt')

t = Markov(words=True)
t.process_file('titles.txt')

for i in range(100):
    x = random.choice([n, g])
    print x.generate_line() + ' - ' + t.generate_line()

[Edit]: removed a redundancy left by earlier refactoring.

And here's the dead simple quodlibet plugin, just to show how cool quodlibet is. Note that quodlibet, unlike for instance the also quite nice Rhythmbox, would allow you to do this (or much more interesting things) with *any* id3 tag, including ones you make up yourself.:

import os
import const
from plugins.songsmenu import SongsMenuPlugin

class AddToListPlugin(SongsMenuPlugin):
    PLUGIN_ID = "Export artist list"
    PLUGIN_NAME = _("Export artist list")
    PLUGIN_DESC = _("Add artist name to artists.txt.")
    PLUGIN_ICON = "gtk-find-and-replace"
    PLUGIN_VERSION = "0.1"

    def player_get_userdir(self):
        """get the application user directory to store files"""
        try:
            return const.USERDIR
        except AttributeError:
            return const.DIR

    def plugin_songs(self, songs):
        f = open(os.path.join(self.player_get_userdir(), "artists.txt"), 'a')
        artists = set()
        for song in songs:
            artist = song("artist")
            if artist in artists:
                continue
            artists.add(artist)
            f.write('%s\n' % artist)

Subscribing to google groups with a non gmail mail address

2009-02-05T10:51:00.000-08:00

Dear lazyweb, I keep running up against this, and it may be documented, but not any place I could find easily:

I want to subscribe to a google group with a non gmail email address, and I can't seem to do it myself. I know as admin of several groups that owners can subscribe or invite people with different email addresses, and I figured that maybe I could just send a mail to [nameofgroup]+subscribe@googlegroups.com, since the +unsubscribe version of that works. I had a brief moment of "doodgemaakt met een blije mus"-erlebnis when I got a confirmation mail, but alas, the link seemed to be invalid. Anyone have any bright ideas?

[edit:] Actually there is a separate page where you can manage all your group subscriptions, and *there* it is possible to select all your registered email addresses on a per group basis. Yay!

Go to http://groups.google.com/groups/mysubs

[edit:] actually still doesn't work, because only my gmail address shows up in the dropdowns, not my other registered and confirmed address.

[edit:] And it suddenly just works now, I noticed the other day. (as in the other email address shows up in the dropdown now.) Cool!

Ubiquity command: lastfm

2008-11-02T08:46:00.000-08:00

I found myself tagging all the songs of the soundtrack of the tv series 'Weeds' on last.fm (as you do,) when I realized the search on last.fm is less than optimal for this purpose. It almost always takes three clicks to get to the page I need, even if I have the exact spelling for an artist/track. Enter Ubiquity, the new firefox extension that allows you to add and use simple commands to your browser. In less than an hour, most of which was spent reading the excellent tutorial, the command was working.

The script can be found here on github.

To use it, open ubiquity with your command key, then type:

lastfm [artistname]

or:

lastfm [artistname] - [title]

to go directly to the artist or track page on last.fm in a new tab. Because of the way ubiquity works, you can also select text on a page, and call the command with the selected text as an argument.

Note: this is pretty rough around the edges for now: If the artist name or the track title contain dashes, it will not end up on the correct page. Also, I'm not sure how I can host this so that others can subscribe to the command, and automatically get changes. I currently don't have any 'real' hosting. Maybe github allows me to do this, but I haven't figured it out yet.

P.S.: Here's the link to the tag, and the one to the soundtrack for True Blood, which is a work in progress. Both have very good and diverse soundtracks, which make for nice impromptu last.fm radio stations:

Weeds tag radio
True Blood tag radio

A better anti-spam method

2008-11-01T09:49:00.000-07:00

Recently an idea on how to combat spam occurred to me, that I haven't heard before. It's very low tech, which I find attractive, and it would never result in false positives, which is the most important shortcoming of all the systems I've tried up until now.

What if everyone in the world had an additional email address, that they would never ever use or give out, but that would be publicized in places where only email address harvesting bots would encounter them. A bit of handwaving here, but in its simplest form, just put your real and your spam email address on your webpage, completely unobfuscated, (perhaps even in mailto: links!) but stipulating (in a clear, but not easily machine readable way,) that people should not send mail to the fake/spam one should be enough.

Then *any* message that arrives in the spam account is necessarily spam. You can now use that account to filter your real account, by removing messages with a body that is identical, or similare enough. Again some handwaving on similar enough, (do this wrong, and voilà: false positives again) but you get the drift.

This kind of filter could be implemented by a provider, where the user would not have to do anything manually, except putting the fake address out there for the harvesters to find, or it could be implemented client side, where the mail client gets all mail from both accounts, and does its thing.

And another thing: I also think the current bayesian filters could be improved upon, by recognizing more/different patterns than just lexical ones. I have an intuition that character based markov chaining could catch a lot of spam I get: I built a small script in college which could reliably distinguish quite a number of languages. That would get rid of all the mail addressed to me in languages I cannot read, which I would classify as spam. Taken further this could also get rid of intentionally misspelt spam, to the detriment of poor spellers that want to send me legitimate mail, or (specific patterns of) html markup in mails, which would get rid of most all the other tricks one could use to show the word 'viagra', without actually writing it.

I might have a go at this, to see if my intuition that these could be better predictors than word counts is correct. (I would train my dutch email and my english email into separate 'ham languages', and everything showing entire unpredicted character sequences, would be unsure/spam. Marking things as spam could train a 'spam' language, so there could be both positive and negative indicators.)

Autoqueue goes cross-player!

2008-08-09T01:28:00.000-07:00

When m' colleague Sylvain expressed an interest in porting my autoqueue plugin for Quod Libet to itunes, I experimented a little with factoring out all the generic parts, and it turned out the player specific stuff isn't all that much, so I decided to do a little work and see what kind of problems I would run into when porting it to another player. I chose Rhythmbox for my experiment, since it's in my Ubuntu anyway, and it has support for python plugins.

Turns out it was pretty easy. I have a large part of the featureset working in less than a day, with a lot of help from this page:

http://live.gnome.org/RhythmboxPlugins/WritingGuide

and example code in Alexandre Rosenfeld's lastfmqueue plugin:

http://code.google.com/p/airmindprojects/source/browse/#svn/trunk/rbplugins/lastfm_queue

which offers similar functionality, but is a little more lightweight (less features/bloat, depending on how you look at it ;)

I also moved autoqueue into it's own repository, since it's now no longer solely a Quod Libet plugin, nor, hopefully, a single developer effort. If you're a rhythmbox (or Quod Libet) user and you're interested in checking an early, but working version out, get the plugin here:

http://code.google.com/p/autoqueue/source/browse/trunk

You'll need autoqueue.py, rhythmbox_autoqueue.py, and rhythmbox_autoqueue.rb-plugin. Drop those in your ~/.gnome2/rhythmbox/plugins directory, start rhythmbox, and activate the autoqueue plugin.

If you have questions, feature requests, or would like to help with porting the plugin to your favorite player, you can contact me directly, or even better, join the autoqueue mailing list here:

http://groups.google.com/group/autoqueue

mp3spider becomes barbipes

2008-08-04T02:36:00.000-07:00

Just a short note: After my colleague Sylvain showed some interest in my mp3spider script, and actually built some really cool new features for it, I decided to split it off into its own little project rather than keep it in my supremely unimaginatively named 'thisfred-python-stuff'.

After a very short google I found this cute little critter:

http://en.wikipedia.org/wiki/Saitis_barbipes

And so, from now on, the mp3spider will be known as barbipes and can be found here:

http://code.google.com/p/barbipes/

Anyone interested in contributing, just drop me a note at my usual username at gmail and I'll give you check-in rights.

Quod Libet Plugins Released!

2008-06-29T06:42:00.000-07:00

After working on them for a long time, and then procrastinating for at least as long on wrapping them up into releasable shape, I'm sort of proud to announce my plugins for Quod Libet, (the best music player I have yet found):

http://code.google.com/p/thisfred-quodlibet-plugins/downloads/list

There are three plugins in there, in order of increasing complexity and interest:

autosearch.py

Very simple plugin, searches for the title of the current song in your library: Good for getting rid of duplicates, and finding possible covers.
lastfmtagger.py

Useful only if you have a last.fm account and make use of tags there. This will synchronize last.fm tags both ways, saving them in a custom 'tag' id3 field in your local files. Since Quod Libet has a great id3 editing interface (Ex Falso, also usable as a stand alone application,) this makes adding and editing tags to songs, artists and albums on last.fm much easier.
autoqueue.py

This gets similar tracks to the ones you play from last.fm and puts them in the queue. It is smart enough not to play the same artists/songs for a configurable time, and has some other options (for instance it can also look up similar songs based on the tags created by the lastfmtagger.py mentioned above.) It works pretty well in creating a consistent yet not wholly predictable listening experience if you have a large and diverse library.

As always, feedback is very welcome, I'm sure there are some bugs left in there, or at the very least some rough edges.

(edit: of course within minutes of posting this I find out last.fm's 2.0 API has gone official, so it appears that once again there is some work to do :) Apparently there's no API for getting similar tracks anymore though. That's a shame...)

last.fm adds event radio

2008-06-19T09:41:00.000-07:00

Cool! Now I don't have to tag the artists for the festivals I'm going to by hand, which always felt a little redundant. (I still might though, since I synchronize the tags with my local library, and it's nice to play all the songs I have locally from a particular edition of a particular festival.)

Anyhoo, urls like (for lowlands 2008, which is going to be very good):

http://www.last.fm/listen/event/436106

allow you to listen to a randomized stream of the entire lineup. A feature I have been rooting for for a long time. Last.fm, once again you come through.

The Musical Gardener's Tools #5: Yet Another Way to Harvest mp3blogs

2008-02-23T03:44:00.000-08:00

Update 2008-03-11: There were a number of things wrong with this script making the spidering *waaaay* slower than it needs to be. Fixed that below, and added threading for both the spidering and downloading, thanks to this cool recipe by Wim Schut which lets me run all the sqlite code in a separate thread. (Important because you can only use sqlite connections in the thread in which they were created.) All of this results in a nice speed-up.

Ok I said I wasn't going to, but I did end up writing a bit of code, although it didn't get too far out of hand. Yet :). It solves *all* of my problems: it does not download files over 30MB in size, and it never downloads the same link twice.

I found this message on the python mailing list, which seemed like a very good start. It almost did what I needed, but not quite, and also the parsing was overcomplicated and didn't catch all links, so I replaced that with a simple regular expression.

I ended up changing most of the code and functionality, (for instance it now stores links in a database.) There's a lot of hard coding in there, which I could factor out if people want to use it, but for now it solves my problems beautifully ;).

It's used with the following syntax:

# initial set up
python spider.py createdb
# add a new blog to be harvested
python spider.py add http://url.of.blog/
# (shallowly) spider all blogs for new links to files
python spider.py
# spider a url to a specific depth (5 for example should get 
# most everything, but will take a while)
python spider.py deepspider 5
# download all files
python spider.py download

A minor problem is that curl doesn't do *minimum* file sizes, and with a lot of broken links it does download something small that isn't really an ogg or mp3 file, but a http response. I can probably solve this better, but for now I call the download from an update script as follows:

python spider.py download
find . -iname "*.mp3" -size "-100k"  -print0 | xargs -0 rm
find . -iname "*.ogg" -size "-100k"  -print0 | xargs -0 rm
find . -iname "*.mp3" -print0 | xargs -0 mp3gain -k -r -f
find . -iname "*.ogg" -print0 | xargs -0 vorbisgain -fr

Translation: download files, throw away suspiciously small ones, mp3/vorbisgain what's left.

Here's the code:

Edit 2008-04-18: Moved the code to google code, so I don't have to update it here. Find the latest version here: spider.py

Reminders via del.icio.us and Yahoo Pipes

2008-02-12T02:50:00.000-08:00

Just thought I'd share this, while we still *have* del.icio.us and yahoo pipes... ;)

A while ago I stumbled agross tagmindr, which seemed like a cool idea: put a custom tag on some url in your del.icio.us account and it will remind you at a certain date to look at that url again. I frequently see announcements on website that say something like check back here on [some date] for [some interesting news]. Having automatic reminders for things like that are great, because the chances of me forgetting otherwise are near 100%. The thing is: the personal feed tagmindr promised me *NEVER WORKED*. That's how I completely forgot about a number of things, and tagmindr itself for a while. No biggie, just not very smart if you wanna get all start-uppy and generate buzz ;)

So I decided, how hard can it be to get this right? Turns out not hard at all! I built a yahoo pipe in under an hour that does exactly the same thing. You just give it your del.icio.us username, and it gives you reminders for anything you tag with the tags 'reminders' and 'remind:yyyy-mm-dd' where you replace the ys and ms and ds with the date you want to be reminded on.

The yahoo pipe is here. Enjoy! (As always, feedback and bug reports very welcome!)

Yahoo pipes rock, del.icio.us rocks. Let's hope Yahoo can hold out. I have nothing against Microsoft per se, but I don't think they ever got the web, and I fear they will screw up the nice and open things (like YUI, for instance) that Yahoo has been developing in the past few years.

Exploratory programming, or my 2 ¢ on arc

2008-02-03T10:38:00.000-08:00

A lot of people have blogged on Paul Graham's new language, arc, the (perceived lack of) new features it brings, and the intentionally non-PC announcement by Mr. Graham. I don't have much to add to that particular debate. It looks like lisp, with some new syntactic sugar, which is fine by me. I like lisp, but I wouldn't want to use it in my day job. Others no doubt do, and their taste is no worse or better than mine.

I think maybe the development time worked against it, in that some features seem less than revolutionary, because other languages got there first. Now lisp has them too, and maybe even better implemented, I'm not one to judge.

What I take issue with is that Mr. Graham explains the lack of some other features by the fact that arc is for exploratory programming only, and those features are somehow a hindrance for that. I think this is just plain wrong: unicode support will hurt noone in their exploratory programming, it will actually help a lot of people a good deal. Mr. Graham quotes Guido van Rossum stating he spent a year implementing unicode. I very much doubt that that quote is correct, but *even if it were*, so what? That means exactly nothing to the exploratory programmer, and only hurts the exploratory *language designer* which I think may be a little closer to what is going on here.

As an exploratory programmer in any language I've ever used, (I do think Mr. Graham correct in saying everyone is,) I can safely say that features have never harmed me, as long as they did not get in the way when I wasn't using them. Unicode support in python doesn't. In fact, python (my favorite language, *and* the one I'm most fluent in by now, so yes, I'm biased) is absolutely fantastic for exploratory programming, exactly because of its huge standard library, which helps you get to the meat of the task at hand, without having to build your own support library first.

The Musical Gardener's Tools #4: Lazyweb, lazyweb on the wall...

2008-01-24T09:07:00.000-08:00

..who is the smartestest wgetter of them all?

I need a little help here. As I've described as part of an earlier post, one of my sources for new music is wget, in combination with an ever growing list of mp3 blog urls. The ever growing part is now slowly starting to become a problem. I ran my update script yesterday evening and it took well over 12 hours to complete. (Mind you, I have fiberoptics to the door, speed is not an issue, at least not at my end.) That is unacceptable, in terms of energy wasted. Also the way it works potentially wastes a lot of bandwidth for the poor blog owners, mostly because files I have deleted are downloaded again, unless they were removed from the blog in the meantime. Note that this hits sites heavier that put up music I don't like or already have, but that should hardly be the measure of all things. Maybe. ;)

I see two ways to solve this:

drastically clean up the list of urls that I harvest from.

This is possible, I do it semi-regularly, but new and interesting mp3 blogs keep popping up, so this is only a short term solution.
filter out the stuff I know I don't want

To some extent, I know what I don't want to download. First of all, long podcasts and extended mixes (let's arbitrarily say, anything over 20MB,) since the way I like to listen to music is at the individual track level, otherwise all my tagging tools and last.fm don't work. Anyway we're getting past the whole idea that (web) music radio is consumed in an order predefined by someone else. More suggestion, less force feeding, kthxbye. (On a tangent: can we get this for news radio: just the news items, not a whole, usually extremely repetitive, bulletin as atomic? True podcasting should let me skip items I'm not interested in/have already heard.) Second of all, for obvious reasons, all the files I've already downloaded but deleted.

Since I am far from a linux command line deity, I thought I would ask here, does anyone have any suggestions on how to start on tackling these two problems, given the script:

wget --timeout=5 -U"Mozilla/5.0" -r -l1 -H -t1 -x -nc -np -P ~/mp3blogs/ -A.mp3,.ogg -erobots=off -i ~/mp3blogs/urls.txt

A: How can I limit the length of mp3s and oggs downloaded in this way to for instance 20MB per file? Keep in mind, throwing them away after downloading is not an option, since I want to prevent the download from happening at all. I don't think wget has a switch for this, so it will probably not be possible in a one liner.

B: I would like to store all of the urls of the files I do download (probably just in a flat text file for now) and then have my script skip them when downloading. Again, I don't think a one liner is possible.

Solutions to either problem are worth a 20$ amazon voucher from me (or somewhere else, I don't really care, as long as I'm out only 40$ total and it's not too much hassle to get it to you.)

I am, of course, the sole judge of this contest, but I will try to be fair. You don't have to give me a whole script, I'm a fairly competent programmer, just not too deep into bash, but if you'll point me at where to start, and I get it to work, that counts as a solution. Although as I've said, it's going to grow beyond a one liner, I would like to keep it a simple script, and I'm not looking for an application. I could build one in Python myself, but I want to keep it zero maintenance, basically too simple to even put the code into subversion.

UPDATE 2008-01-28: I'm now looking into pavuk, which may or may not have all the features I need. If this works, I just earned myself 40$ :)

UPDATE 2008-01-28.1: pavuk, although having rather exotic naming of options and switches, seems to solve A quite nicely, which is a bandwidth (and time, and thus energy) saver. Finding all the right options was made much easier by this guide. I'm still thinking about solving B, there may be options in pavuk to help me with that too.

For completeness' sake, the updated script looks like this (except it should all be one line...):

pavuk -timeout 5000 -identity "Mozilla/5.0" -lmax 1 -retry 1 -dont_leave_dir -cdir ~/mp3blogs/ -asfx .mp3,.ogg -noRobots
 -urls_file ~/mp3blogs/urls.txt -maxsize 30000000 -fnrules F '*' '%h/%d/%n'

Coolendar

2008-01-23T03:53:00.000-08:00

After reading this hypernarrative post about calendar mashups using yahoo pipes, I realized I could make my own filtered calendar feed for stuff events that are recommended to me by various sources, chiefly my last.fm recommendation feed. Since those sources tend to contain more noise than signal, at least for now, (automatic recommendation is hard, I read that somewhere,) and I tend to miss things because they get buried, I decided to take a page out of Wilbert's book, and become the editor of my very own event feed, mostly targeting myself, and perhaps one or two friends.

Since I use thunderbird with the lightning and google calendar provider plugins, which tend to visually clutter when too many events show up, I can now show only this feed there. Once a month I copy everything that looks remotely interesting from the other calendar feeds by hand, and Bob's my uncle. The yahoo pipes part is cool, and I might redirect all the feeds I subscribe to into one big source funnel yet, but for now I don't need it. Also I like to see who recommended me what, so I can unsubscribe from feeds that turn out to be of less interest to me than I thought.

So, without further ado, I present you with: teh coolendar! (The actual ical feed is here, for completeness' sake.)

My top 50 artists for 2007

2008-01-17T02:14:00.000-08:00

In descending order of listening frequency:

Bishop Allen, Bright Eyes, Belle and Sebastian, Beck, Nina Simone, De La Soul, Rilo Kiley, Joni Mitchell, Ween, Steve Earle, Aimee Mann, Johnny Cash, OutKast, John Prine, Casiotone for the Painfully Alone, Hank Williams, Dusty Springfield, The Knife, Johan, Frank Black, Gillian Welch, Tori Amos, Indigo Girls, of Montreal, Beth Orton, Gorillaz, The Flaming Lips, A Tribe Called Quest, Sultans of Ping F.C., Flip Kowlier, The Young Knives, Duvelduvel, The Thermals, Kaiser Chiefs, Billy Bragg, Jacques Brel, Missy Elliott, Elastica, M. Ward, Van Morrison, Devo, Peaches, Nouvelle Vague, Mates of State, Martha Wainwright, The View, Randy Newman, Tom Waits, Editors, Damien Rice

There you have it, gentlemen, what more evidence do you need? Hardly anything very *now*, except maybe for The Thermals, The Young Knives and The View, all of which I personally didn't discover until 2007. Not terribly hip I'm afraid, but last.fm don't lie. :)

last.fm dream job

2008-01-14T09:30:00.000-08:00

Wow, this sounds pretty amazing, it has programming, music metadata obsessiveness, and last.fm. Maybe they'd even let me use Python ;). Shame the timing's a little off, I can't really move to London right now, should I even interview successfully. I wish whoever gets the job a lot of fun!

Premature migration

2007-12-04T08:20:00.000-08:00

Ok, so I'm not ready just yet to go over to jottit.com, cool and python based as the service is, the way it works is in too much flux to run a serious site on it for now. They've just completely changed the way the navigation works, so that my blog entries are no longer sorted, and show meaningless ids instead of titles in the text of the links. Also they now have rss/atom feeds, but they aren't very usable for the public. They show changes to pages, but for a blog, you mostly just want a feed of new entries, and a changes/deletions feed is more of an added feature. So happily back to blogger it is ;)

The Musical Gardener's Tools #3: The Kitchen Sync

2007-06-20T00:53:00.000-07:00

One of the potential downsides of obtaining your music from a large number of mixed quality sources is that your collection will be overrun by crap if you don't aggressively cull the crap. Since I listen to music on at least 4 machines (my laptop, my work desktop, my home desktop and my iAudio M5 hard drive player) synchronisation could become nightmarish: If I delete something from my laptop and I sync with any of the other machines, I don't want the deleted crap to reappear, but I do want new stuff I downloaded to get transferred. The way I solved this is with a few scripts using the wonderful rsync and a bit of self-discipline:

syncing between computers

I have two scripts on my work desktop called hello.sh and goodbye.sh. The former I run every day when I come into the office in the morning and this synchronizes all music from my laptop onto my desktop, including new, changed or deleted files:

#! /bin/sh
rsync -avz --delete laptop:~/ogg/ ~/ogg
~/ogg/mp3blogs/update
./rm_empty
rsync -avz --delete ~/ogg/ laptop:~/ogg

where 'laptop' is the hostname of the laptop, and 'update' and 'rm_empty' are the names of the scripts mentioned in a previous post. So, the script does the following, in order:

synchronize files from laptop to desktop
download new files from selected mp3blogs to the desktop
remove any empty directories under the ogg directory on the desktop
synchronize files from desktop to laptop

That last step is actually redundant when I don't forget to use the accompanying 'goodbye.sh' script when I leave at night, but sometimes I do, when I have to run for a train. The 'goodbye.sh' script is even simpler:


#! /bin/sh
./rm_empty
rsync -avz --delete ~/ogg/ laptop:~/ogg

and does the following:

remove any empty directories under the ogg directory on the desktop
synchronize files from desktop to laptop

syncing between a computer and a music player

For this I wrote a little Python script, mostly because I like Python syntax much better than whatever shell script syntax (yeah, I'm new school), but it could be easily solved differently. The use case here is: all the music on any one of my computers will never fit on the puny 20GB my music player sports. That's ok, because this is only meant to hold the music I *know* I like, and to which I like to relax on the train to and from home. So the problem is we want to synchronize a subset of the music on (for instance) my desktop. I made a script that does this:


#!/usr/bin/env python
from os.path import isdir
from os import listdir, system

local = '/home/eric/ogg'
iaudio = '/media/IAUDIO/MUSIC'

localdirs = listdir(local)
iaudiodirs = listdir(iaudio)

for entry in iaudiodirs:
    iaudio_path = iaudio + '/' + entry
    local_path = local + '/' + entry
    if isdir(iaudio_path):
        if entry not in localdirs:
            print "synching %s from iaudo to local" % iaudio_path
            system('rsync --size-only --delete --delete-excluded \
            --exclude-from= /home/eric/.rsync/exclude -avz \
            --no-group %s/ %s' % (iaudio_path, local_path))
        else:
            print "synching %s from local to iaudo" % entry
            system('rsync --size-only --delete --delete-excluded \
            --exclude-from= /home/eric/.rsync/exclude -avz \
            --no-group %s/ %s' % (local_path, iaudio_path))

With small modifications, this can be made to work with any music player that behaves like an external HD under Linux (obviously paths and directory names need to be changed, I did not try to make this script generic). What it does is run through all the artist directories on my player. If an artist directory exists there that is not on my desktop, it copies it, under the assumption that it is a new artist that I like and picked up somewhere or other. If the artist directory *does* exist, it does the exact reverse: it syncs from the computer *to* the player, under the assumption that I only delete or add single files on the desktop, since it's too much of a hassle to do it on the music player directly. If either of these assumptions are not valid in your case, obviously the script wouldn't work for you without some serious modification.

Metropolis 2007

2007-06-15T03:49:00.000-07:00

Sunday july 1st is the annual free Metropolis festival in Rotterdam. I've never actually been, but heard great things about it, and it looks like this year has another great line-up, so I'm definitely planning on going this time. I've tagged and bagged another radio station:

2007-05-21T06:41:00.000-07:00

I've started tagging the confirmed artists for Lowlands 2007 (as lowlands 2007, surprisingly,) so watch that space for and ever updating radio station of all this year's edition goodness. I'll add the artists as they are officially confirmed. (There are some unconfirmed artists there, obviously tagged by people who know more. Looks like fun though, Cansei de Ser Sexy, yay!)

And here's a radio player widget thingy, for that station:

New last.fm features rock

2007-05-14T04:04:00.000-07:00

There are a few new features last.fm introduced last week (I guess it was last week, it may have actually been earlier, they're sneaky that way.) that make it an even more useful service than it already was.

Along with the introduction of some shiny new widgets that are sure to appeal to the myspace crowd, they threw in some great new functionality:

RSS/Ical feeds for the recommended events.

YAY! Now I can see what artists that I listen to are playing venues near me. This is sure to make my schedule even more hectic and my wallet more empty. upcoming.org has been doing the same thing for a long time, but there are two reasons why I expect last.fm to work better for me: It does only music, whereas upcoming is starting to include a lot of business/tech conferences, which I have other channels for. It filters on my taste or where my friends are going, in addition to location, (upcoming only does locations), and it does that in a smarter way too: it allows you to set a geographic radius, instead of saying, I want to include these and these cities. I'm not sure how much difference this will make in practice, but I do like that I will be notified of stuff happening near me, even if it's happening just outside one of the cities in my area.

One thing I like about upcoming that the last.fm don't yet seem to have is the buttons to send an event directly to a calendar. The ical feed can be used to send all events to a separate calendar, I guess, but I like a little more manual control. The 'Add this to your [foo] calendar' buttons in upcoming are a great thing. (For me foo==google calendar, especially now that it's fully integratable with thunderbird + lightning through this nice plugin. Installation instructions here.)

Configure your overall top artist and top tracks lists to include the last 12, 6 or 3 months only.

This is great if you have been using last.fm for a while, and would like your evolving interests to show up in your profile. For instance, I have listened to a *lot* of Joni Mitchell, and I suppose I will continue to do so, which means that she and other long time favorites tend to keep newer dicoveries out of my charts. Now that I've set the charts to show 12 months, she'll still be in the top 10, but not on 1st place, and other older artists which I've not been listening to quite so much anymore actually have a chance to drop out of the top 50.

The Musical Gardener's Tools #2: More Sources

2007-03-07T09:01:00.000-08:00

My second biggest source for new music is the web, where, with a little work, a lot of high quality free and legal stuff is to be had. Here are some of my tips:

www.last.fm

Easily my favorite website/service of the last years. For anyone still unfamiliar with it, what it does, in a nutshell, is keep track of all music you listen to on your computer (or even on your portable music player,) and generate weekly and lifelong personal and global charts from that.

While people with charts fetishes may feel that's quite exciting already, where last.fm positively shines is what it does with those charts; After a few hundred songs, it starts to compute your musical neighbours, and recommended artists you may or may not have heard of. It lets you listen to a personal 'recommended radio' station, which is in my opinion last.fm's greatest feature. It will play the artists last.fm thinks you might like based on your neighbours, in addition to personal recommendations from other users, and recommendations sent to groups you belong to.

What I usually do is have 'recommendation fridays' where instead of starting my regular music player, I listen to recommendation radio all day. If stuff comes by that I really like, I check whether it's available as a download on last.fm, (there are loads of free downloads,) or see if it's available elsewhere.

See the sidebar on the right for my weekly artist chart, and a link to 'thisfred radio' which you can listen to from any flash enabled browser, or from last.fm's own standalone music player.

www.daytrotter.com

The Daytrotter Sessions are a great and consistently high quality source of unique mp3s. The idea is that bands touring the area stop by at daytrotter, exclusively record three or four songs, which are then put up as free mp3s on the site. The bands are usually on the indie side of the fence, and on the verge of breaking through, although there are some bigger names in the list.

To consistently make available a new interesting session at least every week for a good while now, is a pretty amazing achievement. The new edition is a welcome surprise in my bag o' RSS each week.

A few of my personal favorites:

Casiotone for the Painfully Alone
About. This one just in, and maybe a bit chauvinistic, since they're from the Netherlands. I gather they'll be playing South by Southwest (see below) in Austin this month, so do check them out if you're there (if you are: I'm green with envy,) and in the mood for some high energy melodic bleepcore laptop pop.

South By Southwest Showcase Torrents

I've never been to SxSW, but every year it looks like I'm missing a lot, and I definitely plan to save up and go there one year. That year won't be 2007 unfortunately. *Fortunately*, for us Atlantically challenged Erpians, SxSW makes available a torrent of mp3s from artists that will be playing the festival each year. Apparently, not all of the music industry is clueless. The torrents go back to 2005, and are pretty large. it's some 8GB of music, a *lot* of it very good.

amiestreet.com

Just discovered this today: Amie Street is an mp3 web store with several twists: First of all: DRM-free, which is a sine qua non for me, but not terribly earth shattering. What is interesting is their business model: All mp3s start out as free, as in beer, downloads, but rise in price as they get more recommendations. People recommending the mp3s that get popular get a little kickback, if I understand correctly, which they can use to buy other mp3s. So it literally pays to check out new and unknown stuff, and the less adventurous/miserly users have a pretty good indication of popularity in the price of individual mp3s.

After sifting through some of the free mp3s, I must say the quality is varied to say the least, but that's to be expected. What I think I'll do is shell out some money, and jump in after the first round of sifting through is done, and look for the gems in the 1-10¢ price range. Watch this space for my recommendations.

I do think this might work as a business model, where you let users with little money pay with their time, and vice versa. It does feel right. And they don't just have completely unknown bands on there either. I already saw Barenaked Ladies and Au Revoir Simone advertised.

Your favorite mp3blogs and wget

A slightly more geeky way to get your mp3s, which I originally found here and then slightly adapted to suit my particular needs better.

As noted by Jeffrey in his post, using wget for this in the wrong way can cause bandwidth problems for the sites you are hitting, so use caution: presumably you are targeting those sites because you like the music they make available, causing them problems is probably not the best way to ensure they continue to do so.

The way I call wget is:

wget -U"Mozilla/5.0" -r -l1 -H -t1 -x -nc -np -P ~/mp3blogs/ -A.mp3,.ogg -erobots=off -i ~/mp3blogs/urls.txt

(That should be all on one line.)

My wget call differs from Jeffrey's in the following ways:

I added -nc which stands for 'no clobber', it means it won't re-download files that are already there, which I'm sure makes the site owners happier. I think the original does a checksum check on the files, so it won't reload them, *unless* they have changed. Since I use mp3gain on the files, and almost always correct some tags, that means they would always be downloaded again in my case, losing the changes I made...
I removed -nd and added -x, which forces directories for the entire url path, because I like having the directories over a single directory with all the files: It shows me where the files came from, so I can give kudos for those I like, and if I end up getting a lot of crappy ones from a particular site, I can remove its url from urls.txt. This can mean a lot of empty directories after a while, but I have a script for that too, see below.
I added .ogg to the file mask, just on the off chance that someone out there is providing oggs rather than mp3s.

Some more nice wget tips can be found here on linux.com

After the update, I run the following bash script to remove any empty directory trees that are created by using wget in this way:

#!/bin/bash
LS="$(find ~/mp3blogs -type d -empty)"
echo $LS
while [ -n "$LS" ]; do
    find ~/mp3blogs -type d -empty -print0 | xargs -0 rm -rf
    LS="$(find ~/mp3blogs -type d -empty)"
done

[Edit 2007-08-23:] One thing that script doesn't take into account is album covers: my excellent music player lets me directly delete songs from the hard drive if I decide I don't like them, but when jpegs or playlist files remain in a directory when all the songs have gone, it won't ever get cleaned up. So I wrote a new version, that also takes an argument for the path:

set -u
find $1 -depth -type d | while read dir
do
    songList=`find "$dir" \( -iname '*.ogg' -o -iname '*.mp3' \)`
    if [[ -z "$songList" ]]
    then
        rm -rf "$dir"
    fi
done

Then all that remains is to run the recursive mp3gain and vorbisgain commands I described in my previous post.

Of course I call these 3 commands (and then some I will talk about in an upcoming post) from a single master script, called 'hello', which I run about once a day while I get morning coffee.

The Musical Gardener's Tools #1: Ripping your CDs

2007-02-25T15:24:00.000-08:00

My number one source for music files is still plain old fashioned CDs, and I suspect it will be a while before anything changes that. I like browsing through the new stuff and the bargain bins, and I'll buy pretty much anything if it's cheap enough on the off chance that there's something worthwhile on there. Usually there isn't, but I did find a few gems over the years.

For ripping CDs to ogg vorbis files (my favorite, but mp3s work just the same) I use grip, a great little ripper for Linux, that does everything I must have:

rip to wav, usually even if the CD is 'protected' by some hare-brained DRM, although I make it a point not to consciously buy CDs with DRM, so I haven't tried with very many.
look up metadata over the net, so I don't have to type over the liner notes.
convert to ogg or mp3 with the quality settings of my choice. (Basically it calls the appropriate command line utilities, and you can specify the parameters.)
save the files in a place and with a filename that I can fully specify. (I use oggs/artist/album/artist-album-tracknumber-title, everything lower cased, spaces replaced by underscores. I realize there's some redundancy in there, and the filenames are a tad long, but it does make identifying individual files by their filename easy.)

For those still left on windows, I recommend CDex which I've used in the past and which has a nearly identical feature set. It's also open source.

The next step is volume gain: when you listen to large playlist as opposed to single CDs, large differences in volume quickly become annoying. What I do is normalize the (perceived) volume of all my files, to prevent nasty headphone surprises and constant twiddling of volume knobs.

On Linux there's the command line utilities vorbisgain (for oggs) and mp3gain, probably in your distribution already, I know they are in Ubuntu. I use the following two commands, depending on whether I'm dealing with oggs or mp3s:


find . -name "*.mp3" -print0 | xargs -0 mp3gain -k -r -f #recmp3gain

find . -name "*.ogg" -print0 | xargs -0 vorbisgain -fr #recvorbisgain

The find/xargs combo means that I process all files of the relevant type from the current directory and below. This works for huge numbers of files, where just calling find and the mp3gain or vorbisgain command would not.

The comments at the end, #recmp3gain and #recvorbisgain, are of course not necessary. I just use them so that I can use CTRL-R on the command line, type '#recv' or '#recm' and have the entire command. You can do the same with an alias of course, but somehow I'm always hesitant to pollute the global namespace, and this is a nice alternative.

The options mean find everything, skip the files that have already been adjusted, and check and adjust the rest. For large numbers of files, this can take a while and a lot of CPU. I use global gain rather than album gain, (which would keep the relative loudness of tracks from one album intact,) because then I would have to run the command on each album folder separately, and also it only really matters for classical albums, where the differences in loudness between tracks can be huge. I don't really listen to classical music much, but if you do, you probably want to use album gain.

The last thing I usually do, is check the tags against musicbrainz, with the picard tagger. The nice thing about that tagger is that it will recognize individual files, not just CDs, and the quality of the tag data is usually higher than that of freedb.org, which grip, and a lot of other CD rippers use. Also, it adds its own musicbrainz tags, which can be used by players or other software to look up the most current version of the tags in the musicbrainz database. For instance, the last.fm plugin for my favorite music player[1] uses that to submit information.

[1] More about that in another post.

The Musical Gardener's Tools: Introduction

2007-02-20T10:10:00.000-08:00

A large part of my time is spent either actively or passively listening to music. As a rule, I don't really like radio, because of the repetitive and predictable music on it, and carrying CDs around is impractical, so what I usually do is listen to my collection of oggs and mp3s.

To keep things interesting, I check a lot of sites periodically for new songs to download, and I tend to throw a lot of them away again after listening, the theory being that my collection is forever getting better and better through constantly keeping an eye on what grows there, and culling the crap, sort of like gardening. Or so I imagine.

I'm also a big fan of last.fm, where everything I play is registered, (guilty secrets and all,) and which regularly recommends me some great new bands in return. Because last.fm only works when the id3 tags in your music files are correct, (*and*, I suspect, because I'm anal about categorizing stuff,) I also tend to do a lot of editing of same tags.

Taking the above as a given, I'm constantly searching for the best tools to make my life easier, and I thought I'd share some of my favorites....