Exploratory programming, or my 2 ¢ on arc

A lot of people have blogged on Paul Graham's new language, arc, the (perceived lack of) new features it brings, and the intentionally non-PC announcement by Mr. Graham. I don't have much to add to that particular debate. It looks like lisp, with some new syntactic sugar, which is fine by me. I like lisp, but I wouldn't want to use it in my day job. Others no doubt do, and their taste is no worse or better than mine.

I think maybe the development time worked against it, in that some features seem less than revolutionary, because other languages got there first. Now lisp has them too, and maybe even better implemented, I'm not one to judge.

What I take issue with is that Mr. Graham explains the lack of some other features by the fact that arc is for exploratory programming only, and those features are somehow a hindrance for that. I think this is just plain wrong: unicode support will hurt noone in their exploratory programming, it will actually help a lot of people a good deal. Mr. Graham quotes Guido van Rossum stating he spent a year implementing unicode. I very much doubt that that quote is correct, but *even if it were*, so what? That means exactly nothing to the exploratory programmer, and only hurts the exploratory *language designer* which I think may be a little closer to what is going on here.

As an exploratory programmer in any language I've ever used, (I do think Mr. Graham correct in saying everyone is,) I can safely say that features have never harmed me, as long as they did not get in the way when I wasn't using them. Unicode support in python doesn't. In fact, python (my favorite language, *and* the one I'm most fluent in by now, so yes, I'm biased) is absolutely fantastic for exploratory programming, exactly because of its huge standard library, which helps you get to the meat of the task at hand, without having to build your own support library first.

The Musical Gardener's Tools #4: Lazyweb, lazyweb on the wall...

..who is the smartestest wgetter of them all?

I need a little help here. As I've described as part of an earlier post, one of my sources for new music is wget, in combination with an ever growing list of mp3 blog urls. The ever growing part is now slowly starting to become a problem. I ran my update script yesterday evening and it took well over 12 hours to complete. (Mind you, I have fiberoptics to the door, speed is not an issue, at least not at my end.) That is unacceptable, in terms of energy wasted. Also the way it works potentially wastes a lot of bandwidth for the poor blog owners, mostly because files I have deleted are downloaded again, unless they were removed from the blog in the meantime. Note that this hits sites heavier that put up music I don't like or already have, but that should hardly be the measure of all things. Maybe. ;)

I see two ways to solve this:

1. drastically clean up the list of urls that I harvest from.
   
   This is possible, I do it semi-regularly, but new and interesting mp3 blogs keep popping up, so this is only a short term solution.
   
   
2. filter out the stuff I know I don't want
   
   To some extent, I know what I don't want to download. First of all, long podcasts and extended mixes (let's arbitrarily say, anything over 20MB,) since the way I like to listen to music is at the individual track level, otherwise all my tagging tools and last.fm don't work. Anyway we're getting past the whole idea that (web) music radio is consumed in an order predefined by someone else. More suggestion, less force feeding, kthxbye. (On a tangent: can we get this for news radio: just the news items, not a whole, usually extremely repetitive, bulletin as atomic? True podcasting should let me skip items I'm not interested in/have already heard.) Second of all, for obvious reasons, all the files I've already downloaded but deleted.

Since I am far from a linux command line deity, I thought I would ask here, does anyone have any suggestions on how to start on tackling these two problems, given the script:

wget --timeout=5 -U"Mozilla/5.0" -r -l1 -H -t1 -x -nc -np -P ~/mp3blogs/ -A.mp3,.ogg -erobots=off -i ~/mp3blogs/urls.txt

A: How can I limit the length of mp3s and oggs downloaded in this way to for instance 20MB per file? Keep in mind, throwing them away after downloading is not an option, since I want to prevent the download from happening at all. I don't think wget has a switch for this, so it will probably not be possible in a one liner.

B: I would like to store all of the urls of the files I do download (probably just in a flat text file for now) and then have my script skip them when downloading. Again, I don't think a one liner is possible.

Solutions to either problem are worth a 20$ amazon voucher from me (or somewhere else, I don't really care, as long as I'm out only 40$ total and it's not too much hassle to get it to you.)

I am, of course, the sole judge of this contest, but I will try to be fair. You don't have to give me a whole script, I'm a fairly competent programmer, just not too deep into bash, but if you'll point me at where to start, and I get it to work, that counts as a solution. Although as I've said, it's going to grow beyond a one liner, I would like to keep it a simple script, and I'm not looking for an application. I could build one in Python myself, but I want to keep it zero maintenance, basically too simple to even put the code into subversion.

UPDATE 2008-01-28: I'm now looking into pavuk, which may or may not have all the features I need. If this works, I just earned myself 40$ :)

UPDATE 2008-01-28.1: pavuk, although having rather exotic naming of options and switches, seems to solve A quite nicely, which is a bandwidth (and time, and thus energy) saver. Finding all the right options was made much easier by this guide. I'm still thinking about solving B, there may be options in pavuk to help me with that too.

For completeness' sake, the updated script looks like this (except it should all be one line...):

pavuk -timeout 5000 -identity "Mozilla/5.0" -lmax 1 -retry 1 -dont_leave_dir -cdir ~/mp3blogs/ -asfx .mp3,.ogg -noRobots
 -urls_file ~/mp3blogs/urls.txt -maxsize 30000000 -fnrules F '*' '%h/%d/%n'

Coolendar

After reading this hypernarrative post about calendar mashups using yahoo pipes, I realized I could make my own filtered calendar feed for stuff events that are recommended to me by various sources, chiefly my last.fm recommendation feed. Since those sources tend to contain more noise than signal, at least for now, (automatic recommendation is hard, I read that somewhere,) and I tend to miss things because they get buried, I decided to take a page out of Wilbert's book, and become the editor of my very own event feed, mostly targeting myself, and perhaps one or two friends.

Since I use thunderbird with the lightning and google calendar provider plugins, which tend to visually clutter when too many events show up, I can now show only this feed there. Once a month I copy everything that looks remotely interesting from the other calendar feeds by hand, and Bob's my uncle. The yahoo pipes part is cool, and I might redirect all the feeds I subscribe to into one big source funnel yet, but for now I don't need it. Also I like to see who recommended me what, so I can unsubscribe from feeds that turn out to be of less interest to me than I thought.

So, without further ado, I present you with: teh coolendar! (The actual ical feed is here, for completeness' sake.)

My top 50 artists for 2007

In descending order of listening frequency:

Bishop Allen, Bright Eyes, Belle and Sebastian, Beck, Nina Simone, De La Soul, Rilo Kiley, Joni Mitchell, Ween, Steve Earle, Aimee Mann, Johnny Cash, OutKast, John Prine, Casiotone for the Painfully Alone, Hank Williams, Dusty Springfield, The Knife, Johan, Frank Black, Gillian Welch, Tori Amos, Indigo Girls, of Montreal, Beth Orton, Gorillaz, The Flaming Lips, A Tribe Called Quest, Sultans of Ping F.C., Flip Kowlier, The Young Knives, Duvelduvel, The Thermals, Kaiser Chiefs, Billy Bragg, Jacques Brel, Missy Elliott, Elastica, M. Ward, Van Morrison, Devo, Peaches, Nouvelle Vague, Mates of State, Martha Wainwright, The View, Randy Newman, Tom Waits, Editors, Damien Rice

There you have it, gentlemen, what more evidence do you need? Hardly anything very *now*, except maybe for The Thermals, The Young Knives and The View, all of which I personally didn't discover until 2007. Not terribly hip I'm afraid, but last.fm don't lie. :)

last.fm dream job

Wow, this sounds pretty amazing, it has programming, music metadata obsessiveness, and last.fm. Maybe they'd even let me use Python ;). Shame the timing's a little off, I can't really move to London right now, should I even interview successfully. I wish whoever gets the job a lot of fun!

Premature migration

Ok, so I'm not ready just yet to go over to jottit.com, cool and python based as the service is, the way it works is in too much flux to run a serious site on it for now. They've just completely changed the way the navigation works, so that my blog entries are no longer sorted, and show meaningless ids instead of titles in the text of the links. Also they now have rss/atom feeds, but they aren't very usable for the public. They show changes to pages, but for a blog, you mostly just want a feed of new entries, and a changes/deletions feed is more of an added feature. So happily back to blogger it is ;)

The Musical Gardener's Tools #3: The Kitchen Sync

One of the potential downsides of obtaining your music from a large number of mixed quality sources is that your collection will be overrun by crap if you don't aggressively cull the crap. Since I listen to music on at least 4 machines (my laptop, my work desktop, my home desktop and my iAudio M5 hard drive player) synchronisation could become nightmarish: If I delete something from my laptop and I sync with any of the other machines, I don't want the deleted crap to reappear, but I do want new stuff I downloaded to get transferred. The way I solved this is with a few scripts using the wonderful rsync and a bit of self-discipline:

syncing between computers

I have two scripts on my work desktop called hello.sh and goodbye.sh. The former I run every day when I come into the office in the morning and this synchronizes all music from my laptop onto my desktop, including new, changed or deleted files:

#! /bin/sh
rsync -avz --delete laptop:~/ogg/ ~/ogg
~/ogg/mp3blogs/update
./rm_empty
rsync -avz --delete ~/ogg/ laptop:~/ogg

where 'laptop' is the hostname of the laptop, and 'update' and 'rm_empty' are the names of the scripts mentioned in a previous post. So, the script does the following, in order:

1. synchronize files from laptop to desktop
2. download new files from selected mp3blogs to the desktop
3. remove any empty directories under the ogg directory on the desktop
4. synchronize files from desktop to laptop

That last step is actually redundant when I don't forget to use the accompanying 'goodbye.sh' script when I leave at night, but sometimes I do, when I have to run for a train. The 'goodbye.sh' script is even simpler:

#! /bin/sh
./rm_empty
rsync -avz --delete ~/ogg/ laptop:~/ogg

and does the following:

1. remove any empty directories under the ogg directory on the desktop
2. synchronize files from desktop to laptop

syncing between a computer and a music player

For this I wrote a little Python script, mostly because I like Python syntax much better than whatever shell script syntax (yeah, I'm new school), but it could be easily solved differently. The use case here is: all the music on any one of my computers will never fit on the puny 20GB my music player sports. That's ok, because this is only meant to hold the music I *know* I like, and to which I like to relax on the train to and from home. So the problem is we want to synchronize a subset of the music on (for instance) my desktop. I made a script that does this:

#!/usr/bin/env python
from os.path import isdir
from os import listdir, system

local = '/home/eric/ogg'
iaudio = '/media/IAUDIO/MUSIC'

localdirs = listdir(local)
iaudiodirs = listdir(iaudio)

for entry in iaudiodirs:
    iaudio_path = iaudio + '/' + entry
    local_path = local + '/' + entry
    if isdir(iaudio_path):
        if entry not in localdirs:
            print "synching %s from iaudo to local" % iaudio_path
            system('rsync --size-only --delete --delete-excluded \
            --exclude-from= /home/eric/.rsync/exclude -avz \
            --no-group %s/ %s' % (iaudio_path, local_path))
        else:
            print "synching %s from local to iaudo" % entry
            system('rsync --size-only --delete --delete-excluded \
            --exclude-from= /home/eric/.rsync/exclude -avz \
            --no-group %s/ %s' % (local_path, iaudio_path))

With small modifications, this can be made to work with any music player that behaves like an external HD under Linux (obviously paths and directory names need to be changed, I did not try to make this script generic). What it does is run through all the artist directories on my player. If an artist directory exists there that is not on my desktop, it copies it, under the assumption that it is a new artist that I like and picked up somewhere or other. If the artist directory *does* exist, it does the exact reverse: it syncs from the computer *to* the player, under the assumption that I only delete or add single files on the desktop, since it's too much of a hassle to do it on the music player directly. If either of these assumptions are not valid in your case, obviously the script wouldn't work for you without some serious modification.

Metropolis 2007

Sunday july 1st is the annual free Metropolis festival in Rotterdam. I've never actually been, but heard great things about it, and it looks like this year has another great line-up, so I'm definitely planning on going this time. I've tagged and bagged another radio station:

I've started tagging the confirmed artists for Lowlands 2007 (as lowlands 2007, surprisingly,) so watch that space for and ever updating radio station of all this year's edition goodness. I'll add the artists as they are officially confirmed. (There are some unconfirmed artists there, obviously tagged by people who know more. Looks like fun though, Cansei de Ser Sexy, yay!)

And here's a radio player widget thingy, for that station:

New last.fm features rock

There are a few new features last.fm introduced last week (I guess it was last week, it may have actually been earlier, they're sneaky that way.) that make it an even more useful service than it already was.

Along with the introduction of some shiny new widgets that are sure to appeal to the myspace crowd, they threw in some great new functionality:

RSS/Ical feeds for the recommended events.

YAY! Now I can see what artists that I listen to are playing venues near me. This is sure to make my schedule even more hectic and my wallet more empty. upcoming.org has been doing the same thing for a long time, but there are two reasons why I expect last.fm to work better for me: It does only music, whereas upcoming is starting to include a lot of business/tech conferences, which I have other channels for. It filters on my taste or where my friends are going, in addition to location, (upcoming only does locations), and it does that in a smarter way too: it allows you to set a geographic radius, instead of saying, I want to include these and these cities. I'm not sure how much difference this will make in practice, but I do like that I will be notified of stuff happening near me, even if it's happening just outside one of the cities in my area.

One thing I like about upcoming that the last.fm don't yet seem to have is the buttons to send an event directly to a calendar. The ical feed can be used to send all events to a separate calendar, I guess, but I like a little more manual control. The 'Add this to your [foo] calendar' buttons in upcoming are a great thing. (For me foo==google calendar, especially now that it's fully integratable with thunderbird + lightning through this nice plugin. Installation instructions here.)

Configure your overall top artist and top tracks lists to include the last 12, 6 or 3 months only.

This is great if you have been using last.fm for a while, and would like your evolving interests to show up in your profile. For instance, I have listened to a *lot* of Joni Mitchell, and I suppose I will continue to do so, which means that she and other long time favorites tend to keep newer dicoveries out of my charts. Now that I've set the charts to show 12 months, she'll still be in the top 10, but not on 1st place, and other older artists which I've not been listening to quite so much anymore actually have a chance to drop out of the top 50.