Tuesday 13 February 2007 13:00 GMT
I've moved to a new web space provider, and changed over to WordPress instead of my home-grown DIYBlog system. The URL for the blog with remain www.artificialworlds.net/blog, and, thanks to some beautiful JavaScript hackery, I hope all the permalinks will remain valid, but the RSS feed itself will move to www.artificialworlds.net/blog/?feed=rss2
Wednesday 17 January 2007 16:11 GMT
I have 2 new babies to tell you about:
1. Gnome Simple Stateful Music Player which is a music player that just stays out of your face, playing music off your hard drive without making a database or anything like that. It uses GNOME/GTK, GConf and gstreamer and is written in C, which is great fun.
I think C and Python are actually very similar: they are direct languages that are designed to be simple, and allow you to specify what you want to happen in a very straightforward way.
GSSMP is nearly ready for a 0.1 release - it's simple, but it does most of what I want. Its killer feature is that it remembers what you were listening to and continues from there when you restart. Sounds simple, I know, but sorely lacking from some music programs. It also starts quickly, as opposed to, say, slowly or, for example, very slowly.
2. Another one:
Thursday 21 December 2006 10:48 GMT
I've been really enjoying the GP2X I got for our 5-year anniversary (!) and have got DVD ripping working, along with Quake playing at a decent speed, and lots of downloadable games, notably Spout, which is excellent, and (unlike many games which are really too visually complex to see on a small screen) simple enough to work on a handheld. There are a few other good games too, and lots that I immediately deleted off my SD card. I'd like to see some of the very polished LGames ported - maybe I'll get around to it myself sometime.
Anyway, all of that is peripheral really, because what I really like doing is writing games, not playing them. So far I have imposed two games on the unsuspecting public:
mop(e)snake - a classic snake game with no featuritis - just a snake, apples (pain, actually - read the plot if you're interested...) and you. Try to "fill the screen with green"TM. It's reasonably mature now and works on GP2X, Windows and Linux. There's not a lot left to do, except a bit of polish and fixing any bugs that come up.
duckmaze - a maze game where you can move walls. This is based on a game I played years ago on a handheld, whose name I have forgotten. It's potentially quite fun, but I need to design some more levels - so far it only has 8, and several of them are pretty noddy. The first release includes a level editor (go me), so you can help out with designing more. Actually, you don't even really need the level editor, since the level file format is designed to be fairly human-readable, so potentially you could create levels in a text editor. I'm quite pleased with that, even though no-one is ever going to appreciate it now I've written the level editor.
Download, enjoy, write to the mailing lists with ideas, bugs, code etc.
Monday 18 December 2006 15:15 GMT
Dear LJ,
A few months ago I wrote to you saying we had to break up - you had become too chatty and opinion-filled for my taste.
I have to admit that it turns out I was bluffing.
I love you too much, and today I renewed my subscription.
You did take a significant dip in quality, but things have got better over the last couple of months.
More important, though, is that fact that you started from such a high quality base that even when it dipped, you were still the only Linux publication for me.
Yours forever,
Andy Balaam
Wednesday 19 July 2006 02:16 GMT
I screwed up the RPM, so if you tried that, try the latest one from here: <a href="http://freeguide-tv.sourceforge.net/rc">freeguide-tv.sf.net/rc</a>.
Tuesday 04 July 2006 09:47 GMT
Lots and lots of bug fixes, and some cool features sneaked in by people who obviously didn't read my TODO list ;) means that FreeGuide 0.10.4 is nearly ready.
You can test it by downloading the relevant installer from freeguide-tv.sf.net/rc. I've put up a Linux RPM and tarball, and hopefully a Windows installer and other OS/distro packages will be up soon.
Reasons to try it:
Wednesday 07 June 2006 21:07 GMT
No, I did not just develop a passing interest in data recovery as a leisure activity. Presumably like everyone else, I learnt about it because it was necessary.
A year or two ago my friend's computer was packing up, and he asked me to extract and store all their files for him until he got hold of a new one. These files were basically the whole history of their life in photo, movie, word document etc. form - it totalled about 3.9GB. We were pretty successful about extracting the files from his dying machine by booting into Linux and copying them onto my iRiver. I've been keeping hold of them ever since, with a copy on my iRiver and another on my home machine to be sure.
During the fairly disastrous upgrade of my machine from Fedora Core 3 to Ubuntu Dapper Drake that I did last week (of which, probably, more later) I managed to delete all the little scripts I keep in my home directory to do useful things. (They represent quite a lot of collected wisdom, and losing them has been annoying, but anyway.) One of those scripts called other scripts to delete old stuff I wasn't interested in from my iRiver, and synchronise my local copy of my iRiver's contents with the machine itself.
Obviously, I made a small mistake in re-writing that script, which meant that it started deleting everything older than 2 weeks old off my entire home directory. Since I had gone off to bed while this was running, I was extremely lucky that it hit a read-only file fairly soon so I didn't lose much, but what I did lose was the zip file into which I had put all my friend's files.
Of course, my synchronisation program worked like a dream, running as scheduled that night and merrily deleting the zip from my iRiver too. Lesson 1: Don't let your backup program delete files automatically.
If it had been my own data I would have been gutted, but the thought that I had lost someone else's lifetime collections of photos and sentimental things was pretty horrible.
So I began several long nights of investigating data recovery. I assumed it would be easier to get the files off my Linux ext3 partition, since it is well documented and hackable, but in fact, I learned lesson number 2: You can't recover deleted files off an ext3 partition. That is probably not strictly true, but in practice, it is. The reason is that when you delete a file in ext3 it blanks the record of it from somewhere, so all that's left is the actual data with no index entry. This makes it pretty much impossible to recover anything.
I tried lots of different things, including using grep and strings to search the device directly, and debugfs which appears only to be useful for ext2. The closest I got was a program called foremost which knows what the start and end of each type of file looks like, and searches the raw data for things matching the type of file you are looking for. Foremost seems really cool, and it found lots of things that looked like zip files, but almost none of them actually were valid zips, and it didn't find anything big enough to be the file I wanted anyway.
So much for my Linux drive. Lesson 3: Consider ext2 for a backup partition. It is apparently easy to undelete deleted files on ext2. On the other hand, if the computer crashes while writing to an ext2 drive, you are much more likely to have corrupted data than if you use ext3. Quelle dommage.
I find it quite pleasing that the fact that I had a backup meant not that my files were backed up, which would of course be too easy for a person who has recently been branded Master of Pain (Receiving) by his colleagues, but it did mean I had another deleted file on a different filesystem which I could attempt to recover.
My iRiver (which, incidentally, I love even more after this incident than I did before) uses a FAT32 filesystem, which everyone agrees is rubbish, but which has the highly relevant advantage of being simple and thus less difficult to recover data from.
I made a copy of the entire disk by doing dd /dev/sd1 > myhd.raw and then I was free to attempt recovery on this raw file without fear that I would overwrite something on the real disk.
I hoped it would be easy to get the file back off FAT32, but it turned out to be difficult as well, mainly because the file was so huge. I tried lots of different programs, most of which crashed unceremoniously, or couldn't find my file. Lesson 4: Don't give up. Many of these programs were lying to me - telling me to give up. Had it been my own data, I would have given up. We are several days into my nightly trial now. I was tired.
Then, like a shining beacon of professionalism and respect for data, over the horizon came Autopsy and its underlying Sleuth Kit tools. I installed them, and started the server by typing autopsy and then pointed my web browser at http://localhost:9999 and followed the instructions. It could see my file! I told it to recover it, and my browser said it was downloading the file.
And then it failed. I tried again. It failed.
So I tried with wget. It failed. I tried with curl. It failed.
I realised the problem - the file was over 2GB, which is not supported in some downloading software! I tried with wget again, after confirming that it supported large files. It failed again.
Lesson 5, which really I should have figured out ages ago is: Don't use large files. Especially not large zip files containing all your precious data, obfuscating the files from your recorvery program.
Now, I knew autopsy was running on my local machine and then providing the file I needed as a download, so I knew I could access the file directly by using the tools autopsy was using underneath.
Autopsy, may the Lord bless you for making those tools accessible separately.
Autopsy had told me the inode number of the file I wanted, so a quick read of the icat tool's man page showed me what to do. I ran the command and ended up with a 3.9GB file!
So I unzipped it.
Obviously, it failed.
Full of optimism, I assumed that this really was my file, but that it had been corrupted slightly by having some of its data over-written. I started looking for zip file recovery programs.
To cut a long story short, there are a lot of zip file recovery programs for Windows and Linux, and some of the Windows ones appear to work using Wine, but in the end (which I should have thought of from the beginning) the one that actually worked, rather than tantalising you with the names of the files and then either crashing or charging you money to recover them, is the one written by the guy who invented the zip file. The trial version of PKZip for Windows, running on a Windows machine successfully opened and repaired my damaged archive, with this summary message:
Extracted 3,151 files
Skipped 0 files
62 errors/warnings
Yes, out of 3,151 files, 62 were damaged, and the rest were fine.
It was a long journey, but we made it. Finally, lesson 6: If it's important, have 2 backups.
Thursday 18 May 2006 15:41 GMT
Really encouraged to receive this:
Hello Mr Balaam,
A quick email to thank you for giving the world the brilliant Freeguide.
(I saw in your blog that you have a high volume of email - no reply needed).
I tried Digiguide for a while, loved the principle, found the interface too cluttered. Went back to downloading pages from here and there, and dipping into the Radio Times timeline, which is not quite customisable enough.
I happen to live in a place where I get an unusual mix of channels - UK free satellite + UK terrestrial +Irish terrestrial. I'd never found a single listings service which covers the options.
Then, by chance, I found Freeguide, and I love it. I couldn't ask for more (except, perhaps, for customisable colours for program categories :-) ) Incidentally, one of your blog entries wondered how many painless installs there currently were. Here's one more to be counted (WIn2K, Sun Java v5Update6)
I don't know much about software, but I'm sure this must have taken a vast amount of your time. Thank you for every moment of it. This program will be useful every day, and I'll be passing it on at every chance I get.
Best wishes
Nick Nixon
Monday 15 May 2006 08:44 GMT
Picture the scene: you are writing a Java application and you're trying to do things right, so you use the java.util.logging code to do your logging. To create a logger you do this:
Logger log = Logger.getLogger( "name" );
Everything goes fine until you want to set the logging level from a command-line argument. You set the logging level like so:
log.setLevel( lev );
Where lev is, e.g. Level.FINE.
When you run your program, none of your log messages appear except the ones that appeared before (SEVERE, WARNING, INFO). Nothing appears to have changed.
You read some API docs, and find out that each Logger has a list of handlers. So you call getHandlers() on your Logger.
It doesn't have any.
You take a breath.
You read some more, muttering about Sun engineers having spent too much time at university, and find out that there is a hierarchy of Loggers containing other Loggers.
You simultaneously wonder how that could be useful, and whether some stupid thing is happening here to do with that.
You try log.getParent() and find that it is not null. Eureka! You've got it now:
log.getParent().setLevel( lev );
You hastily recompile, palms sweaty with finally cracking a problem that should not have taken this long.
Nothing has changed. You still don't get your log messages.
OK, that's fine: you know about handlers now, so it's a simple change - we'll do this instead:
log.getParent().getHandlers()[0].setLevel( lev );
You recompile, trying not to get your hopes up.
Nothing.
You consider re-writing your application (which has been in development over 5 years) in any other language. You decide to try one more thing before giving up. What if you need to tell both the Logger and the handler what level to use? It would be pretty awkward, but not outside the realms of possibility.
log.getParent().setLevel( lev );
log.getParent().getHandlers()[0].setLevel( lev );
Not even very hopeful now, you recompile.
Nothing.
This is the moment where you go and find anyone who will understand (or failing that anyone at all) and tell them about it. They wipe the spittle from their faces and appear keen to leave.
You will not be defeated by this. You try every combination you can think of. You start looking into how to log a bug with Sun. You curse and curse again.
A very long time later, you know you should have given up, and you're trying things just out of bloody-mindedness, and you stumble across this:
log.setLevel( lev );
log.getParent().setLevel( lev );
log.getParent().getHandlers()[0].setLevel( lev );
Yes, that's right. To set the log level of the default logger in Java you have to learn about handlers and the logger hierarchy, and you have to set the level in three places.
THREE PLACES.
You swear never to feel superior again when people start having "language wars" about which programming language is better. They are not all the same.
You engage in a relaxing pasttime, such travelling to Sun's development centre and punching the person who "designed" that interface in the face.
Thursday 11 May 2006 11:03 GMT
I've had this problem a couple of times in my Ubuntu machine. I installed it with US keyboard layout, and then changed it to UK later (for my user). Even when I select the correct keyboard model (Generic 105-key (Intl) PC), and layout (United Kingdom International) some keys still act weirdly (e.g. pressing " does nothing, and if I press it twice I get a strange "-like symbol). I fixed it by running:
sudo xmodmap /usr/share/xmodmap/xmodmap.uk
Tuesday 02 May 2006 15:59 GMT
Christian and others have pointed out lots of problems with the source code layout in FreeGuide. The biggest problem is that each plugin has its own source tree which makes it very difficult to set up in an IDE like Eclipse. I knew it needed doing, and the other night I couldn't sleep and I felt like making it happen. Since SVN is so much better than CVS for re-arranging source code (because you can move files and keep their history) and because sourceforge are now offering SVN, I decided the right thing to do would be to migrate to SVN before making the changes.
So far, I have managed to get all the source code to build from the command line (instructions here: freeguide-tv.sourceforge.net/dev/index.php/Build_from_SVN) and I'm working on making the Ant build work again, although there's still a bit to do there.
My aim is for it to be extremely easy to start developing FreeGuide: just download or check out the source code, fire up your editor or IDE, compile and run. That means FreeGuide needs to handle loading plugins from either JARs or directories (without being told which to do), and it needs to know where to look for its lib and doc directories if they are not specified.
Soon, it will Just Work. Then hopefully this will encourage more developers to get involved.
Meanwhile I've decided if I want to build up some momentum I need to balance my limited time between code and email, instead of just fire-fighting the email all the time. Some individual users may not get their questions answered, which I hate, but in the end it would be worse if the project lost interest and faded away. I need to inject some excitement!
Friday 17 March 2006 09:06 GMT
Dear LJ,
I'd like to start by saying thank you very much for the years of quality articles and features that your magazine has provided. I eagerly await the arrival of each month's copy as it jets over the Atlantic to my doorstep. LJ has always provided detailed and informative articles on topics both mainstream and more quirky, but always with the deeply technical edge no other magazine or web site I have found can provide.
I am writing because the April issue proved a grave disappointment. Gone were the technical articles and gone were the interesting projects. In their place I found a plethora of opinion pieces most of which provided no new insight into their subject that I could not have found on Slashdot or OSNews, along with some beginner's articles on how to install a couple of PHP applications. Even the Doc had nothing interesting to say about the yesterday's-news trying-to-be-trendy refrain of "blogs, wikis and podcasts" which seemed to be repeated several times on each page. The articles which did contain decent technical content (notably "Work the shell" and "Remote Temperature Monitoring with Linux") were swamped beneath the river of fluff through which I found myself wading.
When Nicholas Petreley wrote his first uninformed and ill-judged "/etc/rant" I felt I could let it pass since it was just one page of an excellent magazine, but now that his style of writing - opinion rather then content - seems to be spreading to the rest of the magazine, I feel the need to express my disquiet at what he has been doing with this column.
/etc/rant is an article put aside each month for criticising free software projects. The fact that an editor would allow such an article to exist, let alone write it himself, displays a profound and worrying misunderstanding of the community to which he is trying to contribute. Free software projects live and breath through encouragement and just plain getting on with it. As a member of this community, the right way to express dissatisfaction with the way something is being done is to contribute to the project, or (if necessary) launch or encourage an equivalent project that does things differently. By criticising from the sidelines Mr. Petreley reveals himself to be an outsider - as a writer a more appropriate way to contribute to the community (and hence form a useful part of it) would be to praise and encourage projects which are doing the right thing, rather than sucking energy out of valuable and innovative projects which have involved large amounts of volunteer effort such as GCJ and Gnome.
I never thought I would say it, but if this continues I will be cancelling my subscription to LJ. Sadly, I don't think I'll ever find another magazine that will reach the level of quality I have come to expect from you.
Yours sincerely,
Andy Balaam
Tuesday 14 March 2006 16:52 GMT
jdub notes that Linux Journal is going to the dogs. If the only change were to the back page I could get over it and move on, but so far I am about half way through the latest issue (March 2006?) and I've read nothing but opinion pieces.
I can get all the rantish uninformed opinions I need on OSNews. When I want quality technical articles written by people at the top of their field which cover hardcore problem solving and creative projects, then I turn to LJ. That is why I pay REAL MONEY for a paper journal like this. Because it has substance.
If I have to cancel my subscription, where should I go?
Wednesday 08 March 2006 09:37 GMT
Forgot to mention: I got my first non-anonymous donation to the FreeGuide project - Steve Fuller sent in a donation along with a request (but with no implication of linkage ;). Thanks Steve! It never ceases to amaze me how Americans will donate to projects like this. My feeling about British people (certainly myself) is that I would always think if I were going to donate to something I'd make it something "important" like Christian Aid, but actually I do think Free Software is important too (actually in order to achieve some of the same goals that Christian Aid have...). The idea of mixing philanthropy with self-interest, which seems to be natural to Americans, can feel a little dirty to us Europeans, but actually that is stupid.
Anyway, enough philosophy: Steve, your donation will be used to buy myself something nice (e.g. a GameCube game or some online poker money) and it has significantly encouraged me to get stuck back into FreeGuide development (see the previous post ... guilt is rising). You never know, it might even subconsciously encourage me to get to your specific feature request sooner ;).
(The feature request was allowing favourites to match on programme descriptions and other, more specialised fields. I am wondering about how to keep the UI simple for this, but it should be do-able given some time to implement it.)
Wednesday 08 March 2006 09:28 GMT
I have very little time for FreeGuide at the moment. Part of the problem is that there are lots of boring bugs to fix instead of interesting new features, and it's hard to motivate myself to get into it. I am really keen on making it Just Work every time, but since other people have been working on the code and quite a lot of it is unfamiliar, it's a lot of work to work out where to look to fix a problem.
Hopefully when I've purged this poker thing from my system I will find time to get into the latest TODO list. At the moment all my FreeGuide time is going into trying to keep up with helping people on the mailing lists. Joe Blow is doing a fantastic job of providing an initial response to lots of mails, but often people are encountering weird bugs that need a developer to get involved. It would be great if some of the other people involved in development were able to help out, but it's quite a thankless task, and takes away time from developing...
It's frustrating when quite a few of the bugs we have are things I fixed in a previous version that have crept in with new implementations of existing functionality. It's taken ages to get FreeGuide stable, and recently it seems to have taken some steps backwards. My feeling is that the percentage of people who are able to install it with no problems and download listings straight away has gone down considerably. I _really_ want to fix this.
That's why the TODO list for 0.10.4 is pure bugs. We need to make it _work_ before we make it do new stuff. Otherwise people are just going to walk away.
Saying that, the addition of a new experimental vertical listings view by Christian is really exciting. I haven't even found time to try it out yet: I must examine my priorities. FreeGuide has always been my "main" project but I have been neglecting it as time presses in.
The Boy (as I call him) is doing well and has slept well the last 2 nights, giving us a break from the all-over-body pain of extended tiredness. If this is part of a trend towards more sleep, I may get more time to make FreeGuide better.
Tuesday 07 March 2006 10:41 GMT
Forgot to mention that I also have one other optimisation up my sleeve: instead of fully analysing each hand I could only do that when I need it i.e. when the hand type is the same, so I could just return "flush" and only if your opponent also has a flush would I need to say "flush ace high" or whatever. This provision of the detail is actually quite time-consuming, so it may be a significant improvement to avoid it in most cases.
Current timings (without the above optimisation) on a P4 2.4GHz machine:
[andy@andy-p4-linux src]$ time ./poker 5 # Analyse all possible 5-card hands Straight flush: 0.001539% Four of a kind: 0.024010% Full house: 0.144058% Flush: 0.196540% Straight: 0.392465% Three of a kind: 2.112845% Pair of pairs: 4.753902% Pair: 42.256903% High card: 50.117739% real 0m4.058s user 0m3.805s sys 0m0.002s [andy@andy-p4-linux src]$ time ./poker 7 # Analyse all possible 7-card hands Straight flush: 0.016811% Four of a kind: 0.168067% Full house: 2.555057% Flush: 3.039766% Straight: 4.475084% Three of a kind: 4.873941% Pair of pairs: 23.512489% Pair: 43.946865% High card: 17.411920% real 3m43.479s user 3m42.963s sys 0m0.047s
This is nice, because there are 51 times as many 7-card hands as 5-card ones, and the analysis only takes 55 times as long even though it's more complex to analyse larger hands, so it means my analysis scales pretty well with the number of cards.
Tuesday 07 March 2006 09:59 GMT
I've just requested sourceforge space for my latest mini-project: Poker Analysis. I've been a big fan of poker for years, and am pleased to see everyone is catching up with me finally with the boom of Internet poker. I've started playing a little online, and it set me thinking about the algorithmic aspects of analysing the game. Obviously, the real skills of poker are about predicting people's actions, but before you can do that you need to be able to see what hand you've got and what possibilities are out there, and that stuff is entirely deterministic.
I like small, self-contained challenges, so I set about writing a program to play every two-person game of Texas Hold 'Em poker that could ever happen, and ranks the two-card hands you can have in order of how many games they win. Obviously most people have a pretty good idea of which hands are good or bad, but I bet there would be some interesting results to see the full ranking.
Of course, I'm sure lots of people have done this before, but I want to do it too! Also, of course, it doesn't matter how good your hand is in the grand scheme of things, just whether it beats the other hands out there, but bear with me: I'm interested in the results.
The major challenge is getting it to run fast enough, and that is what I'm finding interesting. I've already changed my algorithm several times and made huge improvements in execution time, but to run through all the trillion or so (I think) hands would still take several years to execute on my home machine, so I've got some work still to do.
There are 2 major parts to the algorithm: enumerating all the hands you could get, which sounds easy but is in fact slightly tricky, and difficult to do perfectly if you want to avoid repeating redundant analyses, and finding out what hand you have (e.g. pair, flush). I'm pretty happy with my current implementation of the second, but I've got quite a lot of work to do on the first.
To analyse what hand you've got I'm taking an approach where all the real work is done when you sort the hand, and then looking for pairs or straights is easy since the cards are already next to each other. I then re-sort the cards so that the same suits are next to each other, and that makes it easy to find flushes. The part I'm really happy with is that the way I choose hands to analyse means they are already sorted, and when I transform them into the sorted-by-suit instead of by number order, I have very little work to do the way I've done it. (Some slight extra complication is added by the need for aces to be either high or low in straights.) This means that deciding what hand you've got is pretty quick, and I am able to enumerate and analyse every 5-card poker hand (2.6 million hands) in 4 seconds on my home machine.
I believe I might be able to speed up the analysis a little by changing my encoding of cards which at the moment is just consecutive integers (e.g. 0=2c, 1=3c, 2=4c, ..., 12=Ac, 13=2d, ...) but might be better to have each suit starting at a power of two to allow the use of the shift operator instead of %13 or something, but I'm happy for the moment. Where I need to improve is in writing the code that enumerates all the hands. As I have it at the moment, it plays every 2-card hand against every other 2-card hand for every 5-card set of community cards (but obviously preventing repeats of the same card) but this is hugely redundant (something like 24x?) since the suits are symmetrical. Exactly how much redundancy I can squeeze out of it is what I am thinking about now.
Anyway, hopefully soon the code will be in sourceforge CVS, and when I've achieved my goal of examining all games I'll make a release. Hopefully in the future I'll make it accept as arguments the current situation in a game and output the probabilities of each player winning.
Monday 27 February 2006 10:43 GMT
I use a modestly-modified BashPodder to download my podcasts (Internet radio shows) automatically every night, and sort them into different directories. If anyone is interested in my modifications, do ask and I can send them to you. I haven't sent them to Linc since there seem to be a million modifications, which probably do everything mine do, but better. The only modestly interesting thing I added was the ability to convert from m4a format to mp3 automatically, and that doesn't seem to work, and the Adam Curry PodFinder show is so bad that I realised I didn't want to be able to do it anyway (note no link since it's really not worth it).
Anyway, that brings me on to my main point, which is that I have subscribed to a lot of podcasts, and unsubscribed to a lot too. I have now honed my list to ones that I really like, so why not share?
Basically everything else I've listened to is almost completely not worth it.
Look out this year for loads of corporate AdvertCasts and avoid them.
Monday 27 February 2006 10:14 GMT
I used to do something clever with sendmail, but then spamcop blocked my computer because it apparently wasn't set up properly, so now I use my ISP's smtp server like everyone else. When I need to tunnel in I simply create a tunnel from localhost's port 25 to my isp's smtp server's port 25, and set localhost as my smtp server on the machine from which I am tunneling. It works fine, and there was really no need to use my own sendmail in the first place, it turns out.
Next time: how I made my own webmail server.
TODO for this blog:
Monday 06 February 2006 14:38 GMT
Monday 12 September 2005 11:11 GMT
My ISP provides my email by POP3, so I use fetchmail to download it every 10 minutes. I created a file .fetchmailrc file in my home directory that looks like this:
# Configuration created Wed Jul 27 06:43:07 2005 by fetchmailconf
set postmaster "andy"
set bouncemail
set no spambounce
set properties ""
poll mail.myisp.co.uk with proto POP3
user 'andy@myisp.co.uk' there with password 'password' is 'andy' here
This tells fetchmail to use the username and password given to me by my ISP to download mail into the default location for the user andy on the local machine. This would normally put my mail into /var/spool/mail/andy, or something like that, but to get it into my Maildir (which I talked about in a previous blog entry), I added these lines to a .procmailrc in my home directory:
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$MAILDIR/log
Now when I type the command fetchmail (logged in as andy) I get something like this:
$ fetchmail
fetchmail: Server CommonName mismatch: localhost != mail.myisp.co.uk
fetchmail: Server CommonName mismatch: localhost != mail.myisp.co.uk
fetchmail: Server CommonName mismatch: localhost != mail.myisp.co.uk
fetchmail: No mail for andy@myisp.co.uk at mail.myisp.co.uk
(No idea how to get rid of those errors, but they don't seem to cause any problems.) And if any mail was found, it appears in my Inbox in Thunderbird, as if by magic.
Next time: how I send mail, using sendmail.
Monday 12 September 2005 10:59 GMT
So I've put out FreeGuide version 0.10pre2, and then a little bug-fix release because the Linux RPMs were broken. By calling this 0.10pre2-2 I managed to annoy my faithful Gentoo packager Christian, and myself, but at the time it seemed too much hassle to bump everytjing up to 0.10pre3.
The next release, 0.10.3, is going to be stable, and everyone is finally going to be using and testing our latest code stream. We've had a few bugs reported again 0.10pre2, and we've already got a couple to fix, and then we're going to push it out asap and get everyone migrated up.
I hope people will like the new version - it's a lot faster, and the UI experience is better (in my opinion). I just hope there aren't too many wrinkles in the upgrade path.
Alex has written some good upgrade code, but some of the concepts have changed, and I'm not entirely happy with the depths you have to delve into the Options screen just to play with your XMLTV grabber, but most things should "Just Work" for normal users.
After 0.10.3, I really want to get recording working, using Reuel's plugin. If anyone wants to send me a UK digital TV card, you can be sure it would be put to good use...
Wednesday 31 August 2005 13:19 GMT
On the same machine that was running Dovecot, I set up Thunderbird to be able to access my email. I did this by creating a new email account (which I called "Andy IMAP"), choosing "localhost" as the server, and entering the username I created in /etc/imap.passwd ('brian' in the example I gave) as the username. I set up all the other settings as I wanted them, and clicked OK.
When I clicked "Get New Messages," Thunderbird asked for my password, and I entered the password I had put into /etc/imap.passwd ('sausages' in the example I gave). Now I could see some folders had appeared under the new account I had created, but there was nothing in them.
I tested whether I could read and write emails to the account by dragging an email from another folder (in an old account) to the "Inbox" folder in the new account "Andy IMAP". It copied OK, and when I clicked on Inbox I could see the email in the list, and when I clicked the email its contents appeared in the preview pane. Success! My IMAP account existed.
Next time, how I made emails that people sent to me appear in my new Inbox.
Friday 19 August 2005 12:23 GMT
Recently I have managed to get pretty much my ideal email system set up. This is the kind of thing that would simply be impossible on a Windows box, but (when you know how) is a quick job on Linux. I didn't know how, but now I do (ish). I'm going to do it in separate installments, trying to make them slightly self-contained.
I chose Dovecot because it looked fairly good, and it came high up on Google. Installing it on Fedora Core 3 was just:
yum install dovecot
But then I needed to set up the config file. The only thing I changed looked like this:
default_mail_env = maildir:~/Maildir
I did this because after reading the scare stories about locking when using an mbox-type mailbox, I decided Maildir sounded safer. Maildir holds each mail in a separate file, which sounds good for preventing corruption if something goes wrong, so I chose it. It seems to work ok.
I left the passwords to be plain text for the moment, which meant my /etc/imap.passwd had to look like this:
brian:{PLAIN}sausages:500:500::/home/brian
Where 'brian' would be your username and 'sausages' the password you have decided to use for mail. If you're using plain text passwords, don't use the same password as your login password, because any old packet sniffer will be able to see the mail password.
With this set up, I had a working IMAP server which stored messages in my home dir under the ~/Maildir directory. Of course, there were no messages in there at the moment - that's coming up later. Next time, how I made Mozilla Thunderbird see the messages stored there!
Tuesday 09 August 2005 08:56 GMT
Now that my Christian youth holiday is over, I've got just one more major job to do before I can get back to working on FreeGuide: my thesis corrections. I don't know how long it's going to take, but it will certainly be painful. It might be best to avoid me for a few weeks if you don't like "grumpy Andy".
Friday 22 July 2005 08:08 GMT
I really must get BashPodder working, not only to download my two favourite Linux shows TLLTS and LugRadio but also because Radio 4 has started making Today and the best radio programme out there, From Our Own Correspondent available for podcast download.
Tuesday 19 July 2005 08:12 GMT
Sometime I find myself thinking in Python. Or, accurately, wistfully thinking of Python. Raw C++ is completely hamstrung when it comes to simple concepts like handling strings or dealing with lists (maybe BOOST is better?). Why can't I just do this:
remaining_args = sys.argv[3:7]
I don't even mind if it looks like this:
PyList argv = new PyList( argv, argc );
PyList remaining_args = argv.slice( 3, 7 );
Is this kind of thing possible? If not, someone really should package up the Python library in such a way that it is.
Of course, I'm speaking from a position of total ignorance here: maybe someone has done what I'm talking about with the Python stuff. After a brief look at BOOST, I can say for sure that it continues the STL's tactic of "under-use through obscurity" and "surprisingly missing all the useful features". No slice operator, anyway...
If they have done/do do the Python thing, the most frustrating part then is going to be wishing you could have all the bits of the Python library that are written in Python. Does anyone fancy starting the "GCP" project?
Sunday 17 July 2005 11:06 GMT
I'm now thing of moving off Fedora Core 3 (probably to Ubuntu, or maybe FC4 for the Java stuff) and it turns out I've never written a CD from it, because when I tried it didn't work: I kept getting the message "cdrecord: Success. Cannot open SCSI driver." I finally managed to find a solution. For some reason I need to do this:
cdrecord dev=ATA:1,1,0 -eject -v speed=2 ubuntu-5.04-live-i386.iso
Note the ATA: in the dev bit. No idea why.
Friday 15 July 2005 09:47 GMT
arrived through the post from ebay yesterday. It is superb. Exactly my kind of game, and with loads of features from the old game (one of my favourites of all time), all updated and made even more beautiful. It is incredibly atmospheric, and so far the gameplay has been utterly compelling. Am I more excited about this than the pervious post? Surely not.
Friday 15 July 2005 09:46 GMT
...and I am not a big fan of pictures in blogs, but this is a special occasion:
World domination is nigh. Soon we shall all be Balaams.
Wednesday 13 July 2005 08:46 GMT
I have requested a Sourceforge project for DIYBlog, and it just struck me again how utterly fantastic Sourceforge is. What would the Free and Open Source Software world be like now without Sourceforge? A lot poorer, I reckon.
I'm desperately trying to find time to test FreeGuide for a 0.10.2 release. So far it looks like although my favourites have been imported, they are not working. In fact, when I create a new favourite it works for that run but when I restart the program they are not highlighted. Hmm.
Friday 08 July 2005 08:23 GMT
London is a little quieter than normal this morning, but still pretty busy. It seems like most people are going to work, although some are having to walk. There are definitely fewer tourists on the streets.
Thursday 07 July 2005 12:46 GMT
As a largely unmilitarised country, Britain (like many other Western countries) is a soft target for terrorists. But that very softness makes attacking us in this way pointless: it's like punching a sponge. We won't change - we like our country how it is.
It's possible that by drawing attention to themselves, the people who did this may have changed some aspect of the world, but as for advancing their cause - how can they when the only discernable motive they have is hatred? Does murder really win them support at home? I doubt it.
We in Britain have done and continue to do some terrible things. I wish we would stop, but today isn't going to make any difference either way.
If you or anyone you know was hurt today, I'm sorry.
Thursday 07 July 2005 08:45 GMT
I was speaking to someone over the weekend who works in HR and has been enjoying making various Microsoft Access databases to do useful things for work. I warned her she might get the programming bug and then never be able to stop. Then I realised she already had it when she summed it up beautifully:
"You can make it do things."
I love coding.
Wednesday 06 July 2005 08:34 GMT
For my train journey, I ended up downloading a few emulators that didn't work, and a few games that didn't work, but what did work was the PC version of Prince of Persia, and the PC version of Cannon Fodder, both of which were very enjoyable.
Especially good were some new levels someone had created for PoP - a lot harder than the original ones! It did show how much work went into creating games, even back then, though, since there were a couple of subtle bugs in the new levels that led to you getting trapped with no means of killing yourself, which means your entire game is over and you have to start again, which is quite frustrating.
I did dig out the code for my track and field game, but when it appeared all on one line with weird symbols for the carriage returns, and I contemplated the idea of fixing this (and coding generally) in notepad, I decided to go back to PoP instead.
More on the track and field game later. It's in Python...
Monday 04 July 2005 08:24 GMT
I know it's supposed to be best practice to force users to change their passwords regularly, but how are we supposed to remember them? I wonder whether anyone has done any research into passwords becoming lower quality as people are asked to change them. It just becomes increasingly difficult to think up memorable things that are not obvious, or look for ways of fooling the computer into letting you have a similar password every month.
Friday 01 July 2005 08:14 GMT
We're taking a 4.5 hour train ride to Leeds at the weekend, and then even longer on the way home, so I am hoping to entertain myself using a Windows 2000 laptop with no CD drive. The options I'm thinking of are:
The easy way would have been to run Games Knoppix, but there's no CD drive. Gutted.
Tuesday 28 June 2005 13:10 GMT
We've already had a bug report/feature request for the ATV Palm plugin Alex wrote, so it's obviously going to be pretty popular. It's really great that Alex is doing so much work on FreeGuide at the moment. We are having our disagreements here and there, but FreeGuide is getting a lot better all the time.Tuesday 28 June 2005 13:02 GMT
FreeGuide is hopefully nearing a new release. Alex has been working on the plugins system, and hopefully I will be able to test it this week. If it works ok, we'll release it as unstable, and then with a few bug fixes we should be able to make the next release stable. Alex has refactored the code so that the main program is just another plugin. This is pretty cool as it means we can upgrade everything without restarting the program (hopefully, although in practice we think we may hit problems with resources not being freed). We are currently discussing exactly who should be able to upgrade what plugin when. I am keen that users be able to get a new listings grabber as soon as they need one without needing their admin to do it for them. On the other hand, probably only the admin should be able to upgrade the main app. The way jEdit does these things seems to work pretty well.
Friday 24 June 2005 01:27 GMT
When I wrote DIYBlog it occurred to me that what I was writing would translate into a blog aggregator pretty easily, and so I'm working on making that so. I'd like to start a little community of Java developers who are working on small free and open source software Java programs, so we can start to communicate through our blogs and help each other out with thorny issues e.g. how to get our programs working with an all free software systems (more on that later).
For the moment I'm working on aggregating my own favourite feeds, which you can see here: Planet Andy. All this is/will be running on pure FTP web space, with a little cron job running on my own machine to keep it updated. Of course, that does mean it only works when my machine is turned on...
Thursday 23 June 2005 13:26 GMT
Been discussing with Alex about the dangers of re-implementing - he wants to simplify the command-line arguments code in FreeGuide, but rather than modifying the existing code, he's re-written it. The problem with re-writing is that old code usually has lots of non-obvious bug fixes in it which you will have to re-implement all over again as they are reported. Alex has re-written a few things, and generally he's improved the code structure, but we have both then had extra work to do fixing the bugs that are inevitably introduced. That can seem like a waste of time if they're bugs that I have already fixed in the past.
It always feels easier to write your own new version instead of understanding what's already there, but in reality, unless you're changing the output, it's never a good idea completely to re-write the internals. There's a good article by Joel Spolsky on this.
Wednesday 22 June 2005 11:09 GMT
...for a while.
Woke up at 4.30am today having awful dreams about how none of the code worked. My dreams transposed the problems I am trying to solve at work onto the code I need to write for my corrections. I couldn't stand it any longer so I got up and started work on it, and I've launched an experiment now which should hopefully provide all I need in terms of experimental work. So, it's not as bad as my dreams...
The idea of getting back into the writing part also terrifies me. I guess another few weeks of getting up in the small hours might get it done, but I'm likely to lose all my friends. Obsessing about this is already making me totally inward-focussed. I am trying to make an effort to think about what other people might be feeling, to try and keep my self-pity in check.
Tuesday 21 June 2005 08:07 GMT
Well, I passed, so I should be happy, but I've got to do some corrections before I become a doctor, so I'm feeling disappointed, but really I'm very close to finally achieving what I've worked so hard for. Just one last push.
Monday 20 June 2005 05:56 GMT
Today I find out whether I get a PhD. I've been strangely not-nervous about it, although I've woken up pretty early this morning. It's difficult to `revise' since most of what we'll be talking about is what I wrote. I've been trying to refresh my memory of papers I cited (especially those of my examiners...) and of what I wrote, but it's been so boring I haven't done much. Let's hope I don't regret it later.
I'm fairly hopeful about passing - I would hope my supervisors would not have let me submit if it wasn't going to pass, but I see the viva as being both a rite of passage everyone has to go through (i.e. a horrible experience that my examiners had to go through, so they want to inflict it on me, and a negotiation process about how many corrections I need to do.
I'm not worried about having a hard time today: I tend to stand up reasonably well under face-to-face pressure like this, and I hope I can robustly defend points of which I am confident, and give way in places where I should refine my thinking. I'm more worried about the corrections. I just can't imagine getting back into writing my thesis - I've left it so far behind my thinking, I'm so sick of it, that the idea of writing another chapter or two is horrific. It was bad enough the first time.
Friday 17 June 2005 11:53 GMT
I made a page for the band Pete Beresford and me had when we were at uni. You can download the 3 songs I was getting around to sending to John Peel. If you like your metal instrumental and a tiny bit Propellorheads, it might be for you: Mnemonic. If you like it, let me know and I'll send you some FLAC recordings of those songs (still not great quality, I'm afraid), and/or some of our other songs.
Thursday 16 June 2005 16:19 GMT
Take a look at the latest TODO list. There are several categories of things on it:
I should go through and make sure everything is relevant, but also I need to prioritise what to do before 1.0 if we are ever going to get there. 7s are being done by Alex and others, 2s really need doing if we are going to call it finished, 4s should get cleared up during the beta stage (hopefully), most 5s are likely to get dropped, and 3s and 6s are good potentials for a public vote - then the most wanted ones will get done and the others will have to be post-1.0.
I may go through the list and actually classify them this way, and then we can see what we're doing.
Wednesday 15 June 2005 07:53 GMT
I couldn't sleep last night, I think because my viva for my DPhil (PhD) is coming up on Monday. Work is going to be a struggle, but at least I got a chance to think about FreeGuide.
Alex is working on a plugins system (it's sort-of-working in CVS) that will allow us to split the grabbers out from the main program, and allow people to innovate with new plugins without delving into all the code, which is quite large and a bit messy to say the least. Both of these things are really good, especially because my own time is very limited, so allowing people to work on plugins seperately means progress can continue without me being such a bottleneck.
However, handling plugins means we need to change the way FreeGuide is packaged. I've had some thoughts about how to do it, and I've put them together into an OpenOffice.org Impress presentation (or try the PDF). I'm looking for comments on this before we finalise the system. Once it's established it will be hard to change, so I want to try and get it right.
If there are a couple more sleepless nights we might even see FreeGuide's 0.10.x branch going stable ... Alex has a few more bugs to fix, but we're getting there.
Tuesday 14 June 2005 09:17 GMT
I've made a page about how I'm making this blog, here: DIYBlog. I may even get around to putting it on freshmeat at some point, and if others want to contribute I may even put it on SourceForge.Monday 13 June 2005 16:17 GMT
I think I've managed to work out a system to make RSS work. Basically I write my blog entries into a raw RSS file, and then run a little Python script to add ids, links etc. and generate an HTML page from that. It's been a hassle to write (mainly due to timezones, which I've now decided to ignore) and if anyone's interested in some highly-customisable blogging software that only requires FTP on your web host, get in touch, especially if you're the kind of person who thinks its cool to hand-code your RSS.
Thursday 09 June 2005 13:23 GMT
By Thursday 9th June 2015 30% of corporate desktop computers will be running operating systems (and desktop environments) which satisfy the Open Source Definition.
Wednesday 08 June 2005 08:22 GMT
I really should provide an RSS feed. Does anyone know of a good reference on the simplest form of RSS that I can use to auto-generate some RSS from the HTML of this page? I'm prepared to do a little python coding to make it happen. Give me an email if so.
Speaking of giving me an email, wouldn't it be great if I could have a proper blog that allowed comments? Well, I'd need a web host that allowed PHP or something, and that would cost money. If anyone knows of a good free hosting service (built on Free Software, preferably) let me know. I've registered on advogato.org but although I have finally managed to work out how to post to a blog, it doesn't allow comments, so the only benefit it offers is an RSS feed, and the lack of control is not worth that to me yet. I may get frustrated with my home-grown HTML/FTP solution, but for the moment I'm quite enjoying it.
Wednesday 08 June 2005 09:03 GMT
Having previously been the anonymous benevolent dictator of Guilty Expression I have decided to become non-anonymous and I'll just have to make sure what I write there is either acceptable or I'll have to log out before I write it. Guilty Expression is a web site for people who want somewhere to express the thoughts and feelings they have about God and Christianity that they can't express in their church (if they have one) or with people in real life.
I set up the sire after talking to several of my friends who used to be church-going Christians, but were hurt in one way or another (by their church, or by God in the sense of things happening to them) and are now almost-Christians, or ex-Christians but who still think a lot about these things and want to talk about them without being branded unsound or a heretic. A couple of these people have very genuine relationships with God, which, by their very messed-up-ness are very real, but they obviously suffer from not being part of a community. GE is supposed to provide a way back in to talking to people like them and people who feel more confident in their faith (but are very tolerant of those who aren't).
GE is also a place for Christians to complain about their churches and the attitudes of other Christians, and other people, and almost to let off steam about things in a way that does not have very bad consequences in terms of hurting people or causing division in their church or group.
We've had articles written by people of lots of different ages and backgrounds (from pastors to non-Christians) on loads of topics (from anorexia to the bible to science) and lots of different viewpoints.
My excuse for writing about it on what I hope to be a mainly technical blog is that it presents an interesting challenge in terms of how to build a community. The site is supposed to be for people who are feeling disposessed and out of it, and so they're not likely to be looking for it. The vast majority of visitors and contributors to the site are people I have told about it personally, and it has spread a little from there by word of mouth. My being anonymous has not helped, because it has made it difficult for me to spread the word through my other web presences, and that ti why I've decide to own up. If you have any ideas about where it might be useful to spread the word, let me know by email.
If you're interested in GE, go to the site: www.guiltyexpression.org and consider signing up for the mailing list to hear every now and then when a new article is published.
Tuesday 07 June 2005 08:12 GMT
Getting used to working is an interesting thing. I haven't been as tired as I thoguht I might be, probably partly because I got into a routine of getting up at 7am with my wife (she was working while I did my PhD). However, the weirdness of how much of a job is to do with social pressure is interesting. What I mean is that it takes a lot to get fired (in my job, anyway - I hope...) and you could probably get away with being pretty mediocre and keep your current salary. The things motivating you to do well and work hard are mainly desire for promotion (and within reason your salary will probably increase over time even if you're mediocre) and social pressure. The main way in my work you are motivated is that if you don't do what you were supposed to do one week, you have to survive a difficult moment on Monday when we have our status meeting.
Anyway, none of this is relevant to me because I am really motivated by how fun it is to be working with other people on something that I am reasonably good at, and also because I never have to read or write academic papers again.
Except, of course that I have my viva later this month, and I will have corrections out of that. But I'm trying not to think about it.
Saturday 04 June 2005 16:12 GMT
I thought I'd write a little bit about FreeGuide, since I'm planning to link this page off the FreeGuide page.
I'm the lead developer on FreeGuide, and having just started a new job (which so far is excellent) my free time is seriously limited, especially as my commute is 1.5 hours at the moment. I expect as I settle in I'll find more time, and I also hope my commute may reduce in a few months. In the meantime I've had to fall back into maintenance mode, but fortunately there are several contributors stepping up to add long-requested features. The most prolific contributor has been Alex, who has been working on a number of excellent new features in our development branch. That branch started when Risto refactored the code into an MVC style and optimised the display to speed up massively operation with more than about 10 channels.
That branch has been around _far_ too long, as it's a maintenance headache to look after 2 quite different branches, and it causes confusion when people send in patches against old code.
I really want to make the development branch reasonably bug-free and then declare it stable and retire the 0.8.x branch once and for all. I hope we can do that pretty soon.
Meanwhile, Alex is adding more features, the most notable of which is the plugins system. That will eventually allow you to download and install new listings grabbers without upgrading FreeGuide itself, and plug in loads of other things like exporters, recording schedulers, and loads more. Also excitingly soon to go in is the recording scheduler system developed by Reuel and integrated by Alex.
I want to make 0.10.x stable next release, then get those features in (probably under a 0.11.x name), stabilise them, go through the TODO and make a _short_ list of things to do before 1.0, do them and NOTHING else, and then finally release 1.0. It would be great if the plugin system turned out to be flexible enough to handle all the new thigns people suggest, but if not I plan to keep the 1.0.x branch stable long term, and allow further changes in a 1.1 branch.
The features I think are needed for 1.0 are:
Hopefully not a lot else...
Friday 03 June 2005 16:07 GMT
Can't decide what to think about Google. I was very suspicious about gmail, and this whole portal business is very boring, and _so_ corporate, but you really can't argue with sponsoring open source contributions, can you? If doing this kind of thing starts being seen as the way to impress the developer community (and therefore recruit the geekiest geeks) then everyone wins, and world domination is one step closer.
Thursday 02 June 2005 08:23 GMT
I've decided to start writing a blog, not least to convince people that although my work on open source projects (especially FreeGuide) has slowed down, it has not slipped out of my mind. Quite the opposite: I'm finding my lack of time very frustrating, especially when the train journey in to work every day gives me time to think of lots of excellent ideas, none of which I am going to be able to put into practice. I'll log them here, and maybe someone else will do them. If not, I'll do them when I retire.
I'm hoping to keep this blog as technical as possible, with only occasional daliances into personal matters, since it does my head in to be reading something like Planet Gnome and have to read about people's cats being sick etc.