Thanks, I did have quick look round the South Eastern Finecast set up a few years ago, whilst picking some materials up from him - a really professional operation.
Dodgy fishplates? Try bridging the rail joints with a wire whilst the loco is on the track and see if you get a speed variation..
Are you using the point blades as electrical switches? They can be a bit iffy - try bending up the little bits that wipe on the bottom of the moving blades to make the connection better..
It was mentioned - http://irishrailwaymodeller.com/showthread.php/2592-doing-ground-cover?p=39353&viewfull=1#post39353 - but we were being gentlemen and assuming that you were keeping quiet about it...