I've got a really old/cheap GoPro (not HD). It is rather low res, picture quality is not that great, so I'm thinking about getting a new one.
I'm looking at the GoPro HD Hero, its now about £170. I don't think its worth paying an extra £100 for the HD Hero 2, its not that much better (unless you really need professional quality).
I think a GoPro is best for on a bike, the mounts work well, they have loads of different types of mounts to try.
For software, if using Windows, then Windows Movie Maker is OK for basic stuff, pretty easy to use. I'm also now trying Sony Vegas, which seems rather good, though I've not done much with it yet.
Note you will need a pretty powerful PC with loads of RAM if you want to do much editing of HD video.