"Custom" sounds like something from the installer... It basically just puts files in Program Files and installs EPiServer Deployment Center + a few services.

Have you run EPiServer Deployment Center?

No, I haven't. The process that runs after I've clicked "Custom" doesn't finish because of the errors i mentioned above. You can take a look at the log file. I uploaded it to http://www.viltersten.com/files/text_01.txt and hopefully you'll spot something. I've noticed that the Temp directory I'm executing setup.exe in gets populated by a bunch of files but I understand that they are supposed to be accessed by the installer and not run by me manually.

Also, at the end, after the error's been detected, a rollback is performed, so I doubt anything gets installed at all.

How do I (trouble)shoot such a beast?!

Are the credentials you provided to connect to your database correct ? Step 2 of 6

They're most likely not, since I haven't provided any. There hsan't been any requests from the installer to do that, basically. I just followed the steps described in installation instructions. There are no "step", at least not numbered ones, as far I've seen.'

I understand you're not referring to the steps in the link below. Of those I get as far as to step 6. While performing it, I get the said error.

http://world.episerver.com/Documentation/Items/Installation-Instructions/EPiServer-CMS/Version-6/EPiServer-CMS-60/Installation-Instructions---EPiServer-CMS-6/

Are you definitely running SQL 2008? I had the same issue when I tried to install EPiServer on SQL 2012 (which no longer contains sp_dboption).

Just found this blog post from a google search - http://dotnetcake.blogspot.co.uk/2012/06/installing-episerver-6-r2-on-windows-8.html. Includes a workaround if you are using 2012.

Try to only install "application files" and then run Deployment Center to create a new site.

Yes, it's the DB that's spooking around. The guys at StackOverflow suggested the same thing. There's been a SP in Server 08 which was removed in 12. Sort of a miss not to include a protection in the installer for such a thing. Or maybe I'm too ignorant of Epi to judge, haha.

I'll try the workaround suggested in the link and hopefully, this matter will be resolved. I'm really curious of this whole Epi-thingy so I can't wait to start coding. I've grown a bit tired of building web shops in ASP.NET...

The solution presented by Adam Jenkin word perfectly. I have to admit that the workaround suggested in the link feels very low-tech. Never the less, it resolves the issue, which I'm grateful for. Now, I'll get me some fun with them options in the deployemnt center. I've never done any Epicoding so I'll probably get veeery confused really soon. Oh, joy! :D

If you check system requirements you'll note that SQL 2012 is not supported (yet) with EPi 6 if that is what you are using. I would recommend sticking to the OS, db etc that IS recommended or you will risk running into a bunch of these problems.
Happy coding