Let me see if I got this right. When you browse to the webservice you get a windows login prompt and when you enter your credentials you get redirected to a form login?

Make sure that the user in in the webservices group (defined in web.config under the webserices location) and that the group is added to "Permissions to functions" (admin mode -> config tab).

Also there is an article about configurating and troubleshooting web services:

http://world.episerver.com/Documentation/Items/Tech-Notes/EPiServer-CMS-5/EPiServer-CMS-5-R2/Web-Services/#I Get a Forms Login Prompt, After the Windows Login Prompt 

Yes that's right, I am redirected to the login form.

If I disable form authentication, and set all pages to use basic auth I am able to use the webservice without any problems. However this breaks admin mode, requests for javascript files in /Util/javascript are failing. So the actually logging in and using the webservice works fine. I just can't find a way to disable form authentication for one page, and leave it enabled for the rest. In fact I don't think this is possible.

I was under the impression that logging in with basic auth should create the required cookies and bypass the form login altogether.

I have read the document you posted and tried the following:

If you cannot get past the forms login prompt, verify that forms authentication with Windows accounts actually works. Log on to Edit mode with a local administrator account.

And this does work, I can log in to edit mode with that user.

Thanks

What about this: "Make sure that the user in in the webservices group (defined in web.config under the webserices location) and that the group is added to "Permissions to functions" (admin mode -> config tab)."

Also, basic authentication should be disabled in IIS. Instead you use the EPiServer basic authentication module (so make sure that's added to the http modules). 

Like the document says: "You must use Integrated Windows authentication or follow the steps in Configure EPiServer CMS 5 to Enable Basic Authentication to emulate Basic authentication if you want to use both forms authentication and Web services on the same site." 

Sorry for my late response on this. I managed to get it working in the end, I needed to allow anonymous auth for the web services folder. Thanks for your help.