By "part of the site" I assume you mean a section of EPiServer pages, or is this part some kind of standalone app?
If it's the later I think it might be possible...
Otherwise you can't mix Windows and Forms Authentication. You need to set up the entire EPi-site on two different URLs with different Web.configs and have some code that directs back and forth.
We have a site that is running as Forms Authentication, using SQL membership providers. We have a new requirement where we need part of the site to be accessed by Windows Authentication and auto login a user.
As a test, I've tried creating a sub folder within IIS 7 with a simple HelloWorld.aspx file and converted this sub folder into an application by using the IIS "Convert to Application). I've then set the authentication of this application to be windows only and disabled anonymous access to send a 401 request to the browser. I've also updated the root site web.config to use the MultiplexingRoleProvider with both SqlServerRoleProvider and WindowsRoleProvider.
When I try and browse to my HelloWorld page, I keep getting the error "CS1519: Invalid token ',' in class, struct, or interface member declaration" so something is obviously not configured correctly.
Is this way the correct way to acheive this requirement, is this even possible with EPiServer?