I'm guessing you've already enabled Integrated Windows Authentication in the IIS. Have you tried to just use the Windows membership and role provider (instead of multiplexing)?
The problem is that you cannot mix FormsAuthentication and WindowsAuthentication in the same site, you would have to select either.
One could possibly write a custom authentication handler that performs the NT Challenge/Response on its own before the FormsAuthentication kicks in, but that
would require quite some coding and is not so easily done.