I have done this so it's possible. ;-)
This was before EPiServer supported ADFS so the solution may be easier now.
The down fall is that there is no centralized handling of roles and it's harder to keep track when a role has been assigned to a user and by whom.
Another issue is that roles cannot be assigned to a user before they have logged on for the first time.
We have been requested to setup a POC for a customer in which they would like to see if it is possible to use ADFS to do the authorization to the site but allow episerver to manage the roles / permissions to the site through the site. I think this seems to be a bit of a hassle in that you have 2 seperate parties handling they authentication / authorization but they would like to know if it is possible. Have you or anyone know if this is possible and what the down falls might be to this. Thanks in advance.