I know Mattias Olsson has done some work with this
Maybe something you can build on?
I've used the technique Mattias blogged about but I seem to recall it has a few issues in newer versions of Episerver which will require some resolving (it's all possible though). Scott Reed did similar more recently and so I suspect you wouldn't hit so many issues:
Using EPiServer 7.5, is it somehow possible to restrict which DisplayOptions that should be available for a certain block type or page type? (or say, perhaps for a given ContentArea property).