Haven't tried myself, but I think what you can do is to implement IAttachementHelper, register it to override the default one. In GetSupportedFiletypes you can just return an empty list, and the content will not be indexed.

In case you need help with registering your implementation https://vimvq1987.com/register-your-custom-implementation-the-sure-way/ 

I usually use this:

ServiceLocator.Current.GetInstance<IClient>()
    .Conventions
    .ForInstancesOf<EPiServer.Core.MediaData>()
    .ExcludeField(x => x.SearchAttachment())
    .ExcludeField(x => x.SearchAttachmentText());