I don't think you should be that worried about performance when using controllers for blocks. But if they're not needed, you should avoid them.
You can also create your own view engine and change how the partial views are discovered:
public class ViewEngine : RazorViewEngine
private static readonly string AdditionalPartialViewFormats =
this.PartialViewLocationFormats = this.PartialViewLocationFormats.Union(AdditionalPartialViewFormats).ToArray();
And in Application_Start in Global you can register your custom view engine:
Someone did actually measure the difference https://hacksbyme.net/2017/09/26/performance-when-using-controllers-for-blocks/
Great, both post are useful.
it's much more fun when you throw feature folders in the game...