Originally we went down the controller route and although we didn't experience the same issue that you are having, have you considered switching to using View Components instead, then you can do away with the controller entirely. This is what we decided to do and now we have migrated all of our blocks to be View Components instead. The following article provides lots of information on what View Components are and how you create them.

https://learn.microsoft.com/en-us/aspnet/core/mvc/views/view-components?view=aspnetcore-8.0 

As an example though, take the concept of a simple block as follows:

The structure in the project looks as follows:

with the View Component class itself looking like this:

public sealed class CtaRteBlockViewComponent : BlockComponent<CtaRteBlock>
{
    protected override IViewComponentResult InvokeComponent(CtaRteBlock currentBlock)
    {
        var model = new CtaRteBlockViewModel()
        {
            CtaStyle = currentBlock.CtaStyle,
            PrimaryLink = new CtaViewModel()
            {
                Link = currentBlock.PrimaryLink.GetSingleLink(),
                CtaStyle = currentBlock.CtaStyle,
                CtaButtonStyle = CtaButtonStyle.Primary,
            },
            SecondaryLink = new CtaViewModel()
            {
                Link = currentBlock.SecondaryLink.GetSingleLink(),
                CtaStyle = currentBlock.CtaStyle,
                CtaButtonStyle = CtaButtonStyle.Secondary,
            },
            ContainerClass = currentBlock.CtaStyle.Equals(CtaStyle.Button) ? "rich-text__buttons" : "rich-text__arrows",
            DataComponent = currentBlock.CtaStyle.Equals(CtaStyle.Button) ? "data-component=\"EqualChildren\"" : "",
            AnchorTagTitle = currentBlock.AnchorTagTitle
        };

        return View(model);
    }
}

which has the following View Model:

public sealed class CtaRteBlockViewModel
{
    public CtaStyle CtaStyle { get; set; }

    public CtaViewModel? PrimaryLink { get; set; }

    public CtaViewModel? SecondaryLink { get; set; }

    public string? ContainerClass { get; set; }

    public string? DataComponent { get; set; }

    public string? AnchorTagTitle { get; set; }
}

and the View itself looks like this:

@model CtaRteBlockViewModel

<div class="@Model.ContainerClass" @Model.DataComponent>
    @if (!string.IsNullOrWhiteSpace(Model.AnchorTagTitle))
    {
        <a id="@Model.AnchorTagTitle"></a>
    }
    @if (Model.CtaStyle.Equals(CtaStyle.Button))
    {
        <partial name="_Cta" for="@Model.PrimaryLink" />

        <partial name="_Cta" for="@Model.SecondaryLink" />
    }
    else
    {
        <partial name="_Cta" for="@Model.PrimaryLink" />
    }

</div>

For global components such as header, footer, etc. you create the View Component in the same way but can then invoke the View Component inside your _layout.cshtml file in the following manner:

@await Component.InvokeAsync(typeof(HeaderViewComponent), new { sitePage = Model.CurrentPage })

Hope this helps you.

Bundle of thanks for detailed answer, that will be our last choice due to the level of refactoring requires.

BTW
Wondering are you using Feature Conventions and where cshtml file resides for components?

Longshot, but similar issue came up in this post: https://world.optimizely.com/forum/developer-forum/Developer-to-developer/Thread-Container/2023/5/unable-to-invokecomponent-for-blocktype-in-conversion-project----cms-12/

"I believe the main reason for my problem was that the application output type was wrong. Setting this to "console application" makes the block components load as expected."