So far in my investigation is seems that I would have to override the PromotionEngine and override the Run method to change the logic around which promotions are applied for the OrderGroup. Because of the amount of logic in this method, I'm concerned about the risk of my own introduced issues and breaking changes with Commerce package updates.

This looks quite involved, hoping there is an easier path.

Although, the problem is that the PromotionEngine's Run method implementation using many c# internal things so it's not possible to reproduce as it is. Seems I'm stuck at a dead end with that approach.

You can create an in-memory cart and apply discounts for A and then B and compare the two? Make sure you don't accidentally save the cart in the database.

With that said, I don't believe multi-discount is allowed in B2C commerce.

After much experimentation with the PromotionEngine, I ended up finding it too difficult, if not impossible, to override behaviour to how I needed it. Ultimately it was much simpler, even if less elegant, to just run my own discount calculation logic for discount B after the promotion engine had done its thing - if mine worked out cheaper I just override the discount directly on the cart and update the cart's promotion entries myself. The code overriding the discount on the line looks like this:

var lineItemDiscountAmount = lineItem as ILineItemDiscountAmount;
lineItemDiscountAmount.EntryAmount = myOverrideTotalDiscountAmount;