So far in my investigation is seems that I would have to override the PromotionEngine and override the Run method to change the logic around which promotions are applied for the OrderGroup. Because of the amount of logic in this method, I'm concerned about the risk of my own introduced issues and breaking changes with Commerce package updates.
This looks quite involved, hoping there is an easier path.
Although, the problem is that the PromotionEngine's Run method implementation using many c# internal things so it's not possible to reproduce as it is. Seems I'm stuck at a dead end with that approach.
You can create an in-memory cart and apply discounts for A and then B and compare the two? Make sure you don't accidentally save the cart in the database.
With that said, I don't believe multi-discount is allowed in B2C commerce.
After much experimentation with the PromotionEngine, I ended up finding it too difficult, if not impossible, to override behaviour to how I needed it. Ultimately it was much simpler, even if less elegant, to just run my own discount calculation logic for discount B after the promotion engine had done its thing - if mine worked out cheaper I just override the discount directly on the cart and update the cart's promotion entries myself. The code overriding the discount on the line looks like this:
var lineItemDiscountAmount = lineItem as ILineItemDiscountAmount;
lineItemDiscountAmount.EntryAmount = myOverrideTotalDiscountAmount;
I'm wondering if there is a way to have two separate entry discounts with their own independent conditions and rewards, but when evaluating the discounts, we pick the cheapest one once calculated for an entry.
Meaning, if we have two entry discounts "A" and "B" with their own independent conditions - if there are two items in a basket that both discounts are eligible for, we might find the first item is cheaper for discount "A" whereas for the second line item the discount "B" might be cheaper - in this case the first item should have discount A applied and the second line item should have discount "B" applied.
Key point is that both discounts cannot be applied together - it's only going to be one or the other for any given entry/lineItem.
I don't think priorities and exclusions help with this as that usually means the first applicable discount is applied and the other would be skipped entirely.
I can give better examples if the above doesn't make sense but hopefully this simplified explanation saves getting confused in the weeds - let me know if I need to elaborate more.