Looking in to the implementation and docs https://world.episerver.com/csclasslibraries/commerce/Mediachase.Commerce.Orders.Managers.OrderGroupWorkflowManager?version=10 this only seems to be supported on OrderGroup which is the old non serialized cart. If you change it to IOrderGroup which you should always use there's no support for RunWorkflow.
Quan did say here https://world.episerver.com/forum/developer-forum/Episerver-Commerce/Thread-Container/2018/7/how-to-run-workflow-for-icart/ you don't need to run workflows on 11+ so it may be redundant now.
In the link Scott posted https://world.episerver.com/forum/developer-forum/Commerce/Thread-Container/2018/7/how-to-run-workflow-for-icart/#195426
You can also look into OrderValidationService, it validates the carts (and you can customize what validations you want)
I just enabled the serializable carts mode for our recently upgraded commerce 13.30 solution (upgraded from commerce 11). However, I am getting the error "System.InvalidCastException: 'Unable to cast object of type 'EPiServer.Commerce.Order.Internal.SerializableCart' to type 'Mediachase.Commerce.Orders.OrderGroup'.'" right before we attept to run the workflow. This makes sense but I'm am unsure on how to fix this issue. I have not been able to find any documention about this. This worked fine before switching to serializable carts.
Is it possible use RunWorkflow on Serializable Carts? Do workflows work with serializeable carts? Do I need to cast this to some other type or use a different approach?