Try implementing IMigrationStep instead of an initialization module. It will run only once (unless your migration code fails and you click Retry) and it won't let the site start up until successful.
Even IMigrationStep is not guaranteed to run in only one instance - although it is guaranteed to run successfully once. THis is something we are looking into.
Also it's an internal API, meaning it can be changed without notice. One can argue such useful API should be public, but it is what it is right now (which is why it's not officially documented)
Without looking into the "crash", I can't say anything for sure. But it's possible that not the cache invalidation, simply that production does not have a new code to set the value properly. for example if you add a required field (and update your code to manage that), production does not set that value which blows up when it reaches database. Just a guess by a long shot thought.
We're experiencing a very specific issue when it comes to programmatically adding meta fields (to purchase orders, in this case), specifically in a deployment phase. Our process is:
When we add a new meta field, that will be added in a initialization module and it will be executed after step 1 in the process above (as soon as the editorial server is hit by a request I guess). The time between step 1 and step 4, where the production slot is replaced, is like 10 minutes.
During this period of time, there's a mismatch when it comes to meta fields in the purchase order meta class. The new meta field is created, and thus the stored procedure mdpsp_avto_OrderGroup_PurchaseOrder_Update is changed to accept the new meta field. However, the production slot is not aware of the new meta field yet so if a purchase order is created during this time - the site will crash.
My theory is that meta class changes should trigger cache to be invalidated in a server farm. Our event replication in the server farm seems to work though. Currently, I believe that the event broker isn't enabled until after all init modules has been executed (see EventsInitialization...InitComplete). That would explain why the meta class cache isn't invalidated on other servers (since probably no events are sent).
We're adding meta fields to the purchase order meta class like this: https://github.com/episerver/Foundation/blob/c48e99897bf82c117725d2712719ffd9c2a0addb/src/Foundation.Commerce/Extensions/InitializationEngineExtensions.cs#L64.
Edit - I don't exactly recall the error message, other than the meaning being something like "wrong parameters sent to mdpsp_avto_OrderGroup_PurchaseOrder_Update". This is the stack trace: