Mostly interesting in second question, I want to be able to notify admin user that manual action (recover) is required.

At other environment changing any entry in Commerce manager turns processor in Invalid state. Is it configuration issue?

For my second question:

Looks like there is no any API "events" I can use to track processor state, so I thinking about following solution (not sure that it's the best one):

create a scheduled job that reads state from tblChangeNotificationProcessor table and analyzes ProcessorStatus field (not sure is it any suitable public CRUD API for this)

For first question still can't find the problem, for me Invalid status is something equal to "error".

I was digging a bit in code related to CatalogIndexingChangeNotificationProcessor but still can't understand what is going on there.

Besides this, "background" change queue processing/indexing activity absolutely out of logging, so I can't catch the problem.

Any help?

The invalid state said that the application can’t know if the index is up to date with the current database. So if something happens so the index and the database get out of sync, like when indexing of an items fails this status will be set to invalid.

The public API for the change notification functionality is found by the service locator

var changeNotificationManager =

   ServiceLocator.Current.GetInstance<IChangeNotificationManager>();

Finally, I think I've find out what was the reason of "Invalid" state in case on product change.

Everything works fine if I have only CommerceManager site running, but if I start front site (CMS) catalog indexer gets into "Invalid" state.

The reason was wrong configuation in CMS/Mediachase.Search.config. 

The problem was solved by synchronizing configuration in both Mediachase.Search.config (CommeraceManager/CMS) files.

Background:

- Each site (CMS/Commerce) starts up CatalogIndexingChangeNotificationProcessor

- Processors share state(Invalid/Valid/...) in database table, but each has its' own queue (identified by connectionId)

- If any proccessor fails to do indexing (error you wrote about) shared state becomes "invalid" in database.

Question:

In the way how indexing works now the process itself is being spread out between sites(web boxes). Looks like not a good solution for me.

Is it possible to do indexing only at one specific box (that should accumulate changes from other boxes via Events Replication)?

I don’t know your actual reason why this should be a bad solution for you (maybe if you doing mass updates on several servers at the same time this is probably not the best solution).

To replace the Change Notification functionality you have to reconfigure the IOC container for ICahngeNotificationManager.