I found something. Referencing this comment (https://world.episerver.com/forum/developer-forum/Episerver-Commerce/Thread-Container/2018/4/commerce-catalog-find-fails/#191856) I thought to check if the search location was pointing to the same place in both the CMS and Commerce Manager sites, and using this comment (https://world.episerver.com/forum/developer-forum/Episerver-Commerce/Thread-Container/2018/1/issue-with-catalog-search/#187326) I found that the CMS site had a different index than the Commerce Manager site. (The CMS site had only 2 .cfs files and the CM site had 7).

I repointed the CMS site to the Commerce Manager's index and now I'm seeing all variants.

However, another comment in the first thread said that it's important to point both the Commerce Server and the CMS to the same index location, "Otherwise you would need to run the "Full Search Index" job to index on CMS site." I did run the Full Search Index, and updated it during my debugging session, but it didn't populate the index with variants.

Futhermore, I just confirmed that when I run the "Full Search Index" job on the CMS site, it must be changing the index on the Commerce Manager side because the variants disappear. To get them back I have to Rebuild Index from inside Commerce Manager.

It sounds like your search setting in your cms site is not correct. You should probably look into mediachase.search.config in Configs folder and make sure it is equivalent to your CM site.

also it is true that you can and probably should share the same index between those sites to avoid inconsistency (although lucene is not the best when it comes to handle high level of concurrency)

Hi Quan! Thank you for your answer. 

When I run the Full Search Index most, but not all, of the variants are included. It seems just one type is not getting indexed but the Commerce Manager includes them in the index. So using the same index works as long as no one runs the Full Search Index job to overwrite the index. 

The Mediachase.Search.config files are identical. 

<?xml version="1.0" encoding="utf-8"?>
<Mediachase.Search>
  <SearchProviders defaultProvider="LuceneSearchProvider">
    <providers>
      <add name="LuceneSearchProvider" type="Mediachase.Search.Providers.Lucene.LuceneSearchProvider, Mediachase.Search.LuceneSearchProvider"
        queryBuilderType="Mediachase.Search.Providers.Lucene.LuceneSearchQueryBuilder, Mediachase.Search.LuceneSearchProvider"
        storage="[appDataPath]\Search\ECApplication\" simulateFaceting="true"/>
    </providers>
  </SearchProviders>
  <Indexers basePath="[appDataPath]\Search\ECApplication\">
    <add name="catalog" type="Mediachase.Search.Extensions.Indexers.CatalogIndexBuilder, Mediachase.Search.Extensions"/>
  </Indexers>
</Mediachase.Search>

For now we'll just use the work around, but it would be good to know why some variant types aren't being indexed by the Full Search Index but they are by the Commerce Manager's "Build Index" function.

Hi Nick

I would advice you to use Episerver Find, if in any way possible. It it just so much easier to work with, has more features and it doesn't have concurrency or synchronization issues like you found from local Lucene indices. If your site is in DXC, Find indices are included. If not, it may come at an additional cost.

For a start you can replace the Lucene provider with the Find provider, by changing the configuration and reindexing everything. Unless you are using native Lucene features in your queries, you should be able to keep using your existing SearchProvider queries. Then over time you can migrate all your queries to use the Find .Net SDK, opening the box of very nice features.

It turns out the variants that do not show up in the IndexingService.svc "Full Search Index" job are created programmatically and not through the Commerce Manager. The Commerce Manager sees them but the CMS does not, so they're only included in the Commerce Manager's index and if I run "Full Search Index" it overwrites the Commerce Manager index.

Is there something that maybe was missed when creating the products programmatically that would exclude them from the Full Search Index? Maybe a flag that should have been set? I don't want to rely on "Don't run the Full Search Index" as our solution to this issue.

Here is the code used to create the variant:

var newVariant = CatalogHelper.GetEpiproduct<CustomProduct>(categoryLink, variantContentReference);

newVariant.Code = request.Code;
newVariant.ProductExpirationDateDays = request.ProductExpirationDateDays;
newVariant.Name = request.Name;
newVariant.DisplayName = request.DisplayName;
newVariant.ParentLink = request.ParentProductLink;
newVariant.ContentLink = variantContentReference;
newVariant.DontAllowMultiple = request.DontAllowMultiple;
newVariant.Created = request.DateCreated;
newVariant.Countries = request.Countries;
newVariant.Regions = request.Regions;
newVariant.IsNorthAmerica = request.IsNorthAmerica;
_contentRepository.Service.Save(newVariant, SaveAction.Publish, AccessLevel.Read);

where GetEpiProduct<T> is:

public static T GetEpiproduct<T>(ContentReference categoryLink, ContentReference productLink) where T : IContentData
{
    var epiproduct = productLink.ID == 0 ? _contentRepository.Service.GetDefault<T>(categoryLink) : _contentRepository.Service.Get<T>(productLink);
    return epiproduct;
}

For starter I think you should try to set the StartPublish property. It is automatically set in Commerce Manager but if you are using API you should set it explicitly