Hey!

We have a client that uses alot of legacy find implementations for building and matching indexed fields using query.Filter and the FilterBuilder. This has worked fine but now they want to introduce a new type of filter called "MultiCvlFilter" that we're abit unsure about how to match using .Filter and the FilterBuilder. So I will try to give as much detailed information as possible and show the process steps the filtering takes, maybe somebody has some input that might help us in the right direction.

MultiCvl is basically a field with multiple values that should all individually be matched against a single selected filter.

----------------------------------------------

Frontend

This is an example of what a multicvl filter might look like in the frontend, both ItemMedias and ItemCompatibleWith are multicvl-filters that might appear in one or multiple articles.

----------------------------------------------

Index

Below is what the indexed field looks like in the index, this is a collection of all multicvlfilters that this product or article has, so it's not split up by what the field is named but indexed in a combined field (instead of one field for ItemMedia, one field for ItemCompatibleWith etc.")

"MultiCvlValues$$string": "GII,Classic,Grease,Water hydraulic,Air,Breathing air,Compressed air,Hydraulic oil,Lubrication,Neutral gas,None,Non-explosive water based liquids,Water",

The idea is that if you select a filter that is a multicvlfilter, it should be matched against each individual value in this field. So let's say the selectedFilter ultimately ends up being "Air from ItemMedias" , it should not match the above row, but if I select "Grease" it should.

----------------------------------------------

Step by step

Step 1:
This is the SearchService method that handles the search initially (after the controller passes it here). I've excluded some of the more basic filterings and included what I think is important to see:

        public SearchResult Search<T>(FilterOptionViewModel filterOptions, List<string> availableFilters, List<string> selectedfacets, ContentReference parentLink, bool isProductAreaNode = false) where T : CatalogContentBase
        {
            var language = GetLanguage();
            var query = _client.Search<T>(language);

            if (typeof(T).Equals(typeof(ArticleVariation)))
            {
                query = query.Filter(
                    x => x.SearchModel().AncestorNodes.Match(parentLink.ID.ToString())
                );
            }
            else
            {
                if (parentLink != null)
                {
                    if (isProductAreaNode && typeof(T).Equals(typeof(ProductNode)))
                    {
                        query = query.Filter(
                            new FilterBuilder<T>(
                                _client,
                                new TermFilter(
                                    "ExtendedProductData.ProductAreas$$string",
                                    parentLink.ID.ToString()
                                )
                            )
                        );
                    }
                    else
                    {
                        query = query.Filter(x => x.Ancestors().Match(parentLink.ToString()));
                    }
                }

                query = query.FilterOrphans();
            }

            query = typeof(T).IsSubclassOf(typeof(NodeBase))
                ? query.OrderBy(x => (x as NodeBase).SearchModel().SortPosition)
                : query.CustomOrderBy(filterOptions);

            query = typeof(T).IsSubclassOf(typeof(NodeBase))
                ? query.Filter(x => (x as NodeBase).SearchModel().Searchable.Match(true))
                : query;

            query = query.Skip(filterOptions.Offset).Take(filterOptions.PageSize);

            var facetQuery = query;

            // !!
            // This is where we build the filters based on the selected Filters.
            query = FilterSelected(query, filterOptions.SelectedFilters, availableFilters);

            var result = query.Select(x => x.SearchModel()).GetResult();
            var items = CreateSearchItems(result.Hits).ToList();
            var productCategoryQuery = _client
                .Search<ProductCategory>(language)
                .CustomFilterForVisitor()
                //.Cache()
                .OrderBy(x => x.SearchModel().SortPosition)
                .Select(x => x.SearchModel())
                .Take(100)
                .GetResult();

            var count = items?.Count ?? 0;

            var facetGroups = GetFacetResults(
                filterOptions.SelectedFilters,
                facetQuery,
                availableFilters,
                selectedfacets,
                productCategoryQuery
            );

            return new SearchResult
            {
                Items = items,
                FacetGroups = facetGroups,
                TotalMatching = count > 0 ? count : result.TotalMatching,
            };
        }

----------------------------------------------

Step 2:
This is the FilterSelected method where - in short - the query is extended and funneled down based on what selectedFilters are present. In our case, we have a flag on the filter itself that allows us to determine if the filter is of the new sort (MultiCvl) or not, and pipe it further accordingly.

        private ITypeSearch<T> FilterSelected<T>(ITypeSearch<T> query, List<FilterGroup> options, List<string> availableFilters)
        {
            var facets = GetFacetDefinitions(availableFilters);

            foreach (var facetGroupOption in options)
            {
                IFacet filter = null;

                var isInterval = facetGroupOption.Type.ToString() == "Interval";
                if (isInterval)
                {
                    filter = new NonFacetIntervalDefinition()
                    {
                        FieldName = facetGroupOption.Id,
                        DisplayName = facetGroupOption.Text,
                        SelectedMax = facetGroupOption.SelectedMax,
                        SelectedMin = facetGroupOption.SelectedMin
                    };
                }
                else
                {
                    filter = facets.FirstOrDefault(x => x.FieldName.Equals(facetGroupOption.Id));
                }

                if (filter == null)
                {
                    continue;
                }

                if (facetGroupOption.Items != null && !facetGroupOption.Items.Any(x => x.Selected) && !isInterval)
                {
                    continue;
                }

                if (filter is FacetStringDefinition stringFilter)
                {
                    // !! This is the new type of filter that I explicitly catch to handle it differently
                    // and where we go next.
                    if (filter.IsMultiCvl)
                    {
                        query = stringFilter.MultiCvlFilter(query, facetGroupOption.Items.Where(x => x.Selected).Select(x => x.Text).ToList(), true); 
                    }
                    else
                    {
                        // and these are the normal filters that work since before
                        query = stringFilter.Filter(query, facetGroupOption.Items.Where(x => x.Selected).Select(x => x.Text).ToList(), IsAndFilter(facetGroupOption));
                    }
                }
                else if (filter is FacetStringListDefinition stringListFilter)
                {
                    query = stringListFilter.Filter(
                        query,
                        facetGroupOption.Items.Where(x => x.Selected).Select(x => x.Text).ToList(),
                        IsAndFilter(facetGroupOption)
                    );
                }
                else if (filter is NonFacetIntervalDefinition intervalFilter)
                {
                    var selectedValues = new List<string>();
                    selectedValues.Add(intervalFilter.SelectedMin);
                    selectedValues.Add(intervalFilter.SelectedMax);

                    query = intervalFilter.Filter(query, selectedValues, false);
                }
            }
            return query;
        }

----------------------------------------------

Step 3:
This is the FacetStringDefinition which holds the method MultiCvlFilter<T> from above, and for now all that method does is make it possible to handle this type of filter differently, but right now it just pipes it fordward to the next step just like the normal filters (the Filter<T> method below)

    public class FacetStringDefinition : IFacet
    {
        private readonly Injected<ILocalizationProvider> _localizationProvider;

        private string _displayName;
        public string Name { get; set; }
        public string DisplayName
        {
            get => GetDisplayName();
            set => _displayName = value;
        }

        public string FieldName { get; set; }
        public string RenderType { get; set; }
        public bool HideAsColumn { get; set; }
        public bool IsMultiCvl { get; set; }

        public string GetDisplayName()
        {
            var t = _localizationProvider.Service.GetString("CommerceTranslations.FieldHeaders." + FieldName);
            if (!string.IsNullOrEmpty(t))
            {
                return t;
            }

            return $"[{FieldName}]";
        }

        public ITypeSearch<T> Filter<T>(ITypeSearch<T> query, List<string> selectedValues, bool andFilter)
        {
            if (selectedValues.Any())
            {
                return Services.Search.SearchExtensions.AddStringFilter(query, selectedValues, FieldName, andFilter, false);
            }

            return query;

        }

        public ITypeSearch<T> MultiCvlFilter<T>(ITypeSearch<T> query, List<string> selectedValues, bool andFilter)
        {
            if (selectedValues.Any())
            {
                // !!
                // This is where we end up with multicvl where I've tried to hardcode the fieldname that should be matched against for the next step.
                return Services.Search.SearchExtensions.AddStringFilter(query, selectedValues, "MultiCvlValues", andFilter, false);
            }

            return query;
        }

        public ITypeSearch<T> Facet<T>(ITypeSearch<T> query, EPiServer.Find.Api.Querying.Filter filter) => TermsFacetFor(query, FieldName, typeof(string), filter);
        public ITypeSearch<T> TermsFacetFor<T>(ITypeSearch<T> query, string name, Type type, EPiServer.Find.Api.Querying.Filter filter, int size = 500)
        {
            var fieldName = name;
            if (type != null)
            {
                fieldName = query.Client.GetFullFieldName(name, type);
            }

            return new Search<T, IQuery>(query,
                 context =>
                 {
                     var facetRequest = new TermsFacetFilterRequest(name, filter)
                     {
                         Field = fieldName,
                         Size = size
                     };
                     context.RequestBody.Facets.Add(facetRequest);
                 });
        }
    }

----------------------------------------------

Step 4:
This is the SearchExtensions where we ultimately end up with any type of filter (before this "new" filter, all we had was the regular stringfilter, and the intervalfilter).

    public static class SearchExtensions
    {
        public static string GetFullFieldName(this IClient searchClient, string fieldName) => GetFullFieldName(searchClient, fieldName, typeof(string));
        public static string GetFullFieldName(this IClient searchClient, string fieldName, Type type)
        {
            if (type != null)
            {
                return fieldName + searchClient.Conventions.FieldNameConvention.GetFieldName(Expression.Variable(type, fieldName));
            }

            return fieldName;
        }

        public static ITypeSearch<T> AddIntervalFilter<T>(this ITypeSearch<T> query, string minValue, string maxValue, string fieldName)
        {
            var fullFieldName = $"{fieldName}$$number";
            return query.Filter(GetFilterForInterval<T>(minValue, maxValue, query.Client, fullFieldName));
        }

        public static ITypeSearch<T> AddStringFilter<T>(this ITypeSearch<T> query, List<string> stringFieldValues, string fieldName, bool andFilter = false, bool stringList = true)
        {
            // !!
            // For the MultiCvl-implementation this is where the hardcoded "MultiCvlValues" fieldName will be used.
            var fullFieldName = !stringList ? query.Client.GetFullFieldName(fieldName) : fieldName;

            if (stringFieldValues != null && stringFieldValues.Any())
            {
                return query.Filter(GetFilterForStringList<T>(stringFieldValues, query.Client, fullFieldName, andFilter));
            }

            return query;
        }

        private static FilterBuilder<T> GetFilterForStringList<T>(IEnumerable<string> fieldValues, IClient client, string fieldName, bool andFilter)
        {
            var filters = fieldValues.Select(s => new TermFilter(fieldName, s)).Cast<EPiServer.Find.Api.Querying.Filter>().ToList();
            if (filters.Count == 1)
            {
                return new FilterBuilder<T>(client, filters[0]);
            }

            return andFilter ? new FilterBuilder<T>(client, new AndFilter(filters)) : new FilterBuilder<T>(client, new OrFilter(filters));
        }

        private static FilterBuilder<T> GetFilterForInterval<T>(string minValue, string maxValue, IClient client, string fieldName)
        {
            var decimalMinValue = decimal.Parse(minValue.Replace(".", ","), new System.Globalization.CultureInfo("sv-SE"));
            var decimalMaxValue = decimal.Parse(maxValue.Replace(".", ","), new System.Globalization.CultureInfo("sv-SE"));

            var filter = RangeFilter.Create(fieldName, decimalMinValue, decimalMaxValue);

            filter.IncludeLower = true;
            filter.IncludeUpper = true;

            return new FilterBuilder<T>(client, filter);
        }
    }

So that's basically the entire route the search and filtering takes, I've tried to add // !! comments to the code snippets to guide you. There are other methods as well, but we think that this is where the source of the problem and also the solution for it lies. We are of course open to suggestions and input if you have any, and the ultimate goal is to get the filtering to work in symbiosis with the structure that is already in place, while the long term goal is to rethink the whole approach to filtering.

In the above scenario, let's pretend we've selected "Grease" from the ItemMedias filter, so:

stringFieldValues contains one string value, which is "Grease".

fieldName is MultiCvlValues$$string, we're not storing these as "ItemMedias" or "ItemCompatibleWith" but as one combined field.

So the problem is, I have no idea how I should set up the FilterBuilder to match Grease to this field in the index which contains Grease, as part of the commaseparated string.