Don't think the OrFilter will take care of that entire block below right? It will just work on that specific expression so you will basically have in your case above:

result = SearchClient.Instance.Search()
                                                        .For(query)
                                                        .Filter(x => x.Ancestors().Match(ContentReference.StartPage.ID.ToString())) //AND
                                                        .Filter(x => x.CategoryTag.Match(new ContentReference(category)))              //AND                                         
                                                       ( .Filter(filterBuilder)                                                                                          //OR
                                                        .OrFilter(HashfilterBuilder))                                                                               //AND
                                                        .Filter(x => x.Ancestors().Match(ContentReference.StartPage.ID.ToString())) //AND
                                                        .Filter(x => x.CategoryTag.Match(new ContentReference(category)))              //AND
                                                        .OrderByDescending(x => x.StartPublish)
                                                        .Skip(offSet * pageSize)
                                                        .Take(pageSize)
                                                        .StaticallyCacheFor(TimeSpan.FromMinutes(0))
                                                        .GetPagesResult(LanguageSelector.AutoDetect());     

You can instead use bolean expressions like:

var searchQuery = client.Search<BlogPost>()
    .Filter(
        x => (x.Title.Match("Search") & x.Tags.Match("Find")) 
            | x.Title.Match("Find"));

or similar for you and use a big phat paranthesis around your two groups of filters

Since the only difference is the two filter builders, you could simply change your query to:

                                                  .For(query)
                                                        .Filter(x => x.Ancestors().Match(ContentReference.StartPage.ID.ToString()))
                                                        .Filter(x => x.CategoryTag.Match(new ContentReference(category)))                                                       
                                                        .Filter(filterBuilder | HashfilterBuilder)

If needed, cast the filterBuilders to the "Filter" type.