I think you can achive this if you use Filter for Color and FilterHit for Size. Ref documentation:

Use the Filter method in most cases, as it filters both the search result and the facets. Use FilterHits when you want to apply a filter to search results (the hits), but not to the facets.

Did you ever find a good solution for this? I have the same problem and I feel like this should be easier than it seems to be.

I'm wondering the same, seems like the above is the best way to build the facets in a search and it's more complicated than it needs to be - the best implementation I've seen used multisearch with three separate queries (one with main filters, one with no filters [for all facets] and another for "future facets"). I'm hoping there's a simpler solution and my googling led me to this forum thread. Anyone here know?

After a lot of experimentation and making mistakes along the way, I have managed to build out what I need to achieve what the OP was asking for above. It does involve multisearch and future facets. Basically the future facets search is just utilising an elastic search feature "facet_filter" on each facet that has a selected value so that you filter only on each other filter group. It's hard to explain but here's an example of how the facets part of the JSON going to elastic search like for a search like above:
  "facets": {
    "Make": {
      "terms": {
        "field": "Color$$string",
        "size": 1000
      },
      "facet_filter": {
        "and": [
          {
            "terms": {
              "Size$$string": [
                "Small",
                "Medium"
              ]
            }
          }
        ]
      }
    },
    "BodyType": {
      "terms": {
        "field": "Size$$string",
        "size": 1000
      },
      "facet_filter": {
        "and": [
          {
            "terms": {
              "Color$$string": [
                "Red"
              ]
            }
          }
        ]
      }
    }
  }

The problem is, the Find library doesn't support the "facet_filter" feature (and Find runs on an old v1.4 elastic search from what I can gather) so you need to decompile the built in TermsFacetRequestConverter and make your own extended version of the TermsFacetRequest with filters on it and your own converter to JSON serialize it exactly like TermsFacetRequestConverter did but with your own custom "facet_filter" added on. Here's an example of how my own converter class looks like:

public class TermsFacetFilterRequestConverter : CustomWriteConverterBase<TermsFacetFilterRequest>
{
    private const int MinSize = 0;
    private const int MaxSize = 1000;

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var facetRequest = (TermsFacetFilterRequest)value;
        writer.WriteStartObject();
        writer.WritePropertyName("terms");
        writer.WriteStartObject();
        WriteNonIgnoredProperties(writer, value, serializer);
        if (facetRequest.Fields.IsNotNull() && facetRequest.Fields.Any())
        {
            writer.WritePropertyName("fields");
            WriteArrayValues(writer, facetRequest.Fields, serializer);
        }

        if (facetRequest.AllTerms)
        {
            writer.WritePropertyName("all_terms");
            writer.WriteValue(true);
        }

        if (facetRequest.Size.HasValue)
        {
            if (facetRequest.Size.Value < MinSize)
            {
                throw new InvalidSearchRequestException(string.Format(CultureInfo.InvariantCulture,
                    "Terms facet size can not be set to a lower value than 0. Current value: '{0}'",
                    facetRequest.Size.Value));
            }

            if (facetRequest.Size.Value > MaxSize)
            {
                throw new InvalidSearchRequestException(string.Format(CultureInfo.InvariantCulture,
                    "Terms facet size can not be set to a higher value than 1000. Current value: '{0}'",
                    facetRequest.Size.Value));
            }

            writer.WritePropertyName("size");
            writer.WriteValue(facetRequest.Size.Value);
        }

        writer.WriteEndObject();
        if (facetRequest.FacetFilter.IsNotNull())
        {
            var contract = (JsonObjectContract)serializer.ContractResolver.ResolveContract(value.GetType());
            var property = contract.Properties.FirstOrDefault(x => x.PropertyName.Equals("facet_filter"));
            if (property != null)
            {
                WriteNonIgnoredProperty(serializer, property, facetRequest.FacetFilter, writer);
            }
        }

        writer.WriteEndObject();

        
    }
}

If you decompile the EPiServer.Find.Api.Facets.TermsFacetRequestConverter class - you'll see the only difference above is adding the "facet_filter"

I hope this helps someone - I searched the web and really couldn't find this answer anywhere - and I really hope this is a feature that Optimizely will add to the Find library - it should be a simple Filter  type parameter you pass to the TermsFacet extension method out of the box.