You can only set access rights at catalog and category (node) level. A product (or an entry in general) will inherit access right settings of its true parent. There was a plan to add per-entry setting if there was enough demand, but that's not been materialized 

I can't seem to get the product to inherit access rights from the parent category once restricting it to certain users. When I view the access right for the product under the restricted category it still says Everyone. The product is also still being fetched by the SearchClient.

You'd need to filter search result yourself, for example

Access rights (Find 11) | Optimizely Developer Community

I'm sorry but I think you miss understood me. I just want the product to inherit the access levels from the parent category. No matter what I do the access level for the product remains as Everyone
when I change the access level of the parent category.

Based on the documentation it says:

You cannot set access rights for individual products. Products inherit permissions from their direct parents (but not "linked" parents).

I'm not sure what could be the issue to be honest.

Do you mean this

Yes, that is what I mean. Changing the access level(Visible to) for the parent category of this product does not change the access level for the product(it should say restricted in the visible to label) .

I even tried to create new categories with new products to see if the issue is from a certain category or product, but with no prevail.

Any ideas?

It seems to be a bug. I will look into it and get back to you

The bug is COM-17635, I have asked the Commerce team to look into it