Is the IIS6 site your on your development box. I would imagine it might be the mode in which your application pool is running.
Found the solution, it wasn't caching or something related to that. It seems that the web server was hosted in another segment (with limited bandwidth) then our database server. That in combination that we had some huge images in the database was de reason for the slow performance.
It seems that caching isn't working on my EpiServer Commerce website. When I'm trying to view a product list page, it takes a long time (sometimes up to a minute) to show the page. I think it's because the images aren't cached. I've pretty large images ( 1 - 2 MB), but the images are resized with the MetaImage control :
<cms:MetaImage ID="CatalogImage" ShowThumbImage="true" OpenFullImage="false" PropertyName="PrimaryImage" DataSource='<%#Eval("ItemAttributes.Images")%>' runat="server" />
The website is running on IIS 6.0, I think there is a problem with a configuration setting, because the test website was running on IIS 7 and don't has a problem with loading the product list pages.
Hope that somebody knows the answer.