I've been informed that a bug has been raised internally to look at this issue. When further info in given I will update this post.
This bug is being tracked here http://world.episerver.com/documentation/Release-Notes/ReleaseNote/?releaseNoteId=CMS-4253
Fixed in Update 131 - EPiServer.CMS.Core 9.12.3
Can anyone comment on this or state whether they have seen or resolved a similar issue. Testing was performed against an EPi commerce site
(http://world.episerver.com/releases/episerver---update-114/) and the mainline of Alloy downloaded from the public Git from 9th August 2016.
During performance testing we noticed a very large increase in response times for page requests to an EPiServer commerce site when under load. Given the scenario of a single user accessing a CMS page as a benchmark (the site has been restarted and the page accessed once to initialise and load the page prior to each test) we saw an approximate increase of the response time upwards of 40x when the same request was made by 100 users concurrently. (100 threads running concurrently each requesting the same page twice sequentially and the average response in milliseconds recorded)
This same effect can be seen to slightly lesser extent against the Alloy site where they home page was used, in this case it was an approximate 30x increase. Alloy was used rather than Quicksilver as it should be a less complex site.
When attaching the profiler (in our case RedGate) the major issues seems to be with Structuremap and I believe potentially resource contention. A single request to the homepage for Alloy generates tens of thousands of calls to the Structuremap Container.GetInstance method. Internally this results in calls to SessionCache.GetInstance and RootInstanceGraph.GetDefault which in turn result in calls to Cache.Fill and LifecycleObjectCache.Get. Both these methods rely on locking. With a single user this doesn't present as an issue. As I increase the number of concurrent requests the time waiting within these locking methods rapidly increases.
It appears in newer versions of Structuremap 4+ the Cache class was first used less and then removed to improve resource contention issues. This may indicate there is/was a general issue in Structuremap that is highlighted with EpiServer because of itsusage patterns.
Has anyone else seen this or similar behaviour, or am I missing something obvious that needs to be configured with Epi or IIS to maximise concurrent performance?. (FYI - The server resources are not being tasked during these tests, CPU generally sits around 10%)
I would assume this can be mitigated by implementing output caching or using many micro instances in a webfarm when hosting.