Blog posts by Svante Seleborg

FriendlyUrls and when to call ConvertToExternal and when not to

Fri, 20 Sep 2013 15:32:44 GMT

It’s not always obvious, but most explicit calls by EPiServer applications made to Global.UrlRewriterProvider.ConvertToExternal() are at best superfluous, at worst directly detrimental to both functionality and performance.

First of all the extra code adds complexity that is not required, causing increased thresholds for maintenance and risk of bugs. Secondly, calls to ConvertToExternal() are rather expensive, so they should not be done when they are not needed. Simply more and unnecessary work for both the developer and the server. That can’t be good!

Background

Internally EPiServer uses URLs looking like this: http://www.example.local/Templates/ArticlePage.aspx?id=1234&epslanguage=sv .

A friendly URL for the same page might look like this: http://www.example.local/sv/blogs/friendlyurls-and-converttoexternal/ .

EPiServer works internally with the ‘ugly’ URLs, only at a late stage are links to pages translated to the external friendly form. This is done by something called the HTML rewriter. EPiServer injects a filter to the output stream, intercepting all HTML output, parsing it and finding all URLs with the ‘ugly’ form, and translating them to the friendly form, with the help of the UrlRewriteProvider.

The UrlRewriteProvider thus knows how to convert from the internal (‘ugly’) format to the external (‘friendly’), and back again. Both the rewriter and the UrlRewriteProvider can be extended, modified and replaced by developers, but this is not the topic of this post.

This post is about what EPiServer with the help of the HTML rewriter and the UrlRewriteProvider already does for you, so you do not have to.

Browsers and URLs

When you browse to http://www.example.local/sv/blogs/friendlyurls-and-converttoexternal/, the browser keeps the address around to use as a base for relative and root-relative references. Thus, a link to a resource in this context might be “document.docx” or “/images/picture.png”. The first link will be interpreted by the browser as http://www.example.local/sv/blogs/friendlyurls-and-converttoexternal/document.docx, and the second as http://www.example.local/images/picture.png .

The important part here is that the scheme (HTTP), and the host (www.example.local) are implied by the context the relative URL is used in. This means informally that as long as we refer to resources in the same site, there is no need to specify either the host or the scheme.

This property is often crucial in order to handle situations where proxy servers expose one host-name to the Internet, while the actual EPiServer site uses a different site internally in it’s IIS host. Many sites will unnecessarily have code where the external host name is provided via settings or similar, and all URLs are decorated with this in order to work externally. A better solution is to simply only use root-relative URLs for site-global resources and other pages, and relative URLs for any page-specific resources (not so frequent).

The one common case where you need to specify the external host, is when you need to provide a redirection to a different scheme, i.e. from HTTP to HTTPS or vice-versa. The best way to handle this, is by at an early stage ensure that the outer elements such as the proxy, terminates the SSL connection and ensures via local redirection that the client always uses SSL (HTTPS). This is in cases where HTTPS support is required at all of course. If you really must provide redirection behind a reverse-proxy, try to do this at a single point in the code, and examine if the proxy can be configured to provide the original client-URL via HTTP headers. Many proxies will provide standard headers “X-Forwarded-For” providing the original IP, “X-Forwarded-Host” for the original hostname and “X-Forwarded-Proto” for the original scheme etc. Prefer these over application settings.

Situations where ConvertToExternal is needed

There are essentially two situations where an application may need to call ConvertToExternal explictly.

A URL with a differing scheme than the page being rendered is required. This should be avoided if at all possible, it’s also confusing and annoying for the user.
A URL to a page on the site needs to be made available to JavaScript. This can often also be avoided, and that should be the first option to pursue.

Since EPiServer will parse all outgoing HTML, and also inspect the value of a redirection via Response.Redirect(), all references to images, style sheets, script and same-scheme redirection will be handled automatically.

However, EPiServer cannot parse arbitrary JavaScript and determine what is a URL and what is not. Therefore it does not try.

So, unless your cases fits within one of the two cases above, there is probably no need to call ConvertToExternal. Let EPiServer and it’s HTML Rewriter to the heavy lifting for you. That’s what it’s for.

The type initializer for '' threw an exception

Thu, 27 Oct 2011 10:10:36 GMT

This post describes a rather obscure problem, and its solution, that occurred when we upgraded to ImageVault 3.5.4 with .NET 4.0 and EPiServer CMS 6 R2.

For once, I don’t quite understand all that happens, or rather why, but I’m hoping someone better versed than me can comment on it. Also since we did fix the issue, it may help others with similar problems and not necessarily tied to EPiServer or ImageVault. LeadTools is likely the key component here, which is used by ImageVault, but the situation as such is pretty generic and may pop up in any number of situations where cross AppDomain calls occur.

For readability, I’m showing the actual stack trace at the bottom of this post.

In brief, what seems to happen is that an AppDomain transition is indirectly caused by the call to ImageVault.ImageConverter(). I have not had the time to figure just what code is running in the other AppDomain, or why it needs to. But, the problem is, that when a cross AppDomain call is made, objects attached to the thread needs to be marshaled or serialized across. This is typically done by serialization. The fault we’re seeing is caused by the receiving AppDomain needing a reference to all types needed for deserialization, and it might not have access to the ASP.NET bin folder. This is the case here.

We’re using a custom IPrincipal object, here called Axantum.Identity, which obviously is attached to the thread, and thus gets serialized. But it can’t be deserialized in the other AppDomain, because there it can’t get at our custom type implementing IPrincipal there.

The solution is either to place the assemblies containing the types that needs to get serialized into the GAC, or to use marshaling by having the type in question derive from MarshalByRefObject . We wrapped our IPrincipal implementation inside another decorator that did derive from MarshalByRefObject as well as changed some component types to derive from MarshalByRefObject and problem was solved. We choose not to install into the GAC although that should work as well. I’m not entirely sure about any caveats or performance issues caused by using marshaling by reference instead of serialization – comments appreciated.

Here’s the stack trace, for reference:

ImageStoreNET.WS.Ajax.AjaxEditImageService.GetImageByName - Error converting image (ImageStoreNET.WS.Ajax.AjaxEditImageService) System.TypeInitializationException: The type initializer for '<Module>' threw an exception. ---> System.TypeInitializationException: The type initializer for '<Module>' threw an exception. ---> <CrtImplementationDetails>.ModuleLoadException: The C++ module failed to load while attempting to initialize the default appdomain.
---> System.Runtime.Serialization.SerializationException: Unable to find assembly 'Axantum.Identity, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.

Server stack trace:
   at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly()
   at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name)
   at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable)
   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record)
   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryHeaderEnum binaryHeaderEnum)
   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
   at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(MemoryStream stm)
   at System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage.FixupForNewAppDomain()
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(Byte[] reqStmBuff, SmuggledMethodCallMessage smuggledMcm, SmuggledMethodReturnMessage& smuggledMrm)
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCallback(Object[] args)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.AppDomain.get_Id()
   at <CrtImplementationDetails>.DoCallBackInDefaultDomain(IntPtr function, Void* cookie)
   at <CrtImplementationDetails>.LanguageSupport.InitializeDefaultAppDomain(LanguageSupport* )
   at <CrtImplementationDetails>.LanguageSupport._Initialize(LanguageSupport* )
   at <CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
   --- End of inner exception stack trace ---
   at <CrtImplementationDetails>.ThrowModuleLoadException(String errorMessage, Exception innerException)
   at <CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
   at .cctor()
   --- End of inner exception stack trace ---
   at <CrtImplementationDetails>.ThrowModuleLoadException(String , Exception )
   at <CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
   at .cctor()
   --- End of inner exception stack trace ---
   at ImageConverterService.ImageConverter..ctor()
   at ImageConverterService.ImageConverterService.LoadDataObjectFormatInformation(String fullFilename)
   at ImageStoreNET.Classes.Data.DataFactory.LoadDataObjectFormatInformation(FileInfo file)
   at ImageStoreNET.Classes.Data.DataObjectFormatInformation.Load(FileInfo file)
   at ImageStoreNET.Classes.Data.DataFactory.GetImageInformation(FileInfo file)
   at ImageStoreNET.Classes.Util.ImageUtil.GetImageInformation(String filename, Boolean temp)
   at ImageStoreNET.WS.Ajax.AjaxEditImageService.GetImageInfo(String filename)
   at ImageStoreNET.WS.Ajax.AjaxEditImageService.GetImageByName(String filename, Boolean isUndo, String clientSideUndoScript, Int32 applicationMode2, Int32 dataObjectID)

1Gb ought to be enough for anybody!

Thu, 14 Apr 2011 22:20:41 GMT

If your EPiServer site consumes more than 1Gb, and you can’t explain in measured detail why, you probably have a problem.

I’m often called in when EPiServer sites have problems under load in live production settings. Problems that can’t be reproduced in the development or staging environments. Problems that are not resolved despite upgrading the server memory, running web farms, installing more CPUs or implementing more caching.

Some of the problems concern unchecked memory consumption with forced recycles as a result. Others concerns hangs and crashes, but this post is about high memory situations.

A small-to-medium site should run perfectly fine in 300 – 500Mb, a very large one should do well in 1Gb. I’ve yet to see a site that really requires significantly more than 1Gb.

This has become even a larger issue with the advent of 64-bit Windows and .NET – now servers often have 16Gb or more memory, and few consider why a single EPiServer site with perhaps 100,000 pages would need that much memory.

Do the math! Let’s say the site uses 10Gb of memory, and let’s say your site has 100,000 pages whereof 10,000 is the ‘working set’, i.e. frequently requested directly or indirectly. That means the site is using 1Mb / page! Even with both internal and output caching that’s simply not reasonable except in a few very special cases.

There are many possible reasons for excessive memory usage, but they are not true memory leaks. It’s just a question of your code holding references to more instances of objects than you may be aware of. Some reasons include attaching instance event handlers to static events, excessive string duplication (do consider using String.Intern() for commonly used strings such as URLs and parts thereof), page instance methods adding data to static collections and unawareness of implications of keeping references around to system objects (they are larger than you may realize). Generally speaking, almost all memory problems come from adding object references to static collections either in per URL code or even worse, per request code.

If you do have a site with more than 500 – 700Mb of memory usage, and you don’t really know why, check it! When I say that you should know why, I mean that you should have measured the memory and know what it’s used for – not just assume for example that a caching scheme is responsible.

Measure it! But how? Take a memory dump and use WinDbg from Debugging Tools for Windows. The standard place to start is http://blogs.msdn.com/b/tess/ . Load the dump and SOS.dll and then do:

!dumpheap –stat

!dumpheap –strings

You’re sure to find food for thought…

Some high-level musings on performance in a CMS

Thu, 21 Oct 2010 18:57:36 GMT

I’ve been working for some time with performance optimization of a reasonably large EPiServer site, and it’s been a very learning experience. This is not a post about specific fixes to specific issues, it’s about high-level design principles.

The lessons learned are applicable in many scenarios, but I’d like to share some of my high-level insights in the context of a CMS in general, and EPiServer in particular. While it’s not reasonable to expect any adoption of these principles in EPiServer in any near time, you may find some guidelines on what features to use when expecting heavy growth of the number of pages in your site.

In general, I have found that a common denominator for problems is site enumeration. Site enumeration should simply never be allowed - this precludes the current FindPagesWithCritiera architecture for example, and most of the other ways to produce deep listings. It also disallows the current mirroring, subscription or archival algorithms where the site is enumerated to find updates. All such operations have to be done by other mechanisms, perhaps using variations on the subscriber pattern to inform listeners of updates or placing these operations on a queue that can be processed in linear time in relation to the amount of actual work to do - not the total number of pages.

A different way to express the above – all operations taking measurable amount of time per page used by the application must have sub-linear time complexity in relation to the total number of pages. A truly scalable application will not implement any features that have linear or worse time complexity in relation to the total number of pages, if the time per page is measurable. It can still have linear time complexity as far as the result set is concerned for a given operation.

A good example of a nice way to implement a feature is the blog post describing upcoming improvements of dynamic properties in CMS 6 – that algorithm has linear time complexity in relation to the number of parents that a page has, that’s quite a change from the current implementation where cache revalidation has worse than linear time complexity in relation to the total number of pages!

In EPiServer, loading a page takes significant time. Therefore, any operation that has linear time complexity in relation to the total number of pages will break down quite quickly when the size of the site is significant. If we have 100 000 pages, 100 000 x measurable time == too long. That’s the simple fact.

A corollary that I have found is that while caching is great and required for performance, the first-time hit must still be within acceptable limits. The site must still run without the cache, just not great. Otherwise we end up with situations where the site cannot start at all when under load. EPiServer suffers from this scenario when certain lists and other operations require almost the whole site to be enumerated, causing site startup to be very difficult at times.

The caching corollary also leads to the conclusion that while you can place design limits requiring for example all pages to be resident in memory for optimal performance – it must still perform well enough to actually be able to start even when under maximum load before the site has been loaded into memory.

The rules apply to background jobs as well. In an EPiServer site that runs at the limit, running a job such as mirroring or archival may well act as a denial of service attack by invalidating caches to the extent that the site stops working, or by growing the memory use so a recycle is forced. So, not even those jobs can use site enumeration – also when the time required to run a job is several hours or more, it becomes extremely troublesome.

While the above conclusions may seem self-evident at first glance, please recall that EPiServer implements many features that break these rules, and many EPiServer sites face some serious performance issues when the number of pages grow.

I propose that a good rule to apply when designing a framework such as a programmable CMS, is that if a feature exists it will be used. It’s not sufficient to to document limitations with texts like “Don’t use for large amounts of data”. Either a feature can be used within the design constraints of the framework, or it cannot. Don’t allow different design goals for different parts of the framework. If you can’t make it fast enough, don’t do it.

In EPiServer, there are many features that tend to break down and become unusable as the total number of pages grow – I hope EPiServer over time will work to change the architecture to remove those obstacles to scalability. It’s still amazing what can be done with the current architecture, but at least in my current case I’ve had to work around quite a few issues that are caused by various EPiServer features having linear time complexity or worse in relation to the total number of pages in the site.

A code-quality improving contest!

Wed, 06 Oct 2010 15:31:48 GMT

This is about what I found in EPiServer and why it concerns *you*, and a contest for you to enter in. Not only for the glory, but for the fine prizes!

My wife is very understanding, she let’s me ramble on about my everyday nerdy problems and says nothing if I pull a half-nighter (I don’t do full-nighters any more, the privilege of age aka experience) digging really deep into that memory dump and not even replying when spoken to.

The other day, I received an e-mail from a Swedish on-line forum with the message being essentially: “We’ve been hacked and our password storage is not safe. You should change your password on other sites if it’s also used here”. I went off in a spin, telling her about the simple fact that we’ve known for 40 years how to store passwords securely and irreversibly!

She then says: ‘You’re scaring me. You always go on about simple things like that and how the same mistakes keep being made over and over. It sounds like an industry full of programmers who don’t know their profession.’

I could not really say anything to that. The evidence is that it’s true. Then again I guess it’s human to err. But still… It is scary.

Shameless plug: If you want a tool to simply your life when a sites’ password database gets hacked, or if you just want help with managing all your passwords check out Xecrets on-line password generator and store.

Now to the real topic of this post and another even simpler rule than how to store passwords.

Do not ever catch unspecified exceptions silently.

That’s not a very hard rule to follow. But it’s getting violated every day, hundreds if not thousand times even as you read this.

On a site that I am involved in we started getting reports about intermittent behavior concerning logins.

I asked for logs, then increased logging levels, and more logs. (This was greatly simplified by the consolidated logs produced by the remote logging service.)

After about 4 hours of pouring over the logs, my own code and reflecting on the EPiServer code – I found the following:

public override MembershipUser GetUser(object providerUserKey, bool userIsOnline)
{
    foreach (MembershipProvider provider in this.ActiveMembershipProviders)
    {
        try
        {
            MembershipUser user = provider.GetUser(providerUserKey, userIsOnline);
            if (user != null)
            {
                this.CurrentMembershipUser = user;
                this.CurrentProvider = provider;
                return user;
            }
            continue;
        }
        catch
        {
            continue;
        }
    }
    return null;
}

This causes *any* exception to return as-if the user was not found! This in turn causes rather unpleasant effects in our case, since the calling code now assumes that the user does not exist. And of course it turns out we can get a SQL Server deadlock here!

But the real point is not that I found a bug in EPiServer. My code has bugs too. Lots of them. But… I should hope there are no silent catch-alls!

Everyone now, do a ‘find in files’ in Visual Studio with something like the following regular expression on your code base:

(catch~(:b*\())|(catch:b*\(:b*Exception.*\))

Then, check each and every one and in *all* cases add at least logging! Do not *ever* eat an exception silently.

There is never any reason to consume an exception silently. Never. Let’s repeat that. Never. Ok, just to make things clear: Never. Got it? Never…

Unfortunately this has been said so many times before, and still…

So, let’s have a contest!

Use the above regular expression or whatever tools you have to find and remove silent consuming of unspecified exceptions. The one who posts the most found and fixed will get a computer related book of their choice (within reason) from me personally. The contest lasts for 30 days, and requires at least 3 entrants and > 10 fixes total for the price to be sent. There’s no shame in fixing bugs! I’m going to do the same on my own code base, who knows…

Oh, and I’ll buy a coffee mug for anyone who can provide a convincing argument for a single situation where it’s appropriate to silently consume an unspecified exception.

PS – The contest is open for EPiServer too!

log4net Remote Logging Service released

Thu, 02 Sep 2010 22:26:33 GMT

If you ever had to look at more than one log4net log file from EPiServer, or you have had more than one site log to the same log file, this is for you.

I have released the log4net Remote Logging Server Service project to codeplex. It's not much actual code, but I could not find it ready-made for use anywhere else, so I made this for myself and also for you!

It’s been a huge life- and time-saver for myself in any situation where there is more than one log4net log file to inspect – which is pretty much every project I’ve been involved in…

It’s also solved all my issues with having EPiServer log different sites to the same file (issues such as no logging, overwritten logs at rollover etc).

You can:

Consolidate all your log4net logs from your sites and servers to a single log via the built-in log4net .NET Remoting Appender.
Solve all issues with simultaneous writes to the log file, lost logs during roll-over etc.
Browse all logs from all systems in a single log file.

Get it at http://log4netremotelogging.codeplex.com/, either a ready-made Windows Installer MSI package or the full source code in C# for Visual Studio 2008.

Dynamic Properties performance revisited

Wed, 01 Sep 2010 17:45:18 GMT

Everyone knows dynamic properties are slow and have been so through the ages.

We speeded it up by a factor x100 with a single statement. Read on to know how and why.

My colleague Måns Hjertstrand and myself sat down today with the intent to review options to remove use or work around the slowness of dynamic properties on a site we’re working on.

The site has limited use of dynamic properties, but fairly large number of pages, ca 100 000. It’s an EPiServer CMS 5 site.

We’ve previously analyzed the issue, and found that the Stored Procedure netDynamicPropertyTree was the culprit.

The behavior is very unfortunate, because every time the page cache is cleared (which is frequently) or a page is moved, the cached result of netDynamicPropertyTree is invalidated.

Every time a dynamic property is referred to, it starts by calling netDynamicProperty tree when the cache has been cleared. This only happens the first time between cache invalidations, but it’s much worse than it appears.

We’re running a web farm with 3 servers and 6 enterprise sites. this adds up to 18(!) calls to netDynamicProperty tree every time the cache is cleared.

The last time we visited this issue, a clean call of netDynamicPropertyTree took about 20 seconds. When we had up to 18 instances running at the same time, the total time took so long that the cache often was cleared again before we were done etc etc – basically crashing the server.

The solution then was to hack into the EPiServer code and change the way the cache was reloaded, using the old data in the cache until the new data was delivered from the database. This sort of worked, but the database server still had to work pretty hard, and I had to do some pretty disgusting things in C# with reflection.

Now, with more pages in the database, a clean call of netDynamicProperty tree takes about 133 seconds… It’s simply not working any more!

So Måns hit upon the idea to sneak a peak at the Stored Procedure itself and check what really was the problem and to analyze just what factors affected it.

Surprise! When we ran the code from netDynamicProperties from a SQL Query window it ran in 2-3 seconds. Not 133. Hmm…

To make a long story short – the problem is that a temporary table is created, and when run from a Stored Procedure, SQL Server never figures out it needs an index.

Googling turned up many similar situations where SQL Server is slow when running a stored procedure, but fast when running from a query window. Many solutions were found where the problem was with parameter sniffing (we don’t have any), or ANSI NULL (we’re ok there), recompilation (nope, didn’t help) or suggested use a table variable instead of temp table (nope, slower).

So – the following addition to netDynamicProperties speeded up use of dynamic properties by a factor of almost x100, and totally changed the responsiveness of the whole site.

create clustered index idx_tmpprop_fkParentID on #tmpprop ( fkParentID )

That’s it. Running time down to about 1700 milliseconds instead of 133000. Not quite 100x but when several are running it’s much more!

One would have thought this would have been addressed, considering that dynamic properties have been such a problem for so many years. I guess we struck lucky, considering that neither of us really knows anything about SQL Server.

Oh, I also did the corresponding change to netDynamicPropertiesLoad – it can’t hurt I guess.

Oh, oh – I did say we don’t know much about SQL Server? I have no idea why it apparently does not matter what column we index… Comments appreciated.

Of course we do not recommend you try this at home, after all it does mean voiding your warranty. You might want to ask for this, or something better, to be retrofitted in a hotfix for your version of EPiServer though.

Disclaimer: Your mileage may vary.

What you should not need to know about EPiServer, IIS and Client Certificates

Thu, 05 Feb 2009 11:34:29 GMT

If you configure your EPiServer site to use SSL/HTTPS with client certificates, you may at seemingly random intervals get problems with blank pages – empty white frames, in the edit and admin mode, mostly. This can happen basically at any time, pe...

Logging woes with log4net and enterprise sites

Wed, 21 Jan 2009 16:41:25 GMT

A fundamental issue that has many effects, is that EPiServer in enterprise mode runs several web sites, potentially in different App Pools, with a single common application root directory – and therefore also a single common configuration file. Th...

IsolatedStorage Access Denied

Fri, 31 Oct 2008 10:39:47 GMT

So you're security conscious and want to run your app pool under a custom identity, not just the good old Network Service account? (Actually, this is probably not a good idea, but the client gets what the client wants.) All is well and good, even...

When a 404 Not Found should be a 404 Not Found

Thu, 02 Oct 2008 09:14:57 GMT

The ASP. NET standard behavior with custom error pages is dubious, at best. When a page is not found, it does not say so. It says that the page has been moved (302), and then it typically says either that the page now indeed was found (200) at the...