Thumbnail service is a windows service, it can be installed as an optional component when you deploy CMO on site using Deployment Center.

Thumbhandler.axd HTTP handler is added in web.config for CMO UI virtual path when CMO UI is installed on website:

<configuration>
    <location path="EPiServer/CMOUIPath">
    <system.webServer>
        <handlers>
            <add name="ThumbnailHandler" verb="*" path="thumbhandler.axd" type="EPiServer.Cmo.Cms.ThumbnailHandler.CmoThumbnailHandler" resourceType="Unspecified" requireAccess="Script" />
        </handlers>
      </system.webServer>
    </location>
</configuration>

Right now it is added under:

<location path="episerver/CMO">

However, I installed CMO with a new Alloy site with development center, just to try to get it working. Now, I get a thumbnail, but the thumbnail is a screenshot from an IE error page that says: Navigation to the page was cancled.

I tried to adjust the security options in IE, but it doesn't work. I've also made sure to reserve the service url with urlacl, and made the service log on with my domain user. Still doesn't work, so I'm kinda out of idéas.

Try to enable Debug logging for Thumbnail service and see if you get more information about the problem. Edit config file and add something like this:

<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
  <file value="ServiceLog.txt" />
  <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
  <appendToFile value="true" />
  <rollingStyle value="Date" />
  <datePattern value="yyyyMMdd " />
  <layout type="log4net.Layout.PatternLayout">
    <conversionPattern value="%date [%thread] %-5level %type.%method - %message%newline" />
  </layout>
</appender>

<root>
  <level value="Debug" />
  <appender-ref ref="RollingLogFileAppender" />
</root>

Also you can check if this technote is applicable in your case.

But in general, it sounds like IE configuration issue. Can you browse target CMS pages if you run Internet Explorer on the server where Thumbnail service is deployed?