Import of large sites fails

Join us this Friday for AI in Action at the Virtual Happy Hour! This free virtual event is open to all—enroll now on Academy and don’t miss out.

AI OnAI Off

Home / Forums / Developers (Main Products) Forum / Content Management System 12 /

Import of large sites fails

Carl S

Vote:

We're in the process of changing from CMS 11 to CMS 12 and want to start on a fresh database.

Therefore we want to use the Import data feature, how ever, we get the error "timed out or max file size exceeded".

I've tried multiple things:

https://www.webdavsystem.com/server/documentation/large_files_iis_asp_net

web.config

<configuration>
	<system.webServer>
		<modules>
			<remove name="RequestFilteringModule"/>
		</modules>
		<handlers>
			<remove name="aspNetCore"/>
			<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified"/>
		</handlers>
		<aspNetCore processPath=".\*.exe" stdoutLogEnabled="true" stdoutLogFile=".\logs\stdout" hostingModel="inprocess" requestTimeout="02:00:00">
			<environmentVariables>
				<environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Local" />
			</environmentVariables>
		</aspNetCore>
		<httpErrors errorMode="Detailed" />
		<httpProtocol>
			<customHeaders>
				<remove name="X-Powered-By" />
				<add name="X-UA-Compatible" value="IE=Edge" />
			</customHeaders>
		</httpProtocol>
	</system.webServer>
</configuration>

Startup.cs

// If using Kestrel:
services.Configure<KestrelServerOptions>(options =>
{
	options.Limits.MaxRequestBodySize = null;
	options.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(60);
	options.Limits.RequestHeadersTimeout = TimeSpan.FromMinutes(60);
	options.AllowSynchronousIO = true; // NewtonSoft.Json issue
	options.AddServerHeader = false;
});

// If using IIS:
services.Configure<IISServerOptions>(options =>
{
        options.MaxRequestBodyBufferSize = int.MaxValue;
	options.MaxRequestBodySize = null;
	options.AllowSynchronousIO = true; // NewtonSoft.Json issue
});

// uploading options for EPi
services.Configure<UploadOptions>(options => options.FileSizeLimit = 4294967295L);

Startup.cs

app.UseWhen(context => context.Request.Path.StartsWithSegments(new PathString("/EPiServer"), StringComparison.InvariantCultureIgnoreCase), builder =>
{
	var maxSizeFeature = builder.ServerFeatures.Get<IHttpMaxRequestBodySizeFeature>();
	if (maxSizeFeature is null)
	{
		return;
	}
	maxSizeFeature.MaxRequestBodySize = null;
});

#304346

Edited, Jun 30, 2023 6:01

Quan Mai

Vote:

How big is your file, and what is exactly the error message (including stacktrace if any?)

#304358

Jun 30, 2023 10:59

Carl S - Jul 03, 2023 4:46

Hi Quan,

Everything over 30MB fails. - But some of our sites is 4 GB or so.

Setting 'MaxRequestBodySize' to null is supposed to fix the issue.

No error message in logs. Only the visual message in backoffice "timed out or max file size exceeded".

/ Carl

Paul McGann (Netcel)

Vote:

Hi Carl,

This same question was asked on the forum here with and accepted answer.

Paul

#304449

Jul 02, 2023 18:05

Carl S - Jul 03, 2023 3:20

Hi Paul.

This setting only affects file uploads in the media folders.
And I've already set this by code:

// uploading options for EPi

Carl

Paul McGann (Netcel) - Jul 03, 2023 8:25

Have you tried it with just that configuration setting?

I have this in my current projects and it is working:

"EPiServer": {
"CmsUI": {
"Upload": {
"FileSizeLimit": 104857600
}
}
},

Are you running this under a linux OS?

Carl S - Jul 03, 2023 8:33

No I have not. Setting it by code should be enough. But I can try that.

No. Windows and IIS.

Carl S - Jul 03, 2023 8:38

Your suggestion to add only configuration setting did nothing. Same error.

Quan Mai

Vote:

Just to be sure, did you combine both settings (in IIS and in application layer)?

#304501

Jul 03, 2023 9:33

Carl S - Jul 03, 2023 9:36

Combine both settings? :)

Quan Mai - Jul 03, 2023 9:40

yes, basically enable the big files upload on both IIS and on application layers.

Carl S - Jul 03, 2023 9:45

I have tired multiple variations. But yes.

remove name="RequestFilteringModule" is supposed to remove the max file limit in IIS.

Have also tried requestLimits maxAllowedContentLength="4294967295" in web.config.

And you can see by my code - setting 'MaxRequestBodySize' to null is also supposed to remove the max file limit.

Quan Mai

Vote:

A shot in the dark, but maybe this in addition of your current settings

            services.Configure<FormOptions>(x =>
            {
                x.ValueLengthLimit = 5000; // Limit on individual form values
                x.MultipartBodyLengthLimit = 737280000; // Limit on form body size
                x.MultipartHeadersLengthLimit = 737280000; // Limit on form header size
            });

#304508

Jul 03, 2023 9:53

Carl S - Jul 04, 2023 4:38

Awesome. I set these to the max values and I could upload a 3 GB file.
Thank you!

Carl S

Vote:

web.config

<configuration>
    <system.web>
        <compilation debug="true" />
        <customErrors mode="Off" />
    </system.web>
    <system.webServer>
        <handlers>
            <remove name="aspNetCore"/>
            <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
        </handlers>
        <aspNetCore processPath=".\[NAME].exe" stdoutLogEnabled="true" stdoutLogFile=".\logs\stdout" hostingModel="inprocess" requestTimeout="02:00:00">
            <environmentVariables>
                <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Local" />
            </environmentVariables>
        </aspNetCore>
        <httpErrors errorMode="Detailed" />
        <httpProtocol>
            <customHeaders>
                <remove name="X-Powered-By" />
                <add name="X-UA-Compatible" value="IE=Edge" />
            </customHeaders>
        </httpProtocol>
        <security>
            <requestFiltering removeServerHeader="true">
                <!-- IIS settings, specifies the maximum length of content in a request in bytes, default value is 30000000 -->
                <requestLimits maxAllowedContentLength="4294967295" maxQueryString="32768" maxUrl="65536" />
            </requestFiltering>
        </security>
    </system.webServer>
</configuration>

Startup.cs

services.Configure<UploadOptions>(options => options.FileSizeLimit = 4294967295L);

// If using Kestrel:
services.Configure<KestrelServerOptions>(options =>
{
	options.Limits.MaxRequestBodySize = null;
	options.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(60);
	options.Limits.RequestHeadersTimeout = TimeSpan.FromMinutes(60);
	options.AllowSynchronousIO = true;
	options.AddServerHeader = false;
});

// If using IIS:
services.Configure<IISServerOptions>(options =>
{
	options.MaxRequestBodyBufferSize = int.MaxValue;
	options.MaxRequestBodySize = null;
	options.AllowSynchronousIO = true;
});

services.Configure<FormOptions>(options =>
{
	options.ValueLengthLimit = int.MaxValue; // Limit on individual form values
	options.MultipartBodyLengthLimit = long.MaxValue; // Limit on form body size
	options.MultipartHeadersLengthLimit = int.MaxValue; // Limit on form header size
});

Startup.cs

app.UseWhen(context => context.Request.Path.StartsWithSegments(new PathString("/EPiServer"), StringComparison.InvariantCultureIgnoreCase), builder =>
{
	var maxSizeFeature = builder.ServerFeatures.Get<IHttpMaxRequestBodySizeFeature>();
	if (maxSizeFeature is null || maxSizeFeature.IsReadOnly)
	{
		return;
	}
	maxSizeFeature.MaxRequestBodySize = null;
});

#304582

Edited, Jul 04, 2023 5:00

This topic was created over six months ago and has been resolved. If you have a similar question, please create a new topic and refer to this one.