Custom content provider using FIND - really slow.

Vote:
 

Hello, I've built a custom content provider that fetches data from Find. It's reaaally slow, in edit mode it can take up to 3 minutes to fetch all data(all data = 400 objects...).

This does not seem right, 400 objects(and from FIND!) should load really fast.

Anyhow, my first thought was that I should split the objects in different subfolders so episerver does not load all data right away, but when I try to do that I get the following error message: "Deadlock risk detected. Trying to read a page from the same thread that is already trying to read the same page."

I will post my all methods in my Content provider below, would really appreciate if someone could look at it and maybe give me some pointers... :)

Questions:

  1. Why is it so slow?
  2. Why do I get a deadlock warning when executing GetFolderReference from 
public static ContentFolder GetEntryPoint(string name)
{
	var contentRepository = ServiceLocator.Current.GetInstance < IContentRepository > ();
	var folder = contentRepository.GetBySegment(ContentReference.RootPage, name, LanguageSelector.AutoDetect()) as ContentFolder;
	if (folder == null)
	{
		folder = contentRepository.GetDefault < ContentFolder > (ContentReference.RootPage);
		folder.Name = name;
		contentRepository.Save(folder, SaveAction.Publish, AccessLevel.NoAccess);
	}
	return folder;
}

protected override IContent LoadContent(ContentReference contentLink, ILanguageSelector languageSelector)
{
	var mappedItem = this.identityMappingService.Get(contentLink);
	if (mappedItem != null) {
		var id = mappedItem.ExternalIdentifier.ToString().Split('/').Last();
		var release = this.client.Get < ReleaseModel > (id);
		return ConvertToRelease(release);
	}
	return null;
}

protected override IList < GetChildrenReferenceResult > LoadChildrenReferencesAndTypes(ContentReference contentLink, string languageID, out bool languageSpecific)
{
	languageSpecific = false;
	var releases = GetReleasesFromFind();

	var result = releases.Select(p = >new GetChildrenReferenceResult
	{
		ContentLink = this.identityMappingService.Get(MappedIdentity.ConstructExternalIdentifier(ProviderKey, p.Id.ToString()), true).ContentLink,
		ModelType = typeof(Release)
	}).ToList();
	return result;
}

public Release ConvertToRelease(ReleaseModel releaseModel)
{
	var type = this.contentTypeRepository.Load(typeof(Release));
	var release = this.contentFactory.CreateContent(type, new BuildingContext(type)
	{
	    //Parent = GetFolderReference() this causes a deadlock warning.
		Parent = this.contentRepository.Get < IContent > (EntryPoint),
		
	}) as Release;

	release.Status = VersionStatus.Published;
	release.IsPendingPublish = false;
	release.StartPublish = releaseModel.PublishDate;

	var externalId = MappedIdentity.ConstructExternalIdentifier(ProviderKey, releaseModel.Id.ToString());
	var mappedContent = this.identityMappingService.Get(externalId, true);
	release.ContentLink = mappedContent.ContentLink;
	release.ContentGuid = mappedContent.ContentGuid;

	release.Title = releaseModel.Title;
	release.Name = releaseModel.Title;
	release.Body = releaseModel.Body;
	release.Complete = releaseModel.Complete;
	release.LogoUrl = releaseModel.LogoUrl;

	release.MakeReadOnly();
	return release;
}

// TODO change to query
public List < ReleaseModel > GetReleasesFromFind()
{
	var query = this.client.Search < ReleaseModel > ().Take(1000);
	var batch = query.GetResult();
	var releases = new List < ReleaseModel > ();
	releases.AddRange(batch);
	var totalNumberOfPages = batch.TotalMatching;

	var nextBatchFrom = 1000;
	while (nextBatchFrom < totalNumberOfPages) {
		batch = query.Skip(nextBatchFrom).GetResult();
		releases.AddRange(batch);
		nextBatchFrom += 1000;
	}

	return releases;
}

private IContent GetFolderReference(ReleaseModel releaseModel)
{
	var year = releaseModel.PublishDate.Year.ToString();
	var month = releaseModel.PublishDate.ToString("MMM");

	// check if year folder exists
	var yearFolder = this.contentRepository.GetBySegment(EntryPoint, year, LanguageSelector.AutoDetect()) as ContentFolder;
	var yearFolderReference = yearFolder ? .ContentLink;
	if (yearFolder == null) {
		yearFolderReference = CreateFolder(year, EntryPoint);
	}

	var monthFolder = this.contentRepository.GetBySegment(yearFolderReference, month, LanguageSelector.AutoDetect()) as ContentFolder;
	if (monthFolder != null) {
		return monthFolder;
	}

	var monthFolderReference = CreateFolder(month, yearFolderReference);
	return this.contentRepository.Get < ContentFolder > (monthFolderReference);
}

private ContentReference CreateFolder(string folderName, ContentReference parentReference)
{
	var contentFile = this.contentRepository.GetDefault < ContentFolder > (parentReference);
	contentFile.Name = folderName;
	return this.contentRepository.Save(contentFile, SaveAction.Publish, AccessLevel.NoAccess);
}

[Pasting files is not allowed]

#183658
Oct 19, 2017 9:38
Vote:
 

Try to run Fiddler (https://www.telerik.com/fiddler) to see what it is happening in the communication from the site to Episerver Find.

#183659
Oct 19, 2017 10:49
Vote:
 

The find query is really fast. The problem seems to be the calls to identityMappingService, also, I noticed that LoadChildrenReferencesAndTypes gets called for each item. I've added a return in that method now, if it's a child to the folder I just return a empty list. Not ideal, but it made it a lot faster.

#183660
Oct 19, 2017 10:51
Vote:
 

Ok. Maybe a lot of talking to the DDS is happening in that Service that makes it slow (I do not remember where it keeps the mappings but I guess in the DDS and that is not directly famous to be fast).

#183661
Oct 19, 2017 10:55
Vote:
 

Hey Josef, did you find any solution to this issue? Im getting the same dead lock message. Not using Find but Elastic, so pretty similar setup. 

#188605
Feb 27, 2018 17:16
Vote:
 

No, I ended up skipping the custom content provider altogether, had a lot of other problems as well, sorry..

#188644
Feb 28, 2018 12:47
Vote:
 

Ok, I got it working now. Problem was that I messed up the content folder structure I created in LoadChildrenReferencesAndTypes.

#188698
Mar 01, 2018 11:37
Vote:
 

Hey, Can you please guide me how did you resolved the deadlock issue.

#285158
Aug 08, 2022 10:00
Vote:
 

As always, the first step of fixing a performance problem is to identify the bottleneck. You can use something like dotTrace to profile your code. Looking at the code is not enough to determine where the bottleneck is, and in most of the cases, it is something else entirely (trust me I've been there) 

#285230
Aug 09, 2022 7:53
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.