Dynamic Data Store – Tips, Tricks and Best Practises

In this blog post my aim is to summarize some of my previous posts about the DDS and highlight the things that are good to know and may help you out of potential sticky situations.

What’s the best way to obtain a store instance for a .Net Type?

EPiServer Framework Version 6.2 (CMS 6 R2 / Relate 2 R2) onwards:

• Mark your class with the EPiServerDataStoreAttribute and set the AutomaticallyCreateStore property to true

  using EPiServer.Data.Dynamic;
  [EPiServerDataStore(AutomaticallyCreateStore=true)]
  public class MyType
  {
  }

• Call GetStore on the DynamicDataStoreFactory.Instance or use the GetStore extension method of the System.Type class. As the class was decorated with the EPiServerDataStoreAttribute and the AutomaticallyCreateStore property was set to true, the call to GetStore will trigger the store creation automatically.

  var store = DynamicDataStoreFactory.
               Instance.
                GetStore(typeof(MyType));
  // Or
  var store = typeof(MyType).GetStore();

EPiServer Framework Version 6.1 (CMS 6 R1 / Relate 2 R1) or previous:

Use the CreateStore / GetStore pattern to avoid excessive database calls

using EPiServer.Dynamic.Data;
var store = DynamicDataStoreFactory.
             Instance.
              GetStore(typeof(MyType)) ??
               DynamicDataStoreFactory.
                Instance.
                 CreateStore(typeof(MyType));

What do I do when I change the definition of a class whose instances are stored in the DDS?

It depends a little on what you have changed in your class definition. Adding, removing and changing the names of properties can be handled quickly and efficiently. Changing the CLR Type of a property can be a more heavyweight operation depending upon the conversion that has to be performed and how many instances are stored in the DDS. Changing the CLR Type of a property will only succeed if the underlying database engine supports conversion between the old and new database storage types.

EPiServer Framework Version 6.2 (CMS 6 R2 / Relate 2 R2) onwards:

Add, remove or rename properties or change property CLR Type with less than 100 instances in DDS

Decorate your class with the EPiServerDataStoreAttribute and set the AutomaticallyRemapStore property to true, decorate any renamed properties with the EPiServerDataPropertyRenameAttribute setting the OldName property to the property's previous name

using EPiServer.Data.Dynamic;
[EPiServerDataStore(AutomaticallyRemapStore=true)]
public class MyType
{
   [EPiServerDataPropertyRename(OldName="MyIntProperty")]
   public int MyIntPropertyWithNewName {get; set; }
}

Change property CLR Type with more than 100 instances in DDS

In these cases I recommend you use the PowerShell cmdlet we have provided:

Upgrade-EPiRemapDDSTypes 
 -ApplicationPath pathToWebAppHostingType
 -Type typeNameIncludingNamespace
 -Properties propertiesToBeRenamed

The parameters to the cmdlet can be clarified as follows:

• pathToWebAppHostingType - The full path to the website whose bin folder has an assembly hosting the Type to be remapped, e.g. "C:\EPiServer\Sites\MyEPiServerSite\"
• typeNameIncludingNamespace - The full name including namespace of the Type to be remapped, e.g. "MyNamespace.MyType"
• propertiesToBeRenamed - A query string style set of name-value pairs detailing the properties of the Type that have been renamed, e.g. "MyIntProp=MyNewNameForIntProp&MyStringProp=MyNewNameForStringProp"

I have created a PowerShell script that will execute the Upgrade-EPiRemapDDSTypes cmdlet. You just need to pass the 3 parameters to the script when executing it:

. "<PathToScript>\Upgrade DDS Type.ps1"
   "C:\EPiServer\Sites\MyEPiServerSite\"
   "MyNamespace.MyType"
   "MyIntProp=MyNewNameForIntProp&MyStringProp=MyNewNameForStringProp"

To select the site using a UI wizard, pass $null as the 1st parameter. If no properties have been renamed then pass $null as the 3rd parameter.

. "<PathToScript>\Upgrade DDS Type.ps1"
   $null
   "MyNamespace.MyType"
   $null

The script can be downloaded here.

EPiServer Framework Version 6.1 (CMS 6 R1 / Relate 2 R1) or previous:

Add, remove or rename properties or change property CLR Type with less than 100 instances in DDS

• Option 1.

  Download the DDS Extension assembly I created (alternatively download and compile the source code yourself)

  Decorate your class with the DdsAutoRemapAttribute and decorate any renamed properties with the DdsAutoPropertyRenameAttribute setting the OldName property to the property's previous name

  using EPiServer.Samples.DdsExtensions;
  [DdsAutoRemap]
  public class MyType
  {
     [DdsAutoPropertyRename(OldName="MyIntProperty")]
     public int MyIntPropertyWithNewName {get; set; }
  }

• Option 2.

  Create an EPiServer Initialization Module and call the DDS Remap API for your type

  using EPiServer.Data;
  using EPiServer.Data.Dynamic;
  using EPiServer.Framework;
  using EPiServer.Framework.Initialization;
  [InitializableModule]
  [ModuleDependency(typeof(DataInitialization))]
  public class InitializationModule : IInitializableModule
  {
      #region IInitializableModule Members
      public void Initialize(InitializationEngine context)
      {
          var storeDefinition = 
                 StoreDefinition.Get(
                  DynamicDataStoreFactory.
                   Instance.
                    GetStoreNameForType(typeof(MyType)));
  
          // Does the store need remapping?
          if (storeDefinition != null && 
                !storeDefinition.ValidateAgainstMappings(typeof(MyType), false))
          {
              // Explicity rename any properties first
              if (storeDefinition.GetMapping("MyIntProperty") != null)
              {
                  storeDefinition.RenameProperty("MyIntProperty", 
                      "MyIntPropertyWithNewName");
              }
              // Then do a general remap
              storeDefinition.Remap(typeof(MyType));
              // Finally save the changes
              storeDefinition.CommitChanges();
          }
      }
      public void Preload(string[] parameters)
      {
      }
      public void Uninitialize(InitializationEngine context)
      {
      }
      #endregion
  }

Change property CLR Type with more than 100 instances in DDS

I recommend using Option 2 above but starting the site in a controlled manner (i.e. don't let your first real visitor take the hit). You may need to increase the time-out in the database connection string for remaps where large amounts of data are involved.

How do I control which properties of my class are saved to the DDS?

By default the DDS will save all properties that have both a getter and a setter where the getter is marked public. To override this behavior you have 3 options:

• Option 1.

  Decorate the class with the System.Runtime.Serialization.DataContractAttribute and each property that should be saved, regardless of it's access modifier, with the System.Runtime.Serialization.DataMemberAttribute.

• Option 2.

  Decorate the class with the EPiServer.Data.Dynamic.EPiServerDataContractAttribute and each property that should be saved, regardless of it's access modifier, with the EPiServer.Data.Dynamic.EPiServerDataMemberAttribute. These do exactly the same job but are provided in case the class has already been decorated with DataContract/DataMember and the properties decorated do not match with your DDS requirements.

• Option 3.

  Call the DynamicDataStoreFactory.CreateStore method with a type bag for your type:

  public class SomeType
  {
      public int SomeIntProperty { get; set; }
      public int SomeDerivedValueProperty { get; set; }
  }
  Dictionary<string,Type> typeBag = new Dictionary<string,Type>();
  typeBag.Add("SomeIntProperty", typeof(int));
  var store = DynamicDataStoreFactory.
               Instance.
                CreateStore("MyStoreName", typeBag);

  This is useful when for some reason you cannot change the class definition but still want to do a partial mapping.

What's the best way to store collections?

Saving a collection as a top-level object is currently not supported. So for example, the following will not work as you might expect:

List<string> strings = new List<string>();
strings.Add("One");
strings.Add("Two");
var store = DynamicDataStoreFactory.
             Instance.
              CreateStore("StringStore", typeof(List<string>));
store.Save(strings);

What will happen is that the class List will be reflected over and it's DDS compatible properties extracted. So instead of storing 2 strings you will just store one property called "Capacity"

So the solution to this is to enclose the collection in a container class:

class StringCollectionContainer
{
    public List<string> Strings { get; set; }
}
List<string> strings = new List<string>();
strings.Add("One");
strings.Add("Two");
var container = new StringCollectionContainer();
container.Strings = strings;
var store = DynamicDataStoreFactory.
             Instance.
              CreateStore("StringStore", typeof(StringCollectionContainer));
store.Save(container);

If your collection is likely to get large (you need to judge how big large is) then you should de-couple your objects. Consider the following class structure:

public class ForumTopic
{
    public string Title { get; set; }
    public string CreatedBy { get; set; }
    public DateTime CreatedOn { get; set; }
    List<ForumEntry> Entries { get; set; }
}
public class ForumEntry
{
    public string Text { get; set; }
    public string CreatedBy { get; set; }
    public DateTime CreatedOn { get; set; }
}

If each ForumTopic only has a few ForumEntry instances in the Entries property (let's say less than 50 for arguments sake) then loading a ForumTopic and it's child ForumEntry objects shouldn't be too troublesome.

However, the problem comes when the number of ForumEntry objects per ForumTopic grows to the stage where do you don't want to load them all into memory at the same time. From a user interface perspective you would normally implement paging and therefore only maybe show say 20 posts at a time.

The solution here is to not store the ForumEntry objects as a collection property on the ForumTopic. Instead you would add a new property to the ForumEntry class which would hold the identity of the owning ForumTopic. You would also need to expose the identity on the ForumTopic by implementing the EPiServer.Data.Dynamic.IDynamicData interface:

using EPiServer.Data;
using EPiServer.Data.Dynamic;
public class ForumTopic : IDynamicData
{
    public string Title { get; set; }
    public string CreatedBy { get; set; }
    public DateTime CreatedOn { get; set; }
    #region IDynamicData Members
    public Identity Id { get; set; }
    #endregion
}
public class ForumEntry
{
    public Identity TopicId { get; set; }
    public string Text { get; set; }
    public string CreatedBy { get; set; }
    public DateTime CreatedOn { get; set; }
}

Now it's a matter of setting the ForumTopic's identity on each ForumEntry when the entry is saved.

Paging of ForumEntry's can now be implemented easily using LINQ:

private IEnumerable<ForumEntry> GetForumEntries(ForumTopic topic, int page, int pageSize)
{
    var store = typeof(ForumEntry).GetStore();
    return (from entry in store.Items<ForumEntry>() 
            where entry.TopicId == topic.Id 
            select entry)
            .Skip((page - 1) * pageSize)
            .Take(pageSize);
}

I want to save to more than one store at the same time. Can I use transactions?

You can of course use the System.Transactions.TransactionScope class to save to multiple stores in the same transaction but the DDS also has built-in support by using the ExecuteTransaction method of the data store provider:

using EPiServer.Data.Dynamic;
using EPiServer.Data.Dynamic.Providers;
private void SaveForumTopicAndEntry(ForumTopic topic, ForumEntry entry)
{
    var provider = DataStoreProvider.CreateInstance();
    provider.ExecuteTransaction(() =>
    {
        var topicStore = typeof(ForumTopic).GetStore();
        var entryStore = typeof(ForumEntry).GetStore();
        // Set the DataStoreProvider on both stores so they share the current transaction
        topicStore.DataStoreProvider = provider;
        entryStore.DataStoreProvider = provider;
        topicStore.Save(topic);
        entry.TopicId = topic.Id;
        entryStore.Save(entry);
    });
}

I want to save to the DDS and perform other database updates at the same time. Can I use transactions?

Again, you can use the System.Transactions.TransactionScope class to save to multiple data sources in the same transaction. The DDS also has built-in support by setting an external transaction on the store's provider (only works when the store's provider is a DbDataStoreProvider):

private void SaveForumTopicInTransaction(ForumTopic topic, 
                                         System.Data.IDbTransaction transaction)
{
    var topicStore = typeof(ForumTopic).GetStore();
    DbDataStoreProvider dbProvider = topicStore.DataStoreProvider as DbDataStoreProvider;
    if (dbProvider != null)
    {
        dbProvider.SetExternalTransaction(transaction);
        topicStore.Save(topic);
    }
}

I often use a property in calls to Find and LINQ queries. Can I index that property so searches are faster?

There are three ways to index a property:

• Option 1.

  Decorate the property with the EPiServerDataIndexAttribute:

  [EPiServerDataIndex]
  public Identity TopicId { get; set; }
  

• Option 2.

  Explicitly create the store and pass in a StoreDefinitionParameters instance:

  private DynamicDataStore CreateForumEntryStore()
  {
      StoreDefinitionParameters storeParams = new StoreDefinitionParameters();
      storeParams.IndexNames.Add("TopicId");
      return DynamicDataStoreFactory.Instance.CreateStore(typeof(ForumEntry), 
                                                          storeParams);
  }
  

• Option 3.

  Register a StoreDefinitionParameters instance with the global registry when the application starts. When the store is created the parameters will be picked up and used:

  StoreDefinitionParameters storeParams = new StoreDefinitionParameters();
  storeParams.IndexNames.Add("TopicId");
  GlobalStoreDefinitionParametersMap.
   Instance.
    Add(typeof(ForumEntry), storeParams);
  

I need to keep my DDS data in separate database tables. Can I control this?

Custom Big Tables

The DDS can use as many database tables as you like, one for each entity type if you wish. You may do this for performance reasons (the larger the table, the slower the operation in most cases) or it may be a requirement of the project to keep different types of information seperate for compliance or other reasons.

There is unfortunately no classes to help you create a custom DDS Big Table. You can however re-use the SQL Script that Deployment Center uses when creating a new CMS site. This can be found in the file {Program Files x86}\Framework\{Framework Version}\Database\sql\EPiServer.Data.sql. See the 'create table [dbo].[tblBigTable]' statement.

Your custom Big Table does not need to be exactly the same as the standard Big Table. The minimum requirement to qualify as a DDS Big Table is that is should have the pkId, Row, StoreName and ItemTypes columns with the correct data type and size.

Your additional columns don't need to have any special naming convention but bear in mind the columns you have (or don't have) determine what CLR types can be stored in the table. Consider the following create table statement for a custom Big Table:

create table [dbo].[MyCustomBigTable]
(
  [pkId] bigint not null,
  [Row] int not null CONSTRAINT [DF_MyCustomBigTable_Row] DEFAULT(1) 
      constraint CH_MyCustomBigTable check ([Row]>=1),
  [StoreName] nvarchar(375) not null,
  [ItemType] nvarchar(2000) not null,
  [MyCustomString] nvarchar(max) null,
  [MyCustomInt] int null,
  constraint [PK_MyCustomBigTable] primary key clustered
  (
	[pkId],
	[Row]
  ),
  constraint [FK_MyCustomBigTable_tblBigTableIdentity] foreign key
  (
	[pkId]
  )
  references [tblBigTableIdentity]
  (
	[pkId]
  )
)	

This is a perfectly valid DDS Big Table definition. However, only an object with strings and integers can be stored in it and if the object has more than 1 string or integer then it will be stored in more than 1 row.

Mapping types to custom Big Tables

Once you have your custom Big Table in place you need to tell the DDS to use it for the desired types. There are 3 ways to do this:

• Option 1.

  Decorate the class with the EPiServerDataTableAttribute and optionally each property with the EPiServerDataColumnAttribute:

  [EPiServerDataTable(TableName="tblForumTopic")]
  public class ForumTopic : IDynamicData
  {
      [EPiServerDataColumn(ColumnName="Title")]
      public string Title { get; set; }
      [EPiServerDataColumn(ColumnName = "CreatedBy")]
      public string CreatedBy { get; set; }
      [EPiServerDataColumn(ColumnName = "CreatedOn")]
      public DateTime CreatedOn { get; set; }
      #region IDynamicData Members
      public Identity Id { get; set; }
      #endregion
  }

  If the EPiServerDataColumnAttribute is not used then the properties will be mapped to the next available column with the correct data type as with a the standard Big Table.

• Option 2.

  Explicitly create the store and pass in a StoreDefinitionParameters instance:

  private DynamicDataStore CreateForumTopicStore()
  {
      StoreDefinitionParameters storeParams = new StoreDefinitionParameters();
      storeParams.TableName = "tblForumTopic";
      storeParams.ColumnNamesMap.Add("Title", "Title");
      storeParams.ColumnNamesMap.Add("CreatedBy", "CreatedBy");
      storeParams.ColumnNamesMap.Add("CreatedOn", "CreatedOn");
      return DynamicDataStoreFactory.Instance.CreateStore(typeof(ForumTopic), 
                                                          storeParams);            
  }
  

• Option 3.

  Register a StoreDefinitionParameters instance with the global registry when the application starts. When the store is created the parameters will be picked up and used:

  StoreDefinitionParameters storeParams = new StoreDefinitionParameters();
  storeParams.TableName = "tblForumTopic";
  storeParams.ColumnNamesMap.Add("Title", "Title");
  storeParams.ColumnNamesMap.Add("CreatedBy", "CreatedBy");
  storeParams.ColumnNamesMap.Add("CreatedOn", "CreatedOn");
  GlobalStoreDefinitionParametersMap.
   Instance.
    Add(typeof(ForumTopic), storeParams);

I don't want stores to be named using the classes' namespace and name. I want to name them myself

This can be done in several ways depending upon what you are trying to achive.

Always save Type 'X' to store named 'Y'

• Option 1.

  Decorate the class with the EPiServerDataStoreAttribute and set the StoreName property to the name desired:

  [EPiServerDataStore(StoreName="ForumTopicStore")]
  public class ForumTopic : IDynamicData
  {
  }

• Option 2.

  Register a Type to store name mapping with the global registry when the application starts. When the store is created the mapping will be picked up and used:

  GlobalTypeToStoreMap.Instance.Add(typeof(ForumTopic), "ForumTopicStore");

Save Type 'X' to store named 'Y' when X is the top-level object in the graph

This involves calling CreateStore/GetStore with a string store name as well as a Type:

private void SaveForumTopic(ForumTopic topic)
{
    var topicStore = DynamicDataStoreFactory.Instance.GetStore("ForumTopicStore") ??
        DynamicDataStoreFactory.Instance.CreateStore("ForumTopicStore", 
                                             typeof(ForumTopic));
    topicStore.Save(topic);            
}

However, should a ForumTopic instance be saved as part of an object graph, where it is not the top level object being saved, then the object will be saved in a store whose name is derived from the Type as normal (this includes checking the class for the EPiServerDataStoreAttribute and checking the GlobalTypeToStoreMap for a mapping).

Save Type 'X' to store named 'Y' when X is a child object in the graph

This involves passing a delegate to the DynamicDataStore.Save method. This technique allows you to save the same Type in several different stores depending upon it's context in the object graph:

public class Company
{
    public Person ChiefExecutive { get; set; }
    public List<Person> Directors { get; set; }
    public List<Person> Employees { get; set; }
    // Other properties omitted for brevity......
}
public class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
        
    // Other properties omitted for brevity......
}

private void SaveCompany(Company company)
{
    var companyStore = typeof(Company).GetStore();
    companyStore.Save(company, (propertyName, propertyValue) =>
    {
        switch (propertyName)
        {
            case "ChiefExecutive":
            case "Directors":
                return "Management";
            case "Employees":
                return "Employees";
            // Returning string.Empty causes the 
            // store name to be derived as normal
            default: return string.Empty;
        }
    });
}

Happy DDS'ing!