Hi,

If you were using CMS 11 I would suggest using the KinAndCarta.Webhooks add-on as I suspect that would do what you're looking for. Unfortunately I've not had the time to finish porting that over to CMS12 yet however I can tell you the approach we're taking to implementing that functionality in the CMS 12 version and hopefully that'll point you in the right direction.

In the .Net 4.7 version we were using HostingEnvironment.QueueBackgroundWorkItem to make the HTTP requests in the background but that's no longer available in .Net 5/6 so we're looking at creating a background service using IHostedService and using that service to process queued webhooks, making the required HTTP calls. The webhook objects are queued in response to the standard ContentEvents (plus other custom events where needed) and so don't slow down the UI when publishing or performing other actions which might result in a webhook being fired.

The basics of creating an IHostedService are here:
https://learn.microsoft.com/en-us/dotnet/architecture/microservices/multi-container-microservice-net-applications/background-tasks-with-ihostedservice 

If you're not concerned by adding additional libraries to the project, I suspect you might be able to use something like Quartz or Hangfire to process tasks in the background in response to content events but I've not looked into that approach in any depth.

thanks Paul - I will have a read through that.. 

I am basically trying to invalidate some cache on AWS so need to work out which page urls may have changed when a block/page is published, then make a call to the AWS cache invalidation endpoint with the appropriate urls.

assuming for this use case this is still the correct approach?

You could build the logic in to a scheuled job that you trigger to run via the job service when the page is published, storing any data you need temporarily. As jobs run in the background to churn through what needs done it would then just run in the background until it's figured everything out and invalidated.

Hi Nat,

CDN cache invalidation on publish was one of the use cases that inspired the webhooks add-on so I certainly think your scenario would be a valid one for that kind of approach. There are a few caveats/considerations though. You may already have an approach to these but I think it's worth mentioning them anyway.

When you're looking at what needs to be invalidated from the cache, it's obviously not as simple as just invalidating a single item when you publish that item, particularly where blocks are involved. When you publish a block, you'll need to get all content which references that block and clear that from the cache too. The `IContentRepository.GetReferencesToContent` method can help with that. If you've got blocks nested in other blocks, you'll need to take the nesting into account too.

If you're pulling content onto a page dynamically (e.g. via search, listing children of the current page, etc.), there's no direct link so it's more tricky to work out what needs invalidating. Even if you keep track of everything rendered on every page, it's still tricky to know whether to invalidate the cache for a page due to something new being added which may now form part of your page but didn't previously. A simple example would be a news listing page. When you add a new article, there are no references to that article so, clearing the cache of pages which reference it wouldn't work.

Changing the URL segment of a page would require you to clear the cache of all of that page's descendents as they will all be impacted. It's been a while since I set up an AWS CloudFront CDN but, as I recall, clearing a page and its descendents in a single request is possible.

Changes to anything site-wide (e.g. navigation) would probably need you to drop the cache for all pages.

Finally, it's worth working out the expected volume of invalidation requests you're expecting to fire. I seem to recall there are some restrictions on the number of invalidation requests you can make in a given time period in CloudFront (it may be a financial limitation rather than a technical one though). For certain content, it's probably sufficient to set a sensible cache period and just accept that changes to that content may take that period of time to update.

Hope that's useful.

** UPDATE : Apollogies @Paul, I missed the second post and you had already covered the same thing **

Similar concept to what Scott suggested with a schedule job although another approach would be to use Background Tasks with Hosted Services. If the Optimizely Content Events are wrote to a Service Bus, you can read the messages in the queue and perform what ever action is needed.

    public class ServiceBusHostedService : IHostedService
    {
        private readonly IServiceBusMessageConsumer _serviceBusMessageConsumer;
        private readonly ILogger<ServiceBusHostedService> _logger;

        public ServiceBusHostedService(IServiceBusMessageConsumer serviceBusMessageConsumer, ILogger<ServiceBusHostedService> logger)
        {
            _serviceBusMessageConsumer = serviceBusMessageConsumer;
            _logger = logger;
        }

        public async Task StartAsync(CancellationToken cancellationToken)
        {
            _logger.LogInformation("Starting consumer");
            await _serviceBusMessageConsumer.ConsumeMessagesAsync().ConfigureAwait(false);
        }

        public async Task StopAsync(CancellationToken cancellationToken)
        {
            _logger.LogInformation("Stopping consumer");
            await _serviceBusMessageConsumer.CloseQueueAsync().ConfigureAwait(false);
        }
    }

Message Consumer

    public sealed class ServiceBusMessageConsumer : IServiceBusMessageConsumer, IAsyncDisposable
    {
        private readonly IServiceBusMessageProcessor _serviceBusMessageProcessor;
        private readonly ILogger<ServiceBusMessageConsumer> _logger;
        private readonly ServiceBusSubscriptionOptions _options;
        private ServiceBusClient _client;
        private ServiceBusProcessor _processor;

        public ServiceBusMessageConsumer(IServiceBusMessageProcessor serviceBusMessageProcessor, IOptions<ServiceBusSubscriptionOptions> options, ILogger<ServiceBusMessageConsumer> logger)
        {
            _serviceBusMessageProcessor = serviceBusMessageProcessor;
            _logger = logger;
            _options = options.Value;
        }

        public async Task ConsumeMessagesAsync()
        {
            try
            {
                _client = new ServiceBusClient(_options.ConnectionString);
                _processor = _client.CreateProcessor(_options.Topic, _options.Subscription, new ServiceBusProcessorOptions()
                {
                    AutoCompleteMessages = false
                });

                _processor.ProcessMessageAsync += _serviceBusMessageProcessor.ProcessMessageAsync;
                _processor.ProcessErrorAsync += _serviceBusMessageProcessor.ProcessErrorsAsync;

                await _processor.StartProcessingAsync().ConfigureAwait(false);
            }
            catch (Exception ex)
            {
                _logger.LogError("An error occurred while consuming service bus messages", ex);
            }
        }

        public async ValueTask DisposeAsync()
        {
            if (_processor != null)
            {
                await _processor.DisposeAsync();
            }

            if (_client != null)
            {
                await _client.DisposeAsync();
            }
        }

        public async Task CloseQueueAsync()
        {
            if (_processor?.IsClosed == false)
            {
                await _processor.CloseAsync().ConfigureAwait(false);
            }
        }
    }

thanks all for your help/input

I went for the IHostedService in the end - something like this - hopefully the code for finding the block page references looks about right:

public void OnPublishedContent(object sender, ContentEventArgs e)
    {
      switch (e.Content)
      {
        case BasePage:
          _backgroundTaskQueue.QueueBackgroundWorkItem(token => Task.Run(() => _cacheInvalidationService.InvalidatePageReferences(e.Content), token));
          break;

        case BaseBlock:
          _backgroundTaskQueue.QueueBackgroundWorkItem(token => Task.Run(() => _cacheInvalidationService.InvalidateBlockReferences(e.Content), token));
          break;
      }
    }

public async Task InvalidateBlockReferences(IContent block){
    var contentType = _contentTypeRepository.Load(block.GetType().BaseType);

    var usages = _contentModelUsage.ListContentOfContentType(contentType)
      .Select(x => x.ContentLink.ToReferenceWithoutVersion())
      .Distinct();

    var urls = new List<string>();
    foreach (var usage in usages)
    {
      // Find all pages where the block is used
      var references = _contentSoftLinkRepository.Load(usage, true)
        .Where(x => x.SoftLinkType == ReferenceType.PageLinkReference && !ContentReference.IsNullOrEmpty(x.OwnerContentLink))
        .Select(x => x.OwnerContentLink);

      urls.AddRange(references.Select(reference =>
          _urlResolver.GetUrl(
            reference,
            string.Empty,
            new UrlResolverArguments
            {
              ContextMode = ContextMode.Default,
              ForceAbsolute = true
            }))
        .Where(externalUrl => externalUrl != null));
    }

    await _cacheInvalidator.InvalidateCache(urls); //call AWS using their AmazonCloudFrontClient
}

public class QueuedHostedService : BackgroundService
{
  private readonly ILogger<QueuedHostedService> _logger;

  public QueuedHostedService(IBackgroundTaskQueue taskQueue, ILogger<QueuedHostedService> logger)
  {
    TaskQueue = taskQueue;
    _logger = logger;
  }

  private IBackgroundTaskQueue TaskQueue { get; }

  protected override async Task ExecuteAsync(CancellationToken stoppingToken)
  {
    while (!stoppingToken.IsCancellationRequested)
    {
      var workItem = await TaskQueue.DequeueAsync(stoppingToken);
      try
      {
        await workItem(stoppingToken);
      }
      catch (Exception ex)
      {
        _logger.LogError(ex, $"Error occurred executing {nameof(workItem)}.");
      }
    }
  }
}

public class BackgroundTaskQueue : IBackgroundTaskQueue
{
  private readonly ILogger _logger;
  private readonly SemaphoreSlim _signal = new SemaphoreSlim(0);

  private readonly ConcurrentQueue<Func<CancellationToken, Task>> _workItems = new ConcurrentQueue<Func<CancellationToken, Task>>();

  public BackgroundTaskQueue(ILogger<BackgroundTaskQueue> logger)
  {
    _logger = logger;
  }

  public void QueueBackgroundWorkItem(Func<CancellationToken, Task> workItem)
  {
    _logger.LogInformation("Queuing cache invalidation work item");

    if (workItem == null)
    {
      throw new ArgumentNullException(nameof(workItem));
    }

    _workItems.Enqueue(workItem);
    _signal.Release();
  }

  public async Task<Func<CancellationToken, Task>> DequeueAsync(CancellationToken cancellationToken)
  {
    _logger.LogInformation("Dequeuing cache invalidation work item");

    await _signal.WaitAsync(cancellationToken);
    _workItems.TryDequeue(out var workItem);

    return workItem;
  }
}

thanks all for your help/input

I went for the IHostedService in the end - something like this - hopefully the code for finding the block page references looks about right:

public void OnPublishedContent(object sender, ContentEventArgs e)
    {
      switch (e.Content)
      {
        case BasePage:
          _backgroundTaskQueue.QueueBackgroundWorkItem(token => Task.Run(() => _cacheInvalidationService.InvalidatePageReferences(e.Content), token));
          break;

        case BaseBlock:
          _backgroundTaskQueue.QueueBackgroundWorkItem(token => Task.Run(() => _cacheInvalidationService.InvalidateBlockReferences(e.Content), token));
          break;
      }
    }

public async Task InvalidateBlockReferences(IContent block){
    var contentType = _contentTypeRepository.Load(block.GetType().BaseType);

    var usages = _contentModelUsage.ListContentOfContentType(contentType)
      .Select(x => x.ContentLink.ToReferenceWithoutVersion())
      .Distinct();

    var urls = new List<string>();
    foreach (var usage in usages)
    {
      // Find all pages where the block is used
      var references = _contentSoftLinkRepository.Load(usage, true)
        .Where(x => x.SoftLinkType == ReferenceType.PageLinkReference && !ContentReference.IsNullOrEmpty(x.OwnerContentLink))
        .Select(x => x.OwnerContentLink);

      urls.AddRange(references.Select(reference =>
          _urlResolver.GetUrl(
            reference,
            string.Empty,
            new UrlResolverArguments
            {
              ContextMode = ContextMode.Default,
              ForceAbsolute = true
            }))
        .Where(externalUrl => externalUrl != null));
    }

    await _cacheInvalidator.InvalidateCache(urls); //call AWS using their AmazonCloudFrontClient
}

public class QueuedHostedService : BackgroundService
{
  private readonly ILogger<QueuedHostedService> _logger;

  public QueuedHostedService(IBackgroundTaskQueue taskQueue, ILogger<QueuedHostedService> logger)
  {
    TaskQueue = taskQueue;
    _logger = logger;
  }

  private IBackgroundTaskQueue TaskQueue { get; }

  protected override async Task ExecuteAsync(CancellationToken stoppingToken)
  {
    while (!stoppingToken.IsCancellationRequested)
    {
      var workItem = await TaskQueue.DequeueAsync(stoppingToken);
      try
      {
        await workItem(stoppingToken);
      }
      catch (Exception ex)
      {
        _logger.LogError(ex, $"Error occurred executing {nameof(workItem)}.");
      }
    }
  }
}

public class BackgroundTaskQueue : IBackgroundTaskQueue
{
  private readonly ILogger _logger;
  private readonly SemaphoreSlim _signal = new SemaphoreSlim(0);

  private readonly ConcurrentQueue<Func<CancellationToken, Task>> _workItems = new ConcurrentQueue<Func<CancellationToken, Task>>();

  public BackgroundTaskQueue(ILogger<BackgroundTaskQueue> logger)
  {
    _logger = logger;
  }

  public void QueueBackgroundWorkItem(Func<CancellationToken, Task> workItem)
  {
    _logger.LogInformation("Queuing cache invalidation work item");

    if (workItem == null)
    {
      throw new ArgumentNullException(nameof(workItem));
    }

    _workItems.Enqueue(workItem);
    _signal.Release();
  }

  public async Task<Func<CancellationToken, Task>> DequeueAsync(CancellationToken cancellationToken)
  {
    _logger.LogInformation("Dequeuing cache invalidation work item");

    await _signal.WaitAsync(cancellationToken);
    _workItems.TryDequeue(out var workItem);

    return workItem;
  }
}