Integrating IndexNow with Optimizely Publishing

Overview of IndexNow

IndexNow is a standard for search engines that allows you to submit a URL (along with your API key) to inform the search engines that the page should be indexed. We wanted to automate this by hooking into the PublishedContent event.

The payload follows this format:

Request
POST /IndexNow HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: api.indexnow.org
{
  "host": "www.example.org",
  "key": "e50c2cc6036f4922945f1c62fab66f9e",
  "keyLocation": "https://www.example.org/e50c2cc6036f4922945f1c62fab66f9e.txt",
  "urlList": [
      "https://www.example.org/url1",
      "https://www.example.org/folder/url2",
      "https://www.example.org/url3"
      ]
}

Implementation 

First, we needed a way to host the API key which normally is stored in a plain text (txt) file hosted at the root of the website. I did not want to open up the website to serve up any text files that might happen to be part of the site so I created a new pagetype.

Txtpage

Txtpage.cs is a simple CMS pagetype with a single field for the content of the text file.

[Access(Roles = "Administrators, WebAdmins, CmsAdmins")]
[ContentType(DisplayName = "TxtPage", GUID = "XXXXXXXX-XXX-XXXX-XXXX-XXXXXXXXXXX", Description = "")]
public class TxtPage : PageData
{
        [Display(
            Name = "Content",
            Description = "Txt Content",
            GroupName = PropertyGroups.PageContent,
            Order = 1)]
        public virtual String? Content { get; set; }
}

The cshtml is about as simple as it ever gets, using Html.Raw() to output the content.

@{
    Layout = null;
}
@model Century.Website.Features.Pages.TxtPage.TxtPage

@Html.Raw(Model.Content)

Updating the Name In Url field with the key and .txt extension allows this to mimic the path of an actual text file.

The content displays as so:

API Key

The API Key is stored in appsettings.json. I also added an environment-specific value for enabling/disabling the feature. (You probably don't want lower environment broadcasting themselves to search engines.)

These are pulled from the JSON by the Dependency-injected Configuration class.

var bingKey = _configuration.GetValue<string>("BingAPIKey");

var indexNowEnabled = _configuration.GetValue<bool>("BingIndexNowEnable");

Initialization Module to attach PublishedContent content event 

We make a few checks to ensure that the page being published is something we want to submit to search engines. After checking that the feature is enabled for the current environment we check the interface. Here we make sure that it is NOT a page type excluded from the sitemap (the IExcludeFromSitemap interface comes from the Geta 404Handler package) and ISEOAnalytic is our interface for pagetypes that have a SEO tab with noindex and nofollow settings. If noindex is active we do not want this page submitted.

private void PublishedContent(object sender, ContentEventArgs e)
{
    try
    {
        var indexNowEnabled = _configuration.GetValue<bool>("BingIndexNowEnable");

        if (indexNowEnabled)
        {
            if (e.Content is ISEOAnalytic baseSeo)
            {
                //is set to noindex or is excluded by interface
                var markedNoIndex = e.Content is IExcludeFromSitemap || (baseSeo != null && baseSeo.NoIndex);

                if (!markedNoIndex)
                {
                    _ = SubmitToUrl(e.ContentLink.GetExternalUrl());
                }
            }
        }
    }
    catch (Exception ex)
    {
        Logger.Error($"ERROR in BINGIndexNow: {ex?.Message}, InnerException: {ex?.InnerException}");
    }
}

If it passes all checks we call the SubmitToUrl method passing in the external URL.

We have an IndexNowJson.cs POCO class for submitting as the payload.

public class IndexNowJson
{
    public string host { get; set; }
    public string key { get; set; }
    public string keyLocation { get; set; }
    public string[] urlList { get; set; }
}

This is where the txt page created earlier and the apikey come together. (Obviously the key in appsettings and in the txtpage should be the same.)

var indexNowData = new IndexNowJson
{
    host = $"www.yourdomain.com",
    key = bingKey,
    keyLocation = $"www.yourdomain.com/{bingKey}.txt",
    urlList = new[] { url }
};

We take in the URL, create the payload, and POST it to the IndexNow endpoint. 

//fire and forget
private async Task SubmitToUrl(string url)
{
    var bingKey = _configuration.GetValue<string>("BingAPIKey");

    var indexNowData = new IndexNowJson
    {
        host = $"www.yourdomain.com",
        key = bingKey,
        keyLocation = $"www.yourdomain.com/{bingKey}.txt",
        urlList = new []{ url }
    };

    var jsonClass = JsonConvert.SerializeObject(indexNowData);

    var request = new HttpRequestMessage(HttpMethod.Post, "https://api.indexnow.org/IndexNow");
    request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    request.Content = new StringContent(jsonClass, Encoding.UTF8);
    request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");


    var postResponse = await _httpClient.SendAsync(request);

    Logger.Log(Level.Information, $"Status Code: {postResponse.StatusCode} Reason Phrase: {postResponse.ReasonPhrase}");
}

This is a fire-and-forget situation as we don't want the site to do anything differently if there is a problem submitting to IndexNow.

Conclusion

It is a pretty straightforward process to submit to the IndexNow endpoint. I did find Bing's documentation to be inconsistent but was able to work through it. Thoughts, comments, concerns? Let me know below.