Blog posts by Elias Lundmark

Keeping local environments in sync with your Cloud environments

Thu, 21 Mar 2024 12:08:12 GMT

We recently announced that we are improving the scalability of SQL databases in DXP Cloud Services, this new architecture also enhances our overall security for SQL databases where we are aiming to harden technical controls to maintain confidentiality and integrity of our customers data. This change had an unintended consequence though – it disallows developers from connecting local development environments directly to SQL databases in DXP Cloud Services. We strongly advise against this practice, while the ease of use and flexibility is great, manually managing and storing connection strings and credentials for service users greatly increases the risk of these credentials falling into the wrong hands, allowing potential attackers to access or modify data.

To avoid these risks, our new architecture disallows direct connections from third-party sources to SQL Servers running in DXP Cloud Services. Instead, you should use the paasportal or the API to export your databases and content to use in your local development environments, which are more secure and reliable methods.

How to export content

Via the paasportal

Navigate to https://paasportal.episerver.net and select the project you wish to export a database from
Navigate to the Troubleshoot tab
In the ‘Export Database’ section, select the environment you wish to export the database from, and how long the paasportal should retain this copy.
Once the export is done, click the database file to download it as .bacpac. These files can then be used to import your database to a local SQL server, or an Azure SQL Server.

Via API with Powershell

Navigate to https://paasportal.episerver.net and generate credentials as described here https://docs.developers.optimizely.com/digital-experience-platform/docs/authentication.
Authenticate woth Connect-EpiCloud, Connect-EpiCloud -ClientKey <ClientKey> -ClientSecret <ClientSecret> -ProjectId <ProjectId>
Start a database export with Start-EpiDatabaseExport, for example Start-EpiDatabaseExport -Environment Integration -DatabaseName epicms -Wait
Fetch the download link for the .bacpac with Get-EpiDatabaseExport

Via the API you can also download BLOBs from the storage account, where Get-EpiStorageContainer allows you to list all storage containers and GetEpiStorageContainerSasLink creates a SAS URI that can be used to download BLOBs. For example,

Get-EpiStorageContainerSasLink -ProjectId "2372b396-6fd2-40ca-a955-57871fc497c9" `

-Environment "Integration" `

-StorageContainer "mysitemedia" `

-RetentionHours 2

Import Blobs and Databases to Integration Environments

Mon, 04 Dec 2023 12:05:02 GMT

In this blog, we are going to explore some new extensions to the Deployment API in DXP Cloud Services, specifically the ability to import databases and blobs via the API. Some caveats to consider before we jump into the details,

Blob and database imports are limited to integration and ADE environments
Database imports are only available when using -DirectDeploy.

Uploading and Deploying Databases

You can know supply a bacpac file to Add-EpiDeploymentPackage

$saslink = Get-EpiDeploymentPackageLocation
Add-EpiDeploymentPackage -SasUrl $saslink -Path “C:\MyDatabaseFiles\environmentname.cms.sqldb.20231106.bacpac”

Bacpac is the same format that is used when databases are exported along with the same naming convention, so that any database that is exported can easily be imported again.

Once the upload is done, we can simply pass it to Start-EpiDeployment

Start-EpiDeployment -DeploymentPackage (”environmentname.cms.sqldb.20231106.bacpac”,”cms.app.1.0.0.nupkg”) -TargetEnvironment “Integration” -DirectDeploy

This example deploys a nupkg at the same time, but that can be ommitted to just import a database. E.g., -DeploymentPackage “environmentname.cms.sqldb.20231106.bacpac”. As usual, you will be able to see the status of the deployment in the management portal, or Get-EpiDeployment.

Creating a writeable SAS URI to upload blobs

For quite some time now, we have had the possibility to create readable SAS URIs via the Deployment API. E.g.,

Get-EpiStorageContainerSasLink -Environment “Integration” -StorageContainer “mysitemedia”

You can now add a -Writable flag to this command which enables you to upload blobs to the container as well. You can use this SAS URI to write via HTTPS, or use with Azure Storage Explorer. If you are using Azure Storage Explorer, select connect to a Blob container or Directory

Then select Shared access signature URL (SAS), and paste the writeable SAS URL

Our hope is that this will make it easier to deploy an existing site to our Cloud Services, and allow you to export content from any environment and easily import it again.

Enhancing Database Scalability in DXP Cloud Services

Thu, 02 Nov 2023 13:24:20 GMT

In today's dynamic digital environment, efficient database scalability is more critical than ever. Implementing robust autoscaling for SQL is integral to meeting this need, ensuring our systems remain flexible and responsive. In this blog post, we will delve into how we are enhancing our database infrastructure in DXP Cloud Services to improve scalability and performance.

Autoscaling solutions for SQL databases offer a sophisticated method for handling multiple databases that experience diverse workloads and resource requirements. Instead of provisioning each database with a rigid set of resources, this technology enables a more fluid resource allocation strategy. This adaptability not only mitigates the risks associated with under-provisioning but also maximizes resource utilization and optimizes performance across the board.

The advantages of our new database infrastructure enhancements include:

Management efficiency – Our solution simplifies the operation of hundreds, or even thousands, of databases. It ensures all databases receive the necessary resources, driving optimal performance across various workloads.
Auto-scaling – The system can intuitively allocate more resources during high-demand periods, maintaining high efficiency and responsiveness under fluctuating loads.

In light of these benefits, we are planning a gradual transition to this enhanced database infrastructure over the coming weeks and months. Our goal is to refine our operations and offer superior adaptability to diverse workloads and resource needs. We will initiate this transition with non-production databases, such as those in ADE, integration, and pre-production environments, starting in the SE Central region within the next few weeks.

As businesses and technologies evolve, the need for agility and efficiency only grows. Our upcoming database infrastructure enhancements equip us with the necessary tools to operate DXP Cloud Services at an elevated scale, improving our capacity to meet varying demands. This progress excites us, and we are eager to extend these benefits across our entire platform in 2024.

New Features in DXP Cloud Services

Thu, 13 Apr 2023 13:05:46 GMT

In this article, we’re going to dive into some new features we have released for the management portal and DXP Cloud Services recently.

Preview CMS 12 / Commerce 14 sites during upgrade

Soon after we released the migration tool to allow users to upgrade to CMS 12 and Commerce 14, we realized that many users were adding temporary hostnames to the new .NET Core projects in order to test multi-site setups before going live and make sure everything works as expected with discrete domain names. While this works in practice, it is not an ideal experience as adding temporary domain names can be cumbersome, especially if you have a lot of domain names.

To improve this, we have now added a feature where you can preview your new CMS 12 / Commerce 14 sites using the domain names of the source project. This works similarly to routing rules where you can specify a cookie or query string to reach a deployment slot. For example,

www.domain.com - leads to the source CMS 11 site
www.domain.com/?x-ms-routing-name=preview&x-opti-target=[environmentname] - leads to the target CMS 12 site

Adding x-ms-routing-name with the value of preview and x-opti-target with the value of the target environments name (e.g., elias01mstr123prod) will route you to your new CMS 12 site. If you wish to navigate back to the source site, simply remove the cookies named x-ms-routing-name and x-opti-target.

A bit more intuitively, you can also preview sites directly in the paasportal during the prepare go live phase.

Modernizing Infrastructure for App Services in DXP Cloud Services

Thu, 02 Mar 2023 11:22:11 GMT

Microsoft continuously updates hardware in Azure to keep up with new developments and demand. App Service Plans are no exception to this, the latest and most advanced of the App Service Plans offering is called Premium V3 (Pv3), which offers many benefits and improved features to the predecessor Premium V2 (Pv2).

One of the major benefits of moving to the Pv3 SKU is greatly increased performance and scalability. Pv3 offers improved CPU and memory utilization through the use of modern hardware, allowing for faster and more efficient processing of requests. Additionally, the increased density of virtual machines on the Pv3 SKU enables more capacity per instance – offering 2, 4, and 8 vCPUs per instance options that stretches up to 32GB of memory. Doubling the available capacity per instance compared to the previous generation.

Pv3 also offers improved availability and reliability with the introduction of a zone-redundant architecture. This means that applications hosted on the Pv3 SKU are even more fault-tolerant than ever before, ensuring that applications remain available in the event of a failure.

Pv3, and the underlying Dv4 platform, has been rolling out for new App Service Plans in several regions since early 2021 but is now finally becoming available for existing App Service Plans. We at Optimizely of course plan to take full advantage of the benefits of the Pv3 SKU to improve the scalability, performance, security, and availability of our platform.

DXP environments that have been provisioned during the latter half of 2021 or later, or have migrated to CMS 12 / Commerce 14, are most likely already running on Pv3 as it has been our default SKU for provisioning since it became available (subject to availability in different Azure regions). Over the coming weeks and months, we will also be migrating App Service Plans to Pv3 as it becomes available.

The availability and operation of our customers applications is of great importance to us, and we have developed a migration that is non-disruptive and ensures no downtime - the only thing you may notice is a change of outbound IP addresses. You will be receiving a notification from your Customer Success Manager if your environments are eligible for the upgrade to Pv3. If you do rely on allowlisting outbound IP addresses, we suggest you navigate to the management portal and get an updated list of addresses. If your environment is eligible for an upgrade to Pv3, this list will include the IP addresses post-upgrade as it contains all addresses for all possible app service SKUs.

Outbound IP addresses are available in the management portal under the Troubleshoot tab

We are excited to bring Microsoft’s latest App Service platform to our customers during 2023. Make sure to engage with your Customer Success Manager if you have any questions and follow our status page for regular updates about maintenance.

Developer Preview: Migrations to .NET 5 on DXP Cloud Services

Wed, 09 Feb 2022 11:01:13 GMT

Major upgrades can be daunting and carry risk in many areas; data corruption, breaking integrations and can require downtime. On the other side of the coin, lagging in adopting the latest upgrades carries the risk of poor security posture, obsolescence, incompatibilities and potential software rot. Due to this, we set ourselves a goal when we planned the release of Content Cloud 12 and Commerce Cloud 14 customers that are running in our DXP Cloud Services – make the transition as smooth as possible and minimize risk with the upgrade. This blogpost will give you an insight into our thought process behind our new development and give you a preview of what’s to come.

The first step of this is upgrading the software itself. For this, we created the .NET Upgrade Assistant extension specific to Optimizely scenarios that allows users to go through a guided experience to reduce time and difficulty in the task of modernizing codebases. But how do you actually go about deploying this to production in a manner that carries the least risk?

To tackle this in our DXP Cloud Services, we have been working on a similar guided experience to let existing Content Cloud 11 and Commerce Cloud 13 users deploy sites in parallel to existing environments.

Starting a migration

In the DXP Management Portal, you’ll be greeted with a new tab called Project Migration. This will act as your main hub for administrating the migration process.

Kicking of the migration will provision a parallel project that is completely decoupled from your current environments – complete with integration, preproduction and production environments (including Search & Navigation indexes). This allows you to start deploying your new Content Cloud 12 and Commerce Cloud 14 solutions without affecting status quo.

Migrating environments

Once your new project has been provisioned, you will be able to start deploying to it using the DXP Management portal UI or the deployment API. During any development process, it’s also convenient to access production data to test with “real” data and make sure everything works as expected. Navigating back to the source project in the DXP Management Portal will allow you to copy content from any environment in the source project to any environment in the target project on demand.

Going live

Once you are satisfied that the sites work as expected running on the newest version of Content and Commerce Cloud, it’s time for the moment of truth – going live. In this move, there’s a balance to be struck between incurring downtime and managing data consistency. We want to keep the source site(s) available as long as possible, but at the same time make sure we haven’t left any data behind. As such, kicking off the go live process will do a couple of things,

Put the source site(s) in maintenance mode to make sure no more data is written to Azure resource local to those site(s)
Start a final data transfer over to the target environment
Switch the origin in the Content Delivery Network (CDN) to the new environment for all hostnames coupled with the source environment

And that’s it – your sites should be running on .NET 5 at this point.

We’re still working hard to finish this tooling to allow seamless transitions to our latest software but wanted to put together this preview to convey our thought process and plans. If you have any feedback, we’re happy to receive it through our feedback portal. Contact your Customer Success Manager if you are eager to get started as we are currently running an Early Adopters program for this feature.

FAQ

I have multiple sites and wish to upgrade them one at a time, is that possible?

We are working on how to support this scenario.

Are there any DNS changes involved with this transition?

You may be asked to create some TXT records to validate the domain for the new DXP project, but the final go live event is handled through the CDN.

Is there any risk with starting a migration?

None at all – the source environments are not affected by this until the go live phase, and it is possible to abort a migration at any point.

Is a similar migration required in the future for .NET 6?

No, the new project will support both .NET 5 and 6. We also have mechanisms to detect the targeted .NET version based on the deployed source package to select appropriate version.

Search & Navigation Backend Migrations

Wed, 19 May 2021 14:13:43 GMT

Over the last couple of years, we have been hard at work upgrading the architecture and backend that power Search & Navigation (formerly known as FIND). The most recent development efforts have been on stabilizing the backend we refer to as “V3”, which is discussed in this blogpost. But as the number 3 in “V3” hints at, we have previous versions of our backend, which are still running.

Some of the improvements that we have made in V3 have trickled down into the previous versions, but not all of them. This is mainly due to other sets of constrains within the legacy platforms, such as different public cloud providers and software versions. But of course, we want all our clients and partners to be able to take advantage of our latest and greatest innovations – so over the coming weeks and months we are going to be consolidating our backends to V3.

Historically it has been quite an ordeal to migrate indexes between our backend versions as it required re-indexing of data from scratch. To make this a smoother journey, we have been creating migration tools that moves Search & Navigation indexes from one backend to another – data and all.

What does this mean for you?

Hopefully nothing. Our goal is to make this migration as friction free as possible and at the same time ensure that everything continues to work as expected. The big constrain to accomplish that is ensuring data consistency, and as such have had to make the trade-off of not accepting write operations during an index migration (much like Smooth Deploy). This will prevent indexing from happening for the time period where an index is being migrated, and you may encounter errors stating that read-only mode is activated for your index. In this case, we recommend retrying the indexing job until it goes through. Based on the volume of write operations to our clusters and time to migrate, most users will not notice this migration. There are outliers to this, and in that case, we will be in touch with you to ensure a smooth migration.

Even if we expect this migration to not affect users, we will be posting updates about this work on our status page for the sake of transparency. Since this work is expected to be going on for quite some time, we highly recommend that you subscribe to our status page to follow our progress. We also recommend that you update your client to the latest version available.

P.S. in this blogpost we also mention the next generation of FIND. Not to worry, we are still working hard on this but have taken some detours along the way. More news to come during the second half of this year – EMVPs, keep your eyes and ears open!

Best regards,

Elias Lundmark

Product Manager at Optimizely

SMTP Authentication Changes in DXP

Tue, 01 Dec 2020 16:24:27 GMT

The provider for SMTP services and transactional e-mails in DXP are making some changes around authentication methods during this quarter. The changes will move away from basic authentication with username and password, and instead use API keys for authentication.

So what does this mean for you?

If you are using the SMTP service that is a part of DXP, you will need to make some modifications to your <smtp> section in the web configuration file. Start by navigating to the management portal - within your DXP project and the “API” tab you will now find an option to generate API keys.

After generating an API key (it’s only viewable directly after creation, so save it), grab the username and hostname as well from the management portal. You’re then ready to modify configuration in your deployment packages.

<configuration>
<system.net>
<mailSettings>
<smtp from="yourdefaultreply@address.com">
<network
host="smtp.sendgrid.net"
password="[API key generated in management portal]"
userName="apikey"
port="[587, 465, 25 or 2525]" />
</smtp>
</mailSettings>
</system.net>
</configuration>

And that’s it, you’re all set to deploy to your environments in DXP. Sendgrid does have a hard deadline of January 20th where they will stop supporting basic authentication. If you are currently using basic authentication and cannot make the changes ahead of the deadline, we will run a migration close to the deadline to do this automatically and transform configuration files, but note that we will block any deployments after this migration if we notice that basic authentication is used. This is to ensure that transactional e-mails keep working as expected.

We apologize for the late heads-up for this and our aim is to make this transition as smooth as possible. Thank you for your patience and understanding.

Best regards,
Elias Lundmark
Product Manager, Cloud Services

Demystifying Edge TTL in Cloudflare

Wed, 29 Apr 2020 08:32:52 GMT

Hello World! (pun very much intended)

Long-time lurker, first time poster. I work as a Managed Services Engineer here at Episerver and a common thing we tackle is CDN optimizations in DXP projects and I thought I'd share some general information and basic things to look for when optimizing cache utilization in Cloudflare.

Out of the box, Cloudflare caches the file formats they mention in their docs here, where it acts as a cache to bring assets geographically closer to the end-user to both offload the origin web server as well as speed things up for the user. Cloudflare has around 194 Points of Presences (PoPs) around the globe.

The worst performance offenders are often images as they tend to be the largest assets so getting them as close to the user as possible makes sense, and many sites use image resizing to display one original image on multiple places in different dimensions, and that resize operation is quite resource intensive and consumes a substantial amount of CPU time. The name of the game becomes,

1. Make sure assets are properly cached in the CDN
2. Make sure assets are cached for as long as possible to keep down revalidations (where the CDN has to re-fetch the asset from the origin web server)

Figuring out what is and what is not cached is fairly straight forward because exposes the response header CF-Cache-Status to us, which should return HIT if an asset is served by the CDN cache. See their docs here for more possible responses. Note that you will get a few MISSes before a HIT as it takes some requests for cache to warm up. If you notice constant MISSes and never get a HIT in the CF-Cache-Status header, or if it returns DYNAMIC, your asset might not get cached as expected. "Dr. Flare" is a neat browser plugin to figure out what is and what is not cached and served by Cloudflare.

So how do we tell Cloudflare what to cache and set Time-To-Live (TTL) values to dictate how long an asset is cached? First off the file has to be in the supported format as noted previously and the web server has to return "public" in the Cache-Control header in order to get cached. The TTL is then dictated by max-age is the cache-control header, or it's derived from the Expires header and then displayed in the cache-control header as Max-Age=Expires-Date (seconds).

In the below example, no modifications have been done to the Episerver Alloy template and it defaults to using the Expires header with a 12-hour (43200 seconds) expiration time.

Response from Cloudflare (max-age derived from Date and Expires):

(Invoke-WebRequest "https://www.domainthatgoestocloudflare.com/contentassets/e6c47a7021e64c288fd79956fb477a50/alloymeetbanner.png").headers

Key Value
--- -----
Transfer-Encoding: chunked
Connection: keep-alive
CF-Cache-Status: HIT
Age: 3
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
CF-RAY: 58b1277f8e18caf8-ARN
cf-request-id: 02629703b50000caf851ac8200000001
Cache-Control: public, max-age=43200
Content-Type: image/png
Date: Tue, 28 Apr 2020 13:33:18 GMT
Expires: Wed, 29 Apr 2020 01:33:15 GMT
ETag: "1D5FD02B5411180"
Last-Modified: Wed, 18 Mar 2020 08:53:35 GMT
Set-Cookie: __cfduid=d846441b2dd1d7fc2d424f6dc8dfaa81b1588080798; expires=Thu, 28-May-20 13:33:18 GMT; path=/; domain=....
Server: cloudflare
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

Response from origin web server:

(Invoke-WebRequest "https://www.domainthatbypassescloudflare.com/contentassets/e6c47a7021e64c288fd79956fb477a50/alloymeetbanner.png").headers

Key Value
--- -----
Transfer-Encoding: chunked
Accept-Ranges: bytes
Cache-Control: public
Content-Type: image/png
Date: Tue, 28 Apr 2020 13:34:15 GMT
Expires: Wed, 29 Apr 2020 01:34:14 GMT
ETag: "1D5FD02B5411180"
Last-Modified: Wed, 18 Mar 2020 08:53:35 GMT
Set-Cookie: ARRAffinity=e72ac5b17c6574f3ced950f54941e20cb3d62e26a50b6ece889ededb334459e9;Path=/;HttpOnly;Domain=...
Server: Microsoft-IIS/10.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

12 hours could be sufficient, but if we have hundreds of gigabytes of data that should be revalidated every 12 hours by ~200 PoPs, it gets inefficient very quickly. Mid-tier caching allows PoPs to share data between one another to offload the origin, but 12 hours is a bit low so let's bump that up.

There's two ways of accomplishing this - either increase the time of the Expires header or remove it altogether and set a static max-age in the cache-control header instead. There is an easy way of accomplishing this with staticFile as it allows us to increase the value of Expires with just a few lines in web.config https://world.episerver.com/documentation/developer-guides/CMS/configuration/Configuring-staticFile/.

<configSections>

<section name="staticFile" type="EPiServer.Framework.Configuration.StaticFileSection, EPiServer.Framework.AspNet" allowLocation="true" />
</configSections>

...


<location path="contentAssets">
<staticFile expirationTime="30.0:0:0"/>
</location>

After applying the above configuration to web.config (and purging the cache), we can see that the Expires header is now 30 days instead of 12 hours and max-age updates accordingly from Cloudflare.

(Invoke-WebRequest "https://www.domainthatgoestocloudflare.com/contentassets/e6c47a7021e64c288fd79956fb477a50/alloymeetbanner.png").headers

Key Value
--- -----
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept
CF-Cache-Status: HIT
Age: 3
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
CF-RAY: 58b1451d1d0b75f8-ARN
cf-request-id: 0262a9862e000075f8c383a200000001
Cache-Control: public, max-age=2073597
Content-Type: image/png
Date: Tue, 28 Apr 2020 13:53:31 GMT
Expires: Fri, 22 May 2020 13:53:28 GMT
ETag: "1D5FD02B5411180"
Last-Modified: Wed, 18 Mar 2020 08:53:35 GMT
Set-Cookie: __cfduid=d0fadf111b66d8b5685ad342c62c9393b1588082011; expires=Thu, 28-May-20 13:53:31 GMT; path=/; domain=.....
Server: cloudflare
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

But of course, with great caching power comes great caching responsibility and your mileage may vary with some things.

1. As with any other caching system, we have to consider cache evictions when assets gets updated. Out of the box, IIS includes the ETags header, but there are other ways to also evict cache such as versioned URLs. See our docs on the subject here. I'd encourage anyone to play around with different solutions and see what works best for you and your editors.

2. If you use any third-party plugin, such as ImageResizer or have a custom HTTP module, you might find yourself having to modify headers through that instead. For instance, ImageResizer has its own configuration as described here.