Blog posts by Joona Immonen

Continuous delivery with the Episerver DXC Service

Thu, 04 Jan 2018 08:00:00 GMT

I have put a lot of hours into building up a continuous delivery environment which I can be proud of. I already wrote in my previous blog about how to setup a Jenkins for this purpose. It was a ground work where we start building in this blog post. This blog post is about how to do continuous Delivery with Episerver Digital Experiance Cloud Service but nothing stops you to use this information in some other place.

Continuous delivery (CD) and continuous integration (CI)

For sure there are better references about CI and CD than following will be. Actually, if you have hard time in convincing your product owner to continuous delivery we have a blog for that purpose as well. For me continuous integration means merging work in git repository hourly with coworkers working on the same things. On the other hand continuous delivery means pushing the work to the customer as fast as we can. Developers work on the codebase and then deploy it once they are confident with it. When you speed up the cycle of your deployments, your customer will have new features faster, and you they can steer the direction of your development in a truly agile way. Characteristics of CI and CD are below.

Continuous integration:

Integrate early and often
Make the code available to other developers
Make sure the code builds after merging to the works of other people
Track the quality of your code
Test your code

Continuous delivery:

Push changes to production early and often
Make the code available to your end-users
Make sure that you can deploy at any time
Track the quality of your runtime
Test your deployments with automated release process

I feel that continuous integration is something that software business is doing already with high adaptation rate. Continuous delivery in the other hand is something that many are still learning. Continuous delivery definitely needs more maturity from the development team since you need to automate a lot of things to be able to deploy automatically. You will need a process that makes all kind of sanity checks across the line to trust the deployments on the hand of the automated system. The good thing is once you get on the road it will be a lot easier to adjust the process when you have ability to repeat it multiple times a day.

The hardest question asked about continuous delivery is often: “How can your customer decide when a feature is good enough for release?”. This is something that you need to decide per feature. With Episerver creating new content features is easy. Customer tests the new content types and decides when they serve them well enough to be published. Changing existing content can be done with feature toggles. Of course creating a feature toggle is a change too and can be misimplemented. From this emerges the need for automatic testing. You need to catch errors before they get on the production. The hardest kind of changes are cross-cutting concerns like authentication or database schema changes. You need to be super careful in how do you implement and roll these kind of features to the production. In the end the most important thing is that your development organization (developers and business people) have the correct working habits and the mindset for failing fast and failing often. Some day you will fail and with CD you will be most likely to recover faster.

Having said scary things about deploying features there is the positive side of the coin. When you deploy often you have only a small changeset at the time. It is a lot easier to find the problems on your patch and fix them. Installing the fix is also easy when you have process to take care of all and when manual work is not involved.

the DXC Service

In case you don’t know what the DXC Service is I will describe it briefly. The DXC Service is a managed platform from Episerver to host Episerver CMS in the Azure cloud. Instead of working directly with the Azure platform you will get the services the Episerver thinks that you need. In case that you wish to have more services. Well tough luck. You will need to setup an additional Azure subscription because you won’t be adding them to the DXC Service subscription. Actually you won’t be able to see your production stuff in Azure portal at all. The DXC Service comes with three environments: integration, preproduction and production. The architecture of the DXC service is as following:

The architecture picture was taken directly from their service description. What I would like to highlight here is that they have CDN, they are using Azure Web apps, Azure Storage, Azure Service Bus, Azure SQL and a smtp service. This is a pretty recommended setup when building web applications to Azure.

The DXC Service architecture is actually pretty nice as it is. If you are building up simple CMS solution you won’t be needing much else to be honest. In case you are building extranet, intranet or commerce site then you might end up in a different situation. The main problem is that you won’t be able to add services like Azure Functions, Azure AD, Azure B2C AD or even use the full potential of underlying infrastructure since WebJobs are not supported. You will also have limited visibility on the services you have like Azure Service Bus, Azure Storage and Azure SQL. Episerver does not provide you the technological details about the services you are running but you can of course implement functionality to your production site that tells you how they have been configured (of course there will be security issues with this). Things that you will do with this information is of course in the danger zone. I have been doublechecking what I can do and what I can’t do constantly from Episerver. If you are going to implement a complex project on top of the Episerver DXC Service then you should plan on having an Azure Subscription for deploying all the additional services you might need.

You might be able to read between the lines that I’m not perfectly happy about lost control on the Azure services in the DXC Service. Another thing I’m not perfectly satisfied about is the deployment process. Episerver has decided to reinvent the wheel and implement their own deployment pipeline for the DXC Service. You can do deployments with WebDeploy only to integration environment. From that point onwards you will need to use either their own portal or make service requests (via phone or email) to get deployment into preproduction and finally you need a service request to get the software into the production. Effectively this means that you will have hard times to control the exact moment of a deployment for preproduction and production via continuous delivery. To preproduction environment you will be able to do manual deployments via their own portal and they have been promising to open that for production too. Unfortunately there is no API for that or no supported devops way whatsoever. This means that you either fall back to emailing them or make some serious browser scripting to bypass these problems. We went to the email way because I think that it pushes Episerver towards finding the right solutions for this problem. The problem with email deployment is 24h SLA for registering a ticket (Episerver does not do this automatically) and 24h SLA for fulfilling request which sums up to total of 48h to get the deployment done once email is sent. Which effectively would mean that we need to ask the deployment 48h before if we want it to take place at the exact moment of time. Here is the The Episerver DXC Service description vision about the deployments.

I guess the noteworthy thing is that code swims towards production and content swims towards development. Content arrow towards production means once only in the case of the first deployment. This means that you can replenish your integration and preproduction environment data with the production data. If you wish to do so you need to implement your solution so that you can recover integration and preproduction sites to the correct site settings etc. Episerver won’t do that for you. One another thing to notice is all the stored data from the Episerver Forms and other personal data storages. I would highly encourage to delete completely or scramble that data on the replenish process.

Our team and the process

We are working on a project that has multiple CMS sites on the same DXC Service platform. It is not a simple project and our developer team is somewhat big for an Episerver project. Currently we have seven developers working on the project most of them being full-time and few doing split with some other project. This means that in a busy day there will be a lot of stuff going on on our Git repository. Our working habits wary on the problem at hand. If we are doing straightward and simple things just do it and push it. If we are doing tricky things then creating a branch and having other developer reviewing your work is advised. That is enough background information. Now we should focus on the process that starts on the git push to the master branch. Here is an overall picture of our Jenkins build pipeline.

Jenkins build pipeline has been separated to few stages that somehow represents where things are happening. For most of the stuff those are self-explanatatory but there are few scheduled tasks that are more or less just thrown somewhere. These are long-running or heavy tasks that needs time more than we are willing to give for each deployment. For example ZAP scans can take hours and we want to publish fast and often. So instead of scanning every build we are doing daily scanning. Here is few explanations what is happening.

Build phase

Build trigger task is triggered by git commit
Build tasks builds the software and makes a release to Octopus Deploy
Unit-tests are run after the build and making the release
OWASP-Dependenchy-Check scans nightly if we have any known vulnerabilities in our binaries
Sonar-Tests makes nightly a static code analysis of our current codebase

Dev phase

Deploy makes a deployment with Octopus deploy to our internal test server once unit tests are passed
Smoke-tests are run to see if the site is fine

Test phase

Deploy makes a deployment with Octopus deploy to the DXC Service integration environment once the Dev smoke-tests have passed
Smoke-tests are ran to see if the site is fine
Performance-Tests are run nightly to track on the performance of the website

Staging phase

Deploy sends an email to the Episerver support for an deployment once the Test smoke-tests have passed
Smoke-Tests-Polling-Trigger constantly polls if the version of the site has changed
Smoke-tests are ran whenever polling trigger notices a change in the version of the site
ZAP-scans are run nightly to see if there are any low-hanging security enhancements to be made

Prod phase

Deploy sends an email to the Episerver support for an deployment once per day or in demand
Smoke-Tests-Polling-Trigger constantly polls if the version of the site has changed
Smoke-tests are ran whenever polling trigger notices a change in the version of the site
Data to staging and data to test sends an email to the Episerver support to refresh data of integration and preproduction environment weekly

So from quality perspective to get the software into production it has been surely passed unit-tests and smoke-tests. In addition we have SonarQube, OWASP Dependency Check, OWASP ZAP and jMeter gathering information that needs to be digested. On top of that we of course follow what is happening in the Episerver and in the .NET versions. Here is some kind of picture about different kind of information sources that we are leveraging when building towards top quality.

Building the build pipeline

After the long introductionary presentation of deployment philosophy we finally get the point were I can show some code. We are using the Jenkins Job DSL plugin in building our build pipeline. This means that we have a second repository for infrastructure where we have a groovy script for the Jenkins pipeline. Then we have one job in the Jenkins that builds up the other Jenkins jobs with the groovy script. So let’s see how our phases are from that perspective. Starting from easy side this is how the build trigger looks like.

Build trigger

job(applicationName + ' ' + buildEnvironmentName + ' Trigger') {
	deliveryPipelineConfiguration(buildEnvironmentName, "Build Trigger")
	wrappers {
        deliveryPipelineVersion('\$GIT_REVISION', true)
    }
	scm {
        git {
            remote {
                url(gitRepository)
				credentials(gitCredentialsId)
            }
            branch(branchName)
        }
    }
	triggers {
        scm('* * * * *')
    }
	quietPeriod(5)
	publishers {
		downstream(applicationName + ' ' + buildEnvironmentName, 'SUCCESS')
    }
}

So the job has cron scheduler that constantly polls if there are changes on the git repository and triggers a downstream job if there are. There are a lot of variables used to make it easier to change the naming practices of the whole pipeline. From this point onward I will present only parts of the job DSL to reduce the amount of noise.

Build and octopus release

configure { proj ->
	def builders = proj / builders
	builders << 'hudson.plugins.msbuild.MsBuildBuilder' {
			msBuildName(msBuildInstallationName)
			msBuildFile(buildFile)
			cmdLineArgs('/t:Build /p:RunOctoPack=true /p:OctoPackPackageVersion=1.0.${BUILD_NUMBER} /p:OctoPackPublishPackageToHttp='+oServerUrl+'nuget/packages /p:CmdLineInMemoryStorage=true /p:Configuration=Release /p:OctoPackPublishApiKey=${OCTOPUS_API_KEY} /p:DeployOnBuild=false')
			buildVariablesAsProperties('false')
	}
	def pubs = proj / publishers
	pubs << 'hudson.plugins.octopusdeploy.OctopusDeployReleaseRecorder' {
		serverId(oServerId)
		project(oProject)
		environment('')
		tenant('')
		waitForDeployment('false')
		releaseVersion('1.0.\${BUILD_NUMBER}')
		releaseNotes('true')
		releaseNotesSource('scm')
		channel(oChannel)
		releaseNotesJenkinsLinkback('true')
		deployThisRelease('false')
		packageConfigs { 
			'hudson.plugins.octopusdeploy.PackageConfiguration' {
				packageName(oPackage)
				packageVersion('1.0.\${BUILD_NUMBER}')
			}
		}
		defaultPackageVersion('1.0.\${BUILD_NUMBER}')
	}
}

In case you wondered why I made release before running unit tests. The answer is that I like to have unit-tests and build separated in the pipeline. Still I want to be able to easily track octopus deploy build version in Jenkins and the easiest way is to use build number as a version number. This way every build will have a unique number and I can find out which Octopus release is which Jenkins build. In addition to all this at this point we will do few funny things. We package the git commit hash and the build number into the folder structure and archive the application. Archiving means that later on other Jenkins jobs can access this job from downstream and use the same binaries.

steps {
	powerShell('\$path = "'+buildVersionPath+'"'+
		System.getProperty("line.separator")+
		'If(!(test-path \$path)){ New-Item -ItemType Directory -Force -Path \$path }'+
		System.getProperty("line.separator")+
		'git rev-parse --verify HEAD | Set-Content "\$path/ver.txt"'+
		System.getProperty("line.separator")+
		'\$env:BUILD_NUMBER | Set-Content "\$path/rel.txt"')
}

We use the groovy script to generate a PowerShell script that is able to parse the information that we need and put it on to the folder structure. Finally we archive the whole thing.

publishers {
		archiveArtifacts {
			pattern("**/*")
			exclude("**/.git/**,**/static/node_modules/**,**/static/src/**,**/static/styleguide/**")
			onlyIfSuccessful()
		}
	}

There some excludes added to archiving for getting better IO out of the build server. Without those excludes every build step that uses this archive seemed to last 20-30 minutes longer since we have a lot of node modules and our repository is becoming a rather large one.

Unit tests

steps {
	copyArtifacts(applicationName + ' Build') {
		buildSelector {
			upstreamBuild {
				allowUpstreamDependencies(false)
				fallbackToLastSuccessful(true)
			}
		}
		includePatterns('**/*')
	}
}
configure { project -> 
	def mstest = project / builders / 'org.jenkinsci.plugins.MsTestBuilder' {
		msTestName(msTestVersion)
		testFiles(applicationPrefix+testDllPostFix+'\\bin\\Release\\'+applicationPrefix+testDllPostFix+'.dll')
		resultFile(testFile)
		continueOnFail('false')
		categories('')
		cmdLineArgs('')
	}
	def mstestPublish = project / publishers / 'hudson.plugins.mstest.MSTestPublisher' {
		testResultsFile(testFile)
	}
}

Running unit tests is not especially interesting. First we copy the archived information from the upstream job. Then we just run the tests and make sure that there alerts about failure. We have found having chatbots the most convenient solution for the team.

Deployment with Octopus Deploy

steps {
	powerShell('$ErrorActionPreference = "Stop"'+
		System.getProperty("line.separator")+
		'\$path = "'+buildVerPath+'"'+
		System.getProperty("line.separator")+
		'\$releaseNumber = Get-Content "\$path/rel.txt"'+
		System.getProperty("line.separator")+
		'octo deploy-release --project '+oProject+' --channel '+oChannel+' --releaseNumber 1.0.\$releaseNumber --deployto '+oEnvironment+' --server '+oServerUrl+' --progress --waitfordeployment --apiKey \$Env:OCTOPUS_API_KEY')
}

From the Jenkins perspective the deployment process is quite easy. Just make a call for octo.exe. Of course you need to setup your deployment processes into the Octopus Deploy. For us they are rather simple. For our internal testing environment (we are calling it CI) we use octopus deploy agent and do just some configuration transformation beside normal WebDeploy. For the DXC Service integration environment we deploy to a deployment slot. Clean up few files and finally swap slots. The files we need to clean up are the configuration transformation files for the CI and postdeploy scripts. They are not cleaned up automatically with the Azure Web App as they are with the on-premise IIS server.

The nice thing our setup is the build promotion. Even though we are rebuilding software for unit-tests and smoke-tests we are building only once for deployments. Binary compilation and configuration are separated from each other which means that we can trust that our binaries are the same in each environment and they won’t change because of time relative issues like some 3rd party dependency not being fetched or somebody patching server. Even when we are doing deployments from the Jenkins we can still see and do build promotions from the Octopus Deploy too.

Smoke testing

Smoke testing is done with the same kind of DSL as the unit testing as we are using unit tests for smoke testing too. We just have two separated projects in our solution for different kind of tests. Smoke tests are tests that needs something to be deployed before we can test. We have few basic tests that we do.

Urls in the sitemap returns 200
Different archetypes of pages render with JS ok
Robots.txt is found

We could see that all the pages would render with JS but it would take a really long time. The only point where smoke testing differs from the unit tests is that since Episerver does not always startup so fast we have a wait routine for waiting the server to respond with reasonable http statuscodes after full startup.

def ls = System.getProperty("line.separator")
steps {
	powerShell('\$request = [system.Net.WebRequest]::Create("'+targetAddress+'")'+ls+
		'$result = ""'+ls+
		'for (\$i = 1; \$i -le 100; \$i++) {'+ls+
		'  try {'+ls+
		'    \$result = \$request.GetResponse()'+ls+
		'    if (\$result -is "System.Net.HttpWebResponse" -and \$result.StatusCode -ne "") {'+ls+
		'      break'+ls+
		'    }'+ls+
		'  } catch [System.Net.WebException] {'+ls+
		'    \$result = \$_.Exception.Response '+ls+
		'  }'+ls+
		'  Start-Sleep -s 2'+ls+
		'}'+ls+
		'Write-Host "Status was: \$(\$result.StatusCode)"')
}

Email deployments

def emailSubject = 'Pre-production deployment \$PIPELINE_VERSION'
def emailContent = 'Schedule: ASAP'+
	'\nPriority: Medium'+
	'\nPipeline revision: \$PIPELINE_VERSION'+
	'\nFrom: Integration'+
	'\nTo: Pre-production'+
	'\nDeploy: '+
	'\n\tCode: Yes'+ 
	'\n\tDatabase: No'+
	'\n\tBlobs: No'+ 
	'\n\tDeployment verification url:'+
	'\n\t\thttps://www.google.fi?q=figure+out'+
	'\n\nBest Regards: Jenkins'
publishers {
	extendedEmail {
		recipientList(emailRecipientList)
		replyToList(emailReplyTo)
		defaultSubject(emailSubject)
		defaultContent(emailContent)
		contentType('text/plain')
		preSendScript('$DEFAULT_PRESEND_SCRIPT\nmsg.addHeader("X-Priority", "1 (Highest)");\nmsg.addHeader("Importance", "High");\nmsg.setFrom(new javax.mail.internet.InternetAddress("'+emailSender+'"))')
		triggers {
			beforeBuild()
		}
	}
}

This is pretty close that we got into. We use some Java inside groovy script to setup email being sent as a high priority and to rewrite the sender to be something that we actually want to get responses to. We also have a verification url packaged into emails that we wish the Episerver Support to use to see if the software is fine and rollback if the page does not render correctly. We have same kind of emails for the data replenish too. Instead code we just say deploy data.

Triggering smoke tests

def persistentHash = './persistenthash.txt'
def tempHash = './temphash.txt'
def testFile = './test_if_this_exists.txt'

steps {
	powerShell('if(!(test-path "'+persistentHash+'")) { "init_first_non_existent_hash" | Set-Content "'+persistentHash+'" }'+
		System.getProperty("line.separator")+
		'rm -force "'+testFile+'" -ErrorAction SilentlyContinue'+
		System.getProperty("line.separator")+
		'rm -force "'+tempHash+'" -ErrorAction SilentlyContinue'+
		System.getProperty("line.separator")+
		'curl https://www.google.fi?q=ver.txt -UseBasicParsing | % { $_.Content | Set-Content "'+tempHash+'" }'+
		System.getProperty("line.separator")+
		'if((Get-FileHash "'+persistentHash+'").hash -ne (Get-FileHash "'+tempHash+'").hash) { "true" | Set-Content "'+testFile+'" }'
		)
	conditionalSteps {
		condition {
			fileExists(testFile, BaseDir.WORKSPACE)
		}
		runner('DontRun')
		steps {
			powerShell('curl https://www.google.fi?q=ver.txt -UseBasicParsing | % { $_.Content | Set-Content "'+persistentHash+'" }')
			downstreamParameterized {
				trigger(applicationName + ' ' + stagingEnvironmentName + ' Smoke-Tests')
			}
		}
	}
}

The idea of this PowerShell script is to store a commit hash into our jobs working folder. Then see on every 5 minutes if the hash we have is matching with the one that is on the production server (yeah, I replaced the actual url with google urls). If it does match then nothing happens. If it does not then we create a temporary file. Then we have a conditional step that from Jenkins which checks if the temporary file is present or not. If there is a file it updates the hash in working folder and triggers smoke tests for corresponding environment. This way our smoke tests start running within 5 minutes after the Episerver support has done the deployment.

The nightly stuff

Here is how I run the OWASP ZAP from Jenkins via PowerShell. I use my own PowerShell modules for managing ZAP. There are Jenkins plugins too but I had my own version before they existed so I’m kind of stuck on my own baby. I tried the plugin but it just didn’t do the same thing so I left it and went back to my own scripts.

def lineSeparator = System.getProperty("line.separator")
def zapReport = '(\$env:WORKSPACE+"\\'+zapReportFile+'")'
parentJob.steps {
	powerShell('Import-Module C:\\tools\\powershell-zap-master\\PowerShell-ZAP\\PowerShell-ZAP.psm1'+lineSeparator+
		'Set-ZapLocation "C:\\Program Files\\OWASP\\Zed Attack Proxy\\"'+lineSeparator+
		'Set-ZapReportLocation '+zapReport+lineSeparator+
		'Set-ZapUrlToScan "'+zapUrlToScan+'"'+lineSeparator+
		'# Ensure that daemon is running'+lineSeparator+
		'Start-Zap'+lineSeparator+
		'# Configure policies, this just enables all scanners atm'+lineSeparator+
		'Set-ZapScanPolicies'+lineSeparator+
		'# Do spidering against the url'+lineSeparator+
		'Invoke-ZapSpidering '+lineSeparator+
		'# Do ajax spidering against the url'+lineSeparator+
		'Invoke-ZapAjaxSpidering'+lineSeparator+
		'# Do scanning against the url'+lineSeparator+
		'Invoke-ZapScanning'+lineSeparator+
		'# Save report'+lineSeparator+
		'Save-ZapReport'+lineSeparator+
		'# Destroy scans'+lineSeparator+
		'Remove-ZapCurrentSpider'+lineSeparator+
		'Remove-ZapCurrentScan')
}
parentJob.publishers {
	archiveJunit(zapReportFile) {
		allowEmptyResults()
		retainLongStdout()
		testDataPublishers {
			publishTestStabilityData()
		}
	}
}

For performance monitoring we use my coworkers jmeter-perfotrator which is just a template for jMeter. From the Jenkins the usage is quite simple.

steps {
	 batchFile('call '+jmeterPath+' -n -t "%WORKSPACE%\\Solution Items\\jmeter\\TestUrls.jmx" -j "%WORKSPACE%\\log.txt"')
}
configure { project -> 
		def perfPublish = project / builders / 'hudson.plugins.performance.PerformancePublisher' 
		(perfPublish / 'parsers' / 'hudson.plugins.performance.JMeterParser' / 'glob').value = 'results\\results.jtl'
}

For the dependency checks we use a plugin. In addition you need to setup a job to update the dependency data. After you have the fresh dependency data then you can run the check with DSL like this:

configure { project -> 
	def owaspCheck = project / builders / 'org.jenkinsci.plugins.DependencyCheck.DependencyCheckBuilder' {
		skipOnScmChange(false)
		skipOnUpstreamChange(false)
		scanpath("")
		outdir("")
		datadir("C:\\OWASP_DEPENDENCY_DATA")
		suppressionFile("")
		isAutoupdateDisabled(true)
		includeHtmlReports(true)
	}
	def owaspPublish = project / publishers / 'org.jenkinsci.plugins.DependencyCheck.DependencyCheckPublisher' {
		healthy("")
		unHealthy("")
		thresholdLimit("low")
		pluginName("[DependencyCheck] ")
		defaultEncoding("")
		canRunOnFailed("")
		usePreviousBuildAsReference("true")
		useStableBuildAsReference("false")
		useDeltaValues("true")
		shouldDetectModules("true")
		dontComputeNew("true")
		doNotResolveRelativePaths("false")
		pattern("")
	}
}

For the static code analysis we use the SonarQube. It is fairly easy to setup with the Jenkins. Just wrap your build between the SonarQube begin and end analysis.

// Configure begin analysis
configure { project -> 
		def sonarBegin = project / builders / 'hudson.plugins.sonar.MsBuildSQRunnerBegin' {
			projectKey(sonarProjectKey)
			projectName(sonarProjectName)
			projectVersion(sonarProjectVersion)
			msBuildScannerInstallationName(sonarScannerName)
		}
}
// ... Your build goes here
// Configure end analysis
configure { project -> 
		def sonarEnd = project / builders / 'hudson.plugins.sonar.MsBuildSQRunnerEnd' 
}

Show it on the TV

Finally one of the biggest strenghts of this setup is the ability to show it on the screen with relatively simple DSL (and of course with some Jenkins plugins).

deliveryPipelineView(applicationName + ' delivery Pipeline') {
	pipelineInstances(10)
	showAvatars()
    showChangeLog()
	showTotalBuildTime()
    enableManualTriggers(true)
	allowRebuild()
    showAggregatedPipeline(false)
    pipelines() {
        component(applicationName, applicationName + ' ' + buildEnvironmentName + ' Trigger')
    }
}

buildMonitorView(applicationName + ' build Monitor') {
	description(applicationName+' jobs')
	jobs {
        name(applicationName)
		regex('.*'+applicationName+'.*')
	}
}

There are two views. The pipeline view shows the view that was in the beginning of the blog post where you could see different phases of the continuous delivery process. The build monitor view builds up a screen full of rectangles that are green / yellow / red depending on your jobs status.

We are asking the production deployment in the early morning hours so it usually takes place just before we got to the work. It is kind of peaceful time to do deployments and we have few earlybirds that will notice if something strange is going on when they get to work.

Summarizing

Some characteristics about our project:

More than 100 deployments per month
Go live every workday morning (~20 times a month)
Half a year of live site hosting

We have had a few issues but nothing that would have upset our customer badly. Most of the times if there is an issue our smoke-tests notices it earlier than the Episerver Support does even when they are the ones who are deploying it. Usually the problems have been on the configuration transformation that is done by the Episerver Support with a configuration that we are not aware of. Once or twice we have been forced to make a fix patch immediately after going live. On these rare occasions it takes usually less time to implement the patch than to get episerver installing it. They tell us to call them if we have problem but it is really hard to call and solve the problem at the same time. If we call them, then we need double the people to figure out the situation since the telephone consumes one guy.

In the DXC environment the problem remains that after a failed deployment it takes too much time to get the fixing deployment on the way. Immediate rollbacks do help, but not if we need to wait for a hour or two for that to happen. We have a fix ready on that time and then we would rather go forward-only deployment with the fixed version.

Continuous delivery with the Episerver DXC Service

Thu, 04 Jan 2018 07:00:00 GMT

Continuous delivery (CD) and continuous integration (CI)

Continuous integration:

Integrate early and often
Make the code available to other developers
Make sure the code builds after merging to the works of other people
Track the quality of your code
Test your code

Continuous delivery:

Push changes to production early and often
Make the code available to your end-users
Make sure that you can deploy at any time
Track the quality of your runtime
Test your deployments with automated release process

the DXC Service

Our team and the process

Build phase

Build trigger task is triggered by git commit
Build tasks builds the software and makes a release to Octopus Deploy
Unit-tests are run after the build and making the release
OWASP-Dependenchy-Check scans nightly if we have any known vulnerabilities in our binaries
Sonar-Tests makes nightly a static code analysis of our current codebase

Dev phase

Deploy makes a deployment with Octopus deploy to our internal test server once unit tests are passed
Smoke-tests are run to see if the site is fine

Test phase

Deploy makes a deployment with Octopus deploy to the DXC Service integration environment once the Dev smoke-tests have passed
Smoke-tests are ran to see if the site is fine
Performance-Tests are run nightly to track on the performance of the website

Staging phase

Deploy sends an email to the Episerver support for an deployment once the Test smoke-tests have passed
Smoke-Tests-Polling-Trigger constantly polls if the version of the site has changed
Smoke-tests are ran whenever polling trigger notices a change in the version of the site
ZAP-scans are run nightly to see if there are any low-hanging security enhancements to be made

Prod phase

Deploy sends an email to the Episerver support for an deployment once per day or in demand
Smoke-Tests-Polling-Trigger constantly polls if the version of the site has changed
Smoke-tests are ran whenever polling trigger notices a change in the version of the site
Data to staging and data to test sends an email to the Episerver support to refresh data of integration and preproduction environment weekly

Building the build pipeline

Build trigger

job(applicationName + ' ' + buildEnvironmentName + ' Trigger') {
	deliveryPipelineConfiguration(buildEnvironmentName, "Build Trigger")
	wrappers {
        deliveryPipelineVersion('\$GIT_REVISION', true)
    }
	scm {
        git {
            remote {
                url(gitRepository)
				credentials(gitCredentialsId)
            }
            branch(branchName)
        }
    }
	triggers {
        scm('* * * * *')
    }
	quietPeriod(5)
	publishers {
		downstream(applicationName + ' ' + buildEnvironmentName, 'SUCCESS')
    }
}

Build and octopus release

configure { proj ->
	def builders = proj / builders
	builders << 'hudson.plugins.msbuild.MsBuildBuilder' {
			msBuildName(msBuildInstallationName)
			msBuildFile(buildFile)
			cmdLineArgs('/t:Build /p:RunOctoPack=true /p:OctoPackPackageVersion=1.0.${BUILD_NUMBER} /p:OctoPackPublishPackageToHttp='+oServerUrl+'nuget/packages /p:CmdLineInMemoryStorage=true /p:Configuration=Release /p:OctoPackPublishApiKey=${OCTOPUS_API_KEY} /p:DeployOnBuild=false')
			buildVariablesAsProperties('false')
	}
	def pubs = proj / publishers
	pubs << 'hudson.plugins.octopusdeploy.OctopusDeployReleaseRecorder' {
		serverId(oServerId)
		project(oProject)
		environment('')
		tenant('')
		waitForDeployment('false')
		releaseVersion('1.0.\${BUILD_NUMBER}')
		releaseNotes('true')
		releaseNotesSource('scm')
		channel(oChannel)
		releaseNotesJenkinsLinkback('true')
		deployThisRelease('false')
		packageConfigs { 
			'hudson.plugins.octopusdeploy.PackageConfiguration' {
				packageName(oPackage)
				packageVersion('1.0.\${BUILD_NUMBER}')
			}
		}
		defaultPackageVersion('1.0.\${BUILD_NUMBER}')
	}
}

steps {
	powerShell('\$path = "'+buildVersionPath+'"'+
		System.getProperty("line.separator")+
		'If(!(test-path \$path)){ New-Item -ItemType Directory -Force -Path \$path }'+
		System.getProperty("line.separator")+
		'git rev-parse --verify HEAD | Set-Content "\$path/ver.txt"'+
		System.getProperty("line.separator")+
		'\$env:BUILD_NUMBER | Set-Content "\$path/rel.txt"')
}

We use the groovy script to generate a PowerShell script that is able to parse the information that we need and put it on to the folder structure. Finally we archive the whole thing.

publishers {
		archiveArtifacts {
			pattern("**/*")
			exclude("**/.git/**,**/static/node_modules/**,**/static/src/**,**/static/styleguide/**")
			onlyIfSuccessful()
		}
	}

Unit tests

steps {
	copyArtifacts(applicationName + ' Build') {
		buildSelector {
			upstreamBuild {
				allowUpstreamDependencies(false)
				fallbackToLastSuccessful(true)
			}
		}
		includePatterns('**/*')
	}
}
configure { project -> 
	def mstest = project / builders / 'org.jenkinsci.plugins.MsTestBuilder' {
		msTestName(msTestVersion)
		testFiles(applicationPrefix+testDllPostFix+'\\bin\\Release\\'+applicationPrefix+testDllPostFix+'.dll')
		resultFile(testFile)
		continueOnFail('false')
		categories('')
		cmdLineArgs('')
	}
	def mstestPublish = project / publishers / 'hudson.plugins.mstest.MSTestPublisher' {
		testResultsFile(testFile)
	}
}

Deployment with Octopus Deploy

steps {
	powerShell('$ErrorActionPreference = "Stop"'+
		System.getProperty("line.separator")+
		'\$path = "'+buildVerPath+'"'+
		System.getProperty("line.separator")+
		'\$releaseNumber = Get-Content "\$path/rel.txt"'+
		System.getProperty("line.separator")+
		'octo deploy-release --project '+oProject+' --channel '+oChannel+' --releaseNumber 1.0.\$releaseNumber --deployto '+oEnvironment+' --server '+oServerUrl+' --progress --waitfordeployment --apiKey \$Env:OCTOPUS_API_KEY')
}

Smoke testing

Urls in the sitemap returns 200
Different archetypes of pages render with JS ok
Robots.txt is found

def ls = System.getProperty("line.separator")
steps {
	powerShell('\$request = [system.Net.WebRequest]::Create("'+targetAddress+'")'+ls+
		'$result = ""'+ls+
		'for (\$i = 1; \$i -le 100; \$i++) {'+ls+
		'  try {'+ls+
		'    \$result = \$request.GetResponse()'+ls+
		'    if (\$result -is "System.Net.HttpWebResponse" -and \$result.StatusCode -ne "") {'+ls+
		'      break'+ls+
		'    }'+ls+
		'  } catch [System.Net.WebException] {'+ls+
		'    \$result = \$_.Exception.Response '+ls+
		'  }'+ls+
		'  Start-Sleep -s 2'+ls+
		'}'+ls+
		'Write-Host "Status was: \$(\$result.StatusCode)"')
}

Email deployments

def emailSubject = 'Pre-production deployment \$PIPELINE_VERSION'
def emailContent = 'Schedule: ASAP'+
	'\nPriority: Medium'+
	'\nPipeline revision: \$PIPELINE_VERSION'+
	'\nFrom: Integration'+
	'\nTo: Pre-production'+
	'\nDeploy: '+
	'\n\tCode: Yes'+ 
	'\n\tDatabase: No'+
	'\n\tBlobs: No'+ 
	'\n\tDeployment verification url:'+
	'\n\t\thttps://www.google.fi?q=figure+out'+
	'\n\nBest Regards: Jenkins'
publishers {
	extendedEmail {
		recipientList(emailRecipientList)
		replyToList(emailReplyTo)
		defaultSubject(emailSubject)
		defaultContent(emailContent)
		contentType('text/plain')
		preSendScript('$DEFAULT_PRESEND_SCRIPT\nmsg.addHeader("X-Priority", "1 (Highest)");\nmsg.addHeader("Importance", "High");\nmsg.setFrom(new javax.mail.internet.InternetAddress("'+emailSender+'"))')
		triggers {
			beforeBuild()
		}
	}
}

Triggering smoke tests

def persistentHash = './persistenthash.txt'
def tempHash = './temphash.txt'
def testFile = './test_if_this_exists.txt'

steps {
	powerShell('if(!(test-path "'+persistentHash+'")) { "init_first_non_existent_hash" | Set-Content "'+persistentHash+'" }'+
		System.getProperty("line.separator")+
		'rm -force "'+testFile+'" -ErrorAction SilentlyContinue'+
		System.getProperty("line.separator")+
		'rm -force "'+tempHash+'" -ErrorAction SilentlyContinue'+
		System.getProperty("line.separator")+
		'curl https://www.google.fi?q=ver.txt -UseBasicParsing | % { $_.Content | Set-Content "'+tempHash+'" }'+
		System.getProperty("line.separator")+
		'if((Get-FileHash "'+persistentHash+'").hash -ne (Get-FileHash "'+tempHash+'").hash) { "true" | Set-Content "'+testFile+'" }'
		)
	conditionalSteps {
		condition {
			fileExists(testFile, BaseDir.WORKSPACE)
		}
		runner('DontRun')
		steps {
			powerShell('curl https://www.google.fi?q=ver.txt -UseBasicParsing | % { $_.Content | Set-Content "'+persistentHash+'" }')
			downstreamParameterized {
				trigger(applicationName + ' ' + stagingEnvironmentName + ' Smoke-Tests')
			}
		}
	}
}

The nightly stuff

def lineSeparator = System.getProperty("line.separator")
def zapReport = '(\$env:WORKSPACE+"\\'+zapReportFile+'")'
parentJob.steps {
	powerShell('Import-Module C:\\tools\\powershell-zap-master\\PowerShell-ZAP\\PowerShell-ZAP.psm1'+lineSeparator+
		'Set-ZapLocation "C:\\Program Files\\OWASP\\Zed Attack Proxy\\"'+lineSeparator+
		'Set-ZapReportLocation '+zapReport+lineSeparator+
		'Set-ZapUrlToScan "'+zapUrlToScan+'"'+lineSeparator+
		'# Ensure that daemon is running'+lineSeparator+
		'Start-Zap'+lineSeparator+
		'# Configure policies, this just enables all scanners atm'+lineSeparator+
		'Set-ZapScanPolicies'+lineSeparator+
		'# Do spidering against the url'+lineSeparator+
		'Invoke-ZapSpidering '+lineSeparator+
		'# Do ajax spidering against the url'+lineSeparator+
		'Invoke-ZapAjaxSpidering'+lineSeparator+
		'# Do scanning against the url'+lineSeparator+
		'Invoke-ZapScanning'+lineSeparator+
		'# Save report'+lineSeparator+
		'Save-ZapReport'+lineSeparator+
		'# Destroy scans'+lineSeparator+
		'Remove-ZapCurrentSpider'+lineSeparator+
		'Remove-ZapCurrentScan')
}
parentJob.publishers {
	archiveJunit(zapReportFile) {
		allowEmptyResults()
		retainLongStdout()
		testDataPublishers {
			publishTestStabilityData()
		}
	}
}

For performance monitoring we use my coworkers jmeter-perfotrator which is just a template for jMeter. From the Jenkins the usage is quite simple.

steps {
	 batchFile('call '+jmeterPath+' -n -t "%WORKSPACE%\\Solution Items\\jmeter\\TestUrls.jmx" -j "%WORKSPACE%\\log.txt"')
}
configure { project -> 
		def perfPublish = project / builders / 'hudson.plugins.performance.PerformancePublisher' 
		(perfPublish / 'parsers' / 'hudson.plugins.performance.JMeterParser' / 'glob').value = 'results\\results.jtl'
}

For the dependency checks we use a plugin. In addition you need to setup a job to update the dependency data. After you have the fresh dependency data then you can run the check with DSL like this:

configure { project -> 
	def owaspCheck = project / builders / 'org.jenkinsci.plugins.DependencyCheck.DependencyCheckBuilder' {
		skipOnScmChange(false)
		skipOnUpstreamChange(false)
		scanpath("")
		outdir("")
		datadir("C:\\OWASP_DEPENDENCY_DATA")
		suppressionFile("")
		isAutoupdateDisabled(true)
		includeHtmlReports(true)
	}
	def owaspPublish = project / publishers / 'org.jenkinsci.plugins.DependencyCheck.DependencyCheckPublisher' {
		healthy("")
		unHealthy("")
		thresholdLimit("low")
		pluginName("[DependencyCheck] ")
		defaultEncoding("")
		canRunOnFailed("")
		usePreviousBuildAsReference("true")
		useStableBuildAsReference("false")
		useDeltaValues("true")
		shouldDetectModules("true")
		dontComputeNew("true")
		doNotResolveRelativePaths("false")
		pattern("")
	}
}

For the static code analysis we use the SonarQube. It is fairly easy to setup with the Jenkins. Just wrap your build between the SonarQube begin and end analysis.

// Configure begin analysis
configure { project -> 
		def sonarBegin = project / builders / 'hudson.plugins.sonar.MsBuildSQRunnerBegin' {
			projectKey(sonarProjectKey)
			projectName(sonarProjectName)
			projectVersion(sonarProjectVersion)
			msBuildScannerInstallationName(sonarScannerName)
		}
}
// ... Your build goes here
// Configure end analysis
configure { project -> 
		def sonarEnd = project / builders / 'hudson.plugins.sonar.MsBuildSQRunnerEnd' 
}

Show it on the TV

Finally one of the biggest strenghts of this setup is the ability to show it on the screen with relatively simple DSL (and of course with some Jenkins plugins).

deliveryPipelineView(applicationName + ' delivery Pipeline') {
	pipelineInstances(10)
	showAvatars()
    showChangeLog()
	showTotalBuildTime()
    enableManualTriggers(true)
	allowRebuild()
    showAggregatedPipeline(false)
    pipelines() {
        component(applicationName, applicationName + ' ' + buildEnvironmentName + ' Trigger')
    }
}

buildMonitorView(applicationName + ' build Monitor') {
	description(applicationName+' jobs')
	jobs {
        name(applicationName)
		regex('.*'+applicationName+'.*')
	}
}

Summarizing

Some characteristics about our project:

More than 100 deployments per month
Go live every workday morning (~20 times a month)
Half a year of live site hosting

Installing Jenkins with PowerShell DSC

Tue, 21 Nov 2017 14:40:00 GMT

I have been struggling to find a good reference about how to setup a Jenkins environment in Microsoft environment with an automatic installation script. So, I decided to write a blog post about it. I also wanted to write about how to do continuous integration with Episerver DXC but felt that I need to tell first how to setup Jenkins. Our aim is to put up a Jenkins server that can build .NET FrameWork MVC application and front end. If you are not a reader type of a person, then here is a GitHub link to example scripts.

About the PowerShell DSC

You might not be familiar with the PowerShell DSC. The acronym comes from Desired State Configuration. DSC has been there now for years. Still, in my experience most .NET developers are unfamiliar with DSC. DSC exists for various reasons:

Make scripting less complex
Make scripting look the same over various environments
Make scripts and their parts to be more reusable
Make installation scripts to be idempotent (repeatable)

The DSC is all about setting the state of a machine to be certain. Most used example is to make sure that a service is running or that specific file is found on given location. If you dig further into this world you will find concepts of pull and push servers that would help you to set a farm of machines into certain state. We will not use those but we run the script locally with the help of LCM which is “local configuration manager”. If you are looking for basics of PowerShell DSC then this blog was a well-written one.

DSC resources

Before we start configuration we need to talk about DSC resources. Resources provide you functionality for DSC. For our purpose we are needing at least one that helps us to grab software from Chocolatey (a package repository for windows). These resources would normally be where you start the script from (which might take you back to push and pull servers). In this scenario I just install them locally. Here is what my Install-Modules.ps1 script has inside.

Install-Module cChoco -f
Install-Module xNetworking -f
Install-Module xWebAdministration -f

Three modules that provide you three types of functionality. Something to get stuff from Choco, configuring network things and managing IIS. If you run this for the first time you might be asking whether you want to install nuget package provider for PowerShell. You should, if you want to follow this path. It grabs you the wanted modules from PowerShellGallery.

Our objective

We want to have a Jenkins server running in the end with the following requirements:

Install all the needed dependencies
Install Jenkins itself
Configure Jenkins startup parameters
Setup authentication for Jenkins
Install plugins for Jenkins
Protect Jenkins by settings IIS in front of it
Make sure that all traffic is HTTPS

Once our objective is now clear we can take a look at how I managed to do it.

Starting the scripting

In the following paraghraps, we have the basic structure of my PowerShell script. There are a few important parts in the script. The most important one is the Configuration JENKINS_CI which states that here is my DSC configuration. It has a few steps in it:

State the name of the configuration (JENKINS_CI)
Declare parameters (JenkinsPort with default value of 8080)
Import necessary DscResources (cChoco)
For all the nodes make sure of the following
Make sure NetFrameworkCore WindowsFeature is present
Make sure Choco is installed

Configuration JENKINS_CI
{
    param (
        $JenkinsPort = 8080
	)
	
	Import-DscResource -ModuleName 'cChoco'
	
	Node $AllNodes.NodeName {       
		# Install .NET 3.5
		WindowsFeature NetFrameworkCore 
		{
			Ensure    = "Present" 
			Name      = "NET-Framework-Core"
		}

		# Install Chocolatey
		cChocoInstaller installChoco
		{
			InstallDir = "c:\choco"
			DependsOn = "[WindowsFeature]NetFrameworkCore"
		}
	}
}

You should notice from above that you can mark one resource to be dependent on another with DependsOn. This is super important since we have multiple requirements for actually installing most of the stuff in our configuration later on. Now when we have the basic DSC configuration in place we can call it.

We need to define the AllNodes.NodeName that we referred in the script.
We also need to set the parameters we want for the configuration.
Finally we start the configuration with Start-DscConfiguraiton

$ConfigData = @{
    AllNodes = 
    @(
        @{
            NodeName = "LocalHost"
        }
    )
}

JENKINS_CI -JenkinsPort 8080 -ConfigurationData $ConfigData
Start-DscConfiguration -Path .\JENKINS_CI -Wait -Verbose -Force

All the needed dependencies

We will need a bunch of stuff to be able to build a .NET application.

Java for the Jenkins
Visual Studio and MSBuild for all the build targets and compilation
NodeJs for the front end compilation
Git for grabbing few more resources
Firefox for OWASP Zap
Notepad++ for comfort
Nuget.exe (I like to have a fixed version of this)
Python for … things
Jenkins for the actual goal
Zap for web application security testing automation

Scripting the installation of the software is fairly simple.

# Install Visual Studio, todo: optional features (F#) with param --includeOptional
cChocoPackageInstaller installVisualStudio
{
	Name = "visualstudio2017professional"
	DependsOn = "[cChocoInstaller]installChoco"
}

# Install Visual Studio Web tools 
cChocoPackageInstaller installVisualStudioWebWorkload
{
	Name = "visualstudio2017-workload-netweb"
	Params = "--includeOptional"
	DependsOn = "[cChocoInstaller]installChoco","[cChocoPackageInstaller]installVisualStudio"
}

# Install NuGet
File installNuget 
{
	DestinationPath = "C:\tools\nuget\nuget.exe"
	SourcePath = (Join-Path $InstallConfDirectory "nuget.exe")
	Ensure = "Present"
	Type = "File"
	Checksum = "modifiedDate"
	Force = $true
	MatchSource = $true
}

I think that the above script is quite self-explanatory although I made it shorter for easier reading. The full script can be found here We have a bunch of resources, most of them depending on something earlier. Afterwards we want to have all those resources in place. The only odd thing is in the installVisualStudioWebWorkload part where I add includeOptional parameter for the choco. This is to get F# installed on the target machine as well. It does not come by default. If you need to dig on to those additional options in nuget packages you should head to chocolateys web page and investigate the packages you need to figure out if there is any customization options.

Installing custom made stuff

We have something in our GitHub repositories that we would like to use with Jenkins. Those are marvelous libraries jmeter-perfotrator and powershell-zap. We can fetch files from a git repository directly into Jenkins server with the help of the git client we just installed. Here is an example resource for that.

# Install powershell-zap module 		
Script installPowershellZap 
{
	GetScript = {
		return @{ Result = gci "C:\tools\powershell-zap-master" }
	}
	SetScript = {
		mkdir "C:\tools\powershell-zap-master"
		$gitexe = "${ENV:ProgramFiles}\Git\cmd\git.exe"
		$arguments = 'clone https://github.com/solita/powershell-zap.git "C:\tools\powershell-zap-master"'
		$null = start-process $gitexe $arguments 
	}
	TestScript = {
		Return (Test-Path "C:\tools\powershell-zap-master")
	}
	DependsOn = "[cChocoPackageInstaller]installJdk8","[cChocoPackageInstaller]installGit" 
}

The ugly

The only big letdown that I have had with DSC is manipulating environments PATH variable. You can do it really easily but only once.

Environment setVS2017ToolsPath 
{
	Name = 'PATH'
	Ensure = 'Present'
	Path = $true
	DependsOn = "[cChocoPackageInstaller]installVisualStudio"
	Value = "${ENV:ProgramFiles(x86)}\Microsoft Visual Studio\2017\Professional\Common7\IDE"
}

After you have done that once and try to manipulate the exactly same named environment variable in the same configuration you will get an error. The same applies also for creating and deleting the same file in the same configuration. Instead, you can do a workaround by having multiple configurations or by using script resources in DSC. I have used script resources for this and well it is a bit complex but not worse than it is with traditional scripts.

# Set Java to path
Script SetJavaToPath 
{
	GetScript = {
		return @{ Result = $env:Path }
	}
	SetScript = {
		# Try to find Java bin path and force the result to string 
		[string]$javaBinPath = gci "${Env:ProgramFiles}\Java" -r -filter java.exe | Select Directory | Select-Object -first 1 | % { $_.Directory.FullName }
		# Adds javaBinPath to path variable 
		$newPathValue = $env:Path + ";"+$javaBinPath
		# You might need to reset your console after this 
		[Environment]::SetEnvironmentVariable("Path", $newPathValue, [EnvironmentVariableTarget]::Machine)
		# Add also path to current session
		$env:Path = $newPathValue
	}
	TestScript = {
		# Try to find Java bin path and force the result to string 
		[string]$javaBinPath = gci "${Env:ProgramFiles}\Java" -r -filter java.exe | Select Directory | Select-Object -first 1 | % { $_.Directory.FullName }
		if(-not $env:Path.Contains($javaBinPath))
		{
			# Do update
			Return $False
		}
		# Don't update
		Return $True
	}
	DependsOn = "[cChocoPackageInstaller]installJdk8"
}

Script resources are separated into three stages.

GetScript which returns something to be printed
SetScript which sets the value
TestScript which finds out if we need to set or not.

Test is like unit test, it fails with false which means that DSC needs to run the SetScript. I have multiple copies in my script of this so check them from GitHub repository.

Setup Jenkins startup parameters

Here are all the variables that I have for Jenkins configuration in my script.

param (
	$JenkinsPort = 8080,
	$JenkinsPlugins = @{},
	$JenkinsUsername = "",
	$JenkinsPassword = "",
	$JenkinsXmx = 1024,
	$JenkinsMaxPermSize = 128,
	$InstallConfDirectory = "./",
	$JenkinsInitScriptPath = "",
	$JenkinsUsernameTemplate = "",
	$JenkinsPasswordTemplate = ""
)

Now that you have seen all the variables we will introduce some mindblowing PowerShell magic and use them to setup Jenkins (it was already installed earlier). We can refer to those parameters with $Using:ParamName.

Script SetJenkinsServiceArguments
{
	SetScript = {
		$argString = "-Xrs -Xmx"+$Using:JenkinsXmx+"m -XX:MaxPermSize="+$Using:JenkinsMaxPermSize+"m -Djenkins.install.runSetupWizard=false -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar `"%BASE%\jenkins.war`" --httpPort="+$Using:JenkinsPort+" --webroot=`"%BASE%\war`""
		Write-Verbose -Verbose "Setting jenkins service arguments to $argString"
		
		$Config = Get-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
		$NewConfig = $Config `
			-replace '<arguments>[\s\S]*?<\/arguments>',"<arguments>${argString}</arguments>"
		Set-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml" `
			-Value $NewConfig `
			-Force
		Write-Verbose -Verbose "Restarting Jenkins"
	}
	GetScript = {
		$Config = Get-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
		$Matches = @([regex]::matches($Config, "<arguments>[\s\S]*?<\/arguments>", 'IgnoreCase'))
		$currentMatch = $Matches.Groups[1].Value
		Return @{
			'Result' = $currentMatch
		}
	}
	TestScript = { 
		$Config = Get-Content `
			-Path "${ENV:ProgramFiles(x86)}\Jenkins\Jenkins.xml"
		$Matches = @([regex]::matches($Config, "<arguments>[\s\S]*?<\/arguments>", 'IgnoreCase'))
		$argString = "-Xrs -Xmx"+$Using:JenkinsXmx+"m -XX:MaxPermSize="+$Using:JenkinsMaxPermSize+"m -Djenkins.install.runSetupWizard=false -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar `"%BASE%\jenkins.war`" --httpPort="+$Using:JenkinsPort+" --httpListenAddress=127.0.0.1 --webroot=`"%BASE%\war`""
		$currentMatch = $Matches.Groups[1].Value
		
		Write-Verbose "Current service arguments: $currentMatch"
		Write-Verbose "Should be service arguments: $argString"
		
		If ($argString -ne $currentMatch) {
			# Jenkins port must be changed
			Return $False
		}
		# Jenkins is already on correct port
		Return $True
	}
	DependsOn = "[cChocoPackageInstaller]installJenkins"
}

I admit. The script could be cleaner. Nevertheless it sets startup arguments in jenkins.xml file to have given memorylimit parameters and listen only localhost on the given port. We listen only localhost because I don’t feel Jenkins should be open in any network. I’ll later on show how to setup IIS to be a reverse proxy in front of the Jenkins.

Initial authentication setup for Jenkins

If we would startup the Jenkins at this moment it would not be ready to get api requests from us which we will need to install plugins. So what we need to do is to setup a initialization script to setup authentication. We can do it by making sure that there is groovy script under Jenkins/init.groovy.d/ folder. All the groovy scripts are run on that folder are run on the Jenkins startup. Yes. That is a bit scary. The actual script setup is easy one:

File JenkinsAuthenticationSetup 
{
	DestinationPath = $JenkinsInitScriptPath
	SourcePath = (Join-Path $InstallConfDirectory "solita_jenkins_security_realm.groovy")
	Ensure = "Present"
	Type = "File"
	Checksum = "modifiedDate"
	Force = $true
	MatchSource = $true
	DependsOn = "[cChocoPackageInstaller]installJenkins"
}

So we put a groovy script there and it is set. That script checks the authorization strategy and sets it to be full control once logged in authorization strategy. It also creates a user with given username and password. The script file has placeholders for username and password so we just use again some scripting to replace them with given parameters.

Script SetJenkinsAuthenticationUsername
{
	SetScript = {
		$username = $Using:JenkinsUsername
		(Get-Content $Using:JenkinsInitScriptPath).Replace($Using:JenkinsUsernameTemplate,$username) | Set-Content $Using:JenkinsInitScriptPath
	}
	GetScript = {
		$containsReplacaple = (get-content $Using:JenkinsInitScriptPath) | % {$_ -match $Using:JenkinsUsernameTemplate } | ? { $_ -contains $true }
		$aResult = $containsReplacaple -eq $True
		Return @{
			'Result' = $aResult
		}
	}
	TestScript = {
		$containsReplacaple = (get-content $Using:JenkinsInitScriptPath) | % {$_ -match $Using:JenkinsUsernameTemplate } | ? { $_ -contains $true }
		if($containsReplacaple -eq $True)
		{
			# needs configuration
			Return $False
		}
		Return $True
	}
}
Script SetJenkinsAuthenticationPassword
{
	SetScript = {
		$password = $Using:JenkinsPassword
		(Get-Content $Using:JenkinsInitScriptPath).Replace($Using:JenkinsPasswordTemplate,$password) | Set-Content $Using:JenkinsInitScriptPath 
	}
	GetScript = {
		$containsReplacaple = (get-content $Using:JenkinsInitScriptPath) | % {$_ -match $Using:JenkinsPasswordTemplate } | ? { $_ -contains $true }
		$aResult = $containsReplacaple -eq $True
		Return @{
			'Result' = $aResult
		}
	}
	TestScript = {
		$containsReplacaple = (get-content $Using:JenkinsInitScriptPath) | % {$_ -match $Using:JenkinsPasswordTemplate } | ? { $_ -contains $true }
		if($containsReplacaple -eq $True)
		{
			# needs configuration
			Return $False
		}
		Return $True
	}
}

Start the Jenkins

Nothing fancy here. This is the most common example of DSC usage. Script is really simple. Make sure that Jenkins is started automatically and that it is running.

Service JenkinsService
{
	Name        = "Jenkins"
	StartupType = "Automatic"
	State       = "Running"
	DependsOn = "[cChocoPackageInstaller]installJenkins","[Script]SetJenkinsServiceArguments","[File]JenkinsAuthenticationSetup","[Script]SetJenkinsAuthenticationUsername","[Script]SetJenkinsAuthenticationPassword"
} 

Install plugins

My coworker argued that the Jenkins is bad build server because it does not even give you timestamps on the console log by default. For all of this kind of arguments there is always the same counterargument. There is a plugin for that! We will need plugins. A lot of plugins. Actually more than 40 in our setup which would make with dependencies … well a huge amount of foreign code running on your build server. A full list of plugins can be found at here.

Now when we have happily setup the Jenkins and made sure that it is running we can start using its API to install some plugins. Here is a script that expects to have a list of plugins as a parameter.

Script InstallJenkinsPlugins
{
	SetScript = {
		$plugins = $Using:JenkinsPlugins
		$port = $Using:JenkinsPort
		$password = $Using:JenkinsPassword
		$username = $Using:JenkinsUsername
		
		# Make sure that Jenkins is in the configurated state
		Restart-Service `
			-Name Jenkins
		Start-Sleep -s 15
		
		# Wait a bit for Jenkins to get online 
		$request = [system.Net.WebRequest]::Create("http://localhost:${port}")
		for ($i = 1; $i -le 10; $i++) {
			try {
				   $result = $request.GetResponse()
			} catch [System.Net.WebException] {
				   $result = $_.Exception.Response
			}
			
			if ($result -is "System.Net.HttpWebResponse" -and $result.StatusCode -ne "") {
				$done = "Got status"
				break
			}
			
			Write-Host "Get status attempt number $($i) failed. Retrying..."
			Start-Sleep -s 5
		}
		
		# Install plugins
		
		foreach ($jplug in $plugins) {
			Write-Verbose "installing $jplug"
			java -jar ${ENV:ProgramFiles(x86)}\Jenkins\war\WEB-INF\jenkins-cli.jar  -s "http://localhost:${port}" install-plugin $jplug --username $username --password $password
			# Wait a bit, Jenkins is kind of slow 
			Start-Sleep -s 5
		}
		Write-Verbose -Verbose "Restarting Jenkins"
		Restart-Service `
			-Name Jenkins
	}
	GetScript = {
		Return @{ Result = Get-ChildItem "${ENV:ProgramFiles(x86)}\Jenkins\plugins" | Select Name }
	}
	TestScript = {
		# Check if there are plugins
		$directoryInfo = Get-ChildItem "${ENV:ProgramFiles(x86)}\Jenkins\plugins" | Measure-Object
		# Directory is empty, do the update
		if ($directoryInfo.Count -eq 0) {
			Return $False
		}
		# Do not make update 
		Return $True
	}
	DependsOn = "[cChocoPackageInstaller]installJenkins","[Script]SetJenkinsServiceArguments","[File]JenkinsAuthenticationSetup","[Service]JenkinsService","[Script]SetJenkinsAuthenticationUsername","[Script]SetJenkinsAuthenticationPassword"
}

Now we have actual build server running that listens localhost. We have installed all the tools, all the plugins and setup the authentication so that we can start working on creating build jobs. At this point you might want to delete the authentication initialization script. You can’t do it with the same DSC script because it disallows manipulating same file twice in same configuration. Although you can do it afterwards with a single liner.

Install IIS

I created separated script for this just because I felt that this script is a reusable one. So let’s start from the beginning and introduce some Windows Features that we want to use in a brand new DSC script. The modules that we are using are the same as stated in the beginning so you would need to import them first. Here are the resources that we need:

# Check the windowsfeature names with Get-WindowsFeature
# Install .NET 3.5
WindowsFeature NetFrameworkCore 
{
	Ensure    = "Present" 
	Name      = "NET-Framework-Core"
}
# Install Chocolatey
cChocoInstaller installChoco
{
	InstallDir = "c:\choco"
	DependsOn = "[WindowsFeature]NetFrameworkCore"
}
# Install the IIS role
WindowsFeature IIS
{
	Ensure          = "Present"
	Name            = "Web-Server"
}
# Install the ASP .NET 4.5 role
WindowsFeature AspNet45
{
	Ensure          = "Present"
	Name            = "Web-Asp-Net45"
}
# Install the web management console
WindowsFeature WebManagementConsole
{
	Ensure          = "Present"
	Name            = "Web-Mgmt-Console"
	DependsOn 		= "[WindowsFeature]IIS"
}

Basically we installed IIS WebServer with some tooling. Yet again the full script can be found at here

Setup a website for proxying

The next thing would be to get rid of the “Default Web Site” just because I hate having it laying around. This is really simple task to do.

# Make sure to get rid of default web site
xWebsite DefaultWebSite
{
	Ensure          = "Absent"
	Name            = "Default Web Site"
	State           = "Stopped"
	PhysicalPath    = "C:\inetpub\wwwroot"
	DependsOn = "[WindowsFeature]AspNet45","[WindowsFeature]IIS"
}

Once we have got rid of that ugly thing we can create our own physical folder for the website and also the actual website. I have index.html ready for the proxy site just to make recognizing problems easier.

File JenkinsProxyFolder 
{
	DestinationPath = "C:\inetpub\JenkinsProxy\index.html"
	SourcePath = (Join-Path $InstallConfDirectory "index.html")
	Ensure = "Present"
	Type = "File"
	Checksum = "modifiedDate"
	Force = $true
	MatchSource = $true
}
# Create jenkins proxywebsite
xWebsite JenkinsProxyWebSite
{
	Ensure          = "Present"
	Name            = "JenkinsProxyWebSite"
	State           = "Started"
	PhysicalPath    = "C:\inetpub\JenkinsProxy"
	BindingInfo     = @(
		MSFT_xWebBindingInformation
		{
			Protocol              = "HTTPS"
			Port                  = 443
			CertificateThumbprint = $thumbPrint
			CertificateStoreName  = "My"
		}
	)
	DependsOn = "[WindowsFeature]AspNet45","[WindowsFeature]IIS","[File]JenkinsProxyFolder"
}

You might have noticed that the site was setup with HTTPS and it has a thumbprint as a parameter. I was lazy and did just some manual script step before DSC for installing the certificate into the machine. It requires that you know how to get a .pfx for TLS.

$thumbPrint = "your_thumbprint"
$thumb = Get-ChildItem cert:\localmachine\my | Where { $_.Thumbprint -like $thumbPrint }| Select Thumbprint
if($thumb -eq $NULL) 
{
	$mypwd = Read-Host -AsSecureString "Give password for certificate"
	Import-PfxCertificate -FilePath .\your_certificate.pfx cert:\localMachine\my -Password $mypwd
}

If you have no idea how to get a certificate. Then you might want to try this script. It creates a self-signed certificate and gives you the thumbprint that my script is asking.

New-SelfSignedCertificate -DnsName "www.jenkinstest.fi" -CertStoreLocation "cert:\LocalMachine\My"

With this setup we should have in place a website that is running and responding in 443 port. If you want to support redirects from 80 to 443 too then you need to allow the 80 port too.

UrlRewrite and ApplicationRequestRouting

To be able to finish the setup we will need a few more dependencies. Those are UrlRewrite and IIS-ApplicationRequestRouting. Here is a yet again easy setup for those two.

# Install UrlRewrite
cChocoPackageInstaller UrlRewrite
{
	Name = "urlrewrite"
	DependsOn = "[cChocoInstaller]installChoco"
}
# Install UrlRewrite
cChocoPackageInstaller ApplicationRequestRouting
{
	Name = "iis-arr"
	DependsOn = "[cChocoInstaller]installChoco"
}

And then the fun ends and we will fall back to the ScriptResources and do some configuration madness. First of all we need to make sure that certain server variables are allowed. We can do it in this way:

Script ReWriteRules
{
	#Adds rewrite allowedServerVariables to applicationHost.config
	DependsOn = "[cChocoPackageInstaller]UrlRewrite"
	SetScript = {
		$current = Get-WebConfiguration /system.webServer/rewrite/allowedServerVariables | select -ExpandProperty collection | ?{$_.ElementTagName -eq "add"} | select -ExpandProperty name
		$expected = @("HTTPS", "HTTP_X_FORWARDED_FOR", "HTTP_X_FORWARDED_PROTO", "REMOTE_ADDR")
		$missing = $expected | where {$current -notcontains $_}
		try
		{
			Start-WebCommitDelay 
			$missing | %{ Add-WebConfiguration /system.webServer/rewrite/allowedServerVariables -atIndex 0 -value @{name="$_"} -Verbose }
			Stop-WebCommitDelay -Commit $true 
		} 
		catch [System.Exception]
		{ 
			$_ | Out-String
		}
	}
	TestScript = {
		$current = Get-WebConfiguration /system.webServer/rewrite/allowedServerVariables | select -ExpandProperty collection | select -ExpandProperty name
		$expected = @("HTTPS", "HTTP_X_FORWARDED_FOR", "HTTP_X_FORWARDED_PROTO", "REMOTE_ADDR")
		$result = -not @($expected| where {$current -notcontains $_}| select -first 1).Count
		return $result
	}
	GetScript = {
		$allowedServerVariables = Get-WebConfiguration /system.webServer/rewrite/allowedServerVariables | select -ExpandProperty collection
		return $allowedServerVariables
	}
}

Then once we have that setupped we can make the rewrite rules for routing all the traffic from 443 port to jenkins running in different port.

Script JenkinsReverseProxy
{
	DependsOn = "[cChocoPackageInstaller]UrlRewrite","[cChocoPackageInstaller]ApplicationRequestRouting"
	SetScript = {
		$Name = "HTTPS Reverse Proxy to Jenkins"
		$proxyTargetPath = ("http://localhost:"+$Using:JenkinsPort+"/{R:0}")

		Clear-WebConfiguration -pspath $PsPath -filter "$Filter/rule[@name='$Name']"
		$Filter = "system.webserver/rewrite/rules"
		Clear-WebConfiguration -location $Site -pspath $PsPath -filter "$Filter/rule[@name='$Name']"
		Add-WebConfigurationProperty -location $Site -pspath $PsPath -filter "$Filter" -name "." -value @{name=$Name; patternSyntax='ECMAScript'; stopProcessing='True'}
		Set-WebConfigurationProperty -location $Site -pspath $PsPath -filter "$Filter/rule[@name='$Name']/match" -name url -value "(.*)"
		Set-WebConfigurationProperty -location $Site -pspath $PsPath -filter "$Filter/rule[@name='$Name']/action" -name "type" -value "Rewrite"
		# R:0 Is full phase, R:1 Is the domain with the port and R:2 is the querypart
		Set-WebConfigurationProperty -location $Site -pspath $PsPath -filter "$Filter/rule[@name='$Name']/action" -name "url" -value $proxyTargetPath
	}
	TestScript = {
		$current = Get-WebConfiguration /system.webServer/rewrite/rules | select -ExpandProperty collection | select -ExpandProperty name
		$expected = @("HTTPS to Jenkins")
		$result = -not @($expected| where {$current -notcontains $_}| select -first 1).Count
		return $result
	}
	GetScript = {
		$rules = Get-WebConfiguration /system.webServer/rewrite/rules | select -ExpandProperty collection
		return $rules
	}
}

If you are looking UrlRewrite rules first time that might look confusing. Basic idea is just to make a xml configuration for IIS. Next up we will need to setup an ARR proxy to actually get the proxy functionality enabled. Yet again we go with the ScriptResource and make some XML configuration.

Script EnableARRProxy
{
DependsOn = "[WindowsFeature]WebManagementConsole","[WindowsFeature]IIS","[cChocoPackageInstaller]UrlRewrite","[cChocoPackageInstaller]ApplicationRequestRouting"
	SetScript = {
		$assembly = [System.Reflection.Assembly]::LoadFrom("$env:systemroot\system32\inetsrv\Microsoft.Web.Administration.dll")
		$manager = new-object Microsoft.Web.Administration.ServerManager
		$sectionGroupConfig = $manager.GetApplicationHostConfiguration()

		$sectionName = 'proxy';

		$webserver = $sectionGroupConfig.RootSectionGroup.SectionGroups['system.webServer'];
		if (!$webserver.Sections[$sectionName])
		{
			$proxySection = $webserver.Sections.Add($sectionName);
			$proxySection.OverrideModeDefault = "Deny";
			$proxySection.AllowDefinition="AppHostOnly";
			$manager.CommitChanges();
		}

		$manager = new-object Microsoft.Web.Administration.ServerManager
		$config = $manager.GetApplicationHostConfiguration()
		$section = $config.GetSection('system.webServer/' + $sectionName)
		$enabled = $section.GetAttributeValue('enabled');
		$section.SetAttributeValue('enabled', 'true');
		$manager.CommitChanges();
	}
	TestScript = {
		$assembly = [System.Reflection.Assembly]::LoadFrom("$env:systemroot\system32\inetsrv\Microsoft.Web.Administration.dll")
		$sectionName = 'proxy';
		$manager = new-object Microsoft.Web.Administration.ServerManager
		$sectionGroupConfig = $manager.GetApplicationHostConfiguration()
		$config = $manager.GetApplicationHostConfiguration()
		$section = $config.GetSection('system.webServer/' + $sectionName)
		return ($section -eq $null -and $section.GetAttributeValue('enabled') -eq $False)
	}
	GetScript = {
		$assembly = [System.Reflection.Assembly]::LoadFrom("$env:systemroot\system32\inetsrv\Microsoft.Web.Administration.dll")
		$sectionName = 'proxy';
		$manager = new-object Microsoft.Web.Administration.ServerManager
		$sectionGroupConfig = $manager.GetApplicationHostConfiguration()
		$config = $manager.GetApplicationHostConfiguration()
		$section = $config.GetSection('system.webServer/' + $sectionName)
		return $section.GetAttributeValue('enabled')
	}
}

Only last thing is to actually run the script.

$ConfigData = @{
	AllNodes = 
	@(
		@{
			NodeName = "LocalHost"
		}
	)
}
$thumbPrint = "certificate_thumbprint"
$currentPath = (split-path -parent $MyInvocation.MyCommand.Definition)
$installConfPath = (join-path $currentPath "misc")
IIS_REVERSE_PROXY -ThumbPrint $thumbPrint -InstallConfDirectory $installConfPath -JenkinsPort 8080 -ConfigurationData $ConfigData
Start-DscConfiguration -Path .\IIS_REVERSE_PROXY -Wait -Verbose -Force

Conclusion

I really like how the PowerShell DSC makes some things to be so easy and self-explanatory. What I didn’t like was the need to step down on to creating ScriptResources. Creating a script resource takes time and a lot of testing. You will need to run your script multiple times. This is of course there the idempotent nature of the DSC comes really handy. Although I was able to corrupt my PATH variable (in a VM) into horrible state while testing. Protip would be to get a VM for developing scripts. My usual setup is something like this:

Windows 2016 server running on Hyper-V
Snapshot before running any scripts
Another snapshot when all the software is installed (downloading all the stuff takes ages)
Develop on your own machine, throw in the new script for the VM
Run the script
Repeat 4 and 5 and if you screw up badly then go back to snapshot from point 3.
Finally when you think you are ready go back to snapshot from point 2.

I hope you liked it. All the script is available at our GitHub.

Known problems

Few things that might be broken at the repository. If you are trying to setup the ZAP as a daemon you need to start it manually and change the port to something else since it have same default (8080) as my scripts have. Another thing that might be broken is that later on if you would like to use OWASP dependency check module that was installed. Then you might hit the problem (or not) that Java cacerts does not have let’s encrypt intermediatory certificate. A fix is in the repository in case you need it.

Achieving Azure certification after ARM refresh

Wed, 01 Mar 2017 16:20:00 GMT

I started developing on Azure when the current “old portal” did not yet exist. Back then Azure resources were managed with Silverlight portal. After being around in Azure for years I decided to give a shot for the certifications while I can still earn both MCSE and MCSD with the same trouble. MCSD is expiring at the end of march 2017. Here are some observations from the certification journey.

Overall feeling about the exams

They reflected real life quite well. I was actually surprised that there were only few questions that got me angry. The upsetting ones were either confusing or asked about numbers that I don’t feel like wanting to memorize. If you are not familiar with the Microsoft exams, then you should know how they are structured. There are four things that you will find in every question.

Business problem (enterprise x is developing y to achieve business goal z)
Goal statement (you need to solve issue x)
One or more correct answers
DISTRACTERS (stuff that are not related to problem scenario but rings bells in your head as they are familiar)

Try to extract the essence of the question out and then eliminate the distractors. Some of the distractors were so bad that I was still wondering about them 10 questions after.

Were the questions easy? No. Having years of experience from three generation of Azure portals I just barely passed the first one. Second one went through with a bit better score and the third one with the highest score. I took them in an order that I thought was most familiar to me: development, architecture, infrastructure. What does this tell? They overlap a lot so each round was easier because of more studying on the same subject.

How to mentally prepare yourself to the quetions? Ask yourself questions like “There is a problem with x, where would I find logs?”, “How do i scale this?” or “How do i secure the connection?”. If you can answer those, then you will most likely be successful in eliminating the distractors out of the scenery. Even if the options you are left are nearly the same you have still atleast improved your odds!

About the certificates and the ARM refresh

This is how the old pyramid looked like. And the new one is actually with the exact same format. The lowest tier is the technology specialist tier which you get if you pass a certain kind of exam. The middle tier is solution associate tier which you will get by passing two exams. The highest tier is solution expert tier, which you will get after passing one test and having the solution associate already.

Before studying any further notice that when redesigning the tiers they actually changed the contents of the few relevant exams. Before paying for any training material make sure that the ARM refresh is mentioned. For example books that were preparing for 70-532, 70-533 and 70-534 tests have not been updated. In my experience, you can pretty safely forget the “old portal” and focus on the newer one.

Note that there are also other routes to the same MCSE! You don’t need to take any of the exams that I took.

No more recertification

I have always hated the recertifications. The new tier of Microsoft certificates seems to be lacking of expiration date. I love that. As a developer I get nothing out of recertification for the subject that I have already been certificated. If I have forgotten something during the 2-3 year period then I’m sure that it has not been important. Renewing the internals of the exam won’t help the core of the exam mostly remains the same still. I would rather expand my view of sight by taking a new certificate.

Hint to Episerver: you could learn from the security scene. There you can keep your qualification alive by gathering points from events. Like doing a blog post or speaking in a seminar. That would work for Episerver certifications too. This does not prevent that you could take recertification if you wish. There are much more like assessments, projects, forums, support tickets, and GitHub repositories that you could track. Nothing in this prevents from taking the certification test again.

Three exams, seven certificates

At the edge of certification era change I was able to get seven different certificates with three exams. How is this possible? Well, technology specialist certification is granted from each exam. After two exams solution associate certification is granted. Finally, I was granted solution expert and solution developer certificates from passing all three exams. As a bonus, I also got MCSD App Builder by having MCSA: web applications already in my pocket. The full list is:

Microsoft Specialist: Developing Microsoft Azure Solutions
Microsoft Specialist: Architecting Microsoft Azure Solutions
Microsoft Specialist: Implementing Microsoft Azure Infrastructure solutions
Microsoft Certified Solutions Associate: Cloud Platform
Microsoft Certified Solutions Developer: Azure Solutions Architecting
Microsoft Certified Solutions Developer: App Builder
Microsoft Certified Solutions Expert: Cloud Platform and Infrastructure

Expect to earn at least five certificates by targetting to MCSE.

70-53x overlaps with each other

Despite which number of exams you are going to take, I strongly suggest that you check the materials of each three exam. There were lots of stuff even in Ignite cert preparation videos that would have been beneficial for me in the first place. Here are the list of similarities:

ARM templates
ARM virtual machines
ARM Networking
Storage solutions (especially Azure Storage)
Web apps
Managing identities

I don’t mean to say that the tests are the same. Each exam certainly has its owng angle to the subject. Expect to see code in developer exam, expect to see command-line stuff in infrastructure exam and expect to see design problems in architecture exam. Still, there are lots of questions like “Which is the best for job x in circumstances y?” and the answer would be same regardless of the exam the question was in.

The thing is that you can learn from many of those subjects by learning the ARM templates. So make sure that you study at least this walkthrough

Hot topics

Where to focus? What to learn the best? It’s hard to go wrong with these:

Azure PowerShell (try to setup environments with it directly creating the resources and with arm templates)
ARM templates (use the automation button in portal and try to understand the json)
Storage (I made example clients and I did not need to regret it)
Web apps (especially the pricing tiers, scaling and monitoring)
Virtual machines (pricing tiers, scaling and monitoring)
Hybrid cloud (how to connect on-premise stuff to cloud and vice versa)

What to expect from 70-532 Developing Microsoft Azure solutions

Case studies and code examples. You don’t need to be a good programmer to pass this. You need to know the services listed in the exam and how to do design decisions with them. To be honest, I felt that this was more of a DevOps test than a developer test. There were code yes, but if you think what are the services that needs specific kind of coding to work you don’t have that much to learn. Those things are:

WebJobs
CloudServices (role entrypoints and stuff)
Storage clients
Azure Service Bus clients

The thing is that you need to know a lot about things like DNS, HTTPS, deployments, monitoring and scaling to pass this test. In some organizations this might not be typical for developer. The DevOps-loving Solita of course knows this all!

Link to the exam
Link to the preparation video

What to expect from 70-533 Implementing Microsoft Azure Infrastructure solutions

I was not entirely sure what to expect from this one. I watched The Ignite videos and studied the docs. It was helpful. Notice that the exam materials say that you need to know how to implement Windows and Linux systems. Yes, Linux in Microsoft certification. Luckily, I had used Linux enough that I was able to answer the questions without preparing to answer to them.

Here are a lots of useful stuff at IaaSOpsGuide.
You might be also interested in tools
Link to the exam
Link to the preparation video

What to expect from 70-534 Architecting Microsoft Azure solutions

All I can say is that try to understand purpose of every thing in this picture:

Make extra sure that you understand the purpose of different Azure connectivity options (p2p, p2s, ExpressRoute) and that you study hybrid cloud solutions. There were questions related to nearly every service I service I could think of and it was up to luck if the questions were detailed and hard or merely just about knowing the purpose of a service.

Picture was borrowed from channel9

Link to the exam
Link to the preparation video
Udemy course that helped me to get the overview

How to study

Everyone has their own way to learn. I would still mainly focus on all the materials on under the Azure docs. Then I would definitely watch the preparation videos from Ignite 2016, all three for every certification (with 1.5x speed). And finally I would go out to the portal. I would press buttons N (for new popup) and B (for browse) and setup all the stuff that are mentioned on the exam. I would also definetly press the “automation” button that would show me the json template of the stuff I was about to create. I would also study resource groups json template after I would have set all in place.

There are also a lot of “reading lists” available for the certifications. I checked a few but so many things have changed so recently and so many links were broken that I can’t really recommend those.

Braindumps are no-no for me. But I think that there is kind of “official” practice tests available. You should learn explanations from them if you buy them. I didn’t want to invest to those because I felt confident even without them. Measureup had refreshed atleast some of their tests to the ARM era.

Why?

Why would you want to certificate? Well, the usual list is kind of boring. Partnerships, CV, sales, test your limits etc etc. I took the exams because my company encouraged me to do this. My company also supports certifications by giving a chance to study during worktime. Of course, I also got a chance to refresh my knowledge on the areas that are not so familiar to me.

Most of the time the customer is heavily influencing where to host Episerver. Azure is one option. DXC also runs on top of Azure. There is really no harm to understand in depth Azure as a hosting option or as an option to implement microservices-related to Episerver projects. Azure AD is also a possibility for identity federation.

How much effort for this all?

From 10-15 hours preparation for each test. Then, a few hours to take the actual exam. That was it for me. I got the whole thing done during February 2017.

Episerver developer meetup at Solita

Mon, 10 Oct 2016 15:45:00 GMT

Solita organized a meetup for Finnish Episerver developers in Helsinki on the 29th of September. The meetup consisted of food, beer and presentations. The event was a successor to the spring’s Episerver dev meetup and was as fun and cozy as the first one. This time we had a bit more time in the end for the networking than last time!

Commerce

Johan Hedberg from the Commerce development team visited us and gave a presentation about how to use Commerce to your daily household finance governance. He showed us how to invent your own currency in Commerce, how to give users credit to be used inside the Commerce and how to use the credit for shopping. In addition to that he showed how to use promotion engine in household to give stuff cheaper to household members when they do laundry or clean toilets. It was a very nice presentation. There were not many slides but Johan promised that the fun and educational demo application will be available in some format later. Looking forward for that!

Performance

Tommi Raunio from Solita had a presentation about performance. He first talked about what performance is and why it matters and then divided the topic into three different segments: design & architecture, implementation, and measurement. The slides can be found here .

DevOps

I also had a presentation of my own about DevOps with Episerver. First I talked about how to help new guys in installing development environments. Then I talked about how to script server installations with PowerShell. Then I went a bit deeper and talked about how to build a build pipeline and finally I talked about something about NuGets. The Slides.

Finally, thanks for everyone for the great evening!

Episerver developer meetup at Helsinki

Mon, 25 Apr 2016 02:00:00 GMT

Last week Episerver organized a meetup for Finnish developers in Helsinki. Meetup consisted of pizza, beer and presentations. I loved the meetup and the format was a perfect match to have a cozy atmosphere around the Episerver topic. Only thing that could have been better (next time maybe?) would be to have more time at the end for the networking! Which just means that the actual event was great success!

R.I.P XForms

Mikko Huilaja from Solita had a hour long presentation about new Episerver Forms. He went through how they work, how you can create them, edit them, and also how to extend them. I haven’t had the time before to take look on the new Episerver Forms before, but now I feel I would be ready to implement them any day. Thanks Mikko for this great deep dive into the new Episerver Forms! Slides

Commerce promotion engine

Quan Mai gave us a sneak peek into the upcomming promotion engine enhancement. If I understood correctly, the promotion engine might be included in next Commerce major release. The new system is IContent based and it was promised to be a lot faster than the old system. Debugging and extending it seemed to be thought in a good manner. All this means that there will be even less need to use the old Commerce Manager in the future.

At Solita our Commerce developers absolutely love this news!

Webhooks

James Stout (the egandalf himself) had a cool presentation about webhooks. With just a few lines of code he showed us how to hook into form post event and push its content through Zapier into google spreadsheets. It looked so easy and so painless I wish I would be as cool magician as egandalf is!

Cyberaware development with Episerver

I also had a presentation of my own about secure development. I talked about how developers should also be hobbyist hackers (white hat ones), what secure development lifecycle means, threat modeling and how this all is linked to our Episerver projects. I had only half of the time I thought I had so I hope that people got something out of my fast-forwarded slideshow. Slides

Finally, thanks Episerver for the great evening!

Installing server environment with PowerShell

Thu, 11 Feb 2016 01:00:00 GMT

Most likely, as a reader you are mostly interested in my scripts. Here are my scripts for setting up Windows Server 2012 R2 ready for WebDeployments and running EPiServer ASP.NET site!

Why automate the installation?

This blog is about how to make your brand new Windows Server ready for WebDeployments with just pressing enter once. Cloud services can make your infrastructure lifecycle handling very easy; still people often encounter situations where they need to host our ASP.NET applications in virtual machines or directly on physical hardware. On those situations installation procedures in Windows operating systems are often done with some “clickety click” magic that won’t take too long. Still the “clickety click” installations have lots of long-term problems:

Installations can’t be reproduced
Only the installer knows how he did it
If you have multiple servers you are doing same manual steps multiple times
Base of the installations does not differ much from project to project
After few years when your windows server needs upgrade you will need to repeat this
There is no way to test “clickety click” installations
Windows Server Core installations are rare because people is not used to manage Windows Servers without GUI

One could argue that documentation and clear processes would take care of all the problems above. Maybe they could but I have never seen installation documentation that has 100% coverage over how the installation has been done. Installation script works as document and developers are more likely to update it!

What needs to be installed on a fresh Windows server

Here is short version of my list what I would do for new Windows Server

Install IIS
Install WebPI
Install newest .NET
Install webdeploy
Install various modules for IIS from windows feature list or with WebPI
Install tools for administration (7zip, text editor etc)
Create IIS site and application pool and change some defaults for better

Is there PackageManagement for Windows?

Oh yes there is! It is called Windows PackageManagement (aka OneGet). Unfortunately it is not available for Windows Servers yet. Production preview of Windows Management Framework 5 is already available. Windows PackageManagement is actually a manager that can access multiple type of repositories to have unified way to download and install software from multiple sources. Microsoft MSI installers, Windows Features and software could be all installed with Windows PackageManagement in the future. The thing still missing is grown up economy where all the toys would be easily available in a secure manner. Chocolatey is pretty good already but it is missing a lot of software and I’m still sceptical about security of software packages. Also Chocolatey is not the main delivery channel for many software companies and there might be a huge delay before software is available from there. So instead of using Chocolatey I provide examples how to do silent installations with different type of software packages.

How stuff is installed without Chocolatey

If you are familiar with Windows Server installation you might notice that there are multiple different types of installation involved in the process. Windows features, executables, msi files and software specific plugins are all in the same process. This is the part where I wish that Windows PackageManagement (aka OneGet) or Chocolatey can one day solve unifying installation.

Windows features (e.g. IIS)

The first question you should be looking at is “what are the windows features”? To be able to install the features you must first learn how to find them. Here is a oneliner that will give the available features to you.

Get-WindowsOptionalFeature -Online

Great! Now we know how to find the windows features. Now we just need to change the oneliner a bit to be able to Install them. This is a oneliner for .NET:

Enable-WindowsOptionalFeature -Online -All -FeatureName NetFx4

All-flag means that it will also install required dependencies if any. The .NET delivered among the operating system might not be latest, but the feature needs to be enabled all the same. IIS will need it among some other features. Full list of my users can be found in our example config file at the bottom of the article.

Executables

With windows feature we were able to install the .NET version that is delivered with the operating system. Although that is unlikely the newest one. Instead we need to fetch the newest one from the internet. With executables you never can be quite sure how they should be installed because there is no strictly unified commandline parameters. Here is an example how your executable installation might look like.

$arguments = @(
        "/qn" # display no interface 
        "/norestart"
        "/passive"
		"/S")
Start-Process $exe  -Wait -PassThru -Verb runAs -ArgumentList $arguments

I chose an approach where I just guess what kind of parameters the executable will need for silent installation. I found out that /S is pretty common, but sometimes also /passive is used too. Most likely your executables needs to be run elevated, that can be achieved with “-Verb runAs” parameter for Start-Process function. So even if this worked for my executables, this might be terrible mess for yours (for example if there is checks for arguments being eligible).

MSI files

MSI files are really common way to install stuff in Microsoft environment. They are also unified so we can be more certain that silent installation can be achieved. The installation looks really similar to installation of executables.

$arguments = @(
        "/i" #install product
        "`"$msiFile`""
        "/qn" # display no interface 
        "/norestart"
        "/passive")
Start-Process msiexec.exe -Wait -PassThru -Verb runAs -ArgumentList $arguments

The File to be installed is given as an parameter to msiexec.exe among some other flags. Installer is again run as elevated process to make sure that it can make all the configuration changes it wants.

WebPI modules

To be able to install WebPI modules, you need to first install it. After installation you are able to use it’s cmdline tools to install modules. Before you know what to install, you might need to list the available modules. Here is an example oneliner for listing:

& (Join-Path "$env:programfiles" "microsoft\Web Platform Installer\webpicmd.exe") /List /ListOption:All

Once we get all the available modules we can find out that the name of the needed WebDeploy package is WDeploy36NoSmo. Here we install WebDeploy with oneliner:

& (Join-Path "$env:programfiles" "microsoft\Web Platform Installer\webpicmd.exe") /Install /Products:"WDeploy36NoSmo" /AcceptEula

AcceptEula flag basicly gives you silent installation. You might want to elevate this process too, but I didn’t feel that it was necessary.

How stuff is configured

Once we got our IIS, .NET, WebPI and WebDeploy successfully installed there are still few steps before machine is ready to host our applications. First we need to make sure that the IIS website is set to be able to make deployments.

IIS website

First of all we need a folder for IIS website. We can create it like this:

$folderPath = "C:\MySite"
if(!(Test-Path -Path $folderPath )){
	Write-Verbose "Creating installation folder $folderPath"
	$Null = New-Item -ItemType directory -Path $folderPath
}

Then we need to setup an application pool for the website. It is also good idea to make sure few defaults are changed to better meet the purpose. We like to disable application pool idle timeout to get rid off unexpected expensive first hits. Instead we are using periodic restart to restart the pool when we are not expecting many customers.

Import-Module WebAdministration
New-WebAppPool -Name $appPoolName
# .NET runtime 
Set-ItemProperty "IIS:\AppPools\$appPoolName" -Name "managedRuntimeVersion" `
	-Value $appPoolDotNetVersion
# recycling
Set-ItemProperty "IIS:\AppPools\$appPoolName" -Name Recycling.periodicRestart.time `
	-value ([TimeSpan]::FromMinutes(1440)) # 1 day (default: 1740)
# Clear existing values if any
Clear-ItemProperty "IIS:\AppPools\$appPoolName" -Name Recycling.periodicRestart.schedule 
Set-ItemProperty "IIS:\AppPools\$appPoolName" -Name Recycling.periodicRestart.schedule `
	-Value @{value="$appPoolRestartTime"}
Set-ItemProperty "IIS:\AppPools\$appPoolName" -Name processModel.idleTimeout `
	-value ([TimeSpan]::FromMinutes(0)) # Disabled (default: 20)
# logs from recycling to event log
$recycleLogEvents = "Time,Requests,Schedule,Memory,IsapiUnhealthy,OnDemand,ConfigChange,PrivateMemory"
Set-ItemProperty "IIS:\AppPools\$appPoolName" -Name Recycling.logEventOnRecycle `
	-Value $recycleLogEvents

After that we can create the website. Below there is an example how to create website with possible multiple bindings:

New-WebSite -Name $siteName  `
	-Port 80 `
	-HostHeader $siteName `
	-PhysicalPath $physicalPath  
Set-ItemProperty IIS:\Sites\$siteName -name applicationPool `
	-value $appPoolName
# set bindings (go through all the bindings and create new webbinding for each)
foreach($binding in $bindings)
{
	Write-Verbose "Adding binding $binding"
	$bindingProtocol = "http"
	$bindingIP = "*"
	$bindingPort = "80"
	$bindingHostHeader = $binding.Hostname
	$bindingCreationInfo = New-WebBinding -Protocol $binding.protocol `
		-Name $siteName `
		-IPAddress $bindingIP `
		-Port $bindingPort `
		-HostHeader $bindingHostHeader
}

WebDeploy

Once our website is happily set we can easily configure it for WebDeploy with the WebPI plugin we installed earlier. Scripts for WebDeploy can be found under the installation directory of the product. The scripts take care of problems like giving permissions to the iis site folder for webdeploy, creating user if it does not exist and setting up the WebDeploy configurations. Here is how to use the script with one long oneliner:

& (Join-Path "$env:programfiles" "IIS\Microsoft Web Deploy V3\scripts\SetupSiteForPublish.ps1") `
	-siteName $siteName `
	-siteAppPoolName $appPoolName `
	-deploymentUserName $wdeployUser `
	-deploymentUserPassword $wdeployUserPw `
	-managedRuntimeVersion $appPoolDotNetVersion

Summing up

Instead of huge scripts I like to put my PowerShell stuff to modules and also I like to be able to configure my installations. Thus I created few modules, an example script and an example configuration. I am using XML configuration because it was the way to go earlier with the PowerShell (it was easy to load unlike JSON). Now newer PowerShell has “ConvertFrom-JSON” function that makes it possible to use also JSON configuration.

Here is how to load an XML file:

[xml]$config = get-content $configFile

After the variable with XML file contents is loaded it is easy to access different elements inside it. For example if you would like to access the name of the WebDeploy user it might look something like this:

 $config.Root.IIS.WebDeploy.GetAttribute("Name")

With all this done you could also enable PowerShell remoting on your server and do all the installations without using any RDP connection at all.

Security controls in continuous integration

Fri, 11 Dec 2015 01:00:00 GMT

Continuous integration and continuous delivery have been trendy topics lately. Among other things it is necessary to take care of security in your development process. I want to share few ideas how you can enhance your security in continuous integration in EPiServer CMS projects. Need for speed is a Finnish foundation and its purpose is to find new ways to deliver real time value for customers of Finnish software companies. There was a Q4 review this week at Rovaniemi and my blog post will be about my presentation at there.

Problem domain

In this picture from OWASP it is important to understand that when talking about security, the threat agents in the picture will be trying to effect your business. By understanding the motive of the attacker it will be much easier to estimate what he or she is trying to achieve. Of course there can be natural disasters that are mindless threats but for example botnets are always built for some purpose.

In continuous integration we want to find possible security vulnerabilities and make the build fail to get instant feedback about security problem for the developer.

Security testing as whole

Diagram above is from OWASP Testing guide 4. It presents secure development lifecycle (Microsoft has its own too). There are many things that can’t be handled with automatisation like going through customer security policies, learning industrial standards and creating security architecture but nevertheless there are areas that can be partly automated.

Different tools tested

We tested various tools and divided them to different categories.

Static code analysis

In this category there are tools that go through source code and possibly also the application configuration. They look for problems in code and are meant for ensuring the code quality. They are capable of finding SQL injection possibilities, unvalidated parameters, memory leaks and security misconfigurations but still security testing is not the core idea behind these tools.

FxCop
VisualCodeGrepper
SonarQube
ReSharper commandlinetools

Code quality metrics

Code quality metrics is all about code quality. By having good quality enhances your security indirectly. Complex software is harder to test and bad code is more likely to contain security holes. Duplicated code easily leads to fact that problems are only fixed in one place and left open to another place. Having bugs in your code affects integrity and availability of the software and data. As you might know, the security is often modeled as CIA triangle (confidentiality, integrity and availability).

SonarQube
Code metrics

Configuration and deployment analysis

These tools will test how your system has been set up. MBSA checks if your installation follows best practices and ASA checks what is the attack surface of your application.

Microsoft Baseline Security Analyzer
Attack Surface Analyzer

Vulnerability scanning

These tools are meant for security professionals. Nessus is a network scanner that shines in finding known vulnerabilities in operating systems and application servers. OWASP ZAP is meant for web application security testing. Both of these test directly the security of your application. You should be careful when using these since you might be acting against the law if you scan targets without their permission.

Nessus
OWASP Zed Attack Proxy

Performance testing

Since availability is in the core of security, the performance testing will help you in finding bottlenecks that attacker could use to amplify his or her DOS Denial of Service attack.

jMeter

How do these tools sit on secure development lifecycle?

As it can be clearly seen in the picture the tools can help you in development, deployment and maintenance.

In which layers these tools work

This picture tries to tell you something about defence in depth. Choosing tools only from one category will not be enought to cover all aspects of the security. Also keep in mind that the attacker won’t care if the vulnerability is in the host or in the application. This means that seemless cooperation between development and operations is needed; thus you should embrace DevOps in your workplace to cover these gaps.

How do these tools mitigate OWASP TOP 10 threats

OWASP TOP 10 is a list of common web application vulnerabilities. As it can be seen in the picture the OWASP ZAP is a tool that is created to mitigate these vulnerabilities. Nessus and static code analysis tools can also be helpful. Truth to be told some of these threats are really generic ones. Security misconfiguration for example can mean pretty much anything (network devices, operating systems, application servers, other services in the operating system, etc). Missing function level authorization instead is so tightly coupled to application logic that it is hard to test automatically; the only option is to write your own tests against that.

How do these tools mitigate Cloud Security Alliances “Notorious Nine”

Notorious nine is Cloud Security Alliance’s list of most common threats for cloud computing. In the picture above you can pretty clearly see that most of the threats can’t be mitigated in CI with automized tools. Instead you should read carefully how CSA suggests you to mitigate the threats. Most of the threats are handled with security policies inside companies and only way to truly mitigate them is to educate your employees to be aware of these threats.

How useful our few example projects felt that the tools are in security perspective

This one is interesting. You could think that OWASP ZAP and Nessus would have been better here. But the truth is that the amount of vulnerabilities found with these tools was not big when compared the effort you need to have to use them continuously in continuous integration. Nessus for example was out of the case in projects because hosting was done by 3rd party in these example projects and it makes it hard to make continuous network scans since monitoring of the 3rd party company will be disturbed by this kind of network activity. OWASP ZAP instead reports a great deal of false positives and it is meant to be used manually so its usage in CI is harder than you would think. Example for using it from Jenkins is available at GitHub.

Also when using only OWASP TOP 10 and CSA Notorious nine the usefulness of tools is twisted. The accuracy is great deal better in static code analysis, they can be run a lot faster and tuning them for your needs is easier. OWASP-ZAP can take a lot of time if you run it in a manner that spiders your website, scans the found urls and gathers the report. We have projects that have several thousands of pages which means that going through all of them will really take some time.

How does it feel to use these tools

In the picture there is example of SonarQube dashboard. You might think that there is a great deal of major issues in the code in the project that was scanned with SonarQube. I used three different static code analysis to get that dashboard (ReSharper commandlinetools, FxCop and SonarQubes own static code analysis). 600 of major issues are because of our software didn’t have good enough commentation ratio (above 25% of code should be comments). I personally don’t trust that having that much comments is a good thing; instead readability of code should be embraced and only necessary things should be commented. This way the comments won’t become noise in your eyes and you will keep on reading them.

Well the bottom line is that from the around 6000 reported issues there was maybe few dozen real ones. With all the automized tools you need to spare some time to tune the tools to only report about issues you are really interested on.

Recommendation for implementing continuous security

Try to run static code analysis with few different tools. See what they can find and reflect that to your security needs. Take atleast one into your CI build after tuning it into your needs.

Calculate code metrics and see what they say about your software. My recommendation would be to use SonarQube as a code quality platform that keeps at track how your code quality develops. It will be a difficult task to truly bring down the CI build with these metrics instead you will like to watch how the metrics develop over time.

Try to scan your application manually with few different tools (MSBA, OWASP ZAP, Burp suite, nmap, nessus) and again see what they can find and think if you want to make these checks continuous. You should for example create nightly builds and/or continuous security checks separated from development if they are taking too much time. By learning the tools you can also make only specific tests in CI which would make sure that critical parts of application are secure.

Above all else remember to educate your personnel!

Installing development environment with PowerShell

Fri, 04 Dec 2015 01:00:00 GMT

I will not lie: I love PowerShell! Lately I have been trying to figure out how we can ease the life of .NET developers in EPiServer projetcs. PowerShell and NuGet has been keywords on that topic. Now I want to show you how PowerShell can ease your life! Open your hearth and let the PowerShell flow within!

What are the identified problems

We have few things that are happening all the time.

We are getting new people on our team
We are upgrading our development machines to new ones
We are changing from one project to another
Creation of virtual machines for various reasons is common, they need some tools too

All these above will have some kind of impact of installating software, updating components and configuration of various things. I really wanted to cut down the costs that our company and/or our customers are having because of this.

Enhancement 1: Workstation installation script

Our awesome IT department (you guys are really awesome!) has of course workstation images that they use when they deliver workstations to all employees. Nevertheless there are so many kind of development happening that they don’t have time to support all the variations that different developer archetypes would like to have. In some cases developers won’t even use same tools in same project.

To solve this issue we created a PowerShell Chocolatey script that installs all the wanted tools for .NET developers (others can do their own scripts). Developer can himself comment out unwatend tools or add some others.

Here is an example of it:

Enchancement 2: Project specific environment setup

We have currently many EPiServer projects going on. Common for these projects is the need to support multiple different bindings for the same IIS site. If we bind to *:80 then we are able to develop only one project per machine and we have need to support multiple projects at the same time. Some of the sites have also need to install some certificates for integrations and to create test sites and all other stuff. To achieve this we created a PowerShell Module that can be used.

List of needed functionality

IIS site creation
Application pool creation
IIS site bindings creation
Certificate creation (e.g. for encryption)
Certificate installation
Test that all mandatory .NET and IIS stuff is installed correctly

And here is how it looks like:

What next

As having one PowerShell Module that all projects are dependant might be burdensome; we are inspecting possiblity to deliver this module via NuGet for each project independently. Project could then decide to which version of Module it is dependant. But that would be then a subject for another blog post.

Cool! I want to do that too!

Because we are such a nice guys we put all the scripts into GitHub for everyone to access. You can get them from our GitHub repository

Hello BadUSB

Mon, 21 Sep 2015 02:00:00 GMT

There was a big security flaw in a 2014 Jeep Cherokee. Chrysler didn't have a way to patch cars over the air so they mailed 1.4 million USB drives via the US Postal Service. By doing that they teached their customers to trust USB drives that are delivered by mail. What security professionals are fearing is of course malicious USB devices. Let's peek into world of BadUSB.

What is BadUSB?

BadUSB can for example upload malware to your computer, redirect your internet traffic or pretend to be a keyboard. BadUSB can be any USB device. It is possible for hackers to reprogram micro-controllers of normal devices or cloak their malicious devices to look like regular ones. Probably the most famous use of USB drives in a cyberattack was stuxnet. USB drives were used for getting worm into computers that were not connected to the other world.

How do I get one?

I ordered my BadUSB from hakshop. It is called USB Rubber Ducky Deluxe and it is build for penetration testing purposes for security professionals. In short it is a USB drive that acts as a USB Human Interface Device. When you plug it into your device it presents itself as a keyboard. Within a package there is the BadUSB itself, microSD card and microSD card reader. Because it is a keyboard it works with any device that supports USB keyboard (Windows, Mac, Linux, Android, etc). The payload has to be chosen by knowing the target system because the same keyboard injection won't do desired things in different operating systems.

The device itself looks just like an USB drive

From inside the Ducky is actually embedded device with microSD reader

How to script Ducky

Ducky has its own script language called DuckyScript. It is simple language which allows you to define what keyboard actions you want to script. There is no actual IDE for it but community has created an User Defined Language for Notepad++ which can be found from hak5 forums. My first payload script simply opened PowerShell with Win-R shortcut and downloaded an executable which was then executed.

The script itself is quite self-explanatory.

At the top of the script there are few lines of comments with REM keyword.
Then DELAY for 10s is added because Ducky was anxious to start executing the script before Windows has finished installing HID drivers thus my payload failed without it.
WINDOWS r opens the run prompt and yet again delay is added for waiting the prompt to truly open.
Then PowerShell is written to prompt following with ENTER.
PowerShell is used to invoke a webclient command to download a file and store it under temp folder.
Finally the executable is started with Start-Process command in PowerShell.

There are plenty of ready scripts available. Here are few samples. There is also community made toolkit available. You should check it out since it allows you to generate scripts with certain payload. Toolkit also allows you to compile scripts on the internet but my precautioness didn't let me to test it out.

Building the script for Ducky

This was for me the trickiest part. Wiki tells you to download Duck Encoder. It was straightforwart to use.

java -jar duckencoder.jar -i exploit.txt -o /media/microsdcard/inject.bin

Although there was an annoying problem. It only supported US layout. The script didn't work unless you chose the correct keyboard layout from Windows first. Luckily community has solved this problem. There were newer versions of commandline compiler available. I just needed to pass new parameter for the encoder!

java -jar encoder.jar -i exploit.txt -o inject.bin -l resources\fi.properties

Taking BadUSB into action

After that there is only one more thing to do. To upload the payload for your device. You need to take microSD card from your Ducky device and upload inject.bin into device. After inserting microSD card back the device is ready to use. If you plan to buy multiple microSD cards you should know that there are pretty strict limitations what kind of microSD cards are eligible. The device can't handle big cards at all.

After injecting the BadUSB to a Windows machine the following things will happen.

Run prompt is shown and "powershell" is typed into it

PowerShell runs script to download an executable and then runs it

The executable itself was this time a pretty harmless one that opens notepad and prints an adorable ASCII dragon.

Why BadUSB is so dangerous?

Dangerousness of Human Interface Device USB is that all the user actions are always trusted. How do you bypass UAC in Windows? With user action. Also if the malicious code is injected into USB micro-controller there is no way to antivirus software to be able to scan it.

This kind of BadUSB device is pretty easy to notice because it didn't do what user expected (it wasn't an USB drive). With Ducky there are community made firmwares that can be both USB Drive and USB Keyboard at the same time. This way the victim could download e.g. PDF document from USB drive and after that you could just disrupt him when the HID keyboard starts to build a backdoor for you.

How to protect yourself from BadUSB?

Once your car manufacturer sends you an USB drive via postal service do not plug it in! It is simple as that. You should go to meet your local car service company and ask them to do the needed job. Also you should of course always lock your computer when not using it and never leave it into unsafe location unattended. In an open-plan office it could be pretty easy to add for example keylogger in between your keyboard and computer without you ever noticing it. Modern keyloggers are able to broadcast keypresses through WiFi to the attacker.

One way around would be to whitelist the allowed devices and deny all others. This might be burdensome especially if user does not have privileges to bypass this restriction. It is also common action to disable USB ports from devices that do not need them (e.g. servers). Although that is a bit extreme action and if somebody is in your server room with malicious USB devices the USB device itself is most likely least of your concerns.

Nevertheless the most important thing is to educate your personnel to be aware of this kind of threats. You should also support secure ways to transfer from files from computer to another to ensure that use of USB drives is minimized.

Blog posts by Joona Immonen

Continuous delivery with the Episerver DXC Service

Continuous delivery (CD) and continuous integration (CI)

Continuous integration:

Continuous delivery:

the DXC Service

Our team and the process

Build phase

Dev phase

Test phase

Staging phase

Prod phase

Building the build pipeline

Build trigger

Build and octopus release

Unit tests

Deployment with Octopus Deploy

Smoke testing

Email deployments

Triggering smoke tests

The nightly stuff

Show it on the TV

Summarizing

Continuous delivery with the Episerver DXC Service

Continuous delivery (CD) and continuous integration (CI)

Continuous integration:

Continuous delivery:

the DXC Service

Our team and the process

Build phase

Dev phase

Test phase

Staging phase

Prod phase

Building the build pipeline

Build trigger

Build and octopus release

Unit tests

Deployment with Octopus Deploy

Smoke testing

Email deployments

Triggering smoke tests

The nightly stuff

Show it on the TV

Summarizing

Installing Jenkins with PowerShell DSC

About the PowerShell DSC

DSC resources

Our objective

Starting the scripting

All the needed dependencies

Installing custom made stuff

The ugly

Setup Jenkins startup parameters

Initial authentication setup for Jenkins

Start the Jenkins

Install plugins

Install IIS

Setup a website for proxying

UrlRewrite and ApplicationRequestRouting

Conclusion

Known problems

Achieving Azure certification after ARM refresh

Overall feeling about the exams

About the certificates and the ARM refresh

No more recertification

Three exams, seven certificates

70-53x overlaps with each other

Hot topics

What to expect from 70-532 Developing Microsoft Azure solutions

What to expect from 70-533 Implementing Microsoft Azure Infrastructure solutions

What to expect from 70-534 Architecting Microsoft Azure solutions

How to study

Why?

How was this related to Episerver?

How much effort for this all?

Episerver developer meetup at Solita

Commerce

Performance

DevOps