How to automate Octopus Deploy upgrades

Automating the Octopus Deploy upgrade ensures all essential steps are executed during an upgrade and can reduce the outage window to be around 5-10 minutes. This guide provides the steps necessary to automate the upgrade process.

Overview

This guide was written for upgrading Octopus Deploy on Windows.

Prep work

Before going down the automation path, it is critical to back up the master key and license key. If anything goes wrong, you might need these keys to do a restore. It is better to have the backup and not need it than need the backup and not have it. The master key doesn’t change, while your license key changes, at most, once a year. Back them up once to a secure location and move on.

Backup the Master Key.
Backup the License Key.

Backup the Octopus Master Key

Octopus Deploy uses the Master Key to encrypt and decrypt sensitive values in the Octopus Deploy database. The Master Key is securely stored on the server, not in the database. If the VM hosting Octopus Deploy is somehow destroyed or deleted, the Master Key goes with it.

To view the Master Key, you will need login permissions on the server hosting Octopus Deploy. Once logged in, open up the Octopus Manager and click the view master key button on the left menu.

Save the Master Key to a secure location, such as a password manager or a secret manager.

An alternative means of accessing the Master Key is to run the Octopus.Server.exe show-master-key from the command line. Please note: you will need to be running as an administrator to do that.

Backup the License Key

Like the Master Key, the License Key is necessary to restore an existing Octopus Deploy instance. You can access the License Key by going to Configuration ➜ License. If you cannot access your License Key, please contact support@octopus.com and they can help you recover it.

Upgrading single node Octopus Deploy instances

A single node Octopus Deploy instance is an instance not configured for high availability. The instance is running on a single Windows Server, and as such, you can run this script to:

Check for a new version (exit if the current version is the newest version).
Enable maintenance mode.
Backup all the files if it is a major version change.
Stop the instance.
Backup the database.
Download and install the MSI.
Upgrade the database.
Start the instance back up.

Depending on the duration of the upgrade, your Octopus Server may still be starting up when the script completes. It will still be in maintenance mode, giving you a chance to log in to the Octopus Web Portal and verify things are working as expected.

Replace the variable values at the start of the script with ones applicable to your installation.

$url = 'https://samples.octopus.app'
$apiKey = "API-YOUR-KEY"
$octopusDeployDatabaseName = "OctopusDeploy"
$sqlBackupFolderLocation = "\\ServerStorage\Share\DatabaseBackup"
$fileBackupLocation = "\\ServerStorage\Share\FileBackup"
$downloadDirectory = "${env:Temp}"

# This is the default install location, but yours could be different
$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"
$serverExe = "$installPath\Octopus.Server.Exe"

# Get the latest minor/patch version
$currentVersion = (Invoke-RestMethod "$Url/api").Version
$versions = Invoke-RestMethod "https://octopus.com/download/upgrade/v3"
$upgradeVersion = $versions[-1].Version

if ($upgradeVersion -eq $currentVersion) {
    Write-Host "No new versions found. Quitting..."
    exit
}

# Download the installer
$msiFilename = "Octopus.$upgradeVersion-x64.msi"
Write-Host "Downloading $msiFilename"
Start-BitsTransfer -source "https://download.octopusdeploy.com/octopus/$msiFilename" -Destination "$downloadDirectory\$msiFilename"

# Place Octopus into maintenance mode
if (-not (Invoke-RestMethod -Uri "$url/api/maintenanceconfiguration" -Headers @{'X-Octopus-ApiKey' = $apiKey}).IsInMaintenanceMode) {
    Invoke-RestMethod `
        -Method Put `
        -Uri "$url/api/maintenanceconfiguration" `
        -Headers @{'X-Octopus-ApiKey' = $apiKey} `
        -Body (@{ Id = "maintenance"; IsInMaintenanceMode = $true } | ConvertTo-Json)
}

$versionSplit = $currentVersion -Split "\."
$upgradeSplit = $upgradeVersion -Split "\."

if ($versionSplit[0] -ne $upgradeSplit[0])
{
    Write-Host "Major version upgrade has been detected, backing up all the folders"

    $serverFolders = Invoke-RestMethod -Uri "$url/api/configuration/server-folders/values" -Headers @{'X-Octopus-ApiKey' = $apiKey}

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.LogsDirectory) $fileBackupLocation\TaskLogs /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackupLocation\TaskLogs"
    }

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.ArtifactsDirectory) $fileBackupLocation\Artifacts /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackupLocation\Artifacts"
    }

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.PackagesDirectory) $fileBackupLocation\Packages /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackupLocation\Packages"
    }

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.EventExportsDirectory) $fileBackupLocation\EventExports /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackupLocation\EventExports"
    }
}

# Finish any remaining tasks and stop the service
& $serverExe node --instance="OctopusServer" --drain=true --wait=0
& $serverExe service --instance="OctopusServer" --stop

try{
    # Backup database
    $backupFileName = "$octopusDeployDatabaseName_" + (Get-Date -Format FileDateTime) + '.bak'
    $backupFileFullPath = "$sqlBackupFolderLocation\$backupFileName"

    # OctopusServer is the default instance name.  If you have multiple instances, or are not using the default instance name, change the --instance parameter.
    $instanceConfig = (& $serverExe show-configuration --instance="OctopusServer" --format="JSON") | Out-String | ConvertFrom-Json
    
    $sqlConnection = New-Object System.Data.SqlClient.SqlConnection
    $sqlConnection.ConnectionString = $instanceConfig.'Octopus.Storage.ExternalDatabaseConnectionString'

    $command = $sqlConnection.CreateCommand()
    $command.CommandType = [System.Data.CommandType]'Text'
    $command.CommandTimeout = 0

    Write-Host "Opening the connection"
    $sqlConnection.Open()

    $command.CommandText = "BACKUP DATABASE [$octopusDeployDatabaseName]
    TO DISK = '$backupFileFullPath'
        WITH FORMAT"
    $command.ExecuteNonQuery()

    Write-Host "Successfully backed up the database $octopusDeployDatabaseName"
    Write-Host "Closing the connection"
    $sqlConnection.Close()
}
catch {
    Write-Error $_.Exception
    exit 1
}

# Running the installer
$msiToInstall = "$downloadDirectory\$msiFilename"
Write-Host "Installing $msiToInstall"
$msiExitCode = (Start-Process -FilePath "msiexec.exe" -ArgumentList "/i $msiToInstall /quiet" -Wait -PassThru).ExitCode 
Write-Output "Server MSI installer returned exit code $msiExitCode" 

# Upgrade database and restart service
# OctopusServer is the default instance name.  If you have multiple instances, or are not using the default instance name, change the --instance parameter.
& $serverExe database --instance="OctopusServer" --upgrade
& $serverExe service --instance="OctopusServer" --start
& $serverExe node --instance="OctopusServer" --drain=false

Remove-Item "$downloadDirectory\$msiFilename"

Upgrading High Availability Octopus Deploy instances

Automating the upgrade of a Highly Available Octopus Deploy instance requires more than a single script. A degree of coordination is required to update all the nodes. In addition, specific actions, backing up the database, upgrading the database, enabling / disabling maintenance mode should only happen once.

The recommendation is to use an Octopus Deploy runbook on another instance to upgrade the High Availability instance. You can get a 30-day trial license to do this with an Octopus Cloud instance.

Each node will need a Tentacle installed on it. You will need two roles for this to work.

HAServer: All Tentacles will be assigned to this.
HAServer-Primary: This is the server which does the majority of the work (checking for new versions, upgrading the database, etc).

The same sample script from above will be used, but it will be broken up into steps. For example:

Variables

As this is a runbook, you’ll want to create variables to share across the various steps.

Upgrade.Octopus.Url - the URL of your instance, for example https://samples.octopus.app.
Upgrade.Octopus.ApiKey - the API key of a service account with permissions to turn on and off maintenance mode. Please be sure to make this a sensitive variable!
Upgrade.Download.Folder - The network location of the shared folder to download the MSI to, for example \\YOUR-NAS\ShareName
Upgrade.Database.Backup.Folder - The network location of the shared folder to backup the database to, for example \\YOUR-NAS\DatabaseBackupShare
Upgrade.Database.Name - The name of your Octopus Deploy database. This is used for the backup script.
Upgrade.File.Backup.Folder - The folder name to store the file backups in. For example \\YOUR-NAS\FileBackupShare
Upgrade.Octopus.HasNewVersion - Output variable set in the first step. All other steps will use this variable as a run condition. Example value: #{unless Octopus.Deployment.Error}#{Octopus.Action[Check for new Octopus Version to Download].Output.UpgradeFound}#{/unless}
Upgrade.Download.Msi - Output variable storing the MSI name that is being downloaded. Example value: #{Octopus.Action[Check for new Octopus Version to Download].Output.MSIToDownload}

Process

The upgrade process itself is very similar to upgrading a single node instance. The key difference is the script is broken up into multiple steps.

Aside from step 1, all steps should set a run condition to look at the variable Upgrade.Octopus.HasNewVersion

1. Check for a new version (HAServer-Primary).

# Get the latest minor/patch version
$url = $OctopusParameters["Upgrade.Octopus.Url"]
$downloadFolder = $OctopusParameters["Upgrade.Download.Folder"]

$upgradeMsiList = Get-ChildItem -Path "$downloadFolder\*" -Include *.msi
if ($upgradeMsiList.Count -gt 0)
{
    Set-OctopusVariable -Name "UpgradeFound" -Value $true
    Write-Host "MSIs already exist in the download directory, exiting so they can be installed"
    exit
}

$currentVersion = (Invoke-RestMethod "$Url/api").Version
$currentMajorVersion = $currentVersion.Split('.')
$versions = Invoke-RestMethod "https://octopus.com/download/upgrade/v3" `
    | Where-Object { $_.Version.StartsWith($currentMajorVersion + '.') }
$upgradeVersion = $versions[-1].Version

if ($upgradeVersion -eq $currentVersion) {
    Set-OctopusVariable -Name "UpgradeFound" -Value $false
    Write-Host "No new versions found. Quitting..."
    exit
}

# Download the installer
$msiFilename = "Octopus.$upgradeVersion-x64.msi"
Set-OctopusVariable -Name "MSIToDownload" -value $msiFileName
$outFile = "$downloadFolder\$msiFilename"

if (Test-Path $outFile)
{
    Set-OctopusVariable -Name "UpgradeFound" -Value $true
    Write-Host "The latest version has already been download and is waiting to be installed"
    exit
}

Write-Host "Downloading $msiFilename"
Start-BitsTransfer -source "https://download.octopusdeploy.com/octopus/$msiFilename" -destination $outFile
Set-OctopusVariable -Name "UpgradeFound" -Value $true

2. Put the server into maintenance mode (HAServer-Primary).

$url = $OctopusParameters["Upgrade.Octopus.Url"]
$apiKey = $OctopusParameters["Upgrade.Octopus.ApiKey"]

# This is the default install location, but yours could be different
$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"

# Check to see if this is a re-run, if all the nodes are stopped then this will fail
try
{
    $octopusApi = Invoke-RestMethod -uri "$url/api"
}
catch
{
    Write-Host "Error calling api endpoint for $url, exiting"
    exit
}

# Place Octopus into maintenance mode
if (-not (Invoke-RestMethod -Uri "$url/api/maintenanceconfiguration" -Headers @{'X-Octopus-ApiKey' = $apiKey}).IsInMaintenanceMode) {
    Invoke-RestMethod `
        -Method Put `
        -Uri "$url/api/maintenanceconfiguration" `
        -Headers @{'X-Octopus-ApiKey' = $apiKey} `
        -Body (@{ Id = "maintenance"; IsInMaintenanceMode = $true } | ConvertTo-Json)
}

3. Backup files if major version change (HAServer-Primary).

$url = $OctopusParameters["Upgrade.Octopus.Url"]
$apiKey = $OctopusParameters["Upgrade.Octopus.ApiKey"]
$fileBackupFolder = $OctopusParameters["Upgrade.File.Backup.Folder"]
$msiToDownload = $OctopusParameters["Upgrade.Download.Msi"]

$currentVersion = (Invoke-RestMethod "$Url/api").Version
$versionSplit = $currentVersion -Split "\."

$upgradeSplit = $msiToDownload -Split "\."

if ($versionSplit[0] -ne $upgradeSplit[1])
{
    Write-Host "Major version upgrade has been detected, backing up all the folders"

    $serverFolders = Invoke-RestMethod -Uri "$url/api/configuration/server-folders/values" -Headers @{'X-Octopus-ApiKey' = $apiKey}

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.LogsDirectory) $fileBackUpFolder\TaskLogs /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackUpFolder\TaskLogs"
    }

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.ArtifactsDirectory) $fileBackUpFolder\Artifacts /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackUpFolder\Artifacts"
    }

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.PackagesDirectory) $fileBackUpFolder\Packages /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackUpFolder\Packages"
    }

    $msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.EventExportsDirectory) $fileBackUpFolder\EventExports /mir" -Wait -PassThru).ExitCode
    if ($msiExitCode -ge 8) 
    {
        Throw "Unable to copy files to $fileBackUpFolder\EventExports"
    }
}

4. Stop all nodes (HAServer).

# Upgrading the MSI on a single server updates all instances, shut them all down first
$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"
$serverExe = "$installPath\Octopus.Server.exe"

$instanceList = (& $serverExe list-instances --format="JSON") | Out-String | ConvertFrom-Json
Write-Host "Found $($instanceList.length) Octopus instances"

foreach ($instance in $instanceList)
{
    Write-Host "Stopping $($instance.InstanceName)"
    # Finish any remaining tasks and stop the service
    & $serverExe node --instance="$($instance.InstanceName)" --drain=true --wait=0
    & $serverExe service --instance="$($instance.InstanceName)" --stop
}

5. Backup Database (HAServer-Primary)

$BackupFolderLocation = $OctopusParameters["Upgrade.Database.Backup.Folder"]
$OctopusDatabaseName = $OctopusParameters["Upgrade.Database.Name"]

$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"
$serverExe = "$installPath\Octopus.Server.exe"
$backupFileName = "$OctopusDatabaseName_" + (Get-Date -Format FileDateTime) + '.bak'
$backupFileFullPath = "$backupFolderLocation\$backupFileName"

# OctopusServer is the default instance name.  If you have multiple instances, or are not using the default instance name, change the --instance parameter.
$instanceConfig = (& $serverExe show-configuration --instance="OctopusServer" --format="JSON") | Out-String | ConvertFrom-Json
   
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection
$sqlConnection.ConnectionString = $instanceConfig.'Octopus.Storage.ExternalDatabaseConnectionString'

$command = $sqlConnection.CreateCommand()
$command.CommandType = [System.Data.CommandType]'Text'
$command.CommandTimeout = 0

Write-Host "Opening the connection"
$sqlConnection.Open()

Write-Host "Backing up the database"
$command.CommandText = "BACKUP DATABASE [$OctopusDatabaseName]
  TO DISK = '$backupFileFullPath'
      WITH FORMAT"
$command.ExecuteNonQuery()

Write-Host "Successfully backed up the database $octopusDatabaseName"
Write-Host "Closing the connection"
$sqlConnection.Close()

6. Install the MSI (HAServer).

$downloadFolder = $OctopusParameters["Upgrade.Download.Folder"]
$upgradeMsiList = Get-ChildItem -Path "$downloadFolder\*" -Include *.msi
if ($upgradeMsiList.Count -le 0)
{
    Write-Host "No MSIs found, exiting step"
    exit
}

$msiToInstall = $upgradeMsiList[0]
# Running the installer
Write-Host "Installing $msiToInstall"
$msiExitCode = (Start-Process -FilePath "msiexec.exe" -ArgumentList "/i $msiToInstall /quiet" -Wait -PassThru).ExitCode 
Write-Output "Server MSI installer returned exit code $msiExitCode"

7. Upgrade the database (HAServer-Primary).

$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"
$serverExe = "$installPath\Octopus.Server.exe"

# OctopusServer is the default instance name.  If you have multiple instances, or are not using the default instance name, change the --instance parameter.
& $serverExe database --instance="OctopusServer" --upgrade

As of 2023.2.9755, Octopus will terminate a database upgrade if running nodes are detected. Sometimes the nodes simply haven’t exited cleanly. And it might take a few seconds for Octopus to recognize this. It’s possible to use a simple retry loop to handle this automatically.

$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"
$serverExe = "$installPath\Octopus.Server.exe"

$maxRetry = 3
$retryInterval = 10
$attempts = 0

While ($true) {
    $attempts++

    if ($attempts -gt $maxRetry) {
        Write-Error "Upgrade failed after $maxRetry retries"
        exit 1
    }

    # OctopusServer is the default instance name.  If you have multiple instances, or are not using the default instance name, change the --instance parameter.
    & $serverExe database --instance="OctopusServer" --upgrade
    if ($LastExitCode -ne 0) {
        Write-Warning "Upgrade failed. Retrying in $retryInterval seconds..."
        Start-Sleep -Seconds $retryInterval
    }
}

8. Restart all nodes (HAServer).

# A server could have multiple instances, as we shut them all down earlier, start them all back up
$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"
$serverExe = "$installPath\Octopus.Server.exe"

$instanceList = (& $serverExe list-instances --format="JSON") | Out-String | ConvertFrom-Json
Write-Host "Found $($instanceList.length) Octopus instances"

foreach ($instance in $instanceList)
{
    Write-Host "Starting $($instance.InstanceName)"
    # Finish any remaining tasks and stop the service
    & $serverExe service --instance="$($instance.InstanceName)" --start
    & $serverExe node --instance="$($instance.InstanceName)" --drain=false
}

Triggers and notifications

Now that the upgrade process is in a runbook, you can create a trigger to check once a day or week. In addition, you can set up notifications to notify you when a new version is found and when you need to disable maintenance mode. How you use the runbook is up to you.

Help us continuously improve

Please let us know if you have any feedback about this page.

Send feedback

Page updated on Monday, November 20, 2023

Edit on GitHub