Automating Docker Deployment with Azure Resource Manager

Recently, I had to build a solution where Docker container were appropriate. The idea behind the container is that once there are built you just have to run it. While it's true, my journey was not exactly that, nothing dramatic, only few gotchas that I will explain while sharing my journey.

The Goal

The solution is classic, and this post will focus on a single Virtual Machine (VM). The Linux VM needs a container that automatically runs when the VM starts. Some files first download from a secure location (Azure blob storage) then passed to the container. The solution is deployed using Azure resources manager (ARM). For the simplicity, I will use Nginx container but the same principle applies to any container. Here is the diagram of the solution.

The Solution

I decided to use Azure CLI to deploy since this will be also the command-line used inside the Linux VM, but Azure PowerShell could do the same. We will be deploying an ARM template containing a Linux VM, and two VM-Extension: DockerExtension and CustomScriptForLinux. Once the VM is provisioned, a bash script will be downloaded by CustomScriptForLinux extension from the secure Azure blob storage myprojectsafe, then executed.

The Custom Script

This script is executed with elevated privileges and will do few things for us. Since it contains sensible information, it will be saved in a secure blob storage. I will explain how access it later, first let's have a look to the script itself.

#= mynginx.sh
echo "=== Starting  execution Custom script ==="
mkdir -p /home/docker-nginx/html
echo "... Folder created"
docker run -i -v /home/docker-nginx/html:/home microsoft/azure-cli azure storage blob download -a frankdockerdemo -k "MyGeneratedKEY==" --container myprojectsafe -b "index.html" -d /home
echo "... File downloaded"
docker run --name docker-nginx -p 80:80 -d -v /home/docker-nginx/html:/usr/share/nginx/html nginx
echo "=== End of Custom script ==="

The first command creates a folder where files will be downloaded. Then it downloads from Ducker Hub and installs 'microsoft/azure-cli' container. This container is very useful to execute Azure-CLI command-lines.

It usually gets installed with docker run -it microsoft/azure-cli However, if you execute the command this way, you will have two problems. The initial one, would be that the downloaded files will be in the container instead of the VM. To adjust the situation, we just need to mount the created folder to a folder inside the container with the parameter -v /home/docker-nginx/html:/home . The second problem will be an error related to TTY. We just need to remove the -t parameter to resolve that one. Then pass the Azure-CLI command azure storage blob download and specify in order the Azure Storage account name of the source, the security key to access that storage, the container name, the file name, and the destination folder.

The last command will download again from Ducker Hub and install 'nginx' container. NGINX is a free, open-source, high-performance HTTP server. Since we need to have this container to act as our web server, we will publish the port 80 and mount the folder where the HTML files got previously downloaded. One important detail is the path needs to be absolute, otherwise the file won't be found.
Once the file is complete, Upload it to a secure Azure blob storage, using the method of your choice.

The deployment

As mention earlier, this solution was deployed using ARM template and Azure-CLI. To get started with ARM, read my previous post Azure Resource Manager (ARM) for beginners. Assuming the subscription name is myproject and that the resource group name is frankdemo-rg in the East US region here the commands to be ready for the deployment.

# Login
azure login
# Set the working account to myproject
azure account set myproject
#Create a new Resource Group to deploy
azure group create frankdemo-rg eastus
# Deploy in the previously created RG
azure group deployment create --template-uri https://frankdockerdemo.blob.core.windows.net/deploymentpublic/azuredeploy.json -e azuredeploy.parameters.json -g frankdemo-rg -v

The last command is the one that really launches the deployment. It will use the template azuredeploy.json located in our public Azure storage, and azuredeploy.parameters.json that sits in a local folder and contains all the real values for that particular deployment.

Let's first talk about the template. I used the Azure Quickstart Templates Docker Simple On Ubuntu as base. It already contains a VM, a storage account, all the network plumbing we need, and the Docker Extension. What need to be added is the Custom Script For Linux extensions. This is easily done by adding the follow json node, at the end of the resources array.

{
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "name": "[concat(variables('vmName'), '/', variables('extCustScriptName'))]",
    "apiVersion": "2015-06-15",
    "location": "[resourceGroup().location]",
    "dependsOn": [
        "[resourceId('Microsoft.Compute/virtualMachines/extensions',variables('vmName'),variables('extensionName'))]"
    ],
    "properties": {
       "publisher": "Microsoft.OSTCExtensions",
       "type": "CustomScriptForLinux",
       "typeHandlerVersion": "1.4",
       "settings": {
           "fileUris": ["[parameters('fileUris')]"],
           "commandToExecute": "[parameters('commandToExecute')]"
       },
       "protectedSettings": {
           "storageAccountName": "[parameters('scriptstorageAccountName')]",
           "storageAccountKey": "[parameters('scriptstorageAccountKey')]"
       }
    }
}

The configuration of the extension sits in two sections: settings and protectedSettings. The setting will define where the script will be downloaded from and the command to execute. Since our script contains sensitive information, we put it in a secure Azure blob storage. In the protectedSettings we specify the storage account and the key to access it.

One particularity was the Custom Script extension needed to be added after the docker extension because the script is using docker. That usually done just by adding a reference in the dependsOn section. Nevertheless, in that situation, Resources Manager was not able to find the resource. I used resourceId() an Azure Resource Manager template functions passing resource type, resource name and the next resource name segment because the resource is nested.

Troubbleshooting

While that solution was working for me, you may encounter some problems in your journey to deploy your own solution. Let me share few tips that I learned in mine:

Validate the template before deploying

It may seam obvious but since a deployment can takes few or many minutes to deploy it's a really good thing to validate if before trying to deploy it. To validate your template using Azure-CLI here is the command-line to execute:

azure group template validate --template-uri https://frankdockerdemo.blob.core.windows.net/deploymentpublic/azuredeploy.json -e azuredeploy.parameters.json -g frankdemo-rg -v

Debug you custom script

If you deploy using the verbose option you will have some information, but must of the time you will need to connect to your VM in Azure to read the logs. To do so you will need to ssh simplefranktest.eastus.cloudapp.azure.com, as "simplefranktest" was the name I used for my dns name for public IP. The credentials are those you define as adminUsername/ adminPassword iin your parameter file. Once you are connected, your will find the log file in /var/log/azure/Microsoft.OSTCExtensions.CustomScriptForLinux/[Version]. And if the custom file got correctly downloaded it will be in /var/lib/waagent/Microsoft.OSTCExtensions.CustomScriptForLinux-[Version]/download/[Index]/. In my case [Version] was "1.4.1.0" and [Index] was "0" because I just used one script. To read their content you can use an editor like vi or nano.

Covert your base first

So much things can go wrong when deploying an entire solution. I would strongly suggest to read Troubleshoot common errors when deploying resources to Azure with Azure Resource Manager as it's a fantastic gold mine of information.

Wrapping up

Voila! If everything works as planned, once the VM is fully provisioned, you should be able to navigate to the dnsname and see a web page without having to do anything else.



Enjoy!

---

Here are the complete template and parameter file I used in this sample.

Complete Parameter File

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "newStorageAccountName": {
      "value": "frankteststorage"
    },
    "adminUsername": {
      "value": "frank"
    },
    "adminPassword": {
      "value": "_MyStrongPassword_"
    },
    "dnsNameForPublicIP": {
      "value": "simplefranktest"
    },
    "fileUris": {
      "value": "https://frankdockerdemo.blob.core.windows.net/myprojectsafe/mynginx.sh"
    },
    "commandToExecute": {
      "value": "sh mynginx.sh"
    },
    "scriptstorageAccountName": {
      "value": "frankdockerdemo"
    },
    "scriptstorageAccountKey": {
      "value": "MyGeneratedKEY=="
    }
  }
}

Complete template

{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
    "newStorageAccountName": {
    "type": "string",
    "metadata": {
        "description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed."
    }
    },
    "adminUsername": {
    "type": "string",
    "metadata": {
        "description": "Username for the Virtual Machine."
    }
    },
    "adminPassword": {
    "type": "securestring",
    "metadata": {
        "description": "Password for the Virtual Machine."
    }
    },
    "dnsNameForPublicIP": {
    "type": "string",
    "metadata": {
        "description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
    }
    },
    "ubuntuOSVersion": {
    "type": "string",
    "defaultValue": "14.04.4-LTS",
    "metadata": {
        "description": "The Ubuntu version for deploying the Docker containers. This will pick a fully patched image of this given Ubuntu version. Allowed values: 14.04.4-LTS, 15.10"
    },
    "allowedValues": [
        "14.04.4-LTS",
        "15.10"
    ]
    },
    "fileUris":{
    "type": "string",
    "metadata": {
        "description": "File to download after VM deployed."
    }
    },
    "commandToExecute":{
    "type": "string",
    "metadata": {
        "description": "Command To Execute"
    }
    },
    "scriptstorageAccountName":{
    "type": "string",
    "metadata": {
        "description": "Secure Storage Accountname"
    }
    },
    "scriptstorageAccountKey":{
    "type": "securestring",
    "metadata": {
        "description": "Secure storage Account Key."
    }
    }
},
"variables": {
    "imagePublisher": "Canonical",
    "imageOffer": "UbuntuServer",
    "OSDiskName": "osdiskfordockersimple",
    "nicName": "myVMNic",
    "extensionName": "DockerExtension",
    "extCustScriptName": "CustScriptExt",
    "addressPrefix": "10.0.0.0/16",
    "subnetName": "Subnet",
    "subnetPrefix": "10.0.0.0/24",
    "storageAccountType": "Standard_LRS",
    "publicIPAddressName": "myPublicIP",
    "publicIPAddressType": "Dynamic",
    "vmStorageAccountContainerName": "vhds",
    "vmName": "FrankVM",
    "vmSize": "Standard_D1",
    "virtualNetworkName": "MyVNET",
    "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]",
    "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]"
},
"resources": [
    {
    "type": "Microsoft.Storage/storageAccounts",
    "name": "[parameters('newStorageAccountName')]",
    "apiVersion": "2015-05-01-preview",
    "location": "[resourceGroup().location]",
    "properties": {
        "accountType": "[variables('storageAccountType')]"
    }
    },
    {
    "apiVersion": "2015-05-01-preview",
    "type": "Microsoft.Network/publicIPAddresses",
    "name": "[variables('publicIPAddressName')]",
    "location": "[resourceGroup().location]",
    "properties": {
        "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
        "dnsSettings": {
        "domainNameLabel": "[parameters('dnsNameForPublicIP')]"
        }
    }
    },
    {
    "apiVersion": "2015-05-01-preview",
    "type": "Microsoft.Network/virtualNetworks",
    "name": "[variables('virtualNetworkName')]",
    "location": "[resourceGroup().location]",
    "properties": {
        "addressSpace": {
        "addressPrefixes": [
            "[variables('addressPrefix')]"
        ]
        },
        "subnets": [
        {
            "name": "[variables('subnetName')]",
            "properties": {
            "addressPrefix": "[variables('subnetPrefix')]"
            }
        }
        ]
    }
    },
    {
    "apiVersion": "2015-05-01-preview",
    "type": "Microsoft.Network/networkInterfaces",
    "name": "[variables('nicName')]",
    "location": "[resourceGroup().location]",
    "dependsOn": [
        "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]",
        "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
    ],
    "properties": {
        "ipConfigurations": [
        {
            "name": "ipconfig1",
            "properties": {
            "privateIPAllocationMethod": "Dynamic",
            "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
            },
            "subnet": {
                "id": "[variables('subnetRef')]"
            }
            }
        }
        ]
    }
    },
    {
    "apiVersion": "2015-05-01-preview",
    "type": "Microsoft.Compute/virtualMachines",
    "name": "[variables('vmName')]",
    "location": "[resourceGroup().location]",
    "dependsOn": [
        "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
        "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
    ],
    "properties": {
        "hardwareProfile": {
        "vmSize": "[variables('vmSize')]"
        },
        "osProfile": {
        "computerName": "[variables('vmName')]",
        "adminUsername": "[parameters('adminUsername')]",
        "adminPassword": "[parameters('adminPassword')]"
        },
        "storageProfile": {
        "imageReference": {
            "publisher": "[variables('imagePublisher')]",
            "offer": "[variables('imageOffer')]",
            "sku": "[parameters('ubuntuOSVersion')]",
            "version": "latest"
        },
        "osDisk": {
            "name": "osdisk1",
            "vhd": {
            "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
            },
            "caching": "ReadWrite",
            "createOption": "FromImage"
        }
        },
        "networkProfile": {
        "networkInterfaces": [
            {
            "id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
            }
        ]
        }
    }
    },
    {
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "name": "[concat(variables('vmName'), '/', variables('extensionName'))]",
    "apiVersion": "2015-05-01-preview",
    "location": "[resourceGroup().location]",
    "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
    ],
    "properties": {
        "publisher": "Microsoft.Azure.Extensions",
        "type": "DockerExtension",
        "typeHandlerVersion": "1.0",
        "autoUpgradeMinorVersion": true,
        "settings": { }
    }
    },
    {
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "name": "[concat(variables('vmName'), '/', variables('extCustScriptName'))]",
    "apiVersion": "2015-06-15",
    "location": "[resourceGroup().location]",
    "dependsOn": [
        "[resourceId('Microsoft.Compute/virtualMachines/extensions',variables('vmName'),variables('extensionName'))]"
    ],
    "properties": {
        "publisher": "Microsoft.OSTCExtensions",
        "type": "CustomScriptForLinux",
        "typeHandlerVersion": "1.4",
        "settings": {
        "fileUris": ["[parameters('fileUris')]"],
        "commandToExecute": "[parameters('commandToExecute')]"
        },
        "protectedSettings": {
        "storageAccountName": "[parameters('scriptstorageAccountName')]",
        "storageAccountKey": "[parameters('scriptstorageAccountKey')]"
        }
    }
    }
]
}