How to copy files between Azure subscription from Windows, Linux, OS X, or the cloud

Labels: , , , , , , , , , , , , , , , , ,
(en français: ici)

Copy, Download or Upload from-to any combination of Windows, Linux, OS X, or the cloud


Data is and will always be our primary concern. Whether shaped as text files, images, VM VHDs or any other ways, at some point in time, our data will need to be moved. I already wrote about it previously, and the content of this post is still valuable today, but I wanted to share new options and convert all ground (meaning Linux, Windows and OS X).

Scenarios


Here few scenarios why you would want to move data.
  • Your Microsoft Azure trial is ending, and you wish to keep all the data.
  • You are creating a new web application, and all those images need to be moved to the Azure subscription.
  • You have a Virtual Machine that you would like to move to the cloud or to a different subscription.
  • ...

AZCopy


AzCopy is a fantastic command-line tool for copying data to and from Microsoft Azure Blob, File, and Table storage. At the moment, to write this post AzCopy is only available for Windows users. Another solution will be introduced later in this post for Mac and Linux users. Before AzCopy was only available on Windows. However, recently a second version built with .NET Core Framework is available. The commands are very similar but not exactly the same.

AzCopy on Windows


In his simplest expression, an AzCopy command looks like this:
AzCopy /Source:<source> /Dest:<destination> [Options]
If you earlier have installed an Azure SDK on your machine, you already have it. By default, AzCopy is installed to %ProgramFiles(x86)%\Microsoft SDKs\Azure\AzCopy (64-bit Windows) or %ProgramFiles%\Microsoft SDKs\Azure\AzCopy (32-bit Windows).

If you need only AzCopy for a server, you can download the latest version of AzCopy.
Let's see some frequent usage. First let's say you need do move all those images from your server to an Azure blob storage.
AzCopy /Source:C:\MyWebApp\images /Dest:https://frankysnotes.blob.core.windows.net/blog /DestKey:4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== /S
CopyAllImages

Then to copy those images to another subscription very easy.
AzCopy /Source:https://frankysnotes.blob.core.windows.net/blog /Dest:https://frankshare.blob.core.windows.net/imagesbackup /SourceKey:4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== /DestKey:EwXpZ2uZ3zrjEbpBGDfsefWkj3G2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA== /S

AzCopy Parameters


These examples were simple, but AzCopy is a very powerful tool. I invite you to type one of the following commands to discover more about using AzCopy:
  • For detailed command-line help for AzCopy: AzCopy /?
  • For command-line examples: AzCopy /?:Samples

AzCopy on Linux


Before you could install AzCopy you will need to install .Net Core. This is done very simply with few commands.
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
sudo apt-get update
sudo apt-get install dotnet-sdk-2.0.2
Then to install it, you just need to get it with a wget command, unzip it, and execute the install script.
wget -O azcopy.tar.gz https://aka.ms/downloadazcopyprlinux 
tar -xf azcopy.tar.gz 
sudo ./install.sh
In his simplest expression, the .Net Core version of AzCopy command looks like this:
azcopy --source <source> --destination <destination> [Options]
It is very similar to the original version, but parameters are using -- and - instead of the / and where a : was required, it's now a simple space.

Uploading to Azure


Here an example, to copy a single file GlobalDevopsBootcamp.jpg to an Azure Blob Storage. We pass the full local path to the file into --source, the destination is the full URI, and finally the destination blob storage key. Of course, you could also use SAS token if you prefer.
azcopy \
--source /home/frank/demo/GlobalDevopsBootcamp.jpg \
--destination https://frankysnotes.blob.core.windows.net/blog/GlobalDevopsBootcamp.jpg \
--dest-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA==

Copying Between Azure Subscriptions


To copy the image to a second Azure subscription, we use the command the source is now an Azure Storage URI, and we pass the source and the destination keys:
azcopy \
--source https://frankysnotes.blob.core.windows.net/blog/GlobalDevopsBootcamp.jpg \
--destination https://frankshare.blob.core.windows.net/imagesbackup/GlobalDevopsBootcamp.jpg \
--source-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== \
--dest-key EwXpZ2uZ3zrjEbpBGDfsefWkj3G2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA==

Azure CLI


Azure CLI is a set of cross-platform commands for the Azure Platform. It gives tools to manipulate all Azure components, but this post will focus on azure storage features.

There are two versions of the Azure Command-Line Interface (CLI) currently available:

  • Azure CLI 2.0: written in Python, conpatible only with the Resource Manager deployment model.
  • Azure CLI 1.0: written in Node.js, compatible with both the classic and Resource Manager deployment models.

Azure CLI 1.0 is deprecated and should only be used for support with the Azure Service Management (ASM) model with "classic" resources.

Installing Azure CLI


Let's start by installing Azure CLI. Of course, you can download an installer but since everything is evolving very fast with not getting it from Node Package Manager (npm). The install will be the same, you just need to specify the version if you absolutely need Azure CLI 1.0.

sudo npm install azure-cli -g

AzureCli_installed

To keep the previous scenario, let's try to copy all images to a blob storage. Unfortunately, Azure CLI doesn't offer the same flexibility as AzCopy,and you must upload the file one by one. However, to upload all images from a folder, we can easily put the command in a loop.

for f in Documents/images/*.jpg
do
   azure storage blob upload -a frankysnotes -k YoMjXMDe+694FGgOaN0oaRdOF6s1ktMgkB6pBx2vnAr8AOXm3HTF7tT0NQWvGrWnWj5m4X1U0HIPUIAA==  $f blogimages
done

azurecli_allimages

In the previous command -a was the account name, and -k was the Access key. This two information can easily be found in the Azure portal. From the portal (https://portal.azure.com), select the storage account. In the right band click-on Access keys.

StorageAccessKeys

To copy a file (ex: a VM disk aka VHD) from one storage to another one in a different subscription or region, it's really easy. This time we will use the command azure storage blob copy start and the -a and -k are related to our destination.

azure storage blob copy start 'https://frankysnotes.blob.core.windows.net/blogimages/20151011_151451.MOV' imagesbackup -k EwXpZ2uZ3zrjEbpBGDfsefWkj3GnuFdPCt2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA== -a frankshare

The nice thing about this command is that it's asynchronous. To see the status of your copy just execute the command azure storage blob copy show

azure storage blob copy show -a frankshare -k YoMjXMDe+694FGgOaN0oPaRdOF6s1ktMgkB6pBx2vnAr8AOXm3HTF7tT0NQVxsqhWvGrWnWj5m4X1U0HIPUIAA== imagesbackup 20151011_151451.MOV

CopyStatus1


CopyStatus2

Azure CLI 2.0 (Windows, Linux, OS X, Docker, Cloud Shell)


The Azure CLI 2.0 is Azure's new command-line optimized for managing and administering Azure resources that work against the Azure Resource Manager. Like the previous version, it will work perfectly on Windows, Linux, OS X, Docker but also from the Cloud Shell!


Cloud Shell is available right from the Azure Portal, without any plugging.

Uploading to Azure


The command if the same as the previous version except that now the command is named az. Here an example to upload a single file into an Azure Blob Storage.

az storage blob upload --file /home/frank/demo/CloudIsStrong.jpg \
--account-name frankysnotes \
--container-name blogimages --name CloudIsStrong.jpg \
--account-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA==

Copying Between Subscriptions


Let's now copy the file to another Azure subscription. A think to be aware is that --account-name and --account-key are for the destination, even if it's not specified.
az storage blob copy start \
--source-account-name frankysnotes  --source-account-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== \
--source-container blogimages --source-blob CloudIsStrong.jpg   \
--account-name frankshare  --account-key EwXpZ2uZ3zrjEbpBGDfsefWkj3G2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA== \
--destination-container imagesbackup  \
--destination-blob CloudIsStrong.jpg

In Video Please!


If you prefer, I also have a video version of that post.



One More Thing


Sometimes, we don't need to script things, and a graphic interface is much better. For this kind of situation, the must is the Azure Storage Explorer. It does a lot! Upload, download, and manage blobs, files, queues, tables, and Cosmos DB entities. And it works on Windows, macOS, and Linux!


It's just the beginning


This post was just an introduction to two very powerful tools. I strongly suggest to go read in the official documentation to learn more. Use the comment to share all your questions and suggestion.

References:

Reading Notes #322

Labels: , , , , , , , , , , , ,

IMG_20180328_194043_2Cloud


Databases


Miscellaneous


Books



Unfu*k Yourself: Get Out of Your Head and Into Your Life (Gary John Bishop)

I really enjoyed this book. Strong ideas. No repetition. It goes straight to the point. The narration is awesome.

ASIN: B0731QJ482














Reading Notes #321

Labels: , , , , , , , , , , , , , , , , , ,
ester-eggs-2345859_640

Suggestion of the week



Cloud



Programming



Databases



Miscellaneous


Books



When: The Scientific Secrets of Perfect Timing (Daniel H. Pink)

A really amazing book packed of very interesting advice. Things that you kind of already knew, or at least had a feeling you maybe knew are clearly explained to you.

After reading (or listening) this book, you will know why, and you can decide to fight it or change the when... improve your performance and use your time and energy on something else.

ISBN: 0525589333






Read more »

Reading Notes #320

Labels: , , , , , , , , , , , , , , , , ,
IMG_20180307_195017

Cloud


Programming


Data



Book

    Making Time () - I really enjoyed reading that book. I've been watching Bob's YouTube Channel for a while. ANd was happy to ear about his new book. In Making Time, a little bit like in his videos, Bob explains how he build stuff... but this time instead of a furniture or a room, it's a new career/ business. A short book (88 pages) but a lot of fun.  


Reading Notes #319

Labels: , , , , , , , , , , , , , , , , ,

kUz2asfCloud


Programming


Databases


Miscellaneous



Reading Notes #318 - MVP Summit 2018

Labels: , , , , , , , , , , , ,
CDNMVP2018

Last week, it was the 25 edition of the MVP Summit. An event, where Microsoft invites all his MVP to get to Seattle and spend some time with the products teams and learn on the latest news and best practices.

This year was particularly inspiriting by the Microsoft roadmap, of course,  but even more by all the amazing people a got the chance to meet and discuss with. I'm all pump-up, and I have tons of ideas and projects… more to come.

I already miss you…

Cloud


Programming



Read more »

Reading Notes #317

Labels: , , , , , , , , , , , , ,

729c5a03-6992-401b-a653-7f9b343472e0[1]Suggestion of the week


Cloud


Programming


Miscellaneous



Read more »

Reading Notes #316

Labels: , , , , , , , , , , , , ,

Kubernetes-logoSuggestion of the week


Cloud


Programming


Miscellaneous



Read more »

Reading Notes #315

Labels: , , , , , , , , , , , , , , , ,

36145725596_6536a4f8aa_cSuggestion of the week



Cloud


Programming


Databases


Miscellaneous





Photo credit: nickpettican on Visual Hunt / CC BY-NC-ND

Read more »

Reading Notes #314

Labels: , , , , , , , , , , , , , , , , , ,

MVIMG_20180204_102600Suggestion of the week

  • The Modern Dev Team (Rob Conery) - What a great post. Maybe it's only me getting old ;) but I think we all have these thoughts one day or the other.

Cloud


Programming


Databases

  • My Favorite SQL Prompt Features (MarlonRibunal) - If you never try SqlPrompt and you write SQL in your day to day, stop read this, and go downloading it, or at least read this post that gives you a glimpse of it's feature.

Miscellaneous


Read more »

How to save huge money by shutting down your VM automatically

Labels: , , , , , , , ,
Updated on 2018-03-14

Virtual machines (VM) are used in most solutions nowadays as a [ProcessName] server, temporary machine to run tests or make demos, and sometimes even as a development machine. One of the great benefits of the cloud is that you only pay for what you use. So unlike the old server, that you keep paying for, you won pay virtual machine's CPU for when you turned off! In this post, I explain how to do it with your existing machines and also what to do with all the future one that you will be creating.

(Ce billet en aussi disponible en français.)

Already have a VM up and running, here what to do


From the Azure portal (portal.azure.com), select the Virtual Machine (VM) that you which to edit. Then look at the option panel, on the left, for Auto-Shutdown in the Operations section. You should have something that looks like this:

auto-shutdown

At any time you can enable and disable that functionality, it won’t affect the running VM.

Now, to activate it click on the Enabled. Then Select the time you would like to see the VM shutdown. Be sure to select the good time zone, by default it’s UTC. You can adjust the at for UTC of change the time zone, both options are valid.

Now you could decide to enable the notification. That could be useful if you may want to postpone the shutdown for one or two hours, or integrate the shutdown to another process like backup, cleaning…

To activate the notification option just click on the enabled, and enter the email address. If you want to attach the shutdown to a Logic App or an Azure Functions use the webhook. Here an example of notification email, see the Postpone options link.

emailsample

What if you have many VMs running


Let's say you have already twenty (or more) VMs running, you could have executed a PowerShell script like:


$myMVsName = @("franDev1", "frankBuildserver", "demo_sales2018")

For ($i=0; $i -lt $myMVsName.Length; $i++) {     
    Set-AzureRmDtlAutoShutdownPolicy $myMVsName[$i]
    [...]
}

Update - 2018-03-14
Well, today this is only possible for VM part of a DevTest Labs. Not for "regular" VM. However, I'm sure that day will come pretty quick.Does that mean that you need to go in all your VMs and set it manually? No. You can use an Azure Automation that will stop a list of VM on a regular schedule. A big advantage of this solution is that you can be more creative since it offers a lot more flexibility. You could identify the VM to shutdown base on some TAGS, you could have a different schedule base on the week vs weekend. You could even have a task to start VMs in the morning... More to come on that topic in a future post... If you want to read about how to get started to Azure Automation click here.

Multiple VMs that already exist, no problem

Obviously, if you have multiple virtual machines that already exist it is not very efficient to change their configuration one by one via the portal. Here is a small script to change the configuration of a large amount of VM in one shot.


    '# Login-AzureRmAccount

    $Subscription = Get-AzureRmSubscription -SubscriptionName 'YOUR_SUBSCRIPTION_NAME'
    Select-AzureRmSubscription -Subscription $Subscription.Id

    $selectedVMs = Get-Azurermvm -ResourceGroupName cloud5mins
    foreach($vm in $selectedVMs) 
    { 
        $ResourceGroup = $vm.ResourceGroupName
        $vmName = $vm.Name
        $ScheduledShutdownResourceId = "/subscriptions/$Subscription/resourceGroups/$ResourceGroup/providers/microsoft.devtestlab/schedules/shutdown-computevm-$vmName"
    
        $Properties = @{}
        $Properties.Add('status', 'Enabled')
        $Properties.Add('targetResourceId', $vm.Id)
        $Properties.Add('taskType', 'ComputeVmShutdownTask')
        $Properties.Add('dailyRecurrence', @{'time'= 2100})
        $Properties.Add('timeZoneId', 'Eastern Standard Time')
        $Properties.Add('notificationSettings', @{status='Disabled'; timeInMinutes=60})

        New-AzureRmResource -Location $vm.Location -ResourceId $ScheduledShutdownResourceId -Properties $Properties -Force
    }

The variable $selectedVMs contains all the VMS that we wish to edit. In this sample, I only get VMs contained in the RessourceGroup cloud5mins, but there are no limits to what you can do. You could select all VMs with a specific OS, tags, location, name, etc.

The variable $ScheduledShutdownResourceId will be the identity for the configuration for the auto-shutdown we wish to inject. Note that the provider is microsoft.devtestlab.

Next, we create a collection of properties in $Properties. status the one that active or deactivate the auto-shutdonw. targetResourceId is the resourceID of the VM we target.

The only things left is to specify the time and timezone.

If you prefer, I also have a video version that explains all the steps.

How to shutdown automatically all your existing VMs



End Update

Let's create a VM with the auto-shutdown pre-configured with ARM


Of course, a much more efficient way to set the auto-shutdown is at the creation time by adding a new resource of type Microsoft.DevTestLab/schedules to your template. This option was previously only accessible for DevTestLab, but recently was made available to any VMs.
Here an example of the variables that could be added to your template.

"variables": {

    "ShutdowTime": "21:00",
    "TimeZone": "UTC",
    "emailRecipient": "frank@frankysnotes.com",
    "notificationLocale": "en",
    "timeInMinutes": 30
}

And here an example of Microsoft.DevTestLab/schedules resource. One of these should be added for every VM you wish to auto-shutdown. Because your script is for one server, however, only one instance is required.

{
    "name": "[concat('autoshutdown-', variables('vmName'))]",
    "type": "Microsoft.DevTestLab/schedules",
    "apiVersion": "2017-04-26-preview",
    "location": "[resourceGroup().location]",
    "properties": {
        "status": "Enabled",
        "taskType": "ComputeVmShutdownTask",
        "dailyRecurrence": {
            "time": "[variables('ShutdowTime')]"
        },
        "timeZoneId": "[variables('TimeZone')]",
        "targetResourceId": "[resourceId('Microsoft.Compute/virtualMachines', variables('vmName'))]",
        "notificationSettings": {
            "status": "Enabled",
            "emailRecipient": "[variables('emailRecipient')]",
            "notificationLocale": "[variables('notificationLocale')]",
            "timeInMinutes": "[variables('timeInMinutes')]"
        }
    },
    "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
    ]
}

Reading Notes #313

Labels: , , , , , , , , , , , , , , , , , ,

roy_sky_ansi2Suggestion of the week


Cloud


Programming


Databases


Miscellaneous


Read more »

Reading Notes #312

Labels: , , , , , , , , , , , , , , ,
IoT Hands-on Lab at Ottawa

Suggestion of the week


Cloud


Programming


Miscellaneous

Read more »

Reading Notes #311

Labels: , , , , , , , , , , , , , , , , , , ,

DateTimeImg2Suggestion of the week



Cloud


Programming

  • Styling Social Media Icon Lists in CSS (Mark Heath) - Yeah right, we can read CSS and probably hack some stuff... But it's excellent to learn how to do simple things the good way. And this post shows exactly that.


Data


Miscellaneous


How to Automatically Generate Video Sub-Title in Another Language

Labels: , , , , , , , , , , , , , , , , ,
I recently started a French YouTube channel. Quickly, I got a message asking to add English sub-title, and got also a suggestion to leverage Azure Logic App and some Cognitive Services to help me in that task. I really liked the idea, so I gave it a shot. I recorded myself and in twenty minutes I was done. Even though, it was not the success I was hoping for, the application works perfectly. It's just that speaking in French with a lot of English technical word was a little bite too hard for the Video Indexer. However, If you are speaking only one language in your video that solution would work perfectly. In this post, I will show you how to create that Logic App with Azure Video Indexer and Cognitive Services.

The Idea


Once a video is dropped in an OneDrive folder (or any file system accessible from Azure), a Logic App will get triggered and uploads the file to the Azure Video Indexer, generate a Video Text Tracks (VTT) file, and save this new file in another folder. A second Logic App will get started and use the Translator Text API from Azure Cognitive Service to translate the VTT file, and save it into the final folder.

GenerateSubTitle


The Generation


Before getting started, you will need to create your Video Indexer API. To do this, login to the Video Indexer developer portal, and subscribe at the Video Indexer APIs - Production in the Product tab. You should then get your API keys.


ApiKeyPage

To get more detail on the subscription refer to the documentation. To know the names, parameters, code sample to all the methods available in your new API, click on APIs tab.

APIDetails

Now let's create our first Logic App. I always prefer to start with a blank template, but take what fits you. Any Online file system's trigger will do, in this case I'm using the When a file is created from OneDrive. I got some issue with the trigger. It was not always getting fired by a new file. I tried the When a file is modified trigger, but it didn't solve the problem. If you think, you know what I was doing wrong feel free to leave a comment :).

First reel action is to upload the file to the Azure Video Indexer. We can to that ery easily by using the method Upload video and and index, passing the name and content from the trigger.

Of course, the longer is the video the longer will be the process, so we will need to wait. A way to do that is by adding a waiting loop. Will use the method Get processing state from the Video Indexer and loop until the status is processed. To slow down your loop just add a wait action and set it at tree or five minutes.

When the file is completely processed, it will be time to retrieve the VTT file. This is done in two simple step. First, we will get the URL by calling the method Get the transcript URL, then with a simple HTTP GET we will download the file. The last thing we will need to do will be to save it in a folder where our second Logic App will be watching for new drop.

In the visual designer, the Logic App should look to this.

LogicAppGenerateVTT


The Translation


The second Logic App is very short. Once again, it will get triggered by a new file trigger in our OneDrive Folder. Then it will be time to call our Translator Text API from Azure Cognitive Service. That's to the great Logic App interface it's very intuitive to fill all the parameter for our call. Once we got the translation, we need to save it into our final destination.

The Logic App should look like this.

LogicAppTranslate


Conclusion


It was much easier than I expected. I really like implementing those integration projects with Logic App. It's so easy to "plug" all those APIs together with this interface. And yes like I mentioned in the introduction the result was not "great". I run test with video purely in English (even with my accent) or only in French (no mix) and the result was really good. So I think the problem is really the fact that I mix French and English. I could improve the Indexer by spending time providing files so the service could understand better my "Franglish". However, in twenty minutes, I'm really impressed by the way, in turned out. If you have idea on how to improve this solution, or if you have some questions, feel free to leave a comment. You can also watch my French YouTube video.

All the code is available online on Github - Cloud5VideoHelper.

References:



Reading Notes #310

Labels: , , , , , , , , , , , , , ,
2018

Cloud


Programming


Miscellaneous



How to Fix the [ERROR] Get-ChildItem when Deploying an Azure Resource Group in Visual Studio

Labels: , , , , , , , ,
Lately, I've been having some trouble when deploying from Visual Studio. First, I didn't care since I didn't have time to investigate and also because most of the time using PowerShell or Azure CLI. However, this issue was not usual of Visual Studio, so I decided to see what was the problem and try to fix it.

The Problem


In a solution, I added a simple Azure Resource Group deployment project just like this one.

simpleProject

Then when I try to right-click and do a Deploy...

Deploy

I was having this error message:

 - The following parameter values will be used for this operation:
 - Build started.
 - Project "TestARMProject.deployproj" (StageArtifacts target(s)):
 - Project "TestARMProject.deployproj" (ContentFilesProjectOutputGroup target(s)):
 - Done building project "TestARMProject.deployproj".
 - Done building project "TestARMProject.deployproj".
 - Build succeeded.
 - Launching PowerShell script with the following command:
 - 'D:\Dev\local\TestARMProject\TestARMProject\bin\Debug\staging\TestARMProject\Deploy-AzureResourceGroup.ps1' -StorageAccountName '' -ResourceGroupName 'TestARMProject' -ResourceGroupLocation 'eastus' -TemplateFile 'D:\Dev\local\TestARMProject\TestARMProject\bin\Debug\staging\TestARMProject\azuredeploy.json' -TemplateParametersFile 'D:\Dev\local\TestARMProject\TestARMProject\bin\Debug\staging\TestARMProject\azuredeploy.parameters.json' -ArtifactStagingDirectory '.' -DSCSourceFolder '.\DSC'
 - 
 - 
 - Account          : Frank Boucher
 - SubscriptionName : My Subscription
 - SubscriptionId   : xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
 - TenantId         : xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
 - Environment      : AzureCloud
 - 
 - VERBOSE: Performing the operation "Replacing resource group ..." on target "".
 - VERBOSE: 7:06:33 - Created resource group 'TestARMProject' in location 'eastus'
 - 
 - ResourceGroupName : TestARMProject
 - Location          : eastus
 - ProvisioningState : Succeeded
 - Tags              : 
 - TagsTable         : 
 - ResourceId        : /subscriptions/xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/TestARMProject
 - 
 - Get-ChildItem : Cannot find path 
 - 'D:\Dev\local\TestARMProject\TestARMProject\bin\Debug\staging\TestARMProject\azuredeploy.json' because it does not 
 - exist.
 - At D:\Dev\local\TestARMProject\TestARMProject\bin\Debug\staging\TestARMProject\Deploy-AzureResourceGroup.ps1:108 
 - char:48
 - + ... RmResourceGroupDeployment -Name ((Get-ChildItem $TemplateFile).BaseNa ...
 - +                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 -     + CategoryInfo          : ObjectNotFound: (D:\Dev\local\Te...zuredeploy.json:String) [Get-ChildItem], ItemNotFound 
 -    Exception
 -     + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand
 -  
 - Deploying template using PowerShell script failed.
 - Tell us about your experience at https://go.microsoft.com/fwlink/?LinkId=691202
Apparently the script is failling with Get-ChildItem because my script is missing?! I looked in the folder D:\Dev\local\TestARMProject\TestARMProject\bin\Debug\staging\TestARMProject, and indeed the files are missing! Fixing this is in fact really simple fortunately.

The Solution


The problem is very simple when Visual Studio is building the project, it doen't copies the script files in the build folder (in this case bin\Debug\Staging\). In fact, Visual Studio is doing exactly as we are telling it. Let see the build command for those files. Right-click and select Properties (or Alt+Enter) while azuredeploy.json is selected.

Properties

See the Build Action is set at None change that to Content (for all the scripts). Save and Deploy again.

enjoy

Enjoy!



Reading Notes #309

Labels: , , , , , , , , , , , , ,

419HCloud


Programming


Databases


Miscellaneous



My new Youtube Channel

Labels: , , , ,
I've been blogging for about ten years with you now on this blog, and it's been a pleasure. More recently I started, quietly, a French blog named: Cloud en Français, feel free to have a look.
Today, I'm super excited to share with you my new project: A YouTube channel with original content published every week... in French! Every Wednesday I will be publishing a five-minute video to answer a problem or a question.

Come see me online, subscribe, ask questions...

Cloud en 5 minutes

youtubechannel


Je blogue depuis une dizaine d'années avec vous sur ce blogue, et c'est toujours un plaisir. Plus récemment j'ai commencé, plus silencieusement, un blogue français nommé: Cloud en Français, n'hésitez pas à jeter un coup d'œil.

Aujourd'hui, je suis super excité de partager avec vous mon nouveau projet: une chaîne YouTube avec du contenu original publié chaque semaine... en français! Chaque mercredi, je vais publier une vidéo d'environ cinq minutes pour répondre à un problème ou une question.

Venez me voir en ligne, abonnez-vous, posez des questions ...

Cloud en 5 minutes

Reading Notes #308

Labels: , , , , , , , , , , , , , , , , ,

2017-12-02_00-10-58Suggestion of the week


Cloud


Programming


Miscellaneous


Reading Notes #307

Labels: , , , , , , , , , , , , , ,

MVIMG_20171201_131034Cloud



Programming



Miscellaneous




How to access an SQL Database from an Azure Function and Node.js

Labels: , , , , , ,
The other day, a friend asked me how he could add some functionality to an existing application without having access to the code. It the perfect case to demo some Azure Functions capability, so I jumped on the occasion. Because my friend is a Node.js developer on Linux, and I knew it was supported, I decided to try that combination. I know Node, but I'm definitely not and expert since I don't practice very often.

This post is my journey building that demo. I was out of my comfort zone, coding in Node and working on a Linux machine, but not that far... Because these days, you can "do some Azure" from anywhere.

The Goal


Coding an Azure Function that will connect to an SQL Database (it could be any data source). Using Node.js and tools available on Unbuntu.

Note: In this post, I will be using Visual Studio Code, but you could also create your function directly in the Azure Portal or from Visual Stusio.

Getting Started


If you are a regular reader of this blog, you know how I like Visual Studio Code. It's a great tool available on Mac Linux and Windows and gives you the opportunity to enjoy all its feature from anywhere feeling like if you were in your cozy and familiar environment. If VSCode is not already installed on your machine, go grap your free version on http://code.visualstudio.com.

Many extensions are available for VSCode, and one gives us the capability to code and deploy Azure Function. To install it, open VSCode and select the extension icon and search for Azure Function; it's the one with the yellow lighting and the blue angle brackets.

AzureFunctionExtension

Create the Azure Function


To get started let's great an Azure Function project. By sure to be in the folder where you wish to create your Function App. Open the Command Pallette (Ctrl + Shift + p) and type Azure Function. Select Azure Functions: Create New Project. That will add some configuration files for the Functions App.

Now Let's create a Function. You could reopen again the Command Palette and search for Azure Function: Create Function, but let's use the UI this time. At the bottom left of the Explorer section, you should see a new section called AZURE FUNCTIONS. Click on the little lighting to Create a new Function.

AzureFuncButton

After you specify the Function App name, the Azure subscription and other little essential, a new folder will be added in your folder structure, and the function is created. The code of our function is in the file Index.js. At the moment, of writing this post only Javascript is supported by the VSCode extension.

Open the file index.js and replace all its content by the following code.


var Connection = require('tedious').Connection;
var Request = require('tedious').Request
var TYPES = require('tedious').TYPES;

module.exports = function (context, myTimer) {

    var _currentData = {};

    var config = {
        userName: 'frankadmin',
        password: 'MyPassw0rd!',
        server: 'clouden5srv.database.windows.net',
        options: {encrypt: true, database: 'clouden5db'}
    };

    var connection = new Connection(config);
    connection.on('connect', function(err) {
        context.log("Connected");
        getPerformance();
    });

    function getPerformance() {

        request = new Request("SELECT 'Best' = MIN(FivekmTime), 'Average' = AVG(FivekmTime) FROM RunnerPerformance;", function(err) {
        if (err) {
            context.log(err);}
        });

        request.on('row', function(columns) {
            _currentData.Best = columns[0].value;
            _currentData.Average = columns[1].value;;
            context.log(_currentData);
        });

        request.on('requestCompleted', function () {
            saveStatistic();
        });
        connection.execSql(request);
    }


    function saveStatistic() {

        request = new Request("UPDATE Statistic SET BestTime=@best, AverageTime=@average;", function(err) {
         if (err) {
            context.log(err);}
        });
        request.addParameter('best', TYPES.Int, _currentData.Best);
        request.addParameter('average', TYPES.Int, _currentData.Average);
        request.on('row', function(columns) {
            columns.forEach(function(column) {
              if (column.value === null) {
                context.log('NULL');
              } else {
                context.log("Statistic Updated.");
              }
            });
        });

        connection.execSql(request);
    }

    context.done();
};

The code just to demonstrate how to connect to an SQL Database and do not represent the best practices. At the top, we have some declaration the used the package tedious; I will get back to that later. A that, I've created a connection using the configuration declared just before. Then we hook some function to some event. On connection connect the function getPerformance() is called to fetch the data.

On request row event we grab the data and do the "math", then finally on requestCompleted we call the second sub-function that will update the database with the new value. To get more information and see more example about tedious, check the GitHub repository.

Publish to Azure


All the code is ready; it's now time to publish our function to Azure. One more time you could to that by the Command Palette, or the Extension menu. Use the method of your choice and select Deploy to Function App. After a few seconds only our Function will be deployed in Azure.

Navigate to portal.azure.com and get to your Function App. If you try to Run the Function right now, you will get an error because tedious is not recognized.

Install the dependencies


We need to install the dependencies for the Function App, in this case tedious. A very simple way is to create a package.json file and to use the Kudu console ton install it. Create a package.json file with the following json in it:


{
    "name": "CloudEn5Minutes",
    "version": "1.0.0",
    "description": "Connect to Database",
    "repository": {
       "type": "git",
       "url": "git+https://github.com/fboucher/CloudEn5Minutes.git"
    },
    "author": "",
    "license": "ISC",
    "dependencies": {
        "tedious": "^2.1.1"
    }
}

Open the Kudu interface. You can reach it by clicking on the Function App then the tab Platform features and finally Advanced tools (Kudu). Kudu is also available directly by the URL [FunctionAppNAme].scm.azurewebsites.net (ex: https://clouden5minutes.scm.azurewebsites.net ). Select the Debug console CMD. Than in the top section navigate to the folder home\site\wwwroot. Drag & drop the package.json file. Once the file is uploaded, type the command npm install to download and install all the dependencies declared in our file. Once it all done you should restart the Function App.

Kudu

Wrapping up & my thoughts


There it is, if you go back now to your Function and try to execute it will work perfectly. It's true that I'm familiar with Azure Function and SQL Database. However, for a first experience using Ubuntu and Node.js in the mix, I was expecting more resistance. One more time VSCode was really useful and everything was done with ease.

For those of you that would like to test this exact function, here the SQL code to generate what will be required for the database side.


    CREATE TABLE RunnerPerformance(
        Id           INT IDENTITY(1,1)  PRIMARY KEY,
        FivekmTime   INT
    );

    CREATE TABLE Statistic(
        Id          INT IDENTITY(1,1)  PRIMARY KEY,
        BestTime    INT,
        AverageTime INT
    );

    INSERT Statistic (BestTime, AverageTime) VALUES (1, 1);

    DECLARE @cnt INT = 0;

    WHILE @cnt < 10
    BEGIN
    INSERT INTO RunnerPerformance (FivekmTime)
            SELECT  9+FLOOR((50-9+1)*RAND(CONVERT(VARBINARY,NEWID())));
    SET @cnt = @cnt + 1;
    END;

Video version



References


Subscribe to: Posts (Atom)