Showing posts with label Azure Storage. Show all posts
Showing posts with label Azure Storage. Show all posts

How to copy data in Azure using AzCopy

Labels: , , , , ,

Data is the key to almost all solutions. Obviously, at some point, we will need to move it. And this is when AzCopy  will come to the rescue. In this short post/video I will share how you can securely copy a Zip file (aka. data), from one location (blob storage, AWS) to a blob storage in an Azure subscription (the same subscription or a different one).

What is AzCopy 

AzCopy is a command-line utility that you can use to copy blobs or files to or from a storage account. It can run on Windows, Mac, and Linux. And... It's already available pre-install inside Cloud Shell!

How it works

AzCopy can do many things but let's focus on the "copy" feature. To copy a file from a location to another one here the command to execute:

azcopy copy https://url-source https://url-destination

It looks simple, right? And it is. To keep it secure AzCopy can use Shared Access Signature (SAS) tokens. To get those in Azure, you can execute a command (ex: az storage container generate-sas) or use the Azure Portal. 

Once you are in the Azure portal open the account storage of your source or destination (you will have to do both).

Screen capture showing the steps to create a SAS token
From the option on the left search for Shared access signature or just sas and click on it. Select the type of options you need. A best practice is to allow the minimum requirements. If you know you are only moving files then unchecked the File, Queue, and Table. Same things for the resources types, permissions, and expiry date/time.  Once you are done click the Generate SAS and connection string button.

Use those URLs with the SAS token in your command, and voila!

Video demo


How about Az CLI and PowerShell

If you prefer copying or moving your files using native commands in Azure PowerShell and Az CLI it's also possible here a previous post about that: How to copy files between Azure subscription from Windows, Linux, OS X, or the cloud. There is also a video for this one https://youtu.be/cE8qwnSk9qs


References:


How to copy files between Azure subscription from Windows, Linux, OS X, or the cloud

Labels: , , , , , , , , , , , , , , , , ,
(en français: ici)

Copy, Download or Upload from-to any combination of Windows, Linux, OS X, or the cloud


Data is and will always be our primary concern. Whether shaped as text files, images, VM VHDs or any other ways, at some point in time, our data will need to be moved. I already wrote about it previously, and the content of this post is still valuable today, but I wanted to share new options and convert all ground (meaning Linux, Windows and OS X).

Scenarios


Here few scenarios why you would want to move data.
  • Your Microsoft Azure trial is ending, and you wish to keep all the data.
  • You are creating a new web application, and all those images need to be moved to the Azure subscription.
  • You have a Virtual Machine that you would like to move to the cloud or to a different subscription.
  • ...

AZCopy


AzCopy is a fantastic command-line tool for copying data to and from Microsoft Azure Blob, File, and Table storage. At the moment, to write this post AzCopy is only available for Windows users. Another solution will be introduced later in this post for Mac and Linux users. Before AzCopy was only available on Windows. However, recently a second version built with .NET Core Framework is available. The commands are very similar but not exactly the same.

AzCopy on Windows


In his simplest expression, an AzCopy command looks like this:
AzCopy /Source:<source> /Dest:<destination> [Options]
If you earlier have installed an Azure SDK on your machine, you already have it. By default, AzCopy is installed to %ProgramFiles(x86)%\Microsoft SDKs\Azure\AzCopy (64-bit Windows) or %ProgramFiles%\Microsoft SDKs\Azure\AzCopy (32-bit Windows).

If you need only AzCopy for a server, you can download the latest version of AzCopy.
Let's see some frequent usage. First let's say you need do move all those images from your server to an Azure blob storage.
AzCopy /Source:C:\MyWebApp\images /Dest:https://frankysnotes.blob.core.windows.net/blog /DestKey:4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== /S
CopyAllImages

Then to copy those images to another subscription very easy.
AzCopy /Source:https://frankysnotes.blob.core.windows.net/blog /Dest:https://frankshare.blob.core.windows.net/imagesbackup /SourceKey:4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== /DestKey:EwXpZ2uZ3zrjEbpBGDfsefWkj3G2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA== /S

AzCopy Parameters


These examples were simple, but AzCopy is a very powerful tool. I invite you to type one of the following commands to discover more about using AzCopy:
  • For detailed command-line help for AzCopy: AzCopy /?
  • For command-line examples: AzCopy /?:Samples

AzCopy on Linux


Before you could install AzCopy you will need to install .Net Core. This is done very simply with few commands.
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
sudo apt-get update
sudo apt-get install dotnet-sdk-2.0.2
Then to install it, you just need to get it with a wget command, unzip it, and execute the install script.
wget -O azcopy.tar.gz https://aka.ms/downloadazcopyprlinux 
tar -xf azcopy.tar.gz 
sudo ./install.sh
In his simplest expression, the .Net Core version of AzCopy command looks like this:
azcopy --source <source> --destination <destination> [Options]
It is very similar to the original version, but parameters are using -- and - instead of the / and where a : was required, it's now a simple space.

Uploading to Azure


Here an example, to copy a single file GlobalDevopsBootcamp.jpg to an Azure Blob Storage. We pass the full local path to the file into --source, the destination is the full URI, and finally the destination blob storage key. Of course, you could also use SAS token if you prefer.
azcopy \
--source /home/frank/demo/GlobalDevopsBootcamp.jpg \
--destination https://frankysnotes.blob.core.windows.net/blog/GlobalDevopsBootcamp.jpg \
--dest-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA==

Copying Between Azure Subscriptions


To copy the image to a second Azure subscription, we use the command the source is now an Azure Storage URI, and we pass the source and the destination keys:
azcopy \
--source https://frankysnotes.blob.core.windows.net/blog/GlobalDevopsBootcamp.jpg \
--destination https://frankshare.blob.core.windows.net/imagesbackup/GlobalDevopsBootcamp.jpg \
--source-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== \
--dest-key EwXpZ2uZ3zrjEbpBGDfsefWkj3G2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA==

Azure CLI


Azure CLI is a set of cross-platform commands for the Azure Platform. It gives tools to manipulate all Azure components, but this post will focus on azure storage features.

There are two versions of the Azure Command-Line Interface (CLI) currently available:

  • Azure CLI 2.0: written in Python, conpatible only with the Resource Manager deployment model.
  • Azure CLI 1.0: written in Node.js, compatible with both the classic and Resource Manager deployment models.

Azure CLI 1.0 is deprecated and should only be used for support with the Azure Service Management (ASM) model with "classic" resources.

Installing Azure CLI


Let's start by installing Azure CLI. Of course, you can download an installer but since everything is evolving very fast with not getting it from Node Package Manager (npm). The install will be the same, you just need to specify the version if you absolutely need Azure CLI 1.0.

sudo npm install azure-cli -g

AzureCli_installed

To keep the previous scenario, let's try to copy all images to a blob storage. Unfortunately, Azure CLI doesn't offer the same flexibility as AzCopy,and you must upload the file one by one. However, to upload all images from a folder, we can easily put the command in a loop.

for f in Documents/images/*.jpg
do
   azure storage blob upload -a frankysnotes -k YoMjXMDe+694FGgOaN0oaRdOF6s1ktMgkB6pBx2vnAr8AOXm3HTF7tT0NQWvGrWnWj5m4X1U0HIPUIAA==  $f blogimages
done

azurecli_allimages

In the previous command -a was the account name, and -k was the Access key. This two information can easily be found in the Azure portal. From the portal (https://portal.azure.com), select the storage account. In the right band click-on Access keys.

StorageAccessKeys

To copy a file (ex: a VM disk aka VHD) from one storage to another one in a different subscription or region, it's really easy. This time we will use the command azure storage blob copy start and the -a and -k are related to our destination.

azure storage blob copy start 'https://frankysnotes.blob.core.windows.net/blogimages/20151011_151451.MOV' imagesbackup -k EwXpZ2uZ3zrjEbpBGDfsefWkj3GnuFdPCt2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA== -a frankshare

The nice thing about this command is that it's asynchronous. To see the status of your copy just execute the command azure storage blob copy show

azure storage blob copy show -a frankshare -k YoMjXMDe+694FGgOaN0oPaRdOF6s1ktMgkB6pBx2vnAr8AOXm3HTF7tT0NQVxsqhWvGrWnWj5m4X1U0HIPUIAA== imagesbackup 20151011_151451.MOV

CopyStatus1


CopyStatus2

Azure CLI 2.0 (Windows, Linux, OS X, Docker, Cloud Shell)


The Azure CLI 2.0 is Azure's new command-line optimized for managing and administering Azure resources that work against the Azure Resource Manager. Like the previous version, it will work perfectly on Windows, Linux, OS X, Docker but also from the Cloud Shell!


Cloud Shell is available right from the Azure Portal, without any plugging.

Uploading to Azure


The command if the same as the previous version except that now the command is named az. Here an example to upload a single file into an Azure Blob Storage.

az storage blob upload --file /home/frank/demo/CloudIsStrong.jpg \
--account-name frankysnotes \
--container-name blogimages --name CloudIsStrong.jpg \
--account-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA==

Copying Between Subscriptions


Let's now copy the file to another Azure subscription. A think to be aware is that --account-name and --account-key are for the destination, even if it's not specified.
az storage blob copy start \
--source-account-name frankysnotes  --source-account-key 4YvvYDTg3UUpky8Rj5bDG4KO/R1FdtssxVnunsEd/4rAS04V2LkO0F8mXbddAv39WtCo5LW6JyvfhA== \
--source-container blogimages --source-blob CloudIsStrong.jpg   \
--account-name frankshare  --account-key EwXpZ2uZ3zrjEbpBGDfsefWkj3G2QY5fJcb6kMqV2A0+2TsGno+mk9vEXc5Uw1XiouvAiTS7Kr5OGzA== \
--destination-container imagesbackup  \
--destination-blob CloudIsStrong.jpg

In Video Please!


If you prefer, I also have a video version of that post.



One More Thing


Sometimes, we don't need to script things, and a graphic interface is much better. For this kind of situation, the must is the Azure Storage Explorer. It does a lot! Upload, download, and manage blobs, files, queues, tables, and Cosmos DB entities. And it works on Windows, macOS, and Linux!


It's just the beginning


This post was just an introduction to two very powerful tools. I strongly suggest to go read in the official documentation to learn more. Use the comment to share all your questions and suggestion.

References:

Upgrade an Application Windows Azure OS Family

Labels: , , , , , , , , , , , , , ,
Recently I add to upgrade an web site running in Azure Webrole from Azure OS famille 1.6 to a more recent version. While the migration was not complicated I encounter some little particularity that I found could be interesting to share.

The Context

The website was a Visual Studio 2010 project using Azure SDK 1.6 and a library call AspNetProvider that was part of Microsoft's sample few years ago to manage session and membership. Using the AspNetProvider library the session was saved in Azure blob storage, and the membership was saved in an SQL database.

The Goal

The application must stay a Visual Studio 2010 project, but using the most-recent Azure SDK and Azure Storage Client as possible.

The Solution

  • Azure SDK 2.1
  • Azure.StorageClient 4.0
  • Universal Provider version 2.1
  • OS famille 4

The Journey


Migration from SDK 1.6 to SDK 2.1


Azure SDK version 2.1 is the higher version compatible with Visual Studio 2010. And can be downloaded from Microsoft's website. Once it is installed, just open the project in Visual Studio and right-click on the Azure Project. By clicking on the upgrade button the magic will happen. Some errors could stay but the hard work will be done for you.


Migration from AspNetProvider to UniversalProvider


we need to remove all reference to the AspNetProvider library. Just expand the resources node in the Solution Explorer and delete the reference. One thing important is that since we are using Visual Studio 2010 the latest version of the UniversalProvider we can use is 1.2. More recent version are using .Net 4.5 and this is not compatible with the present solution. To get the reference added to the project just execute the following Nugget command:
Install-Package UniversalProvider -version 1.2

Check the web.config file to clean the membership connections.

Migration of the Azure Storage Client


This one is the easiest, just remove the reference in the reference node and then execute the following Nugget Command:
Install-Package Azure.Storage.Client

Migration of the membership data


The AspNetProvider was using prefixed SQL tables: aspnet_user, aspnet_membership, etc. The new membership manager is using another sets of tables. We must migrate the data from one set to the other one. Here a SQL script that will to exactly that. The script can be run multiple times because it will only copie the unmoved data.
-- ========================================================
-- Description:    Migrate data from asp_* tables 
--                 to the new table used by Universal provider
-- ========================================================

DECLARE @CNT_NewTable AS INT
DECLARE @CNT_OldTable AS INT

-- --------------------------------------------------------
-- Applications -------------------------------------------

INSERT INTO dbo.Applications (ApplicationName, ApplicationId, Description)
    SELECT    n.ApplicationName, n.ApplicationId, n.Description 
    FROM    dbo.aspnet_Applications o 
    LEFT    JOIN dbo.Applications n ON o.ApplicationId = n.ApplicationId
    WHERE    n.ApplicationId IS NULL

SELECT @CNT_NewTable = Count(1) from dbo.Applications 
SELECT @CNT_OldTable = Count(1) from aspnet_Applications

PRINT 'Application Count: ' + CAST(@CNT_NewTable AS VARCHAR) + ' = ' + CAST(@CNT_OldTable AS VARCHAR)

-- -------------------------------------------------------- 
-- Roles --------------------------------------------------

INSERT INTO dbo.Roles (ApplicationId, RoleId, RoleName, Description)
SELECT    o.ApplicationId, o.RoleId, o.RoleName, o.Description 
FROM    dbo.aspnet_Roles o
LEFT JOIN dbo.Roles n ON o.RoleId = n.RoleId
WHERE n.RoleId IS NULL

SELECT @CNT_NewTable = Count(1) from dbo.Roles 
SELECT @CNT_OldTable = Count(1) from aspnet_Roles

PRINT 'Roles Count : ' + CAST(@CNT_NewTable AS VARCHAR) + ' = ' + CAST(@CNT_OldTable AS VARCHAR)

-- --------------------------------------------------------
-- Users --------------------------------------------------

INSERT INTO dbo.Users (ApplicationId, UserId, UserName, IsAnonymous, LastActivityDate)
SELECT o.ApplicationId, o.UserId, o.UserName, o.IsAnonymous, o.LastActivityDate 
FROM dbo.aspnet_Users o LEFT JOIN dbo.Users n ON o.UserId = n.UserID 
WHERE n.UserID IS NULL

SELECT @CNT_NewTable = Count(1) from dbo.Users 
SELECT @CNT_OldTable = Count(1) from aspnet_Users

PRINT 'Users count: ' + CAST(@CNT_NewTable AS VARCHAR) + ' >= ' + CAST(@CNT_OldTable AS VARCHAR)

-- --------------------------------------------------------
-- Memberships --------------------------------------------

INSERT INTO dbo.Memberships (ApplicationId, UserId, Password, 
PasswordFormat, PasswordSalt, Email, PasswordQuestion, PasswordAnswer, 
IsApproved, IsLockedOut, CreateDate, LastLoginDate, LastPasswordChangedDate, 
LastLockoutDate, FailedPasswordAttemptCount, 
FailedPasswordAttemptWindowStart, FailedPasswordAnswerAttemptCount, 
FailedPasswordAnswerAttemptWindowsStart, Comment) 

SELECT o.ApplicationId, o.UserId, o.Password, 
o.PasswordFormat, o.PasswordSalt, o.Email, o.PasswordQuestion, o.PasswordAnswer, 
o.IsApproved, o.IsLockedOut, o.CreateDate, o.LastLoginDate, o.LastPasswordChangedDate, 
o.LastLockoutDate, o.FailedPasswordAttemptCount, 
o.FailedPasswordAttemptWindowStart, o.FailedPasswordAnswerAttemptCount, 
o.FailedPasswordAnswerAttemptWindowStart, o.Comment 
FROM dbo.aspnet_Membership o
LEFT JOIN Memberships n ON  o.ApplicationId = n.ApplicationId
                      AND o.UserId = n.UserId
WHERE n.UserId IS NULL AND n.ApplicationId IS NULL


SELECT @CNT_NewTable = Count(1) from dbo.Memberships 
SELECT @CNT_OldTable = Count(1) from aspnet_Membership

PRINT 'Memberships count: ' + CAST(@CNT_NewTable AS VARCHAR) + ' >= ' + CAST(@CNT_OldTable AS VARCHAR)

-- -------------------------------------------------------
-- UsersInRoles ------------------------------------------
TRUNCATE TABLE dbo.UsersInRoles
INSERT INTO dbo.UsersInRoles SELECT * FROM dbo.aspnet_UsersInRoles


SELECT @CNT_NewTable = Count(1) from dbo.UsersInRoles 
SELECT @CNT_OldTable = Count(1) from aspnet_UsersInRoles

PRINT 'UsersInRoles count: ' + CAST(@CNT_NewTable AS VARCHAR) + ' >= ' + CAST(@CNT_OldTable AS VARCHAR)


Migration from OSFamilly 1 to 4

Open the file .cscfg and edit the OS Family attribute. It's in the ServiceConfiguration node.
<ServiceConfiguration servicename="MyApp" osFamily="4" osVersion="*" ...>


Wrapping up

The only step left is to deploy in the staging environment to see if everything is working as expected. would recommend also to plan to upgrade as soon as possible because the Azure SDK 2.1 official retirement date is November 2015. I hope this post could help you, even if you are migrating from and to a different version. Any comments, suggestions and/or questions are welcome.


~ Frank Boucher


Reading Notes #47

Labels: , , , , , , , , , , , , , , , ,

Balance_cloud

Cloud


Programming

  • An Introduction to NuGet - Nice post that explain quickly what is NuGet and witch tools are available.
  • Why I Hate Unit Testing - What a nightmare! I'm not sure that this post will help the undecided to join the unit testing but this JusMock tool deserved a look.
"This is easy to do with Telerik JustMock. I wrote about using JustMock to get you started in From Legacy to Dependency Injection.
Steve Forte and Joel Semeniuk have a great presentation on this called The Agile Buffet Table."


Miscellaneous

  • A Super-Efficient Email Process (Peter) - Explain a simple method to be more productive and efficient with e-mail management.
  • the Continuous Client (Joshua Topolsky) - So true, I'm looking for a continuous Twitter client for a long time now. I don't understand why this isn't in all application. Congratulation to Kindle for this! You can start reading on one device and continue to another one without effort.
  • 9 Steps to Take When You Loathe Your Own Blog (Guest Blogger) - Having a blog is a lot work. Here are 9 tips to keep you inspired and proliferates.
  • 5 Things to Ask Your Cloud Backup Services Provider - I'm a Mozy user and really happy with it. Of course this post is also a sales pitch but the questions are the good one and should be considered when selecting your backup solution. When did you do your last backup?
  • 5 Steps To Choosing The Right Challenges (Brendon Burchard) - Nice post that give good tips to identify good challenges.



~Frank


Reading Notes #44

Labels: , , , , , , , , , , , , , ,
They Will Bring You In The Cloud
They will bring you in the cloud.

Cloud


Programming

  • Code First model builder versions (Arthur Vickers) - This post explain how to configure EF code first to avoid any breaking code when updating the framework to a newer version. Nice work.
  • Créer son propre Gem et le publier - Very nice tutorial (in French) that explain in detail with a simple case how to create and also publish a Ruby Gem.

Miscellaneous

  • 5 Tricks for a Killer Company Blog - Five simple tips to get started blogging in a business perspective.
  • Hit the Presentation Sweet Spot - Twenty minutes is very short. Maybe it is a good idea for a specific type of presentation... How do you plan a 2-3 formation?
  • Your Personal Brand as a Developer: Implementing (Part 2 of 2) (Jonathan Rozenblit) - Part 2 on an unusual topic but important. Our brand, as a person. On the internet everything is amplified and could be seen by a lot of people very quickly. A post to read "before it is too late".
  • Coping with Email Overload (Peter) - This could be a good advice for a lot of people. While I'm working on a project where co-workers are across the country I not sure I can do this but maybe tweaking it with rules for only those people.

Reading Notes #41

Labels: , , , , , , , , , , , , ,
image from The Cloud Strategy Blog

Cloud

[…]cost efficiency doesn’t equate to cheap or even free. You can determine the cost to be efficient when the equation can’t be changed to reduce cost without also reducing the services or functionality—meaning[…]

    Programming


    Miscellaneous

    • I'm Sure It Will Only Take You A Few Days To Code - Interesting, I didn't think our usual or natural sensor to estimate the complexity of something cannot be use when it comes to estimate software... but it makes senses!
    • What geeks need to tell our parents about shopping online safely and securely (Scott Hanselman) - Nice post. read this post than go talk to your parent, uncle, brothers and sisters all those people that call you when they did nothing but the PC doesn't works anymore... ;)
      • […]HTTPS (SSL) doesn't mean "I can trust this site," it means "this conversation is private." You still might be having a private conversation with Satan. - Scott Hanselman Trust[…]

    Reading Notes #40

    Labels: , , , , , , , , , ,

    Cloud


      Programming


        Miscellaneous


        ~ Frank


        Reading Notes #39

        Labels: , , , , , , , , , , , , ,
        Mozy cloud storage

        Cloud

        […]shows a friendly face to customers and still provides the detailed technical information developers will need in his article Rich Custom Error Handling with ASP.NET. […]
        […]Wade Wegner describes the steps he recommends in his post Using ELMAH in Windows Azure with Table Storage. […]
        […]To learn more about ELMAH, see the MSDN article Using HTTP Modules and Handlers to Create Pluggable ASP.NET Components by Scott Mitchell and Atif Aziz.[…]

          Programming


          Miscellaneous


          ~ Frank


          Reading Notes #38

          Labels: , , , , , , , , , , , ,
          cloud-security

          Cloud


          Programming


          Miscellaneous

          Reading Notes #36

          Labels: , , , , , , , , , , , ,


          Cloud


          Programming


          Miscellaneous



          ~ Frank



          Reading Notes #35

          Labels: , , , , , , , , , , ,

          image

          Cloud


          Programming

          • Structurez votre code Ruby (Bruno Michel) - This nice post introduce the unused Struct. A simple but power class in Ruby.
          • Soyez unique, utilisez des Sets (Bruno Michel) - Wow I didn't know the Set class in Ruby. I got to edit one of my script immediately.
          • Asp.Net MVC 4 beta - Nice post that quickly present some of the main new features in the brand new ASP.Net MVC 4. Also video and slide reference.
            Spending limits are a new feature we added to Windows Azure last month, and ensure that you never have to worry about accidentally going over the resources included in a free offer and being charged.
            You can learn more about the spending limit feature here.
            The ASP.NET MVC 4 Beta release works with VS 2010 and .NET 4.0, and is side-by-side compatible with prior releases of ASP.NET MVC
            Bundling and Minification
            Database Migrations – ASP.NET MVC 4 includes the new Entity Framework 4.3 release,
            Web API
            new support for building mobile web applications and mobile web sites,
            Razor Enhancements – ASP.NET MVC 4 includes V2 of our Razor View engine.
            Async Support and WebSockets
            • Fun with Ruby Block Parameters - Nice post going dipper in Ruby's Blocks capabilities...

            Miscellaneous


            ~ Frank


            Reading Notes #34

            Labels: , , , , , , ,

             


            Cloud


            Programming


            Miscellaneous


            ~Frank


            Subscribe to: Posts (Atom)