[AZURE 70-534 Cheat Sheet and Exam Notes Part-3]: Design an application storage and data access strategy

Series Index

1. Series Index 
   
   1. Azure ARM , networking and GLOBAL Infrastructure (2017-10-01)
   2. Securing Resources and Azure Security (2017-10-01)
   3. Design an application storage and data access strategy (2017-10-01) 
   4. Design advanced applications (2017-10-01)
   5. Design Azure Web and Mobile Apps (2017-10-01)
   6. Design a management, monitoring, and business continuity strategy (2017-10-01)
   7. Architect an Azure Compute infrastructure (2017-10-01)

Azure Storage Services

1. Azure Storage Types
1. RDBMS -SQL Server , Oracle , MySQL , SQL Compact , SQL Lite, PostGreSQL
2. Key Value - Azure Storage(File/Blob), Tables , Cache , Redis , MemCache, Riak 
3. Column Family - HBase , Cassandra
4. Document - Mango, Raven, Couch
5. Graph - NEO4J 
6. Queue Service 
2. All Storage service false under Storage account that is also used for billing 
3. Each Subscription can have multiple Storage Account (soft limit 20)
4. Storage Account can have 500 TB of data 20K IOPS (1 KB Message)
5. Storage account are region specific and also works as storage security boundary 
6. Unlimited Blobs container , table queue etc.
7. Storage Redundancy
1. Azure support four type of  redundancy 
1. Local redundancy 
2. Zonal redundancy 
3. Geographical redundancy 
4. Read Only  Geographical redundancy
8. Storage Account Limits
1. 20 K IOPS
2. Bandwidth Geo Redundant - 10 GB-IN-20GB-OUT
3. Bandwidth Locally Redundant - 20 GB-IN-30GB-OUT
9. Pricing based on storage (per GB/month); replication type (LRS/ZRS/GRS/RA-GRS); bandwidth (ingress is free; egress charged per GB); requests/transactions.

Storage Redundancy And Replication

1. Data stored in multiple locations (minimum 3 copies).
1. LRS (Locally Redundant Storage) synchronously replicates 3 copies data in separate fault and update domains. Use for: low cost; high throughput (less replication); data sovereignty concerns re: transfer out of region. If region goes down, so do all copies.
2. ZRS (Zone Redundant Storage) also 3 copies but in at least 2 facilities (1 or 2 regions). Data durable in case of facility failure.
3. GRS (Globally Redundant Storage) – 6 copies (3 copies in primary region asynchronously replicated to 3 more copies in a secondary region). Data still safe in a secondary region but cannot be read (unless Azure flips primary and secondary in event of catastrophic failure).
4. RA-GRS (Read Access Geo Redundant Storage) – read from secondary copy. -secondary.cloud.core.windows.net domain name.
2. More copies and more bandwidth is more cost! Also:
1. GRS ingress max 10 Gibps (20 egress) but does not impact latency of transactions made to primary location.
2. LRS ingress max 20 Gibps (30 egress)
3. RA-GRS provides a readable secondary copy of your data, but you cannot make updates to it. and it can not be made primary 

Azure Blob Storage 

1. Can store almost any binary data  object like file but not a file system
2. Block are stored in Blob container like folders 
3. Max throughput for individual blob - 60MB sec , 500 trans/sec
4. Create containers inside storage accounts with up to 500TB data per container
5. URL Patterns https://storageaccountname.blob.core.windows.net/containername/blobname
6. There are two type of Blob 
1. Block blobs, with block ID; uploaded and then committed – unless committed doesn’t become part of the blob: max 64MB per upload (blocks <=4MB), max 200GB per blob; Can upload in parallel, better for large blogs (generally) and for sequential streaming of data.
2. Page blobs – collection of 512byte pages. Max size set during creation and initialization (up to 1TB). Write by offset and range – instantly committed. Overwrite single page or up to 4MB at once; Generally used for random read/write operations (e.g. disks in VMs). Page blobs can be created on premium storage for higher IOPs.
7. Access control is via 512bit keys (secret key – used in API calls to sign requests) – two keys so can maintain connectivity whilst regenerate another (i.e. during key rotation).
8. Can have full public read access for anonymous access to blobs in a container; public read access for blobs only (but not list the blobs in the container); no public read access (default – only signed requests allowed); shared access signature – signed URL for access including permissions, start time and expiry time.
9. Lease blob for atomic operations – lease for 15-60 seconds (or infinite). Acquire/renew/change/release (immediately)/break (at lease end).
10. Snapshots – used to create a read-only copy of a blob (multiple snapshots possible but cannot outlive the original blob – i.e. deleting blob deletes the snapshots); charges based on difference.
11. Copy blob to any container within the same storage account (e.g. between environments).
12. Support Three Concurrency
1. Last Win (Default)
2. Optimistic - notification to application when conflict happens
3. Pessimistic - application lock object in advance before update 

Cool Blob Storage Vs (Hot) Blob Storage?

1. Azure Storage offers three storage tiers for Blob object storage so that you can store your data most cost-effectively depending on how you use it. 
2. The Azure hot storage tier is optimized for storing data that is accessed frequently. 
3. The Azure cool storage tier is optimized for storing data that is infrequently accessed and stored for at least a month. The archive storage tier (preview) is optimized for storing data that is rarely accessed and stored for at least six months with flexible latency requirements (on the order of hours). 
4. The archive storage tier can only be used on the blob level, and not on the whole storage account. 
5. https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-storage-tiers

Azure Bulk Data Transfer (server Side Transfer)

1. AzCopy is a command-line utility designed for copying data to and from Microsoft Azure Blob, File, and Table storage using simple commands with optimal performance. You can copy data from one object to another within your storage account, or between storage accounts.+
2. There are two versions of AzCopy that you can download. AzCopy on Windows is built with .NET Framework, and offers Windows style command-line options. AzCopy on Linux is built with .NET Core Framework which targets Linux platforms offering POSIX style command-line options. This article covers AzCopy on Windows.

Azure Table Storage 

1. Massive Scale NoSQL Cloud Store , Support Massive Data Set and Schema Less 
2. Data is partitioned via partition key
3. Max throughput 2000 entities per partition
4. Use Optimistic Concurrency By default   
5. Store data for simple query – NoSQL key-value store – no locks, joins, validation.
6. http://storageaccountname.table.core.windows.net/tablename
7. Generally, use row key to retrieve data.
8. Can partition tables and generate a partition key.
9. Use shared access signatures for querying/adding/updating/deleting/upserting (insert if does not already exist, else update) table entries

Queue Storage 

1. Max Message Size 64 KB , throughput  2000 msg/sec
2. Store and access messages through HTTP/HTTPS calls.
3. Each queue entry up to 64KB in size.
4. Store messages up to 100 TB.
5. Use for an asynchronous list for processing; messaging layer between applications (avoid handshaking – just add to or consume from the queue); or messaging between web and worker roles.
6. http://storageaccountname.queue.core.windows.net/queuename
7. Operations to put (add), get (which makes message invisible), peek (get first entry without making invisible), delete, clear (all), update (visibility timeout or contents) for messages.
8. 
   

Azure File Service (File Storage)

1. Support REST and SMB, Replacement of On Premise file share 
2. URL Pattern https://storageaccountname.file.core.windows.net/sharename/foldername/foldername/filename
3. Through Put 60 MB/S, 1000 IOPS 
4. Port 445 must be open 
5. Support persistent and not persistent credential 
6. Support Mounting as drive 
7. Support both SMB 2.0 and SMB 3.0 (with encryption)
8. File storage – mounted by servers and accessed via API. Provides shared storage for applications using SMB 2.1. Key Use cases for File service is 
1. On-premises apps that rely on file shares migrated to Azure VMs or cloud services without app re-write.
2. Storing shared application settings (e.g. config files) or diagnostc data like logs, metrics and crash dumps.
3. Tools and utils for developing or administering Azure VMs or cloud services.
9. Create shares inside storage accounts – up to 5TB per share, 1TB per file. Unlimited total number of files and folders.
10. 
    

Azure VHD

1. Persistent Disk for IAAS VMs 
2. Stored in Azure Page Blob 
3. Page Blob are Optimized for Random I/O
4. Read /Write are mapped to GET/PUT
5. Support Premium Storage
1. SSD based Single digit latency 
2. 1 TB Disk Blob and support up to 32 TB  stripe
1. making 32 TB max with 50 K IOPS

Azure Disk Storage 

An Azure virtual machine supports attaching a number of data disks. For optimal performance, you will want to limit the number of highly utilized disks attached to the virtual machine to avoid possible throttling.Azure offers two ways to create premium storage disks for VMs:

1

1. Unmanaged disks
1. The original method is to use unmanaged disks. In an unmanaged disk, you manage the storage accounts that you use to store the virtual hard disk (VHD) files that correspond to your VM disks. VHD files are stored as page blobs in Azure storage accounts.
2. Managed disks
1. When you choose Azure Managed Disks, Azure manages the storage accounts that you use for your VM disks. You specify the disk type (Premium or Standard) and the size of the disk that you need. Azure creates and manages the disk for you. You don't have to worry about placing the disks in multiple storage accounts to ensure that you stay within availability limits for your storage accounts. Azure handles that for you.
3. Premium Storage supports DS-series, DSv2-series, GS-series, Ls-series, and Fs-series VMs.
4. Azure Disk Limits 
1. For Azure Managed Disks: 
1. Managed Disks count limit is regional and also depends on the storage type. The default and also the maximum limit is 10,000 per subscription, per region and per storage type. For example, you can create up to 10,000 standard managed disks and also 10,000 premium managed disks in a subscription and in a region.
2. Managed Snapshots and Images are counted against the Managed Disks limit.
2. For standard storage accounts: 
1. A standard storage account has a maximum total request rate of 20,000 IOPS. The total IOPS across all of your virtual machine disks in a standard storage account should not exceed this limit.
3. For premium storage accounts: A premium storage account has a maximum total throughput rate of 50 Gbps. The total throughput across all of your VM disks should not exceed this limit.

Azure Database Services 

SQL Database (PAAS)

1. Relational database service as a service (PaaS) – up to 500 GB per database.
2. Easy provisioning, automatic HA, load balancing, built-in management portal, scalability, use existing skills to deploy database, patching, etc. taken care of so less time to manage, easy sync with offline data.
3. It is not same as SQL Server on a VM though!
   1. Unsupported features may have corresponding features in Azure; some are just not available.
4. Performance model with different tiers: Basic, then Standard S0-S3, Premium P1-P2, P4, P6 (formerly P3).
   1. Measured in Database Thoughput Units (DTUs) – standardised model to help sizing (relative model [like ACU for VMs]).
   2. Only committing to transactions per hour in Basic, per minute in Standard, per second in Premium.
5. Scaling Azure SQL: 
1. Federation is deprecated; 
2. Custom Sharding (create multiple database and use application logic to separate, e.g. based on customer ID); 
3. Elastic Scale (application doesn’t need to be so smart, endpoint is same but multiple applications).
6. Backups:
   1. SQL database creates automatic backup for active database; at least 3 replicas at any one time – one primary replica and two or more secondaries (more if using GRS).
   2. Can restore to point-in-time (self-service capability to restore from automated system – creates new database on same server – zero-cost/zero-admin – number of days depends on service tier – 7, 14, 35 days for basic/standard/premium), 
   3. Geo-restore (restore from geo-redundant backup to any server in any region.
   4. Automatically enabled for all tiers at no extra cost – helps when there is a region outage – estimated recovery time <12h RPO <1h).
7. Also standard geo-replication (protect app from regional outage – one secondary database in Microsoft-defined paired region; secondary is visible but can’t connect to it until failover occurs – discount for secondary DB as offline until failover – standard/premium only with ERT <30s RPO <5s) and active geo-replication (database redundancy within different regions – up to 4 readable secondary servers – asynchronous replication of committed transactions from one DB to another; for write-intensive applications – e.g. load balancing for read-only workloads – premium only with ERT <30s RPO <5s).
   1. Regional disaster – Geo Restore, Standard or Active Geo-Replication.
   2. Online application upgrade – Active Geo replication.
   3. Online application relocation – Active Geo replication.
   4. Read load balancing – Active Geo replication.
8. Security: only available via TCP 1433 – blocked by default – define firewall rules at server and database level to open up (i.e. to own IP address). Can define firewall rules programmatically with T-SQL, REST API and Azure PowerShell.
9. Data encrypted on wire – SSL required all the time
10. Data encrypted at rest – encryption with transparent data encryption – real-time I/O encryption/decryption for data and log files.
11. Only supports SQL Server authentication or Azure AD authentication – i.e. no Windows authentication.
12. First user created (master database principal) cannot be altered or dropped; can configure user-level permissions by logging on to the database and issuing SQL commands.
13. Pricing: DB size plus outbound data transfers (per database, per month) – per hour pricing, so drop DTUs at quiet time.
14. Maximum database size for an Azure SQL Database running on the Premium tier is 500GB but 4TB is currently in public preview.

Document DB,

1. Document DB is now Azure Cosmos DB read more here 
2. Introduction to Azure Cosmos DB: DocumentDB API

MongoDB,

https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction

MySQL;

TBD