Fundamentals of Awazon Web Services

Chapter 1

Cloud Computing

Definition

NIST definition, "cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction."

Automation of highly virtualized environment.

Summarize:

5 Essential Characteristcis

3 Service Models

4 Deployment Models

Essential Characteristics:

1>   On-Demand

2>   Broad Network Access

3>   Resource Pooling

4>   Rapid Elasticity

Rapid expand or decrease the env.

5>   Measured Service

Pay as you go.

 

Service Models

1>   IaaS (Infrastructure as a Service)

2>   PaaS (Platform as a Service)

With the platform as a service, the application is aware of the infrastructure. If a particular portion of the application is suffering, for instance, if the application is under load and it requires additional database services, the application can provision additional database services.

If a portion of the application fails, the application is aware that some portion failed and it can redirect traffic or provision additional resources in order to support the load

3>   SaaS (Software as a Service)

It can allow you to build additional services, but it's already there. There's something there that you are then leveraging to create something else.

Deployment Models

1>   Private

On premises private cloud, private cloud in a public cloud(e.g: AWS).

2>   Hybrid

Can extend on the premises private cloud to the public cloud.

e.g: We have 100 servers in our private cloud, suddenly we need 50 additional servers. Instead of buying equipment, we can extend into the public cloud. This extension is hybrid cloud.

3>   Community

Vertical Cloud. E.g: AWS is having cloud, specifically for govt. So, only govt can access this cloud. ( others: healthcare, finance, etc)

4>   Public

Where you can use things like, for instance, infrastructure as a service from Amazon Web Services. You and other customers will most likely be on the same servers, on the same storage, so again, it's that pooling of resources and the ability to leverage the scale and reach of cloud providers such as Amazon.

 

Chapter 2

Elastic Compute

Definition

Fundamental building block behind AWS or Cloud Computing.

 

For Customer Success Stories:

https://aws.amazon.com/solutions/case-studies/

Regions And Availability Zones

Region: Geographical area that Amazon specifies.

Availability Zones: Data Centre within a Region that Amazon has.

Within a region we have 2 or more availability zones(Data Centre).

Within every region certain services might not be available.

AWS Edge Locations

Not the location where we can place the load, but the locations where we can consume some of the Amazon Services.

 

Chapter 3

Creating EC2 Instance

Provision Free Tier Resource

·       Go to EC2

·       On right upper corner, select preferred Availability zones.

·       Click on “Launch Instance”

·       Select the Free Tier RedHat

·       Choose “Free Tier Eligible” Instance type

·       Select Configure Instance Details (Bottom right)

·       Add Storage

·       Add Tags

·       Configure Security Group

·       Review and Launch

·       Launch

·       Create a new key pair and give a name

·       Create Instance

 

Chapter 4

AWS Storage

Ephemeral Storage

·       Ephemeral means Temporary

·       Another Name = Instance Store

·       Temporary block level storage

Ephemeral volume is connected to an EC2 instance and the storage is located on the same server that is hosting that EC2 instance

·       Data is lost when instance is terminated or stopped, however data will be retained in case of reboot.

·       Free storage with an EC2 instance

·       Ideal for temporary data like buffers, caches, scratch data, etc.

Amazon Simple Storage Services (S3)

·       Internet accessible storage via HTTP/HTTPS

·       Can store audio, video, images, backups ,etc

·       Unlimited bucket size

·       Priced on storage use and transfer out

·       It’s not a file system

Three Types

1>   Standard Storage

·       99.999999999% durability

2>   Standard - IA

·       For infrequently accessed data. Standard - IA has a 30-day minimum retention period and a 128KB minimum object size.

3>   Reduced Redundancy Storage (RRS)

·       Reduced durability

Around 20% cheaper than Standard storage

·       Granular storage type selection

Can select storage type for each file.

Use:

e.g: RRS can be used for backups of non-master copy of pics.

       Can be used to store videos of  various reduced quality

Configuring S3

·       Bucket name should be unique

·       Can give permission

·       Static Website hosting

Elastic Block Storage

• Not internet accessible
• Subsystem of EC2
• Not an independent service
• Persistent FS for EC2
• Does not need to be attached to an instance
• Can be transferred between Availability Zones
• Supports incremental snapshots
• EBS leverages S3 for snapshot storage

Types:

1>   General Purpose (SSD)

2>   Provisioned IOPS (SSD)

3>   Magnetic

Create a Volume and attach to instance

Glacier

• Very cheap storage
• Used for infrequently data, ideal for backups
• Very very slow, retrieval times (4-6 hours)
• High durability
• Cost of restore
• AES 256 bit data encryption
• 0.01 per GB
• Tool for third party to leverage service

Create Glacier Vault, User , Policy and attach policy to Glacier Vault.

 

Chapter 5

AWS Compute Options

Elastic Computer Cloud (EC2)

• Scalable platform for VMs
• Supports windows/linux instances
• Amazon Machine Image (AMI) refers to virtual disk template (ex. OVA, OVF)
• Can export/import your own AMIs

Elastic Compute Unit (ECU)

• Relative measure of processing power
• One ECU is equivalent to 1.0 – 2.0 GHz 2007 Intel Opteron or 2007 Xeon
• Consistent amount of CPU capacity regardless of the hardware
• Each instance type may be based on different physical processor

EC2 Instance Types

Micro Instance: very low memory, limited resources

General Purpose: balance vCPU, memory , etc

Compute Instance: compute intensive

GPU Instances: Video modelling, Healthcare

Memory Optimized:

Storage Optimized

Informative Website: cdn.awsnow.info

Understanding EC2 Pricing

AWS EC2 Pricing:

http://aws.amazon.com/ec2/pricing/

On Demand Instances

• Default type
• Most expensive option
• No commitment
• Prices vary by AWS region
• Billed on an hourly basis

Reserved Instances (RI)

• Less expensive
• Requires a commitment (1 or 3 years)
• Has an upfront cost
• Lower hourly rate
• RI can be sold on the AWS Marketplace
• Larger one can be sold by fragmenting it
• You commit to utilization

Three Types:

1>   Light

2>   Medium

3>   Heavy:

   100% utilization, power off will not save any cost.

Spot Instances

• Unused spot instances
• Very cheap hourly rate
• Not guarantee
• Based on a bid
• Ideal for raw processing power, grid like applications

 

Chapter 6

Understanding AWS Networking, CloudWatch, and Auto Scaling

Virtual Private Cloud (VPC)

• Virtual Network
• Logically isolated network in the AWS cloud
• Control of Network architecture

Control of topology, subnet architecture, IP address routing, gateways

• Enhanced security

Can put security groups on inbound and outbound traffic.

Routing rules, Access Control List(ACL), or other traditional practice.

• Internetwork with other organizations
• Elastic IP address (public IPs)

First one is free, from second its chargeable.

• Enable hybrid cloud (site to site VPN)

Able to extend on premises data centre to connect to VPC on AWS, direct connect, better bandwidth, etc

• VPC cost=$0
• VPN cost is $0.05/hr

IMAGE: VPC

VPC Access (Ways to Connect to VPC from on-premises)

Gateway:

• Internet Gateway (IGW) (Virtual Device)

Ingress and egress

• Virtual Private Gateway (VPG)

AWS side of secure VPN

• Customer Gateway (CG)

Customer side of secure VPN

VPN:

• Direct Connect

Dedicated and isolated

No internet

HA connectivity supported

• Hardware-based VPN

On-premises to AWS over internet

HA connectivity supported

3^rd party brands supported

VPC Network Security

VPC works region wide

Subnet are availability zone specific. Subnet talk to each other by default across availability zone in a region. So, default things need to be studied and restricted as per requirement. It’s the same way we do in our traditional network.

IMAGE: VPC Subnet

• Resources may be getting internal IP address not necessarily public IP address. So, we need to assign specifically as per requirement.

Security Groups

• Resource level traffic firewall

Instance, ELB,etc

• Ingress and Egress
• Stateful

Return traffic allowed

           

Access Control Lists

• Source and Protocol filtering
• Subnet level traffic firewall

Separate inbound and outbound rule set

• Stateless

Traffic strictly filtered

VPC Peering

• Inter-VPC routing
• Same or different AWS account
• Connect to a company subnet, e.g: Consulting
• IP scheme should be different from the connecting VPC
• Transitive Peering is not allowed. VPC3 can’t connect to VPC2 using VPC1.
• In case of acquisition or partner company.

IMAGE: VPC Peering

Elastic Load Balancer

• Region wide load balancer
• Can be used internally or externally

LB traffic going outside or inside resources

• SSL termination and processing

If we are connecting to a load balancer, we need SSL certificate on all the web servers behind LB. Now with SSL termination we are having SSL certificate at load balancer so we don’t need to maintain certificate for all web servers behind the load balancer. So, all encryption decryption happens on load balancer instead of individual web server. This make the web server to perform better.

• ELB EC2 health checks

At the LB level, we can perform health checks on web servers and if found that a particular web server is experiencing performance issue or offline, LB take this web server out of the pool and route the traffic to others.

• Route 53(AWS DNS) performs ELB health checks

Here Route 53 performs health checks of LB and if there is performance issue, it routes the traffic to other LB.

IMAGE: Route 53 LB Health Check

Virtual Private Cloud (VPC)

Refer to Amazon account, show how it can created, nothing much.

Amazon Route 53 (DNS)

• DNS should be up all time
• Worldwide (All Edge Locations) Distributed DNS
• Has an API

Developer and its application wants some automation to DNS like all other AWS service, Route 53 has an API. e.g: GoDaddy wants leverage Amazon DNS server for its application. Now whenever anyone creates a record in DNS, GoDaddy will automatically creates those entries in Route 53.

• Server health checks
• Load Balancing, server round robin

CloudWatch

• Basic Monitoring (7metrics for free, 5 min interval)
• Detailed Monitoring (10 alarms, 1 million API requests, 1 min)

All AWS services has an API, can connect to CloudWatch for monitoring.

• Set alarms and alerts
• Notification via SES, SNS
• Custom monitoring through API

If we have an application to monitor, we can use CloudWatch API to connect our application to it.

• Integrate with Auto Scaling
• Mobile App for basic monitoring and management

Auto Scaling

• Expand or Shrink the EC2 instance on demand

Cloudwatch can monitor and trigger to expand in case 80% threshold is set and breached.

• Can manually schedule configuration also to auto scale. E.g: For month end processing, we set that add 20 additional instances.
• Notification
• It’s free.

It allows to provision other services. Other services are charged.

IMAGE: Auto Scaling

 

Chapter 7

Amazon Database Options, APIs, and Lambda

AWS Database Options

• RDS (Relational Database Services)

Structured Database, tables, tables are linked to other tables.

E.g: Mysql, Oracle, Microsoft SQL, PostgreSQL, Aurora

IMAGE: RDS DB

• Non-RDS

Is a flat file, one table only. Its much faster to query, indexing.

E.g: SimpleDB, DynamoDB(very fast, scalable, reliable), MongoDB, Couchbase.

IMAGE: Non-RDS DB

• In-Memory (Elastic Cache)
• Data Warehousing (Red Shift, Petabytes)
• Custom

Amazon APIs

• Application Programming Interface
• Application to Application communication method
• Almost every AWS service is API capable

e.g: search for S3 API.

• API Wrappers
• API Authentication

Amazon Lambda

• Event Driven Compute Service

Able to take action against an event,

E.g: One file is created in S3. Now an event triggered a script.

• Doesn’t require a VM
• Doesn’t require infrastructure
• Rapid response to events
• Thousands of functions can run
• Run only when needed

Before Lambda

• Provision a fleet of proxy machines to capture uploads
• For each upload, enqueuer a job to process it
• Provision a second fleet of servers to read and process jobs
• Pick a deployment solution
• Plan capacity, accounting for FT, long-term utilization
• Monitoring and Patching
• Migration to new instance types over time

Event Driven Compute

• Stateless, request-driven code called Lambda functions
• Triggered by events:

Put in S3

Write to a DynamoDB table

Transition in an EC2 instance

• Connective tissue for AWS Services

 

Chapter 8

Amazon Simple Services

Simple Email Service (SES)

• Cost effective bulk email services

E.g: Marketing services, advertisement, etc

• Cost based on number of emails sent
• Outbound only email sending service
• Leverage Amazon email reputation

But Amazon do it own filtration that if anyone is abusing it.

• Initially limited to 10,000 emails/day.

Simple Queue Service (SQS)

An application can send N number of messages and if it faster than the receiver then it might hang the receiver. For that we can use this service so that application can release the messages as per receiver bandwidth.

• Fast, reliable, scalable
• Unlimited messages and queue size
• Payload up to 256KB
• Billed in chunks of 64KB payloads
• First 1 million requests are free
• $0.5 / million SQS requests

Simple Notification Service (SNS)

It is a service that will go around between the different services, and it's able to carry messages and notifications between the different services.

It offers push messaging services. This could be triggered by anything.

e.g: 1> There are servers that are creating logs, and you want to be notified when a particular log happens. We can use Simple Notification Services for this.

2> Someone want to be notified when a particular file is stored into S3.

This service decoupled from the rest of the services.

Service-oriented architecture (SAAS): When you need notification services, pull from SNS. Email services, pull from SES. Queueing service, SQS.

Work with HTTP/HTTPS, Email, Email-json, Amazon SQS, Application. If anyone of these option is not available it means that option is not supported in that particular availability zones.

Configuring SNS

Create a Topic ->

Create subscription (select protocol: Email, endpoint: email@email.com) ->

Now Copy the arn details.

Go to glacier vault-> notification and paste this arn in notification. -> Save this.(Tick both of the jobs type)

 

Chapter 9

CloudFront, CloudFormation, Elastic Beanstalk, and CloudTrail

CloudFront

Global content delivery network

Leverages all AWS edge locations

Cache static content

Setup: Create a distribution under CloudFront, fill up all the details. It will generate a URL. This URL users can use to access. Now the requirement might be to use the company/application URL. This can be achieved by using Route 53, my creating a CNAME and assign this Amazon created URL to it. Now users will be using Company URL but Route 53 will redirect it to Amazon URL in the back hand.

CloudFormation

Automate AWS resource provisioning

Free Service

Deleting CloudFormation deletes all instances except data

Setup:

CloudFormation -> Create Stack ->

Elastic Beanstalk

It’s mainly for developers.