According to Microsoft, An Azure storage account contains all of your Azure Storage data objects: blobs, file shares, queues, tables, and disks. The storage account provides a unique namespace for your Azure Storage data that's accessible from anywhere in the world over HTTP or HTTPS. Data in your storage account is durable and highly available, secure, and massively scalable. Read more...
For this blog, we are concerned with only blob storage. We will create a storage account, and one container in it. And also we will create one virtual machine using which we can access the storage account (container). Storage account can't be accessible from the public internet and also we are not whitelisting the subnet in which the virtual machine resides.
All code used in this blog is available at https://github.com/lets-learn-it/terraform-learning/tree/azure/05-private-endpoint
We will create all infrastructure in 4 steps.
- resource group, virtual network, and subnets
- a virtual machine in the public subnet
- storage account with one container
- DNS zone and private endpoint
Directory structure
Resource group, V-net, and subnets
We need one variable, i.e. resource group name. let's create it first. I am hardcoding the value of it.
variable "resource_group_name" {
type = string
default = "qwerty12344321"
}All resources will be created in a single resource group for sake of simplisity. You can create different resource groups if you want.
resource "azurerm_resource_group" "example" {
name = var.resource_group_name
location = "East US"
}Now we need a virtual network and 2 subnets inside it. For one of the subnets, set flag enforce_private_link_endpoint_network_policies to true. It is necessary to create a private endpoint in that subnet.
resource "azurerm_virtual_network" "example" {
name = "example-network"
address_space = ["10.0.0.0/16"]
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
}
# we will create VM in this subnet
resource "azurerm_subnet" "public_subnet" {
name = "public_subnet"
resource_group_name = azurerm_resource_group.example.name
virtual_network_name = azurerm_virtual_network.example.name
address_prefixes = ["10.0.1.0/24"]
}
# we will create private endpoint in this subnet
resource "azurerm_subnet" "endpoint_subnet" {
name = "endpoint_subnet"
resource_group_name = azurerm_resource_group.example.name
virtual_network_name = azurerm_virtual_network.example.name
address_prefixes = ["10.0.2.0/24"]
enforce_private_link_endpoint_network_policies = true
}The virtual machine in the public subnet
Now, we will write a module to create a virtual machine. We will place it in the public subnet and give it public IP so that we can connect it using SSH. for creating a module, create a directory named vm and start creating files in it.
Create public IP first, so that we can attach it to the network interface.
resource "azurerm_public_ip" "public_ip" {
name = format("%s_%s", var.name, "ip")
resource_group_name = var.resource_group_name
location = var.location
allocation_method = "Dynamic"
}Now we can create a network interface and attach public IP (previously created) to it.
resource "azurerm_network_interface" "example" {
name = format("%s_%s", var.name, "network_interface")
location = var.location
resource_group_name = var.resource_group_name
ip_configuration {
name = "internal"
subnet_id = var.subnet_id
private_ip_address_allocation = "Dynamic"
public_ip_address_id = azurerm_public_ip.public_ip.id
}
}As the network interface is available to use, we will create an ubuntu server VM.
resource "azurerm_linux_virtual_machine" "example" {
name = format("%s%s", var.name, "vm")
resource_group_name = var.resource_group_name
location = var.location
size = "Standard_B1s"
admin_username = "adminuser"
network_interface_ids = [
azurerm_network_interface.example.id,
]
admin_ssh_key {
username = "adminuser"
public_key = file(var.public_key_path)
}
os_disk {
caching = "ReadWrite"
storage_account_type = "Standard_LRS"
}
source_image_reference {
publisher = "Canonical"
offer = "UbuntuServer"
sku = "16.04-LTS"
version = "latest"
}
}We want to access this VM using SSH. For that, we need to add the network security group to the network interface. Let's create NSG and attach it to the already created network interface.
resource "azurerm_network_security_group" "nsg" {
name = format("%s_%s", var.name, "nsg")
location = var.location
resource_group_name = var.resource_group_name
security_rule {
name = "allow_ssh_sg"
priority = 100
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "22"
source_address_prefix = "*"
destination_address_prefix = "*"
}
depends_on = [
azurerm_network_interface.example
]
}
resource "azurerm_network_interface_security_group_association" "association" {
network_interface_id = azurerm_network_interface.example.id
network_security_group_id = azurerm_network_security_group.nsg.id
}As vm is a module and we will be passing variables while using it. Let's create these variables so that we can create a reusable module.
variable "resource_group_name" {
type = string
}
variable "public_key_path" {
type = string
}
variable "name" {
type = string
}
variable "subnet_id" {
type = string
}
variable "location" {
type = string
}This vm module will output public_ip of the virtual machine.
output "public_ip" {
value = azurerm_public_ip.public_ip.ip_address
}Our module for the virtual machines is ready. Let's use it.
module "vm" {
# using it from outside of vm directory
source = "./vm/"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
# make sure you have public key at this location
public_key_path = "C:/Users/wv3cxq/.ssh/id_rsa.pub"
name = "demo"
subnet_id = azurerm_subnet.public_subnet.id
}Storage account with one container
Now, we will write a module to create a storage account. for creating a module, create a directory named vm and start creating files in it.
resource "azurerm_storage_account" "storage" {
name = format("%s%s", var.name, "storage9553")
resource_group_name = var.resource_group_name
location = var.location
account_tier = "Standard"
account_replication_type = "GRS"
network_rules {
default_action = "Deny"
ip_rules = var.white_list_ip
}
}As we used variables in the virtual machine module, this module also needs some variables.
variable "resource_group_name" {
type = string
}
variable "name" {
type = string
}
variable "location" {
type = string
}
variable "white_list_ip" {
type = list(string)
default = []
}We will output some values using which we can connect to the storage account using Azure CLI. while outputting these values in production, please mark them as sensitive.
output "primary_connection_string" {
value = azurerm_storage_account.storage.primary_connection_string
}
output "storage_account_id" {
value = azurerm_storage_account.storage.id
}
output "primary_access_key" {
value = azurerm_storage_account.storage.primary_access_key
}
output "storage_account_name" {
value = azurerm_storage_account.storage.name
}Our module for the storage account is now complete. Let's use it to create a storage account and then we will create the container in it.
module "storage_account" {
source = "./storageaccount"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
name = "demo"
white_list_ip = []
}
resource "azurerm_storage_container" "container" {
name = "demo"
storage_account_name = module.storage_account.storage_account_name
container_access_type = "private"
}Now we have a storage account with us. You can check it using a console or add your machine's public IP in white_list_ip and try to access it using Azure CLI. We will see How to access blob storage using Azure CLI? at last of this blog.
DNS zone and private endpoint
Before creating a private endpoint, we need a private DNS zone so that we can create a record of private endpoint in this zone. We also need to link this zone to the virtual network (this is necessary else name resolution will fail).
resource "azurerm_private_dns_zone" "example" {
name = "privatelink.blob.core.windows.net"
resource_group_name = azurerm_resource_group.example.name
}
resource "azurerm_private_dns_zone_virtual_network_link" "network_link" {
name = "vnet_link"
resource_group_name = azurerm_resource_group.example.name
private_dns_zone_name = azurerm_private_dns_zone.example.name
virtual_network_id = azurerm_virtual_network.example.id
}Now the most important part will start. module creation for the private endpoint. We will create a record in this module itself, so that we can achieve reusability of the module.
resource "azurerm_private_endpoint" "endpoint" {
name = format("%s-%s", var.name, "private-endpoint")
location = var.location
resource_group_name = var.resource_group_name
subnet_id = var.subnet_id
private_service_connection {
name = format("%s-%s", var.name, "privateserviceconnection")
private_connection_resource_id = var.private_link_enabled_resource_id
is_manual_connection = false
subresource_names = var.subresource_names
}
}
resource "azurerm_private_dns_a_record" "dns_a" {
name = format("%s-%s", var.name, "arecord")
zone_name = var.private_dns_zone_name
resource_group_name = var.resource_group_name
ttl = 300
records = [azurerm_private_endpoint.endpoint.private_service_connection.0.private_ip_address]
}Let's create variables and outputs for this module also.
variable "resource_group_name" {
type = string
}
variable "name" {
type = string
}
variable "location" {
type = string
}
variable "subnet_id" {
type = string
}
variable "private_link_enabled_resource_id" {
type = string
}
variable "private_dns_zone_name" {
type = string
}
variable "subresource_names" {
type = list(string)
}We need a fully qualified domain name (FQDN) of a private endpoint to access the storage account. Let's output it.
output "dns_a_record" {
value = azurerm_private_dns_a_record.dns_a.fqdn
}Use this module and create a private endpoint with a record in previously created private DNS zone.
module "privateendpoint" {
source = "./privateendpoint/"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
name = "demo"
subnet_id = azurerm_subnet.endpoint_subnet.id
private_link_enabled_resource_id = module.storage_account.storage_account_id
private_dns_zone_name = azurerm_private_dns_zone.example.name
# you can add other subresouce also
subresource_names = ["blob"]
depends_on = [
azurerm_private_dns_zone.example
]
}Everything is in place except output values. We are outputting some values from the module but to get these, we need to output in the root directory also.
output "dns_a_record" {
value = module.privateendpoint.dns_a_record
}
output "primary_connection_string" {
value = module.storage_account.primary_connection_string
}
output "storage_account_id" {
value = module.storage_account.storage_account_id
}
output "public_ip" {
value = module.vm.public_ip
}
output "storage_primary_access_key" {
value = module.storage_account.primary_access_key
}Creating resources
First, run the following command to import all plugins and modules.
terraform initTo check what will get created? run plan. To create a plan, run the following command.
terraform planCheck plan, if everything is OK, you are free to create all resources with a single command.
terraform applyWhile creating resources using the apply command, you may get errors like below. This is coming because we are not allowing our machine which is running terraform apply to access our storage account. That's why terraform is not able to create containers. You can add your public IP in white_list_ip list.
All resources got created. after adding public IP to white_list_ip the list. A total of 15 resources is created. It is showing 1 added and 1 destroyed because I reran apply.
As you can see, I got public IP 20.119.70.83 as output. Let's connect it using SSH. but before that let me show you my storage account's networking from the console. I am allowing only access from my personal computer. Storage account should not be accessible to the virtual machine. But because of the private endpoint, It will access.
If you go to the Private endpoint connections tab, you can find the private endpoint which we created.
Connect to VM and Test access
To connect the virtual machine, run the following command,
ssh -i <private_key_path> adminuser@<public_ip>
To check access to the storage account, we need Azure CLI. But by default, it is not installed on the virtual machine. To install it first, using the following commands,
sudo apt udpate
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bashcheck DNS name resolution is working or not. To check that run the following command. you can find FQDN in output values. We are outputting it as dns_a_record. It should be pointing to private IP in endpoint subnet
nslookup demostorage9553.privatelink.blob.core.windows.net
I tried creating other record, other than
demostorage9553but it wasn't working. If anybody knows why and give me some resource to understand why it is not working? That will be great. I think mapping is based on record. and also pls tell me if there any way to give endpoint to az cli for storage account.
Now use Azure CLI to check access to blob storage. I added one object manually to demo container. Run the following command to access data from demo container. You can get connection string from outputs.
az storage blob list \
--container-name demo
--connection-string <connection_string>
Destroy all resources
terraform destroyReferences
Storage account overview
What is the private endpoint?
Use private endpoints for azure storage
Install Azure CLI in an ubuntu machine
