In this tutorial, we will explore how to create a chat room using the C programming language. We will start by implementing the server, which will allow multiple clients to connect and exchange messages in real time. By following this step-by-step guide, you will gain a better understanding of networking in C and learn how to build a simple chat application.

Full code can be found at https://github.com/lets-learn-it/c-learning/tree/master/16-networking/02-chat-room-PROJECT

Setting up server

The server will handle client connections, message reception, and message broadcasting.

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <string.h>
#include <sys/time.h>
#include <errno.h>

#define MAX_CLIENTS 4

int main(int argc, char const *argv[]) {
  int mastersockfd, connfds[MAX_CLIENTS];
  int activeconnections = 0;

  // Rest of the code...
}

The code starts by including necessary header files and defining the maximum number of clients (MAX_CLIENTS) that can connect to the server. We are initializing mastersockfd which will listen for new connections and connfds is an array of all live TCP connections.

We will create a socket using the AF_INET address family (IPv4) and SOCK_STREAM protocol (TCP). It returns a file descriptor for the socket. If the socket creation fails, an error message is printed, and the program exits.

if((mastersockfd = socket(AF_INET, SOCK_STREAM, 0)) <= 0) {
  perror("error while creating socket...");
  exit(1);
}

Next, we configure the server's address using the sockaddr_in structure. We set the address family, and port number (8081 in this example), and bind it to any available network interface (INADDR_ANY).

serv_addr.sin_family = AF_INET;
serv_addr.sin_port = htons(8081);
serv_addr.sin_addr.s_addr = INADDR_ANY;

To ensure that the socket can be reused immediately after the server is stopped, we set the SO_REUSEADDR option to use the setsockopt function. More can be found on https://stackoverflow.com/questions/2208581/socket-listen-doesnt-unbind-in-c-under-linux

int opt = 1;
setsockopt(mastersockfd, SOL_SOCKET, SO_REUSEADDR, (void *) &opt, sizeof(opt));

The bind function binds the socket to the server's address. If the binding fails, an error message is printed, and the program exits.

if(bind(mastersockfd, (struct sockaddr *) &serv_addr, addrlen) < 0) {
  perror("bind failed...");
  exit(1);
}

We use the listen function to listen for incoming client connections on the socket. The second argument (3 in this case) specifies the maximum number of pending connections that can be queued.

if(listen(mastersockfd, 3) < 0) {
  perror("Listen failed ...");
  exit(1);
}

Asynchronous I/O Multiplexing

As we will be dealing with multiple clients at the same time, How can we know which connection is ready with new data? We want to be notified when the connection is ready with new data or ready for reading. This capability is called I/O multiplexing. and is provided by select and poll functions. More on this can be read at https://notes.shichao.io/unp/ch6/.

We will be using select for working with asynchronous I/O multiplexing.  To begin with, we need to define the necessary variables. The fd_set structure is used to hold the file descriptors. We declare readfds as an fd_set to keep track of the file descriptors available for reading. We also define variables max_fd and readyfds.

fd_set readfds;
int max_fd, readyfds;

In infinite while loop, we will continuously check for incoming connections and messages from clients. At the start of each iteration, we clear the readfds set using FD_ZERO to remove any previously set file descriptors. Next, we add the master socket (mastersockfd) to the readfds set using FD_SET. The master socket is the socket that listens for new client connections. We also update the max_fd variable with the maximum file descriptor value. We iterate through the array of active client sockets (connfds) and add each valid socket to the readfds set using FD_SET. We also update the max_fd value if necessary.

FD_ZERO(&readfds);

// add mastersockfd
FD_SET(mastersockfd, &readfds);
max_fd =  mastersockfd;

for(int i=0;i < activeconnections;i++) {
  if(connfds[i] != 0)
  	FD_SET(connfds[i], &readfds);
  if(connfds[i] > max_fd)
  	max_fd = connfds[i];
}

We call the select function to wait for activity on the file descriptors. It monitors the file descriptors specified in the readfds set and blocks until any of them become available for reading. The max_fd + 1 parameter ensures that select checks all file descriptors from 0 to max_fd. If the select call returns a negative value (readyfds < 0) and the error is not due to an interrupted system call (errno != EINTR), an error message is displayed.

readyfds = select(max_fd + 1, &readfds, NULL, NULL, NULL);

if ((readyfds < 0) && (errno!=EINTR)) {
  printf("select error");
}

We check if the master socket is part of the readfds set using FD_ISSET. If so, it means a new client is attempting to connect. We accept the connection using the accept function and store the new socket descriptor in the connfds array. We also print the client's IP address using inet_ntoa and increment the activeconnections counter.

if(FD_ISSET(mastersockfd, &readfds)) {
  if((connfds[activeconnections] = accept(mastersockfd, (struct sockaddr *) &clientIPs[activeconnections], (socklen_t *) &addrlen)) < 0) {
    perror("accept error...");
    exit(1);
}

  fprintf(stdout, "New connection from %s\n",  inet_ntoa(clientIPs[activeconnections].sin_addr));
  activeconnections++;
}

For each active client socket, we check if it is part of the readfds set using FD_ISSET. If the socket is ready for reading, we clear the input and output buffers using memset. We read data from the client using the read function. If the return value is 0, it means the connection was closed normally, and if it is -1, an error occurred. We handle these cases by printing an error message, marking the connection as closed, and closing the socket. We then continue to the next iteration of the loop.

If the read operation is successful, we retrieve the client's IP address from clientIPs using inet_ntoa and store it in the output buffer. We also print the client's IP address and the received message to the console. We concatenate the client's IP address and the message into the output buffer. Then, we iterate through all active connections and write the message to each client except the sender, using the write function.

for(int i=0;i < activeconnections; i++) {
  // check if connection is active and it is ready to read
  if(connfds[i] != 0 && FD_ISSET(connfds[i], &readfds)) {
    // clear buffer
    memset(inBuffer[i], 0, 1024);
    memset(outBuffer[i], 0, 1024);

    // read returns 0 if connection closed normally
    // and -1 if error
    if(read(connfds[i], inBuffer[i], 1024) <= 0) {
      fprintf(stderr, "%s (code: %d)\n", strerror(errno), errno);
      strncpy(outBuffer[i], inet_ntoa(clientIPs[i].sin_addr), INET_ADDRSTRLEN);
      fprintf(stderr, "Host %s disconnected\n", outBuffer[i]);
      close(connfds[i]);
      connfds[i] = 0;
      continue;
    }

    // get client ip
    strncpy(outBuffer[i], inet_ntoa(clientIPs[i].sin_addr), INET_ADDRSTRLEN);

    fprintf(stdout, "%s: %s", outBuffer[i], inBuffer[i]);

    strcat(outBuffer[i], " : ");
    strcat(outBuffer[i], inBuffer[i]);

    for(int j=0;j<activeconnections;j++) {
       if(connfds[j] != 0 && i != j) {
         write(connfds[j], outBuffer[i], strlen(outBuffer[i]));
       }
    }
  }
}

Setting up client

Firstly, we define a function called readline that reads input from the user until a specified end-of-character is encountered. This function will be used to read input from the user and send it to the server.

int readline(char *buffer, int maxchars, char eoc) {
  int n = 0;
  while(n < maxchars) {
    buffer[n] = getc(stdin);
    if(buffer[n] == eoc)
      break;
    n++;
  }
  return n;
}

Moving on to the main function, we declare some variables and initialize them. sockfd represents the socket file descriptor, serv_addr is a structure that holds the server's address information, and sendline and recvline are character arrays to store the messages to be sent and received, respectively.

int sockfd;
struct sockaddr_in serv_addr;
char sendline[1024], recvline[1024];

We set up the server address by assigning the address family (AF_INET) and the port number (8081) to the serv_addr structure. Additionally, we convert the server IP address from a string format to binary using the inet_pton function. If the conversion fails, an error message is displayed, and the program exits.

serv_addr.sin_family = AF_INET;
serv_addr.sin_port = htons(8081);

if(inet_pton(AF_INET, "127.0.0.1", &serv_addr.sin_addr)<=0) { 
  perror("addre");
  exit(-1);
} 

To establish a connection with the server, we use the connect function. It takes the socket file descriptor (sockfd), the server address structure, and its size as arguments. If the connection fails, an error message is printed, and the program exits.

if(connect(sockfd, (struct sockaddr *) &serv_addr, sizeof serv_addr) < 0) {
  perror("connect error...");
  exit(1);
}

We set up a loop that continuously listens for input from the user and messages from the server. The fd_set data structure is used to monitor the file descriptors for activity. In our case, we monitor the user input (0) and the socket (sockfd) for readability.

Within the loop, we call select to check for any active file descriptors. If select returns a value less than 0 and the error is not due to interruption (EINTR), an error message is printed.

If the user input (stdin) is ready for reading, we call the readline function to read the input from the user and store it in the sendline buffer. Then, we use the write function to send the input to the server.

If the socket is ready for reading, we use the read function to receive data from the server and store it in the recvline buffer. Finally, we print the received data to the standard output using fprintf.

fd_set waitfds;
int readyfds;
while(1) {
  FD_ZERO(&waitfds);

  // add mastersockfd
  FD_SET(sockfd, &waitfds);
  FD_SET(0, &waitfds);

  memset(recvline, 0, 1024);
  memset(sendline, 0, 1024);

  // connfd will be always largest
  readyfds = select(sockfd + 1, &waitfds, NULL, NULL, NULL);
  if ((readyfds < 0) && (errno!=EINTR)) {
    printf("select error");
  }

  // if stdin ready, read it and send
  if(FD_ISSET(0, &waitfds)) {
    readline(sendline, 1024, '\n');
    write(sockfd, sendline, strlen(sendline));
  }
    
  // if socket ready, read it and print
  if(FD_ISSET(sockfd, &waitfds)) {
    read(sockfd, recvline, 1024);
    fprintf(stdout, "%s", recvline);
  }
}

When I embarked on my journey of learning and investing in the stock market, I encountered a multitude of terms and abbreviations that often left me feeling confused. One such set of terms includes the PE ratio, PB ratio, and more. As I continue to learn, I am writing this blog not only for my future reference but also to assist individuals who may be struggling with these concepts.

Total Revenue

Total revenue refers to the overall amount of income or earnings generated by a company during a specific period of time. An increasing total revenue generally indicates that the company is expanding or experiencing growth.

EBITDA (Earnings Before Interest, Taxes, Depreciation, and Amortization)

EBITDA represents the total revenue after deducting various expenses, such as the cost of raw materials, power/fuel costs, employee salaries, and selling and administrative expenses. EBITDA is considered a valuable measure as it provides a clearer view of a company's operational performance by eliminating the effects of financing decisions, accounting methods, and tax regulations.

PBIT/EBIT (Profit/Earnings Before Interest & Taxes)

This metric encompasses EBITDA while also accounting for depreciation and amortization. In simple words, it is EBITDA when depreciation and amortization are also considered.

PBT (Profit Before Taxes)

PBT is an important measure as it helps assess the profitability of a company's core operations before accounting for tax obligations. It provides insights into the financial performance and operating efficiency of the company.

Net Income

Pure profit. Net income, also known as net profit or net earnings, is a financial metric that represents the total profit generated by a company after deducting all expenses, including taxes, interest, and non-operating items, from its total revenue or sales. It provides a comprehensive picture of the company's financial performance.

EPS (Earnings per share)

EPS is calculated by dividing the company's net income by the total number of outstanding shares. It indicates the earnings generated per share.

DPS (Dividend Per Share)

It is a financial metric that represents the amount of cash a company distributes to its shareholders for each share they own as dividends. This metric can be misleading because suppose Company 1 has a current share price of 1000 and gives DPS 10 and Company 2 with a share price of 100 gives DPS 10. Both companies are paying DPS as 1 but as an investor, I may prefer Company 2. That's Why I prefer dividend yield.

Payout Ratio (Total Dividend/Net Income)

The payout ratio is calculated by dividing the total dividends paid by the company by its earnings. The payout ratio is expressed as a percentage. For example, a payout ratio of 50% means that 50% of the company's net income is being paid out as dividends, while the remaining 50% is retained by the company for reinvestment or other purposes.

Dividend Yield

The dividend yield is a financial ratio that measures the return on investment (ROI) from dividends received by shareholders relative to the market price of a company's stock. It indicates the percentage of return an investor can expect to receive in the form of dividends based on the current stock price.

In the same example from DPS, Company 1 has a dividend yield of 1% while Company 2 has a dividend yield of 10%.

PE Ratio (Price to Earnings Ratio)

Price to earnings ratio. It is calculated by dividing the market price per share of a company's stock by its earnings per share (EPS).

The P/E ratio provides insights into how much investors are willing to pay for each rupee of earnings generated by a company. It is often used as a valuation tool to compare the price of a stock to its earnings potential.

PB Ratio (Price to Book Ratio)

It is used to evaluate the market value of a company relative to its book value. It compares the current market price per share of a company's stock to its book value per share.

It's important to note that the interpretation of the PB ratio can vary across industries. Some industries, such as technology or high-growth sectors, tend to have higher PB ratios due to their intangible assets and growth prospects. Other industries, such as utilities or mature industries, may have lower PB ratios due to their tangible asset base and stable cash flows.

The word multi-bagger is common in the stock market. Knowing its meaning, everyone goes in search of multi-bagger shares. So first let us explain the meaning o multi-bagger,

What is multi-bagger stock?

A multi-bagger stock is a stock whose price has multiplied many times its purchase price. If the price doubled, it became two baggers, if the price increased twenty times, then it became twenty baggers, and if it increased 50 times, then it became fifty baggers.

Finding a stock that will multiply in the future is like finding a multi-bagger! Finding such a future multi-bagger means studying the stock that has multiplied many times in the past and buying shares by finding similarities between all the previous multi-baggers!

How to find multi-bagger stock?

Many studies investors have discovered a number of methods using this technique. One of them is the world-famous CANSLIM method invented by William O'Neill. Let us now understand this method -

William O'Neill studied multi-bagger stocks in the United States from 1953 to 1985 and discovered seven similarities. The word CANSLIM was made by taking the initials of these seven things to remember.

CANSLIM in detail

C = Current quarterly earnings per share. The company's quarterly profit should have increased by 20%. Each company publishes its accounts every three months. This year's profit should have increased by at least 20% over the same quarter last year. Eg. March 2017 profit should be at least 20% higher than March 2016 profit.

A = Annual earnings per share. The annual profit of the company should have been increasing a lot every year for the last five years.

N = New things (product, management, price). The company may have launched a new product or the company must have new management or the shares of the company should have recorded a new initial high of the price in the stock market.

S = Shares outstanding. The fewer shares available in the stock market for buying and selling, the fewer the number. If demand increases then supply should be less.

L = Leaders. Choose the leading company in that area/industry.

I = Institutional Ownership. The shares of this company should be invested in three to ten large institutions such as Mutual Funds, Insurance Companies, Pension Funds, and Foreign Institutional Investors. Such institutions study in depth before investing.

M = Market Direction. buy when the stock market is booming.

If any stock fulfilling all the above points, then we can say it is a possible multi-bagger stock.

Hooks are a way to extend the functionality of an application by allowing developers to "hook" into certain events or actions that occur during the application's lifecycle. Hooks can be used to modify the behavior of an application, add new features, or perform custom actions.

For example, imagine you have an application that performs some processing when a user logs in. You could use a hook to add some additional processing when the user logs in, such as sending a welcome email or logging the login event.

Hooks are widely used in various systems, including web applications, operating systems, and programming languages. In this blog post, we will be exploring the hook mechanism and how to implement it in the go programming language.

Hook Mechanism

The hook mechanism consists of three primary components: the hook interface, the hook register function, and the hook implementation. The hook interface defines the operations or blueprint for the hooks. All hooks should implement these operations. The hook register function is used to register custom hooks with the program. The hook implementation contains the custom code to be executed during the hook.

Implementing Hooks in Go

To implement hooks in Go, we'll first define an interface that hooks must implement. This interface will define the Init(), Perform(), and Destroy() methods that hooks must implement. Here is an example of what this interface might look like:

type Hook interface {
  Init()
  Perform()
  Destroy()
}

Next, we'll define a registration function that hooks can use to register themselves with the application. This function will take an instance of the hook interface and add it to a list of registered hooks. Here is an example of what this function might look like:

var hooks []Hook

func Register(h Hook) {
  hooks = append(hooks, h)
}

Now that we have our interface and registration function defined, we can start implementing our hooks. Let's take a look at an example of a simple hook:

//go:build hook_1
// +build hook_1

package hooks

import (
  "fmt"
)

type MyHook1 struct {
  name string
}

func (m *MyHook1) Init() {
  fmt.Println("Initializing Hook 1")
}

func (m *MyHook1) Perform() {
  fmt.Println("Performing Hook 1")
}

func (m *MyHook1) Destroy() {
  fmt.Println("Destroying Hook 1")
}

func init() {
  fmt.Println("Registering hook 1")
  Register(&MyHook1{name: "MyHook1"})
}

Finally, we modify the main function of our program to get all the registered hooks and call their Init(), Perform(), and Destroy() functions.

func main() {
  hooks := hooks.GetHooks()

  fmt.Println("Got hooks", len(hooks))

  for _, v := range hooks {
    v.Init()
  }

  for _, v := range hooks {
    v.Perform()
  }

  for _, v := range hooks {
    v.Destroy()
  }
}

The program first retrieves all the registered hooks and calls their Init() function to initialize the hook-specific data. Then, the program calls the Perform() function of each hook, which performs the hook's functionality. Finally, the program calls the Destroy() function of each hook to clean up the hook.

Regarding the build comments ( //go:build hook_1 or // +build hook_1 ), you can use them to control which hooks are included in the built binary. For example, if you want to include only the first hook in the binary, you can use the following command to build the application:

go build -tags hook_1

This command will include only the files that match the build tag hook_1, which in this case will be our above hook. If you want to include all hooks, you can omit the build tag:

go build

In conclusion, the hook mechanism in Go allows developers to add hooks to their code, which can be executed at specific times during program execution. This mechanism provides developers with flexibility and the ability to add custom snippets. The code snippets and explanations in this blog post should help you get started with implementing hooks in your own Go programs.

If you want to see the complete code with all the hooks, you can check out the GitHub repository here: https://github.com/lets-learn-it/go-learning/tree/master/21-packages-and-modules/04-hook-mechanism. The repository includes a main.go file, a hooks folder containing various hooks, and a register.go file, which is a helper for registering the hooks.

If you are ready to read a lot of web pages then please go to apache Kafka documentation because that will make sense And if you are comfortable watching video lectures then watching learn-apache-Kafka-for-beginners instead of reading this blog will be cool. But if you want to know what is apache Kafka quickly, then stay with me.

Kafka is a pub-sub distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. You can write a lot of messages/events to Kafka and read those at your convenience.

Terminologies

Producers & Consumers

producers/publishers are clients who write data/events to Kafka, while consumers/subscribers are clients who read data/events from Kafka. Both consumers and producers may or may not have knowledge of each other's existence. Both are fully decoupled.

When producers write data to Kafka topic, that data is assigned to partitions randomly unless the key is provided. All events with the same key will be assigned to the same partition.

Brokers

Brokers are machines on which apache Kafka is running and providing its services both to producers and consumers. The system can have multiple brokers (and that is recommended for high availability and fault tolerance) and each broker can be identified by id. Each broker can have one or more topic partitions and those partitions will be replicated across multiple brokers.

If the client connects to any broker (called a bootstrap broker), the client will be connected to the entire cluster.

Topics, Partitions & Offsets

Topics are like tables in a database or virtual hosts in Rabbit MQ. Events are organized and durably stored in topics. Topics can have multiple producers writing events and at the same time, multiple consumers are consuming those events.

Topics are split into partitions like 0,1, 2, 3, ... etc And each message within the partition gets an incremental id called offset. And these offsets have meaning only in that specific partition.

Each topic has a replication factor that states how many times every partition in the topic should be replicated across different brokers. And it should be > 1. If one broker goes down, another broker serves data.

As each partition is replicated across brokers, then who will be the owner of that partition? Here comes the leader for partition, at any time only ONE broker can be the leader for a given partition. Only that leader can receive and serve data for that partition and other brokers will synchronize data.

Suppose in the below image broker with id 101 goes down, then also partition 0 of topic A is already present on the broker with id 102. So broker with id 102 will be the new leader for the partition 0.

Once data is written to a partition, it can't be changed (immutability)

As you can see in the above image, there is a topic that has 4 partitions.

Producers and Message Keys

Producers write data to topics (which are made of partitions). Producers automatically know to which broker & partition to write and in case of broker failure, producers will automatically recover.

Producers can choose to receive acknowledgment of data writes. default is acks=1

• acks=0: no ack (possible data loss)
• acks=1: ack from leader only (limited loss)
• acks=2: leader + replicas (no data loss)

Message Keys

Producers can send keys with messages (string, number, etc). When the key is null and the default partitioner is used, the record will be sent to one of the available partitions of the topic at random. A round-robin algorithm will be used to balance the messages among the partitions.

if the key is sent then all messages with the same key will go to the same partition.

Consumers and Consumer Groups

Consumers read data from the topic (identified by name). Consumers know which broker (leader) to read from and if a broker fails, how to recover. Data is read in order within each partition. The same message can be read multiple times if the consumer wants to.

Consumer Groups

Consumers can read data in consumer groups. Each consumer within the group reads from exclusive partitions. if the group has more consumers than partitions, then some consumers will be inactive and if the group has fewer consumers than partitions, then some consumers will read from multiple partitions.

Consumer Offset

Kafka stores the offsets at which a consumer group has been reading. It tells the consumer where it was the last time in the partition. & which is the next message to read? Offsets are committed live in a Kafka topic __consume_offsets.

When the consumer in the group has processed data received from Kafka, it should be committing the offsets. If consumers die, it will be able to read back from where it left off (thanks to committed consumer offsets).

Apache Kafka Guarantees

• Messages are appended to a topic partition in the order they are sent
• Consumers read messages in order stored in the topic partition.
• With a replication factor of N, producer & consumer can tolerate up to N-1 brokers being down.
• As long as the number of partitions remains constant for a topic (no new partitions), the message with the same key will always go to the same partition.

In this blog post, we will use KEDA (Kubernetes Event-driven Autoscaling) for autoscaling pod count based on Azure Service Bus Queue length. I will be using the local Kubernetes cluster but it should work with Azure Kubernetes Service (AKS) also. All code used in this blog post can be found https://github.com/lets-learn-it/keda-examples/tree/master/azure-service-bus-queue.

Plan of Action as below,

1. Deploy KEDA in Kubernetes cluster
2. Create Azure Service Bus Queue using Terraform
3. Writing Kubernetes configuration files
4. Testing

Deploy KEDA

KEDA can be installed in the Kubernetes cluster using Helm. More info https://keda.sh/docs/2.6/deploy/.  I deployed KEDA 2.6 in Kubernetes 1.22.x.

Azure Service Bus Queue

We need to create a resource group in order to create a service bus namespace & queue inside that namespace. Below is terraform code to achieve that.

resource "azurerm_resource_group" "kedaq-rg" {
  name     = "keda-demo-rg"
  location = "West Europe"
}

resource "azurerm_servicebus_namespace" "keda-namespace" {
  name                = var.servicebus-namespace-name
  location            = azurerm_resource_group.kedaq-rg.location
  resource_group_name = azurerm_resource_group.kedaq-rg.name
  sku                 = "Standard"

}

resource "azurerm_servicebus_queue" "keda-demoq" {
  name                = var.servicebus-queue-name
  namespace_name      = azurerm_servicebus_namespace.keda-namespace.name
  resource_group_name = azurerm_resource_group.kedaq-rg.name

  enable_partitioning = true
}

For the service bus namespace, we can use the default shared access policy but for the queue, we need to create one access policy. & make sure to create manage access policy (as per KEDA docs).

resource "azurerm_servicebus_queue_authorization_rule" "queuerule" {
  name                = "queuerule"
  namespace_name      = azurerm_servicebus_namespace.keda-namespace.name
  queue_name          = azurerm_servicebus_queue.keda-demoq.name
  resource_group_name = azurerm_resource_group.kedaq-rg.name

  # As per KEDA docs,
  # Service Bus Shared Access Policy needs to be of type Manage.
  # Manage access is required for KEDA to be able to get metrics from Service Bus.
  manage = true
  listen = true
  send   = true
}

make sure to declare variables used in the above configuration & also need to do output some values i.e. connection strings.

output "namespace_primary_connection_string" {
    value     = azurerm_servicebus_namespace.keda-namespace.default_primary_connection_string
    sensitive = true
}

output "queue_primary_connection_string" {
    value     = azurerm_servicebus_queue_authorization_rule.queuerule.primary_connection_string
    sensitive = true
}

Create resources

Run plan & apply to create resources in Azure. After running terraform, I got 1 Azure service bus namespace & one queue inside it.

Kubernetes configuration files

I will deploy Nginx as a deployment which will do nothing. We will add messages & remove messages from the queue manually to test scaling.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
spec:
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
      - name: demo-app
        image: nginx
        resources:
          limits:
            memory: "128Mi"
            cpu: "100m"
        ports:
        - containerPort: 8080
        env:
          # This is not required when using triggerauthentication object
          - name: keda-secret
            valueFrom: 
              secretKeyRef:
                name: queue-policy-secret
                key: connection-string

Everything in the above YAML is normal except the environment variable keda-secret. We can authorize KEDA ScaledObject to access our queue in multiple ways. One of the ways needs this secret.

We will see 2 ways to authorize KEDA to access the queue. There is another way of using identity but for simplicity, we will not see this way in this post.

Way 01 Use TriggerAuthentication

First create secret using default primary connection string of Azure service bus namespace, like below

apiVersion: v1
kind: Secret
metadata:
  name: namespace-secret
type: Opaque
data:
  # This is azure service bus namespace connection string
  connection: RW5kcG9pbnQ9c2I6Ly9rZWRhLXNlcnZpY2VidXMtbmFtZXNwYWNlLnNlcnZpY2VidXMud2luZG93cy5uZXQvO1NoYXJlZEFjY2Vzc0tleU5hbWU9Um9vdE1hbmFnZVNoYXJlZEFjY2Vzc0tleTtTaGFyZWRBY2Nlc3NLZXk9bGQ4akIyV042SWRlbzJkMWJobXpYR01SZWRIOXg5ZloremFxSGtmVUQrcz0=

Now, once we have a secret, we can create TriggerAuthentication for that secret.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-servicebus-auth
spec:
  secretTargetRef:
    - key: connection
      name: namespace-secret # name of secret
      parameter: connection # key in secret

The remaining piece in the puzzle is ScaledObject. Which we can create using below YAML,

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    kind: Deployment
    name: demo-app
  pollingInterval: 30
  cooldownPeriod: 60
  minReplicaCount: 1
  maxReplicaCount: 4

  triggers:
  - type: azure-servicebus
    metadata:
      queueName: keda-demoq
      messageCount: "5"
    authenticationRef:
        # reference to TriggerAuthentication
        name: azure-servicebus-auth

That's it, It will create HPA & if HPA is not created then check the logs of the operator. Make sure to check status of ScaledObject.

Testing

I sent 6 messages to the queue using the service bus explorer. And KEDA up scaled pods to 2. To check to downscale, remove some messages from the queue (make sure to wait for cooldownPeriod).

Way 02 Use pod's environment variable

We can use the pod's environment variable for authorization also. I personally discourage using it. In this method, we need the connection string of the queue as an environment variable of deployment. But before starting way 02, remove resources from way 01.

The secret will get changed (not connection string of namespace but of the queue itself. for this reason, we created authorization rule in terraform). And we don't need TriggeredAuthentication.

apiVersion: v1
kind: Secret
metadata:
  name: queue-policy-secret
type: Opaque
data:
  # This is queue connection string
  connection: RW5kcG9pbnQ9c2I6Ly9rZWRhLXNlcnZpY2VidXMtbmFtZXNwYWNlLnNlcnZpY2VidXMud2luZG93cy5uZXQvO1NoYXJlZEFjY2Vzc0tleU5hbWU9cXVldWVydWxlO1NoYXJlZEFjY2Vzc0tleT1pWTNzSlh5OWorL0wvZVhMYmU5RmFseWVDU1pMWHM4WDFQVTZ2dkhzeW1nPTtFbnRpdHlQYXRoPWtlZGEtZGVtb3E=

The only change in ScaledObject is connectionFromEnv parameter & removing TriggerAuthentication reference.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    kind: Deployment
    name: demo-app
  pollingInterval: 30
  cooldownPeriod: 60
  minReplicaCount: 1
  maxReplicaCount: 4

  triggers:
  - type: azure-servicebus
    metadata:
      queueName: keda-demoq
      messageCount: "5"
      # ENV var of deployment
      connectionFromEnv: keda-secret

This time, you can't see anything in front of Authentication but it should show Ready status & it will work.

Cleanup

Make sure to clean all resources, especially those you created in Azure.  No one wants to pay the cost for unused resources.

References

[1] https://keda.sh/docs/2.6/scalers/azure-service-bus/
[2] https://github.com/kedacore/keda/blob/main/pkg/scalers/azure_servicebus_scaler.go

Some time back, I read how to make branching faster? I don't remember where I read it. May be on Reddit. This blog & in the future also, I will try to write similar posts whenever I find new tricks. You can find all code in the GitHub repository.

In this post, we will see the simplest example of finding odd & even numbers. below code will be used for all possible solutions & only conditions will change.

#include<iostream>
using namespace std;

int main() {
    int n = 999999999;

    int even = 0;
    int odd = 0;

    for(int i=0;i<n;i++) {
    // chaning part
        if(i % 2) {
            odd ++;
        } else {
            even ++;
        }
    // changing part ends
    }

    cout << "Even: " << even << endl;
    cout << "Odd: " << odd << endl;

    return 0;
}

Using hyperfine

I will be using hyperfine for finding the performance of code. You can check it https://github.com/sharkdp/hyperfine.

A. Simplest solution

Using if & else with % (remainder operator) is the common solution.

if(i % 2) {
    odd ++;
} else {
    even ++;
}

The simplest solution took an average time of 2.9 seconds.

B. Using bitwise instead of the remainder

Let's try the bitwise operator instead of the remainder operator. Using i & 1 instead of i % 2 will also work.

if(i & 1) {
    odd ++;
} else {
    even ++;
}

It looks like the bitwise condition took less average time but max time is larger than the previous approach.

C. Remove branching completely

It is possible to remove branching completely and use bitwise operations only. In the first condition, we are checking if the number is odd then increment odd counter. We can achieve it using && operator like (i & 1) && odd++ . odd will be incremented when (i & 1) is true. Similarly, we can achieve else condition also.

(i & 1) && odd++;
(i & 1) || even++;

D. Remove duplicate (i & 1)

We can replace 2 statements using a complex one-liner like below.

((i & 1) && odd++) || even++;

And this is worse than the previous one. It took on average 2.681 seconds.

References

[1] Youtube: Branchless Programming: Why "If" is Sloowww... and what we can do about it!

In this blog post, we will start exploring identities & Role-based access control (RBAC) in Azure for accessing different Azure resources from applications. We will create an infrastructure consisting Azure Storage Account which will be having 2 blob storage containers. We will also create one virtual machine which will be allowed as a Reader for that storage account and also a Contributor for one of the blob storage containers.

What is System Assigned Identity?

Azure will give some ID in Azure Active Directory to azure resources which are created by us. This kind of ID will get created at the time of resource created & gets destroyed at the time of resource destruction. only that Azure resources can use this identity to request tokens from Azure AD.

You can't share these IDs with other Azure resources. It is meant for only that resource.

All code used in this post is available at https://github.com/lets-learn-it/terraform-learning/tree/azure/07-system-assigned-identities

Plan of Action

1. Creating resource group, vnet, & 1 public subnet.
2. A virtual machine in public subnet with SSH access allowed & also system assigned identity.
3. Storage account with 2 blob storage containers.
4. Role assignments
5. Checking access from the virtual machine

We will create 2 modules for the Virtual machine & Storage account. Below is my folder structure

Creating RG, Vnet & subnet

Resource Group

All resources will be placed in this single resource group.

resource "azurerm_resource_group" "example" {
  name     = var.resource_group_name
  location = "East US"
}

Vnet & public subnet

We need one public subnet for virtual machines & make sure to specify a service endpoint for storage.

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_subnet" "public_subnet" {
  name                 = "public_subnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.1.0/24"]
  service_endpoints    = ["Microsoft.Storage"]
}

Creating Virtual Machine (Module)

Creating Public IP

We need public IP so that we can SSH into this machine.

resource "azurerm_public_ip" "public_ip" {
  name                = format("%s_%s", var.name, "ip")
  resource_group_name = var.resource_group_name
  location            = var.location
  allocation_method   = "Dynamic"
}

Network Interface

resource "azurerm_network_interface" "example" {
  name                = format("%s_%s", var.name, "network_interface")
  location            = var.location
  resource_group_name = var.resource_group_name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = var.subnet_id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id          = azurerm_public_ip.public_ip.id
  }
}

Network Security Group

While creating NSG, make sure to open port 22 for protocol TCP (for SSH access). Associate network interface (previously created) with NSG.

resource "azurerm_network_security_group" "nsg" {
  name                = format("%s_%s", var.name, "nsg")
  location            = var.location
  resource_group_name = var.resource_group_name

  security_rule {
    name                       = "allow_ssh_sg"
    priority                   = 100 
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }

  depends_on = [
    azurerm_network_interface.example
  ]
}

resource "azurerm_network_interface_security_group_association" "association" {
  network_interface_id      = azurerm_network_interface.example.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

Virtual Machine

Now create a virtual machine with system assigned identity

resource "azurerm_linux_virtual_machine" "example" {
  name                = format("%s%s", var.name, "vm")
  resource_group_name = var.resource_group_name
  location            = var.location
  size                = "Standard_B1s"
  admin_username      = "adminuser"

  network_interface_ids = [
    azurerm_network_interface.example.id,
  ]

  admin_ssh_key {
    username   = "adminuser"
    public_key = file(var.public_key_path)
  }

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }

  identity {
    type = "SystemAssigned"
  }
}

We need to output some variables which are required while creating infrastructure like principal_id for role assignment.

output "public_ip" {
  value = azurerm_public_ip.public_ip.ip_address
}

output "vm_id" {
  value = azurerm_linux_virtual_machine.example.id
}

output "vm_pricipal_id" {
  value = azurerm_linux_virtual_machine.example.identity[0].principal_id
}

Now, use the above module & create a virtual machine but make sure to declare all variables used in that module.

module "vm" {
    source = "./vm/"
    
    resource_group_name = azurerm_resource_group.example.name
    location            = azurerm_resource_group.example.location
    public_key_path     = "<public_key_path>"
    name                = "demo"
    subnet_id           = azurerm_subnet.public_subnet.id
}

Creating Storage Account

Module for the storage account

resource "azurerm_storage_account" "storage" {
  name                     = format("%s%s", var.name, "storage9553")
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "LRS"

  network_rules {
      default_action = "Deny"
      ip_rules = var.white_list_ip
      virtual_network_subnet_ids = var.whitelist_subnet_ids
  }

  tags = {
    environment = "staging"
  }
}

Let's output some variables like storage_account_id

output "storage_account_id" {
    value = azurerm_storage_account.storage.id
}

Using storage account module & create infrastructure & also create 2 blob storage containers. Make sure to add your IP to the list white_list_ip else terraform unable t create containers.

module "storage_account" {
    source = "./storageaccount"

    resource_group_name = azurerm_resource_group.example.name
    location            = azurerm_resource_group.example.location
    name                = "demo12"
    # whitelist ip of machine from which terraform creating infra
    # else terraform apply will fail with 403
    white_list_ip        = ["106.210.242.214"]
    whitelist_subnet_ids = [azurerm_subnet.public_subnet.id]
}

resource "azurerm_storage_container" "container" {
  name                  = "demo"
  storage_account_name  = module.storage_account.storage_account_name
  container_access_type = "private"
  depends_on = [
    module.storage_account
  ]
}

resource "azurerm_storage_container" "container2" {
  name                  = "demo1"
  storage_account_name  = module.storage_account.storage_account_name
  container_access_type = "private"
  depends_on = [
    module.storage_account
  ]
}

Role Assignments

We will assign Reader role to storage account & Storage Blob Data Contributor role to demo1 container. This means we can list containers in the storage account but are only able to write in demo1 container. The above 2 roles are already defined by Azure so no need to create them.

# Read role for storage account
resource "azurerm_role_assignment" "storage" {
    scope = module.storage_account.storage_account_id

    # using azure defined role
    role_definition_name = "Reader"

    principal_id = module.vm.vm_pricipal_id
}

# Write role for container
resource "azurerm_role_assignment" "container" {
    scope = azurerm_storage_container.container2.resource_manager_id

    # using azure defined role
    role_definition_name = "Storage Blob Data Contributor"

    principal_id = module.vm.vm_pricipal_id
}

When you apply the above terraform code, it will create 13 resources.

Checking Access from VM

Connect to the virtual machine using SSH

ssh -i <pvt_key> adminuser@<vm_ip>

Install az CLI in that machine. It depends on your machine's OS. If you used the same OS as I used then check https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt

Login

# login using identity
az login --identity

List containers

# list containers
# make sure that --auth-mode is login
az storage container list --account-name demo12storage9553 --auth-mode login

Upload File in demo1

First create one file.

echo "Hello World" > hello.txt

Now, upload hello.txt to demo1

# upload to demo2.
az storage blob upload --account-name demo12storage9553 --container-name demo1 --name hello-world.txt --file hello.txt --auth-mode login

Upload file in demo

VM doesn't have write access to demo the container. Let's try to upload file in demo

The End

Make sure to destroy all created resources.

Storing and managing logs is one of the important activities in software companies. Because using logs you can find root causes of multiple things like bugs, downtime, cyber attack, etc. Different cloud providers have their own managed logging and monitoring systems like AWS have cloud-watch, cloud-trail, etc and Azure has application insight. But suppose you have hundreds of microservices inside the Kubernetes cluster and you want easy to use, easy to set up logging system which will work exactly the same for all micro-services. You can use fluentd but in this article, I will show you how to use Event hub and Azure Data Explorer to collect and access logs in Azure.

As you can see in the feature image, your application which can be deployed in whether app service, Azure Kubernetes Service, or Virtual Machine will push logs to the Event hub and then Azure data explorer will do storing and querying job for you. In this article, I will run my simple flask application locally and it will push logs to the event hub which we access using azure data explorer.

Plan of action,

1. Create infrastructure using terraform
2. Writing a small flask web app
3. checking logs in ADX

You can find all code used in this article at https://github.com/lets-learn-it/terraform-learning/tree/azure/06-eh-adx-logging

Creating Infrastructure

I am using terraform version Terraform v1.0.11 Add Azure provider and make sure to use the version of azurem > "2.88.1" because we are using $Default in azurerm_kusto_eventhub_data_connection

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "2.88.1"
    }
  }
}

provider "azurerm" {
  features {}
}

Now create one resource group in which all our resources will reside. Like below

resource "azurerm_resource_group" "logs_rg" {
  name     = var.resource_group
  location = "East US"
}

Now create an event hub namespace and event hub in that namespace. We will use the default consumer group for this example.

resource "azurerm_eventhub_namespace" "eh_namespace" {
  name                = var.eh_namespace
  location            = azurerm_resource_group.logs_rg.location
  resource_group_name = azurerm_resource_group.logs_rg.name
  sku                 = "Standard"
  capacity            = 1
  zone_redundant      = true

  tags = var.tags
}

resource "azurerm_eventhub" "eh" {
  name                = var.eh_name
  namespace_name      = azurerm_eventhub_namespace.eh_namespace.name
  resource_group_name = var.resource_group
  partition_count     = 1
  message_retention   = 1
}

data "azurerm_eventhub_consumer_group" "default" {
  name                = "$Default"
  namespace_name      = azurerm_eventhub_namespace.eh_namespace.name
  eventhub_name       = azurerm_eventhub.eh.name
  resource_group_name = var.resource_group
}

We need an Azure data explorer cluster and database in it to store all logs.

resource "azurerm_kusto_cluster" "adx" {
  name                = var.adx_cluster
  location            = azurerm_resource_group.logs_rg.location
  resource_group_name = azurerm_resource_group.logs_rg.name
  engine              = "V3"
  double_encryption_enabled  = var.double_encryption

  sku {
    name     = var.adx_sku_name
    capacity = var.adx_sku_capacity
  }

  tags = var.tags
}

resource "azurerm_kusto_database" "database" {
  name                = var.adx_database
  resource_group_name = var.resource_group
  location            = azurerm_resource_group.logs_rg.location
  cluster_name        = azurerm_kusto_cluster.adx.name
  hot_cache_period    = var.hot_cache_period
  soft_delete_period  = var.soft_delete_period
}

Create variable find and add all variables we are using till now.

variable "resource_group" {
    description = "where we place event hub and azure data explorer"
    type = string
}

variable "adx_cluster" {
    description = "name of adx cluster"
    type = string
}

variable "adx_database" {
    type = string
    description = "name of adx dataset"
}

variable "eh_namespace" {
    type = string
    description = "name of event hub namespace"
}

variable "eh_name" {
    type = string
    description = "name of event hub"
}

variable "double_encryption" {
    type = bool
}

variable "hot_cache_period" {
    type = string
    description = "data will be cached for this no of days"
}

variable "soft_delete_period" {
    type = string
    description = "after these no of days data will be deleted"
}

variable "adx_sku_name" {
    type = string
    description = "type of adx cluster"
}

variable "adx_sku_capacity" {
    type = string
}

variable "tags" {
    type = map(string)
}

variable "adx_eh_connection_name" {
    type = string
}

variable "adx_db_table_name" {
    type = string
}

variable "ingestion_mapping_rule_name" {
    type = string
}

variable "eh_message_format" {
    type = string
    default = "JSON"
}

create terraform.tfvars file for providing values to all variables used,

resource_group = "eh-adx-logs"

adx_cluster = "logscluster"
adx_database = "logsdb"
double_encryption = true
hot_cache_period = "P31D"
soft_delete_period = "P365D"
adx_sku_name = "Standard_D11_v2"
adx_sku_capacity = 2

eh_namespace = "logseventhubns"
eh_name = "logs_eventhub"

adx_eh_connection_name = "adxehconn"
adx_db_table_name = "logs_table"
ingestion_mapping_rule_name = "logs_table_json_ingestion_mapping"

tags = {
    "environment": "prod"
}

To create these resources run a plan and apply.  You will see, terraform is creating 5 resources. (Azure data cluster took 15 min for me 😒)

terraform plan

# then run
terraform apply

Currently, terraform does not support creating tables in the database and mapping for tables so we will create it manually.

To open query editor, go to Azure data explorer cluster in Azure dashboard, and on the left side, you can find databases. In databases, you can find our database, logsdb. double click on it then select Query. In that run following table creation query.

.create table logs_table ( 
	level:string, 
    message:string, 
    loggerName:string, 
    exception:string, 
    applicationName:string, 
    processName:string, 
    processID:string, 
    threadName:string, 
    threadID:string, 
    timestamp:datetime
)

To create ingestion mapping, run the following query

.create table logs_table ingestion json mapping 'logs_table_json_ingestion_mapping' 
'[{"column":"level","Properties":{"path":"$.level"}},{"column":"message","Properties":{"path":"$.message"}},{"column":"loggerName","Properties":{"path":"$.loggerName"}},{"column":"exception","Properties":{"path":"$.exception"}},{"column":"applicationName","Properties":{"path":"$.applicationName"}},{"column":"processName","Properties":{"path":"$.processName"}},{"column":"processID","Properties":{"path":"$.processID"}},{"column":"threadName","Properties":{"path":"$.threadName"}},{"column":"threadID","Properties":{"path":"$.threadID"}},{"column":"timestamp","Properties":{"path":"$.timestamp"}}]'

After creating the table and mapping, we need to create a connection between the event hub and the azure data explorer. While creating infrastructure using terraform, we didn't create it because it need a table and mapping. Now add the following resources in Terraform and run the plan and apply again.

resource "azurerm_kusto_eventhub_data_connection" "eventhub_connection" {
  name                = var.adx_eh_connection_name
  resource_group_name = var.resource_group
  location            = azurerm_resource_group.logs_rg.location
  cluster_name        = azurerm_kusto_cluster.adx.name
  database_name       = azurerm_kusto_database.database.name

  eventhub_id    = azurerm_eventhub.eh.id
  consumer_group = data.azurerm_eventhub_consumer_group.default.name

  table_name        = var.adx_db_table_name
  mapping_rule_name = var.ingestion_mapping_rule_name
  data_format       = var.eh_message_format
}

Flask Application

To push logs to the event hub, we need EventhubHandler for python logging library. Install it using pip as follow

pip install EventhubHandler

Now, import required packages and create a logger. Make sure to use JSONFormatter it because our mapping is expecting JSON from the event hub. I am creating a root level logger so that all logs in all modules will go to the event hub.

from flask import Flask
import logging
from EventhubHandler.handler import EventHubHandler
from EventhubHandler.formatter import JSONFormatter
app = Flask(__name__)

logger = logging.getLogger()

eh = EventHubHandler()
eh.setLevel(logging.DEBUG)

# format will be depends on what you choose at adx
# I am using JSON
formatter = JSONFormatter({"level": "levelname", 
                            "message": "message", 
                            "loggerName": "name", 
                            "processName": "processName",
                            "processID": "process", 
                            "threadName": "threadName", 
                            "threadID": "thread",
                            "timestamp": "asctime",
                            "exception": "exc_info",
                            "applicationName": ""})
eh.setFormatter(formatter)
logger.addHandler(eh)

Write some endpoints so that we can test our flask application. I created /exception endpoint to check how exceptions are getting logged.

@app.route("/")
def hello_world():
    logger.info("inside hello world")
    return "<p>Hello, World!</p>"

@app.get("/exception")
def exception():
    try:
        x = 1 / 0
    except ZeroDivisionError as e:
        logger.exception('ZeroDivisionError: {0}'.format(e))
    return "Exception Occured"

if __name__ == '__main__':
    app.run(host="0.0.0.0")

Save all flask code in one file & name it main.py.  And make sure to set 3 environment variables as follow (for Linux based system),

export applicationName="my-log-app"
export eh_ns_connection_string=<event hub namespace connection string>
export eventhub_name="logs_eventhub"

Run application using the following command,

python main.py

Checking logs

To check logs, we need to run query in that same query editor. We set applicationName=my-log-app. Now using this we will get last 20 min logs

logs_table
| where applicationName contains "my-log-app"
| where timestamp > ago(20m)

When to use & when not

1. If you have 100s of microservices then only. My team is using this and we have 40 microservices. The surprising thing is we have never gone above 10% Azure data cluster utilization.
2. If you have fewer services then it will be too costly per service. check pricing https://azure.microsoft.com/en-in/pricing/details/data-explorer/#pricing

We will see the journey of the c program from source code to executable with help of the GCC compiler. We will see the input and output of all 4 steps involved in the compilation process.

The plan of action for this blog will be as follows,

1. Source c code
2. Preprocessing
3. Compiling
4. Assembling
5. Linking

The compilation is 2nd step of this process and this process is also called compilation overall. 😕

Source C Code

We will create 2 libraries named simple_math and algebra. simple_math consists of 2 functions sum and minus. while algebra consists of a single method evaluate. you can find the source code in the GitHub repository link.

#ifndef __SIMPLE_MATH__
#define __SIMPLE_MATH__

int sum(int a, int b);

int minus(int a, int b);

#endif

#include"algebra.h"
#include<stdio.h>
#include<string.h>

int evaluate(char exp[]) {
  int length = strlen(exp);

  // considering 0+exp
  int left = 0, right = 0;

  // function pointer so that 
  // we can call correct function 
  int (*last_sign)(int, int) = &sum;

  for(int i=0;i<length;i++){
    
    if(exp[i] == '+') {
      left = (*last_sign)(left, right);
      right = 0;
      last_sign = &sum;
    } else if(exp[i] == '-') {
      left = (*last_sign)(left, right);
      right = 0;
      last_sign = &minus;
    } else {
      right = (exp[i] - '0') + (right * 10);
    }
  }
  
  return (*last_sign)(left, right);
}

#include"algebra.h"
#include"simple_math.h"

#include<stdio.h>

#define "12-19+33+57"

int main(){
  printf("12-19+33+57=%d\n",evaluate(EXP));

  printf("Sum of 2 and 3: %d", sum(2, 3));
  return 0;
}

int sum(int a, int b) {
  return a + b;
}

int minus(int a, int b) {
  return a - b;
}

#ifndef __ALGEBRA__
#define __ALGEBRA__

#include"simple_math.h"

int evaluate(char a[]);

#endif

Preprocessing

At preprocessing stage, header files will get added recursively. This means in our example, algebra.h and simple_math.h will get added in the first pass and in the next pass, it will add simple_math.h coming from algebra.h recursively. It will also place value EXP wherever it is present.

Preprocessor processes include files, conditional compilation instructions, and macros. You can preprocess our code in GCC as follow,

# -E Preprocess only; do not compile, assemble or link.
gcc -E main.c -o main_pre.c

You can do the same thing with simple_math.c & algebra.c and create files simple_math_pre.c & algebra_pre.c  respectively.

Compilation

In this stage, we create assembly code from preprocessed files.

# -S Compile only; do not assemble or link.
# this will create assembly code
gcc -S main_pre.c -o main.s

You can open main.s in any text editor and check assembly code generated. Create similar assembly code files for the other 2 source files.

Assembling

During this stage, an assembler is used to translate the assembly instructions to object code. The output consists of actual instructions to be run by the target processor.

# -c Compile and assemble, but do not link.
# if already compiled then only assemble
gcc -c main.c -o main.o

# we cant see content of object file
# seeing object file
# elf format executable and linkable format
objdump -D main.o

Linking

It takes one or more files or libraries as input and combines to produce a single executable file. In this stage, it resolves references to external symbols, assigns final addresses to procedures/functions and variables, and variables, and revises code and data to reflect new addresses (a process called relocation)

# link all object file and libraries
# and create single executable
gcc main.o algebra.o simple_math.o -o main

Linking static & dynamic libraries

This step is not necessary for our example because we are not using any external library.

If you are using static libraries then you can link these static libraries as follow,

# -L is for specifyig path of library
# in our case in same folder 
# ldll means libdll.a
# you can directly user libdll.a instead -ldll
gcc <application_object_files> -L . -ldll
gcc <application_object_files> -L . libdll.a

Or if you are using dynamic or shared libraries then you can link as follows

# ldll is libdll library
# it should present in /usr/lib folder
gcc <application_object_files> -ldll

References

What Really Happens when a C program runs?

According to Microsoft, An Azure storage account contains all of your Azure Storage data objects: blobs, file shares, queues, tables, and disks. The storage account provides a unique namespace for your Azure Storage data that's accessible from anywhere in the world over HTTP or HTTPS. Data in your storage account is durable and highly available, secure, and massively scalable. Read more...

For this blog, we are concerned with only blob storage. We will create a storage account, and one container in it. And also we will create one virtual machine using which we can access the storage account (container). Storage account can't be accessible from the public internet and also we are not whitelisting the subnet in which the virtual machine resides.

All code used in this blog is available at https://github.com/lets-learn-it/terraform-learning/tree/azure/05-private-endpoint

We will create all infrastructure in 4 steps.

1. resource group, virtual network, and subnets
2. a virtual machine in the public subnet
3. storage account with one container
4. DNS zone and private endpoint

Directory structure

Resource group, V-net, and subnets

We need one variable, i.e. resource group name. let's create it first. I am hardcoding the value of it.

variable "resource_group_name" {
  type = string
  default = "qwerty12344321"
}

All resources will be created in a single resource group for sake of simplisity. You can create different resource groups if you want.

resource "azurerm_resource_group" "example" {
  name     = var.resource_group_name
  location = "East US"
}

Now we need a virtual network and 2 subnets inside it. For one of the subnets, set flag enforce_private_link_endpoint_network_policies to true. It is necessary to create a private endpoint in that subnet.

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

# we will create VM in this subnet
resource "azurerm_subnet" "public_subnet" {
  name                 = "public_subnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.1.0/24"]
}

# we will create private endpoint in this subnet
resource "azurerm_subnet" "endpoint_subnet" {
  name                 = "endpoint_subnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/24"]

  enforce_private_link_endpoint_network_policies = true
}

The virtual machine in the public subnet

Now, we will write a module to create a virtual machine. We will place it in the public subnet and give it public IP so that we can connect it using SSH. for creating a module, create a directory named vm and start creating files in it.

Create public IP first, so that we can attach it to the network interface.

resource "azurerm_public_ip" "public_ip" {
  name                = format("%s_%s", var.name, "ip")
  resource_group_name = var.resource_group_name
  location            = var.location
  allocation_method   = "Dynamic"
}

Now we can create a network interface and attach public IP (previously created) to it.

resource "azurerm_network_interface" "example" {
  name                = format("%s_%s", var.name, "network_interface")
  location            = var.location
  resource_group_name = var.resource_group_name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = var.subnet_id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id = azurerm_public_ip.public_ip.id
  }
}

As the network interface is available to use, we will create an ubuntu server VM.

resource "azurerm_linux_virtual_machine" "example" {
  name                = format("%s%s", var.name, "vm")
  resource_group_name = var.resource_group_name
  location            = var.location
  size                = "Standard_B1s"
  admin_username      = "adminuser"

  network_interface_ids = [
    azurerm_network_interface.example.id,
  ]

  admin_ssh_key {
    username   = "adminuser"
    public_key = file(var.public_key_path)
  }

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }

}

We want to access this VM using SSH. For that, we need to add the network security group to the network interface. Let's create NSG and attach it to the already created network interface.

resource "azurerm_network_security_group" "nsg" {
  name                = format("%s_%s", var.name, "nsg")
  location            = var.location
  resource_group_name = var.resource_group_name

  security_rule {
    name                       = "allow_ssh_sg"
    priority                   = 100 
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }

  depends_on = [
    azurerm_network_interface.example
  ]
}

resource "azurerm_network_interface_security_group_association" "association" {
  network_interface_id      = azurerm_network_interface.example.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

As vm is a module and we will be passing variables while using it. Let's create these variables so that we can create a reusable module.

variable "resource_group_name" {
  type = string
}

variable "public_key_path" {
  type = string
}

variable "name" {
  type = string
}

variable "subnet_id" {
  type = string
}

variable "location" {
  type = string
}

This vm module will output public_ip of the virtual machine.

output "public_ip" {
  value = azurerm_public_ip.public_ip.ip_address
}

Our module for the virtual machines is ready. Let's use it.

module "vm" {
	# using it from outside of vm directory
    source = "./vm/"
    
    resource_group_name = azurerm_resource_group.example.name
    location = azurerm_resource_group.example.location
    
    # make sure you have public key at this location
    public_key_path = "C:/Users/wv3cxq/.ssh/id_rsa.pub"
    name = "demo"
    subnet_id = azurerm_subnet.public_subnet.id
}

Storage account with one container

Now, we will write a module to create a storage account. for creating a module, create a directory named vm and start creating files in it.

resource "azurerm_storage_account" "storage" {
  name                     = format("%s%s", var.name, "storage9553")
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "GRS"

  network_rules {
      default_action = "Deny"
      ip_rules = var.white_list_ip
  }
}

As we used variables in the virtual machine module, this module also needs some variables.

variable "resource_group_name" {
  type = string
}

variable "name" {
  type = string
}

variable "location" {
  type = string
}

variable "white_list_ip" {
  type = list(string)
  default = []
}

We will output some values using which we can connect to the storage account using Azure CLI. while outputting these values in production, please mark them as sensitive.

output "primary_connection_string" {
  value = azurerm_storage_account.storage.primary_connection_string
}

output "storage_account_id" {
    value = azurerm_storage_account.storage.id
}

output "primary_access_key" {
    value = azurerm_storage_account.storage.primary_access_key
}

output "storage_account_name" {
  value = azurerm_storage_account.storage.name
}

Our module for the storage account is now complete. Let's use it to create a storage account and then we will create the container in it.

module "storage_account" {
    source = "./storageaccount"

    resource_group_name = azurerm_resource_group.example.name
    location = azurerm_resource_group.example.location
    name = "demo"
    
    white_list_ip = []
}

resource "azurerm_storage_container" "container" {
  name                  = "demo"
  storage_account_name  = module.storage_account.storage_account_name
  container_access_type = "private"
}

Now we have a storage account with us. You can check it using a console or add your machine's public IP in white_list_ip and try to access it using Azure CLI. We will see How to access blob storage using Azure CLI? at last of this blog.

DNS zone and private endpoint

Before creating a private endpoint, we need a private DNS zone so that we can create a record of private endpoint in this zone. We also need to link this zone to the virtual network (this is necessary else name resolution will fail).

resource "azurerm_private_dns_zone" "example" {
  name                = "privatelink.blob.core.windows.net"
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "network_link" {
  name                  = "vnet_link"
  resource_group_name   = azurerm_resource_group.example.name
  private_dns_zone_name = azurerm_private_dns_zone.example.name
  virtual_network_id    = azurerm_virtual_network.example.id
}

Now the most important part will start. module creation for the private endpoint. We will create a record in this module itself, so that we can achieve reusability of the module.

resource "azurerm_private_endpoint" "endpoint" {
  name                = format("%s-%s", var.name, "private-endpoint")
  location            = var.location
  resource_group_name = var.resource_group_name
  subnet_id           = var.subnet_id

  private_service_connection {
    name                           = format("%s-%s", var.name, "privateserviceconnection")
    private_connection_resource_id = var.private_link_enabled_resource_id
    is_manual_connection           = false
    subresource_names              = var.subresource_names
  }
}

resource "azurerm_private_dns_a_record" "dns_a" {
  name                = format("%s-%s", var.name, "arecord")
  zone_name           = var.private_dns_zone_name
  resource_group_name = var.resource_group_name
  ttl                 = 300
  records             = [azurerm_private_endpoint.endpoint.private_service_connection.0.private_ip_address]
}

Let's create variables and outputs for this module also.

variable "resource_group_name" {
  type = string
}

variable "name" {
  type = string
}

variable "location" {
  type = string
}

variable "subnet_id" {
  type = string
}

variable "private_link_enabled_resource_id" {
  type = string
}

variable "private_dns_zone_name" {
  type = string
}

variable "subresource_names" {
  type = list(string)
}

We need a fully qualified domain name (FQDN) of a private endpoint to access the storage account. Let's output it.

output "dns_a_record" {
    value = azurerm_private_dns_a_record.dns_a.fqdn
}

Use this module and create a private endpoint with a record in previously created private DNS zone.

module "privateendpoint" {
    source = "./privateendpoint/"

    resource_group_name = azurerm_resource_group.example.name
    location = azurerm_resource_group.example.location
    name = "demo"

    subnet_id = azurerm_subnet.endpoint_subnet.id
    private_link_enabled_resource_id = module.storage_account.storage_account_id
    private_dns_zone_name = azurerm_private_dns_zone.example.name
    
    # you can add other subresouce also
    subresource_names = ["blob"]

    depends_on = [
      azurerm_private_dns_zone.example
    ]
}

Everything is in place except output values. We are outputting some values from the module but to get these, we need to output in the root directory also.

output "dns_a_record" {
    value = module.privateendpoint.dns_a_record
}

output "primary_connection_string" {
  value = module.storage_account.primary_connection_string
}

output "storage_account_id" {
    value = module.storage_account.storage_account_id
}

output "public_ip" {
    value = module.vm.public_ip
}

output "storage_primary_access_key" {
    value = module.storage_account.primary_access_key
}

Creating resources

First, run the following command to import all plugins and modules.

terraform init

To check what will get created? run plan. To create a plan, run the following command.

terraform plan

Check plan, if everything is OK, you are free to create all resources with a single command.

terraform apply

While creating resources using the apply command, you may get errors like below. This is coming because we are not allowing our machine which is running terraform apply to access our storage account. That's why terraform is not able to create containers. You can add your public IP in white_list_ip list.

All resources got created. after adding public IP to white_list_ip the list. A total of 15 resources is created. It is showing 1 added and 1 destroyed because I reran apply.

As you can see, I got public IP 20.119.70.83  as output. Let's connect it using SSH. but before that let me show you my storage account's networking from the console. I am allowing only access from my personal computer. Storage account should not be accessible to the virtual machine. But because of the private endpoint, It will access.

If you go to the Private endpoint connections tab, you can find the private endpoint which we created.

Connect to VM and Test access

To connect the virtual machine, run the following command,

ssh -i <private_key_path> adminuser@<public_ip>

To check access to the storage account, we need Azure CLI. But by default, it is not installed on the virtual machine. To install it first, using the following commands,

sudo apt udpate
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

check DNS name resolution is working or not. To check that run the following command. you can find FQDN in output values. We are outputting it as dns_a_record. It should be pointing to private IP in endpoint subnet

nslookup demostorage9553.privatelink.blob.core.windows.net

I tried creating other record, other than demostorage9553 but it wasn't working. If anybody knows why and give me some resource to understand why it is not working? That will be great. I think mapping is based on record. and also pls tell me if there any way to give endpoint to az cli for storage account.

Now use Azure CLI to check access to blob storage. I added one object manually to demo container.  Run the following command to access data from demo container. You can get connection string from outputs.

az storage blob list \
  --container-name demo
  --connection-string <connection_string>

Destroy all resources

terraform destroy

References

Storage account overview
What is the private endpoint?
Use private endpoints for azure storage
Install Azure CLI in an ubuntu machine

Before jumping into code, you need to understand what is IaaC i.e. Infrastructure as a Code and what are the advantages of using it.

According to Wikipedia, Infrastructure as code is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. Read more...

Advantages of IaaC 🔥

• Speed and simplicity: by running the simple command you can create/destroy the whole infrastructure.
• Configuration consistency: Every time created infra. will be the same. (if code is same)
• Minimization of risk: Humans tends to do a mistake but computers not.
• Increased efficiency: No need to create infra manually every time.

So Terraform is a tool that is quite cool. In this blog post, we will create an Azure Virtual machine that will reside in the VPC and you can access it using SSH.

Spin Azure VM

you can find all code used in this post at https://github.com/lets-learn-it/terraform-learning/tree/azure/00-vm-instance

We will be using VPC with CIDR 10.0.0.0/16 with 1 subnet with CIDR 10.0.2.0/24. We are putting our VM instance in the network security group which allows SSH inbound connections and all outbound connections.

Add Provider block

Provider block tells terraform which kind of infrastructure we want to create. In our example, we are creating Azure. In terraform block, we can give the plugin version we want to use.

provider "azurerm" {
  features {}
}

Create Resource Group

We will create a resource group at the nearest data center. In my case, India will be the nearest data center. Resource group allows us to create all related resources in a single folder-like structure. If you delete the resource group, all resources in it will get deleted.

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "Central India"
}

V-net and subnet

We will create vnet with CIDR 10.0.0.0/16 with 1 subnet with CIDR 10.0.2.0/24. Our virtual machine will be in this subnet.

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_subnet" "example" {
  name                 = "internal"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/24"]
}

Public IP

To access our virtual machine, we need public IP. public IP can be static or dynamic. Static means after deleting or before creating the resource, IP will be available. but in Dynamic type, it is only available once the resource uses it available. We will be using Dynamic.

resource "azurerm_public_ip" "public_ip" {
  name                = "vm_public_ip"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  allocation_method   = "Dynamic"
}

Network Interface

A Network Interface (NIC) is an interconnection between a Virtual Machine and the underlying software network. An Azure Virtual Machine has one or more network interfaces attached to it. Any NIC can have one or more static or dynamic public and private IP addresses assigned to it. In our case, we are attaching Dynamic public IP to the network interface.

resource "azurerm_network_interface" "example" {
  name                = "example-nic"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.example.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id = azurerm_public_ip.public_ip.id
  }
}

Network Security Group

We are allowing only SSH connections. The lower the priority number, the higher the priority. By default, NSG denies all connections.

resource "azurerm_network_security_group" "nsg" {
  name                = "ssh_nsg"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  security_rule {
    name                       = "allow_ssh_sg"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
}

Associate NSG with interface

you can associate NSG with either subnet or network interface. In our case, the network interface is allowing access to the virtual machine. We will associate NSG with network interface as follow,

resource "azurerm_network_interface_security_group_association" "association" {
  network_interface_id      = azurerm_network_interface.example.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

Create Virtual Machine

Standard_B1s is the cheapest VM available. It will cost less than 1 rupee per hour. Attach network interface which we already created. To access that VM, we need to add ssh key also.

resource "azurerm_linux_virtual_machine" "example" {
  name                = "example-machine"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  size                = "Standard_B1s"
  admin_username      = "adminuser"

  network_interface_ids = [
    azurerm_network_interface.example.id,
  ]

  admin_ssh_key {
    username   = "adminuser"
    public_key = file("~/.ssh/id_rsa.pub")
  }

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

Output

We will output public IP so that we can connect to the machine using SSH.

output "public_ip" {
  value = azurerm_public_ip.public_ip.ip_address
}

Create Infrastructure 🛠

Initialize Terraform Plugin

We need to fetch terraform's Azurerm plugin before we start to create infrastructure. you can initialize the plugin using the following command,

terraform init

Check Plan

before creating AWS EC2, we can check the plan of infrastructure which can tell what resources terraform will create. To check plan, run following command,

terraform plan

Create Infra

Finally, we can create infrastructure. but make sure you are logged in to Azure using the cmd line. Run the following command to create our Azure Virtual machine. When it ask Enter the value: , give it yes.

terraform apply

After applying, you may not get public IP as output. So please run, terraform apply again. does anybody have any idea, why it is happening? I reran it.

you will get public IP as I got 20.204.9.163. We will connect to the machine using SSH.  Just make sure, you know the passphrase of the SSH key pair.

ssh -i ~/.ssh/id_rsa adminuser@20.204.9.163

Azure resource visualizer

You can see all resources created in form of a diagram in the Azure portal. Go to the resource group, you will find the resources visualizer option on the left side.

Destroy Infra

After using Azure VM, we want to destroy it to save money. Terraform can do that with a single command. It will ask you again for confirmation, give it yes

terraform destroy

Before jumping into code, you need to understand what is IaaC i.e. Infrastructure as a Code and what are the advantages of using it.

According to Wikipedia, Infrastructure as code is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. Read more...

Advantages of IaaC 🔥

• Speed and simplicity: by running the simple command you can create/destroy the whole infrastructure.
• Configuration consistency: Every time created infra. will be the same. (if code is same)
• Minimization of risk: Humans tends to do a mistake but computers not.
• Increased efficiency: No need to create infra manually every time.

So Terraform is a tool that is quite cool. In this blog post, we will create an EC2 machine that will reside in the default VPC and you can access it using SSH.

Spin EC2 Instance

According to Wikipedia, Amazon Elastic Compute Cloud (EC2) is a part of Amazon.com's cloud-computing platform, Amazon Web Services (AWS), that allows users to rent virtual computers on which to run their own computer applications.

you can find all code in this post  https://github.com/lets-learn-it/terraform-learning/tree/aws/00-ec2-instance

We will be using default VPC with CIDR 172.31.0.0/16 with 3 subnets. As you can see, we are using the default route table and Internet gateway for sake of simplicity. We are putting our EC2 instance in the security group which allows SSH inbound connections and all outbound connections.

Add Provider block

Provider block tells terraform which kind of infrastructure we want to create. In our example, we are creating AWS. In terraform block, we can give the plugin version we want to use.

provider "aws" {
  # I am using Mumbai region
  region = "ap-south-1"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.37.0"
    }
  }
}

Create key 🗝

To connect EC2 instances using SSH, we need ssh keys. Let's create these first. Run the following command, and type passphrase when it asks

ssh-keygen -t rsa -f ~/.ssh/ec2 

resource "aws_key_pair" "key" {
  key_name   = "parikshits_key"
  public_key = file("~/.ssh/ec2.pub")
}

Use default VPC

AWS creates default VPC in each region. For this example, we will use default VPC.

resource "aws_default_vpc" "default_vpc" {

}

Create Security Group 🛡

We want to connect to our EC2 machine using SSH, So we need to allow ssh traffic to our machine from anywhere.

resource "aws_security_group" "allow_ssh" {
  name        = "allow_ssh"
  description = "Allow ssh inbound traffic"
  
  # using default VPC
  vpc_id      = aws_default_vpc.default_vpc.id

  ingress {
    description = "TLS from VPC"
    
    # we should allow incoming and outoging
    # TCP packets
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    
    # allow all traffic
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "allow_ssh"
  }
}

Create EC2 Instance

We will create t2.micro instance (free tier) in this example.

resource "aws_instance" "my_ec2" {
  ami             = var.ami_id
  instance_type   = "t2.micro"
  
  # refering key which we created earlier
  key_name        = aws_key_pair.key.key_name
  
  # refering security group created earlier
  security_groups = [aws_security_group.allow_ssh.name]

  tags = var.tags
}

Add variables

We are using var.xxx, which are variables used in our code. We need to define these variables.

variable "ami_id" {
  description = "this is ubuntu ami id"
  
  # I am using amazon linux image
  default     = "ami-0a23ccb2cdd9286bb"
}

variable "tags" {
  type = map(string)
  default = {
    "name" = "parikshit's ec2"
  }
}

Output useful info

Terraform can output attributes of resources i.e. public_ip of EC2 instance. We need public IP to connect instances using SSH.

output "arn" {
  value = aws_instance.my_ec2.arn
}

output "public_ip" {
  value = aws_instance.my_ec2.public_ip
}

Create Infrastructure 🛠

Initialize Terraform Plugin

We need to fetch terraform's AWS plugin before we start to create infrastructure. you can initialize the plugin using the following command,

terraform init

Check Plan

before creating AWS EC2, we can check the plan of infrastructure which can tell what resources terraform will create. To check plan, run following command,

terraform plan

Terraform used the selected providers to generate the following execution plan. 
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_default_vpc.default_vpc will be created
  + resource "aws_default_vpc" "default_vpc" {
      + arn                              = (known after apply)
      + assign_generated_ipv6_cidr_block = (known after apply)
      + cidr_block                       = (known after apply)
      + default_network_acl_id           = (known after apply)
      + default_route_table_id           = (known after apply)
      + default_security_group_id        = (known after apply)
      + dhcp_options_id                  = (known after apply)
      + enable_classiclink               = (known after apply)
      + enable_classiclink_dns_support   = (known after apply)
      + enable_dns_hostnames             = (known after apply)
      + enable_dns_support               = true
      + id                               = (known after apply)
      + instance_tenancy                 = (known after apply)
      + ipv6_association_id              = (known after apply)
      + ipv6_cidr_block                  = (known after apply)
      + main_route_table_id              = (known after apply)
      + owner_id                         = (known after apply)
      + tags_all                         = (known after apply)
    }

  # aws_instance.my_ec2 will be created
  + resource "aws_instance" "my_ec2" {
      + ami                          = "ami-0a23ccb2cdd9286bb"
      + arn                          = (known after apply)
      + associate_public_ip_address  = (known after apply)
      + availability_zone            = (known after apply)
      + cpu_core_count               = (known after apply)
      + cpu_threads_per_core         = (known after apply)
      + get_password_data            = false
      + host_id                      = (known after apply)
      + id                           = (known after apply)
      + instance_state               = (known after apply)
      + instance_type                = "t2.micro"
      + ipv6_address_count           = (known after apply)
      + ipv6_addresses               = (known after apply)
      + key_name                     = "parikshits_key"
      + outpost_arn                  = (known after apply)
      + password_data                = (known after apply)
      + placement_group              = (known after apply)
      + primary_network_interface_id = (known after apply)
      + private_dns                  = (known after apply)
      + private_ip                   = (known after apply)
      + public_dns                   = (known after apply)
      + public_ip                    = (known after apply)
      + secondary_private_ips        = (known after apply)
      + security_groups              = [
          + "allow_ssh",
        ]
      + source_dest_check            = true
      + subnet_id                    = (known after apply)
      + tags                         = {
          + "name" = "parikshit's ec2"
        }
      + tenancy                      = (known after apply)
      + vpc_security_group_ids       = (known after apply)

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + enclave_options {
          + enabled = (known after apply)
        }

      + ephemeral_block_device {
          + device_name  = (known after apply)
          + no_device    = (known after apply)
          + virtual_name = (known after apply)
        }

      + metadata_options {
          + http_endpoint               = (known after apply)
          + http_put_response_hop_limit = (known after apply)
          + http_tokens                 = (known after apply)
        }

      + network_interface {
          + delete_on_termination = (known after apply)
          + device_index          = (known after apply)
          + network_interface_id  = (known after apply)
        }

      + root_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }
    }

  # aws_key_pair.key will be created
  + resource "aws_key_pair" "key" {
      + arn         = (known after apply)
      + fingerprint = (known after apply)
      + id          = (known after apply)
      + key_name    = "parikshits_key"
      + key_pair_id = (known after apply)
      + public_key  = "ssh-rsa "
    }

  # aws_security_group.allow_ssh will be created
  + resource "aws_security_group" "allow_ssh" {
      + arn                    = (known after apply)
      + description            = "Allow ssh inbound traffic"
      + egress                 = (known after apply)
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = "TLS from VPC"
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
        ]
      + name                   = "allow_ssh"
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Name" = "allow_ssh"
        }
      + vpc_id                 = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + arn       = (known after apply)
  + public_ip = (known after apply)

─────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, 
so Terraform can't guarantee to take exactly these actions 
if you run "terraform apply" now.

Create Infra

Finally, we can create infrastructure. but make sure you are logged in to AWS using the cmd line. Run the following command to create our AWS EC2 machine. When it ask Enter the value: , give it yes.

terraform apply

you will get public IP as I got 13.232.255.15. We will connect to the machine using SSH.

ssh -i ~/.ssh/ec2 ec2-user@13.232.255.15

Destroy Infra

After using AWS EC2, we want to destroy it to save money. Terraform can do that with a single command. It will ask you again for confirmation, give it yes

terraform destroy

ALFA AWUS036ACS is the cheapest USB Wireless Adapter available in the market which supports dual-band 2.4 and 5Ghz. It supports both monitor mode and packet injection mode.

I recently bought this in India from alfa's official site. https://alfanetwork.co.in/shop/ols/products/alfa-awus036acs. Which uses the Realtek RTL8811AU chipset. I didn't find any resources online for installing drivers for this chipset. I searched the whole day and finally got success. 😃

Features of ALFA AWUS036ACS

• Support monitor and packet injection mode.
• Support both 2.4 (Up to 150Mbps) and 5 GHz (Up to 433Mbps) frequencies.
• Comes with a 1 x dual-band detachable RP-SMA connector.
• Small in size (18 x 45 x 9 mm)

Driver Installation

Check USB Adapter available

lsusb

Download driver source and build

We will use an open-source rtl8812au driver from aircrack-ng.

git clone https://github.com/aircrack-ng/rtl8812au.git

Now you can go in rtl8812au directory.

Build Source and install

sudo make

sudo make install

Troubleshooting

Sometimes you may get an error saying file or directory not found.

make[1]: *** /lib/modules/5.10.0-kali3-amd64/build: No such file or directory. 
Stop

/lib/modules/****/build where **** can be anything depending on your OS.

For solving this problem, run the following commands and try again

sudo apt update
sudo apt upgrade
sudo apt dist-upgrade -y

After running these commands, restart the computer and try to install the driver again.

While sudo make, you may get errors like below

make ARCH=x86_64 CROSS_COMPILE= -C /lib/modules/5.16.0-kali7-amd64/build M=/home/sandbox/github/rtl8812au  modules
make[1]: *** /lib/modules/5.16.0-kali7-amd64/build: No such file or directory.  Stop.
make: *** [Makefile:2244: modules] Error 2

To resolve it, run the below command

sudo apt-get install linux-headers-$(uname -r)

Checking Adapter

Find Interface

iwconfig

Get Info about the adapter

iw list

Put Adapter in Monitor mode

# Turn off interface
sudo ifconfig wlx00c0caadd40c down

# Change mode
sudo iw wlx00c0caadd40c set monitor control

# Turn on interface
sudo ifconfig wlx00c0caadd40c up

# check mode
iwconfig

Try Packet Injection

For performing packet injection, you need to install aircrack-ng

sudo apt install aircrack-ng

For checking packet injection is working or not, connect to wifi and put the adapter in monitor mode then run the following command

# sudo aireplay-ng --test <interface>
sudo aireplay-ng --test wlx00c0caadd40c

This will be the \(3^{rd}\) and last article in neural network and deep learning the series. We will see multiclass classification in this article. Before that, check previous articles first and return to these articles. Below are the links to these previous articles.

Deep L-layer Neural Network from Scratch in Python

Basics of feed-forward neural networks. Generic implementation for L layer deep neural network in NumPy.

AvabodhaParikshit Patil

Still now we only saw binary classification problems (either 1 or 0). In this article, We will see multiclass (means one of the many classes) problem. Let's see directly what is MNIST handwritten digit dataset is. It contains 60000 images of handwritten digits (0-9). Total 10 classes 🙄 i.e. multiclass classification. all images are grayscale and 28x28 in size. You can check some samples in the below image.

Before that take simple network with multiclass output. 4 class neural network is shown in below image. Here, shape of \(\hat{y}\) will be (4, m) where m is number of examples. Generally shape of \(\hat{y}\) is (K, m) where K is number of classes.

Lets revise some terminologies. \(m\) will be number of samples in dataset and \(n_x\) will be size of input then \((n_x, m)\) will be the input matrix size. And \(n_y = K\) will be output size then \((n_y, m)\) will be the output matrix size. As per our above neural network, label will be one of the following column vector because only 1 out of 4 class will be present for a sample in our example. for our convenience, suppose cat, dog, frog and rabbit these are the classes. So \(\begin{bmatrix} 1\ 0\ 0\ 0\ \end{bmatrix}\) is the cat class. This kind of encoding is called as One Hot Encoding because for some particular sample only one value is hot (1) and other are cool (0).

Initialize Variables

In general, our data will look like the following matrices. \(X\) is input and \(Y\) is output. in below equation, \(K\) is number of classes and we use \(K\) and \(n_y\) interchangebly.

Before we jump into forward pass, lets create dataset. We will create random data and at last check whether cost is decreasing or not. If cost is decreasing after each epoch then we can say that our model is working fine.

import numpy as np

# set seed
np.random.seed(95)

# number of examples in dataset
m = 6
# number of classes
K = 4
# input shape
n_x = 3

# input (n_x, m)
X = np.random.rand(n_x, m)

# hypothetical random labels
labels = np.random.randint(K, size=m)

# convert to one hot encoded
Y = np.zeros((4,6))
for i in range(m):
  Y[labels[i]][i] = 1

print("X", X)
print("Labels",labels)
print("Y",Y)

Output

X:
 [[0.22880349 0.19068802 0.88635967 0.7189259  0.53298338 0.8694621 ]
 [0.72423768 0.48208699 0.7560772  0.97473999 0.5083671  0.95849135]
 [0.49426336 0.51716733 0.34406231 0.96975023 0.25608847 0.40327522]]
Labels:  [2 1 3 0 0 3]
Y:
 [[0. 0. 0. 1. 1. 0.]
 [0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 1.]]
 

Now time to randomly intialize weights and biases to 0. We will create units list which tells us, How may neurons in each layers?

# units in each layer
# first number (3) is input dimension
units = [3, 5, 5, 4]

# Total layers except input
L = len(units) - 1

# parameter dictionary
parameters = dict()

for layer in range(1, L+1):
    parameters['W' + str(layer)] =   np.random.rand(units[layer],units[layer-1])
    parameters['b' + str(layer)] = np.zeros((units[layer],1))

Forward Pass

We saw in forward pass in previous article and no change is required for feed forward. So let's copy equations from previous article. Normally people uses Softmax activation function at last layer but for simplicity, we will continue using Sigmoid.

\[
\begin{align*}
Z^{[l]} &= W^{[l]}.a^{[l-1]} + b^{[l]} \\
a^{[l]} &= g^{[l]}{(Z^{[l]})}
\end{align*}
\]

Lets write code for forward pass in python. We will use our generic code from previous article.

cache = dict()
cache['a0'] = X

for layer in range(1, L+1):
    cache['Z' + str(layer)] = np.dot(parameters['W' + str(layer)],cache['a' + str(layer-1)]) 
    						+ parameters['b' + str(layer)]
    cache['a' + str(layer)] = sigmoid(cache['Z' + str(layer)])

Backward Pass

Still what we saw is from previous article. Now the most important part i.e. Loss Function & Cost Function. We already saw difference between loss function and cost function in article 1 of this series. We will use Binary Cross Entropy generalized for \(K\) classes as our loss function. The formula for it is same as Binary Cross Entropy extended for multiple classes. Below, second equation is vectorized version of first equation.

So the cost function will be the addition of all losses over all examples. So cost function will be

Now its time of finding derivative of loss function. Before that check below image so that you can compare binary classification problem with multiclass classification. When we find \(dZ^L\), we are done. As per below image, finding \(dZ^L\) is tricky part and all gradient in layers before it can be found using our regular (generic) code. Check previous article.

In below equations, \(W\) & \(b\) are representing all weights and biases of network.

Above equation can further vectorized for speedup. I will start another series on High Performance Computing as early as possible. In that article, we will see advantages of vectorization.

\[
\begin{align*}
dZ^L &= \frac{1}{m} (\hat{y}-y)
\end{align*}
\]

Its code time 😇. Let's implement backpropagation. You may say that both codes from previous article and this article are same. it's because vectorized equation for \(dZ^L\) is same for both binary classification and multiclass classification. Remember our example with random data doesn't make any sense because it don't have any pattern. We care about cost is decreasing or not?. You can find colab notebook in references and code section. Just care about cost decreasing or not.

def cost(y,y_hat):
  return -np.sum(y*np.log(y_hat) + (1-y)*(np.log(1-y_hat)))

y_hat = cache['a' + str(L)]
cost_ = cost(y,y_hat)
cache['dZ' + str(L)] = (1/m) * (y_hat - Y)
cache['dW' + str(L)] = np.dot(cache['dZ' + str(L)], cache['a' + str(L-1)].T)
cache['db' + str(L)] = np.sum(cache['dZ' + str(L)], axis=1, keepdims=True)

for layer in range(L-1,0,-1):
  cache['dZ' + str(layer)] = np.dot(parameters['W' + str(layer+1)].T, cache['dZ' + str(layer+1)]) 
  							* inv_sigmoid(cache['Z' + str(layer)])
  cache['dW' + str(layer)] = np.dot(cache['dZ' + str(layer)], cache['a' + str(layer-1)].T)
  cache['db' + str(layer)] = np.sum(cache['dZ' + str(layer)], axis=1, keepdims=True)

MNIST Handwritten Digit Recognition

In above random data example, data and output doesn't make any sense so we will see real life example of handwritten digit classification. You already saw about dataset at start of this article.

Load Dataset

Now we will load data. Link to dataset is also given in references and code section.

import numpy as np
np.random.seed(95)

# Load CSV File
data = np.genfromtxt("mnist_train.csv",delimiter = ',')

# first column is labels
y = data[:,0]

# rest 784 columns are features / pixel values.
X = data[:,1:785].T

# Some constants
K = 10  # No of classes
alpha = 0.1  # Learning rate
m = 60000  # No of examples

# convert to one hot encoded
Y = np.zeros((K,m))
for i in range(m):
  Y[int(y[i]),i] = 1

# print shape of input and output/label
print('Shape of X:',X.shape)
print('Shape of y:',y.shape)
print('Shape of Y:',Y.shape)

Output

Shape of X: (784, 60000)
Shape of y: (60000,)
Shape of Y: (10, 60000)

Initialize Weights and Biases

# units in each layer
# first number (784) is input dimension
units = [784, 128, 64, 10]

# Total layers
L = len(units) - 1

# parameter dictionary
parameters = dict()

for layer in range(1, L+1):
    parameters['W' + str(layer)] = np.random.normal(0,1,(units[layer],units[layer-1]))
    parameters['b' + str(layer)] = np.zeros((units[layer],1))

Define Sigmoid and Derivative of Sigmoid Function

def sigmoid(X):
    return 1 / (1 + np.exp(- X))

def inv_sigmoid(X):
    return sigmoid(X) * (1-sigmoid(X))

Forward Pass

cache = dict()
def forward_pass(X):
  cache['a0'] = X

  for layer in range(1, L+1):
    cache['Z' + str(layer)] = np.dot(parameters['W' + str(layer)],cache['a' + str(layer-1)]) + parameters['b' + str(layer)]
    cache['a' + str(layer)] = sigmoid(cache['Z' + str(layer)])

Backward Pass

We will use batch size of 10 samples because of that you will find \(\frac{1}{10}\) instead of \(\frac{1}{m}\) (All data at a time).

def cost(y,y_hat):
  return -np.sum(y*np.log(y_hat) + (1-y)*(np.log(1-y_hat)))

def back_prop(Y):
  y_hat = cache['a' + str(L)]
  cache['dZ' + str(L)] = (1/10)*(y_hat - Y)
  cache['dW' + str(L)] = np.dot(cache['dZ' + str(L)], cache['a' + str(L-1)].T)
  cache['db' + str(L)] = np.sum(cache['dZ' + str(L)], axis=1, keepdims=True)

  for layer in range(L-1,0,-1):
    cache['dZ' + str(layer)] = np.dot(parameters['W' + str(layer+1)].T, cache['dZ' + str(layer+1)]) * inv_sigmoid(cache['Z' + str(layer)])
    cache['dW' + str(layer)] = np.dot(cache['dZ' + str(layer)], cache['a' + str(layer-1)].T)
    cache['db' + str(layer)] = np.sum(cache['dZ' + str(layer)], axis=1, keepdims=True)

Update Weights and Biases

def update_weights():
  for layer in range(1, L+1):
    parameters['W' + str(layer)] = parameters['W' + str(layer)] - alpha * cache['dW' + str(layer)]
    parameters['b' + str(layer)] = parameters['b' + str(layer)] - alpha * cache['db' + str(layer)]

Start Training

epoch = 30

for i in range(epoch):
  cost_tot = 0
  for j in range(6000):
    forward_pass(X[:,j*10:j*10+10])
    cost_tot += cost(Y[:,j*10:j*10+10],cache['a' + str(L)])
    back_prop(Y[:,j*10:j*10+10])
    update_weights()
  if i%5 == 0:
    print('epoch ',i,' ',cost_tot)

Output

epoch  0   119429.14804683565
epoch  5   102128.37509302271
epoch  10   90151.75500527128
epoch  15   86009.41961305328
epoch  20   88218.24177992699
epoch  25   90432.20939203199
epoch  30   92974.92502732007
epoch  35   93986.34837736617
epoch  40   92380.93127681084
epoch  45   90417.26686598927
epoch  50   101933.2601655828

Above output shows, cost is decrasing still \(45^{th}\) epoch but for \(50^{th}\) epoch, it is increased. Reason of this is Learning Rate. If learning rate is too small, training takes more time and if learning rate is too high then what happens you already experienced. Finding not too high, not too low learning rate is difficult problem for particular application. We will see another article with heading Hyper parameter tunning tricks in near future.

Check Some Samples

forward_pass(X[:,0:10])

# axis 0 means max columnwise
# axis 1 means max rowwise
predicted = cache['a3'].argmax(axis=0)
actual = Y[:,:10].argmax(axis=0)

print('Actual Labels: ', actual)
print('Predicted Labels: ',predicted)

Output

Actual Labels:  [5 0 4 1 9 2 1 3 1 4]
Predicted Labels:  [3 0 4 1 4 6 1 3 1 4]

References and Code

[1] Dataset : handwritten digit recognition

You can find code (MNIST)

You can find code (Random Data)