/Private

System Design

2026-09-02T05:29:25.021Z

Introduction

What is System Design?

System design is the blue print of any software system, it is a the strategic process of defining the architecture, components, modules, interfaces, and data flow for a software system to satisfy specified business and technical requirements.

1. Scalability

Often you hear in a discussion “but that doesn’t scale” as the magical word to end an argument. This is often an indication that developers are running into situations where the architecture of their system limits their ability to grow their service. What is it that we really mean by scalability?

A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added. Increasing performance in general means serving more units of work, but it can also be to handle larger units of work, such as when datasets grow.
When it comes to distributes systems adding resources can have many more reasons like to improve the reliability of the offered service and many more.

→ BUT WHY IS SCALABILITY HARD !? Because…

It requires applications and platforms to be designed with scaling in mind, such that adding resources actually results in improving the performance or that if redundancy is introduced the system performance is not adversely affected.
Many algorithms that perform reasonably well under low load and small datasets can explode in cost if either requests rates increase, the dataset grows or the number of nodes in the distributed system increases.

1.1 Types of Scaling

Horizontal and vertical scaling

Horizontal scaling and Vertical scaling both involve adding resources to your computing infrastructure, you must decide which is right for your application. Scaling horizontally and scaling vertically are similar in that they both involve adding computing resources to your infrastructure. There are distinct differences between the two in terms of implementation and performance.

Horizontal scaling means scaling by adding more machines to your pool of resources (also described as “scaling out”), whereas Vertical scaling refers to scaling by adding more power (e.g. CPU, RAM) to an existing machine (also described as “scaling up”)
One of the fundamental differences between the two is that horizontal scaling requires breaking a sequential piece of logic into smaller pieces so that they can be executed in parallel across multiple machines.
In many respects, vertical scaling is easier because the logic really doesn’t need to change. Rather, you’re just running the same code on higher-spec machines.

However, there are many other factors to consider when determining the appropriate approach. So lets explain one by one.

1.1.1 Vertical Scaling

Vertical Scaling basically makes one node stronger, if you have 1 server, make the server stronger with adding more hardware.
Vertical scaling keeps your existing infrastructure but adding more computing power. Your existing code doesn’t need to change — you simply need to run the same code on machines with better specs. Vertical scaling means adding more resources to a single node and adding additional CPU, RAM, and DISK to cope with an increasing workload.
It’s important to keep in mind that you can only increase it to the limits of your server. By scaling up, you increase the capacity of a single machine.
Vertical scaling allows data to live on a single node, and scaling spreads the load through CPU and RAM resources for your machines. But the system has hardware limitations that you can increate CPU and RAM. Also this will highly expensive when you reach the maximum capacity.
If you have get millions of request, in that case having one server won’t be enough, because even the hardware has maximum capacity limitations.

1.1.2 Horizontal Scaling

Horizontal scaling means adding more machines to the resource pool,
rather than simply adding resources by scaling vertically. Vertical scaling gives you the ability to zoom into add more servers to your network, but it also requires you to zoom out by adding a bit more power, CPU, and RAM to the existing infrastructure.
Scaling horizontally gives you scalability but also reliability because you will have more redundancy and mostly its the preferred way to scale in distributed architectures.
If your services are stateless, then its easy and the best practice to Scaling horizontally.
We can just put the different services and have a load balancer that will split the load traffic to different servers.

2. Caching

Caching, in the world of computer systems, is a technique to store a copy of data or computational results that can be retrieved quickly. The cache memory is like a tiny treasure box that keeps the most frequently accessed data ready for you.

Caching adds speed and reduces latency, and lightens the load on the Backend by providing the prep-rocessed data.

2.1 Caching: Mechanisms, Tools, and Techniques

Browser Caching - Browser caching stores webpage resources on a local computer when a user visits a webpage. On the next visit, the browser loads the page from local resources, reducing server load and improving page load speed.Browser caching is powerful but comes with limitations it’s only effective for repeat visits and is entirely client-dependent.
CDN Caching (Content Delivery Network) - CDN caching replicates your data across various geographical locations, reducing the data travel time (latency) by serving it from the nearest server to the user. Big players like Cloudflare and AWS CloudFront provide CDN services. CDN caching shines when you have a global user base. The more distributed the users, the more beneficial a CDN becomes.
Database Query Caching - Database query caching saves the result set of a query in the cache. When the same query is executed, the DBMS first checks the cache. If found, the results are returned from the cache, skipping the actual database hit.
In-Memory Caching (e.g., Redis, Memcached) - In-memory caches store data in RAM, which is much faster than typical disk storage. They are often used for frequently read data, session information, and full-page caching.
Application Caching - Custom build application cache.
Distributed Caching - Application cache, but in microservice architecture.

2.2 Cache Strategies: Types and When to Use What

A well implemented cache can improve the performance of a system but the only deal to make it work efficiently is to choose the correct caching strategy.

Cache-Aside(Lazy Loading) - In the Cache-Aside strategy, the application looks for data in the cache first. If it doesn’t find the data (a cache miss), it loads data from the source, stores it in the cache, and then returns it. This strategy ensures that only requested data is cached, thus preventing the caching of unnecessary data. This strategy is best when reads are much more frequent than writes, and data doesn’t get updated often.
Read-Through (Eager Loading) - In the Read-Through strategy, cache is completely abstracted from the application. When a cache miss occurs, the cache itself is responsible for reading data from the data source, storing it, and then returning it. This approach maintains data consistency and saves the application from interacting with the data source. This strategy works best when the cost of a cache miss is high and when you want to keep your application logic simple.
Write-Through - In the Write-Through strategy, every write operation to the database is simultaneously written to the cache. This ensures the cache always holds the latest data, reducing the chance of serving stale data. This strategy is suitable when write operations are less frequent and you need up-to-date data in cache at all times.
Write-Back (Write-Behind) - In the Write-Back strategy, write operations are first written to the cache, and the cache then writes the data back to the database after a certain delay. This strategy significantly reduces database load. This strategy is ideal when write performance is more critical than data consistency, and the risk of data loss is acceptable.
Refresh-Ahead - In the Refresh-Ahead strategy, cache proactively refreshes data before it gets stale or expires. This strategy ensures that data is always ready in the cache, reducing cache miss latency. This strategy is great when you have predictable data access patterns and can afford the overhead of pre-fetching data.

Choosing the Right Caching Strategy

Now that we know the options, how do we choose the right caching strategy? Several factors come into play:

Scale of the Application: Larger applications with heavy traffic need more robust caching solutions like CDN or distributed caching.
Type of Data: Highly dynamic data might not be the best candidate for caching. On the other hand, data that is expensive to compute or doesn’t change often should be prioritized.
User Distribution: If your users are globally distributed, consider CDN caching. If most of your user base is local, in-memory caching may be more effective.
Infrastructure: Cloud-based applications can take advantage of the built-in caching mechanisms provided by cloud vendors.

3. Load Balancing

In the world of computers and the internet, when a website or online service becomes super popular, it starts receiving a flood of requests from users all over the place. Instead of relying on a single server to handle all these requests, load balancing spreads the workload across multiple servers.

No single server gets overwhelmed with too much traffic.
Every server gets to share the load.
Prevents crashes, slowdowns, and annoying error messages, making sure that users have a smooth experience while browsing websites or using online applications.
Load balancing algorithms decide which server should handle each request, making sure they’re divvied up fairly.

3.1 Types of Load Balancers based on the OSI Model

Network Load Balancer (Load Balancer 4):
- Operates at the Transport Layer (layer 4) of the OSI model.
- It makes decisions based on information such as the source and destination IP addresses, as well as the port numbers of incoming requests.
- The main focus of Layer 4 load balancers is to distribute network traffic efficiently across multiple servers.
- When a user sends a request to access a website or application, the Layer 4 Load Balancer receives the request. It then looks at the transport layer data (IP addresses and ports) to determine which server should handle the request. The load balancer uses various algorithms (e.g., round-robin, least connections) to decide the best server to forward the request to. This ensures that traffic is evenly spread among the servers, improving performance and avoiding overloading any single server.
- Cannot make application specific desicions
Application Load Balancer (Load Balancer 7):
- Operates at the application layer.
- It can make more intelligent decisions based on application-specific data, such as HTTP headers, cookies, and URLs.
- Load Balancers at this level understand application protocols, enabling them to optimize traffic distribution for specific applications or services.
- When a user sends a request, the Layer 7 Load Balancer analyzes the content of the request to gain insights into the application being accessed. For example, it can identify the type of service (e.g., HTTP, HTTPS, FTP) or the specific URL being requested. Using this information, the load balancer can make intelligent decisions about which server is best suited to handle the request. This allows for more advanced load balancing strategies, such as sending certain requests to specialized servers that can handle specific application tasks.
- Higher processing overhead compared to Layer 4 Load Balancer due to content analysis.

3.2 Load Balancing Algorithms

3.2.1 Round Robin:

Transport Layer algorithm.
It maintains a pool of servers for handling incoming requests.
When the request comes the load balancer starts with the first server and sends it the request and then continues sending the requests in the linear or the circular manner.
In a long run each server receives the equal number of requests.
Simple, no sessions required, no external dependency.
No health monitoring, unequal server capability, no geo-location awareness.

3.2.2 Weighted Round Robin:

Expansion of basic round robin algorithm.
It fixes the issue of unequal server capacities by assigning different weights to each server in the pool, the weights represent the relative compute and performance capability of the servers.
When a new request comes in, the load balancer uses the weights to determine which server should handle the request. It cycles through the servers in a circular manner, as in the basic Round Robin algorithm, but with a slight modification. Instead of equal distribution, it takes into account the weights of the servers.
Fine tuned balance, capacity base distribution of requests.
Complex, static config.

3.2.3 IP Hash

Application layer algorithm.
Distribute incoming network traffic across multiple servers based on the source IP address of the client.
From the incoming request the load balancer extracts the ip address of the request.
The IP address is then hashed, meaning it is converted into a unique numerical value using a hash function. The hash function ensures that the same IP address will always produce the same hash value.
The load balancer uses the hashed value to map the client’s IP address to a specific server in the pool. This mapping is typically done by dividing the hash value by the total number of servers and selecting the corresponding server.

3.2.4 Least Connection Algorithm

Based on number of active connection of a server.
The load balancer continuously monitors the number of active connections of each server in the server pool.
When a new request comes in, the load balancer evaluates the current connection count for each server and selects the server with the fewest active connections.

3.2.5 Least Response Time Algorithm

Based on the response time of the server on the previous requests.
The load balancer continuously measures the response times of each server in the pool to previous requests.
When a new request comes in, the load balancer evaluates the response times of each server and selects the server with the shortest response time.

Time to First Byte (TTFB) is the time taken by a server to send the first byte of data in response to a client’s request. It includes the time spent on processing the request, database queries, and any other backend operations before the server starts sending the response data. The Least Response Time load balancing algorithm implicitly takes into account the Time to First Byte as it evaluates the overall response time of each server. The response time includes the TTFB along with the time taken to transmit the complete response data.

3. Database Replication and Partitioning

3.1 Database Replication

Before getting started you must know some keywords for better understanding.

Latency - It is the total time a system takes to process a request and return a response.
Availability - It is a measure of percentage of time a system is operational and accessible to users.
Fault tolerance - It is a system’s ability to continue operating properly without interruption when a hardware, software, or network component fails.
Data Throughput - It is the total volume of data or number of operations a system can process, transfer, or handle successfully within a specific time unit.
Data Consistency - Ensures that all nodes in a distributed system see the same data at the same time, preventing conflicts during concurrent access.
Durability - Ensures that once data is committed, it remains intact, permanent, and recoverable, even during system crashes, power failures, or hardware outages.

What is Database Replication? Database replication in simple language means to store data at one or more places or simply copying the data from one database or from one server to another.

BUT WHY DO WE NEED IT???? Let us take an example situation to understand the need in a better way. I am running an E-comm company and my company is a huge success in India but now i am targeting multi national costumers, so if a person from Australia comes to visit my website he or she will be able to see the products after a large amount of time which is a problem as my servers are hosted in the Indian region. Increasing the resources on my server is not the solution of this problem. The issue of because of network latency.So by replication, we can keep data close to the user’s geographic area to reduce latency. Similar things are done by Netflix and Amazon prime video etc that’s why you people can watch HD movies without any buffering delays.

Replication also solves the problems like Availability & Fault toleration issues and Data throughput issues. So as a CEO of an e-commerce startup, if my application goes down (Unavailable) then my whole business could get impacted and I could have lost thousands of bucks and can lose the trust of our customers which is very bad these days. Hence we need better Fault tolerance in our system.

3.2 Algorithms to implement Database replication

Single Leader Replication(Active-Passive or Master-Slave Replication)
Multi-Leader Replication
Leaderless Replication

System Design MOC

2026-09-02T05:29:25.021Z

🗺️ System Design Map of Content

Welcome to the syllabus and index for System Design.

📄 Notes

System Design

Md Notes MOC

2026-09-02T05:29:25.021Z

🗺️ Md Notes Map of Content

Welcome to the syllabus and index for Md Notes.

📁 Sub-Topics

📄 Notes

Index
README

key

2026-09-02T05:29:25.021Z

bases renderer of Private/md-notes/key

sql basics

2026-09-02T05:29:25.021Z

No description provided

Index for Devops

2026-09-02T05:29:25.020Z

OWASP, GitOps,Infrastructure Monitoring,Kubernetes,Docker,Networking for Devops,Nginx,OWASP,AWS,CI CD,Ansible,,Terraform,Consul,git-cheat-sheet.pdf,Service Mesh

Kubernetes

2026-09-02T05:29:25.020Z

*prerequisite - Docker

Kubernetes (often abbreviated as K8s) is an open-source platform that automates deploying, scaling, and managing containerized applications.

→ Kubernetes is a Cloud Native Computing Foundation (CNCF) graduates project which is responsible for orchestrating containers.

Note

Monolithic architecture is a single, unified unit where all components are tightly coupled, making it easier to develop initially but harder to scale. Microservices break applications into independent, loosely coupled services, allowing for independent deployment and better scalability, but with higher complexity

1. Kubernetes Architecture

Master Node = Headquater Worker Node = Branch(runs docker containers)

etcd - Consistent and highly-available key value store for all API server data.
Controller Manager - Runs controllers to implement Kubernetes API behavior. [Project Manager] Supervises that everything is working fine of not. Works at cluster level
Api server - The core component server that exposes the Kubernetes HTTP API. (communication gateway between the components within the cluster) [Manager]
kube-Scheduler - Looks for Pods not yet bound to a node, and assigns each Pod to a suitable node. [HR]
Kubectl - Cli tool for kubernetes. [Director]
Kubelet - Ensures that Pods are running, including their containers.(works at worker level) [Branch Manager].
Kube-proxy/Service-proxy - Maintains network rules on nodes to implement Services. (helps user to access the pods).
Container runtime - Software responsible for running containers. (Docker).
Container Network Interface (CNI Network): Provides network connectivity, ensuring that pods can communicate with each other across different nodes and that network policies are enforced. Eg- Wavenet, Calico, etc.

2. Kubernetes works in cluster

Types of Kubernetes clusters

Kubeadm - Multiple systems are connnected in a cluster by installing kubeadm on them.
Minikube (local or single ec2)
Kind cluster - kubernetes in docker
EKS/AKS/GKE

→ Kind cluster installation for linux

# For AMD64 / x86_64
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-amd64
# For ARM64
[ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-arm64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

NOTE

Before starting to write kubernetes configs, you must know how to write yml/yaml.

3. How to Write Yet Another Markup Language(yml/yaml)

# key value
key: value
name: xyz
-----------
#list
courses:
  - list1
  - list2
  - list3
-----------
#objects
info:
  name: xyz
  age: 46
  course: CSE

Heading towards practical part…

Creating a kind cluster

We will be using yml/yaml file for creating a cluster.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.31.2
- role: worker
  image: kindest/node:v1.31.2
- role: worker
  image: kindest/node:v1.31.2
- role: worker
  image: kindest/node:v1.31.2
- role: worker
  image: kindest/node:v1.31.2
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP

- Kubernetes Namespaces

|Docker container→Pod→Deployment→service |

             All this inside a namespace for isolation

A Kubernetes namespace is a mechanism for dividing and logically isolating a single physical Kubernetes cluster into multiple virtual sub-clusters.

mradul@mradul-HP-Laptop-15:~/Desktop/GIT/Fast-Translit-Deployment/Nginx$ kubectl get ns
NAME                 STATUS   AGE
default              Active   20m
kube-node-lease      Active   20m
kube-public          Active   20m
kube-system          Active   20m
local-path-storage   Active   20m
nginx                Active   4m49s
mradul@mradul-HP-Laptop-15:~/Desktop/GIT/Fast-Translit-Deployment/Nginx$

default - When no namespace is created the resources are assigned the default namespace.
kube-node-lease - Stores the information of kubernetes clusters.
kube-public - Runs the services that are publicly accessible resources.
kube-system - Runs system level services.
local-path-storage - For storage use.

- Kubernetes Pods

Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. A Pod is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers.

A pod may container one or more application containers which are tightly coupled and run together.
The shared context of a Pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation - the same things that isolate a container.
Pods can simply be understood by the set of containers running on same machine but still completely isolated.
Pods can even run a single container and when needed pods can also run multiple containers which are needed to run together.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

                          pods.yml

To apply the yml manifest files we use kubectl apply -f {filename} Usually we do not create pods directly using the yml file, we instead create them using the workload resources like Deployments, job, ReplicaSet, DaemonSet and StatefulSet(when we need to track the state of the pod running).
Each pod runs the single instance of the application, you must always run multiple pods for any appliation.
The replication of pods is the responsibility of the workloads and the Controllers.
Types of Multi-container pods
- Sidecar Containers: Enhance or extend the main application container, such as a separate log exporter or metrics collector.
- Init Containers: Specialized containers that execute sequentially and run to completion before any app containers start. They typically handle setup scripts, database migrations, or security credentials.
- Ephemeral Containers: Temporary troubleshooting containers that you run interactively inside an existing Pod to diagnose bugs. They cannot be added at Pod creation and lack resource guarantees.

- Kubernetes workloads

Deployment: Makes the pod scalable.

Networking for Devops

2026-09-02T05:29:25.020Z

1. How DNS actually works?

Computer and devices communicate using the IP addresses which are not easy to be remembered by a human, so they use words. Domain Name system brings these two together and gives us our destination site. Lets see how the DNS resolution takes place

We hit a request on our browser for a particular website for example xyz.com .
The browser checks its cache to find the IP address for the website xyz.com .
If the browser does not find the IP of the requested domain name, it requests the Operating System to check its cache for the IP address.
If the Operating system is also not able to find the IP address of the requested website then the, it calls the resolver.
The resolver server is usually your ISP (Internet Service Provider). All resolvers must know one thing: where to locate the root server.
The resolver checks its cache to find the address if it is already there.
If the resolver fails to find it, then the request goes to the root server.
The root server knows where to locate the .COM TLD server. TLD stands for Top-Level Domain.
The root server gives the address of the Top Level Domain server for the requested website.
There are 13 root name servers that exist today,root servers sit at the top of the DNS hierarchy. They are scattered around the globe and operated by 12 independent organisations. They are named [letter].root-servers.net where [letter] ranges from A to M.
This doesn’t mean that we have only 13 physical servers to support the whole internet, each organisation provides multiple physical servers distributed around the globe.
The Resolver now goes to the .com Top Level Domain server to find the address of the requested site.
The Top Level Domain server also does not know the IP address of the xyz.com but it will tell the name servers for that domain name, like ns1.xyz.com ns2.xyz.com ns3.xyz.com ns4.xyz.com
A nameserver is a specialized server in the Domain Name System (DNS) that acts as the internet’s phonebook. It is responsible for storing DNS records (such as IP addresses) for a domain name and answering requests to translate those human-readable names into machine-readable numerical IP addresses. Here a question arises that how did the TDL of .com domain pointed the resolver to the authoritative name servers and how could it make the connection? There are so many .COM domains!? → The answer to all these questions is Domain Registrar, when a domain is purchased the domain registrar reserves the name and communicates to the TDL registry the authoritative name servers.
Now the resolver reached the name server ns1.xyz.com there are more than one server running for a domain as they have to globally manage the load on the website. → If you want to know who are the authoritative name servers for your domain, run a WHOIS query. There are a few websites that provide this information.
The resolver get the IP of the requested domain and saves the IP address and gives it to OS.
The Operating System gives this address to the browser and then the user sees the website.

→ A STRANGE QUESTION ARISES How could the resolver find ‘ns1.xyz.com’ before ‘xyz.com’? Since ‘ns1.xyz.com’ is a subdomain of ‘xyz.com’, how could we resolve ‘ns1.xyz.com’ without resolving ‘xyz.com’ first? Isn’t the search going backwards? HOW CAN WE GET A SUB-DOMAIN WITHOUT GETTING A DOMAIN!?!?!?

The answer is GLUE RECORDS !
When the resolver asked the .COM TLD about xyz.com, extra information was attached to that response.
The resolver got at least one IP address for each name server, these are called glue records.
So the resolver not only got the name of the authoritative name server, it also got the IP address which breaks the circular dependency.

1.1 DNS Records

A Domain Name System (DNS) record is a set of instructions used to connect domain names with internet protocol (IP) addresses within DNS servers. DNS records exist as text-based files known as “zone files” written in DNS syntax. They serve as a record and set of commands on how to handle DNS queries. When a user searches a domain name or URL or takes action related to a domain name in a web browser, this is the beginning of a DNS query. A series of DNS servers then communicate with each other to resolve this query. DNS servers rely on DNS records to connect that user with the corresponding IP address and resolve all other issues. DNS records are stored on authoritative DNS servers also known as authoritative nameservers. They contain information on how frequently a server will refresh the DNS record, known as time-to-live (TTL).

Common types of DNS records:

A records: A records stands for Address records, they are the most commonly used DNS record. They directly connect the IPv4 addresses to the domain name.
AAAA records: They connect the IPv6 addresses to the domain name.
CNAME records: Canonical Name records, direct an alias domain to a canonical domain. This means that this type of record is used to link subdomains to domain A or AAAA records.
DNAME records: Delegation name records, or DNAME records, are used to redirect multiple subdomains with one record and point them to another domain. For example, a DNAME record linking domain.com to example.com will link product.domain.com, trial.domain.com, and blog.domain.com to example.com. These records are helpful in managing largescale domains and in managing domain name changes by ensuring subdomains are properly linked.
CAA records: Certification authority authorization records, or CAA records, allow domain owners to specify which certificate authorities (CAs) can issue certificates for their domain. A CA is an organization that validates the identity of websites and connects them to cryptographic keys by issuing digital certificates
CERT records: Certificate, or CERT records, store certificates that verify the authenticity of all parties involved. This type of record is particularly valuable when securing and encrypting sensitive information.
MX records: Mail exchange, or MX records, direct emails to your domain mail server. These records, along with an email server, allow for the creation of individual email accounts, such as user@example.com, that are linked to the domain (example.com)
NS records: Nameserver, or NS records, show which DNS server is acting as the authoritative nameserver for your domain. Authoritative nameservers contain the final information about a specific domain and its corresponding IP address. An NS record points to all of the different records your domain holds. Without NS records, users will not be able to access your website
SOA records: Start of authority, or SOA records, store important administrative information about a domain. This information can include the domain administrator’s email address, information on domain updates and when a server should refresh its information.
PTR records: Pointer records, or PTR records, work in the opposite direction of A records. They are used to connect an IP address with a domain name, instead of a domain name with an IP address. When a DNS lookup begins with an IP address, it then finds the corresponding hostname. These records are used to detect spam by checking if the IP addresses and associated email addresses are used by legitimate email servers. PTR records must be set up by the server host.
SPF records: Sender policy framework, or SPF records, are used to identify the mail servers that can send emails through your domain. This helps prevent your domain from being used by spammers or for malicious purposes by letting email receivers know that what they are receiving has been authorized.
SRV records: Service, or SRV records, identify a host and port for specific services, such as messaging, for a domain. Ports are virtual connection points that allow digital devices to separate different types of traffic.
ALIAS records: ALIAS records are used to direct your domain name to a host name and not an IP address. For instance, if your domain name is example.com, you can point it to product.differentexample.com using an ALIAS record.
NSEC records: Next secure records, or NSEC records, allow for proof of non-existence. This means that these records exist to confirm that other records do not exist. Being able to confirm the non-existance of a record saves time when searching for specific records.
URLFWD records: URL forwarding (or URL redirecting) is a technique used to make a single web page available via multiple URLs. (NS1 Connect) users can easily set up URL forwarding (HTTP redirects or masking) between zones. There are three types of URL redirects: Permanent (301), temporary (302), or masking.
TXT records: Text, or TXT records, store textual information related to domains and subdomains. Text records allow for the storage of SPF records and email verification records.

2. How HTTP works?

Nginx

2026-09-02T05:29:25.020Z

How to write Nginx config file

Nginx has one master process and several worker processes. The main purpose of the master process is to read and evaluate configuration, and maintain worker processes. Worker processes do actual processing of requests.
The number of worker processes is defined in the configuration file and may be fixed for a given configuration or automatically adjusted to the number of available CPU cores.
The nginx.conf file can be found in /etc/nginx or usr/local/nginx/conf or /usr/local/etc/nginx.

Config file structure

Nginx consists of modules which are controlled by directives specified in the configuration file.

NOTE

Modules provide the actual features and backend functionality, while directives are the configuration instructions you write in your configuration file to control how those modules behave.

Directives are divided into simple directives and block directives. A simple directive consists of the name and parameters separated by spaces and ends with a semicolon.
A block directive has the same structure as a simple directive, but instead of the semicolon it ends with a set of additional instructions surrounded by curly braces.
If a block directive can have other directives inside braces, it is called a context.
Directives placed in the configuration file outside of any contexts are considered to be in the main context.

Serving static content

An important web server task is serving out files.
Depending on the request, files will be served from different local directories.

http {
    server {
    }
}

                      Default config

location /www/ {
  root /data
}

This location block specifies the “/www” prefix compared with the URI from the request. For matching requests, the URI will be added to the path specified in the root directive, that is, to /data, to form the path to the requested file on the local file system. If there are several matching location blocks nginx selects the one with the longest prefix. The location block above provides the shortest prefix, of length one, and so only if all other location blocks fail to provide a match, this block will be used.

http{
    server{
        location /html/ { ;
        root /usr/share/nginx/;
        index index.html;
    }
}

This is already a working configuration of a server that listens on the standard port 80 and is accessible on the local machine at http://localhost/
Do not forget to do sudo systemctl daemon-reload after changing the config.

Nginx Reverse Proxy for an Application Running on localhost:8000

NOTE

→ Proxy - Machine or a set of machines which sit between two systems(user and a backend system or two backend systems, anything). Proxies are used to abstract the complexities or un-trusted environment. → Forward Proxy - Abstracts out the client by acting as a middleware, protects the identity of the client. Whenever the client makes a call the call goes via this proxy to the internet or to some service or machine. The service will only know the IP address of the proxy not of the client. It can be used to restrict the access of websites, at forward proxy caching can be performed as everything is going through it. → Reverse Proxy - The reverse proxy abstracts outs the complexities of the down stream systems. Example “load balancer”, Router (an api gaateway), caching, Abstraction of the infra → Firewall - A firewall is a network security device that separates a trusted internal network from an external network deemed untrustworthy, such as the internet. It regulates incoming and outgoing network traffic based on preset security rules. Firewalls are paramount in shielding networks from unauthorized access, harmful activities, and potential threats, and can exist as hardware, software, software-as-a-service (SaaS), or public or private (virtual) cloud.

Objective

Expose an application running on localhost:8000 through Nginx, allowing clients to access it via port 80.

Architecture

Client
   │
   ▼
Nginx (Port 80)
   │
   ▼
Application (localhost:8000)

Nginx acts as a reverse proxy:

Receives requests from clients.
Forwards requests to the backend application.
Returns the backend’s response to the client.

Nginx Configuration

events {}
 
http {
    server {
        listen 80;
        server_name _;
 
        location / {
            proxy_pass http://127.0.0.1:8000;
 
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

Explanation

1. Listen on Port 80

listen 80;

Tells Nginx to accept incoming HTTP requests on port 80.

2. Forward Requests to Backend

proxy_pass http://127.0.0.1:8000;

Forwards all requests received by Nginx to the application running on port 8000.

Example:

http://server-ip/
        │
        ▼
http://127.0.0.1:8000/

3. Preserve Original Request Information

Host Header

proxy_set_header Host $host;

Passes the original hostname requested by the client.

Example:

Host: example.com

Real Client IP

proxy_set_header X-Real-IP $remote_addr;

Provides the backend with the actual client IP address.

Forwarded Client Chain

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Maintains a list of proxy hops for logging and auditing.

Original Protocol

proxy_set_header X-Forwarded-Proto $scheme;

Indicates whether the client used:

http

https

Validate Configuration

Before reloading Nginx, verify the configuration syntax:

sudo nginx -t

Expected output:

nginx: configuration file syntax is ok
nginx: configuration file test is successful

Reload Nginx

Apply configuration changes:

sudo systemctl restart nginx

sudo nginx -s reload

Verify Backend Service

Check whether the application is listening on port 8000:

ss -tulnp | grep 8000

Example output:

LISTEN 0 128 127.0.0.1:8000

Test the Reverse Proxy

Open:

http://localhost

http://<server-ip>

If configured correctly:

Client → Nginx → Application → Nginx → Client

Troubleshooting

Check Nginx Error Logs

sudo tail -f /var/log/nginx/error.log

Useful for:

Backend connection failures
Invalid configurations
Permission issues

Check Access Logs

sudo tail -f /var/log/nginx/access.log

Useful for:

Verifying incoming requests
Monitoring traffic
Debugging routing issues

Key Concept

A reverse proxy hides the backend service from clients.

Clients interact with:

Nginx (Port 80)

while Nginx internally communicates with:

localhost:8000

This provides:

Centralized request handling
SSL/TLS termination
Load balancing capabilities
Security and access control
Logging and monitoring
Backend abstraction

The core directive that makes Nginx a reverse proxy is:

proxy_pass http://127.0.0.1:8000;

Everything else supports correct request forwarding and backend awareness.

Service Mesh

2026-09-02T05:29:25.020Z

What exactly a service mesh is?

Imagine you are an engineer at a fast-growing e-commerce company like Amazon. Your team started with a monolithic application—a single, large codebase handling user requests, inventory, and payments. As the company grew, you migrated to a microservices architecture with separate services for:

User Service (handles user accounts)
Order Service (processes orders)
Inventory Service (tracks stock)
Payment Service (handles payments)

At first, everything ran smoothly. However, as complexity increased, several problems emerged:

✅ Problem #0: Authentication & Authorization

Each microservice must handle logins & permissions separately, resulting in duplicated and inconsistent security logic.
🔴 Problem #1: Hard-to-Debug Failures

PopCustomers report orders failing randomly. Logs indicate errors in the Payment Service, but it’s unclear whether these issues are from network glitches, load spikes, or faulty service updates.
🔴 Problem #2: Security Risks

Sensitive data, like payment details, is transmitted between services unencrypted. Without proper encryption, interception becomes a major risk.
🔴 Problem #3: Load Balancing Issues

The Order Service is overwhelmed while the Inventory Service remains underutilized. Efficient traffic distribution is missing.
🔴 Problem #4: Slow Rollouts & Deployments

Introducing a new version of the Payment Service poses a risk. Testing it on only 10% of users is ideal, but gradual rollouts without downtime are challenging.

This growing complexity makes managing your microservices painful. This is where a Service Mesh comes in! 🚀

1. Why Do You Need a Service Mesh?

Before service meshes, organizations manually managed service-to-service communication using:

Custom code in each microservice
Reverse proxies (e.g., NGINX, HAProxy)
Load balancers
API gateways

While this approach worked, it had serious limitations:

No standardized way to handle retries, timeouts, and failures.
Security risks due to unencrypted communication.
Limited observability making it hard to trace requests across multiple services.

Real-World Example: Netflix

Netflix’s microservices faced issues like:

Slow response times because of ineffective traffic control.
DDoS threats due to the absence of a central security layer.
Authentication issues from each microservice handling its own login logic.

image.png

2. How Service Mesh Helps

image.png|102

A Service Mesh is an infrastructure layer that manages service-to-service communication within a distributed microservices architecture. It abstracts away networking concerns, thereby enhancing:

Security: Through secure, encrypted communication (mTLS).
Traffic Management: Via intelligent routing, load balancing, and retries.
Observability: With integrated logging, tracing, and monitoring.
Resilience: By using circuit breakers and fault injection.

Key benefits include decoupling networking logic from application code and centralizing security policies.

3. Service Mesh Architecture

image.png

A service mesh consists of two primary components:

3.1 Data Plane (Sidecar Proxies)

Each microservice runs alongside a sidecar proxy that intercepts all incoming and outgoing traffic.

Example: Envoy Proxy

Responsibilities:
- Service Discovery & Load Balancing: Efficiently routes traffic among service instances.
- Retries & Circuit Breaking: Enhances resilience by managing failures.
- Security Enforcement: Uses mTLS for zero-trust security.
- Telemetry Collection: Gathers metrics, logs, and traces.

3.2 Control Plane

Responsibilities:
- Manages and configures the sidecar proxies.
- Applies traffic rules, security policies, and observability settings.

These components work together to enforce consistent communication policies across your microservices.

How It Works: Without vs. With Service Mesh

Without Service Mesh:

Service A → Service B(Custom retry logic, security, and logging embedded in application code)
Service B → Database(Direct connection without security enforcement)

With Service Mesh:

Service A → Envoy Proxy (handles retries, security, monitoring) → Service B
Service B → Envoy Proxy → Database(All communications are secured and monitored)

4. Popular Service Mesh Implementations

image.png

Service Mesh	Proxy Used	Best For
Istio	Envoy	Kubernetes-native applications
Linkerd	Linkerd-proxy	Simplicity and lightweight setups
Consul	Envoy	Multi-cloud and hybrid environments
AWS App Mesh	Envoy	AWS-native applications

For more details, see the Comparison of Service Meshes.

5. Benefits of Service Mesh

5.1 Security: Encrypted & Secure Communication (mTLS)

Problem: Unencrypted traffic is vulnerable to interception.
Example: A banking app transmitting user passwords in plain text.
Solution: Enforce mTLS to ensure all communication is secure and authenticated.

kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT
EOF

5.2 Traffic Control

💡 Benefit: Manage how traffic flows between services, allowing canary deployments, fault injection, and load balancing.

🔀 A/B Testing with Traffic Splitting

📌 Route 80% of traffic to v1 and 20% to v2 of the reviews service

 
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
  namespace: default
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 80
    - destination:
        host: reviews
        subset: v2
      weight: 20
EOF

🔹 Now, when you access productpage, most users will see v1, while 20% get v2.

Example: Netflix gradually releases a new video recommendation algorithm.\

5.3 Observability: Debugging Made Easy

Problem: Locating the root cause of failures across multiple services is challenging.
Example: Uber’s thousands of microservices made debugging nearly impossible.
Solution: Integrate with Jaeger and Kiali for distributed tracing and visualization.

 
istioctl dashboard kiali  # Open the Kiali dashboard for service mesh visualization

5.4 Load Balancing: Efficient Traffic Distribution

Problem: Imbalanced traffic can overwhelm some services while underutilizing others.
Example: Amazon’s checkout service receives millions of requests per minute.
Solution: Dynamically distribute requests using advanced load balancing algorithms.

 
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: checkout-service
spec:
  host: checkout-service
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN  # Directs traffic to the least busy instance

Additional Capabilities

Traffic Management: Intelligent routing, load balancing, retries, and failover.
Security Enhancements: mTLS, role-based access control (RBAC), and centralized authentication/authorization.
Observability & Monitoring: Distributed tracing (Jaeger, Zipkin), logging/metrics collection (Prometheus, Grafana, Kiali), and dynamic service discovery.
Resilience & Fault Tolerance: Circuit breaking, rate limiting, and fault injection for chaos engineering.

7. When to Use a Service Mesh?

Scenario	Use Service Mesh?
Small app with fewer than 5 services	❌ No (Overhead is too high)
10+ microservices	✅ Yes (Provides standardized communication)
Multi-cloud or hybrid deployments	✅ Yes (Simplifies cross-cloud networking)
High-security environments (banking, healthcare)	✅ Yes (Enforces mTLS and centralized authentication)

8. Real-World Benefits & Use Cases

✅ Case Study #1: Airbnb – Scaling Microservices

Problem:

Airbnb had thousands of microservices, making debugging nearly impossible due to scattered logs.
Solution:

They implemented Istio (Service Mesh) along with Jaeger for distributed tracing.
Outcome:

Engineers could trace failures in seconds, reducing incident response time by 70%.

✅ Case Study #2: Stripe – Secure Payment Transactions

Problem:

Stripe processes millions of financial transactions and required end-to-end encryption between microservices.
Solution:

They deployed a Service Mesh with mTLS to encrypt all service-to-service communication automatically.
Outcome:
- Zero unencrypted transactions
- Full compliance with banking security standards

✅ Case Study #3: Netflix – Canary Deployments

Problem:

Netflix needed to test new versions of its recommendation engine on a small subset of users to avoid downtime.
Solution:

They used Istio’s traffic splitting feature to direct 10% of traffic to the new version while maintaining 90% on the stable version.
Outcome:
- Safer rollouts
- No downtime during updates

Conclusion

A Service Mesh—using tools like Istio, Linkerd, or Consul—streamlines inter-service communication in a microservices architecture by abstracting networking, security, and observability away from application code. This approach lets developers focus on core business logic while ensuring robust, secure, and resilient interactions between services.

Feel free to reach out if you need further clarification or want to explore a live demo of these concepts in your environment!

Terraform

2026-09-02T05:29:25.020Z

prerequisite - AWS

Devops MOC

2026-09-02T05:29:25.020Z

🗺️ Devops Map of Content

Welcome to the syllabus and index for Devops.

📄 Notes

Index

2026-09-02T05:29:25.020Z

Make a note of something, Docker, System Design, Kubernetes Terraform,AWS, DBMS, Service Mesh, RabbitMQ, Testing, git-cheat-sheet.pdf, Search Engines, Postgres, GitOps, Consul, Networking for Devops, Building for Scale, OWASP, Ansible, Auth, APIs and its types, Web Security, CI CD, Advance DB concepts, FastAPI, Redis, Nginx, Infrastructure Monitoring, Python, Django, README, key.base , DSA,System Design or try the Importer!

README

2026-09-02T05:29:25.020Z

🎯 Purpose

This repository serves as:

Interview Preparation: Curated notes and explanations for common interview topics
Knowledge Base: Quick reference materials for various technologies
Learning Journal: Organized study materials as you prepare for technical interviews

📚 Directories

DevOps/

Notes on Backend & DevOps tools, methodologies, and practices including:

CI/CD pipelines
Container orchestration
Infrastructure as Code
Monitoring and logging
Cloud platforms and tools

🚀 Getting Started

Browse through the directories to find relevant topics
Each directory contains markdown files with detailed explanations
Use internal links (like [[topic]]) to navigate between related concepts
Add more directories as you expand your preparation materials

📝 Adding Content

As you add more directories and content:

Keep file names descriptive and lowercase with hyphens (e.g., docker-basics.md)
Use clear markdown formatting with headers and code blocks
Link related concepts within the vault using [[link]] syntax
Update this README to reflect new sections

🔧 Planned Additions

💡 Tips for Effective Learning

Review notes regularly before interviews
Practice explaining concepts out loud
Create examples and diagrams for complex topics
Link related concepts together for better understanding
Update notes as you learn new information

📖 How to Use

For Quick Reference: Search within markdown files for specific topics
For Deep Learning: Follow the linked concepts through the vault
For Interview Prep: Review notes in each directory before interviews

Good luck with your interview preparation! 🎓

Python

2026-09-02T05:29:25.019Z

Introduction

print("Hello world")

print(2 + 3)   # addition(+)
print(3 - 1)   # subtraction(-)
print(2 * 3)   # multiplication(*)
print(3 + 2)   # addition(+)
print(3 - 2)   # subtraction(-)
print(3 * 2)   # multiplication(*)
print(3 / 2)   # division(/)
print(3 ** 2)  # exponential(**)
print(3 % 2)   # modulus(%)
print(3 // 2)  # Floor division operator(//)

print(type(10))                  # Int
print(type(3.14))                # Float
print(type(1 + 3j))              # Complex
print(type('Asabeneh'))          # String
print(type([1, 2, 3]))           # List
print(type({'name': 'Asabeneh'}))  # Dictionary
print(type({9.8, 3.14, 2.7}))    # Set
print(type((1,2,3)))             #tuple

NOTE

Python build-in functions will be done after the functions.

Variables

Variables store data in a computer memory. Mnemonic variables are recommended to use in many programming languages. A mnemonic variable is a variable name that can be easily remembered and associated. A variable refers to a memory address in which data is stored. Number at the beginning, special character, hyphen are not allowed when naming a variable. A variable can have a short name (like x, y, z), but a more descriptive name (firstname, lastname, age, country) is highly recommended.

Python Variable Name Rules

A variable name must start with a letter or the underscore character
A variable name cannot start with a number
A variable name can only contain alpha-numeric characters and underscores (A-z, 0-9, and _ )
Variable names are case-sensitive (firstname, Firstname, FirstName and FIRSTNAME) are different variables) → If you want to use reserved keywords as the variable then use a underscore before them. Eg- “_if”

Example List of Valid variables

firstname
lastname
age
country
city
first_name
last_name
capital_city
_if # if we want to use reserved word as a variable
year_2021
year2021
current_year_2021
birth_year
num1
num2

Example list of invalid variables

first-name
first@name
first$name
num-1
1num

Declaring Multiple Variable in a Line

Multiple variables can also be declared in one line:

first_name, last_name, country, age, is_married = 'Asabeneh', 'Yetayeh', 'Helsink', 250, True

print(first_name, last_name, country, age, is_married)
print('First name:', first_name)
print('Last name: ', last_name)
print('Country: ', country)
print('Age: ', age)
print('Married: ', is_married)

Data Types

There are several data types in Python. To identify the data type we use the type built-in function.

# Different python data types
# Let's declare variables with various data types

first_name = 'Asabeneh'     # str
last_name = 'Yetayeh'       # str
country = 'Finland'         # str
city= 'Helsinki'            # str
age = 250                   # int, it is not my real age, don't worry about it

# Printing out types
print(type('Asabeneh'))          # str
print(type(first_name))          # str
print(type(10))                  # int
print(type(3.14))                # float
print(type(1 + 1j))              # complex
print(type(True))                # bool
print(type([1, 2, 3, 4]))        # list
print(type({'name':'Asabeneh'})) # dict
print(type((1,2)))               # tuple
print(type(zip([1,2],[3,4])))    # zip
print(type({1,2,3}))             # set

zip - In Python, there is no native “zip data type.” Instead, zip is a built-in constructor function that returns a specialized iterator called a zip object. This zip object maps corresponding elements from multiple iterables (like lists, tuples, or strings) into an iterator of grouped tuples.

>>> names = ["sid","panth","ani"]
>>> age = ["22","17","21"]
>>> zipped_data = zip(names,age)
>>> print(zipped_data)
<zip object at 0x7f730c059cc0>
>>> print(list(zipped_data))
[('sid', '22'), ('panth', '17'), ('ani', '21')]
>>>

Numbers in python

Number data types in Python:

Integers: Integer(negative, zero and positive) numbers Example: … -3, -2, -1, 0, 1, 2, 3 …
Floating Point Numbers(Decimal numbers) Example: … -3.5, -2.25, -1.0, 0.0, 1.1, 2.2, 3.5 …
Complex Numbers Example: 1 + j, 2 + 4j, 1 - 1j

Operators

A boolean data type represents one of the two values: True or False.

>>> if(2==1):
...   print(True)
... else:
...   print(False)
... 
False
>>>

Assignment Operators
Arithmetic Operators

Addition(+): a + b
Subtraction(-): a - b
Multiplication(*): a * b
Division(/): a / b
Modulus(%): a % b
Floor division(//): a // b
Exponentiation(**): a ** b

Comparison Operators
Logical Operators

Strings

Text is a string data type. Any data type written as text is a string. Any data under single, double or triple quote are strings. There are different string methods and built-in functions to deal with string data types. To check the length of a string use the len() method.

Creating a string

letter = 'P'                # A string could be a single character or a bunch of texts
print(letter)               # P
print(len(letter))          # 1
greeting = 'Hello, World!'  # String could be made using a single or double quote,"Hello, World!"
print(greeting)             # Hello, World!
print(len(greeting))        # 13
sentence = "I hope you are enjoying 30 days of Python Challenge"
print(sentence)

multiline_string = '''I am a teacher and enjoy teaching.
I didn't find anything as rewarding as empowering people.
That is why I created 30 days of python.'''
print(multiline_string)

# Another way of doing the same thing
multiline_string = """I am a teacher and enjoy teaching.
I didn't find anything as rewarding as empowering people.
That is why I created 30 days of python."""
print(multiline_string)

String Concatenation

first_name = 'Amitabh'
last_name = 'Bacchan'
space = ' '
full_name = first_name  +  space + last_name
print(full_name) # Amitabh Bacchan
# Checking the length of a string using len() built-in function
print(len(first_name))  # 8
print(len(last_name))   # 7
print(len(first_name) > len(last_name)) # True
print(len(full_name)) # 16

Escape Sequences in Strings In Python and other programming languages \ followed by a character is an escape sequence. Let us see the most common escape characters:
\n: new line
\t: Tab means(8 spaces)
\: Back slash
\’: Single quote (’)
\”: Double quote (”)
String Formatting

Old Style String Formatting (% Operator) %s - String (or any object with a string representation, like numbers) %d - Integers %f - Floating point numbers “%.number of digitsf” - Floating point numbers with fixed precision.

# Strings only
first_name = 'Asabeneh'
last_name = 'Yetayeh'
language = 'Python'
formated_string = 'I am %s %s. I teach %s' %(first_name, last_name, language)
print(formated_string)

# Strings  and numbers
radius = 10
pi = 3.14
area = pi * radius ** 2
formated_string = 'The area of circle with a radius %d is %.2f.' %(radius, area) # 2 refers the 2 significant digits after the point

python_libraries = ['Django', 'Flask', 'NumPy', 'Matplotlib','Pandas']
formated_string = 'The following are python libraries:%s' % (python_libraries)
print(formated_string) # "The following are python libraries:['Django', 'Flask', 'NumPy', 'Matplotlib','Pandas']"

New Style String Formatting (str.format) This format was introduced in Python version 3.

first_name = 'Asabeneh'
last_name = 'Yetayeh'
language = 'Python'
formated_string = 'I am {} {}. I teach {}'.format(first_name, last_name, language)
print(formated_string)
a = 4
b = 3

print('{} + {} = {}'.format(a, b, a + b))
print('{} - {} = {}'.format(a, b, a - b))
print('{} * {} = {}'.format(a, b, a * b))
print('{} / {} = {:.2f}'.format(a, b, a / b)) # limits it to two digits after decimal
print('{} % {} = {}'.format(a, b, a % b))
print('{} // {} = {}'.format(a, b, a // b))
print('{} ** {} = {}'.format(a, b, a ** b))

# output
4 + 3 = 7
4 - 3 = 1
4 * 3 = 12
4 / 3 = 1.33
4 % 3 = 1
4 // 3 = 1
4 ** 3 = 64

# Strings  and numbers
radius = 10
pi = 3.14
area = pi * radius ** 2
formated_string = 'The area of a circle with a radius {} is {:.2f}.'.format(radius, area) # 2 digits after decimal
print(formated_string)

String Interpolation / f-Strings (Python 3.6+) Another new string formatting is string interpolation, f-strings. Strings start with f and we can inject the data in their corresponding positions.

a = 4
b = 3
print(f'{a} + {b} = {a +b}')
print(f'{a} - {b} = {a - b}')
print(f'{a} * {b} = {a * b}')
print(f'{a} / {b} = {a / b:.2f}')
print(f'{a} % {b} = {a % b}')
print(f'{a} // {b} = {a // b}')
print(f'{a} ** {b} = {a ** b}')

Python Strings as Sequences of Characters

Python strings are sequences of characters, and share their basic methods of access with other Python ordered sequences of objects – lists and tuples. The simplest way of extracting single characters from strings (and individual members from any sequence) is to unpack them into corresponding variables.

language = 'Python'
a,b,c,d,e,f = language # unpacking sequence characters into variables
print(a) # P
print(b) # y
print(c) # t
print(d) # h
print(e) # o
print(f) # n

``` language = 'Python' first_letter = language[0] print(first_letter) # P second_letter = language[1] print(second_letter) # y last_index = len(language) - 1 last_letter = language[last_index] print(last_letter) # n language = 'Python' last_letter = language[-1] print(last_letter) # n second_last = language[-2] print(second_last) # o ```

Slicing Python Strings

language = 'Python'
first_three = language[0:3] # starts at zero index and up to 3 but not include 3
print(first_three) #Pyt
last_three = language[3:6]
print(last_three) # hon
# Another way
last_three = language[-3:]
print(last_three)   # hon
last_three = language[3:]
print(last_three)   # hon

Reversing a string

greeting = 'Hello, World!'
print(greeting[::-1]) # !dlroW ,olleH

Skipping Characters While Slicing

language = 'Python'
pto = language[0:6:2] #[starting index,ending index,steps]
print(pto) # Pto

String Methods

capitalize(): Converts the first character of the string to capital letter.

challenge = 'python is a mastikhor language'
print(challenge.capitalize()) # 'Python is a mastikhor language'

count(): returns occurrences of substring in string, count(substring, start=.., end=..). The start is a starting indexing for counting and end is the last index to count.

challenge = 'thirty days of python'
print(challenge.count('y')) # 3
print(challenge.count('y', 7, 14)) # 1, 
print(challenge.count('th')) # 2

endswith(): Checks if a string ends with a specified ending.

challenge = 'thirty days of python'
print(challenge.endswith('on'))   # True
print(challenge.endswith('tion')) # False

expandtabs(): Replaces tab character with spaces, default tab size is 8. It takes tab size argument.

challenge = 'thirty\tdays\tof\tpython'
print(challenge.expandtabs())   # 'thirty  days    of      python'
print(challenge.expandtabs(10)) # 'thirty    days      of        python'

find(): Returns the index of the first occurrence of a substring, if not found returns -1.

challenge = 'thirty days of python'
print(challenge.find('y'))  # 5
print(challenge.find('th')) # 0

rfind(): Returns the index of the last occurrence of a substring, if not found returns -1.

challenge = 'thirty days of python'
print(challenge.rfind('y'))  # 16
print(challenge.rfind('th')) # 17

index(): Returns the lowest index of a substring, additional arguments indicate starting and ending index (default 0 and string length - 1). If the substring is not found it raises a valueError.

challenge = 'thirty days of python'
sub_string = 'da'
print(challenge.index(sub_string))  # 7
print(challenge.index(sub_string, 9)) # error

rindex(): Returns the highest index of a substring, additional arguments indicate starting and ending index (default 0 and string length - 1)

challenge = 'thirty days of python'
sub_string = 'da'
print(challenge.rindex(sub_string))  # 7
print(challenge.rindex(sub_string, 9)) # error
print(challenge.rindex('on', 8)) # 19

isalnum(): Checks alphanumeric character

challenge = 'ThirtyDaysPython'
print(challenge.isalnum()) # True
 
challenge = '30DaysPython'
print(challenge.isalnum()) # True
 
challenge = 'thirty days of python'
print(challenge.isalnum()) # False, space is not an alphanumeric character
 
challenge = 'thirty days of python 2019'
print(challenge.isalnum()) # False

isalpha(): Checks if all string elements are alphabet characters (a-z and A-Z)

challenge = 'thirty days of python'
print(challenge.isalpha()) # False, space is once again excluded
challenge = 'ThirtyDaysPython'
print(challenge.isalpha()) # True
num = '123'
print(num.isalpha())      # False

isdecimal(): Checks if all characters in a string are decimal (0-9)

challenge = 'thirty days of python'
print(challenge.isdecimal())  # False
challenge = '123'
print(challenge.isdecimal())  # True
challenge = '\u00B2'
print(challenge.isdigit())   # True 
challenge = '12 3'
print(challenge.isdecimal())  # False, space not allowed

isdigit(): Checks if all characters in a string are numbers (0-9 and some other unicode characters for numbers)

challenge = 'Thirty'
print(challenge.isdigit()) # False
challenge = '30'
print(challenge.isdigit())   # True
challenge = '\u00B2'
print(challenge.isdigit())   # True

isnumeric(): Checks if all characters in a string are numbers or number related (just like isdigit(), just accepts more symbols, like ½)

num = '10'
print(num.isnumeric()) # True
num = '\u00BD' # ½
print(num.isnumeric()) # True
num = '10.5'
print(num.isnumeric()) # False

isidentifier(): Checks for a valid identifier - it checks if a string is a valid variable name

challenge = '30DaysOfPython'
print(challenge.isidentifier()) # False, because it starts with a number
challenge = 'thirty_days_of_python'
print(challenge.isidentifier()) # True

islower(): Checks if all alphabet characters in the string are lowercase

challenge = 'thirty days of python'
print(challenge.islower()) # True
challenge = 'Thirty days of python'
print(challenge.islower()) # False

isupper(): Checks if all alphabet characters in the string are uppercase

challenge = 'thirty days of python'
print(challenge.isupper()) #  False
challenge = 'THIRTY DAYS OF PYTHON'
print(challenge.isupper()) # True

join(): Returns a concatenated string

web_tech = ['HTML', 'CSS', 'JavaScript', 'React']
result = ' '.join(web_tech)
print(result) # 'HTML CSS JavaScript React'

web_tech = ['HTML', 'CSS', 'JavaScript', 'React']
result = '# '.join(web_tech)
print(result) # 'HTML# CSS# JavaScript# React'

strip(): Removes all given characters starting from the beginning and end of the string

challenge = 'thirty days of pythoonnn'
print(challenge.strip('noth')) # 'irty days of py'

replace(): Replaces substring with a given string

challenge = 'thirty days of python'
print(challenge.replace('python', 'coding')) # 'thirty days of coding'

split(): Splits the string, using given string or space as a separator

challenge = 'thirty days of python'
print(challenge.split()) # ['thirty', 'days', 'of', 'python']
challenge = 'thirty, days, of, python'
print(challenge.split(', ')) # ['thirty', 'days', 'of', 'python']

title(): Returns a title cased string

challenge = 'thirty days of python'
print(challenge.title()) # Thirty Days Of Python

swapcase(): Converts all uppercase characters to lowercase and all lowercase characters to uppercase characters

challenge = 'thirty days of python'
print(challenge.swapcase())   # THIRTY DAYS OF PYTHON
challenge = 'Thirty Days Of Python'
print(challenge.swapcase())  # tHIRTY dAYS oF pYTHON

startswith(): Checks if String Starts with the Specified String

challenge = 'thirty days of python'
print(challenge.startswith('thirty')) # True
 
challenge = '30 days of python'
print(challenge.startswith('thirty')) # False

Lists

List: is a collection which is ordered and changeable(modifiable). Allows duplicate members. A list is collection of different data types which is ordered and modifiable(mutable). A list can be empty or it may have different data type items.

Creating a list
- A list can be created in two manners → Using the build-in function list() → Using the [] square brackets

# syntax
lst = list()

empty_list = list() # this is an empty list, no item in the list
print(len(empty_list)) # 0

# syntax
lst = []

empty_list = [] # this is an empty list, no item in the list
print(len(empty_list)) # 0

fruits = ['banana', 'orange', 'mango', 'lemon']                     # list of fruits
vegetables = ['Tomato', 'Potato', 'Cabbage','Onion', 'Carrot']      # list of vegetables
animal_products = ['milk', 'meat', 'butter', 'yoghurt']             # list of animal products
web_techs = ['HTML', 'CSS', 'JS', 'React','Redux', 'Node', 'MongDB'] # list of web technologies
countries = ['Finland', 'Estonia', 'Denmark', 'Sweden', 'Norway'] 

# Print the lists and its length
print('Fruits:', fruits)
print('Number of fruits:', len(fruits))
print('Vegetables:', vegetables)
print('Number of vegetables:', len(vegetables))
print('Animal products:',animal_products)
print('Number of animal products:', len(animal_products))
print('Web technologies:', web_techs)
print('Number of web technologies:', len(web_techs))
print('Countries:', countries)
print('Number of countries:', len(countries))

Outputs:

output
Fruits: ['banana', 'orange', 'mango', 'lemon']
Number of fruits: 4
Vegetables: ['Tomato', 'Potato', 'Cabbage', 'Onion', 'Carrot']
Number of vegetables: 5
Animal products: ['milk', 'meat', 'butter', 'yoghurt']
Number of animal products: 4
Web technologies: ['HTML', 'CSS', 'JS', 'React', 'Redux', 'Node', 'MongDB']
Number of web technologies: 7
Countries: ['Finland', 'Estonia', 'Denmark', 'Sweden', 'Norway']
Number of countries: 5

A list can have more than one datatype

 lst = ['Asabeneh', 250, True, {'country':'Finland', 'city':'Helsinki'}] # list containing different data types

Indexing in list

Unpacking list items

lst = ['item1','item2','item3', 'item4', 'item5']
first_item, second_item, third_item, *rest = lst
print(first_item)     # item1
print(second_item)    # item2
print(third_item)     # item3
print(rest)           # ['item4', 'item5']

# First Example
fruits = ['banana', 'orange', 'mango', 'lemon','lime','apple']
first_fruit, second_fruit, third_fruit, *rest = fruits 
print(first_fruit)     # banana
print(second_fruit)    # orange
print(third_fruit)     # mango
print(rest)           # ['lemon','lime','apple']
# Second Example about unpacking list
first, second, third,*rest, tenth = [1,2,3,4,5,6,7,8,9,10]
print(first)          # 1
print(second)         # 2
print(third)          # 3
print(rest)           # [4,5,6,7,8,9]
print(tenth)          # 10
# Third Example about unpacking list
countries = ['Germany', 'France','Belgium','Sweden','Denmark','Finland','Norway','Iceland','Estonia']
gr, fr, bg, sw, *scandic, es = countries
print(gr) 
print(fr)
print(bg)
print(sw)
print(scandic)
print(es)

Slicing Items from a List Positive Indexing: We can specify a range of positive indexes by specifying the start, end and step, the return value will be a new list. (default values for start = 0, end = len(lst) - 1 (last item), step = 1)

fruits = ['banana', 'orange', 'mango', 'lemon']
all_fruits = fruits[0:4] # it returns all the fruits
# this will also give the same result as the one above
all_fruits = fruits[0:] # if we don't set where to stop it takes all the rest
orange_and_mango = fruits[1:3] # it does not include the first index
orange_mango_lemon = fruits[1:]
orange_and_lemon = fruits[::2] # here we used a 3rd argument, step. It will take every 2cnd item - ['banana', 'mango']

Negative Indexing: We can specify a range of negative indexes by specifying the start, end and step, the return value will be a new list.

fruits = ['banana', 'orange', 'mango', 'lemon']
all_fruits = fruits[-4:] # it returns all the fruits
orange_and_mango = fruits[-3:-1] # it does not include the last index,['orange', 'mango']
orange_mango_lemon = fruits[-3:] # this will give starting from -3 to the end,['orange', 'mango', 'lemon']
reverse_fruits = fruits[::-1] # a negative step will take the list in reverse order,['lemon', 'mango', 'orange', 'banana']

Modifying Lists List is a mutable or modifiable ordered collection of items. Lets modify the fruit list.

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits[0] = 'avocado'
print(fruits)       #  ['avocado', 'orange', 'mango', 'lemon']
fruits[1] = 'apple'
print(fruits)       #  ['avocado', 'apple', 'mango', 'lemon']
last_index = len(fruits) - 1
fruits[last_index] = 'lime'
print(fruits)        #  ['avocado', 'apple', 'mango', 'lime']

Checking Items in a List Checking an item if it is a member of a list using in operator. See the example below.

fruits = ['banana', 'orange', 'mango', 'lemon']
does_exist = 'banana' in fruits
print(does_exist)  # True
does_exist = 'lime' in fruits
print(does_exist)  # False

Adding Items to a List To add item to the end of an existing list we use the method append().

# syntax
lst = list()
lst.append(item)

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits.append('apple')
print(fruits)           # ['banana', 'orange', 'mango', 'lemon', 'apple']
fruits.append('lime')   # ['banana', 'orange', 'mango', 'lemon', 'apple', 'lime']
print(fruits)

Inserting Items into a List We can use insert() method to insert a single item at a specified index in a list. Note that other items are shifted to the right. The insert() methods takes two arguments:index and an item to insert.

# syntax
lst = ['item1', 'item2']
lst.insert(index, item)

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits.insert(2, 'apple') # insert apple between orange and mango
print(fruits)           # ['banana', 'orange', 'apple', 'mango', 'lemon']
fruits.insert(3, 'lime')   # ['banana', 'orange', 'apple', 'lime', 'mango', 'lemon']
print(fruits)

Removing Items from a List The remove method removes a specified item from a list

# syntax
lst = ['item1', 'item2']
lst.remove(item)

fruits = ['banana', 'orange', 'mango', 'lemon', 'banana']
fruits.remove('banana')
print(fruits)  # ['orange', 'mango', 'lemon', 'banana'] - this method removes the first occurrence of the item in the list
fruits.remove('lemon')
print(fruits)  # ['orange', 'mango', 'banana']

Removing Items Using Pop The pop() method removes the specified index, (or the last item if index is not specified):

# syntax
lst = ['item1', 'item2']
lst.pop()       # last item
lst.pop(index)

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits.pop()
print(fruits)       # ['banana', 'orange', 'mango']
 
fruits.pop(0)
print(fruits)       # ['orange', 'mango']

Removing Items Using Del The del keyword removes the specified index and it can also be used to delete items within index range. It can also delete the list completely

# syntax
lst = ['item1', 'item2']
del lst[index] # only a single item
del lst        # to delete the list completely

fruits = ['banana', 'orange', 'mango', 'lemon', 'kiwi', 'lime']
del fruits[0]
print(fruits)       # ['orange', 'mango', 'lemon', 'kiwi', 'lime']
del fruits[1]
print(fruits)       # ['orange', 'lemon', 'kiwi', 'lime']
del fruits[1:3]     # this deletes items between given indexes, so it does not delete the item with index 3!
print(fruits)       # ['orange', 'lime']
del fruits
print(fruits)       # This should give: NameError: name 'fruits' is not defined

Clearing List Items The clear() method empties the list:

# syntax
lst = ['item1', 'item2']
lst.clear()

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits.clear()
print(fruits)       # []

Copying a List It is possible to copy a list by reassigning it to a new variable in the following way: list2 = list1. Now, list2 is a reference of list1, any changes we make in list2 will also modify the original, list1. But there are lots of case in which we do not like to modify the original instead we like to have a different copy. One of way of avoiding the problem above is using copy().

# syntax
lst = ['item1', 'item2']
lst_copy = lst.copy()

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits_copy = fruits.copy()
print(fruits_copy)       # ['banana', 'orange', 'mango', 'lemon']

Joining Lists There are several ways to join, or concatenate, two or more lists in Python. → Plus Operator (+)

# syntax
list3 = list1 + list2

positive_numbers = [1, 2, 3, 4, 5]
zero = [0]
negative_numbers = [-5,-4,-3,-2,-1]
integers = negative_numbers + zero + positive_numbers
print(integers) # [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]
fruits = ['banana', 'orange', 'mango', 'lemon']
vegetables = ['Tomato', 'Potato', 'Cabbage', 'Onion', 'Carrot']
fruits_and_vegetables = fruits + vegetables
print(fruits_and_vegetables ) # ['banana', 'orange', 'mango', 'lemon', 'Tomato', 'Potato', 'Cabbage', 'Onion', 'Carrot']

→ Joining using extend() method The extend() method allows to append list in a list. See the example below.

# syntax
list1 = ['item1', 'item2']
list2 = ['item3', 'item4', 'item5']
list1.extend(list2) # ['item1', 'item2', 'item3', 'item4', 'item5']

num1 = [0, 1, 2, 3]
num2= [4, 5, 6]
num1.extend(num2)
print('Numbers:', num1) # Numbers: [0, 1, 2, 3, 4, 5, 6]
negative_numbers = [-5,-4,-3,-2,-1]
positive_numbers = [1, 2, 3,4,5]
zero = [0]
negative_numbers.extend(zero)
negative_numbers.extend(positive_numbers)
print('Integers:', negative_numbers) # Integers: [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]
fruits = ['banana', 'orange', 'mango', 'lemon']
vegetables = ['Tomato', 'Potato', 'Cabbage', 'Onion', 'Carrot']
fruits.extend(vegetables)
print('Fruits and vegetables:', fruits ) # Fruits and vegetables: ['banana', 'orange', 'mango', 'lemon', 'Tomato', 'Potato', 'Cabbage', 'Onion', 'Carrot']

Counting Items in a List The count() method returns the number of times an item appears in a list:

# syntax
lst = ['item1', 'item2']
lst.count(item)

fruits = ['banana', 'orange', 'mango', 'lemon']
print(fruits.count('orange'))   # 1
ages = [22, 19, 24, 25, 26, 24, 25, 24]
print(ages.count(24))           # 3

Finding Index of an Item The index() method returns the index of an item in the list:

# syntax
lst = ['item1', 'item2']
lst.index(item)

fruits = ['banana', 'orange', 'mango', 'lemon']
print(fruits.index('orange'))   # 1
ages = [22, 19, 24, 25, 26, 24, 25, 24]
print(ages.index(24))           # 2, the first occurrence

Reversing a List The reverse() method reverses the order of a list.

# syntax
lst = ['item1', 'item2']
lst.reverse()

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits.reverse()
print(fruits) # ['lemon', 'mango', 'orange', 'banana']
ages = [22, 19, 24, 25, 26, 24, 25, 24]
ages.reverse()
print(ages) # [24, 25, 24, 26, 25, 24, 19, 22]

Sorting List Items To sort lists we can use sort() method or sorted() built-in functions. The sort() method reorders the list items in ascending order and modifies the original list. If an argument of sort() method reverse is equal to true, it will arrange the list in descending order. → sort(): this method modifies the original list

# syntax
lst = ['item1', 'item2']
lst.sort()                # ascending
lst.sort(reverse=True)    # descending

Example:

fruits = ['banana', 'orange', 'mango', 'lemon']
fruits.sort()
print(fruits)             # sorted in alphabetical order, ['banana', 'lemon', 'mango', 'orange']
fruits.sort(reverse=True)
print(fruits) # ['orange', 'mango', 'lemon', 'banana']
ages = [22, 19, 24, 25, 26, 24, 25, 24]
ages.sort()
print(ages) #  [19, 22, 24, 24, 24, 25, 25, 26]
 
ages.sort(reverse=True)
print(ages) #  [26, 25, 25, 24, 24, 24, 22, 19]

sorted(): returns the ordered list without modifying the original list Example:

fruits = ['banana', 'orange', 'mango', 'lemon']
print(sorted(fruits))   # ['banana', 'lemon', 'mango', 'orange']
# Reverse order
fruits = ['banana', 'orange', 'mango', 'lemon']
fruits = sorted(fruits,reverse=True)
print(fruits)     # ['orange', 'mango', 'lemon', 'banana']

Tuples

A tuple is a collection of different data types which is ordered and unchangeable (immutable). Tuples are written with round brackets, (). Once a tuple is created, we cannot change its values. We cannot use add, insert, remove methods in a tuple because it is not modifiable (mutable). Unlike list, tuple has few methods. Methods related to tuples:

tuple(): to create an empty tuple
count(): to count the number of a specified item in a tuple
index(): to find the index of a specified item in a tuple
+ operator: to join two or more tuples and to create a new tuple

Creating a Tuple

Empty tuple: Creating an empty tuple

# syntax
empty_tuple = ()
# or using the tuple constructor
empty_tuple = tuple()

Tuple with initial values

# syntax
tpl = ('item1', 'item2','item3')

fruits = ('banana', 'orange', 'mango', 'lemon')

Tuple length

We use the len() method to get the length of a tuple.

# syntax
tpl = ('item1', 'item2', 'item3')
len(tpl)

Accessing Tuple Items

Positive Indexing Similar to the list data type we use positive or negative indexing to access tuple items.

# Syntax
tpl = ('item1', 'item2', 'item3')
first_item = tpl[0]
second_item = tpl[1]

fruits = ('banana', 'orange', 'mango', 'lemon')
first_fruit = fruits[0]
second_fruit = fruits[1]
last_index =len(fruits) - 1
last_fruit = fruits[last_index]

Negative indexing Negative indexing means beginning from the end, -1 refers to the last item, -2 refers to the second last and the negative of the list/tuple length refers to the first item.

# Syntax
tpl = ('item1', 'item2', 'item3','item4')
first_item = tpl[-4]
second_item = tpl[-3]

fruits = ('banana', 'orange', 'mango', 'lemon')
first_fruit = fruits[-4]
second_fruit = fruits[-3]
last_fruit = fruits[-1]

Slicing tuples

We can slice out a sub-tuple by specifying a range of indexes where to start and where to end in the tuple, the return value will be a new tuple with the specified items.

Range of Positive Indexes

# Syntax
tpl = ('item1', 'item2', 'item3','item4')
all_items = tpl[0:4]         # all items
all_items = tpl[0:]         # all items
middle_two_items = tpl[1:3]  # does not include item at index 3

fruits = ('banana', 'orange', 'mango', 'lemon')
all_fruits = fruits[0:4]    # all items
all_fruits= fruits[0:]      # all items
orange_mango = fruits[1:3]  # doesn't include item at index 3
orange_to_the_rest = fruits[1:]

Range of Negative Indexes

# Syntax
tpl = ('item1', 'item2', 'item3','item4')
all_items = tpl[-4:]         # all items
middle_two_items = tpl[-3:-1]  # does not include item at index 3 (-1)

fruits = ('banana', 'orange', 'mango', 'lemon')
all_fruits = fruits[-4:]    # all items
orange_mango = fruits[-3:-1]  # doesn't include item at index 3
orange_to_the_rest = fruits[-3:]

Changing Tuples to Lists

We can change tuples to lists and lists to tuples. Tuple is immutable if we want to modify a tuple we should change it to a list.

# Syntax
tpl = ('item1', 'item2', 'item3','item4')
lst = list(tpl)

fruits = ('banana', 'orange', 'mango', 'lemon')
fruits = list(fruits)
fruits[0] = 'apple'
print(fruits)     # ['apple', 'orange', 'mango', 'lemon']
fruits = tuple(fruits)
print(fruits)     # ('apple', 'orange', 'mango', 'lemon')

Checking an Item in a Tuple

We can check if an item exists or not in a tuple using in, it returns a boolean.

# Syntax
tpl = ('item1', 'item2', 'item3','item4')
'item2' in tpl # True

fruits = ('banana', 'orange', 'mango', 'lemon')
print('orange' in fruits) # True
print('apple' in fruits) # False
fruits[0] = 'apple' # TypeError: 'tuple' object does not support item assignment

Joining Tuples

We can join two or more tuples using + operator

# syntax
tpl1 = ('item1', 'item2', 'item3')
tpl2 = ('item4', 'item5','item6')
tpl3 = tpl1 + tpl2

fruits = ('banana', 'orange', 'mango', 'lemon')
vegetables = ('Tomato', 'Potato', 'Cabbage','Onion', 'Carrot')
fruits_and_vegetables = fruits + vegetables

Deleting Tuples

It is not possible to remove a single item in a tuple but it is possible to delete the tuple itself using del.

# syntax
tpl1 = ('item1', 'item2', 'item3')
del tpl1

fruits = ('banana', 'orange', 'mango', 'lemon')
del fruits

Sets

Set is a collection of items. Let me take you back to your elementary or high school Mathematics lesson. The Mathematics definition of a set can be applied also in Python. Set is a collection of unordered and un-indexed distinct elements. In Python set is used to store unique items, and it is possible to find the union, intersection, difference, symmetric difference, subset, super set and disjoint set among sets.

Creating a Set

To create an empty set, we use the set() function. Empty curly brackets {} will create a dictionary.

Creating an empty set

# syntax
st = set()

Creating a set with initial items

# syntax
st = {'item1', 'item2', 'item3', 'item4'}

Example:

# syntax
fruits = {'banana', 'orange', 'mango', 'lemon'}

Getting Set’s Length

We use len() method to find the length of a set.

# syntax
st = {'item1', 'item2', 'item3', 'item4'}
len(st)

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
len(fruits)

Accessing Items in a Set

We use loops to access items. We will see this in loop section

Checking an Item

To check if an item exist in a list we use in membership operator.

# syntax
st = {'item1', 'item2', 'item3', 'item4'}
print("Does set st contain item3? ", 'item3' in st) # Does set st contain item3? True

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
print('mango' in fruits ) # True

Adding Items to a Set

Once a set is created we cannot change any items and we can also add additional items.

Add one item using add()

# syntax
st = {'item1', 'item2', 'item3', 'item4'}
st.add('item5')

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
fruits.add('lime')

Add multiple items using update() The update() allows to add multiple items to a set. The update() takes a list argument.

# syntax
st = {'item1', 'item2', 'item3', 'item4'}
st.update(['item5','item6','item7'])

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
vegetables = ('tomato', 'potato', 'cabbage','onion', 'carrot')
fruits.update(vegetables)

Removing Items from a Set

We can remove an item from a set using remove() method. If the item is not found remove() method will raise errors, so it is good to check if the item exist in the given set. However, discard() method doesn’t raise any errors.

# syntax
st = {'item1', 'item2', 'item3', 'item4'}
st.remove('item2')

The pop() methods remove a random item from a list and it returns the removed item. Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
fruits.pop()  # removes a random item from the set

If we are interested in the removed item.

fruits = {'banana', 'orange', 'mango', 'lemon'}
removed_item = fruits.pop()

Clearing Items in a Set

If we want to clear or empty the set we use clear method.

# syntax
st = {'item1', 'item2', 'item3', 'item4'}
st.clear()

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
fruits.clear()
print(fruits) # set()

Deleting a Set

If we want to delete the set itself we use del operator.

# syntax
st = {'item1', 'item2', 'item3', 'item4'}
del st

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
del fruits

Converting List to Set

We can convert list to set and set to list. Converting list to set removes duplicates and only unique items will be reserved.

# syntax
lst = ['item1', 'item2', 'item3', 'item4', 'item1']
st = set(lst)  # {'item2', 'item4', 'item1', 'item3'} - the order is random, because sets in general are unordered

Example:

fruits = ['banana', 'orange', 'mango', 'lemon','orange', 'banana']
fruits = set(fruits) # {'mango', 'lemon', 'banana', 'orange'}

Joining Sets

We can join two sets using the union() or update() method or | symbol .

Union This method returns a new set

# syntax
st1 = {'item1', 'item2', 'item3', 'item4'}
st2 = {'item5', 'item6', 'item7', 'item8'}
st3 = st1.union(st2) #st3 = st1 | st2

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
vegetables = {'tomato', 'potato', 'cabbage','onion', 'carrot'}
print(fruits.union(vegetables)) # {'lemon', 'carrot', 'tomato', 'banana', 'mango', 'orange', 'cabbage', 'potato', 'onion'}
# or using this : print(fruits | vegetables)

Update This method inserts a set into a given set

# syntax
st1 = {'item1', 'item2', 'item3', 'item4'}
st2 = {'item5', 'item6', 'item7', 'item8'}
st1.update(st2) # st2 contents are added to st1

Example:

fruits = {'banana', 'orange', 'mango', 'lemon'}
vegetables = {'tomato', 'potato', 'cabbage','onion', 'carrot'}
fruits.update(vegetables)
print(fruits) # {'lemon', 'carrot', 'tomato', 'banana', 'mango', 'orange', 'cabbage', 'potato', 'onion'}

Finding Intersection Items

Intersection returns a set of items which are in both the sets or using & symbol. See the example

# syntax
st1 = {'item1', 'item2', 'item3', 'item4'}
st2 = {'item3', 'item2'}
st1.intersection(st2) # {'item3', 'item2'}
# or using thia : st1 & st2

Example:

whole_numbers = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
even_numbers = {0, 2, 4, 6, 8, 10}
whole_numbers.intersection(even_numbers) # {0, 2, 4, 6, 8, 10}
 
python = {'p', 'y', 't', 'h', 'o','n'}
dragon = {'d', 'r', 'a', 'g', 'o','n'}
python.intersection(dragon)     # {'o', 'n'}
# python & dragon

Checking Subset and Super Set

A set can be a subset or super set of other sets:

Subset: issubset()
Super set: issuperset

# syntax
st1 = {'item1', 'item2', 'item3', 'item4'}
st2 = {'item2', 'item3'}
st2.issubset(st1) # True
st1.issuperset(st2) # True

Example:

whole_numbers = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
even_numbers = {0, 2, 4, 6, 8, 10}
whole_numbers.issubset(even_numbers) # False, because it is a super set
whole_numbers.issuperset(even_numbers) # True
 
python = {'p', 'y', 't', 'h', 'o','n'}
dragon = {'d', 'r', 'a', 'g', 'o','n'}
python.issubset(dragon)     # False

Checking the Difference Between Two Sets

It returns the difference between two sets or using - symbol .

# syntax
st1 = {'item1', 'item2', 'item3', 'item4'}
st2 = {'item2', 'item3'}
st2.difference(st1) # set() : st2 - st1
st1.difference(st2) # {'item1', 'item4'} => st1\st2  : st2 - st1

Example:

whole_numbers = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
even_numbers = {0, 2, 4, 6, 8, 10}
whole_numbers.difference(even_numbers) # {1, 3, 5, 7, 9}
 
python = {'p', 'y', 't', 'o','n'}
dragon = {'d', 'r', 'a', 'g', 'o','n'}
python.difference(dragon)     # {'p', 'y', 't'}  - the result is unordered (characteristic of sets)
# python - dragon
dragon.difference(python)     # {'d', 'r', 'a', 'g'}
# dragon - python

Finding Symmetric Difference Between Two Sets

It returns the symmetric difference between two sets. It means that it returns a set that contains all items from both sets, except items that are present in both sets, mathematically: (A\B) ∪ (B\A)

# syntax
st1 = {'item1', 'item2', 'item3', 'item4'}
st2 = {'item2', 'item3'}
# it means (A\B)∪(B\A)
st2.symmetric_difference(st1) # {'item1', 'item4'} : st2 ^ st1

Example:

whole_numbers = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
some_numbers = {1, 2, 3, 4, 5}
whole_numbers.symmetric_difference(some_numbers) # {0, 6, 7, 8, 9, 10}
 
python = {'p', 'y', 't', 'h', 'o','n'}
dragon = {'d', 'r', 'a', 'g', 'o','n'}
python.symmetric_difference(dragon)  # {'r', 't', 'p', 'y', 'g', 'a', 'd', 'h'}
# python ^ dragon

Joining Sets

If two sets do not have a common item or items we call them disjoint sets. We can check if two sets are joint or disjoint using isdisjoint() method.

# syntax
st1 = {'item1', 'item2', 'item3', 'item4'}
st2 = {'item2', 'item3'}
st2.isdisjoint(st1) # False

Example:

even_numbers = {0, 2, 4 ,6, 8}
odd_numbers = {1, 3, 5, 7, 9}
even_numbers.isdisjoint(odd_numbers) # True, because no common item
 
python = {'p', 'y', 't', 'h', 'o','n'}
dragon = {'d', 'r', 'a', 'g', 'o','n'}
python.isdisjoint(dragon)  # False, there are common items {'o', 'n'}

Dictionaries

A dictionary is a collection of unordered, modifiable(mutable) paired (key: value) data type.

Creating a Dictionary

To create a dictionary we use curly brackets, {} or the dict() built-in function.

# syntax
empty_dict = {}
# Dictionary with data values
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}

Example:

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_marred':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
    }
    }

The dictionary above shows that a value could be any data types:string, boolean, list, tuple, set or a dictionary.

Dictionary Length

It checks the number of ‘key: value’ pairs in the dictionary.

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
print(len(dct)) # 4

Example:

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_married':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
    }
    }
print(len(person)) # 7

Accessing Dictionary Items

We can access Dictionary items by referring to its key name.

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
print(dct['key1']) # value1
print(dct['key4']) # value4

Example:

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_marred':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
    }
    }
print(person['first_name']) # Asabeneh
print(person['country'])    # Finland
print(person['skills'])     # ['JavaScript', 'React', 'Node', 'MongoDB', 'Python']
print(person['skills'][0])  # JavaScript
print(person['address']['street']) # Space street
print(person['city'])       # Error

Accessing an item by key name raises an error if the key does not exist. To avoid this error first we have to check if a key exist or we can use the get method. The get method returns None, which is a NoneType object data type, if the key does not exist.

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_marred':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
    }
    }
print(person.get('first_name')) # Asabeneh
print(person.get('country'))    # Finland
print(person.get('skills')) #['JavaScript', 'React', 'Node', 'MongoDB', 'Python']
print(person.get('city'))   # None

Adding Items to a Dictionary

We can add new key and value pairs to a dictionary

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
dct['key5'] = 'value5'

Example:

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_marred':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
        }
}
person['job_title'] = 'Instructor'
person['skills'].append('HTML')
print(person)

Modifying Items in a Dictionary

We can modify items in a dictionary

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
dct['key1'] = 'value-one'

Example:

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_marred':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
    }
    }
person['first_name'] = 'Eyob'
person['age'] = 252

Checking Keys in a Dictionary

We use the in operator to check if a key exist in a dictionary

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
print('key2' in dct) # True
print('key5' in dct) # False

Removing Key and Value Pairs from a Dictionary

pop(key): removes the item with the specified key name:
popitem(): removes the last item
del: removes an item with specified key name

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
dct.pop('key1') # removes key1 item
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
dct.popitem() # removes the last item
del dct['key2'] # removes key2 item

Example:

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_marred':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
    }
    }
person.pop('first_name')        # Removes the firstname item
person.popitem()                # Removes the address item
del person['is_married']        # Removes the is_married item

Changing Dictionary to a List of Items

The items() method changes dictionary to a list of tuples.

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
print(dct.items()) # dict_items([('key1', 'value1'), ('key2', 'value2'), ('key3', 'value3'), ('key4', 'value4')])

Clearing a Dictionary

If we don’t want the items in a dictionary we can clear them using clear() method

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
print(dct.clear()) # None

Deleting a Dictionary

If we do not use the dictionary we can delete it completely

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
del dct

Copy a Dictionary

We can copy a dictionary using a copy() method. Using copy we can avoid mutation of the original dictionary.

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
dct_copy = dct.copy() # {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}

Getting Dictionary Keys as a List

The keys() method gives us all the keys of a a dictionary as a list.

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
keys = dct.keys()
print(keys)     # dict_keys(['key1', 'key2', 'key3', 'key4'])

Getting Dictionary Values as a List

The values method gives us all the values of a a dictionary as a list.

# syntax
dct = {'key1':'value1', 'key2':'value2', 'key3':'value3', 'key4':'value4'}
values = dct.values()
print(values)     # dict_values(['value1', 'value2', 'value3', 'value4'])

Conditionals

By default, statements in Python script are executed sequentially from top to bottom. If the processing logic require so, the sequential flow of execution can be altered in two way:

Conditional execution: a block of one or more statements will be executed if a certain expression is true
Repetitive execution: a block of one or more statements will be repetitively executed as long as a certain expression is true. In this section, we will cover if, else, elif statements. The comparison and logical operators we learned in previous sections will be useful here.

If Condition

In python and other programming languages the key word if is used to check if a condition is true and to execute the block code. Remember the indentation after the colon.

# syntax
if condition:
    this part of code runs for truthy conditions

Example: 1

a = 3
if a > 0:
    print('A is a positive number')
# A is a positive number

As you can see in the example above, 3 is greater than 0. The condition was true and the block code was executed. However, if the condition is false, we do not see the result. In order to see the result of the falsy condition, we should have another block, which is going to be else.

If Else

If condition is true the first block will be executed, if not the else condition will run.

# syntax
if condition:
    this part of code runs for truthy conditions
else:
     this part of code runs for false conditions

Example:

a = 3
if a < 0:
    print('A is a negative number')
else:
    print('A is a positive number')

The condition above proves false, therefore the else block was executed. How about if our condition is more than two? We could use elif.

If Elif Else

In our daily life, we make decisions on daily basis. We make decisions not by checking one or two conditions but multiple conditions. As similar to life, programming is also full of conditions. We use elif when we have multiple conditions.

# syntax
if condition:
    code
elif condition:
    code
else:
    code

Example:

a = 0
if a > 0:
    print('A is a positive number')
elif a < 0:
    print('A is a negative number')
else:
    print('A is zero')

Short Hand

# syntax
code if condition else code

Example:

a = 3
print('A is positive') if a > 0 else print('A is negative') # first condition met, 'A is positive' will be printed

Nested Conditions

Conditions can be nested

# syntax
if condition:
    code
    if condition:
    code

Example:

a = 0
if a > 0:
    if a % 2 == 0:
        print('A is a positive and even integer')
    else:
        print('A is a positive number')
elif a == 0:
    print('A is zero')
else:
    print('A is a negative number')

We can avoid writing nested condition by using logical operator and.

If Condition and Logical Operators

# syntax
if condition and condition:
    code

Example:

a = 0
if a > 0 and a % 2 == 0:
        print('A is an even and positive integer')
elif a > 0 and a % 2 !=  0:
     print('A is a positive integer')
elif a == 0:
    print('A is zero')
else:
    print('A is negative')

If and Or Logical Operators

# syntax
if condition or condition:
    code

Example:

user = 'James'
access_level = 3
if user == 'admin' or access_level >= 4:
        print('Access granted!')
else:
    print('Access denied!')

Loops

Life is full of routines. In programming we also do lots of repetitive tasks. In order to handle repetitive task programming languages use loops. Python programming language also provides the following types of two loops:

while loop
for loop

While Loop

We use the reserved word while to make a while loop. It is used to execute a block of statements repeatedly until a given condition is satisfied. When the condition becomes false, the lines of code after the loop will be continued to be executed.

  # syntax
while condition:
    code goes here

Example:

count = 0
while count < 5:
    print(count)
    count = count + 1
#prints from 0 to 4

In the above while loop, the condition becomes false when count is 5. That is when the loop stops. If we are interested to run block of code once the condition is no longer true, we can use else.

  # syntax
while condition:
    code goes here
else:
    code goes here

Example:

count = 0
while count < 5:
    print(count)
    count = count + 1
else:
    print(count)

The above loop condition will be false when count is 5 and the loop stops, and execution starts the else statement. As a result 5 will be printed.

Break and Continue - Part 1

Break: We use break when we like to get out of or stop the loop.

# syntax
while condition:
    code goes here
    if another_condition:
        break

Example:

count = 0
while count < 5:
    print(count)
    count = count + 1
    if count == 3:
        break

The above while loop only prints 0, 1, 2, but when it reaches 3 it stops.

Continue: With the continue statement we can skip the current iteration, and continue with the next:

  # syntax
while condition:
    code goes here
    if another_condition:
        continue

Example:

count = 0
while count < 5:
    if count == 3:
        count += 1
        continue
    print(count)
    count = count + 1

The above while loop only prints 0, 1, 2 and 4 (skips 3).

For Loop

A for keyword is used to make a for loop, similar with other programming languages, but with some syntax differences. Loop is used for iterating over a sequence (that is either a list, a tuple, a dictionary, a set, or a string).

-Using For loop on list

# syntax
for iterator in lst:
    code goes here

Example:

numbers = [0, 1, 2, 3, 4, 5]
for number in numbers: # number is temporary name to refer to the list's items, valid only inside this loop
    print(number)       # the numbers will be printed line by line, from 0 to 5

-Using For loop on string

# syntax
for iterator in string:
    code goes here

Example:

language = 'Python'
for letter in language:
    print(letter)
 
 
for i in range(len(language)):
    print(language[i])

-Using For loop on tuple

# syntax
for iterator in tpl:
    code goes here

Example:

numbers = (0, 1, 2, 3, 4, 5)
for number in numbers:
    print(number)

For loop with dictionary Looping through a dictionary gives you the key of the dictionary.

  # syntax
for iterator in dct:
    code goes here

Example:

person = {
    'first_name':'Asabeneh',
    'last_name':'Yetayeh',
    'age':250,
    'country':'Finland',
    'is_marred':True,
    'skills':['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address':{
        'street':'Space street',
        'zipcode':'02210'
    }
}
for key in person:
    print(key)
 
for key, value in person.items():
    print(key, value) # this way we get both keys and values printed out

-Using For Loop in set

# syntax
for iterator in st:
    code goes here

Example:

it_companies = {'Facebook', 'Google', 'Microsoft', 'Apple', 'IBM', 'Oracle', 'Amazon'}
for company in it_companies:
    print(company)

Break and Continue - Part 2

Short reminder: Break: We use break when we want to stop our loop before it is completed.

# syntax
for iterator in sequence:
    code goes here
    if condition:
        break

Example:

numbers = (0,1,2,3,4,5)
for number in numbers:
    print(number)
    if number == 3:
        break

In the above example, the loop stops when it reaches 3.

Continue: We use continue when we want to skip some of the steps in the iteration of the loop.

  # syntax
for iterator in sequence:
    code goes here
    if condition:
        continue

Example:

numbers = (0,1,2,3,4,5)
for number in numbers:
    print(number)
    if number == 3:
        continue
    print('Next number should be ', number + 1) if number != 5 else print("loop's end") # for short hand conditions need both if and else statements
print('outside the loop')

In the example above, if the number equals 3, the step after the condition (but inside the loop) is skipped and the execution of the loop continues if there are any iterations left.

The Range Function

The range() function is used to return a list of numbers. The range(start, end, step) takes three parameters: starting, ending and increment. By default it starts from 0 and the increment is 1. The range sequence needs at least 1 argument (end). Creating sequences using range

lst = list(range(11))
print(lst) # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
st = set(range(1, 11))    # 2 arguments indicate start and end of the sequence, step set to default 1
print(st) # {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
 
lst = list(range(0,11,2))
print(lst) # [0, 2, 4, 6, 8, 10]
st = set(range(0,11,2))
print(st) #  {0, 2, 4, 6, 8, 10}
 
# for backward from start to end 
lst = list(range(11,0,-2))
print(lst) # [11,9,7,5,3,1]

# syntax
for iterator in range(start, end, step):

Example:

for number in range(11):
    print(number)   # prints 0 to 10, not including 11

Nested For Loop

We can write loops inside a loop.

# syntax
for x in y:
    for t in x:
        print(t)

Example:

person = {
    'first_name': 'Asabeneh',
    'last_name': 'Yetayeh',
    'age': 250,
    'country': 'Finland',
    'is_marred': True,
    'skills': ['JavaScript', 'React', 'Node', 'MongoDB', 'Python'],
    'address': {
        'street': 'Space street',
        'zipcode': '02210'
    }
}
for key in person:
    if key == 'skills':
        for skill in person['skills']:
            print(skill)

For Else

If we want to execute some message when the loop ends, we use else.

# syntax
for iterator in range(start, end, step):
    do something
else:
    print('The loop ended')

Example:

for number in range(11):
    print(number)   # prints 0 to 10, not including 11
else:
    print('The loop stops at', number)

Pass

In python when statement is required (after semicolon), but we don’t like to execute any code there, we can write the word pass to avoid errors. Also we can use it as a placeholder, for future statements.

Example:

for number in range(6):
    pass

Functions

What is a function? Before we start making functions, let us learn what a function is and why we need them?

Defining a Function

A function is a reusable block of code or programming statements designed to perform a certain task. To define or declare a function, Python provides the def keyword. The following is the syntax for defining a function. The function block of code is executed only if the function is called or invoked.

Declaring and Calling a Function

When we make a function, we call it declaring a function. When we start using the it, we call it calling or invoking a function. Functions can be declared with or without parameters.

# syntax
# Declaring a function
def function_name():
    codes
    codes
# Calling a function
function_name()

Function without Parameters

Function can be declared without parameters.

Example:

def generate_full_name ():
    first_name = 'Asabeneh'
    last_name = 'Yetayeh'
    space = ' '
    full_name = first_name + space + last_name
    print(full_name)
generate_full_name () # calling a function
 
def add_two_numbers ():
    num_one = 2
    num_two = 3
    total = num_one + num_two
    print(total)
add_two_numbers()

Function Returning a Value - Part 1

Functions return values using the return statement. If a function has no return statement, it returns None. Let us rewrite the above functions using return. From now on, we get a value from a function when we call the function and print it.

def generate_full_name ():
    first_name = 'Asabeneh'
    last_name = 'Yetayeh'
    space = ' '
    full_name = first_name + space + last_name
    return full_name
print(generate_full_name())
 
def add_two_numbers ():
    num_one = 2
    num_two = 3
    total = num_one + num_two
    return total
print(add_two_numbers())

Function with Parameters

In a function we can pass different data types(number, string, boolean, list, tuple, dictionary or set) as parameters.

Single Parameter: If our function takes a parameter we should call our function with an argument

  # syntax
  # Declaring a function
  def function_name(parameter):
    codes
    codes
  # Calling function
  print(function_name(argument))

Example:

def greetings (name):
    message = name + ', welcome to Python for Everyone!'
    return message
 
print(greetings('Asabeneh'))
 
def add_ten(num):
    ten = 10
    return num + ten
print(add_ten(90))
 
def square_number(x):
    return x * x
print(square_number(2))
 
def area_of_circle (r):
    PI = 3.14
    area = PI * r ** 2
    return area
print(area_of_circle(10))
 
def sum_of_numbers(n):
    total = 0
    for i in range(n+1):
        total+=i
    return total
print(sum_of_numbers(10)) # 55
print(sum_of_numbers(100)) # 5050

Two Parameter: A function may or may not have a parameter or parameters. A function may also have two or more parameters. If our function takes parameters we should call it with arguments. Let us check a function with two parameters:

  # syntax
  # Declaring a function
  def function_name(para1, para2):
    codes
    codes
  # Calling function
  print(function_name(arg1, arg2))

Example:

def generate_full_name (first_name, last_name):
    space = ' '
      full_name = first_name + space + last_name
      return full_name
print('Full Name: ', generate_full_name('Asabeneh','Yetayeh'))
 
def sum_two_numbers (num_one, num_two):
    sum = num_one + num_two
    return sum
print('Sum of two numbers: ', sum_two_numbers(1, 9))
 
def calculate_age (current_year, birth_year):
    age = current_year - birth_year
    return age 
 
print('Age: ', calculate_age(2021, 1819))
 
def weight_of_object (mass, gravity):
    weight = str(mass * gravity)+ ' N' # the value has to be changed to a string first
    return weight
print('Weight of an object in Newtons: ', weight_of_object(100, 9.81))

Passing Arguments with Key and Value

If we pass the arguments with key and value, the order of the arguments does not matter.

# syntax
# Declaring a function
def function_name(para1, para2):
    codes
    codes
# Calling function
print(function_name(para1 = 'John', para2 = 'Doe')) # the order of arguments does not matter here

Example:

def print_fullname(firstname, lastname):
    space = ' '
    full_name = firstname  + space + lastname
    print(full_name)
print_fullname(firstname = 'Asabeneh', lastname = 'Yetayeh')
 
def add_two_numbers (num1, num2):
    total = num1 + num2
    return total
print(add_two_numbers(num2 = 3, num1 = 2)) # Order does not matter

Function Returning a Value - Part 2

If we do not return a value with a function, then our function is returning None by default. To return a value with a function we use the keyword return followed by the variable we are returning. We can return any kind of data types from a function.

Returning a string: Example:

def print_name(firstname):
    return firstname
print_name('Asabeneh') # Asabeneh
 
def print_full_name(firstname, lastname):
    space = ' '
    full_name = firstname  + space + lastname
    return full_name
print_full_name(firstname='Asabeneh', lastname='Yetayeh')

Returning a number:

Example:

def add_two_numbers (num1, num2):
    total = num1 + num2
    return total
print(add_two_numbers(2, 3))
 
def calculate_age (current_year, birth_year):
    age = current_year - birth_year
    return age
print('Age: ', calculate_age(2019, 1819))

Returning a boolean: Example:

def is_even (n):
    if n % 2 == 0:
        return True    # return stops further execution of the function, similar to break 
    return False
print(is_even(10)) # True
print(is_even(7)) # False

Returning a list: Example:

def find_even_numbers(n):
    evens = []
    for i in range(n + 1):
        if i % 2 == 0:
            evens.append(i)
    return evens
print(find_even_numbers(10))

Function with Default Parameters

Sometimes we pass default values to parameters, when we invoke the function. If we do not pass arguments when calling the function, their default values will be used.

# syntax
# Declaring a function
def function_name(param = value):
    codes
    codes
# Calling function
function_name()
function_name(arg)

Example:

def greetings (name = 'Peter'):
    message = name + ', welcome to Python for Everyone!'
    return message
print(greetings())
print(greetings('Asabeneh'))
 
def generate_full_name (first_name = 'Asabeneh', last_name = 'Yetayeh'):
    space = ' '
    full_name = first_name + space + last_name
    return full_name
 
print(generate_full_name())
print(generate_full_name('David','Smith'))
 
def calculate_age (birth_year,current_year = 2021):
    age = current_year - birth_year
    return age 
print('Age: ', calculate_age(1821))
 
def weight_of_object (mass, gravity = 9.81):
    weight = str(mass * gravity)+ ' N' # the value has to be changed to string first
    return weight
print('Weight of an object in Newtons: ', weight_of_object(100)) # 9.81 - average gravity on Earth's surface
print('Weight of an object in Newtons: ', weight_of_object(100, 1.62)) # gravity on the surface of the Moon

Arbitrary Number of Arguments

If we do not know the number of arguments we pass to our function, we can create a function which can take arbitrary number of arguments by adding * before the parameter name.

# syntax
# Declaring a function
def function_name(*args):
    codes
    codes
# Calling function
function_name(param1, param2, param3,..)

Example:

def sum_all_nums(*nums):
    total = 0
    for num in nums:
        total += num     # same as total = total + num 
    return total
print(sum_all_nums(2, 3, 5)) # 10

Default and Arbitrary Number of Parameters in Functions

def generate_groups (team,*args):
    print(team)
    for i in args:
        print(i) 
generate_groups('Team-1','Asabeneh','Brook','David','Eyob')

Dictionary unpacking

You can call a function which has named arguments using a dictionary with matching key names. You do so using **.

# Define a function that takes two arguments: 'name' and 'location'
def greet(name, location):
    # Print a greeting message using the provided arguments
    print("Hi there", name, "how is the weather in", location)
 
# Call the function using keyword arguments
greet(name="Alice", location="New York")  
# Output: Hi there Alice how is the weather in New York
 
# Create a dictionary with keys matching the function's parameter names
my_dict = {"name": "Alice", "location": "New York"}
 
# Call the function using dictionary unpacking
greet(**my_dict)  
# The ** operator unpacks the dictionary, passing its key-value pairs 
# as keyword arguments to the function.
# Output: Hi there Alice how is the weather in New York

Arbitrary Number of Named Arguments

You can also define a function to accept an arbitrary number of named arguments.

def arbitrary_named_args(**args):
    print("I received an arbitrary number of arguments, totaling", len(args))
    print("They are provided as a dictionary in my function:", type(args))
    print("Let's print them:")
    for k, v in args.items():
        print(" * key:", k, "value:", v)

Generally avoid this unless required as it makes it harder to understand what the function accepts and does.

Function as a Parameter of Another Function

#You can pass functions around as parameters
def square_number (n):
    return n ** n
def do_something(f, x):
    return f(x)
print(do_something(square_number, 3)) # 27

Modules

What is a Module

A module is a file containing a set of codes or a set of functions which can be included to an application. A module could be a file containing a single variable, a function or a big code base.

Creating a Module

To create a module we write our codes in a python script and we save it as a .py file. Create a file named mymodule.py inside your project folder. Let us write some code in this file.

# mymodule.py file
def generate_full_name(firstname, lastname):
    return firstname + ' ' + lastname

Create main.py file in your project directory and import the mymodule.py file.

Importing a Module

To import the file we use the import keyword and the name of the file only.

# main.py file
import mymodule
print(mymodule.generate_full_name('Asabeneh', 'Yetayeh')) # Asabeneh Yetayeh

Import Functions from a Module

We can have many functions in a file and we can import all the functions differently.

# main.py file
from mymodule import generate_full_name, sum_two_nums, person, gravity
print(generate_full_name('Asabneh','Yetayeh'))
print(sum_two_nums(1,9))
mass = 100
weight = mass * gravity
print(weight)
print(person['firstname'])

Import Functions from a Module and Renaming

During importing we can rename the name of the module.

# main.py file
from mymodule import generate_full_name as fullname, sum_two_nums as total, person as p, gravity as g
print(fullname('Asabneh','Yetayeh'))
print(total(1, 9))
mass = 100 
weight = mass * g
print(weight)
print(p)
print(p['firstname'])

Import Built-in Modules

Like other programming languages we can also import modules by importing the file/function using the key word import. Let’s import the common module we will use most of the time. Some of the common built-in modules: math, datetime, os,sys, random, statistics, collections, json,re

OS Module

Using python os module it is possible to automatically perform many operating system tasks. The OS module in Python provides functions for creating, changing current working directory, and removing a directory (folder), fetching its contents, changing and identifying the current directory.

# import the module
import os
# Creating a directory
os.mkdir('directory_name')
# Changing the current directory
os.chdir('path')
# Getting current working directory
os.getcwd()
# Removing directory
os.rmdir()

Sys Module

The sys module provides functions and variables used to manipulate different parts of the Python runtime environment. Function sys.argv returns a list of command line arguments passed to a Python script. The item at index 0 in this list is always the name of the script, at index 1 is the argument passed from the command line.

Example of a script.py file:

import sys
#print(sys.argv[0], argv[1],sys.argv[2])  # this line would print out: filename argument1 argument2
print('Welcome {}. Enjoy  {} challenge!'.format(sys.argv[1], sys.argv[2]))

Now to check how this script works I wrote in command line:

python script.py Asabeneh 30DaysOfPython

The result:

Welcome Asabeneh. Enjoy  30DayOfPython challenge!

Some useful sys commands:

# to exit sys
sys.exit()
# To know the largest integer variable it takes
sys.maxsize
# To know environment path
sys.path
# To know the version of python you are using
sys.version

Statistics Module

The statistics module provides functions for mathematical statistics of numeric data. The popular statistical functions which are defined in this module: mean, median, mode, stdev etc.

from statistics import * # importing all the statistics modules
ages = [20, 20, 4, 24, 25, 22, 26, 20, 23, 22, 26]
print(mean(ages))       # ~22.9
print(median(ages))     # 23
print(mode(ages))       # 20
print(stdev(ages))      # ~2.3

Math Module

Module containing many mathematical operations and constants.

import math
print(math.pi)           # 3.141592653589793, pi constant
print(math.sqrt(2))      # 1.4142135623730951, square root
print(math.pow(2, 3))    # 8.0, exponential function
print(math.floor(9.81))  # 9, rounding to the lowest
print(math.ceil(9.81))   # 10, rounding to the highest
print(math.log10(100))   # 2, logarithm with 10 as base

Now, we have imported the math module which contains lots of function which can help us to perform mathematical calculations. To check what functions the module has got, we can use help(math), or dir(math). This will display the available functions in the module. If we want to import only a specific function from the module we import it as follows:

from math import pi
print(pi)

It is also possible to import multiple functions at once

 
from math import pi, sqrt, pow, floor, ceil, log10
print(pi)                 # 3.141592653589793
print(sqrt(2))            # 1.4142135623730951
print(pow(2, 3))          # 8.0
print(floor(9.81))        # 9
print(ceil(9.81))         # 10
print(math.log10(100))    # 2

But if we want to import all the function in math module we can use * .

from math import *
print(pi)                  # 3.141592653589793, pi constant
print(sqrt(2))             # 1.4142135623730951, square root
print(pow(2, 3))           # 8.0, exponential
print(floor(9.81))         # 9, rounding to the lowest
print(ceil(9.81))          # 10, rounding to the highest
print(math.log10(100))     # 2

When we import we can also rename the name of the function.

from math import pi as  PI
print(PI) # 3.141592653589793

String Module

A string module is a useful module for many purposes. The example below shows some use of the string module.

import string
print(string.ascii_letters) # abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
print(string.digits)        # 0123456789
print(string.punctuation)   # !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

Random Module

By now you are familiar with importing modules. Let us do one more import to get very familiar with it. Let us import random module which gives us a random number between 0 and 0.9999… The random module has lots of functions but in this section we will only use random and randint.

from random import random, randint
print(random())   # it doesn't take any arguments; it returns a value between 0 and 0.9999
print(randint(5, 20)) # it returns a random integer number between [5, 20] inclusive

List Comprehension

List comprehension in Python is a compact way of creating a list from a sequence. It is a short way to create a new list. List comprehension is considerably faster than processing a list using the for loop.

# syntax
[expression for i in iterable if condition]

Example:1

For instance if you want to change a string to a list of characters. You can use a couple of methods. Let’s see some of them:

# One way
language = 'Python'
lst = list(language) # changing the string to list
print(type(lst))     # list
print(lst)           # ['P', 'y', 't', 'h', 'o', 'n']
 
# Second way: list comprehension
lst = [i for i in language]
print(type(lst)) # list
print(lst)       # ['P', 'y', 't', 'h', 'o', 'n']

Example:2

For instance if you want to generate a list of numbers

# Generating numbers
numbers = [i for i in range(11)]  # to generate numbers from 0 to 10
print(numbers)                    # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 
# It is possible to do mathematical operations during iteration
squares = [i * i for i in range(11)]
print(squares)                    # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
 
# It is also possible to make a list of tuples
numbers = [(i, i * i) for i in range(11)]
print(numbers)                             # [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)]

Example:2

List comprehension can be combined with if expression

# Generating even numbers
even_numbers = [i for i in range(21) if i % 2 == 0]  # to generate even numbers list in range 0 to 21
print(even_numbers)                    # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
 
# Generating odd numbers
odd_numbers = [i for i in range(21) if i % 2 != 0]  # to generate odd numbers in range 0 to 21
print(odd_numbers)                      # [1, 3, 5, 7, 9, 11, 13, 15, 17, 19]
# Filter numbers: let's filter out positive even numbers from the list below
numbers = [-8, -7, -3, -1, 0, 1, 3, 4, 5, 7, 6, 8, 10]
positive_even_numbers = [i for i in numbers if i % 2 == 0 and i > 0]
print(positive_even_numbers)                    # [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
 
# Flattening a two dimensional array
list_of_lists = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flattened_list = [ number for row in list_of_lists for number in row]
print(flattened_list)    # [1, 2, 3, 4, 5, 6, 7, 8, 9]

Lambda Function

Lambda function is a small anonymous function without a name. It can take any number of arguments, but can only have one expression. Lambda function is similar to anonymous functions in JavaScript. We need it when we want to write an anonymous function inside another function.

Creating a Lambda Function

To create a lambda function we use lambda keyword followed by a parameter(s), followed by an expression. See the syntax and the example below. Lambda function does not use return but it explicitly returns the expression.

# syntax
x = lambda param1, param2, param3: param1 + param2 + param3
print(x(arg1, arg2, arg3))

Example:

# Named function
def add_two_nums(a, b):
    return a + b
 
print(add_two_nums(2, 3))     # 5
# Lets change the above function to a lambda function
add_two_nums = lambda a, b: a + b
print(add_two_nums(2,3))    # 5
 
# Self invoking lambda function
(lambda a, b: a + b)(2,3) # 5 - need to encapsulate it in print() to see the result in the console
 
square = lambda x : x ** 2
print(square(3))    # 9
cube = lambda x : x ** 3
print(cube(3))    # 27
 
# Multiple variables
multiple_variable = lambda a, b, c: a ** 2 - 3 * b + 4 * c
print(multiple_variable(5, 5, 3)) # 22

Lambda Function Inside Another Function

Using a lambda function inside another function.

def power(x):
    return lambda n : x ** n
 
cube = power(2)(3)   # function power now need 2 arguments to run, in separate rounded brackets
print(cube)          # 8
two_power_of_five = power(2)(5) 
print(two_power_of_five)  # 32

Higher Order Functions

In Python functions are treated as first class citizens, allowing you to perform the following operations on functions:

A function can take one or more functions as parameters
A function can be returned as a result of another function
A function can be modified
A function can be assigned to a variable

In this section, we will cover:

Handling functions as parameters
Returning functions as return value from another functions
Using Python closures and decorators

Function as a Parameter

def sum_numbers(nums):  # normal function
    return sum(nums)    # a sad function abusing the built-in sum function :<
 
def higher_order_function(f, lst):  # function as a parameter
    summation = f(lst)
    return summation
result = higher_order_function(sum_numbers, [1, 2, 3, 4, 5])
print(result)       # 15

Function as a Return Value

def square(x):          # a square function
    return x ** 2
 
def cube(x):            # a cube function
    return x ** 3
 
def absolute(x):        # an absolute value function
    if x >= 0:
        return x
    else:
        return -(x)
 
def higher_order_function(type): # a higher order function returning a function
    if type == 'square':
        return square
    elif type == 'cube':
        return cube
    elif type == 'absolute':
        return absolute
 
result = higher_order_function('square')
print(result(3))       # 9
result = higher_order_function('cube')
print(result(3))       # 27
result = higher_order_function('absolute')
print(result(-3))      # 3

You can see from the above example that the higher order function is returning different functions depending on the passed parameter

Python Closures

Python allows a nested function to access the outer scope of the enclosing function. This is is known as a Closure. Let us have a look at how closures work in Python. In Python, closure is created by nesting a function inside another encapsulating function and then returning the inner function. See the example below.

Example:

def add_ten():
    ten = 10
    def add(num):
        return num + ten
    return add
 
closure_result = add_ten()
print(closure_result(5))  # 15
print(closure_result(10))  # 20

Python Decorators

A decorator is a design pattern in Python that allows a user to add new functionality to an existing object without modifying its structure. Decorators are usually called before the definition of a function you want to decorate.

Creating Decorators

To create a decorator function, we need an outer function with an inner wrapper function.

Example:

# Normal function
def greeting():
    return 'Welcome to Python'
def uppercase_decorator(function):
    def wrapper():
        func = function()
        make_uppercase = func.upper()
        return make_uppercase
    return wrapper
g = uppercase_decorator(greeting)
print(g())          # WELCOME TO PYTHON
 
## Let us implement the example above with a decorator
 
'''This decorator function is a higher order function
that takes a function as a parameter'''
def uppercase_decorator(function):
    def wrapper():
        func = function()
        make_uppercase = func.upper()
        return make_uppercase
    return wrapper
@uppercase_decorator
def greeting():
    return 'Welcome to Python'
print(greeting())   # WELCOME TO PYTHON

Applying Multiple Decorators to a Single Function

 
'''These decorator functions are higher order functions
that take functions as parameters'''
 
# First Decorator
def uppercase_decorator(function):
    def wrapper():
        func = function()
        make_uppercase = func.upper()
        return make_uppercase
    return wrapper
 
# Second decorator
def split_string_decorator(function):
    def wrapper():
        func = function()
        splitted_string = func.split()
        return splitted_string
    return wrapper
 
#Decorators will be executed from bottom to top
@split_string_decorator
@uppercase_decorator     # order with decorators is important in this case - .upper() function does not work with lists
def greeting():
    return 'Welcome to Python'
print(greeting())   # ['WELCOME', 'TO', 'PYTHON']

Accepting Parameters in Decorator Functions

Most of the time we need our functions to take parameters, so we might need to define a decorator that accepts parameters.

def decorator_with_parameters(function):
    def wrapper_accepting_parameters(para1, para2, para3):
        function(para1, para2, para3)
        print("I live in {}".format(para3))
    return wrapper_accepting_parameters
 
@decorator_with_parameters
def print_full_name(first_name, last_name, country):
    print("I am {} {}. I love to teach.".format(
        first_name, last_name))
 
print_full_name("Asabeneh", "Yetayeh",'Finland')

Built-in Higher Order Functions

Some of the built-in higher order functions that we cover in this part are map(), filter, and reduce. Lambda function can be passed as a parameter and the best use case of lambda functions is in functions like map, filter and reduce.

Python - Map Function

The map() function is a built-in function that takes a function and iterable as parameters.

    # syntax
    map(function, iterable)

Example:1

numbers = [1, 2, 3, 4, 5] # iterable
def square(x):
    return x ** 2
numbers_squared = map(square, numbers)
print(list(numbers_squared))    # [1, 4, 9, 16, 25]
# Lets apply it with a lambda function
numbers_squared = map(lambda x : x ** 2, numbers)
print(list(numbers_squared))    # [1, 4, 9, 16, 25]

Example:2

numbers_str = ['1', '2', '3', '4', '5']  # iterable
numbers_int = map(int, numbers_str)
print(list(numbers_int))    # [1, 2, 3, 4, 5]

Example:3

names = ['Asabeneh', 'Lidiya', 'Ermias', 'Abraham']  # iterable
 
def change_to_upper(name):
    return name.upper()
 
names_upper_cased = map(change_to_upper, names)
print(list(names_upper_cased))    # ['ASABENEH', 'LIDIYA', 'ERMIAS', 'ABRAHAM']
 
# Let us apply it with a lambda function
names_upper_cased = map(lambda name: name.upper(), names)
print(list(names_upper_cased))    # ['ASABENEH', 'LIDIYA', 'ERMIAS', 'ABRAHAM']

What actually map does is iterating over a list. For instance, it changes the names to upper case and returns a new list.

Python - Filter Function

The filter() function calls the specified function which returns boolean for each item of the specified iterable (list). It filters the items that satisfy the filtering criteria.

    # syntax
    filter(function, iterable)

Example:1

# Lets filter only even nubers
numbers = [1, 2, 3, 4, 5]  # iterable
 
def is_even(num):
    if num % 2 == 0:
        return True
    return False
 
even_numbers = filter(is_even, numbers)
print(list(even_numbers))       # [2, 4]

Example:2

numbers = [1, 2, 3, 4, 5]  # iterable
 
def is_odd(num):
    if num % 2 != 0:
        return True
    return False
 
odd_numbers = filter(is_odd, numbers)
print(list(odd_numbers))       # [1, 3, 5]

# Filter long name
names = ['Asabeneh', 'Lidiya', 'Ermias', 'Abraham']  # iterable
def is_name_long(name):
    if len(name) > 7:
        return True
    return False
 
long_names = filter(is_name_long, names)
print(list(long_names))         # ['Asabeneh']

Python - Reduce Function

The reduce() function is defined in the functools module and we should import it from this module. Like map and filter it takes two parameters, a function and an iterable. However, it does not return another iterable, instead it returns a single value. Example:1

numbers_str = ['1', '2', '3', '4', '5']  # iterable
def add_two_nums(x, y):
    return int(x) + int(y)
 
total = reduce(add_two_nums, numbers_str)
print(total)    # 15

Python Error Types

When we write code it is common that we make a typo or some other common error. If our code fails to run, the Python interpreter will display a message, containing feedback with information on where the problem occurs and the type of an error. It will also sometimes gives us suggestions on a possible fix. Understanding different types of errors in programming languages will help us to debug our code quickly and also it makes us better at what we do.

Let us see the most common error types one by one. First let us open our Python interactive shell. Go to your you computer terminal and write ‘python’. The python interactive shell will be opened.

SyntaxError

Example 1: SyntaxError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'hello world'
  File "<stdin>", line 1
    print 'hello world'
                      ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print('hello world')?
>>>

As you can see we made a syntax error because we forgot to enclose the string with parenthesis and Python already suggests the solution. Let us fix it.

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'hello world'
  File "<stdin>", line 1
    print 'hello world'
                      ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print('hello world')?
>>> print('hello world')
hello world
>>>

The error was a SyntaxError. After the fix our code was executed without a hitch. Let see more error types.

NameError

Example 1: NameError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print(age)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'age' is not defined
>>>

As you can see from the message above, name age is not defined. Yes, it is true that we did not define an age variable but we were trying to print it out as if we had had declared it. Now, lets fix this by declaring it and assigning with a value.

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print(age)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'age' is not defined
>>> age = 25
>>> print(age)
25
>>>

The type of error was a NameError. We debugged the error by defining the variable name.

IndexError

Example 1: IndexError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> numbers = [1, 2, 3, 4, 5]
>>> numbers[5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>>

In the example above, Python raised an IndexError, because the list has only indexes from 0 to 4 , so it was out of range.

ModuleNotFoundError

Example 1: ModuleNotFoundError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import maths
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'maths'
>>>

In the example above, I added an extra s to math deliberately and ModuleNotFoundError was raised. Lets fix it by removing the extra s from math.

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import maths
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'maths'
>>> import math
>>>

We fixed it, so let’s use some of the functions from the math module.

AttributeError

Example 1: AttributeError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import maths
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'maths'
>>> import math
>>> math.PI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'math' has no attribute 'PI'
>>>

As you can see, I made a mistake again! Instead of pi, I tried to call a PI constant from maths module. It raised an attribute error, it means, that the attribute does not exist in the module. Lets fix it by changing from PI to pi.

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import maths
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'maths'
>>> import math
>>> math.PI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'math' has no attribute 'PI'
>>> math.pi
3.141592653589793
>>>

Now, when we call pi from the math module we got the result.

KeyError

Example 1: KeyError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> users = {'name':'Asab', 'age':250, 'country':'Finland'}
>>> users['name']
'Asab'
>>> users['county']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'county'
>>>

As you can see, there was a typo in the key used to get the dictionary value. so, this is a key error and the fix is quite straight forward. Let’s do this!

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> user = {'name':'Asab', 'age':250, 'country':'Finland'}
>>> user['name']
'Asab'
>>> user['county']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'county'
>>> user['country']
'Finland'
>>>

We debugged the error, our code ran and we got the value.

TypeError

Example 1: TypeError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 4 + '3'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>>

In the example above, a TypeError is raised because we cannot add a number to a string. First solution would be to convert the string to int or float. Another solution would be converting the number to a string (the result then would be ‘43’). Let us follow the first fix.

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 4 + '3'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>> 4 + int('3')
7
>>> 4 + float('3')
7.0
>>>

Error removed and we got the result we expected.

ImportError

Example 1: TypeError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from math import power
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'power' from 'math'
>>>

There is no function called power in the math module, it goes with a different name: pow. Let’s correct it:

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from math import power
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'power' from 'math'
>>> from math import pow
>>> pow(2,3)
8.0
>>>

ValueError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> int('12a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '12a'
>>>

In this case we cannot change the given string to a number, because of the ‘a’ letter in it.

ZeroDivisionError

asabeneh@Asabeneh:~$ python
Python 3.9.6 (default, Jun 28 2021, 15:26:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero
>>>

We cannot divide a number by zero.

We have covered some of the python error types, if you want to check more about it check the python documentation about python error types. If you are good at reading the error types, then you will be able to fix your bugs fast and you will also become a better programmer.

Exception Handling

Python uses try and except to handle errors gracefully. A graceful exit (or graceful handling) of errors is a simple programming idiom - a program detects a serious error condition and “exits gracefully”, in a controlled manner as a result. Often the program prints a descriptive error message to a terminal or log as part of the graceful exit, this makes our application more robust. The cause of an exception is often external to the program itself. An example of exceptions could be an incorrect input, wrong file name, unable to find a file, a malfunctioning IO device. Graceful handling of errors prevents our applications from crashing.

We have covered the different Python error types in the previous section. If we use try and except in our program, then it will not raise errors in those blocks.

Backend MOC

2026-09-02T05:29:25.019Z

🗺️ Backend Map of Content

Welcome to the syllabus and index for Backend.

📄 Notes

DSA

2026-09-02T05:29:25.019Z

DSA Revision Cheat Sheet

1. Time Complexity

Common Complexities

Complexity	Typical Algorithm
O(1)	Hash lookup
O(log n)	Binary Search
O(n)	Single traversal
O(n log n)	Sorting, Heap
O(n²)	Two nested loops
O(n³)	Floyd Warshall
O(2ⁿ)	Subset Generation
O(n!)	Permutations

Input Size Guide

n	Expected Complexity
≤ 10	O(n!)
≤ 20	O(2ⁿ)
≤ 500	O(n³)
≤ 5000	O(n²)
≤ 10⁶	O(n log n) / O(n)

2. Arrays

Sliding Window

Use when:

Subarray
Contiguous segment
Longest/Shortest window

Template:

int l = 0;
for(int r = 0; r < n; r++) {
    // add nums[r]
 
    while(condition) {
        // remove nums[l]
        l++;
    }
}

Prefix Sum

prefix[i] = prefix[i-1] + arr[i];
sum(l,r) = prefix[r] - prefix[l-1];

Kadane’s Algorithm

Maximum Subarray Sum

int best = 0;
int cur = 0;
 
for(int x : arr){
    cur = max(x, cur + x);
    best = max(best, cur);
}

Complexity: O(n)

3. Sorting

sort(v.begin(), v.end());

Descending:

sort(v.rbegin(), v.rend());

Custom Comparator:

sort(v.begin(), v.end(),
[](auto a, auto b){
    return a.second < b.second;
});

Complexity: O(n log n)

4. Binary Search

Search Space Pattern

while(l <= r){
    int mid = l + (r-l)/2;
 
    if(condition(mid))
        r = mid - 1;
    else
        l = mid + 1;
}

Recognize:

Minimize maximum
Maximize minimum
Search answer

Complexity: O(log n)

5. Hashing

Unordered Map

unordered_map<int,int> mp;
mp[x]++;

Unordered Set

unordered_set<int> st;

Average Complexity:

Insert O(1)
Search O(1)

6. Linked List

Must Know:

Reverse Linked List

ListNode* prev = NULL;
 
while(head){
    ListNode* nxt = head->next;
    head->next = prev;
    prev = head;
    head = nxt;
}

Fast Slow Pointer

Applications:

Middle Node
Cycle Detection

7. Stack

Applications:

Monotonic Stack

Used For:

Next Greater Element
Largest Rectangle Histogram

stack<int> st;

Complexity: O(n)

8. Queue / Deque

BFS

queue<int> q;
q.push(src);

Sliding Window Maximum

deque<int> dq;

Complexity: O(n)

9. Heap

Max Heap

priority_queue<int> pq;

Min Heap

priority_queue<int,
vector<int>,
greater<int>> pq;

Applications:

K Largest
K Smallest
Dijkstra

Complexity:

Push O(log n)
Pop O(log n)

10. Trees

DFS

void dfs(int node){
    for(auto child : adj[node])
        dfs(child);
}

BFS

queue<int> q;

Must Know:

Height
Diameter
LCA concept
BST properties

11. Graphs

Representation

vector<int> adj[n];

DFS

dfs(node);

BFS

queue<int> q;

Applications:

Connected Components
Shortest Path (Unweighted)
Cycle Detection

12. Topological Sort

Kahn Algorithm

queue<int> q;

Condition:

Directed Acyclic Graph

Complexity:
O(V + E)

13. Union Find (DSU)

int find(int x){
    if(parent[x]==x)
        return x;
    return parent[x]=find(parent[x]);
}

Operations:

Find
Union

Complexity:
Almost O(1)

Applications:

Kruskal
Connected Components

14. Dijkstra

For positive weights

priority_queue<
pair<int,int>,
vector<pair<int,int>>,
greater<pair<int,int>>
> pq;

Complexity:

O((V+E) log V)

15. Dynamic Programming

DP Thinking

State
Transition
Base Case
Memoization / Tabulation

Fibonacci

dp[i] = dp[i-1] + dp[i-2];

Knapsack

dp[i][w]

LIS

O(n log n) preferred

House Robber

Classic 1D DP

16. Backtracking

Template

void solve(){
    if(base){
        ans.push_back(path);
        return;
    }
 
    for(choice){
        choose;
        solve();
        unchoose;
    }
}

Applications:

Subsets
Permutations
Combination Sum

17. Bit Manipulation

Set Bit

n |= (1 << i);

Clear Bit

n &= ~(1 << i);

Toggle Bit

n ^= (1 << i);

Check Bit

(n >> i) & 1

Count Bits

__builtin_popcount(n);

18. Math

GCD

gcd(a,b)

Complexity:
O(log n)

LCM

a / gcd(a,b) * b

Fast Power

while(n){
    if(n&1) ans*=x;
    x*=x;
    n>>=1;
}

Complexity:
O(log n)

Modular Arithmetic

const int MOD = 1e9+7;

Rules:

(a+b)%MOD
(a*b)%MOD
(a-b+MOD)%MOD

19. STL You Must Know

vector
pair
set
unordered_set
map
unordered_map
stack
queue
deque
priority_queue
sort()
lower_bound()
upper_bound()

20. Placement Checklist

Arrays & Strings ✓
Binary Search ✓
Hashing ✓
Linked List ✓
Stack ✓
Queue ✓
Heap ✓
Trees ✓
Graphs BFS/DFS ✓
Topological Sort ✓
DSU ✓
Dijkstra ✓
Dynamic Programming ✓
Backtracking ✓
Bit Manipulation ✓
Math + GCD ✓

If you can solve medium-level questions from all the above topics, you are ready for most college placement coding rounds.

Dsa MOC

2026-09-02T05:29:25.019Z

🗺️ Dsa Map of Content

Welcome to the syllabus and index for Dsa.

📄 Notes

Ansible

2026-09-02T05:29:25.019Z

Ansible is an open source, command-line IT automation software application written in Python. It can configure systems, deploy software, and orchestrate advanced workflows to support application deployment, system updates, and more. Ansible provides open-source automation that reduces complexity and runs everywhere. Using Ansible lets you automate virtually any task. Here are some common use cases for Ansible:

Eliminate repetition and simplify workflows
Manage and maintain system configuration
Continuously deploy complex software
Perform zero-downtime rolling updates

Control node

A system on which Ansible is installed. You run Ansible commands such as ansible or ansible-inventory on a control node.

Inventory

A list of managed nodes that are logically organized. You create an inventory on the control node to describe host deployments to Ansible.

Managed node

A remote system, or host, that Ansible controls.

Ansible uses playbooks to automate tasks, you describe the desired state of the system in the playbook and ansible makes sure that the system remains in that state.

→ As automation technology, Ansible is designed around the following principles:

1. Agent-less architecture

Low maintenance overhead by avoiding the installation of additional software across IT infrastructure.

2. Simplicity

Automation playbooks use straightforward YAML syntax for code that reads like documentation. Ansible is also decentralized, using SSH with existing OS credentials to access remote machines.

3. Scalability and flexibility

Easily and quickly scale the systems you automate through a modular design that supports a large range of operating systems, cloud platforms, and network devices.

4. Idempotence and predictability

When the system is in the state your playbook describes, Ansible does not change anything, even if the playbook runs multiple times.

Docker

2026-09-02T05:29:25.019Z

Introduction

The issue: It works on my machine
The solution: Containerize it
Basically Docker is a tool which helps us to make containers, and these containers run applications inside them.
Docker makes a OS independent environment and makes the system compatible for the libraries that are required

How did Docker came?

There was a company names DotCloud which started facing the classic problem of “it works on my machine”, so they decided to make a tool for building containers and they made it available in pycon in 2013 and in 2017 it became a CNCF tool.

Virtualization vs Containerization

Virtual Machines are used.	Containerization tool like Docker, Podman, Containerd are used.
uses Hypervisor: allocates resources from the system and runs the os on it. Vistual machines has there own operating system.	Docker engine uses the host operating system, do not require there own operating system. The system calls are made via the hosto operating system only.
Virtualization is heavy.	Containers are light weight.

Docker Architecture

Docker Engine: Application container engine, which runs all the applications.
Docker Daemon: Background service to manage all the running applications. Dockerd runs containerd internally
Docker CLI: Interface to interact.
Docker Client: Connects APIs and communicates to docker engine and gives as a GUI view of our containers.

Docker Images and Containers

docker ps (to see the running continers/processes)
docker images (to see the available docker images)

A Docker image is a read-only blueprint that contains the source code, libraries, dependencies, and runtime required to run an application, while a Docker container is the live, runnable instance created from that image. A docker image is written using a Dockerfile.

Docker file has instructions → these instructions create image → this image can be replicated in the form of containers.

Dockerfile → image → container (build) (run)

docker pull hello-world

The docker pull command is used to pull a pre-defined image from the docker hub.

docker run hello-world

This command will run the container for the image.

Adding env to the docker containers

docker run -e MYSQL_ROOT_PASSWORD=root mysql

“Use -d to run the container in the background”

docker stop {container id}
docker start {container id}

How to write a Dockerfile?

FROM {base_image_name:image_version}           #pulls base image

WORKDIR /app                                   #directory in the container                                                       where the application code will                                                  reside

COPY . .                                       #copy code from the current                                                       location to container's workdir

RUN apt-get update && apt-get install -y curl  #executes commands at image                                                       build time

CMD ["python","app.py"]                        #specifies the default command                                                    executed at container runtime

ENTRYPOINT                                     #similar to CMD but it cannot be                                                  overridden

CMD can be override at the time of docker run command, but ENTRYPOINT cannot be overridden, both of these commands are used to inject commands in the running containers.

How to make a docker image from a Dockerfile?

docker build -t test-app .

-t → gives a tag or name to the docker image . → give the context of the Dockerfile (location of the Dockerfile to be used)

docker run test-app

This will run the docker image as a container
Everytime there is a change in the source code the Docker image should be updates using “docker build” command.

→ Port mapping

docker run -p 80:80 test-app

Host Port : Container Port
-p : publish

→ How to get inside a docker container

docker exec -it {container id} bash

-it : interactive terminal
bash : shell name

Docker Networking

Imagine your computer runs multiple docker containers and both the containers are isolated in nature, but if we want both of these containers to communicate we need a network for that.

→ Types of Docker networks

Host (docker m/c and host m/c ‘s networks are same)
Bridge (bridges the network between host and the container, it is the default network for docker)
User defined Bridge (custom bridge)
None (completely isolated container)
MACVLAN (network in which communication happens via 2 MAC addresses, used in docker swarm)
IPVLAN (used in docker swarm)
Overlay (used in docker swarm)

docker network ls

Shows all the available docker networks.

docker network create mynetwork -d bridge

-d : drivers
creates a bridge type network

→ How to give network to the containers?

	docker run -d --name mysql --network two-tier -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=devops mysql:latest

To get information about the network:

docker network inspect two-tier

Docker Volumes and Storage

When we delete our docker containers the data which was not persistent in nature will we erased. For saving our data after a crash we bind our data with our host machine which are called volumes.

docker volume ls

Shows all the docker volumes that are currently available or are in existence.

docker volume create mysql-data

A docker volume can be inspected using the “docker inspect {volume name}”
binding of the container with the host path of the volume:

docker run -d --name mysql --network two-tier -v mysql-data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=devops mysql:latest

Docker Compose

Docker compose is a config file that is a yaml/yml file.

key: value We can make multiple docker containers in a docker compose file.

version: "3.8"

services:
  mysql:
    container_name: mysql
    image: mysql
    environment:
      MYSQL_DATABASE: "devops"
      MYSQL_ROOT_PASSWORD: "root"
    ports:
      - "3306:3306"
    volumes:
      - mysql-data:/var/lib/mysql
    networks:
      - two-tier
    restart: always #no, unless-stopped, on-failure
    healthcheck:
      test: ["CMD", "mysqladmin","ping","-h","localhost","-uroot","-proot"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 60s
  flask:
    build:
      context: .
    container_name: two-tier-backend
    ports:
      - "5000:5000"
    environment:
      MYSQL_HOST: mysql
      MYSQL_USER: root
      MYSQL_PASSWORD: root
	  MYSQL_DB: devops
	networks:
	  - two-tier
	depends_on:
	  - mysql
volumes:
  - mysql-data
networks:
  - two-tier

version: syntax or latest version of docker compose.
Inside services block we mention the names of containers we want to build.
Flask or the other backend images are not made they are build using the context (location of the docker file).
depends_on: This keyword says that the container is dependent on the other container, so the other container should be made first.
healthcheck block: This block makes sure that the container is fully up and working, until this health check is not passed the container is not considered full up.
restart: Docker Compose supports four core restart policies to control how containers behave when they stop, crash, or when the system reboots. Restart policies work as per the results of the healthckecks.
- "no": The default behavior. Docker will never automatically restart the container.
- always: Always restarts the container regardless of why it stopped or its exit code.
- unless-stopped: Similar to always, but respects manual intervention. If you intentionally run docker compose stop, the container will not restart automatically on system or daemon reboots.
- on-failure: Restarts the container only if it exits with a non-zero exit code (indicating an application error or crash). You can optionally specify maximum attempts using on-failure:5

docker compose up

This command is used to build and run the docker compose file.

docker compose up -d

This command is used to build and run docker compose in the background.

docker compose down

Stops the docker containers made using docker compose.

docker compose up -d --build

Forcefully pulls and builds all the docker images again.

docker system prune

Deletes all the unused or stopped containers.

 docker attach {image_id}

attached the containers or container logs to the screen.

Docker Registry

A centralized storage and distribution system used to manage and share Docker images.

Step 1: Docker login Login to docker using the cli.
```
docker login
```
Step 2: Tag the image

docker image tag {old_name_of_the_image}:{version_of_the_image}{docker_username}/{new_image_name}:{version}

Step 3: Docker push Pushes the docker image to the dockerhub
```
docker push {image_tagged_name}:{version}
```
Now we do not need to build the image using the build context in the docker compose we can directly use the “image:” and give it the name of our pushed image.
In windows the Docker desktop holds a simulation of linux, which means that if we want to run a linux based docker image on windows we need to install docker desktop on top priority.

Multi-stage Docker Builds

“Docker image optimization” When we use base images like python:3.7, which are big images which turn out the container to be also big which is a concerning thing in terms of optimization as it takes more time in everything (container building, etc).

#stage 1: base image approx of 994 mb ~ 1.40 gb (total size of image)
FROM python:3.7 AS builder

WORKDIR: /app

COPY requirements.txt .

RUN pip install -r requirements.txt

#stage 2 small base image around 400 mb
FROM python:3.7-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.7/site-packages /usr/local/lib/python3.7/site-packages

COPY . .
ENTRYPOINT ["python","run.py"]

Monitoring and logging in Docker

docker logs {container_id}

Used to see the logs of a particular container.

nohup docker attach {container_id} &

nohup runs a command in the background and sends the data or logs to a file.
It will make a nohup.out named file which will store all the data or logs of the particular contianer.

docker scout and docker init

Base image → new image
The base image may contain some security related issues or vulnerabilities so we need to scan the image for which we use docker scout
docker scout is by default installed in the docker desktop but we need to use plugins to install docker scout in linux.
docker init is a command-line utility provided by Docker Desktop that automatically initializes and generates the necessary configuration boilerplate files to containerize your application.

Index for backend

2026-09-02T05:29:24.996Z

DBMS, Advance DB concepts,APIs and its types,Auth,Building for Scale,Django, FastAPI,Postgres,Python,RabbitMQ,Redis,Search Engines,Testing,Web Security

Genai MOC

2026-09-02T05:29:24.995Z

🗺️ Genai Map of Content

Welcome to the syllabus and index for Genai.

📄 Notes

image-generation

image-generation

2026-09-02T05:29:24.995Z

AI Marketing Image Pipeline — Complete Knowledge Document

Everything you need to understand, explain, and defend this project. Written in plain language with technical depth. Use this to prepare for presentations, peer questions, and viva.

The Problem — Why AI Cannot Generate Text
How a Normal Diffusion Model Works — The Full Story
Our Solution — The Core Idea
The 7 Stage Pipeline — Every Stage Explained
Glyph Injection — The Heart of Everything
LLM Orchestration — How Mistral Drives the System
LoRA — Brand Consistency
Complete Tech Stack with Versions
Dataset and Training Plan
Evaluation Metrics — How We Prove It Works
Production Infrastructure
Questions Peers Will Ask — With Answers

1. The Problem

Why does every AI image tool produce broken text?

When you use Midjourney, DALL-E, Stable Diffusion, or even Flux directly and ask it to write “LAKME DIWALI SALE 50% OFF” on an image, you get something like “LAKMÉ DIWALL SAIE 5O% 0FF.” The letters are wrong, numbers become symbols, spacing breaks.

This is not a bug. It is a fundamental architectural limitation with three separate root causes.

Every diffusion model uses a text encoder to convert your prompt string into numbers the model can understand. Flux uses two encoders. The primary one is CLIP (Contrastive Language Image Pretraining).

CLIP was trained on billions of image-text pairs like this:-

Image of a dog running  →  "a dog running in the park"
Image of a city at night  →  "busy city street at night"
Image of a Lakme product  →  "Lakme cosmetics, beauty, makeup"

CLIP learned to match the meaning of a sentence to the concept in an image. It is extremely good at semantic understanding.

But CLIP was never trained to understand what individual letters look like as visual strokes. For CLIP:-

“LAKME” and “lakme” and “Lakme” are all the same thing — a cosmetics brand concept.
The shapes L, A, K, M, E as individual drawn strokes mean nothing to it.
It has no idea that “L” is a vertical line with a horizontal foot, or that “A” is two diagonal lines with a crossbar.

CLIP sees meaning. It does not see letterforms.

Root Cause 2 — Semantic Drift During Denoising

Image generation runs through 20 to 50 denoising steps. In each step the model makes small adjustments to the latent representation.

When the model tries to render the word “GOLD”, its text embedding says:-

Semantic meaning of GOLD:- shiny, metallic, precious, luxury, yellow

The model does not think “G then O then L then D, four specific shapes.” It thinks “golden shimmer, metallic texture, luxury feel.” Over 50 denoising steps, the semantic concept of gold keeps influencing the generation more and more, while the exact letter shapes drift further and further from correct forms.

By step 50, “GOLD” might have become “G0LD” or “GOELD” or something that looks vaguely gold-coloured but is not readable text.

This is Semantic Drift — the concept overrides the glyph.

Root Cause 3 — VAE Destroys Fine Strokes

Diffusion models do not work directly on pixels. Working on a 1024x1024 image means processing over 3 million numbers — computationally impossible at each denoising step.

Instead the model compresses the image into a Latent Space using a VAE (Variational Autoencoder).

1024x1024 image  →  VAE Encoder  →  128x128 latent  (8x compression)
128x128 latent   →  VAE Decoder  →  1024x1024 image

The VAE was trained on natural images — faces, landscapes, objects, scenes. Text was rare in this training data. So the VAE never learned to preserve the fine details that text requires:-

Thin vertical strokes of letters like I, L, 1
Tight counter shapes inside O, P, B
Serifs — the tiny feet at the ends of strokes
Exact spacing between characters

When a letter “L” goes into the latent space, its thin vertical stroke loses exact geometry. When the VAE decoder reconstructs it, it makes its best guess — which might be “I” or “1” or just a slightly wrong “L.”

The VAE compression destroys exactly the kind of fine detail that text requires.

The Combined Result

All three problems happen simultaneously:-

CLIP does not know what letters look like
Denoising drifts the concept away from exact letterforms
VAE compression destroys fine strokes in reconstruction

This is why Midjourney, DALL-E, Stable Diffusion, and even Flux 1 dev with a direct prompt all produce broken text. It is not a model quality issue. It is a structural issue with how these models were designed and trained.

2. How Diffusion Works — The Full Story

Understanding this is critical because our solution only makes sense once you understand what we are intervening in and exactly where.

Step 1 — Text Encoding

User types:- "luxury Diwali poster, LAKME SALE 50% OFF, gold red theme"

This string passes through two encoders in Flux:-

CLIP-L (400M parameters) — converts the whole sentence into a 768-dimensional semantic vector. Fast, captures overall meaning and style concepts.

T5-XXL (11B parameters) — a large language model encoder that captures more detailed linguistic structure. Better at multi-word concepts and longer prompts. Still not character-aware.

Both outputs are combined into a text embedding — a large numerical vector that represents everything the model knows about your prompt.

"luxury Diwali poster LAKME SALE 50% OFF"
                    ↓
            CLIP + T5-XXL
                    ↓
    [0.23, -0.87, 0.45, 1.23, ...]  ←  4096 numbers

This embedding is what guides the entire generation process.

Step 2 — Latent Space Initialization

The model does not start from a blank canvas. It starts from pure random noise in latent space.

Think of it like a TV with no signal — pure static. The model’s job is to slowly shape this static into a meaningful image, guided by the text embedding.

Random Noise Latent
[random, random, random ...]   ←  128x128x16 numbers
         ↓
This is where generation starts

The 16-channel VAE in Flux is what makes the latent space. 16 channels instead of the 4 channels in older models like SDXL — this means 4 times more information can be stored in the same latent size. This is why Flux generates more detailed images than older models.

Step 3 — The Denoising Process (Where Everything Happens)

The Diffusion Transformer (DiT) — Flux’s 12 billion parameter core model — runs the denoising.

It has two parts:-

19 Double Stream Blocks — Image tokens and text tokens are processed in separate streams but cross-attend to each other. The image learns from the text, the text learns from the image context.

38 Single Stream Blocks — Image and text tokens are merged and processed together. This is where deep integration of style and content happens.

Each denoising step:-

Step 1:   Pure noise. Nothing visible.
Step 5:   Overall brightness and color palette emerging.
Step 15:  Rough shapes and composition forming.
Step 25:  Objects becoming recognizable. Layout set.
Step 40:  Fine textures and edges appearing.
Step 50:  Final clean latent. Ready for decoding.

At each step the model asks:- “Given my text embedding and my current noisy latent, what small change brings me closer to the target image?”

This is what we intercept with ControlNet — at every single one of these steps.

Step 4 — VAE Decoding

After 50 steps the model has a clean latent — a 128x128x16 grid of numbers that represents a high-quality image in compressed form.

The VAE Decoder converts this back to real pixels:-

Clean Latent (128x128x16)
        ↓
    VAE Decoder
        ↓
Final Image (1024x1024 pixels, RGB)

This is where the second VAE problem occurs — fine text strokes that survived denoising can still be blurred or altered during decoding.

Where VAE Appears in the Pipeline

This confuses many people. VAE appears at two points:-

Point	Role	Problem
Implicit at start	Random noise is initialized in VAE latent space	No problem here
Explicit at end	Decodes clean latent to pixels	Can blur fine text strokes

The denoising (the 50 steps) happens entirely inside the latent space, between these two VAE operations.

3. Our Solution

The Core Insight

Every approach to fixing AI text generation tries to make the model generate text better. Character-aware encoders, text-specific fine-tuning, larger models — they all fight the same architectural problems.

We do not ask AI to generate text at all.

We render the text ourselves with pixel-perfect accuracy using Python’s Pillow library. Then we use ControlNet to force Flux to respect every single stroke of that rendered text throughout all 50 denoising steps.

Flux’s only job is to apply style, color, lighting, texture, and composition. It never touches the text itself.

The old question:- "How do we make AI write text correctly?"
Our question:-     "How do we stop AI from writing text entirely?"

This is not a workaround. It is architecturally superior because:-

Text accuracy becomes 100% guaranteed regardless of what Flux does internally
We can use any font we want — system fonts, client brand fonts, decorative fonts
Text effects (shadow, glow, 3D) can be added via edge manipulation before ControlNet
The solution works for any language — Hindi, Tamil, Arabic — without language-specific training

4. The 7 Stage Pipeline

Every stage is a separate, independent component. If one component needs to be upgraded or replaced, the rest of the pipeline does not change.

Stage 1 — Brief Input

What happens:- User types a plain language description.

Example:- “Diwali poster for Lakme, 50% off all products, gold and red luxury theme, include product launch date 28th October”

No processing yet. This is just a string.

Stage 2 — LLM Decomposer

Model:- Mistral 7B v0.3 or LLaMA 3.1 8B
Runtime:- Ollama (fully local, no API cost)

What happens:- The LLM reads the brief and extracts structured information. It understands intent, tone, hierarchy, and content.

Output — structured JSON schema:-

{
  "background_prompt": "luxury Diwali background, deep red bokeh, gold particles, dark atmospheric",
  "canvas_size": [1024, 1024],
  "style_notes": "high-end cosmetics brand, premium feel, festival mood",
  "texts": [
    {
      "content": "LAKME",
      "x": 200,
      "y": 100,
      "font_size": 160,
      "font_weight": "bold",
      "color_instruction": "bright gold metallic"
    },
    {
      "content": "DIWALI SALE",
      "x": 150,
      "y": 320,
      "font_size": 110,
      "font_weight": "heavy",
      "color_instruction": "clean white"
    },
    {
      "content": "50% OFF",
      "x": 180,
      "y": 520,
      "font_size": 200,
      "font_weight": "black",
      "color_instruction": "gold with red glow effect"
    },
    {
      "content": "28th October 2025",
      "x": 300,
      "y": 820,
      "font_size": 45,
      "font_weight": "regular",
      "color_instruction": "soft gold"
    }
  ]
}

Why LLM for this? Because a user brief is messy, implicit, and contextual. “Luxury feel” is not a pixel value — the LLM knows what luxury means visually and translates it into a concrete prompt. A rule-based parser cannot do this.

Stage 3 — Pillow Glyph Renderer

Library:- Python Pillow 10.x
Input:- JSON schema from Stage 2
Output:- White background, black text PNG at 1024x1024

from PIL import Image, ImageDraw, ImageFont
 
def render_glyph(schema: dict, fonts_dir: str) -> Image:
    w, h = schema["canvas_size"]
    glyph = Image.new("RGB", (w, h), color="white")
    draw = ImageDraw.Draw(glyph)
 
    for block in schema["texts"]:
        weight = block["font_weight"]
        font = ImageFont.truetype(f"{fonts_dir}/Inter-{weight.capitalize()}.ttf",
                                   block["font_size"])
        draw.text(
            xy=(block["x"], block["y"]),
            text=block["content"],
            fill="black",     # always black — color is Flux's job via prompt
            font=font
        )
 
    return glyph

Critical point:- Text color in the glyph is always black. The color_instruction fields from the JSON go into the Flux prompt, not into Pillow. ControlNet enforces the shape and position. Flux applies the color.

Why is spelling guaranteed here? Because Pillow renders directly from a font file. It draws the exact vector outlines of each character — no AI, no prediction, no hallucination. The font file says what “L” looks like and Pillow draws exactly that.

Stage 4 — ControlNet Glyph Injection

Model:- InstantX FLUX.1-dev-Controlnet-Union (1.2B parameters)

This is the most technically important stage.

What ControlNet does:-

Takes the white-background black-text glyph PNG
Converts it into a spatial feature map — a mathematical representation of every stroke, every curve, every counter shape in every letter
Injects this feature map into the Flux DiT at every single denoising step

Normal Flux pipeline:-
Noise + Text Embedding  →  50 DiT Steps  →  Clean Latent  →  VAE Decode  →  Image

Our pipeline:-
Noise + Text Embedding + Glyph Features  →  50 DiT Steps  →  Clean Latent  →  VAE Decode  →  Image
                              ↑
                         ControlNet
                    (enforced at every step)

The controlnet_scale parameter:-

This controls how strictly ControlNet enforces the glyph versus how much freedom Flux has.

Scale	Effect
1.0	Text shapes perfectly locked but image looks flat, lifeless, no style
0.75	Text locked, Flux has full creative freedom for style and color
0.5	Text starts to drift — Flux overrides some stroke positions
0.3	Glyph mostly ignored — back to the original problem

0.75 is the sweet spot — validated by this research paper’s findings and practical testing.

Stage 5 — Flux Image Generator

Model:- Flux.1-dev
Parameters:- 12 billion
Architecture:- Diffusion Transformer with Flow Matching

At this stage Flux runs its 50-step denoising process. At every step:-

ControlNet injects the glyph spatial features
Flux uses these features to constrain where text strokes can be
Flux applies all visual styling freely:- background, lighting, color palette, texture, bokeh, depth of field

The LoRA adapter for client brand consistency is also active here — it shifts Flux’s default aesthetic toward the client’s visual identity.

Stage 6 — Quality Gate

Models:-

CLIP ViT-L/14 — semantic similarity scoring
LAION Aesthetic Predictor v2 — visual quality scoring

Every generated image is automatically scored before it reaches the user.

# CLIP score — does image match prompt?
clip_score = cosine_similarity(clip_encode(image), clip_encode(prompt))
# Target: above 0.25
 
# Aesthetic score — is image visually good?
aesthetic_score = aesthetic_predictor(image)
# Target: above 5.5 out of 10
 
if clip_score < 0.25 or aesthetic_score < 5.5:
    # Regenerate with different seed
    # Maximum 3 attempts
    pass

Why is this necessary? Diffusion models are stochastic — the same prompt with a different random seed can produce very different results. Some seeds produce weak compositions. The quality gate catches these automatically.

Stage 7 — Real-ESRGAN Upscaler

Model:- Real-ESRGAN x4plus (17M parameters)
Input:- 1024x1024 image
Output:- 2048x2048 or 4096x4096 image

Real-ESRGAN does not just resize the image. It is a neural upscaler that was trained to hallucinate fine detail — it adds texture, sharpens edges, and improves fine strokes during the upscaling process.

This is especially important for text — the upscaled image has crisper letter edges than a simple bicubic resize would produce.

5. Glyph Injection — The Heart of Everything

This section goes deeper on the technical mechanism because this is what makes the pipeline genuinely novel.

Why ControlNet Works for Text

ControlNet was originally designed to let users guide image generation with spatial conditions — edge maps, depth maps, pose skeletons. The key insight from the research paper you shared (Peong et al., 2024) is that text edges are a perfect spatial condition for ControlNet.

Text is naturally described by its edges — the Canny edge detector on a rendered glyph gives you a precise representation of every stroke. ControlNet was designed to follow edge conditions faithfully, and it does so better than any model specifically trained for text generation.

From Table 2 in that paper:-

Method	F1 Score	NED
GlyphControl	0.4559	0.8114
TextDiffuser	0.7822	0.9192
ControlNet + Text Edge	0.7942	0.9243

ControlNet with text edges outperforms models specifically trained for text generation — without any additional training.

How the Feature Map Works

When ControlNet processes the glyph PNG, it does not just look at whether a pixel is black or white. It encodes the spatial structure into a dense feature map that captures:-

Stroke direction and angle
Stroke width
Junction points (where strokes meet)
Counter shapes (the enclosed spaces inside O, B, P)
Relative positions of all text elements

This feature map is then added to the DiT’s internal representations at every denoising step via residual connections — it does not override what Flux is doing, it adds a constraint that Flux must satisfy.

Edge Manipulation for Text Effects

One powerful extension (from the same paper) is edge manipulation before passing to ControlNet. You can modify the glyph’s edge image to produce effects:-

Manipulation	Effect
Shift edges down and right by N pixels	3D extrusion effect
Dilate edges outward	Outline / border effect
Flip vertically and fade	Reflection effect
Affine rotate and soften	Shadow at custom angle

These effects are achieved with simple Pillow image processing — not additional AI models. Then the modified edge goes to ControlNet and Flux styles it.

6. LLM Orchestration

Why We Need an LLM

Users do not think in JSON. They think in natural language:- “make a wedding invitation for my cousin, classy look, flowers, mention 15th December, Jaipur.”

A rule-based parser cannot extract layout intent, aesthetic direction, font hierarchy, and design philosophy from this. An LLM can.

What the LLM Actually Does

Task 1 — Intent extraction:- Understand what type of design is needed, what mood, what audience.

Task 2 — Layout decomposition:- Decide which text elements are headlines, which are subheads, which are body copy. Assign appropriate font sizes and positions based on visual hierarchy.

Task 3 — Prompt composition:- Write a detailed Flux prompt that captures the background style, color palette, lighting mood, and all the color_instruction fields from the text blocks.

Task 4 — Schema validation:- Ensure the JSON output is syntactically valid and all required fields are present.

Why Ollama and Local LLM

No API cost per request. No latency from external calls. No data privacy concerns for client briefs. A 7B parameter model running locally on A100 responds in 2 to 5 seconds — fast enough for interactive use.

Mistral 7B is used as default because it follows structured JSON output instructions reliably. LLaMA 3.1 8B is the alternative — slightly better reasoning but slightly slower.

LLM Fine-tuning

The base Mistral model does not naturally produce good poster layouts. It needs to be fine-tuned on examples of:-

{
  "input": "Diwali sale poster, Lakme brand, 50% off, gold theme",
  "output": {
    "background_prompt": "...",
    "texts": [...]
  }
}

500 to 2000 such pairs, synthetically generated using GPT-4 or Claude, then used to fine-tune Mistral with QLoRA. This teaches the model:-

Visual hierarchy — headline bigger than subhead bigger than body
Indian design conventions — festivals, weddings, product launches
Brand-appropriate positioning — luxury brands use more whitespace
Font weight vocabulary — when to use bold vs heavy vs black

7. LoRA — Brand Consistency

What LoRA Is

LoRA (Low Rank Adaptation) is a technique for efficiently fine-tuning large models. Instead of updating all 12 billion parameters of Flux, LoRA adds small adapter matrices at specific layers.

Normal fine-tuning:-  Update 12B parameters  →  12GB of new weights
LoRA fine-tuning:-    Update ~100M adapter params  →  ~100MB file

The small adapter file (called a .safetensors file) can be loaded and unloaded at runtime in milliseconds. One Flux base model, many brand adapters.

Training a Brand LoRA

Data needed:- 15 to 100 images of the brand. Product shots, existing ads, brand guidelines visuals.

Caption each image:-

"Lakme cosmetics product shot, clean white background, gold accents, 
premium cosmetics brand, soft lighting, elegant typography"

Training:- kohya-ss or ai-toolkit, 500 to 1500 steps, learning rate 0.0004, rank 16.

What the LoRA learns:- The specific color palette, lighting style, composition preferences, and visual aesthetic of the brand. When loaded during generation, Flux automatically gravitates toward these aesthetics.

What LoRA does NOT do:- It does not learn to generate text correctly. Text accuracy is entirely handled by glyph injection — LoRA only handles style and brand consistency.

8. Complete Tech Stack

Generation Models

Model	Role	Parameters	Precision
Flux.1-dev	Core image generator	12B	bfloat16
CLIP-L	Semantic text encoder	400M	float16
T5-XXL	Character-level text encoder	11B	float16
Flux native VAE	Latent compression and decode	—	bfloat16
ControlNet Union	Glyph injection	1.2B	float16
IP-Adapter	Visual reference conditioning	100M	float16
Real-ESRGAN x4plus	Neural upscaling	17M	float32

Intelligence Models

Model	Role	Parameters
Mistral 7B v0.3	LLM orchestration (primary)	7B
LLaMA 3.1 8B	LLM orchestration (alternative)	8B
CLIP ViT-L/14	Quality gate scoring	400M
LAION Aesthetic Predictor v2	Aesthetic quality scoring	300M
Florence-2-base-ft	Dataset captioning	270M
EasyOCR 1.7.x	Text extraction from dataset images	—

Infrastructure

Component	Technology	Version	Role
Workflow engine	ComfyUI	latest	Visual pipeline orchestration
API layer	FastAPI	0.111.x	REST endpoints
Job queue	Celery	5.3.x	Async task processing
Job store	Redis	7.x	Queue state and result cache
DL backend	PyTorch	2.3.x	Tensor operations
Model loading	Diffusers	0.27.x	Flux inference
LLM runtime	Ollama	0.3.x	Local model serving
Weight format	safetensors	0.4.x	Model storage
Language	Python	3.11.x	All pipeline code

Hardware Requirements

GPU	VRAM	Speed	Use Case
A100 40GB	40GB	15 to 30 sec per image	Production preferred
RTX 4090	24GB	20 to 40 sec per image	Production viable
RTX 3090	24GB	25 to 45 sec per image	Development
CPU only	24GB RAM	20 to 60 min per image	Dev and testing only

9. Dataset and Training Plan

Why We Need a Dataset

Flux.1-dev is a general-purpose image generator. It does not know:-

What good marketing posters look like
How to handle Indian festival aesthetics
What makes a professional wedding card
How to balance text-heavy layouts visually

We need to train it to understand these things via LoRA adapters.

The 5 Training Stages

Stage	Data Needed	Sweet Spot	What It Trains
LLM Fine-tune	500 to 1000 pairs	1000 pairs	Brief to JSON schema conversion
Base Aesthetic LoRA	1000 to 2000 images	2000 images	General marketing poster aesthetic
Text Rendering LoRA	3000 to 5000 images	5000 images	Poster layouts with correct text
Client Brand LoRA	30 to 50 images	50 images	Specific brand visual identity
Style LoRA (optional)	300 to 500 images	500 images	Specific art direction style

Text Rendering LoRA — Most Critical Stage

This is the highest priority training stage. We need 3000 to 10000 high quality images of:-

Marketing posters
Festival sale banners
Wedding invitations
Product launch graphics
Promotional flyers

Sources being collected:-

Pinterest — scrape via gallery-dl or custom agent, target 3000 to 5000 images
Flickr — custom API scraper, target 4000 to 6000 images
LAION-Art — filter for aesthetic score above 7.5 and design keywords, bulk download via img2dataset
Behance — design portfolio site, high quality commercial work

Caption Pipeline

Every image needs a descriptive text caption for training. Captions come from three sources merged together:-

Florence-2-base-ft generates the visual description:-

"wedding invitation card with floral border and gold typography on cream background"

EasyOCR extracts any text visible in the image:-

text "Save The Date", text "Mr & Mrs Sharma", text "15th December"

Filename tags add design category context:-

wedding_invite, formal, serif_font, floral_motif

Final merged caption:-

"wedding invitation card with floral border and gold typography on cream background, 
formal design, serif font, floral motif, text 'Save The Date', text 'Mr and Mrs Sharma', 
text '15th December'"

These captions teach the LoRA to associate visual styles with descriptive text, which improves prompt following during inference.

Collection Scripts

Three scripts are handling data collection:-

flickr_scraper.py — Uses Flickr API (free key required), 34 search queries, resume support, rate limit handling.

pinterest_scraper.py — Uses Pinterest internal API, 34 queries times 150 images each = 5100 target images.

scrape_and_caption_agent.py — Combined scrape and caption pipeline. Scrapes Pinterest and Google Images simultaneously, extracts page context (title, description, alt text), runs Florence-2 plus OCR, produces ready-to-train image-caption pairs.

Training Configuration (kohya-ss)

[training]
model_name = "flux.1-dev"
train_batch_size = 1
learning_rate = 0.0004
max_train_steps = 3000
lora_rank = 16
lora_alpha = 16
save_every_n_steps = 500
mixed_precision = "bf16"
 
[optimizer]
optimizer_type = "AdamW8bit"
 
[dataset]
resolution = 1024
flip_aug = false

Why 3000 steps? Less than 2000 steps — LoRA does not learn enough. More than 5000 steps — risk of overfitting where the model just copies training images instead of generalizing.

10. Evaluation Metrics

Why These Specific Metrics

The pipeline makes two core claims:-

Text will be accurate
Images will be visually high quality

Each claim needs its own measurement.

Dimension 1 — Text Accuracy

How Evaluation Works

After generating an image, we run OCR on it and compare detected text to the text we put into the Pillow glyph renderer.

import easyocr
from jiwer import cer, wer
import editdistance
 
reader = easyocr.Reader(['en', 'hi'])
 
def evaluate(image_path, expected_text):
    results = reader.readtext(image_path)
    detected = " ".join([r[1] for r in results]).upper().strip()
    expected = expected_text.upper().strip()
 
    cer_score  = cer(expected, detected)
    wer_score  = wer(expected, detected)
    ed         = editdistance.eval(expected, detected)
    ned        = 1 - (ed / max(len(expected), len(detected)))
    exact      = 1 if expected == detected else 0
 
    return {"CER": cer_score, "WER": wer_score, "NED": ned, "exact": exact}

CER — Character Error Rate

How many individual characters are wrong.

Formula:-  CER = edit_distance(expected, detected) / length(expected)

Example:-
Expected  →  "DIWALI"   (6 characters)
Detected  →  "DIWALL"   (substituted I with L at position 6)
Edit distance = 1
CER = 1/6 = 0.167   →  16.7% characters wrong

CER = 0.0   →  perfect
CER = 1.0   →  everything wrong
Target:-    CER below 0.05

WER — Word Error Rate

How many complete words are wrong.

Formula:-  WER = word_edit_distance(expected, detected) / word_count(expected)

Example:-
Expected:-  "LAKME DIWALI SALE 50% OFF"   (5 words)
Detected:-  "LAKME DIWALL SAIE 5O% OFF"   (5 words)

LAKME    DIWALL    SAIE    5O%    OFF  
Wrong words = 3

WER = 3/5 = 0.60   →  60% words wrong (bad)

WER = 0.0   →  perfect
Target:-    WER below 0.10

NED — Normalized Edit Distance

From the research paper. Same as CER but normalized differently — higher is better.

Formula:-  NED = 1 - (edit_distance / max(len(expected), len(detected)))

Example:-
Expected:-  "DIWALI"   length 6
Detected:-  "DIWALL"   length 6
Edit distance = 1

NED = 1 - (1/6) = 0.833

NED = 1.0   →  perfect
NED = 0.0   →  completely wrong
Target:-    NED above 0.95

Note:- The paper this pipeline is based on achieved NED 0.9243 with ControlNet plus text edge.
Our pipeline should achieve 0.95 plus since we are using higher quality glyph rendering.

Exact Match Rate — Headline Metric

Strictest metric. The entire detected string must exactly equal the expected string.

Test on 100 images:-
98 images  →  detected text exactly matches expected
2 images   →  one character wrong each

Exact Match Rate = 98/100 = 98%

Target:-  95% plus

Important distinction:- The pipeline guarantees that Pillow renders correct text and ControlNet enforces it. The 5% gap from 100% in target is due to OCR limitations (small text, unusual fonts, background blending) — not pipeline failure.

Dimension 2 — Visual Quality

CLIP Score

Measures semantic alignment between the generated image and the original prompt. Uses cosine similarity between CLIP embeddings of both.

Score range:-  0.0 to 1.0
Target:-       above 0.25
Meaning:-      Image visually represents what the prompt described

Aesthetic Score (LAION Predictor v2)

Measures how visually pleasing the image is, independent of prompt adherence.

Score range:-  0.0 to 10.0
Target:-       above 5.5
Meaning:-      Image meets baseline quality standard for marketing use

FID — Frechet Inception Distance

Measures how similar the distribution of generated images is to real marketing images.

Lower FID = better
FID below 15  →  acceptable
FID below 10  →  good
FID below 5   →  excellent

Measured by comparing feature distributions using InceptionV3
between 1000 generated images and 1000 real marketing images

Dimension 3 — Brand Consistency

For client-specific LoRA evaluation.

CLIP Brand Similarity:-
Encode 10 client reference images and 10 generated images with CLIP
Measure cosine similarity between distributions
Target:-  above 0.7

Human Evaluation (Likert Scale 1 to 5):-
"Does this image look like it belongs to [brand]?"
Target:-  average above 3.5 out of 5

Dimension 4 — Speed

LLM decomposition:-    2 to 5 seconds
Pillow render:-        under 0.5 seconds
Flux generation:-      15 to 40 seconds (A100)
Quality gate:-         2 to 3 seconds
Upscaling:-            5 to 10 seconds
Total:-                25 to 60 seconds per image

Baseline Comparison

This is important for the presentation. Run the same 100 prompts through:-

Vanilla Flux.1-dev (direct prompt, no ControlNet, no glyph)
Our pipeline (full glyph injection system)

Expected results:-

Metric	Vanilla Flux	Our Pipeline
Exact Match Rate	15 to 25%	95%+
CER	0.40 to 0.60	below 0.05
NED	0.60 to 0.75	above 0.95
CLIP Score	0.28	0.26 to 0.30
Aesthetic Score	6.2	6.0 to 6.5

Text accuracy improves dramatically. Visual quality stays similar (since we are using the same Flux base model for styling).

11. Production Infrastructure

Request Flow

Client HTTP Request (FastAPI)
        ↓
Celery pushes job to Redis queue
        ↓
Worker picks up job
        ↓
LLM Decomposer  →  Pillow Glyph  →  ControlNet + Flux  →  Quality Gate  →  Upscaler
        ↓
Result stored in Redis
        ↓
Client polls or receives webhook
        ↓
Image served from S3 / object storage

Why Celery and Redis

Image generation takes 25 to 60 seconds. HTTP requests cannot wait that long — the connection would time out. Celery handles the job asynchronously. The client gets a job ID immediately, then polls for completion.

Redis stores job state and results. Fast in-memory access. Results expire after 24 hours to manage storage.

Scaling

Each worker handles one image at a time (GPU is single-tenant). To scale:-

Add more RunPod or Lambda workers
Redis queue distributes work automatically
FastAPI layer is stateless — can run multiple instances behind a load balancer

12. Questions Peers Will Ask — With Answers

Q: Your ControlNet approach is not new — GlyphControl does the same thing. What is different?

A: Three differences. First, GlyphControl uses a fixed default font for glyph rendering. We use any font the LLM selects based on brand requirements. Second, GlyphControl achieved F1 of 0.4559 in the comparison study — worse than even vanilla ControlNet with text edges. Our approach uses the same ControlNet plus text edge method that achieved 0.7942. Third, GlyphControl has no LLM orchestration layer — it does not understand briefs, it just takes a glyph. Our system takes plain language input and handles the entire workflow.

Q: Why not just use Ideogram 2.0 or Adobe Firefly? They already do text in images.

A: They do text better than Midjourney or DALL-E, but they still fail on non-English text, complex multi-line layouts, and custom font requirements. More importantly, they are closed commercial APIs with per-image pricing. Our system is open-source, self-hosted, fully customizable, and can be trained on client-specific brand data. For an agency producing thousands of images per month, the cost difference alone is significant.

Q: You claim 100% text accuracy but your target is 95% exact match. Is that a contradiction?

A: No. The pipeline guarantees 100% accuracy in text generation — Pillow renders correct text and ControlNet enforces every stroke. The 5% gap in exact match rate is a measurement artifact from OCR limitations:- very small text, unusual decorative fonts, or text that visually blends with a similarly-colored background can confuse the EasyOCR model even when the text in the image is technically correct. These are measurement tool limitations, not pipeline failures. Human visual inspection of the same images would show higher accuracy.

Q: What happens when the LLM produces bad JSON?

A: Schema validation layer. After the LLM outputs JSON, we validate it against a Pydantic schema before it reaches Pillow. If required fields are missing or positions are out of canvas bounds, we either fix them programmatically (clamp positions to canvas) or re-prompt the LLM with the validation error. In practice, fine-tuned Mistral on good examples rarely produces invalid JSON.

Q: VAE still runs at the end. Does it not destroy the text strokes that ControlNet enforced?

A: This is the right question. The VAE decoder can soften fine strokes, especially for very small text. Two mitigations:- First, Flux’s 16-channel VAE preserves more detail than older 4-channel VAEs. Second, Real-ESRGAN upscaling adds detail back and sharpens edges during upscaling. For text below approximately 40px height in the base 1024px image, quality can degrade — our LLM is instructed to set minimum font sizes above this threshold for this reason.

Q: How much does training cost?

A: Text Rendering LoRA at 3000 steps on A100 on RunPod costs approximately 3 to 5 USD per training run. LLM fine-tuning with QLoRA costs similar. Client Brand LoRA at 1000 steps costs under 1 USD. Total to train the full system from scratch:- approximately 15 to 20 USD on cloud GPU.

Q: Can this work for Hindi and other Indian languages?

A: Yes, and this is one of the strongest use cases. Pillow supports any Unicode font including Devanagari (Hindi), Bengali, Tamil, and Telugu. EasyOCR has Hindi support for evaluation. CLIP and T5-XXL understand Hindi prompts. ControlNet does not care what language the text is — it only cares about stroke shapes. The only requirement is a good quality font file for the target language.

Q: Why use a 12B parameter model? Could a smaller model work?

A: Flux.1-schnell is 12B parameters too but distilled for speed (4 steps instead of 50). The reason we use Flux.1-dev and not a smaller model like SDXL is the 16-channel VAE — it preserves more latent detail which matters for fine text strokes. Smaller models with 4-channel VAEs have more reconstruction error and the ControlNet enforcement is less precise. The 12B parameter count is the cost of having a sufficiently capable VAE and DiT.

Q: What is your test set and how did you make sure it does not overlap with training data?

A: Standard 80-10-10 split on the collected dataset. The test set of 10% is held out before any training begins and is never used for model selection or hyperparameter tuning. For evaluation prompts we also generate synthetic prompts (Indian brand names, festival names, product categories) that did not appear in the training corpus — this tests generalization beyond memorization.

Document version 1.0 — Last updated March 2026
Pipeline by:- [Your Name]
For academic and peer presentation use

AI Marketing Image Pipeline — Complete Scripting Knowledge

Har script kya karti hai, kyun karti hai, aur kaise karti hai. Code level understanding for preparation.

Project Folder Structure
Script 1 — Flickr Scraper
Script 2 — Pinterest Scraper
Script 3 — Smart Caption Generator
Script 4 — Scrape and Caption Agent
Script 5 — Glyph Renderer
Script 6 — LLM Decomposer
Script 7 — ControlNet + Flux Pipeline
Script 8 — Quality Gate
Script 9 — Upscaler
Script 10 — Evaluation Runner
Script 11 — FastAPI Server
How All Scripts Connect — Full Pipeline Runner
Dependencies and Install Commands
Common Errors and Fixes

1. Folder Structure

marketing_pipeline/
│
├── data/
│   ├── raw/                    # scraped images (Pinterest, Flickr)
│   ├── captions/               # generated .txt caption files
│   ├── filtered/               # quality-filtered images
│   └── metadata.jsonl          # final training metadata
│
├── models/
│   ├── loras/                  # trained LoRA .safetensors files
│   │   ├── text_rendering.safetensors
│   │   └── lakme_brand.safetensors
│   └── fonts/                  # font .ttf files for Pillow
│       ├── Inter-Bold.ttf
│       ├── Inter-Heavy.ttf
│       ├── Inter-Regular.ttf
│       └── Inter-Black.ttf
│
├── scripts/
│   ├── scraping/
│   │   ├── flickr_scraper.py
│   │   ├── pinterest_scraper.py
│   │   └── scrape_and_caption_agent.py
│   │
│   ├── captioning/
│   │   └── smart_caption.py
│   │
│   ├── pipeline/
│   │   ├── llm_decomposer.py
│   │   ├── glyph_renderer.py
│   │   ├── flux_generator.py
│   │   ├── quality_gate.py
│   │   └── upscaler.py
│   │
│   ├── evaluation/
│   │   └── evaluate.py
│   │
│   └── api/
│       └── server.py
│
├── pipeline_runner.py          # connects everything
├── requirements.txt
└── config.yaml

2. Flickr Scraper

Kya karta hai

Flickr ka public API use karke design-related images download karta hai. 34 search queries run karta hai — poster design, flyer design, wedding invitation, etc. Rate limiting handle karta hai. Resume support hai matlab agar script beech mein crash ho jaaye to dobara wahi se shuru hoti hai.

Kyun Flickr

Flickr Creative Commons images ke liye best source hai. Images high resolution mein available hain aur licensing clear hai. Pinterest ke compared to Flickr ka API proper aur documented hai.

Core Logic Explained

import requests
import os
import time
import json
from pathlib import Path
 
# Flickr API endpoint
FLICKR_API = "https://api.flickr.com/services/rest/"
 
# Ye saari queries run hongi
QUERIES = [
    "poster design typography",
    "flyer design marketing",
    "wedding invitation card",
    "festival sale banner",
    "product launch poster",
    "Diwali banner design",
    "promotional flyer India",
    "event poster design",
    "banner ad design",
    "graphic design poster",
    # ... aur 24 queries
]
 
def search_flickr(query, api_key, page=1, per_page=100):
    """
    Flickr se images search karo.
    
    params dict mein ye fields important hain:-
    - method: flickr.photos.search — ye Flickr ka search endpoint hai
    - text: hamara query string
    - safe_search: 1 — adult content filter
    - content_type: 1 — sirf photos, illustrations nahi
    - media: photos — sirf images
    - extras: url_l,url_o — large aur original URL bhi do
    - format: json
    """
    params = {
        "method": "flickr.photos.search",
        "api_key": api_key,
        "text": query,
        "safe_search": 1,
        "content_type": 1,
        "media": "photos",
        "extras": "url_l,url_o,views,description",
        "per_page": per_page,
        "page": page,
        "format": "json",
        "nojsoncallback": 1
    }
    
    response = requests.get(FLICKR_API, params=params, timeout=10)
    data = response.json()
    
    # data["photos"]["photo"] mein actual results hain
    return data["photos"]["photo"]
 
def download_image(url, save_path):
    """
    Single image download. Stream=True isliye use karte hain
    kyunki badi images ko memory mein ek saath load nahi karna.
    Chunks mein download hoti hai disk pe.
    """
    if os.path.exists(save_path):
        return True  # already downloaded — skip
    
    try:
        response = requests.get(url, stream=True, timeout=15)
        response.raise_for_status()
        
        with open(save_path, 'wb') as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)
        return True
    except Exception as e:
        print(f"Failed: {url} — {e}")
        return False
 
def scrape_flickr(api_key, output_dir, target_per_query=150):
    """
    Main scraping function.
    
    Progress file save karta hai JSON mein — agar script crash ho
    to dobara same images download nahi hogi. Ye resume support hai.
    """
    Path(output_dir).mkdir(parents=True, exist_ok=True)
    
    # Progress tracking
    progress_file = os.path.join(output_dir, "progress.json")
    progress = json.load(open(progress_file)) if os.path.exists(progress_file) else {}
    
    total_downloaded = 0
    
    for query in QUERIES:
        if progress.get(query, 0) >= target_per_query:
            print(f"Skipping '{query}' — already done")
            continue
        
        downloaded_for_query = progress.get(query, 0)
        page = (downloaded_for_query // 100) + 1
        
        print(f"\nQuery: '{query}'")
        
        while downloaded_for_query < target_per_query:
            photos = search_flickr(query, api_key, page=page)
            
            if not photos:
                break
            
            for photo in photos:
                # url_o = original, url_l = large — original prefer karo
                url = photo.get("url_o") or photo.get("url_l")
                if not url:
                    continue
                
                # filename: query_photoId.jpg — unique naam
                safe_query = query.replace(" ", "_")[:30]
                filename = f"{safe_query}_{photo['id']}.jpg"
                save_path = os.path.join(output_dir, filename)
                
                if download_image(url, save_path):
                    downloaded_for_query += 1
                    total_downloaded += 1
                
                if downloaded_for_query >= target_per_query:
                    break
                
                time.sleep(0.5)  # rate limit — Flickr 3600 requests/hour allow karta hai
            
            page += 1
        
        # Progress save karo
        progress[query] = downloaded_for_query
        json.dump(progress, open(progress_file, 'w'))
    
    print(f"\nTotal downloaded: {total_downloaded}")

Kaise Run Karo

# Install
pip install requests
 
# API key required — free at flickr.com/services/apps/create/apply
python flickr_scraper.py --api_key YOUR_KEY --output data/raw/flickr --target 150

3. Pinterest Scraper

Kya karta hai

Pinterest ka internal API use karke design images scrape karta hai. Pinterest ka koi official public API nahi hai production use ke liye, isliye internal BaseSearchResource endpoint use karta hai jo browser bhi use karta hai.

Important Warning

Pinterest scraping unstable hai — unka API structure bina notice ke change ho sakta hai. Isliye gallery-dl alternative bhi suggest kiya hai.

Core Logic Explained

import requests
import os
import time
import json
from pathlib import Path
 
# Pinterest ka internal search API
PINTEREST_SEARCH_URL = "https://www.pinterest.com/resource/BaseSearchResource/get/"
 
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-US,en;q=0.9",
    "Referer": "https://www.pinterest.com/",
    "X-Requested-With": "XMLHttpRequest",
}
 
QUERIES = [
    "diwali sale poster design",
    "wedding invitation India",
    "festival banner design",
    "marketing flyer design",
    "product launch poster",
    # ... more queries
]
 
def search_pinterest(query, bookmark=None, page_size=25):
    """
    Pinterest search. 
    
    bookmark ye Pinterest ka pagination system hai.
    Pehli request mein bookmark=None dete hain.
    Response mein next bookmark aata hai — use karo next page ke liye.
    Jab bookmark nahi aata matlab aur pages nahi hain.
    """
    data = {
        "source_url": f"/search/pins/?q={query}",
        "data": json.dumps({
            "options": {
                "query": query,
                "scope": "pins",
                "page_size": page_size,
                "bookmarks": [bookmark] if bookmark else [],
                "field_set_key": "unauth_react",
                "filters": "",
                "auto_correction_disabled": False,
            },
            "context": {}
        })
    }
    
    response = requests.get(
        PINTEREST_SEARCH_URL,
        headers=HEADERS,
        params=data,
        timeout=15
    )
    
    result = response.json()
    
    # Results in this path
    resource_response = result.get("resource_response", {})
    pins = resource_response.get("data", {}).get("results", [])
    next_bookmark = resource_response.get("bookmark")
    
    return pins, next_bookmark
 
def extract_image_url(pin):
    """
    Pin object se best quality image URL nikalo.
    Pinterest multiple sizes store karta hai.
    Originals > 736x > 474x preference order.
    """
    images = pin.get("images", {})
    
    # Try highest quality first
    for size in ["orig", "736x", "474x", "236x"]:
        if size in images:
            return images[size].get("url")
    
    return None
 
def scrape_pinterest(output_dir, target_per_query=150):
    Path(output_dir).mkdir(parents=True, exist_ok=True)
    progress_file = os.path.join(output_dir, "progress.json")
    progress = json.load(open(progress_file)) if os.path.exists(progress_file) else {}
    
    for query in QUERIES:
        if progress.get(query, 0) >= target_per_query:
            continue
        
        downloaded = 0
        bookmark = None
        
        print(f"Scraping: {query}")
        
        while downloaded < target_per_query:
            try:
                pins, next_bookmark = search_pinterest(query, bookmark)
            except Exception as e:
                print(f"Error: {e} — retrying in 10s")
                time.sleep(10)
                continue
            
            if not pins:
                break
            
            for pin in pins:
                url = extract_image_url(pin)
                if not url:
                    continue
                
                pin_id = pin.get("id", "unknown")
                safe_q = query.replace(" ", "_")[:25]
                save_path = os.path.join(output_dir, f"{safe_q}_{pin_id}.jpg")
                
                if not os.path.exists(save_path):
                    try:
                        img_data = requests.get(url, timeout=10).content
                        with open(save_path, 'wb') as f:
                            f.write(img_data)
                        downloaded += 1
                    except:
                        pass
                else:
                    downloaded += 1
                
                time.sleep(1.5)  # Pinterest rate limit — slower than Flickr
                
                if downloaded >= target_per_query:
                    break
            
            bookmark = next_bookmark
            if not bookmark:
                break
        
        progress[query] = downloaded
        json.dump(progress, open(progress_file, 'w'))
 
# Alternative — gallery-dl command (more stable)
# gallery-dl --dest data/raw/pinterest --range 1-5000 --sleep 2 \
#   "https://www.pinterest.com/search/pins/?q=graphic+design+poster"

4. Smart Caption Generator

Kya karta hai

Har scraped image ke liye ek detailed text caption generate karta hai. Ye caption LoRA training ke liye use hogi. Script do models use karti hai:-

Florence-2-base-ft — visual description generate karta hai (kya dikhta hai image mein)
EasyOCR — image mein jo text hai use extract karta hai

Dono ko merge karke ek rich caption banta hai.

Kyun Captions Important Hain

LoRA training mein har image ke saath ek text file hoti hai same naam se. Jaise poster_001.jpg ke saath poster_001.txt hogi. Is text file mein caption hoti hai. Training ke time pe model image aur caption ka pair dekhta hai aur seekhta hai ki in visual elements ko in words se describe karte hain.

Agar caption poor quality hai — sirf “a poster” — to model kuch khaas nahi seekhega. Agar caption detailed hai — “wedding invitation with gold floral border, serif typography, text ‘Save The Date’” — to model visual style ko descriptive words se link karna seekhta hai. Ye inference ke time pe prompt following improve karta hai.

Core Logic Explained

import torch
from transformers import AutoProcessor, AutoModelForCausalLM
from PIL import Image
import easyocr
import os
import argparse
from pathlib import Path
 
# ─────────────────────────────────────────
# Florence-2 Setup
# ─────────────────────────────────────────
# Florence-2 ek Microsoft ka vision-language model hai.
# Base fine-tuned version (~270M params) use karte hain — fast aur accurate.
# "florence-2-base-ft" mein "ft" = fine-tuned on more tasks.
 
FLORENCE_MODEL = "microsoft/Florence-2-base-ft"
 
def load_florence():
    processor = AutoProcessor.from_pretrained(FLORENCE_MODEL, trust_remote_code=True)
    model = AutoModelForCausalLM.from_pretrained(
        FLORENCE_MODEL,
        torch_dtype=torch.float32,  # CPU pe float32, GPU pe float16
        trust_remote_code=True
    )
    return model, processor
 
def generate_visual_caption(image_path, model, processor):
    """
    Florence-2 se detailed image description generate karo.
    
    Task "<MORE_DETAILED_CAPTION>" se Florence zyada detail deta hai
    simple "<CAPTION>" se zyada. Ye task token model ko batata hai
    ki kitna detailed respond karna hai.
    """
    image = Image.open(image_path).convert("RGB")
    
    task = "<MORE_DETAILED_CAPTION>"
    
    inputs = processor(
        text=task,
        images=image,
        return_tensors="pt"
    )
    
    with torch.no_grad():
        generated_ids = model.generate(
            input_ids=inputs["input_ids"],
            pixel_values=inputs["pixel_values"],
            max_new_tokens=256,
            num_beams=3,       # beam search — 3 candidates consider karta hai
            do_sample=False    # deterministic output
        )
    
    caption = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
    
    # Florence wraps output in task token — clean karo
    caption = processor.post_process_generation(
        caption, task=task, image_size=(image.width, image.height)
    )
    
    return caption[task].strip()
 
# ─────────────────────────────────────────
# EasyOCR Setup
# ─────────────────────────────────────────
# EasyOCR ek open source OCR library hai.
# ['en', 'hi'] matlab English aur Hindi dono detect karega.
# GPU=False kyunki CPU pe bhi fast enough hai OCR ke liye.
 
def load_ocr():
    return easyocr.Reader(['en', 'hi'], gpu=False)
 
def extract_text_from_image(image_path, reader, min_confidence=0.78):
    """
    Image mein visible text detect karo.
    
    reader.readtext() returns list of tuples:-
    [(bounding_box, text, confidence), ...]
    
    Confidence filter important hai — low confidence detections
    mostly garbage hain (random shapes jo OCR ko text jaisi lagti hain).
    
    Additional filters:-
    - Length filter: 1 se 50 characters ke beech
    - Digit ratio: agar text mostly digits hai to likely price/number
    - Mixed script filter: agar ek word mein Hindi aur English mixed hai
    """
    try:
        results = reader.readtext(image_path)
    except Exception:
        return []
    
    clean_texts = []
    
    for (bbox, text, confidence) in results:
        if confidence < min_confidence:
            continue
        
        text = text.strip()
        
        # Length filter
        if len(text) < 2 or len(text) > 50:
            continue
        
        # Garbage filter — mostly symbols or single characters
        alpha_count = sum(1 for c in text if c.isalpha())
        if alpha_count < 1:
            continue
        
        clean_texts.append(text)
    
    return clean_texts
 
# ─────────────────────────────────────────
# Style Tags from Filename
# ─────────────────────────────────────────
# Filename se design style guess karo.
# Ye rough hai but adds useful context to captions.
 
STYLE_KEYWORDS = {
    "wedding": "wedding design, formal, elegant",
    "diwali": "Diwali festival, Indian festival design",
    "sale": "sale promotion, marketing design",
    "poster": "poster design, large format",
    "flyer": "flyer design, promotional",
    "invitation": "invitation card design",
    "banner": "banner design, horizontal format",
    "product": "product photography, commercial",
}
 
def get_style_tags(filename):
    filename_lower = filename.lower()
    tags = []
    for keyword, tag in STYLE_KEYWORDS.items():
        if keyword in filename_lower:
            tags.append(tag)
    return ", ".join(tags)
 
# ─────────────────────────────────────────
# Final Caption Assembly
# ─────────────────────────────────────────
 
def build_caption(visual_description, ocr_texts, style_tags):
    """
    Teen sources merge karke final training caption banao.
    
    Format:-
    [style tags], [visual description], [text "word1"], [text "word2"]
    
    Example output:-
    "wedding invitation card, formal design, elegant gold border on cream background 
    with floral motifs and serif typography, text 'Save The Date', text 'Mr & Mrs Sharma'"
    """
    parts = []
    
    if style_tags:
        parts.append(style_tags)
    
    if visual_description:
        parts.append(visual_description)
    
    for t in ocr_texts[:6]:  # max 6 text detections per image
        parts.append(f'text "{t}"')
    
    return ", ".join(parts)
 
# ─────────────────────────────────────────
# Main Runner
# ─────────────────────────────────────────
 
def caption_dataset(image_folder, output_folder, skip_done=True):
    Path(output_folder).mkdir(parents=True, exist_ok=True)
    
    print("Loading Florence-2...")
    florence_model, florence_processor = load_florence()
    
    print("Loading EasyOCR...")
    ocr_reader = load_ocr()
    
    image_files = list(Path(image_folder).glob("*.jpg")) + \
                  list(Path(image_folder).glob("*.png"))
    
    print(f"Found {len(image_files)} images")
    
    for i, img_path in enumerate(image_files):
        caption_path = Path(output_folder) / (img_path.stem + ".txt")
        
        if skip_done and caption_path.exists():
            continue
        
        try:
            visual = generate_visual_caption(str(img_path), florence_model, florence_processor)
            ocr = extract_text_from_image(str(img_path), ocr_reader)
            tags = get_style_tags(img_path.name)
            caption = build_caption(visual, ocr, tags)
            
            caption_path.write_text(caption, encoding="utf-8")
            
            if (i + 1) % 50 == 0:
                print(f"Progress: {i+1}/{len(image_files)}")
        
        except Exception as e:
            print(f"Error on {img_path.name}: {e}")

Kaise Run Karo

pip install torch transformers pillow easyocr tqdm einops timm
 
# CPU pe (slow — ~40 sec per image)
python smart_caption.py --folder data/raw/flickr --output data/captions/flickr
 
# 7000 images pe estimate: ~3 days on CPU
# A100 pe: ~2 hours

5. Scrape and Caption Agent

Kya karta hai

Ek combined script jo scraping aur captioning ek saath karta hai. Pinterest aur Google Images dono se simultaneously scrape karta hai, page context (title, alt text, description) bhi capture karta hai alongside image, phir Florence plus OCR plus context teeno merge karke caption banata hai.

Kyun Page Context Better Captions Deta Hai

Jab koi image Pinterest pe hoti hai, uske saath hoti hai:-

Pin title: “Beautiful Diwali Party Invitation Card”
Board name: “Indian Festival Designs”
Alt text: “Golden Diwali invitation with diyas and rangoli”

Ye context purely visual description se kaafi better hai. Model ne khud label kiya hai image ko — hum wo label use kar rahe hain caption mein.

Core Logic

import requests
import os
import json
import time
from PIL import Image
from io import BytesIO
from bs4 import BeautifulSoup
from transformers import AutoProcessor, AutoModelForCausalLM
import easyocr
import torch
from pathlib import Path
import argparse
 
class ScrapeAndCaptionAgent:
    
    def __init__(self, output_dir):
        self.output_dir = Path(output_dir)
        self.images_dir = self.output_dir / "images"
        self.captions_dir = self.output_dir / "captions"
        self.metadata_file = self.output_dir / "metadata.jsonl"
        
        self.images_dir.mkdir(parents=True, exist_ok=True)
        self.captions_dir.mkdir(parents=True, exist_ok=True)
        
        print("Loading Florence-2...")
        self.florence_processor = AutoProcessor.from_pretrained(
            "microsoft/Florence-2-base-ft", trust_remote_code=True
        )
        self.florence_model = AutoModelForCausalLM.from_pretrained(
            "microsoft/Florence-2-base-ft",
            torch_dtype=torch.float32,
            trust_remote_code=True
        )
        
        print("Loading EasyOCR...")
        self.ocr = easyocr.Reader(['en', 'hi'], gpu=False)
    
    # ─────────────────────────────────────────
    # Pinterest Scraping with Context
    # ─────────────────────────────────────────
    
    def scrape_pinterest_with_context(self, query, target=500):
        """
        Pinterest se image + metadata ek saath.
        
        Pin object mein ye fields available hain:-
        - images: different size URLs
        - title: pin ka title
        - description: pin ka description
        - board_name: kis board pe hai
        - alt_text: accessibility text (often very descriptive)
        
        Ye sab milke caption quality kaafi improve karte hain.
        """
        results = []
        bookmark = None
        
        while len(results) < target:
            try:
                pins, bookmark = self._pinterest_search(query, bookmark)
            except Exception as e:
                print(f"Pinterest error: {e}")
                time.sleep(30)
                continue
            
            for pin in pins:
                # Image URL
                images = pin.get("images", {})
                url = (images.get("orig") or images.get("736x") or {}).get("url")
                if not url:
                    continue
                
                # Context metadata — ye caption mein jaayega
                context = {
                    "source": "pinterest",
                    "query": query,
                    "title": pin.get("title", ""),
                    "description": pin.get("description", ""),
                    "alt_text": pin.get("alt_text", ""),
                    "board_name": (pin.get("board") or {}).get("name", ""),
                    "image_url": url
                }
                
                results.append(context)
                time.sleep(1.5)
                
                if len(results) >= target:
                    break
            
            if not bookmark:
                break
        
        return results
    
    # ─────────────────────────────────────────
    # Google Images Scraping
    # ─────────────────────────────────────────
    
    def scrape_google_images(self, query, target=100):
        """
        Google Images se scrape karo via regular search.
        
        Note:- Google images scraping fragile hai. URL structure change hoti rehti hai.
        Isliye ye sirf supplementary source hai, primary nahi.
        
        SerpAPI ya ScraperAPI use karna zyada reliable hoga production mein.
        """
        results = []
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
        }
        
        search_url = f"https://www.google.com/search?q={query}&tbm=isch&hl=en"
        
        try:
            response = requests.get(search_url, headers=headers, timeout=10)
            soup = BeautifulSoup(response.text, 'html.parser')
            
            # Google image tags from search results
            for img in soup.find_all("img", limit=target):
                src = img.get("src", "")
                alt = img.get("alt", "")
                
                if src.startswith("http") and not "google" in src:
                    results.append({
                        "source": "google",
                        "query": query,
                        "image_url": src,
                        "alt_text": alt,
                        "title": "",
                        "description": "",
                        "board_name": ""
                    })
        
        except Exception as e:
            print(f"Google scrape error: {e}")
        
        return results
    
    # ─────────────────────────────────────────
    # Download and Process Single Image
    # ─────────────────────────────────────────
    
    def process_item(self, context_dict):
        """
        Ek image download karo aur uski caption banao.
        
        Steps:-
        1. Image download karo
        2. Size validate karo (minimum 256x256, avoid tiny thumbnails)
        3. Florence-2 se visual description
        4. EasyOCR se text extraction
        5. Context (title, alt, description) se tags
        6. Sab merge karke final caption
        7. Image save karo, caption .txt file save karo
        8. metadata.jsonl mein entry append karo
        """
        url = context_dict["image_url"]
        
        try:
            response = requests.get(url, timeout=15, stream=True)
            response.raise_for_status()
            
            img_bytes = response.content
            image = Image.open(BytesIO(img_bytes)).convert("RGB")
            
            # Minimum size check
            if image.width < 256 or image.height < 256:
                return False
            
        except Exception as e:
            return False
        
        # Unique filename from URL hash
        import hashlib
        url_hash = hashlib.md5(url.encode()).hexdigest()[:12]
        source = context_dict["source"]
        img_filename = f"{source}_{url_hash}.jpg"
        img_path = self.images_dir / img_filename
        
        # Save image
        image.save(str(img_path), "JPEG", quality=95)
        
        # Generate caption parts
        visual_desc = self._florence_caption(image)
        ocr_texts = self._ocr_extract(str(img_path))
        context_text = self._context_to_text(context_dict)
        
        # Merge caption
        parts = []
        if context_text:
            parts.append(context_text)
        if visual_desc:
            parts.append(visual_desc)
        for t in ocr_texts[:5]:
            parts.append(f'text "{t}"')
        
        caption = ", ".join(parts)
        
        # Save caption .txt
        caption_path = self.captions_dir / (img_filename.replace(".jpg", ".txt"))
        caption_path.write_text(caption, encoding="utf-8")
        
        # Append to JSONL
        record = {
            "image": str(img_path),
            "caption": caption,
            "source": source,
            "query": context_dict["query"]
        }
        with open(self.metadata_file, 'a') as f:
            f.write(json.dumps(record, ensure_ascii=False) + "\n")
        
        return True
    
    def _context_to_text(self, ctx):
        """Page context se useful tags banao."""
        parts = []
        if ctx.get("title") and len(ctx["title"]) > 3:
            parts.append(ctx["title"].lower())
        if ctx.get("alt_text") and len(ctx["alt_text"]) > 3:
            parts.append(ctx["alt_text"].lower())
        if ctx.get("board_name") and len(ctx["board_name"]) > 3:
            parts.append(ctx["board_name"].lower())
        return ", ".join(parts[:3])  # max 3 context tags
    
    def _florence_caption(self, pil_image):
        """Florence-2 se visual description."""
        try:
            inputs = self.florence_processor(
                text="<MORE_DETAILED_CAPTION>",
                images=pil_image,
                return_tensors="pt"
            )
            with torch.no_grad():
                ids = self.florence_model.generate(
                    **inputs, max_new_tokens=150, num_beams=3
                )
            result = self.florence_processor.batch_decode(ids, skip_special_tokens=False)[0]
            result = self.florence_processor.post_process_generation(
                result, task="<MORE_DETAILED_CAPTION>",
                image_size=(pil_image.width, pil_image.height)
            )
            return result["<MORE_DETAILED_CAPTION>"].strip()
        except:
            return ""
    
    def _ocr_extract(self, image_path):
        """EasyOCR se text extract karo."""
        try:
            results = self.ocr.readtext(image_path)
            return [r[1].strip() for r in results if r[2] > 0.78 and 2 <= len(r[1]) <= 50]
        except:
            return []

Kaise Run Karo

python scrape_and_caption_agent.py \
  --source both \
  --target 5000 \
  --output data/agent_collected \
  --queries "diwali poster,wedding card,sale banner,product launch"

6. Glyph Renderer

Kya karta hai

LLM ka JSON schema le aur ek white background black text PNG banao jo ControlNet ko input milega. Ye pipeline ka Stage 3 hai.

Core Logic Explained

from PIL import Image, ImageDraw, ImageFont
import json
from pathlib import Path
from dataclasses import dataclass
from typing import List, Tuple
import os
 
@dataclass
class TextBlock:
    """Ek text element ki poori information."""
    content: str
    x: int
    y: int
    font_size: int
    font_weight: str          # regular, bold, heavy, black
    color_instruction: str    # ye Flux ke prompt mein jaayega, glyph mein nahi
 
@dataclass
class GlyphSchema:
    """LLM ka poora output schema."""
    canvas_size: Tuple[int, int]
    background_prompt: str
    style_notes: str
    texts: List[TextBlock]
 
class GlyphRenderer:
    
    def __init__(self, fonts_dir: str):
        """
        fonts_dir mein ye files honi chahiye:-
        Inter-Regular.ttf
        Inter-Bold.ttf
        Inter-Heavy.ttf (ya ExtraBold)
        Inter-Black.ttf
        
        Inter font bahut clean hai marketing designs ke liye.
        NotoSans Hindi/Tamil/Bengali ke liye.
        """
        self.fonts_dir = Path(fonts_dir)
        self._font_cache = {}  # same font baar baar load mat karo
    
    def _load_font(self, weight: str, size: int) -> ImageFont.FreeTypeFont:
        """
        Font cache se load karo ya naya load karo.
        
        Weight mapping:-
        regular  →  Inter-Regular.ttf
        bold     →  Inter-Bold.ttf
        heavy    →  Inter-ExtraBold.ttf (800 weight)
        black    →  Inter-Black.ttf (900 weight)
        """
        weight_map = {
            "regular": "Inter-Regular.ttf",
            "light": "Inter-Light.ttf",
            "bold": "Inter-Bold.ttf",
            "semibold": "Inter-SemiBold.ttf",
            "heavy": "Inter-ExtraBold.ttf",
            "black": "Inter-Black.ttf",
        }
        
        font_file = weight_map.get(weight.lower(), "Inter-Bold.ttf")
        font_path = str(self.fonts_dir / font_file)
        
        cache_key = f"{font_file}_{size}"
        if cache_key not in self._font_cache:
            try:
                self._font_cache[cache_key] = ImageFont.truetype(font_path, size)
            except Exception:
                # Fallback to default if font not found
                print(f"Warning: Font {font_file} not found, using default")
                self._font_cache[cache_key] = ImageFont.load_default()
        
        return self._font_cache[cache_key]
    
    def render(self, schema: GlyphSchema) -> Image.Image:
        """
        Schema se glyph image render karo.
        
        White background kyun? ControlNet Canny edge detection use karta hai.
        Black text on white background se sabse clean edges milti hain.
        
        Text color hamesha black (0, 0, 0) kyunki:-
        - Text ka actual color Flux decide karta hai
        - ControlNet sirf shape/position enforce karta hai
        - Color information glyph mein nahi hoti, Flux prompt mein hoti hai
        """
        w, h = schema.canvas_size
        
        # White background
        glyph = Image.new("RGB", (w, h), color=(255, 255, 255))
        draw = ImageDraw.Draw(glyph)
        
        for block in schema.texts:
            font = self._load_font(block.font_weight, block.font_size)
            
            draw.text(
                xy=(block.x, block.y),
                text=block.content,
                fill=(0, 0, 0),   # always black
                font=font
            )
        
        return glyph
    
    def render_from_json(self, json_str: str) -> Tuple[Image.Image, str]:
        """
        LLM ke raw JSON string se directly render karo.
        Returns:- (glyph_image, flux_prompt)
        
        flux_prompt banate waqt color_instructions ko text prompt mein include karo.
        """
        data = json.loads(json_str)
        
        texts = [
            TextBlock(
                content=t["content"],
                x=t["x"],
                y=t["y"],
                font_size=t["font_size"],
                font_weight=t.get("font_weight", "bold"),
                color_instruction=t.get("color_instruction", "white")
            )
            for t in data["texts"]
        ]
        
        schema = GlyphSchema(
            canvas_size=tuple(data["canvas_size"]),
            background_prompt=data.get("background_prompt", ""),
            style_notes=data.get("style_notes", ""),
            texts=texts
        )
        
        glyph = self.render(schema)
        
        # Flux prompt banao — background + style + text colors
        text_color_desc = ", ".join(
            f'{t.content} in {t.color_instruction}'
            for t in texts
        )
        
        flux_prompt = (
            f"{schema.background_prompt}, "
            f"{schema.style_notes}, "
            f"{text_color_desc}, "
            f"professional marketing image, high quality, sharp"
        )
        
        return glyph, flux_prompt
    
    def validate_positions(self, schema: GlyphSchema) -> GlyphSchema:
        """
        LLM kabhi kabhi out-of-bounds positions deta hai.
        Ye method positions ko canvas ke andar clamp karta hai.
        """
        w, h = schema.canvas_size
        validated_texts = []
        
        for block in schema.texts:
            # Ensure text starts within canvas
            x = max(0, min(block.x, w - 50))
            y = max(0, min(block.y, h - block.font_size))
            
            validated_texts.append(TextBlock(
                content=block.content,
                x=x,
                y=y,
                font_size=max(20, min(block.font_size, 400)),  # clamp font size
                font_weight=block.font_weight,
                color_instruction=block.color_instruction
            ))
        
        schema.texts = validated_texts
        return schema

7. LLM Decomposer

Kya karta hai

Ollama ke through Mistral 7B ko call karta hai. User brief in, structured JSON schema out.

Core Logic Explained

import requests
import json
from typing import Optional
 
# Ollama local server — `ollama serve` run karne pe ye URL active hoti hai
OLLAMA_URL = "http://localhost:11434/api/generate"
 
SYSTEM_PROMPT = """You are a professional marketing poster layout designer specializing in Indian markets.
 
Your job: Convert a user's marketing brief into a precise JSON layout schema.
 
Rules:-
1. Always output ONLY valid JSON. No explanation, no markdown, no preamble.
2. Choose font sizes that create clear visual hierarchy. Headline biggest, body smallest.
3. Position elements with proper spacing. Canvas is 1024x1024 pixels.
4. x and y are top-left corner of each text element.
5. font_weight must be one of: regular, bold, heavy, black
6. background_prompt should be detailed and descriptive for Flux image generation.
7. color_instruction describes how Flux should color the text — be specific.
 
Output format (strictly follow this schema):-
{
  "canvas_size": [1024, 1024],
  "background_prompt": "detailed background description for AI image generation",
  "style_notes": "overall aesthetic and mood",
  "texts": [
    {
      "content": "TEXT CONTENT HERE",
      "x": 100,
      "y": 100,
      "font_size": 120,
      "font_weight": "bold",
      "color_instruction": "bright gold metallic with shine"
    }
  ]
}"""
 
def decompose_brief(brief: str, model: str = "mistral") -> Optional[dict]:
    """
    User brief ko JSON schema mein convert karo via Ollama.
    
    Ollama API simple hai:-
    - model: jo model use karna hai (must be pulled: ollama pull mistral)
    - prompt: user ka message
    - system: system instructions
    - stream: False matlab poora response ek saath
    - options: temperature, num_predict etc.
    
    Temperature 0.3 isliye — creative enough for good layouts
    but consistent enough for valid JSON output.
    """
    
    payload = {
        "model": model,
        "prompt": f"Create a poster layout JSON for this brief:\n\n{brief}",
        "system": SYSTEM_PROMPT,
        "stream": False,
        "options": {
            "temperature": 0.3,
            "num_predict": 1024,   # max tokens in response
            "top_p": 0.9,
            "repeat_penalty": 1.1
        }
    }
    
    try:
        response = requests.post(OLLAMA_URL, json=payload, timeout=60)
        response.raise_for_status()
        
        result = response.json()
        raw_output = result["response"].strip()
        
        # Clean response — remove markdown code blocks if present
        if raw_output.startswith("```"):
            raw_output = raw_output.split("```")[1]
            if raw_output.startswith("json"):
                raw_output = raw_output[4:]
        raw_output = raw_output.strip()
        
        # Parse and validate
        schema = json.loads(raw_output)
        
        # Basic validation
        required_keys = ["canvas_size", "background_prompt", "texts"]
        if not all(k in schema for k in required_keys):
            raise ValueError(f"Missing required keys in schema")
        
        if not isinstance(schema["texts"], list) or len(schema["texts"]) == 0:
            raise ValueError("texts must be non-empty list")
        
        return schema
    
    except json.JSONDecodeError as e:
        print(f"LLM produced invalid JSON: {e}")
        print(f"Raw output: {raw_output[:500]}")
        return None
    
    except requests.RequestException as e:
        print(f"Ollama connection error: {e}")
        print("Is Ollama running? Run: ollama serve")
        return None
 
def decompose_with_retry(brief: str, max_attempts: int = 3) -> Optional[dict]:
    """
    Retry logic — LLM kabhi kabhi invalid JSON deta hai.
    3 attempts karo. Agar teeno fail — None return karo.
    """
    for attempt in range(max_attempts):
        result = decompose_brief(brief)
        if result:
            return result
        print(f"Attempt {attempt + 1} failed, retrying...")
    
    print("All attempts failed. Check LLM connection and prompting.")
    return None

Ollama Setup Commands

# Ollama install (Linux)
curl -fsSL https://ollama.ai/install.sh | sh
 
# Model pull
ollama pull mistral        # 4.1GB
ollama pull llama3.1:8b    # 4.7GB
 
# Server start (background mein)
ollama serve &
 
# Test
curl http://localhost:11434/api/generate \
  -d '{"model": "mistral", "prompt": "hello", "stream": false}'

8. ControlNet + Flux Pipeline

Kya karta hai

Ye pipeline ka core hai. Glyph PNG aur Flux prompt lo, ControlNet enforced image generate karo.

Core Logic Explained

import torch
from diffusers import FluxControlNetPipeline, FluxControlNetModel
from diffusers.utils import load_image
from PIL import Image
import numpy as np
 
class FluxGlyphPipeline:
    
    def __init__(
        self,
        flux_model: str = "black-forest-labs/FLUX.1-dev",
        controlnet_model: str = "InstantX/FLUX.1-dev-Controlnet-Union",
        lora_path: str = None,
        device: str = "cuda"
    ):
        self.device = device
        
        print("Loading ControlNet...")
        # ControlNet Union — ek model jo multiple control types support karta hai
        # control_type 0 = canny edges (ye hum use karte hain)
        # control_type 2 = depth map
        # control_type 4 = pose
        self.controlnet = FluxControlNetModel.from_pretrained(
            controlnet_model,
            torch_dtype=torch.bfloat16
        )
        
        print("Loading Flux pipeline...")
        self.pipe = FluxControlNetPipeline.from_pretrained(
            flux_model,
            controlnet=self.controlnet,
            torch_dtype=torch.bfloat16
        )
        self.pipe.to(device)
        
        # Memory optimization — attention slicing
        self.pipe.enable_attention_slicing()
        
        # LoRA load karo agar diya hai
        if lora_path:
            self.load_lora(lora_path)
    
    def load_lora(self, lora_path: str, lora_scale: float = 0.85):
        """
        LoRA adapter load karo.
        
        lora_scale = 0.85 matlab 85% LoRA influence + 15% base model.
        1.0 pe LoRA completely dominant ho jaata hai — sometimes too much.
        0.7 pe LoRA subtle influence deta hai.
        """
        self.pipe.load_lora_weights(lora_path)
        self.pipe.fuse_lora(lora_scale=lora_scale)
        print(f"LoRA loaded from {lora_path} at scale {lora_scale}")
    
    def preprocess_glyph(self, glyph_image: Image.Image) -> Image.Image:
        """
        Glyph PNG ko ControlNet ke liye prepare karo.
        
        ControlNet Canny edge detection use karta hai.
        Lekin hamare glyph pe hum Canny nahi chalate —
        humara glyph already clean black-on-white hai.
        
        Hum directly glyph use karte hain as control image.
        ControlNet internally edges extract kar leta hai.
        
        Size must match generation size.
        """
        return glyph_image.resize((1024, 1024), Image.LANCZOS)
    
    def generate(
        self,
        prompt: str,
        glyph_image: Image.Image,
        controlnet_scale: float = 0.75,
        num_steps: int = 28,
        guidance_scale: float = 3.5,
        seed: int = None,
        width: int = 1024,
        height: int = 1024,
    ) -> Image.Image:
        """
        Core generation function.
        
        Parameters:-
        
        controlnet_scale:-
            0.75 sweet spot. Text locked, style free.
        
        num_steps:-
            28 Flux ke liye good balance (speed vs quality).
            Flux ek flow matching model hai — 50 steps zaruri nahi.
            28-30 steps pe excellent quality milti hai.
        
        guidance_scale:-
            3.5 Flux ke liye standard.
            Higher = prompt ko zyada follow karta hai lekin less diverse.
            Lower = more creative but might ignore prompt.
        
        seed:-
            None = random. Set karo reproducibility ke liye.
        
        control_mode:-
            0 = canny/edge mode for ControlNet Union.
            Ye batata hai ControlNet ko ki is control image ka
            kaunsa type use karna hai.
        """
        control_image = self.preprocess_glyph(glyph_image)
        
        generator = None
        if seed is not None:
            generator = torch.Generator(device=self.device).manual_seed(seed)
        
        # Negative prompt Flux mein directly supported nahi hai
        # lekin kuch wrappers mein hai — ignore karo agar error aaye
        result = self.pipe(
            prompt=prompt,
            control_image=control_image,
            controlnet_conditioning_scale=controlnet_scale,
            control_mode=0,           # canny/edges mode
            num_inference_steps=num_steps,
            guidance_scale=guidance_scale,
            width=width,
            height=height,
            generator=generator,
        )
        
        return result.images[0]
    
    def generate_with_retry(
        self,
        prompt: str,
        glyph_image: Image.Image,
        quality_gate_fn,
        max_attempts: int = 3,
        **kwargs
    ) -> tuple:
        """
        Quality gate ke saath generate karo.
        Agar quality gate fail ho to different seed se retry karo.
        
        Returns:- (image, passed_quality_gate, attempts_used)
        """
        for attempt in range(max_attempts):
            seed = torch.randint(0, 2**32, (1,)).item()
            image = self.generate(prompt, glyph_image, seed=seed, **kwargs)
            
            passed, scores = quality_gate_fn(image, prompt)
            
            if passed:
                return image, True, attempt + 1
            
            print(f"Quality gate failed (attempt {attempt+1}): {scores}")
        
        # Return last attempt even if failed
        return image, False, max_attempts

Memory Management

# Agar GPU VRAM kam ho to ye options use karo
 
# Option 1: Sequential CPU offload
# Slow but works on 8GB VRAM
pipe.enable_sequential_cpu_offload()
 
# Option 2: Model CPU offload
# Faster than sequential, needs ~16GB VRAM peak
pipe.enable_model_cpu_offload()
 
# Option 3: FP8 quantization (RTX 4090 pe)
# pip install optimum-quanto
from optimum.quanto import quantize, qfloat8
quantize(pipe.transformer, weights=qfloat8)

9. Quality Gate

Kya karta hai

Generated image ko do models se score karta hai — CLIP aur LAION Aesthetic Predictor. Agar dono thresholds pass kare to image user ko jaati hai, nahi to regenerate.

import torch
import clip
from PIL import Image
import numpy as np
import requests
from transformers import pipeline as hf_pipeline
 
class QualityGate:
    
    def __init__(self, clip_threshold=0.25, aesthetic_threshold=5.5):
        self.clip_threshold = clip_threshold
        self.aesthetic_threshold = aesthetic_threshold
        
        print("Loading CLIP for quality gate...")
        # CLIP ViT-L/14 — largest CLIP model, most accurate scoring
        self.clip_model, self.clip_preprocess = clip.load("ViT-L/14")
        self.clip_model.eval()
        
        print("Loading Aesthetic Predictor...")
        # LAION Aesthetic Predictor v2
        # Ye ek simple MLP hai jo CLIP embeddings pe trained hai
        # Output: 0-10 aesthetic score
        self.aesthetic_scorer = self._load_aesthetic_predictor()
    
    def _load_aesthetic_predictor(self):
        """
        LAION Aesthetic Predictor v2 load karo.
        
        Ye model CLIP image embeddings leta hai input mein
        aur ek single aesthetic quality score (0-10) output karta hai.
        
        Trained on human ratings of images — logo ne rate kiya
        ki kaunsi images visually pleasing hain.
        """
        try:
            from aesthetic_predictor_v2_5 import convert_v2_5_from_siglip
            # v2.5 uses SigLIP instead of CLIP for better accuracy
            predictor, preprocessor = convert_v2_5_from_siglip(
                low_cpu_mem_usage=True,
                trust_remote_code=True,
            )
            predictor.eval()
            return ("v2.5", predictor, preprocessor)
        except ImportError:
            print("aesthetic_predictor_v2_5 not installed, using basic scoring")
            return None
    
    def clip_score(self, image: Image.Image, prompt: str) -> float:
        """
        CLIP score = cosine similarity between image embedding and text embedding.
        
        Ye measure karta hai ki image semantically kitni close hai prompt se.
        0.25+ ka matlab image prompt ka concept capture kar rahi hai.
        
        Kaise kaam karta hai:-
        1. Image ko CLIP ke through encode karo → 512/768 dim vector
        2. Prompt text ko CLIP ke through encode karo → 512/768 dim vector
        3. In dono vectors ka cosine similarity nikalo
        4. Yahi score hai
        """
        with torch.no_grad():
            # Preprocess
            img_tensor = self.clip_preprocess(image).unsqueeze(0)
            text_tokens = clip.tokenize([prompt], truncate=True)
            
            # Encode
            img_features = self.clip_model.encode_image(img_tensor)
            text_features = self.clip_model.encode_text(text_tokens)
            
            # Normalize
            img_features = img_features / img_features.norm(dim=-1, keepdim=True)
            text_features = text_features / text_features.norm(dim=-1, keepdim=True)
            
            # Cosine similarity
            score = (img_features * text_features).sum().item()
        
        return score
    
    def aesthetic_score(self, image: Image.Image) -> float:
        """
        Image ki visual quality score karo — 0 to 10.
        
        Ye sirf visual quality hai — prompt se match hona zaruri nahi.
        Blurry, ugly, poorly composed images yahan fail hoti hain.
        """
        if self.aesthetic_scorer is None:
            return 6.0  # default pass agar model nahi load hua
        
        version, predictor, preprocessor = self.aesthetic_scorer
        
        with torch.no_grad():
            inputs = preprocessor(images=image, return_tensors="pt")
            score = predictor(**inputs).logits.squeeze().item()
        
        # Normalize to 0-10 range if needed
        if score > 1.0:
            return score  # already 0-10
        else:
            return score * 10  # 0-1 to 0-10
    
    def evaluate(self, image: Image.Image, prompt: str) -> tuple:
        """
        Dono scores nikalo aur decision do.
        Returns:- (passed: bool, scores: dict)
        """
        cs = self.clip_score(image, prompt)
        ae = self.aesthetic_score(image)
        
        scores = {
            "clip_score": round(cs, 4),
            "aesthetic_score": round(ae, 2),
            "clip_passed": cs >= self.clip_threshold,
            "aesthetic_passed": ae >= self.aesthetic_threshold,
        }
        
        passed = scores["clip_passed"] and scores["aesthetic_passed"]
        
        return passed, scores

10. Upscaler

Kya karta hai

Generated 1024px image ko 2048px ya 4096px tak upscale karta hai. Real-ESRGAN sirf resize nahi karta — wo fine detail add karta hai neural network se.

from basicsr.archs.rrdbnet_arch import RRDBNet
from realesrgan import RealESRGANer
from PIL import Image
import numpy as np
import cv2
 
class Upscaler:
    
    def __init__(self, scale: int = 4, model_path: str = "models/RealESRGAN_x4plus.pth"):
        """
        scale = 4 matlab 1024 → 4096
        scale = 2 matlab 1024 → 2048
        
        RRDBNet = Residual in Residual Dense Block Network
        Ye Real-ESRGAN ka backbone architecture hai.
        
        num_block=23 — original model ka configuration
        """
        model = RRDBNet(
            num_in_ch=3,
            num_out_ch=3,
            num_feat=64,
            num_block=23,
            num_grow_ch=32,
            scale=scale
        )
        
        self.upsampler = RealESRGANer(
            scale=scale,
            model_path=model_path,
            model=model,
            tile=512,        # large images ko tiles mein process karo
            tile_pad=10,     # tile boundaries pe artifacts avoid karo
            pre_pad=0,
            half=True        # FP16 — faster, less memory
        )
        
        self.scale = scale
    
    def upscale(self, image: Image.Image) -> Image.Image:
        """
        PIL Image → upscaled PIL Image
        
        Real-ESRGAN OpenCV format use karta hai (BGR, numpy array).
        PIL RGB format use karta hai.
        Convert karna padta hai dono taraf.
        """
        # PIL → OpenCV format
        img_np = np.array(image)              # RGB numpy
        img_bgr = cv2.cvtColor(img_np, cv2.COLOR_RGB2BGR)  # BGR numpy
        
        # Upscale
        output_bgr, _ = self.upsampler.enhance(img_bgr, outscale=self.scale)
        
        # OpenCV → PIL format
        output_rgb = cv2.cvtColor(output_bgr, cv2.COLOR_BGR2RGB)
        return Image.fromarray(output_rgb)

11. Evaluation Runner

Kya karta hai

100 test images pe poori pipeline run karta hai aur sab metrics calculate karta hai. Ye final evaluation script hai jo proof deta hai ki pipeline kaam karti hai.

import easyocr
import json
import torch
import clip
from PIL import Image
from pathlib import Path
import editdistance
from jiwer import cer, wer
import csv
 
class PipelineEvaluator:
    
    def __init__(self):
        print("Loading OCR for evaluation...")
        self.ocr = easyocr.Reader(['en', 'hi'], gpu=False)
        
        print("Loading CLIP for evaluation...")
        self.clip_model, self.clip_preprocess = clip.load("ViT-L/14")
    
    def evaluate_text_accuracy(self, image_path: str, expected_text: str) -> dict:
        """
        Text accuracy metrics — ye sabse important evaluation hai.
        
        OCR se detected text ko expected text se compare karo.
        
        CER = Character Error Rate
        WER = Word Error Rate  
        NED = Normalized Edit Distance (higher is better)
        exact = 1 agar poora match, 0 nahi
        """
        # OCR detect karo
        results = self.ocr.readtext(image_path)
        detected = " ".join([r[1] for r in results if r[2] > 0.5])
        detected = detected.upper().strip()
        expected = expected_text.upper().strip()
        
        if not detected:
            return {"CER": 1.0, "WER": 1.0, "NED": 0.0, "exact": 0,
                    "expected": expected, "detected": "NOTHING_DETECTED"}
        
        # Edit distance
        ed = editdistance.eval(expected, detected)
        
        metrics = {
            "expected": expected,
            "detected": detected,
            "CER": round(cer(expected, detected), 4),
            "WER": round(wer(expected, detected), 4),
            "NED": round(1 - ed / max(len(expected), len(detected)), 4),
            "exact": 1 if expected == detected else 0
        }
        
        return metrics
    
    def evaluate_visual_quality(self, image_path: str, prompt: str) -> dict:
        """CLIP score nikalo."""
        image = Image.open(image_path)
        
        with torch.no_grad():
            img_tensor = self.clip_preprocess(image).unsqueeze(0)
            text_tokens = clip.tokenize([prompt[:77]], truncate=True)
            
            img_feat = self.clip_model.encode_image(img_tensor)
            txt_feat = self.clip_model.encode_text(text_tokens)
            
            img_feat = img_feat / img_feat.norm(dim=-1, keepdim=True)
            txt_feat = txt_feat / txt_feat.norm(dim=-1, keepdim=True)
            
            score = (img_feat * txt_feat).sum().item()
        
        return {"clip_score": round(score, 4)}
    
    def run_full_evaluation(self, test_cases: list, output_csv: str) -> dict:
        """
        test_cases format:-
        [
            {
                "image_path": "outputs/test_001.jpg",
                "expected_text": "DIWALI SALE 50% OFF",
                "prompt": "luxury Diwali poster, gold red theme"
            },
            ...
        ]
        
        Returns summary statistics.
        """
        all_results = []
        
        for i, case in enumerate(test_cases):
            print(f"Evaluating {i+1}/{len(test_cases)}: {case['expected_text'][:30]}")
            
            text_metrics = self.evaluate_text_accuracy(
                case["image_path"], case["expected_text"]
            )
            visual_metrics = self.evaluate_visual_quality(
                case["image_path"], case["prompt"]
            )
            
            result = {**text_metrics, **visual_metrics, "image": case["image_path"]}
            all_results.append(result)
        
        # Summary statistics
        n = len(all_results)
        summary = {
            "total_images": n,
            "exact_match_rate": f"{sum(r['exact'] for r in all_results)/n*100:.1f}%",
            "avg_CER": round(sum(r['CER'] for r in all_results)/n, 4),
            "avg_WER": round(sum(r['WER'] for r in all_results)/n, 4),
            "avg_NED": round(sum(r['NED'] for r in all_results)/n, 4),
            "avg_CLIP_score": round(sum(r['clip_score'] for r in all_results)/n, 4),
        }
        
        # Save detailed results to CSV
        if all_results:
            keys = all_results[0].keys()
            with open(output_csv, 'w', newline='') as f:
                writer = csv.DictWriter(f, fieldnames=keys)
                writer.writeheader()
                writer.writerows(all_results)
        
        print("\n=== EVALUATION SUMMARY ===")
        for k, v in summary.items():
            print(f"{k}: {v}")
        
        return summary

12. FastAPI Server

Kya karta hai

Pipeline ko ek REST API ke roop mein expose karta hai. Client ek POST request bhejta hai brief ke saath, Celery job queue mein task push hota hai, client job ID leta hai, phir poll karta hai result ke liye.

from fastapi import FastAPI, BackgroundTasks, HTTPException
from pydantic import BaseModel
from celery import Celery
import redis
import uuid
import json
 
app = FastAPI(title="Marketing Image Pipeline API")
 
# Redis connection — job store
r = redis.Redis(host="localhost", port=6379, db=0)
 
# Celery — job queue
celery_app = Celery("tasks", broker="redis://localhost:6379/0",
                              backend="redis://localhost:6379/0")
 
class GenerateRequest(BaseModel):
    brief: str                          # "Diwali poster for Lakme 50% off"
    lora_name: str = "text_rendering"   # which LoRA to use
    controlnet_scale: float = 0.75
    upscale: bool = True
 
class JobStatus(BaseModel):
    job_id: str
    status: str          # pending / processing / done / failed
    image_url: str = None
    error: str = None
 
@app.post("/generate", response_model=JobStatus)
async def generate_image(request: GenerateRequest):
    """
    Image generation job submit karo.
    
    Celery task queue mein push karo — async processing.
    Client ko turant job_id milta hai.
    
    Ye pattern isliye use karte hain kyunki image generation
    25-60 seconds leta hai. HTTP request itna wait nahi kar sakti.
    """
    job_id = str(uuid.uuid4())
    
    # Job Celery queue mein push karo
    celery_app.send_task(
        "tasks.generate_image_task",
        kwargs={
            "job_id": job_id,
            "brief": request.brief,
            "lora_name": request.lora_name,
            "controlnet_scale": request.controlnet_scale,
            "upscale": request.upscale
        }
    )
    
    # Initial status Redis mein store karo
    r.set(f"job:{job_id}", json.dumps({"status": "pending"}))
    r.expire(f"job:{job_id}", 86400)  # 24 hour expiry
    
    return JobStatus(job_id=job_id, status="pending")
 
@app.get("/status/{job_id}", response_model=JobStatus)
async def get_status(job_id: str):
    """
    Job ka status check karo.
    Client polling karta hai is endpoint pe.
    """
    data = r.get(f"job:{job_id}")
    
    if not data:
        raise HTTPException(status_code=404, detail="Job not found")
    
    job_data = json.loads(data)
    
    return JobStatus(
        job_id=job_id,
        status=job_data.get("status", "unknown"),
        image_url=job_data.get("image_url"),
        error=job_data.get("error")
    )
 
# Celery task — actual generation happens here
@celery_app.task(name="tasks.generate_image_task")
def generate_image_task(job_id, brief, lora_name, controlnet_scale, upscale):
    """
    Ye actual image generation karta hai.
    Alag process mein run hota hai (worker).
    """
    from scripts.pipeline.llm_decomposer import decompose_with_retry
    from scripts.pipeline.glyph_renderer import GlyphRenderer
    from scripts.pipeline.flux_generator import FluxGlyphPipeline
    from scripts.pipeline.quality_gate import QualityGate
    from scripts.pipeline.upscaler import Upscaler
    
    # Status update
    r.set(f"job:{job_id}", json.dumps({"status": "processing"}))
    
    try:
        # Stage 1: LLM decompose
        schema = decompose_with_retry(brief)
        if not schema:
            raise Exception("LLM decomposition failed")
        
        # Stage 2: Render glyph
        renderer = GlyphRenderer(fonts_dir="models/fonts")
        glyph, flux_prompt = renderer.render_from_json(json.dumps(schema))
        
        # Stage 3: Generate image
        pipeline = FluxGlyphPipeline(
            lora_path=f"models/loras/{lora_name}.safetensors"
        )
        quality_gate = QualityGate()
        
        image, passed, attempts = pipeline.generate_with_retry(
            prompt=flux_prompt,
            glyph_image=glyph,
            quality_gate_fn=quality_gate.evaluate,
            controlnet_scale=controlnet_scale
        )
        
        # Stage 4: Upscale
        if upscale:
            upscaler = Upscaler(scale=2)
            image = upscaler.upscale(image)
        
        # Save image
        output_path = f"outputs/{job_id}.jpg"
        image.save(output_path, "JPEG", quality=95)
        
        # Update status
        r.set(f"job:{job_id}", json.dumps({
            "status": "done",
            "image_url": f"/images/{job_id}.jpg",
            "quality_passed": passed,
            "attempts": attempts
        }))
    
    except Exception as e:
        r.set(f"job:{job_id}", json.dumps({
            "status": "failed",
            "error": str(e)
        }))

13. Full Pipeline Runner

Sab kuch ek jagah

"""
pipeline_runner.py — Complete pipeline test runner
 
Command line se single image generate karo:
python pipeline_runner.py --brief "Diwali sale poster 50% off gold theme" --output test_output.jpg
"""
 
import argparse
import json
from pathlib import Path
 
from scripts.pipeline.llm_decomposer import decompose_with_retry
from scripts.pipeline.glyph_renderer import GlyphRenderer
from scripts.pipeline.flux_generator import FluxGlyphPipeline
from scripts.pipeline.quality_gate import QualityGate
from scripts.pipeline.upscaler import Upscaler
 
def run_pipeline(
    brief: str,
    output_path: str,
    lora_path: str = None,
    controlnet_scale: float = 0.75,
    upscale: bool = True,
    device: str = "cuda"
):
    print(f"\n{'='*50}")
    print(f"PIPELINE START")
    print(f"Brief: {brief}")
    print(f"{'='*50}\n")
    
    # ── Stage 2: LLM Decompose ──────────────────────
    print("[Stage 2] LLM Decomposer...")
    schema = decompose_with_retry(brief)
    
    if not schema:
        print("ERROR: LLM failed to produce valid schema")
        return None
    
    print(f"Schema OK — {len(schema['texts'])} text blocks")
    print(f"Background: {schema['background_prompt'][:80]}...")
    
    # ── Stage 3: Glyph Render ───────────────────────
    print("\n[Stage 3] Glyph Renderer...")
    renderer = GlyphRenderer(fonts_dir="models/fonts")
    glyph, flux_prompt = renderer.render_from_json(json.dumps(schema))
    
    glyph.save("debug_glyph.png")  # debug ke liye save karo
    print(f"Glyph saved to debug_glyph.png")
    print(f"Flux prompt: {flux_prompt[:100]}...")
    
    # ── Stage 4+5: ControlNet + Flux ────────────────
    print("\n[Stage 4+5] ControlNet + Flux Generation...")
    pipeline = FluxGlyphPipeline(
        lora_path=lora_path,
        device=device
    )
    quality_gate = QualityGate()
    
    image, passed, attempts = pipeline.generate_with_retry(
        prompt=flux_prompt,
        glyph_image=glyph,
        quality_gate_fn=quality_gate.evaluate,
        controlnet_scale=controlnet_scale,
        max_attempts=3
    )
    
    print(f"Generation done — Quality gate: {'PASSED' if passed else 'FAILED'}, Attempts: {attempts}")
    
    # ── Stage 6: Upscale ────────────────────────────
    if upscale:
        print("\n[Stage 7] Real-ESRGAN Upscaler (2x)...")
        upscaler = Upscaler(scale=2)
        image = upscaler.upscale(image)
        print(f"Upscaled to {image.size}")
    
    # Save final image
    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    image.save(output_path, "JPEG", quality=95)
    print(f"\nFINAL IMAGE SAVED: {output_path}")
    print(f"Size: {image.size}")
    
    return image
 
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--brief", required=True, help="Marketing brief")
    parser.add_argument("--output", default="output.jpg", help="Output path")
    parser.add_argument("--lora", default=None, help="LoRA path")
    parser.add_argument("--scale", type=float, default=0.75, help="ControlNet scale")
    parser.add_argument("--no-upscale", action="store_true")
    parser.add_argument("--device", default="cuda")
    
    args = parser.parse_args()
    
    run_pipeline(
        brief=args.brief,
        output_path=args.output,
        lora_path=args.lora,
        controlnet_scale=args.scale,
        upscale=not args.no_upscale,
        device=args.device
    )

14. Dependencies and Install Commands

# ── Core ML ─────────────────────────────────────────
pip install torch==2.3.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
 
# ── Diffusion Models ────────────────────────────────
pip install diffusers==0.27.0
pip install transformers==4.40.0
pip install accelerate==0.30.0
pip install safetensors==0.4.3
 
# ── Image Processing ────────────────────────────────
pip install Pillow==10.3.0
pip install opencv-python==4.9.0.80
 
# ── OCR ─────────────────────────────────────────────
pip install easyocr==1.7.1
 
# ── CLIP ────────────────────────────────────────────
pip install git+https://github.com/openai/CLIP.git
 
# ── Upscaling ───────────────────────────────────────
pip install realesrgan
pip install basicsr
 
# ── Caption Generation ──────────────────────────────
pip install einops timm  # Florence-2 dependencies
 
# ── Evaluation ──────────────────────────────────────
pip install jiwer          # CER, WER calculation
pip install editdistance   # edit distance
 
# ── API and Queue ───────────────────────────────────
pip install fastapi==0.111.0
pip install uvicorn==0.29.0
pip install celery==5.3.6
pip install redis==5.0.4
 
# ── Scraping ────────────────────────────────────────
pip install requests==2.31.0
pip install beautifulsoup4==4.12.3
pip install gallery-dl  # Pinterest alternative
 
# ── Dataset Collection (LAION bulk) ─────────────────
pip install img2dataset==1.41.0
 
# ── LLM Runtime ─────────────────────────────────────
# Ollama — separate install
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull mistral
ollama pull llama3.1:8b

Running the Full Stack

# Terminal 1 — Ollama LLM server
ollama serve
 
# Terminal 2 — Redis
redis-server
 
# Terminal 3 — Celery worker
celery -A scripts.api.server worker --loglevel=info --concurrency=1
 
# Terminal 4 — FastAPI server
uvicorn scripts.api.server:app --host 0.0.0.0 --port 8000 --reload
 
# Test single image (no API, direct run)
python pipeline_runner.py \
  --brief "Diwali sale poster for Lakme, 50% off all products, gold red luxury" \
  --output test_output.jpg \
  --lora models/loras/text_rendering.safetensors

15. Common Errors and Fixes

ERROR: CUDA out of memory
FIX: pipe.enable_sequential_cpu_offload()
     Ya VRAM zyada wala GPU use karo (min 24GB for Flux)

ERROR: Ollama connection refused
FIX: ollama serve — run karo pehle
     Check: curl http://localhost:11434/api/tags

ERROR: Florence-2 trust_remote_code error
FIX: trust_remote_code=True parameter add karo
     AutoProcessor aur AutoModelForCausalLM dono mein

ERROR: EasyOCR first run slow
FIX: Pehli baar model download karta hai (~100MB)
     gpu=False set karo CPU pe

ERROR: LLM produces invalid JSON
FIX: decompose_with_retry use karo (3 attempts)
     Temperature 0.2 pe set karo aur zyada strict system prompt likho
     Ya directly schema example dene ki koshish karo few-shot mein

ERROR: ControlNet image size mismatch  
FIX: Glyph image aur generation size same rakhna zaroori hai
     preprocess_glyph() mein resize(1024, 1024) zaroor karo

ERROR: Real-ESRGAN tile artifacts
FIX: tile_pad=20 karo (10 se zyada)
     Large images ke liye tile=256 karo (512 se kam)

ERROR: Celery task not executing
FIX: Worker properly start hai? — celery worker command check karo
     Redis running hai? — redis-cli ping
     Task name exact match karna chahiye — tasks.generate_image_task

Scripting Knowledge Document v1.0
Project: AI Marketing Image Pipeline

Django MOC

2026-09-02T05:29:24.926Z

🗺️ Django Map of Content

Welcome to the syllabus and index for Django.

📄 Notes

djangointernals

djangointernals

2026-09-02T05:29:24.926Z

The Django RESTaurant: From HTTP Order to a JSON Meal

Django REST Framework feels like magic. You write a few lines of code, and a powerful, browsable API appears. But what’s really happening under the hood? Let’s find out.

If you’ve ever used Django REST Framework (DRF), you’ve felt it. You define a Model, a Serializer, and a ViewSet, wire them up with a Router, and suddenly you have a full CRUD API. It feels too easy, almost like you’re cheating.

To demystify the process, let’s think of your web application as a high-end restaurant. Django is the building, the kitchen, and the staff. An API request is a customer’s order. Let’s follow a single GET request from the moment it arrives at the front door to the moment a JSON meal is served.

1. The Maître d’ — The URL Resolver

A customer (an HTTP request) arrives at the restaurant’s address. The first person they meet is the Maître d’ (urls.py). Their job is to look at the customer’s reservation (the URL path, like /api/articles/) and direct them to the correct section of the restaurant.

In Django, this process starts in your project’s root core/urls.py. This file is the main reservation book. For our API, it sees the /api/ prefix and hands the customer off to a specialized manager: the DRF Router. The router knows exactly which chef is responsible for the /articles/ table.

2. The Head Chef — The `ViewSet` (The Brains of the Operation)

The Head Chef (ViewSet) is the central decision-maker in the kitchen. They don’t chop every vegetable, but they hold the master plan. When the order for “all articles” arrives, the Head Chef knows to execute the list() recipe.

This is where the “viewpoint determines the logic of the project.” The ViewSet is the brain, orchestrating the entire workflow by defining three critical things:

What data to work with? (queryset = Article.objects.all()): The chef decides which section of the pantry (which database model) the ingredients will come from.
How should the data be formatted? (serializer_class = ArticleSerializer): The chef points to the specific station responsible for plating and quality control.
Who is allowed to place this order? (permission_classes = [...]): The chef checks the customer’s credentials before starting any work.

The ViewSet contains no data itself; it is pure logic and orchestration. It directs the flow of work.

3. The Recipe Book & The Pantry — The Model & The Database

Following the Head Chef’s instructions, a cook is sent to the pantry (the database) with a shopping list. This list is based on the official recipe book (the Article model in articles/models.py), which defines exactly what an “article” ingredient is made of—a title, some content, an author, etc.

This is the Django ORM (Object-Relational Mapper) in action. It translates the chef’s command (Article.objects.all()) into a raw SQL query, retrieves the data from the database, and brings it back to the kitchen as a collection of Python objects.

4. The Universal Translator & Quality Control — The `Serializer`

This is the most critical station in our RESTaurant. The Serializer acts as both a Universal Translator and a Quality Control Inspector.

As a Translator (for outgoing orders): When the Head Chef sends the raw ingredients (Python Article objects), the translator’s job is to prepare them for the outside world. It follows a strict blueprint (the fields in its Meta class) to arrange the data into a standardized JSON “dish.” It can nest data, like putting author details inside the article dish, to create the rich, complex JSON that allows different projects to communicate seamlessly. The final JSON meal is a universal language that any client—a web frontend, a mobile app, or another server—can understand.

As a Quality Control Inspector (for incoming orders): When a new delivery of ingredients arrives (POST data), this station inspects it first. Does the data match the recipe’s requirements? Is the title a string? Is the content included? If anything is wrong, the delivery is rejected immediately with a 400 Bad Request validation error. This prevents bad data from ever entering your database.

The Serializer doesn’t contain application logic; it is a strict and powerful data transformation and validation tool.

5. The Waiter — The Renderer and The Final Dish

The finished dish is ready. A waiter (Renderer) picks it up. But how do they serve it? They look at the customer.

Is it a person in the dining room using a web browser? The waiter serves the beautiful plate on a fancy platter. This is the BrowsableAPIRenderer, which wraps the JSON in the helpful HTML interface we all know and love.
Is it a takeout order from a script or mobile app? The waiter puts it in a simple, efficient to-go box. This is the JSONRenderer, which serves the raw, unadulterated JSON data.

This process, called Content Negotiation, allows the same endpoint to serve both humans and machines effectively.

The Meal is Served

So, the next time you write a ViewSet and a Serializer and see your API spring to life, you’ll know it’s not magic. It’s the result of a highly efficient, well-structured kitchen staff working in perfect harmony.

Understanding this flow from request to response demystifies the framework and empowers you to step in and customize any part of the recipe.

Happy cooking!

45_Mobile_IP

2026-09-02T05:29:24.810Z

45. Mobile IP

Introduction to Mobile IP

Mobile IP is a communication protocol that allows a mobile device to move from one network to another while maintaining its permanent IP address. This is in contrast to the traditional IP routing, which is based on the assumption that a device’s IP address is fixed. Mobile IP is an open standard defined by the Internet Engineering Task Force (IETF) in RFC 2002.

How Mobile IP Works

Mobile IP works by using two IP addresses for a mobile device: a permanent IP address and a care-of address.

Permanent IP address: The permanent IP address is the IP address that is assigned to the mobile device when it is on its home network.
Care-of address: The care-of address is the IP address that is assigned to the mobile device when it is on a foreign network.

When a mobile device moves from its home network to a foreign network, it registers with a foreign agent on the foreign network. The foreign agent then sends a registration request to the home agent on the home network. The home agent then creates a tunnel to the foreign agent. All packets that are sent to the mobile device’s permanent IP address are then forwarded through this tunnel to the foreign agent. The foreign agent then forwards the packets to the mobile device.

Mobile IP Components

There are three main components of Mobile IP:

Mobile Node (MN): The mobile node is the mobile device that is moving from one network to another.
Home Agent (HA): The home agent is a router on the mobile node’s home network. The home agent is responsible for forwarding packets to the mobile node when it is on a foreign network.
Foreign Agent (FA): The foreign agent is a router on the foreign network that the mobile node is visiting. The foreign agent is responsible for forwarding packets from the home agent to the mobile node.

Mobile IP vs. DHCP

Mobile IP and DHCP are two different protocols that are used to assign IP addresses to devices. The main difference between Mobile IP and DHCP is that Mobile IP allows a mobile device to maintain its permanent IP address while it is moving from one network to another, while DHCP does not.

Advantages of Mobile IP

Mobility: Mobile IP allows a mobile device to move from one network to another while maintaining its permanent IP address.
Transparency: Mobile IP is transparent to the applications that are running on the mobile device.
Scalability: Mobile IP is a scalable protocol that can be used to support a large number of mobile devices.

Disadvantages of Mobile IP

Complexity: Mobile IP can be complex to set up and manage.
Overhead: Mobile IP adds overhead to the packets that are sent to the mobile device.
Security: Mobile IP can be a security risk.

46_Voice_over_IP

2026-09-02T05:29:24.810Z

46. Voice over IP (VoIP)

Introduction to VoIP

Voice over IP (VoIP) is a technology that allows you to make voice calls using a broadband Internet connection instead of a regular (or analog) phone line. Some VoIP services may only allow you to call other people using the same service, but others may allow you to call anyone who has a telephone number - including local, long distance, mobile, and international numbers.

How VoIP Works

VoIP works by converting your voice into a digital signal that can be sent over the internet. The digital signal is then sent to the person you are calling, where it is converted back into an analog signal that can be heard through their phone.

VoIP Protocols

There are many different VoIP protocols, but some of the most common protocols are:

H.323: H.323 is a standard that was developed by the International Telecommunication Union (ITU). H.323 is a complex standard that is not widely used today.
Session Initiation Protocol (SIP): SIP is a standard that was developed by the Internet Engineering Task Force (IETF). SIP is a simpler standard than H.323 and is the most widely used VoIP protocol today.
Real-time Transport Protocol (RTP): RTP is a protocol that is used to transport real-time data, such as audio and video. RTP is often used in conjunction with SIP.
Real-time Transport Control Protocol (RTCP): RTCP is a protocol that is used to monitor the quality of service of an RTP session. RTCP is used to provide feedback to the sender about the quality of the transmission.

VoIP vs. PSTN

VoIP and the public switched telephone network (PSTN) are two different telephone network systems. The main difference between VoIP and the PSTN is the way that they transmit signals. VoIP transmits signals digitally, while the PSTN transmits signals in analog form. This makes VoIP more reliable and provides better voice quality than the PSTN.

Advantages of VoIP

Cost: VoIP can be cheaper than the PSTN, especially for long-distance and international calls.
Features: VoIP offers a number of features that are not available on the PSTN, such as caller ID, call waiting, and voicemail.
Portability: VoIP can be used anywhere there is a broadband Internet connection.

Disadvantages of VoIP

Reliability: VoIP is not as reliable as the PSTN. If your internet connection goes down, you will not be able to make or receive calls.
Emergency calls: VoIP may not be able to connect to 911 or other emergency services.
Security: VoIP can be a security risk.

47_Quality_of_Service

2026-09-02T05:29:24.810Z

47. Quality of Service (QoS)

Introduction to QoS

Quality of Service (QoS) is the ability of a network to provide a certain level of performance to a particular type of traffic. QoS is important for real-time applications, such as voice and video, which are sensitive to delay and jitter. QoS is also important for mission-critical applications, which require a high level of reliability.

QoS Parameters

There are a number of different QoS parameters, but some of the most common parameters are:

Bandwidth: Bandwidth is the amount of data that can be transmitted over a network in a given amount of time. Bandwidth is measured in bits per second (bps).
Latency: Latency is the amount of time that it takes for a packet to travel from the source to the destination. Latency is measured in milliseconds (ms).
Jitter: Jitter is the variation in the delay of received packets. Jitter is measured in milliseconds (ms).
Packet loss: Packet loss is the percentage of packets that are lost in transit. Packet loss is measured as a percentage.

QoS Mechanisms

There are a number of different QoS mechanisms that can be used to provide a certain level of performance to a particular type of traffic. These mechanisms include:

Classification and marking: Classification and marking is the process of identifying and marking different types of traffic. This allows the network to give different levels of priority to different types of traffic.
Congestion management: Congestion management is the process of managing the traffic on a network to prevent congestion. Congestion can cause delay, jitter, and packet loss.
Congestion avoidance: Congestion avoidance is the process of preventing congestion from occurring in the first place. This can be done by using a variety of techniques, such as random early detection (RED).
Shaping and policing: Shaping and policing is the process of controlling the rate at which traffic is sent into the network. This can be used to prevent a single user from consuming all of the bandwidth.
Link efficiency: Link efficiency is the process of using the available bandwidth as efficiently as possible. This can be done by using a variety of techniques, such as compression and fragmentation.

QoS Models

There are two main QoS models:

Integrated Services (IntServ): IntServ is a QoS model that reserves resources on a network for a particular flow of data. IntServ is a connection-oriented model, which means that a connection must be established before data can be transmitted.
Differentiated Services (DiffServ): DiffServ is a QoS model that uses a priority system to give different levels of service to different types of traffic. DiffServ is a connectionless model, which means that data can be transmitted without establishing a connection.

QoS vs. Class of Service (CoS)

QoS and Class of Service (CoS) are two different but related concepts. QoS is the ability of a network to provide a certain level of performance to a particular type of traffic. CoS is a method of classifying and marking traffic so that it can be given a certain level of priority.

48_Network_Monitoring

2026-09-02T05:29:24.810Z

48. Network Monitoring

Introduction to Network Monitoring

Network monitoring is the process of continuously monitoring a computer network for slow or failing components and that notifies the network administrator (via email, SMS or other alarms) in case of outages. Network monitoring is part of network management.

Network Monitoring Tools

There are many different network monitoring tools available. Some of the most popular network monitoring tools are:

Nagios: Nagios is an open-source network monitoring tool. It is one of the most popular network monitoring tools in the world.
Zabbix: Zabbix is an open-source network monitoring tool. It is another popular network monitoring tool.
SolarWinds: SolarWinds is a commercial network monitoring tool. It is a very powerful tool that can be used to monitor a wide variety of devices.
PRTG: PRTG is a commercial network monitoring tool. It is a very easy-to-use tool that can be used to monitor a wide variety of devices.

Network Monitoring Protocols

There are many different network monitoring protocols, but some of the most common protocols are:

Simple Network Management Protocol (SNMP): SNMP is a protocol that is used to manage devices on a network. SNMP is a simple protocol that is easy to use.
Internet Control Message Protocol (ICMP): ICMP is a protocol that is used to send error messages and operational information. For example, the ping command uses ICMP to test the connectivity between two devices.
NetFlow: NetFlow is a protocol that is used to collect IP traffic information. NetFlow can be used to monitor the traffic on a network and to identify potential problems.

Network Monitoring Best Practices

There are a number of different network monitoring best practices, but some of the most important best practices are:

Monitor the right things: You should monitor the things that are important to your business. This includes the availability of your network, the performance of your network, and the security of your network.
Set thresholds: You should set thresholds for the things that you are monitoring. This will allow you to be notified when there is a problem.
Use a variety of tools: You should use a variety of tools to monitor your network. This will give you a more complete picture of the health of your network.
Automate your monitoring: You should automate your monitoring as much as possible. This will free up your time so that you can focus on other things.
Review your monitoring data regularly: You should review your monitoring data regularly. This will help you to identify trends and to prevent problems from occurring.

49_Network_Virtualization

2026-09-02T05:29:24.810Z

49. Network Virtualization

Introduction to Network Virtualization

Network virtualization is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, called a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization.

Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to the software containers on a single system.

Types of Network Virtualization

There are two main types of network virtualization:

External network virtualization: External network virtualization combines many networks, or parts of networks, into a virtual unit. This can be done by using a variety of techniques, such as VLANs, VPNs, and MPLS.
Internal network virtualization: Internal network virtualization provides network-like functionality to the software containers on a single system. This can be done by using a variety of techniques, such as Xen, KVM, and Hyper-V.

VLANs

A VLAN is a virtual LAN. A VLAN is a logical grouping of devices on a network. VLANs can be used to segment a network into smaller, more manageable parts. VLANs can also be used to improve security by isolating the different segments of the network from each other.

VPNs

A VPN is a virtual private network. A VPN is a network that uses a public network, such as the internet, to connect to a private network. VPNs can be used to securely connect to a remote network.

MPLS

MPLS is a multiprotocol label switching. MPLS is a routing technique that directs data from one node to the next based on short path labels rather than long network addresses, avoiding complex lookups in a routing table.

Advantages of Network Virtualization

Flexibility: Network virtualization can make a network more flexible. This is because it allows you to create and manage virtual networks without having to change the physical network.
Agility: Network virtualization can make a network more agile. This is because it allows you to quickly and easily provision and de-provision virtual networks.
Cost savings: Network virtualization can save you money. This is because it allows you to consolidate your network hardware and to reduce your power consumption.

Disadvantages of Network Virtualization

Complexity: Network virtualization can be complex to set up and manage.
Security: Network virtualization can be a security risk.
Performance: Network virtualization can have a negative impact on performance.

50_The_Future_of_Networking

2026-09-02T05:29:24.810Z

50. The Future of Networking

Introduction to the Future of Networking

The future of networking is constantly evolving. New technologies are being developed all the time that are changing the way that we connect to the internet and to each other. In this article, we will take a look at some of the most important trends that are shaping the future of networking.

Trends in Networking

Some of the most important trends in networking include:

The Internet of Things (IoT): The IoT is the network of physical devices, vehicles, home appliances and other items embedded with electronics, software, sensors, actuators, and connectivity which enables these objects to connect and exchange data. The IoT is expected to have a major impact on the future of networking.
Software-Defined Networking (SDN): SDN is an emerging network architecture where network control is decoupled from forwarding and is directly programmable. SDN is expected to make networks more flexible, agile, and programmable.
Network Function Virtualization (NFV): NFV is a network architecture concept that uses the technologies of IT virtualization to virtualize entire classes of network node functions into building blocks that may be connected, or chained, together to create communication services. NFV is expected to make networks more flexible, agile, and cost-effective.
5G: 5G is the next generation of mobile networking. 5G is expected to provide faster speeds, lower latency, and more capacity than 4G.
Artificial Intelligence (AI): AI is being used in a variety of ways to improve the performance of networks. For example, AI can be used to automate tasks, to identify and fix problems, and to optimize the flow of traffic.

The Future of the Internet

The future of the internet is bright. The internet is constantly evolving and becoming more powerful. The trends that we have discussed in this article are just a few of the things that are shaping the future of the internet. As new technologies are developed, the internet will continue to change and to become even more a part of our lives.

The Future of Networking Jobs

The future of networking jobs is also bright. As networks become more complex, there will be a growing demand for skilled network engineers. Network engineers will need to have a strong understanding of the latest networking technologies. They will also need to be able to troubleshoot and solve complex problems.

If you are interested in a career in networking, then now is a great time to get started. There are many different resources available to help you learn about networking. You can find books, articles, and online courses on networking. You can also get certified in networking by taking a certification exam.

Computer Networks MOC

2026-09-02T05:29:24.810Z

Computer Networks Map of Content

Welcome to the syllabus and index for Computer Networks.

Notes

24_Network_Function_Virtualization

2026-09-02T05:29:24.809Z

24. Network Function Virtualization (NFV)

Introduction to NFV

Network function virtualization (NFV) is a network architecture concept that uses the technologies of IT virtualization to virtualize entire classes of network node functions into building blocks that may be connected, or chained, together to create communication services. NFV is complementary to Software-Defined Networking (SDN).

How NFV Works

NFV works by decoupling network functions, such as firewalls, load balancers, and routers, from dedicated hardware devices. These network functions are then run as software on standard x86 servers. This allows for a more flexible and agile network.

NFV Architecture

The NFV architecture is made up of three main components:

Virtualized Network Functions (VNFs): VNFs are the software implementations of network functions. VNFs can be anything from a simple firewall to a complex load balancer.
NFV Infrastructure (NFVI): The NFVI is the hardware and software that the VNFs run on. The NFVI includes the servers, storage, and networking equipment.
NFV Management and Orchestration (MANO): The MANO is responsible for managing the VNFs and the NFVI. The MANO is responsible for instantiating, monitoring, and scaling the VNFs.

NFV vs. SDN

NFV and SDN are complementary technologies. NFV is focused on virtualizing network functions, while SDN is focused on centralizing network control. NFV and SDN can be used together to create a more flexible, agile, and programmable network.

Advantages of NFV

Cost Savings: NFV can help to reduce costs by decoupling network functions from dedicated hardware devices. This allows for the use of standard x86 servers, which are less expensive than dedicated hardware devices.
Agility: NFV makes the network more agile. This means that you can make changes to the network more quickly and easily.
Flexibility: NFV makes the network more flexible. This means that you can create new network services more quickly and easily.
Innovation: NFV is a new and innovative technology. It has the potential to revolutionize the way that we build and manage networks.

Disadvantages of NFV

Complexity: NFV can be complex to set up and manage.
Security: NFV can be a security risk. If a VNF is compromised, then the entire network can be compromised.
Performance: NFV can have a negative impact on performance. This is because the VNFs are running on standard x86 servers, which are not as powerful as dedicated hardware devices.

25_Internet_of_Things

2026-09-02T05:29:24.809Z

25. Internet of Things (IoT)

Introduction to IoT

The Internet of Things (IoT) is the network of physical devices, vehicles, home appliances and other items embedded with electronics, software, sensors, actuators, and connectivity which enables these objects to connect and exchange data. Each thing is uniquely identifiable through its embedded computing system but is able to inter-operate within the existing Internet infrastructure.

How IoT Works

IoT devices are typically connected to the internet through a wireless network, such as Wi-Fi or Bluetooth. Once connected, the devices can be controlled and monitored remotely. For example, you could use your smartphone to turn on the lights in your home, or you could use a web browser to view the temperature in your refrigerator.

IoT Architecture

The IoT architecture is made up of four layers:

Sensing Layer: The sensing layer is where the IoT devices reside. The IoT devices are responsible for collecting data from the physical world.
Network Layer: The network layer is responsible for connecting the IoT devices to the internet.
Data Processing Layer: The data processing layer is responsible for processing the data that is collected by the IoT devices.
Application Layer: The application layer is where the IoT applications and services reside. The applications and services can be anything from a simple web server to a complex video streaming service.

IoT Protocols

There are many different IoT protocols, but some of the most common protocols are:

Message Queuing Telemetry Transport (MQTT): MQTT is a lightweight messaging protocol that is designed for constrained devices. MQTT is often used for IoT applications.
Constrained Application Protocol (CoAP): CoAP is a lightweight application protocol that is designed for constrained devices. CoAP is often used for IoT applications.
Advanced Message Queuing Protocol (AMQP): AMQP is a messaging protocol that is designed for enterprise applications. AMQP is often used for IoT applications.
Data Distribution Service (DDS): DDS is a data-centric messaging protocol that is designed for real-time applications. DDS is often used for IoT applications.

Advantages of IoT

Convenience: IoT can make our lives more convenient. For example, we can use our smartphones to control the lights in our homes, or we can use a web browser to view the temperature in our refrigerators.
Efficiency: IoT can help us to be more efficient. For example, we can use IoT devices to monitor our energy consumption and to make adjustments to save money.
Safety: IoT can help us to be safer. For example, we can use IoT devices to monitor our homes for intruders, or we can use IoT devices to track our children’s location.

Disadvantages of IoT

Security: IoT can be a security risk. If an IoT device is compromised, then it could be used to launch an attack on our network.
Privacy: IoT can be a privacy risk. If an IoT device is compromised, then it could be used to collect personal information about us.
Complexity: IoT can be complex to set up and manage.

26_Cloud_Computing

2026-09-02T05:29:24.809Z

26. Cloud Computing

Introduction to Cloud Computing

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing is a major trend in information technology.

How Cloud Computing Works

Cloud computing works by providing a way for users to access computing resources over the internet. These computing resources can be anything from a simple web server to a complex video streaming service. The user does not need to own or manage the computing resources. Instead, the user pays a monthly fee to a cloud provider.

Cloud Computing Service Models

There are three main cloud computing service models:

Infrastructure as a Service (IaaS): IaaS provides users with access to computing resources, such as servers, storage, and networking. The user is responsible for managing the operating system, applications, and data.
Platform as a Service (PaaS): PaaS provides users with a platform for developing, deploying, and managing applications. The user is responsible for managing the applications and data.
Software as a Service (SaaS): SaaS provides users with access to applications over the internet. The user is not responsible for managing the applications or data.

Cloud Computing Deployment Models

There are four main cloud computing deployment models:

Public Cloud: A public cloud is a cloud that is owned and operated by a third-party cloud provider. Public clouds are available to the general public.
Private Cloud: A private cloud is a cloud that is owned and operated by a single organization. Private clouds are not available to the general public.
Hybrid Cloud: A hybrid cloud is a combination of a public cloud and a private cloud. Hybrid clouds allow organizations to take advantage of the benefits of both public and private clouds.
Community Cloud: A community cloud is a cloud that is shared by a group of organizations with a common interest. Community clouds are not available to the general public.

Advantages of Cloud Computing

Cost Savings: Cloud computing can help to reduce costs by eliminating the need to purchase and maintain your own computing resources.
Scalability: Cloud computing is very scalable. You can easily add or remove computing resources as needed.
Agility: Cloud computing makes your IT more agile. You can make changes to your IT more quickly and easily.
Reliability: Cloud computing is very reliable. Cloud providers have a team of experts who are responsible for keeping the cloud up and running.

Disadvantages of Cloud Computing

Security: Cloud computing can be a security risk. If your cloud provider is compromised, then your data could be at risk.
Privacy: Cloud computing can be a privacy risk. If your cloud provider is compromised, then your data could be at risk.
Lock-in: Cloud computing can lead to lock-in. If you use a proprietary cloud platform, then it can be difficult to switch to another cloud provider.

27_Network_Management

2026-09-02T05:29:24.809Z

27. Network Management

Introduction to Network Management

Network management is the process of managing a network for performance, reliability, and security. Network management is a broad topic that covers a wide range of activities, from monitoring the network to troubleshooting problems.

Network Management Architecture

The network management architecture is made up of three main components:

Managed Devices: Managed devices are the devices that are being managed. Managed devices can be anything from a simple switch to a complex router.
Network Management Station (NMS): The NMS is the software that is used to manage the network. The NMS is responsible for collecting data from the managed devices, for analyzing the data, and for taking action to correct problems.
Network Management Protocol: The network management protocol is the protocol that is used to communicate between the NMS and the managed devices. The most common network management protocol is the Simple Network Management Protocol (SNMP).

Simple Network Management Protocol (SNMP)

SNMP is a protocol that is used to manage devices on a network. SNMP is a simple protocol that is easy to use. SNMP is based on the concept of a manager and an agent.

Manager: The manager is the software that is used to manage the network. The manager is responsible for sending requests to the agents and for receiving responses from the agents.
Agent: The agent is the software that is running on the managed devices. The agent is responsible for collecting data from the managed devices and for sending the data to the manager.

SNMP uses a set of commands to manage devices on a network. The most common SNMP commands are:

GET: The GET command is used to retrieve a value from a managed device.
SET: The SET command is used to set a value on a managed device.
TRAP: The TRAP command is used to send a notification to the manager.

Network Management Functions

There are five main network management functions:

Fault Management: Fault management is the process of detecting, isolating, and correcting faults in the network.
Configuration Management: Configuration management is the process of configuring the network devices.
Accounting Management: Accounting management is the process of tracking the usage of the network resources.
Performance Management: Performance management is the process of monitoring the performance of the network and for taking action to improve the performance.
Security Management: Security management is the process of securing the network from unauthorized access.

Network Management Tools

There are many different network management tools available. Some of the most popular network management tools are:

Nagios: Nagios is an open-source network monitoring tool.
Zabbix: Zabbix is an open-source network monitoring tool.
SolarWinds: SolarWinds is a commercial network management tool.
PRTG: PRTG is a commercial network monitoring tool.

28_Network_Troubleshooting

2026-09-02T05:29:24.809Z

28. Network Troubleshooting

Introduction to Network Troubleshooting

Network troubleshooting is the process of identifying, diagnosing, and resolving problems on a computer network. Network troubleshooting is a systematic process that involves a series of steps. The goal of network troubleshooting is to restore the network to a normal operational state as quickly as possible.

Network Troubleshooting Methodology

There are many different network troubleshooting methodologies, but one of the most common methodologies is the top-down methodology. The top-down methodology starts at the application layer and works its way down to the physical layer. The steps in the top-down methodology are as follows:

Identify the problem: The first step is to identify the problem. This can be done by talking to the users who are experiencing the problem.
Gather information: The next step is to gather information about the problem. This can be done by using a variety of tools, such as ping, traceroute, and nslookup.
Develop a hypothesis: The next step is to develop a hypothesis about the cause of the problem. This can be done by analyzing the information that you have gathered.
Test the hypothesis: The next step is to test the hypothesis. This can be done by making a change to the network and observing the results.
Implement a solution: If the hypothesis is correct, then the next step is to implement a solution. This may involve making a change to the network configuration or replacing a faulty device.
Verify the solution: The final step is to verify the solution. This can be done by testing the network to make sure that the problem has been resolved.

Common Network Problems

There are many different common network problems, but some of the most common problems are:

No connectivity: This is the most common network problem. It can be caused by a variety of factors, such as a faulty cable, a misconfigured network device, or a power outage.
Slow performance: This is another common network problem. It can be caused by a variety of factors, such as a congested network, a faulty network device, or a virus.
Intermittent connectivity: This is a less common network problem, but it can be very frustrating. It can be caused by a variety of factors, such as a loose cable, a faulty network device, or a software bug.

Network Troubleshooting Tools

There are many different network troubleshooting tools available. Some of the most popular network troubleshooting tools are:

Ping: Ping is a command-line tool that is used to test the connectivity between two devices.
Traceroute: Traceroute is a command-line tool that is used to trace the path that a packet takes from the source to the destination.
Nslookup: Nslookup is a command-line tool that is used to query the Domain Name System (DNS).
Wireshark: Wireshark is a graphical network protocol analyzer. It can be used to capture and analyze network traffic.

29_Network_Design

2026-09-02T05:29:24.809Z

29. Network Design

Introduction to Network Design

Network design is the process of planning and designing a computer network. Network design is a complex process that involves a variety of factors, such as the size of the network, the number of users, the types of applications that will be used, and the budget. The goal of network design is to create a network that is reliable, scalable, and secure.

Network Design Models

There are many different network design models, but one of the most common models is the hierarchical network design model. The hierarchical network design model is a three-layer model that consists of the following layers:

Core Layer: The core layer is the backbone of the network. It is responsible for providing high-speed connectivity between the different parts of the network.
Distribution Layer: The distribution layer is responsible for connecting the access layer to the core layer. It is also responsible for providing policy-based connectivity.
Access Layer: The access layer is responsible for connecting the end users to the network.

Network Design Principles

There are many different network design principles, but some of the most important principles are:

Hierarchy: A hierarchical network design is a network design that is divided into layers. This makes the network more scalable and easier to manage.
Modularity: A modular network design is a network design that is made up of modules. This makes the network more flexible and easier to expand.
Resiliency: A resilient network design is a network design that is able to withstand failures. This is achieved by using redundant devices and links.
Security: A secure network design is a network design that is protected from unauthorized access. This is achieved by using firewalls, intrusion detection systems, and other security devices.

Network Design Process

The network design process is a systematic process that involves a series of steps. The steps in the network design process are as follows:

Requirements Gathering: The first step is to gather the requirements for the network. This can be done by talking to the users who will be using the network.
Analysis: The next step is to analyze the requirements. This will help you to determine the size of the network, the number of users, the types of applications that will be used, and the budget.
Design: The next step is to design the network. This will involve creating a logical and physical design of the network.
Implementation: The next step is to implement the network. This will involve installing the network devices and configuring the network.
Testing: The final step is to test the network. This will involve testing the network to make sure that it is working properly.

Network Design Tools

There are many different network design tools available. Some of the most popular network design tools are:

Cisco Packet Tracer: Cisco Packet Tracer is a network simulation tool that can be used to design and test networks.
GNS3: GNS3 is a network simulation tool that can be used to design and test networks.
NetSim: NetSim is a network simulation tool that can be used to design and test networks.

30_Network_Cabling

2026-09-02T05:29:24.809Z

30. Network Cabling

Introduction to Network Cabling

Network cabling is the process of installing and connecting cables to create a computer network. Network cabling is a critical part of any network, as it is the physical medium that carries the data. There are many different types of network cables, and each type has its own advantages and disadvantages.

Types of Network Cables

There are many different types of network cables, but some of the most common types are:

Twisted-Pair Cable: Twisted-pair cable is the most common type of network cable. It is made up of two insulated copper wires that are twisted together. Twisted-pair cable is inexpensive and easy to install.
Coaxial Cable: Coaxial cable is a type of network cable that is used for high-speed data transmission. It is made up of a central copper core, an insulator, a braided metal shield, and an outer cover. Coaxial cable is more expensive than twisted-pair cable, but it is also more resistant to interference.
Fiber-Optic Cable: Fiber-optic cable is a type of network cable that uses light to transmit data. It is made up of a thin strand of glass or plastic that can carry light for long distances. Fiber-optic cable is the most expensive type of network cable, but it is also the fastest and most secure.

Network Cable Standards

There are many different network cable standards, but some of the most common standards are:

Ethernet: Ethernet is the most common network cable standard. It is a family of standards that define the physical layer and data link layer of a computer network.
Token Ring: Token Ring is a network cable standard that was developed by IBM. It is a token-passing standard that is used in a ring topology.
Fiber Distributed Data Interface (FDDI): FDDI is a network cable standard that is used for high-speed data transmission. It is a token-passing standard that is used in a dual-ring topology.

Network Cable Installation

Network cable installation is a complex process that should be performed by a qualified technician. The steps in network cable installation are as follows:

Planning: The first step is to plan the network cable installation. This will involve determining the layout of the network and the types of cables that will be used.
Installation: The next step is to install the network cables. This will involve running the cables through the walls and ceilings.
Termination: The next step is to terminate the network cables. This will involve connecting the cables to the network devices.
Testing: The final step is to test the network cables. This will involve testing the cables to make sure that they are working properly.

Network Cable Troubleshooting

There are many different network cable problems, but some of the most common problems are:

Broken cable: A broken cable is a cable that has been physically damaged. This can be caused by a variety of factors, such as a sharp object, a rodent, or a power surge.
Loose connection: A loose connection is a connection that is not secure. This can be caused by a variety of factors, such as a faulty connector, a misconfigured network device, or a power outage.
Interference: Interference is a type of noise that can disrupt the signal on a network cable. This can be caused by a variety of factors, such as other electronic devices, power lines, or fluorescent lights.

31_Ethernet

2026-09-02T05:29:24.809Z

31. Ethernet

Introduction to Ethernet

Ethernet is a family of computer networking technologies commonly used in local area networks (LAN), metropolitan area networks (MAN) and wide area networks (WAN). It was commercially introduced in 1980 and first standardized in 1983 as IEEE 802.3. Ethernet has since been refined to support higher bit rates, a greater number of nodes, and longer link distances, but retains much backward compatibility. Over time, Ethernet has largely replaced competing wired LAN technologies such as Token Ring, FDDI and ARCNET.

Ethernet Standards

There are many different Ethernet standards, but some of the most common standards are:

10BASE-T: 10BASE-T is an Ethernet standard that uses twisted-pair cable and has a data rate of 10 Mbps.
100BASE-TX: 100BASE-TX is an Ethernet standard that uses twisted-pair cable and has a data rate of 100 Mbps.
1000BASE-T: 1000BASE-T is an Ethernet standard that uses twisted-pair cable and has a data rate of 1 Gbps.
10GBASE-T: 10GBASE-T is an Ethernet standard that uses twisted-pair cable and has a data rate of 10 Gbps.

Ethernet Frame

An Ethernet frame is a data packet that is transmitted over an Ethernet network. An Ethernet frame has the following format:

Preamble: The preamble is a 7-byte field that is used to synchronize the receiver with the sender.
Start Frame Delimiter (SFD): The SFD is a 1-byte field that indicates the start of the frame.
Destination MAC Address: The destination MAC address is a 6-byte field that contains the MAC address of the destination device.
Source MAC Address: The source MAC address is a 6-byte field that contains the MAC address of the source device.
Length/Type: The length/type field is a 2-byte field that indicates the length of the data field or the type of the protocol.
Data: The data field is a variable-length field that contains the data that is being transmitted.
Frame Check Sequence (FCS): The FCS is a 4-byte field that contains a checksum of the frame. The FCS is used to detect errors in the frame.

Carrier Sense Multiple Access with Collision Detection (CSMA/CD)

CSMA/CD is a media access control method used by Ethernet networks. CSMA/CD works as follows:

A device that wants to transmit data first listens to the network to see if it is busy.
If the network is not busy, the device starts to transmit data.
While the device is transmitting data, it continues to listen to the network to see if a collision has occurred.
If a collision occurs, the device stops transmitting data and sends a jam signal.
The device then waits a random amount of time before trying to transmit data again.

Ethernet Switches

An Ethernet switch is a network device that connects multiple Ethernet devices together. An Ethernet switch works by learning the MAC addresses of the devices that are connected to it. When a frame is received, the switch looks at the destination MAC address and forwards the frame to the port that is connected to the destination device. This is in contrast to an Ethernet hub, which forwards the frame to all of the ports.

32_Token_Ring

2026-09-02T05:29:24.809Z

32. Token Ring

Introduction to Token Ring

Token Ring is a local area network (LAN) technology in which all stations are connected in a ring and a token-passing scheme is used to control access to the medium. The token is a special bit pattern that circulates around the ring. A station that wants to transmit data must wait until it receives the token. The station then changes the token to a frame, appends the data to the frame, and sends the frame to the next station in the ring. The frame circulates around the ring until it reaches the destination station. The destination station copies the data from the frame and then sends the frame back to the source station. The source station then removes the data from the frame and changes the frame back to a token.

Token Ring Standards

Token Ring was originally developed by IBM in the 1980s. It was later standardized by the IEEE as IEEE 802.5. The IEEE 802.5 standard defines two types of Token Ring: 4 Mbps and 16 Mbps.

Token Ring Frame

A Token Ring frame has the following format:

Start Delimiter (SD): The SD is a 1-byte field that indicates the start of the frame.
Access Control (AC): The AC is a 1-byte field that contains the token bit, the monitor bit, and the priority bits.
Frame Control (FC): The FC is a 1-byte field that indicates the type of the frame.
Destination MAC Address: The destination MAC address is a 6-byte field that contains the MAC address of the destination station.
Source MAC Address: The source MAC address is a 6-byte field that contains the MAC address of the source station.
Data: The data field is a variable-length field that contains the data that is being transmitted.
Frame Check Sequence (FCS): The FCS is a 4-byte field that contains a checksum of the frame. The FCS is used to detect errors in the frame.
End Delimiter (ED): The ED is a 1-byte field that indicates the end of the frame.
Frame Status (FS): The FS is a 1-byte field that contains the address recognized bit and the frame copied bit.

Token Ring vs. Ethernet

Token Ring and Ethernet are two different LAN technologies. The main difference between Token Ring and Ethernet is the way that they control access to the medium. Token Ring uses a token-passing scheme, while Ethernet uses a CSMA/CD scheme.

Token Ring is a deterministic protocol, which means that each station is guaranteed to be able to transmit data within a certain amount of time. Ethernet is a probabilistic protocol, which means that there is no guarantee that a station will be able to transmit data within a certain amount of time.

Token Ring is more reliable than Ethernet, but it is also more expensive. Ethernet is less reliable than Token Ring, but it is also less expensive.

Decline of Token Ring

Token Ring was once a popular LAN technology, but it has been largely replaced by Ethernet. The decline of Token Ring is due to a number of factors, including the high cost of Token Ring hardware, the complexity of the Token Ring protocol, and the rise of switched Ethernet.

33_Fiber_Distributed_Data_Interface

2026-09-02T05:29:24.809Z

33. Fiber Distributed Data Interface (FDDI)

Introduction to FDDI

Fiber Distributed Data Interface (FDDI) is a standard for data transmission in a local area network. It uses a dual-ring topology and a token-passing scheme to control access to the medium. FDDI was developed in the 1980s as a high-speed alternative to Ethernet and Token Ring. It has a data rate of 100 Mbps.

FDDI Standards

FDDI is standardized by the American National Standards Institute (ANSI) and the International Organization for Standardization (ISO). The FDDI standards define the physical layer and the media access control (MAC) layer of the protocol.

FDDI Topology

FDDI uses a dual-ring topology. The dual-ring topology consists of two independent rings that transmit data in opposite directions. The primary ring is used for data transmission, and the secondary ring is used for backup. If the primary ring fails, the secondary ring can be used to continue data transmission.

FDDI Frame

An FDDI frame has the following format:

Preamble: The preamble is a 1-byte field that is used to synchronize the receiver with the sender.
Start Delimiter (SD): The SD is a 1-byte field that indicates the start of the frame.
Frame Control (FC): The FC is a 1-byte field that indicates the type of the frame.
Destination MAC Address: The destination MAC address is a 6-byte field that contains the MAC address of the destination station.
Source MAC Address: The source MAC address is a 6-byte field that contains the MAC address of the source station.
Data: The data field is a variable-length field that contains the data that is being transmitted.
Frame Check Sequence (FCS): The FCS is a 4-byte field that contains a checksum of the frame. The FCS is used to detect errors in the frame.
End Delimiter (ED): The ED is a 1-byte field that indicates the end of the frame.
Frame Status (FS): The FS is a 3-byte field that contains the error detected bit, the address recognized bit, and the frame copied bit.

FDDI vs. Ethernet

FDDI and Ethernet are two different LAN technologies. The main difference between FDDI and Ethernet is the way that they control access to the medium. FDDI uses a token-passing scheme, while Ethernet uses a CSMA/CD scheme.

FDDI is a deterministic protocol, which means that each station is guaranteed to be able to transmit data within a certain amount of time. Ethernet is a probabilistic protocol, which means that there is no guarantee that a station will be able to transmit data within a certain amount of time.

FDDI is more reliable than Ethernet, but it is also more expensive. Ethernet is less reliable than FDDI, but it is also less expensive.

Decline of FDDI

FDDI was once a popular LAN technology, but it has been largely replaced by Ethernet. The decline of FDDI is due to a number of factors, including the high cost of FDDI hardware, the complexity of the FDDI protocol, and the rise of Fast Ethernet and Gigabit Ethernet.

34_Asynchronous_Transfer_Mode

2026-09-02T05:29:24.809Z

34. Asynchronous Transfer Mode (ATM)

Introduction to ATM

Asynchronous Transfer Mode (ATM) is a switching technique for telecommunication networks. It is a connection-oriented protocol that uses fixed-size cells to transmit data. ATM was developed in the 1980s as a high-speed alternative to traditional circuit-switched and packet-switched networks. It has a data rate of up to 10 Gbps.

ATM Standards

ATM is standardized by the International Telecommunication Union (ITU) and the ATM Forum. The ATM standards define the physical layer, the ATM layer, and the ATM adaptation layer (AAL) of the protocol.

ATM Cell

An ATM cell is a fixed-size packet that is used to transmit data in an ATM network. An ATM cell has a size of 53 bytes. The first 5 bytes of the cell are the header, and the remaining 48 bytes are the payload.

The header of an ATM cell contains the following fields:

Generic Flow Control (GFC): The GFC is a 4-bit field that is used to control the flow of traffic.
Virtual Path Identifier (VPI): The VPI is an 8-bit field that identifies the virtual path.
Virtual Channel Identifier (VCI): The VCI is a 16-bit field that identifies the virtual channel.
Payload Type (PT): The PT is a 3-bit field that indicates the type of the payload.
Cell Loss Priority (CLP): The CLP is a 1-bit field that indicates the priority of the cell.
Header Error Control (HEC): The HEC is an 8-bit field that is used to detect errors in the header.

ATM Adaptation Layer (AAL)

The ATM adaptation layer (AAL) is responsible for adapting the data from the higher layers to the ATM layer. The AAL is divided into two sublayers: the convergence sublayer (CS) and the segmentation and reassembly (SAR) sublayer.

The CS is responsible for providing a specific service to the higher layers. The SAR is responsible for segmenting the data from the higher layers into ATM cells and for reassembling the ATM cells into data at the destination.

There are four types of AALs:

AAL1: AAL1 is used for constant bit rate (CBR) services, such as voice and video.
AAL2: AAL2 is used for variable bit rate (VBR) services, such as compressed voice and video.
AAL3/4: AAL3/4 is used for variable bit rate (VBR) services, such as data.
AAL5: AAL5 is used for variable bit rate (VBR) services, such as data.

ATM vs. Ethernet

ATM and Ethernet are two different networking technologies. The main difference between ATM and Ethernet is the way that they transmit data. ATM uses fixed-size cells, while Ethernet uses variable-size frames.

ATM is a connection-oriented protocol, which means that a connection must be established before data can be transmitted. Ethernet is a connectionless protocol, which means that data can be transmitted without establishing a connection.

ATM is more reliable than Ethernet, but it is also more expensive. Ethernet is less reliable than ATM, but it is also less expensive.

Decline of ATM

ATM was once a popular networking technology, but it has been largely replaced by Ethernet. The decline of ATM is due to a number of factors, including the high cost of ATM hardware, the complexity of the ATM protocol, and the rise of Fast Ethernet and Gigabit Ethernet.

35_Frame_Relay

2026-09-02T05:29:24.809Z

35. Frame Relay

Introduction to Frame Relay

Frame Relay is a standardized wide area network (WAN) technology that specifies the physical and data link layers of digital telecommunications channels using a packet switching methodology. Originally designed for transport across Integrated Services Digital Network (ISDN) infrastructure, it may be used today in a variety of other network interfaces. Frame Relay is a connection-oriented protocol that uses variable-size frames to transmit data.

Frame Relay Standards

Frame Relay is standardized by the International Telecommunication Union (ITU) and the American National Standards Institute (ANSI). The Frame Relay standards define the physical layer and the data link layer of the protocol.

Frame Relay Frame

A Frame Relay frame has the following format:

Flag: The flag is a 1-byte field that indicates the start and end of the frame.
Address: The address is a 2-byte field that contains the data link connection identifier (DLCI).
Data: The data field is a variable-length field that contains the data that is being transmitted.
Frame Check Sequence (FCS): The FCS is a 2-byte field that contains a checksum of the frame. The FCS is used to detect errors in the frame.

Data Link Connection Identifier (DLCI)

A DLCI is a 10-bit number that identifies a virtual circuit. A virtual circuit is a logical connection between two devices on a Frame Relay network. A virtual circuit can be either a permanent virtual circuit (PVC) or a switched virtual circuit (SVC).

Permanent Virtual Circuit (PVC): A PVC is a permanent connection between two devices. A PVC is established by the network administrator and remains in place until it is manually removed.
Switched Virtual Circuit (SVC): An SVC is a temporary connection between two devices. An SVC is established on demand and is torn down when it is no longer needed.

Frame Relay vs. X.25

Frame Relay and X.25 are two different WAN technologies. The main difference between Frame Relay and X.25 is the way that they handle errors. Frame Relay does not perform error correction, while X.25 does. This makes Frame Relay more efficient than X.25, but it also makes it less reliable.

Decline of Frame Relay

Frame Relay was once a popular WAN technology, but it has been largely replaced by other technologies, such as MPLS and Ethernet. The decline of Frame Relay is due to a number of factors, including the high cost of Frame Relay hardware, the complexity of the Frame Relay protocol, and the rise of MPLS and Ethernet.

36_Integrated_Services_Digital_Network

2026-09-02T05:29:24.809Z

36. Integrated Services Digital Network (ISDN)

Introduction to ISDN

Integrated Services Digital Network (ISDN) is a set of communication standards for simultaneous digital transmission of voice, video, data, and other network services over the traditional circuits of the public switched telephone network. It was first defined in 1988 in the CCITT red book. ISDN is a circuit-switched telephone network system, which also provides access to packet switched networks, designed to allow digital transmission of voice and data over ordinary telephone copper wires, resulting in better voice quality than an analog phone can provide.

ISDN Channels

ISDN provides two types of channels:

Bearer Channels (B channels): B channels are used to carry voice, video, and data. Each B channel has a data rate of 64 kbps.
Data Channels (D channels): D channels are used to carry signaling information. Each D channel has a data rate of 16 kbps or 64 kbps.

ISDN Interfaces

There are two types of ISDN interfaces:

Basic Rate Interface (BRI): BRI provides two B channels and one D channel. BRI is intended for home and small business use.
Primary Rate Interface (PRI): PRI provides 23 B channels and one D channel in North America and Japan, and 30 B channels and one D channel in Europe and the rest of the world. PRI is intended for large business use.

ISDN Services

ISDN provides a variety of services, including:

Voice: ISDN provides high-quality voice service.
Video: ISDN can be used for video conferencing.
Data: ISDN can be used for high-speed data transmission.
Fax: ISDN can be used for high-speed fax transmission.

ISDN vs. PSTN

ISDN and the public switched telephone network (PSTN) are two different telephone network systems. The main difference between ISDN and the PSTN is the way that they transmit signals. ISDN transmits signals digitally, while the PSTN transmits signals in analog form. This makes ISDN more reliable and provides better voice quality than the PSTN.

Decline of ISDN

ISDN was once a popular telephone network system, but it has been largely replaced by other technologies, such as DSL and cable modem. The decline of ISDN is due to a number of factors, including the high cost of ISDN hardware, the complexity of the ISDN protocol, and the rise of DSL and cable modem.

37_Digital_Subscriber_Line

2026-09-02T05:29:24.809Z

37. Digital Subscriber Line (DSL)

Introduction to DSL

Digital Subscriber Line (DSL) is a family of technologies that are used to transmit digital data over telephone lines. In telecommunications marketing, the term DSL is widely understood to mean Asymmetric Digital Subscriber Line (ADSL), the most commonly installed DSL technology, for Internet access. DSL service can be delivered simultaneously with wired telephone service on the same telephone line. This is possible because DSL uses higher frequency bands for data. On the customer premises, a DSL filter on each non-DSL outlet blocks any high-frequency interference to enable simultaneous use of voice and DSL services.

Types of DSL

There are many different types of DSL, but some of the most common types are:

Asymmetric Digital Subscriber Line (ADSL): ADSL is the most common type of DSL. It provides a faster download speed than upload speed. ADSL is well-suited for internet access, where users typically download more data than they upload.
Symmetric Digital Subscriber Line (SDSL): SDSL provides the same download speed and upload speed. SDSL is well-suited for business applications, where users may need to upload large files.
High-bit-rate Digital Subscriber Line (HDSL): HDSL is a type of DSL that provides a high-speed connection over a short distance. HDSL is often used to connect two buildings together.
Very-high-bit-rate Digital Subscriber Line (VDSL): VDSL is a type of DSL that provides a very high-speed connection over a short distance. VDSL is often used to provide fiber-to-the-home service.

DSL vs. Cable Modem

DSL and cable modem are two different technologies for providing internet access. The main difference between DSL and cable modem is the way that they transmit data. DSL transmits data over telephone lines, while cable modem transmits data over coaxial cable.

DSL is a dedicated connection, which means that the bandwidth is not shared with other users. Cable modem is a shared connection, which means that the bandwidth is shared with other users. This can cause the speed of a cable modem connection to slow down during peak hours.

DSL is more widely available than cable modem, but it is also more expensive. Cable modem is less widely available than DSL, but it is also less expensive.

Future of DSL

The future of DSL is uncertain. DSL is facing competition from other technologies, such as fiber-to-the-home and wireless. However, DSL is still a viable option for many users, especially in rural areas where other technologies are not available.

38_Cable_Modems

2026-09-02T05:29:24.809Z

38. Cable Modems

Introduction to Cable Modems

A cable modem is a type of network bridge that provides bi-directional data communication via radio frequency channels on a hybrid fiber-coaxial (HFC) and radio frequency over glass (RFoG) infrastructure. Cable modems are primarily used to deliver broadband Internet access in the form of cable Internet, taking advantage of the high bandwidth of a HFC and RFoG network. They are commonly deployed in the Americas, Europe, and Asia.

How Cable Modems Work

A cable modem works by modulating and demodulating data signals for transmission over a coaxial cable. The cable modem is connected to a cable television (CATV) network. The CATV network is a hybrid fiber-coaxial (HFC) network, which means that it uses a combination of fiber-optic cable and coaxial cable.

The fiber-optic cable is used to carry the signal from the headend to the neighborhood. The coaxial cable is used to carry the signal from the neighborhood to the home. The cable modem is connected to the coaxial cable in the home.

The cable modem communicates with a cable modem termination system (CMTS) at the headend. The CMTS is responsible for managing the cable modem network.

Cable Modem Standards

There are many different cable modem standards, but the most common standard is Data Over Cable Service Interface Specification (DOCSIS). DOCSIS is a standard that was developed by CableLabs. DOCSIS defines the physical layer and the media access control (MAC) layer of the protocol.

DOCSIS Versions

There have been several versions of DOCSIS:

DOCSIS 1.0: DOCSIS 1.0 was the first version of DOCSIS. It was released in 1997. It has a maximum download speed of 40 Mbps and a maximum upload speed of 10 Mbps.
DOCSIS 1.1: DOCSIS 1.1 was released in 1999. It added support for Quality of Service (QoS).
DOCSIS 2.0: DOCSIS 2.0 was released in 2001. It has a maximum download speed of 40 Mbps and a maximum upload speed of 30 Mbps.
DOCSIS 3.0: DOCSIS 3.0 was released in 2006. It has a maximum download speed of 1 Gbps and a maximum upload speed of 200 Mbps.
DOCSIS 3.1: DOCSIS 3.1 was released in 2013. It has a maximum download speed of 10 Gbps and a maximum upload speed of 1 Gbps.

Cable Modem vs. DSL

Cable modem and DSL are two different technologies for providing internet access. The main difference between cable modem and DSL is the way that they transmit data. Cable modem transmits data over coaxial cable, while DSL transmits data over telephone lines.

Cable modem is a shared connection, which means that the bandwidth is shared with other users. This can cause the speed of a cable modem connection to slow down during peak hours. DSL is a dedicated connection, which means that the bandwidth is not shared with other users.

Cable modem is less widely available than DSL, but it is also less expensive. DSL is more widely available than cable modem, but it is also more expensive.

39_SONET_SDH

2026-09-02T05:29:24.809Z

39. Synchronous Optical Networking (SONET) and Synchronous Digital Hierarchy (SDH)

Introduction to SONET/SDH

Synchronous Optical Networking (SONET) and Synchronous Digital Hierarchy (SDH) are standardized protocols that transfer multiple digital bit streams synchronously over optical fiber using lasers or highly coherent light from light-emitting diodes (LEDs). At low transmission rates data can also be transferred via an electrical interface.

SONET is the United States version of the standard, and SDH is the international version of the standard. The two standards are very similar, and they are often used interchangeably.

SONET/SDH Hierarchy

SONET/SDH defines a hierarchy of data rates. The base rate is 51.84 Mbps. The other rates are multiples of the base rate.

SONET Level	SDH Level	Data Rate (Mbps)
OC-1	STM-1	51.84
OC-3	STM-1	155.52
OC-12	STM-4	622.08
OC-48	STM-16	2488.32
OC-192	STM-64	9953.28
OC-768	STM-256	39813.12

SONET/SDH Frame

A SONET/SDH frame has a size of 810 bytes. The frame is divided into two parts: the transport overhead and the synchronous payload envelope (SPE).

The transport overhead contains information about the frame, such as the frame synchronization, the section trace, and the section error monitoring.

The SPE contains the data that is being transmitted.

SONET/SDH vs. ATM

SONET/SDH and ATM are two different networking technologies. The main difference between SONET/SDH and ATM is the way that they transmit data. SONET/SDH uses a synchronous time-division multiplexing (TDM) scheme, while ATM uses an asynchronous cell-based scheme.

SONET/SDH is a connection-oriented protocol, which means that a connection must be established before data can be transmitted. ATM is also a connection-oriented protocol.

SONET/SDH is more efficient than ATM for carrying constant bit rate (CBR) traffic, such as voice and video. ATM is more efficient than SONET/SDH for carrying variable bit rate (VBR) traffic, such as data.

Decline of SONET/SDH

SONET/SDH was once a popular networking technology, but it has been largely replaced by other technologies, such as Ethernet and MPLS. The decline of SONET/SDH is due to a number of factors, including the high cost of SONET/SDH hardware, the complexity of the SONET/SDH protocol, and the rise of Ethernet and MPLS.

40_Multiprotocol_Label_Switching

2026-09-02T05:29:24.809Z

40. Multiprotocol Label Switching (MPLS)

Introduction to MPLS

Multiprotocol Label Switching (MPLS) is a routing technique in telecommunications networks that directs data from one node to the next based on short path labels rather than long network addresses, avoiding complex lookups in a routing table. The labels identify virtual links41_Network_Address_Translation (paths) between distant nodes rather than endpoints. MPLS can encapsulate packets of various network protocols, hence the “multiprotocol” reference in its name. MPLS supports a range of access technologies, including T1/E1, ATM, Frame Relay, and DSL.

How MPLS Works

MPLS works by adding a label to each packet. The label is a short, fixed-length value that is used to identify the packet’s path through the network. The routers in the network use the label to forward the packet to the next hop.

The first router in the network, called the ingress router, adds the label to the packet. The last router in the network, called the egress router, removes the label from the packet.

The routers in between the ingress and egress routers, called the label switch routers (LSRs), use the label to forward the packet to the next hop.

MPLS Labels

An MPLS label is a 32-bit value that is divided into four parts:

Label: The label is a 20-bit value that is used to identify the packet’s path through the network.
Experimental (EXP): The EXP is a 3-bit value that is used for quality of service (QoS).
Bottom of Stack (S): The S is a 1-bit value that indicates whether the label is the last label in the stack.
Time to Live (TTL): The TTL is an 8-bit value that is used to prevent packets from looping in the network.

MPLS vs. IP Routing

MPLS and IP routing are two different routing techniques. The main difference between MPLS and IP routing is the way that they forward packets. MPLS forwards packets based on short path labels, while IP routing forwards packets based on long network addresses.

MPLS is more efficient than IP routing, because it does not need to perform complex lookups in a routing table. MPLS is also more scalable than IP routing, because it can support a larger number of nodes.

Advantages of MPLS

Performance: MPLS can improve the performance of a network by reducing the amount of time that it takes to forward packets.
Scalability: MPLS is more scalable than IP routing, because it can support a larger number of nodes.
Quality of Service (QoS): MPLS can be used to provide QoS for different types of traffic.
Traffic Engineering: MPLS can be used to engineer the traffic on a network.
Virtual Private Networks (VPNs): MPLS can be used to create VPNs.

Disadvantages of MPLS

Complexity: MPLS can be complex to set up and manage.
Cost: MPLS can be expensive to implement.

41_Network_Address_Translation

2026-09-02T05:29:24.809Z

41. Network Address Translation (NAT)

Introduction to NAT

Network Address Translation (NAT) is a method of remapping one IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device. The term is most commonly used to refer to the process where a private IP address is translated into a public IP address. This is done to conserve the limited number of public IPv4 addresses.

How NAT Works

NAT works by creating a table that maps private IP addresses to public IP addresses. When a packet from a private IP address is sent to a public IP address, the NAT device looks up the private IP address in the table and replaces it with the corresponding public IP address. The NAT device also creates a new port number for the packet. The new port number is used to distinguish the packet from other packets that are being sent from the same public IP address.

When a packet from a public IP address is sent to a private IP address, the NAT device looks up the public IP address and port number in the table and replaces them with the corresponding private IP address and port number.

Types of NAT

There are three main types of NAT:

Static NAT: Static NAT maps a single private IP address to a single public IP address. Static NAT is often used to allow a device on a private network to be accessible from the internet.
Dynamic NAT: Dynamic NAT maps a pool of private IP addresses to a pool of public IP addresses. Dynamic NAT is often used to allow multiple devices on a private network to share a single public IP address.
Port Address Translation (PAT): PAT is a type of dynamic NAT that maps multiple private IP addresses to a single public IP address by using different port numbers. PAT is the most common type of NAT.

Advantages of NAT

Conserves public IPv4 addresses: NAT can be used to conserve the limited number of public IPv4 addresses.
Increases security: NAT can increase security by hiding the private IP addresses of the devices on a private network.
Flexibility: NAT can be used to create a more flexible network.

Disadvantages of NAT

Complexity: NAT can be complex to set up and manage.
Breaks end-to-end connectivity: NAT breaks the end-to-end connectivity model of the internet.
Can interfere with some applications: NAT can interfere with some applications, such as peer-to-peer applications.

/Private

System Design

Introduction

What is System Design?

1. Scalability

1.1 Types of Scaling

However, there are many other factors to consider when determining the appropriate approach. So lets explain one by one.

1.1.1 Vertical Scaling

1.1.2 Horizontal Scaling

2. Caching

2.1 Caching: Mechanisms, Tools, and Techniques

2.2 Cache Strategies: Types and When to Use What

Choosing the Right Caching Strategy

3. Load Balancing

3.1 Types of Load Balancers based on the OSI Model

3.2 Load Balancing Algorithms

3. Database Replication and Partitioning

3.1 Database Replication

3.2 Algorithms to implement Database replication

Single Leader Replication(Active-Passive or Master-Slave Replication)

System Design MOC

🗺️ System Design Map of Content

📄 Notes

Md Notes MOC

🗺️ Md Notes Map of Content

📁 Sub-Topics

📄 Notes

key

sql basics

Index for Devops

Kubernetes

Kubernetes (often abbreviated as K8s) is an open-source platform that automates deploying, scaling, and managing containerized applications.

1. Kubernetes Architecture

2. Kubernetes works in cluster

3. How to Write Yet Another Markup Language(yml/yaml)

Heading towards practical part…

Creating a kind cluster

- Kubernetes Namespaces

|Docker container→Pod→Deployment→service |

- Kubernetes Pods

- Kubernetes workloads

Networking for Devops

1. How DNS actually works?

1.1 DNS Records

2. How HTTP works?

Nginx

How to write Nginx config file

Config file structure

Serving static content

Nginx Reverse Proxy for an Application Running on localhost:8000

Objective

Architecture

Nginx Configuration

Explanation

1. Listen on Port 80

2. Forward Requests to Backend

3. Preserve Original Request Information

Host Header

Real Client IP

Forwarded Client Chain

Original Protocol

Validate Configuration

Reload Nginx

Verify Backend Service

Test the Reverse Proxy

Troubleshooting

Check Nginx Error Logs

Check Access Logs

Key Concept

Service Mesh

What exactly a service mesh is?

1. Why Do You Need a Service Mesh?

Real-World Example: Netflix

2. How Service Mesh Helps

3. Service Mesh Architecture

3.1 Data Plane (Sidecar Proxies)

3.2 Control Plane

How It Works: Without vs. With Service Mesh

4. Popular Service Mesh Implementations

5. Benefits of Service Mesh