[{"content":"For a long time, \u0026ldquo;make the agent better\u0026rdquo; meant \u0026ldquo;wait for a better model.\u0026rdquo; That is no longer the whole story. The people building serious AI agents now agree on a simple formula:\nAgent = Model + Harness\nThe model is the brain. The harness is everything around it. And here is the surprising part: once your model is good enough, most of your remaining improvement comes from the harness, not the model. A weak model with a great harness beats a great model with a weak harness.\nThis article explains what the harness is, why it matters so much, how to build one, and how to keep it healthy as models keep changing.\nWhat the harness actually is The rule of thumb from LangChain\u0026rsquo;s Vivek Trivedy is short: if you\u0026rsquo;re not the model, you\u0026rsquo;re the harness.\nThe model itself just turns text in into text out. The harness is everything that makes that useful:\nthe system prompt tools, Skills, and MCP servers (plus their descriptions) infrastructure like a filesystem, a sandbox, and a browser orchestration logic (spawning sub-agents, routing between them) hooks that run at fixed points to force certain behavior observability: logs, traces, and cost tracking That last one is easy to skip but important. If you can\u0026rsquo;t see what the agent did, you can\u0026rsquo;t tell whether a change helped. As the practitioners put it: a harness without observability is a harness you can\u0026rsquo;t improve.\nWhy do harnesses exist at all? Because a model on its own can\u0026rsquo;t remember things between runs, can\u0026rsquo;t run code, can\u0026rsquo;t look up today\u0026rsquo;s news, and can\u0026rsquo;t set up its own workspace. The harness fills those gaps. The design habit is always the same: decide what behavior you want, then build the piece that produces it.\ndeepset adds a sharper way to think about it. A harness doesn\u0026rsquo;t just help the model — it changes the job the model has to do. It takes over the three things models are worst at:\nmemory — keeping track of things over time skills — knowing how to do a procedure protocols — following a set structure for interaction The clearest example is memory. Instead of asking a forgetful model to remember something, the harness writes it down and later asks the model to read it. Remembering is hard; reading is easy. The harness quietly swaps the hard job for the easy one.\nThe harness is also the glue that holds these together. It runs the main loop, decides how the limited context budget gets shared, and enforces what each tool is allowed to do.\nflowchart TB M[Model\u0026lt;br/\u0026gt;the brain] M --\u0026gt; H subgraph H[Harness: everything else] direction TB SP[System prompt] T[Tools / Skills / MCP] FS[Filesystem + Git] SB[Sandbox / code execution] CTX[Context management] MEM[Memory + search] ORCH[Orchestration:\u0026lt;br/\u0026gt;sub-agents, routing] HK[Hooks] OBS[Observability:\u0026lt;br/\u0026gt;logs, traces, cost] SP ~~~ T ~~~ FS ~~~ SB ~~~ CTX ~~~ MEM ~~~ ORCH ~~~ HK ~~~ OBS end One split is worth naming early. The builder harness is what ships inside an agent product — the parts above, put together by whoever built it. The user harness is what you build around a coding agent in your own project: the conventions, checks, and guardrails you add so it needs less babysitting. Most day-to-day craft happens in that second layer.\nWhy the harness is where the wins are If the model keeps improving on its own, why fuss over the scaffolding? Because the scaffolding moves the score more.\nHumanLayer\u0026rsquo;s Kyle gave this its catchphrase: the model is probably fine; it\u0026rsquo;s just a skill issue. His point is that most agent failures are setup problems, not model problems. So before you blame the model, look at the harness.\nThe benchmarks agree. On Terminal Bench 2.0, the same model scored much lower in a generic setup than in a purpose-built one. Trivedy\u0026rsquo;s team jumped from Top 30 to Top 5 by changing only the harness. deepset saw the same thing: harness-only tweaks moved agents 20-plus places in the rankings. Their summary of the shift is memorable: AI engineers are becoming environment designers. And deepset\u0026rsquo;s bottom line is that tuning the harness gives you bigger reliability gains than upgrading the model, for far less money.\nThere\u0026rsquo;s one point where the sources openly disagree, and it\u0026rsquo;s worth keeping rather than hiding. Both HumanLayer and deepset say \u0026ldquo;harness engineering\u0026rdquo; and \u0026ldquo;context engineering\u0026rdquo; are two nested, related skills — you need both. But they disagree on which one contains the other:\nHumanLayer: harness engineering is part of context engineering. deepset: context engineering is part of harness engineering — because context work manages what the model sees right now, while the harness also manages how the whole system behaves over time. This article leans toward deepset\u0026rsquo;s view, since \u0026ldquo;Agent = Model + Harness\u0026rdquo; treats context management as one piece inside the harness. But the disagreement is real, and you should know both camps exist.\nThe parts of a builder harness Trivedy\u0026rsquo;s breakdown gives the parts list.\nFilesystem. It does three jobs at once: it stores state that outlives a single run, it lets the agent park information outside the prompt and read it back later, and it acts as shared space between agents and people. Add Git on top and you also get history, diffs, and the ability to undo the agent\u0026rsquo;s work.\nCode execution and sandboxes. Instead of building a custom tool for every task, you give the model a shell and a computer. One general ability lets it solve many problems on its own. But running code needs a safe place, so a sandbox gives it an isolated environment — which also lets the agent run its own work and check the result.\nContext management. The context window is small and gets worse as it fills up (a problem often called \u0026ldquo;context rot\u0026rdquo;). The harness fights back a few ways:\ncompaction — summarize old context to free up room offloading — move bulky tool inputs and outputs to the filesystem progressive disclosure — load a Skill\u0026rsquo;s instructions only when they\u0026rsquo;re needed, instead of paying for everything up front A stronger move is a context reset: wipe the window and start fresh from a short handoff note. This fixes \u0026ldquo;context anxiety,\u0026rdquo; where a model rushes to wrap up as its window fills. deepset simply calls a full window a failure, and its fix is the same: keep state in files and logs, not stuffed in the prompt.\nMemory and search. This decides what knowledge reaches the model. It covers injecting project context (through files like AGENTS.md) and searching the web or MCP tools (like Context7) for anything newer than the model\u0026rsquo;s training cutoff. This is also where the \u0026ldquo;write it down, read it later\u0026rdquo; trick lives. The next frontier is longer memory that lasts weeks or months, so an agent can get up to speed by reading its own past logs.\nLong-horizon execution. To keep working past a single window, you combine the pieces above: files and Git to save progress across restarts; \u0026ldquo;Ralph Loops\u0026rdquo; that re-feed the prompt into a clean window to force the agent to continue; and planning plus self-checking. HumanLayer calls the key idea back-pressure: an agent succeeds when it can check its own work — but that checking has to be cheap and quiet, or it eats up the window too.\nConfiguration and platform Above the raw parts sits configuration, and this is where most users actually shape their harness. According to Addy Osmani, the two highest-leverage knobs are the config file and the tool set.\nKeep the config file short. Treat AGENTS.md (or CLAUDE.md) like a pilot\u0026rsquo;s checklist, not a style guide. HumanLayer keeps theirs under about 60 lines. A study from ETH Zurich even found that most of these files were useless or worse — which argues against auto-generating them and for keeping them tight. Every line should exist because something actually went wrong. Otherwise it\u0026rsquo;s noise.\nUse few, focused tools. Ten sharp tools beat fifty overlapping ones. Too many tools confuse the model and waste context on descriptions — HumanLayer calls this pushing the model into \u0026ldquo;the dumb zone.\u0026rdquo; A good habit: prefer command-line tools the model already knows (like Git or Docker) over custom MCP servers. HumanLayer even built a Linear CLI instead of an MCP server, just to save tokens. There\u0026rsquo;s a security angle too: tool descriptions are fed to the model as trusted text, so a malicious description can hijack the agent. Vetting tools isn\u0026rsquo;t just tidiness — it\u0026rsquo;s a safety boundary.\nHooks are the enforcement layer. They\u0026rsquo;re scripts that run at fixed moments — before a tool call, after an edit, before a commit — and they give you guaranteed behavior from an otherwise unpredictable agent. Advice can be ignored; a pre-commit hook cannot. The rule of thumb: success is silent, failures are verbose. A type-check hook should print nothing when it passes and only speak up when it fails, so it doesn\u0026rsquo;t waste the agent\u0026rsquo;s context.\nSub-agents are widely misunderstood, so this one needs care. The tempting idea is to give each sub-agent a role — a \u0026ldquo;frontend engineer,\u0026rdquo; a \u0026ldquo;backend engineer.\u0026rdquo; HumanLayer is blunt that this doesn\u0026rsquo;t work. The real reason to use a sub-agent is context control. A sub-agent runs a task in its own separate window and hands back only the answer. All the messy middle — failed tries, tool noise — never reaches the main agent. deepset confirms this \u0026ldquo;context firewall\u0026rdquo; idea. The other reason is cost: let an expensive model run the show while cheaper models do the grunt work. deepset takes this further with per-step routing — a strong model for planning, a small fast one for repetitive checking — and calls it standard practice now, not a fancy experiment.\nFinally, the platform trend: Harness-as-a-Service. The industry is moving from LLM APIs (which just return an answer) to harness APIs (which give you a whole runtime) — the Claude Agent SDK, Codex SDK, and OpenAI Agents SDK are examples. You configure these along four dials: system prompt, tools, context, and sub-agents. deepset sees the same shift from the company side: once several teams depend on one harness, it needs access control, secrets management, isolation, and audit trails that prove certain actions always required human sign-off. Their idea of \u0026ldquo;shared agent infrastructure\u0026rdquo; — memory, skills, and protocols maintained by the whole org — is the business case: build a good harness once, share it everywhere.\nBuilding your own harness Birgitta Böckeler gives the clearest framework for the harness you build around a coding agent. Every control you add can be sorted two ways.\nFirst, is it a guide or a sensor?\nGuides act before the agent works — system prompts, how-to docs, conventions. They raise the quality of the first attempt. Sensors act after the agent works — tests, linters, reviews. They let the agent catch and fix its own mistakes. You need both. Guides alone means the agent repeats mistakes nobody checks for. Sensors alone means it never gets steered in the right direction to begin with.\nSecond, is the control computational or inferential?\nComputational controls are exact and fast — tests, linters, type checkers. Cheap and reliable. These are your backbone. Inferential controls use a model to judge — AI review, \u0026ldquo;LLM as judge.\u0026rdquo; They handle fuzzy questions that exact checks can\u0026rsquo;t, but they\u0026rsquo;re slower, pricier, and less predictable. The rule: use exact checks wherever they work, and save the model-based judges for things that truly need judgment. You can make the judges steadier with a few calibrated examples.\nflowchart TB G[Guides — act before\u0026lt;br/\u0026gt;prompts, docs, conventions] A((Agent acts)) S[Sensors — act after\u0026lt;br/\u0026gt;tests, linters, reviews] G --\u0026gt; A A --\u0026gt; S S --\u0026gt;|failures feed back\u0026lt;br/\u0026gt;into the guides| G So your real job isn\u0026rsquo;t to review every output. It\u0026rsquo;s to improve the harness — this is the steering loop. When a problem keeps happening, you fix a control so it can\u0026rsquo;t happen again. Osmani calls this a ratchet: add a rule only after a real failure, and remove it once the model no longer needs it. Hashimoto\u0026rsquo;s test is simple — set things up so the agent never makes that mistake again.\ndeepset spells the loop out: run on real tasks, watch for failures, sort each failure by type, fix the right layer, repeat. Two things make it work: tracing (without logs, you\u0026rsquo;re just guessing) and a sorting step so each fix lands in the right place.\nflowchart TB RUN[Run on real tasks] OBS[Watch failures\u0026lt;br/\u0026gt;using traces] CLASS[Sort each failure by type] UPDATE[Fix the right layer] RUN --\u0026gt; OBS OBS --\u0026gt; CLASS CLASS --\u0026gt; UPDATE UPDATE --\u0026gt; RUN That sorting step deserves its own list, because fixing the wrong layer wastes time. deepset names four failure types:\nFailure What you see What to fix Context Missing the right info at the right time — made up a schema, forgot the goal Retrieval, memory, context management Constraint Had the info but did something wrong — edited files it shouldn\u0026rsquo;t, called a needless tool A guardrail — permission limit, linter rule, scope boundary Verification Produced something that looks right but isn\u0026rsquo;t, and nothing caught it A feedback loop — tests, a validator, a second model as reviewer Planning Took the wrong approach entirely — one step where five were needed, got stuck in a loop Orchestration — break it up, use sub-agents, detect loops deepset\u0026rsquo;s guardrail advice is to constrain more, not less: start strict, loosen as you gain confidence. And all of this only works if you can see what happened — no tracing, no real diagnosis.\nThere\u0026rsquo;s a second way to sort things, this time by what quality you\u0026rsquo;re trying to protect. Böckeler describes three areas, from most solved to least:\nMaintainability — the easy one. Exact checks reliably catch duplication, complexity, and low test coverage. Architecture fitness — checking that the code keeps the shape you want, using \u0026ldquo;fitness functions.\u0026rdquo; Behaviour — whether the code actually works correctly. This is the hard, unsolved one. Today\u0026rsquo;s approach leans too much on AI-written tests that aren\u0026rsquo;t good enough yet. The evidence is concrete: out of the box, \u0026ldquo;Claude is a poor QA agent\u0026rdquo; — it flags a real problem and then talks itself into approving anyway. This is exactly where a human should step in. Even a great harness only partly captures the judgment, accountability, and taste a person brings. The goal isn\u0026rsquo;t to remove the human — it\u0026rsquo;s to point their attention where the automated checks still can\u0026rsquo;t judge for themselves.\nOne more practical note: not every codebase is easy to harness. Strong typing, clear module boundaries, and good frameworks make a codebase easier for agents to work in — Ned Letcher calls these \u0026ldquo;ambient affordances.\u0026rdquo; New projects can build them in from the start. Old, messy codebases are the hardest — and, frustratingly, that\u0026rsquo;s exactly where a harness would help most.\nMulti-agent patterns and grading Anthropic\u0026rsquo;s Prithvi Rajasekaran offers the most hands-on lessons, from building long-running apps.\nThe main pattern is generator/evaluator: one agent builds, a separate agent judges. Splitting the doer from the judge works because agents are bad at grading themselves — they praise their own so-so work. The evaluator scores the output against clear standards, and its feedback drives the next round (they saw 5 to 15 rounds). For app-building, the evaluator got the Playwright tool so it could click through the live app like a real user.\nflowchart TB P[Planner\u0026lt;br/\u0026gt;turns a short prompt into a spec] G[Generator\u0026lt;br/\u0026gt;builds one feature] E[Evaluator\u0026lt;br/\u0026gt;grades it, clicks through the app] DONE[Done\u0026lt;br/\u0026gt;spec satisfied] P --\u0026gt; G G --\u0026gt; E E --\u0026gt;|feedback,\u0026lt;br/\u0026gt;repeat 5–15x| G E --\u0026gt; DONE To grade fuzzy output like visual design, you have to make quality gradable. The team used a four-part rubric, shared by both the builder and the judge: design quality (does it feel like one coherent thing?), originality (real choices vs. generic \u0026ldquo;AI slop\u0026rdquo;), craft (typography, spacing, contrast), and functionality (is it usable?). They weighted design and originality more, since the model was already good at craft and functionality. Giving the judge a few graded examples kept its scores steady. The method has a hard limit, though: it can\u0026rsquo;t grade what the model can\u0026rsquo;t sense — \u0026ldquo;Claude can\u0026rsquo;t actually hear,\u0026rdquo; so musical taste was out of reach.\nScaling this up used three agents: a planner that turns a short prompt into an ambitious spec, a generator that builds one feature at a time, and an evaluator that clicks through and grades. Two things kept them coordinated: a sprint contract (they agreed on what \u0026ldquo;done\u0026rdquo; meant before any coding) and communication through files (the filesystem as shared workspace). The payoff was dramatic — a solo run made a broken game ($9, 20 minutes) while the full harness made a working, polished app ($200, 6 hours). The lesson: an evaluator is worth its cost only when the task is beyond what the model can do alone.\nThis also settles the sub-agent debate. HumanLayer says don\u0026rsquo;t give sub-agents roles; Rajasekaran gives them jobs (plan, build, judge). Both are right. Those jobs earn their keep not because \u0026ldquo;planner\u0026rdquo; is a personality, but because each one creates a clean context boundary. The value is the boundary, not the label.\nWhere this is heading Since harnesses are built to patch a model\u0026rsquo;s weaknesses, better models must make some of those patches pointless — and that\u0026rsquo;s exactly what people report.\nModels and harnesses are increasingly built together: products like Claude Code and Codex are trained with their harness in the loop. That helps, but it also risks overfitting — the best harness for a model isn\u0026rsquo;t always the one it trained with. You can even see this in small things, like a model preferring one edit format over another.\nRajasekaran\u0026rsquo;s practical takeaway: every harness part assumes some model weakness, and that assumption can expire. As Opus 4.5 became 4.6, his team removed parts they no longer needed — context resets and the sprint construct both went, because the newer model could plan on its own and no longer rushed to finish. The guiding rule: find the simplest thing that works, and add complexity only when you must. The subtle point is that the set of useful harnesses doesn\u0026rsquo;t shrink as models improve — it shifts. So whenever a new model lands, revisit your harness. Harness engineering will stick around as a skill, the same way prompt engineering did.\ndeepset sums up the direction as \u0026ldquo;from static scaffolding to dynamic governance\u0026rdquo;: longer memory that spans weeks, shared harness infrastructure across teams, and standard protocols (like MCP) that turn the tool layer into a commodity. The hard open problems are clear too: running hundreds of agents at once, agents that fix their own harness, and assembling tools and context on the fly.\nThe short version Agent = Model + Harness. Once the model is good enough, the harness is where your wins come from. Check the harness before blaming the model. Decide the behavior, then build the part. Every harness piece exists to cover a specific model weakness — memory, skills, or protocol. Build the feedback loop first. Logging, sorting failures, and turning each real mistake into a lasting rule matter more than any one clever feature. Use exact checks; ration the model-based ones. Tests and linters are your cheap, trustworthy backbone. Save \u0026ldquo;LLM as judge\u0026rdquo; for things that truly need judgment. Sub-agents are context boundaries, not personalities. The win is the firewall and the cost savings, not the job title. Harnesses are living things. As models improve, old assumptions expire — so revisit your harness with every model upgrade. The one thing the experts still argue about — whether context engineering contains harness engineering or the other way around — is less a problem to solve than a sign of how young this field is. But everyone agrees on the destination: reliable agents come from engineering the system, not from waiting on the model.\nSources The Anatomy of an Agent Harness — https://www.langchain.com/blog/the-anatomy-of-an-agent-harness Harness Engineering for Coding Agent Users — https://martinfowler.com/articles/harness-engineering.html Harness Design for Long-Running Application Development — https://www.anthropic.com/engineering/harness-design-long-running-apps Agent Harness Engineering — https://addyosmani.com/blog/agent-harness-engineering/ Skill Issue: Harness Engineering for Coding Agents — https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents Harness Engineering: How to Build Reliable AI Agents by Engineering the System, Not the Model — https://www.deepset.ai/blog/harness-engineering ","permalink":"https://rezafarhadur.github.io/posts/agent-harness-engineering/","summary":"Agent = Model + Harness. Once the model is good enough, most of your remaining improvement comes from the harness — everything around the model — not the model itself. What the harness is, why the wins live there, and how to build one.","title":"Agent Harness Engineering for Dummies"},{"content":"Load balancing involves effectively distributing incoming network traffic across a group of backend servers. A load balancer is tasked with distributing the load among the multiple backend servers that have been set up.\nHere we will trying out nginx as a load balancer with default round robin algorithm.\nTASK Run three backend docker container and nginx container Default Load balancing three docker container with Nginx Weighted load balancing Health check (Active health check is for Nginx plus, so using passive) Structer This diagram will give an overview about the system architecture\nIdea We have created three backend server with FastAPI, the code is very simple. All server has one endpoint and return a json response with server name.\n{ \u0026#34;message\u0026#34;: \u0026#34;Hello from server 01\u0026#34; } Source code is availabel here -\u0026gt; https://github.com/rezafarhadur/nginx-load-balancing-docker\nSetup \u0026amp; Run Please make sure docker and docker compose is installed.\nhttps://docs.docker.com/engine/install/ https://docs.docker.com/compose/install/ Run this command\ndocker-compose up --build Now, goto http://localhost/ or http://localhost:80\nPreview,\nNginx Configuration We have created a nginx.conf file that distribute request to all this three servers.\nNginx config:\nupstream api { server server-01:8001; server server-02:8002; server server-03:8003; } server { listen 80; location / { proxy_pass http://api; } } In this case main part focus point is,\nupstream api { server server-01:8001; server server-02:8002; server server-03:8003; } Here by default its using round robin algorithm to distribute the traffic. This way each request made to http://localhost is served by those three servers in a round-robin fashion.\nWeight distribution upstream api { server server-01:8001 weight=5; server server-02:8002 weight=3; server server-03:8003; } By default weight for each server is 1. This involves assigning a weight to each server and then distributing traffic among the servers based on these weights. This ensures that servers with more capacity receive more traffic, that helps to prevent overloading of any one server.\nFor example, here we set the weight of the 1st server to 5, then out of 10 requests, 5 of them (not sequentially though) are redirected to the 1st server and the rest 5 will go to the remaining servers.\nHealth Check upstream api { server server-01:8001; server server-02:8002 max_fails=5 fail_timeout=30s; server server-03:8003; } Load balancer are often used to distribute incoming traffic among servers. Health check is important part to ensure load balancer is routing traffic correctly, it will help to identify which server is down or not functioning properly.\nNginx active health checks are available for Nginx plus.\nFor passive health check, to define a server unavailable we need to define max_fails and failed_timeout\nmax_fails: sets the number of unsuccessful attempts to communicate server that happen in the durantion set in failed_timeout\nfailed_timeout: the time during the specified number of unsuccessful attempts to communicate consider as unavailable.\nAlso we can configure nginx with Least connection , Weighted least connection, IP Hash etc.\nTo check more configuration,\nhttps://nginx.org/en/docs/http/load_balancing.html https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/ Resources:\nhttps://www.nginx.com/resources/glossary/load-balancing/ https://www.nginx.com/resources/glossary/round-robin-load-balancing/ https://www.iankumu.com/blog/nginx-load-balancing/ ","permalink":"https://rezafarhadur.github.io/posts/load-balancing-with-nginx/","summary":"Load balancing involves effectively distributing incoming network traffic across a group of backend servers. A load balancer is tasked with distributing the load among the multiple backend servers that have been set up.","title":"Load Balancing With Nginx"},{"content":"Linux network namespaces are a Linux kernel feature allowing us to isolate network environments through virtualization. Network namespaces are useful for setting up containers or virtual environments. Network namespaces provide isolation of the system resources associated with networking: network devices, IPv4 and IPv6 protocol stacks, IP routing tables, firewall rules and so on.\nLinux kernal (v5.6) has 8 types of namespaces. Network namespace is one of the feature of linux kernal. Most container technologies use this feature to create isolated network stack in the operating system. We can create virtualized network stack with its interfaces, IP range, routing table etc. We can run application in different network stacks.\nVETH devices are virtual ethernet devices. They can act as tunnels between network namespaces to create a bridge to a physical network device in another namespace, but can also be used as standalone network devices. The virtual ethernet device will act as a tunnel between the network namespaces that we will create.\nWe will go thorugh linux network namespace. We want to learn how two network namespace communicate with each other. Also we will see how a network namespace will communicate with outer world.\nOur objectives:\nCreate two network namespace and establish communication between them Add a bridge network device and communicate two/multiple network namespaces via bridge Namespace to root/default namespace communication via bridge Make a communication with outside world via bridge Section 01: Create network namespace\n\u0026gt; sudo ip netns add earth \u0026gt; sudo ip netns add neptune We have created two different namespace, and its like two different computer. To check the list.\n\u0026gt; sudo ip link earth neptune Now we will add a virtual ethernet peer between this two namespace.\n\u0026gt; sudo ip link add earth-veth type veth peer name neptune-veth We created a peer but it\u0026rsquo;s not assigned to any network namespace. Lets assign them. We will assign earth-veth to earth namespace and another end to neptune namespace.\n\u0026gt; sudo ip link set earth-veth netns earth \u0026gt; sudo ip link set neptune-veth netns neptune Lets have a look at the earth network namespace\n\u0026gt; sudo ip netns exec earth ip addr 1: lo: \u0026lt;LOOPBACK\u0026gt; mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 4: earth-veth@if3: \u0026lt;BROADCAST,MULTICAST\u0026gt; mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 26:61:7a:32:8a:72 brd ff:ff:ff:ff:ff:ff link-netnsid 1 When we want to run a command inside a network namespace we can execute our command like this: ip netns exec command or ip -n command\nThe earth namespace has two ethernet devices, lo and earth-veth. Check neptune namespace also. lo is a loopback device. You can imagine it as a virtual network device that is on all systems, even if they aren\u0026rsquo;t connected to any network. And earth-veth is our newly created device. If we check the state they are in DOWN state. Lets up those.\n\u0026gt; sudo ip netns exec earth ip link set dev earth-veth up \u0026gt; sudo ip netns exec earth ip link set dev lo up Run same command in neptune namespace\n\u0026gt; sudo ip netns exec neptune ip link set dev neptune-veth up \u0026gt; sudo ip netns exec neptune ip link set dev lo up Lets check again\n\u0026gt; sudo ip netns exec earth ip addr 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 4: earth-veth@if3: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 26:61:7a:32:8a:72 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::2461:7aff:fe32:8a72/64 scope link valid_lft forever preferred_lft forever Now earth-veth state is UP, check same thing from neptune namespace there neptune-veth state is UP.\nOur virtual interfaces are enable but see they don\u0026rsquo;t have any IP addresses, without IP addresses they can\u0026rsquo;t talk with each other. Lets assign IP address to them.\n\u0026gt; sudo ip netns exec earth ip addr add 10.10.0.10/16 dev earth-veth \u0026gt; sudo ip netns exec neptune ip addr add 10.10.0.20/16 dev neptune-veth Our setup is like this:\nAt this stage if we want to talk between this two namespace, packets will drop. Because they don\u0026rsquo;t have any route so they don\u0026rsquo;t know where the packet will forward. Add route within them.\n\u0026gt; sudo ip netns exec earth ip route add default via 10.10.0.10 dev earth-veth \u0026gt; sudo ip netns exec neptune ip route add default via 10.10.0.20 dev neptune-veth Let test, can they talk with each other?\n\u0026gt; sudo ip netns exec earth ping 10.10.0.20 PING 10.10.0.20 (10.10.0.20) 56(84) bytes of data. 64 bytes from 10.10.0.20: icmp_seq=1 ttl=64 time=0.018 ms 64 bytes from 10.10.0.20: icmp_seq=2 ttl=64 time=0.031 ms 64 bytes from 10.10.0.20: icmp_seq=3 ttl=64 time=0.031 ms ^C --- 10.10.0.20 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2089ms rtt min/avg/max/mdev = 0.018/0.026/0.031/0.008 ms From neptune namespace\n\u0026gt; sudo ip netns exec neptune ping 10.10.0.10 PING 10.10.0.10 (10.10.0.10) 56(84) bytes of data. 64 bytes from 10.10.0.10: icmp_seq=1 ttl=64 time=0.041 ms 64 bytes from 10.10.0.10: icmp_seq=2 ttl=64 time=0.049 ms 64 bytes from 10.10.0.10: icmp_seq=3 ttl=64 time=0.041 ms ^C --- 10.10.0.10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2041ms rtt min/avg/max/mdev = 0.041/0.043/0.049/0.008 ms YES !!! It\u0026rsquo;s working, they can talk with each other based on IP routing table.\n\u0026gt; sudo ip netns exec earth ip route 10.10.0.0/16 dev earth-veth proto kernel scope link src 10.10.0.10 ARP table\n\u0026gt; sudo ip netns exec earth Address HWtype HWaddress Flags Mask Iface 10.10.0.20 ether 12:6c:8c:49:15:2c C earth-veth \u0026gt; sudo ip netns exec neptune Address HWtype HWaddress Flags Mask Iface 10.10.0.10 ether 26:61:7a:32:8a:72 C neptune-veth Lets try to ping earth namespace IP from default network namespace (host namespace)\n\u0026gt; ping 10.10.0.10 PING 10.10.0.10 (10.10.0.10): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3 ^C --- 10.10.0.10 ping statistics --- 5 packets transmitted, 0 packets received, 100.0% packet loss It\u0026rsquo;s not working because default network namespace don\u0026rsquo;t know this ip, there are no routing between default and earth or neptune namespace.\nDelete current namespaces and move to our next step.\n\u0026gt; ip netns delete earth \u0026gt; ip netns delete neptune Section 02: Till now we have go through basic network namespace fundamentals where two network namespace can talk with each other. Now if we need to connect three or more namespaces in the same machine, then in our earlier method we have to hardcoded connection in between namespaces. In this situation we need switch between namespaces. A “switch” in the networking world is a device which makes communication between devices in a “transparent” fashion by creating a dedicated logical link between ports. The “switch” inside a linux virtual machine is often referred to as a “linux bridge” A bridge network is a Link Layer device which forwards traffic between network segments. A bridge can be a hardware device or a software device running within a host machine’s kernel. So, we need a virtual networking bridge. We will create two network namespace (we can create multiple here also) and a virtual bridge device (it will act as a network switch).\nLets create namespace, virtual ethernet cable, assign IP to them. Like we did earlier. We will create two namespace for this demonstration but we can do this with multiple namespaces.\n\u0026gt; sudo ip netns add earth \u0026gt; sudo ip netns add neptune \u0026gt; sudo ip link add earth-veth type veth peer name earth-br-veth \u0026gt; sudo ip link add neptune-veth type veth peer name neptune-br-veth Add veth device to namespaces\n\u0026gt; sudo ip link set earth-veth netns earth \u0026gt; sudo ip link set neptune-veth netns neptune \u0026gt; sudo ip netns exec earth ip addr add 10.10.0.10/16 dev earth-veth \u0026gt; sudo ip netns exec neptune ip addr add 10.10.0.20/16 dev neptune-veth Here we have connected one side of the veth to a namespace but otherside is still not connected to any namespace. Those other two side we will connect them to bridge device.\nCreate a bridge device and connect those to bridge.\n\u0026gt; sudo ip link add planet-br type bridge \u0026gt; sudo ip link set earth-br-veth master planet-br \u0026gt; sudo ip link set neptune-br-veth master planet-br The setup looks like this\nWe have to up all of the virtual devices to connect.\n\u0026gt; sudo ip link set planet-br up \u0026gt; sudo ip link set earth-br-veth up \u0026gt; sudo ip link set neptune-br-veth up \u0026gt; sudo ip netns exec earth ip link set dev lo up \u0026gt; sudo ip netns exec earth ip link set dev earth-veth up \u0026gt; sudo ip netns exec neptune ip link set dev lo up \u0026gt; sudo ip netns exec neptune ip link set dev neptune-veth up Now all devices are up. You can check their status run ip addr command in those namespace. Lets try to ping between the two namespaces:\n\u0026gt; sudo ip netns exec earth ping 10.10.0.20 -c1 PING 10.10.0.20 (10.10.0.20) 56(84) bytes of data. 64 bytes from 10.10.0.20: icmp_seq=1 ttl=64 time=0.076 ms --- 10.10.0.20 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms \u0026gt; sudo ip netns exec neptune ping 10.10.0.10 -c1 PING 10.10.0.10 (10.10.0.10) 56(84) bytes of data. 64 bytes from 10.10.0.10: icmp_seq=1 ttl=64 time=0.107 ms --- 10.10.0.10 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.107/0.107/0.107/0.000 ms with -c1 command it will transmit 1 packet only\nNotice one thing here, we didn\u0026rsquo;t add any default route here but still its working because of bridge involved here. Check route inside of the namespace there already route added and also ARP is populated.\n\u0026gt; sudo ip netns exec earth ip route 10.10.0.0/16 dev earth-veth proto kernel scope link src 10.10.0.10 \u0026gt; sudo ip netns exec earth arp Address HWtype HWaddress Flags Mask Iface 10.10.0.20 ether 3e:a6:7c:d7:d7:b9 C earth-veth Okay, two namespaces can communicate with each other. Can default/root network namespace can communicate with those namespaces?\n\u0026gt; ping 10.10.0.10 -c1 PING 10.10.0.10 (10.10.0.10) 56(84) bytes of data. --- 10.10.0.10 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms So it\u0026rsquo;s not working because there is no routing between host/root namespace and those two namespace.\nSection 03 earth namespace to host namespace communication Here we want to stablish a communication between earth or neptune namespace to root/host namespace via bridge virtual device. Bridge will work as a gateway.\nIn our case 192.168.0.108 IP address is assigned to eth0 of host namespace\n\u0026gt; ip addr show eth0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 08:00:27:0e:82:b9 brd ff:ff:ff:ff:ff:ff inet 192.168.0.108/24 brd 192.168.0.255 scope global dynamic noprefixroute enp0s3 valid_lft 5658sec preferred_lft 5658sec inet6 fe80::aa79:2130:6242:699/64 scope link noprefixroute valid_lft forever preferred_lft forever Ping this root namespace ip from earth namespace\n\u0026gt; sudo ip netns exec earth ping 10.10.0.112 -c1 connect: Network is unreachable This error is expected, check route\n\u0026gt; sudo ip netns exec earth route Destination Gateway Genmask Flags Metric Ref Use Iface 10.10.0.100 0.0.0.0 255.255.0.0 U 0 0 0 e-v We got the issue, the host IP doesn\u0026rsquo;t have any route table entry in earth namespace. He don\u0026rsquo;t know this IP and can\u0026rsquo;t eastablish a connection. So we need to add a default gateway so that which IP addresses are not matching they forward via bridge device (planet-br). We didn\u0026rsquo;t assign any ip to our bridge (planet-br). First add am IP and then add default gateway to earth and neptune namespace.\n\u0026gt; sudo ip addr add 10.10.0.1/16 dev planet-br \u0026gt; sudo ip netns exec earth ip route add default via 10.10.0.1 \u0026gt; sudo ip netns exec neptune ip route add default via 10.10.0.1 Check route\n\u0026gt; sudo ip netns exec earth route Destination Gateway Genmask Flags Metric Ref Use Iface default 10.10.0.1 0.0.0.0 UG 0 0 0 ev 10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ev Lets test again\n\u0026gt; sudo ip netns exec earth ping 192.168.0.108 -c1 PING 192.168.0.108 (192.168.0.108) 56(84) bytes of data. 64 bytes from 192.168.0.108: icmp_seq=1 ttl=64 time=0.105 ms --- 192.168.0.108 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.105/0.105/0.105/0.000 ms Wahhh !!! its working. Now we can communicate between earth or neptune namespace with host namespace.\nLets move outside of the box. Till now we have eastablished connection between two namespaces and then eastablished connection with host/root namespace. Can this namespace (earth or neptune) can eastablish connection with any outside world. We want to connect them with internet. We will ping google IP (8.8.8.8) from earth and neptune nametune.\nSection 04 earth namespace to internet communication Lets ping 8.8.8.8 from earth namespace\n\u0026gt; sudo ip netns exec earth ping 8.8.8.8 -c1 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. --- 8.8.8.8 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms The response is different here, its not like Network is unreachable but it seems that the packet is stuck somewhere, we have to find where the packet get stuck. To debug packet flow we have a very useful tool called tcpdump, a powerful comman-line packet analyzer.\nIn our scenario, ping command goes from earth interface to host via planet-br. So first lets debug from planet-br, does the packet came here.\nrun ping command from earth namespace as we did earlier and in a new ssh session run this command from host\n\u0026gt; sudo tcpdump -i planet-br icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on planet-br, link-type EN10MB (Ethernet), capture size 262144 bytes 13:54:26.471001 IP 10.10.0.10 \u0026gt; dns.google: ICMP echo request, id 2197, seq 1, length 64 13:54:27.481254 IP 10.10.0.10 \u0026gt; dns.google: ICMP echo request, id 2197, seq 2, length 64 13:54:28.504693 IP 10.10.0.10 \u0026gt; dns.google: ICMP echo request, id 2197, seq 3, length 64 Looks like planet-br is receiving the packets so lets debug our host eth0 interface.\n\u0026gt; sudo tcpdump -i eth0 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel hmmm\u0026hellip;. packets are not reaching to eth0. But why? Okay found an issue, IP forwarding is disabled.\n\u0026gt; sudo vi /proc/sys/net/ipv4/ip_forward Change the value 0 to 1 and save it.\nWe can use the following sysctl command to enable or disable Linux IP forwarding on our system. Keep in mind that this setting is changed instantly at runtime. Also, the result will not be preserved after rebooting the system.\n# to disable \u0026gt; sudo sysctl -w net.ipv4.ip_forward=0 # to enable \u0026gt; sudo sysctl -w net.ipv4.ip_forward=0 Check the eth0 interface again (make sure ping command is running)\n\u0026gt; sudo tcpdump -i eth0 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes 14:17:47.367679 IP 10.10.0.10 \u0026gt; dns.google: ICMP echo request, id 2287, seq 1, length 64 14:17:48.377826 IP 10.10.0.10 \u0026gt; dns.google: ICMP echo request, id 2287, seq 2, length 64 14:17:49.400595 IP 10.10.0.10 \u0026gt; dns.google: ICMP echo request, id 2287, seq 3, length 64 Finally eth0 is recieving the packets but hold on packet is still stuck in earth interface. See tcpdump is only getting request logs but there are no reply logs here. From above we saw that packets is tring to reach google dns with private ip address 10.10.0.10. For private ip address google dns server can\u0026rsquo;t reach back to earth interface beacuse same private ip address can use million of devices. So we need a public ip to reach to google dns. Here comes Network Address Translation (NAT), this will convert our private ip to public ip (in our case it will use ISP public ip). We need to add a Source NAT (SNAT) rule in the POSTROUTING chain.\n\u0026gt; sudo iptables --table nat -A POSTROUTING -s 10.10.0.0/16 ! -o planet-br -j MASQUERADE MASQUERADE hides everything “behind” the host. You’d do that to supply Internet to multiple hosts when you only have one uplink IP address. This tech is used on most consumer-grade Internet access routers, dubbed “NAT”. (MASQUERADE)\nSo we added a SNAT rule, its appending (-a) POSTROUTING rule and action(-j) is MASQUERADE where source (-s) network is 10.10.0.0/16 via planet-br output (-o).\nNow lets ping again\n\u0026gt; sudo ip netns exec earth ping 8.8.8.8 -c3 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=33.4 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=33.7 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=35.0 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 33.405/34.077/35.045/0.717 ms Wuuhuu\u0026hellip;\u0026hellip; we finally did it. Now we are able to communicate with outside world.\nNow lets move a step ahead. We will run a server in earth namespace and we will try to access that from outside. Run a simple server http server using python that is listening on port 8000.\n\u0026gt; sudo ip netns exec earth python3 -m http.server --bind 10.10.0.10 8000 Serving HTTP on 10.10.0.10 port 8000 (http://10.10.0.10:8000/) ... Check can we access this from host, open a new session and run this.\n\u0026gt; telnet 10.10.0.10 8000 Trying 10.10.0.10... Connected to 10.10.0.10. Escape character is \u0026#39;^]\u0026#39;. Its connected, check using curl\n\u0026gt; curl 10.10.0.10:8000 \u0026lt;!DOCTYPE HTML PUBLIC \u0026#34;-//W3C//DTD HTML 4.01//EN\u0026#34; \u0026#34;http://www.w3.org/TR/html4/strict.dtd\u0026#34;\u0026gt; \u0026lt;html\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta http-equiv=\u0026#34;Content-Type\u0026#34; content=\u0026#34;text/html; charset=utf-8\u0026#34;\u0026gt; \u0026lt;title\u0026gt;Directory listing for /\u0026lt;/title\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;h1\u0026gt;Directory listing for /\u0026lt;/h1\u0026gt; \u0026lt;hr\u0026gt; \u0026lt;ul\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.bash_history\u0026#34;\u0026gt;.bash_history\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.cache/\u0026#34;\u0026gt;.cache/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.config/\u0026#34;\u0026gt;.config/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.dmrc\u0026#34;\u0026gt;.dmrc\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.gconf/\u0026#34;\u0026gt;.gconf/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.gnupg/\u0026#34;\u0026gt;.gnupg/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.local/\u0026#34;\u0026gt;.local/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;.mozilla/\u0026#34;\u0026gt;.mozilla/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Desktop/\u0026#34;\u0026gt;Desktop/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Documents/\u0026#34;\u0026gt;Documents/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Downloads/\u0026#34;\u0026gt;Downloads/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Music/\u0026#34;\u0026gt;Music/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Pictures/\u0026#34;\u0026gt;Pictures/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Public/\u0026#34;\u0026gt;Public/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;sudo\u0026#34;\u0026gt;sudo\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Templates/\u0026#34;\u0026gt;Templates/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;Videos/\u0026#34;\u0026gt;Videos/\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;/ul\u0026gt; \u0026lt;hr\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; Nice we got the response that means our http server is running and listening on port 8000.\nLets try to access this url from browser. In our case, virtual box ip address is 192.168.0.108, so we will hit the url http://192.168.0.108:8000/ from web browser.\nhmmm\u0026hellip; its not accessible from browser. We need to add destination ip nat (DNAT) like we did previously SNAT to ping into google dns\n\u0026gt; sudo iptables --table nat -A PREROUTING -d 192.168.0.108 -p tcp -m tcp --dport 8000 -j DNAT --to-destination 10.10.0.10:8000 meaning, any request mathched (-m) tcp in destination (-d) ip 192.168.0.108 (eth0 ip) with destination port (-dport) 8000 will jump (-j) in DNAT rule to the destination 10.10.0.10:8000\nWe can add port forwarding rule here as well like\n\u0026gt; sudo iptables --table nat -A PREROUTING -d 192.168.0.108 -p tcp -m tcp --dport 3000 -j DNAT --to-destination 10.10.0.10:8000 So it will forward 192.168.0.108:3000 to 10.10.0.10:8000\nLets try againg from browser.\nYaaaaaaa\u0026hellip; We did it. We have configured ingress and egress traffic flow. Congratulations..!!!\nTo summarize we basically looked at how namespaces are used to create isolation on the same machine and how the linux bridge and iptables is used to forward and masquerade packets to enable communications to the outside world.\nResources Tcpdump -\u0026gt; https://hackertarget.com/tcpdump-examples/ IPTables -\u0026gt; https://medium.com/skilluped/what-is-iptables-and-how-to-use-it-781818422e52 NAT -\u0026gt; https://whatismyipaddress.com/nat Newwork Namespace -\u0026gt; https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/ Newwork Namespace -\u0026gt; https://blogs.igalia.com/dpino/2016/04/10/network-namespaces/ https://www.gilesthomas.com/2021/03/fun-with-network-namespaces https://itnext.io/create-your-own-network-namespace-90aaebc745d https://medium.com/@abhishek.amjeet/container-networking-using-namespaces-part1-859d317ca1b8 https://adil.medium.com/container-networking-under-the-hood-network-namespaces-6b2b8fe8dc2a https://iximiuz.com/en/posts/container-networking-is-simple/ https://www.suse.com/c/rancher_blog/introduction-to-container-networking/ ","permalink":"https://rezafarhadur.github.io/posts/network-namespace/","summary":"Network namespaces are useful for setting up containers or virtual environments. Network namespaces provide isolation of the system resources associated with networking: network devices, IPv4 and IPv6 protocol stacks, IP routing tables, firewall rules and so on.","title":"An introduction to linux Network Namespace with hands-on"},{"content":"When we signup on website its send a email for confirmation to active an account. Or sometime need to change password or change email of an account.\nHere i will show you how to send a confirmation email when someone register on your web app that deploy on Django.\nI will discuss about the normal way to deploy this. But there other option also to deploy this. Like django-registration, django-registration-redux, django-allauth application. Those application made this very easy, also integrated with authentication, account management, social account authentication etc.\nLets start\nI have a project called mysite and an app called blog.\nThere is my project directory\ndjango_blog blog admin.py forms.py models.py urls.py views.py templates signup.html acc_active_email.html django_blog settings.py urls.py manage.py Configure settings Firstly we configure email host server in in settings.py for sending confirmation email.\nmysite/settings.py\nEMAIL_USE_TLS = True EMAIL_HOST = \u0026#39;smtp.gmail.com\u0026#39; EMAIL_HOST_USER = \u0026#39;youremail@gmail.com\u0026#39; EMAIL_HOST_PASSWORD = \u0026#39;yourpassword\u0026#39; EMAIL_PORT = 587 Added this lines in your settings.py file.\nHere, I used gmail smtp host server, you can use others smtp server also. If you get error then follow this answer, its says that allow less secure apps and display unlock captcha\nCreate Tokens Now we have to create token that we will send for confirmation.\nCreate a new files named tokens.py in blog app\nblog/tokens.py\nfrom django.contrib.auth.tokens import PasswordResetTokenGenerator from django.utils import six class AccountActivationTokenGenerator(PasswordResetTokenGenerator): def _make_hash_value(self, user, timestamp): return (six.text_type(user.pk) + six.text_type(timestamp)) account_activation_token = AccountActivationTokenGenerator() Basically it extends PasswordResetTokenGenerator to create token.\nDjango forms I think best simple way to implement a user registration page on Django is using UserCreationForm. Here is my forms.py file.\nblog/forms.py\nfrom django import forms from django.contrib.auth.forms import UserCreationForm from django.contrib.auth.models import User class SignupForm(UserCreationForm): email = forms.EmailField(max_length=200, help_text=\u0026#39;Required\u0026#39;) class Meta: model = User fields = (\u0026#39;username\u0026#39;, \u0026#39;email\u0026#39;, \u0026#39;password1\u0026#39;, \u0026#39;password2\u0026#39;) I have import UserCreationForm forms and User models. Then added an extra field email in SignupForm. This email field take email address while registering for email confirmation.\nblog/views.py\nfrom django.http import HttpResponse from django.shortcuts import render, redirect from django.contrib.auth import login, authenticate from .forms import SignupForm from django.contrib.sites.shortcuts import get_current_site from django.utils.encoding import force_bytes, force_text from django.utils.http import urlsafe_base64_encode, urlsafe_base64_decode from django.template.loader import render_to_string from .tokens import account_activation_token from django.contrib.auth.models import User from django.core.mail import EmailMessage def signup(request): if request.method == \u0026#39;POST\u0026#39;: form = SignupForm(request.POST) if form.is_valid(): user = form.save(commit=False) user.is_active = False user.save() current_site = get_current_site(request) subject = \u0026#39;Activate your blog account.\u0026#39; message = render_to_string(\u0026#39;acc_active_email.html\u0026#39;, { \u0026#39;user\u0026#39;:user, \u0026#39;domain\u0026#39;:current_site.domain, \u0026#39;uid\u0026#39;: urlsafe_base64_encode(force_bytes(user.pk)), \u0026#39;token\u0026#39;: account_activation_token.make_token(user), }) # user.email_user(subject, message) toemail = form.cleaned_data.get(\u0026#39;email\u0026#39;) email = EmailMessage(subject, message, to=[toemail]) email.send() return HttpResponse(\u0026#39;Please confirm your email address to complete the registration\u0026#39;) else: form = SignupForm() return render(request, \u0026#39;signup.html\u0026#39;, {\u0026#39;form\u0026#39;: form}) Here it got the form information using POST method, then valid it. Notice that i have write user.is_active = False so that user can\u0026rsquo;t login without email confirmation.\nThen write email subject, message and send it by EmailMessage() function. Email message create by a template.\nblog/templates/acc_active_email.html:\n{% autoescape off %} Hi {{ user.username }}, Please click on the link to confirm your registration, http://{{ domain }}{% url \u0026#39;activate\u0026#39; uidb64=uid token=token %} {% endautoescape %} This template create a email body with activate link that will send for application.\nActivate function User will get an activate link to their email address. Now we have to active their account through activation link.\nBy clicking on activation link the user send to the activate view.\nblog/views.py\ndef activate(request, uidb64, token): try: uid = force_text(urlsafe_base64_decode(uidb64)) user = User.objects.get(pk=uid) except(TypeError, ValueError, OverflowError, User.DoesNotExist): user = None if user is not None and account_activation_token.check_token(user, token): user.is_active = True user.save() login(request, user) # return redirect(\u0026#39;home\u0026#39;) return HttpResponse(\u0026#39;Thank you for your email confirmation. Now you can login your account.\u0026#39;) else: return HttpResponse(\u0026#39;Activation link is invalid!\u0026#39;) Added this activate function after signup function in blog/views.py file. This function will check token if it valid then user will active and login. Notice that i have write user.is_active=True. Before confirming email this was False.\nUrls blog/urls\nfrom django.conf.urls import url from . import views urlpatterns = [ url(r\u0026#39;^$\u0026#39;, views.home, name=\u0026#39;home\u0026#39;), url(r\u0026#39;^signup/$\u0026#39;, views.signup, name=\u0026#39;signup\u0026#39;), url(r\u0026#39;^activate/(?P\u0026lt;uidb64\u0026gt;[0-9A-Za-z_\\-]+)/(?P\u0026lt;token\u0026gt;[0-9A-Za-z]{1,13}-[0-9A-Za-z]{1,20})/$\u0026#39;, views.activate, name=\u0026#39;activate\u0026#39;), ] This is the urls for this app. Last url for email activate confirmation.\nSign up Template Finally we have create the signup template, that user use for registration.\nblog/templates/signup.html\n{% block content %} \u0026lt;h2\u0026gt;Sign up\u0026lt;/h2\u0026gt; \u0026lt;form method=\u0026#34;post\u0026#34;\u0026gt; {% csrf_token %} {% for field in form %} \u0026lt;p\u0026gt; {{ field.label_tag }}\u0026lt;br\u0026gt; {{ field }} {% if field.help_text %} \u0026lt;small style=\u0026#34;display: none\u0026#34;\u0026gt;{{ field.help_text }}\u0026lt;/small\u0026gt; {% endif %} {% for error in field.errors %} \u0026lt;p style=\u0026#34;color: red\u0026#34;\u0026gt;{{ error }}\u0026lt;/p\u0026gt; {% endfor %} \u0026lt;/p\u0026gt; {% endfor %} \u0026lt;button type=\u0026#34;submit\u0026#34;\u0026gt;Sign up\u0026lt;/button\u0026gt; \u0026lt;/form\u0026gt; {% endblock %} This template will shown like this\n","permalink":"https://rezafarhadur.github.io/posts/django-registration-with-confirmation-email/","summary":"When we signup on website its send a email for confirmation to active an account. Or sometime need to change password or change email of an account.","title":"Django registration with confirmation email"},{"content":"Donald Knuth once wrote these unfortunate words:\nWe should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.\nProfiling is a technique for measuring execution times and numbers of invocations of procedures. Simply profiling is a way to identify time issues in applications.\nWhy you need a profiler? It’s all about improving the performance of your code. Its hard to find out where complex program spending time by just looking at code. It helps to find where hot code is and what can you do to speed it up. Optimizing your code without measuring where it’s taking time is useless exercise.\nProfiling help us answer questions such as: Where application using their time? how much memory is the program using. how much time is spent on functions? how much time is spent on statements? Which part of code need to optimize? Python has couple of profiler built in standard library, like profile and cProfile cProfile is a handy tool and recommend for most users. We will focus on cProfile here.\nProfiling code with cProfile cProfile is very handy tool for profiling python code. We just need to import cProfile and used it. Here we will go through a very simple example.\nimport cProfile cProfile.run('1+1') First import cProfile and profile 1+1 by calling cProfile.run(). It give us:\n3 function calls in 0.000 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;) 1 0.000 0.000 0.000 0.000 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} Here we some two numbers. The first line shows it call 3 functions and take 0 seconds. Next line ordered by standard name, indicates that the text string in the far right column was used to sort the output - according to docs Heading columns are, * ncall - the number of call made * tottime - total time spent in the given function * percall - quotient of tottime by ncall * cumtime - cumulative time spent in this and all subfunctions * percall - quotient of cumtime by primitive calls. A primitive call is one that was not induced via recursion. * filename:lineno(function) - respective data of each function\nThis example calls in 0 sec so there is no measurement. Lets see another example,\n!/usr/bin/python3 import time def main(): sum = 0 for i in range(5): sum += sleep_func(i) return sum def sleep_func(sleep_time): time.sleep(sleep_time) return sleep_time if __name__ == '__main__': main() When the program execute it will run main function. Inside that there is a loop that will call sleep_func function 5 times and finally return the summation. In sleep_func it call time.sleep that will pause program. Lets profile this and see what results come,\npython3 -m cProfile profiling.py\nresults:\n15 function calls in 10.007 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 5 0.000 0.000 10.007 2.001 cprofiling.py:12(sleep_func) 1 0.000 0.000 10.007 10.007 cprofiling.py:3(\u0026lt;module\u0026gt;) 1 0.000 0.000 10.007 10.007 cprofiling.py:5(main) 1 0.000 0.000 10.007 10.007 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {built-in method builtins.print} 5 10.007 2.001 10.007 2.001 {built-in method time.sleep} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} Now we can see that our program takes 10.007 seconds and call 15 functions. And all details are here. In details we can see sleep_func is most calls function that calls time.sleep and take enough time to run. So we identify that sleep_func is the culprit that taking most of time in our program. we know that the best way to speed up the function is to remove the time.sleep call.\nIn our results its hard to find out which function calls most or taking most total time. If the result is sorted by ncalls or tottime then we can find it easily. Luckily there -s argument exists. just run this,\npython3 -m cProfile -s calls profiling.py\n15 function calls in 10.011 seconds Ordered by: call count ncalls tottime percall cumtime percall filename:lineno(function) 5 0.000 0.000 10.010 2.002 cprofiling.py:12(sleep_func) 5 10.010 2.002 10.010 2.002 {built-in method time.sleep} 1 0.000 0.000 10.011 10.011 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {built-in method builtins.print} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 10.011 10.011 cprofiling.py:5(main) 1 0.000 0.000 10.011 10.011 cprofiling.py:3(\u0026lt;module\u0026gt;) Now we get a sorted result that sorted by calls. If you want to sort by tottime just replace calls with tottime\nNow if we want save the results in a file pass -o arguments and output file name. Like,\npython3 -m cProfile -s tottime -o output.txt profiling.py\nUnfortunately you can’t read the data. To read the data we need pstats,\nimport pstats p = pstats.Stats('output.txt') p.strip_dirs().sort_stats(-1).print_stats() This will give a result that we seen previously. To know more about pstats\nNow cProfile module help us to identify where the hot code is and take steps to speed it up.\nThere are also several other 3rd party modules that are good for profiling such as the line_profiler , memory_profiler , Interactive continuous profiler .\nPROFILING RESOURCES Python Documentation profile, cProfile and pstats Python 102: How to Profile Your Code ","permalink":"https://rezafarhadur.github.io/posts/profiling-python-code/","summary":"Profiling is a technique for measuring execution times and numbers of invocations of procedures. Simply profiling is a way to identify time issues in applications.","title":"Profiling python code"},{"content":"Twitter is a very popular social media. There are many bots on twitter. Here we will create a bot using Python and a python library called Tweepy . Tweepy is a great open source library which provide access to Twitter API and its help more on streaming API support. “Tweepy is an easy-to-use Python library for accessing the Twitter API”\nThis blog is orginaly published on medium. To read more, go here :)\n","permalink":"https://rezafarhadur.github.io/posts/twitter-bot-with-python-twepy/","summary":"Tweepy is an easy-to-use Python library for accessing the Twitter API","title":"Create a simple Twitter Bot with Python and Tweepy"},{"content":"Hi there 👋\nI\u0026rsquo;m Farhadur Reza — a Senior Software Engineer at Optimizely, based in Dhaka, Bangladesh.\nI spend my days building and scaling backend systems, and I care deeply about clean architecture, reliable infrastructure, and writing code that\u0026rsquo;s easy to reason about. This blog, The Backend Blueprint, is where I write it all down: deep dives, notes to my future self, and the lessons I pick up along the way.\nBio.link — link to all my social accounts.\nContact me — reach out with any questions or suggestions.\n","permalink":"https://rezafarhadur.github.io/about/","summary":"\u003cp\u003eHi there 👋\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m \u003cstrong\u003eFarhadur Reza\u003c/strong\u003e — a Senior Software Engineer at \u003ca href=\"https://www.optimizely.com\"\u003eOptimizely\u003c/a\u003e, based in Dhaka, Bangladesh.\u003c/p\u003e\n\u003cp\u003eI spend my days building and scaling backend systems, and I care deeply about clean architecture, reliable infrastructure, and writing code that\u0026rsquo;s easy to reason about. This blog, \u003cstrong\u003eThe Backend Blueprint\u003c/strong\u003e, is where I write it all down: deep dives, notes to my future self, and the lessons I pick up along the way.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"https://bio.link/rezafarhadur\"\u003eBio.link\u003c/a\u003e — link to all my social accounts.\u003c/p\u003e","title":"About"},{"content":"You can reach out to me at rzfarhadur [at] gmail [dot] com\nOr Say hi in DM @ Linkedin\n","permalink":"https://rezafarhadur.github.io/contact/","summary":"\u003cp\u003eYou can reach out to me at rzfarhadur [at] gmail [dot] com\u003c/p\u003e\n\u003cp\u003eOr Say hi in DM @ \u003ca href=\"https://linkedin.com/in/rezafarhadur\"\u003eLinkedin\u003c/a\u003e\u003c/p\u003e","title":"Contact"}]