Posts

Validating AI-Generated Distributed Systems - Part 1: Beyond Unit Tests

Image
Overview Writing code is becoming the easiest part of software engineering. AI agents can now generate features, refactor code, and fix bugs with very little manual involvement. But how do we validate that the generated code is actually correct? For a small application, we can run UTs, API tests, and a few automated UI tests to get reasonable confidence. But in highly distributed systems involving microservices, distributed DBs, queues, Kubernetes, and multiple cloud PaaS resources, the problem becomes much more complicated. The API returns a successful response, and the UI shows an order confirmation. Does that mean the feature is correct? Not necessarily. The problem isn't just validating a function's output. It's about validating the behaviour of the complete system, from product coverage to real-world edge cases in a high-scale environment. In this blog, we'll explore the different layers of validation we need and how some existing testing tools approach this proble...

Agentic AI: Harness and Reliability(with code) - Part 3

Image
This is fully written by me without any help from AI, except for learning the concepts. This blog covers everything everyone should be aware of in the AI world, and it took me weeks to learn it so I could make it easy for you. > Practice agentic AI concepts hands-on for interviews and real-world development on AgenticPrep.io . To understand the basics of agents and how to build one, check out Part 2 of the Agentic AI series here . Overview In Part 2, we read about the following: How agents remembers past conversations How agents personalize the replies for you History and checkpointing In this blog , we will be targeting how to make AI Agents more reliable and production ready using Harnesses. Limitation of Agents Harnesses target the major limitations of agents i.e. they are single shot and don't validate the results before you give any output to the user. If you ask an agent to write something, it will do that. But it will not validate the results even if you ask it to valida...

AI Agents: Memory, Persistence and Checkpointing - Part 2

Image
This is fully written by me without any help from AI, except for learning the concepts. This blog covers everything everyone should be aware of in the AI world, and it took me weeks to learn it so I could make it easy for you. > Practice agentic AI concepts hands-on for interviews and real-world development on AgenticPrep.io . To understand the basics of agents and how to build one, check out Part 1 of the Agentic AI series here .  Overview In Part 1, we read about the following: What are LLMs and how do they work How LLMs transform into agents to do real-world work Limitations of LLMs and AI Agents In this blog , we will be targeting one of the major limitations of LLM-based agents - memory, persistence, and checkpointing. LLMs are stateless, so they don't remember their past state and chats whenever they are called(through APIs), yet tools like OpenAI ChatGPT, Claude Code, Codex, etc. remember: The whole chat context from the very first message to the current message, even aft...

AI Agents: What Actually Happens Internally - Part 1(Basics)

Image
This is fully written by me without any help from AI, except for learning the concepts. This blog consists of all the things that everyone should be aware of in the AI world, which took me weeks to learn to make it easy for you. > Practice agentic AI concepts hands-on for interviews and real-world development on AgenticPrep.io . Overview In the world of AI, LLMs are not sufficient. LLMs are simple(and yet complex) until you convert them to agents. This is where all the powers are unlocked, and LLMs can be used to do anything and everything in the world. This blog explains how a chatbot(or simple LLM) is actually converted into an AI Agent to do real-world, productive things. I'll explain from the basics what LLMs are, how they should be used, and how agents are created, but on a high level: LLMs(or large language models) are models trained(like Gemini, Opus, etc) on a huge amount of data and can be considered as "a brain in a glass jar" that can dictate what to do. Ag...

Replication Explained - Designing Distributed Systems (Part 2)

Image
For Part 1, check this out Designing Distributed File Storage Systems - Explained with Google File System(GFS) > Practice agentic AI concepts hands-on for interviews and real-world development on AgenticPrep.io . Overview: Replication in distributed systems involves the process in which we can have multiple copies of data at various locations to guard against unknown and random hardware failures, hence ensuring the availability of the system as a whole. While considering replication, we have to consider the basic assumption that machines will have uncorrelated failures , otherwise, replication will not help in any way. Replication should be considered or not, the number of replicas, etc all depends on the use case and the amount of inconvenience or how much it will cost you if you lose the data and compute power at a given point in time. Replication is achieved, intuitively , when we have two servers, one primary and a replica server, and we have to keep them in synchronize in a w...

Designing Distributed File Storage Systems - Explained with Google File System(GFS) - Part 1

Image
Part 2: Replication-Explained: Designing Distributed systems Overview: Original paper: The Google File System Authors: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung > Practice agentic AI concepts hands-on for interviews and real-world development on AgenticPrep.io . A distributed file system(DFS) is a file system that allows you to access files stored in multiple hosts distributed across them. In contrast to other data stores like MySQL or NoSQL which can also be made distributed, the data is DFS is stored in raw form and as files instead of a table or a fixed-document format. The use cases where DFS is used are very vast. From storing media content like Youtube videos to Instagram images to blogs like these, anything that can be stored as a file, is big and is not suitable to store in other data stores, DFS is generally a preferred choice for everyone. Key points to consider while designing a DFS: Understanding from the perspective of using a distributed file system, the...

Privacy Protection Algorithms

Objective: The objective of this work is to understand automated text anonymization system for protection of personal information of users and after anonymization, should still remain relevant in syntactic and semantics terms, without losing the conveyed meaning of text. Anonymized data can be used in many tasks such as data mining, machine learning, analysis, etc without revealing the identity of entities involved in the creation of data and be useful to improve the accuracy of the applied data analysis tool.   > Practice agentic AI concepts hands-on for interviews and real-world development on AgenticPrep.io . Approach: Beyond simply recognizing entities, variation in the types of documents (e.g., financial or medical) and type of identifying information poses a challenge for automated systems. One common approach is to develop domain specific anonymization tools, where one can utilize knowledge about the structure and information content of documents to construct h...