Testing Distributed Systems at Scale: Running Thousands of Tests Without Expensive Deployments(Part 2)

In-Process Integration Testing

Overview

In Part 1, we discussed why validating a distributed application needs more than UTs and automated UI tests. We need to validate network interactions, DB state, business logic, and internal service communication.

But there is another important problem - how do we execute these tests efficiently?

Let's assume we have an application with 20 microservices, PostgreSQL, queues and external integrations with AWS and Salesforce.

A traditional E2E test might require us to deploy the services, configure networking, set up dependencies, initialize test data, execute the workflow, and clean up the environment.

When AI agents are generating and modifying code continuously, deploying the entire system for every iteration can become very expensive in terms of time and infrastructure cost.

One alternative is in-process integration testing, where we run the real application logic of multiple services inside a single process, instead of deploying every service separately.

In this blog, we'll explore how it can work with Python and FastAPI, even though the concept is applicable to any language or environment, and how to handle the external dependencies and the limitations we need to keep in mind.

UTs vs integration tests vs E2E tests

The main difference is which boundaries we are testing.

1. Unit tests: Validate methods, classes or modules in isolation. They are fast, but usually mock external dependencies.

2. In-process integration tests: Execute real application code across multiple modules or services, with in-memory communication and selected real dependencies.

3. Deployed E2E tests: Execute the application in a real environment with networking, service processes and infrastructure configurations.

Deployed E2E tests are useful, but they are also expensive to set up and execute. We don't want to run every deployed test after every code change.

The idea is not to eliminate E2E tests, but to execute as much functional integration testing as possible without a deployed environment.

How does in-process integration testing work?

Let's assume the application uses:

- Python for the application code.

- FastAPI for HTTP APIs.

- PostgreSQL as the database.

- Multiple services communicating over HTTP.

- A few external APIs and queues.

Normally, each FastAPI application runs behind a server like Uvicorn. Service A makes a network call to reach Service B.

But FastAPI is built on ASGI. Using httpx.ASGITransport, an HTTPX client can call an ASGI application directly in the same Python process.

```python

import httpx

import pytest

from fastapi import FastAPI

app = FastAPI()

@app.get("/inventory/{item_id}")

def get_inventory(item_id: int):

    return {"item_id": item_id, "quantity": 10}

@pytest.mark.asyncio

async def test_inventory():

    transport = httpx.ASGITransport(app=app)

    async with httpx.AsyncClient(

        transport=transport,

        base_url="http://inventory"

    ) as client:

        response = await client.get("/inventory/5")

        assert response.json()["quantity"] == 10

```

In this example:

- The request still goes through FastAPI routing and request/response validation.

- We execute the actual API code, not a mocked response.

- There is no TCP port, Uvicorn process or deployed service.

For multiple services, we'll need a routing layer that maps requests like `http://inventory` and `http://orders` to their corresponding in-memory FastAPI applications.

`ASGITransport` doesn't automatically provide multi-service routing. We can build it using custom HTTPX transports, dependency injection or another routing abstraction. Conceptually, this is similar to the idea of in-memory gRPC transports such as `bufconn` in Go.

What about DBs, queues and third-party systems?

These are more difficult because they maintain state outside the application process.

1. PostgreSQL

We can launch PostgreSQL in a lightweight local Docker container and create the required schema and data before running a test.

Why not use SQLite? It is easier to start, but the transaction behaviour, data types, constraints and SQL support can differ from PostgreSQL. A test passing on SQLite doesn't necessarily mean it will pass on PostgreSQL.

For test isolation, we have a few options:

- Separate schemas or databases in a shared PostgreSQL instance.

- One PostgreSQL container per test worker.

- One container per test where stronger isolation is required.

Launching a separate container for every test can become expensive. The choice depends on how much state needs to be isolated and how easy failures are to reproduce.

2. Third-party simulators

Applications might integrate with AWS, Azure, Salesforce, Jira or other external systems.

We can build or reuse generic simulators for the supported external APIs. For example, an S3 simulator can be reused by any application that needs the same S3 operations.

The simulator only needs to implement the relevant behaviour, and we can extend it as new APIs are required.

But simulators don't always behave exactly like real cloud services. We still need a smaller set of contract or deployed integration tests against actual dependencies.

3. Queues and asynchronous workflows

Queues introduce behaviours like retries, ordering, acknowledgements, and eventual consistency.

For simple functional workflows, a local or in-memory queue simulator might be enough. But to verify queue semantics and failure behaviour, we may need to use a real queue implementation in a separate test environment.

Running tests in parallel

Now assume multiple AI agents are changing different parts of the application and hundreds of tests need to run.

We can create separate Python worker processes, each loading its own set of in-memory services. The main challenge is isolating dependencies that maintain state.

For example:

- Worker 1 uses its own test DB and queue state.

- Worker 2 has a separate DB or schema and independent queue state.

- Worker 3 runs another set of tests with its own data.

We don't need a separate Kubernetes deployment for every worker. However, using separate DB containers for every test might still be costly, so we should support sharing where safe and isolation where required.

Another option worth exploring is an in-process DB, but that can introduce differences from PostgreSQL and make some failures harder to reproduce.

Selective testing - similar to Bazel

Executing all tests after every small code change can still be expensive, even with in-process execution.

Build systems like Bazel use dependency information to avoid unnecessary work. We can apply a similar idea to testing:

1. Identify the changed files, services and API contracts.

2. Map them to dependent services and business workflows.

3. Execute the tests impacted by those changes.

4. Run the broader regression suite periodically and before important releases.

For example, a change in the notification service might only require notification-related functional tests and a few order-placement workflows.

This won't be perfect. Some failures come from unexpected dependencies, configuration changes or shared state. So selective testing should reduce frequent feedback time, not permanently replace the complete regression suite.

What will in-process testing miss?

There are important limitations:

- Network failures, timeouts and connection-pool behaviour.

- Kubernetes service discovery, ingress, TLS and deployment configuration.

- Cross-process concurrency, resource limits and distributed failure scenarios.

- Actual latency and throughput under production-like load.

- Differences between simulators and real external services.

We also need to be careful about side effects of executing multiple services in one Python process, such as shared globals or background tasks behaving differently from production.

In-process test latency should not be used as a measure of production API performance.

Conclusion

In a distributed system, integration testing can become expensive not because the assertions are complicated, but because setting up the environment requires deploying and managing multiple services and dependencies.

For Python and FastAPI applications, ASGI-based in-process communication is one way to remove a lot of that overhead while still executing real application code.

There are open questions around multi-service routing, DB isolation, lifecycle management, queue behaviour and third-party simulators. But starting with one technology stack lets us solve these problems incrementally.

As AI agents generate more code and run more iterations, the speed and cost of validating each change will matter, along with how much behaviour the tests actually cover.