21 posts tagged with "Software Testing"

TL;DR: AI coding agents are incredibly powerful at reading and writing source code, but when it comes to understanding what code actually does at runtime, they are blind. They can read `repository.findAll()` in the source code, but they cannot see the 40-query N+1 explosion it causes in production. This post explores how AI routinely breaks APIs and how connecting agents to real runtime data using BitDive and the Model Context Protocol (MCP) dramatically improves their reliability.

TL;DR: Bumping the Spring Boot version in pom.xml takes 10 seconds. Finding out that the upgrade silently changed your JSON serialization, broke your security filters, or altered your error responses takes days. This post covers seven categories of silent breaking changes that Spring Boot upgrades introduce, and a practical workflow to catch them before production.

TL;DR: Your integration test hits the happy path: 200 OK, correct response body, all good. But what happens when the downstream returns a 404? A 500? A 503 with retry-after? If your service parses the error response body, and the body format changes silently, you will find out at 3 AM when the retry storm starts. Error contracts are just as important as success contracts, and almost nobody tests them.

TL;DR: Your service compiles. Your unit tests pass. Your integration tests are green. But a single line in your ObjectMapper configuration just changed what every outgoing HTTP request looks like. The downstream service cannot parse the payload anymore, and you will find out in production. Here are five Jackson configuration changes that cause this, with exact before/after JSON for each.

TL;DR: The most dangerous bugs in microservices are not inside a service. They are between services. A code change can make a service pass all its local tests while silently altering what it sends to downstream APIs: different payload, missing header, changed error format. These regressions are invisible to unit tests, hard to catch with contract tests, and expensive in production. BitDive detects them by capturing real HTTP exchanges in execution traces and comparing them before and after a code change.

TL;DR: Boot the full Spring context. Stub only what lives outside your service boundary: Feign clients, external HTTP APIs, outbound Kafka. Then hit the service through its real HTTP endpoint and verify the entire chain: controller, validation, service logic, @Transactional, repository, database write, response serialization. This is what Spring calls an integration test. It catches the class of bugs that unit tests structurally miss: broken configs, silent serialization changes, transaction proxy bypass, security filter misconfiguration, and DTO contract drift.

TL;DR: In real-world Spring projects, the "unit vs integration" debate almost always stems from the fact that "integration testing" has become a catch-all term for everything from @SpringBootTest with Testcontainers to full-blown E2E runs on staging environments. To stop arguing and start shipping, we need to draw a clear line in the sand regarding responsibility.

A unit test answers one question: "Is the logic correct in total isolation?" It deliberately cuts infrastructure out of the equation.

A component test answers another: "Does the component work as a system within its own boundaries, including its Spring wiring, configurations, serialization, transactions, and data access?"

If you only have units, you'll inevitably get burned at the seams. If you only have component tests, you'll pay with execution time, flakiness, and painful debugging. The winning strategy is simple: unit tests provide the speed and density of logic verification; component tests provide the confidence that the "real assembly" actually works.

TL;DR: As AI models like Claude, GPT-4, and Gemini write more of our code, the bottleneck has shifted from writing to verifying. Traditional mock-heavy tests are too fragile for AI-native workflows. BitDive provides the Real Runtime Data needed to turn actual execution states into deterministic JUnit tests, enabling a safe and autonomous development loop.

The Verification Gap in AI-Native Development​

In 2026, the industry has reached a tipping point. AI assistants can now create 1,000 lines of functional code in seconds. However, verifying that this code doesn't break subtle production invariants remains a manual, slow process.

We call this the Verification Gap.

We’re excited to introduce a major upgrade to the BitDive ecosystem: native Postman support.

You can now create a Postman Collection from your BitDive-generated JUnit replay tests with a single click. Postman export is an additional execution format alongside the BitDive runner, derived from the same Real Runtime Data replay artifacts, so you can rerun and share endpoint requests without changing your regression suite.

BitDive automatically creates unit and integration tests from recorded behavior, and it can now also export the tested application’s endpoint requests as Postman collections. This lets you invoke endpoints quickly, capture new executions back into BitDive, validate Consumer-Driven Contracts against real scenarios, and compare “before” vs “after” behavior with diffs after a fix.

To solve the Verification Crisis, teams must move from manual mocking to Runtime Context Sharing. By integrating BitDive via Model Context Protocol (MCP), AI agents gain access to real execution traces, allowing them to propose surgical fixes and self-verify their work against Real Runtime Data. This is more than just automation; it is the Deterministic Verification Layer required for the AI-native developer.

"We're now cooperating with AIs and usually they are doing the creation and we as humans are doing the verification. It is in our interest to make this loop go as fast as possible. So, we're getting a lot of work done."

. Andrej Karpathy: Software Is Changing (Again)

This quote describes a shift that is already visible in many teams. Code creation has accelerated. Verification and validation increasingly become the bottleneck.

With AI tools, writing code is often not the limiting factor anymore. The hard part is proving that what was generated is correct, safe, and maintainable.