Skip to main content

BitDive vs. Contract Testing (Pact): Real API Behavior vs. Predefined Examples

Contract testing tools like Pact and Spring Cloud Contract have become a standard practice for preventing API breakage between microservices. They solve a real problem: verifying that a provider still conforms to what consumers expect.

But there is a class of API regressions that static contracts structurally cannot detect, because they happen at the runtime level: serialization changes, header drift, error body mutations, and altered call sequences. This is exactly where BitDive's trace-based approach fills the gap.

The Comparison at a Glance​

DimensionBitDiveContract Testing (Pact)
What it comparesReal HTTP exchanges captured at runtimePredefined contract examples
Data sourceActual serialized payloads from running servicesHand-written or consumer-driven examples
Serialization changesDetected (Jackson, ObjectMapper, enum, date/time)Not covered unless example includes exact field
Header driftFull header comparison as part of the contractRarely tested in contracts
Error contract changesDetected (status code, error body format)Only if a negative example exists
Call sequence changesDetected (order and count of API calls)Not covered
Request body driftField-level diff on real serialized JSONOnly fields defined in the contract
Null handling changesDetected from actual payloadsOnly if explicit example covers it
Setup effortRecord a trace (no contract writing)Write consumer expectations per interaction
MaintenanceRe-record when behavior changesUpdate contracts when API evolves
AI integrationNative MCP for agent verificationNone

What Contract Testing Does Well​

Contract testing is effective when you need to:

  • Formalize expectations between teams that own different services.
  • Prevent breaking changes in public APIs where consumers define what they need.
  • Shift left by catching incompatibilities before deployment.
  • Document API behavior as executable specifications.

Pact, Spring Cloud Contract, and similar tools excel at this because they force explicit agreement on what a provider must return for specific inputs.

Where Contract Testing Falls Short​

1. Contracts Only Cover What You Write​

A Pact contract contains specific request/response examples. If a field is not in the example, its behavior is not verified.

Common blind spots:

  • A field that was always present but is now null due to a mapper change.
  • An enum that serialized as "ACTIVE" but now serializes as 0 after a Jackson update.
  • A BigDecimal that was "19.99" but is now 19.99 (string vs number).
  • A date that was "2026-02-28" but is now 1740700800000 after enabling WRITE_DATES_AS_TIMESTAMPS.

These regressions pass contract tests because the contract did not specify the exact serialization format for those fields.

BitDive catches them because it compares the actual serialized JSON from the real HTTP exchange.

2. Headers Are Invisible​

Most contract tests focus on request/response bodies. But in enterprise systems, critical behavior depends on headers:

  • Authorization / X-Api-Key
  • Content-Type (especially charset changes)
  • X-Correlation-Id / traceparent
  • Tenant headers, locale headers, feature flags

A missing or renamed header can completely change downstream behavior. Contract tests rarely verify headers. BitDive traces include full request and response headers.

3. Error Paths Are Undertested​

Contract tests usually verify the happy path. Negative scenarios require explicit examples, which teams often skip.

A typical regression: the downstream changed its error response from {"code": "NOT_FOUND", "message": "..."} to a plain 404 Not Found HTML page. The upstream service cannot parse the error and enters uncontrolled behavior.

BitDive's before/after comparison covers error responses because they are part of the captured trace.

4. Call Sequence Is Not Validated​

Contract tests verify individual interactions in isolation. They do not validate the order in which a service calls its dependencies during a business scenario.

Example: after refactoring, a service that used to call GET /accounts then POST /transactions now calls them in reverse order, or skips the accounts call entirely. Contract tests for each endpoint pass individually. The business scenario is broken.

BitDive links API calls to a full trace, so it detects sequence changes, missing calls, and unexpected new calls.

5. Infrastructure Changes Are Invisible​

Some of the most dangerous API regressions come not from business logic changes but from infrastructure updates:

  • Jackson version upgrade
  • ObjectMapper reconfiguration
  • DTO mapper library update (MapStruct, ModelMapper)
  • Spring Boot version bump changing default serialization
  • Include.NON_NULL toggled on or off

These changes affect every outgoing HTTP call but touch zero business code. Contract tests stay green. BitDive detects the runtime payload difference.

How BitDive Detects API Regressions​

BitDive's approach is fundamentally different from contract validation:

  1. Capture a trace of a business scenario (includes all outgoing HTTP calls with full payload details).
  2. This trace becomes the baseline runtime API contract.
  3. After a code change, replay the scenario or capture a new trace.
  4. Compare before/after across every layer:
    • Endpoint and HTTP method
    • Request headers
    • Serialized request body (field-level JSON diff)
    • Response status
    • Response headers
    • Response body (field-level JSON diff)
    • Call sequence (order, count, presence/absence)
  5. Classify differences: acceptable changes (new optional field, updated trace header) vs. breaking changes (missing required field, altered error body, changed status code).

The result is a deterministic, automated comparison of real API behavior, not a validation against predefined examples.

When to Use Each Approach​

Use Contract Testing (Pact) if:​

  • You need formal API governance between teams that own separate services.
  • Your API is public or versioned and consumers define their expectations.
  • You want to document expected provider behavior as executable specs.
  • You are in a polyglot environment where services use different languages.

Use BitDive if:​

  • You want to detect infrastructure-driven regressions (Jackson, ObjectMapper, DTO mappers) that contracts miss.
  • You need full HTTP exchange comparison including headers, error bodies, and serialization details.
  • You are doing refactoring, library upgrades, or framework migrations and need to verify nothing changed in inter-service communication.
  • You use AI-assisted development and want agents to verify that their changes did not alter API behavior via MCP.
  • You want zero-maintenance regression checks without writing and updating contract examples.

Use Both Together:​

Contract testing defines the expected contract. BitDive verifies that the actual runtime behavior still matches after every code change. Together, they provide both formal governance and runtime safety.

Frequently Asked Questions​

Does BitDive replace Pact?​

No. Pact validates that a service conforms to a predefined contract. BitDive detects whether actual API behavior changed after a code modification. They solve different problems and work well together: Pact for governance, BitDive for runtime regression detection.

Can BitDive detect API breaking changes that Pact misses?​

Yes. BitDive captures actual serialized HTTP exchanges at runtime. This means it detects regressions caused by Jackson configuration changes, DTO mapper updates, missing headers, altered error bodies, and changed call sequences, all of which fall outside the scope of predefined contract examples.

How does BitDive compare API behavior before and after a code change?​

BitDive captures execution traces that include all outgoing HTTP calls: endpoint, HTTP method, headers, serialized request body, response status, response headers, and response body. After a code change, the same scenario is replayed and a new trace is captured. BitDive performs a field-level diff across all these layers.

Does BitDive work with Spring Cloud Contract?​

BitDive is compatible with any Spring Boot application. If you already use Spring Cloud Contract for provider verification, BitDive adds a runtime safety layer that catches regressions outside the contract scope, particularly useful during infrastructure changes, upgrades, and AI-assisted modifications.

Testing Strategy Full-Cycle Testing: From Traces to Regression Safety

Glossary What is a Runtime API Contract?

Last updated on by turmyshevd