Skip to main content

System Architecture and Secure Data Processing

Client Library Integration​

The system provides a dedicated Java library for client integration.

Integration Details​

Maven Integration​

<dependency>
    <groupId>io.bitdive</groupId>
    <artifactId>bitdive-producer-spring-3</artifactId>
    <version>0.0.15</version>
</dependency>

Usage​

  1. Add the dependency to your Java application
  2. Initialize the client with appropriate configuration
  3. Use the provided methods for secure file processing

System Architecture Overview​

The system consists of multiple microservices designed to handle secure file processing, storage, and data management.

Architecture Diagram​

BitDive system architecture showing the interaction between core services, infrastructure services, and security components

Key Components​

Core Services​

  • Frontend: User interface service
  • File Acceptor: Handles file reception and signature verification
  • Flink Load: Processes files and manages database operations
  • Monitoring API: Handles UI data processing and monitoring

Infrastructure Services​

  • Vault: Manages secrets and SSL certificates
  • PostgreSQL: Primary database for structured data
  • MinIO: Object storage for files
  • Keycloak: Identity and access management

Security Features​

  • Automated SSL certificate generation on container startup
  • Vault-based SSL certificate management
  • Secure service-to-service communication

File Transfer Process​

This section describes the secure process of transferring files from client to server.

Process Diagram​

Secure file transfer process flow diagram showing steps from client preparation to server storage

Process Steps​

  1. Initial File Preparation

    • Java application prepares and archives files
    • Files are prepared for secure transfer

  2. Key Management

    • Public key retrieval for signature verification (24-hour rotation)
    • Encryption and signing key management
    • Local key storage for high-volume processing

  3. Security Measures

    • AES/GCM/NoPadding encryption
    • SHA256withECDSA file signatures
    • SSL-secured transfer

  4. File Storage

    • Signature verification on receipt
    • Encrypted storage in MinIO

File Processing and Database Integration​

Details the process of processing files and integrating data into the database.

Process Diagram​

File processing workflow diagram illustrating the steps from retrieval through database integration

Process Flow​

  1. File Retrieval

    • Secure retrieval from MinIO storage
    • Encrypted file handling

  2. Processing

    • Secure key retrieval from Vault
    • File decryption
    • Data processing and grouping

  3. Database Integration

    • SSL certificate management (24-hour rotation)
    • AES CBC mode encryption
    • Secure PostgreSQL integration

Frontend Operations​

Describes the frontend interaction with backend services.

Process Diagram​

Frontend operations diagram showing the interaction between UI, API, and security layers

Security Considerations​

  1. SSL Implementation

    • SSL connection support
    • Company-wide certificate distribution required
    • Browser trust management

  2. Data Flow

    • Secure communication with Monitoring API
    • SSL-protected database queries
    • Encrypted data handling

Important Notes​

  • SSL certificates must be properly distributed to prevent browser trust issues
  • All production data access requires SSL encryption
  • Regular certificate rotation ensures security
  • Browser security warnings will appear without proper certificate setup

Security Best Practices​

  1. Regular certificate rotation (24-hour cycle)
  2. Encrypted data storage
  3. Secure key management
  4. SSL implementation throughout
  5. Regular security audits
  6. Proper certificate distribution

System Requirements​

  • Docker and Docker Compose
  • Java Runtime Environment
  • SSL Certificate Infrastructure
  • Adequate Storage for MinIO
  • PostgreSQL Database System
  • Network Security Infrastructure

Troubleshooting​

  1. Browser Trust Issues

    • Verify SSL certificate distribution
    • Check certificate validity
    • Ensure proper certificate installation

  2. File Transfer Issues

    • Verify key availability
    • Check SSL certificate status
    • Confirm proper encryption

  3. Database Connection Issues

    • Verify SSL configuration
    • Check certificate rotation
    • Confirm security protocol compliance