Rapid expansion of data in digital ecosystems today demands scalable and efficient architectures for real-time ingestion and analytics. In this paper, we propose and design a hybrid architecture deployed by Apache Kafka and intelligent data systems, which support on-the-fly big data ingestions, computations and query processing. The system is based on using distributed stream processing with Kafka for high-throughput ingestion of data into intelligent databases like PostgreSQL with AI-based indexing and query optimization for context-aware querying at low-latencies. In this work, we present a modular pipeline that enables schema evolution, fault-tolerance, and intelligent workload management. Experiments show that we achieved high performance gains on both data ingestion speed, query response latency and system scalability over the traditional ETL based solutions. This combination architecture forms the basis for mission-critical applications that require real-time insights, such as IoT analytics, financial monitoring and healthcare informatics applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Hybrid Architecture for Real-Time Data Ingestion and Querying Using Kafka and Intelligent Database Systems

  • Santhosh Kumar Somarapu,
  • Harish Chava,
  • Raghu Ram Bojanapalli

摘要

Rapid expansion of data in digital ecosystems today demands scalable and efficient architectures for real-time ingestion and analytics. In this paper, we propose and design a hybrid architecture deployed by Apache Kafka and intelligent data systems, which support on-the-fly big data ingestions, computations and query processing. The system is based on using distributed stream processing with Kafka for high-throughput ingestion of data into intelligent databases like PostgreSQL with AI-based indexing and query optimization for context-aware querying at low-latencies. In this work, we present a modular pipeline that enables schema evolution, fault-tolerance, and intelligent workload management. Experiments show that we achieved high performance gains on both data ingestion speed, query response latency and system scalability over the traditional ETL based solutions. This combination architecture forms the basis for mission-critical applications that require real-time insights, such as IoT analytics, financial monitoring and healthcare informatics applications.