Confluent announced new capabilities for building a Streamhouse, a new type of data architecture that keeps business context current for AI agents, applications and analytics as they make decisions and take action. With Lightning Tables now generally available on AWS, Confluent completes its five-pillar Data Streaming Platform, bringing capabilities to Stream, Connect, Process, Govern and now Query data together in a unified platform for building a Streamhouse. Confluent sits at the centre of IBM’s data strategy, combining real-time data streaming with IBM’s integration, governance and AI capabilities across hybrid environments.
Why enterprises need a Streamhouse
Data lakes, warehouses and lakehouses were built primarily with reporting or other batch jobs in mind. As AI agents take on more work across the enterprise, they must make decisions using information spread across systems that update at different times. When those systems have different views of the current state of the business, agents can act on outdated or conflicting information. Those actions can carry into other workflows.
A Streamhouse addresses this architectural problem by continuously capturing, processing, governing and serving current business context across agents, applications and analytics. Consider the following illustrative example of an airline managing a canceled flight:
- Today, it has different systems holding conflicting views of the same operation. Flight operations track where the aircraft is, scheduling tracks when it will depart, and reservations tracks which seats are available. When those systems update at different times, an autonomous rebooking agent can assemble an itinerary that looks valid in each system but does not work in practice: a seat is available, but the aircraft cannot arrive in time for the passenger’s connection. Customer-service teams and notification systems can then act on those same conflicting views, spreading the disruption.
- With a Streamhouse, each change can update a shared picture of the airline’s operations. When an aircraft’s location changes, a departure is delayed or a seat is booked, the source system sends that update into a continuous flow of events. A Streamhouse combines those updates into current business context and makes it available to rebooking agents, customer-service teams and applications as they work. Instead of piecing together separately updated information, they can check aircraft location, departure time and seat availability together before offering a passenger a new itinerary.
Enterprises are already seeing results. Accenture uses a Streamhouse architecture powered by Confluent to connect data across its global business.
“Operating across more than 120 countries, our global enterprise was previously constrained by legacy batch and point-to-point architectures where duplicative data extractions drove up operational overhead and analytics data lagged. Moving to a Streamhouse architecture powered by Confluent decouples data producers from consumers, transforming high-volume event streams into governed, reusable real-time data products that flow seamlessly across operational and analytical systems. With Confluent handling infrastructure automation and providing seamless enterprise connectivity, our developers operate in a standardised environment without the friction of building custom integrations or micro-managing data movement. This continuous data foundation has dropped our time-to-insight from months to days and slashed reporting times, empowering our teams to focus on delivering high-value AI for our 800,000 employees,” said Thomas Mahaffey, SAI Chief Data Strategist, Accenture.
What makes a Streamhouse different
A Streamhouse is built on three architectural principles: current, mission-critical and decentralised. It maintains continuously updated business context, is engineered to production service levels, and meets data where it lives without requiring consolidation first. It works alongside existing data warehouses and lakehouses, adding a real-time operational layer for decisions and actions.
Confluent’s Data Streaming Platform Now Offers a Complete Streamhouse
Confluent’s Data Streaming Platform (DSP) provides a complete, unified foundation for building a Streamhouse. At its core is Apache Kafka, which captures and streams business events. On its own, Kafka leaves teams to assemble everything else needed to turn those events into current business context. The DSP brings that together in one platform, now adding Query as its fifth pillar alongside Stream, Connect, Process and Govern. Core capabilities and latest additions to each platform pillar include:
- Stream, built on Apache Kafka and the Kora engine, provides the durable, scalable foundation for capturing and streaming business events across systems with sub-millisecond latency in Confluent internal testing; actual latency varies by configuration and workload.
- Connect, with 150+ fully managed and custom connectors, brings data from enterprise systems into Kafka and delivers it downstream. Hybrid connector management extends how teams manage those connections.
- Process uses Apache Flink to combine, enrich and derive business context close to the source as data changes. Additional batch execution capabilities help initialise streaming workloads from existing data.
- Govern helps teams discover, classify and secure data through lineage, catalog, quality, encryption, schema and access-control capabilities.
- Query serves current business context to analytics, agents and applications.
Query puts streaming data to work for analytics, agents and applications, each through the interface that fits the job:
- Confluent Tableflow makes streaming data available to lakehouses and warehouses as open table formats, ready for analytics.
- Real-Time Context Engine serves current business context to AI agents through the Model Context Protocol (MCP).
- Lightning Tables gives applications low-latency SQL and API access to the current state of the business, so they can respond as conditions change.
Lightning Tables: Fast access to the current state of the business
Generally available on AWS, Lightning Tables on Confluent Cloud is a read-only serving layer that durably stores and materialises Apache Kafka data into tables, keeping them current as changes arise. It makes the current state of the business available for low-latency queries. It features:
- Data freshness and query performance: In Confluent internal testing, approximately 100 milliseconds from new data creation to serving, with query latency of 10-100 milliseconds and throughput of 1,000+ queries per second.
- Lower cost and complexity: Serve continuously with a fully managed serving layer, without separate ETL pipelines, databases, or caches.
- Multiple query interfaces: Query through SQL and a direct API.
- Unified governance: Apply schemas, lineage and access policies from data capture through serving.
- Schema evolution: Manage schema changes to help prevent breaks in downstream applications.
- Feature store: Compute features in Flink and store them for machine learning use cases, including real-time personalisation.
“The next era of enterprise AI will be won by the agents that know what’s happening in the business the moment they act,” said Shaun Clowes, General Manager, Confluent and Data, IBM. “That’s what a Streamhouse can deliver. With Lightning Tables, our Data Streaming Platform gives each agent and application a live, governed view of the business. This means companies can act on fraud signals sooner, serve customers in the moment and run operations on what’s true now versus what was true yesterday.”
Discussion about this post