Dynatrace announced the launch of OpenPipeline, a new core technology that provides customers with a single pipeline to manage petabyte-scale data ingestion into the Dynatrace platform to fuel secure and cost-effective analytics, AI, and automation.
Dynatrace OpenPipeline empowers business, development, security, and operations teams with full visibility into and control of the data they are ingesting into the Dynatrace platform while preserving the context of the data and the cloud ecosystems where they originate. Additionally, it evaluates data streams five to ten times faster than legacy technologies. As a result, organisations can better manage the ever-increasing volume and variety of data emanating from their hybrid and multi-cloud environments and empower more teams to access the Dynatrace platform’s AI-powered answers and automations without requiring additional tools.
“Data is the lifeblood of our business. It contains valuable insights and enables automation that frees our teams from low-value tasks,” said Alex Hibbitt, Engineering Director, SRE and Fulfillment, albelli-Photobox Group. “However, we face challenges in managing our data pipelines securely and cost effectively. Adding OpenPipeline to Dynatrace extends the value of the platform. It enables us to manage data from a broad spectrum of sources alongside real-time data collected natively in Dynatrace, all in one single platform, allowing us to make better-informed decisions.”
According to Gartner, “Modern workloads generate increasing volumes—hundreds of terabytes and even petabytes each day—of telemetry originating from a variety of sources. This threatens to overwhelm the operators responsible for availability, performance, and security. The cost and complexity associated with managing this data can be more than $10 million per year in large enterprises.”
Creating a unified pipeline to manage this data is challenging due to the complexity of modern cloud architectures. This complexity and the proliferation of monitoring and analytics tools within organisations can strain budgets. At the same time, organisations need to comply with security and privacy standards, such as GDPR and HIPAA, in relation to their data pipelines, analytics, and automations. Despite these challenges, stakeholders across organisations are seeking more data-driven insights and automation to make better decisions, improve productivity, and reduce costs. Therefore, they require clear visibility and control over their data while managing costs and maximising the value of their existing data analytics and automation solutions.
Dynatrace OpenPipeline works with other core Dynatrace platform technologies, including the Grail data lakehouse, Smartscape topology, and Davis hypermodal AI, to address these challenges by delivering the following benefits:
- Petabyte scale data analytics: Leverages patent-pending stream processing algorithms to achieve significantly increased data throughputs at petabyte scale.
- Unified data ingest: Enables teams to ingest and route observability, security, and business events data–including dedicated Quality of Service (QoS) for business events–from any source and in any format, such as Dynatrace OneAgent, Dynatrace APIs, and OpenTelemetry, with customisable retention times for individual use cases.
- Real-time data analytics on ingest: Allows teams to convert unstructured data into structured and usable formats at the point of ingest—for example, transforming raw data into time series or metrics data and creating business events from log lines.
- Full data context: Enriches and retains the context of heterogeneous data points—including metrics, traces, logs, user behavior, business events, vulnerabilities, threats, lifecycle events, and many others—reflecting the diverse parts of the cloud ecosystem where they originated.
- Controls for data privacy and security: Gives users control over which data they analyse, store, or exclude from analytics and includes fully customisable security and privacy controls, such as automatic and role based PII masking, to help meet customers’ specific needs and regulatory requirements.
- Cost-effective data management: Helps teams avoid ingesting duplicate data and reduces storage needs by transforming data into usable formats—for example, from XML to JSON—and enabling teams to remove unnecessary fields without losing any insights, context, or analytics flexibility.
“OpenPipeline is a powerful addition to the Dynatrace platform,” said Bernd Greifeneder, CTO at Dynatrace. “It enriches, converges, and contextualises heterogeneous observability, security, and business data, providing unified analytics for these data and the services they represent. As with the Grail data lakehouse, we architected OpenPipeline for petabyte-scale analytics. It works with Dynatrace’s Davis hypermodal AI to extract meaningful insights from data, fueling robust analytics and trustworthy automation. Based on our internal testing, we believe OpenPipeline powered by Davis AI will allow our customers to evaluate data streams five to ten times faster than legacy technologies. We also believe that converging and contextualising data within Dynatrace makes regulatory compliance and audits easier while empowering more teams within organisations to gain immediate visibility into the performance and security of their digital services.”
Dynatrace OpenPipeline is expected to be generally available for all Dynatrace SaaS customers within 90 days of this announcement, starting with support for logs, metrics, and business events. Support for additional data types will follow. Visit the Dynatrace OpenPipeline blog for more information.
