by Przemek Tomczak
The onslaught of data continues. Predictions vary but some estimate the world will be generating 175 zettabytes annually by 2025, of which 90% would be from IoT alone. Drawing context and insights quickly is ever more important, and ever more challenging.
What is driving this demand is that organizations are deploying high-resolution sensors to improve the efficiency and effectiveness of their operations and products through analytics and automation.
Organizations are challenged with supporting analytics on higher data volumes cost-effectively. To address these challenges organizations had two options:
- Use more and larger infrastructure – more servers with a lot more DRAM to cache more data
- Use expensive high-performance storage and network infrastructure
Now we have a new option for consideration by performance-demanding customers – Intel® Optane™ persistent memory (PMem). We partnered with Intel to test their persistent memory with our Streaming Analytics platform running mission-critical industrial workloads.
Key takeaways
Use of Intel Optane persistent memory as a block-storage device with KX Streaming Analytics delivers 4× to 12× improved analytics performance over high performance NVMe storage – similar performance to DRAM for query workloads. For key data-processing workloads, we found DRAM requirements were significantly reduced. This lets organizations support more demanding analytic workloads on more data with less infrastructure.
Summary results
Analytics
- Performed within 10% of DRAM for queries involving table joins
- Performed 4× to 12× faster than 24 NVMe storage devices in RAID configuration
- DRAM performed 3× to 10× faster when performing single- threaded calculations and aggregations on data
Data processing and I/O operations
Processed 1.6× more data per second than NVMe-only storage where data was read from PMem and written to NVMe storage
- 2× to 10× faster reading data from files in parallel
- Seed of reading data similar to page cache (DRAM)
- Single-threaded file-write performance within 10% in both configurations
- Multithreaded file-write performance 42% slower
Infrastructure resources
- Required 37% less RAM to complete key I/O-intensive data processing
- Required no page cache for querying or retrieving data stored in PMem
Business benefits
- Collect and process more data with higher velocity sensors and assets
- Accelerate analytics and queries on recent data by 4× to 12×
- Reduce cost of infrastructure running with less servers and DRAM to support data processing and analytic workloads
- Align infrastructure more closely to the value of data by establishing a storage tier between DRAM and NVMe or SSD backed performance block storage
Organizations should consider Intel Optane persistent memory where there is a need to accelerate analytic performance beyond what is available with NVMe or SSD storage.
Please click on this link to code.kx.com for more details
About Streaming Analytics
KX Streaming Analytics has been at the forefront of helping organizations ingest, process, analyze, and deliver actionable insights on massive amounts of real-time and historical data. KX Streaming Analytics is an integrated platform that runs in the cloud and/or on-premises to enable organizations to reduce costs and improve productivity through anomaly detection, analytics, predictions, corrective action and automation based on both real-time and historical data. It provides full life-cycle data ingestion, processing, analytics, and data management enabling mission-critical operations, 24×7, with no downtime and no data loss – all of your data, in any environment, instantly.
About Intel Optane persistent memory
Intel Optane persistent memory (PMem) redefines traditional architectures, offering a large and persistent memory tier at an affordable cost. With breakthrough performance levels in memory-intensive workloads, virtual machine density, and fast storage capacity, Intel Optane persistent memory accelerates IT transformation to support the demands of the data era.
PMem is an affordable alternative to expensive DRAM. It can provide huge capacity and accommodate demanding workloads and emerging tools such as in-memory databases.
PMem is propelling major infrastructure consolidation. The increase in memory size from Intel® Optane™ media is an opportunity to consolidate workloads. Concentrating them on fewer nodes reduces deployments and improves use of processors previously underutilized because of memory constraints.
PMem is the world’s smallest and fastest storage device sitting on a memory bus. Architects and developers are using it to innovate with restart and replication, and to break system bottlenecks.
