SAP signaled a deeper embrace of big data platforms on Wednesday at SAPPHIRE with the announcement that it will create a deeper integration with Hadoop using Apache Spark as a low-latency, in-memory connector and data-filtering mechanism.
The announcement, made by SAP Executive Board Member Berndt Leukert, signals that SAP is counting on Hadoop as a high-scale storage container in Internet-of-Things (IoT) deployments and other scenarios where data-collection and data-processing needs are geographically distributed. Spark, the open-source, in-memory streaming and analysis platform, will be used as an intelligent processing engine to provide accelerated access to data in Hadoop, said Leukert.
SAP’s Berndt Leukert discusses new uses of Hadoop and Spark at Sapphire.
In an oil & gas and utility IoT deployments, for example, diverse sets of sensors in far-flung locations stream out readings at high scale. Hadoop provides high-scale, low-cost storage. Spark would provide low-latency connectivity to that data. But sensor data is diverse, lacking standardized formats, so Spark would also be used to filter, transform and harmonize the data, sending only the crucial signals on to a core IoT app running on the Hana Cloud Platform.
SAP has long had Hana data connectors for Hadoop, and it also has a Spark connector certified by Databricks, the commercial development-and-support company behind Spark. The architecture announced Wednesday signals a deeper level of integration and a deeper commitment to work with these platforms. Irfan Khan, senior vice president and general manager of SAP Big Data, confirmed that in a briefing with Constellation Research, and he noted that the edge-to-core processing approach could also be applied in non-IoT scenarios such as retail price optimization.
SAP has yet to flesh out the exact details of its new architecture, but David Parker, global VP, SAP Platform Solutions, told Constellation that SAP stream processing and mobile database technologies will also figure in the architecture, with lightweight choices at the edge and SAP Hana at the core.
MyPOV: There’s no shortage of big vendors declaring their intention to support IoT deployments these days, but SAP’s announcement shows that it has thought through the workloads and bottlenecks likely to be encountered in real-world IoT deployments. (Indeed, it’s pretty clear that co-innovation customers such as Siemens have played a significant role in evolving SAP’s thinking.) And what’s good for IoT will surely bring advances in other high-scale, real-time applications. It’s a welcome sign of maturation that points toward interesting and innovative deployments ahead.
