03-11-2017

Data is becoming ever increasingly abundant. Last year, the International Data Corporation (IDC) published their Digital Universe estimate for the amount of data created in the world. For 2025, they forecasts a whopping 180 Zettabytes of data to be created annually, jumping from just 44 Zettabytes to be created in 2020 . More strikingly, the IoT data that is analyzed and used to change business processes in 2025 will be as big as all the data created in 2020.


The ever increasing data flow will challenge us to ingest and analyze large volumes of high velocity data. SAP Smart Data Streaming is SAP’s enterprise ready solution for real-time data processing, that is up to this challenge.

Introducing SAP Smart Data Streaming

Smart Data Streaming (SDS) is a real-time processing engine designed to run inside a SAP HANA platform. The engine can receive or retrieve events from multiple sources, combine them, analyze them, and send them to a front-end application, database, or other processing system. It is designed to handle events at a large scale at high performance, which makes it interesting for, e.g., IoT streams, click streams, social media streams or business transactions streams. Simply said, SAP SDS is the SAP onsite solution when immediate action needs to be taken, or real-time insights need to be created.

Technically speaking, any multi-node HANA platform can be extended with SDS functionality. With a simple configuration, HANA nodes can be assigned to run the SDS service, creating a streaming cluster inside the HANA platform. The streaming node(s) will not be part of the HANA core nodes, which run the database service. Consequently, the streaming hosts do not have to run on HANA-certified servers, which gives you some flexibility in choosing the best fit for performance versus financial cost of the machine. It is possible to assign multiple nodes to SDS, for scalability. Load-balancing comes out of the box. For even easier access to Smart Data Streaming technology, SAP also offers the product as part of the SAP Cloud Platform.

Stream processing versus traditional ETL

It could be said that a streaming processor is in its basic form no different than a regular ETL tool, as it moves data from A to B. After all, SAP Data Services (BODS) can also perform realtime jobs. However, the main difference is that both tools have a significantly different approach as how to manage their (system) resources. A Streaming Processor like SDS is optimized for immediate processing of single events, whereas the former is optimized for moving large datasets at a time. For the fastest processing of real-time data, SAP SDS will outperform traditional ETL tools.

Organisational considerations

Taking the real-time aspect into account, there will be some differences in how an organisation should manage an SDS instance compared to an ETL instance. A real-time processing system needs to be monitored continuously, and SLA’s need to be set clearly.

With batch-ETL processes, organizations would usually create a daily report of all successful (and sometimes unsuccessful) data loads, after which the support organisation would (re)start failed loads a few hours later. With real-time processing, waiting a few hours before fixing an isolated problem could potentially mean that thousands of events are lost, or that the ‘event processing chain’ gets clogged, thereby harming performance of the complete system.

Next to differences in support processes, the development of SDS requires a different kind of developer. He or she should be much more proficient in coding than traditionally needed for SAP BI developers. CCL script offers a different paradigm than SQL and BODS script. Also, for creating custom (not SAP-provided) connectors to source or target systems, in-depth Java programming is required. The developer role is therefore much broader than before.

Conclusion

Concluding, SAP Smart Data Streaming is a beautiful product for real-time event processing. As part of the HANA platform, it is easy to set up. It provides excellent performance and scalability.

Do you want to know more about SAP SDS or event processing? Have a look at the documentation at SAP or get in touch. You can even try SDS for free, using SAP HANA Express Edition.

Please note SAP seems to be relabelling SAP Smart Data Streaming to SAP HANA Streaming Analytics.