• Login
    Your ProfileLogout
  • join
  • about reply
Storm Reply Logo
Menu
    Choose language:
    • about Reply
    • join
    Storm Reply Logo
    Focus On

    IoT Data Stream on AWS Cloud ingestion and streaming processing

    24 August 2020

    FOCUS ON: Events,

    IoT devices are growing therefore more and more appliances starting from cars and machineries up to wearable such as watches are now smart and connected. This huge number of devices generates a huge number of messages which needs to be ingested, stored and processed. Classical ingestion mechanisms based on traditional servers are not suitable to sustain such a high volume and highly variable workload.

    By leveraging on AWS services Reply has been able to implement a data ingestion platform fully based on AWS services, scalable and highly available. Reply solution is composed of modules:

    • Ingestion Module: set of inbound channels for data
    • Processing module: complex event processor
    • Storage and forward module: outbound module forwarding data according to the matched complex event rules

    DATA INGESTION:

    • Messages
      • AWS IoT Basic Ingest
      • AWS IoT Core
      • AWS Kinesis Data Stream
      • AWS Kinesis Firehose
    • Binary Files
      • AWS S3
    • Video
      • AWS Kinesis Video Stream
    • Database
      • AWS Glue
    • SFTP
      • AWS Transfer SFTP
    • Custom Data Source
      • AWS Glue

    DATA PROCESSING:

    • Messages
      • AWS Kinesis Data Analytics (Apache Flink Application)
      • Custom Data Source
        • AWS Spark Streaming Job

    DATA STORAGE AND FORWARDING:

    • Storage
      • AWS S3
      • AWS Dynamo
      • AWS RDS
      • AWS RedShift
      • Forwarding (outbound)
        • AWS Kinesis Data Stream
        • Active MQ
        • HTTPS (Post)
        • AWS SQS
        • AWS SNS

    COMPLEX EVENT PROCESSOR AT SPEED

    Reply best practice in IoT Data Ingestion allows to ingest data via multiple channels and uses AWS Kinesis as collector for all data.

    Once ingested data needs to be processed in order to identify events which might be combination of multiple events eventually generated by multiple devices:
    example: when all lights are OFF and the thermostat is over 30°C turn the AC on or to conditionally forward messages to multiple consumer applications:
    example: Send all temperature message to the thermal application, when for 10 times in a row the temperature is above 30°C send the message also to the alarm monitor application

    Reply solution includes a Complex Event processor implemented on AWS Kinesis Data Analytics which is able to apply complex rules and handle new rules dynamically at scale being able to process incoming messages at very high frequency.

    Moreover, in IoT latency might be very important for some use cases as we want to know that a specific alarm is on immediately, to satisfy this requirement, Reply solution has a configurable maximum latency which tunes AWS resources to comply with the specific requirement being able to achieve and end2end sub second latency event for high message volumes.

    RELATED CONTENTS

    30.11.2020 - 18.12.2020 / Live Event

    Event

    AWS Re:Invent

    Reply will be attending the event "AWS RE:Invent", as a sponsor, with a virtual booth within the Partner Pavilion. Sign up for the event and come and meet Reply to discover the potential of the Cloud!

    Microsoft Workload on AWS

    About 70% of Enterprise Applications are Windows based and about 82% of these applications are still on premise (IDC). At Storm Reply we work with businesses to help them build, migrate, deploy, scale, and manage Microsoft Workload on AWS.

    02.07.2020 / Fully Digital

    Event

    Webinar: AWS + Flink: Enrich and Dispatch Streaming Data

    How to use AWS services at the edge to unlock data and send it to the cloud for further processing is explained in this IoT Webinar series offered by AWS. On 2 July 2020 Storm Reply spotlights an existing implementation that takes thousands of messages per second in the cloud, enriches them, applies user-defined rules, and sends the data to multiple destinations with a latency of less than one second.

     
     
     
     
    Reply ©​​ 2021 - Company Information
    • Abou​t Reply​​​​
    • Investors​​​
    • Newsroom
    • Follow Reply on
    ​
    • ​About Storm ​Reply​
    • Privacy Policy
    • Information (Client)
    • Information (Supplier)
    ​