Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka

Dorian Beganovic Kafka, Snowflake, Spark

Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka Streaming architecture In this post we will build a system that ingests real time data from Twitter, packages it as JSON objects and sends it through a Kafka Producer to a Kafka Cluster. A Spark Streaming application will then consume those tweets in JSON format and stream them ...

Big Data News: Convergence with Mapr and Faster Stateful Streaming Processes with Spark

Uli Bethke Big Data, DFS, HUG Ireland, MapR, Spark

Mapr on Impedance Mismatch and how convergence is achieved for layered architecture along with Databricks on using the new Spark API “mapWithState” for faster Stateful Spark Streaming As our big data world comes to the end of another week, the team at Sonra have been once again impressed by the weeks highlights in big data. Mapr has shared its insights ...

Big Data News: Hadoop User Group (HUG) Ireland’s morning briefing and MapR’s new Product Launch called “Streams”

Uli Bethke Hadoop, HUG Ireland, MapR, News, Open Source Software, Technology

HUG Ireland’s great morning briefing with Alexey Grishchenko and MapR is now directly competing with Apache Kafka Last week was a busy one in operations for the Sonra team with all hands on deck, which is why we are a little late writing about a powerful HUG Ireland morning briefing with Alexey Grishchenko, which was sponsored by Sonra and very ...

Big Data News: Apache Kafka 0.9 Release and Ebay Open Sourcing of Pulsar Reporting

Uli Bethke Big Data, HUG Ireland, Open Source Software, Technology

Open Source movers this week with Kafka 0.9 release along with Ebay’s extension of Pulsar open source with Pulsar Reporting  The world does not stand still too long in Big Data as more open source projects come to fruition adding value and substance to our community’s open source movement.  Apache Kafka was one such movement with its 0.9 version release ...

Big Data News - MesosCon (Europe), Unix Philosophy of Distributed Data and Advanced Algorithms

Uli Bethke Big Data, Data Science, Unix

Apache Mesos (Europe) Conference Announcement, Martin Kleppmann on Distributed Unix Philosophy and Advanced Algorithms… today’s view into tomorrow’s world!!..  So as the week draws to a close, we are left with a feeling that the pace of understanding in our community and the progress it brings has not slowed one bit! Accordingly, we finish the week on a good note ...