Will Apache Myriad’s release spark a Big Data Evolution?. . . and Storm, Kafka, Akka, and Prometheus took centre stage at HUG Ireland’s final 2015 event
Once again, the team here at Sonra have being impressed with our big data industry where sharing of knowledge, collaboration and openness has seen the Apache Myriad (community) launch not only a new product for our industry, but arguably launch a new direction for our industry in terms of how clusters may work as one rather than as a layered many.
Apache Myriad is a wonderful example of how open collaboration has taken a problem and turned it into a solution, which solves the silo data problem folks will have had with Mesos and Yarn not getting along. Myriad works with Mesos in a very slick manner whilst sharing resources with Yarn thus becoming a true global resource manager to manage your data center including Hadoop clusters. With Myriad, Mesos and Yarn are now talking to each other like good friends with immediate benefits for the data center. It can now run Hadoop elastically across data center resources, have no application silos and run multiple compute frameworks in tandem. It’s noteworthy features also allow sharing of Data Center services and if needed, reallocating resources within virtual clusters (after all… sharing is caring!). Finally, it can run multiple Yarn sub-clusters (aka Hadoop Virtual Clusters) in the same DFS and Myriad is easy to integrate with no need to rewrite Yarn apps. With the community version release, it’s no surprise that the directional indicator for better data management is being heard by many including MapR whose engineers have been instrumental in Apache Myriad’s design. Mapr are one of the first to announce their intention to integrate this technology into their commercial technology stack. Exciting times with more to follow in 2016…
December 14th saw Hadoop User Group (HUG) Ireland run its final event of the year at Accenture in Grand Canal Square, Silicon Docks, Dublin 2. A wonderful food spread was put on for attendees by Accenture, after which we heard three fascinating speeches from Wu (Post Doctoral Researcher with CeADAR) on his use of Storm/Spark Streaming and Trident in the real-time streaming of SMS messages by a telecoms carrier with view to identify spam. Brian stole the show with a fascinating overview of Prometheus and what it can do for Hadoop clusters independent of Zookeeper. An important design takeaway Brian gave on Prometheus was its platform independence from Hadoop Clusters, which allows it to independently monitor cluster health, network health and application health continuously, thus not just altering to failures. Floran rounded up the talks with a great overview of his project with Real Impact Analytics on the creation of a Data Simulator for a telco company using Spark, Kafka and Akka. His use of big data was primarily to drive network optimization but also aid marketing campaigns through the data platform they were building. Fascinating stuff indeed! HUG Ireland’s first 2016 event is on Jan 11th at Synchronoss on Elastic Big Data Processing with Myriad and Mesos. ETL Use Cases and Hadoop. Hope to see you all there for a great evening!
Needless to say, it’s been a fascinating year for our community with innovation, sharing and talks well received by all as we head to the Christmas break. As this day next week is Christmas, we don’t want to disturb you from your festive treats and thus will return with big data treats on the Big Data News by Sonra in January! Merry Christmas all and have a wonderful new year! See you in 2016...
We are a Big Data company based in Ireland. We are experts in data lake implementations, clickstream analytics, real time analytics, and data warehousing on Hadoop and Spark. We also run the Hadoop User Group (HUG) Ireland. We can help with your Big Data implementation. You can get in touch today, we would love to hear from you!