From Predict Conference 2015 to Data Mining and Innovation at Twitter
There is no doubt that before I went to represent Sonra at Predict Conference 2015, which was organised by Creme Global. I was a little dubious about the efficacy of the conference along with its outcomes for both Sonra and HUG Ireland. Would it allow us to effectively deliver the message for our business and also for the community group to like minded individuals who interact and/or are passionate about Big Data and the Hadoop framework??
My doubts were set aside on the first morning when I saw the congregation of committed data professionals in one place discussing big data issues, collaborating and doing business under one roof. With fears allayed, we went to work bringing the message of Sonra and HUG Ireland to the event, which was well received. The great speaking quality at the event covering many big data areas including food, nutrition, healthcare plus more. This, along with the quality of the exhibitors presenting their business messages made for a successful event in my view. Also of note at Predict Conference 2015 was the international meeting of minds on what constitutes ethical practice in big data and analytics. The concept of a charter of ethics is a critical juncture in the evolution of our industry given that the ability for harm is just as powerful as the ability to do good with big data processing and analytics. There is no doubt in my mind that big data has a big future given the talent that is in the industry, many of which were at Predict Conference 2015.
In our weekly review, we came across a fascinating paper in data mining by Adrian Colyer. It is an interesting overview of data mining and some of the basics of modern text analytics performed against reviews looking for data driven insights into their content from the analysis. Adrian covers natural language processing and part of speech analytics (POS). The concept of correlation applied to words in a sentence, gleaning insight from their composition (noun/verb/adjective) and correlation to one another is truly amazing especially in the article’s “customer review” examples, where the analytics is covering paragraphs. Who ever thought that word tagging, extraction/pruning, association, frequency count, and application to the model proposed could produce such insights built on prior experience, which is poured into the modelling. Use of product features, opinions and sentiment all shine new insights into data sets explored through the model. A fascinating dive into data driven insights and text analytics.
Another fascinating article came from Twitter’s publication on their high performance log replication layer called DistributedLog. Twitter’s Leigh Stewart @l4stewar gave a detailed most interesting analysis of Twitter’s distributed architecture using “Bookkeeper”, “Distributed Log” and “Manhattan” Coordinator to Client features. The architecture bears a striking resemblance to HDFS with a layered approach to data replication, data locality and indexing replicated data record locations on tables. Distributed log service acts (if taken in concert with BookKeeper) in a similar fashion to ZooKeeper on HDFS but has its own features that are Twitter centric as one might expect from the Social Media giant! Some features of distributed log service was log replication amongst nodes that are data locality compliant. The Manhattan controller controls the distribution amongst the nodes and keeps the logs up-to date. The quorum replication feature is a neat feature of the system giving log replication across nodes a faster write capability maintaining low latency architecture and allowing higher levels of abstraction to use the indexing capability of the logs. Lower latency is achieved by distributing rather than replicating logs across nodes in a tiered manner.
The article is action packed with features that define Twitter’s distributed architecture, which eventually consistent, partitioned but highly available with low latency. The architecture is a nice flavour of modern distribution by Twitter that ticks nearly all the boxes from the article’s content. What is unclear in the article is how Manhattan controller behaves should it lose contact with distributed log and bookkeeper. One would assume, replication and destruction of the controller would happen so a new connection could be re-established by the new controller instance, but the article does not shine light on this. Needless to say, it’s reasonable to assume the clever architecture at Twitter would not overlook such a basic point of failure threat in its design. Kudos to Twitter for a well designed and fascinating layered architecture for its distributed computing needs.
So, with the weekend upon us once more, its great to think of the yesterdays achievements and tomorrow’s innovations. I for one am looking forward to another week of great things to come!
We are a Big Data company based in Ireland. We are experts in data lake implementations, clickstream analytics, real time analytics, and data warehousing on Hadoop. We can help with your Big Data implementation. Get in touch.
We also run the Hadoop User Group Ireland. If you are interested to attend or present register on the Meetup website.