Big Data News - Apache Ignite™, High Fashion PCA’s and Bloom Filters… tailoring your big data approach!

Uli Bethke Big Data, Data Science, Data Science, DFS, Distributed Computing

Significantly reduce your storage requirements using PCA and exponentially ignite your processing speed on Spark


Sally Khudairi, VP of Apache Software Foundation


An innovative use case for principal component analysis from Eigen Style was delivered in a blog post by Grace Avery. The objects in question are pictures of models modelling womenswear and how components of the pictures can be "modelled" across many pictures using principle component analysis. Across Eigen Style’s 807 pictures, there are pictures of the same frame size but different model poses along with different dress sizes. Grace goes on to point out how principal components of all pictures were established. In addition, she advises that using PCA the pictures were rebuilt to a degree of accuracy that makes for enormous memory savings (tens of thousands of pixels to just a few in the components) along with accurate representations of the pictures using PCA. Thinking about it, it makes perfect sense, each picture shares commonality components with each other and using PCA, the modelling of the similar pictures makes for memory and cost savings for Eigen Style using big data and in particular Principal Component Analysis.

Exciting times are never too far away from the open source community and Apache™ once again is at the epicentre of attention. Sally Khudairi, VP of Apache Software Foundation presented Apache™ Ignite™ in a fascinating article on the new top project at Apache™. The project from Apache incubator rose from the ranks of the 350 ongoing projects to become a Top Level Project (TLP) and it’s not hard to see why! Ignite™ is a distributed computing framework leveraging the power of other Apache products like Hadoop, Mesos, Spark and Yarn. It is an “in memory computing and transacting” framework for large datasets harnessing in-memory processing, in-memory file storage and uses RAM primarily for storage rather than processing. Sally outlines the performance improvements as far outpacing disc and flash making big data “fast data”.


Apache Ignite™ will truly gives your processing power a 100x performance increase running on Spark when used as an in-memory layer between Spark workers and the underlying frameworks like Yarn, Mesos, Cloud, etc. Ignite™ has the ability to allow shared state in an application (embedded state), that can survive sessions thus be deployable across other Spark applications (stand alone state), e.g. A Spark application such as Zeppelin can share its state with the Spark Thrift server.

Ignite™ has another cool feature being its Ignite File System (IGFS), which is an innovative iteration of what essentially is enhanced HDFS. The notable innovations are in its in-memory caching and how it splits data blocks to cache automatically maintaining data locality in its distributed in-memory cache. Very cool stuff indeed!

Another great feature on Ignite™ is the ability to store data off heap! It’s advanced indexing allows indexing for off heap storage thus reducing garbage collection overhead and increasing performance. As Spark does not support SQL indexing, Ignite™ comes to the rescue with SQL query support and as you can plug it directly into Spark. Your value proposition for Spark has thus increased with SQL query indexing at a much faster rate with the in-memory race horse called Ignite™! The key difference is the in-memory indexing of Ignite™ which negates the need for Spark to scan a whole data set for a single query result, which in big data is a huge performance boost! Its a powerful tool and exciting times at the Apache Software Foundation.

Sketching Data Structures by László Kozma is our final stop of the day where he explores the data science process of sketching large data sets with memory efficient sketching techniques. His insight are impressive into ‘bloom filters’ and ‘count min’ sketches. This big data enthusiast explores the hashing of tables for bloom filters and discusses their effectiveness along with the practical definition and use case for count min.

There is no doubt in my mind that the infusion of enthusiasm, focus and energy in our community will lead to the unveiling of more and more of the universe's secrets using old formula’s and new innovations that lead the way, not only in big data but the evolving world as we know it!

About Sonra

We are a Big Data company based in Ireland. We are experts in data lake implementations, clickstream analytics, real time analytics, and data warehousing on Hadoop. We can help with your Big Data implementation. Get in touch.

We also run the Hadoop User Group Ireland. If you are interested to attend or present register on the Meetup website.