Technology and Big Data movers... A week of wonder in review...
This week saw a number of exciting ideas and announcements reach the Tech community that signals how it can positively impact the world. Taking a look at some of the highlights, I think we can get excited about the following:
Burak Yavuz , Software Engineer at Databricks blogged about an import tool for Spark Packages (http://spark-packages.org). Spark Packages combines libraries from GitHub, PyPI and Maven Central into one central location. If you are going overlay Databricks on top of Apache Spark, then this feature is a must have. Burak detailed an apparently smooth import process for your desired library onto Databricks, which looks “user friendly” and details dependencies along with the opportunity to exclude any such dependencies that may cause a clash in your operating environment. Burak’s detailing of the import process makes for interesting reading for anybody thinking of using open source libraries and Databricks on-top of Apache Spark for scalable projects. Using 3rd Party Libraries in Databricks: Spark Packages and Maven Libraries by Burak Yavuz makes for a good read!
On the same day, ExtremeTech covered the production launch of Xpoint™. Intel and Micron Technology announced a ground breaking product into production, which promises to impress. 3D Xpoint™ (pronounced “CrossPoint”) is a Memory feature that by Intel/Micron’s account is:
- Non-Volatile – You can power down but unlike standard DRAM, in-memory storage persists saving your state.
- Stackable – The higher density allows read write operations x8 times faster then DRAM and does not have a transistor requirement lessening power consumption. Compared to NAND Flash however, density levels and performance are more comparable.
- Highly Durable – Can last x1000 times longer then NAND flash memory and is noted like NAND flash memory to be scalable.
For Intel and Micron, it appears be a game changer in performance and durability yet there are competing technologies that occupy the space from companies such as Samsung. However, if the claims of Intel & Micron are proven in live use cases, then the power consumption and performance returns on Xpoint™ are definitely a space to watch out for in scalable computing. Intel, Micron reveal Xpoint, a new memory architecture that could outclass DDR4 and NAND by Joel Hruska on July 28 makes for stimulating reading also. I would also recommend the official release article by Intel, Intel and Micron Produce Breakthrough Memory Technology, which is also highly informative.
The site “O’Reilly Radar” saw Marie Beaugureau share her thoughts on Dell’s new open source project and new offering called Doradus. Doradus (OLAP) is a query and storage framework designed to work on top of Cassandra distributed file/database system. With no single point of failure, Doradus is supported by Cassandra, which is offering a very stable yet fast parallel processing rate and extreme compression of up-to 99% on a single node according to Marie. This extremely dense storage ability allows large amounts of data to be stored in a single node making it’s querying very fast compared to a distributed data query. The claim is that up-to 1 million objects per node, per second can be stored making data warehousing effectively instantaneous! Whilst impressive, the average object size is not clear in these claims and could lead to misunderstanding on how far Doradus will go in live environment. Also, in use cases with distribution of data across nodes, there will be by design a performance overhead depending on data size, distribution used (data locality) and fault tolerance configuration. Use case testing is advised to frame Doradus against scope requirements. On the same note, the single node performance claims against the backdrop of a distributed file system, which can be highly fault tolerated, sharded and replicated are indeed impressive.
Whilst producing yet another ‘flavour of SQL’, Doradus Query Language (DQL is a Lucene based query language) claims to impress on query speed with up-to 10 million objects per second searched on multi level aggregate queries computing multi level expressions in one efficient iteration over the data, which is organised by the framework into multi dimensional cubes for better access and compression performance. This open source project by Dell certainly has allot of features worth considering in any architecture that is data intensive and speed is critical in a successful delivery of the architecture to the business. If your use case is tested and fits, then follow up should be the order of the day on Doradus. Finding new ways to shrink disk space for storing partitionable data by Marie Beaugureau is a must read for those scouting new architecture opportunities for your business.
It’s nice to think we can have such awesome developments in technology in the space of a week or so! Maybe next week will see more 'solved problems' come to market in a manner that is not only evolutionary, it wets our appetite for times to come!
We are a Big Data company based in Ireland. We are experts in data lake implementations, clickstream analytics, real time analytics, and data warehousing on Hadoop. We can help with your Big Data implementation. Get in touch.