These days Hortonworks with their IPO and Cloudera sitting on $1bn of cash grab all the headlines. However,the real visionary in the field is someone else. Someone blasting the previous world record in TeraSort . A Hadoop distribution on both Amazon Web Services and the Google Compute Engine. A company that Google is invested in. While their competitors have been in skirmishes with each other, MapR has been quietly working away and innovating.
MapR-FS: Features and benefits compared to HDFS
The key innovation underpinning many of the other cool features of the MapR distribution and the subject of today's blog post is the MapR-FS (the proprietary file system of the MapR distro). It allows for random writes! Yes, you have read correctly. You can update a file sitting on MapR-FS. While HDFS is a rather limited file system that only lets you append data to a file, MapR-FS can write to a file at any offset.
This innovation allows for all sorts of cool things.
- File system metadata is distributed (think of it in terms of many mini name nodes). No central name node is needed. This eliminates name node bottlenecks.
- MapR-FS is written in C. No JVM garbage collection choking.
- NFS mount. You can mount the MapR-FS locally and read directly from it or write directly to it.
- MapR-FS implements POSIX. There is no need to learn any new commands. Your Linux administrator can apply existing knowledge to navigate the file system. You can view the content on MapR-FS using standard Unix commands, e.g. to view the contents of a file on MapR-FS you can just use tail <file_name>.
- While MapR-FS is proprietary it is compatible with the Hadoop API. You don't have to rewrite your applications if you want to migrate to MapR. hadoop fs -ls /user on MapR-FS works the same as ls /user.
- You can directly load the data into the file system. No need to set down the data on the local file system first. Guess what? Using NFS mounts there is no distinction between MapR-FS and the local filesystem. MapR-FS in a way is the local filesystem. No additional tools such as Flume etc. are needed to ingest data.
- True and consistent snapshots. Run point in time queries against your snapshots.
One other manifestation of the power of MapR-FS is the fact that an RDBMS such as the Vertica MPP engine can run directly against files stored on MapR-FS. Unthinkable of for HDFS. Other offerings that claim to run their MPP RDBMS against Hadoop often just have a connector that hooks into HDFS and then copies the data into their own storage layer or creates a copy of the data on HDFS. Not exactly what I mean by running your data warehouse on top of Hadoop. Similarly, Lucidworks, a Solr based Enterprise Search solution lets you run their search indexes directly atop MapR-FS.
There are other unique features of MapR that are superior to the other distributions: MapR-DB, Volumes & Multi-Tenancy, Enterprise Security, Drill (the latest innovation) and many more.
In summary. The MapR-FS brings us one step closer to the vision of bringing the processing to the data rather than the other way around. Running the MapR distrubution gives you a true competitive advantage. If your only use case is batch and if you are just appending data you are probably fine with HDFS and the other distros. However, if you have a requirement to run mixed workloads and have different use cases you need something that is more flexible. While vanilla Hadoop ticks some of the checkboxes such as NFS mounting and Snapshots they had to be implemented as workarounds due to the described limitations in HDFS. With MapR-FS and the MapR Hadoop distribution you are ready to enter the next stage of Big Data processing.
The MapR Hadoop distribution really rocks. We are very excited about this product!
Download the Community edition.
Download the MapR sandbox.
Other useful links discussing MapR-FS:
Blog Post: Comparing MapR-FS to HDFS.
Blog Post: Get Real with Hadoop: Read-Write File System.
MapR-FS White Paper.
Video: Comparison of MapR-FS and HDFS
Video: Comparing MapR FS and HDFS NFS and Snapshots
Video: MC Srivas Hadoop Summit 2011 Design, Scale and Performance of MapR's Distribution for Apache Hadoop
Contact us for a demo of MapR.