WebMapReduce is a simple web-based user interface for creating and submitting Hadoop Map-Reduce jobs in practically any language. It is ideally suited for use in the introductory computer science classroom, requiring very little programming experience to write massively parallel programs.
Some of its features include:
WebMapReduce runs on top of an installation of Hadoop, Apache’s implementation of MapReduce. It manages many of the details of using Hadoop, such as compiling code, configuring and submitting jobs, and reading output, mitigating the need to perform these tasks through the command line. It also uses a modified version of Hadoop Streaming, combined with special wrappers, to allow jobs to be written in any language.
WebMapReduce is split into two components, a frontend and a backend:
|Technology||Django (Python) web app||Java web service|
Interacting with the user:
Interacting with Hadoop:
|Location||Any WSGI-capable webserver that can communicate with the backend||Head node of Hadoop cluster|
|Communication||Apache Thrift-based RPC protocol|
One of the reasons for the separate components is flexibility: the frontend can run, for example, on your school’s public webserver, while the backend can be run on a cluster that is not widely accessible.
Of course, the two components can also be installed on the same machine. As will be explained later, this is sometimes desirable for simplicity and security. This installation guide will cover both cases.
WebMapReduce was developed at St. Olaf College by Patrick Garrity ‘12 and Tim Yates ‘12 starting in 2009. Stephen N. Lee ‘14 contributed various bugfixes to version 2.0, while the C# plugin was contributed by Boyang Wei ‘13. It is free software released under the Apache License, version 2.0.