1. Introduction

1.1. What is WebMapReduce?

WebMapReduce is a simple web-based user interface for creating and submitting Hadoop Map-Reduce jobs in practically any language. It is ideally suited for use in the introductory computer science classroom, requiring very little programming experience to write massively parallel programs.

Some of its features include:

• Simplified Map-Reduce model: WebMapReduce offers the features of Map-Reduce that are crucial to the core concept, without details that add to the learning curve.
• Extensible language support: Mappers and reducers can be written in practically any language. All that is needed to support a new language is a simple wrapper library, possibly with an API for users to easily perform common tasks such as string processing.
• Adaptable APIs: In addition, preincluded languages can be customized to alter their APIs—for example, in order to support libraries already introduced in a class, or to suit a particular teaching style or programming paradigm. Purely functional, imperative, or object-oriented strategies are all fair game.
• Customizable authentication: It is possible to authenticate users against an LDAP server or through PAM, which itself provides many options, including traditional Unix authentication, database-based authentication, and many others.

1.2. Architecture

WebMapReduce runs on top of an installation of Hadoop, Apache’s implementation of MapReduce. It manages many of the details of using Hadoop, such as compiling code, configuring and submitting jobs, and reading output, mitigating the need to perform these tasks through the command line. It also uses a modified version of Hadoop Streaming, combined with special wrappers, to allow jobs to be written in any language.

WebMapReduce is split into two components, a frontend and a backend:

	Frontend	Backend
Technology	Django (Python) web app	Java web service
Responsibilities	Interacting with the user: Authentication Submission & monitoring workflow Reporting results/errors	Interacting with Hadoop: Preparing code (e.g., compiling) Submitting jobs Reading job output
Location	Any WSGI-capable webserver that can communicate with the backend	Head node of Hadoop cluster
Communication	Apache Thrift-based RPC protocol

One of the reasons for the separate components is flexibility: the frontend can run, for example, on your school’s public webserver, while the backend can be run on a cluster that is not widely accessible.

Of course, the two components can also be installed on the same machine. As will be explained later, this is sometimes desirable for simplicity and security. This installation guide will cover both cases.

1.3. About

WebMapReduce was developed at St. Olaf College by Patrick Garrity ‘12 and Tim Yates ‘12 starting in 2009. Stephen N. Lee ‘14 contributed various bugfixes to version 2.0, while the C# plugin was contributed by Boyang Wei ‘13. It is free software released under the Apache License, version 2.0.