Hoop, Hadoop HDFS over HTTP 0.1.0-SNAPSHOT - Client Setup

[ Go Back ]

Client Configuration

Add Hoop Client JARs to the CLASSPATH.

The Hoop Client JARs are available in the Hoop distro client-lib> directory.

They are also available in Cloudera's Maven repository:

  • Repo URL: https://repository.cloudera.com/content/repositories/releases
  • Group Id: com.cloudera.hoop
  • Artifact Id: hoop-client
  • Version: 0.1.0-SNAPSHOT
  • Type: jar
      ...
  <dependency>
      <groupId>com.cloudera.hoop</groupId>
      <artifactId>hoop-client</artifactId>
      <version>0.1.0-SNAPSHOT</version>
  </dependency>
  ...

Create a File System object that connects to the Hoop Server:

  ...
  Configuration conf = new Configuration();
  conf.set("fs.http.impl", HoopFileSystem.class.getName());
  FileSystem fs = FileSystem.get(new URI("http://myhoophost:14000"), conf);
  ...

The returned FileSystem object connects to the Hoop server.

All file system operations are supported.

Using Hadoop hadoop to access HDFS via Hoop

Set the HADOOP_CLASSPATH environment variable with all the JARs from Hoop's /client-lib directory. For example:

$ cd <HOOP_HOME>
$ export HADOOP_CLASSPATH="`pwd`/client-lib/alfredo-0.1.5.jar:`pwd`/client-lib/hoop-client-0.1.0-SNAPSHOT.jar:`pwd`/client-lib/json-simple-1.1.jar"

NOTE: The commons-codec JAR file is part of Hadoop's CLASSPATH, thus no need to include it.

Invoke hadoop fs setting the Namenode URI and registering Hoop's file system implementation. For example:

$ hadoop fs -Dfs.default.name=http://<HOOP_HOSTNAME>:14000 -Dfs.http.impl=com.cloudera.hoop.client.fs.HoopFileSystem -lsr
-rw-rw-rw-   1 babu staff             966 1969-12-31 16:00 /user/babu/foofoo.txt
drwxr-xr-x   - babu supergroup          0 2011-07-14 12:43 /user/babu/x

NOTE: For a more permanent setup, add the 2 properties to your local hdfs-site.xml.

[ Go Back ]