Hoop - Hoop, Hadoop HDFS over HTTP 0.1.0-SNAPSHOT

Hoop, Hadoop HDFS over HTTP 0.1.0-SNAPSHOT - HTTP REST API

Security

Hoop enforces authentication using Alfredo. Out of the box Alfredo supports both pseudo authentication and Kerberos HTTP SPNEGO authentication.

Pseudo Authentication

With pseudo authentication the user name must be specified in the user.name=<USERNAME> query string parameter of a Hoop URL. For example:

$ curl "http://<HOOP_HOST>:14000?op=homedir&user.name=babu"

Kerberos HTTP SPNEGO Authentication

Kerberos HTTP SPENGO authentication requires a tool or library supporting Kerberos HTTP SPNEGO protocol.

IMPORTANT: If using curl, the curl version being used must support GSS (curl -V prints out 'GSS' if it supports it).

For example:

$ kinit
Please enter the password for tucu@LOCALHOST:
$ curl --negotiate -u foo "http://<HOOP_HOST>:14000?op=homedir"
Enter host password for user 'foo':

NOTE: the -u USER option is required by the --negotiate but it is not used. Use any value as USER and when asked for the password press [ENTER] as the password value is ignored.

Remembering Who I Am (Establishing an Authenticated Session)

As most authentication mechanisms, Alfredo authenticates users once and issues a short-lived authentication token to be presented in subsequent requests. This authentication token is a signed HTTP Cookie.

When using tools like curl, the authentication token must be stored on the first request doing authentication, and submitted in subsequent requests. To do this with curl the -b and -c options to save and send HTTP Cookies must be used.

For example, the first request doing authentication should save the received HTTP Cookies.

Using Pseudo Authentication:

$ curl -b ~/.hoopauth "http://<HOOP_HOST>:14000?op=homedir&user.name=babu"

Using Kerberos HTTP SPNEGO authentication:

$ curl --negotiate -u foo -b ~/.hoopauth "http://<HOOP_HOST>:14000?op=homedir"

Then, subsequent requests forward the previously received HTTP Cookie:

$ curl -c ~/.hoopauth "http://<HOOP_HOST>:14000?op=list"

File System Operations

The HTTP REST API is fully demonstrated below using the Unix curl command.

HOOP HTTP status codes follow HTTP guidelines.

HOOP payload responses are in JSON format.

For optional parameters, the default values are between '*', for example 'overwrite=*true*|false'.

All the Examples below assume the User is already already authenticated, refer to the Remembering Who I Am section.

Read File

HTTP Request:

$ curl -X GET -c ~/.hoopauth "http://<HOOP_HOST>:14000/<PATH>[?<OPTION>[&<OPTION>]*]"

OPTIONS:

offset=*-1*|<LONG> ('-1' means the whole file)

len=*-1*|<LONG> ('-1' means from the beginning of the file)

Sucessful HTTP Response:

HTTP/1.1 200 OK
Location: http://<HOOP_HOST>:14000/PATH
Content-Type: application/octet-stream
Transfer-Encoding: chunked

<FILE CONTENTS>

Write File

HTTP Request:

$ curl -X POST -c ~/.hoopauth "http://<HOOP_HOST>:14000/<PATH>?op=create[&<OPTION>]*" \
  --data-binary @data.txt  --header "content-type: application/octet-stream"

OPTIONS:

overwrite=*true*|false
replication=*-1*|<SHORT>
blocksize=*-1*|<LONG>
permission=*default*|<-rwxrwxrw>
The content-type header must be specified othewise curl will use application/x-www-form-urlencoded and the servlet-container will attempt parsing the inputstream as a form POST.

Sucessful HTTP Response:
```
HTTP/1.1 201 Created
Location: http://<HOOP_HOST>:14000/PATH
Content-Type: application/json
Content-Length: 0
```

Append File

HTTP Request:

$ curl -X PUT -c ~/.hoopauth "http://<HOOP_HOST>:14000/<PATH>?op=append" \
  --data-binary @data.txt --header "content-type: application/octet-stream"

The content-type header must be specified othewise curl will use application/x-www-form-urlencoded and the servlet-container will attempt parsing the inputstream as a form POST.

Sucessful HTTP Response:

HTTP/1.1 200 OK
Location: http://<HOOP_HOST>:14000/PATH
Content-Type: application/json
Content-Length: 0

Rename File/Directory

HTTP Request:

$ curl -X PUT -c ~/.hoopauth "http://<HOOP_HOST>:14000/<OLDPATH>?op=rename&to=<NEWPATH>"

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{"rename":true}

Delete File/Directory

HTTP Request:

$ curl -X DELETE -c ~/.hoopauth \
  "http://<HOOP_HOST>:14000/<PATH>?op=delete[&recursive=*true*|false]"

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{"delete":true}

List Directory

HTTP Request:

$curl -X GET -c ~/.hoopauth "http://<HOOP_HOST>:14000/<PATH>?op=list[&filter=<FILTER>]"

If specifying a filter use Hadoop's GlobFilter syntax.

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

[
  {
    "path" : "http:\/\/<HOOP_HOST>:14000\/<PATH>\/foo.txt"
    "isDir" : false,
    "len" : 966,
    "owner" : "babu",
    "group" : "supergroup",
    "permission" : "-rw-r--r--",
    "accessTime" : 1310671662423,
    "modificationTime" : 1310671662423,
    "blockSize" : 67108864,
    "replication" : 3
  }
]

Make Directory

HTTP Request:

$ curl -X POST -c ~/.hoopauth \
  "http://<HOOP_HOST>:14000/<PATH>?op=mkdirs[&permission=*default*|<-rwxrwxrw>]"

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{"mkdirs":true}

File Status

HTTP Request:

$ curl -X GET -c ~/.hoopauth "http://<HOOP_HOST>:14000/<PATH>?op=status"

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{
  "path" : "http:\/\/<HOOP_HOST>:14000\/<PATH>"
  "isDir" : false,
  "len" : 966,
  "owner" : "babu",
  "group" : "supergroup",
  "permission" : "-rw-r--r--",
  "accessTime" : 1310671662423,
  "modificationTime" : 1310671662423,
  "blockSize" : 67108864,
  "replication" : 3
}

Home Directory

HTTP Request:

$ curl -X GET -c ~/.hoopauth "http://<HOOP_HOST>:14000/?op=homedir"

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{"homeDir":"http:\/\/<HOOP_HOST>:14000\/user\/babu"}

Set Owner

HTTP Request:

$ curl -X PUT -c ~/.hoopauth \
  "http://<HOOP_HOST>:14000/<OLDPATH>?op=setowner&owner=<OWNER>&group=<GROUP>"

If 'owner' or 'group' are not specified, the current value is preserved.

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 0

Set Permission

HTTP Request:

$ curl -X PUT -c ~/.hoopauth \
  "http://<HOOP_HOST>:14000/<PATH>?op=setpermission&permission=*default*|<-rwxrwxrw>"

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 0

Set Times

HTTP Request:

$ curl -X PUT -c ~/.hoopauth \
  "http://<HOOP_HOST>:14000/<PATH>?op=settimes&mtime=<LONG>&atime=<LONG>"

If 'mtime' or 'atime' are not specified, the current value is preserved.

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 0

Set Replication

HTTP Request:

$ curl -X PUT -c ~/.hoopauth \
  "http://<HOOP_HOST>:14000/<PATH>?op=setreplication&replication=<SHORT>"

If 'replication' is not set the default value is used.

Sucessful HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{"setReplication":true}

File System Command Failures

A failed Hoop file system command produces a detailed response in JSON besides the HTTP error code, for example:

HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close

{
  "statusCode":500,
  "reason":"Internal Server Error",
  "message":"H03: FileSystemExecutor error, ...: Non-super user cannot change owner.",
  "exception":"HadoopException"
}

[ Go Back ]