Overview
Create and manage replication policies using Cloudera Replication Manager.
Version information
Version : 0.9.112
License information
License : Apache 2.0
Terms of service : https://www.cloudera.com/legal/commercial-terms-and-conditions.html
URI scheme
Schemes : HTTPS
Consumes
-
application/json
Produces
-
application/json
Paths
Resume all replication tasks defined by the policy.
POST /api/v1/replicationmanager/activatePolicy
Description
Resumes all replication tasks defined by the policy if the policy is currently suspended.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Collect diagnostic bundle for a policy.
POST /api/v1/replicationmanager/collectDiagnosticBundle
Description
Triggers the collection of diagnostic bundle for a policy.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Create an ABFS Cloud Credential.
POST /api/v1/replicationmanager/createAbfsCredential
Description
Creates a new ABFS cloud credential.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Create an AWS Cloud Credential.
POST /api/v1/replicationmanager/createAwsCredential
Description
Creates a new AWS cloud credential.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Create a Replication Policy.
POST /api/v1/replicationmanager/createPolicy
Description
Creates a new replication policy with the given name on a specific cluster.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Delete Cloud Credential.
POST /api/v1/replicationmanager/deleteCredential
Description
Permanently deletes a specific cloud credential.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Delete Replication Policy.
POST /api/v1/replicationmanager/deletePolicy
Description
Permanently deletes a specific replication policy.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Download diagnostic bundle for a policy.
POST /api/v1/replicationmanager/downloadDiagnosticBundle
Description
Downloads the previously collected diagnostic bundle if available. Before calling, the get-command-status command shall be polled until the command’s status shows 'DOWNLOADABLE_WITH_CLI'. If the status is 'DOWNLOADABLE_WITH_URL', the result can be downloaded with the given result data URL only.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Get the status of the given CM command.
POST /api/v1/replicationmanager/getCommandStatus
Description
Return the current status of the given CM command.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Get Cloud Credentials from a Cluster.
POST /api/v1/replicationmanager/getCredentials
Description
Returns cloud credentials on a specific cluster.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Lists Cloud Credentials.
POST /api/v1/replicationmanager/listAllCredentials
Description
Provides a detailed list of cloud credentials across all available clusters.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Lists Cluster Service Statuses.
POST /api/v1/replicationmanager/listClusterServiceStatuses
Description
Provides a list of cluster service (e.g. HDFS, Hive, YARN, etc.) statuses.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Lists all clusters.
POST /api/v1/replicationmanager/listClusters
Description
Provides a detailed list of all available clusters.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Get All Replication Policies.
POST /api/v1/replicationmanager/listPolicies
Description
Provides a list of all replication policies across all available clusters.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Stop all replication tasks defined by the policy.
POST /api/v1/replicationmanager/suspendPolicy
Description
Stops all replication tasks defined by the policy. Replication will resume after activating the policy.
Parameters
Type | Name | Schema |
---|---|---|
Body |
input |
Responses
HTTP Code | Description | Schema |
---|---|---|
200 |
Describes a successful response to a valid request. |
|
default |
The default response on an error. |
Definitions
AbfsCloudCredentialType
ABFS cloud credential type.
Type : enum (ACCESSKEY, CLIENTKEY)
ActivatePolicyRequest
Activate a replication policy.
Name | Description | Schema |
---|---|---|
clusterCrn |
CRN of the cluster where the policy is. |
string |
policyName |
Policy name to resume. |
string |
ActivatePolicyResponse
Response object for resumePolicy.
Type : object
Alert
Cloudera Manager alerting behavior.
Name | Description | Schema |
---|---|---|
onAbort |
Alert on abort. |
boolean |
onFailure |
Alert on failure. |
boolean |
onStart |
Alert on start. |
boolean |
onSuccess |
ALert on success. |
boolean |
ApiError
API error.
Name | Description | Schema |
---|---|---|
clusterUrl |
Cluster URL. |
string |
code |
Error code. |
integer (int32) |
message |
Error message. |
string |
requestId |
Request ID. |
string |
status |
Error status. |
string |
AwsCloudCredentialType
AWS cloud credential type.
Type : enum (IAM, ACCESSKEY)
Cluster
Describes a Datalake, Datahub or Classic cluster.
Name | Description | Schema |
---|---|---|
cloudAccountProvider |
Cloud account provider. |
enum (AWS, ADLS, WASB, ABFS, GCS) |
clusterManagerType |
Cluster Manager type. |
enum (CLOUDERA_MANAGER, AMBARI) |
clusterManagerUrl |
Cluster Manager URL. |
string |
clusterType |
Cluster type. |
enum (CLASSIC, SDX, DISTROX) |
clusterUrl |
Cluster URL. |
string |
cmName |
Cloudera Manager name. |
string |
codProperties |
Additional properties for operational databases. |
|
crn |
Cluster CRN. |
string |
dataCenter |
Cluster data center. |
string |
description |
Cluster description. |
string |
environmentCrn |
CRN of the cluster’s environment. |
string |
knoxUrl |
Knox URL if the cluster was registered with one, null otherwise. |
string |
location |
Cluster location. |
|
name |
Cluster name. |
string |
replicationEngineType |
Replication engine type. |
enum (CM, DLM_ENGINE) |
stackCrn |
Stack CRN. |
string |
stackMajorVersion |
Stack major version. |
string |
stackVersion |
Stack version. |
string |
totalHosts |
Total hosts on the cluster. |
integer (int32) |
uuid |
Cluster UUID from Cloudera Manager if the cluster has CM backend. |
string |
ClusterServiceStatuses
Describes the status of services running on a cluster.
Name | Description | Schema |
---|---|---|
clusterCrn |
Cluster CRN. |
string |
serviceStatuses |
List of service statuses. |
< ServiceInfo > array |
CmPolicySubmitUser
CM Policy submit user.
Name | Description | Schema |
---|---|---|
sourceUser |
Source user. |
string |
userName |
Username. |
string |
CodProperties
Additional properties for operational databases.
Name | Description | Schema |
---|---|---|
operationalDatabaseName |
Name of the corresponding operational database.. |
string |
CollectDiagnosticBundleRequest
Triggers a diagnostic bundle collect command for the given cluster and policyName.
Name | Description | Schema |
---|---|---|
clusterCrn |
The CRN of the target cluster. |
string |
policyName |
Name of the policy. |
string |
CollectDiagnosticBundleResponse
Response object for collect-diagnostic-bundle.
Name | Description | Schema |
---|---|---|
active |
Whether the diagnostic bundle command is active. |
boolean |
commandId |
ID of the diagnostic bundle collection command issued against the CM. Polling get-command-status with this command id returns the current state of the command. |
integer (int64) |
name |
Name of the policy. |
string |
resultMessage |
Diagnostic bundle collection command result message. |
string |
startTime |
Start time of the diagnostic bundle collection command. |
string |
CreateAbfsCredentialRequest
Request object for creating ABFS cloud credentials. Currently supported cloud credentials: ABFS Access Key for Ambari clusters, ABFS Client Secret for CM clusters.
Name | Description | Schema |
---|---|---|
accessKey |
ABFS access key. |
string |
clientId |
Client ID of an Active Directory service principal account. |
string |
clientSecretKey |
Client Key of an Active Directory service principal account. |
string |
clusters |
List of cluster CRNs where the credential should be created. |
< string > array |
name |
Name of the credential. |
string |
storageAccountName |
ABFS storage account name. |
string |
tenantId |
Tenant ID of an Active Directory service principal account. |
string |
type |
Type of the credential. Additional required parameters by type: ACCESSKEY: storageAccountName, accessKey; CLIENTKEY: clientId, clientSecretKey, tenantId. Superfluous parameters are ignored. Providing ACCESSKEY will create an ABFS Access Key based account in Ambari. Providing CLIENTKEY will create an Azure Active Directory Service Principal account in Cloudera Manager. ACCESSKEY is only supported on Ambari, while CLIENTKEY is only supported on Cloudera Manager. |
CreateAbfsCredentialResponse
Response object for creating ABFS credentials.
Name | Description | Schema |
---|---|---|
errors |
The list of response errors in case of a partial failure while creating the credential. |
< ApiError > array |
CreateAwsCredentialRequest
Request object for creating AWS IAM or Access & Secret Key cloud credentials.
Name | Description | Schema |
---|---|---|
accessKey |
AWS access key. |
string |
clusters |
List of cluster CRNs where the credential should be created. |
< string > array |
name |
Name of the credential. |
string |
secretKey |
AWS secret key. |
string |
type |
Type of the credential. Additional required parameters by type: IAM: none; ACCESSKEY: accessKey, secretKey. Superfluous parameters are ignored. Providing IAM will create an IAM Role-based Authentication account in Cloudera Manager. Providing ACCESSKEY will create an AWS Access Key Authentication account in Cloudera Manager. |
CreateAwsCredentialResponse
Response object for creating AWS credentials.
Name | Description | Schema |
---|---|---|
errors |
The list of response errors in case of a partial failure while creating the credential. |
< ApiError > array |
CreatePolicyRequest
Create a replication policy.
Name | Description | Schema |
---|---|---|
clusterCrn |
CRN of the cluster where the policy will be created. |
string |
policyDefinition |
Policy definition. |
|
policyName |
Name of the new policy. |
string |
CreatePolicyResponse
Response object for createPolicy.
Type : object
CredentialCollection
Cloud Credentials listing.
Name | Description | Schema |
---|---|---|
credentials |
Cloud credential details. |
< CredentialDetails > array |
results |
Number of cloud credentials in the response. |
integer (int64) |
totalResults |
Total number cloud credentials. |
integer (int64) |
CredentialDetails
Cloud credential details for by-name or by-id cloud credential request.
Name | Description | Schema |
---|---|---|
authType |
Authentication type. |
string |
configs |
Configurations. |
< string, string > map |
creationTime |
Creation time. |
string (date-time) |
customConfigs |
Custom configurations. |
< CustomConfigs > array |
id |
Credential id. |
string |
lastModifiedTime |
Last modified time. |
string (date-time) |
name |
Credential name. |
string |
provider |
Credential provider. |
string |
CredentialsResponse
Response object for cloud credentials.
Name | Description | Schema |
---|---|---|
clusterCrn |
Cluster CRN. |
string |
clusterUrl |
Cluster URL. |
string |
credentials |
Cloud credentials. |
|
replicationEngineType |
Replication engine type. |
enum (CM, DLMENGINE) |
CustomConfigs
Custom key/value configurations.
Name | Description | Schema |
---|---|---|
confidential |
Whether the value is confidential. |
boolean |
name |
Name. |
string |
value |
Value. |
string |
DeleteCredentialRequest
Request object for deleting a credential.
Name | Description | Schema |
---|---|---|
name |
Name of the cloud credential to delete. |
string |
DeleteCredentialResponse
Response object for delete credential operation.
Name | Description | Schema |
---|---|---|
errors |
The list of response errors in case of a partial failure while deleting the credential. |
< ApiError > array |
DeletePolicyRequest
Delete a replication policy.
Name | Description | Schema |
---|---|---|
clusterCrn |
CRN of the cluster where the policy is. |
string |
policyName |
Policy name to delete. |
string |
DeletePolicyResponse
Response object for deletePolicy.
Type : object
DownloadDiagnosticBundleRequest
Request to download the diagnostic bundle for the given cluster and command id.
Name | Description | Schema |
---|---|---|
clusterCrn |
The CRN of the target cluster. |
string |
commandId |
The command ID from the response of collect-diagnostic-bundle command. |
string |
DownloadDiagnosticBundleResponse
Response object for download-diagnostic-bundle. On certain clusters this response will show if the download is only available directly from CM.
Name | Description | Schema |
---|---|---|
bundleFile |
Base64 encoded content of the diagnostic bundle zip file. A script like this can be used to save the response as file 'cat response.json | jq -r '.bundleFile' | base64 -D > bundle.zip' |
string (byte) |
Error
An object returned on an error.
Name | Description | Schema |
---|---|---|
code |
The error code. |
string |
message |
The error message. |
string |
ErrorHandling
Error handling behavior.
Name | Description | Schema |
---|---|---|
abortOnError |
Abort policy on error. |
boolean |
abortOnSnapshotDiffFailures |
Abort policy if there are snapshot diff failures. |
boolean |
skipChecksumChecks |
Skip checksum checks. |
boolean |
skipListingChecksumChecks |
Skip checksum checks for copy-listing. |
boolean |
GetCommandStatusRequest
Get the current status of any CM command.
Name | Description | Schema |
---|---|---|
clusterCrn |
The CRN of the target cluster. |
string |
commandId |
ID of a CM command on the cluster. |
string |
GetCommandStatusResponse
Response object for get command status.
Name | Description | Schema |
---|---|---|
active |
Whether the command is still active. |
boolean |
bundleStatus |
The current status of the command. |
enum (IN_PROGRESS, DOWNLOADABLE_WITH_URL, DOWNLOADABLE_WITH_CLI, FAILED) |
bundleStatusMessage |
Further information about the current command status. |
string |
commandId |
Id of the CM command. |
integer (int64) |
endTime |
End time of the CM command. |
string |
name |
Name of the policy. |
string |
resultDataUrl |
Some commands have result data URL for downloading the diagnostic bundle. On certain CM versions the bundle download is only available through this URL, but not with download-diagnostic-bundle operation. |
string |
resultMessage |
Result message of the command. |
string |
startTime |
Start time of the CM command. |
string |
success |
Whether the diagnostic collection is successful. |
boolean |
GetCredentialsRequest
Request object to retrieve specific cloud credentials.
Name | Description | Schema |
---|---|---|
clusterCrn |
CRN of the cluster where the credential is. |
string |
credentialId |
Id of the credential. If both name and Id are specified, the Id is considered. |
string |
credentialName |
Name of the credential. If both name and Id are specified, the Id is considered. |
string |
GetCredentialsResponse
Response for credential request.
Name | Description | Schema |
---|---|---|
credentials |
List of credentials. |
< CredentialDetails > array |
HdfsArguments
Extra properties for HDFS.
Name | Description | Schema |
---|---|---|
alert |
Alerting behavior. |
|
deletePolicy |
Deletion behavior. |
enum (KEEP_DELETED_FILES, DELETE_TO_TRASH, DELETE_PERMANENTLY) |
errorHandling |
Error handling strategy. |
|
exclusionFilters |
Exclusion filters in glob format. |
< string > array |
logPath |
Job log path. |
string |
mapReduceService |
Name of the mapreduce service. |
string |
path |
HDFS path. Mandatory for HDFS policies, otherwise optional. |
string |
preserve |
Properties to preserve. |
|
replicationStrategy |
Replication strategy. |
enum (DYNAMIC, STATIC) |
HiveArguments
Hive replication arguments.
Name | Description | Schema |
---|---|---|
databasesAndTables |
Scope of the replication. |
< HiveDatabaseAndTables > array |
numThreads |
Number of threads to use during replication. |
integer (int64) |
sentryPermissions |
Include or exclude Sentry permissions. |
enum (INCLUDE, EXCLUDE) |
skipUrlPermissions |
Skip URL permissions. |
boolean |
HiveDatabaseAndTables
Hive replication scope.
Name | Description | Schema |
---|---|---|
database |
Database to replicate. |
string |
tablesExcludeRegex |
Regex to match excluded tables. |
string |
tablesIncludeRegex |
Regex to match included tables. |
string |
ListAllCredentialsRequest
Request object for listing all cloud credentials on all clusters.
Type : object
ListAllCredentialsResponse
Response object for list credentials request.
Name | Description | Schema |
---|---|---|
credentials |
List of credentials by cluster. |
< CredentialsResponse > array |
unreachableClusters |
List of errors for unreachable clusters. |
< ApiError > array |
ListClusterServiceStatusesRequest
Request object to list cluster statuses.
Type : object
ListClusterServiceStatusesResponse
Response object for listing of cluster statuses.
Name | Description | Schema |
---|---|---|
statuses |
List of statuses by cluster. |
< ClusterServiceStatuses > array |
ListClustersRequest
Request to list clusters.
Type : object
ListClustersResponse
Response object for list clusters request.
Name | Description | Schema |
---|---|---|
clusters |
List of clusters. |
< Cluster > array |
unreachableClusters |
List of errors for unreachable clusters. |
< ApiError > array |
ListPoliciesRequest
Request object for listing all policies.
Type : object
ListPoliciesResponse
Response object for a list policies request.
Name | Description | Schema |
---|---|---|
policies |
List of policies. |
< Policy > array |
policyErrors |
List of policy errors. |
< ApiError > array |
unreachableClusters |
List of errors for unreachable clusters. |
< ApiError > array |
Location
Location object.
Name | Description | Schema |
---|---|---|
city |
City. |
string |
country |
Country. |
string |
id |
Location ID. |
integer (int64) |
latitude |
Latitude. |
number (double) |
longitude |
Longitude. |
number (double) |
Policy
Replication policy definition.
Name | Description | Schema |
---|---|---|
creationTime |
Policy creation time. |
string (date-time) |
customProperties |
Custom replication properties. |
< string, string > map |
description |
Policy description. |
string |
endTime |
End time of policy run or when it repeats. |
string (date-time) |
executionType |
Policy execution type. |
string |
frequency |
Frequency in seconds when the policy runs. |
integer (int64) |
jobs |
List of policy runs. |
< PolicyInstanceResponse > array |
name |
Policy name. |
string |
nextRun |
Next expected run of the policy. |
string |
plugins |
Plugins. |
< string > array |
policyId |
Policy id. |
string |
report |
Policy report. |
|
sourceCluster |
Source cluster of replication. |
string |
sourceClusterCrn |
CRN of the source cluster. |
string |
sourceDataset |
Source dataset of replication. |
< string > array |
startTime |
Policy start time. |
string (date-time) |
status |
Current status of the policy. |
string |
target |
Replication target details. |
|
targetCluster |
Target cluster of replication if not cloud. |
string |
targetClusterCrn |
CRN of the target cluster. |
string |
targetDataset |
Target dataset of replication. |
< string > array |
type |
Policy type. |
string |
PolicyDefinition
Policy creation request definition.
Name | Description | Schema |
---|---|---|
cloudCredential |
Cloud credentials to use if replicating to Cloud. |
string |
cloudEncryptionAlgorithm |
The type of cloud encryption algorithm. Can be used only for HDP/Ambari managed clusters. Supported values are AES256,SSE-KMS. |
string |
cloudEncryptionKey |
Cloud encryption key. |
string |
cmPolicySubmitUser |
CM policy submit user. |
|
description |
Description of the policy. |
string |
distcpMapBandwidth |
Maximum bandwidth to use for replication. |
integer (int64) |
distcpMaxMaps |
Maximum Map jobs to use for replication. |
integer (int64) |
enableSnapshotBasedReplication |
Enable snapshot based replication. |
boolean |
endTime |
End time. |
string (date-time) |
frequencyInSec |
Policy’s frequency in seconds. |
integer (int64) |
hiveExternalTableBaseDirectory |
Hive external table base directory. |
string |
name |
Name of the new policy. |
string |
plugins |
Plugins. |
< string > array |
queueName |
YARN queue name if not default. |
string |
sourceCluster |
Name of source cluster in the format: "dataCenterName$cluster name", for example "DC-Europe$My Source 42. |
string |
sourceDataset |
Source dataset to replicate. |
|
startTime |
Start time. |
string (date-time) |
targetCluster |
Name of destination cluster in the format: "dataCenterName$cluster name", for example "DC-US$My Destination 17". |
string |
targetDataset |
Dataset to replicate to. |
string |
tdeSameKey |
TDE same key. |
boolean |
type |
Policy replication type. |
enum (FS, HIVE, HBASE) |
PolicyInstanceResponse
Policy execution details.
Name | Description | Schema |
---|---|---|
completedSteps |
Number of completed steps. |
integer (int32) |
endTime |
Command end time. |
string (date-time) |
id |
Policy instance command id. |
string |
jobType |
Job type. |
string |
message |
Output of the policy command. |
string |
policyId |
Policy id. |
string |
startTime |
Command start time. |
string (date-time) |
status |
Policy command status. |
string |
totalSteps |
Total number of steps. |
integer (int32) |
trackingInfo |
JSON data to track polciy status. |
string |
PolicyReport
Policy report.
Name | Description | Schema |
---|---|---|
lastFailedInstance |
Last failed instance. |
|
lastSucceededInstance |
Last successful instance. |
PolicyReportDetails
Policy report details.
Name | Description | Schema |
---|---|---|
endTime |
End time. |
string (date-time) |
status |
Status of the command. |
string |
Preserve
Properties to preserve after replication.
Name | Description | Schema |
---|---|---|
blockSize |
Block size. |
boolean |
extendedAttributes |
Extended attributes. |
boolean |
permissions |
Permissions. |
boolean |
replicationCount |
Replication count. |
boolean |
ServiceInfo
Service status definition.
Name | Description | Schema |
---|---|---|
serviceName |
Service name. |
string |
status |
Service status. |
string |
SourceDataset
Extra properties for the source dataset.
Name | Description | Schema |
---|---|---|
hdfsArguments |
HDFS arguments for HDFS and Hive replication. |
|
hiveArguments |
Arguments for Hive replication, null if HDFS replication. |
SuspendPolicyRequest
Suspend a replication policy.
Name | Description | Schema |
---|---|---|
clusterCrn |
CRN of the cluster where the policy is. |
string |
policyName |
Policy name to suspend. |
string |
SuspendPolicyResponse
Response object for suspendPolicy.
Type : object
TargetDetails
Target details.
Name | Description | Schema |
---|---|---|
environmentCrn |
Target’s environment CRN. |
string |
warehouseCrn |
Target’s warehouse CRN. |
string |