Cloudera Data Engineering

fullAccessGroups
optional

Groups with full access.

< string > array

fullAccessUsers
optional

Users with full access.

< string > array

users
optional

Workload usernames of CDP users granted access to the Virtual Cluster.

< string > array

viewOnlyGroups
optional

Groups with view only access.

< string > array

viewOnlyUsers
optional

Users with view only access.

< string > array

cloudPlatform
required

The cloud platform where a backed-up CDE service was enabled.

string

created
required

The time when the backup was created.

string

creator
required

The workload name of the user who created this backup.

string

creatorCrn
required

CRN of of the user who created this backup.

string

description
optional

Backup description.

string

environmentCrn
required

CRN of CDP environment with which a backed-up CDE service was associated.

string

environmentName
required

Name of CDP environment with which a backed-up CDE service was associated.

string

errorMessage
optional

Error message, only set when there is an error.

string

id
required

Backup ID.

integer (int64)

serviceID
required

Service ID of a backed-up CDE service.

string

serviceName
required

Name of a backed-up CDE service.

string

serviceVersion
required

Version of a backed-up CDE service.

string

status
required

Backup status.

string

credentialFilter
optional

Filter credentials for backup using 'fieldname(operator)argument' syntax. Here, 'fieldname' is one of the credential fields (e.g. name, type, etc.), and 'operator' is one of ('eq', 'noteq', 'lte', 'lt', 'gte', 'gt', 'in', 'notin', 'like', 'rlike', 'is', 'isnot'). Multiple filters are ANDed. For example, 'name(eq)test-credential'. This will only take effect if credentials are backed up, as controlled by the 'includeCredentials' option.

< string > array

includeCredentialSecrets
optional

Whether to backup credential secrets. Secrets are obfuscated while storing. By default, credential secrets are backed up.
Default : true

boolean

includeCredentials
optional

Whether to backup credentials. By default, credentials are backed up.
Default : true

boolean

includeJobResources
optional

Whether to backup job resources. By default, job resources are backed up.
Default : true

boolean

includeJobs
optional

Whether to backup jobs. By default, jobs are backed up.
Default : true

boolean

includeResourceCredentials
optional

Whether to backup resource credentials. By default, resource credentials are backed up.
Default : true

boolean

includeResources
optional

Whether to backup resources. By default, resources are backed up.
Default : true

boolean

jobFilter
optional

Filter jobs for backup using 'fieldname(operator)argument' syntax. Here, 'fieldname' is one of the job fields (e.g. name, type, etc.), and 'operator' is one of ('eq', 'noteq', 'lte', 'lt', 'gte', 'gt', 'in', 'notin', 'like', 'rlike', 'is', 'isnot'). Multiple filters are ANDed. For example, 'name(eq)SparkPi'. This will only take effect if jobs are backed up, as controlled by the 'includeJobs' option.

< string > array

resourceFilter
optional

Filter resources for backup using 'fieldname(operator)argument' syntax. Here, 'fieldname' is one of the resource fields (e.g. name, type, status, etc.), and 'operator' is one of ('eq', 'noteq', 'lte', 'lt', 'gte', 'gt', 'in', 'notin', 'like', 'rlike', 'is', 'isnot'). Multiple filters are ANDed. For example, 'name(eq)test-resource'. This will only take effect if resources are backed up, as controlled by the 'includeResources' option.

< string > array

validateArchive
optional

Whether to validate backup archive. By default, backup archive is not validated.
Default : false

boolean

backupID
required

The ID of the backup to cancel.

integer (int64)

chartName
optional

Name of the chart that has to be overridden, for eg- "dex-app", "dex-base".

string

overrides
optional

Space separated key-value pairs for overriding chart values. The key and the value must be separated using a colon(:) For eg- "airflow.enabled:true safari.enabled:true".

string

chartName
optional

Name of the chart that has to be overridden, for eg- "dex-app", "dex-base".

string

overrides
optional

Space separated key value-pairs for overriding chart values. The key and the value must be separated using a colon(:) For eg- "airflow.enabled:true safari.enabled:true".

string

backupPath
optional

Backup file relative path. The path should not include file name, and be relative to the service backup location. The service backup location can be found in the service description obtained via the 'describe-service' command.

string

backupVcContentOptions
optional

Backup virtual cluster content options.

BackupContentOptions

description
optional

Backup description.

string

includeVc
optional

Whether to backup service virtual clusters. By default, virtual clusters are backed up.
Default : true

boolean

serviceId
required

Service ID of the service to backup.

string

timeout
optional

Backup operation timeout in seconds.

integer (int32)

backupID
required

Backup ID, which can be used for log retrieval, restoration, deletion, and backup cancellation.

integer (int64)

aclUsers
optional

Comma-separated Workload usernames of CDP users to be granted access to the Virtual Cluster.

string

chartValueOverrides
optional

Chart overrides for creating a virtual cluster.

< ChartValueOverridesRequest > array

clusterId
required

Cluster id of the CDE service where virtual cluster has to be created.

string

cpuRequests
required

Cpu requests for autoscaling.

string

fullAccessGroups
optional

Groups with full access.

< string > array

fullAccessUsers
optional

Users with full access.

< string > array

memoryRequests
required

Memory requests for autoscaling - eg. 30Gi.

string

name
required

Name of the virtual cluster.

string

runtimeSpotComponent
optional

Used to describe where the Driver and the Executors would run. By default the Driver would run on on-demand instances and the Executors on spot instances. Setting it to ALL will run both the Driver and the Executors on spot instances whereas setting it to NONE should run both the Driver and the Executor on on-demand instances. Currently applicable for aws services only. Use this option only on services with spot instances enabled.

enum (ALL, NONE)

smtpConfigs
optional

SMTP Configurations for Airflow Email Alerts.

SmtpConfigRequest

sparkVersion
optional

Spark version for the virtual cluster. Currently supported spark versions are SPARK2(deprecated), SPARK3 and SPARK3_3. This feature is only supported in CDE-1.7.0 and beyond. SPARK3_3 is supported in CDE-1.19 and beyond.

enum (SPARK2, SPARK2_4, SPARK3, SPARK3_0, SPARK3_1, SPARK3_2, SPARK3_3)

vcTier
optional

Tier of the virtual cluster. Currently supported tiers are CORE and ALLP. CORE tiered virtual cluster enables operational deployment via batch jobs. ALLP virtual clusters are all-purpose virtual clusters supporting both operational batch jobs and interactive sessions. This feature is only supported in CDE-1.19.0 and beyond.

enum (ALLP, CORE)

viewOnlyGroups
optional

Groups with view only access.

< string > array

viewOnlyUsers
optional

Users with view only access.

< string > array

Vc
optional

Created Virtual Cluster

VcDescription

azureFilesFQDN
optional

Azure File Share’s server address. Defaults to '<storageaccount>.file.core.windows.net'.

string

resourceGroup
required

Resource Group of the Storage Account.

string

storageAccountName
required

Azure Storage Account of the File Share.

string

backupID
required

The ID of the backup to delete.

integer (int64)

clusterId
required

Cluster id of the CDE service where virtual cluster was enabled.

string

vcId
required

Virtual cluster id

string

status
optional

status of virtual cluster deletion.

string

backupID
required

The ID of the backup to describe.

integer (int64)

backup
required

Service backup.

Backup

clusterId
required

Cluster id of the service to be described.

string

service
optional

Description of the service

ServiceDescription

clusterId
required

Cluster id of the CDE service where vc was created.

string

vcId
required

Virtual cluster id.

string

vc
optional

Description of the virtual cluster.

VcDescription

clusterId
required

cluster Id of the service to disable.

string

force
optional

Force disable cde service.
Default : false

boolean

status
optional

Status of deletion

string

chartValueOverrides
optional

Chart overrides for enabling a service.

< ChartValueOverridesRequest > array

customAzureFilesConfigs
optional

CDE uses a default public File Shares storage provisioned by AKS. Enable this option to use your own public/private File Shares.

CustomAzureFilesConfigs

deployPreviousVersion
optional

If set to "true", the previous version of the CDE service will be deployed.
Default : false

boolean

enablePrivateNetwork
optional

Create a fully private CDE instance on either Amazon or Azure. This includes services such as Kubernetes, MySQL, etc. For Azure, this will also enable virtual network (VNet) access via private endpoints and private link.

boolean

enablePublicEndpoint
optional

Creates a CDE endpoint (Load Balancer) in a publicly accessible subnet. If set false, the endpoint will be created in a private subnet and you will need to setup access to the endpoint manually in your cloud account.
Default : false

boolean

enableWorkloadAnalytics
optional

If set false, diagnostic information about job and query execution is sent to Cloudera Workload Manager. Anonymization can be configured under Environments / Shared Resources / Telemetry. Refer documentation for more info at https://docs.cloudera.com/workload-manager/cloud/index.html.
Default : false

boolean

env
required

CDP environment where cde service should be enabled.

string

initialInstances
optional

Initial Instances when the service is enabled.

integer (int32)

initialSpotInstances
optional

Initial spot Instances when the service is enabled.

integer (int32)

instanceType
required

Instance type of the cluster for CDE Service.

string

loadbalancerAllowlist
optional

List of CIDRs that would be allowed to access the load balancer.

< string > array

maximumInstances
required

Maximum Instances for the CDE Service.

integer (int32)

maximumSpotInstances
optional

Maximum Spot Instances for the CDE Service.

integer (int32)

minimumInstances
required

Minimum Instances for the CDE Service.

integer (int32)

minimumSpotInstances
optional

Minimum Spot instances for the CDE Service.

integer (int32)

name
required

Name of the CDE Service.

string

networkOutboundType
optional

Network outbound type. Currently 'udr' is the only supported.

enum (UDR)

rootVolumeSize
optional

EBS volume size in GB.

integer (int32)

skipValidation
optional

Skip Validation check.
Default : false

boolean

subnets
optional

List of Subnet IDs of CDP subnets to use for the kubernetes worker node.

< string > array

tags
optional

User defined labels that tag all provisioned cloud resources.

< string, string > map

useSsd
optional

Instance local storage (SSD) would be used for the workload filesystem (Example - spark local directory). In case the workload requires more space than what’s available in the instance storage, please use an instance type with sufficient instance local storage or choose an instance type without SSD and configure the EBS volume size. Currently supported only for aws services.

boolean

whitelistIps
optional

List of CIDRs that would be allowed to access kubernetes master API server.

< string > array

service
optional

Service details of the created service.

ServiceDescription

code
optional

The error code.

string

message
optional

The error message.

string

backupID
required

The ID of the backup.

integer (int64)

pageSize
optional

The size of each page.
Minimum value : 1
Maximum value : 1000

integer (int32)

startingToken
optional

A token to specify where to start paginating. This is the nextToken from a previously truncated response.

string

logs
required

Contains the logs of the backup operation.

< string > array

nextToken
optional

The token to use when requesting the next set of results. If there are no additional results, the string is empty.

string

clusterId
required

cluster id of the cde service.

string

kubeconfig
optional

kubeconfig for the cluster.

string

pageSize
optional

The size of each page.
Minimum value : 1
Maximum value : 500

integer (int32)

serviceID
required

The service ID of the service.

string

startingToken
optional

A token to specify where to start paginating. This is the nextToken from a previously truncated response.

string

logs
required

Contains the logs of the service initialization operation.

< string > array

nextToken
optional

The token to use when requesting the next set of results. If there are no additional results, the string is empty.

string

filter
optional

Filter backups using 'fieldname(operator)argument' syntax. Here 'fieldname' is one of ('id', 'clusterID', 'cdeVersion', 'environment', 'environmentCrn', 'created', 'status') and 'operator' is one of ('eq', 'noteq', 'lte', 'lt', 'gte', 'gt', 'in', 'notin', 'like', 'rlike', 'is', 'isnot'). Multiple filters are ANDed. For example, 'status(eq)failed'.

< string > array

order
optional

Whether ordering is ascending ('ASC') or descending ('DESC'), default 'DESC'.

Order

orderBy
optional

The 'fieldname' to order by, default "id". Here 'fieldname' is one of ('id', 'clusterID', 'cdeVersion', 'environment', 'environmentCrn', 'created', 'status').
Default : "id"

string

pageSize
optional

The size of each page.
Minimum value : 1
Maximum value : 500

integer (int32)

startingToken
optional

A token to specify where to start paginating. This is the nextToken from a previously truncated response.

string

backups
required

The backups.

< Backup > array

nextToken
optional

The token to use when requesting the next set of results. If there are no additional results, the string is empty.

string

removeDeleted
optional

Filter out deleted CDE services from the list.
Default : false

boolean

services
optional

List of services.

< ServiceSummary > array

clusterId
required

Cluster id of the service.

string

vcs
optional

List of virtual clusters.

< VcSummary > array

backupID
required

The ID of the backup to restore from.

integer (int64)

environmentCrn
required

CRN of the CDP environment with which a restored CDE service will be associated. Currently, CDE service can be restored only to the same CDP environment, with which a backed-up service was associated.

string

serviceId
optional

Specify the restored service ID. If not specified, the service ID from the backup will be reused.

string

serviceName
optional

Specify the restored service name. If not specified, the service name from the backup will be reused.

string

serviceID
required

The ID of the service being restored.

string

backupLocation
optional

Base location for the service backup archives.

string

chartValueOverrides
optional

Chart overrides for the Virtual Cluster.

< ChartValueOverridesResponse > array

cloudPlatform
optional

The cloud platform where the CDE service is enabled.

string

clusterFqdn
optional

FQDN of the CDE service.

string

clusterId
required

Cluster Id of the CDE Service.

string

creatorCrn
optional

CRN of the creator.

string

creatorEmail
optional

Email address of the creator of the CDE service.

string

dataLakeAtlasUIEndpoint
optional

Endpoint of Data Lake Atlas.

string

dataLakeFileSystems
optional

The Data lake file system.

string

enablingTime
optional

Timestamp of service enabling.

string

environmentCrn
optional

CRN of the environment.

string

environmentName
required

CDP Environment Name.

string

loadbalancerAllowlist
optional

Comma-separated CIDRs that would be allowed to access the load balancer.

string

logLocation
optional

Location for the log files of jobs.

string

name
required

Name of the CDE Service.

string

networkOutboundType
optional

Network outbound type. Currently 'udr' is the only supported.

string

previousVersionDeployed
optional

The "true" value indicates that the previous version of the CDE service was requested to be deployed.

boolean

privateClusterEnabled
optional

If true, the CDE service was created with fully private Azure services (AKS, MySQL, etc.).

boolean

publicEndpointEnabled
optional

If true, the CDE endpoint was created in a publicly accessible subnet.

boolean

resources
optional

Resources details of CDE Service.

ServiceResources

ssdUsed
optional

If true, SSD would have been be used for workload filesystem.

boolean

status
optional

Status of the CDE service.

string

subnets
optional

List of Subnet IDs of the CDP subnets used by the kubernetes worker node.

string

tenantId
optional

CDP tenant ID.

string

whitelistIps
optional

List of CIDRs that would be allowed to access kubernetes master API server.

string

workloadAnalyticsEnabled
optional

If true, diagnostic information about job and query execution is sent to Cloudera Workload Manager.

boolean

initial_instances
optional

Initial instances for the CDE service.

string

initial_spot_instances
optional

Initial Spot Instances for the CDE Service.

string

instance_type
optional

Instance type of the CDE service.

string

max_instances
optional

Maximum instances for the CDE service.

string

max_spot_instances
optional

Maximum Number of Spot instances.

string

min_instances
optional

Minimum Instances for the CDE service.

string

min_spot_instances
optional

Minimum number of spot instances for the CDE service.

string

root_vol_size
optional

Root Volume Size.

string

clusterId
required

Cluster Id of the CDE Service.

string

creatorEmail
optional

Email Address of the CDE creator.

string

enablingTime
optional

Timestamp of service enabling.

string

environmentName
required

CDP Environment Name.

string

name
required

Name of the CDE Service.

string

status
required

Status of the CDE service.

string

email
required

Sender’s email address.

string

host
required

SMTP host.

string

password
optional

SMTP password.

string

port
optional

SMTP port.
Default : 25

integer (int32)

ssl
optional

Use SSL to secure the connection to the email server.
Default : false

boolean

startTls
optional

Use SMTP STARTTLS command to encrypt the mail.
Default : true

boolean

username
optional

SMTP username.

string

email
required

Sender’s email address.

string

host
required

SMTP host.

string

port
optional

SMTP port.
Default : 25

integer (int32)

ssl
optional

Use SSL to secure the connection to the email server.
Default : false

boolean

startTls
optional

Use SMTP STARTTLS command to encrypt the mail.
Default : true

boolean

username
optional

SMTP username.

string

clusterId
required

Cluster ID of the service to update.

string

loadbalancerAllowlist
optional

List of CIDRs that would be allowed to access the load balancer.

< string > array

maximumInstances
optional

Maximum number of instances for the CDE service.

integer (int32)

maximumSpotInstances
optional

Maximum number of spot instances for the CDE service.

integer (int32)

minimumInstances
optional

Minimum number of instances for the CDE service.

integer (int32)

minimumSpotInstances
optional

Minimum number of spot instances for the CDE service.

integer (int32)

whitelistIps
optional

List of CIDRs that would be allowed to access kubernetes master API server.

< string > array

operationId
optional

Deprecated. The operation ID of the update request.

string

status
optional

Status of update.

string

aclUsers
optional

Comma-separated Workload usernames of CDP users to be granted access to the Virtual Cluster.

string

clusterId
required

Cluster ID of the CDE service where Virtual Cluster was enabled.

string

fullAccessGroups
optional

Groups with full access.

< string > array

fullAccessUsers
optional

Users with full access.

< string > array

vcId
required

Virtual Cluster ID

string

viewOnlyGroups
optional

Groups with view only access.

< string > array

viewOnlyUsers
optional

Users with view only access.

< string > array

Vc
optional

Updated Virtual Cluster.

VcDescription

VcUiUrl
optional

URL of the CDE Virtual Cluster UI.

string

accessControl
optional

Access control object for the Virtual Cluster.

AccessControlResponse

chartValueOverrides
optional

Chart overrides for the Virtual Cluster.

< ChartValueOverridesResponse > array

clusterId
required

Cluster ID of the CDE service that contains the Virtual Cluster.

string

creationTime
optional

Time of creation of the virtual Cluster.

string

creatorEmail
optional

Email address of the creator of Virtual Cluster.

string

creatorID
optional

ID of the creator of Virtual Cluster.

string

creatorName
optional

Name of the creator of the Virtual Cluster.

string

historyServerUrl
optional

Spark History Server URL for the Virtual Cluster.

string

livyServerUrl
optional

Livy Server URL for the Virtual Cluster.

string

resources
optional

Resources details of CDE Service.

VcResources

safariUrl
optional

Safari URL for the Virtual Cluster.

string

smtpConfig
optional

SMTP Configurations for Airflow Email Alerts.

SmtpConfigResponse

sparkVersion
optional

Spark version for the virtual cluster.

string

status
optional

Status of the Virtual Cluster.

string

vcApiUrl
optional

Url for the Virtual Cluster APIs.

string

vcId
required

Virtual Cluster ID.

string

vcName
required

Name of the CDE Virtual Cluster.

string

vcTier
optional

Tier of the Virtual Cluster.

string

actualCpuRequests
optional

Actual CPU request for the VC. This accounts for other dex apps(eg. livy, airflow), that run in the virtual cluster.

string

actualMemoryRequests
optional

Actual Memory request for the VC. This accounts for other dex apps(eg. livy, airflow), that run in the virtual cluster.

string

cpuRequests
optional

The CPU requests for VC for running spark jobs.

string

memRequests
optional

The Memory requests for VC for running spark jobs.

string

VcUiUrl
optional

Url for the Virtual Cluster UI.

string

clusterId
required

Cluster ID of the CDE service that contains the virtual cluster.

string

creatorEmail
optional

Email Address of the Creator.

string

creatorName
optional

Name of the Creator.

string

status
optional

Status of the Virtual Cluster.

string

vcId
required

Virtual Cluster ID.

string

vcName
required

Name of the CDE Virtual Cluster.

string