create-vw¶

subtitle: Create a Virtual Warehouse.
version: 0.9.128

Description¶

Create a Virtual Warehouse.

Synopsis¶

  create-vw
--cluster-id <value>
--dbc-id <value>
--vw-type <value>
--name <value>
[--instance-type <value>]
[--image-version <value>]
[--t-shirt-size <value>]
[--node-count <value>]
[--availability-zone <value>]
[--enable-unified-analytics | --no-enable-unified-analytics]
[--impala-options <value>]
[--impala-ha-settings <value>]
[--autoscaling <value>]
[--config <value>]
[--query-isolation-options <value>]
[--tags <value>]
[--resource-pool <value>]
[--platform-jwt-auth | --no-platform-jwt-auth]
[--impala-query-log | --no-impala-query-log]
[--ebs-llap-spill-gb <value>]
[--hive-server-ha-mode <value>]
[--cli-input-json <value>]
[--generate-cli-skeleton]

Options¶

--cluster-id (string)

ID of cluster where Virtual Warehouse should be created.

--dbc-id (string)

ID of Database Catalog that the Virtual Warehouse should be attached to.

--vw-type (string)

Type of Virtual Warehouse: ‘hive’ or ‘impala’.

Possible values:

hive

impala

--name (string)

Name of the Virtual Warehouse.

--instance-type (string)

Instance type for this Virtual Warehouse. To learn what instance types are allowed to be used for a Hive or an Impala Virtual Warehouse, please use the ‘describe-allowed-instance-types’ command. The command output will list the usable instance types in its ‘hive’ and ‘impala’ fields accordingly.

Form Factors: public

--image-version (string)

version of the Virtual Warehouse.

Form Factors: public

--t-shirt-size (string)

Name of T-shirt size to use, which will determine the number of nodes.

Possible values:

xsmall

small

medium

large

--node-count (integer)

Nodes per compute cluster. If specified, forces ‘template’ to be ‘custom’

--availability-zone (string)

This feature works only for AWS cluster type. An availability zone to host compute instances. If not specified, defaults to a randomly selected availability zone inferred from available subnets. In order to query possible options, see “availabilityZones” field of describe-cluster or list-clusters command response.

--enable-unified-analytics | --no-enable-unified-analytics (boolean)

Enable Unified Analytics. In the case of Hive Virtual Warehouses, this cannot be provided, because this value is inferred. In the case of Impala, this can be set. Passing –query-isolation-options will be considered only if this flag is set to true. If Unified Analytics is enabled then the “enableShutdownOfCoordinator” in –impala-ha-settings is explicitly disabled (ignored) and should not be provided, furthermore the “highAvailabilityMode” in –impala-ha-settings cannot be set to ACTIVE_ACTIVE.

--impala-options (object)

Impala specific options. It cannot be provided for Virtual Warehouse types other than Impala.

spillToS3Uri -> (string)

Set S3 URI in “s3://bucket/path” format and enable spilling to S3. If this is set, “scratchSpaceLimit” cannot be set. Spilling to a remote storage is not supported on Azure, it is available on AWS and PvC platforms. If your workload results in large volumes of intermediate data being written, it is recommended to configure the heavy spilling queries to use a remote storage location rather than the local one since Impala occasionally needs to use persistent storage for writing intermediate files during large sorts, joins, aggregations, or analytic function operations. Note that this value is ignored if the Impala version does not support spilling to S3.

scratchSpaceLimit -> (integer)

Defines the limit for scratch space in GiB needed by Impala for spilling queries. If this is set, “spillToS3Uri” cannot be set. The valid values are between 1-16000 for AWS and 1-16684 for Azure. It will choose the first valid value for which the total disk space is greater than or equal to the specified scratch space limit. The valid scratch space limit values per node in GiBs for AWS are [300, 634, 968, 1302, 1636, 1970, 2304, 2638, 2972, 6550, 7000, 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, 16000]. Valid values for Azure are [300, 684, 812, 1068, 1324, 1836, 2348, 3372, 4396, 8492, 16684].

Shorthand Syntax:

spillToS3Uri=string,scratchSpaceLimit=integer

JSON Syntax:

{
  "spillToS3Uri": "string",
  "scratchSpaceLimit": integer
}

--impala-ha-settings (object)

High Availability settings for the Impala Virtual Warehouse. The values are disregarded for Hive.

highAvailabilityMode -> (string)

Set Impala High Availability mode. If not provided, the default will apply. DISABLED - Disables Impala coordinator and Database Catalog high availability. ACTIVE_PASSIVE - Runs multiple coordinators (one active, one passive) and Database Catalogs (one active, one passive). ACTIVE_ACTIVE - Runs multiple coordinators (both active) and Database Catalogs (one active, one passive). If Unified Analytics is enabled, then this cannot be set to ACTIVE_ACTIVE.

enableShutdownOfCoordinator -> (boolean)

Enables a shutdown of the coordinator. If Unified Analytics is enabled, then this setting is explicitly disabled (ignored) and should not be provided.

shutdownOfCoordinatorDelaySeconds -> (integer)

Delay in seconds before the shutdown of coordinator event happens.

numOfActiveCoordinators -> (integer)

The number of active coordinators.

enableCatalogHighAvailability -> (boolean)

Enables a backup instance for Impala catalog to ensure high availability.

enableStatestoreHighAvailability -> (boolean)

Enables a backup instance for Impala Statestore to ensure high availability.

Shorthand Syntax:

highAvailabilityMode=string,enableShutdownOfCoordinator=boolean,shutdownOfCoordinatorDelaySeconds=integer,numOfActiveCoordinators=integer,enableCatalogHighAvailability=boolean,enableStatestoreHighAvailability=boolean

JSON Syntax:

{
  "highAvailabilityMode": "ACTIVE_PASSIVE"|"ACTIVE_ACTIVE"|"DISABLED",
  "enableShutdownOfCoordinator": true|false,
  "shutdownOfCoordinatorDelaySeconds": integer,
  "numOfActiveCoordinators": integer,
  "enableCatalogHighAvailability": true|false,
  "enableStatestoreHighAvailability": true|false
}

--autoscaling (object)

Auto-scaling configuration for a Virtual Warehouse.

minClusters -> (integer)

Minimum number of available compute groups.

maxClusters -> (integer)

Maximum number of available compute groups.

disableAutoSuspend -> (boolean)

Turn off auto suspend for Virtual Warehouse.

autoSuspendTimeoutSeconds -> (integer)

Auto suspend threshold for Virtual Warehouse.

hiveScaleWaitTimeSeconds -> (integer)

Set wait time before a scale event happens. Either “hiveScaleWaitTimeSeconds” or “hiveDesiredFreeCapacity” can be provided.

hiveDesiredFreeCapacity -> (integer)

Set Desired free capacity. Either “hiveScaleWaitTimeSeconds” or “hiveDesiredFreeCapacity” can be provided.

impalaScaleUpDelaySeconds -> (integer)

Scale up the scaling up threshold in seconds. If not provided defaults will apply.

impalaScaleDownDelaySeconds -> (integer)

Scale down threshold in seconds. If not provided defaults will apply.

impalaExecutorGroupSets -> (object)

Configure executor group sets for workload aware autoscaling.

small -> (object)

Configure independently scaling set of uniformly sized executor groups.

execGroupSize -> (integer)

Set number of executors per executor group.

minExecutorGroups -> (integer)

Set minimum number of executor groups allowed.

maxExecutorGroups -> (integer)

Set maximum number of executor groups allowed.

autoSuspendTimeoutSeconds -> (integer)

Set auto suspend threshold. If not provided defaults will apply.

disableAutoSuspend -> (boolean)

Turn off auto suspend. If not provided defaults will apply.

triggerScaleUpDelay -> (integer)

Set scale up threshold in seconds. If not provided defaults will apply.

triggerScaleDownDelay -> (integer)

Set scale down threshold in seconds. If not provided defaults will apply.

custom1 -> (object)

Configure independently scaling set of uniformly sized executor groups.

execGroupSize -> (integer)

Set number of executors per executor group.

minExecutorGroups -> (integer)

Set minimum number of executor groups allowed.

maxExecutorGroups -> (integer)

Set maximum number of executor groups allowed.

autoSuspendTimeoutSeconds -> (integer)

Set auto suspend threshold. If not provided defaults will apply.

disableAutoSuspend -> (boolean)

Turn off auto suspend. If not provided defaults will apply.

triggerScaleUpDelay -> (integer)

Set scale up threshold in seconds. If not provided defaults will apply.

triggerScaleDownDelay -> (integer)

Set scale down threshold in seconds. If not provided defaults will apply.

custom2 -> (object)

Configure independently scaling set of uniformly sized executor groups.

execGroupSize -> (integer)

Set number of executors per executor group.

minExecutorGroups -> (integer)

Set minimum number of executor groups allowed.

maxExecutorGroups -> (integer)

Set maximum number of executor groups allowed.

autoSuspendTimeoutSeconds -> (integer)

Set auto suspend threshold. If not provided defaults will apply.

disableAutoSuspend -> (boolean)

Turn off auto suspend. If not provided defaults will apply.

triggerScaleUpDelay -> (integer)

Set scale up threshold in seconds. If not provided defaults will apply.

triggerScaleDownDelay -> (integer)

Set scale down threshold in seconds. If not provided defaults will apply.

custom3 -> (object)

Configure independently scaling set of uniformly sized executor groups.

execGroupSize -> (integer)

Set number of executors per executor group.

minExecutorGroups -> (integer)

Set minimum number of executor groups allowed.

maxExecutorGroups -> (integer)

Set maximum number of executor groups allowed.

autoSuspendTimeoutSeconds -> (integer)

Set auto suspend threshold. If not provided defaults will apply.

disableAutoSuspend -> (boolean)

Turn off auto suspend. If not provided defaults will apply.

triggerScaleUpDelay -> (integer)

Set scale up threshold in seconds. If not provided defaults will apply.

triggerScaleDownDelay -> (integer)

Set scale down threshold in seconds. If not provided defaults will apply.

large -> (object)

Configure independently scaling set of uniformly sized executor groups.

execGroupSize -> (integer)

Set number of executors per executor group.

minExecutorGroups -> (integer)

Set minimum number of executor groups allowed.

maxExecutorGroups -> (integer)

Set maximum number of executor groups allowed.

autoSuspendTimeoutSeconds -> (integer)

Set auto suspend threshold. If not provided defaults will apply.

disableAutoSuspend -> (boolean)

Turn off auto suspend. If not provided defaults will apply.

triggerScaleUpDelay -> (integer)

Set scale up threshold in seconds. If not provided defaults will apply.

triggerScaleDownDelay -> (integer)

Set scale down threshold in seconds. If not provided defaults will apply.

Shorthand Syntax:

minClusters=integer,maxClusters=integer,disableAutoSuspend=boolean,autoSuspendTimeoutSeconds=integer,enableUnifiedAnalytics=boolean,hiveScaleWaitTimeSeconds=integer,hiveDesiredFreeCapacity=integer,impalaHighAvailabilityMode=string,impalaScaleUpDelaySeconds=integer,impalaScaleDownDelaySeconds=integer,impalaEnableShutdownOfCoordinator=boolean,impalaShutdownOfCoordinatorDelaySeconds=integer,impalaNumOfActiveCoordinators=integer,impalaEnableCatalogHighAvailability=boolean,impalaExecutorGroupSets={small={execGroupSize=integer,minExecutorGroups=integer,maxExecutorGroups=integer,autoSuspendTimeoutSeconds=integer,disableAutoSuspend=boolean,triggerScaleUpDelay=integer,triggerScaleDownDelay=integer},custom1={execGroupSize=integer,minExecutorGroups=integer,maxExecutorGroups=integer,autoSuspendTimeoutSeconds=integer,disableAutoSuspend=boolean,triggerScaleUpDelay=integer,triggerScaleDownDelay=integer},custom2={execGroupSize=integer,minExecutorGroups=integer,maxExecutorGroups=integer,autoSuspendTimeoutSeconds=integer,disableAutoSuspend=boolean,triggerScaleUpDelay=integer,triggerScaleDownDelay=integer},custom3={execGroupSize=integer,minExecutorGroups=integer,maxExecutorGroups=integer,autoSuspendTimeoutSeconds=integer,disableAutoSuspend=boolean,triggerScaleUpDelay=integer,triggerScaleDownDelay=integer},large={execGroupSize=integer,minExecutorGroups=integer,maxExecutorGroups=integer,autoSuspendTimeoutSeconds=integer,disableAutoSuspend=boolean,triggerScaleUpDelay=integer,triggerScaleDownDelay=integer}}

JSON Syntax:

{
  "minClusters": integer,
  "maxClusters": integer,
  "disableAutoSuspend": true|false,
  "autoSuspendTimeoutSeconds": integer,
  "enableUnifiedAnalytics": true|false,
  "hiveScaleWaitTimeSeconds": integer,
  "hiveDesiredFreeCapacity": integer,
  "impalaHighAvailabilityMode": "ACTIVE_PASSIVE"|"ACTIVE_ACTIVE"|"DISABLED",
  "impalaScaleUpDelaySeconds": integer,
  "impalaScaleDownDelaySeconds": integer,
  "impalaEnableShutdownOfCoordinator": true|false,
  "impalaShutdownOfCoordinatorDelaySeconds": integer,
  "impalaNumOfActiveCoordinators": integer,
  "impalaEnableCatalogHighAvailability": true|false,
  "impalaExecutorGroupSets": {
    "small": {
      "execGroupSize": integer,
      "minExecutorGroups": integer,
      "maxExecutorGroups": integer,
      "autoSuspendTimeoutSeconds": integer,
      "disableAutoSuspend": true|false,
      "triggerScaleUpDelay": integer,
      "triggerScaleDownDelay": integer
    },
    "custom1": {
      "execGroupSize": integer,
      "minExecutorGroups": integer,
      "maxExecutorGroups": integer,
      "autoSuspendTimeoutSeconds": integer,
      "disableAutoSuspend": true|false,
      "triggerScaleUpDelay": integer,
      "triggerScaleDownDelay": integer
    },
    "custom2": {
      "execGroupSize": integer,
      "minExecutorGroups": integer,
      "maxExecutorGroups": integer,
      "autoSuspendTimeoutSeconds": integer,
      "disableAutoSuspend": true|false,
      "triggerScaleUpDelay": integer,
      "triggerScaleDownDelay": integer
    },
    "custom3": {
      "execGroupSize": integer,
      "minExecutorGroups": integer,
      "maxExecutorGroups": integer,
      "autoSuspendTimeoutSeconds": integer,
      "disableAutoSuspend": true|false,
      "triggerScaleUpDelay": integer,
      "triggerScaleDownDelay": integer
    },
    "large": {
      "execGroupSize": integer,
      "minExecutorGroups": integer,
      "maxExecutorGroups": integer,
      "autoSuspendTimeoutSeconds": integer,
      "disableAutoSuspend": true|false,
      "triggerScaleUpDelay": integer,
      "triggerScaleDownDelay": integer
    }
  }
}

--config (object)

Configurations for a service (DBC or VW)

commonConfigs -> (object)

Configurations for an application inside a service.

configBlocks -> (array)

List of ConfigBlocks for the application.

item -> (object)

A piece of configuration stored in the same place (e.g. same file or environment variables).

id -> (string)

ID of the ConfigBlock. Unique within an ApplicationConfig.

content -> (object)

Contents of a ConfigBlock.

keyValues -> (map)

Key-value type configurations.

key -> (string)

value -> (string)

Key-value type configurations.

text -> (string)

Text type configuration.

json -> (string)

JSON type configuration.

applicationConfigs -> (map)

Application specific configurations.

key -> (string)

value -> (object)

Configurations for an application inside a service.

configBlocks -> (array)

List of ConfigBlocks for the application.

item -> (object)

A piece of configuration stored in the same place (e.g. same file or environment variables).

id -> (string)

ID of the ConfigBlock. Unique within an ApplicationConfig.

content -> (object)

Contents of a ConfigBlock.

keyValues -> (map)

Key-value type configurations.

key -> (string)

value -> (string)

Key-value type configurations.

text -> (string)

Text type configuration.

json -> (string)

JSON type configuration.

ldapGroups -> (array)

LDAP Groupnames to be enabled for auth.

item -> (string)

enableSSO -> (boolean)

Should SSO be enabled for this VW.

JSON Syntax:

{
  "commonConfigs": {
    "configBlocks": [
      {
        "id": "string",
        "content": {
          "keyValues": {"string": "string"
            ...},
          "text": "string",
          "json": "string"
        }
      }
      ...
    ]
  },
  "applicationConfigs": {"string": {
        "configBlocks": [
          {
            "id": "string",
            "content": {
              "keyValues": {"string": "string"
                ...},
              "text": "string",
              "json": "string"
            }
          }
          ...
        ]
      }
    ...},
  "ldapGroups": ["string", ...],
  "enableSSO": true|false
}

--query-isolation-options (object)

Configurations for Query Isolation.

maxQueries -> (integer)

Maximum Concurrent Isolated Queries. If not provided, 0 will be applied. The 0 value means the query isoltaion functionality will be disabled.

maxNodesPerQuery -> (integer)

Maximum Nodes Per Isolated Query. If not provided, 0 will be applied. The 0 value means the query isoltaion functionality will be disabled.

Shorthand Syntax:

maxQueries=integer,maxNodesPerQuery=integer

JSON Syntax:

{
  "maxQueries": integer,
  "maxNodesPerQuery": integer
}

--tags (array)

Tags associated with the resources.

Shorthand Syntax:

key=string,value=string ... (separate items with spaces)

JSON Syntax:

[
  {
    "key": "string",
    "value": "string"
  }
  ...
]

--resource-pool (string)

The Resource Pool of the Virtual Warehouse.

Form Factors: private

--platform-jwt-auth | --no-platform-jwt-auth (boolean)

Value of ‘true’ automatically configures the Virtual Warehouse to support JWTs issues by the CDP JWT token provider. Value of ‘false’ does not enable JWT auth on the Virtual Warehouse. If this field is not specified, it defaults to ‘false’.

--impala-query-log | --no-impala-query-log (boolean)

Denotes whether the Virtual Warehouse has the Impala Query Log enabled or not.

Form Factors: public

--ebs-llap-spill-gb (integer)

Provides EBS gp3 volume as temporary storage space for Hive LLAP cache, and improves query performance. Configurable only at Virtual Warehouse creation. Using EBS volumes incurs additional costs.

--hive-server-ha-mode (string)

Set Hive Server High Availability mode in Private Cloud: * DISABLED (default) - Disables Hive Server high availability. * ACTIVE_PASSIVE - Runs two Hive Server instances, one active and one passive. Hive session failover is not supported in this setup.

Form Factors: private

--cli-input-json (string)

Performs service operation based on the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values.

--generate-cli-skeleton (boolean)

Prints a sample input JSON to standard output. Note the specified operation is not run if this argument is specified. The sample input can be used as an argument for --cli-input-json.

Output¶

vwId -> (string)

ID of the new Virtual Warehouse.

Form Factors¶

public, private

create-vw¶

Description¶

Synopsis¶

Options¶

Output¶

Form Factors¶

Table of Contents

Previous topic

Next topic