com.cloudera.director.aws.ec2

Class EC2Provider

java.lang.Object

com.cloudera.director.spi.v1.model.util.AbstractConfigured

com.cloudera.director.spi.v1.provider.util.AbstractResourceProvider<R,T>

com.cloudera.director.spi.v1.provider.util.AbstractInstanceProvider<R,T>

com.cloudera.director.spi.v1.compute.util.AbstractComputeProvider<EC2Instance,EC2InstanceTemplate>

com.cloudera.director.aws.ec2.EC2Provider

All Implemented Interfaces:

com.cloudera.director.spi.v1.compute.ComputeProvider<EC2Instance,EC2InstanceTemplate>, com.cloudera.director.spi.v1.model.Configured, com.cloudera.director.spi.v1.provider.InstanceProvider<EC2Instance,EC2InstanceTemplate>, com.cloudera.director.spi.v1.provider.ResourceProvider<EC2Instance,EC2InstanceTemplate>

public class EC2Provider
extends com.cloudera.director.spi.v1.compute.util.AbstractComputeProvider<EC2Instance,EC2InstanceTemplate>

Compute provider of Amazon EC2 instances.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	EC2Provider.EC2ProviderConfigurationPropertyToken EC2 configuration properties.
protected static class	EC2Provider.SpotAllocationRecord Holds details about the allocation state of a single virtual instance.
protected class	EC2Provider.SpotGroupAllocator Holds state and logic for allocating a group of Spot instances.

Field Summary

Fields

Modifier and Type	Field and Description
protected static List<com.cloudera.director.spi.v1.model.ConfigurationProperty>	CONFIGURATION_PROPERTIES The provider configuration properties.
static int	DEFAULT_SPOT_INSTANCE_PRICE_CHANGE_DURATION_MS The default amount of time to wait, in milliseconds, for a Spot price change when the Spot bid is known to be below the current Spot price.
static int	DEFAULT_SPOT_INSTANCE_REQUEST_DURATION_MS The default spot instance request duration, in milliseconds.
static String	ID The resource provider ID.
static com.cloudera.director.spi.v1.provider.ResourceProviderMetadata	METADATA The resource provider metadata.

Constructor Summary

Constructors

Constructor and Description

EC2Provider(com.cloudera.director.spi.v1.model.Configured configuration, EphemeralDeviceMappings ephemeralDeviceMappings, VirtualizationMappings virtualizationMappings, AWSFilters awsFilters, com.amazonaws.services.ec2.AmazonEC2Client client, com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient identityManagementClient, com.cloudera.director.spi.v1.model.LocalizationContext cloudLocalizationContext) Construct a new provider instance and validate all configurations.

Method Summary

Modifier and Type	Method and Description
void	allocate(EC2InstanceTemplate template, Collection<String> virtualInstanceIds, int minCount)
void	allocateOnDemandInstances(EC2InstanceTemplate template, Collection<String> virtualInstanceIds, int minCount) Atomically allocates multiple regular EC2 instances with the specified identifiers based on a single instance template.
void	allocateSpotInstances(EC2InstanceTemplate template, Collection<String> virtualInstanceIds, int minCount) Atomically allocates multiple EC2 Spot Instances with the specified identifiers based on a single instance template.
protected static com.amazonaws.services.ec2.AmazonEC2Client	configureClient(com.cloudera.director.spi.v1.model.Configured configuration, com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator, com.amazonaws.services.ec2.AmazonEC2Client client, com.cloudera.director.spi.v1.model.LocalizationContext providerLocalizationContext, boolean verify) Configures the specified EC2 client.
protected static com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient	configureIAMClient(com.cloudera.director.spi.v1.model.Configured configuration, com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator, com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient identityManagementClient, com.cloudera.director.spi.v1.model.LocalizationContext providerLocalizationContext, boolean verify) Configures the specified IAM client.
EC2InstanceTemplate	createResourceTemplate(String name, com.cloudera.director.spi.v1.model.Configured configuration, Map<String,String> tags)
void	delete(EC2InstanceTemplate template, Collection<String> virtualInstanceIds)
protected com.cloudera.director.spi.v1.model.Configured	enhanceTemplateConfiguration(String name, com.cloudera.director.spi.v1.model.Configured configuration, com.cloudera.director.spi.v1.model.LocalizationContext templateLocalizationContext)
Collection<EC2Instance>	find(EC2InstanceTemplate template, Collection<String> virtualInstanceIds)
com.amazonaws.services.ec2.AmazonEC2Client	getClient()
AWSFilters	getEC2Filters() Returns the EC2 filters.
EphemeralDeviceMappings	getEphemeralDeviceMappings() Returns the ephemeral device mappings.
com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient	getIdentityManagementClient()
Map<String,com.cloudera.director.spi.v1.model.InstanceState>	getInstanceState(EC2InstanceTemplate template, Collection<String> virtualInstanceIds)
com.cloudera.director.spi.v1.model.ConfigurationValidator	getResourceTemplateConfigurationValidator()
com.cloudera.director.spi.v1.model.Resource.Type	getResourceType()
VirtualizationMappings	getVirtualizationMappings() Returns the virtualization mappings.

Methods inherited from class com.cloudera.director.spi.v1.provider.util.AbstractResourceProvider

getLocalizationContext, getProviderMetadata, getResourceTemplateLocalizationContext, validateResourceTemplateConfiguration

Methods inherited from class com.cloudera.director.spi.v1.model.util.AbstractConfigured

getConfiguration, getConfigurationValue, getConfigurationValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.cloudera.director.spi.v1.provider.ResourceProvider

getLocalizationContext, getProviderMetadata, getResourceTemplateLocalizationContext, validateResourceTemplateConfiguration

Field Detail

CONFIGURATION_PROPERTIES

protected static final List<com.cloudera.director.spi.v1.model.ConfigurationProperty> CONFIGURATION_PROPERTIES

The provider configuration properties.

ID

public static final String ID

The resource provider ID.

METADATA

public static final com.cloudera.director.spi.v1.provider.ResourceProviderMetadata METADATA

The resource provider metadata.

DEFAULT_SPOT_INSTANCE_REQUEST_DURATION_MS

public static final int DEFAULT_SPOT_INSTANCE_REQUEST_DURATION_MS

The default spot instance request duration, in milliseconds.

See Also:

Constant Field Values

DEFAULT_SPOT_INSTANCE_PRICE_CHANGE_DURATION_MS

public static final int DEFAULT_SPOT_INSTANCE_PRICE_CHANGE_DURATION_MS

The default amount of time to wait, in milliseconds, for a Spot price change when the Spot bid is known to be below the current Spot price.

See Also:

Constant Field Values

Constructor Detail

EC2Provider

public EC2Provider(com.cloudera.director.spi.v1.model.Configured configuration,
                   EphemeralDeviceMappings ephemeralDeviceMappings,
                   VirtualizationMappings virtualizationMappings,
                   AWSFilters awsFilters,
                   com.amazonaws.services.ec2.AmazonEC2Client client,
                   com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient identityManagementClient,
                   com.cloudera.director.spi.v1.model.LocalizationContext cloudLocalizationContext)

Construct a new provider instance and validate all configurations.

Parameters:

configuration - the configuration

ephemeralDeviceMappings - the ephemeral device mappings

virtualizationMappings - the virtualization mappings

awsFilters - the AWS filters

client - the EC2 client

identityManagementClient - the IAM client

cloudLocalizationContext - the parent cloud localization context

Throws:

com.cloudera.director.spi.v1.model.exception.InvalidCredentialsException - if the supplied credentials are invalid

com.cloudera.director.spi.v1.model.exception.TransientProviderException - if a transient exception occurs communicating with the provider

com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException - if an unrecoverable exception occurs communicating with the provider

Method Detail

configureClient

protected static com.amazonaws.services.ec2.AmazonEC2Client configureClient(com.cloudera.director.spi.v1.model.Configured configuration,
                                                                            com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator,
                                                                            com.amazonaws.services.ec2.AmazonEC2Client client,
                                                                            com.cloudera.director.spi.v1.model.LocalizationContext providerLocalizationContext,
                                                                            boolean verify)

Configures the specified EC2 client.

Parameters:

configuration - the provider configuration

accumulator - the exception accumulator

client - the EC2 client

providerLocalizationContext - the resource provider localization context

verify - whether to verify the configuration by making an API call

Returns:

the configured client

Throws:

com.cloudera.director.spi.v1.model.exception.InvalidCredentialsException - if the supplied credentials are invalid

com.cloudera.director.spi.v1.model.exception.TransientProviderException - if a transient exception occurs communicating with the provider

com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException - if an unrecoverable exception occurs communicating with the provider

configureIAMClient

protected static com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient configureIAMClient(com.cloudera.director.spi.v1.model.Configured configuration,
                                                                                                             com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator,
                                                                                                             com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient identityManagementClient,
                                                                                                             com.cloudera.director.spi.v1.model.LocalizationContext providerLocalizationContext,
                                                                                                             boolean verify)

Configures the specified IAM client.

Parameters:

configuration - the provider configuration

accumulator - the exception accumulator

identityManagementClient - the IAM client

providerLocalizationContext - the resource provider localization context

verify - whether to verify the configuration by making an API call

Returns:

the configured client

Throws:

com.cloudera.director.spi.v1.model.exception.InvalidCredentialsException - if the supplied credentials are invalid

com.cloudera.director.spi.v1.model.exception.TransientProviderException - if a transient exception occurs communicating with the provider

com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException - if an unrecoverable exception occurs communicating with the provider

getClient

public com.amazonaws.services.ec2.AmazonEC2Client getClient()

getIdentityManagementClient

public com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient getIdentityManagementClient()

getEphemeralDeviceMappings

public EphemeralDeviceMappings getEphemeralDeviceMappings()

Returns the ephemeral device mappings.

Returns:

the ephemeral device mappings

getVirtualizationMappings

public VirtualizationMappings getVirtualizationMappings()

Returns the virtualization mappings.

Returns:

the virtualization mappings

getEC2Filters

public AWSFilters getEC2Filters()

Returns the EC2 filters.

Returns:

the EC2 filters

getResourceTemplateConfigurationValidator

public com.cloudera.director.spi.v1.model.ConfigurationValidator getResourceTemplateConfigurationValidator()

Overrides:

getResourceTemplateConfigurationValidator in class com.cloudera.director.spi.v1.provider.util.AbstractResourceProvider<EC2Instance,EC2InstanceTemplate>

getResourceType

public com.cloudera.director.spi.v1.model.Resource.Type getResourceType()

createResourceTemplate

public EC2InstanceTemplate createResourceTemplate(String name,
                                                  com.cloudera.director.spi.v1.model.Configured configuration,
                                                  Map<String,String> tags)

allocate

public void allocate(EC2InstanceTemplate template,
                     Collection<String> virtualInstanceIds,
                     int minCount)
              throws InterruptedException

Throws:

InterruptedException

find

public Collection<EC2Instance> find(EC2InstanceTemplate template,
                                    Collection<String> virtualInstanceIds)
                             throws InterruptedException

Throws:

InterruptedException

delete

public void delete(EC2InstanceTemplate template,
                   Collection<String> virtualInstanceIds)
            throws InterruptedException

Throws:

InterruptedException

getInstanceState

public Map<String,com.cloudera.director.spi.v1.model.InstanceState> getInstanceState(EC2InstanceTemplate template,
                                                                                     Collection<String> virtualInstanceIds)

enhanceTemplateConfiguration

protected com.cloudera.director.spi.v1.model.Configured enhanceTemplateConfiguration(String name,
                                                                                     com.cloudera.director.spi.v1.model.Configured configuration,
                                                                                     com.cloudera.director.spi.v1.model.LocalizationContext templateLocalizationContext)

Overrides:

enhanceTemplateConfiguration in class com.cloudera.director.spi.v1.provider.util.AbstractResourceProvider<EC2Instance,EC2InstanceTemplate>

allocateOnDemandInstances

public void allocateOnDemandInstances(EC2InstanceTemplate template,
                                      Collection<String> virtualInstanceIds,
                                      int minCount)
                               throws InterruptedException

Atomically allocates multiple regular EC2 instances with the specified identifiers based on a single instance template. If not all the instances can be allocated, the number of instances allocated must be at least the specified minimum or the method must fail cleanly with no billing implications.

Parameters:

template - the instance template

virtualInstanceIds - the unique identifiers for the instances

minCount - the minimum number of instances to allocate if not all resources can be allocated

Throws:

InterruptedException - if the operation is interrupted

allocateSpotInstances

public void allocateSpotInstances(EC2InstanceTemplate template,
                                  Collection<String> virtualInstanceIds,
                                  int minCount)
                           throws InterruptedException

Atomically allocates multiple EC2 Spot Instances with the specified identifiers based on a single instance template. If not all the instances can be allocated, the number of instances allocated must be at least the specified minimum or the method must fail cleanly with no billing implications.

Note: contrary to the contract of the SPI method, there are some cases where despite failing to satisfy the min count there are billing implications, due to non-atomicity of AWS operations. In particular, if we lose connectivity to AWS for an extended period of time after creating Spot instance requests, we will be unable to cancel the requests, and unable to detect that instances have been provisioned. The resulting requests and/or instances may or may not be tagged appropriately, depending on when connectivity was interrupted.

Parameters:

template - the instance template

virtualInstanceIds - the unique identifiers for the instances

minCount - the minimum number of instances to allocate if not all resources can be allocated

Throws:

InterruptedException - if the operation is interrupted