com.cloudera.director.aws.ec2

Class EC2Provider.SpotGroupAllocator

java.lang.Object

com.cloudera.director.aws.ec2.EC2Provider.SpotGroupAllocator

Enclosing class:

EC2Provider

protected class EC2Provider.SpotGroupAllocator
extends Object

Holds state and logic for allocating a group of Spot instances. A new instance is required for each allocation request.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	allocatedInstanceCount The number of allocated instances (including reused orphans).
protected int	minCount The minimum number of instances to allocate if not all resources can be allocated.
protected Date	priceChangeDeadlineTime The latest time to wait for a Spot price change when it is known that the current Spot price exceeds the Spot bid.
protected Date	requestExpirationTime The latest time to wait for Spot instance request fulfillment.
protected Map<String,EC2Provider.SpotAllocationRecord>	spotAllocationRecordsByVirtualInstanceId The map from virtual instance IDs to the corresponding Spot allocation records.
protected Map<String,String>	spotInstancesByUntaggedSpotInstanceRequestId The map from untagged Spot instance request IDs to the corresponding Spot instances.
protected EC2InstanceTemplate	template The instance template.
protected Collection<String>	virtualInstanceIds The virtual instance IDs.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SpotGroupAllocator(EC2InstanceTemplate template, Collection<String> virtualInstanceIds, int minCount, Date requestExpirationTime, Date priceChangeDeadline) Creates a Spot group allocator with the specified parameters.

Method Summary

Modifier and Type	Method and Description
protected void	allocate() Allocates Spot instances.
protected void	cancelSpotRequests(com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator) Cancels any outstanding Spot requests (includes discovered orphans and created requests).
protected void	checkForOrphanedInstances() Identifies reusable Spot instances orphaned by a previous call.
protected Set<String>	checkForOrphanedSpotInstanceRequests() Identifies reusable Spot instance requests orphaned by a previous call.
protected String	determineClientToken(Set<String> virtualInstanceIds) Determines the idempotency client token for the specified virtual instance IDs.
protected Set<String>	determineVirtualInstanceIdsNeedingSpotInstanceRequest() Determines which virtual instance IDs require a Spot instance request.
protected String	getErrorMessage(Exception e) Returns the error message for the specified exception.
protected EC2Provider.SpotAllocationRecord	getSpotAllocationRecord(String virtualInstanceId) Returns the Spot allocation record corresponding to the specified virtual instance ID.
protected Map<String,EC2Provider.SpotAllocationRecord>	initializeSpotAllocationRecordMap(Collection<String> virtualInstanceIds) Initializes the map from virtual instance IDs to the corresponding Spot allocation records.
protected com.amazonaws.services.ec2.model.RequestSpotInstancesRequest	newRequestSpotInstancesRequest(Collection<String> virtualInstanceIds, String clientToken) Builds a RequestSpotInstancesRequest for the specified virtual instance IDs.
protected Set<String>	requestSpotInstances(com.amazonaws.services.ec2.model.RequestSpotInstancesRequest requestSpotInstancesRequest) Requests Spot instances, and returns the resulting Spot instance request IDs.
protected boolean	spotInstanceRequestExists(String spotInstanceRequestId) Checks whether the specified Spot instance request is visible, i.e. can be located by a describe Spot instance request call.
protected void	tagSpotInstanceRequest(List<com.amazonaws.services.ec2.model.Tag> userDefinedTags, String spotInstanceRequestId, String virtualInstanceId) Tags an EC2 Spot instance request.
protected void	tagSpotInstanceRequests(Collection<String> spotInstanceRequestIds, Collection<String> virtualInstanceIds) Tags the Spot instance requests with the specified IDs with the corresponding virtual instance IDs.
protected void	tagSpotInstances() Tags provisioned Spot instances.
protected void	terminateSpotInstances(com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator) Terminates any running Spot instances (includes discovered orphans and allocated instances).
protected void	waitForPrivateIpAddresses() Waits for provisioned Spot instances to have a private IP address.
protected void	waitForSpotInstances(Set<String> pendingRequestIds, boolean cancelling) Waits for pending Spot instance requests to be fulfilled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

template

protected final EC2InstanceTemplate template

The instance template.

virtualInstanceIds

protected final Collection<String> virtualInstanceIds

The virtual instance IDs.

minCount

protected final int minCount

The minimum number of instances to allocate if not all resources can be allocated.

requestExpirationTime

protected final Date requestExpirationTime

The latest time to wait for Spot instance request fulfillment.

priceChangeDeadlineTime

protected final Date priceChangeDeadlineTime

The latest time to wait for a Spot price change when it is known that the current Spot price exceeds the Spot bid.

spotAllocationRecordsByVirtualInstanceId

protected final Map<String,EC2Provider.SpotAllocationRecord> spotAllocationRecordsByVirtualInstanceId

The map from virtual instance IDs to the corresponding Spot allocation records.

spotInstancesByUntaggedSpotInstanceRequestId

protected final Map<String,String> spotInstancesByUntaggedSpotInstanceRequestId

The map from untagged Spot instance request IDs to the corresponding Spot instances.

allocatedInstanceCount

protected int allocatedInstanceCount

The number of allocated instances (including reused orphans).

Constructor Detail

SpotGroupAllocator

protected SpotGroupAllocator(EC2InstanceTemplate template,
                             Collection<String> virtualInstanceIds,
                             int minCount,
                             Date requestExpirationTime,
                             Date priceChangeDeadline)

Creates a Spot group allocator with the specified parameters.

Parameters:

template - the instance template

virtualInstanceIds - the virtual instance IDs

minCount - the minimum number of instances to allocate if not all resources can be allocated

requestExpirationTime - the latest time to wait for Spot instance request fulfillment

priceChangeDeadline - the latest time to wait for a Spot price change when it is known that the current Spot price exceeds the Spot bid

Method Detail

initializeSpotAllocationRecordMap

protected Map<String,EC2Provider.SpotAllocationRecord> initializeSpotAllocationRecordMap(Collection<String> virtualInstanceIds)

Initializes the map from virtual instance IDs to the corresponding Spot allocation records.

Parameters:

virtualInstanceIds - the virtual instance IDs

Returns:

the map from virtual instance IDs to the corresponding Spot allocation records

getSpotAllocationRecord

protected EC2Provider.SpotAllocationRecord getSpotAllocationRecord(String virtualInstanceId)

Returns the Spot allocation record corresponding to the specified virtual instance ID.

Parameters:

virtualInstanceId - the virtual instance ID

Returns:

the Spot allocation record corresponding to the specified virtual instance ID

allocate

protected void allocate()
                 throws InterruptedException

Allocates Spot instances.

Throws:

InterruptedException - if the operation is interrupted

checkForOrphanedInstances

protected void checkForOrphanedInstances()
                                  throws InterruptedException

Identifies reusable Spot instances orphaned by a previous call.

Note: because of AWS's eventual consistency policies, we are not guaranteed to be able to detect all orphans here, but we make a best-faith effort.

Throws:

InterruptedException - if operation is interrupted

checkForOrphanedSpotInstanceRequests

protected Set<String> checkForOrphanedSpotInstanceRequests()

Identifies reusable Spot instance requests orphaned by a previous call.

Returns:

the reusable Spot instance requests orphaned by a previous call

determineVirtualInstanceIdsNeedingSpotInstanceRequest

protected Set<String> determineVirtualInstanceIdsNeedingSpotInstanceRequest()

Determines which virtual instance IDs require a Spot instance request.

Returns:

the virtual instance IDs which require a Spot instance request

determineClientToken

protected String determineClientToken(Set<String> virtualInstanceIds)

Determines the idempotency client token for the specified virtual instance IDs.

Parameters:

virtualInstanceIds - the virtual instance IDs

Returns:

the idempotency token

newRequestSpotInstancesRequest

protected com.amazonaws.services.ec2.model.RequestSpotInstancesRequest newRequestSpotInstancesRequest(Collection<String> virtualInstanceIds,
                                                                                                      String clientToken)

Builds a RequestSpotInstancesRequest for the specified virtual instance IDs.

Parameters:

virtualInstanceIds - the virtual instance IDs

clientToken - the idempotency client token

Returns:

the RequestSpotInstancesRequest

requestSpotInstances

protected Set<String> requestSpotInstances(com.amazonaws.services.ec2.model.RequestSpotInstancesRequest requestSpotInstancesRequest)

Requests Spot instances, and returns the resulting Spot instance request IDs.

Parameters:

requestSpotInstancesRequest - the RequestSpotInstancesRequest

Returns:

the resulting Spot instance request IDs

tagSpotInstanceRequests

protected void tagSpotInstanceRequests(Collection<String> spotInstanceRequestIds,
                                       Collection<String> virtualInstanceIds)
                                throws InterruptedException

Tags the Spot instance requests with the specified IDs with the corresponding virtual instance IDs.

Parameters:

spotInstanceRequestIds - the Spot instance request IDs

virtualInstanceIds - the corresponding virtual instance IDs

Throws:

InterruptedException - if the operation is interrupted

tagSpotInstanceRequest

protected void tagSpotInstanceRequest(List<com.amazonaws.services.ec2.model.Tag> userDefinedTags,
                                      String spotInstanceRequestId,
                                      String virtualInstanceId)
                               throws InterruptedException

Tags an EC2 Spot instance request.

Parameters:

userDefinedTags - the user-defined tags

spotInstanceRequestId - the Spot instance request ID

virtualInstanceId - the virtual instance ID

Throws:

InterruptedException - if the operation is interrupted

spotInstanceRequestExists

protected boolean spotInstanceRequestExists(String spotInstanceRequestId)

Checks whether the specified Spot instance request is visible, i.e. can be located by a describe Spot instance request call.

Parameters:

spotInstanceRequestId - the Spot instance request ID

Returns:

whether the Spot instance request is visible, i.e. can be located by a describe Spot instance request call

waitForSpotInstances

protected void waitForSpotInstances(Set<String> pendingRequestIds,
                                    boolean cancelling)
                             throws InterruptedException

Waits for pending Spot instance requests to be fulfilled.

Parameters:

pendingRequestIds - the pending Spot instance request IDs

cancelling - whether we are in the process of cancelling

Throws:

InterruptedException - if the operation is interrupted

tagSpotInstances

protected void tagSpotInstances()
                         throws InterruptedException

Tags provisioned Spot instances. Expects that the instances already exists or are in the process of being created.

Throws:

InterruptedException - if the operation is interrupted

waitForPrivateIpAddresses

protected void waitForPrivateIpAddresses()
                                  throws InterruptedException

Waits for provisioned Spot instances to have a private IP address.

Throws:

InterruptedException - if the operation is interrupted

terminateSpotInstances

protected void terminateSpotInstances(com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator)
                               throws InterruptedException

Terminates any running Spot instances (includes discovered orphans and allocated instances).

Parameters:

accumulator - the exception condition accumulator

Throws:

InterruptedException - if operation is interrupted

cancelSpotRequests

protected void cancelSpotRequests(com.cloudera.director.spi.v1.model.exception.PluginExceptionConditionAccumulator accumulator)
                           throws InterruptedException

Cancels any outstanding Spot requests (includes discovered orphans and created requests).

Parameters:

accumulator - the exception condition accumulator

Throws:

InterruptedException - if the operation is interrupted

getErrorMessage

protected String getErrorMessage(Exception e)

Returns the error message for the specified exception.

Parameters:

e - the exception

Returns:

the error message for the specified exception