EC2 Scripts

The primary (and only) EC2 script is */ec2/uw-ec2.py. Here we describe it more in-depth.

Tagging Conventions

Upon launch, the script assigns the tags cluster and master to spot requests, instances, and EBS volumes. cluster is the cluster's name, while master is a boolean (True or False) indicating whether or not the resource is the master's.

For n slave machines, if n is 4, 8, 16, 32, 64, or 128, the cluster names are cloud, cld, cw, cx, cy, cz respectively. For all other values, the cluster name is c<n>x. For example, c10x for 10 slaves.

When init is used, the script assigns a name tag to the instances and EBS volumes. The master has <cluster-name>0, while slaves have <cluster-name>i for i ranging from 1 to n. For example, the cluster cw is tagged with cw0 for the master, and cw1, cw2, ..., cw16 for the slaves.

Searching for Instances

To find instances in the EC2 console, enable the cluster, master, and name tags:

Alternatively, directly filter for them using "Search Instances".

Manual Setup

Here we detail how to manually obtain the results that are automated by the script.

Creating a Key Pair (create-kp)

1. Go to the EC2 console and ensure that you are in us-west-2.

2. Select Key Pairs in the sidebar.

3. Create and save the .pem key to your local machine.

Note: We recommend using uwbench as the key name and saving it to */ec2/uwbench.pem. These are the defaults assumed by uw-ec2.py.

Creating a Security Group (create-sg)

1. Go to the EC2 console and select Security Groups.

2. Click on "Create Security Group" and choose inbound settings as pictured:

Note: Like with the key pair, uwbench is the default name assumed by uw-ec2.py.

Creating a cluster (launch)

We'll go over how to launch spot instances: on-demand differs by one check box (see below).

First, select Spot Requests from the sidebar and click "Request Spot Instances".

Master

1. Select an AMI by searching for "uw-master" or by its AMI image ID (see AMI_MASTER in uw-ec2.py).
   

[--master-ami specifies the master's AMI image.]

2. Select "All generations" and choose m1.xlarge as an instance.
   

[-t specifies the instance type.]

3. Now fill in some relevant options:
   

• 1. "Number of instances" is just 1 for the master.
• 2. Check "Request Spot Instances" to get spot instances.
  • 2a. For spot instances, specify a maximum price ($ per hour per instance).
  • 2b. Also specify the launch group---this should be the cluster's name!
• 3. Select us-west-2c for the availability zone.

[-p specifies the spot price, -z specifies the availability zone.]

4. Decide whether you want the master's volume to be deleted automatically upon termination (see also Terminating EC2).
   

[--persist-master-vol is equivalent to unticking the check box.]

5. Assign the appropriate tags, following Tagging Conventions, so that uw-ec2.py works correctly when/if you ever need it.

Note: If this is a spot request, this will NOT tag the instance that is created! So you must do that manually too. Furthermore, this does not tag the associated EBS volume.

6. Assign the appropriate security group.
   

[-g or --security-group specifies the security group name]

7. Double check the configuration details and launch. Be sure to select the correct key pair!
   
   
   [-k or --key-pair specifies the key-pair to associate instances with]

Wait until the spot instances have all started up. (Either monitor the dashboard, Instances, or filter for "Open" requests in Spot Requests.)