EC2 Scripts

The primary (and only) EC2 script is */ec2/uw-ec2.py. Here we describe it more in-depth.

Tagging Conventions

Upon launch, the script assigns the tags cluster and master to spot requests, instances, and EBS volumes. cluster is the cluster's name, while master is a boolean (True or False) indicating whether or not the resource is the master's.

For n slave machines, if n is 4, 8, 16, 32, 64, or 128, the cluster names are cloud, cld, cw, cx, cy, cz respectively. For all other values, the cluster name is c<n>x. For example, c10x for 10 slaves.

When init is used, the script assigns a name tag to the instances and EBS volumes. The master has <cluster-name>0, while slaves have <cluster-name>i for i ranging from 1 to n. For example, the cluster cw is tagged with cw0 for the master, and cw1, cw2, ..., cw16 for the slaves.

Searching for Instances

To find instances in the EC2 console, enable the cluster, master, and name tags:

Alternatively, directly filter for them using "Search Instances".

Manual Setup

Here we detail how to manually obtain the results that are automated by the script. As well, uw-ec2.py's options are highlighted in [...]s.

Creating a Key Pair (create-kp)

1. Go to the EC2 console and ensure that you are in us-west-2.

2. Select Key Pairs in the sidebar.

3. Create and save the .pem key to your local machine.

Note: We recommend using uwbench as the key name and saving it to */ec2/uwbench.pem. These are the defaults assumed by uw-ec2.py.

Creating a Security Group (create-sg)

1. Go to the EC2 console and select Security Groups.

2. Click on "Create Security Group" and choose inbound settings as pictured:

Note: Like with the key pair, uwbench is the default name assumed by uw-ec2.py.

Creating a cluster (launch)

We'll go over how to launch spot instances: on-demand differs by one check box (see step 3).

First, select Spot Requests from the sidebar and click "Request Spot Instances".

Master

1. Select an AMI by searching for "uw-master" or by its AMI image ID (see AMI_MASTER in uw-ec2.py).
   

[--master-ami specifies the master's AMI image.]

2. Select "All generations" and choose m1.xlarge as an instance.
   

[-t specifies the instance type.]

3. Now fill in some relevant options:

• 1. "Number of instances" is just 1 for the master.
• 2. Check "Request Spot Instances" to get spot instances.
  • 2a. For spot instances, specify a maximum price ($ per hour per instance).
  • 2b. Also specify the launch group---this should be the cluster's name!
• 3. Select us-west-2c for the availability zone.

[-p specifies the spot price, -z specifies the availability zone.]

4. Decide whether you want the master's volume to be deleted automatically upon termination (also see Terminating EC2).
   

[--persist-master-vol is equivalent to unticking the check box.]

Note: For spot instances, you may want to uncheck the box so that you don't lose your log files/results when your master gets terminated due to price spikes.

5. Assign the appropriate tags, following Tagging Conventions, so that uw-ec2.py works correctly when/if you ever need it.

Note: If this is a spot request, this will NOT tag the instance that is created! So you must do that manually too. Furthermore, this does not tag the associated EBS volume.

6. Assign the appropriate security group.
   

[-g or --security-group specifies the security group name.]

7. Double check the configuration details and launch. Be sure to select the correct key pair!
   
   

[-k or --key-pair specifies the key-pair to associate instances with.]

Now monitor the dashboard, Instances, or filter for "Open" requests in Spot Requests.) to wait for the spot (or on-demand) instances to start.

Slaves

Starting the slave instances is quite to starting the master.

1. Select an AMI by searching for "uw-slave" or by its AMI image ID (see AMI_SLAVE in uw-ec2.py).
   

[--slave-ami specifies the slave's AMI image.]

2. Same as the master.

3. The options for the slave differ slightly:
   

• 1. "Number of instances" is the number of slaves to launch.
• 2. Same as the master. Ensure that launch group matches the master's!
• 3. Same as the master. Ensure availability zone matches the master's!

[-n specifies the number of slave machines.]

4. Same as the master. Note that uw-ec2.py has no option to disable "Delete on Termination" for slaves' EBS volumes.

5. Like the master, assign the appropriate tags following Tagging Conventions.

Note: Unlike the master, there is no easy way to tag each instance's name separately. However, cluster and master is all that's needed for uw-ec2.py to work correctly. Again, this does not tag EBS volumes (nor instances if the tags are for spot requests.)

6. Same as the master.

7. Same as the master. Review and launch.