Skip to content

EC2 Scripts

Jump to bottom
Young edited this page May 2, 2014 · 33 revisions

The primary (and only) EC2 script is */ec2/uw-ec2.py. Here we describe it more in-depth.

Tagging Conventions

Upon launch, the script assigns the tags cluster and master to spot requests, instances, and EBS volumes. cluster is the cluster's name, while master is a boolean (True or False) indicating whether or not the resource is the master's.

For n slave machines, if n is 4, 8, 16, 32, 64, or 128, the cluster names are cloud, cld, cw, cx, cy, cz respectively. For all other values, the cluster name is c<n>x. For example, c10x for 10 slaves.

When init is used, the script assigns a name tag to the instances and EBS volumes. The master has <cluster-name>0, while slaves have <cluster-name>i for i ranging from 1 to n. For example, the cluster cw is tagged with cw0 for the master, and cw1, cw2, ..., cw16 for the slaves.

Searching for Instances

To find instances in the EC2 console, enable the cluster, master, and name tags: "Show Tags"

Alternatively, directly filter for them using "Search Instances".

Manual Setup

Here we detail how to manually obtain the results that are automated by the script.

Creating a Key Pair (create-kp)

  1. Go to the EC2 console and ensure that you are in us-west-2.

  2. Select Key Pairs in the sidebar.

  3. Create and save the .pem key to your local machine.

Note: We recommend using uwbench as the key name and saving it to */ec2/uwbench.pem. These are the defaults assumed by uw-ec2.py.

Creating a Security Group (create-sg)

  1. Go to the EC2 console and select Security Groups.

  2. Click on "Create Security Group" and choose incoming settings as pictured: "Security Group"

Note: Like with the key pair, uwbench is the default name assumed by uw-ec2.py.

Creating a cluster (launch)

We'll go over how to launch spot instances: on-demand differs by one check box (see below).

First, select Spot Requests from the sidebar and click "Request Spot Instances".

Master

  1. Select an AMI by searching for "uw-master" or the AMI image ID (see AMI_MASTER in uw-ec2.py). "Select AMI"

  2. Follow the illustrated steps below:

Note: Tag names are critical for uw-ec2.py to work correctly! The cluster names for 4, 8, 16, 32, 64, and 128 slaves are cloud, cld, cw, cx, cy, cz. These are special because they are what we experimented with. All other numbers of slaves have the name c<slave>x (e.g., c10x for 10 slaves). The master's name must be cluster-name0 (e.g., cw0) while all slaves' names are cluster-name (e.g., cw).

  1. Wait until the spot instances have all started up. (Either monitor the dashboard, Instances, or filter for "Open" requests in Spot Requests.)

  2. Once all requests are "Active", tag the spot instances using ???. This tags the instances using the tag names of the spot requests.

  3. Run */ec2/uw-ec2.py -n <num_slaves> init, where <num_slaves> is the number of slave machines in the cluster.

Our Results

Running Experiments

Repo Structure

Clone this wiki locally