EC2 Benchmarking

To run the ec2 load tests, you will need the following on your local machine:

• dynomite
• python 2.5 or 2.6 (earlier may work)
• boto 1.5 or better
• optional: nosetests

You will need an active AWS account, and access to an AMI that includes:

• erlang R12B-2 or better
• python 2.5 or 2.6
• ruby
• rake
• thrift

The test file is found in pylibs/test/ec2. It may be run on its own, or via nosetests. The following environment variables can be set to control test behavior and set ec2 connection parameters:

AWS_KEY                 -- your amazon key 
AWS_SECRET_KEY          -- your amazon secret key
AWS_SSH_KEY             -- your amazon ssh key name
AWS_SSH_KEY_PATH        -- path on disk to ssh key file

EC2_AMI                 -- your AMI name. AMI must have erlang R12B-1 or 
                           better, ruby, rake, python 2.5.1 or better, and
                           thrift installed
EC2_INSTANCE_TYPE       -- type of EC2 instances to start
EC2_INSTANCES           -- number of EC2 instances to start
EC2_RUN_TIME            -- length of time to run the load test
EC2_CLIENTS_PER_HOST    -- number of clients per instance
EC2_GET_THRESHOLD       -- If 99.9% of gets are not faster than this # of
                           milliseconds, the test fails
EC2_PUT_THRESHOLD       -- If 99.9% of puts are not faster than this # of
                           milliseconds, the test fails
EC2_DYNOMITE_BUILD_DIR  -- Directory to upload to and build in, on each
                           ec2 instance.
EC2_DYNOMITE_ARGS       -- Extra args to pass to dynomite start script
EC2_DYNOMITE_STORAGE    -- Dynomite storage module to use
EC2_LOAD_SCRIPT_ARGS    -- Extra args for load_thrift script
EC2_RAKE_ARGS           -- Extra args for rake (when building dynamo on each
                           ec2 instance)
EC2_SEPARATE_CLIENT     -- (flag) If true, run clients on separate instances
                           from servers. Default is to run both client and
                           server on same instance.

Command line options, when run as a script (not available under nosetests):

  -h, --help            show this help message and exit
  --aws-key=AWS_KEY, --key=AWS_KEY
                        AWS key (the short one)
  --aws-secret-key=AWS_SECRET_KEY, --secret-key=AWS_SECRET_KEY
                        AWS secret key (the long one)
  --aws-ssh-key=AWS_SSH_KEY, --ssh-key=AWS_SSH_KEY
                        Name of AWS ssh key pair to use
  --aws-ssh-key-path=AWS_SSH_KEY_PATH, --ssh-key-path=AWS_SSH_KEY_PATH
                        Path on disk to AWS ssh private key file associated
                        with chosen ssh key pair name
  --ec2-ami=EC2_AMI, --ami=EC2_AMI
                        ID of the AMI to use when starting ec2 instances
  --ec2-type=EC2_TYPE, --type=EC2_TYPE
                        Type of instances to start (default: m1.small)
  --ec2-instances=EC2_INSTANCES, --instances=EC2_INSTANCES
                        Number of instances to start (default: 4)
  --ec2-run-time=EC2_RUN_TIME, --run-time=EC2_RUN_TIME
                        Number of seconds to run the test (default: 300)
  --ec2-clients=EC2_CLIENTS, --clients=EC2_CLIENTS
                        Number of client threads per host (default: 10)
  --get-threshold=GET_THRESHOLD
                        If 99.9% of gets are not faster than this # of
                        milliseconds, the test fails (default: 300)
  --put-threshold=PUT_THRESHOLD
                        If 99.9% of puts are not faster than this # of
                        milliseconds, the test fails (default: 300)
  --dynomite-build-dir=DYN_DIR, --dynomite-dir=DYN_DIR
                        Directory to upload to and build in, on each ec2
                        instance
  --dynomite-storage=DYNOMITE_STORAGE
                        Dynomite storage module to use
  --dynomite-args=DYNOMITE_ARGS
                        Extra args to pass to dynomite start script on each
                        node
  --load-args=LOAD_ARGS
                        Extra args to pass to load script on each node
  --rake-args=RAKE_ARGS
                        Extra args to pass to rake when building dynomite on
                        each node
  --separate-client     Run clients on separate instances from servers

Example and recent results:

python test/ec2/ec2_load_test.py --ec2-ami=xxx --type=c1.medium --run-time=600

[... lots of progress output snipped ... ]

gets: 23502 puts: 23502 collisions: 0
get avg: 14.1821250.3ms median: 7.9438690.3ms 99.9: 228.2829280.3ms
put avg: 19.9633930.3ms median: 11.6100310.3ms 99.9: 191.6468140.3ms
gets:
  10% <   1.790ms
  20% <   2.918ms
  30% <   4.905ms
  40% <   6.386ms
  50% <   7.941ms
  60% <  10.555ms
  70% <  13.275ms
  80% <  18.824ms
  90% <  29.427ms
 100% < 3685.608ms
puts:
  10% <   3.440ms
  20% <   4.979ms
  30% <   6.438ms
  40% <   8.214ms
  50% <  11.607ms
  60% <  15.936ms
  70% <  22.062ms
  80% <  31.926ms
  90% <  47.970ms
 100% < 3778.776ms