Test Execution

---

Errthquake tests send HTTP request command sequences to your web services for hundreds or even thousands of simultaneous virtual users.

Subscription plans limit the number of simultaneous virtual users in a single test, but multiple tests can run concurrently to achieve a higher overall total.

Execute tests in the Errthquake portal on the Workspace Overview and Test Results pages.

Test Types

• Duration Test: Specify the input file, configuration file, simultaneous user count, and the duration of the test. The test will run for the duration specified. If it uses all test inputs before the requested duration, it will begin reusing inputs.
• Benchmark Test: Specify the input file, configuration file, and simultaneous user count only. The test configuration will be run once for each input.
• Interactive Test: Specify the input file, configuration file, and an initial simultaneous user count. Once the test has started, it will accept updates to the user count while the test is running. All tests can be stopped early, but interactive tests run until explicitly stopped.

Test Parameters

• Test Input: Choose from the uploaded test input CSV files
• Test Configuration: Choose from the uploaded test configuration YAML files
• Test Location: Choose from the available cloud compute locations
• Test Duration (required for duration tests): The duration time for test execution
• Ramp-Up Time (optional for all tests): The time that should be taken to ramp the test up to the simultaneous user count. The ramp-up time is taken within the test duration time rather than prior to the test duration time.
• Simultaneous Users: The number of virtual users simultaneously running the configured test.

Errthquake runs the command sequences in the test configuration file in parallel for the number of simultaneous users requested for a test. Customer subscription plans specify the upper limit for simultaneous users, and the number of entries in a CSV test input also sets a practical upper limit for simultaneous users in a test.

Subscription Plans

• Free Plan: Maximum 40 virtual users
• Developer Plan: Maximum 1000 virtual users

Test Execution Examples

Given this CSV test input:

user1,password1,abbeyroad,The Beatles,Abbey Road,42.58
user2,password2,thriller,Michael Jackson,Thriller,25.71
user3,password3,nightattheopera,Queen,A Night at the Opera,39.23
user4,password4,darksideofthemoon,Pink Floyd,The Dark Side of the Moon,48.99
user5,password5,ziggystartdust,David Bowie,Ziggy Stardust,31.84
user6,password6,somegirls,The Rolling Stones,Some Girls,37.62
user7,password7,hwy61revisited,Bob Dylan,Highway 61 Revisited,29.38
user8,password8,purplerain,Prince,Purple Rain,21.97

Although any number of simultaneous virtual users can be requested, 8 CSV entries means only 8 simultaneous command sequences will be executed.

For these examples, assume the test configuration YAML defines a test sequence that uses 1 minute of clock time.

The examples use durations and virtual user counts that divide evenly, but this is not required in actual test operation.

Duration Test Examples

• 2 simultaneous users, 2 minute test duration

  • user1 and user2 inputs will run in parallel, then user3 and user4 inputs will run in parallel.
  • Given a 1 minute command sequence and 2 simultaneous users, user1 and user2 will finish after the first minute, and user3 and user4 will finish after the second minute. This uses 2 minutes of clock time, so the test ends without running command sequences for the remaining CSV inputs.

• 2 simultaneous users, 4 minute test duration

  • Given 2 simultaneous users, this test will execute test inputs at the same rate as the prior tests: 2 per minute.
  • Given a 1 minute command sequence and 2 simultaneous users, user1 and user2 will finish after the first minute, user3 and user4 will finish after the second minute, and so on. Command sequences will finish for all 8 users after 4 minutes, which matches the requested test duration.

• 2 simultaneous users, 8 minute test duration

  • Given 2 simultaneous users, this test will execute test inputs at the same rate as the prior tests: 2 per minute.
  • Given the 8 minute test duration, the test inputs will begin a second cycle after the fourth minute and complete the second cycle at the eighth minute. Each test input is used twice to fill the requested 8 minute duration.

• 2 simultaneous users, 10 minute test duration

  • Given 2 simultaneous users, this test will execute test inputs at the same rate as the prior tests: 2 per minute.
  • Given the 10 minute test duration, the test inputs will begin a second cycle afer the fourth minute and a third cycle after the eighth minute. After two minutes of the third cycle, the test duration reaches 10 minutes and ends.

• 4 simultaneous users, 4 minute test duration

  • user1, user2, user3, and user4 inputs will run in parallel. user5, user6, user7, and user8 inputs will run in parallel. All test inputs will be used after 2 minutes of clock time.
  • Given the 4 minute test duration, each test input will be used twice during the test.

• 8 simultaneous users, 4 minute test duration

  • All 8 test inputs are used in parallel.
  • Given the 4 minute test duration, each test input will be used four times during the test.

Benchmark Test Examples

Benchmark tests run the test command sequence once for each user in the CSV test input.

For these examples, assume the same command sequnce that uses 1 minute of clock time.

• 2 simultaneous users

  • Given 2 simultaneous users and 8 test input entries, the total test duration will be 4 minutes.

• 4 simultaneous users

  • Given 4 simultaneous users and 8 test input entries, the total test duration will be 2 minutes.

Interactive Test Examples

Interactive tests start with an initial number of simultaneous users and accept adjustments to the number of simultaneous users. They recycle CSV input like duration tests, but only end when the customer explicitly stops the test.

• 2 initial simultaneous users, adjusted to 4 simultaneous users after 1 minute.

  • Command sequnces are executed for user1 and user2 inputs during the first minute.
  • Command sequences for user3, user4, user5, and user6 are executed upon the 4-user adjustment.
  • Command sequences for user7, user8, user1, and user2 are executed when the previous 4 sequences finish.
  • Command sequences continue for 4 simultaneous users until the test is stopped.

• 8 initial simultaneous users, adjusted to 4 simultaneous users after 1 minute

  • Command sequences are executed for all 8 CSV inputs during the first minute.
  • Command sequences for user1, user2, user3, and user4 are executed upon the 4-user adjustment.
  • Command sequences for user5, user6, user7, and user8 are executed when the previous 4 sequences finish.
  • Command sequences continue for 4 simultaneous users until the test is stopped.

Adding Ramp-Up Time to a Test

Ramp-up time is time used to bring a test from 1 virtual user to the requested number of virtual users.

For these examples, assume the same command sequence that uses 1 minute of clock time.

• 8 simultaneous users, 28 second ramp-up

  • Test begins with 1 virtual user and the 28 second ramp-up is divided across the remaining 7 simultaneous users.
  • A new virtual user is added in parallel at 4 second intervals until 7 more users have been added and command sequences are running for 8 simultaneous virtual users.

• 7 simultaneous users, 1 minute ramp-up

  • Test begins with 1 virtual user and the 1 minute ramp-up is divided across the remaining 6 simultaneous users.
  • A new virtual user is added in parallel at 10 second intervals until 6 more users have been added and command sequences are running for 7 simultaneous virtual users.