Benchmarks

A special set of RSpec scripts used to test a running instance of Scholarsphere.

Setup

Detailed setup instructions are available from our documentation.

Bundler will cover the gem dependencies but there are a few that will need to be installed using brew.

bundle install
brew cask install chromedriver
brew install geckodriver

On the server that you intending to test, go to the Rails console and make sure that the FPS user is registered with Devise:

bundle exec rails c production

Create An New User

If this is first time you're running the tests on this system, or the user doesn't exist, create a new one using the configured Selenuim username:

> User.create(email: "#{SELENIUM_USERNAME}@psu.edu", login: "#{SELENIUM_USERNAME}", ldap_available: 1, ldap_last_update: (DateTime.now + 2.years))

Updating An Existing User

If a few days have passed, you'll need to update the user. This prevents Scholarsphere from querying LDAP, which will cause an error with the uploading tests.

> User.where(login: "#{SELENIUM_USERNAME}").first.update(ldap_available: 1, ldap_last_update: DateTime.now)

Environment Variables

There are two users and passwords that must be set for all the tests to run successfully. The Selenium user information can be found in the setup information and the Box credentials are documented here.

The variables can be set in the shell session, or stored in your ~/.bash_profile:

SELENIUM_USERNAME=[fps-user]
export SELENIUM_USERNAME
SELENIUM_PASSWORD=[password]
export SELENIUM_PASSWORD
BOX_USERNAME=[scholarsphere-box-user]
export BOX_USERNAME
BOX_PASSWORD=[password]
export BOX_PASSWORD

Running the Tests

Execute the benchmarks by entering this directory and running the rspec commands. If available on the host, New Relic data can be gathered as well. By default, tests run against the QA server, in a headless state using Chrome.

cd benchmark
bundle exec rspec spec/solr_query_spec.rb
bundle exec rspec spec/local_upload_spec.rb
bundle exec rspec spec/cloud_upload_spec.rb

Test Variations

You can pass a number of different variables to the test in order to hit different endpoints, use other browsers, or run in a head-"full" state to watch the browser interaction. The variables are:

BENCHMARK_URL="[full url of endpoint]"
BENCHMARK_HEADLESS="[true|false]"
BENCHMARK_BROWSER="[chrome|firefox]"

Note: The browser variable only applies to upload tests. The Solr query test uses both Firefox and Chrome.

Example 1: Running in a non-headless state:

BENCHMARK_HEADLESS="false" bundle exec rspec spec/[etc]

Example 2: Test Firefox against another server:

BENCHMARK_URL="https://other.server.edu/" BENCHMARK_BROWSER="firefox" bundle exec rspec spec/[etc]

Logging

There is also a log of activity maintained for the upload tests. You can see the progress of a test, even in a headless state by monitoring the log file. In a terminal window:

tail -f benchmark_info.log

Then, run your tests in another terminal window and refer to the log during the tests.

Reporting

Results from each of the tests. This is the timing reported by RSpec when executing each test.

Date	System	Release	Browser	Solr Queries	Local Upload	Cloud Upload
08/29/18	Production	3.5	chrome	2' 28.4"	3' 42.2"	3' 41.4"
08/29/18	Production	3.5	firefox	2' 32.9"	3' 06.0"	3' 49.2"
08/30/18	QA	a39db9f	chrome	1' 45.7"	7' 29"	5" 43"
08/30/18	QA	a39db9f	firefox	1' 54.1"	7' 42"	5' 37"
08/31/18	QA	3.5	chrome	1' 50.1"	3' 14.6"	4' 11.9"
08/31/18	QA	3.5	firefox	2' 4.8"	3' 10.8"	4' 8.5"
09/05/18	DCE	829bc2e	chrome	1' 53.6	4' 06.6"	3' 27.2"
09/05/18	DCE	829bc2e	firefox	1' 52.0	3' 35.2"	3' 22.1"

Notes

08/30/18 - 08/31/18

QA was running slowly on the 30th because of certain IPV6 firewall rules. When these rules were updated on the 31st, its performance increased significantly.