[TOC]
The chrome.test
extension API is a limited testing framework implemented as
an extension API. It is primarily used in order to provide testing
functionality used in writing extension API tests by exercising an API directly
in JS. See also writing extension tests.
All tests must have some limited C++ portion (in order to kick off and drive
the test). In the most basic form, this C++ test only needs to load the
extension and wait for the response from the testing framework. This is easily
accomplished through either ExtensionApiTest::RunExtensionTest()
or
ExtensionApiTest::RunExtensionSubtest()
(and their variants).
ExtensionApiTest::RunExtensionTest()
loads a test extension and then waits
for the result from the testing framework.
ExtensionApiTest::RunExtensionSubtest()
loads a test extension, navigates to
a subpage of that extension, and then waits for the result from the testing
framework.
More advanced tests may require more synchronization between the C++ and the JS
(e.g., to set up or verify state on the C++ side before continuing on the JS
side). To do this, leverage either extensions::ResultCatcher
, which waits
for the next result from the testing framework (sent using either
chrome.test.notifyPass()
, chrome.test.notifyFail()
, or waiting for the
result of chrome.test.runTests()
, which calls notifyPass()
or
notifyFail()
automatically), or extensions::ExtensionTestMessageListener
,
which waits for a message sent by chrome.test.sendMessage()
. See
writing extension tests for more information.
The API provides a variety of different methods, used for different purposes.
chrome.test.notifyPass()
and chrome.test.notifyFail()
are used in order to
pass the result of running the JS test to the C++. This is the result that
RunExtensionTest()
, RunExtensionSubtest()
, and the ResultCatcher
wait on.
In order to explicitly pass this result, call chrome.test.notifyPass()
or
chrome.test.notifyFail()
.
chrome.tabs.create(() => {
<Verify state>
chrome.test.notifyPass();
});
notifyPass()
and notifyFail()
may also be implicitly called by the testing
API, as described in the sections below.
chrome.test.runTests()
is used to run a sequence of individual, smaller JS
tests, and then passes the result to the browser by automatically calling
chrome.test.notifyPass()
or chrome.test.notifyFail()
. notifyPass()
will
be called if and only if all individual tests pass; notifyFail()
will be
called if any test fails. A test may fail if an assertion fails, if there is
an unexpected runtime error, or if chrome.test.fail()
is called explicitly.
chrome.test.runTests()
takes an array of functions, and runs them serially.
This means that these functions may be independent, or may implicitly rely on
one another. The output of running these individual tests is printed through
console.log()
s, which enables tracing how far a test suite progresses.
Each individual test function passed to runTests()
will execute, and then
wait for that specific function to pass or fail. Passing is indicated by
calling chrome.test.succeed()
within each test function (not
chrome.test.notifyPass()
, which will automatically indicate the entire JS
test passes, and may mask failures - see also the Do's And Don't's. Failure
is indicated by calling chrome.test.fail()
, a failed assertion, or through
an unexpected runtime error or API error (indicated in
chrome.runtime.lastError
). Each test function must signal success or failure;
otherwise the test will hang (and eventually timeout).
A sample test suite may look like this.
let tabId;
chrome.test.runTests([
function createNewTab() {
chrome.tabs.create({url: 'http://example.com'}, (tab) => {
chrome.test.assertNoLastError();
<verify |tab| properties>
tabId = tab.id;
chrome.test.succeed();
});
},
function queryTab() {
chrome.tabs.query({url: 'http://example.com'}, (tabs) => {
chrome.test.assertNoLastError();
<verify |tabs|>
chrome.test.succeed();
});
},
function removeTab() {
chrome.tabs.remove(tabId, () => {
chrome.test.assertNoLastError();
chrome.test.succeed();
});
},
]);
The chrome.test API
provides a number of basic assertion methods.
Asserts that the given condition is true, printing out the optional error message if it is not.
Asserts that the given condition is false, printing out the optional error message if it is not.
Asserts that the provided value matches the expected value. If expected
is
an object, this will perform a deep-equals check (i.e., verifying that two
objects are logically equivalent, rather than have the same address). If the
expected value does not match the actual value, this will print out the
expected and actual values (through JSON.stringify()
for objects).
Asserts that chrome.runtime.lastError
is undefined, printing out the error
otherwise.
Asserts that chrome.runtime.lastError.message
is equivalent to
expectedError
, printing out the expected and actual errors otherwise.
Asserts that executing fn
with the context object of self
(if defined) and
the specified arguments
throws a runtime error, which is then validated
against expectedError
. expectedError
may be either a string (which must
match exactly) or a RegExp
.
Important Notes:
callbackPass()
andcallbackFail()
should absolutely only be used inside ofchrome.test.runTests()
.callbackPass()
andcallbackFail()
are no longer as useful as they were, and sometimes result in less readable, more surprising code. Think twice (or even thrice!) before using them.
callbackPass()
and callbackFail()
were primarily used when tests needed to
wait on two non-deterministic functions. For instance, consider the
(contrived) following test:
chrome.test.runTests([
function step1() {
chrome.tabs.create({url: 'http://example.com'}, () => {
<verify state>
});
chrome.storage.local.set({foo: 'bar'}, () => {
<verify state>
});
// Somehow end the test when both the tab creation and storage set have
// finished.
},
function step2() {
...
}
]);
Here, we want to have step1()
finish after both the new tab has been created
and a storage value has been set (again, this is admittedly contrived). There
is no hard guarantee about which function will finish first, so putting a
chrome.test.succeed() call in either may result in succeeding and continuing to
the next step too early. Putting a chrome.test.succeed()
call in both will
result in badness (see the Do's And Don't's).
callbackPass()
lets the testing infrastructure handle this. callbackPass()
takes a function to invoke as an argument, and wraps it. When callbackPass()
is used (or any other function that increments an internal callback counter),
the testing infrastructure keeps track of that callback and waits for all
outstanding callbacks to be called before automatically continuing to the next
test. A version of this test using callbackPass()
may look like this.
chrome.test.runTests([
function step1() {
chrome.tabs.create({url: 'http://example.com'}, callbackPass(() => {
<verify state>
}));
chrome.storage.local.set({foo: 'bar'}, callbackPass(() => {
<verify state>
}));
},
function step2() {
...
}
]);
Now, chrome.test.succeed()
will be internally called by the testing
infrastructure once both callbacks have been invoked.
callbackFail()
behaves similarly to callbackPass()
, except that it
predominantly takes an expected error message that will be set in
chrome.runtime.lastError
(though it takes a secondary argument of an optional
function to invoke).
The most obvious advantage to using callbackPass()
and callbackFail()
is
that is eliminates the need for more complex callback management. In addition
to keeping track of the callbacks, callbackPass() also automatically checks
that there was no API error raised in the callback, obviating the need for a
call to chrome.test.assertNoLastError()
. This can lead to more succinct
code.
callbackPass()
and callbackFail()
can both lead to much less readable and
obvious code. Extension APIs (and JavaScript in general) are already heavily
asynchronous, and use of callbackPass()
and callbackFail()
can lead to even
more confusion about when a test will finish, or which order items will be
executed in. Coupled with the internal callback counter, hazards around
multiple calls to chrome.test.succeed()
or chrome.test.fail()
, and others,
and it is often much more clear to avoid using callbackPass()
and
callbackFail()
. By comparison, having a single call to
chrome.test.succeed()
in an extension test function makes it more clear when
success is met, and what the final operation is. Additionally, there's no good
way to sequence multiple operations - there is only a single internal "callback
counter", which all callback-based utility methods share. Finally, having
multiple ways to signal a test is "done" leads to frequent developer and
reviewer confusion and misuse (e.g., calling chrome.test.succeed()
within a
callbackPass()
- see the Do's And Don't's).
In the example above, it's unclear why the storage.set()
call and
tabs.create()
call need to be within the same test. They could instead be
two separate test functions, passed serially in the array to
chrome.test.runTests()
.
chrome.test.runTests([
function createTab() {
chrome.tabs.create({url: 'http://example.com'}, callbackPass(() => {
<verify state>
chrome.test.succeed();
}));
},
function initializeStorage() {
chrome.storage.local.set({foo: 'bar'}, callbackPass(() => {
<verify state>
chrome.test.succeed();
}));
},
function nextStep() {
...
}
]);
The chrome.test
callback mechanism was created before Promises were
available as a language feature in JavaScript. Promises can provide similar
guarantees while providing more explicit direction to the reader about when a
test will finish.
chrome.test.runTests([
function step1() {
let tabPromise = new Promise((resolve) => {
chrome.tabs.create({url: 'http://example.com'}, () => {
<verify state>
resolve();
}));
});
let storagePromise = new Promise((resolve) => {
chrome.storage.local.set({foo: 'bar'}, () => {
<verify state>
resolve();
}));
});
Promise.all([tabPromise, storagePromise]).then(() => {
chrome.test.succeed();
});
},
function step2() {
...
}
]);
This solution is somewhat more verbose, but makes it more clear on which criteria the test is waiting, and can allow for different "sequences" of waiting. This will also be much more streamlined when test APIs themselves can return promises.
The chrome.test.runTests()
allows for asynchronous functions and using the
await keyword. In conjunction with the Extension system's asynchronous APIs,
this can lead to reasonably concise and readable code, such as the example
below:
const tab =
await new Promise(resolve => chrome.tabs.create({url: url}, resolve));
<verify state>
This, too, will be even more readable with Promise-based APIs.
chrome.test.listenOnce()
and chrome.test.listenForever()
are utility
functions used when waiting on different events. Like callbackPass()
and
callbackFail()
above, they use the test API's internal callback counter,
allowing the test to finish automatically once all callbacks have been invoked.
Naturally, they share all the same disadvantages as well.
listenOnce()
waits for the event to be invoked a single time, and then
removes the listener and reduces the internal callback counter. Calling
listenForever()
adds the listener and returns a function to be invoked at any
point; invoking this function will remove the listener and decrement the
callback counter.
chrome.test.getConfig()
retrieves the current configuration of the test
setup. This may be necessary for a variety of reasons - a common one is
needing the port on which the test server is running in order to construct
URLs, as below.
chrome.test.getConfig((config) => {
let url = `http://example.com:${config.testServer.port}/simple.html`;
createTab(url);
});
chrome.test.sendMessage()
is used to communicate with the C++ side of the
browser test, allowing us to force synchronicity if necessary. It should be
used with ExtensionTestMessageListener on the C++ side.
// test.cc:
IN_PROC_BROWSER_TEST_F(...) {
LoadExtension(...);
GURL url = GetASpecialURL();
ExtensionTestMessageListener listener("clicked", /*will_reply=*/true);
ClickAction();
ASSERT_TRUE(listener.WaitUntilSatisfied());
listener.Reply(url.spec());
...
}
// extension_script.js:
chrome.action.onClicked.addListener(() => {
chrome.test.sendMessage('clicked', (specialUrl) => {
useSpecialUrl(specialUrl);
});
});
One of the advantages of having the chrome.test.runTests()
method and
infrastructure is that we can write small, unit-test style tests without
needing to pay the cost of an extra browser test execution (expensive) for
each individual test case. Because of this, we can write very targeted,
easy-to-consume, understandable test cases, rather than a single behemoth test
case.
assertTrue()
and assertFalse()
provide the least information in the logs
("expected true, found false"-style messages). Assert-style methods like
assertEq()
, assertLastError()
, and assertNoLastError()
can provide much
more detail (such as the actual value for assertEq()
, or the error for
assertNoLastError()
).
Many tests were written years and years ago. Modern JS practices, such as
let
and const
, arrow functions, template literals, and more, generally
increase readability. Don't feel shy about using them just because other
examples don't. (The caveat to this is that these tests should not use
anything that isn't approved for Chromium JS use,
unless the use of it is explicitly necessary for the test.)
chrome.test.notifyPass()
(or chrome.test.notifyFail()
) will finish the
entire suite. It should not be used with chrome.test.runTests()
. Instead,
use chrome.test.succeed().
If the callback counter is incremented anywhere in a test function, the test
infrastructure will automatically invoke chrome.test.succeed()
when the
counter reaches zero. Calling chrome.test.succeed()
in addition to
callbackPass()
, callbackFail()
, listenOnce()
, or listenForever()
can
result in unpredictable behavior, or masking failures.
Consider the following code:
chrome.foo.asyncFunction(..., callbackPass(() => {
chrome.foo.asyncFunction2(..., () => {
// Extra stuff
});
});
In this case, the callback counter will wait for the call to asyncFunction()
to complete and execute the inner function, and will then immediately end the
test case. The callback from asyncFunction2()
will be invoked after the test
case is over, which can lead to flakiness or false-negatives.
If you have to have nested functions, each should use callbackPass()
or
callbackFail()
. Better yet, migrate away from callbackPass()
and
callbackFail()
with these alternatives.
While it's possible to have multiple notifyPass()
calls in a single C++
browser test (by using multiple calls to RunExtensionTest()
or using multiple
ResultCatchers
), and by extension possible to mix runTests()
with
notifyPass()
with sendMessage()
with everything else, it's generally a bad
idea. It makes the flow of control incredibly difficult to parse for author,
reviewer, and future reader, and frequently leads to subtle bugs being missed.
Instead, if a complex chain of steps is needed, use either
chrome.test.runTests()
or chrome.test.sendMessage()
(if C++ coordination is
needed).
It can be tempting to set up a sequence of a 20-step sequence in
chrome.test.runTests()
, but this makes it very difficult to understand the
total flow, harder to debug, and increases the risk of timing out (from simply
doing too much). If a test relies on multiple steps in runTests()
, have a
dedicated browser test for those related steps, and restrict them to a
reasonable number. If a test has a series of unrelated steps, keep them small
and targeted (in good unit testing behavior).