tests: new section

Closes curl#233

SUMMARY.md

@@ -265,4 +265,17 @@
     * [Content Encoding](internals/content-encoding.md)
     * [Structs](internals/structs.md)
     * [Resolving host names](internals/resolving.md)
+    * [Tests](internals/tests.md)
+        * [Test file format](internals/tests/file-format.md)
+        * [Build tests](internals/tests/build.md)
+        * [Run tests](internals/tests/run.md)
+        * [Debug builds](internals/tests/debug.md)
+        * [Test servers](internals/tests/servers.md)
+        * [curl tests](internals/tests/curl.md)
+        * [libcurl tests](internals/tests/libcurl.md)
+        * [Unit tests](internals/tests/unit.md)
+        * [Valgrind](internals/tests/valgrind.md)
+        * [Continuous Integration](internals/tests/ci.md)
+        * [Autobuilds](internals/tests/autobuilds.md)
+        * [Torture](internals/tests/torture.md)
 * [Index](bookindex.md)

internals.md

@@ -19,3 +19,4 @@ library do something new.
  * [Content Encoding](internals/content-encoding.md)
  * [Structs](internals/structs.md)
  * [Resolving host names](internals/resolving.md)
+ * [Tests](internals/tests.md)

internals/tests.md

@@ -0,0 +1,28 @@
+# Tests
+
+The curl test suite is a fundamental cornerstone in our development
+process. It helps us verify that existing functionality is still there like
+before, and we use it to check that new functionality behaves as expected.
+
+With every bugfix and new feature, we ideally also create one or more test
+cases.
+
+The test suite is custom made and tailored specifically for our own purposes
+to allow us to test curl from every possible angle that we think we need. It
+does not rely on any third party test frameworks.
+
+The tests are meant to be possible to build and run on virtually all platforms
+available.
+
+* [Test file format](tests/file-format.md)
+* [Build tests](tests/build.md)
+* [Run tests](tests/run.md)
+* [Debug builds](tests/debug.md)
+* [Test servers](tests/servers.md)
+* [curl tests](tests/curl.md)
+* [libcurl tests](tests/libcurl.md)
+* [Unit tests](tests/unit.md)
+* [Valgrind](tests/valgrind.md)
+* [Continuous Integration](tests/ci.md)
+* [Autobuilds](tests/autobuilds.md)
+* [Torture](tests/torture.md)

internals/tests/autobuilds.md

@@ -0,0 +1,25 @@
+# Autobuilds
+
+Volunteering individuals run the **autobuilds**. This is a script that runs
+automatically that:
+
+- checks out the latest code from the git repository
+- builds everything
+- runs the test suite
+- send the full log over email to the curl server
+
+As they are then run on different platforms with different build options, they
+offer an extra dimension of feedback on curl build health.
+
+## Check status
+
+All logs are parsed, managed and displayed on [the curl
+site](https://curl.se/dev/builds.html).
+
+## Legacy
+
+We started the autobuild system in 2003, a decade before [CI jobs](ci.md)
+started becoming a serious alternative.
+
+Now, the autobuilds are more of a legacy system as we are moving more and more
+into a world with CI and more direct and earlier feedback.

internals/tests/build.md

@@ -0,0 +1,13 @@
+# Build tests
+
+Before you can run any tests you need to build curl but also build the test
+suite and its associated tools and servers.
+
+Most conveniently, you can just build and run them all by issuing `make test`
+in the build directory root but if you want to work more on tests or perhaps
+even debug one, you may want to jump into the `tests` directory and work from
+within that. Build it all and run test `144` like this:
+
+    cd tests
+    make
+    ./runtests.pl 144

internals/tests/ci.md

@@ -0,0 +1,28 @@
+# Continuous Integration (CI)
+
+For every pull request submitted to the curl project on GitHub and for every
+commit pushed to the master branch in the git repository, a vast amount of
+virtual machines fire up, check out that code from git, build it with
+different options and run the test suite and make sure that everything is
+working fine.
+
+We run CI jobs on several different operation systems, including Linux, macOS,
+Windows, Solaris and FreeBSD.
+
+We run jobs that build and test many different (combinations of) backends.
+
+We have jobs that use different ways of building: autotools, cmake,
+winbuild, Visual Studio, etc.
+
+We verify that the distribution tarball works.
+
+We run source code analyzers.
+
+## Failing builds
+
+Unfortunately, due to the complexity of everything involved we often have one
+or two CI jobs that seemingly are stuck "permafailing", that seems to be
+failing the jobs on a permanent basis.
+
+We work hard to make them not, but it is a tough job and we often see red
+builds even for changes that should otherwise be "all green".

internals/tests/curl.md

@@ -0,0 +1,16 @@
+# curl tests
+
+The standard test in the suite is the "curl test". They all invoke a curl
+command line and verifies that everything it sends, gets back and returns are
+exactly as expected. Any mismatch and the test is considered a fail and the
+script shows details about the error.
+
+What the test features in the `<client><command>` section is what is used in
+the command line, verbatim.
+
+The `tests/log/commands.log` is handy to look at after a run, as it contains
+the full command line that was run in the test.
+
+If you want to make a test that does not invoke the curl command line tool,
+then you should consider the [libcurl tests](libcurl.md) or [unit
+tests](unit.md) instead.

internals/tests/debug.md

@@ -0,0 +1,38 @@
+# Debug builds
+
+When we speak of *debug builds*, we usually refer to curl builds that done
+with debug code and symbols still present. We strongly recommend you do this
+if you want to work with curl development as it makes it easier to test and
+debug.
+
+You make a debug build using configure like this:
+
+    ./configure --enable-debug
+
+Debug-builds make it possible to run individual test cases with gdb with
+runtests.pl, which is handy - especially for example if you can make it crash
+somewhere as then gdb can catch it and show you exactly where it happens etc.
+
+Debug-builds are also built a little different than regular *release builds*
+in that they contain some snippets of code that makes curl easier to test. For
+example it allows the test suite to override the random number generator so
+that testing for values that otherwise are random actually work. Also, the
+unit tests only work on debug builds.
+
+## Memdebug
+
+Debug builds also enable the *memdebug* internal memory tracking and debugging
+system.
+
+When switched on, the memdebug system outputs detailed information about a lot
+of memory-related options into a logfile, so that it can be analyzed and
+verified after the fact. Verified that all memory was freed, all files were
+closed and so on.
+
+This is a poor-man's version of valgrind but does not at all compare with its
+features. It is however fairly portable and low-impact.
+
+In a debug build, the memdebug system is enabled by curl if the
+`CURL_MEMDEBUG` environment variable set to a file name, which is used for the
+log. The test suite sets this variable for us (see `tests/memdump`) and
+verifies it after each test run, if present.

internals/tests/file-format.md

@@ -0,0 +1,40 @@
+# Test file format
+
+Each curl test is designed in a single text file using an XML-like
+format.
+
+Each test file is called `tests/data/testNUM` where `NUM` is a unique
+numerical test identifier. Each test has to use its own dedicated number. The
+number has no meaning other than identifying the test.
+
+The test file defines exactly what command line or tool to run, what test
+servers to invoke and how they should respond, exactly what protocol exchange
+that should happen, what output and return code to expect and much more.
+
+Everything is written within their dedicated tags like this when the name is
+set:
+
+    <name>
+    HTTP with host name written backwards
+    </name>
+
+So far, the [test file format
+specification](https://github.com/curl/curl/blob/master/tests/FILEFORMAT.md)
+is only available in the git repository.
+
+## keywords
+
+Every test has one or more `<keywords>` set in the top of the file. They are
+meant to be "tags" that identify features and protocols that are tested by
+this test case. `runtests.pl` can be made to run only tests that match (or do
+not match) such keywords.
+
+## Preprocessed
+
+Under the hood, each test input file is "preprocessed" at startup by
+runtests.pl. This means that variables, macros and keywords are expanded and a
+temporary version of the file is stored in `tests/log/tesyNUM` - and *that*
+file is then used by all the test servers etc.
+
+This processing allows the test format to offer features like `%repeat` to
+create really big test files without bloating the input files correspondingly.

internals/tests/libcurl.md

@@ -0,0 +1,12 @@
+# libcurl tests
+
+A libcurl test is a stand-alone C program that uses the public libcurl API to
+do something.
+
+Apart from that, everything else is tested, verified and checked the same way
+[curl tests](curl.md) are.
+
+Since these C programs are usually built and run on a plethora of different
+platforms, considerations might need to be taken.
+
+All libcurl test programs are kept in `tests/libtests`

internals/tests/run.md

@@ -0,0 +1,33 @@
+# Run tests
+
+The main script that runs tests is called `tests/runtests.pl` and some of its
+more useful features are:
+
+## Run a range of tests
+
+Run test 1 to 27:
+
+    ./runtests.pl 1 to 27
+
+Run all tests marked as `SFTP`:
+
+    ./runtests.pl SFTP
+
+Run all tests **not** marked `FTP`:
+
+    ./runtests.pl '!FTP'
+
+## Run a specific test with gdb
+
+    ./runtests.pl -g 144
+
+It starts up gdb, you can set break-points etc and then type `run` and off it
+goes and performs the entire thing through the debugger.
+
+## Run a specific test without valgrind
+
+The test suite will use valgrind by default if it finds it, which is an
+excellent way to find problems but it also makes the test run much
+slower. Sometimes you want to do it faster:
+
+    ./runtests.pl -n 144

internals/tests/servers.md

@@ -0,0 +1,17 @@
+# Test servers
+
+A large portion of the curl test suit actually runs curl command lines that
+interact with servers that are started on the local machine for testing
+purposes only during the test, and that are shut down again at the end of the
+test round.
+
+The test servers are custom servers written for this purpose that speak HTTP,
+FTP, IMAP, POP3, SMTP, TFTP, MQTT, SOCKS proxies and more.
+
+All test servers are controlled via the test file: which servers that each
+test case needs to have running to work, what they should return and how they
+are supposed to act for each test.
+
+The test servers typically log their actions in dedicated files in
+`tests/log`, and they can be useful to check out if your test does not act the
+way you want.

internals/tests/torture.md

@@ -0,0 +1,41 @@
+# Torture tests
+
+When curl is built [debug enabled](debug.md), it offers a special kind of
+testing. The tests we call **torture** tests. Do not worry, it is not quite as
+grim as it may sound.
+
+They verify that libcurl and curl exit paths work without any crash or memory
+leak happening,
+
+The torture tests work like this:
+
+ - run the single test as-is first
+ - count the number of invoked *fallible* functions
+ - rerun the test once for every falling function call
+ - make each fallible function call return error, one by one
+ - verify that there is no leak or crash
+ - continue until all fallible functions have been made to fail
+
+This test can take a seriously long time. I advice you to switch off
+[valgrind](valgrind.md) when trying this out.
+
+## Rerun a specific failure
+
+If a single test fails, `runtests.pl` will identify exactly which "round" that
+triggered the problem and by using the `-t` as shown, you can run a command
+line that when invoked *only* fails that particular fallible function.
+
+## Shallow
+
+To make this way of testing a little more practical, the test suite also
+provides a `--shallow` option. This lets the user set a maximum number of
+fallible functions to fail per test case. If there are more invokes to fail
+than is set with this value, the script will randomly select which ones to
+fail.
+
+As a special feature, as randomizing things in tests can be uncomfortable, the
+script will use a random seed based on year + month, so it will remain the
+same for each calendar month. Convenient, as if you rerun the same test with
+the same `--shallow` value it will run the same random tests.
+
+You can force a different seed with runtests' `--seed` option.

internals/tests/unit.md

@@ -0,0 +1,15 @@
+# Unit tests
+
+**Unit tests only work on [debug builds](debug.md).**
+
+Unit tests are tests that use functions that are libcurl internal and
+therefore not part of any public API, headers or external documentation.
+
+If the internal function you want to test is made `static`, they should
+instead be set `UNITTEST` - which then makes debug builds not use static for
+them and they then become accessible to test from unit tests.
+
+We provide a set of convenience functions and macros for unit tests to make it
+quick and easy to write them.
+
+All unit test programs are kept in `tests/unit`

internals/tests/valgrind.md

@@ -0,0 +1,11 @@
+# Valgrind
+
+Valgrind is a popular and powerful tool for debugging programs and especially
+their use and abuse of memory.
+
+`runtests.pl` automatically detects if valgrind is installed on your system
+and will by default run tests using valgrind if found. You can pass `-n` to
+runtests to disable the use of valgrind.
+
+Valgrind makes execution much slower, but it is an excellent tool to find
+memory leaks and use of uninitialized memory.