What kind of stdlib unittest integration is there
That's an easy answer… none!
There are plans to add some collector support for collecting TestCase? classes and converting them dynamically to functest compliant tests, but that isn't finished yet.
This question usually comes from one of two angles.
- I want to integrate something from functest, or something written in functest to an existing unittest setup. Or the other around.
- The python stdlib unittest framework is the standard python testing framework and all should bend their testing needs to work with it or you are a horrible evil person.
The main reason there is no unittest integration is because too many of the features in functest conflict with the way unittest requires you to author tests. The pdb debugging, simple tracebacks for failures, regular assertions, etc, are all features that the boilerplate heavy unittest ( and for that matter all xUnit implementation ) require. When we do add support for collecting and running these tests there will have to be a lot of magic to dynamically swap out the unittest pieces and replace them with normal asserts and calls to functest's framework.
Ah, but you say "Well I could add all these functest features to unittest by hacking it up". Yes, you could strap them to unittest by replacing all the different unittest classes that are available for hacking, i mean customizing 
, and you could even write a collector on top of them that could mimic the functest collector. But none of your changes would be compatible with anyone else's hacked up unittest framework. So I ask, "What is the point of standardizing on a framework that has zero interop?".
Functest provides various points of customization and extensions that are actually compatible between other customizations and extensions others may have done. Functest even ensures that tests written using one set of customizations will work when swapping out certain other customizations.
How much of functest can be customized?
Out of the box windmill provides you with a very flexible collector and setup/teardown system. A lot of what other people do with extensions has builtin facilities in functest.
Beyond that functest allows you to customize the runner and the reports. Future versions of functest will allow for custom collectors as well.
Most importantly, all of these customizations interop across other functest test repositories.
Certain pieces of functest cannot be overwritten, and this is intentional. Although hard to work with, unittest does allow you to, in some way, overwrite everything.
- Functest doesn't allow you to change the manor in which test functions are executed.
- Functest doesn't allow you to change the manor in which setup/teardown functions are executed.
- Functest doesn't allow you to change the manor in which dependencies are calculated for a collected test function.
- Functest doesn't allow you to customize the trigger for signaling a test failure (asserts)
1 and 2 are simple. Test functions get executed as test() and setup/teardown's get executed as function(obj). You cannot add arguments to be passed or change the arguments that are passed. You can use decorators or setup/teardown to add attributes to test modules/functions, some of which may correspond to standard customizations in functest.
3 means you can't change which setup/teardowns get called in dependency chain. A collector will return a test module to run ( and all of it's sub-modules ) and a dependency chain, which is an ordered list of modules that are defined in a module hierarchy above the one being executed. A collector can omit a module, or remove an attibute it doesn't want functest to run, but you can't change the way in which the dependency chain is iterated over and run.
4 means you MUST use regular assertions for comparisons to cause explicit test failures. Test failures are triggered by python exceptions, which illustrates one of the main limitations of functest and also illustrates the largest difference between functional testing and unit testing.
Functional tests should organized in to chunks of ordered lines of code in functions, and the order matters. It doesn't matter if line 4 or line 46 in a single test function fails, if those lines were kept in a single function and not broken into two functions then line 46 depends on line 4 and nothing useful comes from debugging line 46 or line 4 is failing.
Or in simpler terms, in functional testing units should stop after a failure and continue to the next unit, in unit testing the separation of functions in to unittests isn't necessarily ordered and test units should not fail in place but continue on because future failures in the unit may be unique and unrelated to previous failures.
