This is a first draft for an AccUnit TestRunner.

480px

Contents 1 Details 1.1 Reponsibilites and Input/Output 1.2 Conceptual 1.3 Technical 1.4 Data structures 1.5 Open issues

Details

(Gathered mainly as results of the discussion between Josef and Paul on Jan 13th, 2012)

TO BE ORDERED AND REVIEWED

Reponsibilites and Input/Output

• TestManager
  • Responsibility: Orchestrates the parts described below
  • Responsibilites in particular:
    • A2: listening to TestCollector and maintaing the list of test related info in the repository
    • B0: instantiates the TestRunner for each new test run
    • B1: providing a list of tests to be run by the TestRunner
      • some sources: all tests from the repository, selection from a test tree, test according to the current selection of the IDE, a prioritizer component, selection by tags, ...)
    • B3: creating an instance for the current TestRun (actually before starting)
    • B2: starting the TestRunner (as reaction to some user action)
    • B4: listening to the outputs of the TestRunner and maintaining the current TestRun object
    • B5: adding the current TestRun object to the list of test runs in the repository
  • TO BE CLARIFIED: Should this work as a facade?
  • Events:
    • BeforeCollectingTests (maybe better: BeforeScanningProject? - there are also the test-related modules)
    • AfterCollectingTests (maybe better: AfterScanningProject? - reasoning as above)
    • AddedTestRunToRepository(ITestRun)

• TestCollector
  • Responsibility: Collects the relevant information for running tests. (rephrased: In order to run a test, no one has to go back to the VBProject to look up something)
  • Input: Events from a component in VbeProjectManagement or just a list of all modules in the VBProject
  • Output: IObservable<ITest>/Events
    • FoundTestMethod(ITest)
    • LostTestMethod(ITest)
    • FoundTestRelatedMethod(IAttributedMethod)
    • LostTestRelatedMethod(IAttributedMethod)
  • maybe better name: ProjectScanner
    • there are not just the test classes itself, but also the test-related modules
  • TestManager listens to the output and maintains the list of all test related info in the repository

• Repository.AllTests
  • Responsibility: Represents the current structure of the VBProject with respect to tests
  • Flat list (no fixture-tests hierarchy), hierarchies must be derived by UI components
  • Input: Method calls by the TestManager
  • Output: Enumeration/IObservable<ITest>/Events:
    • AddedTest(ITest)
    • RemovedTest(ITest)
    • Cleared

• Repository.TestRelated (NOT IN THE DRAWING)
  • Responsibility: Represents all modules in the VBProject that are just test-related, but not actually test fixtures.
  • global info needed by the TestRunner
  • Input: method calls by the TestManager in reaction to events from TestCollector
  • Output: none
  • The TestManager instantiates the TestRunner for each run with this information

• TestRunner
  • Responsibility: Coordinates the calls to the methods of the test suite (tests and tag hook handler)
  • Note: Its the TestExecutor that actually invokes the methods via COM, not the TestRunner!
  • Input:
    • List of tests (subset of test in Repository.AllTests)
    • List of test related modules (Repository.TestRelated)
  • Output: IObservable<ITestResult>/Events
    • StartingTestRun(bool cancel)
    • StartedTestRun(ITestRun)
    • TagSetChanged(IEnumerable<ITagChangeInfo>)
    • FinishedTestRun(ITestRun)
    • ITestRun steadiliy populated during the test run
  • The calling of tag hook handlers (see below) is also managed by the TestRunner

• TestExecutor
  • Responsibility: Is able to instantiate classes and invoke methods and understands exceptions from failed assertions
• Input:
  • name of the class to instantiate/terminate
  • name of the method to execute (along with parameters in the case of a call for a row test)
  • Don't use ITest to pass the input parameters.
    • That's too much (attributes not needed generally, rowdata needed without name of the row or ignored state)
    • DIFFERENT FROM THE DRAWING
• Output:
  • True: method call succeeded
  • False: an assertion failed (assertion message is provided to the TestRunner)
  • Non-Assert exceptions are passed to the TestRunner as is
    • Future extension point: special handling for exceptions

Conceptual

• No support of FixtureSetup/FixtureTeardown
  • Seldom used, and when, then often misused (violating independency of tests)
  • More global setup/cleanup (as setting up backend for test) via tag-hooks

• Tags applied on module level inherit to all children (test methods and rows in row tests)

• Support for tag hierarchies?
  • path notation: e.g. DatabaseTests/ADO/Oracle

• No guarantee how often a test class will be instantiated during a test run
  • discourage to use class scope (Initialize/Terminate and Fields)

• Tag Hooks
  • Every time a tag appears during a test run, a setup tag-hook is called (if available)
  • Every time a tag vanishes during a test run, a teardown tag-hook is called (if available)
  • Tag hook handler
    • reside in class modules annotated with AccUnit:TestRelated
    • are tagged with AccUnit:Setup("MyTag"), and AccUnit:Teardown("MyTag"), resp.

• Test related modules (AccUnit:TestRelated) can be standard modules or class modules
  • Test related modules with tag hooks (AccUnit:Setup("MyTag"), AccUnit:Teardown("MyTag")) must reside in class modules
    • This helps to avoid global state enduring over the end of a test run (still could be done by copy state to fields in standard module)

Technical

• New assembly AccUnit.Framework
  • No References to COM (SimplyVbUnit/Access/TLI)

• No IFixture
  • By now, no relevant information besides the name
  • The name of the fixture is a property of ITest

• AccUnit:Rollback and AccUnit:ClickingMsgBox implemented as kind of plugins to the structure

Data structures

• IAttributedMethod
  • modulename
  • methodname
  • attributes

• ITest
  • fixturename
  • methodname
  • attributes
  • tags (specialized subset of attributes)
  • rowdata (specialized subset of attributes)

• ITestRun
  • ProjectInfo (Name of the file, version info?, ...)
  • List<ITestResult>
  • State (prepared, running, finished, canceled, aborted)
  • StartedAt
  • EndedAt
  • Summary

• ITagChangeInfo
  • Tag (string)
  • Change (appeared/disappeared)

• ITestResult
  • ITest
  • status (success, failed, error, ignored) ... error = runtime error; failed = error by asserts
  • message (??? rowtest: summary or null ?)
  • rowdata (testresults of rows)
  • StartedAt
  • EndedAt
  • Summary ??? ... summary of row tests?

Open issues

• Test Reactive Extensions in a VBE AddIn