Current revision as of 13:42, 15 January 2012

This is a first draft for an AccUnit TestRunner.

Details

(Gathered mainly as results of the discussion between Josef and Paul on Jan 13th, 2012)

TO BE ORDERED AND REVIEWED

Reponsibilites and Input/Output

• TestManager
  • Responsibility: Orchestrates the parts described below
  • Responsibilites in particular:
    • A2: listening to TestCollector and maintaing the list of test related info in the repository
    • B0: instantiates the TestRunner for each new test run
    • B1: providing a list of tests to be run by the TestRunner
      • some sources: all tests from the repository, selection from a test tree, test according to the current selection of the IDE, a prioritizer component, selection by tags, ...)
    • B3: creating an instance for the current TestRun (actually before starting)
    • B2: starting the TestRunner (as reaction to some user action)
    • B4: listening to the outputs of the TestRunner and maintaining the current TestRun object
    • B5: adding the current TestRun object to the list of test runs in the repository
  • TO BE CLARIFIED: Should this work as a facade?
  • Events:
    • BeforeCollectingTests (maybe better: BeforeScanningProject? - there are also the test-related modules)
    • AfterCollectingTests (maybe better: AfterScanningProject? - reasoning as above)
    • AddedTestRunToRepository(ITestRun)

• TestCollector
  • Responsibility: Collects the relevant information for running tests. (rephrased: In order to run a test, no one has to go back to the VBProject to look up something)
  • Input: Events from a component in VbeProjectManagement or just a list of all modules in the VBProject
  • Output: IObservable<ITest>/Events
    • FoundTestMethod(ITest)
    • LostTestMethod(ITest)
    • FoundTestRelatedMethod(IAttributedMethod)
    • LostTestRelatedMethod(IAttributedMethod)
  • maybe better name: ProjectScanner
    • there are not just the test classes itself, but also the test-related modules
  • TestManager listens to the output and maintains the list of all test related info in the repository

• Repository.AllTests
  • Responsibility: Represents the current structure of the VBProject with respect to tests
  • Flat list (no fixture-tests hierarchy), hierarchies must be derived by UI components
  • Input: Method calls by the TestManager
  • Output: Enumeration/IObservable<ITest>/Events:
    • AddedTest(ITest)
    • RemovedTest(ITest)
    • Cleared

• Repository.TestRelated (NOT IN THE DRAWING)
  • Responsibility: Represents all modules in the VBProject that are just test-related, but not actually test fixtures.
  • global info needed by the TestRunner
  • Input: method calls by the TestManager in reaction to events from TestCollector
  • Output: none
  • The TestManager instantiates the TestRunner for each run with this information

• TestRunner
  • Responsibility: Coordinates the calls to the methods of the test suite (tests and tag hook handler)
  • Note: Its the TestExecutor that actually invokes the methods via COM, not the TestRunner!
  • Input:
    • List of tests (subset of test in Repository.AllTests)
    • List of test related modules (Repository.TestRelated)
  • Output: IObservable<ITestResult>/Events
    • StartingTestRun(bool cancel)
    • StartedTestRun(ITestRun)
    • TagSetChanged(IEnumerable<ITagChangeInfo>)
    • FinishedTestRun(ITestRun)
    • ITestRun steadiliy populated during the test run
  • The calling of tag hook handlers (see below) is also managed by the TestRunner

• TestExecutor
  • Responsibility: Is able to instantiate classes and invoke methods and understands exceptions from failed assertions
• Input:
  • name of the class to instantiate/terminate
  • name of the method to execute (along with parameters in the case of a call for a row test)
  • Don't use ITest to pass the input parameters.
    • That's too much (attributes not needed generally, rowdata needed without name of the row or ignored state)
    • DIFFERENT FROM THE DRAWING
• Output:
  • True: method call succeeded
  • False: an assertion failed (assertion message is provided to the TestRunner)
  • Non-Assert exceptions are passed to the TestRunner as is
    • Future extension point: special handling for exceptions

Conceptual

• No support of FixtureSetup/FixtureTeardown
  • Seldom used, and when, then often misused (violating independency of tests)
  • More global setup/cleanup (as setting up backend for test) via tag-hooks

• Tags applied on module level inherit to all children (test methods and rows in row tests)

• Support for tag hierarchies?
  • path notation: e.g. DatabaseTests/ADO/Oracle

• No guarantee how often a test class will be instantiated during a test run
  • discourage to use class scope (Initialize/Terminate and Fields)

• Tag Hooks
  • Every time a tag appears during a test run, a setup tag-hook is called (if available)
  • Every time a tag vanishes during a test run, a teardown tag-hook is called (if available)
  • Tag hook handler
    • reside in class modules annotated with AccUnit:TestRelated
    • are tagged with AccUnit:Setup("MyTag"), and AccUnit:Teardown("MyTag"), resp.

• Test related modules (AccUnit:TestRelated) can be standard modules or class modules
  • Test related modules with tag hooks (AccUnit:Setup("MyTag"), AccUnit:Teardown("MyTag")) must reside in class modules
    • This helps to avoid global state enduring over the end of a test run (still could be done by copy state to fields in standard module)

Technical

• New assembly AccUnit.Framework
  • No References to COM (SimplyVbUnit/Access/TLI)

• No IFixture
  • By now, no relevant information besides the name
  • The name of the fixture is a property of ITest

• AccUnit:Rollback and AccUnit:ClickingMsgBox implemented as kind of plugins to the structure

Data structures

• IAttributedMethod
  • modulename
  • methodname
  • attributes

• ITest
  • fixturename
  • methodname
  • attributes
  • tags (specialized subset of attributes)
  • rowdata (specialized subset of attributes)

• ITestRun
  • ProjectInfo (Name of the file, version info?, ...)
  • List<ITestResult>
  • State (prepared, running, finished, canceled, aborted)
  • StartedAt
  • EndedAt
  • Summary

• ITagChangeInfo
  • Tag (string)
  • Change (appeared/disappeared)

• ITestResult
  • ITest
  • Status (success, failed, error, ignored) ... error = runtime error; failed = error by asserts
  • Message (??? rowtest: summary or null ?)
  • Rowdata (testresults of rows ... message + status of rows?)
  • StartedAt
  • EndedAt

Open issues

• Test Reactive Extensions in a VBE AddIn