Implement CKTester

[Garry Yao]

CKTester is our internal universal testing swarm, it should be capable of running tests for different projects with different testing tools easily while remain simplicity, below are some basic ideas of the project based on the previous effort on 'FCKtest'.

Test Cell

A test cell is basically a plain html file reside in individual project's testing source, in which test suites/test cases are written. It'll be executed inside an iframe/popup of the fort so full page life cycle ( from page load to page destroy ) is guaranteed when performing the test.

Each cell has only key responsibility of reporting the sucess/failure/time of it's execution to the fort, this is done by cross-frame communication with parent/opener window.

The cell's testing environment, including testing target, runtime libraries, etc. will be all injected by the fort, initiatively or passively, the only required effort in the cell is to include the bootstrap file.

<script type="text/javascript" src="{CKTester_ROOT}/bootstrap.js"></script>

A cell could be running in two modes:

1. Managed Mode : It's the default mode when it's running in a batch of cells by the forge, where the cell is running in an iframe.
2. Standalone Mode : This mode is adapted when a cell is running separately, on the opposite of 'Managed Mode', the cell will be running inside a popup window, and it doesn't require to report to the fort.

Profile

A tests profile is a js file contains information about running a bunch of test cells. It a function declaration which return a configuration object, contains:

1. Cell variables : An hash of variable names and their values which are often embeded in profile paths to indicate location.
2. Cells list : A list of key/values:
   • Key : The test cell file path, where all the paths are relative to the runner;
   • Value : An array of tags.
3. Cell resolvers : An array of function which were used to manipulate each cell's attributes at run-time, e.g. Inject necessary dependencies for cells when encounter certain tags. Note that The fort already has a few default resolvers, usages including : converting the cell path to tags; calculate resource path for each cell, etc.

A profile will basically looks like:

	CKTester.profile = function ()
	{
		return
		{
			variables : { 'CKEDITOR_ROOT' : 'path-to-editor-root' },
			cells : [
				[ 'editor/tt/unit/plugins/htmldataprocessor/1' ], 		// Implicit tag declaration.
				[ '3210.html' , [ 'tt', 'unit', 'htmldataprocessor'] ]  // Explicit tag declaration.
			]
			cellResolvers : [ ... ]
		};
	}

It's important to note that tag plays an important role as the following indicators:

1. Environments( dependencies ) of a cell, e.g. if the cell has tags 'unit' and 'editor', it denote a unit test running by YUITest, along with the CKEditor testing stage is required to run the test;
2. Categories of the cell, when it's been run in a runner with a specific criteria. E.g. A url criteria as '​http://t/runner?cells=tt,plugins' will run only the tests of category 'ticket' and 'plugins' ( with 'ticket' and 'plugins' tags );
3. Any reasonable denotation.

Fort

The Test Fort is a bridge between the runner tools/framework ( e.g. Selenium ) underneath, and the testing resources, so it's above the level of any TC or TS. It's resided in CKTester project.

The fort is responsible for delegating the running of all the tests to the under-laying tools/framework and collecting the testing result neutrally. It consist of a UI to interact with and providing report.

The runner is required to be running with a specified profile, from which it will register all the defined tests cells in it. Upon requesting by a specific criteria on running, it filtering down the registered cells to only execute the satisfied ones.

Generally, before runner execute each test cell, an extensible cell resolver is working out the required/requested resources of a specific cell by analysing the tags, guarantee the cell always has all the dependencies it should receive. These resources are typically composed of :

• The testing target
• The testing library
• Any resources dedicated to the cell.

Testing Request

A request is simply a page load of the fort, which form the begin of the testing life cycle.

1. The request must use url param to specify the location of profile file relative of the fort.
2. The request url could optionally consist of a criteria which is a list of tags or path to filter the test cells to run, it could be

specified in various ways:

• A comma separated param in URL;
• Specified through UI of the test runner page;

Some example testing requests look like ( I've remove the 'profile' param for clear understanding ):

• Running only the ticket tests of unit test type: ​http://t/fort.html?cells=tt,unit
• Running all the stable tests : ​http://t/fort.html?cells=stable
• Running two specific tests : ​http://t/fort.html?cells=../editor/tt/unit/htmldataprocessor/1,../editor/dd/selenium/visual/1
  Note that since it's not convenient to specify a cell path within URL, it's a shortcut that open the cell url directly would have it redirect to correct url back to the forge.

To-Be Implemented Features

1. Auto sync URL with criteria input UI;
2. Stop/Skip running tests capability;
3. Separate file for customized( local ) profile variables definition to avoid frequent SVN merge of the default profile file.
4. Fort APIs for communication with a test server.

[Garry Yao]

Little adjustments.

[Garry Yao]

Update description according to implementation:

1. Rename 'runner' to 'fort' for disambiguation;
2. Adding 'running mode' concepts of cell;
3. Simplify 'criteria' pattern;
4. Adding 'cell shortcut' concept;

[Garry Yao]

Initial implementation commited with [4088] and [4089].

[Garry Yao]

Update with [4098], make it possible to run individual cell as soon as the cell starts.

[Garry Yao]

Adding some suggested features from Tobiasz after talking with him.

[Garry Yao]

There're several changes has been decided after a dicussion with Fred:

1. Fixing the following critical bugs
   • Fort is loading unrelated resources even before running cells;
2. Adding the following pragmatic features:
   • Be able to run a specific cell directly without relaying on profile, the cell will be self-bootstrapping;
   • Easily tag definition in cell html file;
   • Introducing dev tool for generate testing profile file from scanning test directory.

The changes are effective with [4100].

[Garry Yao]

After a dicussion with Fred, the profile file's cell path could be further simplified by making it a convention that the runner folder will reside in the root of test branch.

[ '../tt/4227/4227',[ 'editor','unit','all' ]  ]
// The path change to relative to the profile file( test root).
[ 'tt/4227/4227',[ 'editor','unit','all' ]  ]

[Garry Yao]

The changes regarding the above comments are commited with [4215] and [4216].

[Garry Yao]

Typos: [4125] and [4126].

[Garry Yao]

Commit the following changes with [4128] and [4129];

1. 'CKTester' object was renamed to 'CKTESTER';
2. Move generic testing functions from CKEditor project test file( core/test.js ) to CKTester.
3. Adding profile template file for CKEditor test branch.
4. Make SVN ignoring 'variables.js' file.

[Garry Yao]

Thanks for Tobias pointing out that the [generic testing API file ​http://svn.fckeditor.net/CKTester/runners/yuitest/test.js] is missing, compensate with [4132].

[Garry Yao]

Thanks for Tobias again for firing a bug about cell path calculation in IE:

Fixing it with [4142] and [4143].

[Garry Yao]

Fixing a bug where asking for absolute path in cell page cause error in all IEs with [4158].