Module talk:QuickTest

Errors in UI on modules like Module:Yesno

{{Edit request}} This module causes error messages to appear in the Scribunto UI navbox for modules implemented as anonymous functions, for example see Module:Yesno. To fix this, please change the Lua code line from...

local testFunction = m['runTests']

to...

local testFunction = ((type(m) == 'table') and m['runTests']) or nil

—RP88 (talk) 02:14, 31 December 2014 (UTC)

 Done -- Rillke^(q?) 03:41, 31 December 2014 (UTC)

Thanks! —RP88 (talk) 03:56, 31 December 2014 (UTC)

QuickTests not working in resource modules that do not contain any function, but that only return data

It is impossible to define a QuickTest in modules that don't return an array of functions (plus optional data with specific keys), but only data.

Currently the code uses this test:

local testFunction = ( (type(m) == 'table') and m['runTests'] ) or nil -- the "or nil" is purely esthetic, it just replaces "false" by "nil" (wrong in my opinion)

However, it should preferably search for the runTest member in the metatable and not as a direct members. This way we could map a runTests function in those data modules by setting the "runTests" key in the metatable without adding any standard member in the array.

local testFunction = (type(m) == 'table') and (getmetatable(m).quickTests or m['runTests'])

In my opinion, using the metatable for such usage should be highly prefered. In the code of modules, instead of defining

function p:runTests()...end

(which not only define the function with a parameter "p", but also expose the function as a member of the table p returned by the module when it is loaded, with its key equal to the defined function name, so making it really public and polluting the returned resources when the goal of the module is only to return data), we would define

local function autoTests(p)...end -- you can name this function as you want, because it is purely local in the module and not exposed

setmetable(p, 'quickTests', autoTests) -- reference the function name you've created just above under the 'quickTests' metatable key

This is a very simple modification to do to the very few existing modules (only 7 have them with succesful results!) using QuickTests.

For now those modules are only tested by adding a separate module in a "/testcases" subpage (the most used way). And there's no way to unify both methods.

Note that "testcases" are generally not intended to be "quickTests", they may expose problems that are still to solve later, but that do not affect the validity of "quickTests", whose role is just to verify that these modules are functional for its active and documetned use cases and do not fail on load ("quickTests" should work in most locales but could be restricted to run only in the English locale, if this limitation is documented in the module):

These limitations in modules are generally listed in the documentation page exposing unsolved problems or things not completely tested, except in "/testcases" for demonstrating these problems and limitations or to test experimental features or additional features still in development.

These extension features are NOT to be tested in "quickTests" that should only test STABLE features (that all further modifications in the module should preserve, notably assertions), and it does not really mattter if there are some "testcases" failing (developed experimentaly in another separate module), as long as all "quickTests" pass !

For this reason, the "testcases" module should not be loaded by quickTests (but if they are loaded and the load fails, the quickTests should not fail, it should fail only if the testcases loaded correctly and jsut some testcases for STABLE features are then tested in the quickTests and fail). However the situation should be the reverse: "testcases" modules should load the main module, and will first run its quickTests (and expose its result) before continuning with other extended tests.

The various "testcases" in the "/testcases" module should not be executed at all as long as "quickTests" built directly in the main module are failing (the priority for module developers being to make sure that all basic STABLE features are still working, before testing and exposing problematic cases or experimental features and extensions (whose code may in fact be in the "/sandbox" version of the main module).

This also allows a "/testcases" module to compare the result of the current stable version of the main module, with a modified version in "/sandbox, or with the results of some old templates that the extension should also deprecate and replace in the module.

So in summary: "quickTests" do not have not exactly the same goal as "testcases". It is important to distinguish stable features (quickTests) and experimental features and extensions (or even corrections) in developement. And frequently we need several separate "/testcases" modules, or several other wiki pages (preferably in the "Commons:" namespace or in a "User:xxx/subpage" for the developer, as they are not really part of "/doc" intended for the documentation of stable featrures only), due to some limitations of MediaWiki in terms of resources (cannot load more than 20 languages, limits of sizes during template expansions, limits of sizes for the detailed results), and also because "/testcases" frequently have to perform tests by loading the modules differently or using various versions of costly templates. The "/testcases" module may have its own "/doc" page with links to these various testcases results pages.

-- verdy_p (talk) 14:11, 15 April 2015 (UTC)

Yeah, the purpose of these tests is primarily smoke testing, although this may include unit testing of features deemed to be stable and I furthermore agree that these tests should be executed prior to executing other tests on /testcases, which in turn may test for features not yet fully working. I like the idea you brought up about the comparison of the sandbox page with the current module. If you need any help in getting things settled in this direction, please let me know. Concept bought :) -- Rillke^(q?) 14:17, 17 April 2015 (UTC)

Thanks for agreeing the concept. I see that you have also accepted my proposed change to use the metatable to reference the quicktests.

However, I do not think that QuickTests are primarily for "smoke testing", but just testing that the module is functional for its essential stable functions that have been tested successfully in the past or that perform basic assertion states. The reply to quickTests is only to determine if the module is functional or not, and these tests could be used to disable temporarily a module completely and disable other modules depending on it for their own stability tests.

The "/testcases" suites don't have these limitations, and they can perform many more extensive tests, including for demonstrating things that are still not supported, or exhibit problems that won't be solved or features that won't be implemented or cannot be implemented (for example because of ambiguities or because we need deeper modifications elsewhere on the site, or in MediaWiki itself).

These "/testcases are in fact intended for discussions (when there are different ways to solve problems or implement extensions, and there are distinct opinions and a decision to take, or tests of performances to do) and that's why we put the results of "/testcases" suites in their own talk page (a "/testcases" submodule has a "/testcases/doc" subpage which can provide lists of links to other testcases, when all of them cannot run simultaneously, you can just create "/testcases/2", "/testcases/bug", "/testcases/extension proposal", and so on, each suite with its own talk page showing their results, and link all of theses suites from "/testcases/doc")

There will also be testcases only intended to test the behavior of deprecated features (useful during transition time, to see if the behavior is still acceptable when a better replacement feature is being deployed) and I don't think that these deprecated features should even be tested in "QuickTests" as they are no longer essential and stable (the main Module doc page may explain how to do the transition and will document deprecated parameters or values that should no longer be used).

• Problems detected in QuickTests should be solved in priority.
• Problems detected in "/testcases" should never have any emergency, they may succeed temporarily and fail again later without impacting lots of pages, except the few ones that are being tracked by the module developer (these pages may be listed in the /doc subpage of such "/testcases" suite, or may be tracked using specific tracking categories if the modules generate some wiki code, but in most cases, problems will only affect the talk page calling the "/testcases" suite of the same name and reporting the results). If these problems affect more pages, the failing feature should be documented in the "/doc" of the main module (with also a link to the appropriate "/testcases" talk page demonstrating and explaining the problem with more details.

verdy_p (talk) 20:23, 17 April 2015 (UTC)

"Tried to write global result."

{{Edit request}} Please change (what is currently) line 51 from:

	result = p.testModule(frame)

to:

	local result = p.testModule(frame)

It's currently triggering an error on Module:Documentation due to Module:No globals. —SamB (talk) 19:52, 14 May 2015 (UTC)

 Done Awesome! Thank you! Rillke^(q?) 22:05, 14 May 2015 (UTC)