Reproducers

From IcedTea

Jump to: navigation, search

Except classical unit test, which are worthy to be submitted with each line of code, there is also set of reproducers which are trying to catch regressions as soon as possible. The gate to add unit tests is directory icedtea-web/tests/netx/unit and the rest is just java+junit programming. But the system of reproducers needs some more information then is described on main page

Reproducers are here to simulate some specific behavior, which is nearly impossible to simulate in unit-tests (mostly because of class-loaders magic). They should be as small as possible and demonstrate one problem as clearly as possible. Each fix should have an associated reproducer. Failing reproducers can be added before a fix to help track the issue, use the 'KnownToFail' annotation to differentiate failures here from a regression.

Contents

1 Location

Each reproducer is in separate direcotry inside subdirectory of icedtea-web/tests/reproducers according of its purpose. Currently there are three and half purposes supported - (unlimited number of) signed, simple and custom. Btw there is nice read-me file inside icedtea-web/tests/reproducers ;)

  • In simple reproducers are just prepared.
  • But the jars produced from signed are also signed before the launching begins. Signed reproducers are enumerated inside Makefile.am in SIGNED_REPRODUCER variable. Each directory here will be signed by its own certificate (those certificates are injected during tests runs intio ~/.icedtea/security/trusted.certs so are used also for applets testing). So you can use reproducers containing unlimited number of signatures.
  • There is one more magic directory - custom - those reproducers are not processed automatically, but must have its own custom Makefile. Those makefiles are then called from prepare-reproducers target and have to compile and deploy themselves. Directory structure inside reproducer ('s directories in custom directory) is not forced by any way. Even deploy must be handled by your own custom Makefile, so use this approach carefully. Is hard to maintain.

For as much as possible use standard simple/signed ones (eg for compilation, deploy of resources, certificates...)

2 Prepare target and directory structure

Each reproducer must have its own directory eg icedtea-web/tests/jnlp_tests/simple/MyGloriousReproducer. Inside this one must be three directories, each with its purpose - resources, srcs and testcases.

Before tests run, each reproducer is prepared:

  • All *.java files inside srcs are compiled. On the classpath is just built netx.jar and classes from icedtea-web/tests/netx/test-extensions/ (see Virtual server below).
  • All tests from icedtea-web/tests/netx/test-extensions-tests are compiled and added to test run
  • All .class files compiled from srcs and all non java files just copied from srcs are packed into jar with coresponding directory tree with name as is their directory (MyGloriousReproducer if example above used)
  • All java files from testcases are compiled. Classpath is same as in case of srcs directory.
  • All files from resources (with their directory tree) and all jars produced from srcs are then copied into the $ICEDTEA_WEB_HOME/tests.build/jnlp_test_server directory.

When resources are in subdirectories, they are copied with all the tree to jnlp_test_server. But if reproducer class files are in directories, then those are interpreted as packages. There exists "dots" name convention for placing simple and signed jars to subdirectories (like reversed packages;): If you will have eg:

|-- {simple|signed}
|   `-- dir2.MyGloriousReproducer
|       |-- resources
|       |   |-- res1.txt
|       |   `-- dir1
|       |      `-- res2.txt}}
|       |-- srcs/
|       |   `-- package1
|       |       `-- source.java
|       `-- testcase
|           `testcase.java
.

then it will "deploy" as

|-- jnlp_test_server
|   |-- dir2
|   |   `-- MyGloriousReproducer.jar
|   |
|   |-- res1.txt
|   |-- dir2.MyGloriousReproducer.jar
|   |-- dir1
|   |   `-- res2.txt
.   .
.   .
.   .

where the duplication of identical dir2.MyGloriousReproducer.jar and MyGloriousReproducer.jar is intentional, and inside those jars will be package package1 with class package1.source in package1/source.class. Of course in the case of dir1 and dir2 with the same name, their contents will be merged. You can have as many dots as you wish (my.another.deep.jar will then be deployed as .../my/another/deep/jar.jar).

Please note, testcases cannot have directories/packages.

3 Virtual server

All the engine (located in icedtea-web/tests/netx/test-extensions/) is serving just one (of course many-times bended) purpose - to launch javaws/browsers with some jnlp/html as url with some timeout and to examine stdout/stderr of its run. Work on integration of awt.Robot (10% done). To be able to launch javaws with some url as parameter, we need some virtual server. This one is provided by icedtea-web/tests/netx/test-extensions/net/sourceforge/jnlp/ServerAccess.java class, and is running on random ( -D specified if wanted) port in -D specified directory as its root (icedtea-web/tests.build/jnlp_test_server) with -D'ed javaws location and specified browsers (see lower).

4 Testcases

Well - we have our (signed) jar and all resources inside icedtea-web/tests.build/jnlp_test_server. But how to test them? For this purpose is testcase. All files compiled from */*/testcase directory are send to junit engine which launches all @Test methods upon them. So what you should do inside testcase @Test method is to launch virtualserver, launch your prepared jnlp file and examine stdout/err. It sounds complicated, but most of this is handled by jnlp test's engine.

Please be sure, that your testcases are following logging as described here (unlike the files in */srcs - those are using System.out/err as usual).

5 Coding

Currently there is plenty of reproducers which, as examples are serving as best learning material. Just few hints:

  • default server is always running as singleton inside ServerAccess, so use something like private static ServerAccess server = new ServerAccess(); for accessing this with all workarround
  • jnlp (html files for applets) are supposed to be inside resources, and YES you must prepare them against the jar and launch them inside testcase
  • eg ServerAccess.ProcessResult pr = server.executeJavawsHeadless("/myresourcefile.jnlp"); is nice launch example ;)
  • You can always launch new instance on different port (and/or directory) by ServerAccess.getIndependentInstance();
  • You can easily run html/jnlp/whatever file by set of methods server.executeProcess/Browser/Javaws*
  • You can attach listeners to process for asynchronous communication
  • -headless and -Xtrusall are your friends
  • net.sourceforge.jnlp.ProcessResult have all what is needed when process is finished.
  • use Assert.something to check the results
  • feel free to find inspiration in already existing reproducers!
  • When writing testcases that fail if an exception is thrown, please avoid matching all exceptions as a failure case, especially in browser tests. Exceptions can occur when the process is killed (InterruptedException, etc).
  • When writing reproducer which have not clearly determined end or need to wait for timeout - especially browser test - try to use Closing listeners

6 Annotations

There are several annotations which make running of reproducers more easy and help to connect them to issues. All those annotations are for test-cases classes and/or test methods themselves.

  • @NeedsDisplay (method) - signalize to test-runner that this test needs DISPLAY variable correctly set and working. You must use it especially for all those cases:
    • running browser
    • running applet by jnlp file
    • runing javaws in diferent then headless mode
  • @KnownToFail (method)- one must use this annotation when reproducer is prepared for issue which is not going to be fixed immediately. Such a failing test is then marked in results and so its "red" does not scare others. Such reproducers are forcing developers to fix the issue and are making the fix much easier. If you fix the issue reproduced by some reproducer, please remove the KnownToFail annotation too. Whatever - the test results will scream at you if you have not done so;)
  • @Bug (method/class) - this annotation serves as connection with bug in some bugzilla or some discussion forum or webpage. Inside this annotation you can have bug-numbers (as described in NEWS file) or urls. Special care si taken for distro-pkg-dev mailing list url. When you are preparing reproducer for a bug, please provide Bug annotation with bugzial ids or to discussion where it was approved. It helps to track an issue a lot.
  • @TestInBrowser (method) - This is most powerful annotation. If you need to run test upon browser, then it is probably needed to be tested in various browsers or in some particular browser and it should be somehow adaptable to various system configurations (see this for implementation details). This annotation is using all configured system browser and iterate executeBrowser method upon them
    • The TestInBrowser has one parameter - testIn - array of browsers on which testcase should be launched eg.:
      • @TestInBrowser(testIn={Browsers.FIREFOX}) will launch the test method upon firefox if installed (if not installed default unset_browser will be used)
      • @TestInBrowser(testIn={Browsers.ONE}) will launch the test method upon one browser chosen randomly from all installed and configured browsers
      • @TestInBrowser(testIn={Browsers.ALL}) will launch the test method upon all installed and configured browsers
      • @TestInBrowser(testIn={Browsers.FIREFOX,Browsers.OPERA,Browsers.CHROME}) will run executeBrowser on installed and configured subset from firefox, opera and google-chrome.
    • the annotation can be configured by configure. There are "two" switches. When method is using TestInBrowser, it MUST extends BrowserTest class!
      • --with-browser-tests=yes|no|all|one
        • yes - default - will run executeBrowser exactly as described in TestInBrowser
        • no - all TestInBrowser annotated tests will be ignored
        • one - all TestInBrowser will behave as Browsers.ONE, with one exception - in case that set of exact browsers is specified, then one element of this set will be used
        • all - all TestInBrowser will behave as Browsers.ALL
      • --with-{browser}=yes|no|/path/to/bin - where {browser} stands for name of browser (eg --with-firefox=no --with-opera=/bin/my/opera)
        • yes - default - browser will be used if exists on system
        • no - browser will not be used
        • /path/to/bin - you can specify exact path to your prefered browser binary

7 Configuring browsers

Some tests are launching your browser to test applets behaviour. To be sure that your browser is using correct java plugin, you need to have correct symlinks (As described here ). For testing purposes there is simplification:

For global links (will affect all users of your system, thats why you need to be root) you can use target links, which will affect all supported browser. (Using mozilla-filesystm, treating Opera individually). The restore-global-links will return your previous configuration to link previous (stable?) icedtea-web plugin.

./autogen.sh
./configure ...
make
make check
make install
sudo make links
make run-netx-dist-tests
sudo make restore-global-links

or - if you do not care about Opera, ore can not be root - you can use user-links targets, to prepare icedtea-web plugin, which will affect only single user. But Opera is not supporting those links.

./autogen.sh
./configure ...
make
make check
make install
make user-link
make run-netx-dist-tests
make restore-user-links

Both those approaches will link libjavaplugin.so to your just build, in --prefix location. There is test to catch that you are using different then just build one.

Except this links is also useful to tune browser a bit. Especially session restore is good to disable. For firefox (which is most troubling browser) you can follow its knowledge base

8 Running

Since you have prepared your resources (jnlp(html)) files, your srcs fiels (and so your jar) and your testcase (which is lunching jnlp/html files and testing theirs outputs, you need to run them. Most easily you

8.1 run from commandline

It is easy, just build as follows:

./autogen.sh
./configure --prefix=<Location to destination where testing installation will be saved eg.: /home/user/icedtea-web-image> (--disable-docs will increase speed of your testing quite a lot)
make
make check
make install
make run-netx-dist-tests

In case that compilation of your srcs or testcase directory fails, the make is not interrupted, but you will quickly find this inside output files. All teh logs (stderr.log and stdout.log) and xml result files can be found in icedtea-web/tests.build/netx/jnlp_testsengine. The html file with results is in icedtea-web/tests.build/index_reproducers.hmtl

If you do not want to run whole testsuite of reproducers (eg because You are debugging your new reproducer) then you can use netx-dist-tests-whitelist. This file contains list (separated by space) of regualr expressions to be processed by expr match. If your testcase file name is matching at least one regex written here, then it will be executed (all reproducers are compiled and deployed) inside junit tests run. By default there is .* and so all testcase files will be run.

8.2 run from IDE

In case that you have somewhere run-netx-dist-tests'd copy of icedtea-web and just built jar, you can easily point you ide to those by -D and Xbootclasspath java parameters. Just set them to you ide as java arguments for launch.

  • -Dtest.server.dir=/where/is/your/build//icedtea-web/tests.build/jnlp_test_server
  • -Djavaws.build.bin=/prefixed/installed/icedtea-web-image/bin/javaws
  • -Dused.browsers=browsers:you:want:to:test
  • (not necessary for runnig/debugging testcases) -Xbootclasspath/a:/prefixed/installed/icedtea-web-image/share/icedtea-web/netx.jar:/rhino/oranother/JSengine/js.jar


With -Xbootclasspath and net.sourceforge.jnlp.runtime.Boot as main class, you can easily debug netx itself. But beware, you are always stepping against by-make prepared netx.jar

To walk around the "by-make prepared netx.jar" when running from IDE, you can easily add your just build classes before netx.jar. eg:

-Xbootclasspath/a:/somewhere/your_just_built_classes_directory:/usr/share/java/js.jar:/your_by?make_installed_icedtea-web-image/share/icedtea-web/netx.jar

Please note, to have installe dnetx.jar is still important, as there is eg deployment configurration, and you should have -Xnofork optin added on launching commandline

Warning - before running reproducers, be sure, that some other netx.jar is not already on your classpath. This happens almost always when you have installed icedtea6 < 1.10 or icedtea-web 1.0.X. You can test that by running java net.sourceforge.jnlp.runtime.Boot. You should see some sort of ClassNotFoundException if netx.jar is not already on your classpath. If you have icedtea6 < 1.10 installed, remove (renaming is not enough) netx.jar from your icedtea6 installation (you can always replace it later by your own one or by reinstalling icedtea6). If you have icedtea-web 1.0.X installed, uninstall icedtea-web. If you do not remove this netx.jar, then you will be testing this installed jar instead your built one. You are also likely to get lots of spurious failures as a result of trying to access classes across jars.

9 Code coverage

If you want to see how you have helped to icedtea web, or what remains to be tested, you can run

./autogen.sh
./configure --prefix=<Location to destination where testing installation will be saved eg.: /home/user/icedtea-web-image> (--disable-docs will increase speed of your testing quite a lot)
make
make check
make install
make run-netx-dist-tests
make run-test-code-coverage-jacoco

This will actually run run-unit-test-code-coverage-jacoco and run-reproducers-test-code-coverage-jacoco and then merge theirs results, which can be found inside icedtea-web/tests.build/coverage.xml (or icedtea-web/tests.build/netx/jnlp_testengine|unit for partial ones) and icedtea-web/tests.build/coverage or icedtea-web/tests.build/netx/jnlp_testsengine/coverage or icedtea-web/tests.build/netx/unit/coverage directory in html files with all the sources. Be aware that run run-unit-test-code-coverage and run-reproducers-test-code-coverage re-run the tests targets and overwrite theirs results, which can be now wrong.

The legacy emma targets are still avaiabel for legacy systems make run-test-code-coverage and run-unit-test-code-coverage and run-reproducers-test-code-coverage

Currently the best comamnd is

vim netx-dist-tests-whitelist;
./autogen.sh ;
./configure --disable-docs --prefix=~/icedtea-web-image/ --with-browser-tests=all --with-firefox=no ;
make ;
make check;
make  run-unit-test-code-coverage-jacoco ;
make run-cpp-unit-tests;
make install;
sudo make links ;
make run-netx-dist-tests ;
make run-reproducers-test-code-coverage-jacoco

10 Description of LiveConnect reproducers

These reproducers are automatized versions of LiveConnect Java<->JavaScript communication interactive tests - see http://icedtea.classpath.org/wiki/LiveConnectReproducers for details

11 Browser tweeks

  • disable session restore for each browser
    • epiphany is not alowing you to do so - solution is to put into ~/.config/epiphnay/session-restore.xml only about:blank and set read permissions for everybody.. Nasty but perfectly working
    • epiphany is not alowing you to do so - solution is to put into ~/.config/epiphnay/ession_state.xml only about:blank and set read permissions for everybody..
      • do not remove all winsow/embed nodes.:
<session>
	 <window x="0" y="24" width="1918" height="1048" active-tab="1" role="epiphany-window-30e238b1">
	 	 <embed url="about:blank" title="nope"/>
	 </window>
</session>
  • chrome and chromium have click to play feature for applets. It is necessary to allow icedtea-web for ever and for every case (about:plugins)
  • FIREFOX have click to play feature for applets. It is necessary to allow icedtea-web for ever and for every case about:config plugin.state.java set to 2
Personal tools