JCKDistilled

From IcedTea

Jump to: navigation, search

Table of Contents

  1. Introduction
  2. Typographic Conventions
  3. Unpacking the Test Suite
  4. Preparing to Run Tests
    1. Compiler
    2. Devtools
    3. Runtime
  5. Running the Test Suite
    1. Graphical User Interface
    2. Command-line
  6. Special test package install and setup
  7. VNC issues
    1. Sound
    2. Window manager
  8. Configuration Templates
  9. Running JCK with small memory

Introduction

The JCK contains three main areas of certification: compiler, devtools, and runtime. These three components and the JCK tests which describe them are contained in the JCK User's Guide in JCK-component-version/doc/jck.pdf, where component is "compiler", "runtime", or "devtools" the only version described here is 6b (6a is an older no longer available version. If you don't have 6b then please get that version. There are subtle differences between versions and only 6b is current.)

This document details the steps necessary to configure and run the certification tests for all three components.

The compiler and devtools test suites are pretty straightforward. They require only minimal editing of the JCK configuration template files and run on only one host. The runtime tests, on the other hand, are quite an adventure.

The runtime test suite requires the use of three networked machines in order to run successfully. For simplicity's sake, this document assumes the main test machine is foo; the remote test agent machine is bar, and the third (mainly unused) machine is baz.

We are going to assume that the JCK sources are installed into /jck/version. This will help simplify sharing configuration information amongst developers. The test suite being used is assumed to be installed in $JCKDIR, e.g., /jck/6b/JCK-compiler-6b and the workspaces for the test suites will be $JCKDIR/work.

Typographic Conventions

This guide uses the following typographic conventions:

Typeface Meaning Example
AaBbCc123 Code examples or file names foo$ mkdir sample_directory
Make sure that /jck/6b/bin is added to the path.
AaBbCc123 Variable placeholder rm -rf /jck/version
AaBbCc123 Titles of documentation JCK User's Guide
$ User command prompt - when commands are to be executed by a user account foo$ vi foobar.txt
# Super-user command prompt - when commands are to be executed by the super-user (root). May also be done via the sudo command. foo# vi /etc/security/limits.conf

Unpacking the Test Suite

Use the "-o" option to unpack the test suite into the /jck/version directory:

foo# mkdir -p /jck/6b foo# chmod -R 777 /jck foo$ java -jar jck_sample-6b-fcs-tck-b24-rt-23_apr_2008.jar -o /jck/6b  

You will now have the sample JCK runtime test suite installed in /jck/6b/sampleJCK-runtime-6b. I recommend starting with the sample test suite (especially for the runtime component) to check your configuration for errors before delving into the full test suite run (which is over 40,000 tests for the runtime component). There are three samples one each for compiler, tools and runtime, so in total you will have 6 test suites in /jck/6b/ (JCK-compiler-6b, JCK-runtime-6b, sampleJCK-devtools-6b, JCK-devtools-6b sampleJCK-compiler-6b and sampleJCK-runtime-6b) these will consume about 2GB when installed.

There will probably be an exclude list that you need to install. This exists because some of the tests in the JCK were found to be invalid, and these are listed in the exclude list for the version of the JCK you are using. In the configuration file for each testsuite there is an entry for the exclude list. Make sure you have the latest version: it will be in a file called something like jck-6b-upd-exc-22_jan_2009.zip. Unpack it and install it as /jck/6b/jdk6b.jtx:

foo$ unzip jck-6b-upd-exc-22_jan_2009.zip inflating: jdk6b.jtx foo$ cp jdk6b.jtx /jck/6b/jdk6b.jtx  

NOTE: There may be an "alternative bundle" for the devtools and runtime components. If there is, download the latest bundles from the Sun Partner website. After installing the devtools and runtime test suites:

foo$ cd path/to/alt01-zips foo$ unzip -o jck-devtools-6b-alt01-date.zip -d /jck/6b/JCK-devtools-6b foo$ unzip -o jck-runtime-6b-alt01-date.zip -d /jck/6b/JCK-runtime-6b  

In order to facilitate later steps, complete the following commands to create the general filesystem hierarchy that will be used for running the JCK:

foo$ cd /jck/6b foo$ mkdir bin lib resources foo$ ln -s /path/to/openjdk/control/build/linux-i586/j2sdk-image ./jdk  

One last thing to do. The JavaTest agent running on bar requires more open files than just the standard 1024 that are allowed. If you type "ulimit -n" at a terminal and it reports 1024 (or less), you need to allow for more. Run "ulimit -n 2048". If that gives you an error, you need to add a line to /etc/security/limits.conf to allow yourself or your group the ability to open more files.

For my own convenience, I have set both the hard and soft limits for the "users" group to 2048 open files. You can do this on Fedora 8 by executing the following command (and adding the user that will run the JavaTest agent to the group users if it isn't already):

bar# echo "@users hard nofile 2048" \   >> /etc/security/limits.conf bar# echo "@users soft nofile 2048" \   >> /etc/security/limits.conf  

You must then re-login for the changes to take effect.

Preparing to Run Tests

The JCK User's Guide describes steps necessary to prepare testing machines for running the JCK test suites. Unfortunately, these are arranged more for the convenience of JCK maintainers and developers than JCK users. This section will attempt to distill the information contained in the JCK User's Guide to a more manageable (and automatable) form.

Preparing for the Compiler Test Suite

The compiler test suite is pretty easy to run. Using the layout generated in this document, the compiler configuration template should be sufficient to just load and run, but you will need to start the activation daemon on the test host:

foo$ rmid -J-Dsun.rmi.activation.execPolicy=none &  

Preparing for the Devtools Test Suite

The devtools tests are pretty simple: you do not need the JavaTest remote agent, activation daemon, or anything else to run the tests. Simply load the configuration file and go.

Preparing for the Runtime Test Suite

This is the tricky part. As mentioned in the Introduction, the runtime tests require three properly configured machines in order to run successfully. These machines are:

  • foo - the machine on which the JDK will be tested

  • bar - the machine that will run the passive javatest agent

  • baz - a third machine which simply must be "reachable" (see port list)

  1. Disable firewall services on all three computers

    2. Sun recommends disabling all firewall services on all three computers, and that's what I've done below. If you do not want to do this, you must at least allow access to the following machines and ports:

    bar:1908/tcp used by the passive JavaTest agent
    bar:ICMP or bar:7/tcp
    baz:ICMP or baz:7/tcp     		for use by distributed network tests (if not using ICMP, an echo server must be running)

    foo# /sbin/service iptables stop bar# /sbin/service iptables stop baz# /sbin/service iptables stop foo# /sbin/service ip6tables stop bar# /sbin/service ip6tables stop baz# /sbin/service ip6tables stop foo# /sbin/chkconfig iptables off bar# /sbin/chkconfig iptables off baz# /sbin/chkconfig iptables off foo# /sbin/chkconfig ip6tables off bar# /sbin/chkconfig ip6tables off baz# /sbin/chkconfig ip6tables off

  2. Make test sources available to the remote javatest host (bar).

    3. You have two options here, either export your test sources via NFS from the test host (foo) or simply copy them to the remote javatest host (bar). I've chosen to use NFS:

    foo# cat >> /etc/exports << EOF > /jck *(rw) > EOF foo# /sbin/service nfs start

    Now mount the test sources available to bar:

    bar# mount foo:/jck /jck

    Obviously, if the sources already live in a place that is NFS reachable, you should not need to do anything except create the map file. But for the sake of simplicity, I recommend you follow the procedure outlined in this document.

  3. Create the map file

    4. The map file is used to map filenames from the test host (foo) into the namespace of the filesystem on the remote javatest host (bar). This is accomplished by means of a map file which gives a list of filesystem name mappings from the host (foo) to the remote host (bar). If you are following the steps in this document, you should only need to create an empty file, since we mount the test sources via NFS in the same place on all machines.

    bar$ touch /jck/6b/mapfile

  4. Set up a Kerberos 5 domain controller

    5. Several of the authentication tests require a Kerberos 5 server with two users. We are going to create a new JCKTEST domain running on the main test host, foo.mydomain.com.

    foo# yum install krb5-server foo# vi /etc/krb5.conf

    Edit the [REALMS] section, adding the new JCKTEST domain:

    [REALMS] JCKTEST = { kdc=foo.mydomain.com admin_server=foo.mydomain.com }

    You may need to add a DNS domain-to-realm mapping in the [domain_realm] section.

    Some recent Kerberos domain controllers also have a [libdefaults] section in the /etc/krb5.conf file. It needs to be updated too, especially the line default_realm:

    [libdefaults] default_realm = JCKTEST

    Next, create and initialize your new Kerberos domain, populating it with two users for use by the test suite. Please note, that paths to kdb5_util and kadmin.local are system dependent (as in Fedora 15 and Fedora 16 case). It means that kadmin.local, for example, is sometimes stored in /usr/sbin/ and not in /usr/kerberos/sbin/:

    foo# /usr/kerberos/sbin/kdb5_util create -r JCKTEST -s Loading random data this might take some time to complete Initializing database '/var/kerberos/krb5kdc/principal' for realm 'JCKTEST', master key name 'K/M@JCKTEST' You will be prompted for the database Master Password. It is important that you NOT FORGET this password. Enter KDC database master key: enter "jcktest" Re-enter KDC database master key to verify: re-enter "jcktest" foo# /usr/kerberos/sbin/kadmin.local -r JCKTEST Authenticating as principal keiths/admin@JCKTEST with password. kadmin.local: ank user1 WARNING: no policy specified for user1@JCKTEST; defaulting to no policy Enter password for principal "user1@JCKTEST": enter "user1" Re-enter password for principal "user1@JCKTEST": re-enter "user1" Principal "user1@JCKTEST" created. kadmin.local: ank user2 WARNING: no policy specified for usser2@JCKTEST; defaulting to no policy Enter password for principal "user2@JCKTEST": enter "user2" Re-enter password for principal "user2@JCKTEST": re-enter "user2" Principal "user2@JCKTEST" created. kadmin.local: quit

    Note, that service prnicipals are unrelated to users. Feel free to add test service principals, while in kadmin.local -r JCKTEST eg:

    admin.local: addprinc -randkey http/service.redhat.com WARNING: no policy specified for http/service.redhat.com@JCKTEST; defaulting to no policy Principal "http/service.redhat.com@JCKTEST" created.

    Now edit the server configuration:

    foo# vi /var/kerberos/krb5kdc/kdc.conf replace EXAMPLE.COM with JCKTEST foo# vi /etc/sysconfig/krb5kdc   change the line to read KRB5REALM=JCKTEST

    Some recent Kerberos domain controllers should be configured differently, especially in case the default_realm is set in krb5.conf file.

    Finally, start the Kerberos server:

    foo# /sbin/service krb5kdc start Starting Kerberos 5 KDC: [ OK ] foo# /sbin/chkconfig krb5kdc on

  5. Compile native code

    6. There are many tests which require native code be built and massaged into various jar files for use by the runtime test suite. If you've followed the directions in this document, the following command should do everything you need to do:

    foo$ /path/to/icedtea6/contrib/jck/compile-native-code.sh /jck/6b/sampleJCK-runtime-6b

    Warning: If you are running the sampleJCK test suite and later switch to the real JCK-runtime test suite, you must recompile everything or you will get test failures.

  6. Start miscellaneous servers needed by the test suite

    7. foo$ rmid -J-Dsun.rmi.activation.execPolicy=none & foo$ tnameserv -ORBInitialPort 1234

  7. Start the remote javatest agent on the remote host bar

    8. NOTE: You must use JDK 1.6.0_04 (or higher) to run the remote JavaTest Agent.

    To start the GUI client:

    bar$ java -cp $JCKDIR/lib/javatest.jar -Djava.security.policy=$JCKDIR/lib/jck.policy -Djavatest.security.allowPropertiesAccess=true com.sun.javatest.agent.AgentFrame

    You will need to manually configure passive mode, specify the map file, etc.

    To start the terminal version of the test client:

    bar$ java -cp $JCKDIR/lib/javatest.jar -Djava.security.policy=$JCKDIR/lib/jck.policy -Djavatest.security.allowPropertiesAccess=true com.sun.javatest.agent.AgentMain -passive [-map mapfile -trace]

    See the JavaTest Agent User's Guide in $JCKDIR/doc/javatest/javatestAgent.pdf for more information on using the JavaTest agents.

  8. Make sure that keyboard autorepeat is off, either in GNOME through the Keyboard Accessibility Features under System -> Preferences -> Personal, or through running "xset -r". This is needed for runtime test: api/java_awt/interactive/event/EventTests.html#EventTest0019.

Running the Test Suite

Now the fun part: getting some results! If you have followed this document closely, and I have not forgotten to tell you anything, you should be able to start running the tests:

  1. Run the GUI test harness

    2. foo$ cd /jck/6b foo$ java -Xms128m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=512m -jar $JCKDIR/lib/javatest.jar

    NOTE: You want to allocate more memory to the virtual machine. While playing with runtime testing, the test harness would sit there doing nothing. Hours later, I noticed a bunch of OutOfMemoryExceptions. Hence, we now pass additional options to java when starting the harness.

    NOTE: For JCK 6b Devtools testing, you must run the tests from the devtools JCK directory, i.e., cd /jck/6b/JCK-devtools-6b before running the JavaTest GUI.

  2. Load the appropriate test suite

    3. Use File->Open Test Suite ... and select the test suite you would like to use, e.g., /jck/6b/sampleJCK-runtime-6b.

  3. Create or specify the work directory and load the configuration template

    4. I recommend not altering the configuration template files. Instead, copy the file and use that for your configuration. For example, I simply cp runtime-template.jti runtime.jti and load this file. Once you find a configuration that works (or works better), make sure you keep a backup of it lying around, just in case you need to tweak your configuration and it breaks. [Don't ask me how I know this.] The configuration files are simply plain text, so you can diff them if you need to compare different versions.

    You can edit the configuration files using a text editor, but you must remove the checksum variable at the top. If you forget to do this, JavaTest will complain that the configuration is corrupted or invalid and it will refuse to load it. When JavaTest sees an unchecksummed configuration file, it will load it and add a new checksum. So just remember to erase the checksum at the top of the configuration file if you ever edit by hand.

    Use the Configuration->Load Configuration ... menu to load the configuration template (or the copy of it that you made), runtime-template.jti. If you haven't already specified a work directory, the test harness will first ask you to create or select one. I usually use a work directory contained in the test suite folder, such as /jck/6/sampleJCK-runtime-6/work.

  4. Check/edit the default configuration

    5. The template files are currently configured to do trial runs (not certification runs) of all tests (except interactive test in the runtime test suite). If you wish to run other tests, you can modify these using the configuration editor. You can edit the current configuration by either selecting Configure->Change Configuration ...->Other Values ... or hitting ^E to bring up the list of all the interview questions. If you simply wish to tweak the keyword list, use the Keyword ... menu item instead: it's a lot nicer interface (mainly because it gives you a list of all the defined keywords).

    WARNING: NEVER press "DONE" or "NEXT" in the configuration editor on the screen titled "Time Factor" (roughly) or later. Your test harness will lock up somewhere around here. It is okay to press done on any other screen of the configuration editor. This is a known bug in IcedTea and OpenJDK http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=67

    The compiler and devtools test suites are pretty straight-forward. Simply loading the template file and clicking run should get the tests running. [Don't forget to start the activation daemon for the compiler tests!] For the runtime tests, however, you will need to edit the configuration before you can run it. The following table lists all the options you will need to modify:

    Option Default Description
    KDC Host Name foo* host name of the KDC server
    Local Host Name foo* host name of the local machine
    Local Host IP Address 1.2.3.4 IP address of the local machine
    Test Host 1 Name bar* host name of a reachable test host (only needs to respond to ping or echo server – see Configuring and Running the Runtime Test Suite)
    Test Host 1 IP 1.2.3.5 IP address of above machine
    Test Host 2 Name baz* host name of second test host
    Test Host 2 IP 1.2.3.6 IP address of above machine
    Remote Host bar* host name of the machine running the javatest remote agent
    ORB Host foo* host name of the machine running the Java RMI Activation daemon (rmid)
    Printing Tests Yes optionally omit printing tests
    JDWP Command-line Options -agentlib:jdwp=server=n,transport=dt_socket,address=bar:35000,suspend=y replace bar with the name of your remote javatest host

    * I recommend using FQDN for hostnames. Some of the tests fail if you don't.

  5. Run tests

    6. Press the Run button or Run Tests->Start to start testing.

Good luck!

Running the Test Suite from the Command Line

 You can also perform testing without the JavaTest GUI. The following will allow just the VM runtime tests to be executed from the command-line:

java -jar $JCKDIR/lib/javatest.jar -verbose:progress -config /jck/6b/runtime-config.jti \ -testSuite $JCKDIR -workDir $JCKDIR/work -excludeList /jck/6b/jdk6b.jtx -tests vm -runtests -writeReport $JCKDIR/report

Test specific package installs and setup

Some tests need specific packages installed, or not installed, or run with specific settings or window managers to PASS. Remember that all test runs have been done on Fedora 8 x86 or x86_64, other releases might need a different set of packages installed/uninstalled to get a full PASS.

  • Compiler test: lang/CLSS/clss411/ciss41104/clss41104a.html#clss41104src - Should be added to exclude list, make sure you have the very latest from https://java-partner.sun.com.

  • Runtime tests: api/java_awt/interactive/FontTests.html#FontTests, api/javax_swing/interactive/JEditorPaneTests.html#JEditorPane, api/javax_swing/interactive/JLabelTests.html#JLabel, api/javax_swing/interactive/JTextAreaTests.html#JTextArea - make sure you have the package dejavu-fonts installed

  • Runtime test: api/javax_swing/SwingUtilities/descriptions.html#other api/javax_swing/basic/BasicLabelUI/index.html#protected make sure you do NOT have the package ghostscript-fonts installed.

  • Runtime test: api/java_security/SecureRandom/SecureRandomTests.html#misc might need a really, really long time or enough entropy. Set timeout factor to something really high (20), but make sure no other test fails because of timeout or the whole run will take ages.

VNC issues

Sound

If you are running the test suite over VNC then you may need to change the ownership of the various sound devices to your user:

sudo chown $USER /dev/{audio,dsp,mixer,sequencer,sequencer2,snd/*}

Window Manager

One of the AWT tests seems to fail when run on Metacity over VNC. One solution is to use TWM, but be sure to add RandomPlacement to your ~/.twmrc or tests will time out waiting for you to place windows.

JCK Configuration Templates

If you are carefully following the instructions in this document, you might find these (mostly) ready-to-go configuration templates useful. Please be sure to read the preparation instructions for any manual configuration that might be needed with these files.

Running JCK with small memory

The following describes my experiences running the JCK tests on the ARM Babbage board which has only 512K memory. In a word, "painful", however I hope the following may be of benefit to others trying to run the tests on the ARM Babbage board, or other platforms with similarly constrained memory systems.

The first thing is that you cannot run the Test Harness and the tests on the same board. The Test Harness is quite memory intensive, so you need to run that on a separate machine, probably your X86 desktop. The configuration can be quite confusing at times.

Firstly, set up 2 directories, /jck and /jdk. /jck contains all the tests as unpacked from the JCK distribution, I unpacked them into /jck/6b. /jdk contains the JDK you are testing on your test platform, in this case, the ARM board. On your X86 box which runs the test harness /jdk should be a link to the Sun reference JDK.

/jck should be NFS mounted so the same view is available on the test platform and the test harness platform and should be writable on both.

/jdk is not NFS mounted, but is private to each machine. The idea behind this is so that when you enter /jdk in response to a question asking you to enter the location of the JDK you can just enter /jdk without worrying about whether it is asking you for the location of a JDK on the test platform or the test harness platform. I tied myself in knots because it was trying to execute the ARM JDK on the X86 test harness, because it didn't recognise it as a binary it tried to execute 'java' as a shell script. The resulting error messages were not elucidating.

  • /jck -> Test suite, NFS mounted across all machines and wirtable by all machines.
  • /jdk -> Local directory or link pointing to the appropriate JDK for that machine.

Below are the configurations I am using for the compiler, devtools and runtime respectively.

(note: the above files are served from my home server camswl.com, could some kind person with the knowledge please upload them to iced-tea)

In the above configurations, the following machine names are used.

orac3 - The board used to run the runtime tests
babbage-karmic - The board used to run the compiler tests
babbage-davem-3 - The board used to run the devtools tests
e101646 - My desktop PC (the name was chosen by IT).

Starting the Test agent on each test board

On each test board you will need to start a test agent which communicates with the test harness and runs tests on behalf of the test harness. Here are the commands I used to start the test agent for each of compiler, devtools and runtime.

compiler
cd /jck
/jdk/bin/java -cp /jck/6b/JCK-compiler-6b/lib/javatest.jar:/jckb/JCK-compiler-6b/lib/jtjck.jar:. -Xmx50m com.sun.javatest.agent.AgentFrame -passive -trace -start -map /dev/null
devtools
cd /jck/6b/JCK-devtools-6b
/jdk/bin/java -cp /jck/6b/JCK-devtools-6b/lib/javatest.jar:/jck/6b/JCK-devtools-6b/lib/jtjck.jar:. -Xmx50m com.sun.javatest.agent.AgentFrame -passive -trace -start -map /dev/null
runtime
cd /jck/6b/JCK-runtime-6b/classes
/jdk/bin/java -cp /jck/6b/JCK-runtime-6b/lib/javatest.jar:/jck/6b/JCK-runtime-6b/lib/jtjck.jar:. -Xmx400m com.sun.javatest.agent.AgentFrame -passive -trace -start -map /dev/null

Note the difference between the runtime command and the compiler and devtools commands. For the compiler and devtools command I specify -Xmx50m because the agent is just a small piece of code which runs the test as a separate process. However, for the runtime tests I specify -Xmx400m because the tests are run as part of the agent, a separate jvm process is not started for each runtime test.

This leads to the next point of confusion. The compiler and devtools tests are run in MultiJVM mode. The runtime tests are always run in SingleJVM mode.

The reason for this is that the runtime tests *always* use an agent. If you try to select MultiJVM mode for the runtime tests the interview will not even ask you whether you are running them locally, or not. You must be running them locally in MultiJVM mode because the Test Harness will start a new agent for each test. It cannot do that on a remote machine.

The compiler and devtools tests must be run in MultiJVM mode because it needs to start up an instance of 'javac' or the appropriate tool for each test (note: there is an option for running the compiler and devtools in SingleJVM mode, but I failed to get that to work. If you succeed please add a note here).

Note on the test interview

The following are some notes on the test interview for each of the compiler, devtools and runtime tests. I am only highlighting differences for running the tests on separate machines from the test harness here. Otherwise, use the configuration as described in JCK_Distilled.

Compiler or Devtools

Test Execution Mode
Select 'MultiJVM'
Testing Locally
Select 'No'
Test Platform
Select 'Linux'
Agent Type
Select 'Passive'
Passive Host
Enter the name of your test board (eg. 'babbage-karmic')
Default Agent Port
Select 'Yes'

Runtime

Test Execution Mode
Select 'SingleJVM'
File seperator
/
Path seperator
:
Agent Type
Select 'Passive'
Passive Host
Enter the name of your test board (eg. 'orac3')
Default Agent Port
Select 'Yes'
Note: The runtime interview does not ask whether you are running the tests locally or not. It assumes that if you are using SingleJVM mode you are not running the tests locally.

Running the test Harness

The following is the command I use to start the Test Harness on my X86 box.

java -jar /jck/6b/JCK-runtime-6b/lib/javatest.jar -newdesktop

  • Do specify the -newdesktop argument. If you do not specify this it will try to pick up the work directories it had open the last time you ran it, which if you are like me, you have probably removed.
  • Do run all of the individual test suites (compiler, devtools, runtime) in a single instance of the Test Harness. The first time I ran all 3 I started 3 instances of the Test Harness on my X86 box and brought it to its knees. To run additional test suites in a single test harness goto File->Open->Test Suite, then goto File->Create Work Directory..., then goto Configure->Load Configuration... in that order. You will then get a new tab at the bottom with the new test suite and can switch between test suites using the tabs at the bottom.

Other Notes

  • The runtime tests require a DISPLAY of at least 1280X1024. The Babbage boards I was using did not have this. If you do not have this you will need to specify a remote X DISPLAY that does. An alternative would be to use xvfb, but I haven't got that working yet. Please add a note here if you have.

Retrieved from "https://icedtea.classpath.org/wiki/JCKDistilled"
Personal tools