JCKDistilled

From IcedTea

Jump to: navigation, search

Table of Contents

  1. Introduction
  2. Typographic Conventions
  3. Unpacking the Test Suite
  4. Preparing to Run Tests
    1. Compiler
    2. Devtools
    3. Runtime
  5. Running the Test Suite
    1. Graphical User Interface
    2. Command-line
  6. Special test package install and setup
  7. VNC issues
    1. Sound
    2. Window manager
  8. Configuration Templates

Introduction

The JCK contains three main areas of certification: compiler, devtools, and runtime. These three components and the JCK tests which describe them are contained in the JCK User's Guide in JCK-component-version/doc/jck.pdf, where component is "compiler", "runtime", or "devtools" the only version described here is 6b (6a is an older no longer available version. If you don't have 6b then please get that version. There are subtle differences between versions and only 6b is current.)

This document details the steps necessary to configure and run the certification tests for all three components.

The compiler and devtools test suites are pretty straightforward. They require only minimal editing of the JCK configuration template files and run on only one host. The runtime tests, on the other hand, are quite an adventure.

The runtime test suite requires the use of three networked machines in order to run successfully. For simplicity's sake, this document assumes the main test machine is foo; the remote test agent machine is bar, and the third (mainly unused) machine is baz.

We are going to assume that the JCK sources are installed into /jck/version. This will help simplify sharing configuration information amongst developers. The test suite being used is assumed to be installed in $JCKDIR, e.g., /jck/6b/JCK-compiler-6b and the workspaces for the test suites will be $JCKDIR/work.

Typographic Conventions

This guide uses the following typographic conventions:

Typeface Meaning Example
AaBbCc123 Code examples or file names foo$ mkdir sample_directory
Make sure that /jck/6b/bin is added to the path.
AaBbCc123 Variable placeholder rm -rf /jck/version
AaBbCc123 Titles of documentation JCK User's Guide
$ User command prompt - when commands are to be executed by a user account foo$ vi foobar.txt
# Super-user command prompt - when commands are to be executed by the super-user (root). May also be done via the sudo command. foo# vi /etc/security/limits.conf

Unpacking the Test Suite

Use the "-o" option to unpack the test suite into the /jck/version directory:

foo# mkdir -p /jck/6b foo# chmod -R 777 /jck foo$ java -jar jck_sample-6b-fcs-tck-b24-rt-23_apr_2008.jar -o /jck/6b  

You will now have the sample JCK runtime test suite installed in /jck/6b/sampleJCK-runtime-6b. I recommend starting with the sample test suite (especially for the runtime component) to check your configuration for errors before delving into the full test suite run (which is over 40,000 tests for the runtime component). There are three samples one each for compiler, tools and runtime, so in total you will have 6 test suites in /jck/6b/ (JCK-compiler-6b, JCK-runtime-6b, sampleJCK-devtools-6b, JCK-devtools-6b sampleJCK-compiler-6b and sampleJCK-runtime-6b) these will consume about 2GB when installed.

There will probably be an exclude list that you need to install. This exists because some of the tests in the JCK were found to be invalid, and these are listed in the exclude list for the version of the JCK you are using. In the configuration file for each testsuite there is an entry for the exclude list. Make sure you have the latest version: it will be in a file called something like jck-6b-upd-exc-22_jan_2009.zip. Unpack it and install it as /jck/6b/jdk6b.jtx:

foo$ unzip jck-6b-upd-exc-22_jan_2009.zip inflating: jdk6b.jtx foo$ cp jdk6b.jtx /jck/6b/jdk6b.jtx  

NOTE: There may be an "alternative bundle" for the devtools and runtime components. If there is, download the latest bundles from the Sun Partner website. After installing the devtools and runtime test suites:

foo$ cd path/to/alt01-zips foo$ unzip -o jck-devtools-6b-alt01-date.zip -d /jck/6b/JCK-devtools-6b foo$ unzip -o jck-runtime-6b-alt01-date.zip -d /jck/6b/JCK-runtime-6b  

In order to facilitate later steps, complete the following commands to create the general filesystem hierarchy that will be used for running the JCK:

foo$ cd /jck/6b foo$ mkdir bin lib resources foo$ ln -s /path/to/openjdk/control/build/linux-i586/j2sdk-image ./jdk  

One last thing to do. The JavaTest agent running on bar requires more open files than just the standard 1024 that are allowed. If you type "ulimit -n" at a terminal and it reports 1024 (or less), you need to allow for more. Run "ulimit -n 2048". If that gives you an error, you need to add a line to /etc/security/limits.conf to allow yourself or your group the ability to open more files.

For my own convenience, I have set both the hard and soft limits for the "users" group to 2048 open files. You can do this on Fedora 8 by executing the following command (and adding the user that will run the JavaTest agent to the group users if it isn't already):

bar# echo "@users hard nofile 2048" \   >> /etc/security/limits.conf bar# echo "@users soft nofile 2048" \   >> /etc/security/limits.conf  

You must then re-login for the changes to take effect.

Preparing to Run Tests

The JCK User's Guide describes steps necessary to prepare testing macihnes for running the JCK test suites. Unfortunately, these are arranged more for the convenience of JCK maintainers and developers than JCK users. This section will attempt to distill the information contained in the JCK User's Guide to a more managable (and automatable) form.

Preparing for the Compiler Test Suite

The compiler test suite is pretty easy to run. Using the layout generated in this document, the compiler configuration template should be sufficient to just load and run, but you will need to start the activation daemon on the test host:

foo$ rmid -J-Dsun.rmi.activation.execPolicy=none &  

Preparing for the Devtools Test Suite

The devtools tests are pretty simple: you do not need the JavaTest remote agent, activation daemon, or anything else to run the tests. Simply load the configuration file and go.

Preparing for the Runtime Test Suite

This is the tricky part. As mentioned in the Introduction, the runtime tests require three properly configured machines in order to run successfully. These machines are:

  • foo - the machine on which the JDK will be tested

  • bar - the machine that will run the passive javatest agent

  • baz - a third machine which simply must be "reachable" (see port list)

  1. Disable firewall services on all three computers

    2. Sun recommends disabling all firewall services on all three computers, and that's what I've done below. If you do not want to do this, you must at least allow access to the following machines and ports:

    bar:1908/tcp used by the passive JavaTest agent
    bar:ICMP or bar:7/tcp
    baz:ICMP or baz:7/tcp     		for use by distributed network tests (if not using ICMP, an echo server must be running)

    foo# /sbin/service iptables stop bar# /sbin/service iptables stop baz# /sbin/service iptables stop foo# /sbin/service ip6tables stop bar# /sbin/service ip6tables stop baz# /sbin/service ip6tables stop foo# /sbin/chkconfig iptables off bar# /sbin/chkconfig iptables off baz# /sbin/chkconfig iptables off foo# /sbin/chkconfig ip6tables off bar# /sbin/chkconfig ip6tables off baz# /sbin/chkconfig ip6tables off

  2. Make test sources available to the remote javatest host (bar).

    3. You have two options here, either export your test sources via NFS from the test host (foo) or simply copy them to the remote javatest host (bar). I've chosen to use NFS:

    foo# cat >> /etc/exports << EOF > /jck *(rw) > EOF foo# /sbin/service nfs start

    Now mount the test sources available to bar:

    bar# mount foo:/jck /jck

    Obviously, if the sources already live in a place that is NFS reachable, you should not need to do anything except create the map file. But for the sake of simplicity, I recommend you follow the procedure outlined in this document.

  3. Create the map file

    4. The map file is used to map filenames from the test host (foo) into the namespace of the filesystem on the remote javatest host (bar). This is accomplished by means of a map file which gives a list of filesystem name mappings from the host (foo) to the remote host (bar). If you are following the steps in this document, you should only need to create an empty file, since we mount the test sources via NFS in the same place on all machines.

    bar$ touch /jck/6b/mapfile

  4. Set up a Kerberos 5 domain controller

    5. Several of the authentication tests require a Kerberos 5 server with two users. We are going to create a new JCKTEST domain running on the main test host, foo.mydomain.com.

    foo# yum install krb5-server foo# vi /etc/krb5.conf

    Edit the [REALMS] section, adding the new JCKTEST domain:

    [REALMS] JCKTEST = { kdc=foo.mydomain.com admin_server=foo.mydomain.com }

    You may need to add a DNS domain-to-realm mapping in the [domain_realm] section.

    Next, create and initialize your new Kerberos domain, populating it with two users for use by the test suite.

    foo# /usr/kerberos/sbin/kdb5_util create -r JCKTEST -s Loading random data this might take some time to complete Initializing database '/var/kerberos/krb5kdc/principal' for realm 'JCKTEST', master key name 'K/M@JCKTEST' You will be prompted for the database Master Password. It is important that you NOT FORGET this password. Enter KDC database master key: enter "jcktest" Re-enter KDC database master key to verify: re-enter "jcktest" foo# /usr/kerberos/sbin/kadmin.local -r JCKTEST Authenticating as principal keiths/admin@JCKTEST with password. kadmin.local: ank user1 WARNING: no policy specified for user1@JCKTEST; defaulting to no policy Enter password for principal "user1@JCKTEST": enter "user1" Re-enter password for principal "user1@JCKTEST": re-enter "user1" Principal "user1@JCKTEST" created. kadmin.local: ank user2 WARNING: no policy specified for usser2@JCKTEST; defaulting to no policy Enter password for principal "user2@JCKTEST": enter "user2" Re-enter password for principal "user2@JCKTEST": re-enter "user2" Principal "user2@JCKTEST" created. kadmin.local: quit

    Now edit the server configuration:

    foo# vi /var/kerberos/krb5kdc/kdc.conf replace EXAMPLE.COM with JCKTEST foo# vi /etc/sysconfig/krb5kdc   change the line to read KRB5REALM=JCKTEST

    Finally, start the Kerberos server:

    foo# /sbin/service krb5kdc start Starting Kerberos 5 KDC: [ OK ] foo# /sbin/chkconfig krb5kdc on

  5. Compile native code

    6. There are many tests which require native code be built and massaged into various jar files for use by the runtime test suite. If you've followed the directions in this document, the following command should do everything you need to do:

    foo$ /path/to/icedtea6/contrib/jck/compile-native-code.sh /jck/6b/sampleJCK-runtime-6b

    Warning: If you are running the sampleJCK test suite and later switch to the real JCK-runtime test suite, you must recompile everything or you will get test failures.

  6. Start miscellaneous servers needed by the test suite

    7. foo$ rmid -J-Dsun.rmi.activation.execPolicy=none & foo$ tnameserv -ORBInitialPort 1234

  7. Start the remote javatest agent on the remote host bar

    8. NOTE: You must use JDK 1.6.0_04 (or higher) to run the remote JavaTest Agent.

    To start the GUI client:

    bar$ java -cp $JCKDIR/lib/javatest.jar -Djava.security.policy=$JCKDIR/lib/jck.policy -Djavatest.security.allowPropertiesAccess=true com.sun.javatest.agent.AgentFrame

    You will need to manually configure passive mode, specify the map file, etc.

    To start the terminal version of the test client:

    bar$ java -cp $JCKDIR/lib/javatest.jar -Djava.security.policy=$JCKDIR/lib/jck.policy -Djavatest.security.allowPropertiesAccess=true com.sun.javatest.agent.AgentMain -passive [-map mapfile -trace]

    See the JavaTest Agent User's Guide in $JCKDIR/doc/javatest/javatestAgent.pdf for more information on using the JavaTest agents.

  8. Make sure that keyboard autorepeat is off, either in GNOME through the Keyboard Accessibility Features under System -> Preferences -> Personal, or through running "xset -r". This is needed for runtime test: api/java_awt/interactive/event/EventTests.html#EventTest0019.

Running the Test Suite

Now the fun part: getting some results! If you have followed this document closely, and I have not forgotten to tell you anything, you should be able to start running the tests:

  1. Run the GUI test harness

    2. foo$ cd /jck/6b foo$ java -Xms128m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=512m -jar $JCKDIR/lib/javatest.jar

    NOTE: You want to allocate more memory to the virtual machine. While playing with runtime testing, the test harness would sit there doing nothing. Hours later, I noticed a bunch of OutOfMemoryExceptions. Hence, we now pass additional options to java when starting the harness.

    NOTE: For JCK 6b Devtools testing, you must run the tests from the devtools JCK directory, i.e., cd /jck/6b/JCK-devtools-6b before running the JavaTest GUI.

  2. Load the appropriate test suite

    3. Use File->Open Test Suite ... and select the test suite you would like to use, e.g., /jck/6b/sampleJCK-runtime-6b.

  3. Create or specify the work directory and load the configuration template

    4. I recommend not altering the configuration template files. Instead, copy the file and use that for your configuration. For example, I simply cp runtime-template.jti runtime.jti and load this file. Once you find a configuration that works (or works better), make sure you keep a backup of it lying around, just in case you need to tweak your configuration and it breaks. [Don't ask me how I know this.] The configuration files are simply plain text, so you can diff them if you need to compare different versions.

    You can edit the configuration files using a text editor, but you must remove the checksum variable at the top. If you forget to do this, JavaTest will complain that the configuration is corrupted or invalid and it will refuse to load it. When JavaTest sees an unchecksummed configuration file, it will load it and add a new checksum. So just remember to erase the checksum at the top of the configuration file if you ever edit by hand.

    Use the Configuration->Load Configuration ... menu to load the configuration template (or the copy of it that you made), runtime-template.jti. If you haven't already specified a work directory, the test harness will first ask you to create or select one. I usually use a work directory contained in the test suite folder, such as /jck/6/sampleJCK-runtime-6/work.

  4. Check/edit the default configuration

    5. The template files are currently configured to do trial runs (not certification runs) of all tests (except interactive test in the runtime test suite). If you wish to run other tests, you can modify these using the configuration editor. You can edit the current configuration by either selecting Configure->Change Configuration ...->Other Values ... or hitting ^E to bring up the list of all the interview questions. If you simply wish to tweak the keyword list, use the Keyword ... menu item instead: it's a lot nicer interface (mainly because it gives you a list of all the defined keywords).

    WARNING: NEVER press "DONE" or "NEXT" in the configuration editor on the screen titled "Time Factor" (roughly) or later. Your test harness will lock up somewhere around here. It is okay to press done on any other screen of the configuration editor. This is a known bug in IcedTea and OpenJDK http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=67

    The compiler and devtools test suites are pretty straight-forward. Simply loading the template file and clicking run should get the tests running. [Don't forget to start the activation daemon for the compiler tests!] For the runtime tests, however, you will need to edit the configuration before you can run it. The following table lists all the options you will need to modify:

    Option Default Description
    KDC Host Name foo* host name of the KDC server
    Local Host Name foo* host name of the local machine
    Local Host IP Address 1.2.3.4 IP address of the local machine
    Test Host 1 Name bar* host name of a reachable test host (only needs to respond to ping or echo server – see Configuring and Running the Runtime Test Suite)
    Test Host 1 IP 1.2.3.5 IP address of above machine
    Test Host 2 Name baz* host name of second test host
    Test Host 2 IP 1.2.3.6 IP address of above machine
    Remote Host bar* host name of the machine running the javatest remote agent
    ORB Host foo* host name of the machine running the Java RMI Activation daemon (rmid)
    Printing Tests Yes optionally omit printing tests
    JDWP Command-line Options -agentlib:jdwp=server=n,transport=dt_socket,address=bar:35000,suspend=y replace bar with the name of your remote javatest host

    * I recommend using FQDN for hostnames. Some of the tests fail if you don't.

  5. Run tests

    6. Press the Run button or Run Tests->Start to start testing.

Good luck!

Running the Test Suite from the Command Line

 You can also perform testing without the JavaTest GUI. The following will allow just the VM runtime tests to be executed from the command-line:

java -jar $JCKDIR/lib/javatest.jar -verbose:progress -config /jck/6b/runtime-config.jti \ -testSuite $JCKDIR -workDir $JCKDIR/work -excludeList /jck/6b/jdk6b.jtx -tests vm -runtests -writeReport $JCKDIR/report

Test specific package installs and setup

Some tests need specific packages installed, or not installed, or run with specific settings or window managers to PASS. Remember that all test runs have been done on Fedora 8 x86 or x86_64, other releases might need a different set of packages installed/uninstalled to get a full PASS.

  • Compiler test: lang/CLSS/clss411/ciss41104/clss41104a.html#clss41104src - Should be added to exclude list, make sure you have the very latest from https://java-partner.sun.com.

  • Runtime tests: api/java_awt/interactive/FontTests.html#FontTests, api/javax_swing/interactive/JEditorPaneTests.html#JEditorPane, api/javax_swing/interactive/JLabelTests.html#JLabel, api/javax_swing/interactive/JTextAreaTests.html#JTextArea - make sure you have the package dejavu-fonts installed

  • Runtime test: api/javax_swing/SwingUtilities/descriptions.html#other api/javax_swing/basic/BasicLabelUI/index.html#protected make sure you do NOT have the package ghostscript-fonts installed.

  • Runtime test: api/java_security/SecureRandom/SecureRandomTests.html#misc might need a really, really long time or enough entropy. Set timeout factor to something really high (20), but make sure no other test fails because of timeout or the whole run will take ages.

VNC issues

Sound

If you are running the test suite over VNC then you may need to change the ownership of the various sound devices to your user:

sudo chown $USER /dev/{audio,dsp,mixer,sequencer,sequencer2,snd/*}

Window Manager

One of the AWT tests seems to fail when run on Metacity over VNC. One solution is to use TWM, but be sure to add RandomPlacement to your ~/.twmrc or tests will time out waiting for you to place windows.

JCK Configuration Templates

If you are carefully following the instructions in this document, you might find these (mostly) ready-to-go configuration templates useful. Please be sure to read the preparation instructions for any manual configuration that might be needed with these files.

Retrieved from "http://icedtea.classpath.org/wiki/JCKDistilled"
Personal tools