readme.htm

# Derby Functional Tests

## Package: org.apache.derbyTesting

- 1. Introduction

- 2. Quickstart

- 2.1 running tests

- 2.2 building derbyTesting package

- 3. More details on running the derby functional tests

- 3.1 Running 1 test

- 3.2 Running suites of tests

- 4. Harness internals for developers

- 4.1 Test types

- 4.2 Supporting files for tests

- 4.3 <testname>_app.properties

- 4.4 <testname>_derby.properties

- 4.5 tmp files, out files, master files, and canons

- 4.6 Masking and comparing

- 4.7 Adding a new test

- 4.8 Suites and adding a new suite

- 4.9 Running with a new jvm

- 4.10 Skipping a test

- 4.11 Frameworks

- 4.12 Some test harness properties

- 4.13 SecurityManager testing by default

- 4.14 Testing with Network Server on a remote host

- 4.15 Encoding Issues

- 5. Compatibility Tests

- 6. JDBC4 Tests

## 1. Introduction

This document describes functionality of the derby functional testing package org.apache.derbyTesting. This package is based on the functional tests in use at IBM for testing the Cloudscape product before its contribution to ASF.

In the following, instructions are geared towards a unix environment. For other environments, some details may need to be adjusted. For instance, the document may refer to $ANT_HOME, for DOS, this would be %ANT_HOME%.

In the following the top directory under which the subversion tree is placed is referred to as ${derby.source} - see also the derby BUILDING.txt.

The version of the classes and supporting files of the derbyTesting package have to match the version of the classes of the derby package. Thus you either need to build all jars yourself, or get all jar files from the Derby site at the same time when available.

## 2. QuickStart

### 2.1 running tests

The derbyTesting package enables you to run 1 test or a suite of tests. Before you can run, you need to setup your environment:

- Obtain a jdk or jre (based on jdk 1.3.1 specification See Note1 or higher). Add the bin directory to your $PATH. Currently supported are:

jdk131 - Sun HotSpot jdk1.3.1
jdk141 - Sun HotSpot jdk1.4.1
jdk142 - Sun HotSpot jdk1.4.2
jdk15 - Sun HotSpot jdk1.5
ibm131 - IBM Classic jdk1.3.1
ibm141 - IBM Classic jdk1.4.1
ibm142 - IBM Classic jdk1.4.2
j9_13 - WCTME jvm (available with IBM Websphere Studio Device Developer, 5.6), version 2.1
j9_22 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition, 5.7), version 2.2
j9_foundation - WCTME jvm (available with IBM Websphere Client Technology Micro Edition, 5.7), version 2.2, foundation library

- cd into a directory that does not have any colons or spaces in it.

- set $CLASSPATH to include the following jars:

  - jakarta-oro-2.0.8.jar oromatcher, obtain from http://jakarta.apache.org/oro/index.html, or follow this link for zip file, or tar.gz)

  - derbyTesting.jar

    test files and classes

  - derby.jar
    main derby package classes

  - derbytools.jar
    derby tools classes for tools like ij and dblook

  - derbynet.jar
    derby network server classes

  - derbyclient.jar
    derby client classes

  - db2jcc.jar and db2jcc_license_c.jar
    IBM Universal JDBC Driver classes. (See IBM developerworks for download)
    These jars are optional. The tests using the IBM Universal JDBC driver are not run if these jar files are not present in your classpath

  - derbyLocale_*.jar
    locale files holding translated messages.

  - junit.jar
    assertion-based test machinery.

For example:

  (note that $jardir is only a convenience variable and that the set CLASSPATH command below has carriage returns for formatting reasons):
  set jardir=/local/derbyjar
  set CLASSPATH="$jardir/derby.jar:$jardir/derbytools.jar:$jardir/derbynet.jar:$jardir/derbyclient.jar:
  $jardir/derbyTesting.jar:/local/derby/tools/java/jakarta-oro-2.0.8.jar:
  $jardir/derbyLocale_de_DE.jar:$jardir/derbyLocale_es.jar:$jardir/derbyLocale_fr.jar:
  $jardir/derbyLocale_it.jar:$jardir/derbyLocale_ja_JP.jar:$jardir/derbyLocale_ko_KR.jar:
  $jardir/derbyLocale_pt_BR.jar:$jardir/derbyLocale_zh_CN.jar:$jardir/derbyLocale_zh_TW.jar:
  $CLASSPATH
  set PATH=/local/jdk141/bin:$PATH

To run 1 test:

syntax:
java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunTest <testdir>/<testname>
where

- <testproperty> are test specific properties, such as 'framework' for the RunTest class.

- <testdir> is one of the directories under functionTests/tests where the actual test is located

- <testname> is the actual name of the test

examples:
to run the test supersimple against the embedded driver:
java org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql
To run a test with network server, using the derbyclient driver, add -Dframework=DerbyNetClient to the run. The test harness will to start network server at port 1527 or connect to a running one, run the test, and stop network server thereafter.
for example:
java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql

A successful run will have a .pass file, and the output to the console will show no difference between expected and actual test result. A failed test run will have at least a .fail file and the output to the console will show the difference between expected and actual result.

To run a suite:

syntax:
java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunSuite <testsuite>
where

- <testproperty> are test specific properties, such as 'verbose' for the RunSuite class.

- <testsuite> is one of the suites under org/apache/derbyTesting/suites

for example for running the suite derbylang:
java -Dverbose=true org.apache.derbyTesting.functionTests.harness.RunSuite derbylang

Each suite run should be started in a clean directory. The test output directory will not be emptied out before testing is begun, although individual test files and result files will be cleaned out and overwritten.

To run a suite with J2ME/CDC/Foundation:

(note that the command below has carriage returns for formatting reasons):
syntax:
<java> -Xbootclasspath/a:$jardir/<jdbc_jar> -Dbootcp=$jardir/<jdbc_jar>
-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource
-Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource"
org.apache.derbyTesting.functionTests.harness.RunSuite <testsuite>
where

- <java> is the jvm with any jvm flags. For testing with CDC/Foundation Profile using j9 jvm from IBM, this will be 'j9 -jcl:foun10'.

- <jdbc_jar> is the jar file which implements the JDBC API for CDC/Foundation Profile (JSR169).

- <testsuite> is one of the suites under org/apache/derbyTesting/suites

for example for running the suite derbyall:
j9 -jcl:foun10 -Xbootclasspath/a:$jardir/jdbc.jar -Dbootcp=$jardir/jdbc.jar
-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource
-Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource"
org.apache.derbyTesting.functionTests.harness.RunSuite derbyall

The suites provided are:

- derbylang:

  - basic functionality of language implementation in derby.

  - Mostly .sql type tests.

  - tested on a variety of hardware takes from 1.15m to 2.00 hours

- derbynetclientmats

  - basic network server tests using the derby client

  - variety of tests, including some from derbylang suite

  - tested on a variety of hardware takes from 15 to 30 minutes

- derbynetmats

  - basic network server tests using the IBM Universal JDBC driver

  - variety of tests, including some from derbylang suite

  - tested on a variety of hardware takes from 15 to 30 minutes

- derbynetautostart

  - tests network server functionality without requiring network server framework

- propertyinfo

  - runs test to get property information

- storeall

  - tests for storage area

  - includes:

    - storemats: most basic quick verification tests.

    - storemore: more extensive storage tests

    - storetests: set of store tests grouped together because they do not each need to create a new database

  - tested on a variety of hardware takes from 25 to 50 minutes

- xa

  - tests the xa implementation. There is both a storage and language element to these tests

  - tested on a variety of hardware takes from 2 to 4 minutes

- storeunit

  - tests store-related unit tests. Runs from 8 to 15 minutes

- unit

  - tests 4 general functionality unit tests. runs from 5 to 10 minutes

- jdbcapi

  - tests implementation of jdbc api such as Connection class implementation, Metadata etc.

  - takes from 20 to 40 minutes

- jdbc20

  - tests implementation of features from the jdbc 20 specification

  - takes 2 to 5 minutes

- jdk14

  - tests implementation of features from the jdk14 specification

  - takes 2 to 5 minutes

- demo, simpledemo

  - tests the SimpleApp example

  - simpledemo runs SimpleApp itself - and thus has a different default resource package name (namely, no package) than all the other tests. Hence it needed its own suite.properties file.

  - takes 30 to 1 minute

- nist

  - test obtained from the NIST SQL suite v 6.0

  - takes 5 to 10 minutes

- encryptionAll

  - takes 30 to 55 minutes

  - runs a few encryption tests plus the following encryption tests suites

    - encryption

      - runs the storemats, sysinfo and multi suites in encryption scheme DESede

      - takes 25 to 40 minutes

    - encryptionAES - tests AES encryption scheme

    - encryptionBlowfish - tests Blowfish encryption scheme

    - encryptionCFB - tests CFB encryption scheme

    - encryptionDES - tests DES encryption scheme

    - encryptionECB - tests ECB encryption scheme

    - encryptionOFB - tests OFB encryption scheme

- multi

  - runs a simple test case with 10 threads

  - runs for 10 minutes, then shuts down all threads

- derbytools

  - tests for dblook, ij, and import/export utilities

  - takes 5 to 10 minutes

- i18nTest

  - tests that characters outside simple ascii scope do not result in errors.

  - takes 5 to 10 minutes

- encodingTests

- runs tests with derbyTesting=UTF-16.

- takes 2 to 5 minutes

- derbyall

  - contains all suites typically run by all developers

  - tested on a variety of hardware takes from 3.00 - 6.00 hours

- largeData

  - Contains tests that deal with large amounts of data and thus require more machine resources. This suite is NOT run as part of 'derbyall' because the tests it contains require either 1) more machine resources than what the typical Derby developer might have, and/or 2) a significant amount of time to run, and thus shouldn't be run every night.

  - As tests are added to this quite, it could require more and more time to run (several minutes to several hours to several days), which is why it is NOT included as part of the derbyall suite. Currently the largedata/LobLimits.java test takes about 17Gb of disk space. On a linux machine with 2.8Ghz Intel Xeon CPU, 4Gb RAM , Linux machine and IBM 1.4.2 JVM with default memory heap size, the test ran for about 4.5 hrs. If the test is successful, it will cleanup the database and other files.

- See Note2

A successful run with all tests passing will have no *.fail files created, the <testsuite>_fail.txt file will be empty, and the <testsuite>_report.txt file will show no failures in the Summary results section.

-------snippet from derbylang_report.txt -----
-----------------------------------------------------------
Summary results:
Test Run Started: 2004-11-10 11:27:55.0
Test Run Duration: 00:04:09
129 Tests Run
100% Pass (129 tests passed)
0% Fail (0 tests failed)
0 Suites skipped

### 2.2 building derbyTesting package

To build the derbyTesting package:

- follow all the steps in the derby BUILDING.txt.

This is some typical output for the ant build process.

> cd /local/derby/java/testing
> ant.ksh
Searching for build.xml ...
Buildfile: /local/derby/java/testing/build.xml
compile:
[javac] Compiling 30 source files to /local/derby/classes
...
[copy] Copying 1 file to /local/derby/classes/org/apache/derbyTesting/funct
ionTests
BUILD SUCCESSFUL
Total time: 10 minutes 3 seconds

Building using the ant all target places all files, that is, classes, but also supporting files such as expected output (*.out), sql test files (*.sql), properties files and any data files used in individual tests into the classes directory so they can all be found using the CLASSPATH.

Once you have built the derbyTesting package, you can make a derbyTesting.jar using the jar build target at the ${derby.source}level.

This will look something like:

c:> ant derbytestingjar
Searching for build.xml ...
Buildfile: C:\derby\build.xml
initjars:
[mkdir] Created dir: C:\derby\jars\
[mkdir] Created dir: C:\derby\jars\lists
[echo] Revision number set to exported
[echo] .
derbytestingjar:
[echo] Beginning derbytesting.jar build
.....
BUILD SUCCESSFULL

## 3. More details on running the derby functional tests

The functional tests are run using a class called 'RunTest'. This class calls a number of other classes. A group of tests, called a 'suite' is executed using a class called 'RunSuite'.

### 3.1 Running 1 test

See section 2.1 for the basic steps to run 1 test.

To pass on system level properties to the test harness, use the test harness property -DtestSpecialProps. For example, to ensure extra information is appended to the log:

java -Dframework=DerbyNetClient -DtestSpecialProps=derby.infolog.append=true org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql

Tests will be executed in the current directory. When running a test using the network server and derby client, i.e. -Dframework=DerbyNetClient, the test will run in a subdirectory (automatically created) 'DerbyNetClient'.

The test will normally create the following:

- a database. The default name is 'wombat'. However, the name may be different depending on certain properties passed in to the test harness.

- a .out file: the final result file

- a .tmp file; the initial result file, before any modification to prevent irrelevant differences has been applied (before 'masking').

- a .diff file; the differences between the .out and the master file with expected output it is compared to.

- a .pass or .fail file. This file lists the test if it passes under .pass, and under .fail if the output in .out is different from the expected output in the master.

- .tmpmstr file. This file is a copy of the master file created in local encoding, and in the case of networkserver, massaged to eliminate irrelevant differences. This is the file the .out file is compared with.

possibly created:

- additional files used in a specific test may get copied over to the test directory. These normally do not get cleaned up.

- .err and .out files in network server database files for any additional error output.

- one or more database directories

- .utf8out file - see section on Encoding Issues.

When the test is successful, cleanup will occur unless the test harness property -Dkeepfiles=true is used. Cleanup will attempt to cleanup all files except for .pass.
See Note3.

A successful run (this example is from a dos environment) would look for instance like:

c:> derbyTesting.functionTests.harness.RunTest lang/supersimple.sql
C:\derby\run2
supersimple
-- listing properties --
derby.locks.deadlockTimeout=3
derby.locks.waitTimeout=3
*** Start: supersimple jdk1.4.2_03 2004-11-10 16:51:02 ***
The test should be running...
MasterFileName = master/supersimple.out
*** End: supersimple jdk1.4.2_03 2004-11-10 16:51:25 ***

A Test Failure shows the diff, creates a .fail file, does not create a .pass file, and does not cleanup any files upon completion. The output might look like this:

c:> derbyTesting.functionTests.harness.RunTest lang/supersimple.sql
C:\derby\run2
supersimple
-- listing properties --
derby.locks.deadlockTimeout=3
derby.locks.waitTimeout=3
*** Start: supersimple jdk1.4.2_03 2004-11-10 16:54:39 ***
The test should be running...
MasterFileName = master/supersimple.out
10 del
< 10
10a10
> 1
Test Failed.
*** End: supersimple jdk1.4.2_03 2004-11-10 16:55:02 ***

### 3.2 Running a suite of tests

See section 2.1 for a basic explanation on how to run a suite of tests.

Tests will be run in a subdirectory with the name of the test suite under the current directory. Eg. for derbylang suite, a directory derbylang will be created. While the tests are run, information about the run is inserted into a <testsuite>.sum file. When all tests have completed summary files are created <testsuite>_pass.txt, _fail.txt, and _diff.txt files are created as well as a <testsuite>_report.txt with additional details. Some of the information is duplicate. Also, a .skip file will be created holding a list of the tests that were skipped (for more details on this, see the section on skipping tests).

RunSuite does not empty the top level directory before running. Thus, if another suite was run in the same directory at an earlier time, the resulting summary files might contain results for more than the current run. Therefore it is important to run each suite in a clean directory.

Sample output from RunSuite:

c:> $ java org.apache.derbyTesting.functionTests.harness.RunSuite derbylang
Top suite: derbylang
Suite to run: derbylang:derbylang
Now do RunList
Now run the suite's tests
Run the tests...
Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtoprepo
rtdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/altertable.sql
Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtopreportdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/arithmetic.sql
...(.more tests)....
Generated report: derbylang_report.txt

This output does not show whether the tests passed or failed. The Summary section in <testsuite>_report.txt shows the statistics of the passed vs. failed tests, the summary <testsuite>_*.txt files list the tests that passed and failed.

## 4. Harness internals for developers

The following is intended for people who have the subversion tree available and want to add or modify tests.

The test harness executing one test basically does the following in sequence:

- identify test to run

- identify properties to run with

- copy needed support files

- find the expected output

- if network server, start network server

- run the test, creating the database

- if network server, shutdown the server

- modify the output based on Sed class and _sed.properties file for the test

- compare expected output with actual output

- if pass, cleanup.

### 4.1 Test types

The test harness recognizes, or will recognize tests with the following extensions:

- .java tests that run in a separate jvm.

- .sql tests that run using ij

- .sql2 related to .sql

- .multi multi threaded tests. There is currently only 1 test being run. The multi test functions a little differently from .java and .sql* tests in that RunTest starts a separate harness class called MultiTest to control the details of the run. Also, the actual test files live under org/apache/derbyTesting/functionTests/multi/stress, rather than org/apache/derbyTesting/functionTests/tests.

- .unit unit tests. The unit tests actually refer to <testname>_derby.properties files under org/apache/derbyTesting/functionTests/tests/unit that activate the actual unit test harness and tests under org/apache/derbyTesting/unitTests. These tests test more underlying functionality than the (rest of the) functionTests, which are more geared toward how end-users might use functionality.

- .junit junit tests. These require junit.jar to run and are actually run with junit.textui.TestRunner. The actual tests are .java files.

### 4.2 Supporting files for tests

Various additional files may be used by a test, for instance, to create large data values, to test using of jar files and the like. Any files that need to be accessed by a particular test that are not accessed from the classpath need to be listed under supportfiles= in the <testname>_app.properties file.
Tests can refer to classes without being in the classpath, and sql tests can use the ij command 'run resource ' to execute additional .sql files without changes to the _app.properties files.

For example, in the file (org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:
supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar

To support running on non-ISO-5889 systems, the harness copies files ending in sql, .view, .multi, .properties, .txt and .policy into local encoding; all other files are copied into UTF-8 encoding.

### 4.3 <testname>_app.properties

Every test directory has a default_app.properties. This file is for system level properties generic to all the tests in that test directory.

If a test requires different system level properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _app.properties

For example, for the test tools/dblook_test.java, there is a properties file called tools/dblook_test_app.properties

### 4.4 <testname>_derby.properties

Every test directory has a default_derby.properties. This file is for derby specific properties common to all the tests in that test directory.
If a test requires different derby properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _derby.properties

### 4.5 tmp files, out files, master files, and canons

The test's output will be put into a file testname.tmp. Then the output is modified if masking is required and the result is put into a .out file.
The expected output is found by examining the following directories, based on certain input

- functionTests/master/framework/jcc_version/jvmcode

- functionTests/master/framework/jcc_version/earlier_jvmcode

- functionTests/master/framework/jcc_version

- functionTests/master/framework/jvmcode

- functionTests/master/framework/earlier_jvmcode

- functionTests/master/jvmcode

- functionTests/master

For example, if we are running a test and the flag -Dframework=DerbyNet is used, to use network server and the IBM Universal JDBC Driver ('jcc'), and the jvm we are using is Sun's jdk 142, and the jcc version is 2.4 (not available at time of writing) then the search for the master to compare with starts in the functionTests/derbynet/jcc2.4/jdk14 directory. If a .out file with the same name as the test is found in that directory, that master is taken. If there is no such file in that directory, search continues in the directory functionTests/derbynet/jcc2.4/jdk13 if it exists.

If there is no file there, nor for any other jcc directory, it will continue to derbynet/jdk14, and the search is continued for earlier jvm versions.
If we are not running network server, the DerbyNet and jcc_version directories are not traversed.

The version details do not go into the subversion level, i.e. running with jdk141 or jdk142 is expected to have the same behavior.

This functionality supports dealing with minor differences in behavior caused by minor differences in behavior in the underlying jvms, jcc versions, differences between results returned through network server vs. embedded and minor differences between a debug and non debug (jar) build.

However, having a large number of these files means a maintenance problem. Every time test output changes due to modifications to derby or to the test, all output files in all directories need to be updated accordingly. If at all possible, irrelevant differences should be masked out, or the test should be written so that the output does not reflect such items.

Suggestions to minimize canons:

- create test specific masking

- ensure test data has a specific correct returned order; or an order by should be added to a query

- when writing java tests, ensure only pertinent output is reflected.

### 4.6 Masking and comparing

Tests often fail because of unimportant differences, such as process ids, statement ids, timestamps. The derby functional test harness provides for masking of these differences at 2 levels:

1. overall level. Masking required in all, or many tests can be achieved using the class Sed in the test harness directory. This class can either delete a reference present in the .tmp file from the .out file, or replace it with a generic string.

2. test specific level. To make masking for only one test, a (testname)_sed.properties file can be created which allows to either remove a string from the output or to replace it.

The diff is executed between the final resulting output and the master file found copied into local encoding as .tmpmstr.

### 4.7 Adding a new test

To add a new test:

- create the test file (e.g. newfunctest.java or newfunctest.sql) in the appropriate tests subdirectory

- list any files needed that are not .sql or .java files in a supportfiles entry in a test specific _app.properties file. e.g. newfunctest_app.properties: supportfiles=xyz.jar

- list any specific derby properties in a test specific _derby.properties file.

- add the properties files to the copyfiles.ant file in the test specific directory

- run the test. The first time around, the test will fail because no master file will be found.

- if the output is correct, copy it to the master directory. If you are running the on a non-ISO5889 encoded system, run the test with -DgenerateUTF8Out=true. This will create a .utf8out file in fixed UTF-8 encoding - use that file to copy into svn.

- Build. This will copy the newly created master into a matching directory structure in the classes directory and, when building jars, the master files will be included in the derbyTesting.jar, so it can be found by the harness at run time. Note that there is no copyfiles.ant file needed for the master directory, all .out files are automatically copied to the classes directory on build.

- run the test again. Investigate if any differences need to be masked out using a test specific sed.properties file (e.g. newfunctest_sed.properties). If so, ensure this is added to copyfiles.ant.

- ensure the test cleans up any testobjects created. This is important for running the tests in suites with a remote server, or with useprocess=false, because in those cases, tests run against databases in the same directory. Even if you do not anticipate your test to run in those configurations, it makes good practice. You may also want to protect your test from other tests leaving things behind by removing them before running also. The class org.apache.derbyTesting.functionTests.util.TestUtil has a cleanUpTest method that may be useful.

- ensure that connections can be made with all supported jvms. This includes the J2ME/CDC/JSR169 configuration, which only has java.sql.Datasource available, no java.sql.DriverManager. One can use the method ij.startJBMS() to get an appropriate connection, or TestUtil.getConnection() if one needs to specify connection properties different from what is defined in the <test>_app.properties file. For instance, to shutdown a database one can use:

TestUtil.getConnection("wombat","shutdown=true");

- ensure that the test does not cause any problems on non-ISO-5889 systems. See section 4.15.

- ensure the test does not have hard coded 'localhost' for network connections. Instead, use the method TestUtil.getHostName() to find if a hostName was specified for remote server testing.

- add the test to a specific suites/*.xml file, maintaining proper xml syntax.

- run the suite, and correct any problems found.

- make sure you svn add all new files, and that the svn:eol-style is native for all text files, and not for any binary files.

### 4.8 Suites and Adding a new suite

A suite constitutes of a <suitename>.properties file and/or a <suitename>.runall file in the org/apache/derbyTesting/functionTests/suites directory. The .properties files hold references to other .properties files, or .runall files, the .runall files are the actual lists of tests.

The lowest level suite always needs to have a .runall file.

For example, the derbyall suite is only a derbyall.properties file that refers to other suites in the 'suites' property:

-----------------------derbyall.properties--------------- suites=derbylang derbynetclientmats derbynetmats storeall xa derbytools derby.debug.true=enableBtreeConsistencyCheck derby.stream.error.logSeverityLevel=0

The derbylang suite is only a derbylang.runall, which lists the tests. The derbynetclientmats suite has both a .runall and a .properties file, so some additional properties can be specified that are true for all tests in that suite.

------------------derbynetclientmats.properties-----------------
framework=DerbyNetClient
suites=derbynetclientmats derbynetmats
jdk12test=true
runwithj9=false
timeout=60

To add a suite, you need to create at least a <suite>.runall file, which lists the actual tests, or a properties file that refers to other suites that do have a .runall file. The suite should be added into the directory ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.

### 4.9 Running with a new jvm

Currently, the supported jvms are:

jdk131 - Sun HotSpot jdk1.3.1 - class: jdk13
jdk141 - Sun HotSpot jdk1.4.1 - class jdk14
jdk142 - Sun HotSpot jdk1.4.2 - class jdk14
jdk15 - Sun HotSpot jdk1.5 - class jdk15
ibm131 - IBM Classic jdk1.3.1 - class ibm13
ibm141 - IBM Classic jdk1.4.1 - class ibm14
ibm142 - IBM Classic jdk1.4.1 - class ibm14
j9_13 - WCTME jvm (available with IBM Websphere Studio Device Developer, 5.6), version 2.1 - class j9_13
j9_22 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition, 5.7), version 2.2 - class j9_22
j9_foundation - WCTME jvm (available with IBM Websphere Client Technology Micro Edition, 5.7), version 2.2, foundation library - class j9_foundation

The classes above are subclasses of org.apache.derbyTesting.functionTests.harness.jvm. The name at the front is just a convention.

To run a test with a jvm that does not have a matching class under org.apache.derbyTesting.functionTests.harness, do the following:

- just run the tests as if there is a jvm class. The harness will default to using the jdk14 class. Unlikely, but possibly there are no differences

- if there are failures showing that cannot be explained any other way but genuine, acceptable jvm differences, do the following:

  - create a subclass of org.apache.derbyTesting.functionTests.harness.jvm. In this class, specify any jvm specific property settings required

  - compile the new jvm class and run the tests

  - create a new canon directory for any additional canons that need to be created.

  - in rare occasions, other harness changes may be required

  - for any tests that should not run with this environment, add a line in the testname_app.properties file indicating this. For instance to add a line for a jvm called jdk29, it would be like this: runwithjdk29=false. Note that the versioning does not currently extend past 2 digits. For j9 jvms, versioning does not apply currently. For all j9 versions, use runwithj9=false. For j9_foundation, use runwithfoundation=false.

  - Add code in RunTest.java to switch to the new jvm based on values for system and vendor properties

### 4.10 Skipping a test

There are 2 skipping mechanisms in place for different kinds of skipping of a test.

Some tests are written to test specific functionality only available with for instance certain jvms, or, with network server, certain versions of the IBM Universal Driver. To control this, properties can be set for each test, for instance, if a test should not be run when using an ibm jvm, set runwithibmjvm=false. If a test should be run with Sun Hotspot jvm version 14, then set runwithjdk14=true.
The skip setting does not go into the subversion level, i.e. setting runwithjdk141=false has no effect, and setting runwithjdk14 affects runs with jdk141 as well as jdk142.
Other skip reasons are encryption protocols specific to a certain jvm.

The property for skipping a test based on the version of the IBM Universal Driver is "excludeJCC". The keywords "at-or-before" and "at-or-after" can be used to specify which range of JCC versions should be excluded. If neither of these keywords is provided, the default is "at-or-before". For example:

To skip a test when running with any version of the IBM Universal Driver that is 2.4 or earlier:
excludeJCC=at-or-before:2.4
To skip a test when running with any version of the IBM Universal Driver that is 2.0 or later:
excludeJCC=at-or-after:2.0

You can also specify an (optional) jvm clause to further tune the exclusion criteria. This clause starts with the ",when" tag and is followed by a three-part jvm version. In this case, a test will only be skipped if BOTH the JCC clause AND the jvm clause are true. For example:

To skip a test when running with any version of the IBM Universal Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:
excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1

To skip a test when running with any version of the IBM Universal Driver that is 2.0 or earlier, but ONLY if the jvm is 1.5 or later:
excludeJCC=at-or-before:2.0,when-at-or-after:jdk1.5.1

Another skipping mechanism works on entire 'frameworks'. Currently there are only 3 supported in the harness, embedded, network server with the derbyclient driver ('DerbyNetClient') and network server with the IBM Universal JDBC Driver ('DerbyNet'). In the suites directory there are .exclude files for each of the frameworks - if for some reason an exclude file were not there, you would see a warning message show up for every test run. In this 'framework'.exclude file tests can be placed that for some reason need to be excluded from running with that framework. This mechanism enables adding of suites to a framework run even if a few of the tests are not appropriate for that particular framework.

Note that at this time, only skipped suites show up in the .skip result file. This still needs to be corrected.

### 4.11 Frameworks

Currently, there are two frameworks used for network server, DerbyNetClient, which uses the derby Client driver, and DerbyNet, which uses the IBM Universal JDBC Driver.
Setting the framework property will invoke the test harness class NetServer which has the actual configuration (driver name, portnumber etc.) used for the individual frameworks. See Note4.
Setting this framework also causes the search for expected output to include appropriate DerbyNetClient or DerbyNet and jcc version specific subdirectories under the master directory.

### 4.12 Some test harness properties

For a complete set, refer to comments in RunTest.java, but here are some valuable test properties which can be passed to the RunTest class:

- runwith<jvm> - See above section 4.10
- framework - specifies which framework to run with. For example:
  java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.RunTest
  lang/supersimple.sql
- verbose - Shows more detailed output if set to "true". For example:
  java -Dverbose=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql
- keepfiles - Indicates to not clean up any of the files if the test passed if set to "true".
- testSpecialProps - sets additional properties. Several can be set using '^' as separator:
  -DtestSpecialProps=<prop-1>=<value-1>^ ... ^<prop-n>=<value-n>. For example:
  java -DTestSpecialProps=derby.infolog.append=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql
- jvmflags - sets specific jvm properties for each jvm instantiated by the test harness, for instance for setting initial and maximum heap size (note that setting only initial size or only max size may cause incompatibilities with heap settings picked up from harness property files or from the command line), or other properties that need to be passed on to the jvm, separated by '^'. This property can be set on the commandline for either RunTest or RunSuite, or in one of the properties files used by the tests. The property jvmflags when passed on at the command line supercedes the properties set in a suite's propertyfile, and those supercede jvmflags properties set in a subsuite's or test's propertyfile, because the 'highest' level properties are passed to the jvm last. Setting this property for a test that runs with useprocess=false cannot have any effect. Example:
  java -Djvmflags=-Xms32M^-Xmx128M org.apache.derbyTesting.functionTests.RunTest lang/streamingColumn.java
- excludeJCC - See above section 4.10
- useprocess - (default=true) Controls whether RunTest runs the test in a separate VM or in a thread in harness VM. Also does not create subdirectories for each test and thus will attempt to reuse databases with the same name. It is potentially useful for debugging tests. Unit tests are not (yet) runnable with "useprocess=false".
- startServer - allows for Network Server tests to start and shutdown Network Server from the test itself. Default is true - i.e. the test harness will start Network Server.
- noSecurityManager – disable the client JVM from installing a SecurityManager. See section 4.13

- derbyTesting.replacePolicyFile - replace or append the contents of the default policy file derby_tests.policy. default is false, i.e. append. See section 4.13

- hostName - allows for running Network Server on a remote host. See section 4.14.

- derbyTesting.encoding - allows for running the harness with a different encoding. Only supported with jdk15. Example:
  java -DderbyTesting.encoding=UTF-16 org.apache.derbyTesting.functionTests.harness.RunTest jdbcapi/resultset.java
- generateUTF8Out - copies the test .out into UTF-8 encoding. See section 4.15.
- derbyTesting.jar.path - This property is required to run upgrade tests. Set this property to point to the location of jar files from a previous release from which we need to test upgrade. This property needs to be set only if the source files will not be available when running tests. In this case, the jars can be copied to any location with the following relative path ${jar_location}/{majorversion.minorversion}. The property should be set as follows:
  derbyTesting.jar.path=${jar_location}
  If the test is run on the machine where the source files from svn have been built, then it is not required to set this property. The jars checked into svn will be used. If the tests are being run in a location where the jars from the previous release are available in a different location than where the source files for the upgrade tests have been built, the following property needs to be set on the command line for RunSuite or RunTest where the tests are being run:
  -Djvmflags=-DderbyTesting.jar.path={path_to_derby_jars}
  where {path_to_derby_jars} points to a directory where the jars for the previous Derby release being tested for upgrade reside in a subdirectory named according to the major.minor version. For example:
  If the Derby 10.1 jars for upgrade testing reside in /opt/testing/derby/10.1 then the command line needed for the upgrade tests to complete would include the following:
  java -Djvmflags=-DderbyTesting.jar.path=/opt/testing/derby org.apache.derbyTesting.functionTests.harness.RunSuite upgrade
  Please note that if you change the value of derbyTesting.jar.path you will need to remove the file org/apache/derbyTesting/functionTests/tests/upgradeTests/Upgrade_10_1_10_2.properties from your output directory and run 'ant all' for the updated value of the property to be written into that generated file.

###
4.13 SecurityManager testing by default

By default the tests install the standard Java SecurityManager using the system property java.security.manager and use a policy file derby_tests.policy. The default file is sourced at
${derby.source}/java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy

During a test run this policy file is copied into ${user.dir} and used from there.
Optionally a test-specific or suite-specific policy file may be appended or used instead. The name of the policy file remains derby_tests.policy.

A test_specific policy file has the same name as the test, has extension .policy, and is located in the same location as the test. For instance, ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/tests/lang/errorStream.policy is the test-specific policy file for the test errorStream.java. A suite-specific policy file has the same name as the suite, has extension .policy, and is located in the ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites directory.

By default, a test-specific and/or suite-specific policy file is appended to the default policy file (derby_tests.policy). However, if a test's _app.properties or suite's .properties file contains the property derbyTesting.replacePolicyFile=true, then the file derby_tests.policy will be overwritten with the contents of the test- or suite-specific policy file.

There are two environments for the installation of the SecurityManager.

- Server JVM for the network client tests. Always uses a SecurityManager.

- Client side JVM, this is the JVM executing the JDBC calls against any Derby driver. Installs a SecurityManager unless:

- noSecurityManager=true in the test's _app.properties file.Used to disable individual tests that cannot run under the security manager, or if the test has a functional requirement not to run with the SecurityManager. Ideally, few tests will have this property set. Any disabling of the SecurityManager for a test requires a comment in the test's _app.properties file indicating why the test cannot run under the SecurityManager.

- Client JDBC driver is DB2's Universal driver for JDBC. Currently does not install a SecurityManager, no technical reasons, just has not been done.

###
4.14 Testing with Network Server on a remote host

To enable testing against a Network Server running on a remote machine - which is for instance needed to test on IPV6 machines - one can use the property hostName.
By default the network server is assumed not to be running, and the test harness will start Network Server on the local machine for each test - unless the test has been qualified explicitly with the property startServer to false. In that case, the test itself is assumed to be starting Network Server. (This can be set either in a test's property file or with the -D on the commandline). Thus, by default Network Server is started and shutdown in the subdirectory for the framework created for each test, and new databases are created in each test directory (e.g. ./getCurConnJdbc20/wombat). The test policy file also gets copied to the appropriate location.
When one wants to run against a Network Server running on a remote machine, the test harness cannot start or stop it because of security reasons. Thus, the setting of the property hostName automatically sets startServer to false. The hostName is automatically passed on to the actual test run as the property ${derbyTesting.serverhost} which is used in the derby_tests.policy file on the client machine.
The hostName property is not a pre-set value in any suite, some manual steps are required to run tests this way:

1. Extract and/or copy the (java/testing/org/apache/derbyTesting/functionTests/util)derby_tests.policy from the derbyTesting.jar or from the svn source to a location on the host on which you want to run the NetworkServer
2. Set up the classpath for Network Server to include derbynet.jar and derbyTesting.jar (or equivalent classes). You need to have derby.jar available also, although it does not need to be in the classpath.
3. Start Network Server like so when using compiled classes for it:

java -Djava.security.manager -Djava.security.policy=copied_derby_tests.policy -DderbyTesting.codeclasses=file:/derby/classes/ -DderbyTesting.codedir=/derby/classes -DderbyTesting.serverhost=localhost -DderbyTesting.clienthost=clientIPaddress -DderbyTesting.codejar=file://unused/ org.apache.derby.drda.NetworkServerControl -h 0.0.0.0 start
where /derbyt/classes is the directory where the classes can be found, and clientIPaddress is your client machine's IP address or name. In this example, derby.system.home is in the directory where Network Server is started, and the derby_tests.policy file is in that directory too.

or like so when using jars:

java -Dderby.system.home=/home/derbytst/ns -Djava.security.manager -Djava.security.policy=/home/derbytst/ns/copied_derby_tests.policy -DderbyTesting.clienthost=clientIPaddress -DderbyTesting.codejar=file:/derby/tstbld/ -DderbyTesting.codedir=/derby/tstbld -DderbyTesting.serverhost=localhost -DderbyTesting.codeclasses=file://unused/ org.apache.derby.drda.NetworkServerControl -h 0.0.0.0 start
where /derbyt/tstbld is the directory where derbynet.jar is found and derby.system.home points to the directory where your derby.properties are and where you want to have derby.log and any databases go, and clientIPaddress is your client machine's IP address or name.

4. On the client machine, start the tests as usual, but specifying hostName, like so:

java -DhostName=favorite.remotehost.com org.apache.derbyTesting.functionTests.harness.RunSuite derbynetclientmats where favorite.remotehost.com is the host name or ip address.

RunTest can pick up the property in a similar way. You can set -Dframework as usual. You can also run derbynetmats with jcc using -DhostName flag; in that case you do not need to worry about running with a policy file - at the moment, tests run with DerbyNet framework do not run with SecurityManager. Tests in the derbynet(client)mats suite have been modified to run against a remote server as much as possible.
When the test harness encounters the -DhostName, and the value is not null and not localhost, it switches to startServer=false. Also, against a remote host, only the ping operations of the NetworkServerControl is supported. Finally, because on the remote server no new directories can be created by the test harness because of security, databases will be reused. This has a number of implications:

1. the test harness, or individual tests cannot start and stop Network Server.
2. the tests cannot check derby.log, nor data files created on the server by import/export or dblook utilities.
3. as the database location is the same for each test, tests run this way need to be diligent about cleaning up after or will cause failures in later tests. The .java tests can use the TestUtil.cleanUp method to drop any database object without any output being reflected. Test writers need to ensure that the objects are in fact getting dropped or the test needs to be excluded from running remotely. To facilitate cleanup, a method was added to org.apache.derbyTesting.functionTests.util.TestUtil, cleanUpTest(Statement, testObjects[]). This method drops anything passed in the array of strings and disregards any SQLExceptions. The benefit of this is that a possible failure of drop does not stop the test, and does not prevent other objects passed in the string to be dropped. An example of usage of this method:

   static void teardown() throws SQLException {
   Statement stmt = conn.createStatement();
   String[] testObjects = { "table t1", "table t2", "procedure p1", "schema s restrict"};
   TestUtil.cleanUpTest(stmt, testObjects);
   conn.commit();
   stmt.close();
   }

   Note that if something prohibits an object from being dropped, e.g. if there still are locks on it, this will not be noticed with this method - the test writer should still verify that the test objects get cleaned up appropriately.
4. tests cannot use a mix of Network Server and local connections to the same database.
5. tests should not hardcode the string 'localhost' or they cannot be run in this fashion. Tests that need to be run in this fashion can use the result of the TestUtil.getHostName() method instead.
6. Tests can only use the ping option of the NetworkServerControl class.
7. Tests cannot be dependent on test- or suite-specific policy files.

New .exclude files for the current frameworks (i.e. DerbyNetRemote.exclude and DerbyNetClientRemote.exclude) have been created to skip some tests that cannot be run against a remote server because of those implications.

###

### 4.15 Encoding issues

There are three aspects to dealing with encoding issues, how to run a test with specific encoding, what to do to ensure a test can be run without relying on a certain encoding, and how to generate test output in fixed encoding.

- #### Run a test with a specific encoding

To force a specific test to run with a specific encoding one can use the property derbyTesting.encoding. This is currently only supported with jdk15. For instance, to run a specific .java test one time only with -DderbyTesting.encoding:

java -DderbyTesting.encoding=UTF-16 org.apache.derbyTesting.functionTests.harness.RunTest jdbcapi/lobStreams.java

The test encodingTests contains tests that run with derbyTesting.encoding=UTF-16 in the encodingTests.properties file:

derbyTesting.encoding=UTF-16

- #### ensure running on non-ISO-5889 systems is possible

Special care is required to ensure that individual tests can be run successfully on non-ISO-5889 systems. This includes:

- Ensure reading of files is done in an appropriate manner. The test harness will copy supportfiles with extension .*sql, .multi, .view, .txt, .properties, and .policy in local encoding. This means, that accessing such a file should be done with an appropriate mechanism; i.e. from within ij (i.e. a .sql test) use 'run', and from within a .java test use FileReader. The test harness will copy other supportfiles in fixed encoding, i.e. using UTF-8. To access these files and you will have to use an input mechanism that supports specifying of the encoding, such as FileInputStream and BufferedReader. Various .sql tests that run other .sql scripts use the test-specific mechanismn in ij to access files from the classpath: 'run resource'. This construct assumes the files listed are in the classpath - the classes dir or derbyTesting.jar - and they are assumed to be in UTF-8.

- Avoid usage of non-encoding safe constructors for Strings. i.e. Avoid String(byte[]) and String(byte[],int,int) to create a string.

- #### Generating master files on non-ISO-5889 systems

- Special care is required when developing tests and running them on non-ISO-5889 systems. The output files cannot be checked into a svn tree which is assumed to be ISO-5889 compatible. To avoid surprises, use the property -DgenerateUTF8Out which will create a .utf8out file which is a copy of the test output converted into UTF-8. Be careful not to let a file transfer program modify this file (for instance. ftp in ascii mode may try to convert the file).

  java -DgenerateUTF8Out=true org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql

## Notes

Note1:

A number of tests require packages that are optional with jdk13, but included with later jvms: jdbc20ext.jar and jta1_2.jar. When running with jdk131, these need to be downloaded separately & placed in the jre/lib/ext directory of your jvm installation.

Note2:

There is one more suite included: the j9derbynetmats suite is a modification of the derbynetmats suite. It is available to test the network server with the jvm available with IBM's WCTME (Workplace Client Technology, Micro Edition; formerly WSDD), and will be run at IBM. Note that the setup for running the j9derbynetmats tests is very specific to the test harness, not even using the WCTME files in their normal location.
The j9derbynetmats suite is included to serve as an example of splitting the network server process to run with a different jvm than the test client. The j9derbynetmats suite will run with another jvm as client (as defined in the suite properties), but start up network server with the 'j9' jvm files (the reference to 'j9' is based on the executable, j9.exe), based on the property 'serverJvm'. Running this suite requires providing the property bootcp, which is interpreted from the test harness class j9_13. See also section on adding a new jvm setup.

Note3:

Occasionally, cleanup is unsuccessful. This does not constitute a problem in any way, as the harness for most suites starts with a clean database, and clean copies of all files. However, you will see something like this in the output:
Warning: Cleanup failed on baseDir: /local/myrun1/DerbyNet/supersimple.

Note4:

NetServer also has a configuration for connecting to DB2 via jcc - the IBM Universal Driver - and via the older DB2 driver. But there are currently no tests to exercise these settings.