Performance comparison of JITs (Sep 2000)

last-modified: Oct 11 18:21, 2000 JST

This page shows results of performance comparisons among Java runtime systems on Linux.
The old comparison page is here.

The followings runtime systems were evaluated.

IBM JDK 1.3 (build cx130-20000815)
- JITC 3.6 included
IBM JDK 1.1.8 (build l118-20000713)
- JITC 3.0 included
Sun JDK 1.3 (build 1.3.0rc1-b17)
- HotSpot Client VM
- HotSpot Server VM
Kaffe 1.0.6
- JIT v3 included
Blackdown JDK 1.2.2 FCS
- sunwjit.so
  (included in this JDK)
- JBuilder Java 2 JIT 1.2.15
  (developed by Inprise Corp.)
- OpenJIT 1.1.14
- TYA 1.7v2
- shuJIT 0.6.7
- Interpreter
GCJ (in gcc 2.95.2)

I applied the following benchmarks to the runtime systems.

SPEC JVM98
SciMark 2.0
Linpack benchmark
Eratosthenes Sieve

All results are also available in Mirosoft Excel book format: `JIT-eval-001010.xls'.

SPEC JVM98

SPEC JVM98 (http://www.spec.org/osg/jvm98/) is a benchmark suite contains real Java applications.
This benchmark consists of

Modified Lampel-Ziv method (compress),
Java Expert Shell System based on NASA's CLIPS expert shell system (jess),
Database functions performed on memory resident database (db),
Java compiler from the JDK 1.0.2 (javac),
MPEG Audio Layer-3 decoder (mpegaudio),
Ray-tracer where two threads work (mtrt),
and Java parser generator which is now called JavaCC (jack).

The following results were obtained with K6-2 / 400 MHz processor and Linux 2.2.16. These are scores, so higher is better.

Maximum and minimum heap size were set as 32 MB, because the machine has only 64 MB memory. However the Kaffe needs more than 32 MB as heap size. The _201_compress and _213_javac benchmarks stop its execution with OutOfMemoryError if only 32 MB memory is given. I supplied 48 MB to the Kaffe.

Runtime system _227_mtrt _202_jess _201_compress _209_db _222_mpegaudio _228_jack _213_javac Geometric Mean

IBM JDK 1.3 26.0 23.7 14.8 12.4 33.5 30.6 13.2 20.5

HotSpot Server VM 24.2 27.4 13.9 9.53 36.5 35.9 11.6 19.2

HotSpot Client VM 16.1 22.6 13.0 9.19 20.5 29.3 11.3 16.2

IBM JDK 1.1.8 11.7 11.7 13.1 7.56 23.8 16.2 7.20 12.1

OpenJIT 8.13 8.52 10.3 3.35 11.1 8.64 4.31 7.18

JBuilder JIT 8.88 8.32 10.4 3.45 13.9 2.74 3.81 6.28

shuJIT 5.23 6.86 8.71 3.97 10.8 6.78 3.44 6.10

TYA 5.79 6.69 7.67 3.39 8.75 6.66 3.69 5.79

sunwjit 2.30 2.90 9.26 N/A (cannot complete) 7.94 5.82 3.09 4.55

Kaffe 3.27 2.46 16.3 4.28 7.66 2.15 2.72 4.25

interpreter 2.60 2.68 1.97 1.71 2.19 2.34 1.99 2.19

SciMark 2.0

SciMark2 (http://math.nist.gov/scimark2/) is a benchmark suite based on numerical kernels. They were collected by NIST researchers.

The results were produced by Pentium II / 333 MHz and Linux 2.4.0. These are scores, so higher is better.

Runtime system FFT (1024) SOR (100x100) Monte Carlo Sparse matmult (N=1000, nz=5000) LU (100x100) Composite Score

IBM JDK 1.3 36.6 91.5 9.85 44.5 85.7 53.6

IBM JDK 1.1.8 36.5 84.3 7.04 41.8 58.8 45.7

HotSpot Client VM 18.4 84.7 8.32 30.0 50.7 38.4

HotSpot Server VM 19.2 71.3 10.1 24.1 50.7 35.1

GCJ (-O2 with .class) 17.1 42.1 2.29 20.8 26.4 21.7

OpenJIT 8.69 32.7 1.94 10.6 19.4 14.7

JBuilder JIT 10.1 19.7 2.02 12.9 16.8 12.3

shuJIT 6.14 26.5 1.99 11.9 11.3 11.6

sunwjit 9.19 32.9 1.13 10.9 3.00 11.4

Kaffe 5.39 23.5 2.16 10.8 13.1 11.0

GCJ (-O0 with .class) 6.53 20.8 1.63 11.1 10.2 10.0

TYA 4.21 14.7 1.59 8.97 11.9 8.27

interpreter 1.21 3.69 0.642 2.36 2.95 2.17

Linpack benchmark

Linpack Benchmark -- Java Version (http://www.netlib.org/benchmark/linpackjava/).

A Pentium II / 333 MHz processor and Linux 2.4.0 perform as the followings. These values are in mega floating-point operations per second (Mflops), so higher is better.

The scores by runtime systems (i.e. IBM JDK and HotSpot Server VM) which utilize adaptive optimization techniques are not good with the 200x200 smaller problem. I guess that adaptive optimizations could not be applied well at the end of the benchmark, because the Linpack benchmark runs for a short time. ShuJIT does not perform such optimizations in its default setting.

500 x 500 system

IBM JDK 1.3 23.675

IBM JDK 1.1.8 17.994

HotSpot Server VM 16.783

HotSpot Client VM 15.035

GCJ (-O2 with .java) 14.284

GCJ (-O2 with .class) 13.813

shuJIT 11.277

JBuilder JIT 10.734

Kaffe 10.584

OpenJIT 1.1.14 9.455

TYA 9.449

GCJ (-O0 with .java) 9.18

GCJ (-O0 with .class) 7.96

sunwjit 2.362

interpreter 2.361

200 x 200 system

GCJ (-O2 with .java) 26.41

GCJ (-O2 with .class) 24.524

HotSpot Client VM 16.349

IBM JDK 1.1.8 14.928

shuJIT 14.306

OpenJIT 1.1.13 13.464

GCJ (-O0 with .java) 12.716

TYA 11.444

JBuilder JIT 11.257

OpenJIT 1.1.14 10.899

Kaffe 9.81

GCJ (-O0 with .class) 9.671

IBM JDK 1.3 5.675

HotSpot Server VM (with -Xbatch opt.) 3.54

interpreter 2.384

sunwjit 2.266

Eratosthenes Sieve

The TYA archive includes this benchmark as `demo/Sieve.java'.

These results were produced by Pentium 333 MHz and Linux 2.4.0. These values are operations per 10 seconds, so higher is better.

IBM JDK 1.3 2093

IBM JDK 1.1.8 1879

HotSpot Server VM (with -Xbatch option) 1169

JBuilder JIT (JAVA_COMPILER_THRESHOLD set as `0') 1076

Kaffe 1140

HotSpot Client VM 914

OpenJIT 809

sunwjit 755

shuJIT 629

TYA 586

interpreter 124

Related Resources

The Java Performance Report: http://www.javalobby.org/features/jpr/
Java Grande Forum: http://www.javagrande.org/
Java Numerics (from NIST): http://math.nist.gov/javanumerics/
validgh Java numerics page: http://www.validgh.com/java/
Java in Science and Engineering list, Java Grande Forum list: http://mailer.csit.fsu.edu/mailman/listinfo

Runtime system	_227_mtrt	_202_jess	_201_compress	_209_db	_222_mpegaudio	_228_jack	_213_javac	Geometric Mean
IBM JDK 1.3	26.0	23.7	14.8	12.4	33.5	30.6	13.2	20.5
HotSpot Server VM	24.2	27.4	13.9	9.53	36.5	35.9	11.6	19.2
HotSpot Client VM	16.1	22.6	13.0	9.19	20.5	29.3	11.3	16.2
IBM JDK 1.1.8	11.7	11.7	13.1	7.56	23.8	16.2	7.20	12.1
OpenJIT	8.13	8.52	10.3	3.35	11.1	8.64	4.31	7.18
JBuilder JIT	8.88	8.32	10.4	3.45	13.9	2.74	3.81	6.28
shuJIT	5.23	6.86	8.71	3.97	10.8	6.78	3.44	6.10
TYA	5.79	6.69	7.67	3.39	8.75	6.66	3.69	5.79
sunwjit	2.30	2.90	9.26	N/A (cannot complete)	7.94	5.82	3.09	4.55
Kaffe	3.27	2.46	16.3	4.28	7.66	2.15	2.72	4.25
interpreter	2.60	2.68	1.97	1.71	2.19	2.34	1.99	2.19

Runtime system	FFT (1024)	SOR (100x100)	Monte Carlo	Sparse matmult (N=1000, nz=5000)	LU (100x100)	Composite Score
IBM JDK 1.3	36.6	91.5	9.85	44.5	85.7	53.6
IBM JDK 1.1.8	36.5	84.3	7.04	41.8	58.8	45.7
HotSpot Client VM	18.4	84.7	8.32	30.0	50.7	38.4
HotSpot Server VM	19.2	71.3	10.1	24.1	50.7	35.1
GCJ (-O2 with .class)	17.1	42.1	2.29	20.8	26.4	21.7
OpenJIT	8.69	32.7	1.94	10.6	19.4	14.7
JBuilder JIT	10.1	19.7	2.02	12.9	16.8	12.3
shuJIT	6.14	26.5	1.99	11.9	11.3	11.6
sunwjit	9.19	32.9	1.13	10.9	3.00	11.4
Kaffe	5.39	23.5	2.16	10.8	13.1	11.0
GCJ (-O0 with .class)	6.53	20.8	1.63	11.1	10.2	10.0
TYA	4.21	14.7	1.59	8.97	11.9	8.27
interpreter	1.21	3.69	0.642	2.36	2.95	2.17

IBM JDK 1.3	23.675
IBM JDK 1.1.8	17.994
HotSpot Server VM	16.783
HotSpot Client VM	15.035
GCJ (-O2 with .java)	14.284
GCJ (-O2 with .class)	13.813
shuJIT	11.277
JBuilder JIT	10.734
Kaffe	10.584
OpenJIT 1.1.14	9.455
TYA	9.449
GCJ (-O0 with .java)	9.18
GCJ (-O0 with .class)	7.96
sunwjit	2.362
interpreter	2.361

GCJ (-O2 with .java)	26.41
GCJ (-O2 with .class)	24.524
HotSpot Client VM	16.349
IBM JDK 1.1.8	14.928
shuJIT	14.306
OpenJIT 1.1.13	13.464
GCJ (-O0 with .java)	12.716
TYA	11.444
JBuilder JIT	11.257
OpenJIT 1.1.14	10.899
Kaffe	9.81
GCJ (-O0 with .class)	9.671
IBM JDK 1.3	5.675
HotSpot Server VM (with `-Xbatch` opt.)	3.54
interpreter	2.384
sunwjit	2.266

IBM JDK 1.3	2093
IBM JDK 1.1.8	1879
HotSpot Server VM (with `-Xbatch` option)	1169
JBuilder JIT (`JAVA_COMPILER_THRESHOLD` set as `0')	1076
Kaffe	1140
HotSpot Client VM	914
OpenJIT	809
sunwjit	755
shuJIT	629
TYA	586
interpreter	124