Performance comparison of JITs (Jan 2002)

last-modified: Feb 6 11:55, 2002 JST

Back to:

Performance comparison pages:


This page shows results of performance comparisons among Java runtime systems on Linux.

Applied benchmarks:

Evaluated Java runtime systems:

The computer that run all benchmark programs is as follows:

All results are also available in Mirosoft Excel book format: `JIT-eval-020120.xls'.


SPEC JVM98

SPEC JVM98 (http://www.spec.org/osg/jvm98/) is a benchmark suite consists of real Java application programs:

SPEC JVM98 shows scores for each benchmark program and the geometric mean of them. Higher number is better.

Fairness of results

The results of ORP, Jikes RVM and Kaffe are not reportable to SPEC because these runtime systems read benchmark programs and data from a local filesystem. Reporatable results have to be produced by the benchmark which run as a Java applet. ORP and Jikes RVM cannot run a Java applet and Kaffe cannot run the SPEC tools as an applet loaded from a web server. _201_compress and _213_jack benchmarks are very sensitive to where the programs and data are read from. Thus the results of these two benchmarks cannot be compared fairly among all runtime systems.

Heap size

Maximum and minimum heap size were set as 32 MB. However Kaffe needs more than 32 MB as heap size. The _201_compress and _213_javac benchmarks stop its execution with OutOfMemoryError if only 32 MB memory is given. We supplied 48 MB to Kaffe. We also supplied 48 MB to ORP only for _209_db benchmark because ORP cannot complete the benchmark with 32 MB heap. Jikes RVM run this benchmark with 128 MB heap for small objects and 24 MB heap for large objects (-X:h=128 -X:lh=24).

SPEC JVM98 results on Pentium 4 [953x556]
Runtime system_227_ mtrt_202_ jess_201_ compress_209_ db_222_ mpegaudio_228_ jack_213_ javacGeometric Mean
HotSpot Server VM 14716011.721.716715015.158.2
ORP (O3 JIT) 73.468.596.320.616169.419.457.9
ORP (O1 JIT) 57.364.582.720.014370.414.851.1
IBM JDK 1.3 10778.311.731.617910415.050.8
Jikes RVM (FastAdaptiveSemispace) 47.057.774.426.577.963.133.350.8
HotSpot Client VM 76.011710.823.011412613.746.0
Jikes RVM (FastSemispace) 56.047.875.517.591.046.424.945.0
Jikes RVM (OptOptSemispace) 32.721.764.315.661.922.511.127.2
OpenJIT 33.633.010.910.961.037.47.1721.6
shuJIT 20.929.310.311.156.833.17.0019.1
Kaffe 15.211.673.914.436.09.9111.718.7
TYA 15.726.79.719.4034.329.46.8416.0
sunwjit.so 8.0112.110.6N/A43.327.26.714.2
interpreter 11.611.94.886.597.6217.08.939.09

SciMark 2.0

SciMark 2.0 (http://math.nist.gov/scimark2/) is a benchmark suite consists of numerical kernels. They were collected by NIST researchers.

We specified the `-large' option to run the large problem size version. SciMark 2.0 shows results in MFlops (mega floating-point operations per second) and the composite score, which is the arithmetic mean of them. Higher number is better because results were in MFlops. Therefore, SOR, which shows relatively higher value, affects the composite score. Oppositely, FFT and Monte Carlo are not counted very much.

SciMark 2.0 results on Pentium 4 [996x572]
Runtime systemFFT (1048576)SOR (1000x1000)Monte CarloSparse matmult (N=100000, nz=1000000)LU (1000x1000)Composite Score
HotSpot Client VM 19.624928.688.4169111
GCJ (-O2 with .java) 18.92296.56138153109
GCJ (-O2 with .class) 18.72327.28119158107
IBM JDK 1.3.0 19.914520.3158167102
ORP (O3 JIT) 11.315227.110760.5715
HotSpot Server VM 20.215735.920.595.665.8
Jikes RVM (FastSemispace) 11.21323.3576.254.555.4
Jikes RVM (OptOptSemispace) 11.01283.3272.653.253.6
ORP (O1 JIT) 10.311622.954.742.649.3
Jikes RVM (FastAdaptiveSemispace) 7.911303.4272.516.246.0
GCJ (-O0 with .class) 10.874.25.2747.044.936.4
sunwjit.so 16.280.04.0867.210.435.6
OpenJIT 6.7770.45.3465.013.832.3
GCJ (-O0 with .java) 11.068.55.5528.648.032.3
Kaffe 9.8366.93.4138.836.831.1
shuJIT 8.9152.65.4622.434.324.7
TYA 6.5627.23.8325.229.418.4
interpreter 2.6913.43.106.5518.78.88

Linpack benchmark

Linpack Benchmark -- Java Version (http://www.netlib.org/benchmark/linpackjava/). The results are in MFlops (mega floating-point operations per second), so higher is better.

1000 x 1000500 x 500
1000 x 1000 Linpack results on Pentium 4 500 x 500 Linpack results on Pentium 4
[875x573] [905x572]
MFlopssecond
IBM JDK 1.3.0201.0423.33
HotSpot Server VM199.93.35
HotSpot Client VM174.453.83
GCJ (-O2 with .java)162.1014.13
GCJ (-O2 with .class)158.4144.22
OpenJIT68.9779.69
GCJ (-O0 with .java)51.09813.09
GCJ (-O0 with .class)45.48414.70
shuJIT42.9415.57
Jikes RVM (OptOptSemispace)42.50915.73
ORP (O1 JIT)39.61816.88
Jikes RVM (FastSemispace)39.32617.00
Jikes RVM (FastAdaptiveSemispace)39.11517.01
TYA38.53517.35
ORP (O3 JIT)33.68119.85
Kaffe29.96622.31
interpreter11.88856.25
sunwjit.so8.79476.03
MFlopssecond
GCJ (-O2 with .java)175.7510.48
IBM JDK 1.3.0170.0470.49
GCJ (-O2 with .class)167.6670.50
HotSpot Client VM158.1760.53
HotSpot Server VM152.7020.55
OpenJIT66.9061.25
Jikes RVM (FastSemispace)61.2821.368
Jikes RVM (OptOptSemispace)59.711.40
Jikes RVM (FastAdaptiveSemispace)54.1211.549
GCJ (-O0 with .java)52.9251.58
ORP (O3 JIT)48.4311.73
GCJ (-O0 with .class)46.6261.80
ORP (O1 JIT)41.2572.03
shuJIT40.8942.04
Kaffe38.212.19
TYA37.9342.21
interpreter11.3737.15
sunwjit.so8.7679.56

Eratosthenes Sieve

This benchmark is the Sieve.java in the TYA archive. These values are number of operations per 10 seconds, so higher is better.

Eratosthenes Sieve results on Pentium 4 [878x568]
IBM JDK 1.3.07205
GCJ (-O2 with .java)6823
ORP (O3 JIT)6736
GCJ (-O2 with .class)6597
HotSpot Server VM6462
ORP (O1 JIT)6228
Jikes RVM (OptOptSemispace)5925
Jikes RVM (FastSemispace)5817
Kaffe4815
HotSpot Client VM4461
OpenJIT3442
sunwjit.so3127
shuJIT2915
GCJ (-O0 with .java)2669
TYA2505
GCJ (-O0 with .class)2275
Jikes RVM (FastAdaptiveSemispace)1438
interpreter563

Related Resources

The Java Performance Report
http://www.javalobby.org/members/jpr/
Java Grande Forum
http://www.javagrande.org/
Java Numerics (at NIST)
http://math.nist.gov/javanumerics/
validgh Java numerics page
http://www.validgh.com/java/
Java in Science and Engineering list, Java Grande Forum list
http://mailer.csit.fsu.edu/mailman/listinfo

Copyright (C) 2000,2001,2002 Kazuyuki Shudo
<shudo at computer.org>