Performance comparison of JITs (Jan 2002)

last-modified: Feb 6 11:55, 2002 JST

Back to:

Performance comparison pages:

Oct 2004: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium 4 and Opteron.
Jan 2002: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium 4.
Feb 2001: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium III, K6-2 and Crusoe.
Sep 2000: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on K6-2.
Mar 1999: Eratosthenes Sieve, Linpack benchmark and CaffeineMark 3.0 on Pentium.

This page shows results of performance comparisons among Java runtime systems on Linux.

Applied benchmarks:

SPEC JVM98
SciMark 2.0
Linpack benchmark
Eratosthenes Sieve

Evaluated Java runtime systems:

Blackdown JDK 1.3.1_02a FCS
- HotSpot Client VM
- HotSpot Server VM
- sunwjit.so (included in this JDK)
- OpenJIT 1.1.16
- TYA 1.7v3
- shuJIT 0.7.10
- Interpreter of HotSpot Client VM
IBM JDK 1.3 (build cx130-20010925)
- JITC 3.6 included.
ORP 1.0.9 by Intel Microprocessor Research Labs
(released at Jan 17 2002)
- with Jan 10 2002 version of GNU Classpath on CVS as the class library.
Jikes RVM 2.0.2
(released at Jan 17 2002)
- The tested configurations are `FastSemispace', `FastAdaptiveSemispace' and `OptOptSemispace'.
Kaffe 1.0.6
- JIT v3 included.
GCJ
- Jan 15 2002 version on CVS.

The computer that run all benchmark programs is as follows:

A desktop PC
- Intel Pentium 4 / 1.7 GHz
- i850 chipset, 768 MB RDRAM
- Linux 2.4.18-pre3

All results are also available in Mirosoft Excel book format: `JIT-eval-020120.xls'.

SPEC JVM98

SPEC JVM98 (http://www.spec.org/osg/jvm98/) is a benchmark suite consists of real Java application programs:

Modified Lampel-Ziv method (_201_compress),
Java Expert Shell System based on NASA's CLIPS expert shell system (_202_jess),
Database functions performed on memory resident database (_209_db),
Java compiler from the JDK 1.0.2 (_213_javac),
MPEG Audio Layer-3 decoder (_222_mpegaudio),
Ray-tracer where two threads work (_227_mtrt),
and Java parser generator which is now called JavaCC (_228_jack).

SPEC JVM98 shows scores for each benchmark program and the geometric mean of them. Higher number is better.

Fairness of results

The results of ORP, Jikes RVM and Kaffe are not reportable to SPEC because these runtime systems read benchmark programs and data from a local filesystem. Reporatable results have to be produced by the benchmark which run as a Java applet. ORP and Jikes RVM cannot run a Java applet and Kaffe cannot run the SPEC tools as an applet loaded from a web server. _201_compress and _213_jack benchmarks are very sensitive to where the programs and data are read from. Thus the results of these two benchmarks cannot be compared fairly among all runtime systems.

Heap size

Maximum and minimum heap size were set as 32 MB. However Kaffe needs more than 32 MB as heap size. The _201_compress and _213_javac benchmarks stop its execution with OutOfMemoryError if only 32 MB memory is given. We supplied 48 MB to Kaffe. We also supplied 48 MB to ORP only for _209_db benchmark because ORP cannot complete the benchmark with 32 MB heap. Jikes RVM run this benchmark with 128 MB heap for small objects and 24 MB heap for large objects (-X:h=128 -X:lh=24).

[953x556]

Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean

HotSpot Server VM 147 160 11.7 21.7 167 150 15.1 58.2

ORP (O3 JIT) 73.4 68.5 96.3 20.6 161 69.4 19.4 57.9

ORP (O1 JIT) 57.3 64.5 82.7 20.0 143 70.4 14.8 51.1

IBM JDK 1.3 107 78.3 11.7 31.6 179 104 15.0 50.8

Jikes RVM (FastAdaptiveSemispace) 47.0 57.7 74.4 26.5 77.9 63.1 33.3 50.8

HotSpot Client VM 76.0 117 10.8 23.0 114 126 13.7 46.0

Jikes RVM (FastSemispace) 56.0 47.8 75.5 17.5 91.0 46.4 24.9 45.0

Jikes RVM (OptOptSemispace) 32.7 21.7 64.3 15.6 61.9 22.5 11.1 27.2

OpenJIT 33.6 33.0 10.9 10.9 61.0 37.4 7.17 21.6

shuJIT 20.9 29.3 10.3 11.1 56.8 33.1 7.00 19.1

Kaffe 15.2 11.6 73.9 14.4 36.0 9.91 11.7 18.7

TYA 15.7 26.7 9.71 9.40 34.3 29.4 6.84 16.0

sunwjit.so 8.01 12.1 10.6 N/A 43.3 27.2 6.7 14.2

interpreter 11.6 11.9 4.88 6.59 7.62 17.0 8.93 9.09

SciMark 2.0

SciMark 2.0 (http://math.nist.gov/scimark2/) is a benchmark suite consists of numerical kernels. They were collected by NIST researchers.

We specified the `-large' option to run the large problem size version. SciMark 2.0 shows results in MFlops (mega floating-point operations per second) and the composite score, which is the arithmetic mean of them. Higher number is better because results were in MFlops. Therefore, SOR, which shows relatively higher value, affects the composite score. Oppositely, FFT and Monte Carlo are not counted very much.

[996x572]

Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score

HotSpot Client VM 19.6 249 28.6 88.4 169 111

GCJ (-O2 with .java) 18.9 229 6.56 138 153 109

GCJ (-O2 with .class) 18.7 232 7.28 119 158 107

IBM JDK 1.3.0 19.9 145 20.3 158 167 102

ORP (O3 JIT) 11.3 152 27.1 107 60.5 715

HotSpot Server VM 20.2 157 35.9 20.5 95.6 65.8

Jikes RVM (FastSemispace) 11.2 132 3.35 76.2 54.5 55.4

Jikes RVM (OptOptSemispace) 11.0 128 3.32 72.6 53.2 53.6

ORP (O1 JIT) 10.3 116 22.9 54.7 42.6 49.3

Jikes RVM (FastAdaptiveSemispace) 7.91 130 3.42 72.5 16.2 46.0

GCJ (-O0 with .class) 10.8 74.2 5.27 47.0 44.9 36.4

sunwjit.so 16.2 80.0 4.08 67.2 10.4 35.6

OpenJIT 6.77 70.4 5.34 65.0 13.8 32.3

GCJ (-O0 with .java) 11.0 68.5 5.55 28.6 48.0 32.3

Kaffe 9.83 66.9 3.41 38.8 36.8 31.1

shuJIT 8.91 52.6 5.46 22.4 34.3 24.7

TYA 6.56 27.2 3.83 25.2 29.4 18.4

interpreter 2.69 13.4 3.10 6.55 18.7 8.88

Linpack benchmark

Linpack Benchmark -- Java Version (http://www.netlib.org/benchmark/linpackjava/). The results are in MFlops (mega floating-point operations per second), so higher is better.

1000 x 1000 500 x 500

[875x573] [905x572]

MFlops second

IBM JDK 1.3.0 201.042 3.33

HotSpot Server VM 199.9 3.35

HotSpot Client VM 174.45 3.83

GCJ (-O2 with .java) 162.101 4.13

GCJ (-O2 with .class) 158.414 4.22

OpenJIT 68.977 9.69

GCJ (-O0 with .java) 51.098 13.09

GCJ (-O0 with .class) 45.484 14.70

shuJIT 42.94 15.57

Jikes RVM (OptOptSemispace) 42.509 15.73

ORP (O1 JIT) 39.618 16.88

Jikes RVM (FastSemispace) 39.326 17.00

Jikes RVM (FastAdaptiveSemispace) 39.115 17.01

TYA 38.535 17.35

ORP (O3 JIT) 33.681 19.85

Kaffe 29.966 22.31

interpreter 11.888 56.25

sunwjit.so 8.794 76.03

MFlops second

GCJ (-O2 with .java) 175.751 0.48

IBM JDK 1.3.0 170.047 0.49

GCJ (-O2 with .class) 167.667 0.50

HotSpot Client VM 158.176 0.53

HotSpot Server VM 152.702 0.55

OpenJIT 66.906 1.25

Jikes RVM (FastSemispace) 61.282 1.368

Jikes RVM (OptOptSemispace) 59.71 1.40

Jikes RVM (FastAdaptiveSemispace) 54.121 1.549

GCJ (-O0 with .java) 52.925 1.58

ORP (O3 JIT) 48.431 1.73

GCJ (-O0 with .class) 46.626 1.80

ORP (O1 JIT) 41.257 2.03

shuJIT 40.894 2.04

Kaffe 38.21 2.19

TYA 37.934 2.21

interpreter 11.373 7.15

sunwjit.so 8.767 9.56

Eratosthenes Sieve

This benchmark is the Sieve.java in the TYA archive. These values are number of operations per 10 seconds, so higher is better.

[878x568]

IBM JDK 1.3.0 7205

GCJ (-O2 with .java) 6823

ORP (O3 JIT) 6736

GCJ (-O2 with .class) 6597

HotSpot Server VM 6462

ORP (O1 JIT) 6228

Jikes RVM (OptOptSemispace) 5925

Jikes RVM (FastSemispace) 5817

Kaffe 4815

HotSpot Client VM 4461

OpenJIT 3442

sunwjit.so 3127

shuJIT 2915

GCJ (-O0 with .java) 2669

TYA 2505

GCJ (-O0 with .class) 2275

Jikes RVM (FastAdaptiveSemispace) 1438

interpreter 563

Related Resources

The Java Performance Report: http://www.javalobby.org/members/jpr/
Java Grande Forum: http://www.javagrande.org/
Java Numerics (at NIST): http://math.nist.gov/javanumerics/
validgh Java numerics page: http://www.validgh.com/java/
Java in Science and Engineering list, Java Grande Forum list: http://mailer.csit.fsu.edu/mailman/listinfo

Runtime system	_227_ mtrt	_202_ jess	_201_ compress	_209_ db	_222_ mpegaudio	_228_ jack	_213_ javac	Geometric Mean
HotSpot Server VM	147	160	11.7	21.7	167	150	15.1	58.2
ORP (O3 JIT)	73.4	68.5	96.3	20.6	161	69.4	19.4	57.9
ORP (O1 JIT)	57.3	64.5	82.7	20.0	143	70.4	14.8	51.1
IBM JDK 1.3	107	78.3	11.7	31.6	179	104	15.0	50.8
Jikes RVM (FastAdaptiveSemispace)	47.0	57.7	74.4	26.5	77.9	63.1	33.3	50.8
HotSpot Client VM	76.0	117	10.8	23.0	114	126	13.7	46.0
Jikes RVM (FastSemispace)	56.0	47.8	75.5	17.5	91.0	46.4	24.9	45.0
Jikes RVM (OptOptSemispace)	32.7	21.7	64.3	15.6	61.9	22.5	11.1	27.2
OpenJIT	33.6	33.0	10.9	10.9	61.0	37.4	7.17	21.6
shuJIT	20.9	29.3	10.3	11.1	56.8	33.1	7.00	19.1
Kaffe	15.2	11.6	73.9	14.4	36.0	9.91	11.7	18.7
TYA	15.7	26.7	9.71	9.40	34.3	29.4	6.84	16.0
sunwjit.so	8.01	12.1	10.6	N/A	43.3	27.2	6.7	14.2
interpreter	11.6	11.9	4.88	6.59	7.62	17.0	8.93	9.09

Runtime system	FFT (1048576)	SOR (1000x1000)	Monte Carlo	Sparse matmult (N=100000, nz=1000000)	LU (1000x1000)	Composite Score
HotSpot Client VM	19.6	249	28.6	88.4	169	111
GCJ (-O2 with .java)	18.9	229	6.56	138	153	109
GCJ (-O2 with .class)	18.7	232	7.28	119	158	107
IBM JDK 1.3.0	19.9	145	20.3	158	167	102
ORP (O3 JIT)	11.3	152	27.1	107	60.5	715
HotSpot Server VM	20.2	157	35.9	20.5	95.6	65.8
Jikes RVM (FastSemispace)	11.2	132	3.35	76.2	54.5	55.4
Jikes RVM (OptOptSemispace)	11.0	128	3.32	72.6	53.2	53.6
ORP (O1 JIT)	10.3	116	22.9	54.7	42.6	49.3
Jikes RVM (FastAdaptiveSemispace)	7.91	130	3.42	72.5	16.2	46.0
GCJ (-O0 with .class)	10.8	74.2	5.27	47.0	44.9	36.4
sunwjit.so	16.2	80.0	4.08	67.2	10.4	35.6
OpenJIT	6.77	70.4	5.34	65.0	13.8	32.3
GCJ (-O0 with .java)	11.0	68.5	5.55	28.6	48.0	32.3
Kaffe	9.83	66.9	3.41	38.8	36.8	31.1
shuJIT	8.91	52.6	5.46	22.4	34.3	24.7
TYA	6.56	27.2	3.83	25.2	29.4	18.4
interpreter	2.69	13.4	3.10	6.55	18.7	8.88

IBM JDK 1.3.0	7205
GCJ (-O2 with .java)	6823
ORP (O3 JIT)	6736
GCJ (-O2 with .class)	6597
HotSpot Server VM	6462
ORP (O1 JIT)	6228
Jikes RVM (OptOptSemispace)	5925
Jikes RVM (FastSemispace)	5817
Kaffe	4815
HotSpot Client VM	4461
OpenJIT	3442
sunwjit.so	3127
shuJIT	2915
GCJ (-O0 with .java)	2669
TYA	2505
GCJ (-O0 with .class)	2275
Jikes RVM (FastAdaptiveSemispace)	1438
interpreter	563