Performance comparison of JITs (Feb 2001)

last-modified: Jan 21 15:40, 2002 JST

Back to:

Performance comparison pages:

Oct 2004: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium 4 and Opteron.
Jan 2002: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium 4.
Feb 2001: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium III, K6-2 and Crusoe.
Sep 2000: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on K6-2.
Mar 1999: Eratosthenes Sieve, Linpack benchmark and CaffeineMark 3.0 on Pentium.

This page shows results of performance comparisons among Java runtime systems on Linux.

The following benchmarks were applied to many Java runtime systems.

SPEC JVM98
SciMark 2.0
Linpack benchmark
Eratosthenes Sieve

The followings runtime systems were evaluated:

IBM JDK 1.3 (build cx130-20001124)
- JITC 3.6 included.
IBM JDK 1.1.8 (build l118-20000713)
- JITC 3.0 included.
Sun JDK 1.3.1 Beta (build 1.3.1beta-b15)
- HotSpot Client VM
- HotSpot Server VM
Blackdown JDK 1.2.2 FCS
- sunwjit.so
  (included in this JDK)
- JBuilder Java 2 JIT 1.2.15
  (developed by Borland Software Corp.)
- OpenJIT 1.1.15
- TYA 1.7v2
- shuJIT 0.7.0
- Interpreter of Classic VM
ORP 20001208 version by Intel Microprocessor Research Labs
Kaffe 1.0.6 by Transvirtual Technologies, Inc.
- JIT v3 included.
GCJ (included in gcc 2.95.2)

All benchmark programs run on the following three computers:

IBM ThinkPad X20, which has
- Intel Mobile Pentium III / 600 MHz
- 128 MB RAM
- Linux 2.4.2-ac5
A desktop PC, which has
- AMD K6-2 / 400 MHz
- 64 MB RAM
- Linux 2.2.16
SONY VAIO PCG-C1 VJ/BP, which has
- Transmeta Crusoe TM5600 / 600 MHz
- 112 MB RAM (except cache area for Code Morphing Software)
- Linux 2.4.2-pre4

All results are also available in Mirosoft Excel book format: `JIT-eval-010311.xls'.

SPEC JVM98

SPEC JVM98 (http://www.spec.org/osg/jvm98/) is a benchmark suite contains real Java applications.
This benchmark consists of

Modified Lampel-Ziv method (_201_compress),
Java Expert Shell System based on NASA's CLIPS expert shell system (_202_jess),
Database functions performed on memory resident database (_209_db),
Java compiler from the JDK 1.0.2 (_213_javac),
MPEG Audio Layer-3 decoder (_222_mpegaudio),
Ray-tracer where two threads work (_227_mtrt),
and Java parser generator which is now called JavaCC (_228_jack).

SPEC JVM98 shows scores for each benchmark program and the geometric mean of them. Higher number is better.

Maximum and minimum heap size were set as 32 MB. However the Kaffe needs more than 32 MB as heap size. The _201_compress and _213_javac benchmarks stop its execution with OutOfMemoryError if only 32 MB memory is given. I supplied 48 MB to the Kaffe.

Pentium III K6-2 Crusoe

Pentium III

Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean

HotSpot Server VM 104 62.3 18.6 20.4 77.4 79.0 19.8 43.6

IBM JDK 1.3 80.6 50.4 19.5 22.1 105 63.6 20.6 42.2

HotSpot Client VM 44.6 48.4 17.9 19.5 59.7 64.1 20.0 34.5

IBM JDK 1.1.8 42.5 33.5 18.9 14.9 71.3 45.7 17.6 30.2

ORP (O3 JIT) 7.53 20.2 58.1 13.2 73.9 25.2 N/A 24.5

ORP (O1 JIT) 6.93 20.4 46.3 12.0 60.5 25.0 N/A 22.2

OpenJIT 20.6 16.8 14.9 6.76 29.0 17.3 6.34 14.1

JBuilder JIT 21.0 16.9 14.8 6.71 28.7 4.71 5.85 11.6

shuJIT 10.2 14.3 11.9 6.67 20.2 14.2 5.55 10.9

Kaffe 8.08 5.37 37.1 8.60 23.0 4.59 6.16 9.88

TYA 8.43 12.2 10.6 5.89 12.6 11.3 5.27 9.01

sunwjit.so 4.32 6.27 13.3 N/A (cannot complete) 17.5 11.3 5.24 8.48

interpreter 5.51 5.30 3.48 3.19 4.43 4.60 3.50 4.20

K6-2

Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean

IBM JDK 1.3 25.7 21.7 14.4 12.2 33.0 28.6 12.6 19.7

HotSpot Server VM 25.7 27.3 14.0 9.77 26.8 37.1 11.1 19.5

HotSpot Client VM 14.9 22.3 13.0 8.98 20.3 27.3 10.2 15.5

IBM JDK 1.1.8 12.7 12.0 12.9 7.56 24.8 15.6 7.45 12.3

ORP (O3 JIT) 3.25 9.54 29.3 6.41 26.7 11.7 N/A 11.0

ORP (O1 JIT) 3.12 9.24 22.1 6.01 23.6 10.8 N/A 9.96

OpenJIT 8.16 8.70 9.88 3.71 11.1 9.06 4.34 7.33

shuJIT 5.43 7.92 8.65 3.89 10.9 7.10 3.53 6.31

JBuilder JIT 9.44 8.43 10.3 3.10 13.5 2.54 3.77 6.14

TYA 5.79 7.08 7.55 3.39 8.90 6.23 3.75 5.79

sunwjit.so 2.32 3.06 9.24 N/A (cannot complete) 8.52 5.91 3.44 4.74

Kaffe 3.29 2.47 18.0 4.44 7.96 2.05 2.82 4.36

interpreter 2.58 2.69 1.95 1.72 2.19 2.29 2.01 2.18

Crusoe

Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean

HotSpot Server VM 55.4 39.3 10.7 10.9 46.4 45.4 9.05 24.2

IBM JDK 1.3 19.6 30.2 10.7 15.1 46.2 37.2 9.47 20.6

IBM JDK 1.1.8 27.4 21.9 10.7 10.2 39.6 28.7 9.23 18.3

HotSpot Client VM 18.0 27.3 9.63 11.0 28.3 35.6 8.78 17.3

ORP (O3 JIT) 5.15 13.1 50.6 7.79 40.4 15.9 N/A 16.1

ORP (O1 JIT) 4.75 12.7 33.3 7.08 32.4 15.2 N/A 13.8

OpenJIT 11.3 11.8 9.04 4.88 15.1 11.7 4.38 8.93

JBuilder JIT 12.3 10.6 8.53 4.42 20.7 2.97 3.87 7.36

Kaffe 4.93 3.87 31.7 6.32 12.2 4.88 4.02 7.11

TYA 6.35 9.03 7.42 4.32 12.1 8.65 3.60 6.83

sunwjit.so 2.77 4.05 8.11 N/A (cannot complete) 10.7 8.05 3.62 5.52

shuJIT 2.96 1.25 10.2 2.55 16.5 9.61 2.00 4.37

interpreter 3.68 3.63 2.01 2.19 2.64 2.68 2.34 2.67

SciMark 2.0

SciMark 2.0 (http://math.nist.gov/scimark2/) is a benchmark suite based on numerical kernels. They were collected by NIST researchers.

I specified the `-large' option to run the large problem size version. SciMark 2.0 shows results in MFlops (mega floating-point operations per second) and the composite score, which is the arithmetic mean of them. Therefore, SOR, which shows relatively higher value, affects the composite score. Oppositely, FFT and Monte Carlo do not very much. Higher number is better because results were in MFlops.

Pentium III K6-2 Crusoe

Pentium III :

Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score

IBM JDK 1.3.0 10.6 80.5 16.7 28.8 37.6 34.9

ORP (O3 JIT) 10.7 80.6 5.24 29.5 37.1 32.6

IBM JDK 1.1.8 10.6 77.2 14.7 24.0 35.6 32.4

HotSpot Server VM 12.6 72.7 19.7 26.3 26.2 31.5

HotSpot Client VM 11.9 78.1 10.5 25.4 26.6 30.5

ORP (O1 JIT) 9.73 77.4 4.19 22.1 22.1 27.1

GCJ (-O2 with .class) 9.45 47.2 3.91 19.2 24.3 20.8

OpenJIT 9.95 43.8 1.61 12.6 22.9 18.2

sunwjit.so 8.47 44.1 1.98 18.4 4.87 15.6

shuJIT 6.30 37.6 3.56 14.5 15.0 15.4

JBuilder JIT (*) 10.0 26.4 3.59 13.0 23.4 15.3

Kaffe 6.21 32.5 4.36 12.2 19.1 14.9

GCJ (-O0 with .class) 6.30 29.2 2.86 12.9 14.1 13.1

TYA 5.52 20.5 2.90 12.3 17.0 11.7

JBuilder JIT 3.42 28.2 3.71 17.3 4.78 11.5

interpreter 2.00 6.92 1.61 4.09 4.78 3.79

K6-2 :

Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score

IBM JDK 1.3.0 5.35 49.9 7.9 15.7 20.4 19.8

ORP (O3 JIT) 4.20 52.9 1.62 18.5 20.3 19.5

IBM JDK 1.1.8 4.12 38.0 7.01 18.1 17.9 17.0

ORP (O1 JIT) 4.07 41.9 1.72 17.5 16.2 16.3

HotSpot Client VM 4.98 33.6 6.32 17.2 17.5 15.9

HotSpot Server VM 4.92 34.6 7.23 13.3 17.1 15.4

GCJ (-O2 with .class) 4.05 34.8 1.13 4.35 16.5 12.2

JBuilder JIT (*) 3.82 19.8 1.49 9.42 14.1 9.73

shuJIT 4.19 18.6 2.23 11.4 11.6 9.61

TYA 3.51 15.8 1.72 10.4 11.7 8.63

OpenJIT 3.51 14.1 1.59 9.20 10.4 7.77

JBuilder JIT 1.65 20.3 1.75 9.68 2.61 7.19

GCJ (-O0 with .class) 3.44 17.1 1.02 4.32 9.30 7.04

Kaffe 3.04 12.8 1.89 7.35 9.88 6.99

sunwjit.so 3.77 15.8 1.11 9.11 2.61 6.47

interpreter 1.07 3.71 0.626 2.21 2.61 2.05

Crusoe :

Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score

ORP (O3 JIT) 4.88 82.3 3.85 27.0 35.0 30.6

HotSpot Server VM 4.69 30.5 12.7 20.0 22.1 28.0

IBM JDK 1.3.0 5.40 62.3 11.6 22.8 34.5 27.3

IBM JDK 1.1.8 5.36 53.1 7.63 20.2 24.9 22.2

HotSpot Client VM 4.57 39.6 7.87 15.8 20.8 17.7

ORP (O1 JIT) 4.72 39.3 3.39 16.9 20.2 16.9

GCJ (-O2 with .class) 4.60 29.6 2.36 16.1 19.2 14.4

shuJIT 4.04 33.9 2.73 13.0 12.7 13.3

JBuilder JIT (*) 4.81 17.4 2.28 10.8 21.8 11.4

OpenJIT 4.36 23.6 2.23 9.02 15.2 10.9

TYA 4.14 17.5 1.96 11.8 18.3 10.7

sunwjit.so 4.66 28.3 1.24 11.5 3.52 9.83

GCJ (-O0 with .class) 4.08 15.8 2.02 11.7 12.5 9.23

JBuilder JIT 1.98 17.4 2.32 17.6 3.53 8.57

Kaffe 3.00 15.2 3.30 8.53 10.3 8.08

interpreter 1.21 5.16 0.808 2.59 3.40 2.63

(*) In default setting, JBuilder JIT compile a method when it is called for the second time. In this case, that compilation threshold was set as 1, then all methods were compiled as soon as called.

Linpack benchmark

Linpack Benchmark -- Java Version (http://www.netlib.org/benchmark/linpackjava/). The problem size is 500 x 500. The results are in MFlops (mega floating-point operations per second), so higher is better.

Pentium III : K6-2 : Crusoe :

ORP (O3 JIT) 38.385

IBM JDK 1.3.0 37.899

IBM JDK 1.1.8 36.057

HotSpot Server VM 35.047

HotSpot Client VM 27.087

ORP (O1 JIT) 25.955

GCJ (-O2 with .class) 25.906

GCJ (-O2 with .java) 23.223

shuJIT 19.606

JBuilder JIT (*) 19.153

JBuilder JIT 18.704

Kaffe 18.584

TYA 16.697

GCJ (-O0 with .java) 16.461

OpenJIT 14.328

GCJ (-O0 with .class) 13.97

sunwjit.so 4.234

interpreter 4.228

ORP (O3 JIT) 21.294

IBM JDK 1.3.0 20.201

HotSpot Client VM 19.075

IBM JDK 1.1.8 18.161

GCJ (-O2 with .class) 17.803

HotSpot Server VM 17.634

ORP (O1 JIT) 17.393

GCJ (-O2 with .java) 17.112

shuJIT 12.785

TYA 11.206

JBuilder JIT (*) 10.982

JBuilder JIT 10.686

Kaffe 9.790

GCJ (-O0 with .java) 9.591

GCJ (-O0 with .class) 9.167

OpenJIT 8.596

sunwjit.so 2.266

interpreter 2.263

HotSpot Server VM 34.570

ORP (O3 JIT) 34.190

IBM JDK 1.3.0 31.599

IBM JDK 1.1.8 25.481

HotSpot Client VM 22.403

ORP (O1 JIT) 21.037

GCJ (-O2 with .class) 18.052

TYA 17.803

GCJ (-O2 with .java) 17.579

shuJIT 16.231

JBuilder JIT (*) 15.968

JBuilder JIT 15.551

GCJ (-O0 with .class) 12.095

Kaffe 10.927

OpenJIT 8.153

GCJ (-O0 with .java) 8.137

sunwjit.so 3.433

interpreter 3.26

(*) In default setting, JBuilder JIT compile a method when it is called for the second time. In this case, that compilation threshold was set as 1, then all methods were compiled as soon as called.

Eratosthenes Sieve

The TYA archive includes this benchmark as `demo/Sieve.java'. These values are in number of operations per 10 seconds, so higher is better.

In this benchmark program, the compilation threshold was set as 1 for JBuilder JIT, then all methods were compiled as soon as called.

Pentium III : K6-2 : Crusoe :

IBM JDK 1.3.0 3780

ORP (O3 JIT) 3495

IBM JDK 1.1.8 3368

GCJ (-O2 with .java) 3117

ORP (O1 JIT) 2607

GCJ (-O2 with .class) 2444

HotSpot Server VM 2178

Kaffe 2036

JBuilder JIT 1973

HotSpot Client VM 1645

OpenJIT 1476

sunwjit.so 1359

shuJIT 1140

GCJ (-O0 with .java) 1040

TYA 1039

GCJ (-O0 with .class) 958

interpreter 224

IBM JDK 1.3.0 2187

ORP (O3 JIT) 2708

IBM JDK 1.1.8 2073

GCJ (-O2 with .java) 2394

ORP (O1 JIT) 1672

GCJ (-O2 with .class) 1532

HotSpot Server VM 1334

Kaffe 1347

JBuilder JIT 1209

HotSpot Client VM 1167

OpenJIT 928

sunwjit.so 667

shuJIT 832

GCJ (-O0 with .java) 785

TYA 670

GCJ (-O0 with .class) 671

interpreter 78

IBM JDK 1.3.0 2427

ORP (O3 JIT) 2419

GCJ (-O2 with .java) 2226

IBM JDK 1.1.8 2059

Kaffe 1936

HotSpot Server VM 1724

ORP (O1 JIT) 1628

GCJ (-O2 with .class) 1590

JBuilder JIT 1389

shuJIT 1355

HotSpot Client VM 1349

sunwjit.so 1244

TYA 1206

GCJ (-O0 with .java) 1142

GCJ (-O0 with .class) 1126

OpenJIT 980

interpreter 102

Related Resources

The Java Performance Report: http://www.javalobby.org/fr/html/frm/javalobby/features/jpr/
Java Grande Forum: http://www.javagrande.org/
Java Numerics (at NIST): http://math.nist.gov/javanumerics/
validgh Java numerics page: http://www.validgh.com/java/
Java in Science and Engineering list, Java Grande Forum list: http://mailer.csit.fsu.edu/mailman/listinfo

Runtime system	_227_ mtrt	_202_ jess	_201_ compress	_209_ db	_222_ mpegaudio	_228_ jack	_213_ javac	Geometric Mean
HotSpot Server VM	104	62.3	18.6	20.4	77.4	79.0	19.8	43.6
IBM JDK 1.3	80.6	50.4	19.5	22.1	105	63.6	20.6	42.2
HotSpot Client VM	44.6	48.4	17.9	19.5	59.7	64.1	20.0	34.5
IBM JDK 1.1.8	42.5	33.5	18.9	14.9	71.3	45.7	17.6	30.2
ORP (O3 JIT)	7.53	20.2	58.1	13.2	73.9	25.2	N/A	24.5
ORP (O1 JIT)	6.93	20.4	46.3	12.0	60.5	25.0	N/A	22.2
OpenJIT	20.6	16.8	14.9	6.76	29.0	17.3	6.34	14.1
JBuilder JIT	21.0	16.9	14.8	6.71	28.7	4.71	5.85	11.6
shuJIT	10.2	14.3	11.9	6.67	20.2	14.2	5.55	10.9
Kaffe	8.08	5.37	37.1	8.60	23.0	4.59	6.16	9.88
TYA	8.43	12.2	10.6	5.89	12.6	11.3	5.27	9.01
sunwjit.so	4.32	6.27	13.3	N/A (cannot complete)	17.5	11.3	5.24	8.48
interpreter	5.51	5.30	3.48	3.19	4.43	4.60	3.50	4.20

Runtime system	FFT (1048576)	SOR (1000x1000)	Monte Carlo	Sparse matmult (N=100000, nz=1000000)	LU (1000x1000)	Composite Score
IBM JDK 1.3.0	10.6	80.5	16.7	28.8	37.6	34.9
ORP (O3 JIT)	10.7	80.6	5.24	29.5	37.1	32.6
IBM JDK 1.1.8	10.6	77.2	14.7	24.0	35.6	32.4
HotSpot Server VM	12.6	72.7	19.7	26.3	26.2	31.5
HotSpot Client VM	11.9	78.1	10.5	25.4	26.6	30.5
ORP (O1 JIT)	9.73	77.4	4.19	22.1	22.1	27.1
GCJ (-O2 with .class)	9.45	47.2	3.91	19.2	24.3	20.8
OpenJIT	9.95	43.8	1.61	12.6	22.9	18.2
sunwjit.so	8.47	44.1	1.98	18.4	4.87	15.6
shuJIT	6.30	37.6	3.56	14.5	15.0	15.4
JBuilder JIT (*)	10.0	26.4	3.59	13.0	23.4	15.3
Kaffe	6.21	32.5	4.36	12.2	19.1	14.9
GCJ (-O0 with .class)	6.30	29.2	2.86	12.9	14.1	13.1
TYA	5.52	20.5	2.90	12.3	17.0	11.7
JBuilder JIT	3.42	28.2	3.71	17.3	4.78	11.5
interpreter	2.00	6.92	1.61	4.09	4.78	3.79