Performance Comparison of Java/.NET Runtimes (Oct 2004)

last-updated: November 20 21:55, 2005 JST

Back to:

Performance comparison pages:


News

November 20, 2005
Linpack.cs and Linpack.java were updated and show correct scores with a large matrix size about 1,300.
Thanks Dr. Tobias Kind.
December 17, 2004
Scores of Mono 1.1.3 were added.
December 12, 2004
Linpack.cs and Linpack.java were added.
October 29, 2004
JRockit was measured and the scores were added.
October 27, 2004
Results of Linpack benchmark in C added.
Numbers of Visual C++ / SciMark 2 updated.
October 25, 2004
Results of SciMark 2 in C# and C added.
October 19, 2004
Mono was measured and the scores were added.
October 14, 2004
This page was initially provided.

This page shows results of performance comparisons among Java runtime systems on Linux and .NET runtime system on Windows.

Applied benchmarks

Evaluated Java runtime systems

Evaluated .NET runtime systems

Evaluated C compilers

Computers

All results are also available in Microsoft Excel (2003) book format: `JIT-eval-20041217.xls'.


SPEC JVM98 (in Java)

SPEC JVM98 (http://www.spec.org/osg/jvm98/) is a benchmark suite consists of real Java application programs:

SPEC JVM98 shows scores for each benchmark program and the geometric mean of them. Higher number is better.

Execution time of each benchmark program is rather short on a today's computer. For example, with IBM JDK 1.4.2, _202_jess, _222_mpegaudio and _228_jack finish only in 2 seconds and _227_mtrt takes only 0.83 second. SPEC JVM98 seems not to be able to evaluate advanced optimizing techniques for server-side applications like profile-guided optimizations.

Fairness of results

The results of Jikes RVM, Kaffe and GCJ are not reportable to SPEC because these runtime systems read benchmark programs and data from a local filesystem. Reportable results have to be produced by the benchmark which run as a Java applet. Jikes RVM and GCJ cannot run a Java applet and Kaffe does not work correctly with the applet version of SPEC JVM98.

_201_compress and _213_jack benchmarks are very sensitive to where those programs and data for them are read from. Thus the results of these two benchmarks cannot be compared fairly among all runtime systems.

Conditions

Benchmark programs and a web server (Apache httpd) ran on the same single computer and those programs and data were loaded from the web server (except Jikes RVM, Kaffe and GCJ).

Initial and maximum heap sizes remain as those default values. Note that one exception is JDK 1.5.0 for x86 on Opteron, whose default values are not enough to run SPEC JVM98 and initial and maximum heap sizes are set as 128 MB.

TYA 1.8 cannot run _200_check, _228_jack and _213_javac and then we use TYA 1.7v3.

Results

Pentium 4

SPEC JVM98 results on Pentium 4
Runtime system_227_ mtrt_202_ jess_201_ compress_209_ db_222_ mpegaudio_228_ jack_213_ javacGeometric Mean
IBM JDK 1.4.2 55725632474.1650247165265
IBM JDK 1.3.1 50921833273.2527235140241
Sun JDK 1.5.0 Server VM 49633130046.353322291.9220
Sun JDK 1.4.2 Server VM 50433229445.551319998.8217
JRockit (-server) 26528928273.7485204123212
JRockit (-client) 19321526070.2476203123190
Sun JDK 1.5.0 Client VM 32522120443.425119296.1162
Sun JDK 1.4.2 Client VM 31818619943.624918190.9155
GCJ (-O2) 12287.018729.0375N/A56.1103
Jikes RVM 10912418238.019594.252.798.6
GCJ (-O0) 10881.912028.7108N/A54.175.0
OpenJIT 82.160.717716.112352.742.562.9
shuJIT 45.853.211717.211451.037.252.2
Kaffe 32.021.319124.810132.921.341.9
TYA 33.946.183.415.167.743.534.441.1
sunwjit 15.523.814516.386.643.833.737.8
interpreter 19.621.213.012.414.728.417.217.4

Opteron (AMD64 binary)

SPEC JVM98 results on Opteron (AMD64 binary)
Runtime system_227_ mtrt_202_ jess_201_ compress_209_ db_222_ mpegaudio_228_ jack_213_ javacGeometric Mean
Sun JDK 1.5.0 Server VM 65027035446.6538351114251
GCJ (-O2) 20710826035.4429N/A77.1138
GCJ (-O0) 13388.899.834.175.7N/A70.177.3
interpreter 27.018.310.710.212.430.216.416.5

Opteron (x86 binary)

SPEC JVM98 results on Opteron (x86 binary)
Runtime system_227_ mtrt_202_ jess_201_ compress_209_ db_222_ mpegaudio_228_ jack_213_ javacGeometric Mean
JRockit (-server) 45028630762.5441316161247
Sun JDK 1.5.0 Server VM 61130826550.0481344106238
JRockit (-client) 23625626464.2453298161217
Sun JDK 1.5.0 Client VM 46723622846.226229497.4189
GCJ (-O2) 13399.618735.1311N/A76.1113
GCJ (-O0) 10886.310733.975.9N/A69.975.0
interpreter 26.618.411.111.012.131.516.716.8

SciMark 2.0 (in Java, C# and C)

SciMark 2.0 (http://math.nist.gov/scimark2/) is a benchmark suite consists of numerical kernels. C version is also available at the NIST site. C# version is available at a web page at Cornell University (http://rotor.cs.cornell.edu/SciMark/).

SciMark 2.0 shows results in MFlops (mega floating-point operations per second) and the composite score, which is the arithmetic mean of them. Higher number is better. Therefore, SOR showing relatively higher value affects the composite score much. Oppositely, FFT and Monte Carlo are not counted very much.

Fairness of results

C version is advantageous compared with Java and C# versions because Java and C# versions use a "synchronized" version of the random number generator, which makes the Monte Carlo Test much slower.

Conditions

We specified the `-large' option to run the benchmarks with the large data set.

We tried "-O2", "-O3" and "-O3 -funroll-loops" options with gcc and "-O2" yielded the best numbers. Similarly, "-O2", "-O3" and "-O3 -xN" tried with icc and "-O2" was the best.

Pentium 4

SciMark 2.0 results on Pentium 4
Runtime systemFFT (1048576)SOR (1000x1000)Monte CarloSparse matmult (N=100000, nz=1000000)LU (1000x1000)Composite Score
C/Visual C++ 27.776190.9384384329
Java/Sun JDK 1.4.2 Server VM 38.1742115356366324
Java/Sun JDK 1.5.0 Server VM 34.474194.3347366316
C/icc (-O2) 30.964185.5391380306
C/gcc (-O2) 31.445184.2385378266
Java/IBM JDK 1.4.2 32.945184.0315370250
Java/JRockit (-server) 35.2430112307358249
Java/IBM JDK 1.3.1 34.744962.1320367247
Java/JRockit (-client) 37.0430113290359246
C#/.NET (Release) 25.945446.9323348240
Java/GCJ (-O2 with .java) 31.940914.6281353218
Java/GCJ (-O2 with .class) 31.940914.4238338206
Java/Sun JDK 1.5.0 Client VM 34.341354.1164327199
Java/Jikes RVM 24.24127.12261279197
Java/Sun JDK 1.4.2 Client VM 34.541438.5165325195
C#/Mono 1.0.2 (JIT) 34.43552.12178245163
C#/Mono 1.1.3 (JIT) 31.83755.24180219162
Java/Kaffe 29.222610.695.3216115
Java/GCJ (-O0 with .java) 27.719313.110615398.6
Java/OpenJIT 28.61248.512355.768.0
Java/GCJ (-O0 with .class) 23.113211.766.183.163.2
Java/sunwjit 31.81485.7359.618.652.7
Java/shuJIT 17.11007.5258.665.449.7
Java/TYA 14.264.25.9350.371.541.2
Java/interpreter 6.6326.94.8819.841.920.0
C#/Mono 1.1.3 (interpreter) 4.1118.40.839.987.468.16
C#/Mono 1.0.2 (interpreter) 4.2216.40.838.248.127.56

Opteron (AMD64 binary)

SciMark 2.0 results on Opteron (AMD64 binary)
Runtime systemFFT (1048576)SOR (1000x1000)Monte CarloSparse matmult (N=100000, nz=1000000)LU (1000x1000)Composite Score
Sun JDK 1.5.0 Server VM 36.966973.3249299265
GCJ (-O2 with .class) 38.248137.0235288216
GCJ (-O2 with .java) 38.148132.4229285213
GCJ (-O0 with .java) 24.414621.680.210976.3
GCJ (-O0 with .class) 21.112017.070.562.558.2
interpreter 5.8424.53.7315.120.213.9

Opteron (x86 binary)

SciMark 2.0 results on Opteron (x86 binary)
Runtime systemFFT (1048576)SOR (1000x1000)Monte CarloSparse matmult (N=100000, nz=1000000)LU (1000x1000)Composite Score
Sun JDK 1.5.0 Server VM 37.865964.4234300259
JRockit (-server) 41.6463121258301237
JRockit (-client) 41.5468121254289235
GCJ (-O2 with .class) 37.748229.8228293214
GCJ (-O2 with .java) 38.147628.9217302212
Sun JDK 1.5.0 Client VM 33.947941.2186204189
GCJ (-O0 with .java) 23.414720.980.999.974.5
GCJ (-O0 with .class) 19.911816.466.465.657.3
interpreter 5.5221.73.6213.719.512.8

Linpack benchmark (in Java, C# and C)

See FAQ on the LINPACK Benchmark. The results are in MFlops (mega floating-point operations per second), so higher is better.

We used a modified version of the Java version (http://www.netlib.org/benchmark/linpackjava/), the C version (http://www.netlib.org/benchmark/linpackc) and ported the Java version to C#:

All these versions of Linpack benchmark are not well optimized (ex. not blocked) and do not exploit potential performance of runtimes/platforms much because the purpose of this benchmarking is not such exploitation but comparison of runtimes/platforms.

Pentium 4

1000 x 1000500 x 500
Linpack (1000x1000) results on Pentium 4 Linpack (500x500) results on Pentium 4
MFlopssecond
C/icc (-O3 -xN)402.5191.66
C/Visual C++ (/Ox /G7 /arch:SSE2)399.9201.67
C/gcc (-O3 -funroll-loops)393.6671.70
C#/.NET (Release)382.0991.75
Java/Sun JDK 1.5.0 Server VM379.9241.76
Java/Sun JDK 1.4.2 Server VM379.2781.763
Java/IBM JDK 1.4.2374.3931.786
Java/IBM JDK 1.3.1360.8561.853
Java/GCJ (-O2 with .java)357.0031.873
Java/GCJ (-O2 with .class)348.8091.917
Java/Sun JDK 1.5.0 Client VM339.9421.967
Java/Sun JDK 1.4.2 Client VM339.081.972
Java/JRockit (-server)318.2612.101
Java/JRockit (-client)318.112.102
C#/Mono 1.1.3 (JIT)313.1132.136
Java/Jikes RVM309.9752.207
C#/Mono 1.0.2 (JIT)230.6512.899
Java/Kaffe211.0023.169
C#/.NET (Debug)195.4093.422
Java/GCJ (-O0 with .java)151.1794.423
Java/OpenJIT130.3445.13
Java/GCJ (-O0 with .class)92.3067.244
Java/shuJIT79.8038.379
Java/TYA71.8159.311
Java/interpreter26.00625.712
Java/sunwjit15.83342.233
C#/Mono 1.1.3 (interpreter)9.4770.611
C#/Mono 1.0.2 (interpreter)6.591101.452
MFlopssecond
C/Visual C++ (/Ox /G7 /arch:SSE2)448.3070.19
C/icc (-O3 -xN)433.9740.19
C/gcc (-O3 -funroll-loops)413.4550.20
Java/GCJ (-O2 with .java)359.80.233
Java/JRockit (-server)349.3060.24
Java/JRockit (-client)349.3060.24
Java/GCJ (-O2 with .class)349.3060.24
Java/Sun JDK 1.5.0 Server VM319.9750.262
Java/IBM JDK 1.4.2317.5510.264
Java/Sun JDK 1.5.0 Client VM317.5510.264
Java/IBM JDK 1.3.1312.8110.268
Java/Sun JDK 1.4.2 Client VM312.8110.268
C#/Mono 1.1.3 (interpreter)301.9980.278
Java/Sun JDK 1.4.2 Server VM297.2810.282
C#/.NET (Release)282.3860.297
C#/Mono 1.0.2 (JIT)222.4290.377
Java/Kaffe198.1880.423
Java/Jikes RVM192.2780.436
C#/.NET (Debug)162.5880.516
Java/GCJ (-O0 with .java)151.5970.553
Java/OpenJIT129.9740.645
Java/GCJ (-O0 with .class)90.9260.922
Java/shuJIT76.771.092
Java/TYA70.5671.188
Java/interpreter25.6533.268
Java/sunwjit15.4535.425
C#/Mono 1.1.3 (interpreter)8.6769.663
C#/Mono 1.0.2 (interpreter)6.56812.764

Opteron

Linpack (500x500 and 1000x1000) results on Opteron
1000 x 1000500 x 500
AMD binaryx86 binaryAMD binaryx86 binary
MFlopssecondMFlopssecondMFlopssecondMFlopssecond
Sun JDK 1.5.0 Server VM 304.2162.198297.5822.247311.6480.269318.7580.263
JRockit (-server) 273.3712.446266.9850.314
JRockit (-client) 273.262.447268.6970.312
GCJ (-O2 with .java) 276.8812.415240.4412.781289.080.29250.9980.334
GCJ (-O2 with .class) 275.1712.43220.6823.03287.10.292225.9660.371
Sun JDK 1.5.0 Client VM 217.6653.072 222.9610.376
GCJ (-O0 with .java) 103.9596.43293.6257.142103.7540.80894.0890.891
GCJ (-O0 with .class) 64.81210.31764.51210.36565.2911.28464.7361.295
interpreter 14.36546.54813.95447.91814.2095.913.7846.082

Eratosthenes Sieve (in Java and C#)

This benchmark is the Sieve.java in the TYA archive. These values are number of operations per 10 seconds, so higher is better.

Pentium 4

Eratosthenes Sieve results on Pentium 4
IBM JDK 1.4.213327
IBM JDK 1.3.113308
JRockit (-server)13197
GCJ (-O2 with .java)13141
JRockit (-client)12991
C#/.NET (Release)12735
GCJ (-O2 with .class)12057
Sun JDK 1.4.2 Server VM11992
Sun JDK 1.5.0 Server VM11174
Jikes RVM9555
C#/Mono 1.1.3 (JIT)9420
Kaffe9024
C#/.NET (Debug)8217
C#/Mono 1.0.2 (JIT)8047
Sun JDK 1.4.2 Client VM7546
Sun JDK 1.5.0 Client VM7275
OpenJIT6587
sunwjit6269
GCJ (-O0 with .java)5488
shuJIT5005
TYA4756
GCJ (-O0 with .class)4350
interpreter515
C#/Mono 1.0.2 (interpreter)343
C#/Mono 1.1.3 (interpreter)337

Opteron

AMD64 binaryx86 binary
Eratosthenes Sieve results on Opteron (AMD64 binary) Eratosthenes Sieve results on Opteron (x86 binary)
GCJ (-O2 with .java)13522
GCJ (-O2 with .class)11934
Sun JDK 1.5.0 Server VM10736
GCJ (-O0 with .java)3471
GCJ (-O0 with .class)2830
interpreter537
JRockit (-server)13460
JRockit (-client)13412
Sun JDK 1.5.0 Server VM12906
GCJ (-O2 with .java)12771
GCJ (-O2 with .class)10653
Sun JDK 1.5.0 Client VM6631
GCJ (-O0 with .java)3934
GCJ (-O0 with .class)3478
interpreter552

Related Resources

The Java Performance Report
http://www.javalobby.org/members/jpr/
Java Grande Forum
http://www.javagrande.org/
Java Numerics (at NIST)
http://math.nist.gov/javanumerics/
validgh Java numerics page
http://www.validgh.com/java/
Java in Science and Engineering list, Java Grande Forum list
http://mailer.csit.fsu.edu/mailman/listinfo

Copyright (C) 2004 Kazuyuki Shudo
<shudo at computer.org>