last-updated: November 20 21:55, 2005 JST
All results are also available in Microsoft Excel (2003) book format: `JIT-eval-20041217.xls'.
SPEC JVM98 shows scores for each benchmark program and the geometric mean of them. Higher number is better.
Execution time of each benchmark program is rather short on a today's computer. For example, with IBM JDK 1.4.2, _202_jess, _222_mpegaudio and _228_jack finish only in 2 seconds and _227_mtrt takes only 0.83 second. SPEC JVM98 seems not to be able to evaluate advanced optimizing techniques for server-side applications like profile-guided optimizations.
_201_compress and _213_jack benchmarks are very sensitive to where those programs and data for them are read from. Thus the results of these two benchmarks cannot be compared fairly among all runtime systems.
Initial and maximum heap sizes remain as those default values. Note that one exception is JDK 1.5.0 for x86 on Opteron, whose default values are not enough to run SPEC JVM98 and initial and maximum heap sizes are set as 128 MB.
TYA 1.8 cannot run _200_check, _228_jack and _213_javac and then we use TYA 1.7v3.
Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean IBM JDK 1.4.2 557 256 324 74.1 650 247 165 265 IBM JDK 1.3.1 509 218 332 73.2 527 235 140 241 Sun JDK 1.5.0 Server VM 496 331 300 46.3 533 222 91.9 220 Sun JDK 1.4.2 Server VM 504 332 294 45.5 513 199 98.8 217 JRockit (-server) 265 289 282 73.7 485 204 123 212 JRockit (-client) 193 215 260 70.2 476 203 123 190 Sun JDK 1.5.0 Client VM 325 221 204 43.4 251 192 96.1 162 Sun JDK 1.4.2 Client VM 318 186 199 43.6 249 181 90.9 155 GCJ (-O2) 122 87.0 187 29.0 375 N/A 56.1 103 Jikes RVM 109 124 182 38.0 195 94.2 52.7 98.6 GCJ (-O0) 108 81.9 120 28.7 108 N/A 54.1 75.0 OpenJIT 82.1 60.7 177 16.1 123 52.7 42.5 62.9 shuJIT 45.8 53.2 117 17.2 114 51.0 37.2 52.2 Kaffe 32.0 21.3 191 24.8 101 32.9 21.3 41.9 TYA 33.9 46.1 83.4 15.1 67.7 43.5 34.4 41.1 sunwjit 15.5 23.8 145 16.3 86.6 43.8 33.7 37.8 interpreter 19.6 21.2 13.0 12.4 14.7 28.4 17.2 17.4
Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean Sun JDK 1.5.0 Server VM 650 270 354 46.6 538 351 114 251 GCJ (-O2) 207 108 260 35.4 429 N/A 77.1 138 GCJ (-O0) 133 88.8 99.8 34.1 75.7 N/A 70.1 77.3 interpreter 27.0 18.3 10.7 10.2 12.4 30.2 16.4 16.5
Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean JRockit (-server) 450 286 307 62.5 441 316 161 247 Sun JDK 1.5.0 Server VM 611 308 265 50.0 481 344 106 238 JRockit (-client) 236 256 264 64.2 453 298 161 217 Sun JDK 1.5.0 Client VM 467 236 228 46.2 262 294 97.4 189 GCJ (-O2) 133 99.6 187 35.1 311 N/A 76.1 113 GCJ (-O0) 108 86.3 107 33.9 75.9 N/A 69.9 75.0 interpreter 26.6 18.4 11.1 11.0 12.1 31.5 16.7 16.8
SciMark 2.0 shows results in MFlops (mega floating-point operations per second) and the composite score, which is the arithmetic mean of them. Higher number is better. Therefore, SOR showing relatively higher value affects the composite score much. Oppositely, FFT and Monte Carlo are not counted very much.
We tried "-O2", "-O3" and "-O3 -funroll-loops" options with gcc and "-O2" yielded the best numbers. Similarly, "-O2", "-O3" and "-O3 -xN" tried with icc and "-O2" was the best.
Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score C/Visual C++ 27.7 761 90.9 384 384 329 Java/Sun JDK 1.4.2 Server VM 38.1 742 115 356 366 324 Java/Sun JDK 1.5.0 Server VM 34.4 741 94.3 347 366 316 C/icc (-O2) 30.9 641 85.5 391 380 306 C/gcc (-O2) 31.4 451 84.2 385 378 266 Java/IBM JDK 1.4.2 32.9 451 84.0 315 370 250 Java/JRockit (-server) 35.2 430 112 307 358 249 Java/IBM JDK 1.3.1 34.7 449 62.1 320 367 247 Java/JRockit (-client) 37.0 430 113 290 359 246 C#/.NET (Release) 25.9 454 46.9 323 348 240 Java/GCJ (-O2 with .java) 31.9 409 14.6 281 353 218 Java/GCJ (-O2 with .class) 31.9 409 14.4 238 338 206 Java/Sun JDK 1.5.0 Client VM 34.3 413 54.1 164 327 199 Java/Jikes RVM 24.2 412 7.12 261 279 197 Java/Sun JDK 1.4.2 Client VM 34.5 414 38.5 165 325 195 C#/Mono 1.0.2 (JIT) 34.4 355 2.12 178 245 163 C#/Mono 1.1.3 (JIT) 31.8 375 5.24 180 219 162 Java/Kaffe 29.2 226 10.6 95.3 216 115 Java/GCJ (-O0 with .java) 27.7 193 13.1 106 153 98.6 Java/OpenJIT 28.6 124 8.5 123 55.7 68.0 Java/GCJ (-O0 with .class) 23.1 132 11.7 66.1 83.1 63.2 Java/sunwjit 31.8 148 5.73 59.6 18.6 52.7 Java/shuJIT 17.1 100 7.52 58.6 65.4 49.7 Java/TYA 14.2 64.2 5.93 50.3 71.5 41.2 Java/interpreter 6.63 26.9 4.88 19.8 41.9 20.0 C#/Mono 1.1.3 (interpreter) 4.11 18.4 0.83 9.98 7.46 8.16 C#/Mono 1.0.2 (interpreter) 4.22 16.4 0.83 8.24 8.12 7.56
Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score Sun JDK 1.5.0 Server VM 36.9 669 73.3 249 299 265 GCJ (-O2 with .class) 38.2 481 37.0 235 288 216 GCJ (-O2 with .java) 38.1 481 32.4 229 285 213 GCJ (-O0 with .java) 24.4 146 21.6 80.2 109 76.3 GCJ (-O0 with .class) 21.1 120 17.0 70.5 62.5 58.2 interpreter 5.84 24.5 3.73 15.1 20.2 13.9
Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score Sun JDK 1.5.0 Server VM 37.8 659 64.4 234 300 259 JRockit (-server) 41.6 463 121 258 301 237 JRockit (-client) 41.5 468 121 254 289 235 GCJ (-O2 with .class) 37.7 482 29.8 228 293 214 GCJ (-O2 with .java) 38.1 476 28.9 217 302 212 Sun JDK 1.5.0 Client VM 33.9 479 41.2 186 204 189 GCJ (-O0 with .java) 23.4 147 20.9 80.9 99.9 74.5 GCJ (-O0 with .class) 19.9 118 16.4 66.4 65.6 57.3 interpreter 5.52 21.7 3.62 13.7 19.5 12.8
We used a modified version of the Java version (http://www.netlib.org/benchmark/linpackjava/), the C version (http://www.netlib.org/benchmark/linpackc) and ported the Java version to C#:
All these versions of Linpack benchmark are not well optimized (ex. not blocked) and do not exploit potential performance of runtimes/platforms much because the purpose of this benchmarking is not such exploitation but comparison of runtimes/platforms.
1000 x 1000 500 x 500 AMD binary x86 binary AMD binary x86 binary MFlops second MFlops second MFlops second MFlops second Sun JDK 1.5.0 Server VM 304.216 2.198 297.582 2.247 311.648 0.269 318.758 0.263 JRockit (-server) 273.371 2.446 266.985 0.314 JRockit (-client) 273.26 2.447 268.697 0.312 GCJ (-O2 with .java) 276.881 2.415 240.441 2.781 289.08 0.29 250.998 0.334 GCJ (-O2 with .class) 275.171 2.43 220.682 3.03 287.1 0.292 225.966 0.371 Sun JDK 1.5.0 Client VM 217.665 3.072 222.961 0.376 GCJ (-O0 with .java) 103.959 6.432 93.625 7.142 103.754 0.808 94.089 0.891 GCJ (-O0 with .class) 64.812 10.317 64.512 10.365 65.291 1.284 64.736 1.295 interpreter 14.365 46.548 13.954 47.918 14.209 5.9 13.784 6.082
IBM JDK 1.4.2 13327 IBM JDK 1.3.1 13308 JRockit (-server) 13197 GCJ (-O2 with .java) 13141 JRockit (-client) 12991 C#/.NET (Release) 12735 GCJ (-O2 with .class) 12057 Sun JDK 1.4.2 Server VM 11992 Sun JDK 1.5.0 Server VM 11174 Jikes RVM 9555 C#/Mono 1.1.3 (JIT) 9420 Kaffe 9024 C#/.NET (Debug) 8217 C#/Mono 1.0.2 (JIT) 8047 Sun JDK 1.4.2 Client VM 7546 Sun JDK 1.5.0 Client VM 7275 OpenJIT 6587 sunwjit 6269 GCJ (-O0 with .java) 5488 shuJIT 5005 TYA 4756 GCJ (-O0 with .class) 4350 interpreter 515 C#/Mono 1.0.2 (interpreter) 343 C#/Mono 1.1.3 (interpreter) 337
AMD64 binary x86 binary
GCJ (-O2 with .java) 13522 GCJ (-O2 with .class) 11934 Sun JDK 1.5.0 Server VM 10736 GCJ (-O0 with .java) 3471 GCJ (-O0 with .class) 2830 interpreter 537
JRockit (-server) 13460 JRockit (-client) 13412 Sun JDK 1.5.0 Server VM 12906 GCJ (-O2 with .java) 12771 GCJ (-O2 with .class) 10653 Sun JDK 1.5.0 Client VM 6631 GCJ (-O0 with .java) 3934 GCJ (-O0 with .class) 3478 interpreter 552