JavaMOPCFG

From FSL
Jump to: navigation, search

Authors:Patrick Meredith, Feng Chen, Dongyun Jin, Grigore Rosu

Contents

Overview

This is the main page for the JavaMOP CFG logic plugin experiments from our paper published in ASE 2008. Note that all of the experiments were performed in JavaMOP1.0.

Paper

Efficient Monitoring of Parametric Context-Free Patterns 
Patrick Meredith, Dongyun Jin, Feng Chen and Grigore Rosu
ASE'08, IEEE/ACM, pp 148-157. 2008 ACM Sigsoft Distinguished Paper
PDF, Experiments, ASE'08 slides(KEY), ASE'08 slides(MOV), ASE'08 slides(PPT), IEEE, ASE'08, BIB


Extended Technical Report


Links

Analysis

System

  • CPU: Pentium 4 2.66Ghz
  • RAM: 1.5GB
  • OS: Ubuntu 7.10

Programs

  • JavaMOP: version of Mar 2008
  • PQL: version of Dec 2006
  • Tracematches: version of Dec 2006
  • Tracematches_2008_03: version of Mar 2008

JavaMOP Property files

The JavaMOP property files in this benchmark can be found at Here

PQL Property Files

The PQL property files in this benchmark can be found at Here

Tracematches Property Files

The PQL property files in this benchmark can be found at Here


Monitoring Statistics (JavaMOP with CFGs)

Number of Instrumentation Points
Property HashMap HasNext SafeIterator ImprovedLeakingSync SafeFileInputStream (*) SafeFileInputStream (**) SafeFileWriter
# of Instrumentation Points

(1) + (2) + (3) + (4)

1721+0+60+1

= 1782

82+43+24+137

= 286

5663+89+1061+294

= 7107

6714+9+1572+60

= 8355

16827+0+379+1

= 17207

315+0+23+1

= 339

46+0+10+1

= 57

  • (1) Instrumentation Points in Dacapo
  • (2) Inter-Typed Points in Dacapo
  • (3) Instrumentation Points in Java RT Library(rt.jar)
  • (4) Inter-Typed Points in Java RT Library(rt.jar)
  • (*) After Class-Level Static Analysis
  • (**) After Method-Level Static Analysis
Number of Events
Benchmark HashMap HasNext SafeIterator ImprovedLeakingSync SafeFileInputStream (**) SafeFileWriter
antlr 0 0 1990 8472 0 385
bloat 361519 146628987 75944328 5587905 259 0
chart 8773 0 569345 634260 0 0
eclipse 20888 0 32759 74630 930 0
fop 17265 0 49959 182407 12 0
hsqldb 0 0 0 0 0 0
jython 443 0 177554 23969673 544 0
luindex 9615 0 82162 1559386 1114 0
lusearch 416 0 405428 1291992 0 32
pmd 11354 36163717 25476563 26291289 10 0
xalan 124155 0 1009649 5146036 13604 0
  • (**) After Method-Level Static Analysis
Number of Partial Matching Sets
Benchmark HashMap HasNext SafeIterator ImprovedLeakingSync SafeFileInputStream (**) SafeFileWriter
antlr 0 0 0 0 0 385
bloat 30708 945737 1894581 0 3 0
chart 3075 0 815 0 0 0
eclipse 13431 0 588 0 8 0
fop 6217 0 79 0 0 0
hsqldb 0 0 0 0 0 0
jython 439 0 50 0 48 0
luindex 9615 0 8788 0 0 0
lusearch 416 0 0 0 0 32
pmd 105 5352848 1950212 0 2 0
xalan 78242 0 0 0 0 0
  • (**) After Method-Level Static Analysis
Number of Monitors
Benchmark HashMap HasNext SafeIterator ImprovedLeakingSync SafeFileInputStream (**) SafeFileWriter
antlr 0 0 0 0 0 385
bloat 361436 73232724 1894581 0 3 0
chart 8729 0 815 0 0 0
eclipse 15836 0 588 0 8 0
fop 16980 0 79 0 0 0
hsqldb 0 0 0 0 0 0
jython 443 0 50 0 48 0
luindex 9615 0 8788 0 0 0
lusearch 416 0 0 0 0 32
pmd 116 10093591 1950212 0 2 0
xalan 95252 0 0 0 0 0
  • (**) After Method-Level Static Analysis
Number of Monitors Analysis

We see the largest correspondence between runtime overhead and number of monitors. This is primarily due to indexing and collapsing monitors.

Maximum Memory Usage of MOP

(Maximum Heap Memory Usage) / (Maximum Non-Heap Memory Usage)

Due to Garbage Collecting, there are some variances.

Benchmark Original HashMap HasNext SafeIterator ImprovedLeakingSync SafeFileInputStream (**) SafeFileWriter
antlr 2.3MB / 10.1MB 2.0MB / 10.6MB 1.8MB / 10.6MB 2.0MB / 10.8MB 2.1MB / 10.7MB 2.4MB / 10.7MB 2.2MB / 10.7MB
bloat 5.6MB / 8.9MB 6.9MB / 8.9MB 5.9MB / 8.7MB 541.0MB / 10.6MB 7.9MB / 10.0MB 5.0MB / 8.9MB 5.6MB / 8.9MB
chart 20.1MB / 11.3MB 20.8MB / 11.4MB 17.0MB / 11.3MB 20.7MB / 11.5MB 17.0MB / 11.3MB 17.8MB / 11.3MB 16.4MB / 11.3MB
eclipse 27.0MB / 22.1MB 30.7MB / 22.2MB 27.4MB / 22.1MB 28.6MB / 22.3MB 28.9MB / 22.1MB 30.7MB / 22.1MB 27.1MB / 22.1MB
fop 12.3MB / 9.1MB 13.2MB / 9.2MB 10.9MB / 9.0MB 10.2MB / 9.1MB 14.6MB / 9.2MB 11.9MB / 9.0MB 12.0MB / 9.0MB
hsqldb 80.8MB / 7.6MB 80.2MB / 7.6MB 76.4MB / 7.5MB 77.5MB / 7.6MB 87.2MB / 7.5MB 78.2MB / 7.5MB 79.3MB / 7.5MB
jython 3.9MB / 19.0MB 4.1MB / 19.0MB 3.8MB / 19.0MB 3.9MB / 19.1MB 4.0MB / 19.2MB 4.0MB / 19.1MB 3.6MB / 19.1MB
luindex 4.2MB / 6.9MB 4.0MB / 7.0MB 4.6MB / 6.9MB 4.7MB / 7.1MB 5.6MB / 7.0MB 4.2MB / 6.9MB 4.6MB / 6.9MB
lusearch 5.2MB / 6.2MB 5.2MB / 6.3MB 5.7MB / 6.2MB 5.3MB / 6.3MB 5.8MB / 6.4MB 5.6MB / 6.3MB 5.7MB / 6.3MB
pmd 22.0MB / 8.6MB 22.3MB / 8.7MB 24.0MB / 8.6MB 888.1MB / 8.9MB 22.2MB / 8.8MB 24.2MB / 8.6MB 22.9MB / 8.6MB
xalan 21.7MB / 10.2MB 23.8MB / 10.5MB 26.2MB / 10.2MB 29.1MB / 10.3MB 24.4MB / 10.3MB 22.0MB / 10.3MB 26.5MB / 10.2MB
  • (**) After Method-Level Static Analysis
Memory Usage Analysis

There are two extreme cases about the memory overhead caused by MOP monitors, namely, bloat-SafeIterator and pmd-SafeIterator, as indicated by the red numbers. Our investigation shows that both programs, bloat and pmd, make intensive use of some vectors and create numerous iterators to do computation over the vectors throughout the whole execution. Note that every creation of the iterator leads to the creation of a monitor instance for SafeIterator using our technique. Hence, a huge number of monitor instances were created in these two benchmarks. While the iterator object is usually used in a small scope and then released, the vectors are not released until the end of the execution, preventing the removal of the created monitor instances. In other words, all the monitor instances created during the execution of bloat and pmd were kept alive until the execution ended, resulting in the observed massive memory usage. On the contrary, we can see that a large number of monitors were also created for bloat-HasNext and pmd-HasNext but with much less memory overhead, because for HasNext, one monitor is created for every iterator object and when the iterator is released, the corresponding monitor will also be removed. Since most iterators were released shortly after creation, only a few monitors existed at the same time during the execution and much lower memory overhead was caused.

Compared with the results of Tracehmatches, we believe there is still some room for improvement with regard to memory usage in our approach. We are working on a new technique that is based on the idea of removing dead monitors using static analysis on the specification.

Performance

HashMap
  • Overhead in Percentage(JDK 1.5)
Benchmark MOP Tracematches Tracematches_2008_03 PQL
antlr 3 -1 0 6
bloat 14 -2 -2 9
chart -1 0 -1 1
eclipse 0 -1 1 1
fop 3 0 0 2
hsqldb 0 19 15 3
jython 0 8 15 23
luindex 1 -2 1 8
lusearch 1 11 8 1
pmd -1 0 3 0
xalan 0 2 1 5
HasNext
  • Overhead in Percentage(JDK 1.5)
Benchmark MOP Tracematches Tracematches_2008_03 PQL
antlr 1 -2 3 2
bloat 1112 2452 5929
chart -1 1 0 3
eclipse 0 0 -1 2
fop 0 0 -1 2
hsqldb 0 20 15 6
jython 0 8 13 0
luindex -2 2 2 93
lusearch -1 9 9 59
pmd 191 136 52 1870
xalan 0 1 2 0
SafeIterator
  • Overhead in Percentage(JDK 1.5)
Benchmark MOP Tracematches Tracematches_2008_03 PQL
antlr 2 0 0 82
bloat 627 11258 8694
chart 2 28 11 50
eclipse -2 0 2 1
fop -1 12 5 24
hsqldb 0 20 17 78
jython 0 16 16 12
luindex 3 8 9 181
lusearch 4 46 34 132
pmd 178 115 175 1334
xalan 1 31 10 53
ImprovedLeakingSync
  • Overhead in Percentage(JDK 1.5)
Benchmark MOP Tracematches Tracematches_2008_03 PQL
antlr 1 N/A(*) N/A(*) N/A(**)
bloat 13 N/A(*) N/A(*) N/A(**)
chart 4 N/A(*) N/A(*) N/A(**)
eclipse 1 N/A(*) N/A(*) N/A(**)
fop 1 N/A(*) N/A(*) N/A(**)
hsqldb 1 N/A(*) N/A(*) N/A(**)
jython 41 N/A(*) N/A(*) N/A(**)
luindex 1 N/A(*) N/A(*) N/A(**)
lusearch 2 N/A(*) N/A(*) N/A(**)
pmd 36 N/A(*) N/A(*) N/A(**)
xalan 3 N/A(*) N/A(*) N/A(**)
  • (*) This property cannot be expressed in Tracematches
  • (**) This property cannot be expressed in PQL
SafeFileInputStream
  • Number of FileInputStream objects
Benchmark # of Objects
antlr 0
bloat 3
chart 0
eclipse 8
fop 0
jython 50
luindex 0
lusearch 0
pmd 2
xalan 0
  • Overhead in Percentage(JDK 1.5)
Benchmark MOP

(w/ class static analysis only)

MOP Tracematches_2008_03

(w/ class static analysis only)

Tracematches_2008_03 PQL
antlr 0 3 -2 -1 113
bloat 2 1 -1 0 132
chart 1 0 0 1 29
eclipse 0 -3 -1 0 3
fop -1 -2 -1 -1 58
hsqldb -1 1 21 21 280
jython 2570 0 85 12 937
luindex -1 -1 5 6 233
lusearch 3 -1 4 7 137
pmd -1 -1 0 1 547
xalan -2 -1 3 3 90
SafeFileWriter
  • Overhead in Percentage(JDK 1.5)
Benchmark MOP Tracematches Tracematches_2008_03 PQL
antlr 2 N/A(*) N/A(*) 22
bloat 27 N/A(*) N/A(*) 97
chart 0 N/A(*) N/A(*) 37
eclipse -2 N/A(*) N/A(*) 1
fop -2 N/A(*) N/A(*) 47
hsqldb 2 N/A(*) N/A(*) 95
jython 1 N/A(*) N/A(*) crashes
luindex 0 N/A(*) N/A(*) 33
lusearch 0 N/A(*) N/A(*) 49
pmd -2 N/A(*) N/A(*) 658
xalan -4 N/A(*) N/A(*) 164
  • (*) This property cannot be expressed in Tracematches
Personal tools
Namespaces

Variants
Actions
Navigation