Difference between revisions of "How Good are the Specs? A Study of the Bug-Finding Effectiveness of Existing Java API Specifications"

From FSL
Jump to: navigation, search
(Created page with "<private> == Submitted to ICSE NIER'15 == <pubbib id='legunsen-marinov-rosu-2015-icse-submission' template='PubDefaultWithAbstractAndTitle'/> </private>")
(4 intermediate revisions by one user not shown)
Line 1: Line 1:
<pubbib id='legunsen-hassan-xu-rosu-marinov-2016-ase' template='PubDefaultWithAbstractAndTitle'/>
== Submitted to ICSE NIER'15 ==
== Submitted to ASE 2016 ==
<pubbib id='legunsen-marinov-rosu-2015-icse-submission' template='PubDefaultWithAbstractAndTitle'/>
<pubbib id='legunsen-hassan-xu-rosu-marinov-2016-ase-submission' template='PubDefaultWithAbstractAndTitle'/>

Latest revision as of 04:32, 9 July 2016

How Good are the Specs? A Study of the Bug-Finding Effectiveness of Existing Java API Specifications
Owolabi Legunsen and Wajih Ul Hassan and Xinyue Xu and Grigore Rosu and Darko Marinov
ASE 2016, IEEE/ACM, pp 602-613. 2016
Abstract. Runtime verification can be used to find bugs early, during software development, by monitoring test executions against formal specifications (specs). The quality of runtime verification depends on the quality of the specs. While previous research has produced many specs for the Java API, manually or through automatic mining, there has been no large-scale study of their bug-finding effectiveness. We present the first in-depth study of the bug-finding effectiveness of previously proposed specs. We used JavaMOP to monitor 182 manually written and 17 automatically mined specs against more than 18K manually written and 2.1M automatically generated tests in 200 open-source projects. The average runtime overhead was under 4.3x. We inspected 652 violations of manually written specs and (randomly sampled) 200 violations of automatically mined specs. We reported 95 bugs, out of which developers already accepted or fixed 74. However, most violations, 82.81% of 652 and 97.89% of 200, were false alarms. Our empirical results show that (1) runtime verification technology has matured enough to incur tolerable runtime overhead during testing, and (2) the existing API specifications can find many bugs that developers are willing to fix; however, (3) the false alarm rates are worrisome and suggest that substantial effort needs to be spent on engineering better specs and properly evaluating their effectiveness.
PDF, Slides(PDF), JavaMOP, DOI, ASE 2016, BIB

Personal tools