100 likes | 170 Views
An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering. Kostadin Damevski , David Shepherd, Lori Pollock. Evaluating Code Search…. Common Approach: Eliminate developers from the evaluation.
E N D
An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering KostadinDamevski, David Shepherd, Lori Pollock
Common Approach: Eliminate developers from the evaluation • The problem with “gold sets”… • Time consuming to create • Created by researchers, not developers • Subjective and context dependent
LET’S CREATE A GOLD SET for “save auctions” JBidMouse.DoSave(Component)JBidMouse.DoAction(Object, String, AuctionEntry)AuctionServer.registerAuction(AuctionEntry) AuctionsManager.saveAuctions()AuctionsManager.backupByDate(String, File) AuctionsManager.preserveFiles(String)AuctionsManager.ensureDirectories(String) AuctionsManager.buildSaveBuffer(XMLElement, XMLElement) AuctionsManager.needSwapSaves(String) AuctionsManager.makeBackupFilename(String String) from jBidWatcher (183 classes)
Our Approach: Humans “Taste-Test” Competing techniques* • *Used to evaluate web search engines
A Taste Test for Code Search The interface does not betray which FLT (A or B) the results Originated from
A Taste Test for Code Search Count clicks to determine a statistically significant preference
Next Steps Verify approach works Use paired interleaving to compare lexical approach to IR-based approach, knowing that the IR-approach should perform better. Widespread data collection* Compare variations on approaches, such as use of different splitters, via widespread distribution and data collection. * Use our Sando Search tool (http://sando.codeplex.com)