Ranking and Reputation Systems in QBF Competition
Massimo Narizzano, Luca Pulina and Armando Tacchella
The 10th Congress of the Italian Association for Artificial Intelligence (AIIA 2007)
Roma, Italy, September 10-13, 2007
Abstract
Systems competitions play a fundamental role in the advancement of the state of the art in several automated reasoning fields. The goal of such events is to answer the question: “Which system should I buy?”. Usually the answer comes as the byproduct of a ranking obtained by considering a pool of problem instances and then aggregating the performances of the systems on each member of the pool. In this paper, we consider voting systems as an alternative to other procedures which are well established in automated reasoning contests. Our research is aimed to compare methods that are customary in the context of social choice, with methods that are targeted to artificial settings, including a new hybrid method that we introduce. Our analysis is empirical, in that we compare the aggregation procedures by computing measures which should account for their effectiveness using the data from the 2005 evaluation of quantified Boolean formulas solvers that we organized. The results of our experiments give useful indications about the relative strengths and weaknesses of the procedures under test, and allow us to infer also some conclusions that are independent of the specific procedure adopted.