August 17, 2005
Question
Answering Passage Retrieval Using Dependency Parsing
22/28
Performance Evaluation
•All improvements are statistically significant (p<0.001)
•MI and EM do not make much difference given our training
data
–EM needs more training data
–MI is more susceptible to noise, so may not scale well
0.3889
0.3889
0.3457
0.3364
0.2716
0.2253
0.1759
0.1975
0.1235
Precision at top one passage
24.07%
24.69%
29.32%
29.63%
32.41%
41.96%
33.02%
37.65%
45.68%
% Incorrect
+138.08
+72.19
+77.83
+72.19
+77.83
+137.85
+72.03
+77.66
+72.03
+77.66
+110.94
+52.57
+57.56
+52.57
+57.56
+108.09
+50.50
+55.43
+50.50
+55.43
+81.25
+31.10
+35.41
+31.10
+35.41
+49.50
+8.14
+11.69
+8.14
+11.69
+33.88
N/A
N/A
N/A
N/A
+38.26
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
% MRR improvement over
MITRE
SiteQ
NUS
SiteQ
NUS
0.4761
0.4756
0.4218
0.4161
0.3625
0.2990
0.2677
0.2765
0.2000
MRR
Rel_EM
(NUS)
Rel_MI
(NUS)
Rel_EM
(MITRE)
Rel_MI
(MITRE)
Rel_Strict
(NUS)
(NUS)
Rel_Strict
(MITRE)
(MITRE)
NUS
SiteQ
MITRE
Passage retrieval systems
Fuzzy matching outperforms strict matching significantly.
