(IAA) Tag: sourceedit |
m (typo) Tag: sourceedit |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
span to S.</ref> |
span to S.</ref> |
||
| + | The algorithm is simple: find subtrees that cover annotated spans and nothing else. Ideally, we can find exactly one subtree for each FEE and FE but due to parse error and other problems, the coverage is not 100%. Researchers therefore consider more than one parses, selecting one with best coverage. In Das et al. (2014), the authors found out that this method can handle 80% cases. |
||
| ⚫ | Fürstenau (2008)<ref>Fürstenau, H. (2008). Enriching Frame Semantic Resources with Dependency Graphs. Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), 1478–1484. Retrieved from http://www.lrec-conf.org/proceedings/lrec2008/</ref> first proposed simple rules to map annotations of FrameNet and SALSA (a German semantically annotated corpus) into dependency structures. Their |
||
| + | |||
| ⚫ | Fürstenau (2008)<ref>Fürstenau, H. (2008). Enriching Frame Semantic Resources with Dependency Graphs. Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), 1478–1484. Retrieved from http://www.lrec-conf.org/proceedings/lrec2008/</ref> first proposed simple rules to map annotations of FrameNet and SALSA (a German semantically annotated corpus) into dependency structures. Their restricted their attention to verbal FEE's and included ad-hoc rules for cyclic dependency paths. The coverage was low: not above 65.1%. They sampled mapped and unmapped examples (100 each) for evaluation and found out that: annotated spans tend to select better parses<ref group=note>Fürstenau (2008), page 1481: "Since correctness does not seem to be affected, this indicates that the additional bracketing information, which is implicit in semantic role annotations in so far as it correlates with constituent boundaries, is able to improve the parse selection process considerably. This is because wrong higher ranking parses are discarded in favour of correct lower ranking ones if the annotation can be mapped to the latter but not the former."</ref> and around 80% unmapped roles are due to parser errors. |
||
Bauer et al. (2012)<ref>Bauer, D., Fürstenau, H., & Rambow, O. (2012). The Dependency-Parsed FrameNet Corpus. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), 3861–3867.</ref> extended previous work by including non-verbal FEE's. However, results reported in this work is not comparable to Fürstenau (2008) in some respects. First, while Fürstenau (2008) accepted only subtrees that exactly match annotated spans, Bauer et al. (2012) allowed partial matches. They chose the parse (among 30 of them) that maximizes F<sub>β</sub> of the match between the nodes in annotated spans and the dominated nodes in subtrees (they didn't report this value). Second, they used a different evaluation methodology in which they hand-annotated the correct heads of 400 sentences and measure accuracy against this data. They found an accuracy of 91% (with an inter-annotator agreement of 94%). |
Bauer et al. (2012)<ref>Bauer, D., Fürstenau, H., & Rambow, O. (2012). The Dependency-Parsed FrameNet Corpus. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), 3861–3867.</ref> extended previous work by including non-verbal FEE's. However, results reported in this work is not comparable to Fürstenau (2008) in some respects. First, while Fürstenau (2008) accepted only subtrees that exactly match annotated spans, Bauer et al. (2012) allowed partial matches. They chose the parse (among 30 of them) that maximizes F<sub>β</sub> of the match between the nodes in annotated spans and the dominated nodes in subtrees (they didn't report this value). Second, they used a different evaluation methodology in which they hand-annotated the correct heads of 400 sentences and measure accuracy against this data. They found an accuracy of 91% (with an inter-annotator agreement of 94%). |
||
| + | |||
| + | The advantages of the corpus of Bauer et al. (2012) include availability<ref group=note>One can find the original dataset [http://www1.ccls.columbia.edu/~rambow/resources/parsed_framenet here] or a copy [https://bitbucket.org/minhlab/dependency-parsed-framenet here].</ref> and ease of use (CoNLL format). |
||
| + | If one uses [[Dependency-based SRL evaluation|dependency-based evaluation]], this dataset is a good choice since the accuracy of identified heads are reported clearly (but note that there is no existing results to compare with). However, if he wants to evaluate against spans (to compare to the literature, for example), this is a poor choice since the conversion from dependency nodes back to spans is surely imperfect and it is hard to estimate the range of error. |
||
== References == |
== References == |
||
Latest revision as of 12:44, 30 November 2015
FrameNet corpus comes without syntactic parse. Annotations are defined over text spans instead. To make use of dependency parse in semantic role labeling, an information source that has proven useful (Johansson & Nugues, 2008[1]), researchers need to align annotated spans to dependency nodes. Even in Das et al. (2014)[2], an approach that works directly on text spans, the authors consider only spans that corresponds to a valid subtree.[note 1]
The algorithm is simple: find subtrees that cover annotated spans and nothing else. Ideally, we can find exactly one subtree for each FEE and FE but due to parse error and other problems, the coverage is not 100%. Researchers therefore consider more than one parses, selecting one with best coverage. In Das et al. (2014), the authors found out that this method can handle 80% cases.
Fürstenau (2008)[3] first proposed simple rules to map annotations of FrameNet and SALSA (a German semantically annotated corpus) into dependency structures. Their restricted their attention to verbal FEE's and included ad-hoc rules for cyclic dependency paths. The coverage was low: not above 65.1%. They sampled mapped and unmapped examples (100 each) for evaluation and found out that: annotated spans tend to select better parses[note 2] and around 80% unmapped roles are due to parser errors.
Bauer et al. (2012)[4] extended previous work by including non-verbal FEE's. However, results reported in this work is not comparable to Fürstenau (2008) in some respects. First, while Fürstenau (2008) accepted only subtrees that exactly match annotated spans, Bauer et al. (2012) allowed partial matches. They chose the parse (among 30 of them) that maximizes Fβ of the match between the nodes in annotated spans and the dominated nodes in subtrees (they didn't report this value). Second, they used a different evaluation methodology in which they hand-annotated the correct heads of 400 sentences and measure accuracy against this data. They found an accuracy of 91% (with an inter-annotator agreement of 94%).
The advantages of the corpus of Bauer et al. (2012) include availability[note 3] and ease of use (CoNLL format). If one uses dependency-based evaluation, this dataset is a good choice since the accuracy of identified heads are reported clearly (but note that there is no existing results to compare with). However, if he wants to evaluate against spans (to compare to the literature, for example), this is a poor choice since the conversion from dependency nodes back to spans is surely imperfect and it is hard to estimate the range of error.
References[]
- ↑ Johansson, R., & Nugues, P. (2008). The effect of syntactic representation on semantic role labeling. In Proceedings of the 22nd International Conference on Computational Linguistics - COLING ’08 (Vol. 1, pp. 393–400). doi:10.3115/1599081.1599131
- ↑ Das, D., Chen, D., Martins, A. F. T., Schneider, N., & Smith, N. A. (2014). Frame-semantic parsing. Computational Linguistics, 40(1), 9–56. doi:10.1162/COLI
- ↑ Fürstenau, H. (2008). Enriching Frame Semantic Resources with Dependency Graphs. Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), 1478–1484. Retrieved from http://www.lrec-conf.org/proceedings/lrec2008/
- ↑ Bauer, D., Fürstenau, H., & Rambow, O. (2012). The Dependency-Parsed FrameNet Corpus. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), 3861–3867.
Notes[]
- ↑ Das et al. (2014), page 31: "In principle, S could contain any subsequence of x, but in this work we only consider the set of contiguous spans that (a) contain a single word or (b) comprise a valid subtree of a word and all its descendants in the dependency parse produced by the MST parser. This covers approximately 80% of arguments in the development data for both data sets." [...] In training, if a labeled argument is not a subtree of the dependency parse, we add its span to S.
- ↑ Fürstenau (2008), page 1481: "Since correctness does not seem to be affected, this indicates that the additional bracketing information, which is implicit in semantic role annotations in so far as it correlates with constituent boundaries, is able to improve the parse selection process considerably. This is because wrong higher ranking parses are discarded in favour of correct lower ranking ones if the annotation can be mapped to the latter but not the former."
- ↑ One can find the original dataset here or a copy here.