Knowledge base
The key idea is that good embeddings to predict triples in a knowledge base are those that cluster similar synsets together. By training a model on knowledge base completion task, researchers hope to find good synset embeddings as byproduct.
Given a triple, e.g. (cat, has_part, tail), let l denotes the left synset, r is the right synset and t is the relation between them.
In the literature, there are two ways to formalize this task:
- Margin-based: A scoring function g should score observed triples greater than random triples at least to some margin, hence: maximize where in L, left entities are replaced by random entities while in R, right entities are randomized .
- Negative sampling: A function f is trained to differentiate between a triple drawn from correct distribution D and one drawn from a uniform distribution N. For example, stands for the probability that a triple comes from D then we minimize negative log-likelihood: .
Many models have been proposed (see Yang et al., 2014[1] for a review):
- Unstructured[2]: Treat all relations indifferently
- RESCAL[3]: TODO
- SE[4]: A relation is represented by two matrices, working as linear transformation on each side:
- SME(LINEAR)[2]: A relation is represented by two vectors, the semantic matching energy function compare two sides:
- SME(BILINEAR)[2]: The representation of relations stays the same but weights are rank 3 tensors:
- LFM[5]: TODO
- TransE[6]: A relation is a translation of the left hand side to the right hand side:
- Neural tensor network[7]: Relations are represented as rank-3 tensors, score consists of two parts: "tensor" part is the tensor product of them with a relation and "neural" part adds up linear combination of entities:
Knowledge base + Text
References
- ↑ Yang, B., Yih, W., He, X., Gao, J., & Deng, L. (2014). Embedding Entities and Relations for Learning and Inference in Knowledge Bases, 12. Computation and Language. Retrieved from http://arxiv.org/abs/1412.6575
- ↑ 2.0 2.1 2.2 A. Bordes, X. Glorot, J. Weston, and Y. Bengio. A semantic matching energy function for learning with multi-relational data. Machine Learning, 2013.
- ↑ M. Nickel, V. Tresp, and H.-P. Kriegel. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning (ICML), 2011.
- ↑ A. Bordes, J.Weston, R. Collobert, and Y. Bengio. Learning structured embeddings of knowl- edge bases. In Proceedings of the 25th Annual Conference on Artificial Intelligence (AAAI), 2011.
- ↑ R. Jenatton, N. Le Roux, A. Bordes, G. Obozinski, et al. A latent factor model for highly multi-relational data. In Advances in Neural Information Processing Systems (NIPS 25), 2012.
- ↑ Bordes, A., Usunier, N., Weston, J., & Yakhnenko, O. (2013). Translating Embeddings for Modeling Multi-relational Data. In N (pp. 1–9).
- ↑ R. Socher, D. Chen, C. D. Manning, and A. Y. Ng. Learning new facts from knowledge bases with neural tensor networks and semantic word vectors. In Advances in Neural Information Processing Systems (NIPS 26), 2013.
- ↑ Wang, Z., Zhang, J., Feng, J., & Chen, Z. (2014). Knowledge Graph and Text Jointly Embedding. In The 2014 Conference on Empirical Methods on Natural Language Processing. ACL – Association for Computational Linguistics. Retrieved from http://research.microsoft.com/apps/pubs/default.aspx?id=228269
- ↑ Bordes, A., & Weston, J. (2012). Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing, 22, 127–135.