Language Models
Since 2017, my team(s) and I have worked on improving and using (increasingly large) language models. The following fields were particular focus topics:
-
Integration of Knowledge into LLMs: I explored methods for integrating structured knowledge sources, such as knowledge graphs, into LLMs. This integration aims to enhance the factual accuracy and reasoning capabilities of LLMs, addressing their tendency to generate factually incorrect or misleading information. I am working on active and passive knowledge retrieval methods over heterogeneous data sources. A major challenge is to preserve the conversational capabilities and naturalness of LLMs when applying retrieval methods.
-
API-augmentation of LLMs: I am investigating the augmentation of LLMs with external tools/APIs (e.g. triggering actions, obtaining real time information). A main challenge in this area is to teach the LLM to follow complex instructions that are on par or better than humans at orchestrating tool workflows.
-
LLMs as Semantic Parsers: Semantic parsing is the task of translating a natural language (NL) utterance into a machine-interpretable representation, e.g. for querying databases or requesting information from APIs. While semantic parsers have been built on top of language models for a long time, LLMs have led to major changes in the field. I am investigating how to minimise the training data requirements for LLM semantic parsers in order to a) reduce the human effort to collect training data, which is usually the bottleneck in building highly accurate production-ready systems, and b) minimise regression of the LLM on other capabilities, such as creative writing skills, when teaching them to perform semantic parses.
-
LLMs as a path towards Artificial General Intelligence (AGI): In my work, I critically examine the potential of neural language models as a path towards AGI. I am working on approaches to improve the reasoning capabilities of LLMs, e.g. for complex question answering and mathematical reasoning, and equipping them with advanced planning capabilities (beyond their typical next token prediction objective). I am also investigating the responsible further development of LLMs to allow humans to maintain control over AGI technology and ensure that their output is safe, ethical, and desirable for the user.
Related Publications
Journal Articles
SGPT: A Generative Approach for SPARQL Query Generation From Natural Language Questions Journal Article
In: IEEE Access, 10 , pp. 70712–70723, 2022.
Language Model Guided Knowledge Graph Embeddings Journal Article
In: IEEE Access, 10 , pp. 76008–76020, 2022.
Introduction to neural network-based question answering over knowledge graphs Journal Article
In: Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 11 (3), 2021.
Inproceedings
Language Models as Controlled Natural Language Semantic Parsers for Knowledge Graph Question Answering Inproceedings
In: ECAI 2023 - 26th European Conference on Artificial Intelligence, September 30 - October 4, 2023, Kraków, Poland - Including 12th Conference on Prestigious Applications of Intelligent Systems (PAIS 2023), pp. 1348–1356, IOS Press, 2023.
Direct Fact Retrieval from Knowledge Graphs without Entity Linking Inproceedings
In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 10038–10055, Association for Computational Linguistics, 2023.
RoMe: A Robust Metric for Evaluating Natural Language Generation Inproceedings
In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 5645–5657, Association for Computational Linguistics, 2022.
DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue Generation Inproceedings
In: Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 2557–2571, Association for Computational Linguistics, 2022.
Contrastive Representation Learning for Conversational Question Answering over Knowledge Graphs Inproceedings
In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pp. 925–934, ACM, 2022.
VOGUE: Answer Verbalization Through Multi-Task Learning Inproceedings
In: Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III, pp. 563–579, Springer, 2021.
Proxy Indicators for the Quality of Open-domain Dialogues Inproceedings
In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 7834–7855, Association for Computational Linguistics, 2021.
Grounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformers Inproceedings
In: The Semantic Web - 18th International Conference, ESWC 2021, Virtual Event, June 6-10, 2021, Proceedings, pp. 323–339, Springer, 2021.
Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks Inproceedings
In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pp. 850–862, Association for Computational Linguistics, 2021.
Context Transformer with Stacked Pointer Networks for Conversational Question Answering over Knowledge Graphs Inproceedings
In: The Semantic Web - 18th International Conference, ESWC 2021, Virtual Event, June 6-10, 2021, Proceedings, pp. 356–371, Springer, 2021.
CHOLAN: A Modular Approach for Neural Entity Linking on Wikipedia and Wikidata Inproceedings
In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pp. 504–514, Association for Computational Linguistics, 2021.
Language Model Transformers as Evaluators for Open-domain Dialogues Inproceedings
In: Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 6797–6808, International Committee on Computational Linguistics, 2020.
Using a KG-Copy Network for Non-goal Oriented Dialogues Inproceedings
In: The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I, pp. 93–109, Springer, 2019.
Learning to Rank Query Graphs for Complex Question Answering over Knowledge Graphs Inproceedings
In: The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I, pp. 487–504, Springer, 2019.
Improving Response Selection in Multi-Turn Dialogue Systems by Incorporating Domain Knowledge Inproceedings
In: Proceedings of the 22nd Conference on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium, October 31 - November 1, 2018, pp. 497–507, Association for Computational Linguistics, 2018.
Deep Query Ranking for Question Answering over Knowledge Bases Inproceedings
In: Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part III, pp. 635–638, Springer, 2018.
Neural Network-based Question Answering over Knowledge Graphs on Word and Character Level Inproceedings
In: Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pp. 1211–1220, ACM, 2017.
