Queryfy: from knowledge graphs to questions using open Large Language Models

3.1 The dataset

For all experiments we used an extended version of the Organizational Graph, a small hand crafted knowledge graph for demonstration purposes. The original graph was developed for [5] and its source is available from the corresponding GitHub Repository.^[5] Using Turtle Syntax (ttl), it describes a fictitious organization called Wonder Org which consists of two departments (research and marketing) and has several employees living in different countries. We expanded it by adding more employees to reinforce the organizational structure that is represented in the graph. The source can be found as org.ttl in the GitHub repository that accompanies this paper.^[6] Its advantages are that it covers a large array of ttl syntax and it is relatively small, making it an easy fit for all current context window sizes, as well as making it easy to verify results by hand.

3.2 Model selection

Since Queryfy is supposed to run on a diverse set of hardware, each with different hardware limitations, we divided its assessment into two sections. First, we start with a limit of 10 billion parameters to analyze its performance with smaller language models. We then conducted a second run with models with up to 34B parameters as this is the maximum number that our current hardware can handle.

For each run, we used the LLM Leaderboard^[7] to find the best performing models that fit within the respective range. If a model offered versions with different context window lengths, we chose the one with the longest window. We also included Microsoft’s Phi-3-Medium model as a canary in the first run because we wanted to see if and how the performance of the models improves with an increasing number of parameters and Phi-3 is one of the best performing models as of now, even in comparison to models far above the 10B parameter limit. The models that were ultimately chosen can be found in Table 1.

We applied the same procedure to select models for the second run. Among these models are larger versions of the previous ones (like Gemma-2-27B), but also new ones like Yi-1.5-34B or InternLM-2.5-20B. The set of models that made it into our pipeline can be seen in Table 2.

3.3 The stages of Queryfy

Queryfy works according a process as visualized in Figure 1. It takes a list of model checkpoints to load from Huggingface as well as a knowledge graph in ttl format as input. In the five steps described in the following it finally generates a json file as explained in Figure 2 as result.

1. Graph2Question (G2Q) Each model is prompted to come up with five questions (this number is configurable). The ttl string of the knowledge graph is part of the prompt. The results are gathered in a list along with metadata about which questions were generated by which model. An example that was generated by Qwen1.5-7b-Chat can be found in the first line of Table 3. The exact prompt we used can be found in Appendix A.1.

2. Question2Answer (Q2A) Each model is prompted to find the answer to every single question directly from the knowledge graph, i.e. not only the questions that it generated itself but also the questions that were generated by all other models. Second line of Table 3 shows an example of a question generated by Phi-3-mini-128k-instruct. See Appendix A.2 for the complete prompt for this stage.

3. Question2SPARQL (Q2S): Iterating over all questions from the list, each model has to generate a SPARQL query via the prompt seen in Appendix A.3 that answers the question.

   An example for a SPARQL query generated by blossom-v5.1-34b can be found in the last line of Table 3.

4. SPARQL2Answer (S2A): The graph was loaded via rdflib and all generated queries were executed against the graph. The results are stored as part of the resulting list, along with any errors that came up during execution. This step does not involve the use of LLMs.

5. Output: This stage currently involves storing the generated questions, queries and answers, after which users have to manually filter the list based on the quality of each question and whether or not the results sets are correct. Research is being done to automate this process based on established criteria from the NLP domain.

Figure 1:

Visualization of the Queryfy pipeline. The first three steps involve the creation of task specific prompts which are sent to a list of LLMs, using the output as the input for the next stage. S2A then executes the generated queries on the given knowledge graph, while output currently consists of formatting and saving the data for later fine-tuning (filtering will be applied at this stage in a future release).

Figure 2:

Close-up of a single entry of the dataset that is produced by Queryfy. Each item consists of four distinct entries as seen in the image and is filtered based on the quality of the natural language question and whether or not the extracted answer matches the one returned by the SPARQL query (both were assessed manually for this paper).

Queryfy contains another optional step called generate_gpt_queries. This helps with testing, but involving GPT runs on production would jeopardize our goal of being vendor independent and avoiding possible data leaking.

For further explanation of the prompts that were used, refer to Appendix A.

4.1 First pipeline execution

The goal of the first run was to assess the current capabilities of models with up to 10B parameters regarding question answering and SPARQL generation over a given knowledge graph. Not all models managed to generate five questions: gemma-7b generated two, mindy-7b gave us three, qwen2-7B three and occiglot-7b two, leaving us with 40 questions in total instead of the expected 50. Out of these 40 questions in total, 9 were discarded for being either too trivial (What is the first name of Charles Turner?), too under-defined (What is the foaf:role of:marketingManager?) or garbage (PREFIX: <https://abc.def/ghi/>), leaving us with 31 questions like:

1. Which department does Charles Turner belong to and what role does he have there?

2. Who is the chief research officer in the organization?

3. What is the country where Francine resides, according to her home address?

The evaluation of the QA capabilities was done manually. For every question we looked at the answers that the LLMs extracted directly from the graph. If a question referred to a person, both the full name as well as the resource designation were accepted as correct (both “Charles Turner” and “:charles”). In case of questions like Who is a member of the Research Department? we accepted mentions of single persons as well as full lists. On the other hand, for the question Who are the members of the Research Department? the models had to provide all names. In any case, if the correct answer was part of the output that the LLM generated and there were no wrong statements within it, we accepted an answer as correct.

We can see from the results in Table 4 that most models are able to answer questions directly if a knowledge graph is given as a source of information. This is an important result for methods like Retrieval Augmented Generation (RAG) because they depend on LLMs being able to extract an answer from a given context. We can also see that models seem to have different capabilities in different areas. For example, while all questions generated by Yi-1.5 were just prefix definitions and therefore unusable, it did really well during question answering. The same can be said about openchat-3.6. All questions started with the string What is the full name of the person who holds the role of which would make for a subpar dataset if this were the only LLM that we used. But it still managed to answer all questions correctly.

When it comes to generating SPARQL queries though, the models struggle a lot. No model was able to generate more than six queries that produced the correct answer to a given question. This is partly due to the fact that these smaller sized LLMs (compared to ChatGPT, Gemini or Claude) tend to ignore commands like Do not generate any other text besides the query. The responses were very chatty and in some cases it was not trivial to extract the raw SPARQL query from the generated text.

Openchat-3.6 and the Qwen models have produced the best results in this run. While they may have only produced a semantically correct query in about 20 % of the time, the amount of syntactic errors they have made is also surprisingly low (only five resp. seven out of 31). Qwen2-7B got the count query wrong and also did not resolve a blank node during one query, returning the ID of the blank node instead. Mistral-7B and Openchat-3.6 also fumbled the count query.

4.2 Second pipeline execution

We kept the questions from the first run and asked the new models to only generate SPARQL queries to answer them. We omitted the question answering (QA) part during this run, because the smaller sized models had already shown very good results and we did not see much room for improvement here. Instead, we focused on the generation of SPARQL queries. The results are shown in Table 5.

Again, no LLM managed to answer the question including a count correctly. However, they were able to use concat if the question asked for full names of people, generating SPARQL snippets like:

BIND(CONCAT(?firstName,’ ’, ?surname) AS ?personName)

We can also very clearly see that increasing the number of parameters leads to an increase in performance. blossom is able to generate SPARQL queries for questions regarding membership in certain departments, showing that it is able to handle blank nodes.