Difference in F1 Score Results Between KnowledgeBase and TripleStore Using Same OWL Ontology

[Quannz]

When loading the same OWL ontology(Carcinogenesis) using two different methods (KnowledgeBase and TripleStore), they return different F1 score for the same concept.

Using the Carcinogenesis ontology with the same test concept:

((∃ hasStructure.⊤) ⊓ (∃ mouse_lymph.rdfs:Literal)) ⊔ (∃ salmonella.{true})

the results differ:

KnowledgeBase Results

• F1 Score: 0.746269
• Precision: 0.632911
• Recall: 0.909091
• True Positives: 50
• False Positives: 29
• Total Instances: 259

TripleStore Results

• F1 Score: 0.556962
• Precision: 0.916667
• Recall: 0.400000
• True Positives: 22
• False Positives: 2
• Total Instances: 127

Steps to Reproduce

1. Load ontology using KnowledgeBase from local OWL file
2. Load same ontology using TripleStore from SPARQL endpoint
3. Execute materialization queries on TripleStore (optional)
4. Query the same concept on both knowledge bases
5. Compare F1 scores

Test script can be found here:

test_case_2.py

To download knowledge graphs

wget https://files.dice-research.org/projects/Ontolearn/KGs.zip -O ./KGs.zip && unzip KGs.zip

[Demirrr]

By default, TripleStore does not use a reasoner but KnowledgeBase does. Please ensure to use the same reasoner