What do Large Language Models actually know?
Project Overview
This research project explores the idea of explainability in AI, looking into what makes a large language model (LLM) choose its outcomes, using Explainable AI (XAI) concepts. More specifically, through the analysis of text simplification and comparing tools: Captum AI and Circuit-Tracer libraries.
What were the key results of your research project?
When identifying areas of divergence, it was clear that this stemmed mainly from content word disagreement and, less importantly, punctuation. Thus, it provides invaluable information on the distinction between influence and importance. We can see that there is no significant correlation between per-token attribution and per-token confidence probability in relation to the resulting tokens. With this, we can infer that in one situation, the context surrounding the final answer from the model can be impacted by different tokens in comparison to a single token's overall importance in the final answer.
GitHub repository: https://github.com/jenellebankas/HPC_repo
How do you feel you have benefited from completing this internship and has it made you consider future career paths?
It has made me more confident to explore areas of interest that I do not know much about, and has allowed me to be more confident in my programming abilities. Ultimately, this internship has provided another insight into a career that I had not heard of beforehand, providing another potential path for the future.
Download presentation slides: