The Data Knight Rises

Here, changing the word "Which" to "Whose" ends up fooling the model, possibly because Carolina is a reasonably common first name, and the answer to a question starting with "Whose" very frequently includes a person's name. The same question, if asked to a human, would most likely elicit a correct response.

Experimental Setup

After generating some adversarial examples, we trained baselines with different amounts of training data, evaluated them, and then fine-tuned on the adversarial data.

Evaluating the fine-tuned models gave us better performance on the test set, and hence, reason to have increased confidence in our Hypothesis:

• The Question Answering System improves with the amount of training data, but even by fine- tuning on only ~100 Adversarial examples, the model consistently outperforms the baseline.

A natural question to ask would be the following:

• What's so special about your "Adversarial" Data? How do we know the same effect could not have been achieved just by using more samples from the training set?

We asked precisely this question and ran control experiments to figure it out. In the control, we used around 100 randomly chosen examples from the training dataset that had not yet been seen by any of the baselines and fine-tuned on them instead of fine-tuning on the Adversarial data. The results showed that the resulting model tends to be very similar to the baselines, sometimes deteriorating in performance but never as good as fine-tuning on Adversarial Data. Intuitively, Adversarial examples are guaranteed to make the model slip up, like a challenging question on a test. The examples used in the control experiments are not guaranteed to be as "challenging", perhaps thus adding lesser value in terms of how much the model can learn from them.

Generative AI (LLMs) for Question Answering

Today, with models like GPT-3.5 (ChatGPT!), their general sense of awareness and realistic sounding dialogue make it tempting to trust whatever the model says as being true. However, this is known to be untrue. Large Language Models (LLMs) are very good at generating language that sounds natural, but they are by no means required to generate content that is factually correct all the time. In a certain sense, Large Language Models (LLMs) hallucinate a reality based on the prompt given to them.

To ensure reliability in systems like this, it is important to carefully engineer a prompt with all the right information, and only the relevant information. The models also have a maximum number of words they can process at the same time. A good way of doing this would be the following:

• Split the input into chunks of a manageable size and store the chunks.
• Given a question, find out the top 3 or 4 chunks in the input that are most "related" to the question.
• Patch together the relevant chunks and the question into a prompt for the LLM and give that as input to it.

We did precisely this, and developed a prototype question-answering system that can be used with large code documentations and be asked to explain how to use certain features without the user having to spend hours sifting through the documentation manually.

Conclusion

The journey of enhancing question-answering systems through Adversarial Machine Learning and integrating them with Large Language Models has been both challenging and rewarding. By exposing the model to challenging examples and guiding its learning process, we've witnessed visible improvements in its performance. This approach not only helps us build more robust systems, but also sheds light on the underlying mechanisms of AI learning. As we continue to enhance these techniques and explore new possibilities, the outlook for AI-driven decision-making at SEB looks promising. This could lead to faster and more accurate access to vital information across the bank's diverse domains.