A five-strong team including Marton Ribary and Eugenio Vaccari (RHUL-LAC) published and presented their experimental findings at JURIX 2023 in Maastricht (NL) about an AI-tool with NLP and LLM components which provides automated legal advice for small businesses in financial distress.
The gold open-access triple peer-reviewed paper with the title “Prompt Engineering and Provision of Context in Domain Specific Use of GPT” was authored by Marton Ribary (RHUL), Eugenio Vaccari (RHUL), Paul Krause (Surrey), Miklos Orban (OPL gunnercooke/Surrey) and Thomas Wood (Fast Data Science/Ulster) and published in Legal Knowledge and Information Systems (Frontiers in Artificial Intelligence and Applications 379). The volume is the edited version of the accepted peer-reviewed papers presented at JURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18-20 December 2023.
This annual conference has become recognized as an international forum where academics and professionals working at the intersection of law and artificial intelligence can exchange knowledge and experience. A total of 92 submissions were received for the conference, of which 18 were selected as long papers, 30 as short papers and 7 as demo papers following a rigorous review process. Topics covered include formal approaches applied to various aspects of legal reasoning; machine learning and information retrieval methods applied to various natural language processing tasks; hybrid approaches to working on the frontier between symbolic and sub-symbolic methods; experimental inquiries into the interfaces between computational systems and legal systems; and network analysis in law.
Our paper experimented with an AI-tool for insolvency law. Large Language Models (LLMs) can appear to generate expert advice on legal matters. However, at closer analysis, some of the advice provided has proven unsound or erroneous. We tested LLMs’ performance in the procedural and technical area of insolvency law in which our team has relevant expertise. This paper demonstrates that statistically more accurate results to evaluation questions come from a design which adds a curated knowledge base to produce quality responses when querying LLMs. We evaluated our bot head-to-head on an unseen test set of twelve questions about insolvency law against the unmodified versions of gpt-3.5-turbo and gpt-4 with a mark scheme similar to those used in examinations in law schools. On the “unseen test set”, the Insolvency Bot based on gpt-3.5-turbo outper-formed gpt-3.5-turbo (p = 1.8%), and our gpt-4 based bot outperformed unmodified gpt-4 (p = 0.05%). These promising results can be expanded to cross-jurisdictional queries and be further improved by matching on-point legal information to user queries. Overall, they demonstrate the
importance of incorporating trusted knowledge sources into traditional LLMs in answering domain-specific queries.
Read the full article (open-access) here: https://doi.org/10.3233/FAIA230979
The book presents the proceedings of JURIX 2023, the 36th International Conference on Legal Knowledge and Information Systems, held from 18–20 December 2023 in Maastricht, the Netherlands.
