Evaluating application of large language models to biomedical patent claim generation

Abstract

Automatic patent claim generation is an emerging application of large language models (LLMs). However, the performances of general-purpose LLMs in this regard remain unclear. Here we empirically evaluate the effectiveness of four different LLMs (two from the LLaMA-2 family and two from the Mistral family) in generating biomedical patent claims. This allows comparisons between LLMs with different sizes and architectures. We show that these open-source LLMs fail to produce correctly styled patent claims despite their reported strengths in natural language tasks. Nevertheless, given selected training data and adequate fine-tuning, even relatively small LLMs can yield high-quality, correctly styled patent claims. Notably, one limitation of LLMs is that they lack the creativity and insights of human drafters. For such a professional task as claim drafting, LLMs should be considered as a digital assistant that requires human oversight.