Functional Tokens: Enhancing Language Models for Function Calling

Table of Links

Abstract and 1. Introduction

2 Related works

3 Methodology and 3.1 Causal language model as a classification model

3.2 Functional token

3.3 Dataset collection

3.4 Model development and training

4 Experiments and 4.1 Android function calls

4.2 Extension to Vehicle, Yelp, and DoorDash function sets

4.3 Full and partial training datasets and 4.4 Full training and LoRA training

4.5 Parallel and nested function call and 4.6 Weighted loss function for special tokens

5 Discussion and future works and References

Appendix

A.1 Android function examples

A.2 Vehicle function examples

3.2 Functional token

The critical innovation presented in this paper is the introduction of the functional token. Drawing parallels to the tokenization of natural language, we propose representing specific functions as functional tokens. We introduce a training methodology for these tokens, inspired by techniques used in natural language models for handling rare words. This training approach is inspired by the word2vec [28] framework, where contextual words enrich a token’s semantic representation. For instance, pretrained language models may initially struggle to recognize specialized terms such as PEGylation and Endosomal Escape from the domain of chemistry. However, these models can learn such terms through causal language modeling, leveraging corpora that include these specialized terms. Similarly, functional tokens can be learned using comparable methods. Our experiments have not identified any limitations on the number of functional tokens that can be defined, allowing users to map any specific function to a token.

By incorporating the functional token, we aim to assign the same importance to functional tokens as other linguistic tokens. Unlike conventional linguistic tokens, our functional token does not possess inherent natural language meaning; instead, it represents specific actions encapsulated within the model. For smaller-scale models like Google Gemma 2B, conducting function calling can be challenging, although they excel at typical language completion tasks. The functional token enables the language model to transform a functional calling method into a standard completion task. As we define the actions for the language model, we can also characterize our model as a small action model. The integration of the functional token allows the model to concentrate on a fixed set of actions and perform effectively on these tasks.