Enhancing Robustness: Leveraging Pydantic Validation Decorator with ActionWeaver

The Pydantic [@validate_call decorator](https://docs.pydantic.dev/latest/concepts/validation_decorator/) enables the validation of function arguments based on the function’s annotations before the function is executed.

ActionWeaver, as a framework centered on function calls, oversees the function calling process for users. Specifically, it includes an ExceptionHandler, empowering users to specify behaviors to exceptions. Together, these components form a powerful combination for developing robust LLM applications.

%load_ext autoreload
%autoreload 2

import os

from typing import List
from uuid import UUID, uuid4

from actionweaver import action
from actionweaver.utils.tokens import TokenUsageTracker
from actionweaver.llms import wrap, ExceptionHandler, ExceptionAction, ChatLoopInfo, Continue, Return
from actionweaver.actions.factories.pydantic_model_to_action import  action_from_model

from pydantic import BaseModel, Field, PrivateAttr, validate_call, field_validator

from openai import OpenAI

from datetime import datetime

from openai import AzureOpenAI, OpenAI

Let’s instantiate an OpenAI client and then encapsulate it within an ActionWeaver wrapper. ActionWeaver will handle the function calling loop.

# Azure OpenAI
# MODEL="gpt-4-32k"
# llm = AzureOpenAI(
#     azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT"),
#     api_key=os.getenv("AZURE_OPENAI_KEY"),
#     api_version="2023-10-01-preview"
# )

# OpenAI
MODEL = "gpt-4"
llm = wrap(OpenAI())

Here, we’ll ask LLM to use the ingest_user_info function, which accepts Pydantic model UserModel as arguments. We aim for the LLM to extract user information from natural language and trigger ingest_user_info, which will then store the validated user information into user_db.

Within the Pydantic model UserModel, we’ve specified certain field_validators to ensure that both the name and phone_number adhere to specific formats.

import re
from pydantic import BaseModel, ValidationError, validator

class UserModel(BaseModel):
    _uid: UUID = PrivateAttr(default_factory=uuid4)
    _created_at: datetime = PrivateAttr(default_factory=datetime.now)
    name: str
    phone_number: str

    @field_validator('name')
    @classmethod
    def validate_name(cls, v: str) -> str:
        # Split the input string into first and last names
        names = v.split()

        # Check if both first and last names are present
        if len(names) != 2:
            raise ValueError('Name must contain a first name and a last name separated by a space')

        # Check if the formatted name is not in uppercase
        if v != v.upper():
            raise ValueError('Name must be in uppercase')

        return v

    @field_validator('phone_number')
    @classmethod
    def validate_phone_number(cls, v: str) -> str:
        # Define a regular expression pattern for a phone number with country code
        pattern = r'^\+\d{1,3}\s*\(\d{3}\)\s*\d{3}-\d{4}$'  # Example: +1 (XXX) XXX-XXXX

        # Check if the phone number matches the pattern
        if not re.match(pattern, v):
            raise ValueError('phone number must be in the format +1 (XXX) XXX-XXXX')
        return v

user_db = []

@action(name="SaveUserInfo", stop=True)
@validate_call
def ingest_user_info(users: List[UserModel]):
    """Save user info to database"""
    user_db.append(users)
    return "success"

Now, let’s attempt to prompt the LLM to call ingest_user_info. We use the following syntax to force the LLM to call function:

ingest_user_info.invoke(
    llm,
    messages=messages,
    model=MODEL,
    stream=False,
    temperature=1,
    exception_handler = ExceptionRetryHandler(3)
)

For further information about this syntax, refer to the documentation at: https://github.com/TengHu/ActionWeaver?tab=readme-ov-file#force-execution-of-an-action

---

As you’ll notice, the input text has a different format and may not pass the field validation.

To enable the LLM to handle this, we’ll define an ExceptionRetryHandler and pass it as argument. This handler will take the exception message as input to the LLM and allow for a maximum number of retries. It empowers the LLM to “self-heal” using the error message and return the desired result.

class ExceptionRetryHandler(ExceptionHandler):
    def __init__(self, max_retry=2):
        self.max_retry = max_retry

    def handle_exception(self, e: Exception, info: ChatLoopInfo) -> ExceptionAction:
        if self.max_retry:
            self.max_retry -= 1

            print(f"\nRetrying. Retries left: {self.max_retry}")
            print(f"Exception raised: {type(e).__name__}: {str(e)}")

            response = info.context['response']
            messages = info.context['messages']
            messages.append(
                    {
                        "role": "tool",
                        "tool_call_id": response.choices[0].message.tool_calls[0].id,
                        "name": response.choices[0].message.tool_calls[0].function.name,
                        "content": f"Exceptions raised: \n{e}",
                    }
                )

            return Continue(functions=info.context['tools'])
        raise e

input = """                Name       Phone Number
0  Dr. Danielle King      (844)055-3780
1        John Miller  +1-268-920-5475x5
2    Michael Johnson  +1-758-232-6153x8
"""

messages = [
    {"role": "user", "content": input}
]

response = ingest_user_info.invoke(
    llm,
    messages=messages,
    model=MODEL,
    stream=False,
    temperature=1,
    exception_handler = ExceptionRetryHandler(3)
)

Retrying. Retries left: 2
Exception raised: ValidationError: 6 validation errors for ingest_user_info
users.0.name
  Value error, Name must contain a first name and a last name separated by a space [type=value_error, input_value='Dr. Danielle King', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.0.phone_number
  Value error, phone number must be in the format +1 (XXX) XXX-XXXX [type=value_error, input_value='(844)055-3780', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.1.name
  Value error, Name must be in uppercase [type=value_error, input_value='John Miller', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.1.phone_number
  Value error, phone number must be in the format +1 (XXX) XXX-XXXX [type=value_error, input_value='+1-268-920-5475x5', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.2.name
  Value error, Name must be in uppercase [type=value_error, input_value='Michael Johnson', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.2.phone_number
  Value error, phone number must be in the format +1 (XXX) XXX-XXXX [type=value_error, input_value='+1-758-232-6153x8', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error

user_db

[[UserModel(name='DANIELLE KING', phone_number='+1 (844) 055-3780'),
  UserModel(name='JOHN MILLER', phone_number='+1 (268) 920-5475'),
  UserModel(name='MICHAEL JOHNSON', phone_number='+1 (758) 232-6153')]]