ChatClient

ChatClient#

ChatClient is a client module for dialogue interaction with large language models, supporting both streaming and non-streaming response retrieval, and suitable for building intelligent dialogue systems, Agent nodes, and other scenarios.

Overview#

The ChatClient module provides a unified interface for calling different backend large language models (LLMs) to generate conversations. It implements a unified calling method for models such as OpenAI and PanGu, and supports features including configuring model parameters, streaming parsing, and HTTP client settings.

Characteristics#

  • Unified Interface: All clients inherit from ChatClientBase, providing a consistent calling method.

  • Streaming Support: Supports streaming responses, suitable for real-time interaction scenarios.

  • Flexible Configuration: Supports flexible configuration of model parameters, HTTP client parameters, and inference parameters.

  • Scalability: Easy to expand new model clients.

  • Asynchronous Support: Implemented using asynchronous interfaces, suitable for high-concurrency scenarios.

Basic Usage#

from evofabric.core.clients import OpenAIChatClient
from evofabric.core.typing import ChatStreamChunk, LLMChatResponse

# init client
client = OpenAIChatClient(
    model="gpt-3.5-turbo",
    client_kwargs={"api_key": "your-api-key"},
    inference_kwargs={"temperature": 0.7}
)

# non-stream create
response = await client.create(messages=[{"role": "user", "content": "hello"}])
print(response.content)

# streaming create
async for chunk in client.create_on_stream(messages=[{"role": "user", "content": "hello"}]):
    if isinstance(chunk, ChatStreamChunk):
        print(f"delta: {chunk.content}")
    elif isinstance(chunk, LLMChatResponse):
        print(f"final reply: {chunk.content}")

Use in Agent#

from evofabric.core.clients import OpenAIChatClient
from evofabric.core.agent import AgentNode

client = OpenAIChatClient(
    model="gpt-3.5-turbo",
    client_kwargs={"api_key": "your-api-key"},
    inference_kwargs={"temperature": 0.7}
)

agent = AgentNode(
    client=client
)

Best Practices#

1. Streaming Response Processing

Applies to scenarios such as real-time display of streaming messages:

async def stream_response(client, messages):
    full_content = ""
    async for chunk in client.create_on_stream(messages=messages):
        if hasattr(chunk, 'delta'):
            full_content += chunk.delta
            print(chunk.delta, end="", flush=True)
    return full_content

2. Parameter Override Mechanism

The kwargs passed during the call will override the inference_kwargs set during initialization:

client = OpenAIChatClient(
    model="gpt-4",
    inference_kwargs={"temperature": 0.7}
)

# Temporarily increase temperature
 response = await client.create(
     messages=[{"role": "user", "content": "Write a poem"}],
     temperature=0.9
 )

3. Error Handling and Retries

It is recommended to add exception handling logic at the calling layer:

import asyncio

async def robust_call(client, messages, retries=3):
    for attempt in range(retries):
        try:
            return await client.create(messages=messages)
        except Exception as e:
            if attempt == retries - 1:
                raise
            await asyncio.sleep(2 ** attempt)
Contents