evofabric.app.rethinker

Contents

evofabric.app.rethinker#

evofabric.app.rethinker.list_ele_overwrite(old: list = MISSING, new: list = MISSING)#

Perform element-wise overwrite merge on a fixed-length list, adhering to the maximum parallel scale limit.

This function merges old and new into a result list, whose length is determined by config.structure.num_parallel. The merge rule is element-wise overwrite:

  • If old is provided, first copy its elements to the result.

  • If new is provided, when the corresponding element in new is not None, overwrite the element at the same position in the result.

Parameters:

  • old (list) – Old State list; if MISSING, ignore.

  • new (list) – New State list; if MISSING, ignore.

Returns:

Element-wise overwritten list, length of config.structure.num_parallel.

Return type:

list

evofabric.app.rethinker.get_agent(client: ChatClientBase, finish_type: Literal['answer', 'select'])#

Build an Agent for encoding/inference based on configuration.

Parameters:

  • client (ChatClientBase) – Chat client base class instance, used to interact with the model.

  • finish_type (Literal['answer', 'select']) – End type, used to specify the Agent’s completion method (e.g., 'answer' or 'select').

Returns:

Built Agent object.

Return type:

Any

evofabric.app.rethinker.build_rethinker_graph()#

Build and configure a computation graph (Graph) supporting dynamic parallel structures.

This graph splits a single query into multiple parallel branches for generation, summarization, criticism, and filtering to obtain the best result. The overall structure includes:

  • Entry node dispatch: Distributes the initial State to all parallel branches.

  • Parallel branches (number equals config.structure.num_parallel): Each branch internally forms a serial pipeline:

    • Solution phase: Cascaded SolutionWithReThinkNode for generating/improving candidate solutions.

    • Summary phase: A GuidedSummaryNode for guided summarization of Solution outputs.

    • Critic phase: Cascaded CriticWithRethinkNode for evaluating and critiquing the summary results with feedback.

  • Aggregation node selector (ConfidenceGuideSelectNode): Receives outputs from the last critic node in each branch and selects the best final result.

Returns:

The constructed computation graph object.

Return type:

Any

async evofabric.app.rethinker.run_rethinker_graph(query: str, query_id: str | None = None, semaphore: asyncio.Semaphore | None = None)#

Asynchronously run the rethinker execution graph to process a single query.

This function builds the rethinker computation graph, initializes log/output directories for each query, injects query metadata into the stream context, and executes the entire graph asynchronously.

When config.exp.output_root is set, output files are organized as follows:

  • output_root/
    • qid00001/

      • node1.json

      • node2.json

      • …

      • result.json

    • qid00002/

    • …

Parameters:

  • query (str) – Input query to be processed by rethinker graph.

  • query_id (Optional[str]) – Unique identifier for the query; automatically generated as UUID if not provided.

  • semaphore (Optional[asyncio.Semaphore]) – Semaphore for synchronization/limiting concurrent execution.

Returns:

Final result returned by rethinker execution graph.

Return type:

Any

class evofabric.app.rethinker.LLMConfig#

Configuration model related to LLM calls, used to describe model name, authentication information, request parameters, and inference parameters.

model_name#

LLM model name used.

Type:

str

api_key#

API Key typically required by OpenAI-compatible backends for authentication.

Type:

str

base_url#

Base URL for LLM API.

Type:

str

csb_token#

CSB authentication token, used for authorization via CSB Token.

Type:

Optional[str]

max_retries#

Maximum retry attempts for transient failure requests.

Type:

int

timeout#

Request timeout in seconds. If None, use backend default.

Type:

Optional[int]

top_p#

Nucleus Sampling parameter to control cumulative probability mass.

Type:

Optional[float]

temperature#

Sampling temperature to control randomness of generation results.

Type:

Optional[float]

max_tokens#

Maximum generated tokens; if None, use backend default.

Type:

Optional[int]

extra_body#

Additional backend-specific request body parameters.

Type:

dict

fast_think#

Whether to enable fast thinking mode: triggered by appending /no_think to the last message.

Type:

bool

output_logp#

Whether to return token-level log probabilities in the response.

Type:

bool

stop_condition#

Stop condition string for streaming output; generation terminates when this string is matched.

Type:

Optional[str]

http_client_kwargs#

Keyword arguments passed to the underlying OpenAI HTTP Client.

Type:

Optional[dict]

stream#

Whether to enable streaming output; must be True when stop_condition is set.

Type:

bool

create_client_kwargs(self) → dict#

Generate the parameter dictionary for initializing the underlying Client.

This method includes basic parameters such as api_key, base_url, max_retries, and timeout; when csb_token is set, it appends the csb-token header to default_headers.

Returns:

Client initialization parameter dictionary.

Return type:

dict

create_inference_kwargs(self) → dict#

Generate the parameter dictionary for inference calls (Inference).

This method includes sampling parameters such as temperature and top_p; when extra_body is non-empty, it injects extra_body; when output_logp is True, it enables logprobs and sets top_logprobs; when max_tokens is not None, it sets max_tokens.

Returns:

Inference call parameter dictionary.

Return type:

dict

class evofabric.app.rethinker.WebParserConfig#

Webpage parsing configuration, used to control scraping, parsing, and content truncation strategies.

llm_input_max_char#

After webpage parsing, the maximum character budget allowed for inputting a single document into the model.

Type:

int

model#

Name of the backend model used for webpage content parsing or downstream inference.

Type:

str

use_jina#

Whether to enable Jina-based webpage content parsing.

Type:

bool

jina_api_key#

Jina service API Key; typically required when use_jina is enabled.

Type:

Optional[str]

ssl_verify#

Whether to verify SSL certificates when making HTTP requests.

Type:

bool

timeout#

Network request timeout (seconds) for fetching web page content.

Type:

int

show_url_content_max_char#

Maximum number of characters allowed per URL; negative values exclude URL content.

Type:

int

class evofabric.app.rethinker.WebSearchConfig#

Web search configuration for controlling authentication and request behavior of the retrieval service.

serper_api_key#

Serper service API Key; used to enable web search requests.

Type:

Optional[str]

retries#

Maximum number of retries when a web search request fails.

Type:

int

ssl_verify#

Whether to verify SSL certificates during web search HTTP requests.

Type:

bool

timeout#

Request timeout (seconds) for a single web search call.

Type:

int

class evofabric.app.rethinker.SolutionConfig#

Problem-solving/reasoning phase configuration, including model and rule settings for stages such as solution generation, reflection, summarization, and selection.

solver_model#

LLM backend model used in the solution generation phase.

Type:

str

critic_model#

LLM backend model used in the critic evaluation phase.

Type:

str

summary_model#

LLM backend model used in the guided summaries phase.

Type:

str

selector_model#

LLM backend model used in the selection phase.

Type:

str

selector_iteration#

Number of iterations in the selection phase, used for gradual convergence and optimization of selection results.

Type:

int

stop_condition#

Regular expression pattern used to extract code blocks from LLM outputs.

Type:

str

selection_condition#

Regular expression pattern used to extract the selected response index from LLM outputs.

Type:

str

answer_condition#

Regular expression pattern for extracting final answer content from LLM output.

Type:

str

max_agent_step#

Maximum number of reasoning/action steps allowed per agent execution.

Type:

int

tool_timeout#

Timeout duration (in seconds) for calling a single tool or external service.

Type:

int

max_empty_response#

Maximum allowed consecutive empty responses; agent execution will terminate upon exceeding this limit.

Type:

int

chat_template#

Multi-turn dialogue serialization template string; used solely for serialization, not as an LLM prompt.

Type:

Optional[str]

class evofabric.app.rethinker.ExpConfig#

Experimental configuration, including dataset input, output directory, and concurrency limits.

input_file_path#

File path of the input dataset used for the experiment.

Type:

str

exp_name#

Experiment name, typically used as a subdirectory name under the output root directory.

Type:

str

output_root#

Output root directory; if None, no cache or output will be written.

Type:

Optional[str]

max_question_thread_limit#

Maximum number of concurrent problems allowed for execution.

Type:

int

max_node_thread_limit#

Maximum number of concurrent nodes allowed during execution.

Type:

int

class evofabric.app.rethinker.StructureConfig#

Reasoning structure configuration, used to control parallel candidate count and iteration counts per phase.

num_parallel#

Number of solution candidates generated in parallel per task.

Type:

int

num_solution_iteration#

Iteration count for generating each candidate solution.

Type:

int

num_critic_iteration#

Iteration count for the Guided reflection phase.

Type:

int

class evofabric.app.rethinker.PromptConfig#

Prompt template configuration, centrally managing prompts for parsing, solving, reflection, summarization, and selection stages.

web_parser_pdf#

Prompt template for parsing PDF webpage content.

Type:

str

web_parser_html#

Prompt template for parsing HTML webpage content.

Type:

str

solver_user_prompt#

User prompt template for the initial problem-solving stage.

Type:

str

solver_twice_user_prompt#

User prompt template for secondary problem-solving (second-pass solution generation).

Type:

str

critic_user_prompt#

User prompt template for the initial reflection stage.

Type:

str

critic_twice_user_prompt#

User prompt template for secondary reflection (second-pass critique/evaluation).

Type:

str

guided_summary_prompt#

Prompt template for guided summaries.

Type:

str

selector_user_prompt#

User prompt template for the selection stage.

Type:

str

selector_iteration_user_prompt#

Iterative selection refinement user prompt template.

Type:

str

class evofabric.app.rethinker.GraphConfig#

Execution Graph (Graph) overall configuration, aggregating LLM resources, web parsing/search, problem-solving stages, structural parameters, experimental parameters, and prompt templates.

llm_resources#

Dictionary of available LLM backend resources; keys are model names, values are corresponding LLMConfig configuration objects.

Type:

dict[str, LLMConfig]

web_parser#

Web content parsing configuration.

Type:

WebParserConfig

Web search configuration.

Type:

WebSearchConfig

solution#

Configuration set for the solution, reflection, summary, and selection phases.

Type:

SolutionConfig

structure#

Iterative reasoning graph structure configuration, including parallelism and selector behavior.

Type:

StructureConfig

exp#

Experimental-level configuration, including I/O paths, output directory, and thread limits.

Type:

ExpConfig

prompts#

Collection of prompt templates used throughout the experiment (parsing, solving, reflection, selection, etc.).

Type:

PromptConfig

class evofabric.app.rethinker.Config#

Experiment configuration manager.

This class is responsible for loading and accessing the complete experiment configuration: it supports loading from YAML files as well as directly loading a GraphConfig instance. It provides convenient access to sub-configurations (LLM resources, web parsing/search, solution settings, structure, experiment I/O, prompts, etc.) and maintains a global semaphore to control node-level concurrency during execution.

property graph#

Obtain the complete GraphConfig configuration object (configuration must be loaded first).

Returns:

Currently loaded graph configuration.

Return type:

GraphConfig

property llm_resources#

Obtain LLM backend resource configuration (configuration must be loaded first).

Returns:

LLM Resource Dictionary.

Return type:

dict[str, LLMConfig]

property web_parser#

Get Web Parsing Configuration (Configuration must be loaded first).

Returns:

Web Parsing Configuration Object.

Return type:

WebParserConfig

Get Web Search Configuration (Configuration must be loaded first).

Returns:

Web Search Configuration Object.

Return type:

WebSearchConfig

property solution#

Get Problem-Solving Phase Configuration (Configuration must be loaded first).

Returns:

Problem-Solving Phase Configuration Object.

Return type:

SolutionConfig

property structure#

Get Structure Configuration (Configuration must be loaded first).

Returns:

Structure Configuration Object.

Return type:

StructureConfig

property exp#

Get Experimental-Level Configuration (Configuration must be loaded first).

Returns:

Experimental-Level Configuration Object.

Return type:

ExpConfig

property prompts#

Get Prompt Template Configuration (Configuration must be loaded first).

Returns:

Prompt Template Configuration Object.

Return type:

PromptConfig

load(self, config_path: str | Path) → None#

Load Experimental Configuration from YAML File.

Parameters:

config_path (Union[str, Path]) – YAML Configuration File Path.

Raises:

FileNotFoundError – Raised when the configuration file does not exist.

loads(self, config: GraphConfig) → None#

Load experiment configuration from an existing GraphConfig instance.

Parameters:

config (GraphConfig) – Pre-constructed GraphConfig instance.

get_semaphore(self) → asyncio.Semaphore#

Obtain the global semaphore for node-level concurrency control.

If not initialized, creates and caches an asyncio.Semaphore based on exp.max_node_thread_limit.

Returns:

asyncio.Semaphore object limiting concurrent node execution.

Return type:

asyncio.Semaphore

evofabric.app.rethinker.config#

Global singleton configuration manager instance for loading and accessing experiment configurations.

Type:

Config

evofabric.app.rethinker.repeat_prompt(prompt: str, repeat_time: int = 3) → str#

Repeat the given prompt text a specified number of times, inserting a fixed English transition prompt between repetitions.

Parameters:

  • prompt (str) – Prompt text to be repeated.

  • repeat_time (int, optional) – Repeat count, must be greater than or equal to 1, default is 3.

Raises:

ValueError – Raise an exception when repeat_time < 1.

Returns:

Complete prompt string after concatenation, with paragraphs separated by two line breaks.

Return type:

str

evofabric.app.rethinker.generate_stop_condition(pattern: str)#

Generate a “streaming stop condition” function based on the given regular expression pattern.

The generated stop condition function uses re.finditer (with re.DOTALL) to match streaming content; once at least one match is detected, it returns True to indicate that generation/parsing should stop.

Parameters:

pattern (str) – The regular expression pattern used to match streaming content.

Returns:

A stop condition function. The function signature is (content: str) -> bool, returning True when a match is detected.

Return type:

callable

class evofabric.app.rethinker.FastSlowThinkOpenAIChatClient#

An OpenAI Chat Client extension implementation supporting “fast thinking” mode.

When fast_think is enabled, /no_think is appended to the content of the last input message to trigger the backend’s fast mode (skipping or reducing reasoning output).

fast_think#

Whether to enable fast thinking mode.

Type:

bool

evofabric.app.rethinker.get_client(model: str)#

Retrieves an OpenAI Chat Client instance corresponding to a specified model name.

This function looks up the corresponding LLMConfig from the global configuration’s llm_resources, then constructs initialization parameters for FastSlowThinkOpenAIChatClient (model name, streaming flag, client parameters, reasoning parameters, etc.). If the configuration contains stop_condition, it additionally injects stream_parser to allow early termination during streaming parsing based on the stop condition.

Parameters:

model (str) – Model identifier name (used to retrieve from the LLM resources dictionary in the configuration).

Raises:

ValueError – Raises an exception when model is not found in the configured LLM resources.

Returns:

The Chat Client instance corresponding to the model.

Return type:

FastSlowThinkOpenAIChatClient

class evofabric.app.rethinker.CodingAgentResult#

The execution result data model for encoding/reasoning agents.

This result object summarizes context information, step count, duration, full conversation logs, raw client responses per round, and (optionally) the final rendered output text from a single agent execution.

ctx#

The execution context captured during the current graph node execution.

Type:

StreamCtx

total_steps#

Total steps of reasoning/actions completed by the agent before termination.

Type:

int

generation_time#

End-to-end execution time of the agent (seconds), including LLM and tool calls.

Type:

float

agent_logs#

Full conversation trajectory log maintained by the agent.

Type:

list

client_responses#

List of raw responses returned by the LLM client for each iteration.

Type:

list

response#

Final output text rendered from the agent log using the configured chat template.

If no chat template is provided, this field is None.

Type:

Optional[str]

class evofabric.app.rethinker.CodingAgent#

A LLM-driven autonomous coding agent supporting multi-step iterative reasoning and native Python code execution.

This agent maintains the full conversation trajectory and repeatedly executes the following process until termination conditions are met:

  1. Invoke LLM backend to generate response;

  2. Parse structured signals (e.g., code blocks or final answers) from model output;

  3. Execute generated Python code via the user-provided executor;

  4. Inject execution results back into the conversation context.

Main capabilities include:

  • Configurable multi-step iterative reasoning (maximum steps configurable)

  • Structured extraction and execution of “tool content (Python code)”

  • Automatic fallback prompt usage to conclude when model output is empty or invalid

  • Final answer detection based on regular expression patterns

  • Detailed step-by-step logging and execution tracking

This agent is designed to be used as an asynchronous callable node within a larger execution graph/workflow system.

client#

LLM conversation client used to initiate generation requests at each agent step.

Type:

ChatClientBase

max_agent_step#

Maximum number of reasoning/action steps allowed for the agent.

Type:

int

max_empty_response#

Maximum allowed consecutive empty responses.

“Empty response” refers to output that matches neither the code pattern nor the answer pattern.

Type:

int

tool_timeout#

Timeout for tool (Python code) execution (seconds).

Type:

int

force_finish_prompt_candidates#

List of fallback prompt candidates used when the agent produces an empty response to prompt the model to provide a final answer and conclude the conversation.

Type:

list

tool_content_pattern#

Regular expression pattern used to extract tool content (code) from agent responses.

Type:

str

answer_pattern#

Regular expression pattern used to extract final answer content from agent responses.

Type:

str

chat_template#

Chat template for formatting agent interaction history into a single string.

If None, the final rendered response field will not be output.

Type:

str

py_exec_handler#

Processing function for executing generated Python code, typically taking (code: str, timeout: int) as input and asynchronously returning an execution result object.

Type:

Callable[[str, int], Awaitable[BaseModel]]

async __call__(self, prompt: str) → CodingAgentResult#

Run the coding agent with a single input prompt.

This method writes the prompt as the first user message in the internal conversation history, then begins subsequent reasoning, tool extraction and execution, result injection, until the termination condition is met.

Parameters:

prompt (str) – Initial user prompt provided to the agent, serving as the starting point for the entire conversation and reasoning process.

Returns:

Structured agent execution result object containing execution context, step count, duration, full logs, raw responses per round, and optionally the final rendered output.

Return type:

CodingAgentResult

class evofabric.app.rethinker.AsyncNodeWithCacheAndConcurrencyLimit#

Asynchronous node base class with built-in caching and concurrency control capabilities.

This class wraps two general capabilities around node execution:

  1. Caching mechanism: Avoids redundant computation by reusing previously persisted results.

  2. Concurrency limit: Controls the maximum number of concurrently executing nodes using a global semaphore.

Subclasses should implement the _run method to define actual node logic; this logic itself does not need to concern itself with caching and concurrency control.

agent#

Coding agent instance, optionally used to complete tasks through multi-round iterative execution of Python code.

Type:

Optional[CodingAgent]

cache_dir_key#

The field name (key) used in state to retrieve the cache directory path.

If the cache directory corresponding to this field can be obtained from state, disk cache will be enabled; default value is "cache_dir".

Type:

Optional[str]

async __call__(self, state: State) → StateDelta#

Execute the node under cache and concurrency control protection.

Execution flow as follows:

  1. Acquire the global semaphore and enter the asynchronous context to enforce concurrency limits.

  2. If cache_dir_key exists and the cache directory can be retrieved from state: - Construct the cache file path based on the current node name (<node_name>.json); - If the cache file exists, load and return the cached result directly; - Otherwise, execute _run, write the result to the cache file, and return.

  3. If cache cannot be enabled (not configured or cache directory cannot be retrieved from state), directly execute _run and return the result.

Parameters:

state (State) – The current execution state object.

Returns:

The state delta (or equivalent change result) produced by this node.

Return type:

StateDelta

async _run(self, state: State) → StateDelta#

Core logic for executing the node (implemented by subclasses).

This method contains only the actual computation or model invocation logic, without handling cache or concurrency control. The base class implementation will directly raise NotImplementedError to indicate that it must be overridden by subclasses.

Parameters:

state (State) – The current execution state object.

Raises:

NotImplementedError – Exception raised when the subclass does not implement this method.

Returns:

The state delta (or equivalent change result) produced by this node.

Return type:

StateDelta

class evofabric.app.rethinker.SolutionWithReThinkNode#

A problem-solving node implementation supporting “revisit/rethink (ReThink)”, inheriting from AsyncNodeWithCacheAndConcurrencyLimit.

This node determines the prompt generation method based on whether last_round is provided:

  • When last_round is empty, indicating the first round of problem-solving, the prompt is constructed using config.prompts.solver_user_prompt.

  • When last_round is non-empty, extract the response content at the specified index from the previous round’s results, parse it, and inject it into config.prompts.solver_twice_user_prompt for secondary solving/rethinking.

The final result will be written to the field corresponding to output_key in the form of a parallel slot list after invoking the agent to execute the prompt.

last_round#

Indicates the key for retrieving the previous round’s answer from which round’s results.

  • If None: indicates the first round, no rethinking is performed;

  • If not None: indicates retrieving the previous round’s answer from state[last_round] and performing rethinking.

Type:

Optional[str]

output_key#

The storage key for the output result in the state increment, default value: "solution".

Type:

str

async _run(self, state: State) → StateDelta#

Executes the problem-solving (or rethinking) logic and returns the state increment.

Execution points:

  • Read query and index from state;

  • If it is the first round (last_round is empty), construct the initial problem-solving prompt;

  • If it is a rethinking round (last_round is non-empty), extract the response at the current index from the previous round’s results, parse it, and inject it into the secondary problem-solving prompt template;

  • Invoke the agent to asynchronously generate the result;

  • Construct a list of length config.structure.num_parallel, placing the current result only at position index, and return it in the form of {output_key: solution}.

Parameters:

state (State) – The current execution state object.

Returns:

Contains the state increment output by this node, with key output_key and value as a parallel slot list.

Return type:

StateDelta

class evofabric.app.rethinker.CriticWithRethinkNode#

Implements a reflection (Critic) node supporting “review/rethinking (ReThink)”, inheriting from AsyncNodeWithCacheAndConcurrencyLimit.

This node is used for critical reflection/reflection on the solution results of a round, and optionally for secondary reflection/rethinking based on the results of the previous round:

  • Extract the content to be reflected from the State field specified by input_key;

  • After cleaning the content (removing thinking/execution fragments, etc.), inject the reflection prompt template;

  • If last_round is empty, use the initial reflection prompt config.prompts.critic_user_prompt; otherwise, read the previous round’s reflection result, trim it, and inject the secondary reflection prompt config.prompts.critic_twice_user_prompt;

  • Call agent to obtain the reflection output and write it as a parallel slot list to output_key.

input_key#

The key for reading the content that needs to be reflected/criticized from the State.

Type:

Optional[str]

last_round#

The key where the previous round’s reflection result is stored.

  • If None: indicates the first round of reflection, no rethinking is performed;

  • If not None: indicates reading the reflection result of the specified round and performing secondary reflection/rethinking.

Type:

Optional[str]

output_key#

The key for storing the reflection (or secondary reflection/rethinking) result in the State increment.

Type:

Optional[str]

async _run(self, state: State) → StateDelta#

Execute the reflection (or secondary reflection/rethinking) logic and return the State increment.

Execution points:

  • Read query and index from state;

  • Read the solution output to be reflected from state[input_key][index]["response"] and clean it;

  • If last_round is None, construct the initial reflection prompt;

  • Otherwise, read state[last_round][index]["response"] as the previous round’s reflection result, parse and trim it, then inject the secondary reflection prompt;

  • Call agent to asynchronously generate the reflection result.

  • Construct a list of length config.structure.num_parallel, placing only the current reflection result in the index slot, and return in the form of {output_key: critic}.

Parameters:

state (State) – The current execution state object.

Returns:

Contains the State increment of the current node’s reflection output, with key output_key and value as a list of parallel slots.

Return type:

StateDelta

class evofabric.app.rethinker.ConfidenceGuideSelectNode#

Implements a confidence-guided selection node, inheriting from AsyncNodeWithCacheAndConcurrencyLimit.

This node is used for selecting among multiple candidate responses and maintaining the full selection history. The overall process typically includes:

  • Read the candidate result list from input_key and parse the answer for each candidate;

  • Construct a Latin square based on the number of candidates to organize the comparison/selection order;

  • Execute the initialization selection process (_init_select) and iterative selection process (_iterative_select);

  • Aggregate historical selection results to extract the candidate set with the highest votes/rank;

  • If multiple candidates are tied, enter the final selection process (_final_select) to determine a single result;

  • Write the selection history, final selected result, and candidate list to output_key.

input_key#

Key for reading the candidate solutions (or intermediate results) list from State, used for subsequent selection.

Type:

Optional[str]

output_key#

Storage key for selection results in State increment, default value "selector".

Type:

str

repeat_prompt#

Prompt repetition count configuration (used to control the repetition/emphasis level of prompts during the selection phase).

Type:

int

async _run(self, state: State) → StateDelta#

Execute the candidate result selection process and return the State increment.

Execution points:

  • Read query from state;

  • Read the candidate list from state[input_key], and parse each candidate’s response into pure answer text;

  • Construct a Latin square based on the number of candidates to organize the comparison/selection order;

  • Execute initial selection and iterative selection sequentially, accumulating _response_history;

  • Calculate the top-ranked candidate set from the current selection history: if the candidate set size is greater than 1, perform final selection to obtain a single result;

  • Return a dictionary with key output_key, whose value contains: selection_history (selection history), selected_response (final selected response), and response_list (candidate answer list).

Parameters:

state (State) – The current execution state object.

Returns:

Include the selection result as a State increment with key output_key.

Return type:

StateDelta

class evofabric.app.rethinker.GuidedSummaryNode#

Implementation of the Guided summary node, inheriting from AsyncNodeWithCacheAndConcurrencyLimit.

This node is used to generate “Guided summary” for candidate solutions:

  • Read the solution output to be summarized from the State field specified by input_key;

  • Clean the output content (remove unnecessary response structures/markers, etc.);

  • Construct the summary prompt using config.prompts.guided_summary_prompt;

  • Initiate a conversation generation request via client to obtain the summary content;

  • Write the summary result as a parallel slot list to output_key.

client#

LLM dialogue client used for generating summary content.

Type:

ChatClientBase

input_key#

Key for reading content to be summarized from State.

Type:

str

output_key#

The storage key for the generated summary result in the State increment.

Type:

str

async _run(self, state: State) → StateDelta#

Generate Guided reflection summary and return State increment.

Execution points:

  • Read query and index from state;

  • Read the content to be summarized from state[input_key][index]["response"] and clean it;

  • Inject the question and answer into the guided_summary_prompt template to construct the prompt;

  • Call client.create to generate the summary, and organize the returned content into a dictionary structure: where response["response"] will be set to response["content"] to unify field naming;

  • Construct a list of length config.structure.num_parallel, writing only the summary result to the index slot;

  • Return in the form of {output_key: solution_summary}.

Parameters:

state (State) – The current execution state object.

Returns:

The State increment containing the summary result, with key output_key and value as a list of parallel slots.

Return type:

StateDelta

class evofabric.app.rethinker.DispatchNode#

Synchronous distribution node, used as a placeholder or distribution entry node in the execution graph.

The current implementation of __call__ does not modify the State, directly returning an empty State increment.

__call__(self, state: State) → StateDelta#

Execute distribution node logic and return State increment.

Parameters:

state (State) – The current execution state object.

Returns:

Empty State increment dictionary.

Return type:

StateDelta

evofabric.app.rethinker.get_dispatch_filter(index: int)#

Generate a distribution filter function for setting state.index.

The returned filter function sets the index field of the incoming state to the specified value and returns the updated state, commonly used to distribute parallel tasks to different slot indices.

Parameters:

index (int) – The index value to be written to state.index.

Returns:

A filter function with signature (state: State) -> State.

Return type:

callable

class evofabric.app.rethinker.BaseBenchmarkEvaluator#

Abstract base class for benchmark evaluation.

This class provides a general evaluation pipeline covering data and result matching, parallel evaluation, exception handling, and statistical aggregation and storage. Subclasses must implement the evaluate_item method to define the specific benchmark evaluation logic.

data_file#

Source dataset file path (JSON).

Type:

Path

result_root#

Root directory where model generation results are stored.

Type:

Path

eval_llm#

LLM configuration for the judge (judge model).

Type:

LLMConfig

max_workers#

Maximum number of threads for parallel evaluation.

Type:

int

max_char#

Maximum number of characters to retain in the response text injected into the judge prompt (for truncation).

Type:

int

max_completion_tokens#

Maximum number of tokens allowed in the judge model’s output.

Type:

int

save_path#

Path to save evaluation results.

Type:

Path

result_extractor#

Optional result extraction function, used to extract model predictions from a single question directory.

If None, default extraction logic is used; this field is excluded from serialization (functions are not serializable).

Type:

Optional[Callable[[str], Optional[str]]]

model_post_init(self, __context: Any) → None#

Hook method after model initialization, used to complete runtime initialization and validation.

This method will:

  • Create and cache the OpenAI client for judging;

  • Validate whether the data file exists (throw an exception if it does not exist);

  • Check if the result root directory exists (log a warning if it does not exist);

  • If result_extractor is not provided, set it to the default extraction logic.

Parameters:

__context (Any) – Context object passed via Pydantic.

Raises:

FileNotFoundError – Throw an exception when data_file does not exist.

evaluate_item(self, data: Dict) → Dict#

Evaluate a single sample (to be implemented by subclasses).

Subclasses must override this method to implement specific benchmark data scoring rules and metadata organization methods.

Parameters:

data (Dict) – A single data sample, typically containing fields such as question, standard answer, and model prediction.

Returns:

Evaluation result dictionary for a single sample, typically containing fields such as score and details.

Return type:

Dict

run(self)#

Run the complete benchmark evaluation process.

This method first matches source data with generated results, then uses a thread pool to evaluate all samples in parallel, and finally aggregates the results statistically and writes them to save_path.

Returns:

No return value.

Return type:

None

class evofabric.app.rethinker.HLEEvaluator#

HLE Benchmark Evaluator, implements single-sample evaluation logic for HLE scenarios.

evaluate_item(self, data: Dict) → Dict#

Evaluates a single HLE sample.

This implementation constructs a judge prompt and uses a judge model for structured parsing, calculating the score based on the correct field in the parsing result.

Parameters:

data (Dict) – A single data sample containing fields such as question, answer, and prediction.

Returns:

Evaluation result dictionary containing score, error, and details.

Return type:

Dict

Raises:

ValueError – Throws an exception when the structured parsing result from the judge model is empty.

class evofabric.app.rethinker.XBenchEvaluator#

XBench Benchmark Evaluator, implements single-sample evaluation logic for XBench scenarios.

evaluate_item(self, data: Dict) → Dict#

Evaluates a single XBench sample.

This implementation calls the judge model to generate judgment text, extracts the “Conclusion (Correct/Incorrect)” field via regular expression, and then calculates the score.

Parameters:

data (Dict) – A single data sample containing fields such as question, answer, and prediction.

Returns:

Evaluation result dictionary containing score, error, and details.

Return type:

Dict

Raises:

ValueError – Throws an exception when the conclusion cannot be parsed from the judge output.

class evofabric.app.rethinker.GaiaEvaluator#

GAIA Benchmark Evaluator, implements single-sample evaluation logic for GAIA scenarios.

evaluate_item(self, data: Dict) → Dict#

Evaluates a single GAIA sample.

This implementation constructs a judge prompt, calls the judge model to generate judgment results, normalizes the output content, and uses correct / non-correct as the scoring criterion.

Parameters:

data (Dict) – A single data sample containing fields such as question, answer, and prediction.

Returns:

Evaluation result dictionary containing score, error, and details.

Return type:

Dict

async evofabric.app.rethinker.execute_python_code(code: str, timeout: int = 300) → CodeResponse#

Asynchronously executes a Python code snippet and returns structured execution results.

This function delegates the actual code execution to a thread pool executor (via the event loop’s run_in_executor), tracks execution time; if an exception occurs during execution, it logs the error and returns the error information in the result.

Parameters:

  • code (str) – Python code string to be executed.

  • timeout (int, optional) – Execution timeout in seconds, default 300.

Returns:

Result object of code execution, containing standard output, error messages, and execution duration.

Return type:

CodeResponse

async evofabric.app.rethinker.web_parse(link: str, query: str) → Dict[str, Any]#

Unified entry point for web content parsing.

This function dynamically selects an appropriate parser based on the input URL, executes the corresponding parsing logic, and returns the structured parsing result. If an unhandled exception occurs during parsing, it logs the error and returns a standardized error result dictionary.

Parameters:

  • link (str) – URL of the web page to be parsed.

  • query (str) – Retrieval/question text in the parsing context, used to extract information related to this context from the page.

Returns:

Parsing result dictionary, typically containing fields such as parsed content (content), related URL list (urls), and score; if an unhandled exception occurs, returns an error dictionary in the form {"content": "System error during parsing"}.

Return type:

Dict[str, Any]

Search the internet using a search engine and return results.

This function constructs the required payload and headers, calls the Serper API for searching, and uses a retry mechanism to improve stability. On success, it typically returns a result dictionary containing the organic field; if retries ultimately return None, it returns an empty list.

Parameters:

  • query (str) – Search query string to be submitted.

  • top_k (int, optional) – Maximum number of returned results, default 10.

Returns:

Returns a dictionary containing search results on success (typically results under the organic key); if the request fails and retries yield no results, returns an empty list.

Return type:

Union[Dict[str, Any], List[Any]]

evofabric.app.rethinker.download_and_read_pdf(url: str) → str#

Synchronously download a PDF from a specified URL and extract its text content.

This function supports both direct PDF links and arXiv abstract page links (which are converted to direct PDF download links). After downloading, it uses PyMuPDF (fitz) to parse the PDF and extract all text.

Parameters:

url (str) – The URL of a PDF file, or the arXiv abstract page URL (e.g., https://arxiv.org/abs/2106.07682).

Returns:

On success, return the extracted PDF text content; on failure, return a descriptive error string (e.g., ‘Failed to read the PDF’).

Return type:

str

Contents