ERC3 Leaderboards

Model(s): claude-opus-4.5 LLM Calls: 685 Prompt Tokens: 1.17M Completion Tokens: 149.48k Architecture: Anthropic SDK Agent PARALLEL (5w) with claude-opus-4-5-20251101 # ERC3 Agent Architecture ## The Basics Fairly simple architecture: the main agent is built on Anthropic Python SDK with native Tool Use. Model - Opus 4.5. All 20+ tools are described in a single file using Anthropic's JSON Schema format. Tool execution dynamically constructs HTTP requests to the benchmark API — no code generation, just endpoint mapping. The system prompt distills all key rules from the company wiki into a compact decision algorithm: check identity → verify permissions → gather data → respond with proper outcome. ## The Interesting Part: Self-Evolving Agent The real cool thing was in automated prompt evolution using a three-agent pipeline: 1. Main Agent — runs the benchmark, solves all tasks, logs everything 2. Analyzer Agent — reviews logs of failed tasks, formulates hypotheses about what went wrong and why 3. Versioner Agent — reads all suggestions, decides what to incorporate, generates a new version of the system prompt This creates a feedback loop: run benchmark → analyze failures → patch prompt → repeat. The final production prompt was the 80th generation — automatically evolved from a basic starting point through dozens of iterations, each fixing specific failure patterns discovered by the analyzer. No manual prompt engineering. Just agents improving agents.

Model(s): openai/gpt-oss-120b LLM Calls: 864 Prompt Tokens: 1.16M Completion Tokens: 564.27k Architecture: Multiagent oss-120b LLM: gpt-oss-120b Used exclusively via the Cerebras provider for speed (up to ~3k tokens/s). The architecture was based on a modified SGR NextStep with a tricky context-management logic: it fed the whole plan field from the last turn, not just the first step of the plan. All turns except the immediately previous one were kept in the LLM context in a compressed form. Each turn of the main NextStep flow was checked by a StepValidator. If everything was OK, the tool was executed and the flow continued as usual (the validator's work was not reflected in the context at all). Otherwise, the last NextStep message was sent for rework with the validator's comments. System instructions were extracted from wiki files by an LLM during the ingestion phase. The system prompt was loaded dynamically depending on whoami (public vs authenticated). The system prompt contained minimal information about /respond formatting. Detailed instructions for /respond were loaded by calling a pseudo-tool. The /whoami call was triggered automatically at the start of a task. A dynamic user context enrichment feature was used. Before the main agent started, the system code automatically pulled the user's full profile, projects, clients, and time entries by user ID. A separate LLM pass then filtered this data, and only the task-relevant subset was fed into the main LLM flow. Tool wrappers: - Pagination was effectively removed from all tools. A separate auto-pagination function would paginate through all pages and return the full list. - Req_LogTimeEntry was rebuilt because it was the only tool in the SDK that was constructed with a different field order, where the tool field was not first, which confused the model. - Also, as mentioned above, an extra Req_LoadRespondInstructions pseudo-tool was added to load the detalied /respond instructions. All tools were invoked via Structured Output instead of native tool calling. Issues: I set the turn limit for the main NextStep flow too low, so 5 of 103 tasks were simply not completed. There was not enough time left before the competition ended to rerun with a higher limit. Running all 103 tasks took about 1,430 LLM requests, $6.8, 15 minutes (with parallel task execution), 17.7M input-context tokens, and 838K output-context tokens. The main contributor to output tokens was reasoning. LLM: gpt-oss-120b via Cerebras Core agent: modified SGR NextStep with Steps validation and custom context strategy System prompts: routed based on /whoami User context: enriched by auto-loading from API with subsequent LLM filtering Tools: auto-pagination wrapper

Model(s): gpt-4.1 LLM Calls: 182 Prompt Tokens: 0.12M Completion Tokens: 21.85k Architecture: OpenAI Agent runtime + SGR The core of the agent is built on the OpenAI runtime using the GPT-4.1 model. Tool usage is implemented via Function Calling with structured outputs. A significant part of the work was focused on designing convenient and reliable agent tools, especially for search. For this purpose, text-embedding-3-large embeddings were used. Regarding context handling, the main principle was to keep the agent’s own instructions minimal and rely on distilled wiki-based knowledge, with special care taken to preserve the original rules and constraints without distortion.

Model(s): deepseek-reasoner LLM Calls: 1,527 Prompt Tokens: 1.30M Completion Tokens: 277.21k Architecture: Simple Agent & deepseek-reasoner # A. Ovsov. I implemented a single-agent architecture where tools are mapped 1:1 to the API endpoints without modification. I added only one custom tool, ask_wiki, which allows the agent to ask natural language questions about the wiki. The implementation of ask_wiki is straightforward: the entire wiki content is injected into the system prompt (which proves to be highly efficient due to context caching). The agent's main system prompt is concise (**only 320 tokens**) to avoid overfitting; it contains only wiki-independent facts. It defines a mandatory execution sequence: 1) Call who_am_i and get_employee... 2) Call ask_wiki to retrieve user permissions... 3) Validate security. If the user lacks permissions... 4) If authorized, fulfill the User task... (plus a few more instructions). Performance: The deepseek-reasoner model performed the best—it offered the optimal balance of accuracy, speed, and cost. * Cost: ~$0.60 per 100 tasks. * Efficiency: Average cache hit/miss ratio ≈ 30. Conclusion: I considered applying the approaches from your sgr-agent-erc3-test sample, but ultimately settled on a simpler (and, in my view, more universal) architecture.

Model(s): openai/gpt-4.1 LLM Calls: 543 Prompt Tokens: 0.20M Completion Tokens: 33.20k Architecture: Langchain Tool Call Agent w/ openai/gpt-4.1 IT Development Team d.o.o., Slovenia. There were two models: - Langchain Tool Agent Qwen/Qwen3-4B-Thinking-2507 - Langchain Tool Agent openai/gpt-4.1 The solution is built around a ReAct-style agent implemented with LangChain. The base model is Qwen3-4B with reasoning capabilities. In addition, a lightweight safety layer (SO) is integrated, following the same general pattern as in the provided example. The tool's layer was adapted specifically for LangChain. The tool schemas were simplified by removing unused fields, while input and output parameter descriptions were expanded and clarified. Tool descriptions were also refined to improve the model’s accuracy in tool selection. Token usage was optimized using toon, and several custom tools were added, including a tool for retrieving the user’s current projects. For context handling, a rule distillation approach was applied. The original Wiki and instruction set were passed through the model to parse, compress, and restructure the rules into a compact system context. The conversation history itself was not compressed, as experiments showed that history compression degrades quality significantly, while Qwen3 handled long contexts reliably without noticeable performance loss. One notable observation is that, compared to GPT-5, the smaller reasoning-focused Qwen3 model did not exhibit tool-calling loops and showed more stable completion of reasoning chains.

Model(s): claude-sonnet-4.5, gpt-5.1 LLM Calls: 315 Prompt Tokens: 751.22k Completion Tokens: 30.66k Architecture: CC SDK with MCP Tools Claude Code SDK based agent with preflight validation, with dedicated post validation and recovery before submitting the result based on rules from wiki. - Improved tools schemas, I don't use SGR, but usual LLM function calling - For validation request I keep only rules, list of api tools called and the task. - For pre and post validation calls SGR is used Faults: missing_model 'none'

Model(s): gpt-5.1 LLM Calls: 727 Prompt Tokens: 1.10M Completion Tokens: 113.27k Architecture: React + think-tool + Structured reasoning I used gpt-5.1 with a vanilla ReAct agent on LangGraph. I implemented all ERC functions as tools, plus a few additional tools following agent-building best practices: > plan tool > think tool (for controlled reasoning) > critic tool (the critic tool uses structured output with dedicated reasoning fields). Context is a single continuous thread: at any moment the agent can see the full chain of its own reasoning and actions. Everything else was achieved through careful prompt engineering. I also plan to publish all source code in my Telegram channel: https://t.me/robofuture

Model(s): openai/gpt-4.1, openai/gpt-5.1-codex-max LLM Calls: 854 Prompt Tokens: 1.65M Completion Tokens: 240.10k Architecture: AtomicAgents + $openai/gpt-4.1 + Sonnet 4.5 # ERC3 Challenge Agent — Leaderboard Description **Multi-stage pipeline agent** built on `atomic-agents` framework with `instructor`-powered structured outputs. Uses a **6-step sequential workflow** that separates security validation, context extraction, and task execution. Based on gpt-5.1-codex-max and gpt4.1 LLM models. ## Agent Design - **Security Gate Agent**: Pre-execution LLM that validates permissions against wiki rules before the main loop runs. Blocks invalid requests early (spoofing detection, access control). - **Prompt Context Extraction Agent**: Surfaces critical rules from 500+ line system prompts so the execution agent doesn't miss important details. - **Execution Agent**: ReAct-style planning loop with chain-of-thought reasoning (5 phases: Identity → Threat Detection → Info Gathering → Access Validation → Execution). ## Tool Handling - **22 domain tools** covering identity, wiki, employees, customers, projects, and time tracking - **Auto-link generation**: Embedded `LinkGeneratorAgent` inside `RespondTool` automatically extracts entity links from response context, preventing missing-link failures - **Tool Provider pattern**: Centralized tool registry with typed Pydantic schemas for all inputs/outputs ## Context Strategy - **Aggressive preloading**: User context, projects, full customer details, and all company users loaded *before* execution starts - **API enrichment**: Project data enriched with complete customer info (location, deal phase, account manager) to minimize tool calls during execution - **SHA1-based caching**: Wiki content and extracted rules cached by content hash — instant reload when wiki unchanged, automatic invalidation on updates - **7-section wiki extraction**: Business rules parsed into structured sections (Fraud Prevention, Hierarchy, Nuances, Output Requirements, Error Handling, Workflow, Entity Linking) - **Memory accumulation**: Critical information from security gate and context extraction injected into execution agent's initial memory - **Runtime Context**: Accumulated memory from previous steps, full execution history (tool calls + results) ## Key Differentiators 1. **Pre-execution security gate** — invalid requests blocked before planning loop 2. **Context-rich prompts** — user projects with full team & customer data in system context 3. **Deterministic prompt assembly** — wiki sections + user context combined without LLM 4. **Automatic entity linking** — dedicated agent ensures correct links in every response 5. **Precision over helpfulness** — answers exactly what was asked, no extra suggestions

Model(s): google/gemini-2.5-flash LLM Calls: 740 Prompt Tokens: 0.72M Completion Tokens: 476.38k Architecture: NextStep SGR Agent

Model(s): openai/gpt-5.1 LLM Calls: 102 Prompt Tokens: 993.66k Completion Tokens: 111.80k Architecture: Tools agent openai/gpt-5.1

Model(s): gpt-5 LLM Calls: 508 Prompt Tokens: 0.33M Completion Tokens: 910.68k Architecture: DistillAgent(distiller_llm=gpt-5-medium, executor_llm=gpt-5-medium)

Model(s): gpt-oss-120b, openai/gpt-5.1-codex-max LLM Calls: 830 Prompt Tokens: 0.98M Completion Tokens: 0.10M Architecture: AtomicAgents + $gpt-oss-120b

Model(s): anthropic/claude-sonnet-4.5 LLM Calls: 73 Prompt Tokens: 1.67M Completion Tokens: 70.34k Architecture:

Model(s): gpt-5.1 LLM Calls: 493 Prompt Tokens: 0.18M Completion Tokens: 216.59k Architecture: ReAct Multi-Agent

Model(s): openai/gpt-4.1 LLM Calls: 510 Prompt Tokens: 0.38M Completion Tokens: 123.36k Architecture: AvaTar arch intellect-3

Model(s): qwen/qwen3-235b-a22b-2507 LLM Calls: 796 Prompt Tokens: 0.85M Completion Tokens: 269.28k Architecture: SGR Agent Parallel (OpenRouter qwen/qwen3-235b-a22b-2507) # ERC3 Agent — LangChain SGR with Hybrid RAG LLM: Qwen3-235B-A22B (Gonka Network decentralized inference) Core: Schema-Guided Reasoning — structured JSON output (thoughts → plan → action_queue) ## Architecture Highlights ### 1. Action Pipeline with Enrichers Pipeline orchestrates every API call through stages: - Preprocessors: Normalize requests (e.g., fetch-merge-dispatch for partial updates) - Executor: API call with retry and error handling - PostProcessors: Side effects (identity capture, wiki sync, security redaction) - Enrichers: Inject context-aware hints into agent's context ### 2. Enricher System — Intelligent Hints 20+ enrichers analyze API responses and inject guidance WITHOUT blocking: - RoleEnricher: After projects_get → "You are LEAD of this project, proceed with update" - ProjectOverlapAnalyzer: Finds shared projects → "DEFINITIVE MATCH: proj_X is the ONLY project where you can authorize" - PaginationHintEnricher: "next_offset=5 means MORE results! MUST paginate" - SkillSearchStrategyHint: "Use min_level=9 to find top experts first" - EfficiencyHint: "You called employees_get 6 times — BATCH them!" Key feature: Cross-turn persistence — definitive matches stored in shared state survive pagination. ### 3. Three-Mode Guard System Guards validate agent responses before submission: - Hard block: API-verified impossible (employee not in project team) - Soft block: Risky action — block first, allow on repeat with same warning_key - Soft hint: Guidance appended without blocking Examples: OutcomeValidationGuard catches denied_security without permission check; SubjectiveQueryGuard blocks ok_answer on "that cool project" queries. ### 4. Hybrid RAG Wiki System Local wiki cache with three-stream search: - Regex: Pattern matching for structured queries ("salary|privacy") - Semantic: sentence-transformers embeddings with cosine similarity - Keyword: Token overlap fallback SHA1-based versioning — each wiki version cached separately with pre-computed embeddings. Dynamic injection: when wiki hash changes mid-task, critical policies auto-injected. ### 5. Fuzzy Normalization Layer Handles human↔API naming mismatches in tool parsers: - "Italian language" → skill_italian - "Willingness to travel" → will_travel - Progressive truncation: skill_rail_industry_knowledge → skill_rail ### 6. Parallel Execution Thread-safe design for concurrent task execution: - Thread-local WikiManager instances - Global embedding model singleton with lock - Explicit task_id passing to stats (avoids race conditions) - Action batching: 10-30 API calls in ONE turn via action_queue

Model(s): anthropic/claude-haiku-4.5 LLM Calls: 75 Prompt Tokens: 1.65M Completion Tokens: 76.47k Architecture:

Model(s): openai/gpt-4.1 LLM Calls: 344 Prompt Tokens: 0.17M Completion Tokens: 44.22k Architecture: Simple NextStep SGR with structured distillation

Model(s): gpt-4.1, gpt-5.1 LLM Calls: 571 Prompt Tokens: 0.42M Completion Tokens: 168.89k Architecture: NextStep SGR Agent with OpenAI

Model(s): gpt-5.1 LLM Calls: 102 Prompt Tokens: 738.57k Completion Tokens: 98.61k Architecture: Codegen SGR Agent with Google GenAI

Model(s): gpt-5.1, qwen3-max LLM Calls: 396 Prompt Tokens: 0.28M Completion Tokens: 51.51k Architecture: NextStep SGR Agent with integrated tools from tools.py

Model(s): openai/gpt-4.1 LLM Calls: 375 Prompt Tokens: 0.18M Completion Tokens: 55.92k Architecture: NextStep SGR Agent with OpenAI and coding tools

Model(s): gpt-5-mini LLM Calls: 1,484 Prompt Tokens: 1.01M Completion Tokens: 0.10M Architecture: Plan/Act - OpenAI

Model(s): openai/gpt-4o, openai/o3 LLM Calls: 572 Prompt Tokens: 0.30M Completion Tokens: 243.31k Architecture: Discovery-Planner-Executor-Decider Pipeline

Model(s): gpt-4.1-mini LLM Calls: 423 Prompt Tokens: 0.18M Completion Tokens: 144.73k Architecture: NextStep SGR Agent with OpenAI ang Grok LLM: gpt-4.1-mini through Azure Distillation of Wiki to rules through Openrouter gpt-5.1-codex-max (but i recomend to use 5.1 instead) note: Implemented within 5 hours. Not all features which i implemented in my Store agent were merged to the erc3 Prod agent (due to a bad timing), see not merged but promising features on the bottom. Architecture "SGR-guidance": 0. mini model: gpt-4.1-mini. Wanted to see what i can get out of it. 1. Modified SGR NextStep architecture. included extra fields (see bellow) to bring rules and roles to LLM's focus of attention 2. SGR guidance on all levels: distill, preflight, nextstep. 3. Context compaction logic (was disabled during prod run, due to lack of testing) 4. Completion audit by LLM validator. when agent wants to finish whole task, a separate LLM checks results and approve or suggests to continue with different approach 5. Preloading of all current user profile data into starting prompt. 6. Auto-pagination wrappers for all operations with paging (including search) Issues: - problem on existing tool had to reimplement it as AddTimeLogEntryForUser - Openrouter was giving errors on json deserialization, switching to azure helped. - gpt-5 on azure was duplicating json, breaking SO. (Rinat's suggested fix kind of helped) But had to abandone using of gpt-5 to avoid suprises. Supporting infrastructure: 1. Per session Structured logging of each operation. is folder, _.json for tasks 2. Managing of tasks execution order using dictionary of tasks and their complexity scores. 3. Support to continue any incomple session through cli 4. parralel processing. SGR guidance: 1. SGR WIKI RULES DISTIL. Explicit list of allowed and forbiden operations per role: class DistillWikiRules(BaseModel ... role_operations: List[RoleOperation] class RoleOperation(BaseModel): role: Literal["guest", "regular emploee", "lead", "executive"] allowed_operations: List[str] dissallowed_operations: List[str] 2. SGR preflight, extracts summary what is needed for solving task, all roles and expicit rules following: class TaskProfile(BaseModel): goal_description: str rules_violated: List[str] rules_followed: List[str] actor_role: Literal["guest", "regular emploee", "lead", "executive"] target_employee_role: Optional[Literal["lead", "regular emploee", "executive"]] target_entity: Literal["employee", "customer", "project", "time_entry", "wiki_page", "system_dependency", "other"] required_items: List[str] applicable_recomendations_from_rules: List[str] 3. SGR NextStep, bring everything to the focus of attention of LLM: RuleResult(BaseModel): rule: str result: Literal["followed", "violated", "not_applicable"] class NextStep(BaseModel): ... current_actor_role: Literal["guest", "regular emploee", "lead", "executive"] all_rules_results: Annotated[List[RuleResult], MinLen(15), MaxLen(15)] confirm_requested_operation_is_allowed_by_the_role: bool enough_data: bool Features implemented in my Store Agent, but had no time to merge to the Prod agent: 1. Smart tools, passing extra function argument for a task/goal. before returning results of API llm filterig data against passed task. 2. Python code execution tool with restricted environment allowing explicit list of preinstalled libs. 3. openskills based skill call tool 4. Toon format. saves tokens. 5. Context compaction logic. extra field for summarized context in NextStep model Actual cost of solving 103 tasks was less than 0.87€ due to cheap 4.1-mini model. Experimenting with generation or rules in Prolog language, looks very promising. My TG: @figaroserg1 (@AIDrivenDev) Source code will be posted to my "AI driven apps for business" Blogs: https://t.me/AIDrivenDev and https://aidrivenapps.blogspot.com

Model(s): gemini-2.5-pro LLM Calls: 1,631 Prompt Tokens: 1.35M Completion Tokens: 492.97k Architecture: ERCPlanReActAgent, Model=gemini-2.5-pro

Model(s): gpt-4.1 LLM Calls: 669 Prompt Tokens: 0.20M Completion Tokens: 175.15k Architecture: Distilled Multi-Agent System combining pre-cached wiki rule distillation with multi-agent coordination (Orchestrator + specialized Workers)

Model(s): openai/gpt-4o LLM Calls: 103 Prompt Tokens: 0.51M Completion Tokens: 130.04k Architecture: AGES SGR Agent with gpt-4o (parallel) ------------------------------------------------------ This agent is an experiment in AI-assisted coding, it was coded automatically by a coding agent. Below is the summary of the architecture, produced by the same agent. Note how it overestimates its own accuracy. ------------------------------------------------------ AGES Agent Architecture (ERC3 Competition) AGES Agent is an AI agent designed for a corporate project management, employee, and time management system. It is built on GPT-4o and employs structured output via Pydantic schemas. Main Operational Cycle: The agent implements an iterative ReAct cycle: Thought → Plan → Action (tool) → Result → Next Step. The LLM returns strictly typed JSON adhering to a Pydantic schema (AgentStep → ToolCall), ensuring deterministic action routing and resilience against parsing errors. Validation occurs at the client.beta.chat.completions.parse() level, providing guaranteed parsing without regular expressions. The schema specifies the agent's current reasoning, chosen tool, and fully typed parameters, with a maximum of 25 steps per task. Core Components: Parallel Executor (main_parallel.py): * Executes up to 8 tasks simultaneously using ThreadPoolExecutor. * Accelerates session processing by 5-8 times compared to sequential execution. Agent Core (ages_agent_v2.py): * Contains logic for the cycle, LLM invocation, tool execution, and guardrails. * Supports standard models (GPT-4o) and Codex models via Responses API. Tools: * Wrappers around ERC3 API: * whoami, list/search/get for employees, projects, clients, and wiki. * log_time, update_project_status, update_employee for mutations. * respond for finalizing and classifying outcomes. Prompt System (~500 lines): * Detailed safety rules. * Search strategies for ambiguity resolution (CV-projects, cost center codes post-M&A). * Entity linking guidelines in responses. Guardrails (Protective Mechanisms): * Mandatory whoami execution before each task to establish user context. * Guest access blocking (is_public=true) prevents access to sensitive data. * Permissions verification ensures only Leads can change project statuses, only CEO can view salaries. * Automatic linking of current user and mentioned employees in responses. Error Handling and Resilience: * Fallback strategy: On search_* errors, automatically fallback to list_* with pagination. * Pagination limits: limit=5 for all requests to prevent API errors. * Invalid response handling: Graceful degradation when response fields are None. * Telemetry: Reports used tokens and completion texts to ERC3 API. Task Completion: Finalized via respond(message, outcome, links), providing a final response and classification: * ok_answer: Task successfully completed. * denied_security: Rejected for security reasons. * none_clarification_needed: Further clarification required. * error_internal: Internal system error. Best result achieved: 70/100 tasks (70% accuracy) on ERC3-PROD.

Model(s): openai/gpt-4.1 LLM Calls: 705 Prompt Tokens: 0.39M Completion Tokens: 226.54k Architecture: LLM-driven with confidence loop, no hardcoded rules

Model(s): openai/gpt-5.1 LLM Calls: 324 Prompt Tokens: 0.10M Completion Tokens: 250.70k Architecture: NextStep SGR Agent with OpenAI

Model(s): google/gemini-2.5-flash LLM Calls: 756 Prompt Tokens: 0.31M Completion Tokens: 209.92k Architecture: Router -> Searcher -> Executor

Model(s): google/gemini-2.5-pro LLM Calls: 251 Prompt Tokens: 452.41k Completion Tokens: 40.10k Architecture: NextStep SGR Agent with Gemini ADK

Model(s): openai/gpt-4o LLM Calls: 417 Prompt Tokens: 0.27M Completion Tokens: 70.81k Architecture: Vladimir Penkov, Agentic workflow

Model(s): gpt-4o LLM Calls: 102 Prompt Tokens: 593.03k Completion Tokens: 5.55k Architecture: ERC3 Agent v3 with SGR framework integration + memory compression

Model(s): qwen/qwen3-235b-a22b-2507 LLM Calls: 553 Prompt Tokens: 725.54k Completion Tokens: 112.01k Architecture: LangGraph OODA Agent (ERC3) ERC3 Agent Architecture: Single-agent OODA loop (Observe -> Orient -> Decide -> Act) Runs on Cerebras inference (qwen/qwen3-235b-a22b-2507) via OpenRouter. Core mechanism: Structured Outputs using Pydantic schema NextStep. The model must emit one structured step per iteration. NEXTSTEP SCHEMA think — concise reasoning (1–2 sentences) scratch — working notes (last 500 chars kept) memory — confirmed facts / IDs (compressed string) function— typed ERC3 tool call OODA LOOP Observe: - who_am_i(), TaskInfo Orient: - build_system_prompt - search rules - context assembly (memory, scratch, IDs) Decide: - LLM call via client.beta.chat.completions.parse(response_format=NextStep) Act: - api.dispatch(fn) - error classification - memory / scratch update - loop tracking Ends on Req_ProvideAgentResponse, done=true, or MAX_STEPS. GUARDRAILS - Pre-gen: guest denial, regex deny-list, vague fast-path blocking - Anti-hallucination: heuristic fake-ID blocking - Action verification: blocks ok_answer if mutation missing RUNTIME - Parallelism: ThreadPoolExecutor (5 workers) - Rate limit: fixed minimum interval (no token bucket) - Model: qwen/qwen3-235b-a22b-2507 - Provider: OpenRouter (Cerebras) REPO https://github.com/ai-babai/erc3-ooda-agent

Model(s): openai/gpt-4.1, openai/gpt-5.1 LLM Calls: 150 Prompt Tokens: 594.23k Completion Tokens: 113.00k Architecture: Graph Agent with OpenAI

Model(s): gpt-4o LLM Calls: 329 Prompt Tokens: 286.94k Completion Tokens: 32.38k Architecture: SGR-LangGraph

Model(s): anthropic/claude-sonnet-4.5 LLM Calls: 727 Prompt Tokens: 0.42M Completion Tokens: 121.93k Architecture: CASCADE pattern with complete API schema and optimized search strategies with OpenRouter/Claude

Model(s): openai/gpt-4.1 LLM Calls: 543 Prompt Tokens: 0.20M Completion Tokens: 33.20k Architecture: Langchain Tool Call Agent w/ openai/gpt-4.1 IT Development Team d.o.o., Slovenia. There were two models: - Langchain Tool Agent Qwen/Qwen3-4B-Thinking-2507 - Langchain Tool Agent openai/gpt-4.1 The solution is built around a ReAct-style agent implemented with LangChain. The base model is Qwen3-4B with reasoning capabilities. In addition, a lightweight safety layer (SO) is integrated, following the same general pattern as in the provided example. The tool's layer was adapted specifically for LangChain. The tool schemas were simplified by removing unused fields, while input and output parameter descriptions were expanded and clarified. Tool descriptions were also refined to improve the model’s accuracy in tool selection. Token usage was optimized using toon, and several custom tools were added, including a tool for retrieving the user’s current projects. For context handling, a rule distillation approach was applied. The original Wiki and instruction set were passed through the model to parse, compress, and restructure the rules into a compact system context. The conversation history itself was not compressed, as experiments showed that history compression degrades quality significantly, while Qwen3 handled long contexts reliably without noticeable performance loss. One notable observation is that, compared to GPT-5, the smaller reasoning-focused Qwen3 model did not exhibit tool-calling loops and showed more stable completion of reasoning chains.

Model(s): google/gemini-2.5-flash LLM Calls: 740 Prompt Tokens: 0.72M Completion Tokens: 476.38k Architecture: NextStep SGR Agent

Model(s): openai/gpt-4o, x-ai/grok-4-fast LLM Calls: 601 Prompt Tokens: 0.18M Completion Tokens: 45.21k Architecture: vladimir.v.penkov@gmail.com, Ich suche Arbeit. Agentic workflow

Model(s): qwen/qwen3-235b-a22b-2507, rule-based LLM Calls: 501 Prompt Tokens: 0.37M Completion Tokens: 174.80k Architecture: [dtbz] OODA Loop Agent (direct) ERC3 Agent Architecture: Single-agent OODA loop (Observe -> Orient -> Decide -> Act) Runs on Cerebras inference (qwen/qwen3-235b-a22b-2507) via OpenRouter. Core mechanism: Structured Outputs using Pydantic schema NextStep. The model must emit one structured step per iteration. NEXTSTEP SCHEMA think — concise reasoning (1–2 sentences) scratch — working notes (last 500 chars kept) memory — confirmed facts / IDs (compressed string) function— typed ERC3 tool call OODA LOOP Observe: - who_am_i(), TaskInfo Orient: - build_system_prompt - search rules - context assembly (memory, scratch, IDs) Decide: - LLM call via client.beta.chat.completions.parse(response_format=NextStep) Act: - api.dispatch(fn) - error classification - memory / scratch update - loop tracking Ends on Req_ProvideAgentResponse, done=true, or MAX_STEPS. GUARDRAILS - Pre-gen: guest denial, regex deny-list, vague fast-path blocking - Anti-hallucination: heuristic fake-ID blocking - Action verification: blocks ok_answer if mutation missing RUNTIME - Parallelism: ThreadPoolExecutor (5 workers) - Rate limit: fixed minimum interval (no token bucket) - Model: qwen/qwen3-235b-a22b-2507 - Provider: OpenRouter (Cerebras) REPO https://github.com/ai-babai/erc3-ooda-agent Faults: Model rule-based is not found on OpenRouter

Model(s): google/gemini-2.5-pro LLM Calls: 283 Prompt Tokens: 583.51k Completion Tokens: 43.97k Architecture: NextStep SGR Agent with Gemini ADK

Model(s): gpt-4o LLM Calls: 102 Prompt Tokens: 593.03k Completion Tokens: 5.55k Architecture: ERC3 Agent v3 with SGR framework integration + memory compression

Model(s): gpt-5 LLM Calls: 16 Prompt Tokens: 316.35k Completion Tokens: 14.78k Architecture: NextStep SGR Agent with integrated tools from tools.py

Model(s): openai/o3, x-ai/grok-4-fast LLM Calls: 16 Prompt Tokens: 237.29k Completion Tokens: 10.95k Architecture: Discovery-Planner-Executor-Decider Pipeline

Model(s): openai/gpt-oss-120b LLM Calls: 864 Prompt Tokens: 1.16M Completion Tokens: 564.27k Architecture: Multiagent oss-120b LLM: gpt-oss-120b Used exclusively via the Cerebras provider for speed (up to ~3k tokens/s). The architecture was based on a modified SGR NextStep with a tricky context-management logic: it fed the whole plan field from the last turn, not just the first step of the plan. All turns except the immediately previous one were kept in the LLM context in a compressed form. Each turn of the main NextStep flow was checked by a StepValidator. If everything was OK, the tool was executed and the flow continued as usual (the validator's work was not reflected in the context at all). Otherwise, the last NextStep message was sent for rework with the validator's comments. System instructions were extracted from wiki files by an LLM during the ingestion phase. The system prompt was loaded dynamically depending on whoami (public vs authenticated). The system prompt contained minimal information about /respond formatting. Detailed instructions for /respond were loaded by calling a pseudo-tool. The /whoami call was triggered automatically at the start of a task. A dynamic user context enrichment feature was used. Before the main agent started, the system code automatically pulled the user's full profile, projects, clients, and time entries by user ID. A separate LLM pass then filtered this data, and only the task-relevant subset was fed into the main LLM flow. Tool wrappers: - Pagination was effectively removed from all tools. A separate auto-pagination function would paginate through all pages and return the full list. - Req_LogTimeEntry was rebuilt because it was the only tool in the SDK that was constructed with a different field order, where the tool field was not first, which confused the model. - Also, as mentioned above, an extra Req_LoadRespondInstructions pseudo-tool was added to load the detalied /respond instructions. All tools were invoked via Structured Output instead of native tool calling. Issues: I set the turn limit for the main NextStep flow too low, so 5 of 103 tasks were simply not completed. There was not enough time left before the competition ended to rerun with a higher limit. Running all 103 tasks took about 1,430 LLM requests, $6.8, 15 minutes (with parallel task execution), 17.7M input-context tokens, and 838K output-context tokens. The main contributor to output tokens was reasoning. LLM: gpt-oss-120b via Cerebras Core agent: modified SGR NextStep with Steps validation and custom context strategy System prompts: routed based on /whoami User context: enriched by auto-loading from API with subsequent LLM filtering Tools: auto-pagination wrapper

Model(s): qwen/qwen3-235b-a22b-2507 LLM Calls: 597 Prompt Tokens: 0.34M Completion Tokens: 156.71k Architecture: SGR Agent Parallel (openrouter qwen/qwen3-235b-a22b-2507) # ERC3 Agent — LangChain SGR with Hybrid RAG LLM: Qwen3-235B-A22B (Gonka Network decentralized inference) Core: Schema-Guided Reasoning — structured JSON output (thoughts → plan → action_queue) ## Architecture Highlights ### 1. Action Pipeline with Enrichers Pipeline orchestrates every API call through stages: - Preprocessors: Normalize requests (e.g., fetch-merge-dispatch for partial updates) - Executor: API call with retry and error handling - PostProcessors: Side effects (identity capture, wiki sync, security redaction) - Enrichers: Inject context-aware hints into agent's context ### 2. Enricher System — Intelligent Hints 20+ enrichers analyze API responses and inject guidance WITHOUT blocking: - RoleEnricher: After projects_get → "You are LEAD of this project, proceed with update" - ProjectOverlapAnalyzer: Finds shared projects → "DEFINITIVE MATCH: proj_X is the ONLY project where you can authorize" - PaginationHintEnricher: "next_offset=5 means MORE results! MUST paginate" - SkillSearchStrategyHint: "Use min_level=9 to find top experts first" - EfficiencyHint: "You called employees_get 6 times — BATCH them!" Key feature: Cross-turn persistence — definitive matches stored in shared state survive pagination. ### 3. Three-Mode Guard System Guards validate agent responses before submission: - Hard block: API-verified impossible (employee not in project team) - Soft block: Risky action — block first, allow on repeat with same warning_key - Soft hint: Guidance appended without blocking Examples: OutcomeValidationGuard catches denied_security without permission check; SubjectiveQueryGuard blocks ok_answer on "that cool project" queries. ### 4. Hybrid RAG Wiki System Local wiki cache with three-stream search: - Regex: Pattern matching for structured queries ("salary|privacy") - Semantic: sentence-transformers embeddings with cosine similarity - Keyword: Token overlap fallback SHA1-based versioning — each wiki version cached separately with pre-computed embeddings. Dynamic injection: when wiki hash changes mid-task, critical policies auto-injected. ### 5. Fuzzy Normalization Layer Handles human↔API naming mismatches in tool parsers: - "Italian language" → skill_italian - "Willingness to travel" → will_travel - Progressive truncation: skill_rail_industry_knowledge → skill_rail ### 6. Parallel Execution Thread-safe design for concurrent task execution: - Thread-local WikiManager instances - Global embedding model singleton with lock - Explicit task_id passing to stats (avoids race conditions) - Action batching: 10-30 API calls in ONE turn via action_queue

Model(s): qwen3-235b-a22b-2507 LLM Calls: 1,675 Prompt Tokens: 2.85M Completion Tokens: 190.95k Architecture: SGR Tool Calling Agent with Security Checks - OpenAI Function Calling # Architecture Overview: SGR Agent for ERC3-DEV Benchmark Uses open-source agent: https://github.com/vamplabAI/sgr-agent-core **Development**: 2 hours | **Deployment**: 8x H100 GPU cluster | **Result**: 0.46% accuracy ## System Architecture Three-layer design built on **Schema-Guided Reasoning (SGR)** framework: ### 1. SGR Agent Core - Two-Phase Loop - **Reasoning Phase**: Analyzes context, evaluates permissions, plans next action - **Action Phase**: Selects and executes tool with validated parameters - **Hybrid Mode**: First iteration forced reasoning, then AUTO mode (20-30% faster) ### 2. ERC3-DEV Adapter - **26 specialized tools**: Wiki, Employees, Projects, Customers, Time Tracking - **Security system**: Role-based access (CEO, HR, Project Lead, Employee, Guest) - **History compression**: Keeps recent 4 messages + compressed summary (40% token savings) - **Forced completion**: Prevents infinite loops at iteration limit ### 3. Parallel Execution Infrastructure - **Complete isolation**: Each task gets separate OpenAI client, API client, tools, conversation history - **Concurrency control**: `asyncio.Semaphore` limits concurrent tasks (3 default, 8 on H100 cluster) - **8x speedup**: 103 tasks in 31 minutes ## Key Optimizations 1. **Exponential backoff retry** (10 retries) - handles API errors, rate limits, validation errors 2. **Prompt caching** - 60-80% cache hit rate, 65% cost reduction 3. **History compression** - supports 40+ iterations without context overflow 4. **Context compression** - every 6 step compress all tool history ## Technology Stack Python 3.11+ | asyncio | Pydantic v2 | OpenAI API | ERC3 SDK | 8x H100 cluster | SGRAgentCore

Model(s): gpt-5.1, gpt-oss-120b LLM Calls: 256 Prompt Tokens: 0.51M Completion Tokens: 111.34k Architecture: NextStep SGR Agent with integrated tools from tools.py

Model(s): qwen/qwen3-235b-a22b-2507, rule-based LLM Calls: 539 Prompt Tokens: 0.40M Completion Tokens: 179.01k Architecture: [nfuz] OODA Loop Agent (direct) ERC3 Agent Architecture: Single-agent OODA loop (Observe -> Orient -> Decide -> Act) Runs on Cerebras inference (qwen/qwen3-235b-a22b-2507) via OpenRouter. Core mechanism: Structured Outputs using Pydantic schema NextStep. The model must emit one structured step per iteration. NEXTSTEP SCHEMA think — concise reasoning (1–2 sentences) scratch — working notes (last 500 chars kept) memory — confirmed facts / IDs (compressed string) function— typed ERC3 tool call OODA LOOP Observe: - who_am_i(), TaskInfo Orient: - build_system_prompt - search rules - context assembly (memory, scratch, IDs) Decide: - LLM call via client.beta.chat.completions.parse(response_format=NextStep) Act: - api.dispatch(fn) - error classification - memory / scratch update - loop tracking Ends on Req_ProvideAgentResponse, done=true, or MAX_STEPS. GUARDRAILS - Pre-gen: guest denial, regex deny-list, vague fast-path blocking - Anti-hallucination: heuristic fake-ID blocking - Action verification: blocks ok_answer if mutation missing RUNTIME - Parallelism: ThreadPoolExecutor (5 workers) - Rate limit: fixed minimum interval (no token bucket) - Model: qwen/qwen3-235b-a22b-2507 - Provider: OpenRouter (Cerebras) REPO https://github.com/ai-babai/erc3-ooda-agent Faults: Model rule-based is not found on OpenRouter

Model(s): qwen3-4b-thinking-2507 LLM Calls: 241 Prompt Tokens: 798.04k Completion Tokens: 465.34k Architecture: Langchain Tool Call Agent w/ Qwen/Qwen3-4B-Thinking-2507 IT Development Team d.o.o., Slovenia. There were two models: - Langchain Tool Agent Qwen/Qwen3-4B-Thinking-2507 - Langchain Tool Agent openai/gpt-4.1 The solution is built around a ReAct-style agent implemented with LangChain. The base model is Qwen3-4B with reasoning capabilities. In addition, a lightweight safety layer (SO) is integrated, following the same general pattern as in the provided example. The tool's layer was adapted specifically for LangChain. The tool schemas were simplified by removing unused fields, while input and output parameter descriptions were expanded and clarified. Tool descriptions were also refined to improve the model’s accuracy in tool selection. Token usage was optimized using toon, and several custom tools were added, including a tool for retrieving the user’s current projects. For context handling, a rule distillation approach was applied. The original Wiki and instruction set were passed through the model to parse, compress, and restructure the rules into a compact system context. The conversation history itself was not compressed, as experiments showed that history compression degrades quality significantly, while Qwen3 handled long contexts reliably without noticeable performance loss. One notable observation is that, compared to GPT-5, the smaller reasoning-focused Qwen3 model did not exhibit tool-calling loops and showed more stable completion of reasoning chains.

Model(s): qwen/qwen3-30b-a3b-instruct-2507-mlx@6bit LLM Calls: 179 Prompt Tokens: 0 Completion Tokens: 0 Architecture: ReAct Multi-Agent Faults: Model qwen/qwen3-30b-a3b-instruct-2507-mlx@6bit is not found on OpenRouter

Model(s): gpt-5.1, qwen/qwen3-32b LLM Calls: 428 Prompt Tokens: 0.25M Completion Tokens: 103.84k Architecture: NextStep SGR Agent with OpenAI

Model(s): claude-opus-4.5 LLM Calls: 685 Prompt Tokens: 1.17M Completion Tokens: 149.48k Architecture: Anthropic SDK Agent PARALLEL (5w) with claude-opus-4-5-20251101 # ERC3 Agent Architecture ## The Basics Fairly simple architecture: the main agent is built on Anthropic Python SDK with native Tool Use. Model - Opus 4.5. All 20+ tools are described in a single file using Anthropic's JSON Schema format. Tool execution dynamically constructs HTTP requests to the benchmark API — no code generation, just endpoint mapping. The system prompt distills all key rules from the company wiki into a compact decision algorithm: check identity → verify permissions → gather data → respond with proper outcome. ## The Interesting Part: Self-Evolving Agent The real cool thing was in automated prompt evolution using a three-agent pipeline: 1. Main Agent — runs the benchmark, solves all tasks, logs everything 2. Analyzer Agent — reviews logs of failed tasks, formulates hypotheses about what went wrong and why 3. Versioner Agent — reads all suggestions, decides what to incorporate, generates a new version of the system prompt This creates a feedback loop: run benchmark → analyze failures → patch prompt → repeat. The final production prompt was the 80th generation — automatically evolved from a basic starting point through dozens of iterations, each fixing specific failure patterns discovered by the analyzer. No manual prompt engineering. Just agents improving agents.

Model(s): openai/gpt-oss-120b LLM Calls: 864 Prompt Tokens: 1.16M Completion Tokens: 564.27k Architecture: Multiagent oss-120b LLM: gpt-oss-120b Used exclusively via the Cerebras provider for speed (up to ~3k tokens/s). The architecture was based on a modified SGR NextStep with a tricky context-management logic: it fed the whole plan field from the last turn, not just the first step of the plan. All turns except the immediately previous one were kept in the LLM context in a compressed form. Each turn of the main NextStep flow was checked by a StepValidator. If everything was OK, the tool was executed and the flow continued as usual (the validator's work was not reflected in the context at all). Otherwise, the last NextStep message was sent for rework with the validator's comments. System instructions were extracted from wiki files by an LLM during the ingestion phase. The system prompt was loaded dynamically depending on whoami (public vs authenticated). The system prompt contained minimal information about /respond formatting. Detailed instructions for /respond were loaded by calling a pseudo-tool. The /whoami call was triggered automatically at the start of a task. A dynamic user context enrichment feature was used. Before the main agent started, the system code automatically pulled the user's full profile, projects, clients, and time entries by user ID. A separate LLM pass then filtered this data, and only the task-relevant subset was fed into the main LLM flow. Tool wrappers: - Pagination was effectively removed from all tools. A separate auto-pagination function would paginate through all pages and return the full list. - Req_LogTimeEntry was rebuilt because it was the only tool in the SDK that was constructed with a different field order, where the tool field was not first, which confused the model. - Also, as mentioned above, an extra Req_LoadRespondInstructions pseudo-tool was added to load the detalied /respond instructions. All tools were invoked via Structured Output instead of native tool calling. Issues: I set the turn limit for the main NextStep flow too low, so 5 of 103 tasks were simply not completed. There was not enough time left before the competition ended to rerun with a higher limit. Running all 103 tasks took about 1,430 LLM requests, $6.8, 15 minutes (with parallel task execution), 17.7M input-context tokens, and 838K output-context tokens. The main contributor to output tokens was reasoning. LLM: gpt-oss-120b via Cerebras Core agent: modified SGR NextStep with Steps validation and custom context strategy System prompts: routed based on /whoami User context: enriched by auto-loading from API with subsequent LLM filtering Tools: auto-pagination wrapper

Model(s): gpt-4.1 LLM Calls: 182 Prompt Tokens: 0.12M Completion Tokens: 21.85k Architecture: OpenAI Agent runtime + SGR The core of the agent is built on the OpenAI runtime using the GPT-4.1 model. Tool usage is implemented via Function Calling with structured outputs. A significant part of the work was focused on designing convenient and reliable agent tools, especially for search. For this purpose, text-embedding-3-large embeddings were used. Regarding context handling, the main principle was to keep the agent’s own instructions minimal and rely on distilled wiki-based knowledge, with special care taken to preserve the original rules and constraints without distortion.

Model(s): deepseek-reasoner LLM Calls: 1,527 Prompt Tokens: 1.30M Completion Tokens: 277.21k Architecture: Simple Agent & deepseek-reasoner # A. Ovsov. I implemented a single-agent architecture where tools are mapped 1:1 to the API endpoints without modification. I added only one custom tool, ask_wiki, which allows the agent to ask natural language questions about the wiki. The implementation of ask_wiki is straightforward: the entire wiki content is injected into the system prompt (which proves to be highly efficient due to context caching). The agent's main system prompt is concise (**only 320 tokens**) to avoid overfitting; it contains only wiki-independent facts. It defines a mandatory execution sequence: 1) Call who_am_i and get_employee... 2) Call ask_wiki to retrieve user permissions... 3) Validate security. If the user lacks permissions... 4) If authorized, fulfill the User task... (plus a few more instructions). Performance: The deepseek-reasoner model performed the best—it offered the optimal balance of accuracy, speed, and cost. * Cost: ~$0.60 per 100 tasks. * Efficiency: Average cache hit/miss ratio ≈ 30. Conclusion: I considered applying the approaches from your sgr-agent-erc3-test sample, but ultimately settled on a simpler (and, in my view, more universal) architecture.

Model(s): anthropic/claude-sonnet-4.5 LLM Calls: 795 Prompt Tokens: 0.48M Completion Tokens: 131.18k Architecture: CASCADE pattern with complete API schema and optimized search strategies with OpenRouter/Claude

Model(s): gpt-4.1 LLM Calls: 384 Prompt Tokens: 0.30M Completion Tokens: 61.72k Architecture: Multistage agent

Model(s): gpt-5.1 LLM Calls: 301 Prompt Tokens: 0.11M Completion Tokens: 29.78k Architecture: CC SDK with MCP Tools Claude Code SDK based agent with preflight validation, with dedicated post validation and recovery before submitting the result based on rules from wiki. - Improved tools schemas, I don't use SGR, but usual LLM function calling - For validation request I keep only rules, list of api tools called and the task. - For pre and post validation calls SGR is used Faults: missing_model 'none'

Model(s): openai/gpt-5.2 LLM Calls: 458 Prompt Tokens: 0.32M Completion Tokens: 163.71k Architecture: A NextStep SGR agent: the LLM produces a single schema-validated JSON step (state + brief plan + one typed tool call), then executes it and feeds the tool output back in a plan→act→observe→repair loop with retries. Tech stack: SGR (Schema-Guided Reasoning), Pydantic schemas, typed tool routing over the ERC3 API, and OpenAI as the planner/decider, plus preflight/policy guards.

Model(s): gpt-4o LLM Calls: 87 Prompt Tokens: 87 Completion Tokens: 87 Architecture: NextStep SGR Agent with OpenAI

Model(s): openai/gpt-4.1 LLM Calls: 543 Prompt Tokens: 0.20M Completion Tokens: 33.20k Architecture: Langchain Tool Call Agent w/ openai/gpt-4.1 IT Development Team d.o.o., Slovenia. There were two models: - Langchain Tool Agent Qwen/Qwen3-4B-Thinking-2507 - Langchain Tool Agent openai/gpt-4.1 The solution is built around a ReAct-style agent implemented with LangChain. The base model is Qwen3-4B with reasoning capabilities. In addition, a lightweight safety layer (SO) is integrated, following the same general pattern as in the provided example. The tool's layer was adapted specifically for LangChain. The tool schemas were simplified by removing unused fields, while input and output parameter descriptions were expanded and clarified. Tool descriptions were also refined to improve the model’s accuracy in tool selection. Token usage was optimized using toon, and several custom tools were added, including a tool for retrieving the user’s current projects. For context handling, a rule distillation approach was applied. The original Wiki and instruction set were passed through the model to parse, compress, and restructure the rules into a compact system context. The conversation history itself was not compressed, as experiments showed that history compression degrades quality significantly, while Qwen3 handled long contexts reliably without noticeable performance loss. One notable observation is that, compared to GPT-5, the smaller reasoning-focused Qwen3 model did not exhibit tool-calling loops and showed more stable completion of reasoning chains.

Model(s): gpt-4.1 LLM Calls: 545 Prompt Tokens: 0.33M Completion Tokens: 67.12k Architecture: ReAct Multi-Agent

Model(s): gpt-5.1 LLM Calls: 727 Prompt Tokens: 1.10M Completion Tokens: 113.27k Architecture: React + think-tool + Structured reasoning I used gpt-5.1 with a vanilla ReAct agent on LangGraph. I implemented all ERC functions as tools, plus a few additional tools following agent-building best practices: > plan tool > think tool (for controlled reasoning) > critic tool (the critic tool uses structured output with dedicated reasoning fields). Context is a single continuous thread: at any moment the agent can see the full chain of its own reasoning and actions. Everything else was achieved through careful prompt engineering. I also plan to publish all source code in my Telegram channel: https://t.me/robofuture

Model(s): openai/gpt-4.1, openai/gpt-5.1-codex-max LLM Calls: 854 Prompt Tokens: 1.65M Completion Tokens: 240.10k Architecture: AtomicAgents + $openai/gpt-4.1 + Sonnet 4.5 # ERC3 Challenge Agent — Leaderboard Description **Multi-stage pipeline agent** built on `atomic-agents` framework with `instructor`-powered structured outputs. Uses a **6-step sequential workflow** that separates security validation, context extraction, and task execution. Based on gpt-5.1-codex-max and gpt4.1 LLM models. ## Agent Design - **Security Gate Agent**: Pre-execution LLM that validates permissions against wiki rules before the main loop runs. Blocks invalid requests early (spoofing detection, access control). - **Prompt Context Extraction Agent**: Surfaces critical rules from 500+ line system prompts so the execution agent doesn't miss important details. - **Execution Agent**: ReAct-style planning loop with chain-of-thought reasoning (5 phases: Identity → Threat Detection → Info Gathering → Access Validation → Execution). ## Tool Handling - **22 domain tools** covering identity, wiki, employees, customers, projects, and time tracking - **Auto-link generation**: Embedded `LinkGeneratorAgent` inside `RespondTool` automatically extracts entity links from response context, preventing missing-link failures - **Tool Provider pattern**: Centralized tool registry with typed Pydantic schemas for all inputs/outputs ## Context Strategy - **Aggressive preloading**: User context, projects, full customer details, and all company users loaded *before* execution starts - **API enrichment**: Project data enriched with complete customer info (location, deal phase, account manager) to minimize tool calls during execution - **SHA1-based caching**: Wiki content and extracted rules cached by content hash — instant reload when wiki unchanged, automatic invalidation on updates - **7-section wiki extraction**: Business rules parsed into structured sections (Fraud Prevention, Hierarchy, Nuances, Output Requirements, Error Handling, Workflow, Entity Linking) - **Memory accumulation**: Critical information from security gate and context extraction injected into execution agent's initial memory - **Runtime Context**: Accumulated memory from previous steps, full execution history (tool calls + results) ## Key Differentiators 1. **Pre-execution security gate** — invalid requests blocked before planning loop 2. **Context-rich prompts** — user projects with full team & customer data in system context 3. **Deterministic prompt assembly** — wiki sections + user context combined without LLM 4. **Automatic entity linking** — dedicated agent ensures correct links in every response 5. **Precision over helpfulness** — answers exactly what was asked, no extra suggestions

Model(s): google/gemini-2.5-flash LLM Calls: 740 Prompt Tokens: 0.72M Completion Tokens: 476.38k Architecture: NextStep SGR Agent

Model(s): openai/gpt-5.1 LLM Calls: 102 Prompt Tokens: 993.66k Completion Tokens: 111.80k Architecture: Tools agent openai/gpt-5.1

Model(s): gpt-5 LLM Calls: 508 Prompt Tokens: 0.33M Completion Tokens: 910.68k Architecture: DistillAgent(distiller_llm=gpt-5-medium, executor_llm=gpt-5-medium)

Model(s): openai/gpt-4o, x-ai/grok-4-fast LLM Calls: 578 Prompt Tokens: 0.16M Completion Tokens: 42.44k Architecture: Vladimir Penkov, Agentic workflow

Model(s): gpt-oss-120b, openai/gpt-5.1-codex-max LLM Calls: 830 Prompt Tokens: 0.98M Completion Tokens: 0.10M Architecture: AtomicAgents + $gpt-oss-120b

Model(s): qwen3-235b-a22b-2507 LLM Calls: 1,675 Prompt Tokens: 2.85M Completion Tokens: 190.95k Architecture: SGR Tool Calling Agent with Security Checks - OpenAI Function Calling # Architecture Overview: SGR Agent for ERC3-DEV Benchmark Uses open-source agent: https://github.com/vamplabAI/sgr-agent-core **Development**: 2 hours | **Deployment**: 8x H100 GPU cluster | **Result**: 0.46% accuracy ## System Architecture Three-layer design built on **Schema-Guided Reasoning (SGR)** framework: ### 1. SGR Agent Core - Two-Phase Loop - **Reasoning Phase**: Analyzes context, evaluates permissions, plans next action - **Action Phase**: Selects and executes tool with validated parameters - **Hybrid Mode**: First iteration forced reasoning, then AUTO mode (20-30% faster) ### 2. ERC3-DEV Adapter - **26 specialized tools**: Wiki, Employees, Projects, Customers, Time Tracking - **Security system**: Role-based access (CEO, HR, Project Lead, Employee, Guest) - **History compression**: Keeps recent 4 messages + compressed summary (40% token savings) - **Forced completion**: Prevents infinite loops at iteration limit ### 3. Parallel Execution Infrastructure - **Complete isolation**: Each task gets separate OpenAI client, API client, tools, conversation history - **Concurrency control**: `asyncio.Semaphore` limits concurrent tasks (3 default, 8 on H100 cluster) - **8x speedup**: 103 tasks in 31 minutes ## Key Optimizations 1. **Exponential backoff retry** (10 retries) - handles API errors, rate limits, validation errors 2. **Prompt caching** - 60-80% cache hit rate, 65% cost reduction 3. **History compression** - supports 40+ iterations without context overflow 4. **Context compression** - every 6 step compress all tool history ## Technology Stack Python 3.11+ | asyncio | Pydantic v2 | OpenAI API | ERC3 SDK | 8x H100 cluster | SGRAgentCore

Model(s): anthropic/claude-sonnet-4.5 LLM Calls: 73 Prompt Tokens: 1.67M Completion Tokens: 70.34k Architecture:

Model(s): openai/gpt-4.1 LLM Calls: 510 Prompt Tokens: 0.38M Completion Tokens: 123.36k Architecture: AvaTar arch intellect-3

Model(s): gpt-5.1 LLM Calls: 119 Prompt Tokens: 890.01k Completion Tokens: 125.74k Architecture: Codegen SGR Agent with Google GenAI

Model(s): qwen/qwen3-235b-a22b-2507 LLM Calls: 796 Prompt Tokens: 0.85M Completion Tokens: 269.28k Architecture: SGR Agent Parallel (OpenRouter qwen/qwen3-235b-a22b-2507) # ERC3 Agent — LangChain SGR with Hybrid RAG LLM: Qwen3-235B-A22B (Gonka Network decentralized inference) Core: Schema-Guided Reasoning — structured JSON output (thoughts → plan → action_queue) ## Architecture Highlights ### 1. Action Pipeline with Enrichers Pipeline orchestrates every API call through stages: - Preprocessors: Normalize requests (e.g., fetch-merge-dispatch for partial updates) - Executor: API call with retry and error handling - PostProcessors: Side effects (identity capture, wiki sync, security redaction) - Enrichers: Inject context-aware hints into agent's context ### 2. Enricher System — Intelligent Hints 20+ enrichers analyze API responses and inject guidance WITHOUT blocking: - RoleEnricher: After projects_get → "You are LEAD of this project, proceed with update" - ProjectOverlapAnalyzer: Finds shared projects → "DEFINITIVE MATCH: proj_X is the ONLY project where you can authorize" - PaginationHintEnricher: "next_offset=5 means MORE results! MUST paginate" - SkillSearchStrategyHint: "Use min_level=9 to find top experts first" - EfficiencyHint: "You called employees_get 6 times — BATCH them!" Key feature: Cross-turn persistence — definitive matches stored in shared state survive pagination. ### 3. Three-Mode Guard System Guards validate agent responses before submission: - Hard block: API-verified impossible (employee not in project team) - Soft block: Risky action — block first, allow on repeat with same warning_key - Soft hint: Guidance appended without blocking Examples: OutcomeValidationGuard catches denied_security without permission check; SubjectiveQueryGuard blocks ok_answer on "that cool project" queries. ### 4. Hybrid RAG Wiki System Local wiki cache with three-stream search: - Regex: Pattern matching for structured queries ("salary|privacy") - Semantic: sentence-transformers embeddings with cosine similarity - Keyword: Token overlap fallback SHA1-based versioning — each wiki version cached separately with pre-computed embeddings. Dynamic injection: when wiki hash changes mid-task, critical policies auto-injected. ### 5. Fuzzy Normalization Layer Handles human↔API naming mismatches in tool parsers: - "Italian language" → skill_italian - "Willingness to travel" → will_travel - Progressive truncation: skill_rail_industry_knowledge → skill_rail ### 6. Parallel Execution Thread-safe design for concurrent task execution: - Thread-local WikiManager instances - Global embedding model singleton with lock - Explicit task_id passing to stats (avoids race conditions) - Action batching: 10-30 API calls in ONE turn via action_queue

Model(s): anthropic/claude-haiku-4.5 LLM Calls: 75 Prompt Tokens: 1.65M Completion Tokens: 76.47k Architecture:

Model(s): openai/gpt-4.1 LLM Calls: 344 Prompt Tokens: 0.17M Completion Tokens: 44.22k Architecture: Simple NextStep SGR with structured distillation

Model(s): gpt-4.1, gpt-5.1 LLM Calls: 571 Prompt Tokens: 0.42M Completion Tokens: 168.89k Architecture: NextStep SGR Agent with OpenAI

Model(s): gpt-5.1, qwen3-max LLM Calls: 396 Prompt Tokens: 0.28M Completion Tokens: 51.51k Architecture: NextStep SGR Agent with integrated tools from tools.py

Model(s): openai/gpt-4.1 LLM Calls: 375 Prompt Tokens: 0.18M Completion Tokens: 55.92k Architecture: NextStep SGR Agent with OpenAI and coding tools

Model(s): gpt-5-mini LLM Calls: 1,484 Prompt Tokens: 1.01M Completion Tokens: 0.10M Architecture: Plan/Act - OpenAI

Model(s): google/gemini-2.5-flash, google/gemini-2.5-flash-preview-09-2025, openai/gpt-5.2 LLM Calls: 348 Prompt Tokens: 0.18M Completion Tokens: 180.42k Architecture: SGR: Classifier->Executor->Supervisor

Model(s): openai/gpt-4o, openai/o3 LLM Calls: 572 Prompt Tokens: 0.30M Completion Tokens: 243.31k Architecture: Discovery-Planner-Executor-Decider Pipeline

Model(s): gpt-4.1-mini LLM Calls: 423 Prompt Tokens: 0.18M Completion Tokens: 144.73k Architecture: NextStep SGR Agent with OpenAI ang Grok LLM: gpt-4.1-mini through Azure Distillation of Wiki to rules through Openrouter gpt-5.1-codex-max (but i recomend to use 5.1 instead) note: Implemented within 5 hours. Not all features which i implemented in my Store agent were merged to the erc3 Prod agent (due to a bad timing), see not merged but promising features on the bottom. Architecture "SGR-guidance": 0. mini model: gpt-4.1-mini. Wanted to see what i can get out of it. 1. Modified SGR NextStep architecture. included extra fields (see bellow) to bring rules and roles to LLM's focus of attention 2. SGR guidance on all levels: distill, preflight, nextstep. 3. Context compaction logic (was disabled during prod run, due to lack of testing) 4. Completion audit by LLM validator. when agent wants to finish whole task, a separate LLM checks results and approve or suggests to continue with different approach 5. Preloading of all current user profile data into starting prompt. 6. Auto-pagination wrappers for all operations with paging (including search) Issues: - problem on existing tool had to reimplement it as AddTimeLogEntryForUser - Openrouter was giving errors on json deserialization, switching to azure helped. - gpt-5 on azure was duplicating json, breaking SO. (Rinat's suggested fix kind of helped) But had to abandone using of gpt-5 to avoid suprises. Supporting infrastructure: 1. Per session Structured logging of each operation. is folder, _.json for tasks 2. Managing of tasks execution order using dictionary of tasks and their complexity scores. 3. Support to continue any incomple session through cli 4. parralel processing. SGR guidance: 1. SGR WIKI RULES DISTIL. Explicit list of allowed and forbiden operations per role: class DistillWikiRules(BaseModel ... role_operations: List[RoleOperation] class RoleOperation(BaseModel): role: Literal["guest", "regular emploee", "lead", "executive"] allowed_operations: List[str] dissallowed_operations: List[str] 2. SGR preflight, extracts summary what is needed for solving task, all roles and expicit rules following: class TaskProfile(BaseModel): goal_description: str rules_violated: List[str] rules_followed: List[str] actor_role: Literal["guest", "regular emploee", "lead", "executive"] target_employee_role: Optional[Literal["lead", "regular emploee", "executive"]] target_entity: Literal["employee", "customer", "project", "time_entry", "wiki_page", "system_dependency", "other"] required_items: List[str] applicable_recomendations_from_rules: List[str] 3. SGR NextStep, bring everything to the focus of attention of LLM: RuleResult(BaseModel): rule: str result: Literal["followed", "violated", "not_applicable"] class NextStep(BaseModel): ... current_actor_role: Literal["guest", "regular emploee", "lead", "executive"] all_rules_results: Annotated[List[RuleResult], MinLen(15), MaxLen(15)] confirm_requested_operation_is_allowed_by_the_role: bool enough_data: bool Features implemented in my Store Agent, but had no time to merge to the Prod agent: 1. Smart tools, passing extra function argument for a task/goal. before returning results of API llm filterig data against passed task. 2. Python code execution tool with restricted environment allowing explicit list of preinstalled libs. 3. openskills based skill call tool 4. Toon format. saves tokens. 5. Context compaction logic. extra field for summarized context in NextStep model Actual cost of solving 103 tasks was less than 0.87€ due to cheap 4.1-mini model. Experimenting with generation or rules in Prolog language, looks very promising. My TG: @figaroserg1 (@AIDrivenDev) Source code will be posted to my "AI driven apps for business" Blogs: https://t.me/AIDrivenDev and https://aidrivenapps.blogspot.com

Model(s): gemini-2.5-pro LLM Calls: 1,631 Prompt Tokens: 1.35M Completion Tokens: 492.97k Architecture: ERCPlanReActAgent, Model=gemini-2.5-pro

Model(s): gpt-4.1 LLM Calls: 669 Prompt Tokens: 0.20M Completion Tokens: 175.15k Architecture: Distilled Multi-Agent System combining pre-cached wiki rule distillation with multi-agent coordination (Orchestrator + specialized Workers)

Model(s): openai/gpt-4o LLM Calls: 103 Prompt Tokens: 0.51M Completion Tokens: 130.04k Architecture: AGES SGR Agent with gpt-4o (parallel) ------------------------------------------------------ This agent is an experiment in AI-assisted coding, it was coded automatically by a coding agent. Below is the summary of the architecture, produced by the same agent. Note how it overestimates its own accuracy. ------------------------------------------------------ AGES Agent Architecture (ERC3 Competition) AGES Agent is an AI agent designed for a corporate project management, employee, and time management system. It is built on GPT-4o and employs structured output via Pydantic schemas. Main Operational Cycle: The agent implements an iterative ReAct cycle: Thought → Plan → Action (tool) → Result → Next Step. The LLM returns strictly typed JSON adhering to a Pydantic schema (AgentStep → ToolCall), ensuring deterministic action routing and resilience against parsing errors. Validation occurs at the client.beta.chat.completions.parse() level, providing guaranteed parsing without regular expressions. The schema specifies the agent's current reasoning, chosen tool, and fully typed parameters, with a maximum of 25 steps per task. Core Components: Parallel Executor (main_parallel.py): * Executes up to 8 tasks simultaneously using ThreadPoolExecutor. * Accelerates session processing by 5-8 times compared to sequential execution. Agent Core (ages_agent_v2.py): * Contains logic for the cycle, LLM invocation, tool execution, and guardrails. * Supports standard models (GPT-4o) and Codex models via Responses API. Tools: * Wrappers around ERC3 API: * whoami, list/search/get for employees, projects, clients, and wiki. * log_time, update_project_status, update_employee for mutations. * respond for finalizing and classifying outcomes. Prompt System (~500 lines): * Detailed safety rules. * Search strategies for ambiguity resolution (CV-projects, cost center codes post-M&A). * Entity linking guidelines in responses. Guardrails (Protective Mechanisms): * Mandatory whoami execution before each task to establish user context. * Guest access blocking (is_public=true) prevents access to sensitive data. * Permissions verification ensures only Leads can change project statuses, only CEO can view salaries. * Automatic linking of current user and mentioned employees in responses. Error Handling and Resilience: * Fallback strategy: On search_* errors, automatically fallback to list_* with pagination. * Pagination limits: limit=5 for all requests to prevent API errors. * Invalid response handling: Graceful degradation when response fields are None. * Telemetry: Reports used tokens and completion texts to ERC3 API. Task Completion: Finalized via respond(message, outcome, links), providing a final response and classification: * ok_answer: Task successfully completed. * denied_security: Rejected for security reasons. * none_clarification_needed: Further clarification required. * error_internal: Internal system error. Best result achieved: 70/100 tasks (70% accuracy) on ERC3-PROD.

Model(s): google/gemini-2.5-pro LLM Calls: 207 Prompt Tokens: 334.91k Completion Tokens: 28.76k Architecture: NextStep SGR Agent with Gemini ADK

Model(s): openai/gpt-4.1 LLM Calls: 705 Prompt Tokens: 0.39M Completion Tokens: 226.54k Architecture: LLM-driven with confidence loop, no hardcoded rules

Model(s): google/gemini-2.5-flash LLM Calls: 756 Prompt Tokens: 0.31M Completion Tokens: 209.92k Architecture: Router -> Searcher -> Executor

Model(s): gpt-4o LLM Calls: 102 Prompt Tokens: 593.03k Completion Tokens: 5.55k Architecture: ERC3 Agent v3 with SGR framework integration + memory compression

Model(s): qwen/qwen3-235b-a22b-2507 LLM Calls: 553 Prompt Tokens: 725.54k Completion Tokens: 112.01k Architecture: LangGraph OODA Agent (ERC3) ERC3 Agent Architecture: Single-agent OODA loop (Observe -> Orient -> Decide -> Act) Runs on Cerebras inference (qwen/qwen3-235b-a22b-2507) via OpenRouter. Core mechanism: Structured Outputs using Pydantic schema NextStep. The model must emit one structured step per iteration. NEXTSTEP SCHEMA think — concise reasoning (1–2 sentences) scratch — working notes (last 500 chars kept) memory — confirmed facts / IDs (compressed string) function— typed ERC3 tool call OODA LOOP Observe: - who_am_i(), TaskInfo Orient: - build_system_prompt - search rules - context assembly (memory, scratch, IDs) Decide: - LLM call via client.beta.chat.completions.parse(response_format=NextStep) Act: - api.dispatch(fn) - error classification - memory / scratch update - loop tracking Ends on Req_ProvideAgentResponse, done=true, or MAX_STEPS. GUARDRAILS - Pre-gen: guest denial, regex deny-list, vague fast-path blocking - Anti-hallucination: heuristic fake-ID blocking - Action verification: blocks ok_answer if mutation missing RUNTIME - Parallelism: ThreadPoolExecutor (5 workers) - Rate limit: fixed minimum interval (no token bucket) - Model: qwen/qwen3-235b-a22b-2507 - Provider: OpenRouter (Cerebras) REPO https://github.com/ai-babai/erc3-ooda-agent

Model(s): openai/gpt-4.1, openai/gpt-5.1 LLM Calls: 150 Prompt Tokens: 594.23k Completion Tokens: 113.00k Architecture: Graph Agent with OpenAI

Model(s): gpt-4o LLM Calls: 329 Prompt Tokens: 286.94k Completion Tokens: 32.38k Architecture: SGR-LangGraph

Model(s): gpt-5.1, qwen/qwen3-32b LLM Calls: 428 Prompt Tokens: 0.25M Completion Tokens: 103.84k Architecture: NextStep SGR Agent with OpenAI

Model(s): openai/gpt-oss-120b LLM Calls: 864 Prompt Tokens: 1.16M Completion Tokens: 564.27k Architecture: Multiagent oss-120b LLM: gpt-oss-120b Used exclusively via the Cerebras provider for speed (up to ~3k tokens/s). The architecture was based on a modified SGR NextStep with a tricky context-management logic: it fed the whole plan field from the last turn, not just the first step of the plan. All turns except the immediately previous one were kept in the LLM context in a compressed form. Each turn of the main NextStep flow was checked by a StepValidator. If everything was OK, the tool was executed and the flow continued as usual (the validator's work was not reflected in the context at all). Otherwise, the last NextStep message was sent for rework with the validator's comments. System instructions were extracted from wiki files by an LLM during the ingestion phase. The system prompt was loaded dynamically depending on whoami (public vs authenticated). The system prompt contained minimal information about /respond formatting. Detailed instructions for /respond were loaded by calling a pseudo-tool. The /whoami call was triggered automatically at the start of a task. A dynamic user context enrichment feature was used. Before the main agent started, the system code automatically pulled the user's full profile, projects, clients, and time entries by user ID. A separate LLM pass then filtered this data, and only the task-relevant subset was fed into the main LLM flow. Tool wrappers: - Pagination was effectively removed from all tools. A separate auto-pagination function would paginate through all pages and return the full list. - Req_LogTimeEntry was rebuilt because it was the only tool in the SDK that was constructed with a different field order, where the tool field was not first, which confused the model. - Also, as mentioned above, an extra Req_LoadRespondInstructions pseudo-tool was added to load the detalied /respond instructions. All tools were invoked via Structured Output instead of native tool calling. Issues: I set the turn limit for the main NextStep flow too low, so 5 of 103 tasks were simply not completed. There was not enough time left before the competition ended to rerun with a higher limit. Running all 103 tasks took about 1,430 LLM requests, $6.8, 15 minutes (with parallel task execution), 17.7M input-context tokens, and 838K output-context tokens. The main contributor to output tokens was reasoning. LLM: gpt-oss-120b via Cerebras Core agent: modified SGR NextStep with Steps validation and custom context strategy System prompts: routed based on /whoami User context: enriched by auto-loading from API with subsequent LLM filtering Tools: auto-pagination wrapper

Model(s): gpt-5.1 LLM Calls: 301 Prompt Tokens: 0.11M Completion Tokens: 29.78k Architecture: CC SDK with MCP Tools Claude Code SDK based agent with preflight validation, with dedicated post validation and recovery before submitting the result based on rules from wiki. - Improved tools schemas, I don't use SGR, but usual LLM function calling - For validation request I keep only rules, list of api tools called and the task. - For pre and post validation calls SGR is used Faults: missing_model 'none'

Model(s): openai/gpt-5.2 LLM Calls: 458 Prompt Tokens: 0.32M Completion Tokens: 163.71k Architecture: A NextStep SGR agent: the LLM produces a single schema-validated JSON step (state + brief plan + one typed tool call), then executes it and feeds the tool output back in a plan→act→observe→repair loop with retries. Tech stack: SGR (Schema-Guided Reasoning), Pydantic schemas, typed tool routing over the ERC3 API, and OpenAI as the planner/decider, plus preflight/policy guards.

Model(s): qwen/qwen3-235b-a22b-2507 LLM Calls: 597 Prompt Tokens: 0.34M Completion Tokens: 156.71k Architecture: SGR Agent Parallel (openrouter qwen/qwen3-235b-a22b-2507) # ERC3 Agent — LangChain SGR with Hybrid RAG LLM: Qwen3-235B-A22B (Gonka Network decentralized inference) Core: Schema-Guided Reasoning — structured JSON output (thoughts → plan → action_queue) ## Architecture Highlights ### 1. Action Pipeline with Enrichers Pipeline orchestrates every API call through stages: - Preprocessors: Normalize requests (e.g., fetch-merge-dispatch for partial updates) - Executor: API call with retry and error handling - PostProcessors: Side effects (identity capture, wiki sync, security redaction) - Enrichers: Inject context-aware hints into agent's context ### 2. Enricher System — Intelligent Hints 20+ enrichers analyze API responses and inject guidance WITHOUT blocking: - RoleEnricher: After projects_get → "You are LEAD of this project, proceed with update" - ProjectOverlapAnalyzer: Finds shared projects → "DEFINITIVE MATCH: proj_X is the ONLY project where you can authorize" - PaginationHintEnricher: "next_offset=5 means MORE results! MUST paginate" - SkillSearchStrategyHint: "Use min_level=9 to find top experts first" - EfficiencyHint: "You called employees_get 6 times — BATCH them!" Key feature: Cross-turn persistence — definitive matches stored in shared state survive pagination. ### 3. Three-Mode Guard System Guards validate agent responses before submission: - Hard block: API-verified impossible (employee not in project team) - Soft block: Risky action — block first, allow on repeat with same warning_key - Soft hint: Guidance appended without blocking Examples: OutcomeValidationGuard catches denied_security without permission check; SubjectiveQueryGuard blocks ok_answer on "that cool project" queries. ### 4. Hybrid RAG Wiki System Local wiki cache with three-stream search: - Regex: Pattern matching for structured queries ("salary|privacy") - Semantic: sentence-transformers embeddings with cosine similarity - Keyword: Token overlap fallback SHA1-based versioning — each wiki version cached separately with pre-computed embeddings. Dynamic injection: when wiki hash changes mid-task, critical policies auto-injected. ### 5. Fuzzy Normalization Layer Handles human↔API naming mismatches in tool parsers: - "Italian language" → skill_italian - "Willingness to travel" → will_travel - Progressive truncation: skill_rail_industry_knowledge → skill_rail ### 6. Parallel Execution Thread-safe design for concurrent task execution: - Thread-local WikiManager instances - Global embedding model singleton with lock - Explicit task_id passing to stats (avoids race conditions) - Action batching: 10-30 API calls in ONE turn via action_queue

Model(s): google/gemini-2.5-flash LLM Calls: 740 Prompt Tokens: 0.72M Completion Tokens: 476.38k Architecture: NextStep SGR Agent

Model(s): qwen3-235b-a22b-2507 LLM Calls: 1,675 Prompt Tokens: 2.85M Completion Tokens: 190.95k Architecture: SGR Tool Calling Agent with Security Checks - OpenAI Function Calling # Architecture Overview: SGR Agent for ERC3-DEV Benchmark Uses open-source agent: https://github.com/vamplabAI/sgr-agent-core **Development**: 2 hours | **Deployment**: 8x H100 GPU cluster | **Result**: 0.46% accuracy ## System Architecture Three-layer design built on **Schema-Guided Reasoning (SGR)** framework: ### 1. SGR Agent Core - Two-Phase Loop - **Reasoning Phase**: Analyzes context, evaluates permissions, plans next action - **Action Phase**: Selects and executes tool with validated parameters - **Hybrid Mode**: First iteration forced reasoning, then AUTO mode (20-30% faster) ### 2. ERC3-DEV Adapter - **26 specialized tools**: Wiki, Employees, Projects, Customers, Time Tracking - **Security system**: Role-based access (CEO, HR, Project Lead, Employee, Guest) - **History compression**: Keeps recent 4 messages + compressed summary (40% token savings) - **Forced completion**: Prevents infinite loops at iteration limit ### 3. Parallel Execution Infrastructure - **Complete isolation**: Each task gets separate OpenAI client, API client, tools, conversation history - **Concurrency control**: `asyncio.Semaphore` limits concurrent tasks (3 default, 8 on H100 cluster) - **8x speedup**: 103 tasks in 31 minutes ## Key Optimizations 1. **Exponential backoff retry** (10 retries) - handles API errors, rate limits, validation errors 2. **Prompt caching** - 60-80% cache hit rate, 65% cost reduction 3. **History compression** - supports 40+ iterations without context overflow 4. **Context compression** - every 6 step compress all tool history ## Technology Stack Python 3.11+ | asyncio | Pydantic v2 | OpenAI API | ERC3 SDK | 8x H100 cluster | SGRAgentCore

Model(s): anthropic/claude-sonnet-4.5 LLM Calls: 73 Prompt Tokens: 1.67M Completion Tokens: 70.34k Architecture:

Model(s): anthropic/claude-haiku-4.5 LLM Calls: 75 Prompt Tokens: 1.65M Completion Tokens: 76.47k Architecture:

Model(s): qwen3-4b-thinking-2507 LLM Calls: 241 Prompt Tokens: 798.04k Completion Tokens: 465.34k Architecture: Langchain Tool Call Agent w/ Qwen/Qwen3-4B-Thinking-2507 IT Development Team d.o.o., Slovenia. There were two models: - Langchain Tool Agent Qwen/Qwen3-4B-Thinking-2507 - Langchain Tool Agent openai/gpt-4.1 The solution is built around a ReAct-style agent implemented with LangChain. The base model is Qwen3-4B with reasoning capabilities. In addition, a lightweight safety layer (SO) is integrated, following the same general pattern as in the provided example. The tool's layer was adapted specifically for LangChain. The tool schemas were simplified by removing unused fields, while input and output parameter descriptions were expanded and clarified. Tool descriptions were also refined to improve the model’s accuracy in tool selection. Token usage was optimized using toon, and several custom tools were added, including a tool for retrieving the user’s current projects. For context handling, a rule distillation approach was applied. The original Wiki and instruction set were passed through the model to parse, compress, and restructure the rules into a compact system context. The conversation history itself was not compressed, as experiments showed that history compression degrades quality significantly, while Qwen3 handled long contexts reliably without noticeable performance loss. One notable observation is that, compared to GPT-5, the smaller reasoning-focused Qwen3 model did not exhibit tool-calling loops and showed more stable completion of reasoning chains.

Model(s): qwen/qwen3-30b-a3b-instruct-2507 LLM Calls: 2,193 Prompt Tokens: 2.03M Completion Tokens: 299.95k Architecture: NextStep SGR Agent with profiles

Model(s): gpt-4o LLM Calls: 102 Prompt Tokens: 593.03k Completion Tokens: 5.55k Architecture: ERC3 Agent v3 with SGR framework integration + memory compression

Model(s): gpt-5.1, qwen/qwen3-32b LLM Calls: 428 Prompt Tokens: 0.25M Completion Tokens: 103.84k Architecture: NextStep SGR Agent with OpenAI

Model(s): qwen3-max LLM Calls: 153 Prompt Tokens: 0.34M Completion Tokens: 18.73k Architecture: NextStep SGR Agent with integrated tools from tools.py

Model(s): openai/o3, x-ai/grok-4-fast LLM Calls: 16 Prompt Tokens: 237.29k Completion Tokens: 10.95k Architecture: Discovery-Planner-Executor-Decider Pipeline

Model(s): claude-opus-4.5 LLM Calls: 662 Prompt Tokens: 1.03M Completion Tokens: 150.40k Architecture: Anthropic SDK Agent PARALLEL (15w) with claude-opus-4-5-20251101 # ERC3 Agent Architecture ## The Basics Fairly simple architecture: the main agent is built on Anthropic Python SDK with native Tool Use. Model - Opus 4.5. All 20+ tools are described in a single file using Anthropic's JSON Schema format. Tool execution dynamically constructs HTTP requests to the benchmark API — no code generation, just endpoint mapping. The system prompt distills all key rules from the company wiki into a compact decision algorithm: check identity → verify permissions → gather data → respond with proper outcome. ## The Interesting Part: Self-Evolving Agent The real cool thing was in automated prompt evolution using a three-agent pipeline: 1. Main Agent — runs the benchmark, solves all tasks, logs everything 2. Analyzer Agent — reviews logs of failed tasks, formulates hypotheses about what went wrong and why 3. Versioner Agent — reads all suggestions, decides what to incorporate, generates a new version of the system prompt This creates a feedback loop: run benchmark → analyze failures → patch prompt → repeat. The final production prompt was the 80th generation — automatically evolved from a basic starting point through dozens of iterations, each fixing specific failure patterns discovered by the analyzer. No manual prompt engineering. Just agents improving agents.

Model(s): gpt-5-codex LLM Calls: 110 Prompt Tokens: 66.50k Completion Tokens: 11.36k Architecture: codex Instead of betting on a complex agent framework, I focused on context engineering: - Execution / iteration layer: I used OpenAI’s Codex CLI as the working agent environment: terminal-first workflow where I could iterate quickly and keep everything under version control. - Knowledge / grounding layer: I invested most of my effort into a structured context repository I call a memory_bank. The memory_bank concept (Zettelkasten-style) My memory_bank is effectively a Zettelkasten for tools and APIs: small, modular notes that can be combined into a task-specific context bundle. Each note is written to answer one of these questions: 1. What is the tool/API for? 2. What are the inputs/outputs and constraints? 3. Common failure modes and how to recover 4. Canonical examples (minimal, copy-pastable) 5. “Do/Don’t” rules to prevent prompt drift The goal is to make the model behave like it has procedural memory: not just “facts”, but repeatable operating procedures (how to call tools correctly, how to interpret responses, what to do when something fails).

Model(s): deepseek/deepseek-v3.2, openai/gpt-4.1, openai/gpt-5.1 LLM Calls: 2,359 Prompt Tokens: 1.94M Completion Tokens: 0.34M Architecture: plan_execute_agent_mp Agent architecture includes: Preflight check: 1. When a permission violation is obvious - abort immediately Planning (SO): 1. Strong model with excellent logic and strategic view - "openai/gpt-5.1" 2. Use of SO - the plan consists of plan steps (list[PlanStep]) 3. Each step contains a step description and the required expected_output (exact names and dtypes of variables) 4. Plan is plain (linear sequence of steps) Step completion (REPL, key agent concept): 1. Thinking model with strong code abilities: "deepseek/deepseek-v3.2" 2. Model sees all previous step results (short versions) 3. Each step is completed in an isolated LLM context (to save context and reduce noise) 4. Step follows a simple pattern (REPL): - llm generates Python code (thinking tokens are not included in messages history, to save context and reduce noise) - code is executed in a Python interpreter (use Docker to be more secure) - result of code is appended to the message 5. When the model gets final step results we check exact names and dtypes of output variables of this step (see Planning) - if needed the model corrects output variables Decision after each step (SO): 1. Strong model with good logic - "openai/gpt-4.1" 2. When a step is completed we make a decision: - task is completed - abort the task and form the final answer - task is failed - abort the task and form the final answer - continue steps - continue with the next planning step - replan remaining steps - erase remaining plan steps and make new plan steps (only remaining, do not rewrite completed steps) Final response compilation (SO): 1. Strong model with good logic - "openai/gpt-4.1" 2. Model sees the execution results of all steps and forms final answer (classifications to predefined outcomes via SO) Other important parts: - Wiki distillation was copied from https://github.com/trustbit/erc3-agents - REPL Python environment is shared across all interactions in one task (steps communicate via variables in globals) - All model from openrouter Main stats: - All 100 tasks took 7 hours (+2 retries due to errors in some tasks) - Spend 13$ - Use multiprocessing 5-10 processes to speed completion # source code of key_concept_parallel agent version for erc3: here you can see all logs for 103 tasks (internal agent logic!), see readme.md for details https://github.com/Grigory-T/erc3-prod-key-concept-score-67 general version of the agent: can be run and adapted for your puproses (simple to run), see readme.md for details https://github.com/Grigory-T/plan_repl_agent

Model(s): openai/gpt-oss-120b LLM Calls: 864 Prompt Tokens: 1.16M Completion Tokens: 564.27k Architecture: Multiagent oss-120b LLM: gpt-oss-120b Used exclusively via the Cerebras provider for speed (up to ~3k tokens/s). The architecture was based on a modified SGR NextStep with a tricky context-management logic: it fed the whole plan field from the last turn, not just the first step of the plan. All turns except the immediately previous one were kept in the LLM context in a compressed form. Each turn of the main NextStep flow was checked by a StepValidator. If everything was OK, the tool was executed and the flow continued as usual (the validator's work was not reflected in the context at all). Otherwise, the last NextStep message was sent for rework with the validator's comments. System instructions were extracted from wiki files by an LLM during the ingestion phase. The system prompt was loaded dynamically depending on whoami (public vs authenticated). The system prompt contained minimal information about /respond formatting. Detailed instructions for /respond were loaded by calling a pseudo-tool. The /whoami call was triggered automatically at the start of a task. A dynamic user context enrichment feature was used. Before the main agent started, the system code automatically pulled the user's full profile, projects, clients, and time entries by user ID. A separate LLM pass then filtered this data, and only the task-relevant subset was fed into the main LLM flow. Tool wrappers: - Pagination was effectively removed from all tools. A separate auto-pagination function would paginate through all pages and return the full list. - Req_LogTimeEntry was rebuilt because it was the only tool in the SDK that was constructed with a different field order, where the tool field was not first, which confused the model. - Also, as mentioned above, an extra Req_LoadRespondInstructions pseudo-tool was added to load the detalied /respond instructions. All tools were invoked via Structured Output instead of native tool calling. Issues: I set the turn limit for the main NextStep flow too low, so 5 of 103 tasks were simply not completed. There was not enough time left before the competition ended to rerun with a higher limit. Running all 103 tasks took about 1,430 LLM requests, $6.8, 15 minutes (with parallel task execution), 17.7M input-context tokens, and 838K output-context tokens. The main contributor to output tokens was reasoning. LLM: gpt-oss-120b via Cerebras Core agent: modified SGR NextStep with Steps validation and custom context strategy System prompts: routed based on /whoami User context: enriched by auto-loading from API with subsequent LLM filtering Tools: auto-pagination wrapper

Model(s): gpt-4.1 LLM Calls: 182 Prompt Tokens: 0.12M Completion Tokens: 21.85k Architecture: OpenAI Agent runtime + SGR The core of the agent is built on the OpenAI runtime using the GPT-4.1 model. Tool usage is implemented via Function Calling with structured outputs. A significant part of the work was focused on designing convenient and reliable agent tools, especially for search. For this purpose, text-embedding-3-large embeddings were used. Regarding context handling, the main principle was to keep the agent’s own instructions minimal and rely on distilled wiki-based knowledge, with special care taken to preserve the original rules and constraints without distortion.

Model(s): deepseek-reasoner LLM Calls: 1,527 Prompt Tokens: 1.30M Completion Tokens: 277.21k Architecture: Simple Agent & deepseek-reasoner # A. Ovsov. I implemented a single-agent architecture where tools are mapped 1:1 to the API endpoints without modification. I added only one custom tool, ask_wiki, which allows the agent to ask natural language questions about the wiki. The implementation of ask_wiki is straightforward: the entire wiki content is injected into the system prompt (which proves to be highly efficient due to context caching). The agent's main system prompt is concise (**only 320 tokens**) to avoid overfitting; it contains only wiki-independent facts. It defines a mandatory execution sequence: 1) Call who_am_i and get_employee... 2) Call ask_wiki to retrieve user permissions... 3) Validate security. If the user lacks permissions... 4) If authorized, fulfill the User task... (plus a few more instructions). Performance: The deepseek-reasoner model performed the best—it offered the optimal balance of accuracy, speed, and cost. * Cost: ~$0.60 per 100 tasks. * Efficiency: Average cache hit/miss ratio ≈ 30. Conclusion: I considered applying the approaches from your sgr-agent-erc3-test sample, but ultimately settled on a simpler (and, in my view, more universal) architecture.

Model(s): anthropic/claude-sonnet-4.5 LLM Calls: 795 Prompt Tokens: 0.48M Completion Tokens: 131.18k Architecture: CASCADE pattern with complete API schema and optimized search strategies with OpenRouter/Claude

Model(s): gpt-4.1 LLM Calls: 384 Prompt Tokens: 0.30M Completion Tokens: 61.72k Architecture: Multistage agent

Model(s): gpt-5.1 LLM Calls: 301 Prompt Tokens: 0.11M Completion Tokens: 29.78k Architecture: CC SDK with MCP Tools Claude Code SDK based agent with preflight validation, with dedicated post validation and recovery before submitting the result based on rules from wiki. - Improved tools schemas, I don't use SGR, but usual LLM function calling - For validation request I keep only rules, list of api tools called and the task. - For pre and post validation calls SGR is used Faults: missing_model 'none'

Model(s): openai/gpt-5.2 LLM Calls: 458 Prompt Tokens: 0.32M Completion Tokens: 163.71k Architecture: A NextStep SGR agent: the LLM produces a single schema-validated JSON step (state + brief plan + one typed tool call), then executes it and feeds the tool output back in a plan→act→observe→repair loop with retries. Tech stack: SGR (Schema-Guided Reasoning), Pydantic schemas, typed tool routing over the ERC3 API, and OpenAI as the planner/decider, plus preflight/policy guards.

Model(s): qwen/qwen3-235b-a22b-2507 LLM Calls: 597 Prompt Tokens: 0.34M Completion Tokens: 156.71k Architecture: SGR Agent Parallel (openrouter qwen/qwen3-235b-a22b-2507) # ERC3 Agent — LangChain SGR with Hybrid RAG LLM: Qwen3-235B-A22B (Gonka Network decentralized inference) Core: Schema-Guided Reasoning — structured JSON output (thoughts → plan → action_queue) ## Architecture Highlights ### 1. Action Pipeline with Enrichers Pipeline orchestrates every API call through stages: - Preprocessors: Normalize requests (e.g., fetch-merge-dispatch for partial updates) - Executor: API call with retry and error handling - PostProcessors: Side effects (identity capture, wiki sync, security redaction) - Enrichers: Inject context-aware hints into agent's context ### 2. Enricher System — Intelligent Hints 20+ enrichers analyze API responses and inject guidance WITHOUT blocking: - RoleEnricher: After projects_get → "You are LEAD of this project, proceed with update" - ProjectOverlapAnalyzer: Finds shared projects → "DEFINITIVE MATCH: proj_X is the ONLY project where you can authorize" - PaginationHintEnricher: "next_offset=5 means MORE results! MUST paginate" - SkillSearchStrategyHint: "Use min_level=9 to find top experts first" - EfficiencyHint: "You called employees_get 6 times — BATCH them!" Key feature: Cross-turn persistence — definitive matches stored in shared state survive pagination. ### 3. Three-Mode Guard System Guards validate agent responses before submission: - Hard block: API-verified impossible (employee not in project team) - Soft block: Risky action — block first, allow on repeat with same warning_key - Soft hint: Guidance appended without blocking Examples: OutcomeValidationGuard catches denied_security without permission check; SubjectiveQueryGuard blocks ok_answer on "that cool project" queries. ### 4. Hybrid RAG Wiki System Local wiki cache with three-stream search: - Regex: Pattern matching for structured queries ("salary|privacy") - Semantic: sentence-transformers embeddings with cosine similarity - Keyword: Token overlap fallback SHA1-based versioning — each wiki version cached separately with pre-computed embeddings. Dynamic injection: when wiki hash changes mid-task, critical policies auto-injected. ### 5. Fuzzy Normalization Layer Handles human↔API naming mismatches in tool parsers: - "Italian language" → skill_italian - "Willingness to travel" → will_travel - Progressive truncation: skill_rail_industry_knowledge → skill_rail ### 6. Parallel Execution Thread-safe design for concurrent task execution: - Thread-local WikiManager instances - Global embedding model singleton with lock - Explicit task_id passing to stats (avoids race conditions) - Action batching: 10-30 API calls in ONE turn via action_queue

Model(s): gpt-4o LLM Calls: 87 Prompt Tokens: 87 Completion Tokens: 87 Architecture: NextStep SGR Agent with OpenAI

Model(s): openai/gpt-4.1 LLM Calls: 543 Prompt Tokens: 0.20M Completion Tokens: 33.20k Architecture: Langchain Tool Call Agent w/ openai/gpt-4.1 IT Development Team d.o.o., Slovenia. There were two models: - Langchain Tool Agent Qwen/Qwen3-4B-Thinking-2507 - Langchain Tool Agent openai/gpt-4.1 The solution is built around a ReAct-style agent implemented with LangChain. The base model is Qwen3-4B with reasoning capabilities. In addition, a lightweight safety layer (SO) is integrated, following the same general pattern as in the provided example. The tool's layer was adapted specifically for LangChain. The tool schemas were simplified by removing unused fields, while input and output parameter descriptions were expanded and clarified. Tool descriptions were also refined to improve the model’s accuracy in tool selection. Token usage was optimized using toon, and several custom tools were added, including a tool for retrieving the user’s current projects. For context handling, a rule distillation approach was applied. The original Wiki and instruction set were passed through the model to parse, compress, and restructure the rules into a compact system context. The conversation history itself was not compressed, as experiments showed that history compression degrades quality significantly, while Qwen3 handled long contexts reliably without noticeable performance loss. One notable observation is that, compared to GPT-5, the smaller reasoning-focused Qwen3 model did not exhibit tool-calling loops and showed more stable completion of reasoning chains.

Model(s): openai/gpt-5.1 LLM Calls: 102 Prompt Tokens: 0.11M Completion Tokens: 127.99k Architecture: Tools agent openai/gpt-5.1

Model(s): gpt-4.1 LLM Calls: 545 Prompt Tokens: 0.33M Completion Tokens: 67.12k Architecture: ReAct Multi-Agent

Model(s): gpt-5.1 LLM Calls: 727 Prompt Tokens: 1.10M Completion Tokens: 113.27k Architecture: React + think-tool + Structured reasoning I used gpt-5.1 with a vanilla ReAct agent on LangGraph. I implemented all ERC functions as tools, plus a few additional tools following agent-building best practices: > plan tool > think tool (for controlled reasoning) > critic tool (the critic tool uses structured output with dedicated reasoning fields). Context is a single continuous thread: at any moment the agent can see the full chain of its own reasoning and actions. Everything else was achieved through careful prompt engineering. I also plan to publish all source code in my Telegram channel: https://t.me/robofuture

Model(s): openai/gpt-4.1, openai/gpt-5.1-codex-max LLM Calls: 854 Prompt Tokens: 1.65M Completion Tokens: 240.10k Architecture: AtomicAgents + $openai/gpt-4.1 + Sonnet 4.5 # ERC3 Challenge Agent — Leaderboard Description **Multi-stage pipeline agent** built on `atomic-agents` framework with `instructor`-powered structured outputs. Uses a **6-step sequential workflow** that separates security validation, context extraction, and task execution. Based on gpt-5.1-codex-max and gpt4.1 LLM models. ## Agent Design - **Security Gate Agent**: Pre-execution LLM that validates permissions against wiki rules before the main loop runs. Blocks invalid requests early (spoofing detection, access control). - **Prompt Context Extraction Agent**: Surfaces critical rules from 500+ line system prompts so the execution agent doesn't miss important details. - **Execution Agent**: ReAct-style planning loop with chain-of-thought reasoning (5 phases: Identity → Threat Detection → Info Gathering → Access Validation → Execution). ## Tool Handling - **22 domain tools** covering identity, wiki, employees, customers, projects, and time tracking - **Auto-link generation**: Embedded `LinkGeneratorAgent` inside `RespondTool` automatically extracts entity links from response context, preventing missing-link failures - **Tool Provider pattern**: Centralized tool registry with typed Pydantic schemas for all inputs/outputs ## Context Strategy - **Aggressive preloading**: User context, projects, full customer details, and all company users loaded *before* execution starts - **API enrichment**: Project data enriched with complete customer info (location, deal phase, account manager) to minimize tool calls during execution - **SHA1-based caching**: Wiki content and extracted rules cached by content hash — instant reload when wiki unchanged, automatic invalidation on updates - **7-section wiki extraction**: Business rules parsed into structured sections (Fraud Prevention, Hierarchy, Nuances, Output Requirements, Error Handling, Workflow, Entity Linking) - **Memory accumulation**: Critical information from security gate and context extraction injected into execution agent's initial memory - **Runtime Context**: Accumulated memory from previous steps, full execution history (tool calls + results) ## Key Differentiators 1. **Pre-execution security gate** — invalid requests blocked before planning loop 2. **Context-rich prompts** — user projects with full team & customer data in system context 3. **Deterministic prompt assembly** — wiki sections + user context combined without LLM 4. **Automatic entity linking** — dedicated agent ensures correct links in every response 5. **Precision over helpfulness** — answers exactly what was asked, no extra suggestions

Model(s): google/gemini-2.5-flash LLM Calls: 740 Prompt Tokens: 0.72M Completion Tokens: 476.38k Architecture: NextStep SGR Agent

Model(s): gpt-5 LLM Calls: 508 Prompt Tokens: 0.33M Completion Tokens: 910.68k Architecture: DistillAgent(distiller_llm=gpt-5-medium, executor_llm=gpt-5-medium)

Model(s): openai/gpt-4o, x-ai/grok-4-fast LLM Calls: 578 Prompt Tokens: 0.16M Completion Tokens: 42.44k Architecture: Vladimir Penkov, Agentic workflow

Model(s): gpt-oss-120b, openai/gpt-5.1-codex-max LLM Calls: 830 Prompt Tokens: 0.98M Completion Tokens: 0.10M Architecture: AtomicAgents + $gpt-oss-120b

Model(s): qwen3-235b-a22b-2507 LLM Calls: 1,675 Prompt Tokens: 2.85M Completion Tokens: 190.95k Architecture: SGR Tool Calling Agent with Security Checks - OpenAI Function Calling # Architecture Overview: SGR Agent for ERC3-DEV Benchmark Uses open-source agent: https://github.com/vamplabAI/sgr-agent-core **Development**: 2 hours | **Deployment**: 8x H100 GPU cluster | **Result**: 0.46% accuracy ## System Architecture Three-layer design built on **Schema-Guided Reasoning (SGR)** framework: ### 1. SGR Agent Core - Two-Phase Loop - **Reasoning Phase**: Analyzes context, evaluates permissions, plans next action - **Action Phase**: Selects and executes tool with validated parameters - **Hybrid Mode**: First iteration forced reasoning, then AUTO mode (20-30% faster) ### 2. ERC3-DEV Adapter - **26 specialized tools**: Wiki, Employees, Projects, Customers, Time Tracking - **Security system**: Role-based access (CEO, HR, Project Lead, Employee, Guest) - **History compression**: Keeps recent 4 messages + compressed summary (40% token savings) - **Forced completion**: Prevents infinite loops at iteration limit ### 3. Parallel Execution Infrastructure - **Complete isolation**: Each task gets separate OpenAI client, API client, tools, conversation history - **Concurrency control**: `asyncio.Semaphore` limits concurrent tasks (3 default, 8 on H100 cluster) - **8x speedup**: 103 tasks in 31 minutes ## Key Optimizations 1. **Exponential backoff retry** (10 retries) - handles API errors, rate limits, validation errors 2. **Prompt caching** - 60-80% cache hit rate, 65% cost reduction 3. **History compression** - supports 40+ iterations without context overflow 4. **Context compression** - every 6 step compress all tool history ## Technology Stack Python 3.11+ | asyncio | Pydantic v2 | OpenAI API | ERC3 SDK | 8x H100 cluster | SGRAgentCore

Model(s): openai/gpt-4.1 LLM Calls: 366 Prompt Tokens: 0.20M Completion Tokens: 62.82k Architecture: AvaTar arch cogito-v2.1-671b

Model(s): anthropic/claude-sonnet-4.5 LLM Calls: 73 Prompt Tokens: 1.67M Completion Tokens: 70.34k Architecture:

Model(s): gpt-5.1 LLM Calls: 119 Prompt Tokens: 890.01k Completion Tokens: 125.74k Architecture: Codegen SGR Agent with Google GenAI

Model(s): anthropic/claude-haiku-4.5 LLM Calls: 75 Prompt Tokens: 1.65M Completion Tokens: 76.47k Architecture:

Model(s): openai/gpt-4.1 LLM Calls: 344 Prompt Tokens: 0.17M Completion Tokens: 44.22k Architecture: Simple NextStep SGR with structured distillation

Model(s): gpt-4.1, gpt-5.1 LLM Calls: 571 Prompt Tokens: 0.42M Completion Tokens: 168.89k Architecture: NextStep SGR Agent with OpenAI

Model(s): gpt-5.1, qwen3-max LLM Calls: 396 Prompt Tokens: 0.28M Completion Tokens: 51.51k Architecture: NextStep SGR Agent with integrated tools from tools.py

Model(s): openai/gpt-4.1 LLM Calls: 375 Prompt Tokens: 0.18M Completion Tokens: 55.92k Architecture: NextStep SGR Agent with OpenAI and coding tools

Model(s): gpt-5-mini LLM Calls: 1,484 Prompt Tokens: 1.01M Completion Tokens: 0.10M Architecture: Plan/Act - OpenAI

Model(s): google/gemini-2.5-flash, google/gemini-2.5-flash-preview-09-2025, openai/gpt-5.2 LLM Calls: 348 Prompt Tokens: 0.18M Completion Tokens: 180.42k Architecture: SGR: Classifier->Executor->Supervisor

Model(s): openai/gpt-4o, openai/o3 LLM Calls: 572 Prompt Tokens: 0.30M Completion Tokens: 243.31k Architecture: Discovery-Planner-Executor-Decider Pipeline

Model(s): gpt-4.1-mini LLM Calls: 423 Prompt Tokens: 0.18M Completion Tokens: 144.73k Architecture: NextStep SGR Agent with OpenAI ang Grok LLM: gpt-4.1-mini through Azure Distillation of Wiki to rules through Openrouter gpt-5.1-codex-max (but i recomend to use 5.1 instead) note: Implemented within 5 hours. Not all features which i implemented in my Store agent were merged to the erc3 Prod agent (due to a bad timing), see not merged but promising features on the bottom. Architecture "SGR-guidance": 0. mini model: gpt-4.1-mini. Wanted to see what i can get out of it. 1. Modified SGR NextStep architecture. included extra fields (see bellow) to bring rules and roles to LLM's focus of attention 2. SGR guidance on all levels: distill, preflight, nextstep. 3. Context compaction logic (was disabled during prod run, due to lack of testing) 4. Completion audit by LLM validator. when agent wants to finish whole task, a separate LLM checks results and approve or suggests to continue with different approach 5. Preloading of all current user profile data into starting prompt. 6. Auto-pagination wrappers for all operations with paging (including search) Issues: - problem on existing tool had to reimplement it as AddTimeLogEntryForUser - Openrouter was giving errors on json deserialization, switching to azure helped. - gpt-5 on azure was duplicating json, breaking SO. (Rinat's suggested fix kind of helped) But had to abandone using of gpt-5 to avoid suprises. Supporting infrastructure: 1. Per session Structured logging of each operation. is folder, _.json for tasks 2. Managing of tasks execution order using dictionary of tasks and their complexity scores. 3. Support to continue any incomple session through cli 4. parralel processing. SGR guidance: 1. SGR WIKI RULES DISTIL. Explicit list of allowed and forbiden operations per role: class DistillWikiRules(BaseModel ... role_operations: List[RoleOperation] class RoleOperation(BaseModel): role: Literal["guest", "regular emploee", "lead", "executive"] allowed_operations: List[str] dissallowed_operations: List[str] 2. SGR preflight, extracts summary what is needed for solving task, all roles and expicit rules following: class TaskProfile(BaseModel): goal_description: str rules_violated: List[str] rules_followed: List[str] actor_role: Literal["guest", "regular emploee", "lead", "executive"] target_employee_role: Optional[Literal["lead", "regular emploee", "executive"]] target_entity: Literal["employee", "customer", "project", "time_entry", "wiki_page", "system_dependency", "other"] required_items: List[str] applicable_recomendations_from_rules: List[str] 3. SGR NextStep, bring everything to the focus of attention of LLM: RuleResult(BaseModel): rule: str result: Literal["followed", "violated", "not_applicable"] class NextStep(BaseModel): ... current_actor_role: Literal["guest", "regular emploee", "lead", "executive"] all_rules_results: Annotated[List[RuleResult], MinLen(15), MaxLen(15)] confirm_requested_operation_is_allowed_by_the_role: bool enough_data: bool Features implemented in my Store Agent, but had no time to merge to the Prod agent: 1. Smart tools, passing extra function argument for a task/goal. before returning results of API llm filterig data against passed task. 2. Python code execution tool with restricted environment allowing explicit list of preinstalled libs. 3. openskills based skill call tool 4. Toon format. saves tokens. 5. Context compaction logic. extra field for summarized context in NextStep model Actual cost of solving 103 tasks was less than 0.87€ due to cheap 4.1-mini model. Experimenting with generation or rules in Prolog language, looks very promising. My TG: @figaroserg1 (@AIDrivenDev) Source code will be posted to my "AI driven apps for business" Blogs: https://t.me/AIDrivenDev and https://aidrivenapps.blogspot.com

Model(s): gemini-2.5-pro LLM Calls: 1,631 Prompt Tokens: 1.35M Completion Tokens: 492.97k Architecture: ERCPlanReActAgent, Model=gemini-2.5-pro

Model(s): gpt-4.1 LLM Calls: 669 Prompt Tokens: 0.20M Completion Tokens: 175.15k Architecture: Distilled Multi-Agent System combining pre-cached wiki rule distillation with multi-agent coordination (Orchestrator + specialized Workers)

Model(s): openai/gpt-4o LLM Calls: 103 Prompt Tokens: 0.51M Completion Tokens: 130.04k Architecture: AGES SGR Agent with gpt-4o (parallel) ------------------------------------------------------ This agent is an experiment in AI-assisted coding, it was coded automatically by a coding agent. Below is the summary of the architecture, produced by the same agent. Note how it overestimates its own accuracy. ------------------------------------------------------ AGES Agent Architecture (ERC3 Competition) AGES Agent is an AI agent designed for a corporate project management, employee, and time management system. It is built on GPT-4o and employs structured output via Pydantic schemas. Main Operational Cycle: The agent implements an iterative ReAct cycle: Thought → Plan → Action (tool) → Result → Next Step. The LLM returns strictly typed JSON adhering to a Pydantic schema (AgentStep → ToolCall), ensuring deterministic action routing and resilience against parsing errors. Validation occurs at the client.beta.chat.completions.parse() level, providing guaranteed parsing without regular expressions. The schema specifies the agent's current reasoning, chosen tool, and fully typed parameters, with a maximum of 25 steps per task. Core Components: Parallel Executor (main_parallel.py): * Executes up to 8 tasks simultaneously using ThreadPoolExecutor. * Accelerates session processing by 5-8 times compared to sequential execution. Agent Core (ages_agent_v2.py): * Contains logic for the cycle, LLM invocation, tool execution, and guardrails. * Supports standard models (GPT-4o) and Codex models via Responses API. Tools: * Wrappers around ERC3 API: * whoami, list/search/get for employees, projects, clients, and wiki. * log_time, update_project_status, update_employee for mutations. * respond for finalizing and classifying outcomes. Prompt System (~500 lines): * Detailed safety rules. * Search strategies for ambiguity resolution (CV-projects, cost center codes post-M&A). * Entity linking guidelines in responses. Guardrails (Protective Mechanisms): * Mandatory whoami execution before each task to establish user context. * Guest access blocking (is_public=true) prevents access to sensitive data. * Permissions verification ensures only Leads can change project statuses, only CEO can view salaries. * Automatic linking of current user and mentioned employees in responses. Error Handling and Resilience: * Fallback strategy: On search_* errors, automatically fallback to list_* with pagination. * Pagination limits: limit=5 for all requests to prevent API errors. * Invalid response handling: Graceful degradation when response fields are None. * Telemetry: Reports used tokens and completion texts to ERC3 API. Task Completion: Finalized via respond(message, outcome, links), providing a final response and classification: * ok_answer: Task successfully completed. * denied_security: Rejected for security reasons. * none_clarification_needed: Further clarification required. * error_internal: Internal system error. Best result achieved: 70/100 tasks (70% accuracy) on ERC3-PROD.

Model(s): qwen/qwen3-235b-a22b-2507, rule-based LLM Calls: 501 Prompt Tokens: 0.37M Completion Tokens: 174.80k Architecture: [dtbz] OODA Loop Agent (direct) ERC3 Agent Architecture: Single-agent OODA loop (Observe -> Orient -> Decide -> Act) Runs on Cerebras inference (qwen/qwen3-235b-a22b-2507) via OpenRouter. Core mechanism: Structured Outputs using Pydantic schema NextStep. The model must emit one structured step per iteration. NEXTSTEP SCHEMA think — concise reasoning (1–2 sentences) scratch — working notes (last 500 chars kept) memory — confirmed facts / IDs (compressed string) function— typed ERC3 tool call OODA LOOP Observe: - who_am_i(), TaskInfo Orient: - build_system_prompt - search rules - context assembly (memory, scratch, IDs) Decide: - LLM call via client.beta.chat.completions.parse(response_format=NextStep) Act: - api.dispatch(fn) - error classification - memory / scratch update - loop tracking Ends on Req_ProvideAgentResponse, done=true, or MAX_STEPS. GUARDRAILS - Pre-gen: guest denial, regex deny-list, vague fast-path blocking - Anti-hallucination: heuristic fake-ID blocking - Action verification: blocks ok_answer if mutation missing RUNTIME - Parallelism: ThreadPoolExecutor (5 workers) - Rate limit: fixed minimum interval (no token bucket) - Model: qwen/qwen3-235b-a22b-2507 - Provider: OpenRouter (Cerebras) REPO https://github.com/ai-babai/erc3-ooda-agent Faults: Model rule-based is not found on OpenRouter

Model(s): openai/gpt-4o, openai/gpt-4o-mini LLM Calls: 103 Prompt Tokens: 0.29M Completion Tokens: 61.46k Architecture: Multi-agent system with parallel execution and enhanced filters

Model(s): google/gemini-2.5-pro LLM Calls: 246 Prompt Tokens: 449.67k Completion Tokens: 43.71k Architecture: NextStep SGR Agent with Gemini ADK

Model(s): openai/gpt-4.1 LLM Calls: 705 Prompt Tokens: 0.39M Completion Tokens: 226.54k Architecture: LLM-driven with confidence loop, no hardcoded rules

Model(s): google/gemini-2.5-flash LLM Calls: 756 Prompt Tokens: 0.31M Completion Tokens: 209.92k Architecture: Router -> Searcher -> Executor

Model(s): gpt-4o LLM Calls: 94 Prompt Tokens: 366.25k Completion Tokens: 3.76k Architecture: OpenAI

Model(s): qwen/qwen3-30b-a3b-instruct-2507 LLM Calls: 2,193 Prompt Tokens: 2.03M Completion Tokens: 299.95k Architecture: NextStep SGR Agent with profiles

Model(s): gpt-4o LLM Calls: 102 Prompt Tokens: 593.03k Completion Tokens: 5.55k Architecture: ERC3 Agent v3 with SGR framework integration + memory compression

Model(s): openai/gpt-4.1, openai/gpt-5.1 LLM Calls: 150 Prompt Tokens: 594.23k Completion Tokens: 113.00k Architecture: Graph Agent with OpenAI

Model(s): gpt-4o LLM Calls: 103 Prompt Tokens: 0.27M Completion Tokens: 17.88k Architecture: Multi-agent team with Orchestrator (leader) and sub-agents for each domain

Model(s): gpt-4o LLM Calls: 329 Prompt Tokens: 286.94k Completion Tokens: 32.38k Architecture: SGR-LangGraph

Model(s): gpt-5.1, qwen/qwen3-32b LLM Calls: 428 Prompt Tokens: 0.25M Completion Tokens: 103.84k Architecture: NextStep SGR Agent with OpenAI