arxiv:2610.09872

LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

Published on Oct 7

· Submitted by

Leiming Fu on Oct 19

Fudan University

Upvote

116

Authors:

Leiming Fu ,

Abstract

LiveMACEBench evaluates persistent LLM agents in live financial markets using decision traces to reveal gaps between outcomes and underlying capabilities.

Generated by thinkingmachines/Inkling-Small

Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Five frontier LLMs operate along continuous trajectories under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. We evaluate them through both realized outcomes and mechanism-specific diagnostics derived from complete decision traces. Across 30 days of live evaluation, we find a pronounced outcome-capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use. Trace-level diagnostics further expose distinct bottlenecks across capabilities, demonstrating that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability. LiveMACEBench makes this distinction measurable, turning live markets from a performance leaderboard into a diagnostic environment for agent capability

View arXiv page View PDF GitHub 9 Add to collection

Community

raymondfu1453

Paper author Paper submitter 1 day ago

We introduce LiveMACE, a benchmark that uses live financial markets to evaluate LLM agent capabilities. Static benchmark questions can enter training data; future market events have yet to occur. This makes evolving markets a natural environment for testing how agents act under genuinely new conditions.

However, realized returns alone do not characterize an agent’s capabilities: they reflect both evolving market conditions and multiple aspects of agent behavior. LiveMACE combines simulated trading in cryptocurrency and US-equity markets with process-aware evaluation of four capabilities: tool use, persistent memory, rule following, and multi-agent collaboration. It records agents’ decisions and interactions, then evaluates the traces behind their outcomes.

Across five frontier LLMs and 30 days of live evaluation, we find an outcome–capability gap: higher returns do not consistently imply stronger capabilities, and similar returns can arise from different patterns of mechanism use. Our goal is to make live markets useful as an evaluation environment for agents, with measurements that go beyond the return leaderboard.

The framework is open source, and we welcome feedback on evaluating persistent agents in evolving environments.

💻 Source Code

researchstudio-bot

about 24 hours ago

This is an automated message from the ResearchStudio team.

We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.