Testing Interchangeability in LLM Agent Teams

The study examines whether members of two-agent LLM teams remain interchangeable when their roles, underlying model, and general experience are held comparable, and whether changing partner identity affects task outcomes, coordination efficiency, and recovery after rotation. The evaluated setting uses persistent private text notebooks, role division, and the Collab-Overcooked and Hanabi tasks.

The central measurement challenge is to separate three effects that can otherwise appear together: disturbance caused by changing personnel, loss of general task experience, and mismatch with coordination conventions formed with a particular partner. The study also treats task performance and coordination efficiency as distinct outcomes, and asks whether they recover at the same rate after a member exchange.

The target is therefore an operational assessment of interchangeability within the specified two-agent collaboration setting, rather than a general claim about all teams, memory designs, protocols, or task environments.

The core proposal is a controlled “swap test.” Agents first form separate teams while retaining persistent private notes. A role-matched member and that member’s notes are then exchanged, while model, role, and broadly comparable experience are held fixed. The design routes several comparison conditions to different counterfactual purposes: intact teams provide the formed-team reference; placebo controls the disturbance of personnel change; swap measures the effect of partner replacement; cleared and amnesia conditions probe the role of partner-related information; and naive provides a new-member experience baseline.

The framework represents general experience with (V), partner-specific residual value with (\Pi), and their relative ratio with (\rho=\Pi/V). Related measures are applied to coordination cost, including communication required per unit of progress. The same comparison logic is extended to seat-level effects, message categories, protocol signatures, and repeated post-swap episodes. This lets the study examine not only whether performance changes, but also where additional coordination occurs and how task performance and communication efficiency recover over time.

The authors interpret the residual coordination cost as evidence that teams may form partner-specific coordination conventions. That interpretation remains an explanatory claim rather than a uniquely established mechanism: the notebook categorization is operational, and removing or clearing notes can also change the amount or content of context available to the agent.

Within the tested scope, the evaluation reports a separation between task outcomes and coordination efficiency. Member exchange has a relatively small effect on task scores, while communication cost per unit of progress increases by about 16%–63%. In Hanabi, the coordination cost after exchange is higher than for an inexperienced replacement baseline. The reported effects are larger in more highly coupled task settings, but coupling is accompanied by task-complexity differences, so the comparison cannot by itself establish a causal effect of coupling strength.

The reported changes are not distributed uniformly across the team. Losses concentrate in the seat that initiates the plan, additional communication is produced mainly by the member who remains, and the post-exchange protocol signature moves toward that of the donating team. These observations provide descriptive localization of the coordination burden and support the study’s distinction between task success and coordination cost.

Recovery is monitored across ten post-exchange rounds. The evaluation reports that task scores recover earlier than coordination efficiency, using changes relative to a placebo baseline and a 1% recovery threshold. Thus, a team can regain task performance while still paying elevated communication costs. The frozen progress assessment treats the swap, counterfactual controls, outcome measures, cost measures, and recovery tracking as a complete executable assessment path for the stated scope; this does not establish broad deployment effectiveness or transfer beyond that scope.

The strongest unresolved issue concerns mechanism attribution. The observed swap residual is consistent with outdated partner models or mismatched partner-specific conventions, but it may also reflect differences in task-note content, team-specific execution state, or context length. Clearing notes is therefore not a pure intervention on partner conventions if it simultaneously changes the available text and shortens the context. Protocol signatures measure observable surface behavior and cannot by themselves rule out teams that share a surface protocol while using different internal strategies.

The findings are limited to the tested two-agent, role-divided settings with persistent private text notebooks, including Collab-Overcooked and Hanabi. The materials do not determine whether the same results hold for teams of three or more, formation histories longer than 20 rounds, systems without partner notes, fixed communication protocols, different memory designs, or other task families. The coupled-task comparison also cannot be read as an isolated causal test of coupling.

Further evidence could use length-matched neutral text, context-preserving note replacement, controlled movement of partner information, and direct tests relating note changes to coordination costs. More repetitions and longer recovery observation would also strengthen validation. These open questions constrain the mechanism explanation and generalization; they do not remove the reported within-scope distinction between task-result recovery and coordination-cost recovery.