Embodied agents empowered by large language models (LLMs) have recently demonstrated remarkable success in executing object rearrangement tasks in household environments. The primary objective of embodied agents is to provide assistance to users while interacting with the physical world. But do such tasks truly reflect the challenges in providing meaningful assistance to the users?

To provide personalized assistance, embodied agents must understand the unique semantics that users assign to the physical world (e.g., favorite cup, breakfast routine) by leveraging prior interaction histories to interpret dynamic, real-world instructions. Therefore, understanding embodied agents' memory utilization capabilities is crucial for developing personalized embodied agents.

We construct Memento, an end-to-end two-stage personalized embodied agent evaluation framework comprising both single-memory and joint-memory tasks to evaluate task performance and quantify how memory utilization affects performance on personalized assistance tasks.

Research Questions: We address the following three key questions:

• RQ1: Can current LLM-powered embodied agents effectively utilize memory to perform personalized assistance tasks?
• RQ2: What are the key factors that negatively affect embodied agents' memory utilization capabilities?
• RQ3: How can we design memory architectures to better support personalized assistance tasks?

(1) Memory Acquisition Stage: Agents perform conventional object rearrangement tasks with instructions that contain sufficient personalized knowledge. These episodes are defined as ϵ_acq = (S, I_acq, g) where ϕ(I_acq) → g, meaning agents can directly infer the goal from instructions. During this stage, agents establish a reference performance baseline while accumulating episodic memory h_acq from their interactions.

(2) Memory Utilization Stage: Agents perform identical tasks but use underspecified instructions ϵ_util = (S, I_util, g). These tasks can only be completed when agents recall and apply relevant personalized knowledge, requiring the extended grounding function ϕ(I_util, h_acq) → g.

Through this evaluation process, we are able to quantify the agent's memory utilization capability by comparing performance between the two stages.

(1) Single-memory tasks: Require information from one episodic memory. These tasks test the agent's ability to retrieve and apply knowledge from a single previous interaction.

(2) Joint-memory tasks: Necessitate synthesis from two distinct memories, formulated as ϵ_util^joint = (S, I_util^joint, [gⁱ;g^j]) where i, j denote corresponding reference episodes from the acquisition stage. These tasks evaluate the agent's capability to integrate information from multiple past experiences.

We conduct experiments without memory usage to validate our framework design, confirming that agents struggle significantly with underspecified instructions when memory is not available.

We categorize personalized knowledge into two types to evaluate different reasoning challenges:

(1) Object Semantics: Tests agents' ability to identify objects based on personal meaning (e.g., my cup, my favorite running gear).

(2) User Patterns: Assesses agents' capacity to reconstruct goals using behavioral patterns (e.g., my remote work setup, my cozy dinner atmosphere).

LLM-powered embodied agents struggle to utilize personalized knowledge effectively. Even GPT-4o shows a 30.5% drop in success rate for joint-memory tasks, with all models experiencing over 20% performance decline. The increased planning cycles and simulation steps indicate misinterpretation of instructions and inefficient exploration.

Agents perform well on object semantics tasks but struggle significantly with user patterns, indicating limitations in understanding sequential behavioral information.

Agents tend to rely on commonsense knowledge rather than personalized memory, leading to inconsistent performance when commonsense doesn't align with user preferences.

As the number of retrieved memories increases, performance degrades across all models, indicating that agents struggle with information overload and increasingly rely on commonsense knowledge rather than personalized memories.

Joint-memory tasks show significantly larger performance drops than single-memory tasks, with GPT-4o declining by 30.5%. Agents struggle to coordinate multiple memories, highlighting the need for improved memory architectures.

Building upon our finding that episodic memory is essential, we devise a hierarchical knowledge graph-based user profile memory module that manages personalized knowledge independently to provide cleaner, more accessible information to the agent.

Design Challenge: Organizing knowledge relationships effectively is crucial. Sequential knowledge like user patterns requires sophisticated memory structure to prevent information loss or failing to adapt to evolving knowledge.

Solution: We construct personalized knowledge as a hierarchical knowledge graph with three levels:

• Top level: Users
• Middle level: Knowledge types (object semantics and user patterns)
• Bottom level: Specific elements (objects, patterns, locations)

The graph connects these levels using hierarchical edges for structural relationships and temporal edges for sequential ordering within user patterns, enabling systematic representation and easy updates of personalized knowledge.

Key Finding: User profile memory significantly enhances embodied agents' memory utilization capability. Agents achieve substantial performance improvements across all models for both single-memory and joint-memory tasks, with especially notable gains on user pattern tasks where sequential structure previously posed challenges.