AI Agent Memory Explained: What Agents Should Remember

In 30 seconds

What it is Agents should remember progress and preferences, never secrets or bank details.

You use it when You are designing an agent that works across several sessions and need rules for what it stores and for how long.

Next in this section Conversational vs Autonomous AI Agents: Which One Fits the Job? →

Memory should hold progress and preferences, with a retention date. Bank details and secrets stay in the system of record.

Part 14 of 26 in SAP Business AI, Explained.

An agent with no memory asks the same questions every time you come back. One that remembers everything becomes a privacy and audit problem. Good design sits in between: the right few things, kept for the right length of time.

I think of agent memory as a small, governed notebook of progress and preferences. It shouldn’t become a second copy of your business records.

What an agent should remember

Short-term memory is the current conversation. It lets the agent follow a multi-step exchange, and it can be logged for audit.
Long-term memory is what the agent carries into the next session: facts, decisions, summaries and preferences that save people from repeating themselves.
The real work is in the rules: what may be written, what never may, who sees it and when it’s deleted.

The everyday picture

Think of a good HR officer who looks after new joiners. During a meeting they take notes (short-term memory). Afterwards they keep a short file card for each joiner: “Laptop ordered, policy signed, prefers Arabic, induction on Monday” (long-term memory).

The joiner’s bank account and passport numbers never go on that card. They sit in the personnel file, locked in the HR system. The card points to the file without copying it.

How memory moves through a session

SAP describes SAP Agent Memory service as a persistent memory layer for agents, built on SAP HANA Cloud. It stores two kinds of record. Messages are chronological conversation logs grouped by session, and memories are durable entries you can search by meaning across sessions (SAP Community, Agent Memory). Another SAP post adds a third need, workflow checkpoints that capture the agent’s state at decision points (SAP Community, Memory Meets Orchestration).

The session runs. Each turn is kept as short-term memory so the agent can follow the thread.
At useful moments the agent proposes a short memory, such as a step completed, a preference stated, or a decision with its reason.
A write rule filters it. Your design decides what’s allowed. Bank details, identity numbers and passwords are blocked, while references to records get through.
In the next session, the agent searches memories by meaning and pulls back the few that matter instead of replaying the whole history.
Retention deletes memories when their purpose ends, and people can see, correct or delete what’s held about them.

Noor’s first week at NextSAP

NextSAP Utilities is the fictional Dubai multi-utility (electricity, water, gas and district cooling) used across this site’s SAP content. Its new-joiner onboarding agent starts when a hire is confirmed in SAP SuccessFactors and works at L3, meaning it acts, with approval for anything that changes a record. Mariam, the HR director, expects about 45 joiners a month across the UAE and India.

Noor, a new field technician at a Dubai depot, starts on Monday 16 November. Over three sessions the agent records “PPE kit and handheld meter-reading device requested, 9 Nov”, “Code of conduct acknowledged, 10 Nov”, “Prefers Arabic for documents” and “4 of 11 onboarding tasks complete”. When she comes back on 12 November, it picks up at task 5 without asking again.

In session two she types her IBAN (international bank account number) into the chat and asks whether it’s correct. The agent doesn’t store it. It sends her to the secure payroll form in SuccessFactors, where the record has its own access rules, and the session log is masked for that turn. All the memory holds is “Bank details submitted via payroll form, 10 Nov.”

Her onboarding memories expire 90 days after her start date. Daniel from Internal Audit can still trace what happened, because the system of record and the masked logs sit under their own retention policy.

In the pilot, new joiners answer about 30% fewer repeated questions, and Hana, the AI governance lead, can show exactly which data the agent may keep.

What goes where

Kind of information	Short-term (session)	Long-term memory	System of record only
The current back-and-forth	Yes	No	No
Progress on a multi-step task	Yes	Yes, with a retention date	Status may also live there
Stated preferences (language, format)	Yes	Yes, user can edit or delete	Optional
Decisions and their reasons	Yes	Yes, as a short summary	Yes, if they change a record
Bank details, passwords, identity numbers	Masked	Never	Yes
Salary, health, disciplinary details	Masked	Never	Yes
Copies of business documents	No	No, store a reference instead	Yes

Where a notebook stops helping

Good memory saves people from repeating themselves, keeps prompts short and leaves auditors a trail of what was said and decided.

It doesn’t replace the system of record, though. If memory and the record disagree, the record wins. If I had to pin one design rule above the desk, it would be this: remember progress and preferences, never secrets, and let the system of record hold the truth. Memory isn’t safe by default either. Write rules and retention are your design work, and people keep their rights over data held about them.

Explain it to your CFO

The helper keeps a short notebook so staff don’t repeat themselves. It isn’t a copy of our files, and bank details never go into it.
Every note has an expiry date, and people can see and delete what’s held about them.

Settle these before go-live

What is each agent allowed to remember? I start with an allow-list of progress, preferences and decision summaries, and block everything else.
How long do memories live? Until the purpose ends plus a short grace period, such as 90 days after onboarding completes.
Who can see an agent’s memory? The user it’s about, the agent’s business owner for support, and auditors through a logged request.

Go deeper

Technical detail for architects and developers

The SAP Agent Memory post says all data is scoped to an agent and invoker pair, so memories are isolated per agent and per user. It describes an OData (Open Data Protocol) REST API (application programming interface) secured with OAuth 2.0 client credentials, an optional Python client in the SAP Cloud SDK for Python, configurable retention, and audit logging (SAP Community, Agent Memory). Embeddings are generated in SAP HANA Cloud and retrieved by vector similarity.

On availability, the post announces the service as generally available on a bring-your-own SAP HANA Cloud model, with a fully managed offering described as planned. It describes native Joule Studio integration as “planned towards 2027”. The Memory Meets Orchestration post also mentions an open-source checkpoint saver for LangGraph that persists agent state through the service (SAP Community, Memory Meets Orchestration). Check the current status in the Joule capability watch.

Company-level context is a different thing. SAP News describes SAP Company Memory as a preview (SAP News). Don’t confuse shared organisational context with a single agent’s per-user memory. They need different owners and access rules.

The OWASP Top 10 for LLM Applications 2025 lists Sensitive Information Disclosure (LLM02) and Vector and Embedding Weaknesses (LLM08) (OWASP). For memory, I translate that into controls: filter writes so sensitive data never gets in, scope reads so one user’s memories never surface for another, and treat recalled memories as context rather than instructions so a poisoned memory can’t steer actions.

When agents hand work to each other, don’t pass memory wholesale. The A2A (Agent2Agent) specification frames agents as sharing declared capabilities and exchanged information rather than internal state, and lists retention policies and deletion requests among its data privacy practices (A2A specification). Send only what the receiving agent needs for the task.

For the write rule itself, I recommend a small classifier or pattern check before every memory write. It blocks account numbers, identity numbers and credentials, requires a retention tag, and logs the write with the session identifier.

Words you’ll hear

Short-term memory: the conversation in the current session, kept so the agent can follow along.
Long-term memory: facts, decisions and preferences an agent keeps across sessions.
Semantic search: finding stored items by meaning rather than exact words.
Checkpoint: a saved snapshot of where an agent is in a task, so it can resume.
Retention policy: the rule for how long data is kept before deletion.
System of record: the business application that holds the official data, such as SAP SuccessFactors for employee records.
Embedding: a numeric representation of text that lets software compare meanings.

Keep going

Independent explainer, not affiliated with or endorsed by SAP SE. SAP, Joule and other SAP product names are trademarks of SAP SE.

Frequently asked questions

What is the difference between short-term and long-term agent memory?

Short-term memory is the conversation in the current session. Long-term memory is a small set of facts, decisions and preferences the agent keeps across sessions and looks up by meaning when relevant.

Is agent memory the same as the business record?

No. The system of record, such as the employee record or the invoice, holds the truth. Memory holds working notes about progress and preferences, and should point to the record rather than copy it.

What should an AI agent never remember?

Secrets and highly sensitive personal data: bank details, passwords, identity document numbers, salary figures and health information. Those belong in the system of record with its own controls.

Can a user see or delete what an agent remembers about them?

SAP's post on SAP Agent Memory service says users can inspect, edit or delete their memories, and that retention is configurable. Design your agent and process so that this is actually offered.