Implemented conversation memory and loading of historical conversations

[Disviel]

Close #43
Close #44

This pull request introduces persistent conversation history to the actor chat system and improves the backend infrastructure for handling conversation messages and debugging. It adds a new API for retrieving conversation messages, updates the actor and SSE endpoints to support conversation context, and implements message storage and retrieval using the database. Additionally, it introduces index creation methods for all major MongoDB-backed databases to improve query performance and adds a debug endpoint for inspecting database contents.

Conversation message persistence and retrieval:

• Added a new API endpoint at api/conversations/messages to fetch conversation messages, with support for pagination and input validation. (packages/ema-ui/src/app/api/conversations/messages/route.ts)
• Updated the actor input and SSE endpoints to require and handle conversationId, ensuring messages are associated with the correct conversation. (packages/ema-ui/src/app/api/actor/input/route.ts, packages/ema-ui/src/app/api/actor/sse/route.ts, packages/ema-ui/src/app/chat/page.tsx) [1] [2] [3] [4] [5] [6] [7] [8] [9]
• Refactored ActorWorker to persist messages to the database instead of in-memory, and to retrieve conversation history from the database. (packages/ema/src/actor.ts) [1] [2] [3] [4] [5] [6]

Database enhancements and infrastructure:

• Added createIndices methods to all major MongoDB-backed database classes to allow programmatic creation of indices for improved performance. (packages/ema/src/db/base.ts, packages/ema/src/db/mongo.actor.ts, packages/ema/src/db/mongo.conversation.ts, packages/ema/src/db/mongo.conversation_message.ts, packages/ema/src/db/mongo.long_term_memory.ts, packages/ema/src/db/mongo.short_term_memory.ts) [1] [2] [3] [4] [5] [6]
• Enhanced listConversationMessages to support sorting and limiting results, enabling efficient pagination and ordering. (packages/ema/src/db/base.ts, packages/ema/src/db/mongo.conversation_message.ts) [1] [2]

Debugging and utility endpoints:

• Added a debug API endpoint to inspect MongoDB collections and their contents, facilitating easier development and debugging. (packages/ema-ui/src/app/api/debug/db/route.ts)

Other improvements:

• Fixed asynchronous call to server.login() in the user login endpoint. (packages/ema-ui/src/app/api/users/login/route.ts)

[Copilot]

Pull request overview

This PR implements persistent conversation history for the EverMemoryArchive system, enabling historical conversations to be loaded and maintained across sessions. It introduces database-backed message storage, comprehensive index creation for improved query performance, and a new API for retrieving conversation history.

Changes:

• Refactored ActorWorker to persist and retrieve conversation messages from MongoDB instead of in-memory storage
• Added createIndices() methods to all MongoDB-backed database classes for improved query performance
• Introduced new /api/conversations/messages endpoint for retrieving historical conversation messages with pagination support
• Updated actor endpoints to require conversationId parameter, ensuring proper conversation context tracking

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/ema/src/actor.ts	Refactored to use database-backed message persistence and added conversationId/userName/actorName parameters
packages/ema/src/server.ts	Updated actor caching to use composite key (userId:actorId:conversationId) and added bulk index creation
packages/ema/src/db/base.ts	Added IndexableDB interface and extended ListConversationMessagesRequest with sort/limit parameters
packages/ema/src/db/mongo.*.ts	Added createIndices() methods to all MongoDB database classes
packages/ema/src/db/mongo.conversation_message.ts	Enhanced listConversationMessages with sorting and limit support
packages/ema/src/memory/utils.ts	Fixed bufferMessageFromEma to use reply.response instead of stringifying entire reply object
packages/ema/src/tests/skills/memory.spec.ts	Updated test to include new ActorWorker constructor parameters
packages/ema/src/server.spec.ts	Updated login test to await async login() call
packages/ema-ui/src/app/api/conversations/messages/route.ts	New endpoint for retrieving conversation messages with validation and pagination
packages/ema-ui/src/app/api/actor/input/route.ts	Added conversationId parameter to request schema and getActor call
packages/ema-ui/src/app/api/actor/sse/route.ts	Added conversationId parameter to request schema and getActor call
packages/ema-ui/src/app/api/users/login/route.ts	Fixed to await async login() call
packages/ema-ui/src/app/api/debug/db/route.ts	New debug endpoint for inspecting MongoDB collections
packages/ema-ui/src/app/chat/page.tsx	Added conversation history loading on mount and updated SSE connection with conversationId

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 10 comments.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (2)

packages/ema-ui/src/app/api/actor/sse/route.ts:77

• This endpoint lacks authentication and authorization checks. Any user can subscribe to actor events for any conversation by providing arbitrary userId, actorId, and conversationId values, potentially allowing unauthorized access to private conversation streams. Add authentication to verify the requesting user has permission to access the specified conversation.

export const GET = getQuery(ActorSseRequest)(async (query) => {
  const server = await getServer();
  const actor = await server.getActor(
    Number.parseInt(query.userId),
    Number.parseInt(query.actorId),
    Number.parseInt(query.conversationId),
  );
  const encoder = new TextEncoder();
  /* The handle to unsubscribe from the actor events. */
  let eventCallback: (event: ActorAgentEvent) => void;

  const customReadable = new ReadableStream({
    start(controller) {
      eventCallback = (event) => {
        if (event.kind !== "emaReplyReceived") {
          return;
        }
        controller.enqueue(
          encoder.encode(`data: ${JSON.stringify(event)}\n\n`),
        );
      };
      actor.events.on("agent", eventCallback);
    },
    cancel() {
      if (eventCallback) {
        actor.events.off("agent", eventCallback);
      }
    },
  });

  return new Response(customReadable, {
    headers: {
      Connection: "keep-alive",
      "Content-Encoding": "none",
      "Cache-Control": "no-cache, no-transform",
      "Content-Type": "text/event-stream; charset=utf-8",
    },
  });
});

packages/ema-ui/src/app/api/actor/input/route.ts:67

• This endpoint lacks authentication and authorization checks. Any user can send inputs to any actor by providing arbitrary userId and actorId values, potentially allowing unauthorized access to conversations. Add authentication to verify the requesting user matches the userId in the request body.

export const POST = postBody(ActorInputRequest)(async (body) => {
  const server = await getServer();
  const actor = await server.getActor(
    body.userId,
    body.actorId,
    body.conversationId,
  );

  // Processes input.
  await actor.work(body.inputs);

  return new Response(JSON.stringify({ success: true }), {
    status: 200,
    headers: { "Content-Type": "application/json" },
  });
});