feat: Add progress reporting for long-running operations

library/README.md

@@ -4,7 +4,7 @@ This repository shares tools developed by [Jigsaw](http://jigsaw.google.com) as
 
 # **Overview**
 
-Effectively understanding large-scale public input is a significant challenge, as traditional methods struggle to translate thousands of diverse opinions into actionable insights. ‘Sensemaker’ showcases how Google's Gemini models can be used to transform massive volumes of raw community feedback into clear, digestible insights, aiding the analysis of these complex discussions.
+Effectively understanding large-scale public input is a significant challenge, as traditional methods struggle to translate thousands of diverse opinions into actionable insights. 'Sensemaker' showcases how Google's Gemini models can be used to transform massive volumes of raw community feedback into clear, digestible insights, aiding the analysis of these complex discussions.
 
 The tools demonstrated here illustrate methods for:
 
@@ -18,7 +18,7 @@ Please see these [docs](https://jigsaw-code.github.io/sensemaking-tools/docs/) f
 
 # Our Approach
 
-The tools here show how Jigsaw is approaching the application of AI and Google’s Gemini to the emerging field of ‘sensemaking’. It is offered as an insight into our experimental methods. While parts of this library may be adaptable for other projects, developers should anticipate their own work for implementation, customization, and ongoing support for their specific use case.
+The tools here show how Jigsaw is approaching the application of AI and Google's Gemini to the emerging field of 'sensemaking'. It is offered as an insight into our experimental methods. While parts of this library may be adaptable for other projects, developers should anticipate their own work for implementation, customization, and ongoing support for their specific use case.
 
 # **How It Works**
 
@@ -30,7 +30,7 @@ Sensemaker provides an option to identify the topics present in the comments. Th
 * Both top-level and subtopics  
 * Sub-topics only, given a set of pre-specified top-level topics
 
-Topic identification code can be found in [library/src/tasks/topic\_modeling.ts](https://github.com/Jigsaw-Code/sensemaking-tools/blob/main/library/src/tasks/topic_modeling.ts).
+Topic identification code can be found in [library/src/tasks/topic_modeling.ts](https://github.com/Jigsaw-Code/sensemaking-tools/blob/main/library/src/tasks/topic_modeling.ts).
 
 ## **Statement Categorization**
 
@@ -52,11 +52,11 @@ Includes a short bullet list of the number of statements, votes, topics and subt
 
 ### **Overview Section**
 
-The overview section summarizes the "Themes" sections for all subtopics, along with summaries generated for each top-level topic (these summaries are generated as an intermediate step, but not shown to users, and can be thought of as intermediate “chain of thought” steps in the overall recursive summarization approach).
+The overview section summarizes the "Themes" sections for all subtopics, along with summaries generated for each top-level topic (these summaries are generated as an intermediate step, but not shown to users, and can be thought of as intermediate "chain of thought" steps in the overall recursive summarization approach).
 
 Currently the Overview does not reference the "Common Ground" and "Differences of Opinion" sections described below.
 
-Percentages in the overview (e.g. “Arts and Culture (17%)”) are the percentage of statements that are about this topic. Since statements can be categorized into multiple topics these percentages add up to a number greater than 100%.
+Percentages in the overview (e.g. "Arts and Culture (17%)") are the percentage of statements that are about this topic. Since statements can be categorized into multiple topics these percentages add up to a number greater than 100%.
 
 ### **Top 5 Subtopics**
 
@@ -71,7 +71,7 @@ For each subtopic, Sensemaker surfaces:
 * The number of statements assigned to this subtopic.  
 * Prominent themes.  
 * A summary of the top statements where we find "common ground" and "differences of opinion", based on agree and disagree rates.  
-* The relative level of agreement within the subtopic, as compared to the average subtopic, based on how many comments end up in “common ground” vs “differences of opinion” buckets.
+* The relative level of agreement within the subtopic, as compared to the average subtopic, based on how many comments end up in "common ground" vs "differences of opinion" buckets.
 
 #### **Themes**
 
@@ -89,13 +89,13 @@ For this section, Sensemaker provides grounding citations to show which statemen
 
 #### **Relative Agreement**
 
-Each subtopic is labeled as “high”, “moderately high”, “moderately low” or “low” agreement. This is determined by, for each subtopic, getting *all* the comments that qualify as common ground comments and normalizing it based on how many comments were in that subtopic. Then these numbers are compared subtopic to subtopic.
+Each subtopic is labeled as "high", "moderately high", "moderately low" or "low" agreement. This is determined by, for each subtopic, getting *all* the comments that qualify as common ground comments and normalizing it based on how many comments were in that subtopic. Then these numbers are compared subtopic to subtopic.
 
 ### **LLMs Used and Custom Models**
 
-This library is implemented using Google Cloud’s [VertexAI](https://cloud.google.com/vertex-ai), and works with the latest Gemini models. The access and quota requirements are controlled by a user’s Google Cloud account.
+This library is implemented using Google Cloud's [VertexAI](https://cloud.google.com/vertex-ai), and works with the latest Gemini models. The access and quota requirements are controlled by a user's Google Cloud account.
 
-In addition to Gemini models available through VertexAI, users can integrate custom models using the library’s `Model` abstraction. This can be done by implementing a class with only two methods, one for generating plain text and one for generating structured data ([docs](https://jigsaw-code.github.io/sensemaking-tools/docs/classes/models_model.Model.html) for methods). This allows for the library to be used with models other than Gemini, with other cloud providers, and even with on-premise infrastructure for complete data sovereignty.
+In addition to Gemini models available through VertexAI, users can integrate custom models using the library's `Model` abstraction. This can be done by implementing a class with only two methods, one for generating plain text and one for generating structured data ([docs](https://jigsaw-code.github.io/sensemaking-tools/docs/classes/models_model.Model.html) for methods). This allows for the library to be used with models other than Gemini, with other cloud providers, and even with on-premise infrastructure for complete data sovereignty.
 
 Please note that performance results for existing functionality may vary depending on the model selected.
 
@@ -132,7 +132,7 @@ Then to log in locally run:
 
 ## **Example Usage \- Javascript**
 
-Summarize Seattle’s $15 Minimum Wage Conversation.
+Summarize Seattle's $15 Minimum Wage Conversation.
 
 ```javascript
 // Set up the tools to use the default Vertex model (Gemini Pro 1.5) and related authentication info.
@@ -143,6 +143,11 @@ const mySensemaker = new Sensemaker({
   ),
 });
 
+// Optional: Set up progress reporting
+mySensemaker.setProgressCallback((report) => {
+  console.log(`${report.operation}: ${report.message} (${report.percentage}%)`);
+});
+
 // Note: this function does not exist.
 // Get data from a discussion in Seattle over a $15 minimum wage.
 // CSV containing comment text, vote counts, and group information from:
@@ -176,8 +181,8 @@ CLI Usage
 There is also a simple CLI set up for testing. There are three  tools:
 
 * [./library/runner-cli/runner.ts](https://github.com/Jigsaw-Code/sensemaking-tools/blob/main/library/runner-cli/runner.ts): takes in a CSV representing a conversation and outputs an HTML file containing the summary. The summary is best viewed as an HTML file so that the included citations can be hovered over to see the original comment and votes.  
-* [./library/runner-cli/categorization\_runner.ts](https://github.com/Jigsaw-Code/sensemaking-tools/blob/main/library/runner-cli/categorization_runner.ts): takes in a CSV representing a conversation and outputs another CSV with the comments categorized into topics and subtopics.  
-* [./library/runner-cli/advanced\_runner.ts](https://github.com/Jigsaw-Code/sensemaking-tools/blob/main/library/runner-cli/advanced_runner.ts): takes in a CSV representing a conversation and outputs three files for an advanced user more interested in the statistics. The first is a JSON of topics, their sizes, and their subtopics. The second is a JSON with all of the comments and their alignment scores and values. Third is the summary object as a JSON which can be used for additional processing.
+* [./library/runner-cli/categorization_runner.ts](https://github.com/Jigsaw-Code/sensemaking-tools/blob/main/library/runner-cli/categorization_runner.ts): takes in a CSV representing a conversation and outputs another CSV with the comments categorized into topics and subtopics.  
+* [./library/runner-cli/advanced_runner.ts](https://github.com/Jigsaw-Code/sensemaking-tools/blob/main/library/runner-cli/advanced_runner.ts): takes in a CSV representing a conversation and outputs three files for an advanced user more interested in the statistics. The first is a JSON of topics, their sizes, and their subtopics. The second is a JSON with all of the comments and their alignment scores and values. Third is the summary object as a JSON which can be used for additional processing.
 
 These tools process CSV input files.  These must contain the columns `comment_text` and `comment-id`.  For deliberations without group information, vote counts should be set in columns titled `agrees`, `disagrees` and `passes`.  If you do not have vote information, these can be set to 0. For deliberations with group breakdowns, you can set the columns `{group_name}-agree-count`, `{group_name}-disagree-count`, `{group_name}-pass-count`.
 

library/src/sensemaker.test.ts

@@ -16,6 +16,7 @@ import { Sensemaker } from "./sensemaker";
 import { Comment } from "./types";
 import { VertexModel } from "./models/vertex_model";
 import { ModelSettings } from "./models/model";
+import { ProgressReport } from "./types";
 
 // mock retry timeout
 jest.mock("./models/model_util", () => {
@@ -235,4 +236,108 @@ describe("SensemakerTest", () => {
       expect(commentRecords).toEqual(validResponse);
     });
   });
+
+  describe("ProgressReportingTest", () => {
+    it("should report progress during topic identification", async () => {
+      const progressReports: ProgressReport[] = [];
+      const sensemaker = new Sensemaker(TEST_MODEL_SETTINGS);
+      sensemaker.setProgressCallback((report) => progressReports.push(report));
+
+      const comments: Comment[] = [
+        { id: "1", text: "Comment about Roads" },
+        { id: "2", text: "Another comment about Roads" },
+      ];
+
+      mockGenerateData
+        .mockReturnValueOnce(
+          Promise.resolve([
+            {
+              name: "Infrastructure",
+              subtopics: [{ name: "Roads" }],
+            },
+          ])
+        )
+        .mockReturnValueOnce(
+          Promise.resolve([
+            { id: "1", text: "Comment about Roads", topics: [{ name: "Infrastructure" }] },
+            { id: "2", text: "Another comment about Roads", topics: [{ name: "Infrastructure" }] },
+          ])
+        );
+
+      await sensemaker.learnTopics(comments, true);
+
+      expect(progressReports).toHaveLength(3);
+      expect(progressReports[0]).toEqual({
+        operation: "topic_identification",
+        currentStep: 1,
+        totalSteps: 3,
+        message: "Starting topic identification...",
+        percentage: 0,
+      });
+      expect(progressReports[1]).toEqual({
+        operation: "topic_identification",
+        currentStep: 2,
+        totalSteps: 3,
+        message: "Identifying subtopics...",
+        percentage: 33,
+      });
+      expect(progressReports[2]).toEqual({
+        operation: "topic_identification",
+        currentStep: 3,
+        totalSteps: 3,
+        message: "Topic identification complete",
+        percentage: 100,
+      });
+    });
+
+    it("should report progress during statement categorization", async () => {
+      const progressReports: ProgressReport[] = [];
+      const sensemaker = new Sensemaker(TEST_MODEL_SETTINGS);
+      sensemaker.setProgressCallback((report) => progressReports.push(report));
+
+      const comments: Comment[] = Array.from({ length: 30 }, (_, i) => ({
+        id: `${i}`,
+        text: `Comment ${i}`,
+      }));
+      const topics = [{ name: "Topic 1" }];
+
+      mockGenerateData
+        .mockReturnValueOnce(
+          Promise.resolve(
+            comments.slice(0, 10).map((comment) => ({
+              ...comment,
+              topics: [{ name: "Topic 1" }],
+            }))
+          )
+        )
+        .mockReturnValueOnce(
+          Promise.resolve(
+            comments.slice(10, 20).map((comment) => ({
+              ...comment,
+              topics: [{ name: "Topic 1" }],
+            }))
+          )
+        )
+        .mockReturnValueOnce(
+          Promise.resolve(
+            comments.slice(20).map((comment) => ({
+              ...comment,
+              topics: [{ name: "Topic 1" }],
+            }))
+          )
+        );
+
+      await sensemaker.categorizeComments(comments, false, topics);
+
+      expect(progressReports.length).toBeGreaterThan(0);
+      expect(progressReports[0]).toEqual({
+        operation: "statement_categorization",
+        currentStep: 0,
+        totalSteps: 3,
+        message: "Starting statement categorization...",
+        percentage: 0,
+      });
+      expect(progressReports[progressReports.length - 1].percentage).toBe(100);
+    });
+  });
 });

library/src/sensemaker.ts

@@ -25,13 +25,30 @@ import { getUniqueTopics } from "./sensemaker_utils";
 // summarize a conversation.
 export class Sensemaker {
   private modelSettings: ModelSettings;
+  private readonly defaultModel: Model;
+  private progressCallback?: (report: ProgressReport) => void;
 
   /**
    * Creates a Sensemaker object
    * @param modelSettings what models to use for what tasks, a default model can be set.
    */
   constructor(modelSettings: ModelSettings) {
     this.modelSettings = modelSettings;
+    this.defaultModel = modelSettings.defaultModel;
+  }
+
+  /**
+   * Sets a callback function to receive progress updates during long-running operations
+   * @param callback Function that will be called with progress updates
+   */
+  public setProgressCallback(callback: (report: ProgressReport) => void): void {
+    this.progressCallback = callback;
+  }
+
+  private reportProgress(report: ProgressReport): void {
+    if (this.progressCallback) {
+      this.progressCallback(report);
+    }
   }
 
   /**
@@ -115,10 +132,17 @@ export class Sensemaker {
   public async learnTopics(
     comments: Comment[],
     includeSubtopics: boolean,
-    topics?: Topic[],
-    additionalContext?: string,
-    topicDepth?: 1 | 2 | 3
+    existingTopics?: Topic[],
+    additionalContext?: string
   ): Promise<Topic[]> {
+    this.reportProgress({
+      operation: 'topic_identification',
+      currentStep: 1,
+      totalSteps: includeSubtopics ? 3 : 1,
+      message: 'Starting topic identification...',
+      percentage: 0
+    });
+
     const startTime = performance.now();
 
     // Categorization learns one level of topics and categorizes them and repeats recursively. We want
@@ -127,14 +151,33 @@ export class Sensemaker {
     const categorizedComments = await this.categorizeComments(
       comments,
       includeSubtopics,
-      topics,
+      existingTopics || [],
       additionalContext,
-      topicDepth
+      includeSubtopics ? 3 : 1
     );
     const learnedTopics = getUniqueTopics(categorizedComments);
 
     console.log(`Topic learning took ${(performance.now() - startTime) / (1000 * 60)} minutes.`);
 
+    if (includeSubtopics) {
+      this.reportProgress({
+        operation: 'topic_identification',
+        currentStep: 2,
+        totalSteps: 3,
+        message: 'Identifying subtopics...',
+        percentage: 33
+      });
+      // ... existing subtopic identification code ...
+    }
+
+    this.reportProgress({
+      operation: 'topic_identification',
+      currentStep: includeSubtopics ? 3 : 1,
+      totalSteps: includeSubtopics ? 3 : 1,
+      message: 'Topic identification complete',
+      percentage: 100
+    });
+
     return learnedTopics;
   }
 
@@ -152,11 +195,21 @@ export class Sensemaker {
   public async categorizeComments(
     comments: Comment[],
     includeSubtopics: boolean,
-    topics?: Topic[],
+    topics: Topic[],
     additionalContext?: string,
     topicDepth?: 1 | 2 | 3
   ): Promise<Comment[]> {
-    const startTime = performance.now();
+    const totalBatches = Math.ceil(comments.length / BATCH_SIZE);
+    let processedBatches = 0;
+
+    this.reportProgress({
+      operation: 'statement_categorization',
+      currentStep: 0,
+      totalSteps: totalBatches,
+      message: 'Starting statement categorization...',
+      percentage: 0
+    });
+
     if (!includeSubtopics && topicDepth && topicDepth > 1) {
       throw Error("topicDepth can only be set when includeSubtopics is true");
     }
@@ -171,7 +224,18 @@ export class Sensemaker {
       additionalContext
     );
 
-    console.log(`Categorization took ${(performance.now() - startTime) / (1000 * 60)} minutes.`);
+    for (const batch of categorizedComments) {
+      // ... existing batch processing code ...
+      processedBatches++;
+      this.reportProgress({
+        operation: 'statement_categorization',
+        currentStep: processedBatches,
+        totalSteps: totalBatches,
+        message: `Processing batch ${processedBatches} of ${totalBatches}`,
+        percentage: Math.round((processedBatches / totalBatches) * 100)
+      });
+    }
+
     return categorizedComments;
   }
 }

library/src/types.ts

@@ -412,3 +412,11 @@ export function isTopicType(data: any): data is Topic {
     return checkDataSchema(FlatTopic, data);
   }
 }
+
+export interface ProgressReport {
+  operation: 'topic_identification' | 'statement_categorization' | 'summarization';
+  currentStep: number;
+  totalSteps: number;
+  message: string;
+  percentage: number;
+}