Flowlord Dashboard

[jbsmith7741]

User description

• Adds a SQLite db to use instead of a memory cache for
  • alerting
  • workflow management
  • non-finished task detection
• New Dashboard
  • Alerts: Historical alerted tasks with ability
  • Files: that have processed and the match process
  • Tasks: that have run for a given day
  • Workflows: details about all phases and validate on each phase
  • About: details about cache, and overall system

---

PR Type

Enhancement, Tests, Documentation

---

Description

• Implements comprehensive SQLite database backend replacing in-memory cache for persistent storage of tasks, alerts, file messages, and workflow phases

• Adds new web dashboard UI with embedded HTML templates and static files for viewing alerts, files, tasks, workflows, and system information

• Introduces database schema versioning, migration system, and optimization features including PRAGMA settings, date indexing, and size statistics

• Refactors task master and handler components to use SQLite queries instead of in-memory storage

• Adds alert deduplication, incomplete task detection, and atomic duration tracking for notification frequency management

• Implements workflow phase caching with file loading, checksum tracking, and refresh functionality

• Enhances date range formatting with automatic granularity detection (hourly/daily/monthly)

• Improves cron schedule validation using dedicated parser

• Updates Go toolchain to 1.24.0 and adds modernc.org/sqlite dependency

• Upgrades CircleCI configuration to version 2.1 with Docker build workflow

• Includes comprehensive test suite covering SQLite operations, HTML rendering, date formatting, and workflow caching

---

Diagram Walkthrough

flowchart LR
  A["In-Memory Cache"] -->|"Replace with"| B["SQLite Database"]
  B -->|"Schema & Migration"| C["Database Layer"]
  C -->|"Query"| D["Task Master"]
  C -->|"Query"| E["Handler"]
  D -->|"Store"| F["Tasks & Alerts"]
  E -->|"Render"| G["Dashboard UI"]
  G -->|"Display"| H["Alerts, Files, Tasks, Workflows, About"]
  I["HTML Templates"] -->|"Embedded in"| E
  J["Static Assets"] -->|"CSS & JS"| G

Loading

File Walkthrough

	Relevant files
Enhancement	16 files sqlite.go SQLite database implementation for persistent task caching apps/flowlord/cache/sqlite.go Implements comprehensive SQLite database backend for task caching, replacing in-memory storage Adds schema versioning and migration system to handle database evolution Provides methods for storing and querying tasks, alerts, file messages, and workflow phases Includes database optimization features like PRAGMA settings, date indexing, and size statistics Implements alert deduplication and incomplete task detection with efficient JOIN queries +1576/-0 handler.go Web dashboard UI with embedded templates and static files apps/flowlord/handler.go Adds embedded HTML templates and static file serving for web dashboard pages Implements new HTML rendering functions for alerts, files, tasks, workflows, and about pages Adds template helper functions for formatting dates, durations, and bytes Integrates gzip compression middleware and static file serving with chi router Replaces old memory cache references with SQLite database queries +639/-34 taskmaster.go Task master refactoring for SQLite and alert management    apps/flowlord/taskmaster.go Migrates from memory cache to SQLite database backend Refactors notification handling to use database-stored alerts instead of in-memory lists Adds atomic duration tracking for notification frequency management Implements incomplete task detection via CheckIncompleteTasks() method Updates workflow scheduling to query phases from database instead of in-memory cache +153/-136 workflow.go SQLite-based workflow phase caching implementation              apps/flowlord/cache/workflow.go Implements workflow phase caching using SQLite database with methods for searching, retrieving, and managing workflow phases Adds Phase and PhaseDB structs to represent workflow phases with validation and helper methods Implements workflow file loading, checksum tracking, and refresh functionality to detect file changes Provides database operations for storing and querying workflow phases with dependency tracking +582/-0  tmpl.go Enhanced date range formatting with granularity detection tmpl/tmpl.go Refactors PrintDates() function to automatically detect granularity (hourly/daily/monthly) and format date ranges accordingly Adds isConsecutive() and formatTime() helper functions for granularity-aware date formatting Improves handling of date ranges with gaps, duplicates, and cross-month/year boundaries Simplifies logic by detecting granularity in a single pass through the data +144/-43 cache.go Cache data structures for alerts and task tracking              apps/flowlord/cache/cache.go Adds new data structures AlertRecord and SummaryLine for dashboard alert display Defines TaskJob and Stat structs for task tracking and statistics Comments out legacy in-memory cache implementation while preserving it for reference +42/-18  files.go File message persistence to database                                          apps/flowlord/files.go Adds collection of task IDs and task names during file matching process Calls AddFileMessage() on taskCache to persist file processing information to database Tracks which tasks were created from matched files for audit trail +19/-1    main.go File utility enhancements and stat command                              apps/utils/fz/main.go Adds new stat command to display file statistics in JSON format Simplifies cp() function to use io.Copy() instead of line-by-line scanning Reorganizes imports for better readability Improves error messages for file operations +21/-17  job.go Cron schedule validation improvements                                        apps/flowlord/job.go Replaces manual cron schedule validation with cronParser.Parse() for more robust validation Removes string field counting logic in favor of direct parser validation Improves error handling with formatted error messages +3/-5      workflow.go Workflow cache directory detection method                                workflow/workflow.go Adds IsDir() method to Cache struct to determine if workflow path is a directory Enables callers to distinguish between single file and directory-based workflow configurations +5/-0      style.css Dashboard stylesheet with comprehensive UI styling              apps/flowlord/handler/static/style.css Comprehensive CSS stylesheet for Flowlord dashboard with navigation, tables, and UI components Includes styles for calendar date picker, collapsible sections, and responsive layouts Defines color schemes, typography, and interactive element states Provides table column width management and cell expansion functionality +1290/-0 task_summary.tmpl Task summary template for dashboard display                            apps/flowlord/handler/task_summary.tmpl HTML template for task topic summary section with collapsible display Shows task metrics including completion count, error count, alerts, and execution times Implements job-level breakdown with visual metric cards +51/-0    workflow.tmpl Workflow phases dashboard HTML template with filtering      apps/flowlord/handler/workflow.tmpl New HTML template for workflow phases dashboard Displays workflow file summary with phase counts Implements filtering by workflow file, task type, and job Includes sortable table with columns for workflow, topic, job, rule, dependencies, retry, template, and status Provides expandable/collapsible fields for long text with copy-to-clipboard functionality Client-side JavaScript for sorting, filtering, and field expansion +529/-0  alert.tmpl Alert dashboard HTML template with date filtering                apps/flowlord/handler/alert.tmpl New HTML template for alerts dashboard Displays alert summary grouped by task type with time ranges Sortable table with columns for ID, task type, job, message, alerted time, and task time Date filter for historical alert viewing Expandable message and ID fields with copy-to-clipboard functionality Client-side sorting and filtering with URL parameter persistence +337/-0  task.js Task page JavaScript functionality and filtering logic      apps/flowlord/handler/static/task.js New JavaScript module for task page functionality Implements table sorting with URL parameter persistence Provides filter initialization from server configuration Handles task type and job filter dropdowns with dynamic population Supports expandable fields and copy-to-clipboard operations Includes result filtering and pagination support +348/-0  task.tmpl Task summary dashboard HTML template with pagination          apps/flowlord/handler/task.tmpl New HTML template for task summary dashboard Displays task statistics with clickable stat cards for filtering Implements pagination for large task result sets Provides task type and job filtering with dynamic dropdown population Sortable table with columns for ID, type, job, message, result, info, meta, and timestamps Expandable fields for long text content with copy-to-clipboard support +167/-0
Tests	8 files cache_test.go SQLite database test suite for date queries                            apps/flowlord/cache/cache_test.go Replaces old memory cache tests with new SQLite database tests Adds comprehensive test for DatesByType() method covering tasks, alerts, and files Tests date index rebuilding and backward compatibility methods Validates query results and error handling for invalid data types +95/-210 workflow_test.go Workflow cache unit tests and validation                                  apps/flowlord/cache/workflow_test.go Comprehensive test suite for workflow caching functionality including file loading, phase validation, and refresh operations Tests for phase retrieval, child task discovery, and file path handling Validates TOML parsing, cron expression validation, and workflow file management Tests for handling multiple workflow files and subdirectories +333/-0  handler_test.go Handler tests with SQLite cache and HTML validation            apps/flowlord/handler_test.go Updates test setup to use SQLite cache instead of in-memory workflow cache Adds HTML generation tests for alert, files, task, workflow, and about pages with validation Implements validateHTML() helper to check HTML structure and template execution Adds loadTaskViewData() function to load test data from JSON files +293/-3  taskmaster_test.go TaskMaster tests updated for SQLite cache integration        apps/flowlord/taskmaster_test.go Updates test initialization to use SQLite cache with in-memory database instead of workflow.New() Modifies test cases to work with new taskCache field in taskMaster struct Updates cron schedule expectations in test cases to reflect new format Adds job field to task test inputs for job-specific workflow matching +16/-44  tmpl_test.go Extended date formatting test coverage                                      tmpl/tmpl_test.go Adds comprehensive test cases for PrintDates() function covering daily records with offsets, duplicates, and monthly ranges Tests cross-year monthly ranges, cross-month daily ranges, and single timestamp edge cases Validates handling of empty input and mixed granularity scenarios +43/-0    app_test.go Test expectations updated for hourly date format                  apps/utils/recap/app_test.go Updates test expectations to include hour component in date format (e.g., 2020/01/02T00 instead of 2020/01/02) Affects task summary output format for both single and multiple task test cases +4/-4      files_test.go File matching test updated for job field                                  apps/flowlord/files_test.go Updates test expectations to include Job field in task struct for file matching results Reflects new task structure with explicit job assignment +1/-1      tasks.json Sample task records test data file                                              apps/flowlord/test/tasks.json New test data file with 13 sample task records Includes various task types (csv2json, filecopy, transform, bigquery) Demonstrates different result states (complete, error, warn, alert, running) Contains calculated fields for task duration and queue time Provides realistic timestamps and task metadata for testing +210/-0
Configuration changes	2 files main.go Configuration updates for SQLite database integration        apps/flowlord/main.go Adds SQLite configuration to options struct with LocalPath and TaskTTL settings Initializes database connection during application startup Adds Host configuration option for dashboard URL generation Removes old memory cache initialization in favor of database-backed cache +14/-4    config.yml CircleCI configuration upgrade with Docker build workflow .circleci/config.yml Upgraded CircleCI configuration from version 2 to 2.1 Introduced reusable executor go-executor for Go 1.23 environment Separated test and build jobs with distinct responsibilities Added Docker build job with Docker Hub authentication and image push Implemented workflow orchestration with test_and_build pipeline Added tag-based filtering for both test and build jobs +41/-5
Documentation	3 files Workflow_plan.md Workflow SQLite persistence architecture documentation      apps/flowlord/Workflow_plan.md Detailed specification for SQLite-based workflow persistence system Documents database schema design with workflow files and phases tables Outlines implementation strategy maintaining 100% interface compatibility Provides example queries and migration approach for seamless replacement +366/-0  sqlite_spec.md SQLite technical specification and database schema design apps/flowlord/sqlite_spec.md Comprehensive technical specification for SQLite integration in flowlord Documents completed implementation of task records, alert records, and file message history Defines database schema with tables for alert_records, file_messages, and task_records Includes SQL query examples and view definitions for task duration calculations Outlines database maintenance plan with backup and retention policies Documents web dashboard endpoints and UI implementation status +492/-0  testing-framework.md Testing framework documentation and best practices guide  testing-framework.md Comprehensive documentation of testing patterns and frameworks used in task-tools Details hydronica/trial framework as primary testing tool with examples Documents standard Go testing patterns and example functions Provides conversion strategies from testify/assert to trial framework Includes best practices for test structure, error testing, and time handling Outlines migration plan for converting existing tests +334/-0
Dependencies	2 files go.sum Dependency updates and SQLite support packages                      apps/go.sum Updated google/pprof to version 0.0.0-20250317173921-a4b03ec1a45e Removed jbsmith7741/uri v0.6.0 and added v0.6.1 Added new dependencies: mattn/go-isatty v0.0.20, ncruces/go-strftime v0.1.9, remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec Updated golang.org/x/exp, golang.org/x/mod, and golang.org/x/tools to newer versions Added modernc.org packages for SQLite support (cc, ccgo, fileutil, gc, libc, mathutil, memory, opt, sortutil, sqlite, strutil, token) +39/-8    go.mod Go module updates and SQLite dependency addition                  apps/go.mod Updated Go toolchain from 1.23.3 to 1.24.0 Added modernc.org/sqlite v1.37.0 as direct dependency Updated indirect dependencies including golang.org/x/exp, golang.org/x/mod, golang.org/x/tools Added new indirect dependencies for SQLite support and utilities +11/-4
Additional files	9 files README.md +20/-2    schema.sql +138/-0  about.tmpl +138/-0  files.tmpl +147/-0  header.tmpl +78/-0    calendar.js +163/-0  utils.js +130/-0  go.work.sum +0/-7      jobs.toml +6/-2

---

[qodo-code-review]

PR Compliance Guide 🔍

(Compliance updated until commit 41691aa)

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Cross-site scripting Description: HTML endpoints (e.g., htmlAlert, htmlFiles, htmlTask, htmlWorkflow, htmlAbout) render templates using unescaped data from database and query params without any output sanitization or CSP, potentially enabling stored or reflected XSS if attacker-controlled content (like task Msg, file paths, IDs, or template data) includes HTML/JS. handler.go [371-477] Referred Code dt, _ := time.Parse("2006-01-02", r.URL.Query().Get("date")) if dt.IsZero() { dt = time.Now() } alerts, err := tm.taskCache.GetAlertsByDate(dt) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return } // Get dates with alerts for calendar highlighting datesWithData, _ := tm.taskCache.DatesByType("alerts") w.WriteHeader(http.StatusOK) w.Header().Set("Content-Type", "text/html") w.Write(alertHTML(alerts, dt, datesWithData)) } // htmlFiles handles GET /web/files - displays file messages for a specific date ... (clipped 86 lines)
	Missing security headers Description: Static files and HTML templates are served without setting security headers (e.g., Content-Security-Policy, X-Content-Type-Options, X-Frame-Options), increasing risk of script injection, MIME sniffing, and clickjacking across the new dashboard endpoints. handler.go [94-104] Referred Code // Enable gzip compression for all responses router.Use(middleware.Compress(5)) // Static file serving - serve embedded static files // Create a sub-filesystem that strips the "handler/" prefix staticFS, err := fs.Sub(StaticFiles, "handler/static") if err != nil { log.Fatal("Failed to create static filesystem:", err) } router.Handle("/static/*", http.StripPrefix("/static/", http.FileServer(http.FS(staticFS))))
	Stored XSS via DB Description: Task records are inserted/updated using request/producer-controlled fields (info, meta, msg) which are later displayed in HTML; without input validation or output encoding this can store malicious payloads leading to stored XSS in the dashboard. tasks.go [62-95] Referred Code result, err := s.db.Exec(` INSERT INTO task_records (id, type, job, info, result, meta, msg, created, started, ended) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT (TYPE, job, id, created) DO UPDATE SET result = excluded.result, meta = excluded.meta, msg = excluded.msg, started = excluded.started, ended = excluded.ended `, t.ID, t.Type, t.Job, t.Info, t.Result, t.Meta, t.Msg, t.Created, t.Started, t.Ended, ) if err != nil { log.Printf("ERROR: Failed to insert task record: %v", err) return } // Update date index for this task's date ... (clipped 13 lines)
	Information disclosure Description: Slack alert summary includes HostName and port to construct an internal URL; if externally reachable or misconfigured, this may disclose internal host details to third parties and aid reconnaissance. taskmaster.go [548-571] Referred Code func (tm *taskMaster) sendAlertSummary(alerts []sqlite.AlertRecord) error { if len(alerts) == 0 { return nil } // build compact summary using existing logic summary := sqlite.BuildCompactSummary(alerts) // format message similar to current Slack format var message strings.Builder message.WriteString(fmt.Sprintf("see report at %v:%d/web/alert?date=%s\n", tm.HostName, tm.port, time.Now().Format("2006-01-02"))) for _, line := range summary { message.WriteString(fmt.Sprintf("%-35s%5d %s\n", line.Key+":", line.Count, line.TimeRange)) } // send to Slack if configured log.Println(message.String()) if tm.slack != nil { if err := tm.slack.Notify(message.String(), slack.Critical); err != nil { ... (clipped 3 lines)
	Sensitive information exposure Description: The workflowFiles endpoint echoes arbitrary file contents based on format and path logic without strict allowlisting or authentication, potentially exposing sensitive workflow files or enabling path traversal if other code paths allow attacker-influenced fName. handler.go [326-368] Referred Code w.WriteHeader(http.StatusNoContent) return } pth := tm.path // support directory and single file for workflow path lookup. if tm.taskCache.IsDir() { pth += "/" + fName } sts, err := file.Stat(pth, tm.fOpts) if err != nil { w.WriteHeader(http.StatusNoContent) return } w.Header().Set("Content-Type", "text/plain") if sts.IsDir { w.WriteHeader(http.StatusOK) files, _ := file.List(pth, tm.fOpts) for _, f := range files { b, a, _ := strings.Cut(f.Path, tm.path) w.Write([]byte(b + a + "\n")) ... (clipped 22 lines)
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
🔴	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Error Exposure: User-facing handlers write raw error strings directly to HTTP responses without structured context or safe messaging, lacking graceful degradation and consistent error handling. Referred Code alerts, err := tm.taskCache.GetAlertsByDate(dt) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return } // Get dates with alerts for calendar highlighting datesWithData, _ := tm.taskCache.DatesByType("alerts") w.WriteHeader(http.StatusOK) w.Header().Set("Content-Type", "text/html") w.Write(alertHTML(alerts, dt, datesWithData)) } Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Sensitive Errors: Handlers return internal error details directly to end users (e.g., database/template errors via err.Error()), potentially exposing internal implementation details. Referred Code alerts, err := tm.taskCache.GetAlertsByDate(dt) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return } // Get dates with alerts for calendar highlighting datesWithData, _ := tm.taskCache.DatesByType("alerts") w.WriteHeader(http.StatusOK) w.Header().Set("Content-Type", "text/html") w.Write(alertHTML(alerts, dt, datesWithData)) } Learn more about managing compliance generic rules or creating your own custom rules
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Missing Audit: Newly added web routes and DB-backed operations (alerts, files, tasks, workflows) do not include explicit audit logging of user actions (e.g., viewing pages, fetching workflow files), making it unclear whether critical access/read events are traceable. Referred Code router.Get("/", tm.htmlAbout) router.Get("/info", tm.Info) router.Get("/refresh", tm.refreshHandler) router.Post("/backload", tm.Backloader) router.Get("/workflow/*", tm.workflowFiles) router.Get("/workflow", tm.workflowFiles) router.Get("/notify", func(w http.ResponseWriter, r *http.Request) { sts := stats{ AppName: "flowlord", Version: tools.Version, RunTime: gtools.PrintDuration(time.Since(tm.initTime)), } b, _ := json.Marshal(sts) if err := tm.slack.Notify(string(b), slack.OK); err != nil { w.Write([]byte(err.Error())) } }) router.Get("/status", func(w http.ResponseWriter, r *http.Request) { w.Write([]byte("ok")) }) router.Get("/task/{id}", tm.taskHandler) ... (clipped 6 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Log Content Risk: Logging of task filters and summaries (including IDs, jobs, and messages) may include sensitive task metadata without evidence of redaction or structured logging. Referred Code // Single consolidated log with all metrics log.Printf("Task page: date=%s filters=[id=%q type=%q job=%q result=%q] total=%d filtered=%d page=%d/%d query=%v render=%v size=%.2fMB", date.Format("2006-01-02"), filter.ID, filter.Type, filter.Job, filter.Result, counts.Total, totalCount, filter.Page, totalPages, queryTime, renderTime, float64(htmlSize)/(1024*1024)) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Input Validation: Query parameters (e.g., date, page, filters) are parsed and used to query/render without explicit validation or sanitization in handlers, relying on downstream DB layer protections not visible here. Referred Code dt, _ := time.Parse("2006-01-02", r.URL.Query().Get("date")) if dt.IsZero() { dt = time.Now() } alerts, err := tm.taskCache.GetAlertsByDate(dt) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return } // Get dates with alerts for calendar highlighting datesWithData, _ := tm.taskCache.DatesByType("alerts") w.WriteHeader(http.StatusOK) w.Header().Set("Content-Type", "text/html") w.Write(alertHTML(alerts, dt, datesWithData)) } // htmlFiles handles GET /web/files - displays file messages for a specific date func (tm *taskMaster) htmlFiles(w http.ResponseWriter, r *http.Request) { ... (clipped 71 lines) Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

---

Previous compliance checks

Compliance check up to commit 53c04ba

Security Compliance
⚪	Stored XSS risk Description: HTML is rendered with html/template but user-controlled alert content (e.g., alert.msg, type, job) is directly injected into templates without explicit sanitization or output escaping guidance; while html/template auto-escapes by default, ensure templates do not use "safe" pipelines or unescaped HTML actions that could enable stored XSS via database-stored alert fields. handler.go [374-392] Referred Code dt, _ := time.Parse("2006-01-02", r.URL.Query().Get("date")) if dt.IsZero() { dt = time.Now() } alerts, err := tm.taskCache.GetAlertsByDate(dt) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return } // Get dates with alerts for calendar highlighting datesWithData, _ := tm.taskCache.DatesByType("alerts") w.WriteHeader(http.StatusOK) w.Header().Set("Content-Type", "text/html") w.Write(alertHTML(alerts, dt, datesWithData)) }
	Information disclosure Description: Database table and index size details are exposed via HTTP in aboutHTML, potentially leaking internal schema names and sizing information that could aid attackers in profiling and DoS planning; restrict access to these diagnostics or require authentication. sqlite.go [1466-1528] Referred Code } defer rows.Close() var tables []string for rows.Next() { var tableName string if err := rows.Scan(&tableName); err != nil { return nil, err } tables = append(tables, tableName) } var stats []TableStat var totalTableSize int64 for _, tableName := range tables { // Get row count var rowCount int64 err := s.db.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM %s", tableName)).Scan(&rowCount) if err != nil { continue // Skip tables we can't read ... (clipped 42 lines)
	Reflected XSS risk Description: Summary grouping builds keys from task types and jobs derived from user/task metadata and returns them to the UI; if templates display these values without strict escaping, an attacker could inject HTML/JS via job or type fields—verify template escaping and avoid marking such content as safe HTML. sqlite.go [534-552] Referred Code job := t.Job if job == "" { v, _ := url.ParseQuery(t.Meta) job = v.Get("job") } key := strings.TrimRight(t.Type+":"+job, ":") stat, found := data[key] if !found { stat = &Stats{ CompletedTimes: make([]time.Time, 0), ErrorTimes: make([]time.Time, 0), ExecTimes: &DurationStats{}, } data[key] = stat } stat.Add(t) }
	Untrusted input handling Description: Task summary generation parses meta query strings from tasks and surfaces values to the UI; ensure meta-derived fields are treated as untrusted and escaped in templates to prevent injection and avoid logging sensitive metadata. handler.go [520-582] Referred Code // generateSummaryFromTasks creates a summary of tasks grouped by type:job func generateSummaryFromTasks(tasks []cache.TaskView) map[string]*cache.Stats { summary := make(map[string]*cache.Stats) for _, t := range tasks { // Get job from TaskView.Job or extract from Meta job := t.Job if job == "" { if meta, err := url.ParseQuery(t.Meta); err == nil { job = meta.Get("job") } } // Create key in format "type:job" key := strings.TrimRight(t.Type+":"+job, ":") // Get or create stats for this type:job combination stat, found := summary[key] if !found { stat = &cache.Stats{ CompletedTimes: make([]time.Time, 0), ... (clipped 42 lines)
	DoS via DB locks Description: PRAGMA settings (page_size, auto_vacuum) are applied without enforcing WAL mode and busy timeouts; under concurrent access this can lead to database lock contention and potential denial of service—set a busy timeout (PRAGMA busy_timeout) and consider WAL mode for safer concurrent reads. sqlite.go [94-107] Referred Code // Set a smaller page size to reduce DB file size _, err = db.Exec("PRAGMA page_size = 4096;") if err != nil { return fmt.Errorf("failed to set page size: %w", err) } // Enable auto vacuum to reclaim space when records are deleted _, err = db.Exec("PRAGMA auto_vacuum = INCREMENTAL;") if err != nil { return fmt.Errorf("failed to set auto vacuum: %w", err) } o.db = db
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🔴	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Ignored errors: Database operations frequently ignore or only log errors without propagating or adding context (e.g., Add uses log.Printf and returns, GetTask swallows errors, Recycle/CheckIncompleteTasks return empty stats on error), reducing robustness and debuggability. Referred Code func (s *SQLite) Add(t task.Task) { if t.ID == "" { return } s.mu.Lock() defer s.mu.Unlock() // Use UPSERT to handle both new tasks and updates result, err := s.db.Exec(` INSERT INTO task_records (id, type, job, info, result, meta, msg, created, started, ended) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT (TYPE, job, id, created) DO UPDATE SET result = excluded.result, meta = excluded.meta, msg = excluded.msg, started = excluded.started, ended = excluded.ended `, t.ID, t.Type, t.Job, t.Info, t.Result, t.Meta, t.Msg, ... (clipped 234 lines) Learn more about managing compliance generic rules or creating your own custom rules
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Limited auditing: New DB operations and web actions are not consistently logged with user/context, though some events are logged; verify that critical actions (task insert/update, alert creation, migration, backup/restore, web views) include user id and outcome in structured logs. Referred Code func (s *SQLite) Add(t task.Task) { if t.ID == "" { return } s.mu.Lock() defer s.mu.Unlock() // Use UPSERT to handle both new tasks and updates result, err := s.db.Exec(` INSERT INTO task_records (id, type, job, info, result, meta, msg, created, started, ended) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT (TYPE, job, id, created) DO UPDATE SET result = excluded.result, meta = excluded.meta, msg = excluded.msg, started = excluded.started, ended = excluded.ended `, t.ID, t.Type, t.Job, t.Info, t.Result, t.Meta, t.Msg, ... (clipped 322 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Generic names: Some identifiers like isLocal and data maps in template rendering are generic and may benefit from clearer names, though most function and type names are meaningful. Referred Code var isLocal = false // getBaseFuncMap returns a template.FuncMap with all common template functions func getBaseFuncMap() template.FuncMap { return template.FuncMap{ // Time formatting functions "formatFullDate": func(t time.Time) string { return t.Format(time.RFC3339) }, "formatTimeHour": func(t time.Time) string { return t.Format("2006-01-02T15") }, // Duration formatting "formatDuration": gtools.PrintDuration, // Size formatting "formatBytes": func(bytes int64) string { if bytes < 0 { return "0 B" } return humanize.Bytes(uint64(bytes)) }, ... (clipped 777 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Error exposure: HTML handlers write raw error messages to the response (e.g., htmlAlert/files/task/about respond with err.Error()), which may expose internal details to end users; user-facing messages should be generic with detailed logs internally. Referred Code dt, _ := time.Parse("2006-01-02", r.URL.Query().Get("date")) if dt.IsZero() { dt = time.Now() } alerts, err := tm.taskCache.GetAlertsByDate(dt) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return } // Get dates with alerts for calendar highlighting datesWithData, _ := tm.taskCache.DatesByType("alerts") w.WriteHeader(http.StatusOK) w.Header().Set("Content-Type", "text/html") w.Write(alertHTML(alerts, dt, datesWithData)) } // htmlFiles handles GET /web/files - displays file messages for a specific date ... (clipped 74 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Verbose logging: The taskHTML function logs detailed request filters and sizes which could be acceptable, but there is no evidence of structured logging or safeguards against sensitive data in logs. Referred Code htmlSize := buf.Len() renderTime := time.Since(renderStart) // Single consolidated log with all metrics log.Printf("Task page: date=%s filters=[type=%q job=%q result=%q] tasks=%d filtered=%d page=%d/%d query=%v render=%v size=%.2fMB", date.Format("2006-01-02"), taskType, job, result, totalAllTasks, totalFilteredTasks, page, totalPages, queryTime, renderTime, float64(htmlSize)/(1024*1024)) return buf.Bytes() Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Input validation: While SQL statements use parameters to avoid injection, HTTP handlers parse query params without validation and some SQL uses fmt.Sprintf for column names (DatesByType) which is controlled by internal switch, but overall input sanitation and authorization are not evident. Referred Code // DatesByType returns a list of dates (YYYY-MM-DD format) that have data for the specified type // dataType can be "tasks", "alerts", or "files" // This uses the date_index table for instant lookups func (s *SQLite) DatesByType(dataType string) ([]string, error) { s.mu.Lock() defer s.mu.Unlock() var column string switch dataType { case "tasks": column = "has_tasks" case "alerts": column = "has_alerts" case "files": column = "has_files" default: return nil, fmt.Errorf("invalid data type: %s (must be 'tasks', 'alerts', or 'files')", dataType) } query := fmt.Sprintf(` SELECT date ... (clipped 7 lines) Learn more about managing compliance generic rules or creating your own custom rules

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Security	Prevent potential SQL injection vulnerabilities To prevent potential SQL injection, replace fmt.Sprintf with %s with %q to properly quote table names in SQL queries. apps/flowlord/cache/sqlite.go [1484] // In GetTableStats // Get row count var rowCount int64 - err := s.db.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM %s", tableName)).Scan(&rowCount) + err := s.db.QueryRow(fmt.Sprintf("SELECT COUNT(*) FROM %q", tableName)).Scan(&rowCount) if err != nil { continue // Skip tables we can't read } ... // In getIndexSize func (s *SQLite) getIndexSize(tableName string) int64 { // Get all indexes for this table rows, err := s.db.Query(fmt.Sprintf(` SELECT name FROM sqlite_master - WHERE type='index' AND tbl_name='%s' AND name NOT LIKE 'sqlite_%%' + WHERE type='index' AND tbl_name=%q AND name NOT LIKE 'sqlite_%%' `, tableName)) ... } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a potential SQL injection vulnerability due to unsafe string formatting and proposes the correct fix, which is a critical security best practice.	High
Possible issue	Ensure atomic database index rebuild Wrap the database operations in RebuildDateIndex within a single transaction to ensure the index rebuild is atomic and prevent data corruption on failure. apps/flowlord/cache/sqlite.go [1308-1374] // RebuildDateIndex scans all tables and rebuilds the date_index table // This should be called once during migration or can be exposed as an admin endpoint func (s *SQLite) RebuildDateIndex() error { s.mu.Lock() defer s.mu.Unlock() log.Println("Starting date_index rebuild...") + tx, err := s.db.Begin() + if err != nil { + return fmt.Errorf("failed to begin transaction: %w", err) + } + defer tx.Rollback() // Rollback on error + // Clear existing index - _, err := s.db.Exec("DELETE FROM date_index") + _, err = tx.Exec("DELETE FROM date_index") if err != nil { return fmt.Errorf("failed to clear date_index: %w", err) } // Populate from task_records // First insert the dates, then update the has_tasks flag - _, err = s.db.Exec(` + _, err = tx.Exec(` INSERT OR IGNORE INTO date_index (date, has_tasks) SELECT DISTINCT DATE(created), 1 FROM task_records `) if err != nil { return fmt.Errorf("failed to populate date_index from tasks: %w", err) } ... + if err := tx.Commit(); err != nil { + return fmt.Errorf("failed to commit transaction: %w", err) + } log.Println("Successfully rebuilt date_index") return nil } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a race condition that could lead to data inconsistency and proposes wrapping the database operations in a transaction to ensure atomicity, which is a critical fix for data integrity.	Medium
	Prevent resource leaks with defer In the cp function, use defer to close both the reader and writer to prevent resource leaks if an error occurs during the copy or close operations. apps/utils/fz/main.go [149-169] func cp(from, to string, opt *file.Options) error { if to == "" || from == "" { return fmt.Errorf(usage) } r, err := file.NewReader(from, opt) if err != nil { return fmt.Errorf("init reader err: %w", err) } + defer r.Close() + w, err := file.NewWriter(to, opt) if err != nil { return fmt.Errorf("init writer err: %w", err) } - _, err = io.Copy(w, r) - if err != nil { + defer w.Close() + + if _, err = io.Copy(w, r); err != nil { return fmt.Errorf("copy err: %w", err) } - if err := w.Close(); err != nil { - return fmt.Errorf("close writer err: %w", err) - } - return r.Close() + + // The deferred w.Close() will handle closing the writer. + // We still need to return its error if it occurs. + return w.Close() } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a potential resource leak due to improper error handling and proposes using defer, which is the idiomatic Go solution for ensuring resource cleanup.	Medium
	Prevent silent cron job failures In NewJob, add a check to ensure the bJob.Schedule field is not empty before parsing, preventing the silent failure of jobs created without a valid cron schedule. apps/flowlord/job.go [70-87] func (tm *taskMaster) NewJob(ph workflow.Phase, path string) (cron.Job, error) { bJob := &Cronjob{ Workflow: path, Topic: ph.Topic(), Template: ph.Template, } err := uri.Unmarshal(ph.Rule, bJob) if err != nil { return nil, err + } + + if bJob.Schedule == "" { + return nil, errors.New("cron schedule is missing or empty") } if _, err := cronParser.Parse(bJob.Schedule); err != nil { return nil, fmt.Errorf("cron: %w", err) } // return Cronjob if not batch params if bJob.Every == 0 { return bJob, nil } return NewBatch(bJob) } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a silent failure mode where a job with an empty cron schedule is considered valid but never runs, and the proposed check prevents this by enforcing a non-empty schedule.	Medium
	Fix incorrect duration formatting in SQL Correct the SQL duration formatting by replacing the incorrect use of strftime with the time() function and the 'unixepoch' modifier. apps/flowlord/sqlite_spec.md [223-236] -- Format task duration as HH:MM:SS -strftime('%H:%M:%S', - CAST((julianday(task_records.ended) - julianday(task_records.started)) * 24 * 60 * 60 AS INTEGER) / 3600 || ':' || - CAST((julianday(task_records.ended) - julianday(task_records.started)) * 24 * 60 * 60 AS INTEGER) % 3600 / 60 || ':' || - CAST((julianday(task_records.ended) - julianday(task_records.started)) * 24 * 60 * 60 AS INTEGER) % 60 -) as task_time, +time(CAST((julianday(task_records.ended) - julianday(task_records.started)) * 24 * 60 * 60 AS INTEGER), 'unixepoch') as task_time, -- Calculate queue time in seconds CAST((julianday(task_records.started) - julianday(task_records.created)) * 24 * 60 * 60 AS INTEGER) as queue_seconds, -- Format queue duration as HH:MM:SS -strftime('%H:%M:%S', - CAST((julianday(task_records.started) - julianday(task_records.created)) * 24 * 60 * 60 AS INTEGER) / 3600 || ':' || - CAST((julianday(task_records.started) - julianday(task_records.created)) * 24 * 60 * 60 AS INTEGER) % 3600 / 60 || ':' || - CAST((julianday(task_records.started) - julianday(task_records.created)) * 24 * 60 * 60 AS INTEGER) % 60 -) as queue_time, +time(CAST((julianday(task_records.started) - julianday(task_records.created)) * 24 * 60 * 60 AS INTEGER), 'unixepoch') as queue_time, Apply / Chat Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a bug in the SQL query where strftime is misused, which would lead to incorrect time formatting, and provides a simpler, correct implementation using the time() function.	Medium
	Fix incorrect database transaction handling Fix incorrect database transaction handling in removeWorkflow by removing the deferred tx.Rollback() and instead explicitly rolling back the transaction only on error paths. apps/flowlord/cache/workflow.go [532-554] // removeWorkflow removes a workflow and its phases from the database func (s *SQLite) removeWorkflow(filePath string) error { // Start transaction tx, err := s.db.Begin() if err != nil { return err } - defer tx.Rollback() // Remove phases first _, err = tx.Exec("DELETE FROM workflow_phases WHERE file_path = ?", filePath) if err != nil { + tx.Rollback() return err } // Remove workflow file record _, err = tx.Exec("DELETE FROM workflow_files WHERE file_path = ?", filePath) if err != nil { + tx.Rollback() return err } return tx.Commit() } Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies an incorrect transaction handling pattern where Rollback() is called even after a successful Commit(), and the proposed fix is the standard, robust way to handle SQL transactions in Go.	Medium
General	✅ Avoid duplicating business logic Suggestion Impact: The commit removed the custom generateSummaryFromTasks implementation and its duplicated stats logic, switching to a new flow where task statistics are obtained via tm.taskCache.GetTaskSummaryByDate and passed through taskStats into the template. This effectively eliminates the duplicated logic the suggestion targeted. code diff: -// generateSummaryFromTasks creates a summary of tasks grouped by type:job -func generateSummaryFromTasks(tasks []cache.TaskView) map[string]*cache.Stats { - summary := make(map[string]*cache.Stats) - - for _, t := range tasks { - // Get job from TaskView.Job or extract from Meta - job := t.Job - if job == "" { - if meta, err := url.ParseQuery(t.Meta); err == nil { - job = meta.Get("job") - } - } - - // Create key in format "type:job" - key := strings.TrimRight(t.Type+":"+job, ":") - - // Get or create stats for this type:job combination - stat, found := summary[key] - if !found { - stat = &cache.Stats{ - CompletedTimes: make([]time.Time, 0), - ErrorTimes: make([]time.Time, 0), - ExecTimes: &cache.DurationStats{}, - } - summary[key] = stat - } - - // Convert TaskView to task.Task for processing - taskTime := tmpl.TaskTime(task.Task{ - ID: t.ID, - Type: t.Type, - Job: t.Job, - Info: t.Info, - Result: task.Result(t.Result), - Meta: t.Meta, - Msg: t.Msg, - Created: t.Created, - Started: t.Started, - Ended: t.Ended, - }) - - // Process based on result type - if t.Result == "error" { - stat.ErrorCount++ - stat.ErrorTimes = append(stat.ErrorTimes, taskTime) - } else if t.Result == "complete" { - stat.CompletedCount++ - stat.CompletedTimes = append(stat.CompletedTimes, taskTime) - - // Add execution time for completed tasks - if t.Started != "" && t.Ended != "" { - startTime, err1 := time.Parse(time.RFC3339, t.Started) - endTime, err2 := time.Parse(time.RFC3339, t.Ended) - if err1 == nil && err2 == nil { - stat.ExecTimes.Add(endTime.Sub(startTime)) - } - } - } - // Note: warn and alert results don't contribute to execution time stats - } - - return summary -} Refactor generateSummaryFromTasks to use the existing stat.Add() method for calculating statistics, eliminating duplicated logic and improving maintainability. apps/flowlord/handler.go [520-582] // generateSummaryFromTasks creates a summary of tasks grouped by type:job func generateSummaryFromTasks(tasks []cache.TaskView) map[string]*cache.Stats { summary := make(map[string]*cache.Stats) for _, t := range tasks { -... + // Get job from TaskView.Job or extract from Meta + job := t.Job + if job == "" { + if meta, err := url.ParseQuery(t.Meta); err == nil { + job = meta.Get("job") + } + } + + // Create key in format "type:job" + key := strings.TrimRight(t.Type+":"+job, ":") + + // Get or create stats for this type:job combination + stat, found := summary[key] + if !found { + stat = &cache.Stats{ + CompletedTimes: make([]time.Time, 0), + ErrorTimes: make([]time.Time, 0), + ExecTimes: &cache.DurationStats{}, + } + summary[key] = stat + } + // Convert TaskView to task.Task for processing - taskTime := tmpl.TaskTime(task.Task{ + tsk := task.Task{ ID: t.ID, Type: t.Type, Job: t.Job, Info: t.Info, Result: task.Result(t.Result), Meta: t.Meta, Msg: t.Msg, Created: t.Created, Started: t.Started, Ended: t.Ended, - }) - - // Process based on result type - if t.Result == "error" { - stat.ErrorCount++ - stat.ErrorTimes = append(stat.ErrorTimes, taskTime) - } else if t.Result == "complete" { -... } - // Note: warn and alert results don't contribute to execution time stats + // Use the existing Add method to avoid duplicating logic + stat.Add(tsk) } return summary } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies duplicated logic for calculating statistics and proposes a refactoring to use the existing stat.Add() method, which improves code maintainability and reduces redundancy.	Medium
Update