Implement support for new epoch columns to StreamDBClient queries #216

Tiihott · 2025-08-25T12:14:11Z

Description

The aim of this pull request is to implement support for new epoch time columns to logfile metadata queries done by StreamDBClient. Original logdate and logtime implementations are prone to timezone issues where the result of the query changes depending on the session timezone, using epoch time resolves this issue.

Includes:

Added support for using epoch_hour column as a source for logtime result value.
Removed support for using old logdate and logtime implementation.
New tests for asserting the query changes work as expected.
Renamed and disabled old logdate implementation tests.
Fixed weightedOffsetTest which was not using whole hours for logtime values.

Resolves #213
Resolves #214
Resolves #284

Checklists

Testing

General

I have checked that my test files and functions have meaningful names.
I have checked that each test tests only a single behavior.
I have done happy tests.
I have tested only my own code.
I have tested at least all public methods.

Assertions

I have checked that my tests use assertions and not runtime overhead.
I have checked that my tests end in assertions.
I have checked that there is no comparison statements in assertions.
I have checked that assertions are in tests and not in helper functions.
I have checked that assertions for iterables are outside of for loops and both sides of the iteration blocks.
I have checked that assertions are not tested inside consumers.

Testing Data

I have tested algorithms and anything else with the possibility of unbound growth.
I have checked that all testing data is local and fully replaceable or reproducible or both.
I have checked that all test files are standalone.
I have checked that all test-specific fake objects and classes are in the test directory.
I have checked that my tests do not contain anything related to customers, infrastructure or users.
I have checked that my tests do not contain non-generic information.
I have checked that my tests do not do external requests and are not privately or publicly routable.

Statements

I have checked that my tests do not use throws for exceptions.
I have checked that my tests do not use try-catch statements.
I have checked that my tests do not use if-else statements.

Java

I have checked that my tests for Java uses JUnit library.
I have checked that my tests for Java uses JUnit utilities for parameters.

Other

I have only tested public behavior and not private implementation details.
I have checked that my tests are not (partially) commented out.
I have checked that hand-crafted variables in assertions are used accordingly.
I have tested Object Equality.
I have checked that I do not have any manual tests or I have a valid reason for them and I have explained it in the PR description.

Code Quality

I have checked that my code follows metrics set in Procedure: Class Metrics.
I have checked that my code follows metrics set in Procedure: Method Metrics.
I have checked that my code follows metrics set in Procedure: Object Quality.
I have checked that my code does not have any NULL values.
I have checked my code does not contain FIXME or TODO comments.

StrongestNumber9 · 2025-08-27T07:35:38Z

Does not include:

MariaDB TestContainers implementation for StreamDBClient tests, tests that are run on workstation require a working MariaDB instance to be set up. Workflows have the correct MariaDB setup already present for running the tests.

I think this is not very good approach as this could introduce security issues for workstation users when running improperly configured MariaDB, like unauthenticated access but open to the internet. Also while the CI starts from scratch everytime using well known steps, the database developer is using might get out of sync with schema changes or have lingering data which makes the tests unreliable.

Example of the actual issue with out of date database like I had:

[ERROR] epochHourNullTest  Time elapsed: 0.157 s  <<< ERROR!
org.jooq.exception.DataAccessException: SQL [insert into `journaldb`.`logfile` (`id`, `logdate`, `expiration`, `bucket_id`, `path`, `object_key_hash`, `host_id`, `original_filename`, `archived`, `file_size`, `sha256_checksum`, `archive_etag`, `logtag`, `source_system_id`, `category_id`, `uncompressed_file_size`, `epoch_hour`, `epoch_expires`, `epoch_archived`) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; (conn=3) Unknown column 'epoch_hour' in 'INSERT INTO'
        at com.teragrep.pth_06.planner.StreamDBClientTest.epochHourNullTest(StreamDBClientTest.java:296)
Caused by: java.sql.SQLSyntaxErrorException: (conn=3) Unknown column 'epoch_hour' in 'INSERT INTO'
        at com.teragrep.pth_06.planner.StreamDBClientTest.epochHourNullTest(StreamDBClientTest.java:296)

Concider implementing it like described in https://github.com/teragrep/mvn_01/tree/main/testcontainers

StrongestNumber9

The tests requires quite a lot of cleaning and adding the testcontainers as they are very mentally expensive to parse, especially with the duplicated codeblocks

StrongestNumber9 · 2025-08-27T07:48:33Z

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

+        HostRecord hostRecord = new HostRecord(UShort.valueOf(1), "testHost1");
+        ctx.insertInto(JOURNALDB.HOST).set(hostRecord).execute();
+
+        // Set logdate to 2023-10-04 instead of the correct 2023-10-05 to emulate timezone issues, and test if epoch_hour takes priority or not.


Sounds like something worth of two distinct tests

Changed the test configuration to use predetermined UTC-4 timezone for all tests instead of whatever the system the test runs on decides to use as session timezone.
epochHourNullTest() now tests if the results are affected by the UTC-4 session timezone as expected when old logdate/logtime implementation is used.
epochHourTimezoneTest() tests if logtime and logdate results are unaffected by the UTC-4 timezone.

StrongestNumber9 · 2025-08-27T07:51:48Z

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

+        long logtime = hourRange.get(0).get(8, Long.class);
+        Assertions.assertEquals(1696471200L, logtime);
+        Date logdate = hourRange.get(0).get(5, Date.class);


Extracting these hourRanges to single use variable looks a bit noisier than it has to be

Removed single use variables in commit 8f96a57

StrongestNumber9 · 2025-08-27T07:53:51Z

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

+        long logtime = hourRange.get(0).get(8, Long.class);
+        Assertions.assertEquals(1696471200L, logtime);
+        Date logdate = hourRange.get(0).get(5, Date.class);
+        Assertions.assertEquals(Date.valueOf(date), logdate);


Can this Date.valueOf(date) be changed into literal of some sort instead of using output of a function with a variable passed as the value? This is to avoid some bizarre scenario where both Date.valueOf(date) and logdate ends up being null or 0 or something else similarly unexpected

Changed Date.valueod() to literal string format instead of LocalDate in commit 8f96a57

StrongestNumber9 · 2025-08-27T07:55:56Z

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

+                "example.log-@1696471200-2023100423.log.gz",
+                new Timestamp(2025, 8, 13, 16, 18, 22, 0),
+                ULong.valueOf(120L),
+                "e2I8CnejWweTSk8tmo4tNkDO2fU7RajqI111FPlF7Mw=",


What is this value?

Random sha256Checksum value, replaced with simple description string in commit 381c1d3

StrongestNumber9 · 2025-08-27T07:56:02Z

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

+                new Timestamp(2025, 8, 13, 16, 18, 22, 0),
+                ULong.valueOf(120L),
+                "e2I8CnejWweTSk8tmo4tNkDO2fU7RajqI111FPlF7Mw=",
+                "5ddea0496b0a9ad1266b26da3de9f573",


What is this value?

Random archiveEtag value, replaced with simple description string in commit 381c1d3

StrongestNumber9 · 2025-08-27T07:56:54Z

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

+        LogfileRecord logfileRecord = new LogfileRecord(
+                ULong.valueOf(1),
+                Date.valueOf(LocalDate.of(2023, 10, 4)),
+                new Date(2125 - 1900, 7, 13),


Looks like magic numbers

replaced with easier to read Date.valueOf() in commit 381c1d3

StrongestNumber9 · 2025-08-27T08:01:08Z

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

+        // Init mandatory Config object with the minimum options required for testing StreamDBClient.
+        Map<String, String> opts = new HashMap<>();
+        opts.put("S3endPoint", "mock");
+        opts.put("S3identity", "mock");
+        opts.put("S3credential", "mock");
+        opts.put("DBusername", streamDBUsername);
+        opts.put("DBpassword", streamDBPassword);
+        opts.put("DBurl", streamDBUrl);
+        opts.put("DBstreamdbname", streamdbName);
+        opts.put("DBjournaldbname", journaldbName);
+        opts.put("queryXML", "<index value=\"example\" operation=\"EQUALS\"/>");
+        opts.put("archive.enabled", "true");
+        Config config = new Config(opts);
+        // Add test data to journaldb and streamdb, there are several tables in both databases.
+        Settings settings = new Settings()
+                .withRenderMapping(new RenderMapping().withSchemata(new MappedSchema().withInput("streamdb").withOutput(config.archiveConfig.dbStreamDbName), new MappedSchema().withInput("journaldb").withOutput(config.archiveConfig.dbJournalDbName), new MappedSchema().withInput("bloomdb").withOutput(config.archiveConfig.bloomDbName)));
+        final Connection connection = Assertions
+                .assertDoesNotThrow(
+                        () -> DriverManager
+                                .getConnection(
+                                        config.archiveConfig.dbUrl, config.archiveConfig.dbUsername,
+                                        config.archiveConfig.dbPassword
+                                )
+                );
+        final DSLContext ctx = DSL.using(connection, SQLDialect.MYSQL, settings);
+
+        BucketRecord bucketRecord = new BucketRecord(UShort.valueOf(1), "bucket1");
+        ctx.insertInto(JOURNALDB.BUCKET).set(bucketRecord).execute();
+
+        ctx
+                .query(String.format("INSERT INTO %s.category (id, name) VALUES (1, 'testCategory')", journaldbName))
+                .execute();
+        ctx
+                .query(
+                        String
+                                .format(
+                                        "INSERT INTO %s.source_system (id, name) VALUES (2, 'testSourceSystem2')",
+                                        journaldbName
+                                )
+                )
+                .execute();
+


These could be deduplicated either in beforeEach or as some buildDSLContext function. Repeating 40 idential lines per test is not very easy to mentally parse

Moved all the test data insertions that are not directly relevant to the tests to beforeEach in commit 2fbb4de
Created a private method for initializing Config object with test options in commit 1f2b0a6

Tiihott · 2025-08-28T07:28:34Z

Implemented testcontainers to the tests. As testcontainers do not support managing several databases inside the same testcontainer using the methods available on the testcontainer object, the streamdb database had to be initialized using a separate CREATE_STREAMDB_DB.sql script that is called during container startup.

Note: New databases can't be created after the container is already started as the query results in error with access denied for the user.

Tiihott · 2025-08-28T08:31:35Z

Cleaned up beforeEach further by moving all journaldb table creation and test data population inside the script that is called during container startup.

StrongestNumber9 · 2025-08-28T10:28:56Z

The tests looks a lot better now, good job with them! I still couldn't run the project as I use podman and not docker but I'll look into what was missing

StrongestNumber9 · 2025-08-28T11:59:29Z

I was able to make it work with podman, opened ticket a ticket for enhancing documentation in teragrep/mvn_01#84

I think this looks acceptable but I would like @kortemik to give another sanity check

Tiihott · 2025-09-12T07:34:37Z

Noticed an error in logdate definition for StreamDBClient queries. The epoch based logdate that is not affected by session timezone should actually be sourced from epoch_hour column instead of epoch_archived column.
Fixed in commit 13cfba8

Tiihott · 2025-09-15T11:08:26Z

Rebased to main.

Tiihott · 2025-10-01T05:58:43Z

Rebased to main.

Tiihott · 2025-11-03T13:02:43Z

Rebased to main, updated the new SteamDBClient tests in the rebase.
MariaDB TestContainers implementation for StreamDBClient tests was added to main.

elliVM

Some comments

src/main/java/com/teragrep/pth_06/planner/StreamDBClient.java

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

elliVM

LGTM!

kortemik · 2025-11-19T11:43:44Z

please rebase

Tiihott · 2025-11-20T10:50:08Z

Rebased to main.

kortemik · 2025-11-20T11:48:17Z

proper indexes are missing for logfile table regardin epoch_hour, coalesce will call non-indexed column epoch_hour and cause full table scan. additionally coalesce will cause the search to be non-search argument able meaning b-tree scan can not be used and therefore this approach will not work.

see more at https://dba.stackexchange.com/questions/162024/is-coalesce-sargable-now

kortemik · 2025-11-20T11:48:43Z

src/main/java/com/teragrep/pth_06/planner/NestedTopNQuery.java

        logger.debug("NestedTopNQuery.getTableStatement exit");
+        final Field<Date> logdateFunction = DSL
+                .field(
+                        "CAST(date_add('1970-01-01', interval {0} second) as DATE)", Date.class,


mariadb has FROM_UNIXTIME, please use it

kortemik · 2025-11-20T11:49:04Z

src/main/java/com/teragrep/pth_06/planner/StreamDBClient.java

-
+        final Field<Date> logdateFunction = DSL
+                .field(
+                        "CAST(date_add('1970-01-01', interval {0} second) as DATE)", Date.class,


mariadb has FROM_UNIXTIME, please use it

Using FROM_UNIXTIME forces the logdateFunction result to be tied to session timezone. date_add is used as replacement that is not affected by the session timezone.

Date type is assumed to be session timezone, where do we use this and why? why do not we just pass the epoch as bigint?

…me field creation. REBASE Implemented coalesce with epoch_hour converted to date for defining logdate of the logfile. REBASE

…hen available. REBASE

…sing date arithmetics that completely ignores session timezone. Added epochHourTimezoneTest. REBASE

…TimezoneTest().

…ed and disabled.

…est().

…lEpochTest().

…TableNullEpochTest().

Tiihott · 2026-01-09T10:48:22Z

Rebased and removed the functionality for using old logtime implementation for now.
Renamed and disabled tests that were using the old logtime implementation, and added new epoch logtime tests to replace them.
Updated PR description with the changes.

elliVM · 2026-01-12T08:35:15Z

I think we should consider changing the query contract from a calendar-base (Date) model to an explicit epoch range model (e.g. startEpochHourInclusive - endEpochHourExclusive). This would eliminate the need for date reconstruction and casting in SQL, which can introduce unwanted dependencies on the MariaDB session timezone. the epoch_hour column should be treated as the single source of truth for the time-based semantics, with any calendar or date conversions performed outside of the database layer.

I suggest this because it is frequently non-obvious whether MariaDB applies session or system timezone rules when casting or deriving the DATE values.

@kortemik since the epoch_hour migration is in process, do you think it's acceptable to switch at this time?

src/main/java/com/teragrep/pth_06/planner/NestedTopNQuery.java

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java

elliVM

Looks good to me, question still open about switching to epoch range based query logic. Perhaps can be done in another issue

kortemik · 2026-01-23T13:37:37Z

src/main/java/com/teragrep/pth_06/planner/StreamDBClient.java

-
+        final Field<Date> logdateFunction = DSL
+                .field(
+                        "CAST(date_add('1970-01-01', interval {0} second) as DATE)", Date.class,


Date type is assumed to be session timezone, where do we use this and why? why do not we just pass the epoch as bigint?

Tiihott self-assigned this Aug 25, 2025

Tiihott requested a review from StrongestNumber9 August 27, 2025 06:34

StrongestNumber9 requested changes Aug 27, 2025

View reviewed changes

Tiihott requested a review from StrongestNumber9 August 28, 2025 07:29

StrongestNumber9 requested a review from kortemik August 28, 2025 11:59

Tiihott force-pushed the epoch_reads branch from 13cfba8 to c064d8b Compare September 15, 2025 11:08

Tiihott force-pushed the epoch_reads branch from c064d8b to 703bd47 Compare October 1, 2025 05:58

Tiihott force-pushed the epoch_reads branch from 703bd47 to ba83138 Compare November 3, 2025 12:59

Tiihott requested a review from elliVM November 3, 2025 13:05

elliVM requested changes Nov 4, 2025

View reviewed changes

src/main/java/com/teragrep/pth_06/planner/StreamDBClient.java Outdated Show resolved Hide resolved

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java Show resolved Hide resolved

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java Show resolved Hide resolved

Tiihott requested a review from elliVM November 6, 2025 10:09

elliVM previously approved these changes Nov 6, 2025

View reviewed changes

kortemik requested a review from eemhu November 19, 2025 11:43

Tiihott added the review Issues or pull requests waiting for a review label Nov 19, 2025

eemhu previously approved these changes Nov 20, 2025

View reviewed changes

Tiihott dismissed stale reviews from eemhu and elliVM via 2c5ffbc November 20, 2025 10:49

Tiihott force-pushed the epoch_reads branch from 120b37a to 2c5ffbc Compare November 20, 2025 10:49

kortemik requested changes Nov 20, 2025

View reviewed changes

Tiihott removed the review Issues or pull requests waiting for a review label Nov 20, 2025

Tiihott added 8 commits January 9, 2026 12:37

Implemented coalesce with epoch_hour column usage for synthetic logti…

78cc53b

…me field creation. REBASE Implemented coalesce with epoch_hour converted to date for defining logdate of the logfile. REBASE

Fixed pullToSliceTable to allow using epoch_hour for Date filtering w…

1b6c734

…hen available. REBASE

Implemented epochHourTest and epochHourNullTest. REBASE

38dd66a

Fixed timezone issues with logdate generation from epoch_hour field u…

9ded5ce

…sing date arithmetics that completely ignores session timezone. Added epochHourTimezoneTest. REBASE

Removed debugging println() from tests.

73dffc1

Removed wildcards from imports. REBASE

4ea2f10

Added connection closing to cleanup.

7cc1d86

Implemented the usage of two different session timezones to epochHour…

25c9c63

…TimezoneTest().

Tiihott force-pushed the epoch_reads branch from 2c5ffbc to 25c9c63 Compare January 9, 2026 10:37

Tiihott added 6 commits January 9, 2026 12:38

Removed support for old logtime source, only epoch_hour supported.

c20d6e7

Added new epoch tests to replace old null epoch tests that were renam…

d2bf0af

…ed and disabled.

Added second bucket to test data.

0305210

Fixed test setup in weightedOffsetTest() and weightedOffsetNullEpochT…

32ada06

…est().

Added setIncludeBeforeEpochTest() to replace setIncludeBeforeEpochNul…

387875c

…lEpochTest().

Added deleteRangeFromSliceTableTest() to replace deleteRangeFromSlice…

97871bf

…TableNullEpochTest().

Tiihott requested a review from elliVM January 9, 2026 10:48

Tiihott requested a review from kortemik January 12, 2026 09:05

Tiihott added the review Issues or pull requests waiting for a review label Jan 13, 2026

elliVM requested changes Jan 14, 2026

View reviewed changes

src/main/java/com/teragrep/pth_06/planner/NestedTopNQuery.java Show resolved Hide resolved

src/main/java/com/teragrep/pth_06/planner/NestedTopNQuery.java Outdated Show resolved Hide resolved

src/test/java/com/teragrep/pth_06/planner/StreamDBClientTest.java Show resolved Hide resolved

Tiihott removed the review Issues or pull requests waiting for a review label Jan 14, 2026

Tiihott added 2 commits January 14, 2026 14:55

Renamed logtime usage to epochHour.

2fa6133

Refactored Config options to be copied from the global options in tests.

9bfe420

Tiihott requested a review from elliVM January 14, 2026 13:33

elliVM approved these changes Jan 14, 2026

View reviewed changes

Tiihott added the review Issues or pull requests waiting for a review label Jan 14, 2026

kortemik requested changes Jan 23, 2026

View reviewed changes

Implement support for new epoch columns to StreamDBClient queries #216

Are you sure you want to change the base?

Implement support for new epoch columns to StreamDBClient queries #216

Conversation

Tiihott commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Testing

General

Assertions

Testing Data

Statements

Java

Other

Code Quality

Uh oh!

StrongestNumber9 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StrongestNumber9 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tiihott commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tiihott commented Aug 28, 2025

Uh oh!

StrongestNumber9 commented Aug 28, 2025

Uh oh!

StrongestNumber9 commented Aug 28, 2025

Uh oh!

Tiihott commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tiihott commented Sep 15, 2025

Uh oh!

Tiihott commented Oct 1, 2025

Uh oh!

Tiihott commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elliVM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elliVM left a comment

Choose a reason for hiding this comment

Uh oh!

kortemik commented Nov 19, 2025

Tiihott commented Aug 25, 2025 •

edited

Loading

StrongestNumber9 commented Aug 27, 2025 •

edited

Loading

Tiihott commented Aug 28, 2025 •

edited

Loading

Tiihott commented Sep 12, 2025 •

edited

Loading

Tiihott commented Nov 3, 2025 •

edited

Loading

kortemik commented Nov 20, 2025 •

edited

Loading

Tiihott Nov 20, 2025 •

edited

Loading