Epoch migration mode #274

elliVM · 2025-12-11T08:36:57Z

Description

Adds epoch migration mode, which can be used to modify archive reads to return only the first event in each S3 object.

The goal is to return only the first event in the S3 object. The epoch value of this first event can then be used by the user to migrate missing epoch metadata in the S3 objects.

The event message — and consequently the resulting _raw column — is left empty, as it is not required in this mode.

General

I have checked that my test files and functions have meaningful names.
I have checked that each test tests only a single behavior.
I have done happy tests.
I have tested only my own code.
I have tested at least all public methods.

Assertions

I have checked that my tests use assertions and not runtime overhead.
I have checked that my tests end in assertions.
I have checked that there is no comparison statements in assertions.
I have checked that assertions are in tests and not in helper functions.
I have checked that assertions for iterables are outside of for loops and both sides of the iteration blocks.
I have checked that assertions are not tested inside consumers.

Testing Data

I have tested algorithms and anything else with the possibility of unbound growth.
I have checked that all testing data is local and fully replaceable or reproducible or both.
I have checked that all test files are standalone.
I have checked that all test-specific fake objects and classes are in the test directory.
I have checked that my tests do not contain anything related to customers, infrastructure or users.
I have checked that my tests do not contain non-generic information.
I have checked that my tests do not do external requests and are not privately or publicly routable.

Statements

I have checked that my tests do not use throws for exceptions.
I have checked that my tests do not use try-catch statements.
I have checked that my tests do not use if-else statements.

Java

I have checked that my tests for Java uses JUnit library.
I have checked that my tests for Java uses JUnit utilities for parameters.

Other

I have only tested public behavior and not private implementation details.
I have checked that my tests are not (partially) commented out.
I have checked that hand-crafted variables in assertions are used accordingly.
I have tested Object Equality.
I have checked that I do not have any manual tests or I have a valid reason for them and I have explained it in the PR description.

Code Quality

I have checked that my code follows metrics set in Procedure: Class Metrics.
I have checked that my code follows metrics set in Procedure: Method Metrics.
I have checked that my code follows metrics set in Procedure: Object Quality.
I have checked that my code does not have any NULL values.
I have checked my code does not contain FIXME or TODO comments.

src/main/java/com/teragrep/pth_06/task/s3/RowConverter.java

elliVM · 2025-12-15T06:40:32Z

rebased, improved test to include multiple messages per s3 object to ensure only first message is selected for the result

src/test/java/com/teragrep/pth_06/EpochMigrationTest.java

src/main/java/com/teragrep/pth_06/task/s3/RowConverter.java

elliVM · 2026-01-14T13:34:26Z

rebased

elliVM · 2026-01-23T09:11:03Z

rebased

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

kortemik · 2026-01-23T11:09:36Z

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

+            rowWriter.write(8, new EventToOrigin().asUTF8StringFrom(rfc5424Frame));
+        }
+        else {
+            rowWriter.write(0, 0L);


instead of 0, use the timestamp from the object path. path has always present elements which are parseable because of the object structure

example path is

2007/10-08/sc-99-99-14-110/f17/f17.logGLOB-2007100800.log.gz

segments 2007/10-08/... are extractable with

^(?<year>\d{4})/(?<month>\d{2})-(?<day>\d{2})/(?>.*)$

time should be 00:00:00 of that day. we can then attempt extracting the hour from the path itself and assume europe/helsinki as the time. logGLOB-2007100800.log.gz includes the specific hour which could be extracted with

^(?>.*-)(?<year>\d{4})(?<month>\d{2})(?<day>\d{2})(?<hour>\d{2})(?>\..*)$

further thinking, please add this information to the metadata as well if it is available as it is much easier to do this extraction on pth_06 side.

Extraction now prioritizes hourly data when available at the end of the path; otherwise, it falls back to the date from the initial segments.

The path-derived timestamp is returned in the _time column for non-syslog events, with missing hour information defaulting to 00:00:00 in the Europe/Helsinki timezone.

please add information which one of these was used, path or hourformat

src/main/java/com/teragrep/pth_06/task/s3/EventMetadata.java

elliVM · 2026-01-27T07:14:26Z

rabased

src/main/java/com/teragrep/pth_06/task/s3/EventMetadata.java

…atches

…leanup unused objects

… JSON metadata payload for _raw column

…ct is non-syslog, use new mock objects

elliVM · 2026-01-28T06:54:01Z

rebased

elliVM · 2026-01-28T06:55:07Z

path extracted value now uses ZonedDateTime
c85ddcf

kortemik · 2026-01-28T10:10:31Z

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

+        if (LOGGER.isDebugEnabled()) {
+            LOGGER.debug("Parser syslog event <[{}]>", rfc5424Frame.toString());
+        }
+        final RFC5424Timestamp rfc5424Timestamp = new RFC5424Timestamp(rfc5424Frame.timestamp);


this uses rfc5424Frame even though the content might not be in syslog format, is there a test case for non syslog format object case?

added non syslog format testing. This instantiates the timestamp here but only calls it later inside the isSyslog block, would it be better to move it inside the block?

it would be more context coherent to move it inside it.

Tiihott

There were some missing object equality tests, methods that could be set to private and few smaller things to fix. The RowConverterImpl and EpochMigrationRowConverter classes could also use some refactoring but I think that should be done in a separate PR.

src/main/java/com/teragrep/pth_06/task/ArchiveMicroBatchInputPartitionReader.java

src/main/java/com/teragrep/pth_06/task/s3/EventToSource.java

src/test/java/com/teragrep/pth_06/EpochMigrationTest.java

src/test/java/com/teragrep/pth_06/task/s3/EventMetadataTest.java

src/main/java/com/teragrep/pth_06/task/s3/PathExtractedTimestamp.java

src/test/java/com/teragrep/pth_06/task/s3/PathExtractedTimestampTest.java

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

src/main/java/com/teragrep/pth_06/task/s3/RowConverterImpl.java

…format test for epoch migration testing

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

… it from the main JSON

kortemik · 2026-01-30T13:24:42Z

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

+                        );
+                readAttempted = true;
+                isSyslogFormat = false;
+                returnValue = true;


return value must be false, if this was not syslog format, right? there is no next frame to load then

make a test case for this

kortemik · 2026-01-30T13:26:49Z

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

+        if (!readAttempted) {
+            try {
+                boolean nextResult = rfc5424Frame.next();
+                readAttempted = true;


please move readAttempted out of the try block, if rfc5424Frame.next(); throws and this will not be set to true

make a test case for this

kortemik · 2026-01-30T13:27:53Z

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

+                boolean nextResult = rfc5424Frame.next();
+                readAttempted = true;
+                isSyslogFormat = nextResult;
+                returnValue = true;


returnValue must be set by the rfc5424Frame.next(); instead of manually setting to true.

isSyslogFormat can be set to true if rfc5424Frame.next(); does not throw.

make a test case for this

kortemik · 2026-01-30T13:30:18Z

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

+                                "ParseException at object: <[{}]>/<[{}]>\n message: <{}>", bucket, path,
+                                exception.getMessage()
+                        );
+                readAttempted = true;


setting readAttempted is not necessary in catch block when it is set before the try block

kortemik · 2026-01-30T13:33:44Z

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

+        if (LOGGER.isDebugEnabled()) {
+            LOGGER.debug("Parser syslog event <[{}]>", rfc5424Frame.toString());
+        }
+        final RFC5424Timestamp rfc5424Timestamp = new RFC5424Timestamp(rfc5424Frame.timestamp);


it would be more context coherent to move it inside it.

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java

kortemik · 2026-01-30T13:50:04Z

src/test/java/com/teragrep/pth_06/planner/MockDBNonSyslogRowImpl.java

+public final class MockDBNonSyslogRowImpl implements MockDBRow {
+
+    private final MockDBRow origin;
+    private final static Comparator<MockDBRow> COMPARATOR = Comparator


does not comprea isSyslog. that must be a private static final boolean field here to make it part of the comparison and that must be part of equals and hashCode

kortemik · 2026-01-30T13:51:38Z

src/test/java/com/teragrep/pth_06/planner/MockDBRowImpl.java


+    @Override
+    public boolean isSyslog() {
+        return true;


make this private static final boolean field and add to compareTo, equals and hashCode methods.

kortemik · 2026-01-30T13:57:29Z

src/main/java/com/teragrep/pth_06/task/s3/EventMetadata.java

+        if (isSyslogFormat) {
+            final EpochMicros epochMicros = new EpochMicros(rfc5424Timestamp);
+            timestampBuilder
+                    .add("original", String.valueOf(rfc5424Frame.timestamp))


rename original to rfc5242timestamp

elliVM self-assigned this Dec 11, 2025

elliVM linked an issue Dec 11, 2025 that may be closed by this pull request

implement epoch migration mode option #249

Open

elliVM requested a review from eemhu December 11, 2025 08:47

eemhu reviewed Dec 11, 2025

View reviewed changes

src/main/java/com/teragrep/pth_06/task/s3/RowConverter.java Outdated Show resolved Hide resolved

elliVM requested a review from eemhu December 15, 2025 06:14

elliVM force-pushed the epoch-migration-mode branch from 26276ff to cae62ff Compare December 15, 2025 06:39

eemhu reviewed Dec 15, 2025

View reviewed changes

src/test/java/com/teragrep/pth_06/EpochMigrationTest.java Outdated Show resolved Hide resolved

eemhu previously approved these changes Dec 17, 2025

View reviewed changes

elliVM added the review Issues or pull requests waiting for a review label Dec 18, 2025

elliVM requested a review from kortemik December 18, 2025 07:21

kortemik requested changes Dec 21, 2025

View reviewed changes

src/main/java/com/teragrep/pth_06/task/s3/RowConverter.java Show resolved Hide resolved

src/main/java/com/teragrep/pth_06/task/s3/RowConverter.java Outdated Show resolved Hide resolved

elliVM removed the review Issues or pull requests waiting for a review label Dec 29, 2025

elliVM dismissed eemhu’s stale review via f26de61 January 14, 2026 12:39

elliVM force-pushed the epoch-migration-mode branch from f26de61 to 28273f1 Compare January 14, 2026 13:13

elliVM requested a review from eemhu January 14, 2026 13:34

elliVM requested a review from Tiihott January 19, 2026 05:58

elliVM force-pushed the epoch-migration-mode branch from 50bb345 to 22b8e93 Compare January 23, 2026 09:10

kortemik requested changes Jan 23, 2026

View reviewed changes

elliVM force-pushed the epoch-migration-mode branch from 48c9631 to 47409b1 Compare January 27, 2026 07:13

kortemik requested changes Jan 27, 2026

View reviewed changes

src/main/java/com/teragrep/pth_06/task/s3/EventMetadata.java Show resolved Hide resolved

elliVM added 6 commits January 28, 2026 08:53

add epoch fetching from s3 WIP

0ba7ec4

implement epochMigrationMode option logic

12fceba

limit to single next() call on epoch migration mode, WIP one off in b…

bced82f

…atches

move first event only logic to RowConverter, clean up get() method, c…

dae353d

…leanup unused objects

make new objects and object ctor values final

8bc1b29

new objects became irrelevant without more responsibilities, removed

bec728e

elliVM added 10 commits January 28, 2026 08:53

remove unwanted logger message

22a1340

refactor next() to have single point of return

310634d

add testing with data containing multiple messages per s3 objects

494843a

assert epoch migration mode epoch results for all rows

8cb6d6e

restore task package

d9e3e35

create separate EpochMigrationRowConverter for migration process, add…

8596970

… JSON metadata payload for _raw column

add unit tests for new objects + cleanup and better naming

9e055d0

add equals and hashcode for new objects and testing

82ec60e

add path extracted value to metadata and to _time column when s3 obje…

311ad0f

…ct is non-syslog, use new mock objects

return ZonedDateTime instead of epoch in the path-extracted value

c85ddcf

elliVM force-pushed the epoch-migration-mode branch from c5d8feb to c85ddcf Compare January 28, 2026 06:53

kortemik requested changes Jan 28, 2026

View reviewed changes

Tiihott requested changes Jan 30, 2026

View reviewed changes

elliVM added 2 commits January 30, 2026 14:19

implement mock support for non-syslog format s3 data, add non-syslog …

64acefd

…format test for epoch migration testing

fix EpochMigrationTest.testNonSyslogEvent assertions

d001e23

elliVM commented Jan 30, 2026

View reviewed changes

src/main/java/com/teragrep/pth_06/task/s3/EpochMigrationRowConverter.java Show resolved Hide resolved

elliVM added 2 commits January 30, 2026 15:02

cleanup code, add equals and hahscode for PathExtractedTimestamp

70aeb18

use timestampJson variable in EventMetadataTest instead of collecting…

0b9e31d

… it from the main JSON

elliVM requested a review from Tiihott January 30, 2026 13:11

kortemik requested changes Jan 30, 2026

View reviewed changes

Epoch migration mode #274

Are you sure you want to change the base?

Epoch migration mode #274

Uh oh!

Conversation

elliVM commented Dec 11, 2025

Description

General

Assertions

Testing Data

Statements

Java

Other

Code Quality

Uh oh!

Uh oh!

elliVM commented Dec 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elliVM commented Jan 14, 2026

Uh oh!

elliVM commented Jan 23, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kortemik Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elliVM commented Jan 27, 2026

Uh oh!

Uh oh!

elliVM commented Jan 28, 2026

Uh oh!

elliVM commented Jan 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tiihott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kortemik Jan 30, 2026 •

edited

Loading