Skip to content

Conversation

@elliVM
Copy link
Contributor

@elliVM elliVM commented Dec 18, 2025

Description

Logtime function throws exception in case functions fail, forcing the SQL engine to be run in a throw no exceptions mode.
This change aims to allow the logtime function to use SQL null when parsing is not possible.

  • Extracts the logtime function into it's own object for easier unit testing
  • Checks if parse is available, otherwise uses SQL null

@elliVM elliVM self-assigned this Dec 18, 2025
@elliVM elliVM linked an issue Dec 18, 2025 that may be closed by this pull request
@elliVM elliVM requested a review from Tiihott December 18, 2025 14:01
Field<Long> asField() {
final String unixTimestamp = "UNIX_TIMESTAMP(STR_TO_DATE("
+ "IFNULL(SUBSTRING(REGEXP_SUBSTR({0},'^\\\\d{4}\\\\/\\\\d{2}-\\\\d{2}\\\\/[\\\\w\\\\.-]+\\\\/([^\\\\p{Z}\\\\p{C}]+?)\\\\/([^\\\\p{Z}\\\\p{C}]+)(-@)?(\\\\d+|)-(\\\\d{4}\\\\d{2}\\\\d{2}\\\\d{2})'), -10, 10), '1970010100'), '%Y%m%d%H'))";
return DSL.field("COALESCE(" + unixTimestamp + ", 0)", Long.class, pathField);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

making 1970 as a magick value is not really an option, better way would be to return a "sql null" in case extraction is not possible.

using

SELECT REGEXP_REPLACE(         '2010/01-08/sc-99-99-14-40/f17_v2/f17_v2.logGLOB-2010011601.log.gz','^\\d{4}\\/\\d{2}-\\d{2}\\/[\\w\\.-]+\\/([^\\p{Z}\\p{C}]+?)\\/([^\\p{Z}\\p{C}]+)(-@)?(\\d+|)-(\\d{4}\\d{2}\\d{2}\\d{2}).*',         '\\5' );
REGEXP_REPLACE(         '2010/01-08/sc-99-99-14-40/f17_v2/f17_v2.logGLOB-2010011601.log.gz','^\\d{4}\\/\\d{2}-\\d{2}\\/[\\w\\.-]+\\/([^\\p{Z}\\p{C}]+?)\\/([^\\p{Z}\\p{C}]+)(-@)?(\\d+|)-(\\d{4}\\d{2}\\d{2}\\d{2}).*',         '\\5' );

for example would provide better results. we should rather check that if extraction was successful and that it parses to date and then to unix date.

these can be verified with if(value regex 'valid value', unixtime(value), null) style

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactored to check for valid timestamp, otherwise returning SQL null

@elliVM elliVM force-pushed the logtime-function-avoid-excetpion branch from 5ebd3c3 to fa51ffc Compare January 7, 2026 09:52
@elliVM
Copy link
Contributor Author

elliVM commented Jan 7, 2026

rebased

Tiihott
Tiihott previously approved these changes Jan 7, 2026
Copy link
Contributor

@Tiihott Tiihott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logtime is handled as sql null when the value extraction is not possible. Tests pass and seem fine.
LGTM

@elliVM elliVM added the review Issues or pull requests waiting for a review label Jan 8, 2026
@elliVM elliVM requested a review from kortemik January 8, 2026 11:36
@elliVM elliVM force-pushed the logtime-function-avoid-excetpion branch from fa51ffc to fe78746 Compare January 23, 2026 09:16
@elliVM
Copy link
Contributor Author

elliVM commented Jan 23, 2026

rebased

@kortemik
Copy link
Member

select REGEXP_REPLACE(path, '^(?>.*-)([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})(?>\..*)$', '\\1-\\2-\\3T\\4') from logfile;

@elliVM elliVM removed the review Issues or pull requests waiting for a review label Jan 26, 2026
@elliVM elliVM requested a review from Tiihott January 27, 2026 07:11
@elliVM
Copy link
Contributor Author

elliVM commented Jan 27, 2026

select REGEXP_REPLACE(path, '^(?>.*-)([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})(?>\..*)$', '\\1-\\2-\\3T\\4') from logfile;

added

@elliVM elliVM force-pushed the logtime-function-avoid-excetpion branch from 0c6b07d to 6a0315f Compare January 27, 2026 07:12
@elliVM
Copy link
Contributor Author

elliVM commented Jan 27, 2026

rebased

@elliVM elliVM force-pushed the logtime-function-avoid-excetpion branch from 6a0315f to 8566092 Compare January 28, 2026 06:55
@elliVM
Copy link
Contributor Author

elliVM commented Jan 28, 2026

rebased

Copy link
Contributor

@Tiihott Tiihott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elliVM elliVM added the review Issues or pull requests waiting for a review label Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review Issues or pull requests waiting for a review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

avoid throwing sql exceptions in logtimeFunction

3 participants