Skip to content

Strange variable behaviour #14

New issue
New issue
Open
Open
Strange variable behaviour#14
@jbfavre

Description

@jbfavre
jbfavre
opened on Mar 3, 2022

👋
First, thanks for the tools, it's very useful.
However, I'm facing a quite strange behaviour with variables.

Let say I've logs like:

07:09:36 caps,info router21-1: selected channel 2412/20/gn(10dBm) (fixed) 
07:09:36 caps,info router21-2: selected channel 5180/20-Ceee/ac/P(23dBm) (fixed) 
07:09:37 caps,info router42-1: selected channel 2412/20/gn(10dBm) (fixed) 
07:09:38 caps,info router41-1: selected channel 2412/20/gn(10dBm) (fixed) 
07:09:38 caps,info router41-2: selected channel 5180/20-Ceee/ac/P(23dBm) (fixed) 
07:56:58 caps,info router42-1: selected channel 2412/20/gn(10dBm) (fixed) 
07:58:25 caps,info router42-1: selected channel 2412/20/gn(10dBm) (fixed) 
07:59:28 caps,info router41-1: selected channel 2412/20/gn(10dBm) (fixed) 
07:59:28 caps,info router41-2: selected channel 5180/20-Ceee/ac/P(23dBm) (fixed) 
08:00:45 caps,info router41-1: selected channel 2412/20/gn(10dBm) (fixed) 
08:00:45 caps,info router41-2: selected channel 5180/20-Ceee/ac/P(23dBm) (fixed) 
08:02:18 caps,info router42-1: selected channel 2412/20/gn(10dBm) (fixed) 
08:02:19 caps,info router41-1: selected channel 2412/20/gn(10dBm) (fixed) 
08:02:19 caps,info router41-2: selected channel 5180/20-Ceee/ac/P(23dBm) (fixed) 
08:06:35 caps,info router42-1: selected channel 2412/20/gn(10dBm) (fixed) 
08:06:35 caps,info router41-1: selected channel 2412/20/gn(10dBm) (fixed) 
08:06:35 caps,info router41-2: selected channel 5180/20-Ceee/ac/P(23dBm) (fixed)

And I want to use variables to get:

17 <time> caps,info <router_name> selected channel <wifiChannel> (fixed)

Let's start with time as in the README file:

> logmine -v "<time>:/\\d{2}:\\d{2}:\\d{2}/"
17 <time> caps,info router21-1: selected channel 2412/20/gn(10dBm) (fixed)

Cool, let's add router_name:

> logmine -v "<time>:/\\d{2}:\\d{2}:\\d{2}/" "<router_name>:/router\\d{2}-\\d{1}/"
17 <time> caps,info <router_name> selected channel 2412/20/gn(10dBm) (fixed)

Great. Now, add wifiChannel:

> logmine -v "<time>:/\\d{2}:\\d{2}:\\d{2}/" "<router_name>:/router\\d{2}-\\d{1}/" \
               "<wifiChannel>:/\\d{4}\\/\\d{2}\\/\\w{2}\\(\\d{2}cBm\\)/"
Traceback (most recent call last):
  File "/home/jbfavre/.local/bin/logmine", line 6, in <module>
    run()
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/run.py", line 47, in run
    return logmine.run(input_files)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/log_mine.py", line 18, in run
    clusters = self.processor.process(files)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/processor.py", line 25, in process
    return self.process_pipe()
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/processor.py", line 84, in process_pipe
    clusterer = Clusterer(**self.cluster_config)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/clusterer.py", line 18, in __init__
    self.preprocessor = Preprocessor(variables)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/preprocessor.py", line 16, in __init__
    self.variables = [
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/preprocessor.py", line 17, in <listcomp>
    (tuple[0], re.compile(tuple[1])) for tuple in parsed_variables
  File "/usr/lib/python3.9/re.py", line 252, in compile
    return _compile(pattern, flags)
  File "/usr/lib/python3.9/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.9/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.9/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.9/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/lib/python3.9/sre_parse.py", line 644, in _parse
    if not sourcematch("}"):
  File "/usr/lib/python3.9/sre_parse.py", line 251, in match
    self.__next()
  File "/usr/lib/python3.9/sre_parse.py", line 245, in __next
    raise error("bad escape (end of pattern)",
re.error: bad escape (end of pattern) at position 5

Wooops… let's try "<wifiChannel>:/\\d{4}\/\\d{2}\/\\w{2}\\(\\d{2}cBm\\)/" (only one \ escape before internal /)

logmine -v "<time>:/\\d{2}:\\d{2}:\\d{2}/" \
               "<router_name>:/router\\d{2}-\\d{1}/" \
               "<wifiChannel>:/\\d{4}\/\\d{2}\/\\w{2}\\(\\d{2}cBm\\)/"
Traceback (most recent call last):
  File "/home/jbfavre/.local/bin/logmine", line 6, in <module>
    run()
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/run.py", line 47, in run
    return logmine.run(input_files)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/log_mine.py", line 18, in run
    clusters = self.processor.process(files)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/processor.py", line 25, in process
    return self.process_pipe()
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/processor.py", line 84, in process_pipe
    clusterer = Clusterer(**self.cluster_config)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/clusterer.py", line 18, in __init__
    self.preprocessor = Preprocessor(variables)
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/preprocessor.py", line 16, in __init__
    self.variables = [
  File "/home/jbfavre/.local/lib/python3.9/site-packages/logmine_pkg/preprocessor.py", line 17, in <listcomp>
    (tuple[0], re.compile(tuple[1])) for tuple in parsed_variables
  File "/usr/lib/python3.9/re.py", line 252, in compile
    return _compile(pattern, flags)
  File "/usr/lib/python3.9/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.9/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.9/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.9/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/lib/python3.9/sre_parse.py", line 644, in _parse
    if not sourcematch("}"):
  File "/usr/lib/python3.9/sre_parse.py", line 251, in match
    self.__next()
  File "/usr/lib/python3.9/sre_parse.py", line 245, in __next
    raise error("bad escape (end of pattern)",
re.error: bad escape (end of pattern) at position 5

Nope, doesn't work either…
Let's go without any escape before internal /:

> logmine -v "<time>:/\\d{2}:\\d{2}:\\d{2}/" \
               "<router_name>:/router\\d{2}-\\d{1}/" \
               "<wifiChannel>:/\\d{4}/\\d{2}/\\w{2}\\(\\d{2}cBm\\)/"
17 <time> caps,info <router_name> selected channel <wifiChannel> (fixed)

Cool, it works !
But…wait a minute… if I do not escape internal /, how does Python knows it's not the end of the regexp ?
More: I'm looking for 4 digits followed by / then 2 digits followed by / then 2 letters.
So, how is it that 5180/20-Ceee/ac/P could ever match and get replaced?

Thanks in advance for the explanation, I'm stuck here 🤔

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions