[chore] Drop py27 support and update lark dependency from lark-parser to lark>=1.3.1 #82

Mikaayenson · 2025-11-14T17:45:23Z

Summary

Resolves #81

This PR updates the EQL library to use the modern lark package (v1.3.1) instead of the deprecated lark-parser (v0.12.0), fixes compatibility issues with the new Lark version, and drops support for Python 2.7 and Python < 3.8.

Changes

Dependency Updates

Migrated from lark-parser~=0.12.0 to lark>=1.3.1
- The lark-parser package has been deprecated and replaced by lark
- Updated all imports and usage to work with the new package

Compatibility Fixes for Lark 1.3.1

Lark 1.3.1 handles optional grammar rules differently than lark-parser 0.12.0. When optional rules (marked with [...] in the grammar) are empty, Lark 1.3.1 includes them in the parse tree as None values, whereas the old version would omit them entirely. This required fixes to:

Fixed macro parameter parsing: Macros with empty parameter lists (e.g., macro TRUE()) now parse correctly. The implementation filters out None values before visiting parameter nodes, resulting in cleaner, more maintainable code.
Fixed pipe argument parsing: Pipes with no arguments (e.g., | count) now handle empty argument lists correctly using the or [] pattern for cleaner null handling.
Fixed Schema.current(): Ensures it always returns a valid Schema object, never None, by falling back to EMPTY_SCHEMA when both the stack and default are None.
Fixed KvTree position information: Added compatibility properties (line, end_line, column, end_column) to access position information from Lark 1.3.1's meta attribute, with fallback to token-based position information.

Python Version Support

BREAKING: Dropped support for Python 2.7 and Python < 3.8
Minimum required Python version is now 3.8
Removed all Python 2.7 compatibility code and comments:
- Removed encoding checks in eql/ast.py
- Simplified input function in eql/shell.py (removed raw_input fallback)
- Removed Python 2/3 compatibility type checking in eql/utils.py
- Removed Python 2 support code from tests
Updated documentation to reflect Python 3.8+ requirement

CI/CD Updates

Updated GitHub Actions workflows:
- Updated actions/checkout from v2 to v5
- Updated actions/setup-python from v4 to v5
- Removed Python 2.7 workflow file (.github/workflows/pythonpackage27.yml)
- Updated Python version matrix to test 3.8, 3.9, 3.10, 3.11, and 3.12
- Added setuptools to dependency installation (required for Python 3.12)
- Removed deprecation warning suppression (no longer needed)

Testing

All 616 unit tests pass ✅
Verified compatibility with endpoint-rules and detection-rules repositories:
- eql 0.9.20 installs correctly in both repos
- All EQL parsing functionality works correctly
- Empty macro parameters work correctly
- Empty pipe arguments work correctly
- Rule conversion and validation tests pass
All linting checks pass ✅

Technical Details

Why the Changes Were Needed

Lark 1.3.1 handles optional grammar rules differently than lark-parser 0.12.0. When optional rules (marked with [...] in the grammar) are empty, Lark 1.3.1 includes them in the parse tree as None values, whereas the old version would omit them entirely. This required fixes to:

Macro parsing: Empty parameter lists were being parsed as [None] instead of []
Pipe parsing: Empty argument lists were being parsed as [None] instead of []
Schema handling: Added fallback to ensure Schema.current() never returns None
Position information: Lark 1.3.1 stores position information in node.meta instead of directly on the node

Implementation Approach

The fixes for Lark 1.3.1 compatibility follow a consistent pattern:

Filter None before visiting: Instead of visiting nodes that may contain None and filtering afterward, we filter None values from the children list before calling visit(). This results in cleaner code and better performance.
Use or [] pattern: For cases where visit() may return None, we use the or [] pattern to provide a default empty list, making the code more Pythonic and concise.
Position information compatibility: The KvTree class now checks for position information in Lark 1.3.1's meta attribute first, then falls back to inspecting child tokens, ensuring compatibility with both old and new Lark versions.

Files Changed

Core Changes:

eql/parser.py: Fixed macro and pipe parsing, added KvTree compatibility properties
eql/schema.py: Fixed Schema.current() to always return a valid schema
setup.py: Updated dependencies, removed Python 2.7 support, fixed deprecated APIs
eql/__init__.py: Updated version to 0.9.20

Cleanup:

eql/ast.py: Removed Python 2.7 encoding check
eql/shell.py: Simplified input function
eql/utils.py: Removed Python 2/3 compatibility code
tests/test_python_engine.py: Removed Python 2 support code
tests/test_cli.py: Updated mock import to use unittest.mock

Documentation:

CHANGELOG.md: Added comprehensive changelog entry
README.md: Updated Python version requirement and version example
docs/index.rst: Updated Python version requirement
docs/cli.rst: Removed Python 2.7 notes

CI/CD:

.github/workflows/pythonpackage.yml: Updated to Python 3.8+, newer actions, removed deprecation warnings
.github/workflows/pythonpackage27.yml: Deleted (Python 2.7 no longer supported)

Configuration:

setup.cfg: Added linting exemptions for style issues (W504, E241, etc.)

Testing

EQL Unit Tests

python setup.py -q test
# Result: 616 passed in ~8 minutes

Integration Testing

✅ Verified eql 0.9.20 installs correctly in endpoint-rules
✅ Verified eql 0.9.20 installs correctly in detection-rules
✅ All EQL parsing features work correctly:
- Basic queries
- Queries with pipes (including empty arguments)
- Sequence queries
- Queries with functions
- Queries with macros (including empty parameter macros)
✅ Rule conversion tests pass in endpoint-rules
✅ All required parser classes available for detection-rules (KvTree, LarkToEQL, NodeInfo, TypeHint)

Breaking Changes

Important

Python 2.7 and Python < 3.8 are no longer supported
Users must upgrade to Python 3.8 or higher

Migration Guide

If you're using Python 2.7 or Python < 3.8, you must upgrade to Python 3.8+ before using eql 0.9.20.

No code changes are required for existing Python 3.8+ users - the API remains the same.

Related Issues

Migrates from deprecated lark-parser package to lark
Fixes compatibility with Lark 1.3.1's handling of optional grammar rules
Aligns with modern Python development practices by dropping EOL Python versions

- Migrated from deprecated lark-parser package to lark package - Updated dependency in setup.py from lark-parser~=0.12.0 to lark>=1.3.1 - Bumped version to 0.9.20 - Updated CHANGELOG.md with migration details The code already uses the correct import paths (from lark import ...), so no code changes were required. All functionality tested and verified.

setup.cfg

eric-forte-elastic · 2025-11-14T21:12:06Z

Not seeing any reason to keep Python 2 support yet, looks like that will not cause issues. OpenCTI is on py3 as well as Eland and Geneve (requirements are pinned anyway so it would not break their build for someone wanting to use py2).

…te 'line'

…to update_lark_dependency

eric-forte-elastic · 2025-11-14T21:24:49Z

Just checking, there are still some references to Python 2.7 in the current state of the code. For instance: ast.py and in docs too.

eql/parser.py

cavokz · 2025-11-17T10:21:01Z

I run the geneve tests with this preliminary eql branch, everything looks fine. Such tests are quite picky with the reproducibility of the data generation, if they do not complain it means that data that was generated (and therefore the corresponding eql rules ASTs) did not change of a single bit.

eql/__init__.py

eql/parser.py

CHANGELOG.md

eric-forte-elastic · 2025-11-17T18:07:20Z

Nit. May want to remove Python 2.7 references from docs/_static/eql-crash-course.slides.html and CONTRIBUTING.md.

eric-forte-elastic · 2025-11-17T18:31:28Z

Unit testing results 🟢

Details

========================================================================================================= test session starts ==========================================================================================================
platform linux -- Python 3.12.12, pytest-9.0.1, pluggy-1.6.0 -- /tmp/eql/env/eql-build/bin/python
cachedir: .pytest_cache
rootdir: /tmp/eql
configfile: setup.cfg
plugins: cov-7.0.0, typeguard-4.3.0
collected 616 items                                                                                                                                                                                                                    


------------------------------------------------------------------------------------------------ generated xml file: /tmp/eql/junit.xml ------------------------------------------------------------------------------------------------
============================================================================================================ tests coverage ============================================================================================================
___________________________________________________________________________________________ coverage: platform linux, python 3.12.12-final-0 ___________________________________________________________________________________________

Name                  Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------
eql/__init__.py          16     16      0      0     0%   2-70
eql/ast.py              691    331    204     20    58%   2-79, 85-99, 103, 107, 111, 127, 132, 136, 140, 146-152, 157, 161, 170-179, 185-187, 208, 229, 234-241, 247-248, 261-262, 267-277, 281-286, 290-294, 298-303, 307-325, 331-332, 343, 347-357, 361, 374-376, 381-387, 392, 396, 400-410, 421, 424->426, 428-429, 433-434, 442, 445, 456-464, 474-480, 484, 487, 493, 495->500, 502, 513-532, 542-555, 561, 565-566, 573-574, 579-592, 600-608, 610->618, 612->618, 620, 627-634, 639-646, 651-657, 663, 667-671, 685, 688->713, 698->713, 706->713, 710->713, 715, 718->730, 727->730, 732-748, 752, 765-771, 778, 781, 792-799, 806-821, 827-829, 833-853, 862, 870-875, 882, 886-891, 902-903, 906->911, 909->911, 913, 926-931, 940, 949-957, 964-978, 989-997, 1010-1017, 1022-1034, 1039, 1045-1050, 1059, 1064-1069, 1078-1079, 1083-1084, 1088-1096, 1100-1105, 1114-1120, 1126-1137, 1146, 1151-1166, 1178, 1213, 1221-1224, 1250, 1255, 1262->exit, 1264, 1267, 1279, 1288-1290
eql/build.py             70     35     18      4    49%   2-72, 79, 95, 105, 107, 123, 130->133
eql/engine.py           858    137    358     21    85%   2-23, 27, 36-39, 67, 79-83, 89, 93, 99, 101->104, 105-109, 117-120, 124-125, 137-138, 145->exit, 152-153, 160->exit, 165, 195-196, 204, 230, 239-240, 247, 256, 266, 276, 280, 288, 294, 327, 343, 349-350, 354, 372, 389, 412, 424, 433, 437, 441, 459, 462-463, 468, 490, 513, 523-524, 531, 554-555, 568, 572, 583, 599, 612, 632, 647, 661-662, 687, 693, 703->707, 715-716, 732->735, 746-747, 751, 761, 808, 855, 894, 906, 937, 948-966, 980, 1029, 1038, 1071, 1086, 1119, 1147, 1151, 1155, 1160, 1165, 1170, 1176, 1194, 1199, 1208, 1217, 1238->exit, 1241, 1252, 1263, 1267-1272, 1277
eql/errors.py            22     11      2      0    54%   2-29, 45-57
eql/etc/__init__.py       9      8      0      0    11%   2-8, 13-17
eql/events.py            23     10      6      1    62%   2-12, 24->29, 31, 37-41
eql/functions.py        361    218    102     14    49%   2-22, 27-36, 41-49, 53-56, 59-68, 74-106, 108->exit, 112-122, 130-133, 147-176, 192-202, 207->214, 218-219, 221->231, 225->224, 233-234, 240, 263-273, 280-287, 295-304, 306->exit, 310-321, 333-342, 348-363, 374-375, 389-393, 396, 403-417, 419->exit, 423-430, 432->exit, 436-454, 460-469, 473-486, 492-499, 501->exit, 505-516, 518->exit, 523->exit, 527-536, 545-555, 573-574, 586-587, 593-594, 603
eql/loader.py            16     10      2      1    39%   2-13, 16->18, 21, 29-31
eql/main.py             100     31     20      4    66%   35-37, 44-52, 64, 72-74, 82-88, 93-104, 147, 149
eql/optimizer.py        120     31     58      3    81%   2-14, 19, 26-27, 41-42, 50-51, 59-60, 80-81, 89-90, 94-95, 120-121, 136-137, 140, 148, 151, 161-162, 178
eql/parser.py           943    223    460     53    78%   2-76, 81, 87, 91, 95-96, 99-100, 105-111, 116-122, 127-133, 138-147, 176, 201-202, 209-210, 220, 227-228, 232, 250, 278, 297->289, 303, 312-320, 327, 337, 363, 367, 373->exit, 388-393, 397, 401, 406, 444, 460, 471, 493->509, 522, 526, 535, 539, 543, 550, 555, 559, 571, 580, 583, 603, 611-612, 629, 637, 646, 662, 666, 682, 692, 707, 758->769, 771, 779, 794-797, 804, 809, 813, 817, 821->826, 823, 831, 837, 847, 853, 860, 874->882, 879->882, 884, 971->943, 996, 1015, 1018, 1033->1041, 1045, 1049, 1056, 1076, 1080, 1085, 1087, 1094, 1097, 1153, 1161, 1165, 1173, 1179, 1220, 1244-1289, 1296, 1300, 1304, 1307, 1334, 1339, 1344, 1346, 1353->1364, 1361-1362, 1369, 1379-1385, 1422->1425, 1439-1477, 1494, 1508, 1520, 1525, 1530, 1545, 1553->1555, 1558, 1571, 1585, 1608-1611, 1614, 1617, 1623-1625, 1630-1634
eql/pipes.py             75     51     16      5    38%   2-19, 24-39, 60-69, 73, 76-77, 80, 84-93, 97, 100-101, 104, 108-125, 129, 146-153
eql/schema.py           217     85    148     29    63%   2-40, 48, 52, 62, 64-66, 69, 74, 79, 82-83, 94-95, 103, 106-110, 115, 127, 133->exit, 135, 139, 141, 148-153, 166-167, 174, 180, 182-200, 206-208, 221, 232-233, 236, 239-240, 247, 251-252, 258->256, 263, 271-275, 278, 282, 286, 287->exit, 290-291, 298, 306-308
eql/signatures.py        15      7      6      0    67%   2-13
eql/transpilers.py      108     82     14      1    25%   2-52, 59-60, 64-88, 100, 110, 116, 122-129, 137-182
eql/types.py             73     35     24      3    61%   2-18, 22-38, 47, 51-59, 74, 84, 90, 103, 113, 118, 127-133
eql/utils.py            250     95    110     23    65%   2-34, 39, 44, 49, 54, 65, 72, 75, 79, 84, 89, 99, 103-104, 110-113, 136-140, 154-157, 167-174, 178->177, 182, 191->197, 194-195, 198-200, 209, 219-220, 226, 242, 253, 256, 269, 278, 281, 286, 292-295, 333, 342, 345, 350, 359, 362, 367, 376, 384, 396-404, 410-411, 415-416, 420-421, 425-426, 433, 436, 443, 449
eql/walkers.py          174     60     58      7    70%   2-28, 38, 44, 57-60, 66, 74-83, 86, 96, 99, 102, 105, 108, 111, 119, 122, 124->127, 146-157, 174, 181, 186, 193->exit, 198-201, 206, 211-214, 222->exit, 229-237, 248, 258-261, 267, 273
-----------------------------------------------------------------
TOTAL                  4141   1476   1606    189    69%
Coverage HTML written to dir htmlcov
Coverage XML written to file coverage.xml
=================================================================================================== 616 passed in 853.89s (0:14:13) ====================================================================================================

Search CLI functions 🟢

Details

eql> search
 ..> any where host.os.type == "linux" and event.dataset == "kubernetes.audit_logs" and kubernetes.audit.verb == "delete"
 ..> and kubernetes.audit.objectRef.resource == "events" and kubernetes.audit.stage == "ResponseComplete"
 ..> 
 eql>

eql/parser.py

eric-forte-elastic

🟢 Manual review, spot testing queries 🟢 , new properties look good. LGTM! 👍

Nit in the changelog should there be a reference to the new properties of KvTree? Not sure if necessary or not.

Mikaayenson · 2025-11-17T20:23:58Z

@cavokz want to do a final test before closing this out?

cavokz

@cavokz want to do a final test before closing this out?

All Geneve's tests passed fine. For me it's a go.

cavokz · 2025-11-18T16:34:22Z

👏

Mikaayenson requested a review from eric-forte-elastic November 14, 2025 17:47

Mikaayenson self-assigned this Nov 14, 2025

Mikaayenson added the enhancement New feature or request label Nov 14, 2025

Mikaayenson added 9 commits November 14, 2025 11:55

update ci

7c219b5

add setuptools

fc7e5a4

update linting requirements

3253ae6

support flake8 api 3.x for linting

5dd4323

bump min req to py 3.8

6d025fb

lint

6c541c5

more lint updates

d96bffa

update tests

ab63c86

readd dependencies

ca738c6

Mikaayenson commented Nov 14, 2025

View reviewed changes

setup.cfg Outdated Show resolved Hide resolved

Update setup.cfg

8a729ee

Mikaayenson added 2 commits November 14, 2025 15:18

Add support to address AttributeError: 'KvTree' object has no attribu…

3b0fc30

…te 'line'

Merge branch 'update_lark_dependency' of github.com:endgameinc/eql in…

223ceb6

…to update_lark_dependency

Mikaayenson changed the title ~~Update lark dependency from lark-parser to lark>=1.3.1~~ WIP: Update lark dependency from lark-parser to lark>=1.3.1 Nov 14, 2025

Mikaayenson added 2 commits November 14, 2025 16:16

additional support for lark 1.3.1

c4b3e6e

remove obvious py27 support

4bca725

Mikaayenson commented Nov 15, 2025

View reviewed changes

eql/parser.py Show resolved Hide resolved

Mikaayenson changed the title ~~WIP: Update lark dependency from lark-parser to lark>=1.3.1~~ [FR] Update lark dependency from lark-parser to lark>=1.3.1 Nov 17, 2025

Mikaayenson changed the title ~~[FR] Update lark dependency from lark-parser to lark>=1.3.1~~ [chore] Update lark dependency from lark-parser to lark>=1.3.1 Nov 17, 2025

terrancedejesus reviewed Nov 17, 2025

View reviewed changes

eql/__init__.py Outdated Show resolved Hide resolved

terrancedejesus reviewed Nov 17, 2025

View reviewed changes

eql/parser.py Outdated Show resolved Hide resolved

terrancedejesus approved these changes Nov 17, 2025

View reviewed changes

code cleanup

2c69216

Mikaayenson commented Nov 17, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Update CHANGELOG.md

c072185

Mikaayenson changed the title ~~[chore] Update lark dependency from lark-parser to lark>=1.3.1~~ [chore] Drop py27 support and update lark dependency from lark-parser to lark>=1.3.1 Nov 17, 2025

update docs

44754ae

eric-forte-elastic reviewed Nov 17, 2025

View reviewed changes

eql/parser.py Show resolved Hide resolved

eric-forte-elastic approved these changes Nov 17, 2025

View reviewed changes

cavokz approved these changes Nov 18, 2025

View reviewed changes

Mikaayenson merged commit 7d8a09c into master Nov 18, 2025
5 checks passed

[chore] Drop py27 support and update lark dependency from lark-parser to lark>=1.3.1 #82

[chore] Drop py27 support and update lark dependency from lark-parser to lark>=1.3.1 #82

Uh oh!

Conversation

Mikaayenson commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Dependency Updates

Compatibility Fixes for Lark 1.3.1

Python Version Support

CI/CD Updates

Testing

Technical Details

Why the Changes Were Needed

Implementation Approach

Files Changed

Testing

EQL Unit Tests

Integration Testing

Breaking Changes

Migration Guide

Related Issues

Uh oh!

Uh oh!

eric-forte-elastic commented Nov 14, 2025

Uh oh!

eric-forte-elastic commented Nov 14, 2025

Uh oh!

Uh oh!

cavokz commented Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eric-forte-elastic commented Nov 17, 2025

Uh oh!

eric-forte-elastic commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

eric-forte-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

Mikaayenson commented Nov 17, 2025

Uh oh!

cavokz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cavokz commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Mikaayenson commented Nov 14, 2025 •

edited

Loading

eric-forte-elastic commented Nov 17, 2025 •

edited

Loading