Feature/2777 multiline strings

[karimnabli]

Gauge Multiline String Feature Implementation

Overview

This implementation of native multiline string support for Gauge, addressing a long-standing feature request. The implementation allows users to include multiline content in step arguments using triple-quote syntax (""").
The token (""") must be in a single line, to define that start and the end of the multiline.

Changes Made Across Three Projects

1. Gauge Core (gauge repository)

parser/lex.go

• Added multiline detection: New extractMultilineContent() function that identifies and captures content between """ delimiters
• Enhanced tokenization: Modified GenerateTokens() to handle multiline content as implicit step arguments
• Scanner integration: Uses both scanner-based parsing and line indexing for multiline lookahead

parser/specparser.go

• Implicit argument handling: Updated CreateStepUsingLookup() to detect when a step has multiline content without explicit parameter placeholders
• Fragment creation: Added logic to create proper fragments for multiline parameters
• Parameter processing: Distinguishes between implicit multiline strings and classic multiline steps

parser/stepParser.go

• Parameter type extraction: Enhanced extractStepValueAndParameterTypes() to handle multiline content
• Step processing: Modified processStep() to skip parameter processing for implicit multiline strings
• Validation: Added logic to validate multiline step arguments

gauge/arg.go

• New parameter type: Added MultilineString constant to support the new parameter type
• Type handling: Extended argument type system to recognize multiline content

gauge/protoConverters.go

• Protocol support: Added Multiline_String parameter type in convertToProtoParameter() and makeParameterCopy()
• Fragment conversion: Ensures multiline strings are properly converted to protocol buffer format

2. Gauge Protocol (gauge-proto repository)

• New parameter type: Added Parameter_Multiline_String to the protocol buffer definition
• Extended message handling: Updated protocol definitions to support multiline string parameters

3. HTML Report Plugin (html-report repository)

• Multiline rendering: Enhanced toFragments() function to detect and properly render multiline strings in HTML reports
• Fragment handling: Added multilineFragmentKind support with proper content preservation
• Cross-platform compatibility: Consistent rendering across Windows, Linux, and macOS

3.1 Spec level

3.1.1 Multiline icon representation

3.1.2 Multiline hover

3.1.3 Multiline expansion

3.1 Concept level

Similar to the spec one

Technical Implementation Details

Parsing Logic

1. Detection: Lexer identifies """ following a step as multiline block start
2. Capture: Content between """ delimiters is captured as a single string, preserving all formatting
3. Tokenization: Multiline content is attached to the preceding step as an implicit argument in stepToken.Args[0]
4. Parameter Processing: Steps with multiline arguments skip regular parameter processing

Parameter Handling

• Implicit arguments: Multiline strings are treated as implicit parameters (no {} placeholders needed)
• Protocol mapping: They're mapped to gauge_messages.Parameter_Multiline_String parameter type
• Content preservation: Whitespace, indentation, and special characters are preserved exactly as written
• Fragment creation: Creates proper gauge_messages.Fragment with parameter type for rendering

Key Design Decisions

1. Scanner-based approach: Maintained the original scanner-based parsing while adding multiline support
2. Implicit arguments: No need for parameter placeholders in spec files for multiline strings
3. Backward compatibility: Classic multiline step merging continues to work with parameter processing
4. State management: Proper handling of parser state during multiline content extraction

Usage Examples

Before (Workarounds)

* Step with JSON data
    |json                       |
    |{                         |
    |  "name": "Gauge",        |
    |  "type": "Testing"       |
    |}                         |

After (Native Support)

* Step with JSON data
"""
{
  "name": "Gauge", 
  "type": "Testing Framework"
}
"""

Multiple Use Cases

* Step with XML data
"""
<user>
  <name>John</name>
  <age>30</age>
</user>
"""

* Step with code snippet
 """
function hello() {
  console.log("Hello, Gauge!");
  return true;
}
"""

* Step with plain text
 """
This is a long description
that spans multiple lines
with preserved formatting.
"""

Benefits Achieved

1. Improved Readability: No more misusing tables for non-tabular data
2. Semantic Correctness: Proper representation of multiline content as strings
3. Consistency: Aligns with common patterns in modern programming languages and markup formats
4. Flexibility: Supports JSON, XML, YAML, code snippets, configuration files, and any text content
5. Preservation: Exact whitespace, indentation, and formatting maintained
6. Developer Experience: Natural syntax that developers expect from modern tools

Backward Compatibility

• Fully backward compatible: All existing specs continue to work unchanged
• No breaking changes: Classic multiline step merging with parameter processing still works
• Progressive enhancement: New functionality available when """ syntax is used
• Test preservation: All existing 205 parser tests pass, plus new multiline tests

Testing Coverage

The implementation includes comprehensive tests covering:

Core Functionality

• Basic multiline string extraction
• Empty multiline strings
• Multiline strings with special characters

Edge Cases

• Mixed classic multiline steps and implicit multiline strings
• Protocol buffer conversion for multiline parameters
• Parameter copy functionality
• Content with braces, quotes, and special symbols

Integration

• Full parsing pipeline from spec text to executable steps
• Protocol buffer serialization/deserialization
• Fragment creation and parameter handling

Test Status:

• ✅ Gauge Core: Comprehensive unit tests added for multiline parsing
• ✅ HTML Report: Full test coverage for multiline fragment rendering
• 🔄 gauge-proto: Protocol changes tested indirectly through integration tests

Protocol Impact

• New parameter type: Added Parameter_Multiline_String to the protocol buffer definition
• Extended conversion: Updated convertToProtoParameter() and makeParameterCopy() to handle the new type
• Maintained compatibility: All existing parameter types continue to work unchanged
• Runner support: Uses established extension patterns that existing runners can support

Performance Considerations

• Efficient parsing: Multiline content is processed in a single pass
• Memory efficient: Content is stored as strings without unnecessary copies
• Minimal overhead: Only activates multiline parsing when """ is detected

Review Notes

• All existing tests pass (205 parser tests + new multiline tests)
• Backward compatibility verified
• No breaking changes to existing APIs
• Follows existing code patterns and conventions
• Cross-platform compatibility confirmed

Next Steps

• Gauge-proto PR - [ ] Html-reprot PR

This implementation finally provides native multiline string support in Gauge while maintaining the framework's commitment to readability, usability, and backward compatibility. The feature aligns Gauge with modern testing tool expectations and eliminates the need for workarounds when dealing with structured data or code snippets in specifications.

[karimnabli]

my bad: i have pushed the following line in the go.mod
replace github.com/getgauge/gauge-proto/go/gauge_messages => ../gauge-proto/go/gauge_messages
as I was developing locally and referencing the build in my machine.

[chadlwilson]

What’s going on here? Is this releasable and backwards compat?

I haven’t followed what’s happening, but want to release other fixes.

[karimnabli]

Yes it is.

[chadlwilson]

OK, released as 4.3.4 since I think this PR was forgotten to be marked as a release candidate, despite having the version bumped.