protosearch reference

This document describes the complete protosearch API.

API

protosearch exposes two extensions.

Extension	Message	Description
`protosearch.field`	`protosearch.Field`	Manage field configuration
`protosearch.index`	`protosearch.Index`	Manage index configuration

`field`

protosearch.Field is a message with the following fields:

Field	Type	Description
`name`	`string`	Rename a field in the mapping.
`mapping`	`protosearch.FieldMapping`	Define mapping field parameters.
`target`	`repeated protosearch.Target`	Configure a literal mapping for a specific target.

The protoc-gen-protosearch plugin compiles these message options to a JSON file containing the document mapping.

The simplest way to annotate a field is:

string uid = 1 [(protosearch.field) = {}];

This will generate a basic field mapping with no parameters except for type. See type inference below.

If you do not annotate a protobuf field with (protosearch.field) options, it will be excluded from the mapping.

`name`

The name field lets you rename a protobuf field in the compiled mapping.

string uid = 1 [(protosearch.field).name = "user_uid"];

{
  "properties": {
    "user_uid": {
      "type": "keyword"
    }
  }
}

`mapping`

In most cases, you will need to use mapping to define field parameters. FieldMapping supports the most common mapping parameters with one important difference:

It does not support properties, because the plugin supports defining object and nested fields as protobuf message fields.

Certain fields, namely dynamic, index_options, and term_vector, are enums. All provide a default UNSPECIFIED value. The plugin will not output an enum parameter if it has the default UNSPECIFIED value.

If you need to generate a parameter that is not in this list, see target below.

Field	Type	Description
`type`	`string`	The field type. If omitted, the plugin infers the type from the protobuf field type.
`analyzer`	`string`	Analyzer used at index time. Applies to `text` fields.
`boost`	`double`	Boost a field's score at index time.
`coerce`	`bool`	Whether to coerce values to the declared mapping type. Applies to numeric and date fields.
`copy_to`	`repeated string`	Copy this field's value to the named field.
`doc_values`	`bool`	Whether to store doc values for sorting and aggregation.
`dynamic`	`protosearch.Dynamic`	How to handle unknown subfields. Applies to `object` fields.
`eager_global_ordinals`	`bool`	Whether to load global ordinals at refresh time.
`enabled`	`bool`	Whether to parse and index the field.
`fielddata`	`bool`	Whether to use in-memory fielddata for sorting and aggregations. Applies to `text` fields.
`fields`	`map<string, FieldMapping>`	A multi-field mapping.
`format`	`string`	The date format. Applies to `date` and `date_nanos` fields.
`ignore_above`	`int32`	Do not index strings longer than this length. Applies to `keyword` fields.
`ignore_malformed`	`bool`	Ignore invalid values instead of rejecting the document.
`index_options`	`protosearch.IndexOptions`	Which information to store in the index. Applies to `text` fields.
`index_phrases`	`bool`	Whether to index bigrams separately. Applies to `text` fields.
`index_prefixes`	`protosearch.IndexPrefixes`	Index term prefixes to speed up prefix queries. Applies to `text` fields.
`index`	`bool`	Whether to index the field.
`meta`	`map<string, string>`	Metadata about the field.
`normalizer`	`string`	Normalize `keyword` fields with this normalizer.
`norms`	`bool`	Whether to store field length norms for scoring.
`null_value`	`google.protobuf.Value`	Replace explicit `null` values with this value at index time.
`position_increment_gap`	`int32`	A gap inserted between elements in an array to prevent spurious matches. Applies to `text` fields.
`search_analyzer`	`string`	Analyzer used at search time.
`similarity`	`string`	The scoring algorithm.
`store`	`bool`	Whether to store this field separately from `_source`.
`subobjects`	`bool`	Whether dotted field names are interpreted as nested subobjects.
`term_vector`	`protosearch.TermVector`	Whether to store term vectors.

`dynamic`

protosearch.Dynamic is an enum with the following values:

DYNAMIC_TRUE
DYNAMIC_FALSE
DYNAMIC_STRICT
DYNAMIC_RUNTIME

`index_options`

protosearch.IndexOptions is an enum with the following values:

INDEX_OPTIONS_DOCS
INDEX_OPTIONS_FREQS
INDEX_OPTIONS_POSITIONS
INDEX_OPTIONS_OFFSETS

`index_prefixes`

protosearch.IndexPrefixes is a message with the following fields:

Field	Type	Description
`min_chars`	`int32`	Minimum prefix length to index.
`max_chars`	`int32`	Maximum prefix length to index.

`term_vector`

protosearch.TermVector is an enum with the following values:

TERM_VECTOR_NO
TERM_VECTOR_YES
TERM_VECTOR_WITH_POSITIONS
TERM_VECTOR_WITH_OFFSETS
TERM_VECTOR_WITH_POSITIONS_OFFSETS
TERM_VECTOR_WITH_POSITIONS_PAYLOADS
TERM_VECTOR_WITH_POSITIONS_OFFSETS_PAYLOADS

`target`

The target field gives you complete control over how a protobuf field compiles to a mapping property.

It is a message with the following fields:

Field	Type	Description
`label`	`string`	A human-readable label used to target that particular mapping with `--protosearch_opt=target=<label>`.
`json`	`string`	A literal JSON string containing the mapping.

Use this to define more complex mapping types, or specify parameters that are not supported in FieldMapping. You can also use this to define mappings for different clusters or vendors. You can specify this field more than once.

For example, you might want to represent a Point object as a geo_point in Elasticsearch and an xy_point in OpenSearch. You can create targets for both mappings:

Point origin = 1 [(protosearch.field) = {
  target: {
    label: "elasticsearch"
    json: '{"type": "point"}'
  }
  target: {
    label: "opensearch"
    json: '{"type": "xy_point"}'
  }
}];

With --protosearch_opt=target=elasticsearch:

{
  "properties": {
    "origin": {
      "type": "point"
    }
  }
}

With --protosearch_opt=target=opensearch:

{
  "properties": {
    "origin": {
      "type": "xy_point"
    }
  }
}

If target does not match an existing label, the plugin falls back on the common mapping parameters.

`index`

protosearch.Index is a message with the following fields:

Field	Type	Description
`mapping`	`protosearch.IndexMapping`	Define index mapping parameters.

`mapping`

protosearch.IndexMapping is a message with the following fields:

Field	Type	Description
`date_detection`	`bool`	Whether to detect date strings as `date` fields.
`dynamic`	`protosearch.Dynamic`	How to handle unknown fields.
`dynamic_date_formats`	`repeated string`	Date formats to use for dynamic date detection.
`_field_names`	`protosearch.IndexFieldNames`	Controls the `_field_names` metadata field.
`_meta`	`map<string, string>`	Metadata about the index mapping.
`numeric_detection`	`bool`	Whether to detect numeric strings as numeric fields.
`_routing`	`protosearch.IndexRouting`	Controls the `_routing` metadata field.
`_source`	`protosearch.IndexSource`	Controls the `_source` metadata field.

dynamic uses the same protosearch.Dynamic enum as field.mapping.dynamic.

`_field_names`

protosearch.IndexFieldNames is a message with the following fields:

Field	Type	Description
`enabled`	`bool`	Whether to enable the `_field_names` metadata field.

`_routing`

protosearch.IndexRouting is a message with the following fields:

Field	Type	Description
`required`	`bool`	Whether to require routing for all document operations.

`_source`

protosearch.IndexSource is a message with the following fields:

Field	Type	Description
`compress`	`bool`	Whether to compress stored source data. OpenSearch only.
`compress_threshold`	`string`	Minimum source size to trigger compression. OpenSearch only.
`enabled`	`bool`	Whether to store the `_source` field.
`excludes`	`repeated string`	Fields to exclude from the stored `_source`.
`includes`	`repeated string`	Fields to include in the stored `_source`.
`mode`	`protosearch.SourceMode`	How to store the `_source` field.

`mode`

protosearch.SourceMode is an enum with the following values:

SOURCE_MODE_DISABLED
SOURCE_MODE_STORED
SOURCE_MODE_SYNTHETIC

Type inference

If type is not specified, protoc-gen-protosearch will infer a field type from the protobuf type.

Protobuf	Elasticsearch
`string`	`keyword`
`bool`	`boolean`
`int32`, `sint32`, `sfixed32`	`integer`
`uint32`, `fixed32`	`long`
`int64`, `sint64`, `sfixed64`	`long`
`uint64`, `fixed64`	`unsigned_long`
`float`	`float`
`double`	`double`
`bytes`	`binary`
message	`object`
enum	`keyword`
`google.protobuf.Timestamp`	`date`

Diagnostics

The plugin validates some field options and collects diagnostics during compilation. Errors (EXXX) are fatal; protoc will exit with an error code and will not produce any output. The plugin prints warnings (WXXX) to standard output.

Errors

E001

The specified value is invalid for this parameter. The plugin will report the reason.

E002

target.json is not valid JSON.

E003

target.json is not a JSON object.

Warnings

W001

name is invalid.

Names must match the pattern [@a-z][a-z0-9_]*(\.[a-z0-9_]+)*. These are all allowed names:

@timestamp
foo
foo_bar
foo.bar.baz
foo_123

W002

The target label does not correspond to a known target.

`protoc-gen-protosearch`

With protoc-gen-protosearch installed on your $PATH, you can compile mappings like so:

protoc -I proto/ --plugin=protoc-gen-protosearch --protosearch_out=. proto/example/article.proto

Specify --protosearch_opt=target=<label> to compile the mapping for a specific target.

protoc -I proto/ --plugin=protoc-gen-protosearch --protosearch_out=. --protosearch_opt=target=<label> proto/example/article.proto

The plugin pretty-prints output by default. Specify --protosearch_opt=pretty=false to disable this.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

protosearch reference

API

`field`

`name`

`mapping`

`dynamic`

`index_options`

`index_prefixes`

`term_vector`

`target`

`index`

`mapping`

`_field_names`

`_routing`

`_source`

`mode`

Type inference

Diagnostics

Errors

E001

E002

E003

Warnings

W001

W002

`protoc-gen-protosearch`

FilesExpand file tree

reference.md

Latest commit

History

reference.md

File metadata and controls

protosearch reference

API

field

name

mapping

dynamic

index_options

index_prefixes

term_vector

target

index

mapping

_field_names

_routing

_source

mode

Type inference

Diagnostics

Errors

E001

E002

E003

Warnings

W001

W002

protoc-gen-protosearch

`field`

`name`

`mapping`

`dynamic`

`index_options`

`index_prefixes`

`term_vector`

`target`

`index`

`mapping`

`_field_names`

`_routing`

`_source`

`mode`

`protoc-gen-protosearch`