Importing ES custom template failed

[northalpha]

While trying to import your pfsense_custom_template into my ES via cerebro, i am getting following error:

[2018-06-26 12:58:30,262][DEBUG][action.admin.indices.template.put] [Sabra] failed to put template [pfsense-custom] MapperParsingException[Failed to parse mapping [message]: No handler for type [keyword] declared on field [PFSENSE_UDP_DATA]]; nested: MapperParsingException[No handler for type [keyword] declared on field [PFSENSE_UDP_DATA]]; at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:291) at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.validateAndAddTemplate(MetaDataIndexTemplateService.java:213) at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.access$200(MetaDataIndexTemplateService.java:57) at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService$2.execute(MetaDataIndexTemplateService.java:157) at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:480) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:784) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: MapperParsingException[No handler for type [keyword] declared on field [PFSENSE_UDP_DATA]] at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:307) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:222) at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:118) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:549) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:319) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:289)

i am using ES 2.4.6 with cerebro 0.8.1 and graylog 2.4.5

Anyone an idea what is wrong here?

[mipsou]

I send template by curl
curl -X PUT "localhost:9200/_template/template_1" -H 'Content-Type: application/json' -d'…content_pfsense-custom…'
The answer is
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [keyword] declared on field [PFSENSE_UDP_DATA]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [message]: No handler for type [keyword] declared on field [PFSENSE_UDP_DATA]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [keyword] declared on field [PFSENSE_UDP_DATA]"}},"status":400}

i am using ES 2.4.6 with cerebro 0.8.1 and graylog 2.4.5

[mipsou]

I try to update the template with this code
I test now.
{ "order": -1, "template": "pfsense_*", "settings": { "index": { "analysis": { "analyzer": { "analyzer_keyword": { "filter": "lowercase", "tokenizer": "keyword" } } }, "max_result_window": "1000000" } }, "mappings": { "message": { "_source": { "enabled": true }, "dynamic_templates": [ { "internal_fields": { "mapping": { "type": "keyword" }, "match": "gl2_*" } }, { "store_generic": { "mapping": { "index": "not_analyzed" }, "match": "*" } } ], "properties": { "reason": { "type": "string" }, "PFSENSE_UDP_DATA": { "type": "string" }, "gl2_remote_ip": { "type": "string" }, "gl2_remote_port": { "type": "string" }, "icmp_unreachport_dest_ip": { "type": "string" }, "icmp_unreachport_protocol": { "type": "string" }, "source": { "analyzer": "analyzer_keyword", "index": "analyzed", "type": "string" }, "dest_ip_geolocation": { "copy_to": "dst_location", "type": "string" }, "gl2_source_input": { "type": "string" }, "PFSENSE_ICMP_ECHO_REQ_REPLY": { "type": "string" }, "PFSENSE_PROTOCOL_DATA": { "type": "string" }, "ack_number": { "type": "string" }, "ip_ver": { "type": "string" }, "ecn": { "type": "string" }, "dest_ip_city_name": { "type": "string" }, "tcp_flags": { "type": "string" }, "PFSENSE_ICMP_UNREACHPORT": { "type": "string" }, "src_ip_city_name": { "type": "string" }, "PFSENSE_ICMP_DATA": { "type": "string" }, "action": { "type": "string" }, "gl2_source_node": { "type": "string" }, "src_ip_geolocation": { "copy_to": "src_location", "type": "string" }, "id": { "type": "string" }, "dest_port": { "type": "string" }, "PFSENSE_IGMP_DATA": { "type": "string" }, "offset": { "type": "string" }, "level": { "type": "long" }, "streams": { "index": "not_analyzed", "type": "string" }, "PFSENSE_TCP_DATA": { "type": "string" }, "PFSENSE_ICMP_RESPONSE": { "type": "string" }, "icmp_unreachport_dest_ip_geolocation": { "type": "string" }, "PFSENSE_ICMP_TYPE": { "type": "string" }, "iface": { "type": "string" }, "tcp_window": { "type": "string" }, "icmp_unreachport_port": { "type": "string" }, "icmp_echo_id": { "type": "string" }, "dest_ip": { "type": "string" }, "proto": { "type": "string" }, "PFSENSE_LOG_DATA": { "type": "string" }, "icmp_type": { "type": "string" }, "PFSENSE_IPv4_SPECIFIC_DATA": { "type": "string" }, "flags": { "type": "string" }, "rule": { "type": "string" }, "icmp_unreachport_dest_ip_city_name": { "type": "string" }, "PFSENSE_IP_DATA": { "type": "string" }, "tcp_options": { "type": "string" }, "PFSENSE_IP_SPECIFIC_DATA": { "type": "string" }, "src_ip": { "type": "string" }, "PFSENSE_LOG_ENTRY": { "type": "string" }, "proto_id": { "type": "string" }, "tracker": { "type": "string" }, "tos": { "type": "string" }, "timestamp": { "format": "yyyy-MM-dd HH:mm:ss.SSS", "type": "date" }, "direction": { "type": "string" }, "data_length": { "type": "string" }, "length": { "type": "string" }, "message": { "analyzer": "standard", "index": "analyzed", "type": "string" }, "icmp_unreachport_dest_ip_country_code": { "type": "string" }, "ttl": { "type": "string" }, "icmp_echo_sequence": { "type": "string" }, "sequence_number": { "type": "string" }, "src_location": { "type": "geo_point" }, "src_port": { "type": "string" }, "dest_ip_country_code": { "type": "string" }, "dst_location": { "type": "geo_point" }, "src_ip_country_code": { "type": "string" }, "full_message": { "analyzer": "standard", "index": "analyzed", "type": "string" }, "facility": { "type": "string" }, "real_timestamp": { "format": "yyyy-MM-dd HH:mm:ss", "type": "date" } } } }, "aliases": {} }

[mipsou]

I confirm. It work perfectly.

[unbaiat]

template loads ok but no data in dashboard. Testbed: mongodb 4.0.0, elasticsearch 5.6.10, graylog 2.4.6, Java 1.8.0.181, Debian 8.11 x64.

[opc40772]

@unbaiat First, there are additional items that need to be installed from the command line... Grafana Panels. You need these:
grafana-cli plugins install grafana-worldmap-panel
grafana-cli plugins install savantly-heatmap-panel

[fabioccoelho]

Hi, this code doesn´t works for me. I´m get follow error:

{ "error": { "root_cause": [ { "type": "mapper_parsing_exception", "reason": "Root mapping definition has unsupported parameters: [message : {_source={enabled=true}, dynamic_templates=[{internal_fields={mapping={type=keyword}, match=gl2_*}}, {store_generic={mapping={index=not_analyzed}, match=*}}], properties={PFSENSE_UDP_DATA={type=keyword}, reason={type=keyword}, gl2_remote_ip={type=keyword}, gl2_remote_port={type=keyword}, icmp_unreachport_dest_ip={type=keyword}, icmp_unreachport_protocol={type=keyword}, dest_ip_geolocation={copy_to=dst_location, type=text}, source={fielddata=true, analyzer=analyzer_keyword, type=text}, gl2_source_input={type=keyword}, PFSENSE_ICMP_ECHO_REQ_REPLY={type=keyword}, PFSENSE_PROTOCOL_DATA={type=keyword}, ack_number={type=keyword}, ip_ver={type=keyword}, ecn={type=keyword}, dest_ip_city_name={type=keyword}, tcp_flags={type=keyword}, PFSENSE_ICMP_UNREACHPORT={type=keyword}, PFSENSE_ICMP_DATA={type=keyword}, src_ip_city_name={type=keyword}, action={type=keyword}, gl2_source_node={type=keyword}, id={type=keyword}, src_ip_geolocation={copy_to=src_location, type=string}, dest_port={type=keyword}, PFSENSE_IGMP_DATA={type=keyword}, offset={type=keyword}, level={type=long}, streams={type=keyword}, PFSENSE_TCP_DATA={type=keyword}, PFSENSE_ICMP_RESPONSE={type=keyword}, icmp_unreachport_dest_ip_geolocation={type=keyword}, PFSENSE_ICMP_TYPE={type=keyword}, iface={type=keyword}, tcp_window={type=keyword}, icmp_unreachport_port={type=keyword}, icmp_echo_id={type=keyword}, dest_ip={type=keyword}, proto={type=keyword}, PFSENSE_LOG_DATA={type=keyword}, icmp_type={type=keyword}, PFSENSE_IPv4_SPECIFIC_DATA={type=keyword}, flags={type=keyword}, rule={type=keyword}, PFSENSE_IP_DATA={type=keyword}, icmp_unreachport_dest_ip_city_name={type=keyword}, tcp_options={type=keyword}, PFSENSE_IP_SPECIFIC_DATA={type=keyword}, src_ip={type=keyword}, PFSENSE_LOG_ENTRY={type=keyword}, proto_id={type=keyword}, tracker={type=keyword}, tos={type=keyword}, direction={type=keyword}, timestamp={format=yyyy-MM-dd HH:mm:ss.SSS, type=date}, data_length={type=keyword}, length={type=keyword}, icmp_unreachport_dest_ip_country_code={type=keyword}, message={analyzer=standard, type=text}, ttl={type=keyword}, icmp_echo_sequence={type=keyword}, sequence_number={type=keyword}, src_location={type=geo_point}, src_port={type=keyword}, dest_ip_country_code={type=keyword}, dst_location={type=geo_point}, src_ip_country_code={type=keyword}, full_message={analyzer=standard, type=text}, facility={type=keyword}, real_timestamp={format=yyyy-MM-dd HH:mm:ss, type=date}}}]" } ], "type": "mapper_parsing_exception", "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [message : {_source={enabled=true}, dynamic_templates=[{internal_fields={mapping={type=keyword}, match=gl2_*}}, {store_generic={mapping={index=not_analyzed}, match=*}}], properties={PFSENSE_UDP_DATA={type=keyword}, reason={type=keyword}, gl2_remote_ip={type=keyword}, gl2_remote_port={type=keyword}, icmp_unreachport_dest_ip={type=keyword}, icmp_unreachport_protocol={type=keyword}, dest_ip_geolocation={copy_to=dst_location, type=text}, source={fielddata=true, analyzer=analyzer_keyword, type=text}, gl2_source_input={type=keyword}, PFSENSE_ICMP_ECHO_REQ_REPLY={type=keyword}, PFSENSE_PROTOCOL_DATA={type=keyword}, ack_number={type=keyword}, ip_ver={type=keyword}, ecn={type=keyword}, dest_ip_city_name={type=keyword}, tcp_flags={type=keyword}, PFSENSE_ICMP_UNREACHPORT={type=keyword}, PFSENSE_ICMP_DATA={type=keyword}, src_ip_city_name={type=keyword}, action={type=keyword}, gl2_source_node={type=keyword}, id={type=keyword}, src_ip_geolocation={copy_to=src_location, type=string}, dest_port={type=keyword}, PFSENSE_IGMP_DATA={type=keyword}, offset={type=keyword}, level={type=long}, streams={type=keyword}, PFSENSE_TCP_DATA={type=keyword}, PFSENSE_ICMP_RESPONSE={type=keyword}, icmp_unreachport_dest_ip_geolocation={type=keyword}, PFSENSE_ICMP_TYPE={type=keyword}, iface={type=keyword}, tcp_window={type=keyword}, icmp_unreachport_port={type=keyword}, icmp_echo_id={type=keyword}, dest_ip={type=keyword}, proto={type=keyword}, PFSENSE_LOG_DATA={type=keyword}, icmp_type={type=keyword}, PFSENSE_IPv4_SPECIFIC_DATA={type=keyword}, flags={type=keyword}, rule={type=keyword}, PFSENSE_IP_DATA={type=keyword}, icmp_unreachport_dest_ip_city_name={type=keyword}, tcp_options={type=keyword}, PFSENSE_IP_SPECIFIC_DATA={type=keyword}, src_ip={type=keyword}, PFSENSE_LOG_ENTRY={type=keyword}, proto_id={type=keyword}, tracker={type=keyword}, tos={type=keyword}, direction={type=keyword}, timestamp={format=yyyy-MM-dd HH:mm:ss.SSS, type=date}, data_length={type=keyword}, length={type=keyword}, icmp_unreachport_dest_ip_country_code={type=keyword}, message={analyzer=standard, type=text}, ttl={type=keyword}, icmp_echo_sequence={type=keyword}, sequence_number={type=keyword}, src_location={type=geo_point}, src_port={type=keyword}, dest_ip_country_code={type=keyword}, dst_location={type=geo_point}, src_ip_country_code={type=keyword}, full_message={analyzer=standard, type=text}, facility={type=keyword}, real_timestamp={format=yyyy-MM-dd HH:mm:ss, type=date}}}]", "caused_by": { "type": "mapper_parsing_exception", "reason": "Root mapping definition has unsupported parameters: [message : {_source={enabled=true}, dynamic_templates=[{internal_fields={mapping={type=keyword}, match=gl2_*}}, {store_generic={mapping={index=not_analyzed}, match=*}}], properties={PFSENSE_UDP_DATA={type=keyword}, reason={type=keyword}, gl2_remote_ip={type=keyword}, gl2_remote_port={type=keyword}, icmp_unreachport_dest_ip={type=keyword}, icmp_unreachport_protocol={type=keyword}, dest_ip_geolocation={copy_to=dst_location, type=text}, source={fielddata=true, analyzer=analyzer_keyword, type=text}, gl2_source_input={type=keyword}, PFSENSE_ICMP_ECHO_REQ_REPLY={type=keyword}, PFSENSE_PROTOCOL_DATA={type=keyword}, ack_number={type=keyword}, ip_ver={type=keyword}, ecn={type=keyword}, dest_ip_city_name={type=keyword}, tcp_flags={type=keyword}, PFSENSE_ICMP_UNREACHPORT={type=keyword}, PFSENSE_ICMP_DATA={type=keyword}, src_ip_city_name={type=keyword}, action={type=keyword}, gl2_source_node={type=keyword}, id={type=keyword}, src_ip_geolocation={copy_to=src_location, type=string}, dest_port={type=keyword}, PFSENSE_IGMP_DATA={type=keyword}, offset={type=keyword}, level={type=long}, streams={type=keyword}, PFSENSE_TCP_DATA={type=keyword}, PFSENSE_ICMP_RESPONSE={type=keyword}, icmp_unreachport_dest_ip_geolocation={type=keyword}, PFSENSE_ICMP_TYPE={type=keyword}, iface={type=keyword}, tcp_window={type=keyword}, icmp_unreachport_port={type=keyword}, icmp_echo_id={type=keyword}, dest_ip={type=keyword}, proto={type=keyword}, PFSENSE_LOG_DATA={type=keyword}, icmp_type={type=keyword}, PFSENSE_IPv4_SPECIFIC_DATA={type=keyword}, flags={type=keyword}, rule={type=keyword}, PFSENSE_IP_DATA={type=keyword}, icmp_unreachport_dest_ip_city_name={type=keyword}, tcp_options={type=keyword}, PFSENSE_IP_SPECIFIC_DATA={type=keyword}, src_ip={type=keyword}, PFSENSE_LOG_ENTRY={type=keyword}, proto_id={type=keyword}, tracker={type=keyword}, tos={type=keyword}, direction={type=keyword}, timestamp={format=yyyy-MM-dd HH:mm:ss.SSS, type=date}, data_length={type=keyword}, length={type=keyword}, icmp_unreachport_dest_ip_country_code={type=keyword}, message={analyzer=standard, type=text}, ttl={type=keyword}, icmp_echo_sequence={type=keyword}, sequence_number={type=keyword}, src_location={type=geo_point}, src_port={type=keyword}, dest_ip_country_code={type=keyword}, dst_location={type=geo_point}, src_ip_country_code={type=keyword}, full_message={analyzer=standard, type=text}, facility={type=keyword}, real_timestamp={format=yyyy-MM-dd HH:mm:ss, type=date}}}]" } }, "status": 400 }
Anyone can help me?

Thanks.

Gray version: 4.0
ES version: 7.x