Corrupted Database

Hi,
I'm trying to update the mmdb database, as there is missing data in some ips, as Graylog needs those information to treat the logs.
However, after I update the mmdb, the graylog lookup table doesn't work.
The result from the test is this:

```
{
  "single_value": null,
  "multi_value": null,
  "string_list_value": null,
  "has_error": false,
  "ttl": 9223372036854776000
}
```

normally, it would be this:

```
{
  "single_value": "TZ",
  "multi_value": {
    "continent": {
      "code": "AF",
      "geoname_id": 6255146,
      "names": {
        "de": "Afrika",
        "ru": "Африка",
        "pt-BR": "África",
        "ja": "アフリカ",
        "en": "Africa",
        "fr": "Afrique",
        "zh-CN": "非洲",
        "es": "África"
      }
    },
    "country": {
      "confidence": null,
      "geoname_id": 149590,
      "is_in_european_union": false,
      "iso_code": "TZ",
      "names": {
        "de": "Tansania",
        "ru": "Танзания",
        "pt-BR": "Tanzânia",
        "ja": "タンザニア連合共和国",
        "en": "Tanzania",
        "fr": "Tanzanie",
        "zh-CN": "坦桑尼亚",
        "es": "Tanzania"
      }
    },
    "traits": {
      "autonomous_system_number": null,
      "autonomous_system_organization": null,
      "connection_type": null,
      "domain": null,
      "ip_address": "41.59.208.37",
      "is_anonymous": false,
      "is_anonymous_proxy": false,
      "is_anonymous_vpn": false,
      "is_anycast": false,
      "is_hosting_provider": false,
      "is_legitimate_proxy": false,
      "is_public_proxy": false,
      "is_residential_proxy": false,
      "is_satellite_provider": false,
      "is_tor_exit_node": false,
      "isp": null,
      "mobile_country_code": null,
      "mobile_network_code": null,
      "network": "41.59.0.0/16",
      "organization": null,
      "user_type": null,
      "user_count": null,
      "static_ip_score": null
    },
    "represented_country": {
      "confidence": null,
      "geoname_id": null,
      "is_in_european_union": false,
      "iso_code": null,
      "names": {},
      "type": null
    },
    "registered_country": {
      "confidence": null,
      "geoname_id": 149590,
      "is_in_european_union": false,
      "iso_code": "TZ",
      "names": {
        "de": "Tansania",
        "ru": "Танзания",
        "pt-BR": "Tanzânia",
        "ja": "タンザニア連合共和国",
        "en": "Tanzania",
        "fr": "Tanzanie",
        "zh-CN": "坦桑尼亚",
        "es": "Tanzania"
      }
    }
  },
  "string_list_value": null,
  "has_error": false,
  "ttl": 9223372036854776000
}
```

what I have done:

1. installed the app:
```
wget "https://github.com/ipinfo/mmdbctl/releases/download/mmdbctl-1.4.4/mmdbctl_1.4.4_linux_amd64.tar.gz" && tar xvf mmdbctl_1.4.4_linux_amd64.tar.gz && rm mmdbctl_1.4.4_linux_amd64.tar.gz && mv mmdbctl_1.4.4_linux_amd64 /usr/local/bin/mmdbctl
sudo apt-get update && sudo apt-get install jq
```

2. Exported:
`mmdbctl export --in ${importfrom} --out ${exportto}`

3. Changed the content using python:
```
import json

input_file = '/opt/scripts/tmp/GeoLite2-City.json'
output_file = '/opt/scripts/tmp/GeoLite2-City-Changes.json'

def add_subdivisions_to_json_line(json_line):
    data = json.loads(json_line)
    original_data = data.copy()  # Make a copy of the original data for comparison

    if 'subdivisions' not in data:
        country_info = data.get('country', {})
        if country_info:
            subdivisions = [{
                "geoname_id": country_info.get("geoname_id"),
                "iso_code": country_info.get("iso_code"),
                "names": {"en": country_info["names"]["en"]}
            }]
            data['subdivisions'] = subdivisions

    return data, data != original_data  # Return both the data and a flag indicating if a change was made

with open(input_file, 'r') as infile, open(output_file, 'w') as outfile:
    for line in infile:
        updated_data, is_changed = add_subdivisions_to_json_line(line)
        if is_changed:
            updated_line = json.dumps(updated_data)
            outfile.write(updated_line + '\n')

print(f"Processed file saved as {output_file}")
```

4. Imported the changes to the mmdb:
`mmdbctl import --disable-metadata-pointers false --in ${updatedjson} --out ${mmdbbkp}`

5. Tests
`mmdbctl metadata GeoLite2-City-updated.mmdb`
- Binary Format 2.0
- Database Type ipinfo GeoLite2-City-updated.mmdb
- IP Version    6
- Record Size   32
- Node Count    3971781
- Description
    en ipinfo GeoLite2-City-updated.mmdb
- Languages     en
- Build Epoch   1721934123

`mmdbctl metadata GeoLite2-City.mmdb`
- Binary Format 2.0
- Database Type GeoLite2-City
- IP Version    6
- Record Size   28
- Node Count    3971880
- Description
    en GeoLite2City database
- Languages     de, en, es, fr, ja, pt-BR, ru, zh-CN
- Build Epoch   1721747951

`mmdbctl read -f json-pretty 41.59.208.37 GeoLite2-City.mmdb # Original File`
{
  "continent": {
    "code": "AF",
    "geoname_id": 6255146,
    "names": {
      "de": "Afrika",
      "en": "Africa",
      "es": "África",
      "fr": "Afrique",
      "ja": "アフリカ",
      "pt-BR": "África",
      "ru": "Африка",
      "zh-CN": "非洲"
    }
  },
  "country": {
    "geoname_id": 149590,
    "iso_code": "TZ",
    "names": {
      "de": "Tansania",
      "en": "Tanzania",
      "es": "Tanzania",
      "fr": "Tanzanie",
      "ja": "タンザニア連合共和国",
      "pt-BR": "Tanzânia",
      "ru": "Танзания",
      "zh-CN": "坦桑尼亚"
    }
  },
  "location": {
    "accuracy_radius": 1000,
    "latitude": -6.8227,
    "longitude": 39.2936,
    "time_zone": "Africa/Dar_es_Salaam"
  },
  "registered_country": {
    "geoname_id": 149590,
    "iso_code": "TZ",
    "names": {
      "de": "Tansania",
      "en": "Tanzania",
      "es": "Tanzania",
      "fr": "Tanzanie",
      "ja": "タンザニア連合共和国",
      "pt-BR": "Tanzânia",
      "ru": "Танзания",
      "zh-CN": "坦桑尼亚"
    }
  }
}

`mmdbctl read -f json-pretty 41.59.208.37 GeoLite2-City-updated.mmdb # New File`
{
  "continent": {
    "code": "AF",
    "geoname_id": 6255146,
    "names": {
      "de": "Afrika",
      "en": "Africa",
      "es": "África",
      "fr": "Afrique",
      "ja": "アフリカ",
      "pt-BR": "África",
      "ru": "Африка",
      "zh-CN": "非洲"
    }
  },
  "country": {
    "geoname_id": 149590,
    "iso_code": "TZ",
    "names": {
      "de": "Tansania",
      "en": "Tanzania",
      "es": "Tanzania",
      "fr": "Tanzanie",
      "ja": "タンザニア連合共和国",
      "pt-BR": "Tanzânia",
      "ru": "Танзания",
      "zh-CN": "坦桑尼亚"
    }
  },
  "location": {
    "accuracy_radius": 1000,
    "latitude": -6.8227,
    "longitude": 39.2936,
    "time_zone": "Africa/Dar_es_Salaam"
  },
  "network": "41.59.208.0/20",
  "registered_country": {
    "geoname_id": 149590,
    "iso_code": "TZ",
    "names": {
      "de": "Tansania",
      "en": "Tanzania",
      "es": "Tanzania",
      "fr": "Tanzanie",
      "ja": "タンザニア連合共和国",
      "pt-BR": "Tanzânia",
      "ru": "Танзания",
      "zh-CN": "坦桑尼亚"
    }
  },
  "subdivisions": [
    {
      "geoname_id": 149590,
      "iso_code": "TZ",
      "names": {
        "en": "Tanzania"
      }
    }
  ]
}

Graylog log:

WARN  [MaxmindDataAdapter] Unable to look up city data for IP address /123.185.49.44, returning empty result.
java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "parameters[index]" is null
        at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:450) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decodeMap(Decoder.java:341) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decode(Decoder.java:89) ~[graylog.jar:?]
        at com.maxmind.db.NoCache.get(NoCache.java:17) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decode(Decoder.java:116) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:434) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decodeMap(Decoder.java:341) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[graylog.jar:?]
        at com.maxmind.db.Decoder.decode(Decoder.java:76) ~[graylog.jar:?]
        at com.maxmind.db.Reader.resolveDataPointer(Reader.java:411) ~[graylog.jar:?]
        at com.maxmind.db.Reader.getRecord(Reader.java:185) ~[graylog.jar:?]
        at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:280) ~[graylog.jar:?]
        at com.maxmind.geoip2.DatabaseReader.getCity(DatabaseReader.java:365) ~[graylog.jar:?]
        at com.maxmind.geoip2.DatabaseReader.city(DatabaseReader.java:348) ~[graylog.jar:?]
        at org.graylog.plugins.map.geoip.MaxMindIPLocationDatabaseAdapter.maxMindCity(MaxMindIPLocationDatabaseAdapter.java:39) ~[graylog.jar:?]
        at org.graylog.plugins.map.geoip.MaxmindDataAdapter.doGet(MaxmindDataAdapter.java:186) ~[graylog.jar:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.get(LookupDataAdapter.java:143) ~[graylog.jar:?]
        at org.graylog2.lookup.LookupTable.lambda$lookup$0(LookupTable.java:73) ~[graylog.jar:?]
        at org.graylog2.lookup.caches.CaffeineLookupCache.lambda$get$2(CaffeineLookupCache.java:161) ~[graylog.jar:?]
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:167) ~[graylog.jar:?]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2688) ~[graylog.jar:?]
        at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source) [?:?]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686) [graylog.jar:?]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669) [graylog.jar:?]
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) [graylog.jar:?]
        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) [graylog.jar:?]
        at org.graylog2.lookup.caches.CaffeineLookupCache.get(CaffeineLookupCache.java:182) [graylog.jar:?]
        at org.graylog2.lookup.LookupTable.lookup(LookupTable.java:73) [graylog.jar:?]
        at org.graylog2.lookup.LookupTableService$Function.lookup(LookupTableService.java:635) [graylog.jar:?]
        at org.graylog2.inputs.extractors.LookupTableExtractor.run(LookupTableExtractor.java:66) [graylog.jar:?]
        at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:226) [graylog.jar:?]
        at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:79) [graylog.jar:?]
        at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:105) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:167) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:137) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:107) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:52) [graylog.jar:?]
        at org.graylog2.shared.buffers.PartitioningWorkHandler.onEvent(PartitioningWorkHandler.java:52) [graylog.jar:?]
        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167) [graylog.jar:?]
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]

mmdb Sizes:
-rw-r--r--  1 root root 237M Jul 26 00:38 GeoLite2-City-updated.mmdb
-rw-r--r--  1 root root  50M Jul 25 07:28 GeoLite2-City.mmdb

json sizes:
-rw-r--r-- 1 root root 963M Jul 25 14:35 GeoLite2-City-Changes.json
-rw-r--r-- 1 root root 3.8G Jul 25 08:36 GeoLite2-City-Dump.json


**Why is the database in some way corrupted that Graylog can't read but the command line can??**


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Corrupted Database #35

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Corrupted Database #35

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions