Understand and mitigate OpenSearch _id length restriction (inherited from Elastic)

[nsoft]

This limitation on ID's is likely to be inconvenient for JesterJ deployments if data is contained in deep directories or otherwise identified by long URL's. This ticket covers investigating and mitigating this restriction.

[nsoft]

Apparently despite the wording in the docs, this limitation is meant to apply outside the bulk api Docs could be improved if they said: "In OpenSearch 2.9 the bulk API began enforcing the 512 byte limit on _id (to match other APIs)"

opensearch-project/OpenSearch#6595

[nsoft]

Some interesting discussion of Id's and memory usage here: elastic/elasticsearch#33049 but it sounds a bit like it's a concern in indexing many small documents? (I guess that's what they mean by dense?) and also it's centered on auto-generated ID's which are usually only acceptable for things like event or log data that should never be updated/rewritten anyway... (or perhaps for systems that re-index everything every time and no downstream system ever looks at _id) Clearly other use cases exist and would have other priorities.

[nsoft]

Apparently the genesis of this restriction relates to being able to paste a request into a browser???

elastic/elasticsearch#16034

That seems like an upside down priority to me. If folks have long ID's they should not expect to be pasting these things into browsers in the first place. sigh

Pasting into a browser ought to be a dev convenience side line, not a application limiting use case (IMHO). Also here's a SO answer that suggests that ONLY IE was likely to be affected by this 7 years before this limit was implemented... https://stackoverflow.com/a/417184 EDIT: in fairness the table detailing other popular browsers doesn't show up till november 2016, 10 months later, but it's clear from the current content, and the 5% Edge/IE market share that this decision has not aged well.

So... it's been baked in for 9 years, and dates back to elastic. Time to think a bit.