Support join query based on refid field

[takesson]

The refid field contains index IDs. They can be a mix of references to specific revisions and latest revision.

It would be very useful to be able to make a single query that returns all references / dependencies. Currently, it seems difficult because there is no ID field that contains both latest and revision-specific IDs.

• id: always revision-specific
• idhead: always latest, interestingly, this field remains even when head=false.
• Could perhaps be useful, if possible to design a query that resolves graphics baseline.

I propose a new field:

• idjoin: multivalue field that can contain one or two IDs
• head=true: contains both id and idhead value (item can be referenced in 2 ways)
• head=false: contains only id (item can only be referenced with revision)

Vital question: Can indexing update the idjoin field when setting head=false? There are certain limitations to what updates can do.

[takesson]

@solsson just requesting comments

[solsson]

Are the inline docs on the refid field outdated? It says that it can be resolved to idhead.

I don't remember the limitations of update queries. We still want Solr 4 compatibility, right? I'm pretty sure it was less allowing than http://yonik.com/solr/atomic-updates/ ("updated for Solr 5"). I guess we want the add operation? Looks like supported in https://github.com/Reposoft/repos-indexing/blob/v0.12.1/src/main/java/se/repos/indexing/twophases/IndexingDocIncrementalSolrj.java#L175, but committed with a rather odd message f3d0b3e.

[takesson]

Are the inline docs on the refid field outdated? It says that it can be resolved to idhead.
Copied from the applicable ref_* field by the extracting handlers, and resolved to "idhead" field values. Revision locked references should be excluded because they can not be affected.

This is correct but it says "Revision locked references should be excluded". This is true for use-cases like, "can I delete this item". For other use-case we want a report of all references, e.g. "I want to include all graphics when exporting this document" (might include revision locked graphics). That would require 2 join queries today. I would like a single join query.

We still want Solr 4 compatibility, right?

Undecided. I am considering documenting a known limitation in CMS 4.2. Would enable us to gather a number of indexing enhancements and deliver in CMS 5.0 with a modern SolR.

I guess we want the add operation?

I think we want remove. When flipping head field to false we want to remove the head-ID from idjoin.

[takesson]

I guess we want the add operation?

I think we want remove. When flipping head field to false we want to remove the head-ID from idjoin.

Actually, this field contains only things that are known about the previous version. We can replace the field content. Unless boolean (head flag) is a special case in SolR 4 this should be possible. Well, idjoin would also be multivalue so we are not sure if SolR 4 supports update of multivalue fields at all.

[takesson]

I just had a flash of insight. We have for a long time had challenges with indexing always returning IDs that are revision-locked, regardless of the head flag. There are inherent challenges in determining whether the ID should have revision of not.

When requesting single item from svn backend, the result depends on the query:

• Query for latest will return an item with ID without revision (or exception if it no longer exists).
• Query for specific revision will return that revision.

When requesting multiple items from svn backend:

• Typically used to list content in folders, obviously returns rev-less items when listing a head-folder.
• There are TODO comments in the API debating how revision should be handled when requesting historical folder list.

Requests to the indexing can be much more complex, simple first:

• Search for a word, only head.
• Reporting queries are typically either head or non-head.
• Used-in: typically only head. Can be more complex but mostly either head or non-head. Not quite sure.
• Dependencies: Can be a mix of head and non-head. Very challenging because each result item should have head-id or rev depending on how they were referenced in the XML, not based on whether they happen to be head right now. Also, an XML could reference the same file multiple times with a mix of head and rev, then the reporting result needs to contain them both.

Conclusion

• We should index 2 items for each commit: 1. A revision-specific item (head=false). 2. A head-item (head=true).
• We no longer need to modify the head flag.
• Maybe we want to keep the head flag or introduce a similar flag that we actually modify (flip to false) when an item is no longer head. It could still be useful to see if an item is head even when requesting historical items (not sure we use that today?).
• Indexing the head item is standard SolR indexing that overwrites the existing item. The id field will be equal to idHead. The item needs to be deleted when deleted in Subversion.
• If doing reporting without filter on head, there could be duplicates (same item in both forms). This is quite rare, not sure which use-cases. Can be deduplicated by grouping on idRev (if we introduce such a field).

Comments:

• This is a semi-reversal to svnhead / svnrev with the massive difference that all items are in the same core and with virtually exactly the same set of fields.
• The current head implementation has its challenges and they are very obvious now when moving dependency tracking towards indexing.
• There will potentially be other challenges with this approach, need to investigate when we make queries that does not have a head filter.
• How does this relate to indexing-driven move tracking? Needs investigation.

[takesson]

Discussed with Staffan over some ale.

We agreed that having both historical and head would be a reasonable change. The current design was a slight over-correction from the svnhead / svnrev separation.

Staffan noted that the current code that flips head flag is likely a handler in the indexing pipeline. It might not be very intrusive to write an alterntive such handler that instead forks the indexdoc.

Note that changes are also needed in cms-reporting where we currently have a hack that removes revision from the itemid of head items.

[takesson]

Simonsoft CMS ticket in Trac: #1310

[takesson]

I have implemented the above change adding 2 SolR objects per changed item. Actually, the head item is overwritten in SolR when modified, having an ID without revision.

Since the complexity of flipping the head flag is now removed, we are one huge step closer to supporting the head flag on folders. That is important for enabling index-based browsing and some other features.

However, the simple form of enabling head flag on folders will likely not represent the path revision (field rev) the same way as subversion, where a file modification impacts the path revision of all parent folders.

Is it important to represent path revision identical to Subversion? @solsson what do you think?

[takesson]

I have created a test case documenting the current implementation of path revision 'rev' and commit revision 'revc' of folders when a containing file is modified.

Considering different aspects, I don't think path revision is vital for folders. PR #7 does not change 'rev' and 'revc' for folders. I expect that 'rev' = 'revc' always for folders. However, I am not entirely sure after folder copy (children of the copied folder).

• Implementing the head flag for folders is important in order to allow simple browsing of the HEAD folder structure. Historical browsing of folder structure is complex regardless of path revision 'rev', requires group-by queries.
• Folders do NOT keep references to their files in SolR. Files keep references to their folder parents. So path revision is not important in order to query which files are in a folder at a certain revision.
• Many modern object storage backends (S3, ...) do NOT model folders at all. A migration to non-subversion backend would likely obsolete the concept of path revision for folders anyway. Metadata on folders would likely need to be stored elsewhere or in/on a special file object.

I am considering PR #7 completed. Will now propagate to higher level testing.