3.16 - Rebuild Issue

[yvesgoupil]

There is a problem with Solr re-indexing (Admin portal > Metadata Service >Index). It works with stop-gap metadatas but in the case where GISCs metadata has been harvested previously and only one metadata contains an error, in the bandbox for example, the re-indexing crashes and the whole metadatas are not accessible anymore.
The only solution I found to be able to have the metadata available again by making new harvestings is to delete the tables concerning the metadata in the database.

• Stop Solr
• Connect to the Database and :
  OpenWIS=# delete FROM metadata where istemplate = 'n' ;
  OpenWIS=# delete FROM openwis_pattern_metadata_mapping ;
  OpenWIS=# delete FROM openwis_product_metadata ;
• Restart Solr and the admin portal .

[yvesgoupil]

• Here is the error in admin porta openwis.log:
  2023-01-30 15:06:22,114 ERROR [geonetwork.index] - User: admin4. Exception while rebuilding index, going to rebuild it: Error from server at http://wism31-sidev:8983/solr/core2: ERROR: [doc=urn:x-wmo:md:int.wmo.wis::isin11lfpw] Error adding field '_geometry'='POLYGON ((5.07 89.9, 5.07 0, 5.07 0, 5.07 89.9, 5.07 89.9))' msg=Unable to parse shape given formats "lat,lon", "x y" or as WKT because org.locationtech.spatial4j.exception.InvalidShapeException: Invalid polygon, all points are coplanar input: POLYGON ((5.07 89.9, 5.07 0, 5.07 0, 5.07 89.9, 5.07 89.9)) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://wism31-sidev:8983/solr/core2: ERROR: [doc=urn:x-wmo:md:int.wmo.wis::isin11lfpw] Error adding field '_geometry'='POLYGON ((5.07 89.9, 5.07 0, 5.07 0, 5.07 89.9, 5.07 89.9))' msg=Unable to parse shape given formats "lat,lon", "x y" or as WKT because org.locationtech.spatial4j.exception.InvalidShapeException: Invalid polygon, all points are coplanar input: POLYGON ((5.07 89.9, 5.07 0, 5.07 0, 5.07 89.9, 5.07 89.9)) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:649) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106) at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71) at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85) at org.openwis.metadataportal.kernel.search.index.solr.SolrIndexManager.addToServer(SolrIndexManager.java:376) at org.openwis.metadataportal.kernel.search.index.solr.SolrIndexManager.add(SolrIndexManager.java:341) at org.openwis.metadataportal.kernel.search.SearchManagerImpl.index(SearchManagerImpl.java:342) at org.openwis.metadataportal.kernel.search.SearchManagerImpl.synchronizeDocs(SearchManagerImpl.java:576) at org.openwis.metadataportal.kernel.search.SearchManagerImpl.rebuildIndex(SearchManagerImpl.java:852) at org.openwis.metadataportal.kernel.search.AsyncSearchManagerImpl$1.run(AsyncSearchManagerImpl.java:82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

[abrmh]

Changes has been made and committed to the branch 3.16 to avoid the Solr to crash when rebuilding indexes.

1. The Admin and User portal wars have to be deployed
2. A file called wrongmetada.log is created and will contain all wrong metadata's
3. A file log4.cfg should look like as text bellow:

Default Logging Configuration

log4j.rootLogger = OFF

GEONETWORK SETTINGS

log4j.logger.geonetwork = INFO, jeeves
log4j.logger.geonetwork.search = WARN
log4j.logger.geonetwork.editorexpandelement = WARN
log4j.logger.geonetwork.editoraddelement = WARN
log4j.logger.geonetwork.index = INFO
log4j.logger.geonetwork.csw = WARN
log4j.logger.geonetwork.mef = WARN
log4j.logger.geonetwork.z3950server = WARN
log4j.logger.geonetwork.z3950 = WARN
log4j.logger.geonetwork.sru = INFO
log4j.logger.geonetwork.sru.search = INFO
log4j.logger.geonetwork.oai = INFO
log4j.logger.geonetwork.datamanager = INFO

JEEVES SETTINGS

log4j.logger.jeeves = INFO, jeeves
log4j.logger.jeeves.service = INFO
log4j.logger.jeeves.dbms = WARN
log4j.logger.jeeves.dbmspool = WARN
log4j.logger.jeeves.xlinkprocessor = INFO

OPENWIS SETTINGS

log4j.logger.Login = INFO, jeeves
log4j.logger.statistics = INFO, statistics
log4j.logger.openwis = INFO, jeeves
log4j.logger.openwis.metadataaligner = INFO, metadata
log4j.logger.openwis.extract.productmetadata = INFO, metadata
log4j.logger.openwis.wrong.productmetadata = INFO, wrongmetadata
log4j.logger.openwis.harvester.executor = INFO, metadata
#log4j.logger.openwis.service.provider = DEBUG, jeeves
#log4j.logger.openwis.service.management = DEBUG, jeeves

SPRING

log4j.logger.org.springframework = WARN, jeeves

JZKIT SETTINGS

log4j.logger.com.k_int=WARN, jeeves
log4j.logger.org.jzkit=WARN, jeeves
log4j.logger.org.jzkit.a2j=WARN, jeeves
log4j.logger.org.jzkit.search.impl.LRUCache = INFO, jeeves

JEEVES APPENDER

log4j.appender.jeeves =org.apache.log4j.DailyRollingFileAppender
log4j.appender.jeeves.DatePattern='.'yyyy-MM-dd
log4j.appender.jeeves.file = ${user.home}/logs/openwis.log
log4j.appender.jeeves.layout = org.apache.log4j.PatternLayout
log4j.appender.jeeves.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

METADATA APPENDER

log4j.appender.metadata =org.apache.log4j.DailyRollingFileAppender
log4j.appender.metadata.DatePattern='.'yyyy-MM-dd
log4j.appender.metadata.file = ${user.home}/logs/metadata.log
log4j.appender.metadata.layout = org.apache.log4j.PatternLayout
log4j.appender.metadata.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

WRONG METADATA APPENDER

log4j.appender.wrongmetadata =org.apache.log4j.DailyRollingFileAppender
log4j.appender.wrongmetadata.DatePattern='.'yyyy-MM-dd
log4j.appender.wrongmetadata.file = ${user.home}/logs/wrongmetadata.log
log4j.appender.wrongmetadata.layout = org.apache.log4j.PatternLayout
log4j.appender.wrongmetadata.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

STATS APPENDER

log4j.appender.statistics = org.apache.log4j.RollingFileAppender
log4j.appender.statistics.maxBackupIndex= 1
log4j.appender.statistics.maxFileSize= 1024KB
log4j.appender.statistics.file = ${user.home}/logs/stats.csv
log4j.appender.statistics.append = false
log4j.appender.statistics.layout = org.apache.log4j.PatternLayout
log4j.appender.statistics.layout.ConversionPattern=%d{ISO8601};%m%n

CONSOLE SETTINGS

log4j.appender.console = org.apache.log4j.ConsoleAppender
log4j.appender.console.layout = org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

[abrmh]

Solr is indexing all metadata when doing harvesting or rebuild without crashing , it list all wrong metadata's in wrongmetadata.log file but it the incorrect MDs are not corrected in database so it is up to admin to correct them.

[yvesgoupil]

I've deployed the last commit a65f519 made by Brahim, I tested this fix and it works perfectly. Here is the usecase:

Stop data feed
Stop user and admin portals
Stop solr
Delete MDs in database
OpenWIS=# delete FROM metadata where istemplate = 'n' ;
OpenWIS=# delete FROM openwis_pattern_metadata_mapping ;
OpenWIS=# delete FROM openwis_product_metadata ;
Launch solr
Start user and admin portals
Restart the data feed
Harvest the MF set for example
See that metadata with errors appear in the wrongmetadata.log.
Check the number of metadata of the test set in the catalog on the admin portal.
Do a Re-build on the admin portal
Check that the same metadata with errors appear again in the log wrongmetadata.log.
Check that the number of test set metadata in the catalog on the admin portal has not changed.
There has been no Solr crash.

I propose that we consider this issue resolved.

[yvesgoupil]

I set the label to "close me".

Yves.