valid HTML entity   causing errors

[jmckenna]

• example valid HTML:
  <span style='font-weight:bold;padding-left:50px'>&nbsp;&nbsp;&nbsp; Position : </span>Project Scientist<br/>

  • source record

• causes invalid character entity error in archetype run:

  {"file":"/home/fils/src/gomods/gleaner/internal/summoner/sitemaps/sitemap_ng.go:58","func":"github.com/gleanerio/gleaner/internal/summoner/sitemaps.DomainSitemap","level":"error","msg":"XML syntax error on line 271: invalid character entity \u0026nbsp;","time":"2024-05-29T16:45:32Z"}
  {"file":"/home/fils/src/gomods/gleaner/internal/summoner/acquire/resources.go:151","func":"github.com/gleanerio/gleaner/internal/summoner/acquire.getSitemapURLList","level":"error","msg":"Error parsing sitemap index at: https://incois.gov.in/essdp/ViewMetadata?fileid=7d051f4f-98cd-4578-861a-6bfb5f6c03afXML syntax error on line 271: invalid character entity \u0026nbsp;","time":"2024-05-29T16:45:32Z"}
  {"file":"/home/fils/src/gomods/gleaner/internal/summoner/acquire/resources.go:75","func":"github.com/gleanerio/gleaner/internal/summoner/acquire.ResourceURLs","level":"error","msg":"Error getting sitemap urls for: incoisXML syntax error on line 271: invalid character entity \u0026nbsp;","time":"2024-05-29T16:45:32Z"}
  

• causes undefined entity error in sitemapCheck

  2024-05-29 13:49:38,803 | WARNING | sitemaps.py:530 | sitemap_to_df | undefined entity: line 271, column 49 https://incois.gov.in/essdp/ViewMetadata?fileid=7d051f4f-98cd-4578-861a-6bfb5f6c03af
  

related to iodepo/odis-arch#396

[jmckenna]

cc @fils @pbuttigieg as I hit this when trying to harvest the INCOIS partner. (sitemap)

[jmckenna]

Similar issues for Gleaner with un-closed HTML tags for <meta>, <link>, <img>, <input type="button"

It seems that we need a Gleaner switch for something like -ignore-html-tag-errors (?)

[pbuttigieg]

@fils if it's valid HTML, it shouldn't cause any issues for import. Could you confirm the issue?

[fils]

These look like invalid entries in the sitemap.xml not errors in the HTML?

Both the Go and Python code seem to fail on their sitemap. I did pull their sitemap and it looks OK on inspection but it claims the unicode character \u0026 was used in line 305.

{"file":"/home/fils/src/Projects/gleaner.io/gleaner/internal/summoner/sitemaps/sitemap_ng.go:58","func":"github.com/gleanerio/gleaner/internal/summoner/sitemaps.DomainSitemap","level":"error","msg":"XML syntax error on line 305: invalid character entity \u0026copy;","time":"2024-06-05T06:46:55-05:00"}

Still trying to confirm this.

[pbuttigieg]

\u0026 is the & - perhaps there's some sort of weird character sneaking in?

[fils]

@jmckenna @pbuttigieg this isn't a sitemap, it's a sitemap index.

They need to make this into a sitemap like at https://sitemaps.org/protocol.html with

<?xml version="1.0" encoding="UTF-8"?>

<[urlset](https://sitemaps.org/protocol.html#urlsetdef) xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

   <[url](https://sitemaps.org/protocol.html#urldef)>

      <[loc](https://sitemaps.org/protocol.html#locdef)>http://www.example.com/</loc>

      <[lastmod](https://sitemaps.org/protocol.html#lastmoddef)>2005-01-01</lastmod>

      <[changefreq](https://sitemaps.org/protocol.html#changefreqdef)>monthly</changefreq>

      <[priority](https://sitemaps.org/protocol.html#prioritydef)>0.8</priority>

   </url>

</urlset>

not the sitemap index mode which is an index of sitemaps.

[fils]

@jmckenna @pbuttigieg
Just to follow up on this. Our XML guidance in the book was wrong. I can't believe it has been that way all this time. I updated it and may even push this out for publishing here later today as it was so bad to have that there.

[jmckenna]

@jmckenna @pbuttigieg this isn't a sitemap, it's a sitemap index.

Wow, good catch (even though all other sitemap validators accept this, Gleaner does not). I did notice this early on but since all of the validators accepted it like this, I assumed it was valid. It makes sense to me now though, how Gleaner was thinking.