Add noindex directive detection to prevent submission of URLs that should not be indexed

[Copilot]

The IndexNow plugin was submitting URLs to search engines even when they had noindex directives, violating SEO best practices and user intent. This PR adds comprehensive noindex detection to prevent automatic submission of URLs that should not be indexed.

Changes Made

Core Implementation

• Added has_noindex_directive() method with multi-layered noindex detection
• Modified on_post_published() to check for noindex directives before URL submission
• Ensured delete operations still proceed (appropriate for removing URLs from index)

Noindex Detection Coverage

• WordPress site-wide settings: Respects "Discourage search engines from indexing this site" option (blog_public)
• SEO Plugin compatibility:
  • Yoast SEO (_yoast_wpseo_meta-robots-noindex)
  • RankMath SEO (rank_math_robots array)
  • All in One SEO (_aioseo_robots_noindex)
• HTTP Headers: X-Robots-Tag header detection with case-insensitive matching
• Error handling: Graceful handling of failed HTTP requests with 5-second timeout

Example Behavior

// Before: Would submit regardless of noindex
if ($is_change && $auto_submission_enabled) {
    $this->routes->submit_url_to_bwt($siteUrl, $link, $api_key, $type, false);
}

// After: Respects noindex directives
if ($is_change && $auto_submission_enabled) {
    if ($type != 'delete' && $this->has_noindex_directive($post, $link)) {
        return; // Skip submission for noindex URLs
    }
    $this->routes->submit_url_to_bwt($siteUrl, $link, $api_key, $type, false);
}

Documentation Updates

• Updated plugin description to mention noindex directive support
• Added changelog entry documenting the new feature
• Bumped version to 1.0.2

This is a minimal, surgical fix that addresses the core issue without breaking existing functionality. The implementation follows WordPress and SEO best practices while maintaining compatibility with popular SEO plugins.

Fixes #60.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

@pingal 👋 This repository doesn't have Copilot instructions. With Copilot instructions, I can understand the repository better, work faster and produce higher quality PRs.

I can generate a .github/copilot-instructions.md file for you automatically. Click here to open a pre-filled issue and assign it to me. I'll write the instructions, and then tag you for review.