[Resolv] Decoded URL not add in URI list

[jm-edard]

Version:
SpamAssassin version 3.4.1
running on Perl version 5.24.1
with: DecodeShortURLs.pm version 0.11

Test: mail with a url byt.ly link to "google.com"

Result:

mars  7 12:11:16.308 [29615] dbg: plugin: loading Mail::SpamAssassin::Plugin::URIDNSBL from @INC
mars  7 12:11:17.640 [29615] dbg: config: fixed relative path: /etc/spamassassin/DecodeShortURLs.pm
mars  7 12:11:17.641 [29615] dbg: plugin: loading Mail::SpamAssassin::Plugin::DecodeShortURLs from /etc/spamassassin/DecodeShortURLs.pm
mars  7 12:11:20.811 [29615] dbg: plugin: Mail::SpamAssassin::Plugin::DecodeShortURLs=HASH(0x5561f0d2ae08) implements 'parsed_metadata', priority -1
mars  7 12:11:21.055 [29615] dbg: DecodeShortURLs: Found http://bit.ly/1dNVPAW => http://www.google.com/
(...) 
mars  7 12:12:56.822 [46930] dbg: uridnsbl: considering host=bit.ly, domain=bit.ly
mars  7 12:12:56.898 [46930] dbg: uridnsbl: complete_dnsbl_lookup URIBL_WS_SURBL DNSBL:bit.ly:multi.surbl.org
mars  7 12:12:56.898 [46930] dbg: uridnsbl: complete_dnsbl_lookup URIBL_CR_SURBL DNSBL:bit.ly:multi.surbl.org
(...) 

• uridnsbl scan byt.li
• SHORT_URIBL match
• DecodeShortURLs Found the link
  BUT uridnsbl doesn't scan google.com !

# spamassassin -D <test.eml  2>&1 | grep -i "google"
mars  7 12:15:27.662 [14082] dbg: DecodeShortURLs: Found cached http://bit.ly/1dNVPAW => http://www.google.com/

I have try with more domain, this is the same, the decoded url is not add to URL to scan on the mail.

Regards

[karolaug]

Hi, I just tested it by shortening url blacklisted by URIBL using bit.ly and sending e-mail with it. I got hits: HAS_SHORT_URL, SHORT_URIBL, URIBL_BLACK, so it looks like it is working. I am using Perl 5.26.1, SA 3.4.1, DecodeShortURLs.pm version 0.11. Are you sure that you run DecodeShortURLs.pm with -1 priority, so it can pass the decoded URLs for further processing on time? Karol

[jm-edard]

Thank you very mutch.
DecodeShortURLs.pm run with -1 priority
But i have a rules witch -100 priority , than call 1 (only 1) URIBL scan.
My problem is certainly at this level.

So i have test to put -200 priority to DecodeShortURLs.pm with:
$self->register_method_priority ('parsed_metadata', -200);

But the problem is the same !
Regards

[jm-edard]

Hello again,
I have completely disabled the rules that call URIBL priority -100. I also deleted "sa-compile" in doubt. It's always the same !
In addition, I see that "DecodeShortURLs: Found cached" is calculated first before any call to uridnsbl. I do not think the problem comes from the "priority".
Regards

[jm-edard]

I have find my origin problem, it was :
mars 7 17:29:50.780 [59110] warn: plugin: eval failed: Undefined subroutine &Mail::SpamAssassin::Util::RegistrarBoundaries::trim_domain called at /usr/share/perl5/Mail/SpamAssassin/Util.pm line 1236.

I have fin a patch here, that solve my problem : https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7195

--- DecodeShortURLs.pm.orig     2015-05-15 19:19:07.000000000 +0300
+++ DecodeShortURLs.pm  2015-05-15 19:20:19.000000000 +0300
@@ -446,7 +446,7 @@

   # At this point we have a new URL in $response
   $pms->got_hit('HAS_SHORT_URL');
-  _add_uri_detail_list($pms, $location);
+  $self->_add_uri_detail_list($pms, $location);

   # Set chained here otherwise we might mark a disabled page or
   # redirect back to the same host as chaining incorrectly.
@@ -458,7 +458,7 @@
     my($host) = ($short_url =~ /^(https?:\/\/\S+)\//);
     $location = "$host/$location";
     dbg("Looks like a local redirection: $short_url => $location");
-    _add_uri_detail_list($pms, $location);
+    $self->_add_uri_detail_list($pms, $location);
     return $location;
   }

@@ -490,7 +490,7 @@
 # Beware.  Code copied from PerMsgStatus get_uri_detail_list().
 # Stolen from GUDO.pm
 sub _add_uri_detail_list {
-  my ($pms, $uri) = @_;
+  my ($self, $pms, $uri) = @_;
   my $info;

   # Cache of text parsed URIs, as previously used by get_uri_detail_list().
@@ -502,7 +502,7 @@
     [Mail::SpamAssassin::Util::uri_list_canonify (undef, $uri)];

   foreach (@{$info->{cleaned}}) {
-    my ($dom, $host) = Mail::SpamAssassin::Util::uri_to_domain($_);
+    my ($dom, $host) = $self->{main}->{registryboundaries}->uri_to_domain($_);

     if ($dom && !$info->{domains}->{$dom}) {
       # 3.4 compatibility as per Marc Martinec

[cepheid666]

I'm pretty sure those changes are already included in the version of DecodeShortURLs.pm that is downloadable from trunk. Are you sure you are/were running the latest version? It's still listed as 0.11 but was modified in 2017, not 2015.

[jm-edard]

Ah yes ! Actually I do not have the right version. I only relied on the version number 0.11. It's really a shame not to make it evolve with the corrections, you could have called it "0.11.1" for example!

[cepheid666]

"you" is not me, I'm just an end-user like you... but agreed, if changes were made, the version number should ideally increment.

[cepheid666]

Since this issue is resolved, @jm-edard, you should close it. There is a "Close Issue" button at the bottom of the comment box. Cheers.