Wait for popen3 execution of goodhosts

[conradolandia]

I block malware sites and ads from the internet using a large hosts file (600.000+ lines), redirecting all these bad hosts to 0.0.0.0. When I try to use goodhosts, it just freezes and whatever process I was on won't go on. If I remove most of the entries form /etc/hosts, the process completes without problems. Is this a bug or expected behavior?

[Mte90]

I think that https://github.com/goodhosts/cli doesn't support a big hosts file at all.
I used to have an hosts file huge like you but was like 100000~ lines and was working.
I can't do anything for this as this is not a vagrant issue but the cli tool that the plugin uses.

[conradolandia]

Thank you, will take the problem there.

[tomjn]

@Mte90 how do you know it's goodhosts itself that's stalling?

[Mte90]

So I had in the past a lot of hosts inside the file and goodhosts was parsing all of them for rewriting (clean as example) and this was changing the whole file and delay everything.
As the plugin just call the command with the hosts we need to add the stall should be on the goodhosts side as need to parse 600000~ urls for cleaning and look if there are duplicates and so on.

At the end I removed everything on my hosts file and there are no issues anymore, also as goodhosts development is stalled I didn't opened a ticket for it.

[tomjn]

@Mte90 that's cleaning, we don't clean anymore on VVV where this was first seen, haven't for a long time.

But even ignoring that, you've shared an anecdote, no actual data has been shared proving it, just assumptions. Instead, @conradolandia should be able to test the theory by trying to add a host with the goodhosts CLI to see if it still hangs. If so, it's the CLI. If not, it's the vagrant plugin.

You're also assuming they have the latest version of the plugin, we don't know that

[tomjn]

@conradolandia grab the latest goodhosts from https://github.com/goodhosts/cli/releases and try to use it to add a host, e.g. goodhosts add 127.0.0.1 example.test

[tomjn]

and run vagrant plugin list --local to get the version of the plugin

[conradolandia]

Results:

Downloaded the goodhosts binary from https://github.com/goodhosts/cli/releases, where it reports the last version as 1.0.7.
It adds the entry with no problems, even in the 600.000+ lines file, where it takes a couple seconds, but it makes the addition. Removing also works fine.

$ vagrant plugin list --local
vagrant-goodhosts (1.1.0, local)

[Mte90]

How much times takes? maybe is the ruby api that has issues with the timeout or similar.

[conradolandia]

~ wc -l /etc/hosts
610748 /etc/hosts
~ time sudo goodhosts add 127.0.0.1 loquesea.test
hosts entry added: 127.0.0.1 loquesea.test
sudo goodhosts add 127.0.0.1 loquesea.test  3,61s user 0,37s system 156% cpu 2,545 total

[Mte90]

So I guess that the issue is at https://github.com/goodhosts/vagrant/blob/master/lib/vagrant-goodhosts/GoodHosts.rb#L119
A solution could be execute the command in a different thread but this not ensure that it works always.

Looking at the doc a pid is created but we are no printing it, https://ruby-doc.org/stdlib-2.4.2/libdoc/open3/rdoc/Open3.html#method-c-popen3

We can add a code that wait for the execution of the process like this https://stackoverflow.com/questions/11710542/wait-for-popen3-process-to-finish but I am not sure.

[tomjn]

we should wait for the command to finish before proceeding otherwise how do we know the command is finished, and how do we avoid multiple attempts happening at the same time from the same vagrant command