GameScanners - Socket Exception

[nitral]

The Game Scanners are currently unreliable and due to some reason work on one machine and not on the other. The machine currently hosting the hub is facing problems. For example, it works absolutely fine on my Windows 7 machine. However when I tried it in my Ubuntu 12.04LTS, it immediately broke down!

On my Linux machine, while preparing sockets, it pops up:

java.net.SocketException: Too many open files
    at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
    at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:82)
    at java.net.DatagramSocket.createImpl(DatagramSocket.java:322)
    at java.net.DatagramSocket.<init>(DatagramSocket.java:179)
    at CODServerQueryDemo.prepareSockets(CODServerQueryDemo.java:140)
    at CODServerQueryDemo.main(CODServerQueryDemo.java:26)

It is able to catch this. Execution moves forward.
Then it successfully creates the template request packet.
Then it breaks down again while sending packets through these sockets.

java.lang.NullPointerException
    at CODServerQueryDemo.sendRequests(CODServerQueryDemo.java:106)
    at CODServerQueryDemo.main(CODServerQueryDemo.java:41)

It handles this too. Execution moves forward.
The main thread sleeps for 20ms.
Then, it breaks down again while getting responses and it not able to recover.

Exception in thread "main" java.lang.NullPointerException
    at CODServerQueryDemo.getResponse(CODServerQueryDemo.java:75)
    at CODServerQueryDemo.main(CODServerQueryDemo.java:48)

I have JRE build 1.7.0_10-b18 and javac version 1.7.0_10
The original host machine scored a time of ~ 6 seconds in scanning the subnets mentioned in the source code of the scanners.
I believe the reason is that the program is not able to open that many sockets. Currently, it tries opens 5610 sockets. One for each IP on the network. That's just too much! Perhaps, a way out is to not make that many socket and reuse the ones we create.
We need to rethink the strategy we use to ping servers for information. Or at least rethink the flow of the current strategy so it works.

Ideas?

[sidcode]

Can we open a socket only after someone types the command +cod, +cs, +dota? Is there a way to parse the incoming messages, and then add Event Listeners to take action-- open a socket for that IP, and then return the game server details. Why do we need to have an open socket for every ip? Refreshing periodically for messages sounds like an awful waste of space and time. Also, I guess the problem may be because of OpenJDK/ IcedTea framework on Ubuntu.

[nitral]

@sidcode The problem also exists in Windows 7 and 8 apparently, as we have also tested on them. On those machines, I personally installed the latest official Oracle JRE and received the same stack trace message. Although, there are other Windows 7 machines where it has worked flawlessly in ~ 6 seconds.
To properly satisfy a +cs, +cod, +csgo and +dota query, we need to scan the entire network. Although, an incoming query can be parsed and actions can be taken accordingly. There are 5610 people with the potential to create a server which deserves to be on the list.
None can be ignored.
However, I do agree that opening a socket for every IP was a bad idea.
Any ideas on what strategy we should implement?

[sidcode]

@nitral How many of those 5610 people actually create a server?
Here's one unique ( and possibly whacky ) thing we can do: Wait for a few days, scan the nicks who actually create a server- people who participate in that kind of discussion ( eg. server jaldi banao yaar ); use these people as a training set, and use some ML classification algorithm to know about the potential server creating users. So we open sockets for only these people, the Learning set would only improve with time, and we can save on space.

Alternatively: Search for key-phrases like "server", "cod", "cs", "jaldi banao", "koi to banao", in the main chat: parse those queries, and then only create a socket for those IPs.

[nitral]

@sidcode It's not about if they actually create a server. Making a Game Server is nothing actually.
As far as the ML strategy is concerned, same thing. It is non-deterministic. The point is that if anyone makes a server, it should show up. Regardless if the host knows about the hub or not. The host may even shut down his DC client to save bandwidth. Using a non-deterministic approach here would mean that a live server may not be scanned. That is unacceptable.
People typing "jaldi janao" or "koi to banao" aren't gonna host Game Servers and hence there is no point in opening a socket for them.

[nitral]

One of the premature/temporary fixes I have come accross recently is to enable the OS to open more files. This lets it open more sockets, which are treated as files by the kernel.
Starting a shell using this does the trick:
sudo sh -c "ulimit -n 18432 && exec su $LOGNAME"
Maybe we can include this in the launcher shell script and land it on master (as a temporary fix of course)

[nitral]

Temporary fix for Linux Kernels landed on master with: 4ee8f83
I guess a permanent fix will require a comeplete revamp on the way requests are sent and received.
Temporary fix for Windows still pending.

This issue will just be concerned now with temporary fixes to the problem. Permanent fix issue will be filed separately once the temporary fix is completed for Windows.

[nitral]

I changed the Source Engine Server Scanner script to use a configurable number of sockets (config.cfg) [f272229]. The idea is to re-use the same sockets (opened once) in a round-robin scheduled manner to send and receive datagram packets.
If this is the final fix we are going with, then this needs to be applied to CODServerScanner as well. So, I'm keeping the bug open for now.