Web Scraping issue

[MNSleeper]

@aubertc , I have a bit of an issue with the web-scraping aspect of things. I was trying get a mini-implementation going, but couldn't get fileUtils to be recognized. But that's a problem I ran into before, and you sent a minimal app that I was able to get working in the past as a fix. I went and redownloaded it, fired it up, and it didn't work. The error I get it:

_> Exception in thread "main" java.lang.Error: Unresolved compilation problem:

    FileUtils cannot be resolved

    at com.mycompany.app.App.main(App.java:23)_

I don't understand why it just stops working. I know I was able to get a scraped HTML page, but now I won't even compile. I know thismight be old ground, but do you have an solutions/ideas why this is happening?

[aubertc]

I was trying get a mini-implementation

Please share the code, I can't do much without looking at it.

[aubertc]

I guess you mean this one?

package com.mycompany.app;
import org.apache.commons.io.FileUtils;
import java.io.File;
import java.io.IOException;
import java.net.URL;

public class App {
    public static void main(String[] args) {
        try {
            URL url = new URL("https://www.nsf.gov/awardsearch/download.jsp"); // don't just assign the string to the url: create a URL object with it.
            String Fname = "example.html"; // Give some extension to your file (here, I added ".html").
            File destination = new File(Fname); // You could avoid creating a string, but ok.
            FileUtils.copyURLToFile(url, destination);
        } catch (IOException e) { // You were not catching exceptions, which is weird to me.
            e.printStackTrace();
        }
    }
}

[MNSleeper]

My apologies. Yes, that was the implementation I tried to execute but failed.

[aubertc]

That may be because maven is not in charge of finding that library, and your "basic" java installation can't find that file.

To use maven to create a small test, do:

mvn archetype:generate -DgroupId=com.popbr.fileUtilsTest -DartifactId=fileUtilsTest -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

2. In the pom.xml file, add

     <dependency>
	<groupId>commons-io</groupId>
    	<artifactId>commons-io</artifactId>
    	<version>2.11.0</version>
     </dependency>

in the <dependencies> group,
3. In fileUtilsTest/src/main/java/com/popbr/fileUtilsTest/App.java copy-and-paste the code

package com.popbr.fileUtilsTest;
import org.apache.commons.io.FileUtils;
import java.io.File;
import java.io.IOException;
import java.net.URL;

public class App {
    public static void main(String[] args) {
        try {
            URL url = new URL("https://www.nsf.gov/awardsearch/download.jsp"); // don't just assign the string to the url: create a URL object with it.
            String Fname = "example.html"; // Give some extension to your file (here, I added ".html").
            File destination = new File(Fname); // You could avoid creating a string, but ok.
            FileUtils.copyURLToFile(url, destination);
        } catch (IOException e) { // You were not catching exceptions, which is weird to me.
            e.printStackTrace();
        }
    }
} 

5. Finally, run mvn compile and mvn exec:java -Dexec.mainClass="com.popbr.app.App".

[aubertc]

(I assume that by "fire it up", you meant "using command-line java")

[aubertc]

I had a look at https://github.com/popbr/data-integration/tree/main/fileUtilsTest and have a few comments:

1. The naming gives the command mvn exec:java -Dexec.mainClass="com.popbr.fileUtilsTest.App", which is not very consistent with your previous naming schemes.
2. I obtained the error java.lang.NullPointerException: Cannot read the array length because "fileN" is null. I believe I need to understand why you declare
   

   data-integration/fileUtilsTest/src/main/java/com/popbr/fileUtilsTest/App.java

   Line 18 in d953007

   String Fdestination = "target" + File.separator +"downloads" + File.separator;

   
   before overriding it with "hard-coded" values at
   

   data-integration/fileUtilsTest/src/main/java/com/popbr/fileUtilsTest/App.java

   Line 19 in d953007

   Fdestination = "C:" + File.separator + "Users" + File.separator + "sleep" + File.separator + "Desktop" + File.separator + "fileUtilsTest" + File.separator + "target" + File.separator + "downloads";

   
   Can't we do something more modular?

[MNSleeper]

1. To the naming of it:

mvn archetype:generate -DgroupId=com.popbr.fileUtilsTest -DartifactId=fileUtilsTest -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

was the command I used, which you suggested in the reply here #13 (comment). I do intend to integrate this program's functions with the larger program once its bugs/functions are worked out and it works fully.

2. I apologize for the lack of modularity, I wasn't thinking back when I tested this. I did that hardcoded value for testing when it was throwing files in odd places. I can fix it later tonight

[MNSleeper]

The hardcoding problem has been removed. As a bonus, the unzipped files now go directly into the downloads folder, not some extra folder in the downloads folder.