Crash with very small input files

[hamishmorgan]

Byblo was run on approximately 1% of the fruit test data set. During one of these runs byblo crashed giving the following output. Some data was lost so it can't be confirmed, but it seems likely this caused by the filtered entries file containing only 1 entry. This resulted in an empty sims files, and consequently KNN became unhappy.

Need test case to confirm.

Here's the whole output, in case something's relevant in the rest.

user1@ubuntu:~/Desktop/NLP/bybloStats$ sudo python ./byblo_statistics.py ../Data/bnc-gramrels-fruit/bnc-gramrels-fruit -p 0.01
***************************************************************************
BYBLO STATISTICS TOOL
***************************************************************************


>> start:eventsStats - full file size= 584000 bytes

   >> start:browseEvents
   Creating file for 1.0 %
   375 entries and 375 features
   >> end:browseEvents
   Writing statistics in ./stats/bnc-gramrels-fruit-%0-01.stats.events

>> end:eventsStats

Statistics files after feature extraction
./stats/bnc-gramrels-fruit-%0-01.stats.events ...



>> start:bybloStats

   >> start:singleBybloRun 
/home/user1/Desktop/NLP/Byblo-2.0.1/byblo.sh 
/home/user1/Desktop/NLP/Data/bnc-gramrels-fruit/bnc-gramrels-fruit-%0-01
/home/user1/Desktop/NLP/bybloStats/thesauri
/home/user1/Desktop/NLP/Byblo-2.0.1
[INFO] 
=== Running full thesaurus build === 

[INFO] 
Configuration:
 * Input instances file: /home/user1/Desktop/NLP/Data/bnc-gramrels-fruit/bnc-gramrels-fruit-%0-01
 * Output directory: /home/user1/Desktop/NLP/bybloStats/thesauri (42.8GiB free)
 * Temporary directory: /home/user1/Desktop/NLP/bybloStats/thesauri/tempdirtmp-421668544681582743.tmp (42.8GiB free)
 * Character encoding: UTF-8
 * Num. Threads: 6
 * Memory used=5.5MiB, allocated=989.9MiB, available=1.9GiB
 * Java Spec: Java Platform API Specification 1.6, Sun Microsystems Inc.
 * Java VM: OpenJDK Client VM 20.0-b12, Sun Microsystems Inc.
 * Java Runtime: OpenJDK Runtime Environment 1.6.0_24-b24
 * OS: Linux 3.2.0-26-generic-pae i386
 * Start time: 5:05:21 PM BST Monday, July 9, 2012


[INFO] 
=== Stage 1 of 6: Enumerating Strings ===

[INFO] 
Configuration:
 * Input instances file: /home/user1/Desktop/NLP/Data/bnc-gramrels-fruit/bnc-gramrels-fruit-%0-01
 * Output enumerated instances file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.enumerated
 * Output entry index: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.entry-index
 * Output feature index: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.feature-index
 * Start time: 5:05:22 PM BST Monday, July 9, 2012
 * Memory used=5.9MiB, allocated=989.9MiB, available=1.9GiB


[INFO] pipe: (unknown% complete)
[INFO] pipe: Processed 375 items. (unknown% complete)
[INFO] pipe: Processed 375 items. (unknown% complete)
[INFO] 
Stats:
 * End time: 5:05:22 PM BST Monday, July 9, 2012
 * Ellapsed time: 00:00:0.3650
 * Memory used=11.4MiB, allocated=989.9MiB, available=1.9GiB


[INFO] 
=== Stage 2 of 6: Counting ===

[INFO] 
Configuration:
 * Input instances file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.enumerated
 * Output entries file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.entries
 * Output features file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.features
 * Output events file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.events
 * Chunk size: 1,000,000 instances
 * Start time: 5:05:22 PM BST Monday, July 9, 2012
 * Memory used=6.0MiB, allocated=989.9MiB, available=1.9GiB


[WARN] nThreads (6) > availableProcessors (1)
[INFO] xcount: Mapping to small count tasks (0% complete)
[INFO] xcount: Merging and aggregating results (0% complete)
[INFO] xcount: Merging and aggregating results (0% complete)
    count: (0% complete)
[INFO] xcount: Merging and aggregating results (0% complete)
    count: Read 375 instances (0% complete)
[INFO] xcount: Merging and aggregating results (44% complete)
    count: Writing entries. (44% complete)
[INFO] xcount: Merging and aggregating results (46% complete)
    count: Writing features. (46% complete)
[INFO] xcount: Merging and aggregating results (72% complete)
    count: Writing events. (72% complete)
[INFO] xcount: Merging and aggregating results (100% complete)
[INFO] xcount: Merging and aggregating results (33% complete)
    sort: (unknown% complete)
[INFO] xcount: Merging and aggregating results (66% complete)
[INFO] xcount: Merging and aggregating results (50% complete)
    sort: (unknown% complete)
[INFO] xcount: Merging and aggregating results (50% complete)
    sort: (unknown% complete)
    sort: (unknown% complete)
[INFO] xcount: Merging and aggregating results (75% complete)
    sort: (unknown% complete)
[INFO] xcount: Merging and aggregating results (100% complete)
[INFO] 
Stats:
 * End time: 5:05:22 PM BST Monday, July 9, 2012
 * Ellapsed time: 00:00:0.3510
 * Memory used=38.7MiB, allocated=989.9MiB, available=1.9GiB


[INFO] 
=== Stage 3 of 6: Filtering ===

[INFO] 
Configuration:
 * Input entries file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.entries
 * Input features file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.features
 * Input events file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.events
 * Output entries file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.entries.filtered
 * Output features file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.features.filtered
 * Output events file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.events.filtered
 * Chunk size: 1,000,000 instances
 * Min. Entry Freq: 10
 * Min. Feature Freq: 10
 * Min. Event Freq: 0
 * Entry Pattern: null
 * Feature Pattern: null
 * Entry Whitelist: null
 * Feature Whitelist: null
 * Start time: 5:05:23 PM BST Monday, July 9, 2012
 * Memory used=6.0MiB, allocated=989.9MiB, available=1.9GiB


[INFO] Running filtering.
[INFO] filter: (0% complete)
[INFO] filter: Running filtering pass (#1). (0% complete)
[INFO] filter: Filtering entries. (0% complete)
[INFO] filter: Accepted 3 of 22 entries. (0% complete)
[INFO] filter: Accepted 3 of 22 entries. (16% complete)
[INFO] filter: Filtering events from. (16% complete)
[INFO] filter: Accepted 173 of 237 events. (16% complete)
[INFO] filter: Accepted 173 of 237 events. (28% complete)
[INFO] filter: Filtering features. (28% complete)
[INFO] filter: Accepted 4 of 216 features. (28% complete)
[INFO] filter: Accepted 4 of 216 features. (37% complete)
[INFO] filter: Filtering events from. (37% complete)
[INFO] filter: Filtering events from. (44% complete)
[INFO] filter: Filtering entries. (44% complete)
[INFO] filter: Accepted 3 of 4 entries. (44% complete)
[INFO] filter: Accepted 3 of 4 entries. (55% complete)
[INFO] filter: Running filtering pass (#2). (55% complete)
[INFO] filter: Filtering features. (55% complete)
[INFO] filter: Accepted 4 of 5 features. (55% complete)
[INFO] filter: Accepted 4 of 5 features. (66% complete)
[INFO] filter: Copying final entries file. (66% complete)
[INFO] filter: Copying finaly events file. (77% complete)
[INFO] filter: Copying final features file. (88% complete)
[INFO] filter: Copying final features file. (100% complete)
[INFO] filter: Copying final features file. (100% complete)
[INFO] 
Stats:
 * End time: 5:05:23 PM BST Monday, July 9, 2012
 * Ellapsed time: 00:00:0.1040
 * Memory used=6.0MiB, allocated=989.9MiB, available=1.9GiB
 * Start time: 5:05:23 PM BST Monday, July 9, 2012


[INFO] 
=== Stage 4 of 6: All-Pairs ===

[INFO] 
Configuration:
 * Input entries file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.entries.filtered
 * Input features file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.features.filtered
 * Input events file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.events.filtered
 * Ouput sims file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.sims
 * Measure: Lin
 * Accept sims range: 0.1 to ?
 * Start time: 5:05:23 PM BST Monday, July 9, 2012
 * Memory used=6.0MiB, allocated=989.9MiB, available=1.9GiB


[INFO] Running all-pairs similarity.
[INFO] Loading features file /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.features.filtered
[INFO] threaded-allpairs: Reading threaded all-pairs. (0% complete)
[INFO] threaded-allpairs: Queueing chunk pair 1 and 1 (0% complete)
[INFO] threaded-allpairs: Starting chunk pair 1 and 1 (50% complete)
[INFO] threaded-allpairs: Completed chunk pair 1 and 1 (100% complete)
[INFO] threaded-allpairs: Finished (90% complete)
[INFO] Completed all-pairs similarity search.
[INFO] 
Stats:
 * End time: 5:05:23 PM BST Monday, July 9, 2012
 * Ellapsed time: 00:00:0.1280
 * Memory used=11.4MiB, allocated=989.9MiB, available=1.9GiB


[INFO] 
=== Stage 5 of 6: K-Nearest-Neighbours ===

[INFO] 
Configuration:
 * Input sims file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.sims
 * Ouput neighbours file: /home/user1/Desktop/NLP/bybloStats/thesauri/bnc-gramrels-fruit-%0-01.sims.neighbours
 * K: 100
 * Start time: 5:05:23 PM BST Monday, July 9, 2012
 * Memory used=6.0MiB, allocated=989.9MiB, available=1.9GiB


[WARN] nThreads (6) > availableProcessors (1)
[INFO] knn-sims: (0% complete)
Exception in thread "main" java.lang.NullPointerException: Argument 'sourceFile' is null.
    at uk.ac.susx.mlcl.lib.Checks.checkNotNull(Checks.java:203)
    at uk.ac.susx.mlcl.lib.tasks.FileCopyTask.setSrcFile(FileCopyTask.java:118)
    at uk.ac.susx.mlcl.byblo.commands.AbstractExternalSortCommand.runTask(AbstractExternalSortCommand.java:295)
    at uk.ac.susx.mlcl.lib.tasks.AbstractTask.run(AbstractTask.java:75)
    at uk.ac.susx.mlcl.lib.AbstractCommandTask.runCommand(AbstractCommandTask.java:73)
    at uk.ac.susx.mlcl.byblo.commands.ExternalSortEventsCommand.runCommand(ExternalSortEventsCommand.java:85)
    at uk.ac.susx.mlcl.byblo.FullBuild.runKNN(FullBuild.java:689)
    at uk.ac.susx.mlcl.byblo.FullBuild.runCommand(FullBuild.java:335)
    at uk.ac.susx.mlcl.lib.commands.AbstractCommand.runCommand(AbstractCommand.java:163)
    at uk.ac.susx.mlcl.byblo.FullBuild.main(FullBuild.java:232)
   Byblo failed on input file: /home/user1/Desktop/NLP/Data/bnc-gramrels-fruit/bnc-gramrels-fruit-%0-01
   Fail Code: 1