java.lang.OutOfMemoryError with mapping large file in FileChannel #78
Description
When calling FileChannel.map on a large file, OutOfMemoryError is thrown.
Am unable to replicate this problem under on my own system.
[INFO]
=== Running full thesaurus build ===
[INFO]
Configuration:
* Input instances file: /home/user1/Desktop/NLP/Data/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2
* Output directory: /home/user1/Desktop/NLP/bybloStats/output1/thesauri (27.0GiB free)
* Temporary directory: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/tempdirtmp-8511953953287541189.tmp (27.0GiB free)
* Character encoding: UTF-8
* Num. Threads: 6
* Memory used=5.5MiB, allocated=989.9MiB, available=1.9GiB
* Java Spec: Java Platform API Specification 1.6, Sun Microsystems Inc.
* Java VM: OpenJDK Client VM 20.0-b12, Sun Microsystems Inc.
* Java Runtime: OpenJDK Runtime Environment 1.6.0_24-b24
* OS: Linux 3.2.0-26-generic-pae i386
* Start time: 22:59:04 o'clock BST Wednesday, 11 July 2012
[INFO]
=== Stage 1 of 6: Enumerating Strings ===
[INFO]
Configuration:
* Input instances file: /home/user1/Desktop/NLP/Data/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2
* Output enumerated instances file: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2.enumerated
* Output entry index: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2.entry-index
* Output feature index: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2.feature-index
* Start time: 22:59:05 o'clock BST Wednesday, 11 July 2012
* Memory used=5.9MiB, allocated=989.9MiB, available=1.9GiB
Exception in thread "main" java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:803)
at uk.ac.susx.mlcl.lib.io.CharFileChannel.insureMapped(CharFileChannel.java:402)
at uk.ac.susx.mlcl.lib.io.CharFileChannel.read(CharFileChannel.java:316)
at uk.ac.susx.mlcl.lib.io.Lexer.insureRemaining(Lexer.java:494)
at uk.ac.susx.mlcl.lib.io.Lexer.advance0(Lexer.java:294)
at uk.ac.susx.mlcl.lib.io.Lexer.advance(Lexer.java:279)
at uk.ac.susx.mlcl.lib.io.TSV$Source.<init>(TSV.java:221)
at uk.ac.susx.mlcl.byblo.io.TokenPairSource.open(TokenPairSource.java:102)
at uk.ac.susx.mlcl.byblo.io.BybloIO.openInstancesSource(BybloIO.java:55)
at uk.ac.susx.mlcl.byblo.commands.IndexingCommands$IndexInstances.openSource(IndexingCommands.java:240)
at uk.ac.susx.mlcl.byblo.commands.AbstractCopyCommand.runCommand(AbstractCopyCommand.java:128)
at uk.ac.susx.mlcl.byblo.commands.IndexingCommands.runCommand(IndexingCommands.java:76)
at uk.ac.susx.mlcl.byblo.FullBuild.runIndex(FullBuild.java:405)
at uk.ac.susx.mlcl.byblo.FullBuild.runCommand(FullBuild.java:294)
at uk.ac.susx.mlcl.lib.commands.AbstractCommand.runCommand(AbstractCommand.java:163)
at uk.ac.susx.mlcl.byblo.FullBuild.main(FullBuild.java:232)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:800)
... 15 more
Moved back to /home/user1/Desktop/NLP/bybloStats
Byblo failed on input file: /home/user1/Desktop/NLP/Data/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2
Fail Code: 1