Problem about big files

[liugui]

Thanks for your nice work! I use the sequitur of C++ version to compress my file, and it works well with small files about 240MB. However, when I use it to compress a larger file, for example, around 2.7GB, it becomes slower and slower, even more than 25 hours, and finally stopped. Here is my command line:

 ./sequitur -c < ../data/compress/normal.qual > ../data/compress/normal.qual.sequitur 

the size of normal.qual is 2.7G. Here is the log:

-1877 MB processed, 0.02 MB/s, 0.510 collisions/lookup, 77.63% occupancy
-1876 MB processed, 0.02 MB/s, 0.500 collisions/lookup, 77.66% occupancy
-1875 MB processed, 0.02 MB/s, 0.490 collisions/lookup, 77.69% occupancy
-1874 MB processed, 0.02 MB/s, 0.479 collisions/lookup, 77.72% occupancy
-1873 MB processed, 0.02 MB/s, 0.469 collisions/lookup, 77.75% occupancy
-1872 MB processed, 0.01 MB/s, 0.459 collisions/lookup, 77.78% occupancy
-1871 MB processed, 0.01 MB/s, 0.448 collisions/lookup, 77.81% occupancy
-1870 MB processed, 0.01 MB/s, 0.437 collisions/lookup, 77.84% occupancy
-1869 MB processed, 0.01 MB/s, 0.426 collisions/lookup, 77.88% occupancy
-1868 MB processed, 0.01 MB/s, 0.415 collisions/lookup, 77.91% occupancy
-1867 MB processed, 0.01 MB/s, 0.404 collisions/lookup, 77.94% occupancy
-1866 MB processed, 0.01 MB/s, 0.393 collisions/lookup, 77.97% occupancy
-1865 MB processed, 0.01 MB/s, 0.382 collisions/lookup, 78.01% occupancy
-1864 MB processed, 0.01 MB/s, 0.370 collisions/lookup, 78.04% occupancy
-1863 MB processed, 0.01 MB/s, 0.358 collisions/lookup, 78.07% occupancy
-1862 MB processed, 0.01 MB/s, 0.347 collisions/lookup, 78.10% occupancy
-1861 MB processed, 0.01 MB/s, 0.335 collisions/lookup, 78.13% occupancy
-1860 MB processed, 0.01 MB/s, 0.323 collisions/lookup, 78.16% occupancy
-1859 MB processed, 0.01 MB/s, 0.310 collisions/lookup, 78.19% occupancy
-1858 MB processed, 0.01 MB/s, 0.298 collisions/lookup, 78.22% occupancy
-1857 MB processed, 0.01 MB/s, 0.285 collisions/lookup, 78.25% occupancy
-1856 MB processed, 0.01 MB/s, 0.272 collisions/lookup, 78.28% occupancy
-1855 MB processed, 0.01 MB/s, 0.259 collisions/lookup, 78.31% occupancy
-1854 MB processed, 0.01 MB/s, 0.246 collisions/lookup, 78.34% occupancy
-1853 MB processed, 0.01 MB/s, 0.233 collisions/lookup, 78.37% occupancy
-1852 MB processed, 0.01 MB/s, 0.220 collisions/lookup, 78.40% occupancy
-1851 MB processed, 0.01 MB/s, 0.206 collisions/lookup, 78.43% occupancy
-1850 MB processed, 0.01 MB/s, 0.192 collisions/lookup, 78.46% occupancy
-1849 MB processed, 0.01 MB/s, 0.178 collisions/lookup, 78.49% occupancy
-1848 MB processed, 0.01 MB/s, 0.164 collisions/lookup, 78.52% occupancy
-1847 MB processed, 0.01 MB/s, 0.150 collisions/lookup, 78.55% occupancy
-1846 MB processed, 0.01 MB/s, 0.135 collisions/lookup, 78.58% occupancy
-1845 MB processed, 0.01 MB/s, 0.121 collisions/lookup, 78.61% occupancy
-1844 MB processed, 0.01 MB/s, 0.106 collisions/lookup, 78.64% occupancy
-1843 MB processed, 0.01 MB/s, 0.090 collisions/lookup, 78.67% occupancy
-1842 MB processed, 0.01 MB/s, 0.075 collisions/lookup, 78.70% occupancy
-1841 MB processed, 0.01 MB/s, 0.060 collisions/lookup, 78.73% occupancy
-1840 MB processed, 0.01 MB/s, 0.044 collisions/lookup, 78.76% occupancy
-1839 MB processed, 0.01 MB/s, 0.028 collisions/lookup, 78.79% occupancy
-1838 MB processed, 0.01 MB/s, 0.012 collisions/lookup, 78.82% occupancy
-1837 MB processed, 0.01 MB/s, -0.005 collisions/lookup, 78.85% occupancy
-1836 MB processed, 0.01 MB/s, -0.021 collisions/lookup, 78.88% occupancy
-1835 MB processed, 0.01 MB/s, -0.038 collisions/lookup, 78.91% occupancy
-1834 MB processed, 0.01 MB/s, -0.055 collisions/lookup, 78.94% occupancy
-1833 MB processed, 0.00 MB/s, -0.072 collisions/lookup, 78.97% occupancy
-1832 MB processed, 0.00 MB/s, -0.089 collisions/lookup, 79.00% occupancy
-1831 MB processed, 0.00 MB/s, -0.107 collisions/lookup, 79.03% occupancy
-1830 MB processed, 0.00 MB/s, -0.124 collisions/lookup, 79.06% occupancy
-1829 MB processed, 0.00 MB/s, -0.142 collisions/lookup, 79.09% occupancy
-1828 MB processed, 0.00 MB/s, -0.161 collisions/lookup, 79.12% occupancy
-1827 MB processed, 0.00 MB/s, -0.179 collisions/lookup, 79.15% occupancy
-1826 MB processed, 0.00 MB/s, -0.198 collisions/lookup, 79.18% occupancy
-1825 MB processed, 0.00 MB/s, -0.217 collisions/lookup, 79.21% occupancy
-1824 MB processed, 0.00 MB/s, -0.236 collisions/lookup, 79.25% occupancy
-1823 MB processed, 0.00 MB/s, -0.256 collisions/lookup, 79.28% occupancy
-1822 MB processed, 0.00 MB/s, -0.276 collisions/lookup, 79.31% occupancy
-1821 MB processed, 0.00 MB/s, -0.296 collisions/lookup, 79.34% occupancy
-1820 MB processed, 0.00 MB/s, -0.316 collisions/lookup, 79.37% occupancy
-1819 MB processed, 0.00 MB/s, -0.337 collisions/lookup, 79.40% occupancy
-1818 MB processed, 0.00 MB/s, -0.359 collisions/lookup, 79.43% occupancy
-1817 MB processed, 0.00 MB/s, -0.380 collisions/lookup, 79.46% occupancy
-1816 MB processed, 0.00 MB/s, -0.402 collisions/lookup, 79.49% occupancy
-1815 MB processed, 0.00 MB/s, -0.425 collisions/lookup, 79.52% occupancy
-1814 MB processed, 0.00 MB/s, -0.448 collisions/lookup, 79.55% occupancy
-1813 MB processed, 0.00 MB/s, -0.471 collisions/lookup, 79.58% occupancy
-1812 MB processed, 0.00 MB/s, -0.494 collisions/lookup, 79.61% occupancy
-1811 MB processed, 0.00 MB/s, -0.518 collisions/lookup, 79.64% occupancy
-1810 MB processed, 0.00 MB/s, -0.543 collisions/lookup, 79.67% occupancy

At the beginning, the speed is around 3.44MB/s and then becomes slower and slower, finally 0.00MB/s. Besides, the "XXXX MB processed" even becomes negative. How I can solve this problem? Thanks very much for any reply!