Max size of ndarray matrices?

[mars0i]

Is there a maximum size on ndarray matrices? Is this documented somewhere? I'm getting the following error. Note that the matrix seems to have been made successfully, but then I get an out of bounds exception. The total number of cells is 10^5 x 10^5, but the out of bounds error is at a location much less than 10^10. (If it's a Java Array limitation, would it be possible to generate an error when a matrix is created that's too large?)

(ns mwe-ndarray-error
  (:require [clojure.core.matrix :as mx]))

(defn make-mat
  [size]
  (println "In make-mat with size=" size)
  (let [m (mx/new-matrix :ndarray size size)]
    (println "Made matrix, with dimensions" (mx/shape m))
    (doseq [row (range size)
            col (range size)]
      (mx/mset! m col row 1))
    (println "Done initializing matrix")
    m))

(def m (make-env 100000))

; eval (file): ...src/clojure/forage/src/clj/mwe_ndarray_error.clj
; (out) In make-env with size= 100000
; (out) Made matrix, with dimensions [100000 100000]
; (err) Execution error (ArrayIndexOutOfBoundsException) at clojure.core.matrix.impl.ndarray_object.NDArray/set_2d_BANG_ (ndarray_object.clj:14).
; (err) Index 1410100000 out of bounds for length 1410065408
; --------------------------------------------------------------------------------
; eval (effective-root-form): (clojure.repl/pst)
; (err) CompilerException Syntax error macroexpanding at (mwe_ndarray_error.clj:15:8). #:clojure.error{:phase :execution, :line 15, :column 8, :source "mwe_ndarray_error.clj"}
; (err) 	clojure.lang.Compiler$InvokeExpr.eval (Compiler.java:3719)
; (err) 	clojure.lang.Compiler$DefExpr.eval (Compiler.java:457)
; (err) 	clojure.lang.Compiler.eval (Compiler.java:7199)
; (err) 	clojure.lang.Compiler.load (Compiler.java:7653)
; (err) 	clojure.lang.Compiler.loadFile (Compiler.java:7591)
; (err) 	clojure.lang.RT$3.invoke (RT.java:327)
; (err) 	forage.core.env-matrix/eval66341 (NO_SOURCE_FILE:1)
; (err) 	forage.core.env-matrix/eval66341 (NO_SOURCE_FILE:1)
; (err) 	clojure.lang.Compiler.eval (Compiler.java:7194)
; (err) 	clojure.lang.Compiler.eval (Compiler.java:7149)
; (err) 	clojure.core/eval (core.clj:3215)
; (err) 	clojure.core/eval (core.clj:3211)
; (err) Caused by:
; (err) ArrayIndexOutOfBoundsException Index 1410100000 out of bounds for length 1410065408
; (err) 	clojure.lang.RT.aset (RT.java:2381)
; (err) 	clojure.core.matrix.impl.ndarray-object.NDArray (ndarray_object.clj:14)
; (err) 	mwe-ndarray-error/make-env (mwe_ndarray_error.clj:11)
; (err) 	mwe-ndarray-error/make-env (mwe_ndarray_error.clj:4)
; (err) 	clojure.lang.Compiler$InvokeExpr.eval (Compiler.java:3714)
; (err) 	clojure.lang.Compiler$DefExpr.eval (Compiler.java:457)
; (err) 	clojure.lang.Compiler.eval (Compiler.java:7199)
; (err) 	clojure.lang.Compiler.load (Compiler.java:7653)

[mars0i]

Update: The same problem occurs with mget. It doesn't occur with :persistent-vector. I haven't tried other implementations, but I'm guessing that it's specific to ndarrays.

[mikera]

I believe the NDArray maxes out at the Java int size (for array indexes). It should indeed be specific to ndarray. In general, different implementation can have different constraints on array shapes (total size, max number of dimensions etc.)