How to count FLOPs?

[miladdona]

Hi,

How can I count number of floating point operation in kerasdense layers in this library (t3f)?

In normal way we do Y = AX + b in a dense layer in keras:
Y: output
X: input
A: weights (in dense layers is a 2-D matrix in shape (m, n))
b: bias
num of FLOPs: m * n + b

But in t3f I don't know how can I count FLOPs?

Thanks

[Bihaqo]

Hi,

You can ask TF about flops count of some function as discussed here: tensorflow/tensorflow#32809

But when I tried using it on the TT layer, it returns zero: https://colab.research.google.com/drive/1LsXKfmEuPLwEgjmSyzS95Xb4qmw4vwfo?usp=sharing

I'll figure it out..

[Bihaqo]

I think this should be about right: https://colab.research.google.com/drive/16S_SUbIjhnQBFj_r7sCpbwZNHADIzEwX?usp=sharing

(Note that the FLOP count in this example is not very impressive because the layer is super small)

[miladdona]

Thanks for your answer.
Why do you mentioned in this link https://colab.research.google.com/drive/16S_SUbIjhnQBFj_r7sCpbwZNHADIzEwX?usp=sharing Approximate number of flops of a TT-layer.?
Why it is not accurate?!

[Bihaqo]

The bottleneck of the TT-layer is tensor contractions (~= matrix multiplications) and they are counted correctly here. However, there are also much cheaper reshapes and axis transposes and I don't know exactly which and how many of them there are because to implement the layer I use tf.einsum operation which automatically converts tensor contraction rule into a sequence of reshapes / transposes / matrix multiplications. So in the function above I added approximate number of FLOPs that is enough to do one reshape, but this might not be entirely accurate.

[Aelphy]

you can count the flops of einsum using np.einsum_path

[miladdona]

Thank you for your answer.

I had a mistake in the post: num of FLOPs: m * n + b
this is num of params in a dense layer
and num of flops in a dense layer calculate by : 2 * m * n
anyway, in general I found by using this library we decrease the num of params (memory) and increase the num of flops (speed). Am I right?

I have access to this link:
https://t3f.readthedocs.io/en/latest/
and the papers mentioned in this link.
Do you have any other document or tutorial to understand this library (in depth)?

Thanks.
Miladdona

[Aelphy]

depends on the settings, sometimes it is possible to reduce FLOPs as well with really low rank

[miladdona]

Why do you use this statement at the end of for-loop:
input_right *= core_left / core_left
Is this a mistake or no?

[miladdona]

Also, input_left and core_right are equal.
This is correct or no?