Fail when metric given as function

[mpekalski]

I have a model that I compile with

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[matthews_correlation])

and it fails after the first epoch

17/18 [===========================>..] - ETA: 0s - loss: 14.9240 - <function matthews_correlation at 0x7f61611d0c80>: 0.0000e+00Traceback (most recent call last):
  File "<stdin>", line 19, in <module>
  File "/opt/conda/lib/python3.6/site-packages/importance_sampling/training.py", line 137, in fit
    on_scores=on_scores
  File "/opt/conda/lib/python3.6/site-packages/importance_sampling/training.py", line 289, in fit_dataset
    batch_size=batch_size
  File "/opt/conda/lib/python3.6/site-packages/importance_sampling/model_wrappers.py", line 75, in evaluate
    for xi, yi in self._iterate_batches(x, y, batch_size)
  File "/opt/conda/lib/python3.6/site-packages/importance_sampling/model_wrappers.py", line 75, in <listcomp>
    for xi, yi in self._iterate_batches(x, y, batch_size)
  File "/opt/conda/lib/python3.6/site-packages/importance_sampling/model_wrappers.py", line 298, in evaluate_batch
    print(len(outputs))
  File "/opt/conda/lib/python3.6/site-packages/numpy/core/shape_base.py", line 288, in hstack
    return _nx.concatenate(arrs, 1)
ValueError: all the input arrays must have same number of dimensions

when I run with 'accuracy' as a metric e.g.

model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])

everything is fine.

I've tried it even with a dummy metric

def test_metric(x,y):
    return tf.constant(1.0, dtype=tf.float32)

and it also fails.

I am using CUDA 10.0, TF '1.13.0-rc1', Keras 2.2.4 and latest keras-imporatance-sampling installed via pip. Without ImportanceTraning everything runs fine.

[mpekalski]

I mean it finishes training on first batch but then fails on evaluation.

[mpekalski]

So in my case the shapes of outputs in evaluate_batch within morel_wrappers.py are

[(), (128, 1), (128, 1), (128, 1), (1,)]

in case of metrics=["accuracy"] the shapes look like

[(), (128, 1), (128, 1), (128, 1), (128, 1)]
[(), (128, 1), (128, 1), (128, 1), (128, 1)]
[(), (128, 1), (128, 1), (128, 1), (128, 1)]
[(), (128, 1), (128, 1), (128, 1), (128, 1)]
[(), (70, 1), (70, 1), (70, 1), (70, 1)]

[mpekalski]

Tracking further, looks like the metric passed as a function is not applied to each sample in a batch, o

I've added comments around the part that creates MetricLayer to see what is the output for accuracy and my own metric and they differ in shape. Looks like one has batch coefficient (?) and the other does not.

        print(f"metrics: {metrics}")
        metrics = [
            MetricLayer(metric)([y_true, model.get_output_at(0)])
            for metric in metrics
        ]
        print(f"metrics: {metrics}")

Output

metrics: ['accuracy']
metrics: [<tf.Tensor 'metric_layer_1/ExpandDims_1:0' shape=(?, 1) dtype=float32>]

metrics: [<function matthews_correlation at 0x7f65ae953c80>]
metrics: [<tf.Tensor 'metric_layer_1/ExpandDims_1:0' shape=(1,) dtype=float32>]

[mpekalski]

In a call method within layers/metrics.py it looks like the input in both cases 'accuracy' and metric passed as a function gets the same input (in terms of shape)

    def call(self, inputs, mask=None):
        # Compute the metric
        metric = self.metric_func(*inputs)
        print(f"inputs {inputs}")

inputs [<tf.Tensor 'input_2:0' shape=(?, 1) dtype=float32>, <tf.Tensor 'activation_1/Softmax:0' shape=(?, 1) dtype=float32>]

but still metric passed as a function returns different shape than passed as a string.
Seems like unpacking, i.e., *inputs, does not work as intended.

[mpekalski]

For me a fix is to change (in layers/metrics.py)

metric = self.metric_func(*inputs)

to

        metric = K.map_fn(lambda x: self.metric_func(x[0],x[1]),   K.concatenate(inputs, axis=-1))

but this solution is not generic enough. And here you cannot use *x to unpack a tensor.

[mpekalski]

The problem I see now, with my fix is that I do see one value for my metric (for the whole batch) but I think it is wrong as this metric should not be calculated on a single sample but whole batch. Hence, I do not know if my PR really fixed the problem and the problem is with metric, or the other way around.

[mpekalski]

Ok, so the problem is that my metric needs to be evaluated on the whole batch and does not make sense when calculated on a single sample, hence it cannot be fed to mean().

[angeloskath]

Hi,

Thank you for the contribution and sorry for the relatively slow reply. I pushed a fix at branch scalar-metrics (e5dac90). I also added a test. In case you have use case that is not covered, you can add it there. I will merge and do another release in due time.

Thanks,
Angelos

[mpekalski]

No problem. I see your solution is more elegant than mine. :)