Support for ESP32-P4?

[AIWintermuteAI]

Hello, @vikramdattu !
Do you have any information about how NN accelerated inference is going to be handled in the new P4 chip? Will it be included in ESP NN as well?

[vikramdattu]

@AIWintermuteAI yes, the optimizations will be included in esp-nn for esp32-p4.
Once added, these will get included transparently to esp-tflite-micro.
Unfortunately, I do not have any ETA at the moment though.

[nicklasb]

Oh, I just saw this here, great stuff.

[nicklasb]

@vikramdattu
Hi, testing with an ESP32-P4 dev board (320 Mhz) and realized that there probably are not many P4-specific optimizations enabled?

Because I have an inference (YOLOV5) that takes ESP32-S3 40-50 seconds.
However, the ESP32-P4 using defaults (128 K L2 cache) take 100 seconds. With 512 L2 we are down to 63 seconds.

I don't remember exacly the performance of the S3 unoptimized, but It feels like 100 seconds is roughly how much faster (50 %) I would expect the P4 would be unoptimized? Either that, or the P4 is slower, which I certainly don't hope.

[vikramdattu]

@nicklasb we have just pushed the first-cut support for ESP32-P4.

• Enabled generic optimisations from esp-nn for esp32p4 chip
• Convolution function is optimised using inline assembly

With these optimisations you should see 2x-7x performance boost depending on the model architecture. You should see yet better version for convolution and also optimisations for other kernels as well in follow-up releases.

[nicklasb]

@vikramdattu
So now it is down to 35 seconds, so some improvement, certainly, however no where near 2x-7x, so as of yet, it does less per cycle than the S3, so not super impressed yet. ;-)

Are there some other settings I should consider, or is just the code very immature?

For context, these are the operations. It is kind of typical for a simple YOLOv5 (later YOLO versions kees the making the models hybrid model*.

    resolver.AddConv2D();
    resolver.AddDepthwiseConv2D();
    resolver.AddAdd();
    resolver.AddConcatenation();
    resolver.AddResizeNearestNeighbor();
    resolver.AddRelu();
    resolver.AddPad();
    resolver.AddLogistic();
    resolver.AddMul();
    resolver.AddMaxPool2D();
    resolver.AddReshape();
    resolver.AddStridedSlice();
    resolver.AddQuantize();

I am using TFLite Micro, the model has one class, is about 2 mb and is quantized to int8.
I am not sure if I would make any large gains by using ESP-DL on this one.

* Yes, obviously I am considering other technologies given their lackluster interest, but it is also a good way to measure performance.

[vikramdattu]

Hi @nicklasb you're right.
This is just a start. The optimizations are at an early stage, and that too only conv function optimized right now.
This will definitely be better and surpass S3 in a week or two.

[nicklasb]

Hi @nicklasb you're right. This is just a start. The optimizations are at an early stage, and that too only conv function optimized right now. This will definitely be better and surpass S3 in a week or two.

Great, if you report on what operations are being optimized I can help out with up performance numbers and profiling here.
Because from what I've gathered from the marketing material, I am expecting it to end up being about 40 times faster than the S3. :-)

[vikramdattu]

Great, if you report on what operations are being optimized I can help out with up performance numbers and profiling here. Because from what I've gathered from the marketing material, I am expecting it to end up being about 40 times faster than the S3. :-)

Thanks, I will keep you posted on the progress.
BTW, can you share with me the link for the S3 vs P4 documentation you've referred to? I have missed this particular point. Would help me understand the premise for the claim.

[nicklasb]

BTW, can you share with me the link for the S3 vs P4 documentation you've referred to? I have missed this particular point. Would help me understand the premise for the claim.

I was being a bit ironic, but if you market something with "AI extensions", one expects that the extension will provide something that won't be that much slower than much older chips, like existing edge TPUs. I think few consider "AI" be much less than being able to run deep learning models at a reasonable frame rate.

(and obviously, if you were able to, it would make a whole market open up for you)

[vikramdattu]

@nicklasb thanks for the clarification. Actually, ESP32-S3 also has AI/vector instructions. The one included in P4 is a bit refined version, however.

Please find a patch to optimise the convolution function further:

optimize_conv_generic_case.patch

I added time profiling across some more function and following stats are printed for Yolov5-n (2MB model):

Total time = 28978
FC time = 0
DC time = 0
conv time = 7672
Pooling time = 127
Padding time = 297
add time = 367
mul time = 1626
logistic time = 17625
person score:1%, no person score 99%
I (33384) [esp_cli]: Time required for the inference is 28982 ms

Total time printed now is below 30 seconds. One thing you may notice is most of the time now is consumed by Logistics (Sigmoid) kernel.
This Sigmoid kernel is used by Yolo's model architecture for activation, which is not yet optimised by esp-nn. This will eventually be optimised. May I ask you to replace this with other activation method such as RELU which is much higher on CPU resources?

[nicklasb]

Hi!

Ok, so that was faster, 29 seconds, about 6 seconds faster.
However, the boxes change a bit, the confidence seem to lower slightly with the patch.
It has a filtering confidence at 0.7 and an IOU threshold at 0.1:

# P4 v1 - the one in master
boxes = [
    [295.74, 189.43, 304.70, 207.35, 0.78],
    [295.74, 186.44, 304.70, 210.34, 0.71],
    [295.74, 189.43, 304.70, 207.35, 0.80],
    [221.81, 198.39, 226.29, 210.34, 0.71],
    [322.63, 180.47, 474.98, 204.36, 0.80],
    [318.15, 180.47, 479.46, 204.36, 0.77],
]

# P4 Patch 2024-09-18
boxes = [
    [295.74, 189.43, 304.70, 207.35, 0.76],
    [295.74, 189.43, 304.70, 207.35, 0.78],
    [322.63, 180.47, 474.98, 204.36, 0.78],
    [318.15, 180.47, 479.46, 204.36, 0.77],
]

Don't ask me to debug the code, haven't done any major assembler work since my C64 days, almost 40 years ago. :-)

[nicklasb]

WRT RELU, I'll have to retrain it then, afaik it is a bit cumbersome to change from the defaults in Yolo, I'll be back with that next week, have a lot of stuff this weekend.

[nicklasb]

Annoyingly, something has broken in my Yolov5n export and int8 quantization which breaks the model in some way I have yet to be able to fix, I can't successfully export any yolov5 model currently, ReLU() or not. (and the newer versions doesn't export a pure int8 model leading to TFLite complaining on "not supporting a hybrid model").
I will probably have to use an other architecture.
What is it that you are using, sounded like a similar size model?

[vikramdattu]

Hi @nicklasb I used an existing model at GH user already exported from here which fits our requirement.

I used the guide here to achieve the 2.7MB model with official yolov5-n model.

from ultralytics import YOLO

# Load a model
model = YOLO("yolov5n.pt")  # load an official model

# Export the model
model.export(format="tflite", int8=True)

One thing I noticed, if you can afford to use input image size smaller (640-->320), this should speed up model