Massive compile-time issues since we use a lot of fields in params

[mlankenau]

We changed our maru services so that we define the data-structure in params instead of receiving just a Map.

We really like that it is checking the data that is received and that we get atom keys for all maps, but the compile time has increased massively.

Looking at the maru code it looks like a lot of things are going on in the param-definitions. Is there a change to speed up the macro execution?

[falood]

@mlankenau
how many endpoints / params have you defined?
and have you tested whether the runtime performance is good?

[mlankenau]

we have around 50 endpoints. Most of them have a small number of params (1-5). Some crud endpoints receive our whole data-structure. We have a macro that is generating the maru params from our ecto schemas. That are around 30 params in the worst case. Our compile time is 120 seconds right now.

[falood]

I have a demo https://gist.github.com/falood/3ecb2f038b6093e081ad7161e5444f44 which take about 8 seconds to compile.
It will be very helpful for me to debug it if you can show me what's your routes struct.

[mlankenau]

8 seconds sound ok, we will try to extract our code to a toy project. And measure. Thanx alot for your help. Am Do., 31. Jan. 2019 um 16:26 Uhr schrieb Xiangrong Hao < notifications@github.com>:

…

I have a demo https://gist.github.com/falood/3ecb2f038b6093e081ad7161e5444f44 which take about 8 seconds to compile. It will be very helpful for me to debug it if you can show me what's your routes struct. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAouUvJq57-K_B0Wo2b3WK6QYp7XY9IJks5vIwsegaJpZM4aZ_pI> .

[falood]

@mlankenau

how about this issue? can I do any help with this?

[danielkirch]

I am working with @mlankenau on the same project and now I finally had the time to take a closer look at this issue.

The problem seems to come up in combination with the make_plug option. Here is a minimal example with ~200 parameters which takes about 11 seconds to compile on my computer.
(Without make_plug it takes a bit over a second which would be fine.)

api.ex

[falood]

Hi @danielkirch
Thanks for the feedback.
there's only one huge endpoint in your example, can I ask how many endpoints like this in your system?

[danielkirch]

We have one big param structure that we define in a helper module, but we have 5 endpoints that use it. And for each time we use the big param structure it seems to recompile the whole thing because the compile times add up.

(You can check this with the example: put the params into a helper and use it in 2 endpoints, the compile time should be about twice as long as before.)

[falood]

I see, it's expected to take 10 seconds to compile such a complex parameter block, I'll try to decrease the compile time when we include it in several endpoints.

[danielkirch]

Thanks, that would be a great help for us!

I still wonder why the make_plug option makes such a big difference in compile time though. I tried to figure this out by looking at the project myself, but I didn't get very far. Maybe you can give a short explanation for what is happening behind the scenes there?

[danielkirch]

I think I have finally been able to fix this issue (see PR). :D

It seems that the runtime macro generated by the helper was getting a little bloated by repeated quoting and unquoting. In the new version it will first generate the struct and then convert it into a macro. This change cuts down the compile time from about 10 seconds to under 2 seconds.

[falood]

that's awesome, quote and unquote take a lot of time!
Thanks for figure it out!

[danielkirch]

Sadly, this has become more of an issue again after the changes to the compiler in Elixir 1.8, so we will probably just stick to 1.7 for now. :-/