Finding the proper frequency multiplier

[FabricioArendTorres]

Hi,

how do you usually initialize the frequency of the network (leaving aside the magic number 30)?
As it is very much problem-dependent, I'm wondering if it is currently just a trial-and-error task or if there is a more
reasonable approach, even if just for finding general scale (I saw you mentioned an value of 3000 for audio data in another issue).

Thanks already in advance, and also for this nice code-base.

[wenqihuang]

Hi, @vsitzmann

Thanks for the very interesting work! I also have the same question as @FabricioArendTorres. I tried to replace the ReLU in a regression network end up with nonsense outputs. It worked well with ReLU. I think the problem would relate to the selection of frequency multiplier. It would be great help if you can provide some suggestions on finding the number. Many thanks!

Best,
Wenqi

[FabricioArendTorres]

@wenqihuang
After working with it some time (for solving PDEs), I can maybe share a bit my experience.

If you get nonsense, the network usually grossly overfits, and it helps to strongly decrease the parameter.
I'd suggest starting from a lower value, and then slowly increasing it. With a validation set it will be really obvious where the range of usable values lies.

Just be aware that changing the number of units and layers might also require a change in the parameter. I found that after increasing the number of units in each layer, you need to decrease the frequency parameter.

Did not find any golden solution other than basically trying it or using some hyperpameter search. I guess that's also a limitation of this work, compared to Random Fourier Feature layers where the tunable parameters a bit more understandable in their effect.

[wenqihuang]

@FabricioArendTorres Thanks for the experience!