Paper for pytorch applications?

[gnorman7]

Hi all,

Thanks for the nice code and paper! I'm opening this issue as an inquiry to learn what PyTorch applications you have explored. Or in other words, why implement in PyTorch as opposed to other libraries that support automatic differentiation? I'm also curious about what problem sizes you'd expect this to work. I see that the Hessian is only approximated, which should help with parameter scaling, but what about for the scaling based on the number of constraints (like 1000)? Lastly, do you expect good performance on smooth problems as well? While the paper focuses on nonsmooth problems, how much would that negatively impact the performance as opposed to assuming more smoothness? Perhaps a good (expensive) baseline to ground a discussion would be this scipy optimizer:.

Thanks and best,
Grant

[fabian-sp]

Hi Grant,

sorry for the late response. In general, the motivation to use Pytorch was the following: imagine you have trained a model that represents the objective or a constraints function. Now you want to solve an opt problem that involves your trained function. Our package can handle this as it allows objective/constraint to be as general as a Pytorch Module. Given that Pytorch is the most widely used package for training ML models, it seemed the obvious choice.

However, to be honest, this application of having trained NNs as objective/constraints is something that we imagined would become more and more important - simply due to the idea that classical modelling functions (e.g. for predicting a quantity of interest) might be replaced by learned functions in more and more domains.
We haven't tested this on large-scale problems (due to time constraints). If you have a specific application, I would be very happy to know more and help if I can.