Support for POJO, bring it back!

[weimeilin79]

Currently POJO is no longer taken in by Camel K. Instead of restricting and directly rejecting it, can we have a window open for this scenario, provide it as an optional feature.

Without the support for POJO meaning there is no easy DEV mode for Camel K, basically it’s the death of live coding. Instead of seeing the beauty of seeing code directly running on K8s, now the flow goes like this:

Developer creates another project for ANY bean/helper/util class, compiles, and uploads to jitpack, (what if it’s enterprise env? Developers go through another process of uploading it? Meaning the OPS needs to get either Nexus or Jitpack whatever ready? )
Or just do an awful long javascript like all-in-one java class?
Where is the agility? Where is the fast time to deploy?

Yes, separating it could be better for unit testing, (I understand the need for unit testing? But aren’t we pushing BDD for integration code? ) BUT!!! how about the painful process before unit test, during development? It’s not like you can run Camel K route easily locally to test the utility? Meaning the developer has to shift between local and k8s ..

It'll just be easier to do a simple Quarkus java app, at least there is no two separate env I need to work with, and two separate deployments process.

I know there would be problems for Quarkus runtime, and native compile and maybe longer build time. But having to do things separately not only breaks the train of thoughts, but also requires developers to do a lot of extra work.

And another thing is about the positioning with Kamelet, if Camel K used JUST for route? I don’t see a clear definite line between Kamelet and this?

Can we have an option for POJO, say if this is there, then default runtime won't be Quarkus or with Native build. And warning on slower build time.. etc..

[lburgazzoli]

I am the one to be blamed for have introduced and then removed this feature.

When this feature was first introduced, Camel was in its 2.x ages and features like properties binding and configuration over properties were not powerful as it is today, so having a way to easily configure components or other aspect of Camel was really needed and that's why I added it but it was never been intended to let people adding arbitrary code. As today the Camel engine is much more powerful and the reasons for which this has been introduced do not longer exist.

Unfortunately people as start abusing of this feature and I had to deal with lot of issues, some have been solved but some won't as camel-k is not a generic build system and it will never be.

So what issues do we have by letting people adding any arbitrary code through camel-k ?

1. each class is compiled in its own class-loader which make the resulting classes not visible from other classes (this has partially been solved but I don't want to bring yet another modular class-loader which greatly complicates things)
2. dependencies between classes is not taken into account and it will never be so the compilation may fail at runtime if any of your class depend on another class you have added (maybe you are lucky and the order do not unveil the problem but still, it is unpredictable)
3. size of the resulting CR may exceed the limit resources have (bear in mind that all your sources are stored inside etcd)
4. consistency with other languages: should we support adding arbitrary js/groovy/kotlin code ?
5. code is compiled at runtime, so every time the application starts, it will spend some time compiling your code which is not exactly what when your integration needs to quickly scale up
6. you may miss the option to use native compilation

That said I can evaluate to re introduce this feature but certainly not in its previous incarnation and you need to help me to shape an UX that do not encourage users to abuse of this capability and does not introduce yet another unmaintainable feature.

[lburgazzoli]

And another thing is about the positioning with Kamelet, if Camel K used JUST for route? I don’t see a clear definite line between Kamelet and this?

In essence, Kamelets are just like any regular camel component that instead of just providing support to exchnage data with a 3th party system, can be used to package complex routing logic and dependencies hiding the detail from the end user.

I don't see how it conflicts with other things.

[weimeilin79]

Yes, I totally agree that we should not abuse it. But also we don’t want to completely wipe out the flexibility. In order for a better UX and more efficient engine, we need to take a closer look at their behaviour. Because sometimes I don’t code in a certain way, it’ll be good to take a look at some older issue and their problem repo to setup some rules?

I am the one to be blamed for have introduced and then removed this feature.

Not blaming :) Just want to know why?

So what issues do we have by letting people adding any arbitrary code through camel-k ?

1. each class is compiled in its own class-loader which make the resulting classes not visible from other classes (this has partially been solved but I don't want to bring yet another modular class-loader which greatly complicates things)

What is the current limitation? In what situation could it be possible for people to do it?(Sorry not expert in Quarkus app. ) Do we have an example of how their code would need to have more class loaders?

2. dependencies between classes is not taken into account and it will never be so the compilation may fail at runtime if any of your class depend on another class you have added (maybe you are lucky and the order do not unveil the problem but still, it is unpredictable)

Pardon my ignorance, this problem is caused by the previous independent loader issue, right?

3. size of the resulting CR may exceed the limit resources have (bear in mind that all your sources are stored inside etcd)

Yes, we’ll have to define some rules here.

4. consistency with other languages: should we support adding arbitrary js/groovy/kotlin code ?

This probably will need some help from the experts doing these DSL, but most of people I see are doing Java or XML.. (some yaml)

5. code is compiled at runtime, so every time the application starts, it will spend some time compiling your code which is not exactly what when your integration needs to quickly scale up

Understood. This could be a warning.

6. you may miss the option to use native compilation

Another possible option for users to decide.

That said I can evaluate to re introduce this feature but certainly not in its previous incarnation and you need to help me to shape an UX that do not encourage users to abuse of this capability and does not introduce yet another unmaintainable feature.

From reading the cons, IMHO the biggest problem is the class loaders, which could potentially may cause the application not runnable. Others I would think it’s more like an decision for users to decide.

[zregvart]

I think the biggest drawback of Camel K being able to address moderate to high complexity cases is in it's over-reliance on having to fit the whole implementation fit in the CR. I think Camel K should consider supporting container images or archives instead. So instead of putting the code into a CR, I as a developer, should be able to push an image/archive with the code instead, have the operator integrate it into an runnable image.

[lburgazzoli]

This is actually not accurate, we need the route as it is what drives the materialization of various aspects of the integration but we actually recommend providing additional logic through artifacts

On Sat, 21 Nov 2020 at 17:33, Zoran Regvart ***@***.***> wrote: I think the biggest drawback of Camel K being able to address moderate to high complexity cases is in it's over-reliance on having to fit the whole implementation of the solution in the CR. I think Camel K should consider supporting container images or archives instead. So instead of putting the code into a CR, I as a developer, should be able to push an image/archive with the code instead, have the operator integrate it into an runnable image. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1821 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOIJBIOI6B3WMUSLZ5IGNTSQ7TVLANCNFSM4T5CPGOA> .

-- -- Luca Burgazzoli

[zregvart]

This is actually not accurate

Yes, the wording doesn't pay it justice, but I don't think it's inaccurate. Let me rephrase this with an example: we moved to Linux containers that are self contained, i.e. all dependencies that are needed at runtime are present within one container image. This is something we did to achieve idempotency and immutability, which in turn gave us reliability and scalability. The model of having the Camel route the only executable bit of my integration inside of a CR and then ship dependencies separately, I think, goes against that. For development and for production I would like to bundle everything together, much as the container image bundles all dependencies together.

[weimeilin79]

Yes, @zregvart I think we are both thinking about the same thing, it's the packaging to deployment experience, if everything are broken down to too many pieces, and the developer were left to do everything manually, it's not ideal.

[liurui-1]

Thanks @lburgazzoli to mention this issue.
I fully agree and suggest to support POJO again, maybe with some restrictions, especially when Camel v1.4.0 isolate all Java files. As I mentioned in #2317 , we badly need some features to reuse our code. We need to install our product into many customer environments, it is not a good solution to build those constants and utility functions into Maven artifacts.

[oscerd]

Well, it's not good to use multiple class of constants and utility functions in many customer enviroments. If you need to change anything, you'll need to change it everywhere.

[liurui-1]

@oscerd As I mentioned in #2317 , I do not see any people upload their utility class into public Maven just for private use in their own app. And we have the unique version for Camel code and the utility. Camel code is more important. I cannot imagine if you upgrade utility class with Maven but do not upgrade the Camel code at the same time.

[arthurdm]

hey everyone - I agree with a lot of points made here from both sides. There should be a way to reuse POJOs that is lightweight and doesn't require a lot of process, to enable development teams that are pulling from a common source repo within an organization.

At the same time, having all of those POJOs duplicated inside different CRs definitely can quickly hit size limitations. I guess the CR size ceiling is something we could hit even without POJOs, just with an extremely long Route class. There may be some higher architectural decisions there in terms of is Camel K well suitable for complex projects versus smaller re-routing / transformation module?

I think @zregvart made some very good arguments for solving this via container images that have the code + dependencies. The Camel K CR could then just reference the source code image, and the Operator would take over from there in terms of creating a runnable image that has the compiled code + dependencies.

[lburgazzoli]

I think @zregvart made some very good arguments for solving this via container images that have the code + dependencies. The Camel K CR could then just reference the source code image, and the Operator would take over from there in terms of creating a runnable image that has the compiled code + dependencies.

For my understanding, so that I can evaluate better, how this would be different from shipping some POJOs as jar ?

[arthurdm]

For me the two differentiators are:

• all of the code, including the core Route, lives outside of the Integration CR - thus removing the danger of running over the CR limit
• packaging the code and common POJOs in a container would remove the need for setting up a maven artifact for the POJO code