Add new LLM rules #472
Add new LLM rules #472
Conversation
grafana-plugins-platform-bot
bot
moved this from 📬 Triage
to 🔬 In review
in Grafana Catalog Team
| ExpectedAnswer: false, | ||
| }, | ||
| { | ||
| Question: "For panel plugins (check plugin.json type field): Does this plugin include a backend component (Go code in pkg/ directory, .go files in backend/, or any Go backend implementation)? (Panel plugins cannot have backend components in Grafana; only app and datasource plugins can include backend components. If this is a panel plugin with backend code, it must be converted to an app plugin). Provide the specific plugin.json type and backend file locations if found.", |
There was a problem hiding this comment.
feels like this could be a very simple check without needing LLM - e.g. plugin ID = "something-something-panel" and "backend:true"
There was a problem hiding this comment.
I guess so, also it's a so-obvious issue that we may not need to add it here, I'll remove it.
academo
left a comment
academo
left a comment
There was a problem hiding this comment.
many of the questions in this PR are very good but they are better as semgrep rules than questions to the LLM.
semgrep rules are deterministic, if semgrep finds something that matches our search criteria, we know for sure it is a problem (error/warning).
the LLM will constantly miss things or make up errors where there is none and it will start generating a lot of noise we will start ignoring.
also, there are some questions that are absolutely good to have here for the LLM but that require more context. we should specially add examples of what we are looking for if possible in the question itself.
also, @andresmgot this LLMreview will run these questions in all the plugin code, consider if maybe some of these questions you want to move them to the codediff validator that only runs the questions in "new code" and not in all the code.
3 participants
Based on our existing responses to plugin developers.
I have also divided the questions in two: Errors and warnings, to differentiate between things that needs to be fixed or suggestions for future submissions.