Merge 4-0-284 to backup_sync #199
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* [pysrc2cpg] refactor <module>, ANY and __init__ constants * Replace Constants/Defines, cf. feedback suggestion
* minor changes: code format * add "isInline" method for "CallMethods" * add a method "isInline" for CallTraversal
…esses. (joernio#5036) * [javasrc2cpg] Generate TYPE_REF for implicit base of static field accesses. For static field access like `String x = staticField1;` we used to generate a IDENTIFIER with the name of the class declaring the static field as base for the field access operator. Now we generate a TYPE_REF referencing the declaring type. This better represents the actual situation where there is no local variable for a class/type which in practice also resulted in a CPG format error because the IDENTIFIFER node was at least not in all cases linked to a LOCAL. While implementing this i reorganized the code a bit to avoid duplication. Also changes the `equals` semantic for TrackedTypeRef. Before it was based on the actual TYPE_REF node which does not make sense. Instead we need to compare the referenced TYPE itself to check if two different type references in code point to the same TYPE. * Handle statically imported field scenario.
Due to the performance and stability issues with a 2-phase LLK parser approach with ANTLR, this branch, developed over the past two weeks, introduces the usage of a [`parser`](https://github.com/whitequark/parser/tree/master) based approach wrapped in [`ruby_ast_gen`](https://github.com/joernio/ruby_ast_gen). This Gem is used by many popular Ruby linters and static analysis tools, and is compatible with Ruby versions across 1-3. We bundle `ruby_ast_gen` with the frontend under `resources`, and execute the Ruby code using JRuby to maximize portability. If the resource is detected to be in a JAR, `ruby_ast_gen` is written to a temporary directory, from where it is then executed. Both JRuby and native Ruby runs of `ruby_ast_gen` far outperform the old ANTLR version, which would often have worst-case performance for some set of language constructs encountered. Changes include: * Remove ANTLR implementation * Execute `ruby_ast_gen` and parse JSON output * Map the JSON output to the same intermediate AST nodes that ANTLR used to avoid changing `AstCreation` logic * Rename the method name for executable strings (`%x{}`-style strings) to `<operator>.backticks` instead of `exec` as `exec` technically has different semantics * Trim `.code` properties to 60 characters to avoid large `.code` properties. (this is done by `ruby_ast_gen`) * Added `db/migration` to default ignored directories Tests are largely kept the same, and `ruby:test` is green on `codescience` after some small `.code` and `.lineNumber` tweaks. See `dave/ruby/prep-for-ruby-parse-gem`. The `astGenResourceTask` task in `build.sbt` is tied to `Compile / resourceGenerators` as the task fetches the zipped Ruby code and places it under `resources` to be packaged alongside the classes in the JAR. --------- Co-authored-by: Andrei Dreyer <andrei@whirlylabs.com> Co-authored-by: Andrei Dreyer <andrei.dreyer@adagintech.com>
…condition (joernio#5046) * [gosrc2cpg]handle the init code before if condition. * [gosrc2cpg]handle the init code before if condition, fix the test error. * [gosrc2cpg]after formatting.
We migrated to sonatype central due to publishing problems, as suggested by their support. Let's see if this works. https://central.sonatype.com/
* Moved lowering for ||= and &&= to AstCreator * Moved lowering func to AstCreatorHelper trait
* upgrade deps * latest
…nio#5071) * [kotlin2cpg] Dispose the environment as soon as it's not needed * scalafmt
…ode (joernio#5076) * [javasrc2cpg] support flows for field accesses with a TYPE_REF base node * scalafmt * simplify `.where(_.or...)` with `.or(...)` cf. review suggestion Co-authored-by: maltek <1694194+maltek@users.noreply.github.com> --------- Co-authored-by: maltek <1694194+maltek@users.noreply.github.com>
* Convert `FieldIdentifier` nodes to `MemberAccess` if they reach AstCreator for call targets * Handle forwarded args which were a cause for some null pointer exceptions * Fixed `kwrestarg` null value issue, which turned out to be a `**` parameter Resolves joernio#5063 Resolves joernio#5064 Resolves joernio#5065
Signed-off-by: ricekot <github@ricekot.com>
* [ruby] Bitwise Assignment Operators Added handling for `|=`, `&=`, `<<=`, and `>>=` * Added tests
Add modifiers to members --------- Co-authored-by: NextToMinus <NextToMinus@gmail.com> Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com> Co-authored-by: StackOF <sfncat@gmail.com>
…detecting types for PHP in queryDB (joernio#5327) * fix: Add PHPJoern queries for SQL injection, command injection, code injection, unrestricted file upload, and XSS detection * fix: Improve PHP version comparison to handle version strings correctly * fix: Enhance PHP version parsing and update SQL injection query titles for clarity * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the source identification Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the source identification Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the description about the XSS attack Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the description about the code execution attack Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the description about the SQLi attack Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the default iterator accessing Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com> * fix: Add PHP security test suite and update dependencies * fix: Update PHPJoernTests to reflect changes in user input handling * refactor: Improve code formatting and readability in PHP-related files --------- Co-authored-by: David Baker Effendi <davidbakereffendi@hotmail.com>
* Make Gradle dependency fetcher init script more robust * Fetch class jars instead of source jars in init script
* parallelize delombok-ing * Update joern-cli/frontends/javasrc2cpg/src/main/scala/io/joern/javasrc2cpg/util/Delombok.scala Co-authored-by: maltek <1694194+maltek@users.noreply.github.com> --------- Co-authored-by: maltek <1694194+maltek@users.noreply.github.com>
* [jimple2cpg] if a member have ConstantValueTag in SootField info, add the tag value to the member possibleTypes. add tests in MemberTests. * [jimple2cpg] change the constantValue to the member code --------- Co-authored-by: NextToMinus <NextToMinus@gmail.com>
To be consistent with other dynamic calls, adds a dynamic type hint to literals + sets `match` to be `DynamicCallUnknownFullName` to enable dynamic call resolution strategies to kick in.
…oernio#5332) * Removed last File.newTemporaryDirectory instances * code cleanup * fix jimple failing tests * fix jimple tests * Ensure all file streams are closed to allow file delete on windows * fmt * fix gradle destinationDir * code cleanup --------- Co-authored-by: Andrei Dreyer <andrei.dreyer@adagintech.com>
* Removed last File.newTemporaryDirectory instances * code cleanup * fix jimple failing tests * fix jimple tests * Ensure all file streams are closed to allow file delete on windows * fmt * fix gradle destinationDir * code cleanup * remove all usages of better.files.File.newTemporaryFile * remove all usages of better.files.File.newTemporaryFile * remove touch-like function --------- Co-authored-by: Andrei Dreyer <andrei.dreyer@adagintech.com>
…ntent via offset (joernio#5341) * Add sourceCode accessor to nodes and traversals for accessing source code via offset * Update semanticcpg/src/main/scala/io/shiftleft/semanticcpg/language/nodemethods/AstNodeMethods.scala Co-authored-by: Michael Pollmeier <michael@michaelpollmeier.com> * Separate offset and sourceCode tests * Improve offset test descriptions --------- Co-authored-by: Michael Pollmeier <michael@michaelpollmeier.com>
* ghidra: use our fork https://github.com/joernio/ghidra * use first release made from github actions runner https://github.com/joernio/ghidra/actions/runs/13589776349 * next try, this time with mac and win binaries
) * Removed last File.newTemporaryDirectory instances * code cleanup * fix jimple failing tests * fix jimple tests * Ensure all file streams are closed to allow file delete on windows * fmt * fix gradle destinationDir * code cleanup * remove all usages of better.files.File.newTemporaryFile * remove all usages of better.files.File.newTemporaryFile * remove touch-like function * removed all instances of better.files.File.usingTemporaryDirectory/usingTemporaryFile * code cleanup * fix file content for rubysrc2cpg * Attempting to fix windows tests on runner * remove system.gc * fmt * Revert to default scala Using, update writeBytes to use Iterable instead of iterator type for param * Removed default charset to all writeString calls, deleted commented code * fmt * missed a File.usingTemporaryDirectorry * revert retry delete * remove unused charset imports * remove commented out line in workrflow * code cleanup + fmt --------- Co-authored-by: Andrei Dreyer <andrei.dreyer@adagintech.com>
* Add init script that only works with gradle 8 * Fix init script for Gradle <8 * Add default app override * Clean up unnecessary TODOs
No custom operator call anymore. It's semantically just a block of expressions.
This speeds up parsing quite a lot. (OPTION_NO_IMAGE_LOCATIONS, OPTION_SKIP_TRIVIAL_EXPRESSIONS_IN_AGGREGATE_INITIALIZERS) Also cleaned up the logs: - do not log preprocessorErrorCount and parse problem numbers. These numbers might be nice for small code snippets but do not mean anything on real projects when they go to >100 or the like. - only log relative paths (easier to read logs)
…ernio#5359) * upgrade repl: runAfter code execution for scripts * latest * release
* Replaced all instances of better.files.File|DSL etc * fixed pathing issues * fix path issues in gosrc tests * added currentWorkingDirectory util function for Files, fixed ExternalCommandTest * remove import for better-files * moved fileutil from io.joern.x2cpg.utils to io.shiftleft.semanticcpg.utils. Removed last instances of better files * remove .toAbsolutePath.normalize * fix ruby failing test * remove .toAbsolutePath.normalize in TypeStubUtil --------- Co-authored-by: Andrei Dreyer <andrei.dreyer@adagintech.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by Github action