vm/map: fix ENOMEM handling in _vm_mmap

[etiaro]

Description

Previous handling of ENOMEM during physical page allocation ignored that _map_map can merge new allocation with existing entries, leading to invalid map state, accidental virtual address reallocation and leaking physical pages.

Use of dedicated _vm_munmap function instead of manual cleanup ensures reliability and avoids code duplication. Correct manual cleanup would require handling multiple cases separately (lmerge, lmerge

Motivation and Context

This is part of series of PRs that increase kernel stability when system is out of memory, inspired by work on reliability of separation given by partitioning mechanisms and related to fork bomb issue phoenix-rtos/phoenis-rtos-project#560

JIRA: RTOS-1235

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Chore (refactoring, style fixes, git/CI config, submodule management, no code logic changes)

How Has This Been Tested?

• Already covered by automatic testing.
• New test added: (add PR link here).
• Tested by hand on: (list targets here).

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing linter checks and tests passed.
• My changes generate no new compilation warnings for any of the targets.

Special treatment

• This PR needs additional PRs to work (list the PRs, preferably in merge-order).
• I will merge this PR by myself when appropriate.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances kernel stability by refining the error handling within the virtual memory mapping (_vm_mmap) function. It specifically addresses a vulnerability where ENOMEM conditions could lead to an inconsistent virtual memory map, accidental address reallocations, and physical page leaks. The change ensures robust cleanup by utilizing a specialized unmapping function, thereby fortifying the system against out-of-memory scenarios.

Highlights

• Improved ENOMEM Handling: Replaced a multi-step manual cleanup process with a single, dedicated call to _vm_munmap within the _vm_mmap function's error path. This change addresses issues where _map_map could merge new allocations with existing entries, leading to invalid map states, accidental virtual address reallocation, and physical page leaks during out-of-memory conditions.
• Enhanced Kernel Stability: The fix contributes to increased kernel stability when the system is under memory pressure, preventing potential resource leaks and ensuring a more reliable virtual memory management system.

Changelog

• vm/map.c
  • Replaced manual cleanup logic with a call to _vm_munmap when _map_force fails during virtual memory mapping.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request addresses a critical bug in the error handling logic within _vm_mmap when dealing with out-of-memory conditions. The previous manual cleanup mechanism was flawed, particularly in scenarios involving merged memory map entries, which could lead to an invalid map state and resource leaks. The change correctly replaces this with a call to _vm_munmap, ensuring a robust and reliable cleanup by leveraging the existing unmapping functionality. This not only fixes the bug but also simplifies the code and improves its maintainability.

[github-actions]

Unit Test Results

9 525 tests  ±0   8 933 ✅ ±0   53m 7s ⏱️ +24s
  583 suites ±0     592 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 7c753cd. ± Comparison against base commit 523e005.

[etiaro]

Included in larger #749