Skip to content

Improve handling of large files #288

@alewitt2

Description

@alewitt2

Is your feature request related to a problem? Please describe.
when trying to apply a large file (one real world example they were trying to apply a 2.7MB file), the remoteresource-controller container kept getting OOMKilled.

Describe the solution you'd like
dont download the entire file, these large files are usually large because they are --- separated multipart yamls. You should be able to download or stream chunks and process them accordingly.

Describe alternatives you've considered
use smaller files. divide large multipart yamls into smaller files with less yamls.

Additional context
real world sample of container getting killed

remoteresource-controller:
    Image:          registry.ng.bluemix.net/armada-master/remoteresource:1.0.3_b3def35
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 28 Jan 2022 21:10:31 +0000
      Finished:     Fri, 28 Jan 2022 21:13:12 +0000
    Ready:          False
    Restart Count:  41
    Limits:
      cpu:     100m
      memory:  200Mi
    Requests:
      cpu:     40m
      memory:  75Mi

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions