Skip to content

autoscale p1

Jump to bottom
Jesús Daniel Colmenares Oviedo edited this page Mar 27, 2025 · 1 revision

AutoScaling Projects (part 1)

Even when there are hundreds of servers, deploying projects is an easy task, however the problem with this approach is that we are wasting resources and the clients probably only use less than 5% of the resources of our cluster, or on the contrary, you can deploy your project on a few servers which you think is enough until you realize at some point that this is not enough, even worse, some servers can be down at any time for any reason. This is what AutoScale can solve.

AutoScaling in Overlord introduces subtle changes to the workflow from the client's point of view. The client, in its deployment file, only needs to add the autoScale section with some desires such as the minimum and maximum deployments to perform, the rctl(8) rules, the labels, etc. Note that there is a difference between the labels specified in the deployIn section and the labels specified in the autoScale.labels section. In the former, you specify the labels that the API servers must match in order to be responsible for deploying the project and monitoring the project status on each API server on which it is deployed, and in the latter, the project is deployed to the API servers that match the specified labels.

kind: directorProject
datacenters:
  main:
    entrypoint: 'http://127.0.0.1:8888'
    access_token: '<access token>'
deployIn:
  labels:
    - desktop
projectName: hello-http
projectFile: |
  options:
    - virtualnet: ':<random> default'
    - nat:
  services:
    darkhttpd:
      makejail: 'gh+DtxdF/hello-http-makejail'
      options:
        - expose: '8123:80 ext_if:tailscale0 on_if:tailscale0'
        - label: 'overlord.skydns:1'
        - label: 'overlord.skydns.group:hello-http'
        - label: 'overlord.skydns.interface:tailscale0'
      arguments:
        - darkhttpd_tag: 14.2
autoScale:
  replicas:
    min: 2
  labels:
    - services

Note: The above deployment file specifies the labels to use SkyDNS, you can exclude them to avoid this, but in this article we assume that SkyDNS is enabled and configured for use with the overlord.lan. domain.

In the above deployment file we are specifying that API servers matching the desktop label should be responsible for monitoring the status of our project and deploying it. We are also specifying that our project should be deployed only to API servers matching the services label.

# overlord apply -f hello-http.yml
# overlord get-info -f hello-http.yml -t projects --use-autoscale-labels --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: bravo
  labels:
    - all
    - centralita
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_19h25m21s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': '6f027aa1a5'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 9 minutes and 35.07 seconds
        job_id: 12
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: delta
  labels:
    - all
    - service002
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_15h33m53s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': 'a5d6606a2c'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 9 minutes and 13.03 seconds
        job_id: 21
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.
# overlord get-info -f hello-http.yml -t autoscale --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: None
  labels:
    - all
    - desktop
    - vm-only
  projects:
    hello-http:
      autoScale:
        last_update: 20.14 seconds
        operation: COMPLETED
        output:
         message: None
# host hello-http.overlord.lan
hello-http.overlord.lan has address 100.84.92.68
hello-http.overlord.lan has address 100.96.18.2

Deployed and have two which is the minimum. Now one of the API servers is down, so let's look at the changes:

# host hello-http.overlord.lan
hello-http.overlord.lan has address 100.84.92.68
hello-http.overlord.lan has address 100.89.77.98
hello-http.overlord.lan has address 100.96.18.2
# overlord get-info -f hello-http.yml -t projects --use-autoscale-labels --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: bravo
  labels:
    - all
    - centralita
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_19h25m21s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': '6f027aa1a5'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 1 hour, 22 minutes and 30.16 seconds
        job_id: 12
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: echo
  labels:
    - all
    - service001
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_16h26m04s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': '7cfc1c77d2'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 29 minutes and 16.55 seconds
        job_id: 1
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.

Even if an API server is not running, the service is not removed from SkyDNS. Note also the output of the get-info command: centralita is alive but now service001 takes over the execution of our service.

# curl http://hello-http.overlord.lan:8123
Hello, world!
UUID: 56e083e5-f097-44d8-ae97-24cf688e87c8

Let's assume that our API server is back up and running.

# host hello-http.overlord.lan
hello-http.overlord.lan has address 100.84.92.68
hello-http.overlord.lan has address 100.89.77.98
# overlord get-info -f hello-http.yml -t projects --use-autoscale-labels --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: delta
  labels:
    - all
    - service002
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_15h33m53s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': 'a5d6606a2c'}
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: echo
  labels:
    - all
    - service001
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_16h26m04s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': '7cfc1c77d2'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 41 minutes and 2.58 seconds
        job_id: 1
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.

Overlord will destroy projects until it gets only the minimum that is set to two. But what's interesting is that now centralita is gone, but service001 and service002 are there. Overlord doesn't care about this, it just tries to deploy or destroy based on what you configured.

Clone this wiki locally