Skip to content

Move Ansible hosts storage to file system#213

Open
unitmatrix wants to merge 2 commits intofboender:masterfrom
unitmatrix:master
Open

Move Ansible hosts storage to file system#213
unitmatrix wants to merge 2 commits intofboender:masterfrom
unitmatrix:master

Conversation

@unitmatrix
Copy link
Copy Markdown

Currently, ansible-cmdb is storing parsed Ansible hosts and variables in dictionary variable in memory. While this works well for tens and hundreds of hosts, it does not scale for larger inventories with thousands hosts or more. The script would run for over 1 hour and finally crash with memory exception for me. Generating cmdb inventory for 10.000+ hosts is basically impossible.

This change moves hosts management to a separate AnsibleHosts class where it implements Python dictionary so that the data is saved to a temporary directory on disk rather than your RAM thus enabling linear processing of hosts where the free space on disk is your only limit.

This only changes the back-end of the hosts dictionary so that it implements functionality of the regular Python dictionary providing standard methods, such as:

hosts = AnsibleHosts() # create new object
hosts[key] = value     # set new value
item = hosts.get(key)  # get value by key
len = len(hosts)       # get number of items
items = hosts.items()  # iterate-able items
key in hosts           # check if key exists
etc.

This way it does not affect other parts of code which will utilize the dictionary as usual. Unit tests attached to verify the required functionality as well.

Aleksandr Cupacenko added 2 commits March 19, 2020 10:21
…es (10K+) file linear system based processing of parsed hosts data to reduce memory usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant