Repository-Manager: Replace __init__.py by YAML

[zmsdev]

• This PR evaluates a generic YAML creation.
• A late branch (after #14b383f) tries to replace the approach by common library functions: Repository-Manager: Replace __init__.py by ruamel.yaml #425

[drfho]

yes, thus manually much better editable. The quotation-marks around the values seem to arbitray; maybe can be ommited generally.
Just "default" is critical: here the block-scalar | might appropriate for the formerly non-editable multiline codes, e.g.

  type: interface
  default: |
    ##
    return context.getTextFormatDefault()

Ref: https://yaml-multiline.info/

[drfho]

@zmsdev
Analysis & Discussion :

1. if init.py is deleted the whole class def will be ignored from ZMS-repo-manager; thus a transition to yaml will be prohibited
2. the attr sequence in init is sorted alphabetically: this makes is easier for code comparision, but much more difficult for the user/developer, Maybe generally starting with id/name and proceed alpha-sorted may be more intuitive? (sideeffect: any content-models habe to be updated in the code repo; but this has to be done anyway when switching to yaml)
3. the process does not utilize the py-core module 'yaml' for dumping a yaml structure. Why does yaml.dump() not work with the custom fields (str)?
4. some indents may be too deep, e.g. meta_types/roles items are 3x indented

manage_tab_statistics:
  action: %smanage_executeMetacmd?id=manage_tab_statistics
  icon_clazz: icon-cogs fas fa-chart-bar
  id: manage_tab_statistics
  meta_types:
        - ZMS
  name: Statistics
  nodes: {$}
  revision: 5.2.0
  roles:
        - ZMSAdministrator
  stereotype: tab
  title: Content object usage statistics
  Impl:
    - id: manage_tab_statistics
      type: Page Template

[drfho]

@zmsdev : getInitYaml can create invalid yaml

1. '%s' at begining of value
2. list blocks may not start with linebreak

I tried to add a workaround for fixing the ZMS creates yaml
621befd
DISCUSSION: wouldn't it be better, if ZMS could create a generic yaml and not a dialect that needs a set of cleaning steps?

[drfho]

@zmsdev
For discussion I made a copy of getInitYaml()

ZMS/Products/zms/repositoryutil.py

Lines 403 to 435 in 11a895d

	def getInitYaml(self, o):
	"""
	Generate a YAML representation of the given object.
	Args:
	self: The instance of the class containing this method.
	o (dict): A dictionary representing the object to be converted to YAML.
	Returns:
	list: A list of strings representing the YAML structure of the input object.
	"""
	id = o.get('id','?')
	yaml = []
	yaml.append('%s:'%id)
	e = sorted([x for x in o if not x.startswith('__') and x==x.capitalize() and isinstance(o[x], list)])
	keys = sorted([x for x in o if not x.startswith('__') and x not in e])
	for k in keys:
	v = o.get(k)
	if v:
	yv = standard.str_yaml(v, level=1)
	if len(yv.strip()) > 0:
	sep = ' '
	if type(v) is list:
	sep = '\n'
	elif type(v) is dict:
	sep = '\n '
	yaml.append(' %s:%s%s'%(standard.id_quote(k), sep, yv))
	for k in e:
	v = o.get(k)
	if v and isinstance(v, list):
	yaml.append(' %s:'%standard.id_quote(k).capitalize())
	yaml.append(standard.str_yaml([{k: v for k, v in i.items() if k != 'ob'} for i in v if 'id' in i], level=1))
	return yaml

called getInitYamlDump() using an existing YAML-lib avoiding internal YAML-generators ( standard.str_yaml() ) for converting list/dicts-objs (linebreak/indents probs)

ZMS/Products/zms/repositoryutil.py

Lines 438 to 480 in 11a895d

	def getInitYamlDump(self, o):
	"""
	Generate a YAML representation of the given object
	by utilizing the standard yaml library.
	Args:
	self: The instance of the class containing this method.
	o (dict): A dictionary representing the object to be converted to YAML.
	Returns:
	list: A normalized Dictionary representing the YAML structure of the input object.
	"""
	id = o.get('id','?')
	o_init = {id: {}}
	# Ignore acquisition wrapper objects
	e = sorted([x for x in o if not x.startswith('__') and x==x.capitalize() and isinstance(o[x], list)])
	keys = sorted([x for x in o if not x.startswith('__') and x not in e])
	for k in keys:
	if o.get(k):
	o_init[id][k] = o.get(k)
	# Append attribute lists
	for k in e:
	l = o.get(k)
	if l and isinstance(l, list):
	o_init[id][k] = [{k:v for k, v in i.items() if k != 'ob' and v not in [None, '', [], [''], 0] } for i in l]
	# Remove empty attributes and apply LiteralScalarString
	for i in list(o_init[id][k]):
	for k1, v1 in i.items():
	if k1 in ('custom', 'default') and type(v1) is str and str(v1).find('\n') >= 0 or str(v1).find('\r') >= 0:
	o_init[id][k][o_init[id][k].index(i)][k1] = LiteralScalarString(str(v1))
	yaml = YAML()
	yaml.preserve_quotes = True
	stream = io.StringIO()
	yaml.dump(o_init, stream)
	yaml_as_text = stream.getvalue()
	# Create a list of yaml lines because ZMS expects it for comparision line by line
	yaml_list = []
	for line in yaml_as_text.split('\n'):
	yaml_list.append(line)
	return yaml_list

To make the alternative function getInitYamlDump() work I integrated it's call here:

ZMS/Products/zms/repositoryutil.py

Line 270 in 11a895d

l.update(getInitArtefacts(self, o, {'yaml':getInitYamlDump(self, o)}))

What YAML-Lib: As YAML-lib not PYYAML but ruamel.yaml is applied, because PYYAML seems to force element sorting of the source-dict. In genereal: when the output is a normalized YAML it is easier to transform it back to a dict just by yaml.safe_load() :

ZMS/Products/zms/repositoryutil.py

Lines 119 to 120 in 11a895d

	if name.endswith('.yaml'):
	d = yaml.safe_load(filedata)

Any workaround for 'special' YAML dialects are not needed anymore:

ZMS/Products/zms/repositoryutil.py

Lines 91 to 95 in 11a895d

	def safe_yaml_load(data):
	data = re.sub(r'(\w+:\s)(%.*?)(\n)', r'\1"\2"\3', data)
	data = re.sub(r'- \*', r'- "*"', data)
	data = re.sub(r'(\w+: )(\s*?)(- \w+\n)', r'\1\n\2\3', data)
	return yaml.safe_load(data)

Example:
A. ZMSTextarea (same with both functions)

1. Using standard.str_yaml()
   https://github.com/zms-publishing/ZMS/blob/0b1f8afcbb76cec2288a3c3940425e0a461a07c6/Products/zms/conf/metaobj_manager/com.zms.foundation/ZMSTextarea/__init__.yaml
2. . Using yaml-lib
   https://github.com/zms-publishing/ZMS/blob/43cfdf0e6dd2b56502793bfae7eb8967bfdf20dd/Products/zms/conf/metaobj_manager/com.zms.foundation/ZMSTextarea/__init__.yaml

B. ZMSFlexbox (wrong list-block processing with str_yaml() )

standard.str_yaml()	yaml-lib

[zmsdev]

Use PyYAML consequently on reduced repo-dict