RFC - Starkit Configuration.

[NikhilVerma]

Starkit Configuration Format RFC

Introduction

Starkit currently provides a simple command-line configuration, which is insufficient for advanced and real-world CI/CD scenarios. This RFC proposes a new configuration format to support complex configurations and outlines the requirements and expected behavior.

Configuration Requirements

The following features should be supported by the Starkit configuration:

1. Base configurations:
   • Region(s) (multiple region support to be added)
   • Service(s)
   • Format(s) (multiple format support and HTML report to be added)
2. Failure conditions specification using service:criticality pairs to determine pass/fail status of runtime tests
3. Exclusion of specific instances from services (e.g., a specific S3 bucket, EC2 instance, or App Runner instance)

Configuration Format

• Multiple configuration formats should be supported, with JSON and YAML as first-class citizens.
• Internal validation will use JSON format powered by zod.
• Configurations should use snake_case.

Proposed format:

{
	// Used to mark the configuration format and backward compatibility
	"compatibility_date": "2025-01-03",

	"clouds": [
		{
			"provider": "aws",
			"regions": ["us-east-1", "us-west-2"],
			"services": ["s3", "ec2"],
			"allow_failures": [
				// Allow all failures for a service
				{
					"service": "s3",
					"criticality": "critical"
				},
				// Allow high failures for all services
				{
					"service": "*",
					"criticality": "high"
				},
				// Allow failures for an an ARN
				{
					"arn": "arn:aws:s3:::my_bucket",
					"criticality": "high"
				}
			]
		}

		// Similar configs for other providers
	],

	"iac": [
		{
			"type": "terraform",
			"dir": "path/to/terraform",
			"exclude": ["module1", "module2"]
		}
	]
}

Configuration Management

• File naming: .starkit.{json,yaml}
• Config location priority:
  1. CLI specified path
  2. Current directory
  3. User home directory
  4. Environment variable STARKIT_CONFIG_PATH

Version Control

• Use compatibility_date to manage backward compatibility and config versioning.

Considerations

• Determine the depth of configuration allowed (e.g., "region:service:instance" or just "region:service").
• Evaluate the impact of introducing complex configurations on the user experience and documentation.
• Consider the scalability and performance implications of the proposed configuration format.

Implementation

1. Implement support for JSON and YAML configuration formats.
2. Integrate zod for internal JSON validation.
3. Update the CLI to read and validate configurations based on the defined location priority.
4. Implement the proposed configuration structure and default behaviours.
5. Add support for multiple regions, formats, and HTML reporting.
6. Update documentation and examples to reflect the new configuration format.

Rollout Plan

1. Develop and test the new configuration format in a separate branch.
2. Gather feedback from the development team and selected users.
3. Address feedback and make necessary adjustments.
4. Merge the configuration format changes into the main branch.
5. Release a new version of Starkit with the updated configuration format.

Conclusion

Implementing a comprehensive configuration format will enable Starkit to support advanced and real-world CI/CD scenarios. By following the proposed format and requirements, Starkit can offer greater flexibility and customization options to its users.

[lotsofsoup]

Who will review and approve exemptions? This needs to be annotated into the results file for visibility and audit. The user who submits the file can't be the person who approves exemptions.

I guess in the free version it just needs to annotate the report output that the user has disabled certain classes of tests. To be user in enterprise though tjis feature needs to be set up in the context of the pipeline and release owner, and whoever is authorised to approve this sort of thing, i.e. it needs Stance / Provenance to be viable.

[lotsofsoup]

What is meant by supporting regions and services, do we mean aggregates?