Skip to content

feat: typed config overrides #917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
dervoeti wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat: typed config overrides #917

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 59 additions & 21 deletions extra/crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ spec:
anyOf:
- required:
- zookeeperConfigMapName
- {}
- type: object
description: |-
Settings that affect all roles and role groups.
The settings in the `clusterConfig` are cluster wide settings that do not need to be configurable at role or role group level.
Expand All @@ -48,6 +48,16 @@ spec:
description: This field contains OIDC-specific configuration. It is only required in case OIDC is used.
nullable: true
properties:
clientAuthenticationMethod:
default: client_secret_basic
description: 'The client authentication method used when communicating with the token endpoint. Defaults to `client_secret_basic`. The required contents of `clientCredentialsSecret` depend on the chosen method: secret-based methods (`client_secret_basic`, `client_secret_post`, `client_secret_jwt`) expect a client secret, while `private_key_jwt` expects a private key.'
enum:
- client_secret_basic
- client_secret_post
- client_secret_jwt
- private_key_jwt
- none
type: string
clientCredentialsSecret:
description: |-
A reference to the OIDC client credentials secret. The secret contains
Expand Down Expand Up @@ -523,7 +533,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -603,7 +613,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -683,7 +693,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -763,7 +773,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -816,7 +826,7 @@ spec:
type: object
type: object
enableVectorAgent:
description: Wether or not to deploy a container with the Vector log agent.
description: Whether or not to deploy a container with the Vector log agent.
nullable: true
type: boolean
type: object
Expand Down Expand Up @@ -1176,17 +1186,31 @@ spec:
type: object
type: object
configOverrides:
additionalProperties:
additionalProperties:
type: string
type: object
default: {}
description: |-
The `configOverrides` can be used to configure properties in product config files
that are not exposed in the CRD. Read the
[config overrides documentation](https://docs.stackable.tech/home/nightly/concepts/overrides#config-overrides)
and consult the operator specific usage guide documentation for details on the
available config files and settings for the specific product.
properties:
bootstrap.conf:
additionalProperties:
type: string
description: Overrides for the `bootstrap.conf` file.
nullable: true
type: object
nifi.properties:
additionalProperties:
type: string
description: Overrides for the `nifi.properties` file.
nullable: true
type: object
security.properties:
additionalProperties:
type: string
description: Overrides for the `security.properties` file.
nullable: true
type: object
type: object
envOverrides:
additionalProperties:
Expand Down Expand Up @@ -1368,7 +1392,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -1448,7 +1472,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -1528,7 +1552,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -1608,7 +1632,7 @@ spec:
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
description: Custom log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
Expand Down Expand Up @@ -1661,7 +1685,7 @@ spec:
type: object
type: object
enableVectorAgent:
description: Wether or not to deploy a container with the Vector log agent.
description: Whether or not to deploy a container with the Vector log agent.
nullable: true
type: boolean
type: object
Expand Down Expand Up @@ -2021,17 +2045,31 @@ spec:
type: object
type: object
configOverrides:
additionalProperties:
additionalProperties:
type: string
type: object
default: {}
description: |-
The `configOverrides` can be used to configure properties in product config files
that are not exposed in the CRD. Read the
[config overrides documentation](https://docs.stackable.tech/home/nightly/concepts/overrides#config-overrides)
and consult the operator specific usage guide documentation for details on the
available config files and settings for the specific product.
properties:
bootstrap.conf:
additionalProperties:
type: string
description: Overrides for the `bootstrap.conf` file.
nullable: true
type: object
nifi.properties:
additionalProperties:
type: string
description: Overrides for the `nifi.properties` file.
nullable: true
type: object
security.properties:
additionalProperties:
type: string
description: Overrides for the `security.properties` file.
nullable: true
type: object
type: object
envOverrides:
additionalProperties:
Expand Down
4 changes: 2 additions & 2 deletions rust/operator-binary/src/config/jvm.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use stackable_operator::{

use crate::{
config::{JVM_SECURITY_PROPERTIES_FILE, NIFI_CONFIG_DIRECTORY},
crd::{NifiConfig, NifiConfigFragment, NifiNodeRoleConfig},
crd::{NifiConfig, NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig},
security::{
authentication::{STACKABLE_SERVER_TLS_DIR, STACKABLE_TLS_STORE_PASSWORD},
authorization::ResolvedNifiAuthorizationConfig,
Expand All @@ -33,7 +33,7 @@ pub enum Error {
/// Create the NiFi bootstrap.conf
pub fn build_merged_jvm_config(
merged_config: &NifiConfig,
role: &Role<NifiConfigFragment, NifiNodeRoleConfig, JavaCommonConfig>,
role: &Role<NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig, JavaCommonConfig>,
role_group: &str,
authorization_config: Option<&ResolvedNifiAuthorizationConfig>,
) -> Result<JvmArgumentOverrides, Error> {
Expand Down
6 changes: 3 additions & 3 deletions rust/operator-binary/src/config/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ use strum::{Display, EnumIter};

use crate::{
crd::{
HTTPS_PORT, NifiConfig, NifiConfigFragment, NifiNodeRoleConfig, NifiRole,
HTTPS_PORT, NifiConfig, NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig, NifiRole,
NifiStorageConfig, PROTOCOL_PORT, sensitive_properties,
v1alpha1::{self, NifiClusteringBackend},
},
Expand Down Expand Up @@ -116,7 +116,7 @@ pub enum Error {
pub fn build_bootstrap_conf(
merged_config: &NifiConfig,
overrides: BTreeMap<String, String>,
role: &Role<NifiConfigFragment, NifiNodeRoleConfig, JavaCommonConfig>,
role: &Role<NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig, JavaCommonConfig>,
role_group: &str,
authorization_config: Option<&crate::security::authorization::ResolvedNifiAuthorizationConfig>,
) -> Result<String, Error> {
Expand Down Expand Up @@ -756,7 +756,7 @@ pub fn build_state_management_xml(clustering_backend: &NifiClusteringBackend) ->
pub fn validated_product_config(
resource: &v1alpha1::NifiCluster,
version: &str,
role: &Role<NifiConfigFragment, NifiNodeRoleConfig, JavaCommonConfig>,
role: &Role<NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig, JavaCommonConfig>,
product_config: &ProductConfigManager,
) -> Result<ValidatedRoleConfigByPropertyKind, Error> {
let mut roles = HashMap::new();
Expand Down
9 changes: 5 additions & 4 deletions rust/operator-binary/src/controller.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,9 @@ use crate::{
},
crd::{
APP_NAME, BALANCE_PORT, BALANCE_PORT_NAME, Container, HTTPS_PORT, HTTPS_PORT_NAME,
METRICS_PORT, METRICS_PORT_NAME, NifiConfig, NifiConfigFragment, NifiNodeRoleConfig,
NifiRole, NifiStatus, PROTOCOL_PORT, PROTOCOL_PORT_NAME, STACKABLE_LOG_CONFIG_DIR,
METRICS_PORT, METRICS_PORT_NAME, NifiConfig, NifiConfigFragment, NifiConfigOverrides,
NifiNodeRoleConfig, NifiRole, NifiStatus, PROTOCOL_PORT, PROTOCOL_PORT_NAME,
STACKABLE_LOG_CONFIG_DIR,
STACKABLE_LOG_DIR, authentication::AuthenticationClassResolved,
authorization::NifiAccessPolicyProvider, v1alpha1,
},
Expand Down Expand Up @@ -719,7 +720,7 @@ async fn build_node_rolegroup_config_map(
resolved_product_image: &ResolvedProductImage,
authentication_config: &NifiAuthenticationConfig,
authorization_config: &ResolvedNifiAuthorizationConfig,
role: &Role<NifiConfigFragment, NifiNodeRoleConfig, JavaCommonConfig>,
role: &Role<NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig, JavaCommonConfig>,
rolegroup: &RoleGroupRef<v1alpha1::NifiCluster>,
rolegroup_config: &HashMap<PropertyNameKind, BTreeMap<String, String>>,
merged_config: &NifiConfig,
Expand Down Expand Up @@ -843,7 +844,7 @@ async fn build_node_rolegroup_statefulset(
resolved_product_image: &ResolvedProductImage,
cluster_info: &KubernetesClusterInfo,
rolegroup_ref: &RoleGroupRef<v1alpha1::NifiCluster>,
role: &Role<NifiConfigFragment, NifiNodeRoleConfig, JavaCommonConfig>,
role: &Role<NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig, JavaCommonConfig>,
rolegroup_config: &HashMap<PropertyNameKind, BTreeMap<String, String>>,
merged_config: &NifiConfig,
authentication_config: &NifiAuthenticationConfig,
Expand Down
46 changes: 45 additions & 1 deletion rust/operator-binary/src/crd/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ use stackable_operator::{
fragment::{self, Fragment, ValidationError},
merge::Merge,
},
config_overrides::{KeyValueConfigOverrides, KeyValueOverridesProvider},
crd::{authentication::core as auth_core, git_sync},
deep_merger::ObjectOverrides,
k8s_openapi::{
Expand Down Expand Up @@ -100,7 +101,7 @@ pub mod versioned {

// no doc - docs in Role struct.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub nodes: Option<Role<NifiConfigFragment, NifiNodeRoleConfig, JavaCommonConfig>>,
pub nodes: Option<Role<NifiConfigFragment, NifiConfigOverrides, NifiNodeRoleConfig, JavaCommonConfig>>,

// no doc - docs in ProductImage struct.
pub image: ProductImage,
Expand Down Expand Up @@ -522,6 +523,49 @@ pub struct NifiStorageConfig {
pub filebased_repo: PvcConfig,
}

/// Typed config overrides for NiFi configuration files.
///
/// Each field corresponds to a configuration file that supports key-value overrides.
/// The field names match the actual file names used by NiFi.
#[derive(Clone, Debug, Default, Deserialize, Eq, JsonSchema, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct NifiConfigOverrides {
/// Overrides for the `bootstrap.conf` file.
#[serde(rename = "bootstrap.conf")]
pub bootstrap_conf: Option<KeyValueConfigOverrides>,

/// Overrides for the `nifi.properties` file.
#[serde(rename = "nifi.properties")]
pub nifi_properties: Option<KeyValueConfigOverrides>,

/// Overrides for the `security.properties` file.
#[serde(rename = "security.properties")]
pub security_properties: Option<KeyValueConfigOverrides>,
}

impl KeyValueOverridesProvider for NifiConfigOverrides {
fn get_key_value_overrides(&self, file: &str) -> BTreeMap<String, Option<String>> {
match file {
"bootstrap.conf" => self
.bootstrap_conf
.as_ref()
.map(|kv| kv.as_overrides())
.unwrap_or_default(),
"nifi.properties" => self
.nifi_properties
.as_ref()
.map(|kv| kv.as_overrides())
.unwrap_or_default(),
"security.properties" => self
.security_properties
.as_ref()
.map(|kv| kv.as_overrides())
.unwrap_or_default(),
_ => BTreeMap::new(),
}
}
}

#[derive(Clone, Debug, Deserialize, JsonSchema, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct NifiNodeRoleConfig {
Expand Down
1 change: 1 addition & 0 deletions rust/operator-binary/src/security/oidc.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ mod tests {
let oidc = oidc::v1alpha1::ClientAuthenticationOptions {
client_credentials_secret_ref: "nifi-keycloak-client".to_owned(),
extra_scopes: vec![],
client_authentication_method: Default::default(),
product_specific_fields: (),
};

Expand Down
Loading