2025.1.1 AM fails: `Can't open boot keystore`

[jsanhc]

Environment

• MacBook Pro M2 Pro, 16GB, Sequoia 15.3.1
• Minikube v1.34.0
  • cpus: 4
  • disk-size: 40g
  • driver: docker
  • kubernetes:
    • Client Version: v1.32.1
    • Kustomize Version: v5.5.0
    • Server Version: v1.31.0
  • memory: 8g
• Deployment helm procedure
• Forgeops Tag 2025.1.1
  • secret-agent 1.2.0 (currently the version provided in charts/identity-platform/scripts/install-prereqs)

Description

I'm currently deploying using Helm the identity-platform locally using minikube all seems to work properly, the SAC generates properly the secrets.
AM start fails with the below error:
com.sun.identity.common.configuration.ConfigurationException: Configuration store is not available.
Caused by:

"exception":"java.io.IOException: Can't open boot keystore
...
Caused by: java.security.KeyStoreException: Exception trying to fetch key with alias configStorePwd
...
Caused by: java.security.KeyStoreException: Incorrect password used to retrieve secret key with alias configStorePwd

am-keystore has been generated properly with all expected aliases but seems that the password used to import the aliases configstorepwd and dsameuserpwd is not the expected .keypass defined in the SAC, or maybe the importpassword secret-agent command doesn't work properly, these are the only two entries in the SAC that use importpassword command.

I have copied the keystore /home/forgerock/openam/security/keystores/keystore.jceks locally to verify that the password used for the keystore and the key are correct.

Effectively cannot get the password keys configstorepwd and dsameuserpwd, they are the only password keys imported using the cmd alias importpassword, the sourcePath defined in the SAC to import these passwords are ok, the keystore can be opened using the .storepass but is not possible get the aliases configstorepwd and dsameuserpwd using the expected .keypass, the other keys are ok and can be opened with the .keypass.

Using keyStore Explorer and a Simple Java

Java simple class to retrieve the password keys aliases to verification

import javax.crypto.SecretKey;

import java.io.FileInputStream;
import java.math.BigInteger;
import java.security.KeyStore;

public class Main {
    public static void main(String[] args) throws Exception {
        /* Boot
        "keystores" : {
            "default" : {
              "keyStorePasswordFile" : "/home/forgerock/openam/security/secrets/default/.storepass",
              "keyPasswordFile" : "/home/forgerock/openam/security/secrets/default/.keypass",
              "keyStoreType" : "JCEKS",
              "keyStoreFile" : "/home/forgerock/openam/security/keystores/keystore.jceks"
            }
         */
        String fileName = "keystore.jceks";
        char[] storepass = "STOREPASS_VALUE".toCharArray(); // .storepass value 
        char[] keypass = "KEYPASS_VALUE".toCharArray(); // .keypass value
        String alias = "configstorepwd";

        KeyStore ks = KeyStore.getInstance("JCEKS");

        try (FileInputStream fis = new FileInputStream(Main.class.getResource(fileName).getFile())) {
            ks.load(fis, storepass);
            SecretKey secretKey = (SecretKey) ks.getKey(alias, keypass);
            System.out.println(new BigInteger(1, secretKey.getEncoded()).toString(16));
        }
        
    }
}

Exception in thread "main" java.security.UnrecoverableKeyException: java.security.Key: [SECRET] [PBEWithMD5AndDES] [RAW]

AM Stack trace

{"timestamp":"2025-03-02T12:13:35.011Z","level":"ERROR","thread":"main","logger":"com.sun.identity.setup.AMSetupServlet",
"message":"AMSetupServlet.checkConfigProperties","context":"default",
"exception":"java.io.IOException: Can't open boot keystore
    at com.sun.identity.setup.BootstrapData.<init>(BootstrapData.java:95)
    at com.sun.identity.setup.AMSetupServlet.checkConfigProperties(AMSetupServlet.java:344)
    at com.sun.identity.setup.AMSetupServlet.init(AMSetupServlet.java:241)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:984)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:941)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:838)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4193)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4494)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:599)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:571)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:603)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1175)
    at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1888)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
    at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
    at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1086)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:470)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1584)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:312)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:109)
    at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:389)
    at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:336)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:776)
    at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:721)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1203)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1193)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
    at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:749)
    at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:211)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
    at org.apache.catalina.core.StandardService.startInternal(StandardService.java:415)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
    at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:874)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:739)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
Caused by: java.security.KeyStoreException: Exception trying to fetch key with alias configStorePwd
    at org.forgerock.openam.utils.AMKeyProvider.getSecret(AMKeyProvider.java:630)
    at com.sun.identity.setup.BootstrapData.<init>(BootstrapData.java:92)
    ... 46 common frames omitted
Caused by: java.security.KeyStoreException: Incorrect password used to retrieve secret key with alias configStorePwd
    at org.forgerock.openam.utils.AMKeyProvider.getSecret(AMKeyProvider.java:624)
    ... 47 common frames omitted
","transactionId":null}
02-Mar-2025 12:13:35.721 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/am] has finished in [4,669] ms
02-Mar-2025 12:13:35.725 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
02-Mar-2025 12:13:35.730 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8081"]
02-Mar-2025 12:13:35.731 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [4694] milliseconds
com.sun.identity.common.configuration.ConfigurationException: Configuration store is not available.
        at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:124)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
        at org.forgerock.openam.validation.RequestEntitySizeVerificationFilter.doFilter(RequestEntitySizeVerificationFilter.java:66)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
        at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:43)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at java.base/java.lang.Thread.run(Thread.java:840)
10.244.0.1 - - [02/Mar/2025:12:13:39 +0000] "GET /am/json/health/live HTTP/1.1" 500 3511 51ms
com.sun.identity.common.configuration.ConfigurationException: Configuration store is not available.

[lee-baines]

Hi @jsanhc, I need to get some more information to identify what is happening here. Often we find issues running in minikube that results in issues with the platform but these tend to be a symptom. Can you please confirm where you are deploying a default forgeops deployment or are there custom changes made? If custom changes have been made, can you get it working with a default single-instance deployment?
Can you please run bin/debug-logs and provide the resulting html page. This will give me all the environmental info to see what is happening?

[jsanhc]

Hi @lee-baines I can confirm that I'm deploying a default forgeops deployment, single-instance, no custom changes has been made.

git heads c0d8447

git show-ref --heads -s
c0d8447c051df6d471e143c7e6efc34757213707
c0d8447c051df6d471e143c7e6efc34757213707
c0d8447c051df6d471e143c7e6efc34757213707

commit c0d8447c051df6d471e143c7e6efc34757213707 (HEAD -> 2025.1.1-jsanchez, tag: identity-platform-2025.1.1, tag: 2025.1.1, origin/main, origin/HEAD, main)

Install

Create forgeops env --env-name mink --fqdn idp.mink.local --cluster-issuer default-issuer --single-instance

The /helm/mink/values.yaml env:

am:
  replicaCount: 1
ds_cts:
  replicaCount: 1
idm:
  replicaCount: 1
ds_idrepo:
  replicaCount: 1
ig:
  replicaCount: 1
platform:
  ingress:
    hosts:
    - idp.kube.local
    tls:
      issuer:
        kind: ClusterIssuer
        name: default-issuer

helm upgrade --install identity-platform ./ --version 2025.1.1 --namespace identity-platform --values ../../helm/mink/values.yaml

Logs

This is the result of debug-logs:

log-html.txt

[lee-baines]

Thanks for providing this. Can you please try something? This is to eliminate something for me more that anything else. Please just try deleting the AM pod and letting it restart. The reason for this is that we have an open bug where a customer has reported issues before when a container restarts(not a pod) as mounted resources and items saved to Kubernetes dirs remain on the pod during a pod restart.

[jsanhc]

Just happens the same issue in AWS EKS, following your suggestion deleting the AM pod works again, but it's weird isn't? could be related with the kubernetes version?
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.30.9-eks-8cce635

[lee-baines]

I think we've identified the issue which is regarding the truststore-init init container. keystores and truststores get moved around in the main container and when that restarts, because the truststore-init container doesn't get reran, the main container fails to find certain stores. We are working on a fix currently.

[jsanhc]

Hi @lee-baines, I wish this information helps you to identify the issue.

In minikube doesn't fix the problem deleting the am pod, the issue persist, so I did more tests, the error origin it seems to be in the generated secret am-keystore for entries created by secret-agent using the cmd: importpassword.

I'm using the secret-agent by default v1.2.0

I have decoded the secret am-keystore as keystore.jceks and the related secrets am-passwords and ds-passwords defined in the SAC.

This is part of default SAC:

  - name: am-keystore
    keys:
    - name: keystore.jceks
      type: keytool
      spec:
        storeType: jceks
        storePassPath: am-passwords/.storepass
        keyPassPath: am-passwords/.keypass
        keytoolAliases:
          # More aliases ...
          - name: directenctest
            cmd: genseckey
            args: ["-keyalg", "aes", "-keysize", "256"]
          - name: configstorepwd
            cmd: importpassword
            sourcePath: am-passwords/.storepass
          - name: dsameuserpwd
            cmd: importpassword
            sourcePath: ds-passwords/dirmanager.pw

Using as refer the SAC, it should be possible:

• Unlock the keystore.jceks using .storepass value as password: OK
• Unlock the entry genseckey (or any other secret key) using .keypass value as password: OK
• Unlock the entry dsameuserpwd using .keypass value as password: KO
• Unlock the entry configstorepwd using .keypass value as password: KO

Deleting am-passwords and am-keystore secrets before delete the am pod to force the re-generation of these secrets It doesn't work either.

I confirm that I have never managed to deploy successfully the platform 7.5.1 in minikube (kube server 1.31.0) cause of this issue.

[lee-baines]

thanks for the additional information. So to confirm, you're saying that you can resolve the issue in EKS by deleting the AM pod, but you can't resolve the issue in minikube by deleting the pod. You can't get it to work at all in minikube?

[jsanhc]

thanks for the additional information. So to confirm, you're saying that you can resolve the issue in EKS by deleting the AM pod, but you can't resolve the issue in minikube by deleting the pod. You can't get it to work at all in minikube?

Yes, It's that, deleting de AM in EKS resolves the issue, the issue in EKS appeared randomly only once after more than 150 days working, I mean that I can get it work in EKS from scratch in the first deployment without issues, but not in minikube, I can't get it to work at all in minikube.

[lee-baines]

Ok thanks. We do have a ticket to take a look at the AM containers and revise how they work to prevent container failures like this in the future. It might get fixed in our next release ForgeOps 2025.2.0 in a few weeks but if not then, in an immediate maintenance release after that. Please keep an eye on the release notes and ensure you enable Github release notifications if you haven't already

[virgilpintilie]

Hello.Any updates on this issue? Same behavior for me, in EKS this thing happens on every helm update, if i do a pod delete manually am starts ok.In minikube sometimes same error occurs couple of pod restarts in a row and then it works all of a sudden.

[jason-ra]

Also occurring for me in Minikube - on every restart of the pod. Deleting the pod resolves the issue - until next restart. In 7.5.1 and 8.

My workaround has been to take a copy of good ~/openam/security/keystores/keystore.jceks (created on first start which is successful) as keystore2.jceks and inject this into boot.json using docker-entrypoint.sh.

## Replace keystore in boot.json
if [ -f $AM_HOME/security/keystores/keystore2.jceks ]; then
  sed -i -e 's/keystore.jceks/keystore2.jceks/g' ~/openam/config/boot.json
fi

[lee-baines]

Hi, we are still working on this issue. We are also doing some changes in how we provision truststores using cert-manager, so the fixes and changes will all be released as part of ForgeOps 2025.2.0 in the next few weeks. I'll update this ticket when I have more information or an easier workaround

[virgilpintilie]

Hi, a quick followup on this issue with and my workaround.
When the am pod is created the config bootstrap is copied to openam including config/boot.json file.This default file references a boot-keystore at location /home/forgerock/openam/security/keystores/boot/boot-keystore.jceks and .storepass, .keypass at the same location.After tomcat spins up the am application this boot.json file is getting rewritten with the external created keystore and password at location /home/forgerock/openam/security/keystores/keystore.jceks
But on a container restart fbc-init init container is not running so that initial bootstrap is not being copied and boot.json file references /home/forgerock/openam/security/keystores/keystore.jceks.But when am aplication starts it seems to be checking initial boot-keystore.jceks using external storepass which fails so that configstore error occurs.
The workaround used was to always use initial boot.json file so that am can validate that boot-keystore and than the startup process can finalize corectly.

Virgil