Error: no memory of presenting a DNS record / failed to append record / invalid response format

[Darkwing371]

Hi!

Are you sure this plugin is still working and API calls are correct? I'm constantly hitting these errors:

"challenge_type":"dns-01","error":"no memory of presenting a DNS record for "_acme-challenge.sub.domain.tld"
"error":"[sub.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone "sub.domain.tld.": failed to append record {_acme-challenge 0s TXT aaaa-redacted-bbbb-redacted-cccc}: invalid response format"

Full log:

MarMar 04 21:50:09 caddy systemd[1]: Started caddy.service - Caddy.
░░ Subject: A start job for unit caddy.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit caddy.service has finished successfully.
░░ 
░░ The job identifier is 16399.
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.1866305,"msg":"serving initial configuration"}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.1944845,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/root/.local/share/caddy"}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.1946986,"logger":"tls.obtain","msg":"acquiring lock","identifier":"sub.domain.tld"}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.1967802,"logger":"tls.obtain","msg":"lock acquired","identifier":"sub.domain.tld"}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.1969018,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sub.domain.tld"}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.1973372,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"email@domain.tld","ca":"https://acme-v02.api.letsencrypt.org/directory","error":"open /root/.local/share/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/email@domain.tld/email.json: no such file or directory"}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.1973758,"logger":"http","msg":"ACME account has empty status; registering account with ACME server","contact":["mailto:email@domain.tld"],"location":""}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.2003474,"logger":"tls","msg":"finished cleaning storage units"}
Mar 04 21:50:09 caddy caddy[17511]: {"level":"info","ts":1772657409.2010033,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"email@domain.tld","ca":"https://acme-v02.api.letsencrypt.org/directory","error":"open /root/.local/share/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/email@domain.tld/email.json: no such file or directory"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.0944073,"logger":"http","msg":"new ACME account registered","contact":["mailto:email@domain.tld"],"status":"valid"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.1079957,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sub.domain.tld"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"email@domain.tld"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.1080678,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sub.domain.tld"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"email@domain.tld"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.1080933,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/0123456789","account_contact":["mailto:email@domain.tld"]}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.4776862,"msg":"trying to solve challenge","identifier":"sub.domain.tld","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"error","ts":1772657410.6872318,"msg":"cleaning up solver","identifier":"sub.domain.tld",**"challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.sub.domain.tld\"** (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:318\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:363\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.2/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:73"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"error","ts":1772657410.8931262,"logger":"tls.obtain","msg":"**could not get certificate from issuer"**,"identifier":"sub.domain.tld","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[sub.domain.tld] solving challenges: presenting for challenge: **adding temporary record for zone \"sub.domain.tld.\": failed to append record** {_acme-challenge 0s TXT aaaa-redacted-bbbb-redacted-cccc}: **invalid response format** (order=https://acme-v02.api.letsencrypt.org/acme/order/0123456789/9876543210) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.893212,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"email@domain.tld","ca":"https://acme.zerossl.com/v2/DV90","error":"open /root/.local/share/caddy/acme/acme.zerossl.com-v2-dv90/users/email@domain.tld/email.json: no such file or directory"}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.8932629,"logger":"http","msg":"ACME account has empty status; registering account with ACME server","contact":["mailto:email@domain.tld"],"location":""}
Mar 04 21:50:10 caddy caddy[17511]: {"level":"info","ts":1772657410.9039762,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"email@domain.tld","ca":"https://acme.zerossl.com/v2/DV90","error":"open /root/.local/share/caddy/acme/acme.zerossl.com-v2-dv90/users/email@domain.tld/email.json: no such file or directory"}
Mar 04 21:50:11 caddy caddy[17511]: {"level":"info","ts":1772657411.825457,"logger":"http","msg":"generated EAB credentials","key_id":"aaaa-redacted-bbbb"}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"info","ts":1772657412.1289177,"logger":"http","msg":"new ACME account registered","contact":["mailto:email@domain.tld"],"status":"valid"}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"info","ts":1772657412.1313827,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sub.domain.tld"],"ca":"https://acme.zerossl.com/v2/DV90","account":"email@domain.tld"}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"info","ts":1772657412.1314526,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sub.domain.tld"],"ca":"https://acme.zerossl.com/v2/DV90","account":"email@domain.tld"}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"info","ts":1772657412.1314733,"logger":"http","msg":"using ACME account","account_id":"https://acme.zerossl.com/v2/DV90/account/aaaa-redacted-bbbb","account_contact":["mailto:email@domain.tld"]}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"info","ts":1772657412.3227375,"msg":"trying to solve challenge","identifier":"sub.domain.tld","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"error","ts":1772657412.400974,"msg":"cleaning up solver","identifier":"sub.domain.tld","challenge_type":"dns-01","error":"**no memory of presenting a DNS record** for \"_acme-challenge.sub.domain.tld\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:318\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:363\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.2/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:73"}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"error","ts":1772657412.4992433,"logger":"tls.obtain","msg":"**could not get certificate from issuer**","identifier":"sub.domain.tld","issuer":"acme.zerossl.com-v2-DV90","error":"[sub.domain.tld] solving challenges: presenting for challenge: **adding temporary record for zone \"sub.domain.tld.\": failed to append record** {_acme-challenge 0s TXT aaaa-redacted-bbbb-redacted-cccc}: invalid response format (order=https://acme.zerossl.com/v2/DV90/order/aaaa-redacted-bbbb) (ca=https://acme.zerossl.com/v2/DV90)"}
Mar 04 21:50:12 caddy caddy[17511]: {"level":"error","ts":1772657412.499364,"logger":"tls.obtain","msg":"will retry","error":"[sub.domain.tld] Obtain: [sub.domain.tld] solving challenges: **presenting for challenge: adding temporary record for zone \"sub.domain.tld.\": failed to append record** {_acme-challenge 0s TXT aaaa-redacted-bbbb-redactred-cccc}: invalid response format (order=https://acme.zerossl.com/v2/DV90/order/aaaa-redacted-bbbb) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":3.302566101,"max_duration":2592000}

Interestingly, it looks pretty much the same as issue #1 by @fhaala, some months ago. Unfortunately he didn't explain how he solved his issue. I included and compiled according to the docs (of course^tm).

Any ideas?
Thanks a lot!

[ljelschen]

This is now fixed in v0.2.2 of libdns/all-inkl — the root cause was three bugs in the underlying library:

1. SOAP Fault responses not handled — KAS returns a <Fault> XML root element on errors (e.g. record_already_exists, flood limits, auth failures) instead of the normal <KasApiResponse>. The code was not checking for this, so every KAS error silently returned "invalid response format" — which is the error you were seeing.

2. Cache not updated after AppendRecord — After successfully adding the TXT record, the in-memory record cache was never updated with the new record ID. So when Caddy called DeleteRecord to clean up the challenge, it couldn't find the record and failed with "no memory of presenting a DNS record".

3. record_aux set to TTL instead of 0 — For non-MX records, record_aux should be 0. Was incorrectly sending the TTL value, which could cause KAS to reject the request.

The fix has been merged into caddy-dns/all-inkl (PR #3). To use it, rebuild your Caddy binary with xcaddy and you'll get the patched version automatically.

[Darkwing371]

Thanks for taking care! 👍 It works ... better now.

There seem to be two more problems:

1. When I have several domains set in the caddyfile, and the DNS-01-Challenge is performed several times in a row, the KASAPI blocks the request due to its flood protection. I believe the timeout is 2 seconds by default, but I read it is part of the API's response, so it can be respected dynamically. In other words: can you read the flood protection timeout from a KASAPI's response, wait these couple of seconds and only then start the next DNS-01-Challenge utilizing the KASAPI? I guess, this is the way it should be done to be safe.

2. I noticed that all my subdomains work great with the DNS-01-Challenge if they are of the CNAME type. But I have one NS type and it does not want to work. KASAPI responds with "zone_not_found", which is kind of ridiculous. Let's Encrypt explicitly states, that CNAME and NS types are supported. But does the KASAPI call for this need to be a little different, maybe? Do you have an idea what could be wrong?

The caddy journal for this:

░░ A start job for unit caddy.service has finished successfully.
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5861926,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/root/.local/share/caddy","instance":"aaaa-bbbb-cccc-dddd","try_again":1773066983.5861907,"try_again_in":86399.999999585}
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5863013,"logger":"tls","msg":"finished cleaning storage units"}
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5871196,"logger":"tls.obtain","msg":"acquiring lock","identifier":"dyn.domain.tld"}
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5951364,"logger":"tls.obtain","msg":"lock acquired","identifier":"dyn.domain.tld"}
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5953722,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"dyn.domain.tld"}
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5963917,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"letsencrypt@domain.tld"}
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5964305,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"letsencrypt@domain.tld"}
Mar 08 15:36:23 caddy caddy[30181]: {"level":"info","ts":1772980583.5964546,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/123456789","account_contact":["mailto:letsencrypt@domain.tld"]}
Mar 08 15:36:24 caddy caddy[30181]: {"level":"info","ts":1772980584.756392,"msg":"trying to solve challenge","identifier":"dyn.domain.tld","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Mar 08 15:36:26 caddy caddy[30181]: {"level":"error","ts":1772980586.252928,"msg":"cleaning up solver","identifier":"dyn.domain.tld","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.dyn.domain.tld\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:331\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:376\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:149\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.11.2/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:73"}
Mar 08 15:36:26 caddy caddy[30181]: {"level":"error","ts":1772980586.4722908,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dyn.domain.tld","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[dyn.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"dyn.domain.tld.\": failed to append record {_acme-challenge 0s TXT aaaa-bbbb-cccc-dddd}: KAS API error: zone_not_found (order=https://acme-v02.api.letsencrypt.org/acme/order/123456789/987654321) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Mar 08 15:36:26 caddy caddy[30181]: {"level":"info","ts":1772980586.47263,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme.zerossl.com/v2/DV90","account":"letsencrypt@domain.tld"}
Mar 08 15:36:26 caddy caddy[30181]: {"level":"info","ts":1772980586.472643,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme.zerossl.com/v2/DV90","account":"letsencrypt@domain.tld"}
Mar 08 15:36:26 caddy caddy[30181]: {"level":"info","ts":1772980586.472657,"logger":"http","msg":"using ACME account","account_id":"https://acme.zerossl.com/v2/DV90/account/aaa-bbb-ccc","account_contact":["mailto:letsencrypt@domain.tld"]}
Mar 08 15:36:26 caddy caddy[30181]: {"level":"info","ts":1772980586.8800528,"msg":"trying to solve challenge","identifier":"dyn.domain.tld","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Mar 08 15:36:28 caddy caddy[30181]: {"level":"error","ts":1772980588.1432428,"msg":"cleaning up solver","identifier":"dyn.domain.tld","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.dyn.domain.tld\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:331\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:376\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:149\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.11.2/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:73"}
Mar 08 15:36:28 caddy caddy[30181]: {"level":"error","ts":1772980588.2160392,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dyn.domain.tld","issuer":"acme.zerossl.com-v2-DV90","error":"[dyn.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"dyn.domain.tld.\": failed to append record {_acme-challenge 0s TXT 0aaaa-bbbb-cccc-dddd}: KAS API error: zone_not_found (order=https://acme.zerossl.com/v2/DV90/order/aaa-bbb-ccc) (ca=https://acme.zerossl.com/v2/DV90)"}
Mar 08 15:36:28 caddy caddy[30181]: {"level":"error","ts":1772980588.2160823,"logger":"tls.obtain","msg":"will retry","error":"[dyn.domain.tld] Obtain: [dyn.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"dyn.domain.tld.\": failed to append record {_acme-challenge 0s TXT 0aaaa-bbbb-cccc-dddd}: KAS API error: zone_not_found (order=https://acme.zerossl.com/v2/DV90/order/aaa-bbb-ccc) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":4.620908493,"max_duration":2592000}
Mar 08 15:37:28 caddy caddy[30181]: {"level":"info","ts":1772980648.2189507,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"dyn.domain.tld"}
Mar 08 15:37:28 caddy caddy[30181]: {"level":"info","ts":1772980648.2195024,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/123456789","account_contact":["mailto:letsencrypt@domain.tld"]}
Mar 08 15:37:29 caddy caddy[30181]: {"level":"info","ts":1772980649.2578475,"msg":"trying to solve challenge","identifier":"dyn.domain.tld","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 08 15:37:30 caddy caddy[30181]: {"level":"error","ts":1772980650.5375009,"msg":"cleaning up solver","identifier":"dyn.domain.tld","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.dyn.domain.tld\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:331\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:376\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:149\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.11.2/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:73"}
Mar 08 15:37:30 caddy caddy[30181]: {"level":"error","ts":1772980650.7098837,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dyn.domain.tld","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[dyn.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"dyn.domain.tld.\": failed to append record {_acme-challenge 0s TXT aaaa-bbbb-cccc-dddd}: KAS API error: zone_not_found (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/123456789/987654321) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Mar 08 15:37:30 caddy caddy[30181]: {"level":"info","ts":1772980650.7102048,"logger":"http","msg":"using ACME account","account_id":"https://acme.zerossl.com/v2/DV90/account/aaa-bbb-ccc","account_contact":["mailto:letsencrypt@domain.tld"]}

For context, DNS Records look a little something like this:

Name	Type	Value	Result
sub1	CNAME	dyn.domain.tld.	✅
sub2	CNAME	dyn.domain.tld.	✅
sub3	CNAME	dyn.domain.tld.	✅
dyn	NS	dyndns1.kasserver.com.	❌

HTTP-01-Challenge worked with all of them. Funnily enough, also Wildcard Certificate (DNS-01-Challenge) worked with all of them.

Can you look into it again?
Thank you very much in advance!

[ljelschen]

Thanks for the feedback! Glad it's working better.

For the flood delay, I have an idea to make it dynamic.

For issue #2, could you share the relevant part of your Caddyfile (domains redacted is fine)? Specifically interested in how many domains you have configured and whether you're using wildcards. That'll help reproduce the exact scenario. Thanks!

[Darkwing371]

Hi, thanks for getting back!

I must have confused you; I tested several scenarios:

1. HTTP-01-Challenge with a bunch of domains (up to 5)
2. DNS-01-Challenge with a bunch of domains (up to 5)
3. DNS-01-Challenge with a Wildcard Certificate for all of the domains

All these scenarios would need a different Caddyfile configuration, of course. Scenarios 1 and 3 now work great, but scenario 2 still led to some errors, as stated in the post above. It's hard to test scenario 2 further, as long as KASAPI's flood protection blocks the action. I suspect, that the "zone_not_found" error is also due to a race condition because of flood protection. It potentially messes up all kind of things. Satisfying flood protection is the issue that needs to be resolved with priority.

But since you asked, there is really nothing special about my Caddyfile in scenario 2, like:

{
     email letsencrypt@domain.tld
     acme_dns allinkl {
                      kas_username {env.KAS_USERNAME}
                      kas_password {env.KAS_PASSWORD}
                      }
}

sub1.domain.tld {
     respond "sub1"
}

sub2.domain.tld {
     respond "sub2"
}

sub3.domain.tld {
     respond "sub3"
}

sub4.domain.tld {
     respond "sub4"
}

sub5.domain.tld {
     respond "sub5"
}

If you pursue your idea about taking flood protection into account, I sense a strong chance that also the other problem will go away.

Again, thanks a lot for your work!
Looking forward to testing a new version.

[ljelschen]

Thanks for keep testing the new versions!

The KasFloodDelay value from the API response wasn't being read correctly (wrong nesting level in the response structure), so the code always falling back to a hardcoded delay. This is now changed!

I tested locally with 5 domains simultaneously (your exact Caddyfile setup) — all 5 certs obtained cleanly with zero flood_protection errors.

So happy testing!

[Darkwing371]

Nice, flood protection seems not to be a problem any more. Works great, thanks! 👍

But here's the but: unfortunately the issue with DNS Record Type 'NS' cemented itself pretty clearly. 🤔 This is a thing to explore. I rephrase my above comments and post again:

I set Caddyfile up to perform a DNS-01-Challenge, for several subdomains:

{
     email letsencrypt@domain.tld
     acme_dns allinkl {
                      kas_username {env.KAS_USERNAME}
                      kas_password {env.KAS_PASSWORD}
                      }
}

sub1.domain.tld {
     respond "sub1"
}

sub2.domain.tld {
     respond "sub2"
}

sub3.domain.tld {
     respond "sub3"
}

sub4.domain.tld {
     respond "sub4"
}

dyn.domain.tld {
     respond "dyn"
}

Four subdomains are of type CNAME, one is of type NS (pointing to all-inkl’s DynDNS server).

When restarting Caddy and having it performing the DNS-01-Challenge, it works for the CNAME type subdomains, but not for the NS type subdomain. Let’s Encrypt states clearly, that both record types should work and can be used in a DNS-01-Challenge.

Since Let’s Encrypt follows the DNS standards when looking up TXT records for DNS-01 validation, you can use CNAME records or NS records to delegate answering the challenge to other DNS zones. This can be used to delegate the _acme-challenge subdomain to a validation-specific server or zone.

Here’s my DNS Record configuration:

Name	Type	Value	Certificate
sub1	CNAME	dyn.domain.tld.	✅
sub2	CNAME	dyn.domain.tld.	✅
sub3	CNAME	dyn.domain.tld.	✅
sub4	CNAME	dyn.domain.tld.	✅
dyn	NS	dyndns1.kasserver.com.	❌

KASAPI responds with "zone_not_found", resulting in "failed to append record" and "could not get certificate from issuer". Here is the log about this error:

░░ A start job for unit caddy.service has finished successfully.
Mar 10 15:56:22 caddy caddy[36690]: {"level":"info","ts":1773154582.4849677,"logger":"tls.obtain","msg":"acquiring lock","identifier":"dyn.domain.tld"}
Mar 10 15:56:22 caddy caddy[36690]: {"level":"info","ts":1773154582.4865086,"logger":"tls.obtain","msg":"lock acquired","identifier":"dyn.domain.tld"}
Mar 10 15:56:22 caddy caddy[36690]: {"level":"info","ts":1773154582.4866462,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"dyn.domain.tld"}
Mar 10 15:56:22 caddy caddy[36690]: {"level":"info","ts":1773154582.4872496,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"letsencrypt@domain.tld"}
Mar 10 15:56:22 caddy caddy[36690]: {"level":"info","ts":1773154582.4872613,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"letsencrypt@domain.tld"}
Mar 10 15:56:22 caddy caddy[36690]: {"level":"info","ts":1773154582.487272,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/123456789","account_contact":["mailto:letsencrypt@domain.tld"]}
Mar 10 15:56:23 caddy caddy[36690]: {"level":"info","ts":1773154583.6860623,"msg":"trying to solve challenge","identifier":"dyn.domain.tld","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Mar 10 15:56:24 caddy caddy[36690]: {"level":"error","ts":1773154584.0663714,"msg":"cleaning up solver","identifier":"dyn.domain.tld","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.dyn.domain.tld\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:331\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:376\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:149\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.11.2/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:73"}
Mar 10 15:56:24 caddy caddy[36690]: {"level":"error","ts":1773154584.28451,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dyn.domain.tld","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[dyn.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"dyn.domain.tld.\": failed to append record {_acme-challenge 0s TXT aaaa-bbbb-cccc-dddd}: KAS API error: zone_not_found (order=https://acme-v02.api.letsencrypt.org/acme/order/123456789/987654321) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Mar 10 15:56:24 caddy caddy[36690]: {"level":"info","ts":1773154584.2848861,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme.zerossl.com/v2/DV90","account":"letsencrypt@domain.tld"}
Mar 10 15:56:24 caddy caddy[36690]: {"level":"info","ts":1773154584.284908,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["dyn.domain.tld"],"ca":"https://acme.zerossl.com/v2/DV90","account":"letsencrypt@domain.tld"}
Mar 10 15:56:24 caddy caddy[36690]: {"level":"info","ts":1773154584.2849247,"logger":"http","msg":"using ACME account","account_id":"https://acme.zerossl.com/v2/DV90/account/aaa-bbb-ccc","account_contact":["mailto:letsencrypt@domain.tld"]}
Mar 10 15:56:24 caddy caddy[36690]: {"level":"info","ts":1773154584.8745856,"msg":"trying to solve challenge","identifier":"dyn.domain.tld","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Mar 10 15:56:26 caddy caddy[36690]: {"level":"error","ts":1773154586.4517014,"msg":"cleaning up solver","identifier":"dyn.domain.tld","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.dyn.domain.tld\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:331\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:376\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.6/client.go:149\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.25.2/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.11.2/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.25.2/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.25.2/async.go:73"}
Mar 10 15:56:26 caddy caddy[36690]: {"level":"error","ts":1773154586.5101557,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dyn.domain.tld","issuer":"acme.zerossl.com-v2-DV90","error":"[dyn.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"dyn.domain.tld.\": failed to append record {_acme-challenge 0s TXT aaaa-bbbb-cccc-dddd}: KAS API error: zone_not_found (order=https://acme.zerossl.com/v2/DV90/order/12345678987654321) (ca=https://acme.zerossl.com/v2/DV90)"}
Mar 10 15:56:26 caddy caddy[36690]: {"level":"error","ts":1773154586.5102236,"logger":"tls.obtain","msg":"will retry","error":"[dyn.domain.tld] Obtain: [dyn.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"dyn.domain.tld.\": failed to append record {_acme-challenge 0s TXT aaaa-bbbb-cccc-dddd}: KAS API error: zone_not_found (order=https://acme.zerossl.com/v2/DV90/order/h0XOOmqQY6km-fld40Ew_w) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":4.023695566,"max_duration":2592000}

Can you reproduce? Can you test with an NS type DNS record? It’s pretty odd, that this is not working. I can hardly believe, that it maybe a misconfiguration of this NS type on my side.

Thanks again,
hope to hear back on this!

EDIT:
It begins to dawn on me, that the part about CNAME and NS records needs to be read in the following way:

Since Let’s Encrypt follows the DNS standards when looking up TXT records for DNS-01 validation, you can use ( CNAME records ) or ( NS records to delegate answering the challenge to other DNS zones ). This can be used to delegate the _acme-challenge subdomain to a validation-specific server or zone. (Refers to NS records.)

Following the link to eff.org, one can find such a phrasing:

A subdelegate DNS zone is defined using NS records and it effectively delegates the complete control over a part of the zone to an external authority.

So this would mean, that for a domain of an NS record type, no certificate can be issued directly, because Let’s Encrypt expects to be delegated to another domain, where the NS records points to.

This would provide an explanation for the observed behaviour, would you agree?

[cweickhmann]

Just a ping here: I seem to have a similar problem trying to obtain a wildcard certificate.

{
  "level":"error",
  "ts":1776031789.3459105,
  "logger":"tls.obtain",
  "msg":"will retry",
  "error":"[*.<MYDOMAIN>.de] Obtain: [*.<MYDOMAIN>.de] solving challenges: presenting for challenge: adding temporary record for zone \"de.\": failed to append record {_acme-challenge.<MYDOMAIN> 0s TXT pd0-B7hZgMSNEaCBYUvECfUklTQZFO_9EOxxls5H7r4}: KAS API error: zone_syntax_incorrect (order=https://acme.zerossl.com/v2/DV90/order/Yi2KklJ7Y6zKM9wdH4ZV-A) (ca=https://acme.zerossl.com/v2/DV90)",
  "attempt":1,
  "retrying_in":60,
  "elapsed":4.030720733,
  "max_duration":2592000
}

Is it correct, that it says _acme-challenge.<MYDOMAIN> with .de (or even .de.) missing?
Yup, from playing with all-inkl's KAS API that almost certainly is the issue here. Does zone_host lose the TLD somehow? For reference, this is the Caddyfile:

{
  email info@<MYDOMAIN>.de
  acme_dns allinkl {
    kas_username {env.KAS_USERNAME}
    kas_password {env.KAS_PASSWORD}
  }
}

*.<MYDOMAIN>.de {
  @grafana host grafana.<MYDOMAIN>.de
  handle @grafana {
    reverse_proxy grafana:3000
  }

[Darkwing371]

Just a guess here, not sure about your setup as a whole, but:

The error sounds more like a DNS misconfiguration (typo). Have you tried to obtain a single certificate for this subdomain? Does this work in the first place?

The DNS record(s) for it should look like this:

… while "dyn" is the general DynDNS subdomain to point to your home server. (Assuming you have such a setup.)

The Caddyfile:

grafana.domain.de {
    reverse_proxy grafana:3000
  }

I personally stepped away from Wildcard certificates via DNS-01-Challenge – due to the restriction I may have discovered (see my post above). If you get it to work with single certificates, stick with it and just be happy. ;)

[cweickhmann]

No, I did not try it with individual certificates yet. I mainly toyed around with it to get rid of certificate-invalid-errors in my internal network. I'm not, right now at least, interested in having those public.

But it's a very good thing to try. Thanks!

[Darkwing371]

Ok, I'm not sure I understood your use case correctly, but: Let's Encrypt is not meant (and able) to obtain certificates for local domains. They need to be "public" on a webserver and in a DNS record.

If it is only for your personal use in your local network, you may be better off with looking into a thing called "self-signed certificates" and install these in your browsers and operating systems. This way, you can get rid of the "invalid certificate warning" and also can set a very very long expiration date, so that you don't need to care to renew constantly. I believe, it's the better option for you (and Caddy or Lets' Encrypt is actually the wrong tool for this case [you can, however, serve your self-signed certificates with Caddy later]).

[cweickhmann]

It's a bit off topic here, but I beg to differ :-) I'm well aware of self-signed certificates and they are a pain in most browsers because they are essentially not accepted (with Firefox being a particular PITA as it treats self-signed even more suspiciously than un-encrypted), unless you distribute the root cert manually. Also, using a fantasy domain you don't control is not a good idea for a bunch of reasons (see the whole fritz.box issue).
Common practice is to register a public domain and use it for internal stuff too. That's all fine and ACME actually supports using wildcard certificates.

I wanted to use wildcard to expose as little as possible about my private network to the public. But if it works by creating a couple of CNAMEs, like grafana.domain.de, I'll take that. (Haven't tried yet).

Reason for my question was that I couldn't quite understand where the .de. part got lost. And I got stuck analysing that further because I could not figure out how to get log messages from the all-inkl go-module out to the console (only error messages show up, using fmt.Println or log.Printf didn't show up - I have no idea about Go, so there's that ^^).

[Darkwing371]

Alright, I have no answer for your problem with the .de. part or the zone stuff, I'm sorry. But I don't strongly think it's a software problem, since I never came across it myself, during my tests and usage of Caddy and the all-inkl module.

But maybe one last tip: I noticed in Chrome/Chromium, there's a setting to "warn only about insecure public sites":

This eases the warning about local sites not being https. Maybe Firefox has something similar. (And I noticed Brave doesn't seem to have it at all.)

Good luck!

[ljelschen]

Hey, I was actually dealing with the domain suffix (.de.) issue earlier, but I used a bit of a dirty workaround in the program. Right now, whatever comes in gets normalized to ensure there’s a trailing dot:

"zone_host": strings.TrimSuffix(zone, ".") + ".",

(see: https://github.com/libdns/all-inkl/blob/main/client.go)

I think this should really be handled based on what the API expects rather than forcing it unconditionally. I’m a bit short on time at the moment, but once I get a chance, I’ll try to reproduce the error properly and work on a cleaner fix.