chutney

Warning

Status: 🚧 Work in Progress 🚧

This project might undergo changes as we scale the cache to cover 1000+ users at Juspay.

See #17 for current status on reliability.

chutney provides a NixOS + terranix configuration to deploy a reliable Nix binary cache server, powered by attic.

• Getting Started
• Limitations
• Operational Decisions
• Guide
  • Cache Creation
  • Secrets
  • Garbage Collection
  • Migration
• Gotchas
• Discussion

Getting Started

By default, chutney deploys its infrastructure on AWS. To get started, follow the AWS guide.

If you are not using AWS, you can use the standalone NixOS modules, which are decoupled from the terranix configuration. See the NixOS Modules guide.

Limitations

• Unbounded DB and S3 growth

Operational Decisions

Disable Chunking

Chunking saves storage space but costs significantly more due to S3 PUT request pricing. Disabling chunking saves ~95% on S3 PUT cost (see #48 (comment)). One might argue that the storage cost might bite us in the long run, but we don't have to worry about that given we solve #52.

Guide

Administrate cache

Login to attic using the root-token for admin related work:

cd secrets && nix run nixpkgs#attic-client -- login root https://cache.nixos.asia $(agenix -d attic/root-token.age)

Create cache

• Ensure you are logged in as an admin (see Administrate cache)
• Run nix run nixpkgs#attic-client cache create <cache-name>
• SSH into the host and generate the access token, see comments above attic/oss-push-token.age in secrets/secrets.nix. Also see https://docs.attic.rs/tutorial.html#access-control

Secrets

chutney uses agenix for secrets management.

Adding a new secret

Run cd secrets && agenix -e <mysecret.age>

Editing an existing secret

Run just secret-edit and select the key to edit.

Adding a new user/host

Add the new user/host in ./secrets/secrets.nix and run just secrets-rekey to allow the new user/host to decrypt the keys.

Garbage Collection

Note

Auto GC is disabled in chutney. The only way GC will free up space is if you have configured retention-period for your cache. See https://docs.attic.rs/tutorial.html#garbage-collection

Run Garbage Collection once:

ssh root@<public-ip>
sudo -u atticd attic-gc-once

Support more platforms in .terraform.lock.hcl

Currently only darwin_arm64 is supported. To manage infra from other platform/s, follow:

• mv .terraform.lock.hcl .terraform.lock.hcl.bkp
• terraform init
• Add back the extra hashes from .terraform.lock.hcl.bkp to .terraform.lock.hcl

We can't use the terraform providers lock -platform=<platform-1> -platform=<platform-2> ... as this command always fetches and locks the latest aws provider and not the pinned one from nixpkgs (The provider is pinned using terraform.withPlugins in devShells.default ).

Gotchas

Flaky destroy

destroy script can indefinitely keep trying to delete the aws_vpc.chutney, this happens (atleast with Juspay's AWS account) when the vpc has a non-default security group depenedency not managed by terraform. This dependency has to be manually deleted by running nix run .#vpc-sg-cleanup <vpc-id> in another terminal window.

HTTP 524

If your domain uses cloudflare and the requests are proxied through cloudflare, attic push might fail on large Nix store objects withHTTP 524. This is owing to cloudflare's 100 second timeout. You can fix this by changing Proxy status for your domain from Proxied to DNS only in the cloudflare dashboard, as the aforementioned page describes:

If you regularly run HTTP requests that take over 100 seconds to complete (for example, large data exports), move those processes behind a subdomain not proxied (grey clouded) in the Cloudflare DNS app.

Discussion

To discuss this project, post in GitHub Discussions or join the NixOS Asia community.