Secrets management for cloud apps is the practice of storing, delivering, rotating, auditing, and revoking sensitive values such as database passwords, API keys, signing keys, service tokens, SSH private keys, and TLS private keys.
The practical decision is not:
Where can I put a secret?
The better question is:
What is the lightest secrets workflow that still lets us rotate, restrict, audit, and recover safely?
Use this rule:
| Situation | Best starting point |
|---|---|
| One small app, few secrets, trusted operators | Disciplined environment variables |
| Several services, multiple environments, central access control needed | Managed secret store |
| Many services, dynamic secrets, strict audit/revocation, multi-cloud policy | Vault-style system |
| Kubernetes app with cluster-native delivery | Kubernetes Secrets plus encryption/RBAC, or external secret store |
| Docker Swarm service | Docker secrets |
| CI/CD deployment secrets | CI/CD secret store or external secrets manager |
| Production database credentials | Avoid shared long-lived passwords where possible |
The most important point:
Environment variables are a delivery mechanism. They are not a complete secrets management system.
A mature secrets workflow handles the whole lifecycle:
Create → Store → Deliver → Use → Rotate → Revoke → Audit → Recover
What counts as a secret?
A secret is any value that grants access, identity, trust, or privileged capability inside your system.
Common examples:
- database usernames and passwords;
- API keys;
- OAuth client secrets;
- JWT signing secrets;
- session signing keys;
- SSH private keys;
- TLS private keys;
- webhook signing secrets;
- cloud access keys;
- CI/CD deployment tokens;
- container registry credentials;
- service-to-service tokens;
- encryption keys;
- backup access credentials.
A secret is different from normal configuration.
This is configuration:
APP_ENV=production PORT=3000 PUBLIC_BASE_URL=https://example.com
This is a secret:
DATABASE_PASSWORD=... STRIPE_SECRET_KEY=... JWT_SIGNING_SECRET=...
Normal configuration can usually appear in logs, dashboards, and documentation.
Secrets should not.
Why secrets management breaks down
Secrets usually start simple.
A small app has:
DATABASE_URL API_KEY JWT_SECRET
Then the system grows:
- staging is added;
- CI/CD is added;
- a second service is added;
- contractors get access;
- backups are introduced;
- logs are centralized;
- Kubernetes or Docker enters the stack;
- a team member leaves;
- an API key leaks;
- the database password needs rotation.
At that point, the question becomes operational.
Common failure patterns:
.envfiles copied between machines;- production secrets reused in staging;
- secrets committed to Git;
- credentials pasted into Slack, tickets, docs, or screenshots;
- CI logs printing secret values;
- Docker images containing old credentials;
- one database password shared by every service;
- no inventory of what secrets exist;
- no known owner for each secret;
- no tested rotation process;
- no audit trail of secret access;
- backups containing plaintext secret exports.
Secret sprawl is not only a storage problem.
It is an ownership, access, rotation, and recovery problem.
The three main models
Most cloud teams choose between three patterns:
1. Environment variables 2. Vault-style systems 3. Managed secret stores
These are not mutually exclusive.
A common production architecture is:
Managed store or vault ↓ Deployment system ↓ Environment variable or mounted file ↓ Application runtime
In other words, the app may still read a secret from an environment variable, even if the real source of truth is a vault or managed store.
That distinction matters.
Model 1 — Environment variables
Environment variables are the most common starting point for cloud apps.
They work across almost every language and framework:
- Node.js;
- Python;
- PHP;
- Laravel;
- Django;
- Rails;
- Go;
- Java;
- .NET;
- Docker;
- systemd;
- CI/CD tools;
- PaaS platforms.
The Twelve-Factor App popularized storing deploy-specific configuration in the environment, including credentials to external services. The same model also warns against storing deploy-specific config as constants in source code.
What env vars do well
Environment variables are useful because they are:
- simple;
- language-agnostic;
- easy to inject at deploy time;
- easy to separate by environment;
- supported by most hosting models;
- familiar to developers;
- good for early-stage apps.
For a small app running on one Linux VM, this can be enough if access is controlled.
Example:
DATABASE_URL=postgres://app_user:strong-password@10.0.0.5:5432/app REDIS_URL=redis://10.0.0.6:6379 JWT_SECRET=long-random-value
Environment variables are especially useful when:
- the app has few secrets;
- only a small trusted team has server access;
- staging and production are cleanly separated;
- rotation is manual but still manageable;
- there are no strict audit requirements;
- the app is not part of a larger regulated platform.
Where env vars become risky
Environment variables are not enough when lifecycle control matters.
They do not automatically tell you:
- who created the secret;
- who viewed the secret;
- who changed it;
- when it was last rotated;
- whether it is reused elsewhere;
- which service should be allowed to access it;
- whether it should expire;
- whether it was included in backups;
- whether it appeared in logs;
- whether the old value was revoked.
Common env var problems:
.envfiles checked into Git by mistake;- secrets copied into local development;
- production values reused in staging;
- no central secret inventory;
- shell history exposure;
- debug endpoint exposure;
- process inspection risk;
- Docker image build leakage;
- no safe rotation workflow.
The right way to think about env vars:
Good delivery mechanism. Weak lifecycle system.
Best practices for env vars
Use environment variables safely by following these rules:
- Never commit
.envfiles. - Add
.env,.env.*, and local secret files to.gitignore. - Keep
.env.examplewith names only, not values. - Use separate values for dev, staging, and production.
- Limit SSH and dashboard access to production hosts.
- Avoid printing env vars in logs.
- Avoid putting secrets in process arguments.
- Document who owns each production secret.
- Keep a rotation runbook.
- Move to a managed store or vault when rotation/access control becomes painful.
A good .env.example:
DATABASE_URL= REDIS_URL= JWT_SECRET= PAYMENT_API_KEY=
A bad .env.example:
DATABASE_URL=postgres://prod_user:real-password@prod-db:5432/app
Model 2 — Vault-style systems
A vault is a dedicated secrets control plane.
The best-known example is HashiCorp Vault, but the concept matters more than the specific product.
A vault can provide:
- centralized secret storage;
- authentication;
- authorization policies;
- audit logging;
- dynamic secrets;
- lease-based access;
- secret rotation workflows;
- encryption services;
- revocation;
- service identity integration.
The big change is this:
Instead of copying long-lived secrets into apps, services authenticate to the vault and receive allowed secrets under policy.
What vaults do well
Vault-style systems are strongest when secrets have become a platform problem.
They help with:
- service-specific access policies;
- reducing shared credentials;
- dynamic database credentials;
- short-lived tokens;
- audit trails;
- revocation;
- multi-cloud consistency;
- security team review;
- policy-as-code style management;
- platform-wide secret standards.
Example:
App service authenticates to Vault ↓ Vault checks identity and policy ↓ Vault returns short-lived database credentials ↓ App connects to database ↓ Credentials expire or are revoked
This reduces blast radius.
If one service is compromised, it should not automatically expose every credential in the environment.
The real cost of vaults
Vaults are powerful, but they add operational responsibility.
You must manage:
- vault deployment;
- high availability;
- storage backend;
- unseal/recovery process;
- access policies;
- authentication methods;
- audit logs;
- upgrades;
- monitoring;
- backups;
- break-glass access;
- incident response.
For a small team, a vault can become a critical internal service before the team is ready to operate one.
A useful test:
Would running Vault reduce more complexity than it adds?
If the answer is no, start lighter.
When Vault is the right call
Choose a vault-style system when:
- you have many services;
- each service needs different access;
- shared credentials are becoming dangerous;
- dynamic secrets would reduce risk;
- audit trails are required;
- secrets are used across multiple clouds;
- you need short-lived credentials;
- security controls must be centralized;
- the team can operate a critical internal platform.
Do not deploy Vault only because it sounds mature.
Deploy it when you are ready to own it.
Model 3 — Managed secret stores
Managed secret stores sit between env vars and self-operated vaults.
Examples include cloud-native services such as:
- AWS Secrets Manager;
- AWS Systems Manager Parameter Store;
- Google Secret Manager;
- Azure Key Vault;
- provider-specific secret stores;
- platform secret stores in CI/CD systems.
A managed store gives centralized secret storage and access control without making your team operate the secret service itself.
What managed stores do well
Managed stores are often the best fit for small cloud-native teams because they provide:
- centralized inventory;
- IAM-based access control;
- encryption;
- audit logs;
- API-based retrieval;
- rotation support in some cases;
- cloud-native integration;
- less operational overhead than Vault.
They are useful when:
- you already live mostly inside one cloud/platform;
- you want better control than raw env vars;
- you do not want to operate Vault;
- IAM access control is enough;
- rotation and audit are becoming important;
- developers need a standard place to store secrets.
A typical pattern:
Managed secret store ↓ CI/CD retrieves secret at deploy time ↓ App receives secret as env var or file
Or:
App authenticates using instance/workload identity ↓ App retrieves secret at runtime
Limits of managed stores
Managed stores are not perfect.
Trade-offs:
- provider-specific APIs;
- cloud identity lock-in;
- cross-cloud consistency challenges;
- cost per secret/API call in some services;
- rotation may require custom integration;
- runtime dependency on provider availability;
- less flexible dynamic secret model than a full vault.
For many small teams, those trade-offs are acceptable.
The goal is not theoretical portability.
The goal is safer production operations.
Containers and secrets
Containerized apps add extra leak paths.
Avoid putting secrets into:
- Dockerfiles;
- build arguments that end up in image history;
- committed Compose files;
- public container images;
- container labels;
- debug layers;
- image names or tags;
- logs.
Bad pattern:
ENV DATABASE_PASSWORD=real-password
Better pattern:
Secret comes from deployment environment, secret store, or mounted secret file at runtime.
Docker Swarm secrets can centrally manage sensitive data and only grant access to services that need it, but Docker notes that Docker secrets are for Swarm services, not standalone containers.
For plain Docker Compose on a single VM, be careful: a secrets: block may still map to local files depending on the Compose mode. Do not assume it gives the same security model as Swarm or Kubernetes.
Kubernetes secrets
Kubernetes Secrets are designed for small sensitive values such as passwords, tokens, and keys.
They let you keep secrets out of application code and Pod specs.
But Kubernetes Secrets need configuration discipline.
Kubernetes documentation warns that Secrets are stored unencrypted in etcd by default and that API or etcd access can expose them. The docs recommend enabling encryption at rest, configuring least-privilege RBAC, restricting Secret access to specific containers, and considering external secret store providers.
For Kubernetes, use these rules:
- enable encryption at rest for Secrets;
- restrict Secret access with RBAC;
- avoid giving broad namespace access;
- do not let every workload read every Secret;
- prefer mounted files over env vars when rotation behavior matters;
- avoid storing secrets in Helm values files committed to Git;
- review who can create Pods in a namespace;
- consider external secret operators or CSI drivers when secrets need to come from a managed store.
A Kubernetes Secret is useful.
It is not automatically secure by default.
CI/CD secrets
CI/CD is one of the most common places secrets leak.
Pipelines often need:
- deploy keys;
- registry tokens;
- cloud credentials;
- SSH keys;
- database migration credentials;
- webhook tokens;
- package repository tokens.
Rules for CI/CD:
- Store secrets in the CI/CD secret store or external manager.
- Never print secrets in logs.
- Mask values where possible.
- Use short-lived credentials when possible.
- Avoid broad cloud admin keys.
- Separate dev/staging/prod deployment credentials.
- Rotate tokens when team members leave.
- Restrict who can modify pipelines.
- Avoid secrets in pull requests from untrusted forks.
- Keep deployment credentials separate from human credentials.
A pipeline with production secrets is production infrastructure.
Treat it that way.
Comparison: env vars vs vaults vs managed stores
| Criteria | Environment variables | Vault-style system | Managed secret store |
|---|---|---|---|
| Setup complexity | Low | High | Medium |
| Operational overhead | Low | High | Low to medium |
| Central inventory | Weak | Strong | Strong |
| Access policy | Weak to medium | Strong | Medium to strong |
| Auditability | Limited | Strong | Strong |
| Rotation support | Manual/custom | Strong | Medium to strong |
| Dynamic secrets | No | Yes | Limited/provider-specific |
| Multi-cloud consistency | Medium | Strong | Weak to medium |
| Best fit | Small/simple apps | Mature platforms | Cloud-native small teams |
| Main risk | Secret sprawl | Platform overhead | Provider lock-in |
| Skill requirement | Low | High | Medium |
| Incident response | Manual | Strong | Medium to strong |
No option is universally best.
The right answer depends on maturity.
Practical decision framework
Choose environment variables when:
- you run one or two applications;
- the number of secrets is small;
- production access is limited;
- deployment is simple;
- rotation is still manageable;
- audit requirements are low;
- the team is small and trusted;
- you can keep secrets out of Git and logs.
Environment variables are a good first step when paired with discipline.
They become risky when used as the only long-term secrets system.
Choose a managed secret store when:
- you need central storage;
- you want access control without operating Vault;
- secrets are growing across services;
- rotation is becoming important;
- you already depend on one cloud or platform;
- IAM fits your access model;
- audit logs matter;
- your team wants a practical middle ground.
For many small teams, this is the best next step after env vars.
Choose a vault when:
- secrets management is a platform problem;
- many services need different policy boundaries;
- dynamic secrets reduce meaningful risk;
- short-lived credentials are important;
- audit and revocation are required;
- you operate across multiple clouds or environments;
- the team can manage a critical internal control plane.
Vault is the strongest option, but it is not the lightest.
Recommended maturity path
For most small teams:
Stage 1: Disciplined env vars Stage 2: Separate dev/staging/prod values Access-controlled deployment process Secret inventory Stage 3: Managed secret store Rotation runbooks Audit logs Stage 4: Vault-style platform Dynamic secrets Strong policies Centralized audit and revocation
This path avoids over-engineering on day one while still moving toward a stronger security model.
Secret rotation strategy
A rotation strategy answers:
- what rotates;
- how often;
- who owns it;
- what depends on it;
- how to deploy the new value;
- how to revoke the old value;
- how to roll back;
- how to verify success.
For every production secret, document:
Name: Owner: Used by: Environment: Source of truth: Rotation frequency: Rotation steps: Rollback steps: Last rotated: Emergency contact:
Safe rotation pattern
A safe rotation usually looks like this:
1. Create new secret value. 2. Add new value without removing old value. 3. Deploy apps that can use new value. 4. Verify traffic works. 5. Revoke old value. 6. Monitor errors. 7. Update inventory.
Do not rotate by deleting the old value first.
That creates avoidable downtime.
Least privilege for secrets
Each service should access only the secrets it needs.
Bad pattern:
All apps use one shared production database password.
Better pattern:
api-service gets api_db_user. worker-service gets worker_db_user. analytics-service gets analytics_readonly_user.
Benefits:
- smaller blast radius;
- easier revocation;
- clearer ownership;
- better audit trails;
- safer rotation;
- fewer surprise dependencies.
Least privilege applies to humans too.
Not every developer needs read access to production secret values.
Secret inventory
A small secret inventory can be simple.
Start with a table:
| Secret | Environment | Owner | Used by | Source of truth | Rotation |
|---|---|---|---|---|---|
| DATABASE_URL | production | Platform | API | Managed store | 90 days |
| JWT_SECRET | production | Backend | API | Managed store | On incident |
| STRIPE_SECRET_KEY | production | Payments | API | Provider + store | On staff change/incidents |
| S3_ACCESS_KEY | staging | Platform | Worker | Managed store | 90 days |
This is not bureaucracy.
It is how you avoid discovering during an incident that nobody knows what a credential does.
Backup and recovery
Secrets management must include recovery.
Ask:
- How do we restore the secret store?
- Who has break-glass access?
- Where are recovery keys stored?
- Are backups encrypted?
- Are backups access-controlled?
- Are old secrets duplicated in backup exports?
- Can we recover without exposing every secret to every admin?
- How do we rotate after restore?
- How do we revoke after compromise?
Do not create casual plaintext exports of all secrets.
Backups should protect the secret management system without multiplying exposure.
Incident response when a secret leaks
If a secret leaks:
- Identify which secret leaked.
- Determine where it was exposed.
- Revoke or rotate it.
- Deploy the new value.
- Check logs for misuse.
- Search repositories, images, tickets, and logs for copies.
- Remove exposed copies where possible.
- Review access scope.
- Document the incident.
- Improve prevention.
Do not only delete the secret from Git and move on.
If a secret was committed, assume it may have been copied.
Rotate it.
Common mistakes
Mistake 1 — Treating .env as the secret source of truth
A .env file may be a delivery mechanism, but it should not become the only uncontrolled production secret store.
Mistake 2 — Reusing production secrets in staging
Staging should never unlock production resources.
Use separate credentials.
Mistake 3 — Giving every developer production secret access
Access should follow need, not curiosity.
Use deployment systems and role-based access.
Mistake 4 — No rotation plan
If rotation feels impossible, the system is too tightly coupled.
Design rotation before the emergency.
Mistake 5 — Secrets in Docker images
Never bake secrets into container images.
Images move through registries, caches, CI systems, and developer machines.
Mistake 6 — Assuming Kubernetes Secrets are encrypted by default
Kubernetes warns that Secrets are stored unencrypted in etcd by default unless encryption at rest is enabled.
Configure Kubernetes Secrets intentionally.
Mistake 7 — Ignoring logs
Secrets can leak through:
- error logs;
- request logs;
- debug logs;
- CI logs;
- crash dumps;
- APM traces.
Mask and redact sensitive fields.
Mistake 8 — No owner for each secret
Every production secret needs an owner.
Without ownership, rotation and incident response stall.
Mistake 9 — Over-engineering too early
A solo app does not always need a full vault.
Start with disciplined controls, then mature.
Mistake 10 — Under-engineering too long
If you have many services, contractors, production incidents, audits, and manual rotations, raw env vars are no longer enough.
How these choices apply on Raff
Raff does not force one secrets architecture.
That is useful because different teams need different maturity levels.
Small app on one Raff VM
For a single app on a Raff Linux VM:
App on VM ↓ systemd / Docker / app runtime ↓ environment variables
This can be acceptable if:
.envfiles are not committed;- SSH access is limited;
- production and staging values are different;
- file permissions are tight;
- backups are controlled;
- rotation is documented;
- logs do not print secrets.
Growing app with staging and production
When you add staging, workers, CI/CD, background jobs, and private services, move toward centralization.
Pattern:
Secret inventory ↓ Managed secret store or vault ↓ CI/CD or deployment agent ↓ Runtime env vars or mounted files ↓ Separate dev/staging/prod values
This reduces copy-paste and makes rotation easier.
Multi-service app on private networking
For apps split across VMs:
Web VM Worker VM Database VM Internal services Private network
Use separate credentials per service.
Example:
web_db_user worker_db_user readonly_reporting_user backup_user
Do not give every service the same database admin password.
Pair secrets management with private networking and firewall rules.
A secret should not be the only line of defense.
Kubernetes workloads
For Kubernetes workloads, treat Secrets as one layer.
Use:
- encryption at rest;
- RBAC least privilege;
- namespace boundaries;
- service accounts;
- external secret stores when needed;
- careful Helm/GitOps handling;
- secret rotation workflows;
- audit logging.
Do not put production secrets into public Helm values files.
Do not assume base64 encoding is encryption.
Raff API keys and automation
Raff API keys should be treated as production secrets.
Use:
- separate keys for different automation tasks;
- project-scoped access where possible;
- clear naming;
- expiration dates;
- rotation;
- revocation when team members leave;
- no keys in Git;
- no keys in screenshots;
- no keys in shared docs.
Automation keys are secrets.
Treat them like database passwords.
Recommended checklist
Before production, confirm:
- secrets are not in code;
.envfiles are ignored by Git;- dev, staging, and production use different values;
- each secret has an owner;
- production access is limited;
- CI/CD secrets are masked;
- rotation is documented;
- backups do not create uncontrolled plaintext secret exports;
- logs redact sensitive values;
- container images do not contain secrets;
- Kubernetes Secrets are encrypted and access-controlled if used;
- API keys have expiration and revocation process;
- old secrets are removed after rotation;
- incident response steps are known.
What Raff recommends
For most small cloud app teams:
- Start with disciplined env vars only if the app is small.
- Separate dev, staging, and production secrets early.
- Keep secrets out of Git, images, logs, and tickets.
- Create a simple secret inventory.
- Use a managed secret store when centralization and rotation matter.
- Use Vault when dynamic secrets and policy control justify the operational load.
- Rotate secrets with a safe two-phase process.
- Use private networking and firewalls so secrets are not your only boundary.
- Treat Raff API keys as production secrets.
- Keep backups and recovery materials secure.
The goal is not to buy the most advanced tool.
The goal is to reduce exposure while keeping the system operable.
Conclusion
Secrets management is not one tool.
It is a lifecycle.
Environment variables are a useful starting point, but they do not solve inventory, rotation, audit, revocation, or least privilege by themselves.
Managed secret stores are often the right next step for small cloud teams because they centralize control without making the team operate a full secrets platform.
Vault-style systems are strongest when secrets become a platform problem: dynamic credentials, short-lived access, strict audit, multiple services, and stronger policy boundaries.
The best choice is the lightest model that still lets your team answer these questions:
Where does each secret live? Who can access it? Which services use it? How do we rotate it? How do we revoke it? How do we recover it? How do we prove what happened?
If you can answer those clearly, your secrets strategy is on the right path.
