Recently we ran across an issue where a credential we'd just added passed development, CI, and staging yet failed in production. Why? A simple typo in a nested key name and nothing to catch it.

In all environments (except production) we had something like mailer.smtp.password and in production we had mailer.stmp.password. Review didn't catch this because Rails encrypts the whole credential file, keys and values together, so the diff is an opaque blob with nothing intelligible to read. For nested keys, we typically use dig to retrieve them. However, this method doesn't raise on a missing key, it returns nil. The typo failed quietly, in production, after a deploy. By the time it was noticed, the person debugging the nil credential wasn't the person who'd introduced it days earlier.

To prevent this from happening again, we thought about what we could do to catch the drift before it reaches production. If we assume that production is the ultimate source of truth, then any deviation from it is a potential problem. We happen to have a staging environment, which should match production exactly, so we can compare the two and fail if they don't. Development and test environments often do not have all the keys that production does, so we can consider those a subset of production, and allow them to declare fewer keys.

We considered using a git hook, a CI check, or a deploy check as our entry point, but none of them felt right. A git hook would require every developer to install it, and a CI check would require CI to have the production master key, which is a big tradeoff. A deploy check seemed too late to find this kind of problem. The check really belongs where the new key is introduced, which is a developer's machine, and where the framework already has hooks for exactly this kind of state check.

Looking at how Rails handles a pending migration, we realized we could use the same approach. Rails raises on page load in development and again as the test suite boots, so you find out the moment you run anything. Credential drift can surface the same way.

We packaged this into a gem

We packaged the fix as credential_parity (source on GitHub). It compares full leaf paths, applies the reference/mirror/subset rules, installs its middleware through a railtie, and ships a rails_helper hook.

Install is a Gemfile line and one line in your test helper:

gem "credential_parity"

# spec/rails_helper.rb
ActiveRecord::Migration.maintain_test_schema!
require "credential_parity/rspec"

Configuration mirrors the rule described above: reference defaults to :production, mirrors defaults to [:staging] (environments that must match the reference exactly), and subsets defaults to [:development, :test] (environments allowed to declare fewer keys, never more):

CredentialParity.configure do |config|
  config.reference = :production
  config.mirrors    = [:staging]
  config.subsets    = [:development, :test]
end

directory and middleware_environments are there for less common layouts, if your credentials live somewhere other than the default path or you want the per-request check running somewhere besides development. And for the rare legitimate case, a console session on a machine that's missing a key, setting SKIP_CREDENTIAL_PARITY_CHECK makes the check no-op.

Two ActiveSupport traps we hit building it

Both come up in any gem that touches framework internals: requiring a leaf ActiveSupport file without its parent, and a missing predicate that surfaces as an arity error instead of a NoMethodError.

We're already using it and it's been a great help in catching credential drift before it reaches production. Give it a try and let us know what you think!