--rotate-keys does nothing if using AWS KMS

[hovissimo]

Environment

Provide at least:

• Ruby Version.

ruby 2.5.8p224 (2020-03-31 revision 67882) [x86_64-linux]

• Symmetric Encryption Version.

hovisb@stage1:/usr/local/sites/HiringThing/current$ symmetric-encryption --version
Symmetric Encryption v4.3.2
OpenSSL v2.1.2
Environment: staging

• Application/framework names and versions (e.g. Rails, Sinatra, Puma, etc.).

hovisb@stage1:/usr/local/sites/HiringThing/current$ bin/rails --version
Rails 5.2.4.3

• Full Stack Trace, if an exception is being raised.
  No stack trace. See output from gem CLI in screenshot.

Expected Behavior

After running symmetric-encryption --rotate-keys --rolling-deploy --environments staging --app-name hiringthing I expect to find a new cipher value in my config (config/symmetric-encryption.yml) and a new encrypted data key in my key path (~/.symmetric-encryption)
Other than specifying an environment, this is the exact example at Step 1 here: http://encryption.rocketjob.io/key_rotation.html

Actual Behavior

I get a success message from the tool, but there are no new keys on my filesystem and the config file is not updated.

[hovissimo]

I have also tried --env staging and with no environment specified at all. I get identical behavior in each of these cases.
I have also verified that my AWS credentials are good. I can run aws kms generate-data-key-without-plaintext --key-id [REDACTED] --key-spec AES_256 on the very same console and get a good key back.

[hovissimo]

Reviewing the source code, it appears that the bug is here:
https://github.com/rocketjob/symmetric-encryption/blob/c81735ecb3686a7a8d68a4c7f98d71e27177f2ef/lib/symmetric_encryption/keystore.rb#L70

Because I'm using the AWS keystore, I don't have a key_encrypting_key entry or a private_rsa_key entry in my cipher configuration. That said, I don't have any trouble calling SymmetricEncryption::KeyStore::Aws.generate_data_keys directly.

[hovissimo]

To test my theory I added key_encrypting_key: HAAAX to my config yaml, and all of a sudden I started getting somewhere. Worth noting is that even with this hack it still failed to discover the correct key path. It did not read key path from the cli arguments, and it did not read the key path from the existing key_files in the cipher config. I was able to fool it by adding another hacky key to the yaml: key_filename: "/usr/local/sites/HiringThing/releases/current/foo.key". At this point it ALMOST works, but it also ignores the regions in the cipher config and ignores the regions in the cli args. It defaults to all four of us-east-1, us-east-2, us-west-1, us-west-2 - but at least it generates keys and updates the config file.

[reidmorrison]

We moved to docker containers and found the config file to be too cumbersome. So instead of managing and storing the data encryption key in KMS directly we store it in encrypted form in the AWS System Manager Parameter store.

The following code shows how to load the various keys using the Secret Config gem to access the AWS SSM Parameter Store:

# Need to set the cipher before the application is loaded and gem files initialized
# to avoid warning about config file not present.
SecretConfig.configure("symmetric_encryption") do |config|
  SymmetricEncryption.cipher =
    SymmetricEncryption::Cipher.new(
      key:         config.fetch("key", encoding: :base64),
      iv:          config.fetch("iv", encoding: :base64),
      version:     config.fetch("version", type: :integer),
      cipher_name: "aes-256-cbc"
    )
end

# Older encryption keys
%w[old older oldest].each do |path|
  next unless SecretConfig.key?("symmetric_encryption/#{path}/key")

  SecretConfig.configure("symmetric_encryption/#{path}") do |config|
    SymmetricEncryption.secondary_ciphers <<
      SymmetricEncryption::Cipher.new(
        key:         config.fetch("key", encoding: :base64),
        iv:          config.fetch("iv", encoding: :base64),
        version:     config.fetch("version", type: :integer),
        cipher_name: "aes-256-cbc"
      )
  end
end

Then the keys can be updated as needed via Secret Config.
To generate a new key to store in Secret Config (AWS SSM Parameter Store):

new_key = SymmetricEncryption::Key.new(cipher_name: 'aes-256-cbc')
h = {
  "key" => Base64.strict_encode64(new_key.key),
  "iv" => Base64.strict_encode64(new_key.iv),
  "version" => 1
}
puts h.to_yaml

Update the version as needed, and then new keys and key rotation is simply administered directly via Secret Config.

This approach removes the file config/symmetric-encryption.yml entirely.

We have found this approach much simpler, since it no longer requires any config file changes to handle key rotation etc.

[reidmorrison]

If someone is using KMS and wants the key rotation capability, we can pair on a PR to add that capability.