Infrastructure as Code and Secrets Management – Lesson 4

### Lesson 4 Overview
In this lesson we’ll configure a non-HA vault server that was deployed into our ES2 app layer.
In the real world, the vault server should also be HA. This requires that we use a storage
back end like Hashicorp Consul which is out of the scope of this lab.
The Vault server will be the storage location for secrets required by our web servers to access the
database server. Before we can use the vault server, we must
* Connect to our bastion host that has access to the vault & mysql server
* Initialize the vault
* Unseal the vault (and hide the keys)
* Configure the standard secrets back end to store our database host name and port
* Configure the mysql secrets back end to dynamically create database users on MySQL
* Configure a policy that allows the web-role to access these secrets
* Configure aws-ec2 authentication so that our EC2 servers can authenticate to vault
### Lesson 4
  1. Start the ssh-agent to store our keys and store the created for this lab
  • * The ssh-agent is a daemon that runs your provisioning server. It allows you to register
  • a private ssh key, and it will manage providing those keys to additional servers as you connect
  • to them. In this case, we are about to connect to our bastion host, and we don’t want to mess
  • around with running ssh with the “-i” flag all the time.
  • * An interesting fact is that when ssh-agent starts, it outputs environmental variables so that
  • your shell can find the agent. running the command inside of eval $() sets the variable for us.
  • Cool huh?
  • eval $(ssh-agent)
  • ssh-add ~/iac_lab/terraform/ssh/id_rsa
  1. Build an file with environmental variables set for the bastion host to make life easier when we get there
  • * This is a script that pulls all of the outputs defined in terraform file and
  • drops them into a file called bastion.profile. It then copies that file up to the bastion
  • host so that it will be there for us when we log in.
  • * We’re doing this because setting up vault requires a bunch of commands that each need to know
  • something about our infrastructure. This is the easiest way to make the lab work for everyone.
  • In reality, you wouldn’t store things like your database password in a file on the provisioning
  • host or the bastion host
  • cd ~/iac_lab/vault/
  • ./
  • . ./bastion.profile
  1. Connect to the bastion host
  • * See, look ma, no “-i”. Thanks to the ssh agent. We are using the “-A” flag that forwards our
  • private key to the next host. This makes it super easy to connect to other hosts in the app
  • environment from the bastion host
  • ssh -A ec2-user@$BASTION_IP
  1. Set the shell prompt
  • * Set the shell prompt to identify this host so that we can stay oriented
  • echo ‘export PS1=”[\u@\h (BASTION) \W]\$ “‘ >> ~/.bashrc
  • . ~/.bashrc
  1. Set the env variables
  • * Ok, now we are working on the bastion host. It is highly advised that you write down the
  • last octet of the host name so that you can remember where you are working. It is really
  • easy to get lost when you are ssh hopping from host to host in the cloud. 🙂
  • * We copied the bastion.profile file up to the bastion host, now lets source it so that we
  • have all the env VARs set. This is critical for the next 10-15 steps
  • . ~/bastion.profile
  • * Check that it worked. Don’t proceed unless you see some env variables.
  • env | grep MYSQL
  • * EXAMPLE OUTPUT (to confirm that the environment variables are set)
  • ><br>
> MYSQL_PASSWORD=sumthinsecret<br>
  1. Install the vault client on the bastion host
  • * We are using the stock Amazon linux image for our bastion host. Need to install vault binary
  • wget
  • sudo unzip -j vault_* -d /usr/local/bin
  1. Set the VAULT_ADDR env variable and check connectivity
  • * Sure we could have done this in the bastion.profile file. But doing this by steps
  • lets you see that the `vault` command requires the VAULT_HOST env variable to be set
  • export VAULT_ADDR=http://${VAULT_IP}:8200
  • vault status
  • * EXAMPLE OUTPUT (for uninitialized Vault)
> [ec2-user@ip-172-31-35-147 ~]$ vault status<br>
> Error checking seal status: Error making API request.<br>
> URL: GET<br>
> Code: 400. Errors:<br>
> * server is not yet initialized<br>
  • * Specifically it **should say**
  • “server is not yet initialized”. This because we have a brand new vault server.
  1. Initialize the vault
  • * The keys generated by the vault unseal command are only created once, upon the init command,
  • and are **very sensitive**. In a real environment, each Unseal Key should be stored
  • by separate people in the organization, and NEVER stored anywhere near the vault
  • server, or those that access it. For this lab, we’ll just leave it in ./vaultkeys.txt.
  • In real life, get that file off the bastion host as soon as the vault is initialized.
  • vault operator init 2>&1 | egrep ‘^Unseal Key|Initial Root Token’ | tee ./vaultkeys.txt
  • chmod 600 ./vaultkeys.txt
  • * EXAMPLE OUTPUT (for vault initialization)
> [ec2-user@ip-172-31-35-147 ~]$ vault init 2>&1 | egrep ‘^Unseal Key|Initial Root Token’ | tee ./vaultkeys.txt<br>
> Unseal Key 1: MnM25Qj430w3HBrCJp9N0mIxqKxiHy/f0BWjCR1TWkJb<br>
> Unseal Key 2: a35k+PH2oq8Hkp > Unseal Key 3: tt6uGtW0FG1N0sJHmSqsjHgV2yJF3VTaabB3M7NLfnPw<br>
> Unseal Key 4: 5UkDIyxOfzO+KQIdlT3xUxwNXsC2qUW9eN/n8b5YYug8<br>
> Unseal Key 5: 3Wp+JLVx3aOl6/CftQjfF6ofT3EPwpCzOt35suw7D9wB<br>
> Initial Root Token: 51660e02-79be-2966-afd5-8b18609b3a3c<br>
  1. Unseal the vault
  • * At this stage, the vault is initialized, but not unsealed. We use the “Unseal Keys” provide
at init to unseal it
  • * The following commands send 3 of the Unseal Keys to the `vault unseal` command.
In this lab, we are doing it this way for speed.
In the real world, the users who each control one of the keys would log in
and issue the unseal command
  • egrep -m3 ‘^Unseal Key’ vaultkeys.txt | cut -f2- -d: | tr -d ‘ ‘ | while read key; do vault operator unseal ${key}; echo; done
  • * EXAMPLE OUTPUT (for vault unsealing)
> [ec2-user@ip-172-31-35-147 ~]$ egrep -m3 ‘^Unseal Key’ vaultkeys.txt | cut -f2- -d: | tr -d ‘ ‘ | while read key; do vault unseal ${key}; echo; done<br>
> Sealed: true<br>
> Key Shares: 5<br>
> Key Threshold: 3<br>
> Unseal Progress: 1<br>
> Unseal Nonce: 9d25838c-b9d9-f934-0741-182b08205819<br>
> <br>
> Sealed: true<br>
> Key Shares: 5<br>
> Key Threshold: 3<br>
> Unseal Progress: 2<br>
> Unseal Nonce: 9d25838c-b9d9-f934-0741-182b08205819<br>
> <br>
> Sealed: false<br>
> Key Shares: 5<br>
> Key Threshold: 3<br>
> Unseal Progress: 0<br>
> Unseal Nonce:<br>
  • * Verify that the vault is unsealed
  • vault status
  • * EXAMPLE OUTPUT (for vault status)
> [ec2-user@ip-172-31-35-147 ~]$ vault status<br>
> Seal Type: shamir<br>
> Sealed: false<br>
> Key Shares: 5<br>
> Key Threshold: 3<br>
> Unseal Progress: 0<br>
> Unseal Nonce: <br>
> Version: 0.9.1<br>
> Cluster Name: vault-cluster-d6bf2bc8<br>
> Cluster ID: 20850931-24f2-a16a-45e6-027df6797187<br>
> <br>
> High-Availability Enabled: false<br>
  1. Set the vault root user token (aka password) as an environmental variable
  • * Vault has several ways that users can authenticate to it. But the root user always
  • authenticates with the token that was provided at init time. To allow our bastion host
  • to access the vault server, set the VAULT_TOKEN env variable from the vaultkeys.txt file we saved
  • at init
  • export VAULT_TOKEN=`egrep ‘^Initial Root Token:’ ./vaultkeys.txt | awk -F’:’ ‘{print$2}’ | sed s/’ ‘//g`
  1. Write out first secret to the vault database
  • * In this step, we are going to store information for the application servers about
  • where to find the database. The secret/ storage path is simple key/value pairs and can
  • contain almost anything. Here, we’ll store some data in a path called secure/mysql. (mysql could be any name)
  • vault write secret/mysql host=${MYSQL_HOST} port=${MYSQL_PORT} database=${MYSQL_DB}
> [ec2-user@ip-172-31-35-147 ~]$ vault write secret/mysql host=${MYSQL_HOST} port=${MYSQL_PORT} database=${MYSQL_DB}<br>
> Success! Data written to: secret/mysql<br>
  • * Check that we can read the secret back. This is similar to the query that our applications will
  • run when they want to know where to find the database
  • vault read secret/mysql
> [ec2-user@ip-172-31-35-147 ~]$ vault read secret/mysql<br>
> Key Value<br>
> — —–<br>
> refresh_interval 768h0m0s<br>
> database myprojectdevdb<br>
> host<br>
> port 3306<br>
  • * Awesome, we just stored simple key/value pairs in the path secret/mysql. Now lets set up MySQL passwords
  1. Lets configure MySQL secrets back end that will issue dynamic passwords to our application
  • * First, install the mysql client so that we can connect to the database and create a user for vault
  • sudo yum install -y mysql
> Running transaction<br>
> Installing : 1:mariadb-5.5.56-2.amzn2.x86_64<br>
> Verifying : 1:mariadb-5.5.56-2.amzn2.x86_64<br>
> Installed:<br>
> mariadb.x86_64 1:5.5.56-2.amzn2<br>
> Complete!<br>
  • * Now we can create a mysql user & pass for vault service that has ALL privs (including GRANT) on our db.
  • For this lab, we are using the root password configured when we ran `terraform apply` as the vault password.
  • In the real world, these should be different passwords.
  • MYSQL=”mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS -Bsse”
  • * *Note*: if you happen to screw up the copy/paste above and want to delete the user and start over, run
  1. Configure the vault mysql secrets database
  • * Vault is configurable with a variety of storage, authentication and secrets databases
  • Each has to be “mounted” in order to use them. Let’s mount the “mysql” secrets database below
  • so that we can have it create usernames & passwords on the aurora database for us.
  • vault secrets enable mysql
> [ec2-user@ip-172-31-35-147 ~]$ vault mount mysql<br>
> Successfully mounted ‘mysql’ at ‘mysql’!<br>
  • * We have to tell the vault mysql module how to find and log into our mysql database, so that it can create
  • the users and passwords.
  • vault write mysql/config/connection \
  • connection_url=”vault:${MYSQL_VAULT_PASS}@tcp(${MYSQL_HOST}:${MYSQL_PORT})/” \
  • allowed_roles=”readwrite”
> [ec2-user@ip-172-31-35-147 ~]$ vault write mysql/config/connection \<br>
> connection_url=”vault:${MYSQL_VAULT_PASS}@tcp(${MYSQL_HOST}:${MYSQL_PORT})/” \
> allowed_roles=”readwrite”<br>
> The following warnings were returned from the Vault server:<br>
> * Read access to this endpoint should be controlled via ACLs as it will return the connection URL as it is, including passwords, if any.<br>
  • * And when vault’s mysql module creates a user for our web servers, we want the connection to have
  • a short lease (1 hour) in case the user/password is ever comprimised.
  • vault write mysql/config/lease lease=1h lease_max=12h
> [ec2-user@ip-172-31-35-147 ~]$ vault write mysql/config/lease lease=1h lease_max=12h<br>
> Success! Data written to: mysql/config/lease<br>
  • * Finally, we want to create role called “readwrite” that grants our users ALL privs on our database
  • vault write mysql/roles/readwrite \
  • sql=”CREATE USER ‘{{name}}’@’%’ IDENTIFIED BY ‘{{password}}’;GRANT ALL ON ${MYSQL_DB}.* TO ‘{{name}}’@’%’;”
> [ec2-user@ip-172-31-35-147 ~]$ vault write mysql/roles/readwrite \<br>
> sql=”CREATE USER ‘{{name}}’@’%’ IDENTIFIED BY ‘{{password}}’;GRANT ALL ON ${MYSQL_DB}.* TO ‘{{name}}’@’%’;”<br>
> Success! Data written to: mysql/roles/readwrite<br>
  1. Enough already, lets test it out as root
  • * use the vault CLI and root user to read some mysql credentials
  • vault read mysql/creds/readwrite
> [ec2-user@ip-172-31-35-147 ~]$ vault read mysql/creds/readwrite<br>
> Key Value<br>
> — —–<br>
> lease_id mysql/creds/readwrite/5195839e-7274-550d-c4d8-01e58840174c<br>
> lease_duration 1h0m0s<br>
> lease_renewable true<br>
> password 1cd3fa9f-0cd8-cb68-e415-b0a8e80a5533<br>
> username read-root-a9f162<br>
  • * Bingo! Vault created a temporary user for us with a 1 hour lease. If you feel froggy, prove that
  • you can log into mysql with that user. It would look something like
  • `mysql -u read-root-a9f162 -p’1cd3fa9f-0cd8-cb68-e415-b0a8e80a5533 -h myprojectdevdb`
  1. Create a policy for web access to secrets
  • * Now we will create & write a policy call web-policy that allows a user to access the secrets we just configured
  • cat <<EOF > ./web-policy.hcl
  • path “sys/*” {
  • policy = “deny”
  • }
  • path “secret/mysql*” {
  • capabilities = [“read”]
  • }
  • path “mysql/creds/readwrite” {
  • capabilities = [“read”]
  • }
  • EOF
  • vault policy-write web-policy ./web-policy.hcl
  1. Enable the AWS EC2 Auth module
  • * Enable Vault auth module so that we can configure vault to authenticate and ec2 instance
  • vault auth enable aws-ec2
  • * Configure aws-ec2 auth module allow the web-role to authenticate if they coming from a legit
  • EC2 instance, and have our IAM policy (myproject-dev-webinst-role) attached. Assign them to the
  • policy web-policy (that allowed access to our secrets)
  • vault write auth/aws-ec2/role/web-role bound_iam_instance_profile_arn=${WEB_PROFILE_ARN} policies=web-policy
  1. Test a web server
  • * Go to ALB URL and check the vault and database tabs
To get the load balancer’s DNS name, from the provisioning host run the following
  • cd ~/iac_lab/terraform
  • terraform output LOADBALANCER_DNS
  • * Or, from the AWS console naviget to the following: AWS Console > EC2 > Load Balancers > select your ALB > Public DNS
  • * Wait, a minute, they still cant authenticate… The reason is that the ultra simple way we
built this application is that the web server tries to login to vault on build and stores the
credentials in apache ENV vars. Since vault was down when these servers were built, they will
never be able to connect.
  • * To fix this, just terminate the *-webinst EC2 severs and let the autoscaling group launch
some new ones. The new servers should be able to authenticate with vault and database
connectivity should work. From the Provisioning host, you can run the following command
  • aws ec2 describe-instances –region=us-east-2 –filters “Name=tag:Name,Values=’iaclab-dev-webinst'” –query Reservations[*].Instances[*].[InstanceId] –output text | xargs -i aws ec2 terminate-instances –region=us-east-2 –instance-ids {}
  1. Take a screen shot!!!
  • * **WAIT!** If you are completing this lab as part of a Cloud Shift
Strategies facilitated lab, take a screen shot of your running vault
page with login Succeeded and send to
for a chance to receive an AWS voucher
## Lesson 4 Summary
In this lesson, we completed the following tasks:
  1. Started the ssh-agent on the provisioning server and added the ssh keys to connect to bastion host
  2. Pulled output variables from terraform and put into a profile
  3. Connected to the bastion host
  4. Set up temporary environmental variables on bastion host
  5. Installed the vault client on the bastion host
  6. Configured the $VAULT_ADDR enviornmental variable on the bastion host and tested access to vault
  7. Initialized the vault
  8. Unsealed the vault and checked the status of vault unseal
  9. Set the VAULT_TOKEN variable so that we can gain root access to the vault
  10. Wrote a simple secret to secret/mysql with database host name & port number
  11. Installed mysql client & created a vault user that can create more users in Aurora / MySQL
  12. Configured the vault mysql secrets database to create users for us
  13. Tested the temporary credentials from root user
  14. Created a policy for the web servers to access these secretes
  15. Enabled the aws-ec2 auth modules that allow our web servers to authenticate w/ vault
  16. Tested that the web servers can access the secrets and the database
  17. Captured a screen shot for a chance to win $50
In Lesson 5 we’ll clean up