5 Things I Wish I’d Known Before Setting Up Heptio-Authenticator

5 Things I Wish I’d Known Before Setting Up Heptio-Authenticator

If you’re in Operations then I’m sure you’ve already felt the pain of having to manage permissions on dozens of third party applications. When we decided at Lola to move our application to Kubernetes we dreaded having to implement a new set of Role Based Access Controls to manage for all of our engineers. Since we're PCI DSS compliant, we also didn't want ANOTHER set of access controls to be audited.

We stumbled upon heptio-authenticator, which allows us to assign IAM users access to assume specific IAM Roles. The users assume the Role on the command line and generate a STS token. Heptio-authenticator will determine if that Role has access to the Kubernetes cluster.

I spent a few days banging my head against my keyboard trying to figure out why the heptio-authenticator instructions weren’t working for me. Everyone was recommending heptio-authenticator but no one seemed to be hitting the same hurdles we were. Our Kubernetes setup couldn’t be that different from everyone else's, right?

We finally got heptio-authenticator working and I wanted to share what we learned during this process so other people don’t suffer through vague and useless error messages.Here are a few key tips that I wish we had known as we went through the heptio-authenticator instructions:

1. Configure cross account IAM Roles with no permissions.

I found that updating the Trust Policy to allow specific users, groups, or everyone access to assume the role made things easier for us. For our Dev cluster we allow any of the engineers in our AWS account to access the cluster:

Screen Shot 2018-04-18 at 12.07.51 PM

For our other environments we restrict it by IAM Group:

Screen Shot 2018-04-18 at 12.08.45 PM

2. You don’t have to use the default Kubernetes defined group.

When configuring the server, you have to map Roles or IAM users to Kubernetes groups. But you can put in an arbitrary group name in the heptio-authenticator configuration and then map it to a RoleBinding in your cluster.

In this example, we mapped the IAM Role "k8s-dev-access" to the Kubernetes group "lolagroups"

Screen Shot 2018-04-18 at 12.09.59 PM

We then created a RoleBinding that restricts access to the namespace “dev” with “edit” permissions for that namespace. If you attach ClusterRole permissions to a RoleBinding, then you can restrict access to a namespace. You can't do that if you create a ClusterRoleBinding.

Screen Shot 2018-04-18 at 12.10.37 PM


3. Configure a kops hook to deploy heptio-authenticator.

(If you’re not using kops to manage your cluster then you can skip this step.)

For kops configuration, you need to generate the certificates that heptio-authenticator and the Kubernetes server-api will use to trust each other. Once you’ve generated the certs, store them in the S3 bucket you’re using to store the kops state. You can then configure a hook in your kops config to pull down the certs into a directory that is shared by both heptio-authenticator and the server-api.

Screen Shot 2018-04-18 at 12.11.49 PM

Confirm that your heptio-authenticator config is mounting the shared directory:

Screen Shot 2018-04-18 at 12.12.48 PM

Then you’re ready to start up the heptio-authenticator service!

4. Configure AWS CLI before you start. 

You need to have the AWS CLI configured on your machine with your IAM access keys set up. Now that you have the heptio-authenticator service running, you’re ready to set up the client! I like to set up contexts for each environment, so in this case my Dev context will be “katie”

Screen Shot 2018-04-18 at 12.13.40 PM

I use the heptio-authenticator client to generate a token using the IAM Role I created and have it saved to my context.

Screen Shot 2018-04-18 at 12.11.49 PM

I then switch my context to "katie".

Screen Shot 2018-04-18 at 12.14.51 PM

And now I can run commands against our Dev namespace but not our staging or production environments!

5. It’s impossible to troubleshoot error messages.

I mentioned vague and useless error messages…

Screen Shot 2018-04-18 at 12.15.36 PM

This means your token expired and you need to rerun the heptio-authenticator client command to get a fresh AWS token. The token only lasts 15 minutes so definitely set up an alias because you may be running this frequently if you don’t have the Kubernetes Dashboard configured.

You can use heptio-authenticator to add any level of granularity you need by adding users to different IAM groups and adjusting various RoleBindings. During our next PCI audit we’ll be able to point to the IAM console and show which users have access to our Kubernetes clusters without having to dive too deep into Kubernetes permissioning.

If you have any questions about how we set up heptio-authenticator or have any suggestions for better ways to configure it, I’d love to hear from you! Leave a comment below, or send me an email at katie@lola.com.


Subscribe to our blog to get all of Lola.com's updates sent right to your inbox!

Subscribe now

About the Author: Katie Paugh
Katie lives and breathes DevOps. Constantly playing with the newest Hashi Corp tool or writing scripts to automate maintenance of a Kubernetes cluster. She's active in the Boston DevOps community and gives talks at AWS, DevOps, and Ansible MeetUps around Boston.