Migration tips/hints for IRSA

[jess-belliveau]

Hello! An odd issue - we love KIAM and as such, deployed it all over the place and have a few teams relying on it to work with AWS services.

Alas, we are also on the track of switching to IRSA - we are just at the inflection point and will likely kick off a project to figure out the migration path.

From a super quick cursory look, we weren't sure if there was a "zero downtime" migration method. We were wondering if the uswitch team (or others) had any helpful hints or processes they would be willing to share to help with our smooth transition away from KIAM?

While I'm here - huge thanks to the contributers of KIAM, its been great using as a tool and helped our teams consume AWS services easily for many years now.

[Joseph-Irving]

Hey, we recently finished switching everything over to IRSA and shutting down Kiam in our clusters.
You can use both at the same time and IRSA will take precedence due to the way the aws credential chain works, so our method was to leave kiam running, then on an app by app basis we would setup all the IRSA stuff for it, roll that out.
At this point the app still has the kiam annotations but it will be using the IRSA credentials instead.
You can then remove the Kiam annotation for the app and it should continue working with the IRSA creds and you have no downtime.
Once this is done for every app you can then delete Kiam!

[sushama-kothawale]

@Joseph-Irving The above migration tips helps us in our lower environments kiam to irsa migration. We have one doubt before going for prod migration:

1. As per kiam docs, iptables maintained on the node stores AWS creds and can be accessed by pods for temporary access, so after removing kiam annotation from pod, assume role will be done through serviceaccount or is there any intercepting from iptables?

cc: @rhysemmas

[maanti]

This new EKS feature might be useful for anyone doing a migration:
https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-eks-pod-identity/
https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html