AWS EKS EBS volume attach issue in Kubernetes: Persistence Volume Attach issue fix EKS

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that simplifies deploying, managing, and scaling containerized applications using Kubernetes on AWS. In this documentation, we’ll address common issues related to persistent volume attachment and demonstrate how to associate an EBS volume encryption policy with an IAM role for use in EKS.

When attempting to attach a persistent volume to an EKS cluster, you may encounter the error message:

"waiting for a volume to be created, either by external provisioner 'ebs.csi.aws.com' or manually created by the system administrator."

while describing the PVC by running

kubectl describe PVC yourpvcname

This error typically occurs during volume provisioning, preventing successful attachment of the volume to the cluster.

Solution

To resolve the persistent volume attachment issue and attach an EBS volume encryption policy to an IAM role, follow these steps:

1. Associate IAM OIDC Provider

IAM OIDC Provider (Identity and Access Management OpenID Connect Provider) is a service provided by AWS that allows you to integrate your AWS Identity and Access Management (IAM) system with an OpenID Connect (OIDC)-compatible identity provider. It enables authentication and authorization of users and applications in AWS based on the standards of OpenID Connect.

By configuring an IAM OIDC Provider, you establish a trust relationship between AWS and your OIDC IdP. This trust allows AWS services like Amazon EKS, Amazon Cognito, etc., to use OIDC tokens issued by your IdP for authentication and authorization.

For example, Amazon EKS can use the IAM OIDC Provider to authenticate Kubernetes service accounts using OIDC tokens, simplifying access control for pods in your EKS cluster.

In summary, IAM OIDC Provider facilitates secure integration between your existing identity provider using OpenID Connect and AWS IAM, allowing you to use standardized identity protocols for authentication and authorization within the AWS ecosystem.

Before proceeding, ensure you have the necessary permissions. Run the following command to associate the IAM OIDC provider with your EKS cluster:

eksctl utils associate-iam-oidc-provider --region=your_region --cluster=your_cluster --approve

2. Create IAM Service Account

Next, create an IAM service account named “ebs-csi-controller-sa” using eksctl. This account will have the permissions required for EBS volume provisioning:

eksctl create iamserviceaccount \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster your_cluster \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve \
--role-only \
--role-name AmazonEKS_EBS_CSI_Driver

3. Update IAM Policy for EBS Volume Encryption(Optional)

Now, let’s update the IAM service account to include an EBS volume encryption policy:

• Update the IAM policy by adding the following JSON snippet for EBS volume encryption permissions:

the full policy can be viewed here https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/example-iam-policy.json

{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKeyWithoutPlaintext",
"kms:CreateGrant"
],
"Resource": "*"
}

Update the IAM service account using eksctl, adding the encryption policy ARN:

eksctl create iamserviceaccount \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster your_cluster\
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--attach-policy-arn arn:aws:iam::aws:policy/your-encryption-policy-name # Replace with the actual encryption policy ARN
--approve \
--role-only \
--role-name AmazonEKS_EBS_CSI_Driver

4. Create EKS Addon

An EKS addon is a pre-built extension or enhancement for Amazon Elastic Kubernetes Service (EKS) that provides additional functionalities, features, or components to extend the capabilities of your Kubernetes clusters running on EKS. Addons help streamline the deployment and management of specific functionalities, making it easier for users to integrate and utilize various AWS services within their EKS environment.

Examples of EKS Addons:

• Amazon VPC CNI (Container Network Interface): This addon enhances networking for pods by integrating with Amazon VPC, improving performance and security.
• Amazon EBS CSI (Container Storage Interface) Driver: Simplifies the management and use of Amazon Elastic Block Store (EBS) volumes in Kubernetes applications.
• Cluster Autoscaler: Automatically adjusts the size of the EKS cluster by managing the number of nodes based on resource requirements.

Create the EKS addon “aws-ebs-csi-driver” using eksctl and link it to the IAM service account you just created:

eksctl create addon --name aws-ebs-csi-driver --cluster your_cluster --service-account-role-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):role/AmazonEKS_EBS_CSI_Driver --force

By following these steps, you should resolve the persistent volume attachment issue and configure EBS volume encryption policy for your IAM service account, allowing successful encryption and decryption of EBS volumes within your AWS EKS cluster.

Frustrated by EBS volume attachment issues in Kubernetes? This guide provides a step-by-step solution to fix persistent volume attachment problems on AWS EKS. Don’t let storage issues slow you down - read now and get your cluster back on track!