AtooltouseAWSIAMcredentialstoauthenticatetoaKubernetescluster.TheinitialworkonthistoolwasdrivenbyHeptio.TheprojectreceivescontributionsfrommultiplecommunityengineersandiscurrentlymaintainedbyHeptioandAmazonEKSOSSEngineers.
WhydoIwantthis?IfyouareanadministratorrunningaKubernetesclusteronAWS,youalreadyneedtomanageAWSIAMcredentialstoprovisionandupdatethecluster.ByusingAWSIAMAuthenticatorforKubernetes,youavoidhavingtomanageaseparatecredentialforKubernetesaccess.AWSIAMalsoprovidesanumberofnicepropertiessuchasanoutofbandaudittrail(viaCloudTrail)and2FA/MFAenforcement.
IfyouarebuildingaKubernetesinstalleronAWS,AWSIAMAuthenticatorforKubernetescansimplifyyourbootstrapprocess.Youwon'tneedtosomehowsmuggleyourinitialadmincredentialsecurelyoutofyournewlyinstalledcluster.Instead,youcancreateadedicatedKubernetesAdminroleatclusterprovisioningtimeandsetupAuthenticatortoallowclusteradministratorlogins.
HowdoIuseit?AssumingyouhaveaclusterrunninginAWSandyouwanttoaddAWSIAMAuthenticatorforKubernetessupport,youneedto:
CreateanIAMroleyou'llusetoidentifyusers.RuntheAuthenticatorserverasaDaemonSet.ConfigureyourAPIservertotalktoAuthenticator.SetupkubectltouseAuthenticatortokens.1.CreateanIAMroleFirst,youmustcreateoneormoreIAMrolesthatwillbemappedtousers/groupsinsideyourKubernetescluster.TheeasiestwaytodothisistologintotheAWSConsole:
Choosethe"Roleforcross-accountaccess"/"ProvideaccessbetweenAWSaccountsyouown"option.PasteinyourAWSaccountIDnumber(availableinthetoprightintheconsole).Yourroledoesnotneedanyadditionalpoliciesattached.ThiswillcreateanIAMrolewithnopermissionsthatcanbeassumedbyauthorizedusers/rolesinyouraccount.NotetheAmazonResourceName(ARN)ofyourrole,whichyouwillneedbelow.
YoucanalsodothisinasinglestepusingtheAWSCLIinsteadoftheAWSConsole:
#getyouraccountIDACCOUNT_ID=$(awsstsget-caller-identity--outputtext--query'Account')#definearoletrustpolicythatopenstheroletousersinyouraccount(limitedbyIAMpolicy)POLICY=$(echo-n'{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::';echo-n"$ACCOUNT_ID";echo-n':root"},"Action":"sts:AssumeRole","Condition":{}}]}')#createarolenamedKubernetesAdmin(willprintthenewrole'sARN)awsiamcreate-role\--role-nameKubernetesAdmin\--description"Kubernetesadministratorrole(forAWSIAMAuthenticatorforKubernetes)."\--assume-role-policy-document"$POLICY"\--outputtext\--query'Role.Arn'Youcanalsoskipthisstepanduse:
Anexistingrole(suchasacross-accountaccessrole).AnIAMuser(seemapUsersbelow).AnEC2instanceorafederatedrole(seemapRolesbelow).2.RuntheserverTheserverismeanttorunoneachofyourmasternodesasaDaemonSetwithhostnetworkingsoitcanexposealocalhostport.
ForasampleConfigMapandDaemonSetconfiguration,seedeploy/example.yaml.
(Optional)Pre-generateacertificate,key,andkubeconfigIfyou'rebuildinganautomatedinstaller,youcanalsopre-generatethecertificate,key,andwebhookkubeconfigfileseasilyusingaws-iam-authenticatorinit.Thiscommandwillgeneratefilesandplacethemintheconfiguredoutputdirectories.
YoucanrunthisoneachmasternodepriortostartingtheAPIserver.Youcouldalsogeneratethembeforeprovisioningmasternodesandinstallthemintheappropriatehostpaths.
Ifyoudonotpre-generatefiles,aws-iam-authenticatorserverwillgeneratethemondemand.ThisworksbutrequiresthatyourestartyourKubernetesAPIserverafterinstallation.
3.ConfigureyourAPIservertotalktotheserverTheKubernetesAPIintegrateswithAWSIAMAuthenticatorforKubernetesusingatokenauthenticationwebhook.Whenyourunaws-iam-authenticatorserver,itwillgenerateawebhookconfigurationfileandsaveitontothehostfilesystem.You'llneedtoaddasingleadditionalflagtoyourAPIserverconfiguration:
--authentication-token-webhook-config-file=/etc/kubernetes/aws-iam-authenticator/kubeconfig.yamlOnmanyclusters,theAPIserverrunsasastaticpod.Youcanaddtheflagto/etc/kubernetes/manifests/kube-apiserver.yaml.Makesurethehostdirectory/etc/kubernetes/aws-iam-authenticator/ismountedintoyourAPIserverpod.Youmayalsoneedtorestartthekubeletdaemononyourmasternodetopickuptheupdatedstaticpoddefinition:
systemctlrestartkubelet.service4.CreateIAMrole/usertokubernetesuser/groupmappingsThedefaultbehavioroftheserveristosourcemappingsexclusivelyfromthemapUsersandmapRolesfieldsofitsconfigurationfile.SeeFullConfigurationFormatbelowfordetails.
Usingthe--backend-modeflag,youcanconfiguretheservertosourcemappingsfromtwoadditionalbackends:anEKS-styleConfigMap(--backend-mode=EKSConfigMap)orIAMIdentityMappingcustomresources(--backend-mode=CRD).Thedefaultbackend,theserverconfigurationfilethat'smountedbytheserverpod,correspondsto--backend-mode=MountedFile.
Youcanpassacomma-separatedlistofthesebackendstohavetheserversearchtheminorder.Forexample,with--backend-mode=EKSConfigMap,MountedFile,theserverwillsearchtheEKS-styleConfigMapformappingsthen,ifitdoesn'tfindamappingforthegivenIAMrole/user,theserverconfigurationfile.IfamappingforthesameIAMrole/userexistsinmultiplebackends,theserverwillusethemappinginthebackendthatoccursfirstinthecomma-separatedlist.Inthisexample,ifamappingisfoundintheEKSConfigMapthenitwillbeusedwhetherornotaduplicateorconflictingmappingexistsintheserverconfigurationfile.
Notethatwhensettingasinglebackend,theserverwillonlysourcefromthatoneandignoretheotherseveniftheyexist.Forexample,with--backend-mode=CRD,theserverwillonlysourcefromIAMIdentityMappingsandignorethemountedfileandEKSConfigMap.
MountedFileThisisthedefaultbackendofmappingsandsufficientformostusers.SeeFullConfigurationFormatbelowfordetails.
CRD(alpha)ThisbackendmodelseachIAMmappingasanIAMIdentityMappingKubernetesCustomResource.ThisapproachenablesyoutomaintainmappingsinaKubernetes-nativewayusingkubectlortheAPI.Plus,syntaxerrors(likemisalignedYAML)canbemoreeasilycaughtandwon'taffectallmappings.
TosetupanIAMIdentityMappingCRDyou'llfirstneedtoapplytheCRDmanifest:
kubectlapply-fdeploy/iamidentitymapping.yamlWiththeCRDsdeployedyoucanthencreateCustomResourceswhichmodelyourIAMIdentities.See./deploy/example-iamidentitymapping.yaml:
---apiVersion:iamauthenticator.k8s.aws/v1alpha1kind:IAMIdentityMappingmetadata:name:kubernetes-adminspec:#ArnoftheUserorRoletobeallowedtoauthenticatearn:arn:aws:iam::XXXXXXXXXXXX:user/KubernetesAdmin#UsernamethatKuberneteswillseetheuseras,thisisusefulforsetting#upallowedspecificpermissionsfordifferentusersusername:kubernetes-admin#Groupstobeattachedtoyourusers/roles.Forexample`system:masters`to#createclusteradmin,or`system:nodes`,`system:bootstrappers`fornodesto#accesstheAPIserver.groups:-system:mastersEKSConfigMapTheEKS-stylekube-system/aws-authConfigMapservesasthebackend.TheConfigMapisexpectedtobeinexactlythesameformatasinEKSclusters:https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html.Thisisusefulifyou'remigratingfrom/toEKSandwanttokeepyourmappings,orarerunningEKSinadditiontosomeotherAWScluster(s)andwanttohavethesamemappingsineach.
5.SetupkubectltouseauthenticationtokensprovidedbyAWSIAMAuthenticatorforKubernetesThisrequiresa1.10+kubectlbinarytowork.IfyoureceivePleaseenterUsername:whentryingtousekubectlyouneedtoupdatetothelatestkubectl
Finally,oncetheserverissetupyou'llwanttoauthenticate.Youwillstillneedakubeconfigthathasthepublicdataaboutyourcluster(clusterCAcertificate,endpointaddress).Theuserssectionofyourconfiguration,however,shouldincludeanexecsection(refertothev1.10docs)::
#[...]users:-name:kubernetes-adminuser:exec:apiVersion:client.authentication.k8s.io/v1beta1command:aws-iam-authenticatorargs:-"token"-"-i"-"REPLACE_ME_WITH_YOUR_CLUSTER_ID"-"-r"-"REPLACE_ME_WITH_YOUR_ROLE_ARN"#noclientcertificate/keyneededhere!ThismeansthekubeconfigisentirelypublicdataandcanbesharedacrossallAuthenticatorusers.ItmaymakesensetouploadittoatrustedpubliclocationsuchasAWSS3.
Makesureyouhavetheaws-iam-authenticatorbinaryinstalled.Youcaninstallitwithgoget-u-vsigs.k8s.io/aws-iam-authenticator/cmd/aws-iam-authenticator.
Toauthenticate,runkubectl--kubeconfig/path/to/kubeconfig"[...].kubectlwillexectheaws-iam-authenticatorbinarywiththesuppliedparamsinyourkubeconfigwhichwillgenerateatokenandpassittotheapiserver.Thetokenisvalidfor15minutes(theshortestvalueAWSpermits)andcanbereusedmultipletimes.
Youcanalsospecifysessionnamewhengeneratingthetokenbyincluding--session-nameor-sparameter.Thisparametercannotbeusedalongwith--forward-session-name.
Youcanalsoomit-rROLE_ARNtosignthetokenwithyourexistingcredentialswithoutassumingadedicatedrole.ThisisusefulifyouwanttoauthenticateasanIAMuserdirectlyorifyouwanttoauthenticateusinganEC2instanceroleorafederatedrole.
KopsUsageClustersmanagedbyKopscanbeconfiguredtouseAuthenticator.ForusageinstructionsseetheKopsdocumentation.
Howdoesitwork?ItworksusingtheAWSsts:GetCallerIdentityAPIendpoint.ThisendpointreturnsinformationaboutwhateverAWSIAMcredentialsyouusetoconnecttoit.
Clientside(aws-iam-authenticatortoken)WeusethisAPIinasomewhatunusualwaybyhavingtheAuthenticatorclientgenerateandpre-signarequesttotheendpoint.WeserializethatrequestintoatokenthatcanpassthroughtheKubernetesauthenticationsystem.
Serverside(aws-iam-authenticatorserver)ThetokenispassedthroughtheKubernetesAPIserverandintotheAuthenticatorserver's/authenticateendpointviaawebhookconfiguration.TheAuthenticatorservervalidatesalltheparametersofthepre-signedrequesttomakesurenothinglooksfunny.Itthensubmitstherequesttotherealhttps://sts.amazonaws.comserver,whichvalidatestheclient'sHMACsignatureandreturnsinformationabouttheuser.NowthattheserverknowstheAWSidentityoftheclient,ittranslatesthisidentityintoaKubernetesuserandgroupsviaasimplestaticmapping.
ThismechanismisborrowedwithafewchangesfromVault.
WhatisaclusterID?TheAuthenticatorclusterIDisaunique-per-clusteridentifierthatpreventscertainreplayattacks.Specifically,itpreventsoneAuthenticatorserver(e.g.,inadevenvironment)fromusingaclient'stokentoauthenticatetoanotherAuthenticatorserverinanothercluster.
TheclusterIDdoesneedtobeuniqueper-cluster,butitdoesn'tneedtobeasecret.Somegoodchoicesare:
ArandomIDsuchasfromopensslrand16-hexThedomainnameofyourKubernetesAPIserverTheVaultdocumentationalsoexplainsthisattack(seeX-Vault-AWS-IAM-Server-ID).
SpecifyingCredentials&UsingAWSProfilesCredentialscanbespecifiedforusewithaws-iam-authenticatorviaanyofthemethodsavailabletotheAWSSDKforGo.ThisincludesspecifyingAWScredentialswithenviromentvariablesorbyutilizingacredentialsfile.
AWSnamedprofilesaresupportedbyaws-iam-authenticatorviatheAWS_PROFILEenvironmentvariable.Forexample,toauthenticatewithcredentialsspecifiedinthedevprofiletheAWS_PROFILEcanbeexportedorspecifiedexplictly(e.g.,AWS_PROFILE=devkubectlgetall).IfnoAWS_PROFILEisset,thedefaultprofileisused.
TheAWS_PROFILEcanalsobespecifieddirectlyinthekubeconfigfileaspartoftheexecflow.Forexample,tospecifythatcredentialsfromthedevnamedprofileshouldalwaysbeusedbyaws-iam-authenticator,yourkubeconfigwouldincludeanenvkeythatssetstheprofile:
apiVersion:v1clusters:-cluster:server:${server}certificate-authority-data:${cert}name:kubernetescontexts:-context:cluster:kubernetesuser:awsname:awscurrent-context:awskind:Configpreferences:{}users:-name:awsuser:exec:apiVersion:client.authentication.k8s.io/v1beta1command:aws-iam-authenticatorenv:-name:"AWS_PROFILE"value:"dev"args:-"token"-"-i"-"mycluster"Thismethodallowstheappropriateprofiletobeusedimplicitly.Notethatanyenvironmentvariablessetaspartoftheexecflowwilltakeprecedenceoverwhat'salreadysetinyourenvironment.
Noteforfederatedusers:FederatedAWSusersoftenwillhavea"meaningful"attributemappedontotheirassumedrole,suchasanemailaddress,throughtheaccount'sAWSconfiguration.Theseassumedsessionshaveafewparts,theroleidandcaller-specified-role-name.Bydefault,whenafederateduserusesthe--roleoptionofaws-iam-authenticatortoassumeanewrolethecaller-specified-role-namewillbeconvertedtoarandomtokenandtheroleidcarriesthroughtothenewlyassumedrole.
Usingaws-iam-authenticatortoken...--forward-session-namewillmaptheoriginalcaller-specified-role-nameattributeontothenewSTSassumedsession.Thiscanbehelpfulforquicklyattemptingtoassociate"whoperformedactionXontheK8cluster".
Pleasenote,thisshouldnotbeconsidereddefinitiveandneedstobecrossreferencedviatheroleid(whichremainsconsistent)withCloudTraillogsasausercouldpotentiallychangethisontheclientside.
APIAuthorizationfromOutsideaClusterItispossibletomakerequeststotheKubernetesAPIfromaclientthatisoutsidethecluster,bethatusingthebareKubernetesRESTAPIorfromoneofthelanguagespecificKubernetesclients(e.g.,Python).Inordertodoso,youmustcreateabearertokenthatisincludedwiththerequesttotheAPI.Thisbearertokenrequiresyouappendthestringk8s-aws-v1.withabase64encodedstringofasignedHTTPrequesttotheSTSGetCallerIdentityQueryAPI.ThisisthensentitintheAuthorizationheaderoftherequest.SomethingtonotethoughisthattheIAMAuthenticatorexplicitlyomitsbase64paddingtoavoidany=charactersthusguaranteeingastringsafetouseinURLs.BelowisanexampleinPythononhowthistokenwouldbeconstructed:
importbase64importboto3importrefrombotocore.signersimportRequestSignerdefget_bearer_token(cluster_id,region):STS_TOKEN_EXPIRES_IN=60session=boto3.session.Session()client=session.client('sts',region_name=region)service_id=client.meta.service_model.service_idsigner=RequestSigner(service_id,region,'sts','v4',session.get_credentials(),session.events)params={'method':'GET','url':'https://sts.{}.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15'.format(region),'body':{},'headers':{'x-k8s-aws-id':cluster_id},'context':{}}signed_url=signer.generate_presigned_url(params,region_name=region,expires_in=STS_TOKEN_EXPIRES_IN,operation_name='')base64_url=base64.urlsafe_b64encode(signed_url.encode('utf-8')).decode('utf-8')#removeanybase64encodingpadding:return'k8s-aws-v1.'+re.sub(r'=*','',base64_url)#IfmakingaHTTPrequestyouwouldcreatetheauthorizationheadersasfollows:headers={'Authorization':'Bearer'+get_bearer_token('my_cluster','us-east-1')}TroubleshootingIfyourclientfailswithanerrorlikecouldnotgettoken:AccessDenied[...],youcantryassumingtherolewiththeAWSCLIdirectly:
#AWSCLIversionof`aws-iam-authenticatortoken-rarn:aws:iam::ACCOUNT:role/ROLE`:$awsstsassume-role--role-arnarn:aws:iam::ACCOUNT:role/ROLE--role-session-nametestIfthatfails,thereareafewpossibleproblemstocheckfor:
MakesureyourbaseAWScredentialsareavailableinyourshell(awsstsget-caller-identitycanhelptroubleshootthis).
Makesurethetargetroleallowsyoursourceaccountaccess(intheroletrustpolicy).
Makesureyoursourceprincipal(user/role/group)hasanIAMpolicythatallowssts:AssumeRoleforthetargetrole.
Makesureyoudon'thaveanyexplicitdenypoliciesattachedtoyouruser,group,orinAWSOrganizationsthatwouldpreventthests:AssumeRole.
Trysimulatingthests:AssumeRolecallinthePolicySimulator.
FullConfigurationFormatTheclientandserverhavethesameconfigurationformat.Theycansharethesameexactconfigurationfile,sincetherearenosecretsstoredintheconfiguration.
#aunique-per-clusteridentifiertopreventreplayattacks(seeabove)clusterID:my-dev-cluster.example.com#defaultIAMroletoassumefor`aws-iam-authenticatortoken`defaultRole:arn:aws:iam::000000000000:role/KubernetesAdmin#serverlistenerconfigurationserver:#localhostportwheretheserverwillservethe/authenticateendpointport:21362#(default)#statedirectoryforgeneratedTLScertificateandprivatekeysstateDir:/var/aws-iam-authenticator#(default)#output`path`whereageneratedwebhookkubeconfigwillbestored.generateKubeconfig:/etc/kubernetes/aws-iam-authenticator.kubeconfig#(default)#roletoassumebeforequeryingEC2APIinordertodiscovermetadatalikeEC2privateDNSNameec2DescribeInstancesRoleARN:arn:aws:iam::000000000000:role/DescribeInstancesRole#AWSAccountIDstoscrubfromserverlogs.(Defaultstoemptylist)scrubbedAccounts:-"111122223333"-"222233334444"#eachmapRolesentrymapsanIAMroletoausernameandsetofgroups#Eachusernameandgroupcanoptionallycontaintemplateparameters:#1)"{{AccountID}}"isthe12digitAWSID.#2)"{{SessionName}}"istherolesessionname,with`@`characters#transliteratedto`-`characters.#3)"{{SessionNameRaw}}"istherolesessionname,withoutcharacter#transliteration(availableinversion>=0.5).mapRoles:#staticallymaparn:aws:iam::000000000000:role/KubernetesAdmintoclusteradmin-roleARN:arn:aws:iam::000000000000:role/KubernetesAdminusername:kubernetes-admingroups:-system:masters#mapEC2instancesinmy"KubernetesNode"roletouserslike#"aws:000000000000:instance:i-0123456789abcdef0".Onlyusethisifyou#trustthattherolecanonlybeassumedbyEC2instances.IfanIAMuser#canassumethisroledirectly(withsts:AssumeRole)theycancontrol#SessionName.-roleARN:arn:aws:iam::000000000000:role/KubernetesNodeusername:aws:{{AccountID}}:instance:{{SessionName}}groups:-system:bootstrappers-aws:instances#mapnodesthatshouldconformtotheusername"system:node:<private-DNS>".This#requirestheauthenticatortoquerytheEC2APIinordertodiscovertheprivate#DNSoftheEC2instanceoriginatingtheauthenticationrequest.Optionally,you#mayspecifyarolethatshouldbeassumedbeforequeryingtheEC2APIwiththe#key"server.ec2DescribeInstancesRoleARN"(seeabove).-roleARN:arn:aws:iam::000000000000:role/KubernetesNodeusername:system:node:{{EC2PrivateDNSName}}groups:-system:nodes-system:bootstrappers#mapfederatedusersinmy"KubernetesAdmin"roletouserslike#"admin:alice-example.com".TheSessionNameisanarbitraryrolename#likeane-mailaddresspassedbytheidentityprovider.Notethatifthis#roleisassumeddirectlybyanIAMUser(notviafederation),theuser#cancontroltheSessionName.-roleARN:arn:aws:iam::000000000000:role/KubernetesAdminusername:admin:{{SessionName}}groups:-system:masters#mapfederatedusersinmy"KubernetesOtherAdmin"roletouserslike#"alice-example.com".TheSessionNameisanarbitraryrolename#likeane-mailaddresspassedbytheidentityprovider.Notethatifthis#roleisassumeddirectlybyanIAMUser(notviafederation),theuser#cancontroltheSessionName.Notethatthe"{{SessionName}}"macrois#quotedtoensureitisproperlyparsedasastring.-roleARN:arn:aws:iam::000000000000:role/KubernetesOtherAdminusername:"{{SessionName}}"groups:-system:masters#IfunalterableidentificationofanIAMUserisdesirable,youcanmapagainst#AccessKeyID.-roleARN:arn:aws:iam::000000000000:role/KubernetesOtherAdminusername:"admin:{{AccessKeyID}}"groups:-system:masters#eachmapUsersentrymapsanIAMroletoastaticusernameandsetofgroupsmapUsers:#mapuserIAMuserAlicein000000000000touser"alice"ingroup"system:masters"-userARN:arn:aws:iam::000000000000:user/Aliceusername:alicegroups:-system:masters#automaticallymapIAMARNfromtheseaccountstousername.#NOTE:Alwaysusequotestoavoidtheaccountnumbersbeingrecognizedasnumbers#insteadofstringsbytheyamlparser.mapAccounts:-"012345678901"-"456789012345"#sourcemappingsfromthisfile(mapUsers,mapRoles,&mapAccounts)backendMode:-MountedFileCommunity,discussion,contribution,andsupportLearnhowtoengagewiththeKubernetescommunityonthecommunitypage.
Youcanreachthemaintainersofthisprojectat:
SlackMailingListCodeofconductParticipationintheKubernetescommunityisgovernedbytheKubernetesCodeofConduct.
评论