VirtualKubeletisanopensourceKuberneteskubeletimplementationthatmasqueradesasakubeletforthepurposesofconnectingKubernetestootherAPIs.ThisallowsthenodestobebackedbyotherserviceslikeACI,AWSFargate,IoTEdge,TensileKubeetc.TheprimaryscenarioforVKisenablingtheextensionoftheKubernetesAPIintoserverlesscontainerplatformslikeACIandFargate,thoughweareopentoothers.However,itshouldbenotedthatVKisexplicitlynotintendedtobeanalternativetoKubernetesfederation.
VirtualKubeletfeaturesapluggablearchitectureanddirectuseofKubernetesprimitives,makingitmucheasiertobuildon.
WeinvitetheKubernetesecosystemtojoinusinempoweringdeveloperstobuilduponourbase.Joinourslackchannelnamed,virtual-kubelet,withintheKubernetesslackgroup.
Thebestdescriptionis"KubernetesAPIontop,programmableback."
TableofContentsHowItWorksUsageProvidersAdmiraltyMulti-ClusterSchedulerAlibabaCloudECIProviderAzureContainerInstancesProviderAzureBatchGPUProviderAWSFargateProviderElotlKipHashiCorpNomadLiqoOpenStackZunTensileKubeProviderAddingaNewProviderviatheProviderInterfaceTestingUnittestsEnd-to-endtestsKnownquirksandworkaroundsContributingHowItWorksThediagrambelowillustrateshowVirtual-Kubeletworks.
UsageVirtualKubeletisfocusedonprovidingalibrarythatyoucanconsumeinyourprojecttobuildacustomKubernetesnodeagent.
Seegodocforuptodateinstructionsonconsumingthisproject:https://godoc.org/github.com/virtual-kubelet/virtual-kubelet
Thereareimplementationsavailableforseveralproviders,seethosereposfordetailsonhowtodeploy.
CurrentFeaturescreate,deleteandupdatepodscontainerlogs,exec,andmetricsgetpod,podsandpodstatuscapacitynodeaddresses,nodecapacity,nodedaemonendpointsoperatingsystembringyourownvirtualnetworkProvidersThisprojectfeaturesapluggableproviderinterfacedeveloperscanimplementthatdefinestheactionsofatypicalkubelet.
Thisenableson-demandandnearlyinstantaneouscontainercompute,orchestratedbyKubernetes,withouthavingVMinfrastructuretomanageandwhilestillleveragingtheportableKubernetesAPI.
Eachprovidermayhaveitsownconfigurationfile,andrequiredenvironmentalvariables.
ProvidersmustprovidethefollowingfunctionalitytobeconsideredasupportedintegrationwithVirtualKubelet.
Providestheback-endplumbingnecessarytosupportthelifecyclemanagementofpods,containersandsupportingresourcesinthecontextofKubernetes.ConformstothecurrentAPIprovidedbyVirtualKubelet.DoesnothaveaccesstotheKubernetesAPIServerandhasawell-definedcallbackmechanismforgettingdatalikesecretsorconfigmaps.AdmiraltyMulti-ClusterSchedulerAdmiraltyMulti-ClusterSchedulermutatesannotatedpodsinto"proxypods"scheduledonavirtual-kubeletnodeandcreatescorresponding"delegatepods"inremoteclusters(actuallyrunningthecontainers).Afeedbackloopupdatesthestatusesandannotationsoftheproxypodstoreflectthestatusesandannotationsofthedelegatepods.YoucanfindmoredetailsintheAdmiraltyMulti-ClusterSchedulerdocumentation.
AlibabaCloudECIProviderAlibabaCloudECI(ElasticContainerInstance)isaservicethatallowyouruncontainerswithouthavingtomanageserversorclusters.
YoucanfindmoredetailsintheAlibabaCloudECIproviderdocumentation.
ConfigurationFileThealibabaECIproviderwillreadconfigurationfilespecifiedbythe--provider-configflag.
TheexampleconfigurefileisintheECIproviderrepository.
AzureContainerInstancesProviderTheAzureContainerInstancesProviderallowsyoutoutilizebothtypicalpodsonVMsandAzureContainerinstancessimultaneouslyinthesameKubernetescluster.
YoucanfinddetailedinstructionsonhowtosetitupandhowtotestitintheAzureContainerInstancesProviderdocumentation.
ConfigurationFileTheAzureconnectorcanuseaconfigurationfilespecifiedbythe--provider-configflag.TheconfigfileisinTOMLformat,andanexamplelivesinproviders/azure/example.toml.
AWSFargateProviderAWSFargateisatechnologythatallowsyoutoruncontainerswithouthavingtomanageserversorclusters.
TheAWSFargateproviderallowsyoutodeploypodstoAWSFargate.YourpodsonAWSFargatehaveaccesstoVPCnetworkingwithdedicatedENIsinyoursubnets,publicIPaddressestoconnecttotheinternet,privateIPaddressestoconnecttoyourKubernetescluster,securitygroups,IAMroles,CloudWatchLogsandmanyotherAWSservices.PodsonFargatecanco-existwithpodsonregularworkernodesinthesameKubernetescluster.
EasyinstructionsandasampleconfigurationfileisavailableintheAWSFargateproviderdocumentation.Pleasenotethatthisproviderisnotcurrentlysupported.
ElotlKipKipisaproviderthatrunspodsincloudinstances,allowingaKubernetesclustertotransparentlyscaleworkloadsintoacloud.Whenapodisscheduledontothevirtualnode,Kipstartsaright-sizedcloudinstanceforthepod'sworkloadanddispatchesthepodontotheinstance.Whenthepodisfinishedrunning,thecloudinstanceisterminated.
WhenworkloadsrunonKip,yourclustersizenaturallyscaleswiththeclusterworkload,podsarestronglyisolatedfromeachotherandtheuserisfreedfrommanagingworkernodesandstrategicallypackingpodsontonodes.
HashiCorpNomadProviderHashiCorpNomadproviderforVirtualKubeletconnectsyourKubernetesclusterwithNomadclusterbyexposingtheNomadclusterasanodeinKubernetes.Byusingtheprovider,podsthatarescheduledonthevirtualNomadnoderegisteredonKuberneteswillrunasjobsonNomadclientsastheywouldonaKubernetesnode.
Fordetailedinstructions,followtheguidehere.
LiqoProviderLiqoimplementsaproviderforVirtualKubeletdesignedtotransparentlyoffloadpodsandservicesto"peered"Kubernetesremotecluster.Liqoiscapableofdiscoveringneighborclusters(usingDNS,mDNS)and"peer"withthem,orinotherwords,establisharelationshiptosharepartoftheclusterresources.Whenaclusterhasestablishedapeering,anewinstanceoftheLiqoVirtualKubeletisspawnedtoseamlesslyextendthecapacityofthecluster,byprovidinganabstractionoftheresourcesoftheremotecluster.TheprovidercombinedwiththeLiqonetworkfabricextendstheclusternetworkingbyenablingPod-to-Podtrafficandmulti-clustereast-westservices,supportingendpointsonbothclusters.
Fordetailedinstruction,followtheguidehere
OpenStackZunProviderOpenStackZunproviderforVirtualKubeletconnectsyourKubernetesclusterwithOpenStackinordertorunKubernetespodsonOpenStackCloud.YourpodsonOpenStackhaveaccesstoOpenStacktenantnetworksbecausetheyhaveNeutronportsinyoursubnets.EachpodwillhaveprivateIPaddressestoconnecttootherOpenStackresources(i.e.VMs)withinyourtenant,optionallyhavefloatingIPaddressestoconnecttotheinternet,andbind-mountCindervolumesintoapathinsideapod'scontainer.
./bin/virtual-kubelet--provider="openstack"Fordetailedinstructions,followtheguidehere.
TensileKubeProviderTensilekubeiscontributedbytencentgames,whichisproviderforVirtualKubeletconnectsyourKubernetesclusterwithotherKubernetesclusters.ThisproviderenablesusextendingKubernetestoanunlimitedone.Byusingtheprovider,podsthatarescheduledonthevirtualnoderegisteredonKuberneteswillrunasjobsonotherKubernetesclusters'nodes.
AddingaNewProviderviatheProviderInterfaceProvidersconsumethisprojectasalibrarywhichimplementsthecorelogicofaKubernetesnodeagent(Kubelet),andwireuptheirimplementationforperformingtheneccessaryactions.
Thereare3maininterfaces:
PodLifecylceHandlerWhenpodsarecreated,updated,ordeletedfromKubernetes,thesemethodsarecalledtohandlethoseactions.
godoc#PodLifecylceHandler
typePodLifecycleHandlerinterface{//CreatePodtakesaKubernetesPodanddeploysitwithintheprovider.CreatePod(ctxcontext.Context,pod*corev1.Pod)error//UpdatePodtakesaKubernetesPodandupdatesitwithintheprovider.UpdatePod(ctxcontext.Context,pod*corev1.Pod)error//DeletePodtakesaKubernetesPodanddeletesitfromtheprovider.DeletePod(ctxcontext.Context,pod*corev1.Pod)error//GetPodretrievesapodbynamefromtheprovider(canbecached).GetPod(ctxcontext.Context,namespace,namestring)(*corev1.Pod,error)//GetPodStatusretrievesthestatusofapodbynamefromtheprovider.GetPodStatus(ctxcontext.Context,namespace,namestring)(*corev1.PodStatus,error)//GetPodsretrievesalistofallpodsrunningontheprovider(canbecached).GetPods(context.Context)([]*corev1.Pod,error)}ThereisalsoanoptionalinterfacePodNotifierwhichenablestheprovidertoasynchronouslynotifythevirtual-kubeletaboutpodstatuschanges.Ifthisinterfaceisnotimplemented,virtual-kubeletwillperiodicallycheckthestatusofallpods.
ItishighlyrecommendedtoimplementPodNotifier,especiallyifyouplantorunalargenumberofpods.
godoc#PodNotifier
typePodNotifierinterface{//NotifyPodsinstructsthenotifiertocallthepassedinfunctionwhen//thepodstatuschanges.////NotifyPodsshouldnotblockcallers.NotifyPods(context.Context,func(*corev1.Pod))}PodLifecycleHandlerisconsumedbythePodControllerwhichisthecorelogicformanagingpodsassignedtothenode.
pc,_:=node.NewPodController(podControllerConfig)//<--instatiatesthepodcontrollerpc.Run(ctx)//<--startswatchingforpodstobescheduledonthenodeNodeProviderNodeProviderisresponsiblefornotifyingthevirtual-kubeletaboutnodestatusupdates.Virtual-KubeletwillperiodicallycheckthestatusofthenodeandupdateKubernetesaccordingly.
godoc#NodeProvider
typeNodeProviderinterface{//Pingchecksifthenodeisstillactive.//Thisisintendedtobelightweightasitwillbecalledperiodicallyasa//heartbeattokeepthenodemarkedasreadyinKubernetes.Ping(context.Context)error//NotifyNodeStatusisusedtoasynchronouslymonitorthenode.//Thepassedincallbackshouldbecalledanytimethereisachangetothe//node'sstatus.//ThiswillgenerallytriggeracalltotheKubernetesAPIservertoupdate//thestatus.////NotifyNodeStatusshouldnotblockcallers.NotifyNodeStatus(ctxcontext.Context,cbfunc(*corev1.Node))}VirtualKubeletprovidesaNaiveNodeProviderthatyoucanuseifyoudonotplantohavecustomnodebehavior.
godoc#NaiveNodeProvider
NodeProvidergetsconsumedbytheNodeController,whichiscorelogicformanagingthenodeobjectinKubernetes.
nc,_:=node.NewNodeController(nodeProvider,nodeSpec)//<--instantiateanodecontrollerfromanodeproviderandakubernetesnodespecnc.Run(ctx)//<--createsthenodeinkubernetesandstartsuphecontrollerAPIendpointsOneoftherolesofaKubeletistoacceptrequestsfromtheAPIserverforthingslikekubectllogsandkubectlexec.Helpersforsettingthisupareprovidedhere
TestingUnittestsRunningtheunittestslocallyisassimpleasmaketest.
End-to-endtestsCheckouttest/e2eformoredetails.
KnownquirksandworkaroundsMissingLoadBalancerIPaddressesforservicesProvidersthatdonotsupportservicediscoveryKubernetes1.9introducesanewflag,ServiceNodeExclusion,forthecontrolplane'sControllerManager.EnablingthisflagintheControllerManager'smanifestallowsKubernetestoexcludeVirtualKubeletnodesfrombeingaddedtoLoadBalancerpools,allowingyoutocreatepublicfacingserviceswithexternalIPswithoutissue.
WorkaroundClusterrequirements:Kubernetes1.9orabove
EnabletheServiceNodeExclusionflag,bymodifyingtheControllerManagermanifestandadding--feature-gates=ServiceNodeExclusion=truetothecommandlinearguments.
ContributingVirtualKubeletfollowstheCNCFCodeofConduct.SigntheCNCFCLAtobeabletomakePullRequeststothisrepo.
MonthlyVirtualKubeletOfficeHoursareheldat10amPSTonthelastThursdayofeverymonthinthiszoommeetingroom.Checkoutthecalendarhere.
Ourgoogledrivewithdesignspecificationsandmeetingnotesarehere.
Wealsohaveacommunityslackchannelnamedvirtual-kubeletintheKubernetesslack.YoucanalsoconnectwiththeVirtualKubeletcommunityviathemailinglist.
评论