TheNVIDIAdevicepluginforKubernetesisaDaemonsetthatallowsyoutoautomatically:
ExposethenumberofGPUsoneachnodesofyourclusterKeeptrackofthehealthofyourGPUsRunGPUenabledcontainersinyourKubernetescluster.ThisrepositorycontainsNVIDIA'sofficialimplementationoftheKubernetesdeviceplugin.
Pleasenotethat:
TheNVIDIAdevicepluginAPIisbetaasofKubernetesv1.10.TheNVIDIAdevicepluginisstillconsideredbetaandismissingMorecomprehensiveGPUhealthcheckingfeaturesGPUcleanupfeatures...SupportwillonlybeprovidedfortheofficialNVIDIAdeviceplugin(andnotforforksorothervariantsofthisplugin).PrerequisitesThelistofprerequisitesforrunningtheNVIDIAdevicepluginisdescribedbelow:
NVIDIAdrivers~=384.81nvidia-dockerversion>2.0(seehowtoinstallandit'sprerequisites)dockerconfiguredwithnvidiaasthedefaultruntime.Kubernetesversion>=1.10QuickStartPreparingyourGPUNodesThefollowingstepsneedtobeexecutedonallyourGPUnodes.ThisREADMEassumesthattheNVIDIAdriversandnvidia-dockerhavebeeninstalled.
Notethatyouneedtoinstallthenvidia-docker2packageandnotthenvidia-container-toolkit.Thisisbecausethenew--gpusoptionshasn'treachedkubernetesyet.Example:
#Addthepackagerepositories$distribution=$(./etc/os-release;echo$ID$VERSION_ID)$curl-s-Lhttps://nvidia.github.io/nvidia-docker/gpgkey|sudoapt-keyadd-$curl-s-Lhttps://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list|sudotee/etc/apt/sources.list.d/nvidia-docker.list$sudoapt-getupdate&&sudoapt-getinstall-ynvidia-docker2$sudosystemctlrestartdockerYouwillneedtoenablethenvidiaruntimeasyourdefaultruntimeonyournode.Wewillbeeditingthedockerdaemonconfigfilewhichisusuallypresentat/etc/docker/daemon.json:
{"default-runtime":"nvidia","runtimes":{"nvidia":{"path":"/usr/bin/nvidia-container-runtime","runtimeArgs":[]}}}ifruntimesisnotalreadypresent,headtotheinstallpageofnvidia-docker
EnablingGPUSupportinKubernetesOnceyouhaveconfiguredtheoptionsaboveonalltheGPUnodesinyourcluster,youcanenableGPUsupportbydeployingthefollowingDaemonset:
$kubectlcreate-fhttps://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.ymlNote:Thisisasimplestaticdaemonsetmeanttodemonstratethebasicfeaturesofthenvidia-device-plugin.PleaseseetheinstructionsbelowforDeploymentviahelmwhendeployingtheplugininaproductionsetting.
RunningGPUJobsWiththedaemonsetdeployed,NVIDIAGPUscannowberequestedbyacontainerusingthenvidia.com/gpuresourcetype:
apiVersion:v1kind:Podmetadata:name:gpu-podspec:containers:-name:cuda-containerimage:nvcr.io/nvidia/cuda:9.0-develresources:limits:nvidia.com/gpu:2#requesting2GPUs-name:digits-containerimage:nvcr.io/nvidia/digits:20.12-tensorflow-py3resources:limits:nvidia.com/gpu:2#requesting2GPUsWARNING:ifyoudon'trequestGPUswhenusingthedevicepluginwithNVIDIAimagesalltheGPUsonthemachinewillbeexposedinsideyourcontainer.
DeploymentviahelmThepreferredmethodtodeploythedevicepluginisasadaemonsetusinghelm.Instructionsforinstallinghelmcanbefoundhere.
Thehelmchartforthelatestreleaseoftheplugin(v0.9.0)includesanumberofcustomizablevalues.Themostcommonlyoverriddenonesare:
failOnInitError:failthepluginifanerrorisencounteredduringinitialization,otherwiseblockindefinitely(default'true')compatWithCPUManager:runwithescalatedprivilegestobecompatiblewiththestaticCPUManagerpolicy(default'false')legacyDaemonsetAPI:usethelegacydaemonsetAPIversion'extensions/v1beta1'(default'false')migStrategy:thedesiredstrategyforexposingMIGdevicesonGPUsthatsupportit[none|single|mixed](default"none")deviceListStrategy:thedesiredstrategyforpassingthedevicelisttotheunderlyingruntime[envvar|volume-mounts](default"envvar")deviceIDStrategy:thedesiredstrategyforpassingdeviceIDstotheunderlyingruntime[uuid|index](default"uuid")nvidiaDriverRoot:therootpathfortheNVIDIAdriverinstallation(typicalvaluesare'/'or'/run/nvidia/driver')Whensettotrue,thefailOnInitErrorflagfailsthepluginifanerrorisencounteredduringinitialization.Whensettofalse,itprintsanerrormessageandblocksthepluginindefinitelyinsteadoffailing.Blockingindefinitelyfollowslegacysemanticsthatallowtheplugintodeploysuccessfullyonnodesthatdon'thaveGPUsonthem(andaren'tsupposedtohaveGPUsonthem)withoutthrowinganerror.Inthisway,youcanblindlydeployadaemonsetwiththepluginonallnodesinyourcluster,whethertheyhaveGPUsonthemornot,withoutencounteringanerror.However,doingsomeansthatthereisnowaytodetectanactualerroronnodesthataresupposedtohaveGPUsonthem.Failingifaninitilizationerrorisencounteredisnowthedefaultandshouldbeadoptedbyallnewdeployments.
ThecompatWithCPUManagerflagconfiguresthedaemonsettobeabletointeroperatewiththestaticCPUManagerofthekubelet.Settingthisflagrequiresonetodeploythedaemonsetwithelevatedprivileges,soonlydosoifyouknowyouneedtointeroperatewiththeCPUManager.
ThelegacyDaemonsetAPIflagconfiguresthedaemonsettouseversionextensions/v1beta1oftheDaemonSetAPI.ThisAPIversionwasremovedinKubernetesv1.16,soisonlyintendedtoallownewerpluginstorunonolderversionsofKubernetes.
ThemigStrategyflagconfiguresthedaemonsettobeabletoexposeMulti-InstanceGPUs(MIG)onGPUsthatsupportthem.MoreinformationonwhatthesestrategiesareandhowtheyshouldbeusedcanbefoundinSupportingMulti-InstanceGPUs(MIG)inKubernetes.
Note:WithamigStrategyofmixed,youwillhaveadditionalresourcesavailabletoyouoftheformnvidia.com/mig-<slice_count>g.<memory_size>gbthatyoucansetinyourpodspectogetaccesstoaspecificMIGdevice.
ThedeviceListStrategyflagallowsonetochoosewhichstrategythepluginwillusetoadvertisethelistofGPUsallocatedtoacontainer.ThisistraditionallydonebysettingtheNVIDIA_VISIBLE_DEVICESenvironmentvariableasdescribedhere.Thisstrategycanbeselectedviathe(default)envvaroption.Supportwasrecentlyaddedtothenvidia-container-toolkittoalsoallowpassingthelistofdevicesasasetofvolumemountsinsteadofasanenvironmentvariable.Thisstrategycanbeselectedviathevolume-mountsoption.Detailsfortherationalebehindthisstrategycanbefoundhere.
ThedeviceIDStrategyflagallowsonetochoosewhichstrategythepluginwillusetopassthedeviceIDoftheGPUsallocatedtoacontainer.ThedeviceIDhastraditionallybeenpassedastheUUIDoftheGPU.ThisflagletsauserdecideiftheywouldliketousetheUUIDortheindexoftheGPU(asseenintheoutputofnvidia-smi)astheidentifierpassedtotheunderlyingruntime.PassingtheindexmaybedesirableinsituationswherepodsthathavebeenallocatedGPUsbytheplugingetrestartedwithdifferentphysicalGPUsattachedtothem.
Pleasetakealookinthefollowingvalues.yamlfiletoseethefullsetofoverridableparametersforthedeviceplugin.
https://github.com/NVIDIA/k8s-device-plugin/blob/v0.9.0/deployments/helm/nvidia-device-plugin/values.yamlInstallingviahelminstallfromthenvidia-device-pluginhelmrepositoryThepreferredmethodofdeploymentiswithhelminstallviathenvidia-device-pluginhelmrepository.
Thisrepositorycanbeinstalledasfollows:
$helmrepoaddnvdphttps://nvidia.github.io/k8s-device-plugin$helmrepoupdateOncethisrepoisupdated,youcanbegininstallingpackagesfromittodepoloythenvidia-device-plugindaemonset.Belowaresomeexamplesofdeployingthepluginwiththevariousflagsfromabove.
Note:Sincethisisapre-releaseversion,youwillneedtopassthe--develflagtohelmsearchrepoinordertoseethisreleaselisted.
Usingthedefaultvaluesfortheflags:
$helminstall\--version=0.9.0\--generate-name\nvdp/nvidia-device-pluginEnablingcompatibilitywiththeCPUManagerandrunningwitharequestfor100msofCPUtimeandalimitof512MBofmemory.
$helminstall\--version=0.9.0\--generate-name\--setcompatWithCPUManager=true\--setresources.requests.cpu=100m\--setresources.limits.memory=512Mi\nvdp/nvidia-device-pluginUsethelegacyDaemonsetAPI(onlyavailableonKubernetes<v1.16):
$helminstall\--version=0.9.0\--generate-name\--setlegacyDaemonsetAPI=true\nvdp/nvidia-device-pluginEnablingcompatibilitywiththeCPUManagerandthemixedmigStrategy
$helminstall\--version=0.9.0\--generate-name\--setcompatWithCPUManager=true\--setmigStrategy=mixed\nvdp/nvidia-device-pluginDeployingviahelminstallwithadirectURLtothehelmpackageIfyouprefernottoinstallfromthenvidia-device-pluginhelmrepo,youcanrunhelminstalldirectlyagainstthetarballoftheplugin'shelmpackage.Theexamplesbelowinstallthesamedaemonsetsasthemethodabove,exceptthattheyusedirectURLstothehelmpackageinsteadofthehelmrepo.
Usingthedefaultvaluesfortheflags:
$helminstall\--generate-name\https://nvidia.github.com/k8s-device-plugin/stable/nvidia-device-plugin-0.9.0.tgzEnablingcompatibilitywiththeCPUManagerandrunningwitharequestfor100msofCPUtimeandalimitof512MBofmemory.
$helminstall\--generate-name\--setcompatWithCPUManager=true\--setresources.requests.cpu=100m\--setresources.limits.memory=512Mi\https://nvidia.github.com/k8s-device-plugin/stable/nvidia-device-plugin-0.9.0.tgzUsethelegacyDaemonsetAPI(onlyavailableonKubernetes<v1.16):
$helminstall\--generate-name\--setlegacyDaemonsetAPI=true\https://nvidia.github.com/k8s-device-plugin/stable/nvidia-device-plugin-0.9.0.tgzEnablingcompatibilitywiththeCPUManagerandthemixedmigStrategy
$helminstall\--generate-name\--setcompatWithCPUManager=true\--setmigStrategy=mixed\https://nvidia.github.com/k8s-device-plugin/stable/nvidia-device-plugin-0.9.0.tgzBuildingandRunningLocallyThenextsectionsarefocusedonbuildingthedevicepluginlocallyandrunningit.Itisintendedpurelyfordevelopmentandtesting,andnotrequiredbymostusers.Itassumesyouarepinningtothelatestreleasetag(i.e.v0.9.0),butcaneasilybemodifiedtoworkwithanyavailabletagorbranch.
WithDockerBuildOption1,pulltheprebuiltimagefromDockerHub:
$dockerpullnvcr.io/nvidia/k8s-device-plugin:v0.9.0$dockertagnvcr.io/nvidia/k8s-device-plugin:v0.9.0nvcr.io/nvidia/k8s-device-plugin:develOption2,buildwithoutcloningtherepository:
$dockerbuild\-tnvcr.io/nvidia/k8s-device-plugin:devel\-fdocker/Dockerfile\https://github.com/NVIDIA/k8s-device-plugin.git#v0.9.0Option3,ifyouwanttomodifythecode:
$gitclonehttps://github.com/NVIDIA/k8s-device-plugin.git&&cdk8s-device-plugin$dockerbuild\-tnvcr.io/nvidia/k8s-device-plugin:devel\-fdocker/Dockerfile\.RunWithoutcompatibilityfortheCPUManagerstaticpolicy:
$dockerrun\-it\--security-opt=no-new-privileges\--cap-drop=ALL\--network=none\-v/var/lib/kubelet/device-plugins:/var/lib/kubelet/device-plugins\nvcr.io/nvidia/k8s-device-plugin:develWithcompatibilityfortheCPUManagerstaticpolicy:
$dockerrun\-it\--privileged\--network=none\-v/var/lib/kubelet/device-plugins:/var/lib/kubelet/device-plugins\nvcr.io/nvidia/k8s-device-plugin:devel--pass-device-specsWithoutDockerBuild$C_INCLUDE_PATH=/usr/local/cuda/includeLIBRARY_PATH=/usr/local/cuda/lib64gobuildRunWithoutcompatibilityfortheCPUManagerstaticpolicy:
$./k8s-device-pluginWithcompatibilityfortheCPUManagerstaticpolicy:
$./k8s-device-plugin--pass-device-specsChangelogVersionv0.9.0FixbugwhenusingCPUManagerandthedevicepluginMIGmodenotsetto"none"AllowpassinglistofGPUsbydeviceindexinsteadofuuidMovetourfave/clitobuildtheCLISupportsettingcommandlineflagsviaenvironmentvariablesVersionv0.8.2Updatealldockerhubreferencestonvcr.ioVersionv0.8.1FixpermissionerrorwhenusingNewDeviceinsteadofNewDeviceLitewhenconstructingMIGdevicemapVersionv0.8.0RaiseanerrorifadevicehasmigEnabled=truebuthasnoMIGdevicesAllowmig.strategy=singleonnodeswithnon-MIGgpusVersionv0.7.3UpdatevendoringtoincludebugfixfornvmlEventSetWait_v2Versionv0.7.2Fixbugindockfilesforubi8andcentosusingCMDnotENTRYPOINTVersionv0.7.1UpdateallDockerfilestopointtolatestcuda-baseonnvcr.ioVersionv0.7.0Promotev0.7.0-rc.8tov0.7.0Versionv0.7.0-rc.8Permitconfigurationofalternativecontainerregistrythroughenvironmentvariables.Addanalternatesetofgitlab-cidirectivesunder.nvidia-ci.ymlUpdateallk8sdependenciestov1.19.1UpdatevendoringforNVMLGobindingsMoverestartlooptoforcerecreateofpluginsonSIGHUPVersionv0.7.0-rc.7FixbugwhichonlyallowedrunningthepluginonmachineswithCUDA10.2+installedVersionv0.7.0-rc.6Addlogictoskip/erroroutwhenunsupportedMIGdeviceencounteredFixbugtreatingmemoryasmultipleof1000insteadof1024SwitchtousingCUDAbaseimagesAddasetofstandardteststothe.gitlab-ci.ymlfileVersionv0.7.0-rc.5AdddeviceListStrategyFlagtoallowdevicelistpassingasvolumemountsVersionv0.7.0-rc.4Allowonetooverrideselector.matchLabelsinthehelmchartAllowonetooverridetheudateStrategyinthehelmchartVersionv0.7.0-rc.3FailthepluginifNVMLcannotbeloadedUpdateloggingtoprinttostderronerrorAddbesteffortremovalofsocketfilebeforeservingAddlogictoimplementGetPreferredAllocation()callfromkubeletVersionv0.7.0-rc.2Addtheabilitytoset'resources'aspartofahelminstallAddoverridesfornameandfullnameinhelmchartAddabilitytooverrideimagerelatedparametershelmchartAddconditionalsupportforoverridingsecutiryContextinhelmchartVersionv0.7.0-rc.1AddedmigStrategyasaparametertoselecttheMIGstrategytothehelmchartAddsupportforMIGwithdifferentstrategies{none,single,mixed}UpdatevendoredNVMLbindingstolatest(toincludeMIGAPIs)AddlicenseinUBIimageUpdateUBIimagewithcertificationrequirementsVersionv0.6.0UpdateCI,buildsystem,andvendoringmechanismChangeversioningschemetov0.x.xinsteadofv1.0.0-betaxIntroducedhelmchartsasamechanismtodeploythepluginVersionv0.5.0Addanewplugin.ymlvariantthatiscompatiblewiththeCPUManagerChangeCMDinDockerfiletoENTRYPOINTAddflagtooptionallyreturnlistofdevicenodesinAllocate()callRefactordeviceplugintoeventuallyhandlemultipleresourcetypesMovepluginerrorretrytoeventloopsowecanexitwithasignalUpdateallvendoreddependenciestotheirlatestversionsFixbugthatwasinadvertentlyalwaysdisablinghealthchecksUpdateminimaldriverversionto384.81Versionv0.4.0FixesabugwithanilpointerdereferencearoundgetDevices:CPUAffinityVersionv0.3.0ManifestisupdatedforKubernetes1.16+(apps/v1)AddsmorelogginginformationVersionv0.2.0AddstheTopologyfieldforKubernetes1.16+Versionv0.1.0IfgRPCthrowsanerror,thedevicepluginnolongerendsupinanonresponsivestate.Versionv0.0.0ReversionedtoSEMVERasdevicepluginsaren'ttiedtoaspecificversionofkubernetesanymore.Versionv1.11Nochange.Versionv1.10ThedevicePluginAPIisnowv1beta1Versionv1.9ThedevicePluginAPIchangedandisnolongercompatiblewith1.8ErrormessageswereaddedIssuesandContributingCheckouttheContributingdocument!
YoucanreportabugbyfilinganewissueYoucancontributebyopeningapullrequestVersioningBeforev1.10theversioningschemeofthedevicepluginhadtomatchexactlytheversionofKubernetes.Afterthepromotionofdevicepluginstobetathisconditionwaswasnolongerrequired.WequicklynoticedthatthisversioningschemewasveryconfusingforusersastheystillexpectedtoseeaversionofthedevicepluginforeachversionofKubernetes.
Thisversioningschemeappliestothetagsv1.8,v1.9,v1.10,v1.11,v1.12.
WehavenowchangedtheversioningtofollowSEMVER.Thefirstversionfollowingthisschemehasbeentaggedv0.0.0.
Goingforward,themajorversionofthedevicepluginwillonlychangefollowingachangeinthedevicepluginAPIitself.Forexample,versionv1beta1ofthedevicepluginAPIcorrespondstoversionv0.x.xofthedeviceplugin.Ifanewv2beta2versionofthedevicepluginAPIcomesout,thenthedevicepluginwillincreaseitsmajorversionto1.x.x.
Asofnow,thedevicepluginAPIforKubernetes>=v1.10isv1beta1.IfyouhaveaversionofKubernetes>=1.10youcandeployanydevicepluginversion>v0.0.0.
UpgradingKuberneteswiththeDevicePluginUpgradingKuberneteswhenyouhaveadeviceplugindeployeddoesn'trequireyoutodoany,particularchangestoyourworkflow.TheAPIisversionedandisprettystable(thoughitisnotguaranteedtobenonbreaking).StartingwithKubernetesversion1.10,youcanusev0.3.0ofthedeviceplugintoperformupgrades,andKuberneteswon'trequireyoutodeployadifferentversionofthedeviceplugin.Onceanodecomesbackonlineaftertheupgrade,youwillseeGPUsre-registeringthemselvesautomatically.
Upgradingthedevicepluginitselfisamorecomplextask.ItisrecommendedtodrainGPUtasksaswecannotguaranteethatGPUtaskswillsurvivearollingupgrade.HoweverwemakebesteffortstopreserveGPUtasksduringanupgrade.
评论