kubernetes-elasticsearch-cluster_开源项目-程序员客栈

Thisprojectisnolongermaintained

AsofNovember7th,2018,I'vedecidedtoendmycommitmenttomaintainingthisrepoandrelated.

It'sbeen3yearssinceIlastusedElasticsearch,soInolongerhavethemotivationittakestomaintainandevolvethisproject.Also,otherprojectsneedalltheattentionIcangive.

Itwasagreatrun,thankyouall.

kubernetes-elasticsearch-cluster

ElasticsearchclusterontopofKubernetesmadeeasy.

TableofContents(Very)ImportantNotesPre-RequisitesBuildcontainerimage(optional)TestDeployAccesstheservicePodanti-affinityAvailabilityDeploywithHelmInstallplug-insClean-upwithCuratorKibanaFAQTroubleshootingAbstract

Elasticsearchbest-practicesrecommendtoseparatenodesinthreeroles:

Masternodes-intendedforclusteringmanagementonly,nodata,noHTTPAPIDatanodes-intendedforclientusageanddataIngestnodes-intendedfordocumentpre-processingduringingestion

Giventhis,I'mgoingtodemonstratehowtoprovisionaproductiongradescenarioconsistingof3master,2dataand2ingestnodes.

(Very)Importantnotes

Elasticsearchpodsneedforaninit-containertoruninprivilegedmode,soitcansetsomeVMoptions.Forthattohappen,thekubeletshouldberunningwithargs--allow-privileged,otherwisetheinit-containerwillfailtorun.

Bydefault,ES_JAVA_OPTSissetto-Xms256m-Xmx256m.Thisisaverylowvaluebutmanyusers,i.e.minikubeusers,werehavingissueswithpodsgettingkilledbecausehostswereoutofmemory.Onecanchangethisinthedeploymentdescriptorsavailableinthisrepository.

Asofthemoment,KubernetespoddescriptorsuseanemptyDirforstoringdataineachdatanodecontainer.Thisismeanttobeforthesakeofsimplicityandshouldbeadaptedaccordingtoone'sstorageneeds.

ThestatefuldirectorycontainsanexamplewhichdeploysthedatapodsasaStatefulSet.TheseuseavolumeClaimTemplatestoprovisionpersistentstorageforeachpod.

Bydefault,PROCESSORSissetto1.Thismaynotbeenoughforsomedeployments,especiallyatstartuptime.Adjustresources.limits.cpuand/orlivenessProbeaccordinglyifrequired.Notethatresources.limits.cpumustbeaninteger.

Pre-requisitesKubernetes1.11.x(testedwithv1.11.2ontopofVagrant+CoreOS).kubectlconfiguredtoaccesstheKubernetesAPI.Buildimages(optional)

Providingone'sownversionoftheimagesautomaticallybuiltfromthisrepositorywillnotbesupported.Thisisanoptionalstep.Onehasbeenwarned.

TestDeploykubectlcreate-fes-discovery-svc.yamlkubectlcreate-fes-svc.yamlkubectlcreate-fes-master.yamlkubectlrolloutstatus-fes-master.yamlkubectlcreate-fes-ingest-svc.yamlkubectlcreate-fes-ingest.yamlkubectlrolloutstatus-fes-ingest.yamlkubectlcreate-fes-data.yamlkubectlrolloutstatus-fes-data.yaml

Let'scheckifeverythingisworkingproperly:

kubectlgetsvc,deployment,pods-lcomponent=elasticsearchNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEservice/elasticsearchClusterIP10.100.243.196<none>9200/TCP3mservice/elasticsearch-discoveryClusterIPNone<none>9300/TCP3mservice/elasticsearch-ingestClusterIP10.100.76.74<none>9200/TCP2mNAMEDESIREDCURRENTUP-TO-DATEAVAILABLEAGEdeployment.extensions/es-data22221mdeployment.extensions/es-ingest22222mdeployment.extensions/es-master33333mNAMEREADYSTATUSRESTARTSAGEpod/es-data-56f8ff8c97-642bq1/1Running01mpod/es-data-56f8ff8c97-h6hpc1/1Running01mpod/es-ingest-6ddd5fc689-b4s941/1Running02mpod/es-ingest-6ddd5fc689-d8rtj1/1Running02mpod/es-master-68bf8f86c4-bsfrx1/1Running03mpod/es-master-68bf8f86c4-g8nph1/1Running03mpod/es-master-68bf8f86c4-q5khn1/1Running03m

Aswecanassert,theclusterseemstobeupandrunning.Easy,wasn'tit?

Accesstheservice

Don'tforgetthatservicesinKubernetesareonlyacessiblefromcontainersinthecluster.Fordifferentbehavioroneshouldconfigurethecreationofanexternalload-balancer.Whileit'ssupportedwithinthisexampleservicedescriptor,itsusageisoutofscopeofthisdocument,fornow.

Note:ifyouareusingoneofthecloudproviderswhichsupportexternalloadbalancers,settingthetypefieldto"LoadBalancer"willprovisionaloadbalancerforyourService.Youcanuncommentthefieldines-svc.yaml.

kubectlgetsvcelasticsearchNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEelasticsearchClusterIP10.100.243.196<none>9200/TCP3m

FromanyhostontheKubernetescluster(that'srunningkube-proxyorsimilar),run:

curlhttps://10.100.243.196:9200

Oneshouldseesomethingsimilartothefollowing:

{"name":"es-data-56f8ff8c97-642bq","cluster_name":"myesdb","cluster_uuid":"RkRkTl26TDOE7o0FhCcW_g","version":{"number":"6.3.2","build_flavor":"default","build_type":"tar","build_hash":"053779d","build_date":"2018-07-20T05:20:23.451332Z","build_snapshot":false,"lucene_version":"7.3.1","minimum_wire_compatibility_version":"5.6.0","minimum_index_compatibility_version":"5.0.0"},"tagline":"YouKnow,forSearch"}

Orifonewantstoseeclusterinformation:

curlhttps://10.100.243.196:9200/_cluster/health?pretty

Oneshouldseesomethingsimilartothefollowing:

{"cluster_name":"myesdb","status":"green","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}Podanti-affinity

OneofthemainadvantagesofrunningElasticsearchontopofKubernetesishowresilienttheclusterbecomes,particularlyduringnoderestarts.Howeverifalldatapodsarescheduledontothesamenode(s),thisadvantagedecreasessignificantlyandmayevenresultinnodatapodsbeingavailable.

Itisthenhighlyrecommended,inthecontextofthesolutiondescribedinthisrepository,thatoneadoptspodanti-affinityinordertoguaranteethattwodatapodswillneverrunonthesamenode.

Here'sanexample:

spec:affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:-weight:100podAffinityTerm:labelSelector:matchExpressions:-key:componentoperator:Invalues:-elasticsearch-key:roleoperator:Invalues:-datatopologyKey:kubernetes.io/hostnamecontainers:-(...)Availability

IfonewantstoensurethatnomorethannElasticsearchnodeswillbeunavailableatatime,onecanoptionally(changeand)applythefollowingmanifests:

kubectlcreate-fes-master-pdb.yamlkubectlcreate-fes-data-pdb.yaml

Note:ThisisanadvancedsubjectandoneshouldonlyputitinpracticeifoneunderstandsclearlywhatitmeansbothintheKubernetesandElasticsearchcontexts.Formoreinformation,pleaseconsultPodDisruptions.

DeploywithHelm

WARNING:TheHelmchartismaintainedbysomeoneelseinthecommunityandmaynotup-to-datewiththisrepo.

Helmchartsforabasic(non-stateful)ElasticSearchdeploymentaremaintainedathttps://github.com/clockworksoul/helm-elasticsearch.WithHelmproperlyinstalledandconfigured,standingupacompleteclusterisalmosttrivial:

gitclonehttps://github.com/clockworksoul/helm-elasticsearch.githelminstallhelm-elasticsearch

Variousparametersofthecluster,includingreplicacountandmemoryallocations,canbeadjustedbyeditingthehelm-elasticsearch/values.yamlfile.ForinformationaboutHelm,pleaseconsultthecompleteHelmdocumentation.

Installplug-ins

Theimageusedinthisrepoisveryminimalist.However,onecaninstalladditionalplug-insatwillbysimplyspecifyingtheES_PLUGINS_INSTALLenvironmentvariableinthedesiredpoddescriptors.Forinstance,toinstallGoogleCloudStorageandS3plug-insitwouldbelikefollows:

-name:"ES_PLUGINS_INSTALL"value:"repository-gcs,repository-s3"

Note:TheX-Packplugindoesnotcurrentlyworkwiththequay.io/pires/docker-elasticsearch-kubernetesimage.SeeIssue#102

Clean-upwithCurator

Additionally,onecanrunaCronJobthatwillperiodicallyrunCuratortocleanupindices(ordootheractionsontheElasticsearchcluster).

kubectlcreate-fes-curator-config.yamlkubectlcreate-fes-curator.yaml

Please,confirmthejobhasbeencreated.

kubectlgetcronjobsNAMESCHEDULESUSPENDACTIVELAST-SCHEDULEcurator10***False0<none>

Thejobisconfiguredtorunonceadayat1minutepastmidnightanddeleteindicesthatareolderthan3days.

Notes

Onecanchangetheschedulebyeditingthecronnotationines-curator.yaml.Onecanchangetheaction(e.g.deleteolderthan3days)byeditingthees-curator-config.yaml.Thedefinitionoftheaction_file.yamlisquiteself-explainingforsimpleset-ups.Formoreadvancedconfigurationoptions,pleaseconsulttheCuratorDocumentation.

Ifonewantstoremovethecuratorjob,justrun:

kubectldeletecronjobcuratorkubectldeleteconfigmapcurator-configKibana

WARNING:TheKibanasectionismaintainedbysomeoneelseinthecommunityandmaynotup-to-datewiththisrepo.

Deploy

IfKibanadefaultsarenotenough,onemaywanttocustomizekibana.yamlthroughaConfigMap.PleaserefertoConfiguringKibanaforallavailableattributes.

kubectlcreate-fkibana-cm.yamlkubectlcreate-fkibana-svc.yamlkubectlcreate-fkibana.yaml

Kibanawillbecomeavailablethroughservicekibana,andonewillbeabletoaccessitfromwithinthecluster,orproxyitthroughtheKubernetesAPIasfollows:

curlhttps://<API_SERVER_URL>/api/v1/namespaces/default/services/kibana:http/proxy

OnecanalsocreateanIngresstoexposetheservicepubliclyorsimplyusetheservicenodeport.Inthecaseoneproceedstodoso,onemustchangetheenvironmentvariableSERVER_BASEPATHtothematchtheirenvironment.

FAQWhydoesNUMBER_OF_MASTERSdifferfromnumberofmaster-replicas?

Thedefaultvalueforthisenvironmentvariableis2,meaningaclusterwillneedaminimumof2masternodestooperate.Ifaclusterhas3mastersandonedies,theclusterstillworks.Minimummasternodesareusuallyn/2+1,wherenisthenumberofmasternodesinacluster.Ifaclusterhas5masternodes,oneshouldhaveaminimumof3,lessthanthatandtheclusterstops.Ifonescalesthenumberofmasters,makesuretoupdatetheminimumnumberofmasternodesthroughtheElasticsearchAPIassettingenvironmentvariablewillonlyworkonclustersetup.Moreinfo:https://www.elastic.co/guide/en/elasticsearch/guide/1.x/_important_configuration_changes.html#_minimum_master_nodes

HowcanIcustomizeelasticsearch.yaml?

ReadadifferentconfigfilebysettingsenvvarES_PATH_CONF=/path/to/my/config/(seetheElasticsearchdocsformore).Anotheroptionwouldbetobuildone'sownimagefromthisrepository

TroubleshootingNoup-and-runningsite-local

Oneoftheerrorsonemaycomeacrosswhenrunningthesetupisthefollowingerror:

[2016-11-29T01:28:36,515][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler][]uncaughtexceptioninthread[main]org.elasticsearch.bootstrap.StartupException:java.lang.IllegalArgumentException:Noup-and-runningsite-local(private)addressesfound,got[name:lo(lo),name:eth0(eth0)]atorg.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:116)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.cli.Command.main(Command.java:62)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73)~[elasticsearch-5.0.1.jar:5.0.1]Causedby:java.lang.IllegalArgumentException:Noup-and-runningsite-local(private)addressesfound,got[name:lo(lo),name:eth0(eth0)]atorg.elasticsearch.common.network.NetworkUtils.getSiteLocalAddresses(NetworkUtils.java:187)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.common.network.NetworkService.resolveInternal(NetworkService.java:246)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.common.network.NetworkService.resolveInetAddresses(NetworkService.java:220)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.common.network.NetworkService.resolveBindHostAddresses(NetworkService.java:130)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:575)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:182)~[?:?]atorg.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.transport.TransportService.doStart(TransportService.java:182)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.node.Node.start(Node.java:525)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:211)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:288)~[elasticsearch-5.0.1.jar:5.0.1]atorg.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112)~[elasticsearch-5.0.1.jar:5.0.1]...6more[2016-11-29T01:28:37,448][INFO][o.e.n.Node][kIEYQSE]stopping...[2016-11-29T01:28:37,451][INFO][o.e.n.Node][kIEYQSE]stopped[2016-11-29T01:28:37,452][INFO][o.e.n.Node][kIEYQSE]closing...[2016-11-29T01:28:37,464][INFO][o.e.n.Node][kIEYQSE]closed

Thisisrelatedtohowthecontainerbindstonetworkports(defaultsto_local_).Itwillneedtomatchtheactualnodenetworkinterfacename,whichdependsonwhatOSandinfrastructureprovideroneuses.Forinstance,iftheprimaryinterfaceonthenodeisp1p1thenthatisthevaluethatneedstobesetfortheNETWORK_HOSTenvironmentvariable.Pleaseseethedocumentationforreferenceofoptions.

Inordertoworkaroundthis,setNETWORK_HOSTenvironmentvariableinthepoddescriptorsasfollows:

-name:"NETWORK_HOST"value:"_eth0_"#_p1p1_ifinterfacenameisp1p1,_ens4_ifinterfacenameisens4,andsoon.(IPv6)org.elasticsearch.bootstrap.StartupException:BindTransportException

IntermittentfailuresoccurwhenthelocalnetworkinterfacehasbothIPv4andIPv6addresses,andElasticsearchtriestobindtotheIPv6addressfirst.IftheIPv4addressischosenfirst,Elasticsearchstartscorrectly.

Inordertoworkaroundthis,setNETWORK_HOSTenvironmentvariableinthepoddescriptorsasfollows:

-name:"NETWORK_HOST"value:"_eth0:ipv4_"#_p1p1:ipv4_ifinterfacenameisp1p1,_ens4:ipv4_ifinterfacenameisens4,andsoon.

kubernetes-elasticsearch-cluster开源项目

作品详情

重点城市程序员兼职推荐

重点岗位程序员兼职推荐