google-rules-of-machine-learning

GithubmirrorofM.Zinkevich'sgreat"RulesofMachineLearning"styleguide,withextragoodness.

Youcanfindtheterminologyforthisguideinterminology.md.

Youcanfindtheoverviewforthisguideinoverview.md.

Note:Asterisk(*)footnotesaremyown.NumberedfootnotesareMartin's.

Machinelearningiscool,butitrequiresdata.Theoretically,youcantakedatafromadifferentproblemandthentweakthemodelforanewproduct,butthiswilllikelyunderperformbasicheuristics.Ifyouthinkthatmachinelearningwillgiveyoua100%boost,thenaheuristicwillgetyou50%ofthewaythere.Forinstance,ifyouarerankingappsinanappmarketplace,youcouldusetheinstallrateornumberofinstalls.Ifyouaredetectingspam,filteroutpublishersthathavesentspambefore.Don’tbeafraidtousehumaneditingeither.Ifyouneedtorankcontacts,rankthemostrecentlyusedhighest(orevenrankalphabetically).Ifmachinelearningisnotabsolutelyrequiredforyourproduct,don'tuseituntilyouhavedata.

GoogleResearchBlog-The280-Year-OldAlgorithmInsideGoogleTrips

Beforeformalizingwhatyourmachinelearningsystemwilldo,trackasmuchaspossibleinyourcurrentsystem.Dothisforthefollowingreasons:

Forinstance,supposeyouwanttodirectlyoptimizeone-­dayactiveusers.However,duringyourearlymanipulationsofthesystem,youmaynoticethatdramaticalterationsoftheuserexperiencedon’tnoticeablychangethismetric.GooglePlusteammeasuresexpandsperread,resharesperread,plus­-onesperread,comments/read,commentsperuser,resharesperuser,etc.whichtheyuseincomputingthegoodnessofapostatservingtime.Also,notethatanexperimentframework,whereyoucangroupusersintobucketsandaggregatestatisticsbyexperiment,isimportant.SeeRule#12.

Bybeingmoreliberalaboutgatheringmetrics,youcangainabroaderpictureofyoursystem.Noticeaproblem?Addametrictotrackit!Excitedaboutsomequantitativechangeonthelastrelease?Addametrictotrackit!

Asimpleheuristiccangetyourproductoutthedoor.Acomplexheuristicisunmaintainable.Onceyouhavedataandabasicideaofwhatyouaretryingtoaccomplish,moveontomachinelearning.Asinmostsoftwareengineeringtasks,youwillwanttobeconstantlyupdatingyourapproach,whetheritisaheuristicoramachine-learnedmodel,andyouwillfindthatthemachine-­learnedmodeliseasiertoupdateandmaintain(seeRule#16).

Focusonyoursysteminfrastructureforyourfirstpipeline.Whileitisfuntothinkaboutalltheimaginativemachinelearningyouaregoingtodo,itwillbehardtofigureoutwhatishappeningifyoudon’tfirsttrustyourpipeline.

Thefirstmodelprovidesthebiggestboosttoyourproduct,soitdoesn'tneedtobefancy.Butyouwillrunintomanymoreinfrastructureissuesthanyouexpect.Beforeanyonecanuseyourfancynewmachinelearningsystem,youhavetodetermine:

Choosingsimplefeaturesmakesiteasiertoensurethat:

Onceyouhaveasystemthatdoesthesethreethingsreliably,youhavedonemostofthework.Yoursimplemodelprovidesyouwithbaselinemetricsandabaselinebehaviorthatyoucanusetotestmorecomplexmodels.Someteamsaimfora“neutral”firstlaunch:afirstlaunchthatexplicitlyde-­prioritizesmachinelearninggains,toavoidgettingdistracted.

Makesurethattheinfrastructureistestable,andthatthelearningpartsofthesystemareencapsulatedsothatyoucantesteverythingaroundit.Specifically:

Testgettingdataintothealgorithm.Checkthatfeaturecolumnsthatshouldbepopulatedarepopulated.Whereprivacypermits,manuallyinspecttheinputtoyourtrainingalgorithm.Ifpossible,checkstatisticsinyourpipelineincomparisontoelsewhere,suchasRASTA.

Testgettingmodelsoutofthetrainingalgorithm.Makesurethatthemodelinyourtrainingenvironmentgivesthesamescoreasthemodelinyourservingenvironment(seeRule#37).Machinelearninghasanelementofunpredictability,somakesurethatyouhavetestsforthecodeforcreatingexamplesintrainingandserving,andthatyoucanloadanduseafixedmodelduringserving.Also,itisimportanttounderstandyourdata:seePracticalAdviceforAnalysisofLarge,ComplexDataSets.

Oftenwecreateapipelinebycopyinganexistingpipeline(i.e.cargocultprogramming),andtheoldpipelinedropsdatathatweneedforthenewpipeline.Forexample,thepipelineforGooglePlusWhat’sHotdropsolderposts(becauseitistryingtorankfreshposts).ThispipelinewascopiedtouseforGooglePlusStream,whereolderpostsarestillmeaningful,butthepipelinewasstilldroppingoldposts.Anothercommonpatternistoonlylogdatathatwasseenbytheuser.Thus,thisdataisuselessifwewanttomodelwhyaparticularpostwasnotseenbytheuser,becauseallthenegativeexampleshavebeendropped.AsimilarissueoccurredinPlay.WhileworkingonPlayAppsHome,anewpipelinewascreatedthatalsocontainedexamplesfromtwootherlandingpages(PlayGamesHomeandPlayHomeHome)withoutanyfeaturetodisambiguatewhereeachexamplecamefrom.

Usuallytheproblemsthatmachinelearningistryingtosolvearenotcompletelynew.Thereisanexistingsystemforranking,orclassifying,orwhateverproblemyouaretryingtosolve.Thismeansthatthereareabunchofrulesandheuristics.Thesesameheuristicscangiveyoualiftwhentweakedwithmachinelearning.Yourheuristicsshouldbeminedforwhateverinformationtheyhave,fortworeasons.First,thetransitiontoamachinelearnedsystemwillbesmoother.Second,usuallythoserulescontainalotoftheintuitionaboutthesystemyoudon’twanttothrowaway.Therearefourwaysyoucanuseanexistingheuristic:

Ingeneral,practicegoodalertinghygiene,suchasmakingalertsactionableandhavingadashboardpage.

Howmuchdoesperformancedegradeifyouhaveamodelthatisadayold?Aweekold?Aquarterold?Thisinformationcanhelpyoutounderstandtheprioritiesofyourmonitoring.Ifyoulose10%ofyourrevenueifthemodelisnotupdatedforaday,itmakessensetohaveanengineerwatchingitcontinuously.Mostadservingsystemshavenewadvertisementstohandleeveryday,andmustupdatedaily.Forinstance,iftheMLmodelforGooglePlaySearchisnotupdated,itcanhaveanimpactonrevenueinunderamonth.SomemodelsforWhat’sHotinGooglePlushavenopostidentifierintheirmodelsotheycanexportthesemodelsinfrequently.Othermodelsthathavepostidentifiersareupdatedmuchmorefrequently.Alsonoticethatfreshnesscanchangeovertime,especiallywhenfeaturecolumnsareaddedorremovedfromyourmodel.

Manymachinelearningsystemshaveastagewhereyouexportthemodeltoserving.Ifthereisanissuewithanexportedmodel,itisauser­facingissue.Ifthereisanissuebefore,thenitisatrainingissue,anduserswillnotnotice.Dosanitychecksrightbeforeyouexportthemodel.Specifically,makesurethatthemodel’sperformanceisreasonableonheldoutdata.Or,ifyouhavelingeringconcernswiththedata,don’texportamodel.ManyteamscontinuouslydeployingmodelschecktheareaundertheROCcurve(orAUC)beforeexporting.Issuesaboutmodelsthathaven’tbeenexportedrequireane­mailalert,butissuesonauser­facingmodelmayrequireapage.Sobettertowaitandbesurebeforeimpactingusers.

Thisisaproblemthatoccursmoreformachinelearningsystemsthanforotherkindsofsystems.Supposethataparticulartablethatisbeingjoinedisnolongerbeingupdated.Themachinelearningsystemwilladjust,andbehaviorwillcontinuetobereasonablygood,decayinggradually.Sometimestablesarefoundthatweremonthsoutofdate,andasimplerefreshimprovedperformancemorethananyotherlaunchthatquarter!Forexample,thecoverageofafeaturemaychangeduetoimplementationchanges:forexampleafeaturecolumncouldbepopulatedin90%oftheexamples,andsuddenlydropto60%oftheexamples.Playoncehadatablethatwasstalefor6months,andrefreshingthetablealonegaveaboostof2%ininstallrate.Ifyoutrackstatisticsofthedata,aswellasmanuallyinspectthedataonoccasion,youcanreducethesekindsoffailures.*

Ifthesystemislarge,andtherearemanyfeaturecolumns,knowwhocreatedorismaintainingeachfeaturecolumn.Ifyoufindthatthepersonwhounderstandsafeaturecolumnisleaving,makesurethatsomeonehastheinformation.Althoughmanyfeaturecolumnshavedescriptivenames,it'sgoodtohaveamoredetaileddescriptionofwhatthefeatureis,whereitcamefrom,andhowitisexpectedtohelp.

Youhavemanymetrics,ormeasurementsaboutthesystemthatyoucareabout,butyourmachinelearningalgorithmwilloftenrequireasingleobjective,anumberthatyouralgorithmis“trying”tooptimize.Idistinguishherebetweenobjectivesandmetrics:ametricisanynumberthatyoursystemreports,whichmayormaynotbeimportant.SeealsoRule#2.

Youwanttomakemoney,makeyourusershappy,andmaketheworldabetterplace.Therearetonsofmetricsthatyoucareabout,andyoushouldmeasurethemall(seeRule#2).However,earlyinthemachinelearningprocess,youwillnoticethemallgoingup,eventhosethatyoudonotdirectlyoptimize.Forinstance,supposeyoucareaboutnumberofclicks,timespentonthesite,anddailyactiveusers.Ifyouoptimizefornumberofclicks,youarelikelytoseethetimespentincrease.So,keepitsimpleanddon’tthinktoohardaboutbalancingdifferentmetricswhenyoucanstilleasilyincreaseallthemetrics.Don’ttakethisruletoofarthough:donotconfuseyourobjectivewiththeultimatehealthofthesystem(seeRule#39).And,ifyoufindyourselfincreasingthedirectlyoptimizedmetric,butdecidingnottolaunch,someobjectiverevisionmayberequired.

Oftenyoudon'tknowwhatthetrueobjectiveis.Youthinkyoudobutthenyouasyoustareatthedataandside­-by-sideanalysisofyouroldsystemandnewMLsystem,yourealizeyouwanttotweakit.Further,differentteammembersoftencan'tagreeonthetrueobjective.TheMLobjectiveshouldbesomethingthatiseasytomeasureandisaproxyforthe“true”objective.SotrainonthesimpleMLobjective,andconsiderhavinga"policylayer"ontopthatallowsyoutoaddadditionallogic(hopefullyverysimplelogic)todothefinalranking.

Theeasiestthingtomodelisauserbehaviorthatisdirectlyobservedandattributabletoanactionofthesystem:

Avoidmodelingindirecteffectsatfirst:

Finally,don’ttrytogetthemachinelearningtofigureout:

Theseareallimportant,butalsoincrediblyhard.Instead,useproxies:iftheuserishappy,theywillstayonthesitelonger.Iftheuserissatisfied,theywillvisitagaintomorrow.Insofaraswell­beingandcompanyhealthisconcerned,humanjudgementisrequiredtoconnectanymachinelearnedobjectivetothenatureoftheproductyouaresellingandyourbusinessplan,sowedon’tenduphere.

Linearregression,logisticregression,andPoissonregressionaredirectlymotivatedbyaprobabilisticmodel.Eachpredictionisinterpretableasaprobabilityoranexpectedvalue.Thismakesthemeasiertodebugthanmodelsthatuseobjectives(zero­oneloss,varioushingelosses,etcetera)thattrytodirectlyoptimizeclassificationaccuracyorrankingperformance.Forexample,ifprobabilitiesintrainingdeviatefromprobabilitiespredictedinside­-by-­sidesorbyinspectingtheproductionsystem,thisdeviationcouldrevealaproblem.

Forexample,inlinear,logistic,orPoissonregression,therearesubsetsofthedatawheretheaveragepredictedexpectationequalstheaveragelabel(1­momentcalibrated,orjustcalibrated)3.Ifyouhaveafeaturewhichiseither1or0foreachexample,thenthesetofexampleswherethatfeatureis1iscalibrated.Also,ifyouhaveafeaturethatis1foreveryexample,thenthesetofallexamplesiscalibrated.

Withsimplemodels,itiseasiertodealwithfeedbackloops(seeRule#36&).Often,weusetheseprobabilisticpredictionstomakeadecision:e.g.rankpostsindecreasingexpectedvalue(i.e.probabilityofclick/download/etc.).However,rememberwhenitcomestimetochoosewhichmodeltouse,thedecisionmattersmorethanthelikelihoodofthedatagiventhemodel(seeRule#27).

Qualityrankingisafineart,butspamfilteringisawar.*Thesignalsthatyouusetodeterminehighqualitypostswillbecomeobvioustothosewhouseyoursystem,andtheywilltweaktheirpoststohavetheseproperties.Thus,yourqualityrankingshouldfocusonrankingcontentthatispostedingoodfaith.Youshouldnotdiscountthequalityrankinglearnerforrankingspamhighly.Similarly,“racy”contentshouldbehandledseparatelyfromQualityRanking.Spamfilteringisadifferentstory.Youhavetoexpectthatthefeaturesthatyouneedtogeneratewillbeconstantlychanging.Often,therewillbeobviousrulesthatyouputintothesystem(ifaposthasmorethanthreespamvotes,don’tretrieveit,etcetera).Anylearnedmodelwillhavetobeupdateddaily,ifnotfaster.Thereputationofthecreatorofthecontentwillplayagreatrole.

Atsomelevel,theoutputofthesetwosystemswillhavetobeintegrated.Keepinmind,filteringspaminsearchresultsshouldprobablybemoreaggressivethanfilteringspaminemailmessages.Also,itisastandardpracticetoremovespamfromthetrainingdataforthequalityclassifier.

GoogleResearchBlog-LessonslearnedwhileprotectingGmail

Inthefirstphaseofthelifecycleofamachinelearningsystem,theimportantissueistogetthetrainingdataintothelearningsystem,getanymetricsofinterestinstrumented,andcreateaservinginfrastructure.Afteryouhaveaworkingendtoendsystemwithunitandsystemtestsinstrumented,PhaseIIbegins.

Don’texpectthatthemodelyouareworkingonnowwillbethelastonethatyouwilllaunch,oreventhatyouwilleverstoplaunchingmodels.Thusconsiderwhetherthecomplexityyouareaddingwiththislaunchwillslowdownfuturelaunches.Manyteamshavelaunchedamodelperquarterormoreforyears.Therearethreebasicreasonstolaunchnewmodels:

Regardless,givingamodelabitoflovecanbegood:lookingoverthedatafeedingintotheexamplecanhelpfindnewsignalsaswellasold,brokenones.So,asyoubuildyourmodel,thinkabouthoweasyitistoaddorremoveorrecombinefeatures.Thinkabouthoweasyitistocreateafreshcopyofthepipelineandverifyitscorrectness.Thinkaboutwhetheritispossibletohavetwoorthreecopiesrunninginparallel.Finally,don’tworryaboutwhetherfeature16of35makesitintothisversionofthepipeline.You’llgetitnextquarter.

Thismightbeacontroversialpoint,butitavoidsalotofpitfalls.Firstofall,let’sdescribewhatalearnedfeatureis.Alearnedfeatureisafeaturegeneratedeitherbyanexternalsystem(suchasanunsupervisedclusteringsystem)orbythelearneritself(e.g.viaafactoredmodelordeeplearning).Bothofthesecanbeuseful,buttheycanhavealotofissues,sotheyshouldnotbeinthefirstmodel.Ifyouuseanexternalsystemtocreateafeature,rememberthatthesystemhasitsownobjective.Theexternalsystem'sobjectivemaybeonlyweaklycorrelatedwithyourcurrentobjective.Ifyougrabasnapshotoftheexternalsystem,thenitcanbecomeoutofdate.Ifyouupdatethefeaturesfromtheexternalsystem,thenthemeaningsmaychange.Ifyouuseanexternalsystemtoprovideafeature,beawarethattheyrequireagreatdealofcare.Theprimaryissuewithfactoredmodelsanddeepmodelsisthattheyarenon­-convex.Thus,thereisnoguaranteethatanoptimalsolutioncanbeapproximatedorfound,andthelocalminimafoundoneachiterationcanbedifferent.Thisvariationmakesithardtojudgewhethertheimpactofachangetoyoursystemismeaningfulorrandom.Bycreatingamodelwithoutdeepfeatures,youcangetanexcellentbaselineperformance.Afterthisbaselineisachieved,youcantrymoreesotericapproaches.

Oftenamachinelearningsystemisasmallpartofamuchbiggerpicture.Forexample,ifyouimagineapostthatmightbeusedinWhat’sHot,manypeoplewillplus­-one,re-­share,orcommentonapostbeforeitisevershowninWhat’sHot.Ifyouprovidethosestatisticstothelearner,itcanpromotenewpoststhatithasnodataforinthecontextitisoptimizing.YouTubeWatchNextcouldusenumberofwatches,orco­-watches(countsofhowmanytimesonevideowaswatchedafteranotherwaswatched)fromYouTubesearch.Youcanalsouseexplicituserratings.Finally,ifyouhaveauseractionthatyouareusingasalabel,seeingthatactiononthedocumentinadifferentcontextcanbeagreatfeature.Allofthesefeaturesallowyoutobringnewcontentintothecontext.Notethatthisisnotaboutpersonalization:figureoutifsomeonelikesthecontentinthiscontextfirst,thenfigureoutwholikesitmoreorless.

Withtonsofdata,itissimplertolearnmillionsofsimplefeaturesthanafewcomplexfeatures.Identifiersofdocumentsbeingretrievedandcanonicalizedqueriesdonotprovidemuchgeneralization,butalignyourrankingwithyourlabelsonheadqueries..Thus,don’tbeafraidofgroupsoffeatureswhereeachfeatureappliestoaverysmallfractionofyourdata,butoverallcoverageisabove90%.Youcanuseregularizationtoeliminatethefeaturesthatapplytotoofewexamples.

Thereareavarietyofwaystocombineandmodifyfeatures.MachinelearningsystemssuchasTensorFlowallowyoutopre­processyourdatathroughtransformations.Thetwomoststandardapproachesare“discretizations”and“crosses”.

Discretizationconsistsoftakingacontinuousfeatureandcreatingmanydiscretefeaturesfromit.Consideracontinuousfeaturesuchasage.Youcancreateafeaturewhichis1whenageislessthan18,anotherfeaturewhichis1whenageisbetween18and35,etcetera.Don’toverthinktheboundariesofthesehistograms:basicquantileswillgiveyoumostoftheimpact.Crossescombinetwoormorefeaturecolumns.Afeaturecolumn,inTensorFlow'sterminology,isasetofhomogenousfeatures,(e.g.{male,female},{US,Canada,Mexico},etcetera).Acrossisanewfeaturecolumnwithfeaturesin,forexample,{male,female}×{US,Canada,Mexico}.Thisnewfeaturecolumnwillcontainthefeature(male,Canada).IfyouareusingTensorFlowandyoutellTensorFlowtocreatethiscrossforyou,this(male,Canada)featurewillbepresentinexamplesrepresentingmaleCanadians.Notethatittakesmassiveamountsofdatatolearnmodelswithcrossesofthree,four,ormorebasefeaturecolumns.

Crossesthatproduceverylargefeaturecolumnsmayoverfit.Forinstance,imaginethatyouaredoingsomesortofsearch,andyouhaveafeaturecolumnwithwordsinthequery,andyouhaveafeaturecolumnwithwordsinthedocument.Youcancombinethesewithacross,butyouwillendupwithalotoffeatures(seeRule#21).Whenworkingwithtexttherearetwoalternatives.Themostdraconianisadotproduct.Adotproductinitssimplestformsimplycountsthenumberofcommonwordsbetweenthequeryandthedocument.Thisfeaturecanthenbediscretized.Anotherapproachisanintersection:thus,wewillhaveafeaturewhichispresentifandonlyiftheword“pony”isinthedocumentandthequery,andanotherfeaturewhichispresentifandonlyiftheword“the”isinthedocumentandthequery.

Therearefascinatingstatisticallearningtheoryresultsconcerningtheappropriatelevelofcomplexityforamodel,butthisruleisbasicallyallyouneedtoknow.Ihavehadconversationsinwhichpeopleweredoubtfulthatanythingcanbelearnedfromonethousandexamples,orthatyouwouldeverneedmorethan1millionexamples,becausetheygetstuckinacertainmethodoflearning.Thekeyistoscaleyourlearningtothesizeofyourdata:

Statisticallearningtheoryrarelygivestightbounds,butgivesgreatguidanceforastartingpoint.Intheend,useRule#28todecidewhatfeaturestouse.

Unusedfeaturescreatetechnicaldebt.Ifyoufindthatyouarenotusingafeature,andthatcombiningitwithotherfeaturesisnotworking,thendropitoutofyourinfrastructure.Youwanttokeepyourinfrastructurecleansothatthemostpromisingfeaturescanbetriedasfastaspossible.Ifnecessary,someonecanalwaysaddbackyourfeature.Keepcoverageinmindwhenconsideringwhatfeaturestoaddorkeep.Howmanyexamplesarecoveredbythefeature?Forexample,ifyouhavesomepersonalizationfeatures,butonly8%ofyourusershaveanypersonalizationfeatures,itisnotgoingtobeveryeffective.Atthesametime,somefeaturesmaypunchabovetheirweight.Forexample,ifyouhaveafeaturewhichcoversonly1%ofthedata,but90%oftheexamplesthathavethefeaturearepositive,thenitwillbeagreatfeaturetoadd.

Beforegoingontothethirdphaseofmachinelearning,itisimportanttofocusonsomethingthatisnottaughtinanymachinelearningclass:howtolookatanexistingmodel,andimproveit.Thisismoreofanartthanascience,andyetthereareseveralanti-­patternsthatithelpstoavoid.

Thisisperhapstheeasiestwayforateamtogetboggeddown.Whiletherearealotofbenefitstofish-fooding(usingaprototypewithinyourteam)anddog-fooding(usingaprototypewithinyourcompany),employeesshouldlookatwhethertheperformanceiscorrect.Whileachangewhichisobviouslybadshouldnotbeused,anythingthatlooksreasonablynearproductionshouldbetestedfurther,eitherbypayinglaypeopletoanswerquestionsonacrowdsourcingplatform,orthroughaliveexperimentonrealusers.Therearetworeasonsforthis.Thefirstisthatyouaretooclosetothecode.Youmaybelookingforaparticularaspectoftheposts,oryouaresimplytooemotionallyinvolved(e.g.confirmationbias).Thesecondisthatyourtimeistoovaluable.Considerthecostof9engineerssittinginaonehourmeeting,andthinkofhowmanycontractedhumanlabelsthatbuysonacrowdsourcingplatform.

Ifyoureallywanttohaveuserfeedback,useuserexperiencemethodologies.Createuserpersonas(onedescriptionisinBillBuxton’sDesigningSketchingUserExperiences)earlyinaprocessanddousabilitytesting(onedescriptionisinSteveKrug’sDon’tMakeMeThink)later.Userpersonasinvolvecreatingahypotheticaluser.Forinstance,ifyourteamisallmale,itmighthelptodesigna35­-yearoldfemaleuserpersona(completewithuserfeatures),andlookattheresultsitgeneratesratherthan10resultsfor25­-40yearoldmales.Bringinginactualpeopletowatchtheirreactiontoyoursite(locallyorremotely)inusabilitytestingcanalsogetyouafreshperspective.

GoogleResearchBlog-Howtomeasuretranslationqualityinyouruserinterfaces

Oneoftheeasiest,andsometimesmostusefulmeasurementsyoucanmakebeforeanyusershavelookedatyournewmodelistocalculatejusthowdifferentthenewresultsarefromproduction.Forinstance,ifyouhavearankingproblem,runbothmodelsonasampleofqueriesthroughtheentiresystem,andlookatthesizeofthesymmetricdifferenceoftheresults(weightedbyrankingposition).Ifthedifferenceisverysmall,thenyoucantellwithoutrunninganexperimentthattherewillbelittlechange.Ifthedifferenceisverylarge,thenyouwanttomakesurethatthechangeisgood.Lookingoverquerieswherethesymmetricdifferenceishighcanhelpyoutounderstandqualitativelywhatthechangewaslike.Makesure,however,thatthesystemisstable.Makesurethatamodelwhencomparedwithitselfhasalow(ideallyzero)symmetricdifference.

Yourmodelmaytrytopredictclick­-through-­rate.However,intheend,thekeyquestioniswhatyoudowiththatprediction.Ifyouareusingittorankdocuments,thenthequalityofthefinalrankingmattersmorethanthepredictionitself.Ifyoupredicttheprobabilitythatadocumentisspamandthenhaveacutoffonwhatisblocked,thentheprecisionofwhatisallowedthroughmattersmore.Mostofthetime,thesetwothingsshouldbeinagreement:whentheydonotagree,itwilllikelybeonasmallgain.Thus,ifthereissomechangethatimprovesloglossbutdegradestheperformanceofthesystem,lookforanotherfeature.Whenthisstartshappeningmoreoften,itistimetorevisittheobjectiveofyourmodel.

Supposethatyouseeatrainingexamplethatthemodelgot“wrong”.Inaclassificationtask,thiscouldbeafalsepositiveorafalsenegative.Inarankingtask,itcouldbeapairwhereapositivewasrankedlowerthananegative.Themostimportantpointisthatthisisanexamplethatthemachinelearningsystemknowsitgotwrongandwouldliketofixifgiventheopportunity.Ifyougivethemodelafeaturethatallowsittofixtheerror,themodelwilltrytouseit.Ontheotherhand,ifyoutrytocreateafeaturebaseduponexamplesthesystemdoesn’tseeasmistakes,thefeaturewillbeignored.Forinstance,supposethatinPlayAppsSearch,someonesearchesfor“freegames”.Supposeoneofthetopresultsisalessrelevantgagapp.Soyoucreateafeaturefor“gagapps”.However,ifyouaremaximizingnumberofinstalls,andpeopleinstallagagappwhentheysearchforfreegames,the“gagapps”featurewon’thavetheeffectyouwant.

Onceyouhaveexamplesthatthemodelgotwrong,lookfortrendsthatareoutsideyourcurrentfeatureset.Forinstance,ifthesystemseemstobedemotinglongerposts,thenaddpostlength.Don’tbetoospecificaboutthefeaturesyouadd.Ifyouaregoingtoaddpostlength,don’ttrytoguesswhatlongmeans,justaddadozenfeaturesandtheletmodelfigureoutwhattodowiththem(seeRule#21).Thatistheeasiestwaytogetwhatyouwant.

Somemembersofyourteamwillstarttobefrustratedwithpropertiesofthesystemtheydon’tlikewhicharen’tcapturedbytheexistinglossfunction.Atthispoint,theyshoulddowhateverittakestoturntheirgripesintosolidnumbers.Forexample,iftheythinkthattoomany“gagapps”arebeingshowninPlaySearch,theycouldhavehumanratersidentifygagapps.(Youcanfeasiblyusehuman-­labelleddatainthiscasebecausearelativelysmallfractionofthequeriesaccountforalargefractionofthetraffic.)Ifyourissuesaremeasurable,thenyoucanstartusingthemasfeatures,objectives,ormetrics.Thegeneralruleis“measurefirst,optimizesecond”.

Imaginethatyouhaveanewsystemthatlooksateverydoc_idandexact_query,andthencalculatestheprobabilityofclickforeverydocforeveryquery.YoufindthatitsbehaviorisnearlyidenticaltoyourcurrentsysteminbothsidebysidesandA/Btesting,sogivenitssimplicity,youlaunchit.However,younoticethatnonewappsarebeingshown.Why?Well,sinceyoursystemonlyshowsadocbasedonitsownhistorywiththatquery,thereisnowaytolearnthatanewdocshouldbeshown.

Theonlywaytounderstandhowsuchasystemwouldworklong­termistohaveittrainonlyondataacquiredwhenthemodelwaslive.Thisisverydifficult.

Training­-servingskewisadifferencebetweenperformanceduringtrainingandperformanceduringserving.Thisskewcanbecausedby:

WehaveobservedproductionmachinelearningsystemsatGooglewithtraining-­servingskewthatnegativelyimpactsperformance.Thebestsolutionistoexplicitlymonitoritsothatsystemanddatachangesdon’tintroduceskewunnoticed.