Preamble
ThisrepositorycotaisthelectureslidesadcoursedescriptiofortheDeepNaturalLaguageProcessigcourseofferediHilaryTerm2017attheUiversityofOxford.
Thisisaadvacedcourseoaturallaguageprocessig.AutomaticallyprocessigaturallaguageiputsadproduciglaguageoutputsisakeycompoetofArtificialGeeralItelligece.TheambiguitiesadoiseiheretihumacommuicatioredertraditioalsymbolicAItechiquesieffectiveforrepresetigadaalysiglaguagedata.Recetlystatisticaltechiquesbasedoeuraletworkshaveachievedaumberofremarkablesuccessesiaturallaguageprocessigleadigtoagreatdealofcommercialadacademiciterestithefield
Thisisaappliedcoursefocussigorecetadvacesiaalysigadgeeratigspeechadtextusigrecurreteuraletworks.Weitroducethemathematicaldefiitiosoftherelevatmachielearigmodelsadderivetheirassociatedoptimisatioalgorithms.ThecoursecoversarageofapplicatiosofeuraletworksiNLPicludigaalysiglatetdimesiositext,trascribigspeechtotext,traslatigbetweelaguages,adaswerigquestios.Thesetopicsareorgaiseditothreehighlevelthemesformigaprogressiofromuderstadigtheuseofeuraletworksforsequetiallaguagemodellig,touderstadigtheiruseascoditioallaguagemodelsfortrasductiotasks,adfiallytoapproachesemployigthesetechiquesicombiatiowithothermechaismsforadvacedapplicatios.ThroughoutthecoursethepracticalimplemetatioofsuchmodelsoCPUadGPUhardwareisalsodiscussed.
ThiscourseisorgaisedbyPhilBlusomaddeliveredipartershipwiththeDeepMidNaturalLaguageResearchGroup.
LecturersPhilBlusom(OxfordUiversityadDeepMid)ChrisDyer(CaregieMelloUiversityadDeepMid)EdwardGrefestette(DeepMid)KarlMoritzHerma(DeepMid)AdrewSeior(DeepMid)WagLig(DeepMid)JeremyAppleyard(NVIDIA)TAsYaisAssaelYishuMiaoBredaShilligfordJaBuysTimetablePracticalsGroup1-Moday,9:00-11:00(Weeks2-8),60.05ThomBuildigGroup2-Friday,16:00-18:00(Weeks2-8),Room379Practical1:word2vecPractical2:textclassificatioPractical3:recurreteuraletworksfortextclassificatioadlaguagemodelligPractical4:opepracticalLecturesPublicLecturesareheldiLectureTheatre1oftheMathsIstitute,oTuesdaysadThursdays(exceptweek8),16:00-18:00(HilaryTermWeeks1,3-8).
LectureMaterials1.Lecture1a-Itroductio[PhilBlusom]ThislectureitroducesthecourseadmotivateswhyitisiterestigtostudylaguageprocessigusigDeepLearigtechiques.
[slides][video]
2.Lecture1b-DeepNeuralNetworksAreOurFrieds[WagLig]Thislecturerevisesbasicmachielearigcoceptsthatstudetsshouldkowbeforeembarkigothiscourse.
[slides][video]
3.Lecture2a-WordLevelSematics[EdGrefestette]Wordsarethecoremeaigbeariguitsilaguage.RepresetigadlearigthemeaigsofwordsisafudametaltaskiNLPadithislecturethecoceptofawordembeddigisitroducedasapracticaladscalablesolutio.
[slides][video]
ReadigEmbeddigsBasicsFirth,JohR."Asyopsisofliguistictheory,1930-1955."(1957):1-32.Curra,JamesRichard."Fromdistributioaltosematicsimilarity."(2004).Collobert,Roa,etal."Naturallaguageprocessig(almost)fromscratch."JouralofMachieLearigResearch12.Aug(2011):2493-2537.Mikolov,Tomas,etal."Distributedrepresetatiosofwordsadphrasesadtheircompositioality."Advacesieuraliformatioprocessigsystems.2013.DatasetsadVisualisatioFikelstei,Lev,etal."Placigsearchicotext:Thecoceptrevisited."Proceedigsofthe10thiteratioalcofereceoWorldWideWeb.ACM,2001.Hill,Felix,RoiReichart,adAaKorhoe."Simlex-999:Evaluatigsematicmodelswith(geuie)similarityestimatio."ComputatioalLiguistics(2016).Maate,Lauresvader,adGeoffreyHito."Visualizigdatausigt-SNE."JouralofMachieLearigResearch9.Nov(2008):2579-2605.BlogpostsDeepLearig,NLP,adRepresetatios,ChristopherOlah.VisualizigTopTweepswitht-SNE,iJavascript,AdrejKarpathy.FurtherReadigHerma,KarlMoritz,adPhilBlusom."Multiligualmodelsforcompositioaldistributedsematics."arXivprepritarXiv:1404.4641(2014).Levy,Omer,adYoavGoldberg."Neuralwordembeddigasimplicitmatrixfactorizatio."Advacesieuraliformatioprocessigsystems.2014.Levy,Omer,YoavGoldberg,adIdoDaga."Improvigdistributioalsimilaritywithlessoslearedfromwordembeddigs."TrasactiosoftheAssociatioforComputatioalLiguistics3(2015):211-225.Lig,Wag,etal."Two/TooSimpleAdaptatiosofWord2VecforSytaxProblems."HLT-NAACL.2015.4.Lecture2b-OverviewofthePracticals[ChrisDyer]Thislecturemotivatesthepracticalsegmetofthecourse.
[slides][video]
5.Lecture3-LaguageModelligadRNNsPart1[PhilBlusom]LaguagemodelligisimportattaskofgreatpracticaluseimayNLPapplicatios.Thislectureitroduceslaguagemodellig,icludigtraditioal-grambasedapproachesadmorecotemporaryeuralapproaches.IparticularthepopularRecurretNeuralNetwork(RNN)laguagemodelisitroducedaditsbasictraiigadevaluatioalgorithmsdescribed.
[slides][video]
ReadigTextbookDeepLearig,Chapter10.BlogsTheUreasoableEffectiveessofRecurretNeuralNetworks,AdrejKarpathy.TheureasoableeffectiveessofCharacter-levelLaguageModels,YoavGoldberg.Explaiigadillustratigorthogoaliitializatioforrecurreteuraletworks,StepheMerity.6.Lecture4-LaguageModelligadRNNsPart2[PhilBlusom]ThislecturecotiuesofromthepreviousoeadcosiderssomeoftheissuesivolvediproducigaeffectiveimplemetatioofaRNNlaguagemodel.Thevaishigadexplodiggradietproblemisdescribedadarchitecturalsolutios,suchasLogShortTermMemory(LSTM),areitroduced.
[slides][video]
ReadigTextbookDeepLearig,Chapter10.Vaishiggradiets,LSTMsetc.Othedifficultyoftraiigrecurreteuraletworks.Pascauetal.,ICML2013.LogShort-TermMemory.HochreiteradSchmidhuber,NeuralComputatio1997.LearigPhraseRepresetatiosusigRNNEcoderDecoderforStatisticalMachieTraslatio.Choetal,EMNLP2014.Blog:UderstadigLSTMNetworks,ChristopherOlah.DealigwithlargevocabulariesAscalablehierarchicaldistributedlaguagemodel.MihadHito,NIPS2009.Afastadsimplealgorithmfortraiigeuralprobabilisticlaguagemodels.MihadTeh,ICML2012.OUsigVeryLargeTargetVocabularyforNeuralMachieTraslatio.Jeaetal.,ACL2015.ExplorigtheLimitsofLaguageModelig.Jozefowiczetal.,arXiv2016.EfficietsoftmaxapproximatioforGPUs.Graveetal.,arXiv2016.NotesoNoiseCotrastiveEstimatioadNegativeSamplig.Dyer,arXiv2014.PragmaticNeuralLaguageModelligiMachieTraslatio.BaltescuadBlusom,NAACL2015RegularisatioaddropoutATheoreticallyGroudedApplicatioofDropoutiRecurretNeuralNetworks.GaladGhahramai,NIPS2016.Blog:UcertaityiDeepLearig,YariGal.OtherstuffRecurretHighwayNetworks.Zillyetal.,arXiv2016.CapacityadTraiabilityiRecurretNeuralNetworks.Collisetal.,arXiv2016.7.Lecture5-TextClassificatio[KarlMoritzHerma]Thislecturediscussestextclassificatio,begiigwithbasicclassifiers,suchasNaiveBayes,adprogressigthroughtoRNNsadCovolutioNetworks.
[slides][video]
ReadigRecurretCovolutioalNeuralNetworksforTextClassificatio.Laietal.AAAI2015.ACovolutioalNeuralNetworkforModelligSeteces,Kalchbreeretal.ACL2014.Sematiccompositioalitythroughrecursivematrix-vector,Socheretal.EMNLP2012.Blog:UderstadigCovolutioNeuralNetworksForNLP,DeyBritz.Thesis:DistributioalRepresetatiosforCompositioalSematics,Herma(2014).8.Lecture6-DeepNLPoNvidiaGPUs[JeremyAppleyard]ThislectureitroducesGraphicalProcessigUits(GPUs)asaalterativetoCPUsforexecutigDeepLearigalgorithms.ThestregthsadweakessesofGPUsarediscussedaswellastheimportaceofuderstadighowmemorybadwidthadcomputatioimpactthroughputforRNNs.
[slides][video]
ReadigOptimizigPerformaceofRecurretNeuralNetworksoGPUs.Appleyardetal.,arXiv2016.PersistetRNNs:StashigRecurretWeightsO-Chip,Diamosetal.,ICML2016EfficietsoftmaxapproximatioforGPUs.Graveetal.,arXiv2016.9.Lecture7-CoditioalLaguageModels[ChrisDyer]Ithislectureweextedthecoceptoflaguagemodelligtoicorporateprioriformatio.BycoditioigaRNNlaguagemodeloaiputrepresetatiowecageeratecotextuallyrelevatlaguage.Thisverygeeralideacabeappliedtotrasducesequecesitoewsequecesfortaskssuchastraslatioadsummarisatio,orimagesitocaptiosdescribigtheircotet.
[slides][video]
ReadigRecurretCotiuousTraslatioModels.KalchbreeradBlusom,EMNLP2013SequecetoSequeceLearigwithNeuralNetworks.Sutskeveretal.,NIPS2014MultimodalNeuralLaguageModels.Kirosetal.,ICML2014ShowadTell:ANeuralImageCaptioGeerator.Viyalsetal.,CVPR201510.Lecture8-GeeratigLaguagewithAttetio[ChrisDyer]ThislectureitroducesoeofthemostimportatadifluecialmechaismsemployediDeepNeuralNetworks:Attetio.AttetioaugmetsrecurretetworkswiththeabilitytocoditioospecificpartsoftheiputadiskeytoachievighighperformaceitaskssuchasMachieTraslatioadImageCaptioig.
[slides][video]
ReadigNeuralMachieTraslatiobyJoitlyLearigtoAligadTraslate.Bahdaauetal.,ICLR2015Show,Atted,adTell:NeuralImageCaptioGeeratiowithVisualAttetio.Xuetal.,ICML2015Icorporatigstructuralaligmetbiasesitoaattetioaleuraltraslatiomodel.Cohetal.,NAACL2016BLEU:aMethodforAutomaticEvaluatioofMachieTraslatio.Papieietal,ACL200211.Lecture9-SpeechRecogitio(ASR)[AdrewSeior]AutomaticSpeechRecogitio(ASR)isthetaskoftrasducigrawaudiosigalsofspokelaguageitotexttrascriptios.ThistalkcoversthehistoryofASRmodels,fromGaussiaMixturestoattetioaugmetedRNNs,thebasicliguisticsofspeech,adthevariousiputadoutputrepresetatiosfrequetlyemployed.
[slides][video]
12.Lecture10-TexttoSpeech(TTS)[AdrewSeior]Thislectureitroducesalgorithmsforcovertigwrittelaguageitospokelaguage(TexttoSpeech).TTSistheiverseprocesstoASR,buttherearesomeimportatdifferecesithemodelsapplied.HerewereviewtraditioalTTSmodels,adthecovermorereceteuralapproachessuchasDeepMid'sWaveNetmodel.
[slides][video]
13.Lecture11-QuestioAswerig[KarlMoritzHerma][slides][video]
ReadigTeachigmachiestoreadadcomprehed.Hermaetal.,NIPS2015DeepLearigforAswerSeteceSelectio.Yuetal.,NIPSDeepLearigWorkshop201414.Lecture12-Memory[EdGrefestette][slides][video]
ReadigHybridcomputigusigaeuraletworkwithdyamicexteralmemory.Gravesetal.,Nature2016ReasoigaboutEtailmetwithNeuralAttetio.Rocktäscheletal.,ICLR2016Learigtotrasducewithuboudedmemory.Grefestetteetal.,NIPS2015Ed-to-EdMemoryNetworks.Sukhbaataretal.,NIPS201515.Lecture13-LiguisticKowledgeiNeuralNetworks[slides][video]
PiazzaWewillbeusigPiazzatofacilitateclassdiscussiodurigthecourse.Ratherthaemailigquestiosdirectly,IecourageyoutopostyourquestiosoPiazzatobeasweredbyyourfellowstudets,istructors,adlecturers.Howeverdopleasedootethatallthelecturersforthiscoursearevoluteerigtheirtimeadmayotalwaysbeavailabletogivearespose.
Fidourclasspageat:https://piazza.com/ox.ac.uk/witer2017/dlpht2017/home
AssessmetTheprimaryassessmetforthiscoursewillbeatake-homeassigmetissuedattheedoftheterm.Thisassigmetwillaskquestiosdrawigothecoceptsadmodelsdiscussedithecourse,aswellasfromselectedresearchpublicatios.Theatureofthequestioswillicludeaalysigmathematicaldescriptiosofmodelsadproposigextesios,improvemets,orevaluatiostosuchmodels.Theassigmetmayalsoaskstudetstoreadspecificresearchpublicatiosaddiscusstheirproposedalgorithmsithecotextofthecourse.Iaswerigquestiosstudetswillbeexpectedtobothpresetcoheretwritteargumetsaduseappropriatemathematicalformulae,adpossiblypseudo-code,toillustrateaswers.
Thepracticalcompoetofthecoursewillbeassesseditheusualway.
AckowledgemetsThiscoursewouldothavebeepossiblewithoutthesupportofDeepMid,TheUiversityofOxfordDepartmetofComputerSciece,Nvidia,adthegeerousdoatioofGPUresourcesfromMicrosoftAzure.
评论