Deep-Learning-Papers-Reading-Roadmap

IfyouareanewcomertotheDeepLearningarea,thefirstquestionyoumayhaveis"WhichpapershouldIstartreadingfrom?"

HereisareadingroadmapofDeepLearningpapers!

Theroadmapisconstructedinaccordancewiththefollowingfourguidelines:

Youwillfindmanypapersthatarequitenewbutreallyworthreading.

Iwouldcontinueaddingpaperstothisroadmap.

[0]Bengio,Yoshua,IanJ.Goodfellow,andAaronCourville."Deeplearning."AnMITPressbook.(2015).[html](DeepLearningBible,youcanreadthisbookwhilereadingfollowingpapers.)⭐⭐⭐⭐⭐

[1]LeCun,Yann,YoshuaBengio,andGeoffreyHinton."Deeplearning."Nature521.7553(2015):436-444.[pdf](ThreeGiants'Survey)⭐⭐⭐⭐⭐

[2]Hinton,GeoffreyE.,SimonOsindero,andYee-WhyeTeh."Afastlearningalgorithmfordeepbeliefnets."Neuralcomputation18.7(2006):1527-1554.[pdf](DeepLearningEve)⭐⭐⭐

[3]Hinton,GeoffreyE.,andRuslanR.Salakhutdinov."Reducingthedimensionalityofdatawithneuralnetworks."Science313.5786(2006):504-507.[pdf](Milestone,Showthepromiseofdeeplearning)⭐⭐⭐

[4]Krizhevsky,Alex,IlyaSutskever,andGeoffreyE.Hinton."Imagenetclassificationwithdeepconvolutionalneuralnetworks."Advancesinneuralinformationprocessingsystems.2012.[pdf](AlexNet,DeepLearningBreakthrough)⭐⭐⭐⭐⭐

[5]Simonyan,Karen,andAndrewZisserman."Verydeepconvolutionalnetworksforlarge-scaleimagerecognition."arXivpreprintarXiv:1409.1556(2014).[pdf](VGGNet,NeuralNetworksbecomeverydeep!)⭐⭐⭐

[6]Szegedy,Christian,etal."Goingdeeperwithconvolutions."ProceedingsoftheIEEEConferenceonComputerVisionandPatternRecognition.2015.[pdf](GoogLeNet)⭐⭐⭐

[7]He,Kaiming,etal."Deepresiduallearningforimagerecognition."arXivpreprintarXiv:1512.03385(2015).[pdf](ResNet,Veryverydeepnetworks,CVPRbestpaper)⭐⭐⭐⭐⭐

[8]Hinton,Geoffrey,etal."Deepneuralnetworksforacousticmodelinginspeechrecognition:Thesharedviewsoffourresearchgroups."IEEESignalProcessingMagazine29.6(2012):82-97.[pdf](Breakthroughinspeechrecognition)⭐⭐⭐⭐

[9]Graves,Alex,Abdel-rahmanMohamed,andGeoffreyHinton."Speechrecognitionwithdeeprecurrentneuralnetworks."2013IEEEinternationalconferenceonacoustics,speechandsignalprocessing.IEEE,2013.[pdf](RNN)⭐⭐⭐

[10]Graves,Alex,andNavdeepJaitly."TowardsEnd-To-EndSpeechRecognitionwithRecurrentNeuralNetworks."ICML.Vol.14.2014.[pdf]⭐⭐⭐

[11]Sak,Haşim,etal."Fastandaccuraterecurrentneuralnetworkacousticmodelsforspeechrecognition."arXivpreprintarXiv:1507.06947(2015).[pdf](GoogleSpeechRecognitionSystem)⭐⭐⭐

[12]Amodei,Dario,etal."Deepspeech2:End-to-endspeechrecognitioninenglishandmandarin."arXivpreprintarXiv:1512.02595(2015).[pdf](BaiduSpeechRecognitionSystem)⭐⭐⭐⭐

[13]W.Xiong,J.Droppo,X.Huang,F.Seide,M.Seltzer,A.Stolcke,D.Yu,G.Zweig"AchievingHumanParityinConversationalSpeechRecognition."arXivpreprintarXiv:1610.05256(2016).[pdf](State-of-the-artinspeechrecognition,Microsoft)⭐⭐⭐⭐

Afterreadingabovepapers,youwillhaveabasicunderstandingoftheDeepLearninghistory,thebasicarchitecturesofDeepLearningmodel(includingCNN,RNN,LSTM)andhowdeeplearningcanbeappliedtoimageandspeechrecognitionissues.Thefollowingpaperswilltakeyouin-depthunderstandingoftheDeepLearningmethod,DeepLearningindifferentareasofapplicationandthefrontiers.Isuggestthatyoucanchoosethefollowingpapersbasedonyourinterestsandresearchdirection.

#2DeepLearningMethod

[14]Hinton,GeoffreyE.,etal."Improvingneuralnetworksbypreventingco-adaptationoffeaturedetectors."arXivpreprintarXiv:1207.0580(2012).[pdf](Dropout)⭐⭐⭐

[15]Srivastava,Nitish,etal."Dropout:asimplewaytopreventneuralnetworksfromoverfitting."JournalofMachineLearningResearch15.1(2014):1929-1958.[pdf]⭐⭐⭐

[16]Ioffe,Sergey,andChristianSzegedy."Batchnormalization:Acceleratingdeepnetworktrainingbyreducinginternalcovariateshift."arXivpreprintarXiv:1502.03167(2015).[pdf](AnoutstandingWorkin2015)⭐⭐⭐⭐

[17]Ba,JimmyLei,JamieRyanKiros,andGeoffreyE.Hinton."Layernormalization."arXivpreprintarXiv:1607.06450(2016).[pdf](UpdateofBatchNormalization)⭐⭐⭐⭐

[18]Courbariaux,Matthieu,etal."BinarizedNeuralNetworks:TrainingNeuralNetworkswithWeightsandActivationsConstrainedto+1or−1."[pdf](NewModel,Fast)⭐⭐⭐

[19]Jaderberg,Max,etal."Decoupledneuralinterfacesusingsyntheticgradients."arXivpreprintarXiv:1608.05343(2016).[pdf](InnovationofTrainingMethod,AmazingWork)⭐⭐⭐⭐⭐

[20]Chen,Tianqi,IanGoodfellow,andJonathonShlens."Net2net:Acceleratinglearningviaknowledgetransfer."arXivpreprintarXiv:1511.05641(2015).[pdf](Modifypreviouslytrainednetworktoreducetrainingepochs)⭐⭐⭐

[21]Wei,Tao,etal."NetworkMorphism."arXivpreprintarXiv:1603.01670(2016).[pdf](Modifypreviouslytrainednetworktoreducetrainingepochs)⭐⭐⭐

[22]Sutskever,Ilya,etal."Ontheimportanceofinitializationandmomentumindeeplearning."ICML(3)28(2013):1139-1147.[pdf](Momentumoptimizer)⭐⭐

[23]Kingma,Diederik,andJimmyBa."Adam:Amethodforstochasticoptimization."arXivpreprintarXiv:1412.6980(2014).[pdf](Maybeusedmostoftencurrently)⭐⭐⭐

[24]Andrychowicz,Marcin,etal."Learningtolearnbygradientdescentbygradientdescent."arXivpreprintarXiv:1606.04474(2016).[pdf](NeuralOptimizer,AmazingWork)⭐⭐⭐⭐⭐

[25]Han,Song,HuiziMao,andWilliamJ.Dally."Deepcompression:Compressingdeepneuralnetworkwithpruning,trainedquantizationandhuffmancoding."CoRR,abs/1510.001492(2015).[pdf](ICLRbestpaper,newdirectiontomakeNNrunningfast,DeePhiTechStartup)⭐⭐⭐⭐⭐

[26]Iandola,ForrestN.,etal."SqueezeNet:AlexNet-levelaccuracywith50xfewerparametersand<1MBmodelsize."arXivpreprintarXiv:1602.07360(2016).[pdf](AlsoanewdirectiontooptimizeNN,DeePhiTechStartup)⭐⭐⭐⭐

[27]GloratXavier,BengioYoshua,etal."Understandingthedifficultyoftrainingdeepforwardneuralnetworks."ProceedingsofthethirteenthInternationalConferenceonArtificialIntelligenceandStatistics,PMLR9:249-256,2010.[pdf]⭐⭐⭐⭐

[28]Le,QuocV."Buildinghigh-levelfeaturesusinglargescaleunsupervisedlearning."2013IEEEinternationalconferenceonacoustics,speechandsignalprocessing.IEEE,2013.[pdf](Milestone,AndrewNg,GoogleBrainProject,Cat)⭐⭐⭐⭐

[29]Kingma,DiederikP.,andMaxWelling."Auto-encodingvariationalbayes."arXivpreprintarXiv:1312.6114(2013).[pdf](VAE)⭐⭐⭐⭐

[30]Goodfellow,Ian,etal."Generativeadversarialnets."AdvancesinNeuralInformationProcessingSystems.2014.[pdf](GAN,supercoolidea)⭐⭐⭐⭐⭐

[31]Radford,Alec,LukeMetz,andSoumithChintala."Unsupervisedrepresentationlearningwithdeepconvolutionalgenerativeadversarialnetworks."arXivpreprintarXiv:1511.06434(2015).[pdf](DCGAN)⭐⭐⭐⭐

[32]Gregor,Karol,etal."DRAW:Arecurrentneuralnetworkforimagegeneration."arXivpreprintarXiv:1502.04623(2015).[pdf](VAEwithattention,outstandingwork)⭐⭐⭐⭐⭐

[33]Oord,Aaronvanden,NalKalchbrenner,andKorayKavukcuoglu."Pixelrecurrentneuralnetworks."arXivpreprintarXiv:1601.06759(2016).[pdf](PixelRNN)⭐⭐⭐⭐

[34]Oord,Aaronvanden,etal."ConditionalimagegenerationwithPixelCNNdecoders."arXivpreprintarXiv:1606.05328(2016).[pdf](PixelCNN)⭐⭐⭐⭐

[34]S.Mehrietal.,"SampleRNN:AnUnconditionalEnd-to-EndNeuralAudioGenerationModel."arXivpreprintarXiv:1612.07837(2016).[pdf]⭐⭐⭐⭐⭐

[35]Graves,Alex."Generatingsequenceswithrecurrentneuralnetworks."arXivpreprintarXiv:1308.0850(2013).[pdf](LSTM,verynicegeneratingresult,showthepowerofRNN)⭐⭐⭐⭐

[36]Cho,Kyunghyun,etal."LearningphraserepresentationsusingRNNencoder-decoderforstatisticalmachinetranslation."arXivpreprintarXiv:1406.1078(2014).[pdf](FirstSeq-to-SeqPaper)⭐⭐⭐⭐

[37]Sutskever,Ilya,OriolVinyals,andQuocV.Le."Sequencetosequencelearningwithneuralnetworks."Advancesinneuralinformationprocessingsystems.2014.[pdf](OutstandingWork)⭐⭐⭐⭐⭐

[38]Bahdanau,Dzmitry,KyungHyunCho,andYoshuaBengio."NeuralMachineTranslationbyJointlyLearningtoAlignandTranslate."arXivpreprintarXiv:1409.0473(2014).[pdf]⭐⭐⭐⭐

[39]Vinyals,Oriol,andQuocLe."Aneuralconversationalmodel."arXivpreprintarXiv:1506.05869(2015).[pdf](Seq-to-SeqonChatbot)⭐⭐⭐

[40]Graves,Alex,GregWayne,andIvoDanihelka."Neuralturingmachines."arXivpreprintarXiv:1410.5401(2014).[pdf](BasicPrototypeofFutureComputer)⭐⭐⭐⭐⭐

[41]Zaremba,Wojciech,andIlyaSutskever."ReinforcementlearningneuralTuringmachines."arXivpreprintarXiv:1505.00521362(2015).[pdf]⭐⭐⭐

[42]Weston,Jason,SumitChopra,andAntoineBordes."Memorynetworks."arXivpreprintarXiv:1410.3916(2014).[pdf]⭐⭐⭐

[43]Sukhbaatar,Sainbayar,JasonWeston,andRobFergus."End-to-endmemorynetworks."Advancesinneuralinformationprocessingsystems.2015.[pdf]⭐⭐⭐⭐

[44]Vinyals,Oriol,MeireFortunato,andNavdeepJaitly."Pointernetworks."AdvancesinNeuralInformationProcessingSystems.2015.[pdf]⭐⭐⭐⭐

[45]Graves,Alex,etal."Hybridcomputingusinganeuralnetworkwithdynamicexternalmemory."Nature(2016).[pdf](Milestone,combineabovepapers'ideas)⭐⭐⭐⭐⭐

[46]Mnih,Volodymyr,etal."Playingatariwithdeepreinforcementlearning."arXivpreprintarXiv:1312.5602(2013).[pdf])(FirstPapernameddeepreinforcementlearning)⭐⭐⭐⭐

[47]Mnih,Volodymyr,etal."Human-levelcontrolthroughdeepreinforcementlearning."Nature518.7540(2015):529-533.[pdf](Milestone)⭐⭐⭐⭐⭐

[48]Wang,Ziyu,NandodeFreitas,andMarcLanctot."Duelingnetworkarchitecturesfordeepreinforcementlearning."arXivpreprintarXiv:1511.06581(2015).[pdf](ICLRbestpaper,greatidea)⭐⭐⭐⭐

[49]Mnih,Volodymyr,etal."Asynchronousmethodsfordeepreinforcementlearning."arXivpreprintarXiv:1602.01783(2016).[pdf](State-of-the-artmethod)⭐⭐⭐⭐⭐

[50]Lillicrap,TimothyP.,etal."Continuouscontrolwithdeepreinforcementlearning."arXivpreprintarXiv:1509.02971(2015).[pdf](DDPG)⭐⭐⭐⭐

[51]Gu,Shixiang,etal."ContinuousDeepQ-LearningwithModel-basedAcceleration."arXivpreprintarXiv:1603.00748(2016).[pdf](NAF)⭐⭐⭐⭐

[52]Schulman,John,etal."Trustregionpolicyoptimization."CoRR,abs/1502.05477(2015).[pdf](TRPO)⭐⭐⭐⭐

[53]Silver,David,etal."MasteringthegameofGowithdeepneuralnetworksandtreesearch."Nature529.7587(2016):484-489.[pdf](AlphaGo)⭐⭐⭐⭐⭐

[54]Bengio,Yoshua."DeepLearningofRepresentationsforUnsupervisedandTransferLearning."ICMLUnsupervisedandTransferLearning27(2012):17-36.[pdf](ATutorial)⭐⭐⭐

[55]Silver,DanielL.,QiangYang,andLianghaoLi."LifelongMachineLearningSystems:BeyondLearningAlgorithms."AAAISpringSymposium:LifelongMachineLearning.2013.[pdf](Abriefdiscussionaboutlifelonglearning)⭐⭐⭐

[56]Hinton,Geoffrey,OriolVinyals,andJeffDean."Distillingtheknowledgeinaneuralnetwork."arXivpreprintarXiv:1503.02531(2015).[pdf](Godfather'sWork)⭐⭐⭐⭐

[57]Rusu,AndreiA.,etal."Policydistillation."arXivpreprintarXiv:1511.06295(2015).[pdf](RLdomain)⭐⭐⭐

[58]Parisotto,Emilio,JimmyLeiBa,andRuslanSalakhutdinov."Actor-mimic:Deepmultitaskandtransferreinforcementlearning."arXivpreprintarXiv:1511.06342(2015).[pdf](RLdomain)⭐⭐⭐

[59]Rusu,AndreiA.,etal."Progressiveneuralnetworks."arXivpreprintarXiv:1606.04671(2016).[pdf](OutstandingWork,Anovelidea)⭐⭐⭐⭐⭐

[60]Lake,BrendenM.,RuslanSalakhutdinov,andJoshuaB.Tenenbaum."Human-levelconceptlearningthroughprobabilisticprograminduction."Science350.6266(2015):1332-1338.[pdf](NoDeepLearning,butworthreading)⭐⭐⭐⭐⭐

[61]Koch,Gregory,RichardZemel,andRuslanSalakhutdinov."SiameseNeuralNetworksforOne-shotImageRecognition."(2015)[pdf]⭐⭐⭐

[62]Santoro,Adam,etal."One-shotLearningwithMemory-AugmentedNeuralNetworks."arXivpreprintarXiv:1605.06065(2016).[pdf](Abasicsteptooneshotlearning)⭐⭐⭐⭐

[63]Vinyals,Oriol,etal."MatchingNetworksforOneShotLearning."arXivpreprintarXiv:1606.04080(2016).[pdf]⭐⭐⭐

[64]