Erlemar.github.io Data science portfolio

NowIhaveapersonalwebsite!

ThisportfolioisacompilationofnotebookswhichIcreatedfordataanalysisorforexplorationofmachinelearningalgorithms.Aseparatecategoryisforseparateprojects.

Thisismyownprojectusingimagerecognitionmethodsinpractice.Thisisasite(alsoworksonmobile)whereusercandrawadigit,andmachinelearningmodels(FNNandCNN)willtrytorecognizeit.Afterthanmodelscanusethedrawndigitfortrainingtoimprovetheiraccuracy.Liveversionishere.Thecodecanbefoundhere.

AconversationalchatbotintelegramwhichwascreatedforanhonorassignmentofnlpcoursebyHigherSchoolofEconomics.Themainfunctionalityofthebotistodistinguishtwotypesofquestions(questionsrelatedtoprogrammingandothers)andtheneithergiveananswerortalkusingaconversationalmodel.

AvitodemandpredictionwasacompetitiononKagglewherewetriedtopredictsomethinglikedemandbasedonadscontent.Thiscompetitionwasveryinterestingbecauseithadtabulardata,textsandimages.Ontheotherhandthiswasthereasonthecompetitionwasquitedifficult.Myteamreached131thplaceandgotbronzemedal!Hereisalinktomysolution.

ThiswasaRussianinclassKagglecompetitioninthethirdsessionofODSmlcourse.ItsoundedinterestingandItookpartinitreachinga3rdplace.Hereismykagglekernelwithasolution.

SometimeagoKagglelaunchedabigonlinesurveyforkagglersandnowthisdataispublic.Thereweremultiplechoicequestionsandsomeformsforopenanswers.Surveyreceived23k+respondentsfrom147countries.AsaresultwehaveabigdatasetwithrichinformationondatascientistsusingKaggle.InthiskernelIcompareDSinUSA,Russia,Indiaandothercountries.

DonorsChoose.orgempowerspublicschoolteachersfromacrossthecountrytorequestmuch-neededmaterialsandexperiencesfortheirstudents.DonorsChoose.orgreceiveshundredsofthousandsofprojectproposalseachyearforclassroomprojectsinneedoffunding.ThisisacompetitiononKagglewherepeoplecancreateamachinelearningmodeltohelpthisfundwithauto-approvingofapplications.Prizesaregiventotheauthorswiththemostupvotedkernels.HereismykernelwithextensiveEDA,featureengineeringandbuildingmodels.Thiskernelgot2ndplacebythenumberofvotesandIwonGooglePixelbookforit!

Avitochallengeisaboutpredictingdemandforanonlineadvertisementbasedonitsfulldescription(title,description,images,etc.),itscontext(geographicallywhereitwasposted,similaradsalreadyposted)andhistoricaldemandforsimilaradsinsimilarcontexts.Thecompetitionisinterestingduetomanytypesofdatainitwhichallowstobuildvariousmodels.HereismykernelwithEDA,creatingfeaturesandbuildingmodels.

HomeCreditBankoffersachallengeofcreditscoring.Thereisalotofdataaboutapplicantsandtheirpreviousbehavior.Hereismykernel.

SometimeagoKagglehaslaunchedseveral"remakes"ofoldcompetitions.Itmeansthatdatasetsarethesame,butnowweareofferedanopportunitytosimplyexplorethedataandcreatekernelswithnewmethods.OneofthesecompetitionsissentimentanalysisofRottenTomatoesdatasetwith5classes(negative,somewhatnegative,neutral,somewhatpositive,positive).IhavecreatedakernelwithEDAandmodernNNarchitecture:LSTM-CNN.Currentlythiskernelshowsthe5thresultofleaderboard.

InthiscompetitionReutersprovideuniquedata,whichcan'tbeobtainedoutsideofthiscompetition.Wecanseea10yearsworthofnewsandmarketdataonmanycompanies.Thiscompetitioniskernel-only,whichmeansthateveryonehasthesameamountofcomputationalpowerforthiscompetition.InmykernelIhaveanalysedthedataandshowedtrendsofmarketdata.

Inthiscompetitionwegotananonymizeddataset,lateritwasfoundthatithadacertainstructure.InmykernelItriedtoanalyzethedataandcreatednewfeaturesusingNNmodel.

RStudiohostedthiscompetitiontoprovethatmachinelearningalgorithmscanimpactbusinessandhelpmarketing.InmykernelIdidanextensiveEDAandbuildaninterestingLGBmodel.

ThisdatasetwasprovidedbyTheCenterforPolicingEquity.Theyhopethatkagglerswillhelptocreatebettermodels,findsomeuniqueinsightsandimprovegeo-analytics.InmykernelItrytodosuchthings.

Githubnbviewer

Titanic:MachineLearningfromDisasterisaknowledgecompetitiononKaggle.Manypeoplestartedpracticinginmachinelearningwiththiscompetition,sodidI.Thisisabinaryclassificationproblem:basedoninformationaboutTitanicpassengerswepredictwhethertheysurvivedornot.GeneraldescriptionanddataareavailableonKaggle.Titanicdatasetprovidesinterestingopportunitiesforfeatureengineering.

Githubnbviewer

Ghouls,Goblins,andGhosts...Boo!isaknowledgecompetitiononKaggle.Thisisamultipleclassificationproblem:basedoninformationaboutmonsterswepredicttheirtypes.AfuncompetitionforHalloween.GeneraldescriptionanddataareavailableonKaggle.Thisdatasethaslittlenumberofsamples,socarefulfeatureselectionandmodelensemblearenecessaryforhighaccuracy.

Githubnbviewer

OttoGroupProductClassificationChallengeisaknowledgecompetitiononKaggle.Thisisamultipleclassificationproblem.Basedoninformationaboutproductswepredicttheircategory.GeneraldescriptionanddataareavailableonKaggle.Thedataisobfuscated,sothemainquestionliesintheselectionofthemodelforprediction.

Githubnbviewer

Inrealworlditiscommontomeetdatainwhichsomeclassesaremorecommonandothersarerarer.Incaseofaseriousdisbalancepredictionrareclassescouldbedifficultusingstandardclassificationmethods.InthisnotebookIanalysesuchasituation.Ican'tsharethedata,usedinthisanalysis.

Githubnbviewer

Banksstrivetoincreasetheefficiencyoftheircontactswithcustomers.Oneoftheareaswhichrequirethisisofferingnewproductstoexistingclients(cross-selling).Insteadofofferingnewproductstoallclients,itisagoodideatopredicttheprobabilityofapositiveresponse.Thentheofferscouldbesenttothoseclients,forwhomtheprobabilityofresponseishigherthansomethresholdvalue.InthisnotebookItrytosolvethisproblem.

Githubnbviewer

HousePrices:AdvancedRegressionTechniquesisaknowledgecompetitiononKaggle.Thisisaregressionproblem:basedoninformationabouthouseswepredicttheirprices.GeneraldescriptionanddataareavailableonKaggle.Thedatasethasalotoffeaturesandmanymissingvalues.Thisgivesinterestingpossibilitiesforfeaturetransformationanddatavisualization.

Githubnbviewer

LoanPredictionisaknowledgeandlearninghackathononAnalyticsvidhya.DreamHousingFinancecompanydealsinhomeloans.Companywantstoautomatetheloaneligibilityprocess(realtime)basedoncustomerdetailprovidedwhilefillingonlineapplicationform.Basedoncustomer'sinformationwepredictwhethertheyshouldreceivealoanornot.GeneraldescriptionanddataareavailableonAnalyticsvidhya.

Githubnbviewer

CaterpillarTubePricingisacompetitiononKaggle.Thisisaregressionproblem:basedoninformationabouttubeassemblieswepredicttheirprices.GeneraldescriptionanddataareavailableonKaggle.Datasetconsistsofmanyfiles,sothereisanadditionalchallengeincombiningthedatasndselectingthefeatures.

Githubnbviewer

BagofWordsMeetsBagsofPopcornisasentimentalanalysisproblem.Basedontextsofreviewswepredictwhethertheyarepositiveornegative.GeneraldescriptionanddataareavailableonKaggle.Thedataprovidedconsistsofrawreviewsandclass(1or2),sothemainpartiscleaningthetexts.

Githubnbviewer

Naturallanguageprocessinginmachinelearninghelpstoaccomplishavarietyoftasks,oneofwhichisextractinginformationfromtexts.ThisnotebookisanoverviewofseveraltextexplorationmethodsusingEnglishtranslationofJapaneselightnovel"Fate/Zero"asanexample.

Githubnbviewer

ThisnotebookshowshowanewtextcanbegeneratedbasedonagivencorpususinganideaofMarkovchains.Istartwithsimplefirst-orderchainsandwitheachstepimprovemodeltogeneratebettertext.

Githubnbviewer

Thisnotebookshowshowtextcanbesummarizedchoosingseveralmostimportantsentencesfromthetext.Iexplorevariousmethodsofdoingthisbasedonanewsarticle.

Githubnbviewer

Clusteringisanapproachtounsupervisedmachinelearning.ClusteringwithKMeansisoneofalgorithmsofclustering.inthisnotebookI'lldemonstratehowitworks.Datausedisaboutvarioustypesofseedsandtheirparameters.Itisavailablehere.

Githubnbviewer

Thisisasimpleexampleoffeedforwardneuralnetworkwithregularization.ItisbasedonAndrewNg'slecturesonCoursera.IuseddatafromKaggle'schallenge"Ghouls,Goblins,andGhosts...Boo!",itisavailablehere.

Githubnbviewer

Ihaveadatasetwithtelematicinformationabout10carsdrivingduringoneday.Ivisualisedata,searchforinsightsandanalysethebehaviorofeachdriver.Ican'tsharethedata,buthereisthenotebook.Iwanttonoticethatfoliummapcan'tberenderedbynativegithub,butnbviewer.jupytercandoit.

Githubnbviewer

Recommendersaresystems,whichpredictratingsofusersforitems.ThereareseveralapproachestobuildsuchsystemsandoneofthemisCollaborativeFiltering.Thisnotebookshowsseveralexamplesofcollaborativefilteringalgorithms.