Thefollowingisalistoftext-basedfileformatsandcommandlinetoolsformanipulatingeach.
ContentsAwk-likeAwkPOSIXcommandsSQL-basedtoolsOthertoolsCSVJSONXML,HTMLYAML,TOMLConfigurationfiles/etc/hostsINIMultipleformatsLogfilesTemplatingforstructuredtextBonusround:CLIsforsingle-filedatabasesLicenseDisclosureAwk-likeToolsthatworkwithlinesoffieldsseparatedbydelimitersbutdonotnecessarilysupportCSVfieldquoting.
AwkAwkisaPOSIX-standardcommandlinetoolandprogramminglanguage.IfyouuseLinux,macOS,oraBSD,youalmostcertainlyhaveitinstalled.SeebelowforWindows.
Ifyoualreadyknowhowtoprogram,thenawkmanpageisagreatwaytolearnAwkquickly.Whatyoulearnfromitwillapplytootherimplementationsondifferentplatforms.ReaditfirstifyoufeeloverwhelmedbythesheersizeoftheGNUAwkmanual.Awk.infoarchive—anextensiveresourceonAwk.AWKVsNAWKVsGAWK—acomparisonoffeaturespresentindifferentimplementations.busybox-w32includesafullimplementationofPOSIXAwkandothertoolslikesedinasingleWindowsexecutable.GNUAwk5binariesforWindowsbyEZWinPorts.POSIXcommandsNameDescriptioncommSelectthelinescommontotwosortedfilesorthelinescontainedinonlyoneofthem.(Manual:man1commonyoursystem,GNU,FreeBSD.)cutSelectportionsofeachlineinoneormorefiles.(Manual:man1cut,GNU,FreeBSD.)grepSelectthelinesthatmatchordonotmatchapatternfromoneormorefiles.(Manual:man1grep,GNU,FreeBSD.)joinTaketwofilessortedbyacommonfieldandjointheirlinesonthevalueofthatfield.Lineswithvaluesthatdonotappearintheotherfilearediscarded.(Manual:man1join,GNU,FreeBSD.)pasteCombineseveralconsecutivelinesinatextfileintoone.(Manual:man1paste,GNU,FreeBSD.)sortSortlinesbykeyfields.(Manual:man1sort,GNU,FreeBSD.)uniqFindorremoverepeatedlines.(Manual:man1uniq,GNU,FreeBSD.)OthertoolsNameDescriptioncsvquoteSeetheCSVsection.GNUdatamashPerformstatisticaloperationsontextinput.HawkTransformtextfromthecommand-lineusingHaskellexpressions.rqSeetheJSONsection.CSVCSV,TSV,andotherdelimiter-separatedvalueformats.Toolsbelongonthislistiftheysupportfieldquoting.
NameandlinkDescriptioncsv-nix-toolsList*nixsysteminformationsuchasenvironmentvariables,files,processes,networkconnections,usersasCSV.Manipulateandpretty-printCSV.ExecuteCSVrowsascommands.csv2mdConvertCSVtoMarkdowntables.csv2htmlConvertCSVtoHTMLtables.csvfakerGenerateCSVfileswithfakedata.Supportsdifferenttypesoffakedataindifferentlocales:names,cities,jobs,emailaddresses,andothers.csvfix(unofficialmirror)Amultitool.Compare,filter,normalize,split,andvalidateCSVfiles.Reorder,remove,split,andmergefields.Convertdatabetweenfixed-width,multi-line,XML,andDSVformat.GenerateSQLstatements.csvkitcsvkitisasuiteofcommand-linetoolsforconvertingtoandworkingwithCSV:convert,clean,cut,grep,join,sort,stack,format,render,query,analyze,etc.csvquoteTransformCSVtoandfromaformatprocessablewithAwk-liketools.csvtkSearch,sample,cut,join,transpose,andsortCSV/TSVfiles.Renamecolumns.Replacefieldsandgeneratenewfiendsfromexistingfields.Plotdataasvectororrasterhistogramsandbox,line,andscatterplots.ConvertCSVtoMarkdown.ConvertXLSXtoCSV.SplitXLSXsheets.CSVtoTableConvertCSVtoasearchableandsortableHTMLtable.daselSeetheJSONsection.GraphtageSeetheJSONsection.jp(sgreben)Plotdata.SeetheJSONsection.MarioSeetheJSONsection.MCMD(M-Command)Select,sample,cut,join,sort,reformat,andgenerateCSVfiles.Containsalargesetofcommands.Millersed,awk,cut,joinandsortforname-indexeddatasuchasCSVandtabularJSON.pawkProcesstextwithAwk-likepatterns,butPythoncode.rowsAPythonlibrarywithaCLI.Convertbetweenanumberoffileformatsfortabulardata:CSV,XLS,XLSX,ODS,andothers.Querythedata(viaSQLite).Combinetables.Generateschemas.rqSeetheJSONsection.scrubcsvRemovebadlinesfromaCSVfileandnormalizetherest.WritteninRust.tabAnon-Turing-completestaticallytypedprogramminglanguagefordataprocessing.AnalternativetoAwk.teipSelectfields,characterranges,orregularexpressionmatchesfromthestandardinput.Replacethemwiththeoutputofacommand.eBay'sTSVutilitiesFiltering,statistics,sampling,joinsandotheroperationsonTSVfiles.Highperformance,especiallygoodforlargedatasets.WritteninD.tvViewdelimitedfilesintheterminal.VisiDataExploreinteractivelydatainTSV,CSV,XLS,XLSX,HDF5,JSON,andotherformats.Introduction.xsvIndex,slice,analyze,split,andjoinCSVfiles.SQL-basedtoolsSeethebigcomparisontable.Itcovers
AlaSQLCLIcsvqcsvsqlfsqlqRBQLrowsSqawk(dbohdan)sqawk(tjunier)SquawktermsqltrdsqltextqlJSONNameandlinkDescriptionclconfSeetheYAMLsection.daselQueryandupdatedatastructuresfromthecommandline.Comparabletojq/yqbutsupportsJSON,TOML,YAML,andXML.Staticbinariesavailableforreleases.fxRunarbitraryJavaScriptonJSONinput.Standalonebinariesavailable.gojqApureGoimplementationofjq(seebelow).SupportsYAMLinputandoutput.GraphtageCompareandmergetree-likestructuressemantically.SupportsJSON,JSON5,XML,HTML,YAML,andCSV.CanbeusedasaPythonlibrary.gronConvertJSONtoandfromflat,greppablelistsof"path=value"statements.JCConverttheoutputofstandardcommandlinetoolstoJSON.jelloQueryJSONandJSONLineswithPythoncode.Outputtheresultinaline-basedformatsuitableforcreatingBasharrays.Generateagrep-ableschema.jetConvertbetweenandqueryJSON,Clojure'sedn,andTransit.jfqQueryandtransformJSONwiththeJSONatalanguage.jidExploreJSONinteractivelywithfilteringquerieslikejq.jiqExploreJSONinteractivelywithjq.Requiresjq.jjQueryandmodifyvaluesinJSONorJSONLineswithakeypath.jlQueryandmanipulateJSONusingatinyfunctionallanguage.joCreateJSONobjectsfromtheshell.jp(jmespath)QueryJSONwithJMESPath.jp(sgreben)PlotJSONandCSVdataintheterminal.Supportsdifferentkindsofplots:barcharts,linecharts,scatterplots,histograms,andheatmaps.jplotPlotreal-timeJSONdataintheterminal(workswithterminalssupportinggraphicrendering).jqCreateandmanipulateJSONwithafunctional(asin"functionalprogramming")DSL.CanconvertJSONtootherformats.jqlCreateandmanipulateJSONwithaLisp-syntaxDSL.jtblFormatJSONorJSONLinesasaplain-texttable.jtcCreate,manipulate,search,validateJSONwithpathexpressions.CanbeusedasaC++14library.emutoCLItoolsimilartojq.CreateandmanipulateJSONandotherfiles.CanbecompiledtoJavaScript.jshonCreateandmanipulateJSONusinggetopt-stylecommand-lineoptions.json2ConvertJSONtoandfromflat,greppablelistsof"path=value"statements.Modeledafterxml2.jsonaxeCreateandmanipulateJSONwithaPython-basedDSL.Inspiredbyjq.jsonRunarbitraryJavaScriptonJSONinput.json-tableConvertnestedJSONintoCSVorTSVforprocessingintheshell.json.tool(Python3docs)Validateandpretty-printJSON.ThismoduleispartofthestandardlibraryofPython2/3andislikelytobeavailablewhereverPythonisinstalled.jsonwatchTrackchangesinJSONdatafromthecommandline.Workslikewatch-d.lobarExploreJSONinteractivelyorprocessitinbatchwithawrapperforlodash.chain().AnalternativetojqwithaJavaScriptsyntax.MarioManipulateandconvertbetweenCSV,JSON,YAML,TOML,andXMLwithPythoncode.qpysonQueryandmanipulateJSONwithPython.query-jsonAfasterjqimplementationwritteninReasonNative(OCaml).quicktypeInfertheunderlyingmodeloftheJSONandoutputastypesforvariousprogramminglanguagesorJSONSchema.CLIandWebUI.ramda-cliManipulateJSONwiththeRamdafunctionallibrary,andeitherLiveScriptorJavaScriptsyntax.RecordStreamCreate,manipulate,andoutputastreamofrecords,orJSONobjects.CanretrieverecordsfromanSQLdatabase,MongoDB,Atomfeeds,XML,andothersources.rqConvertbetweenApacheAvro,CBOR,CSV,JSON,MessagePack,ProtocolBuffers,TOML,YAML,andAwk-styleplaintext.validjsonValidateorpretty-printJSON.VisiDataExploredatainteractivelydata.SeetheDSV/Othertoolssection.XML,HTMLNameandlinkDescriptiondaselSupportsXML.SeetheJSONsection.GraphtageSeetheJSONsection.hredQueryXMLandHTMLwithaquerylanguagebasedonCSSselectors.html-xml-utilsAnumberofsimpleutilities(likehxcopy,hxpipe,hxunent,hxselect)formanipulatingHTMLandXMLfilesfromW3C.WritteninC,quiteold-fashioned,butstillrelevantandmaintained.MarioSupportsXML.SeetheJSONsection.pupQueryHTMLpageswithCSSselectors.Staticbinariesavailableforreleases.Inspiredbyjq.SaxonQueryXMLandHTMLdatawithXPath.Documentation.sml2ConvertbetweenXMLandSML,asimplifiedXMLrepresentation.TemmeQueryHTMLwithCSS-likeselectorstoextractJSON.TemmeextendsCSSselectorswithvaluecapturepatterns.tidy-html5Validate,fix,andreformatHTML(5),XHTML,andXMLdocuments.ConvertHTMLtoXHTML.tqQueryHTMLwithCSSselectors.XidelQueryormodifyXMLandHTMLpageswithXPath,XQuery3,andCSSselectors.xml-to-json-fastConvertXMLtoJSON.CanhandleverylargeXMLfiles.xml2ConvertXMLandHTMLtoandfromflat,greppablelistsof"path=value"statements.Sourcecodemirror.xmljsonConvertmultipleandlargeXMLfilestoJSON.WritteninSwift.XMLLintQuery(includingXSLT),validateandreformatXMLdocuments.XMLStarletQuery,modify,andvalidateXMLdocuments.xqjqwrapperforXMLdocuments.xsltprocTransformXMLdocumentsusingXSLTandEXSLT.SeealsoGrepandSedEquivalentforXMLCommandLineProcessingonStackOverflow.YAML,TOMLWithaformatconverterlikeRemarshal(below)youcanuseJSONtoolstoprocessYAMLandTOML,butmakesureyoudonotlosedataintheconversion.
NameandlinkDescriptionclconfMergemultipleconfigfilesandextractvaluesfromthemusingpathstring.SupportsJSONandYAML.CanbeusedasaGolibrary.daselSupportsTOMLandYAML.SeetheJSONsection.gojqSupportsYAML.SeetheJSONsection.GraphtageSupportsYAML.SeetheJSONsection.MarioSupportsYAML.SeetheJSONsection.RemarshalConvertbetweenCBOR,JSON,MessagePack,TOML,andYAML.Validateeachoftheformats.Pretty-printJSON,TOML,andYAML.rqSupportsTOMLandYAML.SeetheJSONsection.shyamlQueryYAML.Canoutputnull-terminatedstringsforuseinshellscripts.validtomlValidateTOML.validyamlValidateorpretty-printYAML.yaml-toolsAsetofCLItoolstomanipulateYAMLfiles(merge,delete,etc...)withcommentpreservation,basedonruamel.yaml.yq(kislyuk)jqwrapperforYAML.yq(mikefarah)Query,modify,andmergeYAML.ConverttoandfromJSON.Configurationfiles/etc/hostsNameandlinkDescriptionhostctlAddandremoveentiresin/etc/hosts.Disable(commentout)andenable(uncomment)entires.Notidempotent.Preservesarbitrarycommentsaboveitssectionofthehostsfile.Workswithgroupsofentriescalled"profiles".hostessAddandremoveentiresin/etc/hosts.Disable(commentout)andenable(uncomment)entires.Checkifahostnameexists.Reformatthehostsfile.ConverttheentriestoJSON.Idempotent.Removesarbitrarycomments.hostsAddandremoveentiresin/etc/hosts.Changeahostname'sIPaddress.Idempotent.Preservesarbitrarycomments.CanbeusedasaTcllibrary.ININameandlinkPlatformLicenseDescriptioncfgetAnywithPython2.x?GNUGPLv2+Retrievepropertiesasshellscriptcommandstosetthecorrespondingvariables(with--dumpexports).Retrieveproperties'valuesasplaintext.SubstitutevaluesfromanINIfileinanAutoconf-styletemplate.Supportsplug-ins.Chokesonsectionnamesandkeyswithspaces.confgetLinux,FreeBSDTwo-clauseBSDRetrievepropertiesandsectionsasshellscriptcommandstosetthecorrespondingvariables.Retrieveproperties'valuesasplaintext.Checkforexistenceofproperties.Listsections.Findvaluesthatmatchapattern.Read-only.crudiniAnywithPython2.xGNUGPLv2RetrievepropertiesandsectionsasINIfragmentsorshellscriptcommandstosetthecorrespondingvariables.Retrieveproperties'valuesasplaintext.Setproperties.Removepropertiesandsections.Createemptysections.MergeINIfiles.Changesfilesinplace.inicompWindows,*nixApache2.0CompareINI(andalsoWindows.reg)files.IniFile(DOSversion)Windows(x86,x86-64),MS-DOSClosed-sourcefreewareRetrievepropertiesandsectionsasbatchfilecommandstosetthecorrespondingvariables.Setproperties.Removepropertiesandsections.Changesfilesinplace.initoolLinux,FreeBSD,WindowsMITRetrievepropertiesandsectionsasINIfragments.Retrieveproperties'valuesasplaintext.Setproperties.Checkforexistenceofpropertiesandsections.Removepropertiesandsections.OutputstheupdatedINIfile.MultipleformatsNameandlinkDescriptionAugeasQueryandmodifyanumberoffileformats.NotalloftheformatsareequallywellsupportedbyAugeasandforsomeonlyalimitedsubsetofallvalidfilescanbeparsed.ElektraQueryandmodifyconfigurationfiles.SharesAugeas'limitationswhenitcomestoapplication-specificconfigurationfiles(itusesthesamelenses),buthasbettersupportforgenericformatssuchasJSONandINI.LogfilesNameandlinkDescriptionSquawkQueryApacheandNginxlogfiles.SeetheSQL-basedtoolcomparison.lnavQueryandwatchlogfiles.Hasbatchandinteractivemode.SupportedformatsincludetheCommonLogFormat,CUPSpage_log,syslog,strace,andgenerictimestampedmessages.CanperformSQLqueries.TemplatingforstructuredtextListedbelowarerestrictedprogramminglanguageinterpretersandtemplatingtoolsthatproducestructuredtextoutput.Theyaregenerallyintendedtoremoverepetitioninconfigurationfiles.Theyaredistinctfromunstructedtemplatingtoolslikethejinja2CLIprogram,whichshouldnotbeaddedtothistable.
NameandlinkOutputformatTuring-complete?SyntaxI/ODescriptionCUEJSONYes?ExtendedJSON?AconstraintlanguageforJSONconfigurationdata.CangenerateandvalidatesJSON.DhallJSON,YAMLNoHaskell-inspiredLimitedtoimportinglibrariesfromfilesandHTTP(S)URLs(withprotectionagainstleakingyourdatatotheserver)Astatically-typedfunctionalconfigurationlanguage.Hasastandardformattingtool.jkJSON,YAML,plaintextYesJavaScriptDiskI/OGenerateconfigurationfilesusingJavaScript(V8VM).JsonnetJSON,INI,XML,YAML,plaintextYesExtendedJSONNoneAfunctionalconfigurationlanguage.Hasastandardformattingtool.rjsoneJSON,YAMLNo?ExtendedJSONNoneACLItoolfortheJSON-etemplatinglanguage.yttYAMLNoYAML/PythonhybridNone?AtemplatingtoolforYAMLbuiltupontheStarlarkconfigurationlanguage.Bonusround:CLIsforsingle-filedatabasesNameandlinkDescriptionFileformatFirebirdFirebirdisaFOSSdatabasethatcanbeusedfromasinglefile,likeSQLite."isqlisaprogramthatallowstheusertoissuearbitrarySQLcommands".BinaryFsdbAflat-filedatabaseforshellscripting.Text-based,TSVwithaheaderor"key:value"GNURecutils"[A]setoftoolsandlibrariestoaccesshuman-editable,plaintextdatabasescalledrecfiles."Text-based,roughly"key:value"SDB"[A]simplestringkey/valuedatabasebasedondjb'scdbdiskstorageandsupportsJSONandarraysintrospection."Binarysqlite3(1)"[A]simplecommand-lineutility[...]thatallowstheusertomanuallyenterandexecuteSQLstatementsagainstanSQLitedatabase."BinaryLicenseThecontentsofthisdocumentislicensedundertheCreativeCommonsAttribution4.0InternationalLicense.Bycontributingyouagreetoreleaseyourcontributionunderthislicense.
Disclosurecsv2html,hosts,Sqawk,jsonwatch,Remarshalandinitoolaredevelopedbythecuratorofthisdocument.
评论