FLEX(1) FLEX(1) NAME flex−fastlexicalanalyzergenerator SYNOPSIS flex[−bcdfhilnpstvwBFILTV78+?−C[aefFmr]−ooutput−Pprefix−Sskeleton] [−−help−−version][filename...] OVERVIEW Thismanualdescribesflex,atoolforgeneratingprogramsthatperformpattern-matchingontext. Themanualincludesbothtutorialandreferencesections: Description abriefoverviewofthetool SomeSimpleExamples FormatOfTheInputFile Patterns theextendedregularexpressionsusedbyflex HowTheInputIsMatched therulesfordeterminingwhathasbeenmatched Actions howtospecifywhattodowhenapatternismatched TheGeneratedScanner detailsregardingthescannerthatflexproduces; howtocontroltheinputsource StartConditions introducingcontextintoyourscanners,and managing"mini-scanners" MultipleInputBuffers howtomanipulatemultipleinputsources;howto scanfromstringsinsteadoffiles End-of-fileRules specialrulesformatchingtheendoftheinput MiscellaneousMacros asummaryofmacrosavailabletotheactions ValuesAvailableToTheUser asummaryofvaluesavailabletotheactions InterfacingWithYacc connectingflexscannerstogetherwithyaccparsers Options flexcommand-lineoptions,andthe"%option" directive PerformanceConsiderations Version2.5 April1995 1 FLEX(1) FLEX(1) howtomakeyourscannergoasfastaspossible GeneratingC++Scanners the(experimental)facilityforgeneratingC++ scannerclasses IncompatibilitiesWithLexAndPOSIX howflexdiffersfromAT&TlexandthePOSIXlex standard Diagnostics thoseerrormessagesproducedbyflex(orscanners itgenerates)whosemeaningsmightnotbeapparent Files filesusedbyflex Deficiencies/Bugs knownproblemswithflex SeeAlso otherdocumentation,relatedtools Author includescontactinformation DESCRIPTION flexisatoolforgeneratingscanners:programswhichrecognizedlexicalpatternsintext.flex readsthegiveninputfiles,oritsstandardinputifnofilenamesaregiven,foradescriptionofa scannertogenerate.ThedescriptionisintheformofpairsofregularexpressionsandCcode, calledrules.flexgeneratesasoutputaCsourcefile,lex.yy.c,whichdefinesaroutineyylex(). Thisfileiscompiledandlinkedwiththe−lfllibrarytoproduceanexecutable.Whentheexecutableisrun,itanalyzesitsinputforoccurrencesoftheregularexpressions.Wheneveritfinds one,itexecutesthecorrespondingCcode. SOMESIMPLEEXAMPLES Firstsomesimpleexamplestogettheflavorofhowoneusesflex.Thefollowingflexinputspecifiesascannerwhichwheneveritencountersthestring"username"willreplaceitwiththeuser’s loginname: %% usernameprintf("%s",getlogin()); Bydefault,anytextnotmatchedbyaflexscanneriscopiedtotheoutput,sotheneteffectof thisscanneristocopyitsinputfiletoitsoutputwitheachoccurrenceof"username"expanded. Inthisinput,thereisjustonerule."username"isthepatternandthe"printf"istheaction. The"%%"marksthebeginningoftherules. Here’sanothersimpleexample: intnumlines=0,numchars=0; %% \n++numlines;++numchars; .++numchars; Version2.5 April1995 2 FLEX(1) FLEX(1) %% main() { yylex(); printf("#oflines=%d,#ofchars=%d\n", numlines,numchars); } Thisscannercountsthenumberofcharactersandthenumberoflinesinitsinput(itproducesno outputotherthanthefinalreportonthecounts).Thefirstlinedeclarestwoglobals,"numlines" and"numchars",whichareaccessiblebothinsideyylex()andinthemain()routinedeclared afterthesecond"%%".Therearetworules,onewhichmatchesanewline("\n")andincrements boththelinecountandthecharactercount,andonewhichmatchesanycharacterotherthana newline(indicatedbythe"."regularexpression). Asomewhatmorecomplicatedexample: /*scannerforatoyPascal-likelanguage*/ %{ /*needthisforthecalltoatof()below*/ #include<math.h> %} DIGIT[0-9] ID[a-z][a-z0-9]* %% {DIGIT}+{ printf("Aninteger:%s(%d)\n",yytext, atoi(yytext)); } {DIGIT}+"."{DIGIT}*{ printf("Afloat:%s(%g)\n",yytext, atof(yytext)); } if|then|begin|end|procedure|function{ printf("Akeyword:%s\n",yytext); } {ID}printf("Anidentifier:%s\n",yytext); "+"|"-"|"*"|"/"printf("Anoperator:%s\n",yytext); "{"[ˆ }\n]*"}"/*eatupone-linecomments*/ [\t\n]+/*eatupwhitespace*/ .printf("Unrecognizedcharacter:%s\n",yytext); %% Version2.5 April1995 3 FLEX(1) FLEX(1) main(argc,argv) intargc; char**argv; { ++argv,--argc;/*skipoverprogramname*/ if(argc>0) yyin=fopen(argv[0],"r"); else yyin=stdin; yylex(); } ThisisthebeginningsofasimplescannerforalanguagelikePascal.Itidentifiesdifferenttypes oftokensandreportsonwhatithasseen. Thedetailsofthisexamplewillbeexplainedinthefollowingsections. FORMATOFTHEINPUTFILE Theflexinputfileconsistsofthreesections,separatedbyalinewithjust%%init: definitions %% rules %% usercode Thedefinitionssectioncontainsdeclarationsofsimplenamedefinitionstosimplifythescanner specification,anddeclarationsofstartconditions,whichareexplainedinalatersection. Namedefinitionshavetheform: namedefinition The"name"isawordbeginningwithaletteroranunderscore(’’)followedbyzeroormoreletters,digits,’’,or’-’(dash).Thedefinitionistakentobeginatthefirstnon-white-spacecharac-terfollowingthenameandcontinuingtotheendoftheline.Thedefinitioncansubsequentlybe referredtousing"{name}",whichwillexpandto"(definition)".Forexample, DIGIT[0-9] ID[a-z][a-z0-9]* defines"DIGIT"tobearegularexpressionwhichmatchesasingledigit,and"ID"tobearegular expressionwhichmatchesaletterfollowedbyzero-or-moreletters-or-digits.Asubsequentrefer-enceto {DIGIT}+"."{DIGIT}* isidenticalto ([0-9])+"."([0-9])* andmatchesone-or-moredigitsfollowedbya’.’followedbyzero-or-moredigits. Therulessectionoftheflexinputcontainsaseriesofrulesoftheform: patternaction Version2.5 April1995 4 FLEX(1) FLEX(1) wherethepatternmustbeunindentedandtheactionmustbeginonthesameline. Seebelowforafurtherdescriptionofpatternsandactions. Finally,theusercodesectionissimplycopiedtolex.yy.cverbatim.Itisusedforcompanionroutineswhichcallorarecalledbythescanner.Thepresenceofthissectionisoptional;ifitismissing,thesecond%%intheinputfilemaybeskipped,too. Inthedefinitionsandrulessections,anyindentedtextortextenclosedin%{and%}iscopied verbatimtotheoutput(withthe%{}’sremoved).The%{}’smustappearunindentedonlinesby themselves. Intherulessection,anyindentedor%{}textappearingbeforethefirstrulemaybeusedto declarevariableswhicharelocaltothescanningroutineand(afterthedeclarations)codewhichis tobeexecutedwheneverthescanningroutineisentered.Otherindentedor%{}textintherule sectionisstillcopiedtotheoutput,butitsmeaningisnotwell-definedanditmaywellcausecom-pile-timeerrors(thisfeatureispresentforPOSIXcompliance;seebelowforothersuchfeatures). Inthedefinitionssection(butnotintherulessection),anunindentedcomment(i.e.,alinebeginningwith"/*")isalsocopiedverbatimtotheoutputuptothenext"*/". PATTERNS Thepatternsintheinputarewrittenusinganextendedsetofregularexpressions.Theseare: xmatchthecharacter’x’ .anycharacter(byte)exceptnewline [xyz]a"characterclass";inthiscase,thepattern matcheseitheran’x’,a’y’,ora’z’ [abj-oZ]a"characterclass"witharangeinit;matches an’a’,a’b’,anyletterfrom’j’through’o’, ora’Z’ [ˆ A-Z]a"negatedcharacterclass",i.e.,anycharacter butthoseintheclass.Inthiscase,any characterEXCEPTanuppercaseletter. [ˆ A-Z\n]anycharacterEXCEPTanuppercaseletteror anewline r*zeroormorer’s,whererisanyregularexpression r+oneormorer’s r?zerooroner’s(thatis,"anoptionalr") r{2,5}anywherefromtwotofiver’s r{2,}twoormorer’s r{4}exactly4r’s {name}theexpansionofthe"name"definition (seeabove) "[xyz]\"foo" theliteralstring:[xyz]"foo \XifXisan’a’,’b’,’f’,’n’,’r’,’t’,or’v’, thentheANSI-Cinterpretationof\x. Otherwise,aliteral’X’(usedtoescape operatorssuchas’*’) \0aNULcharacter(ASCIIcode0) \123thecharacterwithoctalvalue123 \x2athecharacterwithhexadecimalvalue2a (r)matchanr;parenthesesareusedtooverride precedence(seebelow) rstheregularexpressionrfollowedbythe Version2.5 April1995 5 FLEX(1) FLEX(1) regularexpressions;called"concatenation" r|seitheranrorans r/sanrbutonlyifitisfollowedbyans.The textmatchedbysisincludedwhendetermining whetherthisruleisthe"longestmatch", butisthenreturnedtotheinputbefore theactionisexecuted.Sotheactiononly seesthetextmatchedbyr.Thistype ofpatterniscalledtrailingcontext". (Therearesomecombinationsofr/sthatflex cannotmatchcorrectly;seenotesinthe Deficiencies/Bugssectionbelowregarding "dangeroustrailingcontext".) ˆ ranr,butonlyatthebeginningofaline(i.e., whichjuststartingtoscan,orrightaftera newlinehasbeenscanned). r$anr,butonlyattheendofaline(i.e.,just beforeanewline).Equivalentto"r/\n". Notethatflex’snotionof"newline"isexactly whatevertheCcompilerusedtocompileflex interprets’\n’as;inparticular,onsomeDOS systemsyoumusteitherfilterout\r’sinthe inputyourself,orexplicitlyuser/\r\nfor"r$". <s>ranr,butonlyinstartconditions(see belowfordiscussionofstartconditions) <s1,s2,s3>r same,butinanyofstartconditionss1, s2,ors3 <*>ranrinanystartcondition,evenanexclusiveone. <<EOF>>anend-of-file <s1,s2><<EOF>> anend-of-filewheninstartconditions1ors2 Notethatinsideofacharacterclass,allregularexpressionoperatorslosetheirspecialmeaning exceptescape(’\’)andthecharacterclassoperators,’-’,’]’,and,atthebeginningoftheclass,’ˆ ’. Theregularexpressionslistedabovearegroupedaccordingtoprecedence,fromhighestprecedence atthetoptolowestatthebottom.Thosegroupedtogetherhaveequalprecedence.Forexample, foo|bar* isthesameas (foo)|(ba(r*)) sincethe’*’operatorhashigherprecedencethanconcatenation,andconcatenationhigherthan Version2.5 April1995 6 FLEX(1) FLEX(1) alternation(’|’).Thispatternthereforematcheseitherthestring"foo"orthestring"ba"followedbyzero-or-morer’s.Tomatch"foo"orzero-or-more"bar"’s,use: foo|(bar)* andtomatchzero-or-more"foo"’s-or-"bar"’s: (foo|bar)* Inadditiontocharactersandrangesofcharacters,characterclassescanalsocontaincharacter classexpressions.Theseareexpressionsenclosedinside[:and:]delimiters(whichthemselves mustappearbetweenthe’[’and’]’ofthecharacterclass;otherelementsmayoccurinsidethe characterclass,too).Thevalidexpressionsare: [:alnum:][:alpha:][:blank:] [:cntrl:][:digit:][:graph:] [:lower:][:print:][:punct:] [:space:][:upper:][:xdigit:] TheseexpressionsalldesignateasetofcharactersequivalenttothecorrespondingstandardC isXXXfunction.Forexample,[:alnum:]designatesthosecharactersforwhichisalnum() returnstrue-i.e.,anyalphabeticornumeric.Somesystemsdon’tprovideisblank(),soflex defines[:blank:]asablankoratab. Forexample,thefollowingcharacterclassesareallequivalent: [[:alnum:]] [[:alpha:][:digit:] [[:alpha:]0-9] [a-zA-Z0-9] Ifyourscanneriscase-insensitive(the−iflag),then[:upper:]and[:lower:]areequivalentto [:alpha:]. Somenotesonpatterns: -Anegatedcharacterclasssuchastheexample"[ˆ A-Z]"abovewillmatchanewlineunless "\n"(oranequivalentescapesequence)isoneofthecharactersexplicitlypresentinthe negatedcharacterclass(e.g.,"[ˆ A-Z\n]").Thisisunlikehowmanyotherregularexpres-siontoolstreatnegatedcharacterclasses,butunfortunatelytheinconsistencyishistori-callyentrenched.Matchingnewlinesmeansthatapatternlike[ˆ "]*canmatchtheentire inputunlessthere’sanotherquoteintheinput. -Arulecanhaveatmostoneinstanceoftrailingcontext(the’/’operatororthe’$’opera-tor).Thestartcondition,’ˆ ’,and"<<EOF>>"patternscanonlyoccuratthebeginningofapattern,and,aswellaswith’/’and’$’,cannotbegroupedinsideparentheses. A’ˆ ’whichdoesnotoccuratthebeginningofaruleora’$’whichdoesnotoccuratthe endofarulelosesitsspecialpropertiesandistreatedasanormalcharacter. Thefollowingareillegal: foo/bar$ <sc1>foo<sc2>bar Notethatthefirstofthese,canbewritten"foo/bar\n". Thefollowingwillresultin’$’or’ˆ ’beingtreatedasanormalcharacter: Version2.5 April1995 7 FLEX(1) FLEX(1) foo|(bar$) foo|ˆ bar Ifwhat’swantedisa"foo"orabar-followed-by-a-newline,thefollowingcouldbeused (thespecial’|’actionisexplainedbelow): foo| bar$/*actiongoeshere*/ Asimilartrickwillworkformatchingafooorabar-at-the-beginning-of-a-line. HOWTHEINPUTISMATCHED Whenthegeneratedscannerisrun,itanalyzesitsinputlookingforstringswhichmatchanyofits patterns.Ifitfindsmorethanonematch,ittakestheonematchingthemosttext(fortrailing contextrules,thisincludesthelengthofthetrailingpart,eventhoughitwillthenbereturnedto theinput).Ifitfindstwoormorematchesofthesamelength,therulelistedfirstintheflex inputfileischosen. Oncethematchisdetermined,thetextcorrespondingtothematch(calledthetoken)ismade availableintheglobalcharacterpointeryytext,anditslengthintheglobalintegeryyleng.The actioncorrespondingtothematchedpatternisthenexecuted(amoredetaileddescriptionof actionsfollows),andthentheremaininginputisscannedforanothermatch. Ifnomatchisfound,thenthedefaultruleisexecuted:thenextcharacterintheinputisconsideredmatchedandcopiedtothestandardoutput.Thus,thesimplestlegalflexinputis: %% whichgeneratesascannerthatsimplycopiesitsinput(onecharacteratatime)toitsoutput. Notethatyytextcanbedefinedintwodifferentways:eitherasacharacterpointerorasacharacterarray.Youcancontrolwhichdefinitionflexusesbyincludingoneofthespecialdirectives %pointeror%arrayinthefirst(definitions)sectionofyourflexinput.Thedefaultis %pointer,unlessyouusethe-llexcompatibilityoption,inwhichcaseyytextwillbeanarray. Theadvantageofusing%pointerissubstantiallyfasterscanningandnobufferoverflowwhen matchingverylargetokens(unlessyourunoutofdynamicmemory).Thedisadvantageisthat youarerestrictedinhowyouractionscanmodifyyytext(seethenextsection),andcallstothe unput()functiondestroysthepresentcontentsofyytext,whichcanbeaconsiderableporting headachewhenmovingbetweendifferentlexversions. Theadvantageof%arrayisthatyoucanthenmodifyyytexttoyourheart’scontent,andcalls tounput()donotdestroyyytext(seebelow).Furthermore,existinglexprogramssometimes accessyytextexternallyusingdeclarationsoftheform: externcharyytext[]; Thisdefinitioniserroneouswhenusedwith%pointer,butcorrectfor%array. %arraydefinesyytexttobeanarrayofYYLMAXcharacters,whichdefaultstoafairlylarge value.Youcanchangethesizebysimply#define’ingYYLMAXtoadifferentvalueinthefirst sectionofyourflexinput.Asmentionedabove,with%pointeryytextgrowsdynamicallyto accommodatelargetokens.Whilethismeansyour%pointerscannercanaccommodatevery largetokens(suchasmatchingentireblocksofcomments),bearinmindthateachtimethescannermustresizeyytextitalsomustrescantheentiretokenfromthebeginning,somatchingsuch tokenscanproveslow.yytextpresentlydoesnotdynamicallygrowifacalltounput()results intoomuchtextbeingpushedback;instead,arun-timeerrorresults. Alsonotethatyoucannotuse%arraywithC++scannerclasses(thec++option;seebelow). Version2.5 April1995 8 FLEX(1) FLEX(1) ACTIONS Eachpatterninarulehasacorrespondingaction,whichcanbeanyarbitraryCstatement.The patternendsatthefirstnon-escapedwhitespacecharacter;theremainderofthelineisitsaction. Iftheactionisempty,thenwhenthepatternismatchedtheinputtokenissimplydiscarded.For example,hereisthespecificationforaprogramwhichdeletesalloccurrencesof"zapme"fromits input: %% "zapme" (Itwillcopyallothercharactersintheinputtotheoutputsincetheywillbematchedbythe defaultrule.) Hereisaprogramwhichcompressesmultipleblanksandtabsdowntoasingleblank,andthrows awaywhitespacefoundattheendofaline: %% [\t]+putchar(’’); [\t]+$/*ignorethistoken*/ Iftheactioncontainsa’{’,thentheactionspanstillthebalancing’}’isfound,andtheaction maycrossmultiplelines.flexknowsaboutCstringsandcommentsandwon’tbefooledbybraces foundwithinthem,butalsoallowsactionstobeginwith%{andwillconsidertheactiontobeall thetextuptothenext%}(regardlessofordinarybracesinsidetheaction). Anactionconsistingsolelyofaverticalbar(’|’)means"sameastheactionforthenextrule." Seebelowforanillustration. ActionscanincludearbitraryCcode,includingreturnstatementstoreturnavaluetowhatever routinecalledyylex().Eachtimeyylex()iscalleditcontinuesprocessingtokensfromwhereit lastleftoffuntiliteitherreachestheendofthefileorexecutesareturn. Actionsarefreetomodifyyytextexceptforlengtheningit(addingcharacterstoitsend--these willoverwritelatercharactersintheinputstream).Thishoweverdoesnotapplywhenusing %array(seeabove);inthatcase,yytextmaybefreelymodifiedinanyway. Actionsarefreetomodifyyylengexcepttheyshouldnotdosoiftheactionalsoincludesuseof yymore()(seebelow). Thereareanumberofspecialdirectiveswhichcanbeincludedwithinanaction: -ECHOcopiesyytexttothescanner’soutput. -BEGINfollowedbythenameofastartconditionplacesthescannerinthecorrespond-ingstartcondition(seebelow). -REJECTdirectsthescannertoproceedontothe"secondbest"rulewhichmatchedthe input(oraprefixoftheinput).Theruleischosenasdescribedabovein"HowtheInput isMatched",andyytextandyylengsetupappropriately.Itmayeitherbeonewhich matchedasmuchtextastheoriginallychosenrulebutcamelaterintheflexinputfile,or onewhichmatchedlesstext.Forexample,thefollowingwillbothcountthewordsinthe inputandcalltheroutinespecial()whenever"frob"isseen: intwordcount=0; %% frobspecial();REJECT; [ˆ \t\n]+++wordcount; WithouttheREJECT,any"frob"’sintheinputwouldnotbecountedaswords,since Version2.5 April1995 9 FLEX(1) FLEX(1) thescannernormallyexecutesonlyoneactionpertoken.MultipleREJECT’sare allowed,eachonefindingthenextbestchoicetothecurrentlyactiverule.Forexample, whenthefollowingscannerscansthetoken"abcd",itwillwrite"abcdabcaba"totheoutput: %% a| ab| abc| abcdECHO;REJECT; .|\n/*eatupanyunmatchedcharacter*/ (Thefirstthreerulessharethefourth’sactionsincetheyusethespecial’|’action.) REJECTisaparticularlyexpensivefeatureintermsofscannerperformance;ifitisused inanyofthescanner’sactionsitwillslowdownallofthescanner’smatching.Furthermore,REJECTcannotbeusedwiththe-Cfor-CFoptions(seebelow). Notealsothatunliketheotherspecialactions,REJECTisabranch;codeimmediately followingitintheactionwillnotbeexecuted. -yymore()tellsthescannerthatthenexttimeitmatchesarule,thecorrespondingtoken shouldbeappendedontothecurrentvalueofyytextratherthanreplacingit.Forexample,giventheinput"mega-kludge"thefollowingwillwrite"mega-mega-kludge"tothe output: %% mega-ECHO;yymore(); kludgeECHO; First"mega-"ismatchedandechoedtotheoutput.Then"kludge"ismatched,butthe previous"mega-"isstillhangingaroundatthebeginningofyytextsotheECHOfor the"kludge"rulewillactuallywrite"mega-kludge". Twonotesregardinguseofyymore().First,yymore()dependsonthevalueofyylengcorrectly reflectingthesizeofthecurrenttoken,soyoumustnotmodifyyylengifyouareusingyymore(). Second,thepresenceofyymore()inthescanner’sactionentailsaminorperformancepenaltyin thescanner’smatchingspeed. -yyless(n)returnsallbutthefirstncharactersofthecurrenttokenbacktotheinput stream,wheretheywillberescannedwhenthescannerlooksforthenextmatch.yytext andyylengareadjustedappropriately(e.g.,yylengwillnowbeequalton).Forexample,ontheinput"foobar"thefollowingwillwriteout"foobarbar": %% foobarECHO;yyless(3); [a-z]+ECHO; Anargumentof0toyylesswillcausetheentirecurrentinputstringtobescannedagain. Unlessyou’vechangedhowthescannerwillsubsequentlyprocessitsinput(using BEGIN,forexample),thiswillresultinanendlessloop. Notethatyylessisamacroandcanonlybeusedintheflexinputfile,notfromothersource files. -unput(c)putsthecharactercbackontotheinputstream.Itwillbethenextcharacter scanned.Thefollowingactionwilltakethecurrenttokenandcauseittoberescanned enclosedinparentheses. Version2.5 April1995 10 FLEX(1) FLEX(1) { inti; /*Copyyytextbecauseunput()trashesyytext*/ char*yycopy=strdup(yytext); unput(’)’); for(i=yyleng-1;i>=0;--i) unput(yycopy[i]); unput(’(’); free(yycopy); } Notethatsinceeachunput()putsthegivencharacterbackatthebeginningoftheinput stream,pushingbackstringsmustbedoneback-to-front. Animportantpotentialproblemwhenusingunput()isthatifyouareusing%pointer(the default),acalltounput()destroysthecontentsofyytext,startingwithitsrightmostcharacter anddevouringonecharactertotheleftwitheachcall.Ifyouneedthevalueofyytextpreserved afteracalltounput()(asintheaboveexample),youmusteitherfirstcopyitelsewhere,orbuild yourscannerusing%arrayinstead(seeHowTheInputIsMatched). Finally,notethatyoucannotputbackEOFtoattempttomarktheinputstreamwithanend-of-file. -input()readsthenextcharacterfromtheinputstream.Forexample,thefollowingis onewaytoeatupCcomments: %% "/*"{ registerintc; for(;;) { while((c=input())!=’*’&& c!=EOF) ;/*eatuptextofcomment*/ if(c==’*’) { while((c=input())==’*’) ; if(c==’/’) break;/*foundtheend*/ } if(c==EOF) { error("EOFincomment"); break; } } } (NotethatifthescanneriscompiledusingC++,theninput()isinsteadreferredtoas yyinput(),inordertoavoidanameclashwiththeC++streambythenameofinput.) -YYFLUSHBUFFERflushesthescanner’sinternalbuffersothatthenexttimethe scannerattemptstomatchatoken,itwillfirstrefillthebufferusingYYINPUT(see Version2.5 April1995 11 FLEX(1) FLEX(1) TheGeneratedScanner,below).Thisactionisaspecialcaseofthemoregeneral yyflushbuffer()function,describedbelowinthesectionMultipleInputBuffers. -yyterminate()canbeusedinlieuofareturnstatementinanaction.Itterminatesthe scannerandreturnsa0tothescanner’scaller,indicating"alldone".Bydefault,yyterminate()isalsocalledwhenanend-of-fileisencountered.Itisamacroandmaybe redefined. THEGENERATEDSCANNER Theoutputofflexisthefilelex.yy.c,whichcontainsthescanningroutineyylex(),anumberof tablesusedbyitformatchingtokens,andanumberofauxiliaryroutinesandmacros.Bydefault, yylex()isdeclaredasfollows: intyylex() { ...variousdefinitionsandtheactionsinhere... } (Ifyourenvironmentsupportsfunctionprototypes,thenitwillbe"intyylex(void)".)Thisdefinitionmaybechangedbydefiningthe"YYDECL"macro.Forexample,youcoulduse: #defineYYDECLfloatlexscan(a,b)floata,b; togivethescanningroutinethenamelexscan,returningafloat,andtakingtwofloatsasarguments.NotethatifyougiveargumentstothescanningroutineusingaK&R-style/non-prototypedfunctiondeclaration,youmustterminatethedefinitionwithasemi-colon(;). Wheneveryylex()iscalled,itscanstokensfromtheglobalinputfileyyin(whichdefaultsto stdin).Itcontinuesuntiliteitherreachesanend-of-file(atwhichpointitreturnsthevalue0)or oneofitsactionsexecutesareturnstatement. Ifthescannerreachesanend-of-file,subsequentcallsareundefinedunlesseitheryyinispointedat anewinputfile(inwhichcasescanningcontinuesfromthatfile),oryyrestart()iscalled. yyrestart()takesoneargument,aFILE*pointer(whichcanbenil,ifyou’vesetup YYINPUTtoscanfromasourceotherthanyyin),andinitializesyyinforscanningfromthat file.Essentiallythereisnodifferencebetweenjustassigningyyintoanewinputfileorusing yyrestart()todoso;thelatterisavailableforcompatibilitywithpreviousversionsofflex,and becauseitcanbeusedtoswitchinputfilesinthemiddleofscanning.Itcanalsobeusedto throwawaythecurrentinputbuffer,bycallingitwithanargumentofyyin;butbetteristouse YYFLUSHBUFFER(seeabove).Notethatyyrestart()doesnotresetthestartcondition toINITIAL(seeStartConditions,below). Ifyylex()stopsscanningduetoexecutingareturnstatementinoneoftheactions,thescanner maythenbecalledagainanditwillresumescanningwhereitleftoff. Bydefault(andforpurposesofefficiency),thescannerusesblock-readsratherthansimplegetc() callstoreadcharactersfromyyin.ThenatureofhowitgetsitsinputcanbecontrolledbydefiningtheYYINPUTmacro.YYINPUT’scallingsequenceis "YYINPUT(buf,result,maxsize)".Itsactionistoplaceuptomaxsizecharactersinthecharacterarraybufandreturnintheintegervariableresulteitherthenumberofcharactersreadorthe constantYYNULL(0onUnixsystems)toindicateEOF.ThedefaultYYINPUTreadsfrom theglobalfile-pointer"yyin". AsampledefinitionofYYINPUT(inthedefinitionssectionoftheinputfile): %{ #defineYYINPUT(buf,result,maxsize)\ {\ intc=getchar();\ Version2.5 April1995 12 FLEX(1) FLEX(1) result=(c==EOF)?YYNULL:(buf[0]=c,1);\ } %} Thisdefinitionwillchangetheinputprocessingtooccuronecharacteratatime. Whenthescannerreceivesanend-of-fileindicationfromYYINPUT,itthenchecksthe yywrap()function.Ifyywrap()returnsfalse(zero),thenitisassumedthatthefunctionhas goneaheadandsetupyyintopointtoanotherinputfile,andscanningcontinues.Ifitreturns true(non-zero),thenthescannerterminates,returning0toitscaller.Notethatineithercase, thestartconditionremainsunchanged;itdoesnotreverttoINITIAL. Ifyoudonotsupplyyourownversionofyywrap(),thenyoumusteitheruse%optionnoyywrap(inwhichcasethescannerbehavesasthoughyywrap()returned1),oryoumustlinkwith −lfltoobtainthedefaultversionoftheroutine,whichalwaysreturns1. Threeroutinesareavailableforscanningfromin-memorybuffersratherthanfiles: yyscanstring(),yyscanbytes(),andyyscanbuffer().Seethediscussionofthembelow inthesectionMultipleInputBuffers. ThescannerwritesitsECHOoutputtotheyyoutglobal(default,stdout),whichmayberedefinedbytheusersimplybyassigningittosomeotherFILEpointer. STARTCONDITIONS flexprovidesamechanismforconditionallyactivatingrules.Anyrulewhosepatternisprefixed with"<sc>"willonlybeactivewhenthescannerisinthestartconditionnamed"sc".For example, <STRING>[ˆ "]*{/*eatupthestringbody...*/ ... } willbeactiveonlywhenthescannerisinthe"STRING"startcondition,and <INITIAL,STRING,QUOTE>\.{/*handleanescape...*/ ... } willbeactiveonlywhenthecurrentstartconditioniseither"INITIAL","STRING",or "QUOTE". Startconditionsaredeclaredinthedefinitions(first)sectionoftheinputusingunindentedlines beginningwitheither%sor%xfollowedbyalistofnames.Theformerdeclaresinclusivestart conditions,thelatterexclusivestartconditions.AstartconditionisactivatedusingtheBEGIN action.UntilthenextBEGINactionisexecuted,ruleswiththegivenstartconditionwillbe activeandruleswithotherstartconditionswillbeinactive.Ifthestartconditionisinclusive, thenruleswithnostartconditionsatallwillalsobeactive.Ifitisexclusive,thenonlyrules qualifiedwiththestartconditionwillbeactive.Asetofrulescontingentonthesameexclusive startconditiondescribeascannerwhichisindependentofanyoftheotherrulesintheflexinput. Becauseofthis,exclusivestartconditionsmakeiteasytospecify"mini-scanners"whichscanpor-tionsoftheinputthataresyntacticallydifferentfromtherest(e.g.,comments). Ifthedistinctionbetweeninclusiveandexclusivestartconditionsisstillalittlevague,here’sa simpleexampleillustratingtheconnectionbetweenthetwo.Thesetofrules: %sexample %% <example>foodosomething(); Version2.5 April1995 13 FLEX(1) FLEX(1) barsomethingelse(); isequivalentto %xexample %% <example>foodosomething(); <INITIAL,example>barsomethingelse(); Withoutthe<INITIAL,example>qualifier,thebarpatterninthesecondexamplewouldn’tbe active(i.e.,couldn’tmatch)wheninstartconditionexample.Ifwejustused<example>to qualifybar,though,thenitwouldonlybeactiveinexampleandnotinINITIAL,whileinthe firstexampleit’sactiveinboth,becauseinthefirstexampletheexamplestartionconditionisan inclusive(%s)startcondition. Alsonotethatthespecialstart-conditionspecifier<*>matcheseverystartcondition.Thus,the aboveexamplecouldalsohavebeenwritten; %xexample %% <example>foodosomething(); <*>barsomethingelse(); Thedefaultrule(toECHOanyunmatchedcharacter)remainsactiveinstartconditions.Itis equivalentto: <*>.|\nECHO; BEGIN(0)returnstotheoriginalstatewhereonlytheruleswithnostartconditionsareactive. Thisstatecanalsobereferredtoasthestart-condition"INITIAL",soBEGIN(INITIAL)is equivalenttoBEGIN(0).(Theparenthesesaroundthestartconditionnamearenotrequired butareconsideredgoodstyle.) BEGINactionscanalsobegivenasindentedcodeatthebeginningoftherulessection.For example,thefollowingwillcausethescannertoenterthe"SPECIAL"startconditionwhenever yylex()iscalledandtheglobalvariableenterspecialistrue: intenterspecial; %xSPECIAL %% if(enterspecial) BEGIN(SPECIAL); <SPECIAL>blahblahblah ...morerulesfollow... Toillustratetheusesofstartconditions,hereisascannerwhichprovidestwodifferentinterpretationsofastringlike"123.456".Bydefaultitwilltreatitasthreetokens,theinteger"123",adot (’.’),andtheinteger"456".Butifthestringisprecededearlierinthelinebythestring"expectfloats"itwilltreatitasasingletoken,thefloating-pointnumber123.456: Version2.5 April1995 14 FLEX(1) FLEX(1) %{ #include<math.h> %} %sexpect %% expect-floatsBEGIN(expect); <expect>[0-9]+"."[0-9]+{ printf("foundafloat,=%f\n", atof(yytext)); } <expect>\n{ /*that’stheendoftheline,so *weneedanother"expect-number" *beforewe’llrecognizeanymore *numbers */ BEGIN(INITIAL); } [0-9]+{ printf("foundaninteger,=%d\n", atoi(yytext)); } "."printf("foundadot\n"); Hereisascannerwhichrecognizes(anddiscards)Ccommentswhilemaintainingacountofthe currentinputline. %xcomment %% intlinenum=1; "/*"BEGIN(comment); <comment>[ˆ *\n]*/*eatanythingthat’snota’*’*/ <comment>"*"+[ˆ */\n]*/*eatup’*’snotfollowedby’/’s*/ <comment>\n++linenum; <comment>"*"+"/"BEGIN(INITIAL); Thisscannergoestoabitoftroubletomatchasmuchtextaspossiblewitheachrule.Ingeneral, whenattemptingtowriteahigh-speedscannertrytomatchasmuchpossibleineachrule,asit’s abigwin. Notethatstart-conditionsnamesarereallyintegervaluesandcanbestoredassuch.Thus,the abovecouldbeextendedinthefollowingfashion: %xcommentfoo %% intlinenum=1; intcommentcaller; "/*"{ Version2.5 April1995 15 FLEX(1) FLEX(1) commentcaller=INITIAL; BEGIN(comment); } ... <foo>"/*"{ commentcaller=foo; BEGIN(comment); } <comment>[ˆ *\n]*/*eatanythingthat’snota’*’*/ <comment>"*"+[ˆ */\n]*/*eatup’*’snotfollowedby’/’s*/ <comment>\n++linenum; <comment>"*"+"/"BEGIN(commentcaller); Furthermore,youcanaccessthecurrentstartconditionusingtheinteger-valuedYYSTART macro.Forexample,theaboveassignmentstocommentcallercouldinsteadbewritten commentcaller=YYSTART; FlexprovidesYYSTATEasanaliasforYYSTART(sincethatiswhat’susedbyAT&Tlex). Notethatstartconditionsdonothavetheirownname-space;%s’sand%x’sdeclarenamesinthe samefashionas#define’s. Finally,here’sanexampleofhowtomatchC-stylequotedstringsusingexclusivestartconditions, includingexpandedescapesequences(butnotincludingcheckingforastringthat’stoolong): %xstr %% charstringbuf[MAXSTRCONST]; char*stringbufptr; \"stringbufptr=stringbuf;BEGIN(str); <str>\"{/*sawclosingquote-alldone*/ BEGIN(INITIAL); *stringbufptr=’\0’; /*returnstringconstanttokentypeand *valuetoparser */ } <str>\n{ /*error-unterminatedstringconstant*/ /*generateerrormessage*/ } <str>\\[0-7]{1,3}{ /*octalescapesequence*/ intresult; (void)sscanf(yytext+1,"%o",&result); Version2.5 April1995 16 FLEX(1) FLEX(1) if(result>0xff) /*error,constantisout-of-bounds*/ *stringbufptr++=result; } <str>\\[0-9]+{ /*generateerror-badescapesequence;something *like’\48’or’\0777777’ */ } <str>\\n*stringbufptr++=’\n’; <str>\\t*stringbufptr++=’\t’; <str>\\r*stringbufptr++=’\r’; <str>\\b*stringbufptr++=’\b’; <str>\\f*stringbufptr++=’\f’; <str>\\(.|\n)*stringbufptr++=yytext[1]; <str>[ˆ \\\n\"]+{ char*yptr=yytext; while(*yptr) *stringbufptr++=*yptr++; } Often,suchasinsomeoftheexamplesabove,youwindupwritingawholebunchofrulesallprecededbythesamestartcondition(s).Flexmakesthisalittleeasierandcleanerbyintroducinga notionofstartconditionscope.Astartconditionscopeisbegunwith: <SCs>{ whereSCsisalistofoneormorestartconditions.Insidethestartconditionscope,everyrule automaticallyhastheprefix<SCs>appliedtoit,untila’}’whichmatchestheinitial’{’.So,for example, <ESC>{ "\\n"return’\n’; "\\r"return’\r’; "\\f"return’\f’; "\\0"return’\0’; } isequivalentto: <ESC>"\\n"return’\n’; <ESC>"\\r"return’\r’; <ESC>"\\f"return’\f’; <ESC>"\\0"return’\0’; Startconditionscopesmaybenested. Threeroutinesareavailableformanipulatingstacksofstartconditions: Version2.5 April1995 17 FLEX(1) FLEX(1) voidyypushstate(intnewstate) pushesthecurrentstartconditionontothetopofthestartconditionstackandswitches tonewstateasthoughyouhadusedBEGINnewstate(recallthatstartcondition namesarealsointegers). voidyypopstate() popsthetopofthestackandswitchestoitviaBEGIN. intyytopstate() returnsthetopofthestackwithoutalteringthestack’scontents. Thestartconditionstackgrowsdynamicallyandsohasnobuilt-insizelimitation.Ifmemoryis exhausted,programexecutionaborts. Tousestartconditionstacks,yourscannermustincludea%optionstackdirective(seeOptions below). MULTIPLEINPUTBUFFERS Somescanners(suchasthosewhichsupport"include"files)requirereadingfromseveralinput streams.Asflexscannersdoalargeamountofbuffering,onecannotcontrolwherethenextinput willbereadfrombysimplywritingaYYINPUTwhichissensitivetothescanningcontext. YYINPUTisonlycalledwhenthescannerreachestheendofitsbuffer,whichmaybealong timeafterscanningastatementsuchasan"include"whichrequiresswitchingtheinputsource. Tonegotiatethesesortsofproblems,flexprovidesamechanismforcreatingandswitching betweenmultipleinputbuffers.Aninputbufferiscreatedbyusing: YYBUFFERSTATEyycreatebuffer(FILE*file,intsize) whichtakesaFILEpointerandasizeandcreatesabufferassociatedwiththegivenfileandlarge enoughtoholdsizecharacters(whenindoubt,useYYBUFSIZEforthesize).Itreturnsa YYBUFFERSTATEhandle,whichmaythenbepassedtootherroutines(seebelow).The YYBUFFERSTATEtypeisapointertoanopaquestructyybufferstatestructure,so youmaysafelyinitializeYYBUFFERSTATEvariablesto((YYBUFFERSTATE)0)ifyou wish,andalsorefertotheopaquestructureinordertocorrectlydeclareinputbuffersinsource filesotherthanthatofyourscanner.NotethattheFILEpointerinthecallto yycreatebufferisonlyusedasthevalueofyyinseenbyYYINPUT;ifyouredefine YYINPUTsoitnolongerusesyyin,thenyoucansafelypassanilFILEpointerto yycreatebuffer.Youselectaparticularbuffertoscanfromusing: voidyyswitchtobuffer(YYBUFFERSTATEnewbuffer) switchesthescanner’sinputbuffersosubsequenttokenswillcomefromnewbuffer.Notethat yyswitchtobuffer()maybeusedbyyywrap()tosetthingsupforcontinuedscanning, insteadofopeninganewfileandpointingyyinatit.Notealsothatswitchinginputsourcesvia eitheryyswitchtobuffer()oryywrap()doesnotchangethestartcondition. voidyydeletebuffer(YYBUFFERSTATEbuffer) isusedtoreclaimthestorageassociatedwithabuffer.(buffercanbenil,inwhichcasetheroutinedoesnothing.)Youcanalsoclearthecurrentcontentsofabufferusing: voidyyflushbuffer(YYBUFFERSTATEbuffer) Thisfunctiondiscardsthebuffer’scontents,sothenexttimethescannerattemptstomatcha tokenfromthebuffer,itwillfirstfillthebufferanewusingYYINPUT. yynewbuffer()isanaliasforyycreatebuffer(),providedforcompatibilitywiththeC++ useofnewanddeleteforcreatinganddestroyingdynamicobjects. Version2.5 April1995 18 FLEX(1) FLEX(1) Finally,theYYCURRENTBUFFERmacroreturnsaYYBUFFERSTATEhandleto thecurrentbuffer. Hereisanexampleofusingthesefeaturesforwritingascannerwhichexpandsincludefiles(the <<EOF>>featureisdiscussedbelow): /*the"incl"stateisusedforpickingupthename *ofanincludefile */ %xincl %{ #defineMAXINCLUDEDEPTH10 YYBUFFERSTATEincludestack[MAXINCLUDEDEPTH]; intincludestackptr=0; %} %% includeBEGIN(incl); [a-z]+ECHO; [ˆ a-z\n]*\n?ECHO; <incl>[\t]*/*eatthewhitespace*/ <incl>[ˆ \t\n]+{/*gottheincludefilename*/ if(includestackptr>=MAXINCLUDEDEPTH) { fprintf(stderr,"Includesnestedtoodeeply"); exit(1); } includestack[includestackptr++]= YYCURRENTBUFFER; yyin=fopen(yytext,"r"); if(!yyin) error(...); yyswitchtobuffer( yycreatebuffer(yyin,YYBUFSIZE)); BEGIN(INITIAL); } <<EOF>>{ if(--includestackptr<0) { yyterminate(); } else { yydeletebuffer(YYCURRENTBUFFER); yyswitchtobuffer( Version2.5 April1995 19 FLEX(1) FLEX(1) includestack[includestackptr]); } } Threeroutinesareavailableforsettingupinputbuffersforscanningin-memorystringsinsteadof files.Allofthemcreateanewinputbufferforscanningthestring,andreturnacorresponding YYBUFFERSTATEhandle(whichyoushoulddeletewithyydeletebuffer()whendone withit).Theyalsoswitchtothenewbufferusingyyswitchtobuffer(),sothenextcallto yylex()willstartscanningthestring. yyscanstring(constchar*str) scansaNUL-terminatedstring. yyscanbytes(constchar*bytes,intlen) scanslenbytes(includingpossiblyNUL’s)startingatlocationbytes. Notethatbothofthesefunctionscreateandscanacopyofthestringorbytes.(Thismaybe desirable,sinceyylex()modifiesthecontentsofthebufferitisscanning.)Youcanavoidthe copybyusing: yyscanbuffer(char*base,yysizetsize) whichscansinplacethebufferstartingatbase,consistingofsizebytes,thelasttwobytes ofwhichmustbeYYENDOFBUFFERCHAR(ASCIINUL).Theselasttwo bytesarenotscanned;thus,scanningconsistsofbase[0]throughbase[size-2],inclusive. Ifyoufailtosetupbaseinthismanner(i.e.,forgetthefinaltwo YYENDOFBUFFERCHARbytes),thenyyscanbuffer()returnsanilpointer insteadofcreatinganewinputbuffer. Thetypeyysizetisanintegraltypetowhichyoucancastanintegerexpression reflectingthesizeofthebuffer. END-OF-FILERULES Thespecialrule"<<EOF>>"indicatesactionswhicharetobetakenwhenanend-of-fileis encounteredandyywrap()returnsnon-zero(i.e.,indicatesnofurtherfilestoprocess).Theaction mustfinishbydoingoneoffourthings: -assigningyyintoanewinputfile(inpreviousversionsofflex,afterdoingtheassignment youhadtocallthespecialactionYYNEWFILE;thisisnolongernecessary); -executingareturnstatement; -executingthespecialyyterminate()action; -or,switchingtoanewbufferusingyyswitchtobuffer()asshownintheexample above. <<EOF>>rulesmaynotbeusedwithotherpatterns;theymayonlybequalifiedwithalistof startconditions.Ifanunqualified<<EOF>>ruleisgiven,itappliestoallstartconditions whichdonotalreadyhave<<EOF>>actions.Tospecifyan<<EOF>>ruleforonlytheinitial startcondition,use <INITIAL><<EOF>> Theserulesareusefulforcatchingthingslikeunclosedcomments.Anexample: %xquote %% ...otherrulesfordealingwithquotes... Version2.5 April1995 20 FLEX(1) FLEX(1) <quote><<EOF>>{ error("unterminatedquote"); yyterminate(); } <<EOF>>{ if(*++filelist) yyin=fopen(*filelist,"r"); else yyterminate(); } MISCELLANEOUSMACROS ThemacroYYUSERACTIONcanbedefinedtoprovideanactionwhichisalwaysexecuted priortothematchedrule’saction.Forexample,itcouldbe#define’dtocallaroutinetoconvert yytexttolower-case.WhenYYUSERACTIONisinvoked,thevariableyyactgivesthe numberofthematchedrule(rulesarenumberedstartingwith1).Supposeyouwanttoprofile howofteneachofyourrulesismatched.Thefollowingwoulddothetrick: #defineYYUSERACTION++ctr[yyact] wherectrisanarraytoholdthecountsforthedifferentrules.Notethatthemacro YYNUMRULESgivesthetotalnumberofrules(includingthedefaultrule,evenifyouuse −s),soacorrectdeclarationforctris: intctr[YYNUMRULES]; ThemacroYYUSERINITmaybedefinedtoprovideanactionwhichisalwaysexecuted beforethefirstscan(andbeforethescanner’sinternalinitializationsaredone).Forexample,it couldbeusedtocallaroutinetoreadinadatatableoropenaloggingfile. Themacroyysetinteractive(isinteractive)canbeusedtocontrolwhetherthecurrent bufferisconsideredinteractive.Aninteractivebufferisprocessedmoreslowly,butmustbeused whenthescanner’sinputsourceisindeedinteractivetoavoidproblemsduetowaitingtofill buffers(seethediscussionofthe−Iflagbelow).Anon-zerovalueinthemacroinvocationmarks thebufferasinteractive,azerovalueasnon-interactive.Notethatuseofthismacrooverrides %optionalways-interactiveor%optionnever-interactive(seeOptionsbelow). yysetinteractive()mustbeinvokedpriortobeginningtoscanthebufferthatis(orisnot)to beconsideredinteractive. Themacroyysetbol(atbol)canbeusedtocontrolwhetherthecurrentbuffer’sscanningcontextforthenexttokenmatchisdoneasthoughatthebeginningofaline.Anon-zeromacro argumentmakesrulesanchoredwith ThemacroYYATBOL()returnstrueifthenexttokenscannedfromthecurrentbufferwill have’ˆ ’rulesactive,falseotherwise. Inthegeneratedscanner,theactionsareallgatheredinonelargeswitchstatementandseparated usingYYBREAK,whichmayberedefined.Bydefault,itissimplya"break",toseparate eachrule’sactionfromthefollowingrule’s.RedefiningYYBREAKallows,forexample,C++ usersto#defineYYBREAKtodonothing(whilebeingverycarefulthateveryruleendswitha "break"ora"return"!)toavoidsufferingfromunreachablestatementwarningswherebecausea rule’sactionendswith"return",theYYBREAKisinaccessible. VALUESAVAILABLETOTHEUSER Thissectionsummarizesthevariousvaluesavailabletotheuserintheruleactions. Version2.5 April1995 21 FLEX(1) FLEX(1) -char*yytextholdsthetextofthecurrenttoken.Itmaybemodifiedbutnotlength-ened(youcannotappendcharacterstotheend). Ifthespecialdirective%arrayappearsinthefirstsectionofthescannerdescription, thenyytextisinsteaddeclaredcharyytext[YYLMAX],whereYYLMAXisamacro definitionthatyoucanredefineinthefirstsectionifyoudon’tlikethedefaultvalue(generally8KB).Using%arrayresultsinsomewhatslowerscanners,butthevalueofyytext becomesimmunetocallstoinput()andunput(),whichpotentiallydestroyitsvaluewhen yytextisacharacterpointer.Theoppositeof%arrayis%pointer,whichisthe default. Youcannotuse%arraywhengeneratingC++scannerclasses(the−+flag). -intyylengholdsthelengthofthecurrenttoken. -FILE*yyinisthefilewhichbydefaultflexreadsfrom.Itmayberedefinedbutdoing soonlymakessensebeforescanningbeginsorafteranEOFhasbeenencountered. Changingitinthemidstofscanningwillhaveunexpectedresultssinceflexbuffersits input;useyyrestart()instead.Oncescanningterminatesbecauseanend-of-filehas beenseen,youcanassignyyinatthenewinputfileandthencallthescanneragainto continuescanning. -voidyyrestart(FILE*newfile)maybecalledtopointyyinatthenewinputfile. Theswitch-overtothenewfileisimmediate(anypreviouslybuffered-upinputislost). Notethatcallingyyrestart()withyyinasanargumentthusthrowsawaythecurrent inputbufferandcontinuesscanningthesameinputfile. -FILE*yyoutisthefiletowhichECHOactionsaredone.Itcanbereassignedbythe user. -YYCURRENTBUFFERreturnsaYYBUFFERSTATEhandletothecurrent buffer. -YYSTARTreturnsanintegervaluecorrespondingtothecurrentstartcondition.You cansubsequentlyusethisvaluewithBEGINtoreturntothatstartcondition. INTERFACINGWITHYACC Oneofthemainusesofflexisasacompaniontotheyaccparser-generator.yaccparsersexpect tocallaroutinenamedyylex()tofindthenextinputtoken.Theroutineissupposedtoreturn thetypeofthenexttokenaswellasputtinganyassociatedvalueintheglobalyylval.Touse flexwithyacc,onespecifiesthe−doptiontoyacctoinstructittogeneratethefiley.tab.hcontainingdefinitionsofallthe%tokensappearingintheyaccinput.Thisfileisthenincludedin theflexscanner.Forexample,ifoneofthetokensis"TOKNUMBER",partofthescanner mightlooklike: %{ #include"y.tab.h" %} %% [0-9]+yylval=atoi(yytext);returnTOKNUMBER; OPTIONS flexhasthefollowingoptions: −bGeneratebacking-upinformationtolex.backup.Thisisalistofscannerstateswhich requirebackingupandtheinputcharactersonwhichtheydoso.Byaddingrulesone canremovebacking-upstates.Ifallbacking-upstatesareeliminatedand−Cfor−CFis used,thegeneratedscannerwillrunfaster(seethe−pflag).Onlyuserswhowishto Version2.5 April1995 22 FLEX(1) FLEX(1) squeezeeverylastcycleoutoftheirscannersneedworryaboutthisoption.(SeethesectiononPerformanceConsiderationsbelow.) −cisado-nothing,deprecatedoptionincludedforPOSIXcompliance. −dmakesthegeneratedscannerrunindebugmode.Wheneverapatternisrecognizedand theglobalyyflexdebugisnon-zero(whichisthedefault),thescannerwillwriteto stderralineoftheform: --acceptingruleatline53("thematchedtext") Thelinenumberreferstothelocationoftheruleinthefiledefiningthescanner(i.e.,the filethatwasfedtoflex).Messagesarealsogeneratedwhenthescannerbacksup,accepts thedefaultrule,reachestheendofitsinputbuffer(orencountersaNUL;atthispoint, thetwolookthesameasfarasthescanner’sconcerned),orreachesanend-of-file. −fspecifiesfastscanner.Notablecompressionisdoneandstdioisbypassed.Theresultis largebutfast.Thisoptionisequivalentto−Cfr(seebelow). −hgeneratesa"help"summaryofflex’soptionstostdoutandthenexits.−?and−−help aresynonymsfor−h. −iinstructsflextogenerateacase-insensitivescanner.Thecaseoflettersgivenintheflex inputpatternswillbeignored,andtokensintheinputwillbematchedregardlessofcase. Thematchedtextgiveninyytextwillhavethepreservedcase(i.e.,itwillnotbefolded). −lturnsonmaximumcompatibilitywiththeoriginalAT&Tleximplementation.Notethat thisdoesnotmeanfullcompatibility.Useofthisoptioncostsaconsiderableamountof performance,anditcannotbeusedwiththe−+,-f,-F,-Cf,or-CFoptions.For detailsonthecompatibilitiesitprovides,seethesection"IncompatibilitiesWithLexAnd POSIX"below.ThisoptionalsoresultsinthenameYYFLEXLEXCOMPAT being#define’dinthegeneratedscanner. −nisanotherdo-nothing,deprecatedoptionincludedonlyforPOSIXcompliance. −pgeneratesaperformancereporttostderr.Thereportconsistsofcommentsregardingfeaturesoftheflexinputfilewhichwillcauseaseriouslossofperformanceintheresulting scanner.Ifyougivetheflagtwice,youwillalsogetcommentsregardingfeaturesthat leadtominorperformancelosses. NotethattheuseofREJECT,%optionyylineno,andvariabletrailingcontext(see theDeficiencies/Bugssectionbelow)entailsasubstantialperformancepenalty;useof yymore(),theˆ operator,andthe−Iflagentailminorperformancepenalties. −scausesthedefaultrule(thatunmatchedscannerinputisechoedtostdout)tobesuppressed.Ifthescannerencountersinputthatdoesnotmatchanyofitsrules,itaborts withanerror.Thisoptionisusefulforfindingholesinascanner’sruleset. −tinstructsflextowritethescanneritgeneratestostandardoutputinsteadoflex.yy.c. −vspecifiesthatflexshouldwritetostderrasummaryofstatisticsregardingthescannerit generates.Mostofthestatisticsaremeaninglesstothecasualflexuser,butthefirstline identifiestheversionofflex(sameasreportedby−V),andthenextlinetheflagsused whengeneratingthescanner,includingthosethatareonbydefault. −wsuppresseswarningmessages. −Binstructsflextogenerateabatchscanner,theoppositeofinteractivescannersgenerated by−I(seebelow).Ingeneral,youuse−Bwhenyouarecertainthatyourscannerwill neverbeusedinteractively,andyouwanttosqueezealittlemoreperformanceoutofit. Ifyourgoalisinsteadtosqueezeoutalotmoreperformance,youshouldbeusingthe −Cfor−CFoptions(discussedbelow),whichturnon−Bautomaticallyanyway. Version2.5 April1995 23 FLEX(1) FLEX(1) −Fspecifiesthatthefastscannertablerepresentationshouldbeused(andstdiobypassed). Thisrepresentationisaboutasfastasthefulltablerepresentation(-f),andforsomesets ofpatternswillbeconsiderablysmaller(andforothers,larger).Ingeneral,ifthepattern setcontainsboth"keywords"andacatch-all,"identifier"rule,suchasintheset: "case"returnTOKCASE; "switch"returnTOKSWITCH; ... "default"returnTOKDEFAULT; [a-z]+returnTOKID; thenyou’rebetteroffusingthefulltablerepresentation.Ifonlythe"identifier"ruleis presentandyouthenuseahashtableorsomesuchtodetectthekeywords,you’rebetter offusing-F. Thisoptionisequivalentto−CFr(seebelow).Itcannotbeusedwith−+. −Iinstructsflextogenerateaninteractivescanner.Aninteractivescannerisonethatonly looksaheadtodecidewhattokenhasbeenmatchedifitabsolutelymust.Itturnsout thatalwayslookingoneextracharacterahead,evenifthescannerhasalreadyseen enoughtexttodisambiguatethecurrenttoken,isabitfasterthanonlylookingahead whennecessary.Butscannersthatalwayslookaheadgivedreadfulinteractiveperformance;forexample,whenausertypesanewline,itisnotrecognizedasanewlinetoken untiltheyenteranothertoken,whichoftenmeanstypinginanotherwholeline. Flexscannersdefaulttointeractiveunlessyouusethe−Cfor−CFtable-compression options(seebelow).That’sbecauseifyou’relookingforhigh-performanceyoushouldbe usingoneoftheseoptions,soifyoudidn’t,flexassumesyou’drathertradeoffabitof run-timeperformanceforintuitiveinteractivebehavior.Notealsothatyoucannotuse−I inconjunctionwith−Cfor−CF.Thus,thisoptionisnotreallyneeded;itisonby defaultforallthosecasesinwhichitisallowed. Youcanforceascannertonotbeinteractivebyusing−B(seeabove). −Linstructsflexnottogenerate#linedirectives.Withoutthisoption,flexpeppersthegeneratedscannerwith#linedirectivessoerrormessagesintheactionswillbecorrectly locatedwithrespecttoeithertheoriginalflexinputfile(iftheerrorsareduetocodein theinputfile),orlex.yy.c(iftheerrorsareflex’sfault--youshouldreportthesesortsof errorstotheemailaddressgivenbelow). −Tmakesflexrunintracemode.Itwillgeneratealotofmessagestostderrconcerningthe formoftheinputandtheresultantnon-deterministicanddeterministicfiniteautomata. Thisoptionismostlyforuseinmaintainingflex. −Vprintstheversionnumbertostdoutandexits.−−versionisasynonymfor−V. −7instructsflextogeneratea7-bitscanner,i.e.,onewhichcanonlyrecognized7-bitcharac-tersinitsinput.Theadvantageofusing−7isthatthescanner’stablescanbeuptohalf thesizeofthosegeneratedusingthe−8option(seebelow).Thedisadvantageisthat suchscannersoftenhangorcrashiftheirinputcontainsan8-bitcharacter. Note,however,thatunlessyougenerateyourscannerusingthe−Cfor−CFtablecompressionoptions,useof−7willsaveonlyasmallamountoftablespace,andmakeyour scannerconsiderablylessportable.Flex’sdefaultbehavioristogeneratean8-bitscanner unlessyouusethe−Cfor−CF,inwhichcaseflexdefaultstogenerating7-bitscanners unlessyoursitewasalwaysconfiguredtogenerate8-bitscanners(aswilloftenbethecase withnon-USAsites).Youcantellwhetherflexgenerateda7-bitoran8-bitscannerby inspectingtheflagsummaryinthe−voutputasdescribedabove. Notethatifyouuse−Cfeor−CFe(thosetablecompressionoptions,butalsousing equivalenceclassesasdiscussedseebelow),flexstilldefaultstogeneratingan8-bit Version2.5 April1995 24 FLEX(1) FLEX(1) scanner,sinceusuallywiththesecompressionoptionsfull8-bittablesarenotmuchmore expensivethan7-bittables. −8instructsflextogeneratean8-bitscanner,i.e.,onewhichcanrecognize8-bitcharacters. Thisflagisonlyneededforscannersgeneratedusing−Cfor−CF,asotherwiseflex defaultstogeneratingan8-bitscanneranyway. Seethediscussionof−7aboveforflex’sdefaultbehaviorandthetradeoffsbetween7-bit and8-bitscanners. −+specifiesthatyouwantflextogenerateaC++scannerclass.SeethesectiononGeneratingC++Scannersbelowfordetails. −C[aefFmr] controlsthedegreeoftablecompressionand,moregenerally,trade-offsbetweensmall scannersandfastscanners. −Ca("align")instructsflextotradeofflargertablesinthegeneratedscannerforfaster performancebecausetheelementsofthetablesarebetteralignedformemoryaccessand computation.OnsomeRISCarchitectures,fetchingandmanipulatinglongwordsismore efficientthanwithsmaller-sizedunitssuchasshortwords.Thisoptioncandoublethesize ofthetablesusedbyyourscanner. −Cedirectsflextoconstructequivalenceclasses,i.e.,setsofcharacterswhichhaveidenticallexicalproperties(forexample,iftheonlyappearanceofdigitsintheflexinputisin thecharacterclass"[0-9]"thenthedigits’0’,’1’,...,’9’willallbeputinthesameequiva-lenceclass).Equivalenceclassesusuallygivedramaticreductionsinthefinaltable/object filesizes(typicallyafactorof2-5)andareprettycheapperformance-wise(onearraylook-uppercharacterscanned). −Cfspecifiesthatthefullscannertablesshouldbegenerated-flexshouldnotcompress thetablesbytakingadvantagesofsimilartransitionfunctionsfordifferentstates. −CFspecifiesthatthealternatefastscannerrepresentation(describedaboveunderthe −Fflag)shouldbeused.Thisoptioncannotbeusedwith−+. −Cmdirectsflextoconstructmeta-equivalenceclasses,whicharesetsofequivalence classes(orcharacters,ifequivalenceclassesarenotbeingused)thatarecommonlyused together.Meta-equivalenceclassesareoftenabigwinwhenusingcompressedtables,but theyhaveamoderateperformanceimpact(oneortwo"if"testsandonearraylook-up percharacterscanned). −CrcausesthegeneratedscannertobypassuseofthestandardI/Olibrary(stdio)for input.Insteadofcallingfread()orgetc(),thescannerwillusetheread()systemcall, resultinginaperformancegainwhichvariesfromsystemtosystem,butingeneralis probablynegligibleunlessyouarealsousing−Cfor−CF.Using−Crcancausestrange behaviorif,forexample,youreadfromyyinusingstdiopriortocallingthescanner (becausethescannerwillmisswhatevertextyourpreviousreadsleftinthestdioinput buffer). −CrhasnoeffectifyoudefineYYINPUT(seeTheGeneratedScannerabove). Alone−Cspecifiesthatthescannertablesshouldbecompressedbutneitherequivalence classesnormeta-equivalenceclassesshouldbeused. Theoptions−Cfor−CFand−Cmdonotmakesensetogether-thereisnoopportu-nityformeta-equivalenceclassesifthetableisnotbeingcompressed.Otherwisethe optionsmaybefreelymixed,andarecumulative. Thedefaultsettingis−Cem,whichspecifiesthatflexshouldgenerateequivalenceclasses andmeta-equivalenceclasses.Thissettingprovidesthehighestdegreeoftablecompres-sion.Youcantradeofffaster-executingscannersatthecostoflargertableswiththefol-lowinggenerallybeingtrue: Version2.5 April1995 25 FLEX(1) FLEX(1) slowest&smallest -Cem -Cm -Ce -C -C{f,F}e -C{f,F} -C{f,F}a fastest&largest Notethatscannerswiththesmallesttablesareusuallygeneratedandcompiledthequickest,soduringdevelopmentyouwillusuallywanttousethedefault,maximalcompression. −Cfeisoftenagoodcompromisebetweenspeedandsizeforproductionscanners. −ooutput directsflextowritethescannertothefileoutputinsteadoflex.yy.c.Ifyoucombine −owiththe−toption,thenthescanneriswrittentostdoutbutits#linedirectives(see the−Loptionabove)refertothefileoutput. −Pprefix changesthedefaultyyprefixusedbyflexforallglobally-visiblevariableandfunction namestoinsteadbeprefix.Forexample,−Pfoochangesthenameofyytexttofootext.Italsochangesthenameofthedefaultoutputfilefromlex.yy.ctolex.foo.c. Hereareallofthenamesaffected: yycreatebuffer yydeletebuffer yyflexdebug yyinitbuffer yyflushbuffer yyloadbufferstate yyswitchtobuffer yyin yyleng yylex yylineno yyout yyrestart yytext yywrap (IfyouareusingaC++scanner,thenonlyyywrapandyyFlexLexerareaffected.) Withinyourscanneritself,youcanstillrefertotheglobalvariablesandfunctionsusing eitherversionoftheirname;butexternally,theyhavethemodifiedname. Thisoptionletsyoueasilylinktogethermultipleflexprogramsintothesameexecutable. Note,though,thatusingthisoptionalsorenamesyywrap(),soyounowmusteitherprovideyourown(appropriately-named)versionoftheroutineforyourscanner,oruse %optionnoyywrap,aslinkingwith−lflnolongerprovidesoneforyoubydefault. −Sskeletonfile overridesthedefaultskeletonfilefromwhichflexconstructsitsscanners.You’llnever needthisoptionunlessyouaredoingflexmaintenanceordevelopment. flexalsoprovidesamechanismforcontrollingoptionswithinthescannerspecificationitself,rather thanfromtheflexcommand-line.Thisisdonebyincluding%optiondirectivesinthefirstsec-tionofthescannerspecification.Youcanspecifymultipleoptionswithasingle%option Version2.5 April1995 26 FLEX(1) FLEX(1) directive,andmultipledirectivesinthefirstsectionofyourflexinputfile. Mostoptionsaregivensimplyasnames,optionallyprecededbytheword"no"(withnointerveningwhitespace)tonegatetheirmeaning.Anumberareequivalenttoflexflagsortheirnegation: 7bit-7option 8bit-8option align-Caoption backup-boption batch-Boption c++-+option casefulor case-sensitiveoppositeof-i(default) case-insensitiveor caseless-ioption debug-doption defaultoppositeof-soption ecs-Ceoption fast-Foption full-foption interactive-Ioption lex-compat-loption meta-ecs-Cmoption perf-report-poption read-Croption stdout-toption verbose-voption warnoppositeof-woption (use"%optionnowarn"for-w) arrayequivalentto"%array" pointerequivalentto"%pointer"(default) Some%option’sprovidefeaturesotherwisenotavailable: always-interactive instructsflextogenerateascannerwhichalwaysconsidersitsinput"interactive".Normally,oneachnewinputfilethescannercallsisatty()inanattempttodetermine whetherthescanner’sinputsourceisinteractiveandthusshouldbereadacharacterata time.Whenthisoptionisused,however,thennosuchcallismade. maindirectsflextoprovideadefaultmain()programforthescanner,whichsimplycalls yylex().Thisoptionimpliesnoyywrap(seebelow). never-interactive instructsflextogenerateascannerwhichneverconsidersitsinput"interactive"(again, nocallmadetoisatty()).Thisistheoppositeofalways-interactive. stackenablestheuseofstartconditionstacks(seeStartConditionsabove). stdinit ifset(i.e.,%optionstdinit)initializesyyinandyyouttostdinandstdout,insteadofthe defaultofnil.Someexistinglexprogramsdependonthisbehavior,eventhoughitisnot compliantwithANSIC,whichdoesnotrequirestdinandstdouttobecompile-timecon-stant. Version2.5 April1995 27 FLEX(1) FLEX(1) yylineno directsflextogenerateascannerthatmaintainsthenumberofthecurrentlinereadfrom itsinputintheglobalvariableyylineno.Thisoptionisimpliedby%optionlexcompat. yywrap ifunset(i.e.,%optionnoyywrap),makesthescannernotcallyywrap()uponanendof-file,butsimplyassumethattherearenomorefilestoscan(untiltheuserpointsyyin atanewfileandcallsyylex()again). flexscansyourruleactionstodeterminewhetheryouusetheREJECToryymore()features. Therejectandyymoreoptionsareavailabletooverrideitsdecisionastowhetheryouusethe options,eitherbysettingthem(e.g.,%optionreject)toindicatethefeatureisindeedused,or unsettingthemtoindicateitactuallyisnotused(e.g.,%optionnoyymore). Threeoptionstakestring-delimitedvalues,offsetwith’=’: %optionoutfile="ABC" isequivalentto-oABC,and %optionprefix="XYZ" isequivalentto-PXYZ.Finally, %optionyyclass="foo" onlyapplieswhengeneratingaC++scanner(−+option).Itinformsflexthatyouhavederived fooasasubclassofyyFlexLexer,soflexwillplaceyouractionsinthememberfunction foo::yylex()insteadofyyFlexLexer::yylex().ItalsogeneratesayyFlexLexer::yylex() memberfunctionthatemitsarun-timeerror(byinvokingyyFlexLexer::LexerError())if called.SeeGeneratingC++Scanners,below,foradditionalinformation. Anumberofoptionsareavailableforlintpuristswhowanttosuppresstheappearanceof unneededroutinesinthegeneratedscanner.Eachofthefollowing,ifunset(e.g.,%optionnounput),resultsinthecorrespondingroutinenotappearinginthegeneratedscanner: input,unput yypushstate,yypopstate,yytopstate yyscanbuffer,yyscanbytes,yyscanstring (thoughyypushstate()andfriendswon’tappearanywayunlessyouuse%optionstack). PERFORMANCECONSIDERATIONS Themaindesigngoalofflexisthatitgeneratehigh-performancescanners.Ithasbeenoptimized fordealingwellwithlargesetsofrules.Asidefromtheeffectsonscannerspeedofthetablecompression−Coptionsoutlinedabove,thereareanumberofoptions/actionswhichdegradeperformance.Theseare,frommostexpensivetoleast: REJECT %optionyylineno arbitrarytrailingcontext patternsetsthatrequirebackingup %array %optioninteractive %optionalways-interactive Version2.5 April1995 28 FLEX(1) FLEX(1) ’ˆ ’beginning-of-lineoperator yymore() withthefirstthreeallbeingquiteexpensiveandthelasttwobeingquitecheap.Notealsothat unput()isimplementedasaroutinecallthatpotentiallydoesquiteabitofwork,whileyyless() isaquite-cheapmacro;soifjustputtingbacksomeexcesstextyouscanned,useyyless(). REJECTshouldbeavoidedatallcostswhenperformanceisimportant.Itisaparticularly expensiveoption. Gettingridofbackingupismessyandoftenmaybeanenormousamountofworkforacomplicatedscanner.Inprincipal,onebeginsbyusingthe−bflagtogeneratealex.backupfile.For example,ontheinput %% fooreturnTOKKEYWORD; foobarreturnTOKKEYWORD; thefilelookslike: State#6isnon-accepting-associatedrulelinenumbers: 23 out-transitions:[o] jam-transitions:EOF[\001-np-\177] State#8isnon-accepting-associatedrulelinenumbers: 3 out-transitions:[a] jam-transitions:EOF[\001-‘b-\177] State#9isnon-accepting-associatedrulelinenumbers: 3 out-transitions:[r] jam-transitions:EOF[\001-qs-\177] Compressedtablesalwaysbackup. Thefirstfewlinestellusthatthere’sascannerstateinwhichitcanmakeatransitiononan’o’ butnotonanyothercharacter,andthatinthatstatethecurrentlyscannedtextdoesnotmatch anyrule.Thestateoccurswhentryingtomatchtherulesfoundatlines2and3intheinputfile. Ifthescannerisinthatstateandthenreadssomethingotherthanan’o’,itwillhavetobackup tofindarulewhichismatched.Withabitofheadscratchingonecanseethatthismustbethe stateit’sinwhenithasseen"fo".Whenthishashappened,ifanythingotherthananother’o’is seen,thescannerwillhavetobackuptosimplymatchthe’f’(bythedefaultrule). ThecommentregardingState#8indicatesthere’saproblemwhen"foob"hasbeenscanned. Indeed,onanycharacterotherthanan’a’,thescannerwillhavetobackuptoaccept"foo". Similarly,thecommentforState#9concernswhen"fooba"hasbeenscannedandan’r’doesnot follow. Thefinalcommentremindsusthatthere’snopointgoingtoallthetroubleofremovingbacking upfromtherulesunlesswe’reusing−Cfor−CF,sincethere’snoperformancegaindoingso withcompressedscanners. Thewaytoremovethebackingupistoadd"error"rules: Version2.5 April1995 29 FLEX(1) FLEX(1) %% fooreturnTOKKEYWORD; foobarreturnTOKKEYWORD; fooba| foob| fo{ /*falsealarm,notreallyakeyword*/ returnTOKID; } Eliminatingbackingupamongalistofkeywordscanalsobedoneusinga"catch-all"rule: %% fooreturnTOKKEYWORD; foobarreturnTOKKEYWORD; [a-z]+returnTOKID; Thisisusuallythebestsolutionwhenappropriate. Backingupmessagestendtocascade.Withacomplicatedsetofrulesit’snotuncommontoget hundredsofmessages.Ifonecandecipherthem,though,itoftenonlytakesadozenorsorulesto eliminatethebackingup(thoughit’seasytomakeamistakeandhaveanerrorruleaccidentally matchavalidtoken.Apossiblefutureflexfeaturewillbetoautomaticallyaddrulestoeliminate backingup). It’simportanttokeepinmindthatyougainthebenefitsofeliminatingbackinguponlyifyou eliminateeveryinstanceofbackingup.Leavingjustonemeansyougainnothing. Variabletrailingcontext(whereboththeleadingandtrailingpartsdonothaveafixedlength) entailsalmostthesameperformancelossasREJECT(i.e.,substantial).Sowhenpossiblearule like: %% mouse|rat/(cat|dog)run(); isbetterwritten: %% mouse/cat|dogrun(); rat/cat|dogrun(); oras %% mouse|rat/catrun(); mouse|rat/dogrun(); Notethatherethespecial’|’actiondoesnotprovideanysavings,andcanevenmakethingsworse (seeDeficiencies/Bugsbelow). Anotherareawheretheusercanincreaseascanner’sperformance(andonethat’seasiertoimplement)arisesfromthefactthatthelongerthetokensmatched,thefasterthescannerwillrun. Thisisbecausewithlongtokenstheprocessingofmostinputcharacterstakesplaceinthe(short) innerscanningloop,anddoesnotoftenhavetogothroughtheadditionalworkofsettingupthe scanningenvironment(e.g.,yytext)fortheaction.RecallthescannerforCcomments: Version2.5 April1995 30 FLEX(1) FLEX(1) %xcomment %% intlinenum=1; "/*"BEGIN(comment); <comment>[ˆ *\n]* <comment>"*"+[ˆ */\n]* <comment>\n++linenum; <comment>"*"+"/"BEGIN(INITIAL); Thiscouldbespedupbywritingitas: %xcomment %% intlinenum=1; "/*"BEGIN(comment); <comment>[ˆ *\n]* <comment>[ˆ *\n]*\n++linenum; <comment>"*"+[ˆ */\n]* <comment>"*"+[ˆ */\n]*\n++linenum; <comment>"*"+"/"BEGIN(INITIAL); Nowinsteadofeachnewlinerequiringtheprocessingofanotheraction,recognizingthenewlinesis "distributed"overtheotherrulestokeepthematchedtextaslongaspossible.Notethatadding rulesdoesnotslowdownthescanner!Thespeedofthescannerisindependentofthenumberof rulesor(modulotheconsiderationsgivenatthebeginningofthissection)howcomplicatedthe rulesarewithregardtooperatorssuchas’*’and’|’. Afinalexampleinspeedingupascanner:supposeyouwanttoscanthroughafilecontaining identifiersandkeywords,oneperlineandwithnootherextraneouscharacters,andrecognizeall thekeywords.Anaturalfirstapproachis: %% asm| auto| break| ...etc... volatile| while/*it’sakeyword*/ .|\n/*it’snotakeyword*/ Toeliminatetheback-tracking,introduceacatch-allrule: %% asm| auto| break| ...etc... volatile| while/*it’sakeyword*/ Version2.5 April1995 31 FLEX(1) FLEX(1) [a-z]+| .|\n/*it’snotakeyword*/ Now,ifit’sguaranteedthatthere’sexactlyonewordperline,thenwecanreducethetotal numberofmatchesbyahalfbymergingintherecognitionofnewlineswiththatoftheother tokens: %% asm\n| auto\n| break\n| ...etc... volatile\n| while\n/*it’sakeyword*/ [a-z]+\n| .|\n/*it’snotakeyword*/ Onehastobecarefulhere,aswehavenowreintroducedbackingupintothescanner.Inparticular,whileweknowthattherewillneverbeanycharactersintheinputstreamotherthanletters ornewlines,flexcan’tfigurethisout,anditwillplanforpossiblyneedingtobackupwhenithas scannedatokenlike"auto"andthenthenextcharacterissomethingotherthananewlineora letter.Previouslyitwouldthenjustmatchthe"auto"ruleandbedone,butnowithasno "auto"rule,onlya"auto\n"rule.Toeliminatethepossibilityofbackingup,wecouldeither duplicateallrulesbutwithoutfinalnewlines,or,sinceweneverexpecttoencountersuchaninput andthereforedon’thowit’sclassified,wecanintroduceonemorecatch-allrule,thisonewhich doesn’tincludeanewline: %% asm\n| auto\n| break\n| ...etc... volatile\n| while\n/*it’sakeyword*/ [a-z]+\n| [a-z]+| .|\n/*it’snotakeyword*/ Compiledwith−Cf,thisisaboutasfastasonecangetaflexscannertogoforthisparticular problem. Afinalnote:flexisslowwhenmatchingNUL’s,particularlywhenatokencontainsmultiple NUL’s.It’sbesttowriteruleswhichmatchshortamountsoftextifit’santicipatedthatthetext willoftenincludeNUL’s. Anotherfinalnoteregardingperformance:asmentionedaboveinthesectionHowtheInputis Matched,dynamicallyresizingyytexttoaccommodatehugetokensisaslowprocessbecauseit presentlyrequiresthatthe(huge)tokenberescannedfromthebeginning.Thusifperformanceis vital,youshouldattempttomatch"large"quantitiesoftextbutnot"huge"quantities,wherethe cutoffbetweenthetwoisatabout8Kcharacters/token. GENERATINGC++SCANNERS flexprovidestwodifferentwaystogeneratescannersforusewithC++.Thefirstwayistosimply compileascannergeneratedbyflexusingaC++compilerinsteadofaCcompiler.Youshould Version2.5 April1995 32 FLEX(1) FLEX(1) notencounteranycompilationserrors(pleasereportanyyoufindtotheemailaddressgivenin theAuthorsectionbelow).YoucanthenuseC++codeinyourruleactionsinsteadofCcode. Notethatthedefaultinputsourceforyourscannerremainsyyin,anddefaultechoingisstilldone toyyout.BothoftheseremainFILE*variablesandnotC++streams. YoucanalsouseflextogenerateaC++scannerclass,usingthe−+option(or,equivalently, %optionc++),whichisautomaticallyspecifiedifthenameoftheflexexecutableendsina’+’, suchasflex++.Whenusingthisoption,flexdefaultstogeneratingthescannertothefile lex.yy.ccinsteadoflex.yy.c.ThegeneratedscannerincludestheheaderfileFlexLexer.h,which definestheinterfacetotwoC++classes. Thefirstclass,FlexLexer,providesanabstractbaseclassdefiningthegeneralscannerclass interface.Itprovidesthefollowingmemberfunctions: constchar*YYText() returnsthetextofthemostrecentlymatchedtoken,theequivalentofyytext. intYYLeng() returnsthelengthofthemostrecentlymatchedtoken,theequivalentofyyleng. intlineno()const returnsthecurrentinputlinenumber(see%optionyylineno),or1if%optionyylinenowasnotused. voidsetdebug(intflag) setsthedebuggingflagforthescanner,equivalenttoassigningtoyyflexdebug(see theOptionssectionabove).Notethatyoumustbuildthescannerusing%optiondebug toincludedebugginginformationinit. intdebug()const returnsthecurrentsettingofthedebuggingflag. Alsoprovidedarememberfunctionsequivalenttoyyswitchtobuffer(),yycreatebuffer() (thoughthefirstargumentisanistream*objectpointerandnotaFILE*),yyflushbuffer(), yydeletebuffer(),andyyrestart()(again,thefirstargumentisaistream*objectpointer). ThesecondclassdefinedinFlexLexer.hisyyFlexLexer,whichisderivedfromFlexLexer.It definesthefollowingadditionalmemberfunctions: yyFlexLexer(istream*argyyin=0,ostream*argyyout=0) constructsayyFlexLexerobjectusingthegivenstreamsforinputandoutput.Ifnot specified,thestreamsdefaulttocinandcout,respectively. virtualintyylex() performsthesameroleisyylex()doesforordinaryflexscanners:itscanstheinput stream,consumingtokens,untilarule’sactionreturnsavalue.IfyouderiveasubclassS fromyyFlexLexerandwanttoaccessthememberfunctionsandvariablesofSinside yylex(),thenyouneedtouse%optionyyclass="S"toinformflexthatyouwillbe usingthatsubclassinsteadofyyFlexLexer.Inthiscase,ratherthangenerating yyFlexLexer::yylex(),flexgeneratesS::yylex()(andalsogeneratesadummy yyFlexLexer::yylex()thatcallsyyFlexLexer::LexerError()ifcalled). virtualvoidswitchstreams(istream*newin=0, ostream*newout=0)reassignsyyintonewin(ifnon-nil)andyyouttonewout (ditto),deletingthepreviousinputbufferifyyinisreassigned. intyylex(istream*newin,ostream*newout=0) firstswitchestheinputstreamsviaswitchstreams(newin,newout)andthen returnsthevalueofyylex(). Inaddition,yyFlexLexerdefinesthefollowingprotectedvirtualfunctionswhichyoucanredefine inderivedclassestotailorthescanner: Version2.5 April1995 33 FLEX(1) FLEX(1) virtualintLexerInput(char*buf,intmaxsize) readsuptomaxsizecharactersintobufandreturnsthenumberofcharactersread.To indicateend-of-input,return0characters.Notethat"interactive"scanners(seethe−B and−Iflags)definethemacroYYINTERACTIVE.IfyouredefineLexerInput() andneedtotakedifferentactionsdependingonwhetherornotthescannermightbe scanninganinteractiveinputsource,youcantestforthepresenceofthisnamevia #ifdef. virtualvoidLexerOutput(constchar*buf,intsize) writesoutsizecharactersfromthebufferbuf,which,whileNUL-terminated,mayalso contain"internal"NUL’sifthescanner’srulescanmatchtextwithNUL’sinthem. virtualvoidLexerError(constchar*msg) reportsafatalerrormessage.Thedefaultversionofthisfunctionwritesthemessageto thestreamcerrandexits. NotethatayyFlexLexerobjectcontainsitsentirescanningstate.Thusyoucanusesuch objectstocreatereentrantscanners.Youcaninstantiatemultipleinstancesofthesame yyFlexLexerclass,andyoucanalsocombinemultipleC++scannerclassestogetherinthesame programusingthe−Poptiondiscussedabove. Finally,notethatthe%arrayfeatureisnotavailabletoC++scannerclasses;youmustuse %pointer(thedefault). HereisanexampleofasimpleC++scanner: //AnexampleofusingtheflexC++scannerclass. %{ intmylineno=0; %} string\"[ˆ \n"]+\" ws[\t]+ alpha[A-Za-z] dig[0-9] name({alpha}|{dig}|\$)({alpha}|{dig}|[.\-/$])* num1[-+]?{dig}+\.?([eE][-+]?{dig}+)? num2[-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)? number{num1}|{num2} %% {ws}/*skipblanksandtabs*/ "/*"{ intc; while((c=yyinput())!=0) { if(c==’\n’) ++mylineno; elseif(c==’*’) { if((c=yyinput())==’/’) Version2.5 April1995 34 FLEX(1) FLEX(1) break; else unput(c); } } } {number}cout<<"number"<<YYText()<<’\n’; \nmylineno++; {name}cout<<"name"<<YYText()<<’\n’; {string}cout<<"string"<<YYText()<<’\n’; %% intmain(int/*argc*/,char**/*argv*/) { FlexLexer*lexer=newyyFlexLexer; while(lexer->yylex()!=0) ; return0; } Ifyouwanttocreatemultiple(different)lexerclasses,youusethe−Pflag(ortheprefix= option)torenameeachyyFlexLexertosomeotherxxFlexLexer.Youthencaninclude <FlexLexer.h>inyourothersourcesonceperlexerclass,firstrenamingyyFlexLexerasfollows: #undefyyFlexLexer #defineyyFlexLexerxxFlexLexer #include<FlexLexer.h> #undefyyFlexLexer #defineyyFlexLexerzzFlexLexer #include<FlexLexer.h> if,forexample,youused%optionprefix="xx"foroneofyourscannersand%optionprefix="zz"fortheother. IMPORTANT:thepresentformofthescanningclassisexperimentalandmaychangeconsiderablybetweenmajorreleases. INCOMPATIBILITIESWITHLEXANDPOSIX flexisarewriteoftheAT&TUnixlextool(thetwoimplementationsdonotshareanycode, though),withsomeextensionsandincompatibilities,bothofwhichareofconcerntothosewho wishtowritescannersacceptabletoeitherimplementation.Flexisfullycompliantwiththe POSIXlexspecification,exceptthatwhenusing%pointer(thedefault),acalltounput() destroysthecontentsofyytext,whichiscountertothePOSIXspecification. Inthissectionwediscussalloftheknownareasofincompatibilitybetweenflex,AT&Tlex,and thePOSIXspecification. flex’s−loptionturnsonmaximumcompatibilitywiththeoriginalAT&Tleximplementation,at thecostofamajorlossinthegeneratedscanner’sperformance.Wenotebelowwhichincompatibilitiescanbeovercomeusingthe−loption. flexisfullycompatiblewithlexwiththefollowingexceptions: Version2.5 April1995 35 FLEX(1) FLEX(1) -Theundocumentedlexscannerinternalvariableyylinenoisnotsupportedunless−lor %optionyylinenoisused. yylinenoshouldbemaintainedonaper-bufferbasis,ratherthanaper-scanner(single globalvariable)basis. yylinenoisnotpartofthePOSIXspecification. -Theinput()routineisnotredefinable,thoughitmaybecalledtoreadcharactersfollow-ingwhateverhasbeenmatchedbyarule.Ifinput()encountersanend-of-filethenormal yywrap()processingisdone.A‘‘real’’end-of-fileisreturnedbyinput()asEOF. InputisinsteadcontrolledbydefiningtheYYINPUTmacro. Theflexrestrictionthatinput()cannotberedefinedisinaccordancewiththePOSIX specification,whichsimplydoesnotspecifyanywayofcontrollingthescanner’sinput otherthanbymakinganinitialassignmenttoyyin. -Theunput()routineisnotredefinable.ThisrestrictionisinaccordancewithPOSIX. -flexscannersarenotasreentrantaslexscanners.Inparticular,ifyouhaveaninteractive scannerandaninterrupthandlerwhichlong-jumpsoutofthescanner,andthescanneris subsequentlycalledagain,youmaygetthefollowingmessage: fatalflexscannerinternalerror--endofbuffermissed Toreenterthescanner,firstuse yyrestart(yyin); Notethatthiscallwillthrowawayanybufferedinput;usuallythisisn’taproblemwith aninteractivescanner. AlsonotethatflexC++scannerclassesarereentrant,soifusingC++isanoptionfor you,youshouldusetheminstead.See"GeneratingC++Scanners"abovefordetails. -output()isnotsupported.OutputfromtheECHOmacroisdonetothefile-pointer yyout(defaultstdout). output()isnotpartofthePOSIXspecification. -lexdoesnotsupportexclusivestartconditions(%x),thoughtheyareinthePOSIXspeci-fication. -Whendefinitionsareexpanded,flexenclosestheminparentheses.Withlex,thefollow-ing: NAME[A-Z][A-Z0-9]* %% foo{NAME}?printf("Foundit\n"); %% willnotmatchthestring"foo"becausewhenthemacroisexpandedtheruleisequivalent to"foo[A-Z][A-Z0-9]*?"andtheprecedenceissuchthatthe’?’isassociatedwith"[A-Z0-9]*".Withflex,therulewillbeexpandedto"foo([A-Z][A-Z0-9]*)?"andsothestring "foo"willmatch. Notethatifthedefinitionbeginswithˆ orendswith$thenitisnotexpandedwith parentheses,toallowtheseoperatorstoappearindefinitionswithoutlosingtheirspecial meanings.Butthe<s>,/,and<<EOF>>operatorscannotbeusedinaflexdefinition. Version2.5 April1995 36 FLEX(1) FLEX(1) Using−lresultsinthelexbehaviorofnoparenthesesaroundthedefinition. ThePOSIXspecificationisthatthedefinitionbeenclosedinparentheses. -Someimplementationsoflexallowarule’sactiontobeginonaseparateline,iftherule’s patternhastrailingwhitespace: %% foo|bar<spacehere> {foobaraction();} flexdoesnotsupportthisfeature. -Thelex%r(generateaRatforscanner)optionisnotsupported.Itisnotpartofthe POSIXspecification. -Afteracalltounput(),yytextisundefineduntilthenexttokenismatched,unlessthe scannerwasbuiltusing%array.ThisisnotthecasewithlexorthePOSIXspecification.The−loptiondoesawaywiththisincompatibility. -Theprecedenceofthe{}(numericrange)operatorisdifferent.lexinterprets"abc{1,3}" as"matchone,two,orthreeoccurrencesof’abc’",whereasflexinterpretsitas"match ’ab’followedbyone,two,orthreeoccurrencesof’c’".Thelatterisinagreementwiththe POSIXspecification. -Theprecedenceoftheˆ operatorisdifferent.lexinterprets"ˆ foo|bar"as"matcheither ’foo’atthebeginningofaline,or’bar’anywhere",whereasflexinterpretsitas"match either’foo’or’bar’iftheycomeatthebeginningofaline".Thelatterisinagreement withthePOSIXspecification. -Thespecialtable-sizedeclarationssuchas%asupportedbylexarenotrequiredbyflex scanners;flexignoresthem. -ThenameFLEXSCANNERis#define’dsoscannersmaybewrittenforusewitheither flexorlex.ScannersalsoincludeYYFLEXMAJORVERSIONand YYFLEXMINORVERSIONindicatingwhichversionofflexgeneratedthescanner (forexample,forthe2.5release,thesedefineswouldbe2and5respectively). ThefollowingflexfeaturesarenotincludedinlexorthePOSIXspecification: C++scanners %option startconditionscopes startconditionstacks interactive/non-interactivescanners yyscanstring()andfriends yyterminate() yysetinteractive() yysetbol() YYATBOL() <<EOF>> <*> YYDECL YYSTART YYUSERACTION YYUSERINIT #linedirectives %{}’saroundactions multipleactionsonaline Version2.5 April1995 37 FLEX(1) FLEX(1) plusalmostalloftheflexflags.Thelastfeatureinthelistreferstothefactthatwithflexyou canputmultipleactionsonthesameline,separatedwithsemi-colons,whilewithlex,thefollow-ing foohandlefoo();++numfoosseen; is(rathersurprisingly)truncatedto foohandlefoo(); flexdoesnottruncatetheaction.Actionsthatarenotenclosedinbracesaresimplyterminated attheendoftheline. DIAGNOSTICS warning,rulecannotbematchedindicatesthatthegivenrulecannotbematchedbecauseitfollowsotherrulesthatwillalwaysmatchthesametextasit.Forexample,inthefollowing"foo" cannotbematchedbecauseitcomesafteranidentifier"catch-all"rule: [a-z]+gotidentifier(); foogotfoo(); UsingREJECTinascannersuppressesthiswarning. warning,−soptiongivenbutdefaultrulecanbematchedmeansthatitispossible(perhapsonly inaparticularstartcondition)thatthedefaultrule(matchanysinglecharacter)istheonlyone thatwillmatchaparticularinput.Since−swasgiven,presumablythisisnotintended. rejectusedbutnotdetectedundefinedoryymoreusedbutnotdetectedundefined-Theseerrors canoccuratcompiletime.TheyindicatethatthescannerusesREJECToryymore()butthat flexfailedtonoticethefact,meaningthatflexscannedthefirsttwosectionslookingforoccurrencesoftheseactionsandfailedtofindany,butsomehowyousnucksomein(viaa#includefile, forexample).Use%optionrejector%optionyymoretoindicatetoflexthatyoureallydo usethesefeatures. flexscannerjammed-ascannercompiledwith−shasencounteredaninputstringwhichwasn’t matchedbyanyofitsrules.Thiserrorcanalsooccurduetointernalproblems. tokentoolarge,exceedsYYLMAX-yourscanneruses%arrayandoneofitsrulesmatcheda stringlongerthantheYYLMAXconstant(8Kbytesbydefault).Youcanincreasethevalueby #define’ingYYLMAXinthedefinitionssectionofyourflexinput. scannerrequires−8flagtousethecharacter’x’-Yourscannerspecificationincludesrecognizing the8-bitcharacter’x’andyoudidnotspecifythe−8flag,andyourscannerdefaultedto7-bit becauseyouusedthe−Cfor−CFtablecompressionoptions.Seethediscussionofthe−7flag fordetails. flexscannerpush-backoverflow-youusedunput()topushbacksomuchtextthatthescanner’s buffercouldnotholdboththepushed-backtextandthecurrenttokeninyytext.Ideallythe scannershoulddynamicallyresizethebufferinthiscase,butatpresentitdoesnot. inputbufferoverflow,can’tenlargebufferbecausescannerusesREJECT-thescannerwaswork-ingonmatchinganextremelylargetokenandneededtoexpandtheinputbuffer.Thisdoesn’t workwithscannersthatuseREJECT. fatalflexscannerinternalerror--endofbuffermissed-Thiscanoccurinanscannerwhichis reenteredafteralong-jumphasjumpedout(orover)thescanner’sactivationframe.Beforereen-teringthescanner,use: yyrestart(yyin); Version2.5 April1995 38 FLEX(1) FLEX(1) or,asnotedabove,switchtousingtheC++scannerclass. toomanystartconditionsin<>construct!-youlistedmorestartconditionsina<>construct thanexist(soyoumusthavelistedatleastoneofthemtwice). FILES −lfllibrarywithwhichscannersmustbelinked. lex.yy.c generatedscanner(calledlexyy.consomesystems). lex.yy.cc generatedC++scannerclass,whenusing-+. <FlexLexer.h> headerfiledefiningtheC++scannerbaseclass,FlexLexer,anditsderivedclass, yyFlexLexer. flex.sklskeletonscanner.Thisfileisonlyusedwhenbuildingflex,notwhenflexexecutes. lex.backup backing-upinformationfor−bflag(calledlex.bckonsomesystems). DEFICIENCIES/BUGS Sometrailingcontextpatternscannotbeproperlymatchedandgeneratewarningmessages("dangeroustrailingcontext").Thesearepatternswheretheendingofthefirstpartoftherule matchesthebeginningofthesecondpart,suchas"zx*/xy*",wherethe’x*’matchesthe’x’at thebeginningofthetrailingcontext.(NotethatthePOSIXdraftstatesthatthetextmatched bysuchpatternsisundefined.) Forsometrailingcontextrules,partswhichareactuallyfixed-lengtharenotrecognizedassuch, leadingtotheabovementionedperformanceloss.Inparticular,partsusing’|’or{n}(suchas "foo{3}")arealwaysconsideredvariable-length. Combiningtrailingcontextwiththespecial’|’actioncanresultinfixedtrailingcontextbeing turnedintothemoreexpensivevariabletrailingcontext.Forexample,inthefollowing: %% abc| xyz/def Useofunput()invalidatesyytextandyyleng,unlessthe%arraydirectiveorthe−loptionhas beenused. Pattern-matchingofNUL’sissubstantiallyslowerthanmatchingothercharacters. Dynamicresizingoftheinputbufferisslow,asitentailsrescanningallthetextmatchedsofarby thecurrent(generallyhuge)token. Duetobothbufferingofinputandread-ahead,youcannotintermixcallsto<stdio.h>routines, suchas,forexample,getchar(),withflexrulesandexpectittowork.Callinput()instead. Thetotaltableentrieslistedbythe−vflagexcludesthenumberoftableentriesneededtodeterminewhatrulehasbeenmatched.ThenumberofentriesisequaltothenumberofDFAstatesif thescannerdoesnotuseREJECT,andsomewhatgreaterthanthenumberofstatesifitdoes. REJECTcannotbeusedwiththe−for−Foptions. Theflexinternalalgorithmsneeddocumentation. SEEALSO lex(1),yacc(1),sed(1),awk(1). JohnLevine,TonyMason,andDougBrown,Lex&Yacc,O’ReillyandAssociates.Besuretoget the2ndedition. Version2.5 April1995 39 FLEX(1) FLEX(1) M.E.LeskandE.Schmidt,LEX−LexicalAnalyzerGenerator AlfredAho,RaviSethiandJeffreyUllman,Compilers:Principles,TechniquesandTools,Addison-Wesley(1986).Describesthepattern-matchingtechniquesusedbyflex(deterministicfinite automata). AUTHOR VernPaxson,withthehelpofmanyideasandmuchinspirationfromVanJacobson.OriginalversionbyJefPoskanzer.Thefasttablerepresentationisapartialimplementationofadesigndone byVanJacobson.TheimplementationwasdonebyKevinGongandVernPaxson. Thankstothemanyflexbeta-testers,feedbackers,andcontributors,especiallyFrancoisPinard, CaseyLeedom,RobertAbramovitz,StanAdermann,TerryAllen,DavidBarker-Plummer,John Basrai,NealBecker,NelsonH.F.Beebe,benson@odi.com,KarlBerry,PeterA.Bigot,Simon Blanchard,KeithBostic,FredericBrehm,IanBrockbank,KinCho,NickChristopher,BrianClapper,J.T.Conklin,JasonCoughlin,BillCox,NickCropper,DaveCurtis,ScottDavidDaniels, ChrisG.Demetriou,TheoDeraadt,MikeDonahue,ChuckDoucette,TomEpperly,LeoEskin, ChrisFaylor,ChrisFlatters,JonForrest,JeffreyFriedl,JoeGayda,KavehR.Ghazi,Wolfgang Glunz,EricGoldman,ChristopherM.Gould,UlrichGrepel,PeerGriebel,JanHajic,Charles Hemphill,NOROHideo,JarkkoHietaniemi,ScottHofmann,JeffHonig,DanaHudes,Eric Hughes,JohnInterrante,CerielJacobs,MichalJaegermann,SakariJalovaara,JeffreyR.Jones, HenryJuengst,KlausKaempf,JonathanI.Kamens,TerrenceOKane,AmirKatz, ken@ken.hilco.com,KevinB.Kenny,SteveKirsch,WinfriedKoenig,MarqKole,RonaldLamprecht,GregLee,RohanLenard,CraigLeres,JohnLevine,SteveLiddle,DavidLoffredo,Mike Long,MohamedelLozy,BrianMadsen,Malte,JoeMarshall,BengtMartensson,ChrisMetcalf, LukeMewburn,JimMeyering,R.AlexanderMilowski,ErikNaggum,G.T.Nicol,LandonNoll, JamesNordby,MarcNozell,RichardOhnemus,KarstenPahnke,SvenPanne,RolandPesch,WalterPelissero,GaumondPierre,EsmondPitt,JefPoskanzer,JoeRahmeh,JarmoRaiha,Frederic Raimbault,PatRankin,RickRichardson,KevinRodgers,KaiUweRommel,JimRoskind, AlbertoSantini,AndreasScherer,DarrellSchiebel,RafSchietekat,DougSchmidt,PhilippeSchnoebelen,AndreasSchwab,LarrySchwimmer,AlexSiegel,EckehardStolz,Jan-ErikStrvmquist, MikeStump,PaulStuart,DaveTallman,IanLanceTaylor,ChrisThewalt,RichardM.Timoney, JodiTsai,PaulTuinenga,GaryWeik,FrankWhaley,GerhardWilhelms,KentWilliams,Ken Yap,RonZellar,NathanZelle,DavidZuhn,andthosewhosenameshaveslippedmymarginal mail-archivingskillsbutwhosecontributionsareappreciatedallthesame. ThankstoKeithBostic,JonForrest,NoahFriedman,JohnGilmore,CraigLeres,JohnLevine, BobMulcahy,G.T.Nicol,FrancoisPinard,RichSalz,andRichardStallmanforhelpwithvarious distributionheadaches. ThankstoEsmondPittandEarleHortonfor8-bitcharactersupport;toBensonMarguliesand FredBurkeforC++support;toKentWilliamsandTomEpperlyforC++classsupport;toOve EwerlidforsupportofNUL’s;andtoEricHughesforsupportofmultiplebuffers. ThisworkwasprimarilydonewhenIwaswiththeRealTimeSystemsGroupattheLawrence BerkeleyLaboratoryinBerkeley,CA.ManythankstoallthereforthesupportIreceived. Sendcommentstovern@ee.lbl.gov. Version2.5 April1995 40