How do I keep asking for characters for my lexer?

Learning
1

I have:

{
  (*
  We can put (ocaml) code here that can be used in the actions below.
  We won't use it here.

  #use "example.ml";;
  *)
}
rule main = parse
  | ['0'-'9']+ { print_string "Int\n"}
  | ['0'-'9']+'.'['0'-'9'] {print_string "Float\n"}
  | ['a'-'z']+ { print_string "String\n"}
  | _{ main lexbuf }
  {
    let newlextbuf = (Lexing.from_channel stdin) in
    main newlextbuf
  }

but it stops after my first toke during top:

asdf 1243 4.17
String
- : unit = ()

why is that? how do I keep asking for more?

2

The usual purpose of a lexer is to return one token. So the code will return a value after each token unless you take other steps. Your final catch-all rule is an example of taking other steps; it looks for another token rather than returning.

You could add main lexbuf to all your rules. However, then you won’t have a provision for stopping. You might want to handle eof in another rule that doesn’t call main lexbuf.

3

I don’t understand the code I pasted.

Is the:

  {
    let newlextbuf = (Lexing.from_channel stdin) in
    main newlextbuf
  }

special? What if I called it:

  {
    let banana = (Lexing.from_channel stdin) in
    main banana
  }

?

4

These two code fragments have the same meaning. The name newlextbuf is just a name like any other name you might have used.

On the other hand, stdin is a predefined name, and so it has a special meaning. The difference is that this fragment defines the name newlextbuf but it only uses the name stdin (which thus has to be defined already).