Docs: update API descriptions.

skvadrik · Oct 1, 2024 · b8ed9e4 · b8ed9e4
1 parent 12ba6f8
commit b8ed9e4
Show file tree

Hide file tree

Showing 11 changed files with 751 additions and 188 deletions.
diff --git a/doc/manpage.rst.in b/doc/manpage.rst.in
@@ -39,12 +39,12 @@ Syntax
 
 .. include:: @top_srcdir@/doc/manual/syntax/syntax.rst_
 
-Program interface
-=================
+Program interface (API)
+=======================
 
-.. include:: @top_srcdir@/doc/manual/syntax/api1.rst_
-.. include:: @top_srcdir@/doc/manual/syntax/api2_RE2C_LANG.rst_
-.. include:: @top_srcdir@/doc/manual/syntax/api3.rst_
+.. include:: @top_srcdir@/doc/manual/api/api1.rst_
+.. include:: @top_srcdir@/doc/manual/api/api2_RE2C_LANG.rst_
+.. include:: @top_srcdir@/doc/manual/api/api3.rst_
 
 Options
 =======
@@ -70,10 +70,6 @@ Blocks and directives
 =====================
 .. include:: @top_srcdir@/doc/manual/directives/directives.rst_
 
-API primitives
-==============
-.. include:: @top_srcdir@/doc/manual/api/interface.rst_
-
 Configurations
 ==============
 .. include:: @top_srcdir@/doc/manual/configurations/configurations.rst_

diff --git a/doc/manual/api/api1.rst_ b/doc/manual/api/api1.rst_
@@ -0,0 +1,14 @@
+
+The generated code interfaces with the outer program with the help of
+*primitives*, collectively referred to as the **API**.
+Which primitives should be defined for a particular program depends on multiple
+factors, including the complexity of regular expressions, input representation,
+buffering and the use of various features. All the necessary primitives should
+be defined by the user in the form of macros, functions, variables or any other
+suitable form that makes the generated code syntactically and semantically
+correct. re2c does not (and cannot) check the definitions, so if anything is
+missing or defined incorrectly, the generated program may have compile-time or
+run-time errors.
+This manual provides examples of API definitions in the most common cases.
+
+.. _generic_api:
diff --git a/doc/manual/api/api2_c.rst_ b/doc/manual/api/api2_c.rst_
@@ -0,0 +1,99 @@
+
+re2c has three API flavors that define the core set of primitives used by a
+program:
+
+**Simple API**
+    This is the default API for C/C++ backend. It consists of primitives
+    ``YYCURSOR``, ``YYMARKER``, ``YYCTXMARKER`` and ``YYLIMIT``, which
+    should be defined as pointers of type ``YYCTYPE*``.
+
+    |
+
+**Record API**
+    (*added in version 4.0*)
+    Record API is useful in cases when lexer state must be stored in a struct.
+    It is enabled with option ``--api record`` or configuration
+    ``re2c:api = record;``. This API consists of a variable ``yyrecord`` (the
+    name can be overridden with ``re2c:variable:yyrecord``) that should be
+    defined as a struct with fields ``yycursor``, ``yymarker``, ``yyctxmarker``,
+    ``yylimit`` (only the fields used by the generated code need to be defined,
+    and their names can be configured).
+
+    |
+
+**Generic API**
+    (*added in version 0.14*)
+    This is the most flexible API provided by re2c. It is enabled with
+    ``--api generic`` option or ``re2c:api = generic;`` configuration.
+    This API contains primitives for generic operations:
+    ``YYPEEK``,
+    ``YYSKIP``,
+    ``YYBACKUP``,
+    ``YYBACKUPCTX``,
+    ``YYSTAGP``,
+    ``YYSTAGN``,
+    ``YYMTAGP``,
+    ``YYMTAGN``,
+    ``YYRESTORE``,
+    ``YYRESTORECTX``,
+    ``YYRESTORETAG``,
+    ``YYSHIFT``,
+    ``YYSHIFTSTAG``,
+    ``YYSHIFTMTAG``,
+    ``YYLESSTHAN``.
+    Generic API supports two styles that determine the form in which the
+    primitives should be defined:
+
+    |
+
+    **Free-form**
+        Free-form style is enabled with configuration
+        ``re2c:api:style = free-form;``.
+        In this style interface primitives should be defined as free-form
+        pieces of code with interpolated variables of the form ``@@{var}`` or
+        optionally just ``@@`` if there is a single variable. The set of
+        variables is specific to each primitive. Here's how free-form generic
+        API can be defined in terms of pointers ``cursor``, ``limit``,
+        ``marker`` and ``ctxmarker``:
+
+        .. code-block:: c
+
+            /*!re2c
+              re2c:define:YYPEEK       = "*cursor";
+              re2c:define:YYSKIP       = "++cursor;";
+              re2c:define:YYBACKUP     = "marker = cursor;";
+              re2c:define:YYRESTORE    = "cursor = marker;";
+              re2c:define:YYBACKUPCTX  = "ctxmarker = cursor;";
+              re2c:define:YYRESTORECTX = "cursor = ctxmarker;";
+              re2c:define:YYRESTORETAG = "cursor = ${tag};";
+              re2c:define:YYLESSTHAN   = "limit - cursor < @@{len}";
+              re2c:define:YYSTAGP      = "@@{tag} = cursor;";
+              re2c:define:YYSTAGN      = "@@{tag} = NULL;";
+              re2c:define:YYSHIFT      = "cursor += @@{shift};";
+              re2c:define:YYSHIFTSTAG  = "@@{tag} += @@{shift};";
+            */
+
+        |
+
+    **Function-like**
+        Historically function-like style is the default one; it also can be
+        enabled with configuration ``re2c:api:style = functions;``. In this
+        style primitives should be defined as functions or macros with
+        parentheses, accepting the necessary arguments. Here's how function-like
+        generic API can be defined in terms of pointers ``cursor``, ``limit``,
+        ``marker`` and ``ctxmarker`` using preprocessor macros:
+
+        .. code-block:: c
+
+            #define  YYPEEK()                 *cursor
+            #define  YYSKIP()                 ++cursor
+            #define  YYBACKUP()               marker = cursor
+            #define  YYRESTORE()              cursor = marker
+            #define  YYBACKUPCTX()            ctxmarker = cursor
+            #define  YYRESTORECTX()           cursor = ctxmarker
+            #define  YYRESTORETAG(tag)        cursor = tag
+            #define  YYLESSTHAN(len)          limit - cursor < len
+            #define  YYSTAGP(tag)             tag = cursor
+            #define  YYSTAGN(tag)             tag = NULL
+            #define  YYSHIFT(shift)           cursor += shift
+            #define  YYSHIFTSTAG(tag, shift)  tag += shift
diff --git a/doc/manual/api/api2_go.rst_ b/doc/manual/api/api2_go.rst_
@@ -0,0 +1,98 @@
+
+re2go has three API flavors that define the core set of primitives used by a
+program:
+
+**Simple API**
+    (*added in version 4.0*)
+    This is a basic API that can be enabled with option ``--api simple`` or
+    configuration ``re2c:api = simple;``. It consists of the following
+    primitives: ``YYINPUT`` (which should be defined as a sequence of code
+    units, e.g. a string) and ``YYCURSOR``, ``YYMARKER``, ``YYCTXMARKER``,
+    ``YYLIMIT`` (which should be defined as indices in ``YYINPUT``).
+
+    |
+
+**Record API**
+    (*added in version 4.0*)
+    Record API is useful in cases when lexer state must be stored in a struct.
+    It is enabled with option ``--api record`` or configuration
+    ``re2c:api = record;``. This API consists of a variable ``yyrecord`` (the
+    name can be overridden with ``re2c:variable:yyrecord``) that should be
+    defined as a struct with fields ``yyinput``, ``yycursor``, ``yymarker``,
+    ``yyctxmarker``, ``yylimit`` (only the fields used by the generated code
+    need to be defined, and their names can be configured).
+
+    |
+
+**Generic API**
+    (*added in version 0.14*)
+    This is the default API for the Go backend. It is enabled with
+    ``--api generic`` option or ``re2c:api = generic;`` configuration.
+    This API contains primitives for generic operations:
+    ``YYPEEK``,
+    ``YYSKIP``,
+    ``YYBACKUP``,
+    ``YYBACKUPCTX``,
+    ``YYSTAGP``,
+    ``YYSTAGN``,
+    ``YYMTAGP``,
+    ``YYMTAGN``,
+    ``YYRESTORE``,
+    ``YYRESTORECTX``,
+    ``YYRESTORETAG``,
+    ``YYSHIFT``,
+    ``YYSHIFTSTAG``,
+    ``YYSHIFTMTAG``,
+    ``YYLESSTHAN``.
+    Generic API supports two styles that determine the form in which the
+    primitives should be defined:
+
+    !
+
+    **Free-form**
+        Free-form style is the default one. It can also be enabled with
+        configuration ``re2c:api:style = free-form;``.
+        In this style interface primitives should be defined as free-form
+        pieces of code with interpolated variables of the form ``@@{var}`` or
+        optionally just ``@@`` if there is a single variable. The set of
+        variables is specific to each primitive. Here's how free-form generic
+        API can be defined in terms of integer variables ``cursor``, ``limit``,
+        ``marker``, ``ctxmarker`` and a string (or a byte slice) ``data``:
+
+        .. code-block:: go
+
+            /*!re2c
+              re2c:define:YYPEEK       = "data[cursor]";
+              re2c:define:YYSKIP       = "cursor++";
+              re2c:define:YYBACKUP     = "marker = cursor";
+              re2c:define:YYRESTORE    = "cursor = marker";
+              re2c:define:YYBACKUPCTX  = "ctxmarker = cursor";
+              re2c:define:YYRESTORECTX = "cursor = ctxmarker";
+              re2c:define:YYRESTORETAG = "cursor = ${tag}";
+              re2c:define:YYLESSTHAN   = "limit - cursor < @@{len}";
+              re2c:define:YYSTAGP      = "@@{tag} = cursor";
+              re2c:define:YYSTAGN      = "@@{tag} = -1";
+              re2c:define:YYSHIFT      = "cursor += @@{shift}";
+              re2c:define:YYSHIFTSTAG  = "@@{tag} += @@{shift}";
+            */
+
+        |
+
+    **Function-like**
+        Function-like style is enabled with configuration
+        ``re2c:api:style = functions;``. In this style primitives should be
+        defined as closures accepting the necessary arguments. Here's how
+        function-like generic API can be defined in terms of integer variables
+        ``cursor``, ``limit``, ``marker``, ``ctxmarker`` and a string (or a byte
+        slice) ``data``:
+
+        .. code-block:: go
+
+            YYPEEK := func() byte { return data[cursor] }
+            YYSKIP := func() { cursor++ }
+            YYBACKUP := func() { marker = cursor }
+            YYRESTORE := func() { cursor = marker }
+            YYBACKUPCTX := func() { ctxmarker = cursor }
+            YYRESTORECTX := func() { cursor = ctxmarker }
+            YYLESSTHAN := func(n int) bool { return limit-cursor < n }
+            YYSHIFT := func(n int) { cursor += n }
diff --git a/doc/manual/api/api2_rust.rst_ b/doc/manual/api/api2_rust.rst_
@@ -0,0 +1,67 @@
+
+re2rust has three API flavors that define the core set of primitives used by a
+program:
+
+**Simple API**
+    (*added in version 4.0*)
+    This is a basic API that can be enabled with option ``--api simple`` or
+    configuration ``re2c:api = simple;``. It consists of the following
+    primitives: ``YYINPUT`` (which should be defined as a sequence of code
+    units, e.g. a string) and ``YYCURSOR``, ``YYMARKER``, ``YYCTXMARKER``,
+    ``YYLIMIT`` (which should be defined as indices in ``YYINPUT``).
+
+    |
+
+**Record API**
+    (*added in version 4.0*)
+    Record API is useful in cases when lexer state must be stored in a struct.
+    It is enabled with option ``--api record`` or configuration
+    ``re2c:api = record;``. This API consists of a variable ``yyrecord`` (the
+    name can be overridden with ``re2c:variable:yyrecord``) that should be
+    defined as a struct with fields ``yyinput``, ``yycursor``, ``yymarker``,
+    ``yyctxmarker``, ``yylimit`` (only the fields used by the generated code
+    need to be defined, and their names can be configured).
+
+    |
+
+**Generic API**
+    (*added in version 0.14*)
+    This is the default API for the Rust backend. It is enabled with
+    ``--api generic`` option or ``re2c:api = generic;`` configuration.
+    This API contains primitives for generic operations:
+    ``YYPEEK``,
+    ``YYSKIP``,
+    ``YYBACKUP``,
+    ``YYBACKUPCTX``,
+    ``YYSTAGP``,
+    ``YYSTAGN``,
+    ``YYMTAGP``,
+    ``YYMTAGN``,
+    ``YYRESTORE``,
+    ``YYRESTORECTX``,
+    ``YYRESTORETAG``,
+    ``YYSHIFT``,
+    ``YYSHIFTSTAG``,
+    ``YYSHIFTMTAG``,
+    ``YYLESSTHAN``.
+    For example, if the input is a byte slice ``buffer: &[u8]``, variables
+    ``cursor``, ``limit``, ``marker`` and ``ctxmarker`` of type ``usize``
+    represent input positions, and a constant ``NONE`` represents invalid
+    position, then generic API can be defined as follows:
+
+    .. code-block:: rust
+
+        /*!re2c
+          re2c:define:YYPEEK       = "*buffer.get_unchecked(cursor)";
+          re2c:define:YYSKIP       = "cursor += 1;";
+          re2c:define:YYBACKUP     = "marker = cursor;";
+          re2c:define:YYRESTORE    = "cursor = marker;";
+          re2c:define:YYBACKUPCTX  = "ctxmarker = cursor;";
+          re2c:define:YYRESTORECTX = "cursor = ctxmarker;";
+          re2c:define:YYRESTORETAG = "cursor = @@{tag};";
+          re2c:define:YYLESSTHAN   = "limit - cursor < @@{len}";
+          re2c:define:YYSTAGP      = "@@{tag} = cursor;";
+          re2c:define:YYSTAGN      = "@@{tag} = NONE;";
+          re2c:define:YYSHIFT      = "cursor = (cursor as isize + @@{shift}) as usize;";
+          re2c:define:YYSHIFTSTAG  = "@@{tag} = (@@{tag} as isize + @@{shift}) as usize;";
+        */