DefGen can now find Code Prim defs in C files

DefGen/src/defgen/core.clj

@@ -70,14 +70,44 @@
                          (def-compare (defs key1) (defs key2))))
         defs))
 
-(defn load-def-file
-  "Returns a map of all of the definitions in the given file, indexed by token id."
+(defn load-def-c-file
+  [path]
+  ;; Partition the lines into consecutive groups of EDN data
+  ;; (***-prefixed lines) and non-EDN data.  The file will always
+  ;; follow the pattern non-EDN, EDN, non-EDN, EDN, ... due to the
+  ;; fact that the comment must begin and end on lines that do *not*
+  ;; contain EDN data.  We use this to our advantage and strip out the
+  ;; odd number elements in the sequence after the partition
+  ;; operation.  Then we strip the comment leader from each line, join
+  ;; all of the lines together, and finally run each joined line
+  ;; through the EDN reader.
+  (->> path
+       slurp
+       s/split-lines
+       (partition-by #(some? (re-matches #"^\s*\*\*\*.*" %)))
+       rest
+       (take-nth 2)
+       (map (fn [lines] (map #(s/replace % #"^\s*\*\*\*(.*)$" "$1")
+                             lines)))
+       (map s/join)
+       (map edn/read-string)
+       (map parse-def)))
+
+(defn load-def-edn-file
   [path]
   (with-open [rdr (java.io.PushbackReader. (clojure.java.io/reader path))]
     (loop [defs []]
       (if-let [next-def (edn/read {:eof nil} rdr)]
         (recur (conj defs (parse-def next-def)))
-        (into {} (map #(vector (% :id) %) defs))))))
+        defs))))
+
+(defn load-def-file
+  "Returns a map of all of the definitions in the given file, indexed by token id.  The file may be either an EDN file (ending in .edn) or a C file (ending in .c).  If the latter, then ***-prefixed comments will be read in order to locate EDN data."
+  [path]
+  (let [defs (if (= ".c" (fs/extension path))
+               (load-def-c-file path)
+               (load-def-edn-file path))]
+    (into {} (map #(vector (% :id) %) defs))))
 
 
 ;; =====================================================================

Makefile.cygwin

@@ -11,13 +11,13 @@ test: test/enforthtest
 sertest: test/enforthserialtest
 	test/enforthserialtest --abort
 
-utility/enforth_definitions.h: primitives/core.edn primitives/core-ext.edn primitives/double.edn primitives/enforth.edn primitives/string.edn primitives/tools.edn
+utility/enforth_definitions.h: enforth.c primitives/core.edn primitives/core-ext.edn primitives/double.edn primitives/enforth.edn primitives/string.edn primitives/tools.edn
 	cd DefGen && lein run ../utility $(addprefix ../, $+)
 
-utility/enforth_jumptable.h: primitives/core.edn primitives/core-ext.edn primitives/double.edn primitives/enforth.edn primitives/string.edn primitives/tools.edn
+utility/enforth_jumptable.h: enforth.c primitives/core.edn primitives/core-ext.edn primitives/double.edn primitives/enforth.edn primitives/string.edn primitives/tools.edn
 	cd DefGen && lein run ../utility $(addprefix ../, $+)
 
-utility/enforth_tokens.h: primitives/core.edn primitives/core-ext.edn primitives/double.edn primitives/enforth.edn primitives/string.edn primitives/tools.edn
+utility/enforth_tokens.h: enforth.c primitives/core.edn primitives/core-ext.edn primitives/double.edn primitives/enforth.edn primitives/string.edn primitives/tools.edn
 	cd DefGen && lein run ../utility $(addprefix ../, $+)
 
 test/enforthsimple: enforth.h enforth.c utility/enforth_definitions.h utility/enforth_jumptable.h utility/enforth_tokens.h test/enforthsimple.c

TODO.md

@@ -1,6 +1,7 @@
 # Before Release
 
-* Modify DefGen to read code primitive EDN data from `/****`-prefixed comments in the `enforth.c` file.  Then rename the `primitives` directory to `definitions` and have it only include ROM definitions.
+* Move all Code Primitive EDN data into `enforth.c`.
+* Rename the `primitives` directory to `definitions` now that it only contains ROM definitions.
 * Create some sort of iterate-over-the-dictionary word that takes an XT (`FOUND?`, in the case of `FIND-WORD`) and stops iterating when the word returns true?  Use this for both `FIND-WORD` and `WORDS`.
 * Most of `FOUND-FFIDEF?` is just `FOUND?`; we should find a way to merge that code.
   * FFI definition names are stored normally (forward order) which means that `FOUND?` cannot use `STRING~XT` for comparing FFIs.  We should put definitions in forward order and then just do subtraction to jump to the start of the definition.  Then we can use `FOUND?` for everything.
@@ -14,16 +15,19 @@
 * Improve the stack checking code.
   * First, the code is probably too aggressive and may not let us use the last stack item.
   * Second, we have the macro scattered everywhere, but it would be better if the stack sizes were in an extra byte in the definition header and then checked in a single place right before DISPATCH\_TOKEN.  This may make the logic small enough to include on AVRs (although it will add ~240 bytes to the size of the ROM Definition block).
+* Rewrite `DUMP` to use `BEGIN/REPEAT` instead of `DO/LOOP`; eliminates `PIQDO`, `PILOOP`, and `PIPLUSLOOP`.
 * Consider additional de-duplication of the Code Prims and ROM Definitions.
   * `I` could compile `R@` instead of providing its own token.  Same thing with `(DO)` and `2>R`
   * `TOKEN,` could be `C@`.
   * `(LOOP)` could maybe always be `(+LOOP)` with a `:charlit 1` in front?
   * DefGen could implement the above optimizations, that way the code always reads nicely, even though it is being rewritten during compilation.  Note that this doesn't work for `I`.
-  * Rewriting `DUMP` to use `BEGIN/REPEAT` instead of `DO/LOOP` eliminates `PIQDO`, `PILOOP`, and `PIPLUSLOOP`.
   * Is there any benefit to defining `DOICONSTANT` for storing constants in ROM PFAs?  Currently we define tokens or words that calculate and return constants.
+  * Can we eliminate some of the `DOUBLE` words that are used for things like parsing and output?
 * Fix tracing now that kDefinitionNames has gone away.
+* Create a `:profile` property on each word that lets us build smaller versions of the ROM.  For example, maybe you eliminate `DOUBLE` support or the `TOOLS` so that you can get down to something that fits on ATtiny85.  Code Primitives should be included in this as well and then jump table entries for elided primitives should not be generated (so that the compiler will remove that code).
 * Consider creating EnforthDuino.cpp/.h wrappers to make it easier to interact with Enforth in the Arduino environment.  Mostly just to wrap the serial code, allocate the dictionary block, implement EEPROM-backed load/save, etc.
 * Add comments to all of the `.edn` files.
+* Move to [Arduino 1.5 library format](https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification) now that 1.0.6 supports that format?
 
 # After Release
 
@@ -52,4 +56,6 @@
   * I wonder if we can find a way to predefine a set of trampolines in Flash instead of in RAM?  *i.e.,* we reserve the last 32 tokens for precompiled trampolines and then provide a simplified way to build up that flash array.  This table-based method would actually work since it would just be a list of other addresses (which conveniently we already have thanks to the `FFIDEF_*` vars that are being used for the linked list).  This would give users a way to modify their enforth compile to predefine externals in a way that consumes no RAM.  You still need to define the FFIs, but you don't need to reference them at runtime.
   * This feels like a good balance between ROM and RAM: you can access any FFI at runtime if you are willing to consume memory on that (which is probably fine during development) and then you switch to a ROM-based FFI primitive once you know you'll be using an FFI a lot.  This breaks your flash, of course, but your source is unchanged (and we could make the `EXTERNAL:` word just do nothing in the case where you are trying to reference a ROM-based FFI primitive).
   * This makes the ATtiny85 possible again, because we'll just define the primitives that we care about as ROM primitives.  The theory here is that we'll only get 256 bytes or something for the dictionary on ATtiny85 and so we don't want to waste 6 bytes on every Arduino function that we want to call.  I don't know if I buy that though, because the real concern on the ATtiny85 is ROM and we're way over our limit at the moment.
+  * This would allow `LOAD` and `SAVE` to be FFIs without consuming RAM -- they would just be the first two tokens in this extra table.
+  * All of this should be much easier now that we have the new XT format.  We just need a way to tack some extra stuff onto the definitions table (or use a new XT flag to target this extra definition block).  We can probably implement this using macros since we don't actually need the names here (it's just a bunch of FFI trampolines one after the other).  Probably similar to how the names table worked way-back-when.
 * Refactor the DefGen code to make it easier to load in the definitions and then traverse them for analysis purposes.  First analysis: output a GraphViz file that shows the calling patterns between all of the words.

enforth.c

@@ -430,6 +430,17 @@ void enforth_resume(EnforthVM * const vm)
 
         goto *(void *)pgm_read_word(&primitive_table[token]);
 
+        /* -------------------------------------------------------------
+         * (HALT) [Enforth] ( i*x -- i*x ) ( R: j*x -- j*x )
+         *
+         * Stops and exits the VM.  State is preserved on the stack,
+         * allowing the VM to be resumed by enforth_resume.
+         *
+        ***{:token :phalt
+        *** :name "(HALT)"
+        *** :args [[] []]
+        *** :flags #{:headerless}}
+         */
         PHALT:
         {
             /* Push TOS onto the stack. */
@@ -553,6 +564,10 @@ void enforth_resume(EnforthVM * const vm)
         }
         continue;
 
+        /* -------------------------------------------------------------
+        ***{:token :dup
+        *** :args [[:x] [:x :x]]}
+         */
         DUP:
         {
             CHECK_STACK(1, 2);

primitives/core.edn

@@ -272,8 +272,6 @@
 
 {:token :drop :args [[:x1 :x2] [:x1]]}
 
-{:token :dup :args [[:x] [:x :x]]}
-
 {:token :else
  :flags #{:immediate}
  :source ": ELSE ( C: orig1 -- orig2 )

primitives/enforth.edn

@@ -592,8 +592,6 @@
  :args [[:n1 :n2] []]
  :flags #{:headerless}}
 
-{:token :phalt :flags #{:headerless}}
-
 {:token :piqdo
  :name "(I?DO)"
  :flags #{:headerless}}