Stacken/c64-livecoding
6
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Add TOKEN-ATOMIC for better tokenization statistics

Browse source
This commit is contained in:
John Lorentzson 2025-07-09 14:37:47 +02:00
parent 2b0bcecc66
commit 352f9e897e
1 changed files with 3 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
wip-duuqnd/user-side-compiler/tokenizer.lisp
View file

@ -65,8 +65,10 @@ reading immediately. Should be a subset of *SPECIAL-TOKEN-CHARS*.")
;; Special syntax tokens ;; Special syntax tokens
(defclass token-atomic (token) ())
(defmacro define-atomic-token (name) (defmacro define-atomic-token (name)
`(defclass ,name (token) ())) `(defclass ,name (token-atomic) ()))
(define-atomic-token token-end-of-statement) (define-atomic-token token-end-of-statement)