kitten.sh
1<div align="center">
2 <img alt="reghex" width="250" src="docs/reghex-logo.png" />
3 <br />
4 <br />
5 <strong>
6 The magical sticky regex-based parser generator
7 </strong>
8 <br />
9 <br />
10 <br />
11</div>
12
13Leveraging the power of sticky regexes and JS code generation, `reghex` allows
14you to code parsers quickly, by surrounding regular expressions with a regex-like
15[DSL](https://en.wikipedia.org/wiki/Domain-specific_language).
16
17With `reghex` you can generate a parser from a tagged template literal, which is
18quick to prototype and generates reasonably compact and performant code.
19
20_This project is still in its early stages and is experimental. Its API may still
21change and some issues may need to be ironed out._
22
23## Quick Start
24
25##### 1. Install with yarn or npm
26
27```sh
28yarn add reghex
29# or
30npm install --save reghex
31```
32
33##### 2. Add the plugin to your Babel configuration _(optional)_
34
35In your `.babelrc`, `babel.config.js`, or `package.json:babel` add:
36
37```json
38{
39 "plugins": ["reghex/babel"]
40}
41```
42
43Alternatively, you can set up [`babel-plugin-macros`](https://github.com/kentcdodds/babel-plugin-macros) and
44import `reghex` from `"reghex/macro"` instead.
45
46This step is **optional**. `reghex` can also generate its optimised JS code during runtime only!
47
48##### 3. Have fun writing parsers!
49
50```js
51import { match, parse } from 'reghex';
52
53const name = match('name')`
54 ${/\w+/}
55`;
56
57parse(name)('hello');
58// [ "hello", .tag = "name" ]
59```
60
61## Concepts
62
63The fundamental concept of `reghex` are regexes, specifically
64[sticky regexes](https://www.loganfranken.com/blog/831/es6-everyday-sticky-regex-matches/)!
65These are regular expressions that don't search a target string, but instead match at the
66specific position they're at. The flag for sticky regexes is `y` and hence
67they can be created using `/phrase/y` or `new RegExp('phrase', 'y')`.
68
69**Sticky Regexes** are the perfect foundation for a parsing framework in JavaScript!
70Because they only match at a single position they can be used to match patterns
71continuously, as a parser would. Like global regexes, we can then manipulate where
72they should be matched by setting `regex.lastIndex = index;` and after matching
73read back their updated `regex.lastIndex`.
74
75> **Note:** Sticky Regexes aren't natively
76> [supported in any versions of Internet Explorer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/sticky#Browser_compatibility). `reghex` works around this by imitating its behaviour, which may decrease performance on IE11.
77
78This primitive allows us to build up a parser from regexes that you pass when
79authoring a parser function, also called a "matcher" in `reghex`. When `reghex` compiles
80to parser code, this code is just a sequence and combination of sticky regexes that
81are executed in order!
82
83```js
84let input = 'phrases should be parsed...';
85let lastIndex = 0;
86
87const regex = /phrase/y;
88function matcher() {
89 let match;
90 // Before matching we set the current index on the RegExp
91 regex.lastIndex = lastIndex;
92 // Then we match and store the result
93 if ((match = regex.exec(input))) {
94 // If the RegExp matches successfully, we update our lastIndex
95 lastIndex = regex.lastIndex;
96 }
97}
98```
99
100This mechanism is used in all matcher functions that `reghex` generates.
101Internally `reghex` keeps track of the input string and the current index on
102that string, and the matcher functions execute regexes against this state.
103
104## Authoring Guide
105
106You can write "matchers" by importing the `match` import from `reghex` and
107using it to write a matcher expression.
108
109```js
110import { match } from 'reghex';
111
112const name = match('name')`
113 ${/\w+/}
114`;
115```
116
117As can be seen above, the `match` function, is called with a "node name" and
118is then called as a tagged template. This template is our **parsing definition**.
119
120`reghex` functions only with its Babel plugin, which will detect `match('name')`
121and replace the entire tag with a parsing function, which may then look like
122the following in your transpiled code:
123
124```js
125import { _pattern /* ... */ } from 'reghex';
126
127var _name_expression = _pattern(/\w+/);
128var name = function name() {
129 /* ... */
130};
131```
132
133We've now successfully created a matcher, which matches a single regex, which
134is a pattern of one or more letters. We can execute this matcher by calling
135it with the curried `parse` utility:
136
137```js
138import { parse } from 'reghex';
139
140const result = parse(name)('Tim');
141
142console.log(result); // [ "Tim", .tag = "name" ]
143console.log(result.tag); // "name"
144```
145
146If the string (Here: "Tim") was parsed successfully by the matcher, it will
147return an array that contains the result of the regex. The array is special
148in that it will also have a `tag` property set to the matcher's name, here
149`"name"`, which we determined when we defined the matcher as `match('name')`.
150
151```js
152import { parse } from 'reghex';
153parse(name)('42'); // undefined
154```
155
156Similarly, if the matcher does not parse an input string successfully, it will
157return `undefined` instead.
158
159### Nested matchers
160
161This on its own is nice, but a parser must be able to traverse a string and
162turn it into an [Abstract Syntax Tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree).
163To introduce nesting to `reghex` matchers, we can refer to one matcher in another!
164Let's extend our original example;
165
166```js
167import match from 'reghex';
168
169const name = match('name')`
170 ${/\w+/}
171`;
172
173const hello = match('hello')`
174 ${/hello /} ${name}
175`;
176```
177
178The new `hello` matcher is set to match `/hello /` and then attempts to match
179the `name` matcher afterwards. If either of these matchers fail, it will return
180`undefined` as well and roll back its changes. Using this matcher will give us
181**nested abstract output**.
182
183We can also see in this example that _outside_ of the regex interpolations,
184whitespace and newlines don't matter.
185
186```js
187import { parse } from 'reghex';
188
189parse(hello)('hello tim');
190/*
191 [
192 "hello",
193 ["tim", .tag = "name"],
194 .tag = "hello"
195 ]
196*/
197```
198
199### Regex-like DSL
200
201We've seen in the previous examples that matchers are authored using tagged
202template literals, where interpolations can either be filled using regexes,
203`${/pattern/}`, or with other matchers `${name}`.
204
205The tagged template syntax supports more ways to match these interpolations,
206using a regex-like Domain Specific Language. Unlike in regexes, whitespace
207and newlines don't matter, which makes it easier to format and read matchers.
208
209We can create **sequences** of matchers by adding multiple expressions in
210a row. A matcher using `${/1/} ${/2/}` will attempt to match `1` and then `2`
211in the parsed string. This is just one feature of the regex-like DSL. The
212available operators are the following:
213
214| Operator | Example | Description |
215| -------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
216| `?` | `${/1/}?` | An **optional** may be used to make an interpolation optional. This means that the interpolation may or may not match. |
217| `*` | `${/1/}*` | A **star** can be used to match an arbitrary amount of interpolation or none at all. This means that the interpolation may repeat itself or may not be matched at all. |
218| `+` | `${/1/}+` | A **plus** is used like `*` and must match one or more times. When the matcher doesn't match, that's considered a failing case, since the match isn't optional. |
219| `\|` | `${/1/} \| ${/2/}` | An **alternation** can be used to match either one thing or another, falling back when the first interpolation fails. |
220| `()` | `(${/1/} ${/2/})+` | A **group** can be used to apply one of the other operators to an entire group of interpolations. |
221| `(?: )` | `(?: ${/1/})` | A **non-capturing group** is like a regular group, but the interpolations matched inside it don't appear in the parser's output. |
222| `(?= )` | `(?= ${/1/})` | A **positive lookahead** checks whether interpolations match, and if so continues the matcher without changing the input. If it matches, it's essentially ignored. |
223| `(?! )` | `(?! ${/1/})` | A **negative lookahead** checks whether interpolations _don't_ match, and if so continues the matcher without changing the input. If the interpolations do match the matcher is aborted. |
224
225We can combine and compose these operators to create more complex matchers.
226For instance, we can extend the original example to only allow a specific set
227of names by using the `|` operator:
228
229```js
230const name = match('name')`
231 ${/tim/} | ${/tom/} | ${/tam/}
232`;
233
234parse(name)('tim'); // [ "tim", .tag = "name" ]
235parse(name)('tom'); // [ "tom", .tag = "name" ]
236parse(name)('patrick'); // undefined
237```
238
239The above will now only match specific name strings. When one pattern in this
240chain of **alternations** does not match, it will try the next one.
241
242We can also use **groups** to add more matchers around the alternations themselves,
243by surrounding the alternations with `(` and `)`
244
245```js
246const name = match('name')`
247 (${/tim/} | ${/tom/}) ${/!/}
248`;
249
250parse(name)('tim!'); // [ "tim", "!", .tag = "name" ]
251parse(name)('tom!'); // [ "tom", "!", .tag = "name" ]
252parse(name)('tim'); // undefined
253```
254
255Maybe we're also not that interested in the `"!"` showing up in the output node.
256If we want to get rid of it, we can use a **non-capturing group** to hide it,
257while still requiring it.
258
259```js
260const name = match('name')`
261 (${/tim/} | ${/tom/}) (?: ${/!/})
262`;
263
264parse(name)('tim!'); // [ "tim", .tag = "name" ]
265parse(name)('tim'); // undefined
266```
267
268Lastly, like with regexes, `?`, `*`, and `+` may be used as "quantifiers". The first two
269may also be optional and _not_ match their patterns without the matcher failing.
270The `+` operator is used to match an interpolation _one or more_ times, while the
271`*` operators may match _zero or more_ times. Let's use this to allow the `"!"`
272to repeat.
273
274```js
275const name = match('name')`
276 (${/tim/} | ${/tom/})+ (?: ${/!/})*
277`;
278
279parse(name)('tim!'); // [ "tim", .tag = "name" ]
280parse(name)('tim!!!!'); // [ "tim", .tag = "name" ]
281parse(name)('tim'); // [ "tim", .tag = "name" ]
282parse(name)('timtim'); // [ "tim", tim", .tag = "name" ]
283```
284
285As we can see from the above, like in regexes, quantifiers can be combined with groups,
286non-capturing groups, or other groups.
287
288### Transforming as we match
289
290In the previous sections, we've seen that the **nodes** that `reghex` outputs are arrays containing
291match strings or other nodes and have a special `tag` property with the node's type.
292We can **change this output** while we're parsing by passing a function to our matcher definition.
293
294```js
295const name = match('name', (x) => x[0])`
296 (${/tim/} | ${/tom/}) ${/!/}
297`;
298
299parse(name)('tim'); // "tim"
300```
301
302In the above example, we're passing a small function, `x => x[0]` to the matcher as a
303second argument. This will change the matcher's output, which causes the parser to
304now return a new output for this matcher.
305
306We can use this function creatively by outputting full AST nodes, maybe even like the
307ones that resemble Babel's output:
308
309```js
310const identifier = match('identifier', (x) => ({
311 type: 'Identifier',
312 name: x[0],
313}))`
314 ${/[\w_][\w\d_]+/}
315`;
316
317parse(name)('var_name'); // { type: "Identifier", name: "var_name" }
318```
319
320We've now entirely changed the output of the parser for this matcher. Given that each
321matcher can change its output, we're free to change the parser's output entirely.
322By **returning a falsy value** in this matcher, we can also change the matcher to not have
323matched, which would cause other matchers to treat it like a mismatch!
324
325```js
326import match, { parse } from 'reghex';
327
328const name = match('name')((x) => {
329 return x[0] !== 'tim' ? x : undefined;
330})`
331 ${/\w+/}
332`;
333
334const hello = match('hello')`
335 ${/hello /} ${name}
336`;
337
338parse(name)('tom'); // ["hello", ["tom", .tag = "name"], .tag = "hello"]
339parse(name)('tim'); // undefined
340```
341
342Lastly, if we need to create these special array nodes ourselves, we can use `reghex`'s
343`tag` export for this purpose.
344
345```js
346import { tag } from 'reghex';
347
348tag(['test'], 'node_name');
349// ["test", .tag = "node_name"]
350```
351
352**That's it! May the RegExp be ever in your favor.**