liqe

Lightweight and performant Lucene-like parser, serializer and search engine...

SearchFull-text search
gajus/liqeliqe

README

liqe

Travis build status Coveralls NPM version Canonical Code Style Twitter Follow

Lightweight and performant Lucene-like parser, serializer and search engine.

Motivation
Usage
Query Syntax
  Liqe syntax cheat sheet
  Keyword matching
  Number matching
  Range matching
  Wildcard matching
  Boolean operators
Serializer
AST
Utilities
Compatibility with Lucene
Recipes
  Handling syntax errors
  Highlighting matches
Development
Tutorials

Motivation


Originally built Liqe to enable Roarr log filtering via cli. I have since been polishing this project as a hobby/intellectual exercise. I've seen it being adopted by various CLI and web applications that require advanced search. To my knowledge, it is currently the most complete Lucene-like syntax parser and serializer in JavaScript, as well as a compatible in-memory search engine.

Liqe use cases include:

parsing search queries
serializing parsed queries
searching JSON documents using the Liqe query language (LQL)

Note that the Liqe AST is treated as a public API, i.e., one could implement their own search mechanism that uses Liqe query language (LQL).

Usage


  1. ```ts
  2. import {
  3.   filter,
  4.   highlight,
  5.   parse,
  6.   test,
  7. } from 'liqe';

  9. const persons = [
  10.   {
  11.     height: 180,
  12.     name: 'John Morton',
  13.   },
  14.   {
  15.     height: 175,
  16.     name: 'David Barker',
  17.   },
  18.   {
  19.     height: 170,
  20.     name: 'Thomas Castro',
  21.   },
  22. ];
  23. ```

Filter a collection:

  1. ```ts
  2. filter(parse('height:>170'), persons);
  3. // [
  4. //   {
  5. //     height: 180,
  6. //     name: 'John Morton',
  7. //   },
  8. //   {
  9. //     height: 175,
  10. //     name: 'David Barker',
  11. //   },
  12. // ]
  13. ```

Test a single object:

  1. ```ts
  2. test(parse('name:John'), persons[0]);
  3. // true
  4. test(parse('name:David'), persons[0]);
  5. // false
  6. ```

Highlight matching fields and substrings:

  1. ```ts
  2. test(highlight('name:john'), persons[0]);
  3. // [
  4. //   {
  5. //     path: 'name',
  6. //     query: /(John)/,
  7. //   }
  8. // ]
  9. test(highlight('height:180'), persons[0]);
  10. // [
  11. //   {
  12. //     path: 'height',
  13. //   }
  14. // ]
  15. ```

Query Syntax


Liqe uses Liqe Query Language (LQL), which is heavily inspired by Lucene but extends it in various ways that allow a more powerful search experience.

Liqe syntax cheat sheet


  1. ```rb
  2. # search for "foo" term anywhere in the document (case insensitive)
  3. foo

  5. # search for "foo" term anywhere in the document (case sensitive)
  6. 'foo'
  7. "foo"

  9. # search for "foo" term in `name` field
  10. name:foo

  12. # search for "foo" term in `full name` field
  13. 'full name':foo
  14. "full name":foo

  16. # search for "foo" term in `first` field, member of `name`, i.e.
  17. # matches {name: {first: 'foo'}}
  18. name.first:foo

  20. # search using regex
  21. name:/foo/
  22. name:/foo/o

  24. # search using wildcard
  25. name:foo*bar

  27. # boolean search
  28. member:true
  29. member:false

  31. # null search
  32. member:null

  34. # search for age =, >, >=, <, <=
  35. height:=100
  36. height:>100
  37. height:>=100
  38. height:<100
  39. height:<=100

  41. # search for height in range (inclusive, exclusive)
  42. height:[100 TO 200]
  43. height:{100 TO 200}

  45. # boolean operators
  46. name:foo AND height:=100
  47. name:foo OR name:bar

  49. # unary operators
  50. NOT foo
  51. -foo
  52. NOT foo:bar
  53. -foo:bar
  54. name:foo AND NOT (bio:bar OR bio:baz)

  56. # implicit AND boolean operator
  57. name:foo height:=100

  59. # grouping
  60. name:foo AND (bio:bar OR bio:baz)
  61. ```

Keyword matching


Search for word "foo" in any field (case insensitive).

  1. ```rb
  2. foo
  3. ```

Search for word "foo" in the name field.

  1. ```rb
  2. name:foo
  3. ```

Search for name field values matching /foo/i regex.

  1. ```rb
  2. name:/foo/i
  3. ```

Search for name field values matching f*o wildcard pattern.

  1. ```rb
  2. name:f*o
  3. ```

Search for phrase "foo bar" in the name field (case sensitive).

  1. ```rb
  2. name:"foo bar"
  3. ```

Number matching


Search for value equal to 100 in the height field.

  1. ```rb
  2. height:=100
  3. ```

Search for value greater than 100 in the height field.

  1. ```rb
  2. height:>100
  3. ```

Search for value greater than or equal to 100 in the height field.

  1. ```rb
  2. height:>=100
  3. ```

Range matching


Search for value greater or equal to 100 and lower or equal to 200 in the height field.

  1. ```rb
  2. height:[100 TO 200]
  3. ```

Search for value greater than 100 and lower than 200 in the height field.

  1. ```rb
  2. height:{100 TO 200}
  3. ```

Wildcard matching


Search for any word that starts with "foo" in the name field.

  1. ```rb
  2. name:foo*
  3. ```

Search for any word that starts with "foo" and ends with bar in the name field.

  1. ```rb
  2. name:foo*bar
  3. ```

Boolean operators


Search for phrase "foo bar" in the name field AND the phrase "quick fox" in the bio field.

  1. ```rb
  2. name:"foo bar" AND bio:"quick fox"
  3. ```

Search for either the phrase "foo bar" in the name field AND the phrase "quick fox" in the bio field, or the word "fox" in the name field.

  1. ```rb
  2. (name:"foo bar" AND bio:"quick fox") OR name:fox
  3. ```

Serializer


Serializer allows to convert Liqe tokens back to the original search query.

  1. ```ts
  2. import {
  3.   parse,
  4.   serialize,
  5. } from 'liqe';

  7. const tokens = parse('foo:bar');

  9. // {
  10. //   expression: {
  11. //     location: {
  12. //       start: 4,
  13. //     },
  14. //     quoted: false,
  15. //     type: 'LiteralExpression',
  16. //     value: 'bar',
  17. //   },
  18. //   field: {
  19. //     location: {
  20. //       start: 0,
  21. //     },
  22. //     name: 'foo',
  23. //     path: ['foo'],
  24. //     quoted: false,
  25. //     type: 'Field',
  26. //   },
  27. //   location: {
  28. //     start: 0,
  29. //   },
  30. //   operator: {
  31. //     location: {
  32. //       start: 3,
  33. //     },
  34. //     operator: ':',
  35. //     type: 'ComparisonOperator',
  36. //   },
  37. //   type: 'Tag',
  38. // }

  40. serialize(tokens);
  41. // 'foo:bar'
  42. ```

AST


  1. ```ts
  2. import {
  3.   type BooleanOperatorToken,
  4.   type ComparisonOperatorToken,
  5.   type EmptyExpression,
  6.   type FieldToken,
  7.   type ImplicitBooleanOperatorToken,
  8.   type ImplicitFieldToken,
  9.   type LiteralExpressionToken,
  10.   type LogicalExpressionToken,
  11.   type RangeExpressionToken,
  12.   type RegexExpressionToken,
  13.   type TagToken,
  14.   type UnaryOperatorToken,
  15. } from 'liqe';
  16. ```

There are 11 AST tokens that describe a parsed Liqe query.

If you are building a serializer, then you must implement all of them for the complete coverage of all possible query inputs. Refer to the built-in serializer for an example.

Utilities


  1. ```ts
  2. import {
  3.   isSafeUnquotedExpression,
  4. } from 'liqe';

  6. /**
  7. * Determines if an expression requires quotes.
  8. * Use this if you need to programmatically manipulate the AST
  9. * before using a serializer to convert the query back to text.
  10. */
  11. isSafeUnquotedExpression(expression: string): boolean;
  12. ```

Compatibility with Lucene


The following Lucene abilities are not supported:

Fuzzy Searches
Proximity Searches
Boosting a Term

Recipes


Handling syntax errors


In case of a syntax error, Liqe throws SyntaxError.

  1. ```ts
  2. import {
  3.   parse,
  4.   SyntaxError,
  5. } from 'liqe';

  7. try {
  8.   parse('foo bar');
  9. } catch (error) {
  10.   if (error instanceof SyntaxError) {
  11.     console.error({
  12.       // Syntax error at line 1 column 5
  13.       message: error.message,
  14.       // 4
  15.       offset: error.offset,
  16.       // 1
  17.       offset: error.line,
  18.       // 5
  19.       offset: error.column,
  20.     });
  21.   } else {
  22.     throw error;
  23.   }
  24. }
  25. ```

Highlighting matches


Consider using [highlight-words](https://github.com/tricinel/highlight-words) package to highlight Liqe matches.

Development


Compiling Parser


If you are going to modify parser, then use npm run watch to run compiler in watch mode.

Benchmarking Changes


Before making any changes, capture the current benchmark on your machine using npm run benchmark. Run benchmark again after making any changes. Before committing changes, ensure that performance is not negatively impacted.


Tutorials


Building advanced SQL search from a user text input