MarkdownDB

Turn markdown files into structured, queryable data (SQL and JSON). Build r...

MarkdownDatabase
datopian/markdowndbmarkdowndb.commddb

README

MarkdownDB


MarkdownDB is a javascript library that turns markdown files into structured queryable databaase (SQL-based and simple JSON). It helps you build rich markdown-powered sites easily and reliably. Specifically it:

- Parses your markdown files to extract structured data (frontmatter, tags etc) and builds a queryable index either in JSON files or a local SQLite database
- Provides a lightweight javascript API for querying the index and using the data files into your application

Features and Roadmap


- [x] Index a folder of files - create a db index given a folder of markdown and other files
  - [x] Command line tool for indexing: Create a markdowndb (index) on the command line v0.1
  - [x] SQL(ite) index v0.2
  - [x] JSON index v0.6
  - [ ] BONUS Index multiple folders (with support for configuring e.g. prefixing in some way e.g. i have all my blog files in this separate folder over here)
  - [x] Configuration for Including/Excluding Files in the folder

Extract structured data like:

- [x] Frontmatter metadata: Extract markdown frontmatter and add in a metadata field
  - [ ] deal with casting types e.g. string, number so that we can query in useful ways e.g. find me all blog posts before date X
- [x] Tags: Extracts tags in markdown pages
  - [x] Extract tags in frontmatter v0.1
  - [x] Extract tags in body like #abc v0.5
- [x] Links: links between files like hello or wikilink style [[xyz]] so we can compute backlinks or deadlinks etc (see #4) v0.2
- [x] Tasks: extract tasks like this - [ ] this is a task (See obsidian data view) v0.4

Data enhancement and validation

- [x] Computed fields: add new metadata properties based on existing metadata e.g. a slug field computed from title field; or, adding a title based on the first h1 heading in a doc; or, a type field based on the folder of the file (e.g. these are blog posts). cf https://www.contentlayer.dev/docs/reference/source-files/define-document-type#computedfields.
- [ ] 🚧 Data validation and Document Types: validate metadata against a schema/type so that I know the data in the database is "valid" #55
  - [ ] BYOT (bring your own types): i want to create my own types ... so that when i get an object out it is cast to the right typescript type

Quick start


Have a folder of markdown content


For example, your blog posts. Each file can have a YAML frontmatter header with metadata like title, date, tags, etc.

  1. ```md
  2. ---
  3. title: My first blog post
  4. date: 2021-01-01
  5. tags: [a, b, c]
  6. author: John Doe
  7. ---

  9. # My first blog post

  11. This is my first blog post.
  12. I'm using MarkdownDB to manage my blog posts.
  13. ```

Index the files with MarkdownDB


Use the npm mddb package to index Markdown files into an SQLite database. This will create a markdown.db file in the current directory. You can preview it with any SQLite viewer, e.g. https://sqlitebrowser.org/.

  1. ```bash
  2. # npx mddb <path-to-folder-with-your-md-files>
  3. npx mddb ./blog
  4. ```

Watching for Changes


To monitor files for changes and update the database accordingly, simply add the --watch flag to the command:
  1. ```bash
  2. npx mddb ./blog --watch
  3. ```
This command will continuously watch for any modifications in the specified folder (./blog), automatically rebuilding the database whenever a change is detected.

Query your files with SQL...


E.g. get all the files with with tag a.

  1. ```sql
  2. SELECT files.*
  3. FROM files
  4. INNER JOIN file_tags ON files._id = file_tags.file
  5. WHERE file_tags.tag = 'a'
  6. ```

...or using MarkdownDB Node.js API in a framework of your choice!


Use our Node API to query your data for your blog, wiki, docs, digital garden, or anything you want!

Install mddb package in your project:

  1. ```bash
  2. npm install mddb
  3. ```

Now, once the data is in the database, you can add the following script to your project (e.g. in /lib folder). It will allow you to establish a single connection to the database and use it across you app.

  1. ```js
  2. // @/lib/mddb.mjs
  3. import { MarkdownDB } from "mddb";

  5. const dbPath = "markdown.db";

  7. const client = new MarkdownDB({
  8.   client: "sqlite3",
  9.   connection: {
  10.     filename: dbPath,
  11.   },
  12. });

  14. const clientPromise = client.init();

  16. export default clientPromise;
  17. ```

Now, you can import it across your project to query the database, e.g.:

  1. ```js
  2. import clientPromise from "@/lib/mddb";

  4. const mddb = await clientPromise;
  5. const blogs = await mddb.getFiles({
  6.   folder: "blog",
  7.   extensions: ["md", "mdx"],
  8. });
  9. ```

Computed Fields


This feature helps you define functions that compute additional fields you want to include.

Step 1: Define the Computed Field Function


Next, define a function that computes the additional field you want to include. In this example, we have a function named addTitle that extracts the title from the first heading in the AST (Abstract Syntax Tree) of a Markdown file.

  1. ```javascript
  2. const addTitle = (fileInfo, ast) => {
  3.   // Find the first header node in the AST
  4.   const headerNode = ast.children.find((node) => node.type === "heading");

  6.   // Extract the text content from the header node
  7.   const title = headerNode
  8.     ? headerNode.children.map((child) => child.value).join("")
  9.     : "";

  11.   // Add the title property to the fileInfo
  12.   fileInfo.title = title;
  13. };
  14. ```

Step 2: Indexing the Folder with Computed Fields


Now, use the client.indexFolder method to scan and index the folder containing your Markdown files. Pass the addTitle function in the computedFields option array to include the computed title in the database.

  1. ```javascript
  2. client.indexFolder(folderPath: "PATH_TO_FOLDER", customConfig: { computedFields: [addTitle] });
  3. ```