node-llama-cpp

Run AI models locally on your machine with node.js bindings for llama.cpp. ...

Machine Learning
withcatai/node-llama-cppwithcatai.github.io/node-llama-cpp/node-llama-cpp

README

node-llama-cpp Logo

node-llama-cpp


Run AI models locally on your machine

Pre-built bindings are provided with a fallback to building from source with cmake

Features

Run a text generation model locally on your machine
Metal and CUDA support
Pre-built binaries are provided, with a fallback to building from source without node-gyp or Python
Chat with a model using a chat wrapper
Use the CLI to chat with a model without writing any code
Up-to-date with the latest version of llama.cpp. Download and compile the latest release with a single CLI command.
Force a model to generate output in a parseable format, like JSON, or even force it to follow a specific JSON schema

Documentation

Getting started guide
API reference
CLI help
Changelog
Roadmap

Installation

  1. ```bash
  2. npm install --save node-llama-cpp
  3. ```

This package comes with pre-built binaries for macOS, Linux and Windows.

If binaries are not available for your platform, it'll fallback to download the latest version of llama.cpp and build it from source with cmake.
To disable this behavior set the environment variable NODE_LLAMA_CPP_SKIP_DOWNLOAD to true.

Usage

  1. ```typescript
  2. import {fileURLToPath} from "url";
  3. import path from "path";
  4. import {LlamaModel, LlamaContext, LlamaChatSession} from "node-llama-cpp";

  6. const __dirname = path.dirname(fileURLToPath(import.meta.url));

  8. const model = new LlamaModel({
  9.     modelPath: path.join(__dirname, "models", "codellama-13b.Q3_K_M.gguf")
  10. });
  11. const context = new LlamaContext({model});
  12. const session = new LlamaChatSession({context});


  15. const q1 = "Hi there, how are you?";
  16. console.log("User: " + q1);

  18. const a1 = await session.prompt(q1);
  19. console.log("AI: " + a1);


  22. const q2 = "Summerize what you said";
  23. console.log("User: " + q2);

  25. const a2 = await session.prompt(q2);
  26. console.log("AI: " + a2);
  27. ```

For more examples, see the getting started guide


Contributing

To contribute to node-llama-cpp read the contribution guide.

Acknowledgements

llama.cpp: ggerganov/llama.cpp