Cogitator Documentation¶

Overview¶

Cogitator is a Python toolkit for experimenting and working with chain-of-thought (CoT) prompting methods in large language models (LLMs). CoT prompting improves LLM performance on complex tasks (like question-answering, reasoning, and problem-solving) by guiding the models to generate intermediate reasoning steps before arriving at the final answer. Additionally, it can be used to improve the interpretability of LLMs by providing insight into the model's reasoning process. The toolkit aims to make it easier to use popular CoT strategies and frameworks for research or integrating them into AI applications.

Features¶

Provides unified sync/async API for CoT strategies
Supports using OpenAI and Ollama as LLM providers
Supports structured model output with Pydantic validation
Includes a customizable benchmarking framework (see benches)
Includes implementations of popular CoT strategies and frameworks like

The diagram below shows a high-level overview of Cogitator's workflow.

Cogitator Architecture

Installation¶

Cogitator can be installed via pip using the following command:

pip install cogitator

To run the unit tests and benchmarks, development dependencies are needed that can be installed with the following command:

git clone https://github.com/habedi/cogitator && cd cogitator

# Set up Python environment
pip install poetry
poetry install --with dev

# Run the tests to make sure everything is working (optional)
poetry run pytest

Examples¶

Check the examples for usage examples on how to use the library with different LLM providers and CoT strategies.

API Reference¶

The Cogitator library's functionality is organized into several modules:

LLM Providers (cogitator.model)
- BaseLLM: Base LLM provider class that defines a common interface for all providers.
- OpenAILLM: LLM provider implementation for using OpenAI models (like gpt-4o-mini and gpt-4o).
- OllamaLLM: LLM provider implementation for using Ollama models (like Llama, Gemma, and Qwen).
CoT Strategies (cogitator.strategies)
- AutoCoT: An implementation of the automatic CoT prompting strategy.
- CDWCoT: An implementation of the clustered distance-weighted CoT framework.
- GraphOfThoughts: An implementation of the graph of thoughts CoT framework.
- LeastToMost: An implementation of the least-to-most prompting strategy.
- SelfConsistency: An implementation of the self-consistency prompting strategy.
- TreeOfThoughts: An implementation of the tree of thoughts CoT framework.
Data Formatting and Validation (cogitator.schemas)
- Schemas: A set of Pydantic models that are used for validating structure of outputs from LLMs.
Utilities
- Embedding: A set of tools for embedding prompt text which is used by strategies like AutoCoT and CDWCoT.
- Clustering: Includes clustering algorithms for grouping similar embeddings that is used during the training phase of strategies like AutoCoT and CDWCoT.
- Functions: A set of utility functions for working with the library.

Extra Resources¶

Benchmarking: Learn how to configure and run the performance evaluation framework.
Contributing: Find guidelines for contributing to the Cogitator project.