> ## Documentation Index
> Fetch the complete documentation index at: https://harisfazillah.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Reference: dsom_token_auditor.py

> Tiktoken-based token efficiency and context load calculation script. Deep State of Mind framework. Deep State of Mind framework. Deep State of Mind fram...

Tiktoken-based token efficiency and context load calculation script.

## Description

The `dsom_token_auditor.py` script compares non-DSOM "Bloated" context loads against optimised DSOM configurations (which utilise episodic resumes and progressive disclosures). It calculates savings percentages.

## Script path

`tools/dsom_token_auditor.py`

## CLI signature

```bash theme={null}
uv run --with tiktoken python tools/dsom_token_auditor.py

```

## Outputs

Prints an analysis report:

* Token counts for bloated scenario structures.
* Token counts for progressive disclosure scenario structures.
* Percentage and absolute token savings achieved per execution turn.

## Dependencies

* **tiktoken:** Fast BPE tokenization library.

## Internal Python API

### `count_tokens(text, model="gpt-4")`

Counts tokens for a given string using optimised tokenisation with the specified model's encoder.

* **Arguments:** `text` (raw string), `model` (defaults to `"gpt-4"`).
* **Returns:** Integer representation of token count.

### `generate_bloated_context()`

Generates a raw string simulating chat history and massive file loads.

### `generate_dsom_context()`

Generates a raw string simulating episodic records and relative link references.

***

*Deep State of Mind (DSOM) For My AI Protocol | Harisfazillah Jamel (LinuxMalaysia) | 2026-08-14*
*Standard: UK English | DBP-standard Bahasa Melayu Malaysia (Piawai) | GNU General Public License v3.0*


## Related topics

- [Reference Material Index](/reference/index.md)
- [DSOM Token Efficiency & Integration Report](/governance/dsom-token-efficiency-report.md)
- [DSOM Token Performance Playbook](/governance/dsom-token-performance-playbook.md)
- [🕵️ Palace Auditor Skill](/skills/palace-auditor.md)
- [🎓 DSOM Team Masterclass: Project Creation, GitOps & Multi-Agent Collaboration](/tutorials/team-dsom-masterclass.md)