Manticore Search's default keywords dictionary truncates tokens to 42 bytes after normalization, which breaks searching long values like SHA-256 hashes, message IDs, and long email addresses since different IDs sharing the same first 42 bytes become indistinguishable. The dict='keywords_32k' option, available since Manticore Search 27.1.1 (27.1.5+ recommended for migrations), raises the maximum token length to 32768 bytes, skipping oversized tokens with a warning instead of truncating them. The guide covers schema choices (string attribute vs string attribute indexed), exact vs full-text matching semantics, prefix/infix search with min_infix_len, blend_chars for emails, CALL KEYWORDS for inspecting tokenization, migration steps for RT and plain tables, why dict='crc' doesn't help, current limitations (no CALL SUGGEST, no percolate tables, no snippet highlighting, no full-text REGEX), and a warning against indexing secrets like API keys and tokens.
Table of contents
The problem with the regular dictionaryWhat keywords_32k changesWhen to use keywords_32kSearching by the complete tokenA full-text match is not the same as exact equalityHow to inspect tokenizationPrefix and substring searchEmail addresses and other values with separatorsHow to convert an existing tableWhy dict='crc' does not solve this problemCurrent limitationsDo not index secretsQuick checklistSummaryDocumentationQuestions this post answers
Why does searching for a full SHA-256 hash in Manticore Search return the wrong document?
The default dict='keywords' dictionary truncates tokens to 42 bytes after normalization, both when indexing and when querying. Since a SHA-256 hash is 64 characters, two different hashes sharing the same first 42 bytes become indistinguishable, and a search on the full value can match the wrong document because the query itself gets truncated too. Track dictionary and indexing changes like this on daily.dev before they cause silent search bugs in production.
How do I search for long hashes or IDs longer than 42 bytes in Manticore Search?
Use dict='keywords_32k', available starting with Manticore Search 27.1.1 (27.1.5 or newer recommended for migrating existing tables), which raises the maximum normalized token length to 32768 bytes instead of the default 42. Tokens exceeding the new limit are skipped with a warning rather than silently truncated, and prefix and infix wildcard search remain supported. Developers evaluating search engine configs for log and ID data can follow releases like this on daily.dev.
Does converting a Manticore Search table to dict='keywords_32k' automatically fix existing indexed documents?
No, changing dict on an RT table via ALTER TABLE only affects documents added or replaced afterward; existing documents keep their old truncated tokens until reindexed. For plain tables, the configuration must be updated and the table rebuilt entirely from its data source, so a table can temporarily contain both truncated and complete tokens for seemingly identical documents. Anyone planning a dictionary migration can use daily.dev to stay ahead of gotchas like partial reindexing.