system.tokenizers

View as MarkdownEdit this pageReport an issueSet up docs MCP

Description

Shows all available tokenizers. These can be used in functions tokens, hasAllTokens, hasAnyTokens, and the text index.

Columns

name (String) — Name of the tokenizer

Example

SELECT * FROM system.tokenizers;

┌─name────────────┐
│ ngrams          │
│ splitByNonAlpha │
│ sparseGrams     │
│ tokenbf_v1      │
│ ngrambf_v1      │
│ array           │
│ splitByString   │
│ sparse_grams    │
│ asciiCJK        │
│ icu             │
│ japanese        │
└─────────────────┘