1.38.0tokenize
Splits a string into word-like tokens using Unicode-aware letter, number, mark, and casing boundaries.
tokenize(value):string[]Word characters
A character is considered part of a token when it is a Unicode letter, Unicode number, or Unicode mark. Characters outside these categories terminate the current token.
Lowercase to uppercase boundaries
A boundary is created when the current character is uppercase, the previous character is lowercase or a number, and the next character is lowercase.
Uppercase runs
Consecutive uppercase characters are analyzed as a run. Additional prefix and run-length checks determine whether an uppercase sequence should be separated, which allows acronym-like sequences to remain together in relevant cases.
Separators
Characters that are not Unicode letters, numbers, or marks terminate the current token. Empty tokens are not added to the result.
Unicode handling
The input is converted with Array.from before processing, so iteration operates on Unicode code points rather than UTF-16 code units. The character classification uses Unicode property escapes.
Parameters
valuestringrequiredThe string to split into tokens.
Returns
string[]An array of tokens extracted from the input string.
Examples
Basic usage
Separators such as spaces are used to separate tokens.
tokenize("hello world");
// ["hello", "world"]Punctuation
Non-word characters separate the surrounding word characters.
tokenize("hello-world!");
// ["hello", "world"]CamelCase
A lowercase-to-uppercase transition can create a word boundary.
tokenize("helloWorld");
// ["hello", "World"]PascalCase
PascalCase identifiers are split according to the casing boundary rules.
tokenize("HelloWorld");
// ["Hello", "World"]Acronyms
Uppercase runs are handled specially to avoid splitting every uppercase character into a separate token.
tokenize("parseHTTPResponse");
// ["parse", "HTTP", "Response"]Numbers
Unicode number characters are treated as word characters.
tokenize("version2Update");
// ["version2", "Update"]Unicode text
Unicode letters are preserved as word characters.
tokenize("سلام دنیا");
// ["سلام", "دنیا"]Empty input
An empty string produces no tokens.
tokenize("");
// []