Dangit. Now I got unterminated string errors etc. I understand /why/ (tokenize is buggy, doh).
Its not a bug if you read the (*¤%/) documentation of tokenizer it becomes clear.
The firts sting is what to tokenize, the second string si NOT the pattern to seek but a list of things to search for So if you were to do
tokenize "aqschbq" "abc" $target;
then you would get the list:
"q","s","h" and "q".
This is intentional, exactly what they MEANT TO DO ergo no bug (it would be a bug if it was intended to work like you think it was not). Just because you think something works like you think does mean it was even supposed to. Alas that means:
I was sure to disambiguish the separator with ||| :?
No disabiguation theres just no such thing in the tool that can tokenize based on sequences. So unfortunately the tool will never do what you want.
I would suggest you use a character mayas guaranteed to never use i would choose something like \b (bell, note its not 2 characters its one escape sequance) its pretty damn guaranteed not to be used by almost anybody. Maya also never uses \n in names this ones perhaps even better, and natural, as it makes debugging the strings extremely easy just dump into any place an a human will see them as lines (same applies to \t), Other things guaranteed never to exist in mayas names are !#¤%&/()='´`£@, and so on
tokenize "test|with|real|separator\btest|cool|separator" "\b" $a;