tokenizer: capture non-ASCII identifiers

[smola]

\w captures only ASCII letters and numbers. Changed to [[:alnum:]]
to capture any Unicode letter and digit. This makes the tokenizer
work properly on non-English based langiages (e.g. 1C Enterprise).

[Alhadis]

\w captures only ASCII letters and numbers.

That's dependent on the regex engine in question. Ruby uses Oniguruma, which is Unicode-aware by default, IIRC. Did you test to see if \w doesn't match word-characters in other alphabets?

[smola]

@Alhadis If I run the test I added without the fix, this is its output:

  1) Failure:
TestTokenizer#test_utf8_tokens [test/test_tokenizer.rb:118]:
Expected: ["Функция", "الكون"]
  Actual: []

[pchaigno]

@Alhadis [[:alnum:]] is safer in any case, no?

[Alhadis]

Possibly; it seems like Oniguruma only defaults to Unicode in Atom, but not Ruby... never mind then. :)

[smola]

Closing. I guess #3846 makes this obsolete.