Using \x in string literals is almost always a bad idea, but using it in regular expressions is particularly dangerous.

Consider this “don’t do” example in R 4.2.1 or earlier:

text <- "Hello\u00a0R"
gsub("\xa0", "", text)

a0 is the code point of the Unicode “NO-BREAK SPACE” and the example runs in UTF-8 locale. The intention is to remove the space; a slightly more complicated variant has been discussed on the R-devel mailing list about half a year ago.

The result is "Hello R", the space is not removed. While slightly contrived, this example gives a clue why:

text <- "Only ASCII <,>,a and digits: <a0><a1><a2>"
gsub("\xa0", "", text)

The result is ""Only ASCII <,>,a and digits: <a1><a2>", so the “<a0>” portion of the string is removed. The problem is that R converts the \xa0 in the regular expression to ASCII string "<a0>" before passing it to the regular expression engine.

R does it because the string is invalid. First, the parser expands \xa0 into the byte a0 and when gsub needs to convert the string to UTF16-LE for use with TRE, it can’t, because a0 is invalid UTF-8 (and we are running in UTF-8 locale). The code point a0 is instead encoded as c2 a0 in UTF-8. Thus, R escapes the invalid byte to "<a0>" and produces a valid UTF16-LE string, but that’s not the one intended. Additional checks are now in place in R-devel so that R would actually report an error (more later below) instead of escaping invalid bytes.

The origin of the problem is, however, a user error. The pattern is itself an invalid string. Likely this used to work when R was used in Latin-1 locale, where the a0 byte represents the no-break space, and maybe it was only tested there but not in other locales. Latin-1 in recent R would be used very rarely and hence this issue would now bite users much more.

To somewhat mitigate this issue, one could pass \x to the regular expression engine, so double the backslash in the regular expression. \\x is an ASCII string, hence will always be valid. But, see below.

By default (perl=FALSE, fixed=FALSE), POSIX extended regular expressions as described in ?regex are used, and those are not documented to support the \x escapes. While the currently used implementation, TRE, supports them, one should hence not use the feature (e.g. as a prevention against the case the implementation in R switches to a different engine). So, for this one should use the Perl regular expressions (perl=TRUE), which has other advantages, so this is not limiting modulo that one has to remember.

Notable advantages of Perl regular expressions include that one usually saves the encoding conversions to UTF-16LE (and back) and that one has access to Unicode properties. So, when re-visiting existing code to fix issues like this one, it may be beneficial to switch to Perl regular expressions, anyway (but this requires some care as the expressions are not exactly the same, see ?regex).

But, worse yet for using \\x escapes, it has the risk that the interpretation depends on the mode of the regular expression engine and can hence be still locale-specific. This example works in ISO-8859-2 (result is "cesky") but not in UTF-8 locale:

text <- "\u010desky"
text <- iconv(text, from="UTF-8", to="")
gsub("\\xe8", "c", text, perl=TRUE)

This works in UTF-8 locale, but not in ISO-8859-2:

text <- "\u010desky"
text <- iconv(text, from="UTF-8", to="")
gsub("\\x{010d}", "c", text, perl=TRUE)

The reason is that the first time the Perl regular expression is run in a locale mode (e8 is the code of “LATIN SMALL LETTER C WITH CARON”), the second time they run in UTF mode (where 010d is the code, the code point number).

Which mode is used depends on the locale and on the input strings. One can force the UTF mode by ensuring one of the input is in UTF-8 (excluding ASCII). But, still, if the text argument is a vector with multiple elements, we can’t pick just any element to convert to UTF-8, we have to pick one that is not ASCII. Or, all of them may be ASCII, then we would have to convert the pattern or the replacement. So perhaps convert all inputs explicitly to UTF-8? Technically, this would work, but is it worth the hassle?

There is a much easier way to ensure that the result is locale-independent (and that one of the inputs is UTF-8, specifically, the pattern):

text <- "Hello\u00a0R"
gsub("\u00a0", "", text)

and

text <- "\u010desky"
text <- iconv(text, from="UTF-8", to="")
gsub("\u010d", "c", text)

This works with the default (POSIX extended) regular expressions, with Perl regular expressions and with “fixed” expressions, because the Unicode (UTF-8) character is created by the parser. (\\x does not work with fixed expressions and is only documented to work with Perl regular expressions).

In principle, the first example of removing the no-break space could probably be often generalized to refer also to other kinds of spaces, e.g.  via Unicode properties, which are supported by the Perl regular expressions.

> text <- "Hello\u00a0R"
gsub("\xa0", "", text)
Error in gsub("\xa0", "", text) : 'pattern' is invalid
In addition: Warning message:
In gsub("\xa0", "", text) : unable to translate '<a0>' to a wide string