If the user needs a hard guarantee of not-horrible performance, they need to know that they
should use a matcher
that can use a non-backtracking algorithm, and need to know what
properties of an ARE will ensure a non-backtracking match. Fortunately, most AREs can be
matched in a non-backtracking way (AREs containing backreferences being the most prominent exception).

If the user is in a situation where it is OK for the matcher to simply run out of time or memory
and fail, they can use whatever matcher they desire—and indeed, most backtracking matchers perform
acceptably for most AREs and input strings. The problem is that an ARE that works perfectly fine
with a backtracking matcher on most input strings may well suddenly blow up in time or space
when presented with just the “wrong” one. The user who hasn’t anticipated and planned for
this possibility will find themselves in a bad situation.

CS researchers now know how to build matchers that perform at least as well as backtracking
matchers on all inputs, that implement the same semantics as backtracking matchers,
and that run in time proportional only to the length of the input string for a wide and predictable
variety of AREs. These matchers are not terribly difficult to build, and source code is available. It would
behoove those choosing
ARE matchers to choose that kind: there is no downside. Users of ARE-based software that
care about performance need to
be aware of which matching algorithm is being used, and to control what sort of ARE and/or input
string is fed to the matcher so that they get the performance guarantees they need.

Efficiency is a running theme through the whole book, because as I wrote (in the quote from the book, cited above), “So long as you know what you can expect from it (something this chapter will show you), you know all you need to care about.“. There are plenty of cases for which an NFA that uses backtracking is order(N) on the length of the string, and cases, too, where changing one character of the regex turns it into a never-ending computational bomb. Knowing what you can expect from a regex engine encompasses all that. —Jeffrey

It’s not that the regular expression that Russ Cox used in his article cannot be transformed (it can be, and trivially at that). Rather, the point is that when using NDFAs and DFAs, the type of exponential blow up that happens in Perl’s backtracking implementation doesn’t happen. Indeed, it cannot happen.

Jeffrey is right in that the end user shouldn’t necessarily have to be aware of the underlying theory in order to use the implementation (though it couldn’t hurt). However, that doesn’t mean that it’s okay to abuse the theory by calling it something that it most definitely is not. Words have meaning, terms have definitions, and just because you choose to ignore them doesn’t make you correct. Millions of people say “ain’t” every day; that doesn’t make it proper English.

The fact is that, as much as practitioners may abuse them, these terms have formal definitions. Those definitions don’t need to, nor will they, change just because Perl screwed it up by making regular expressions not regular anymore. It’s really not the computer scientists fault. It’s Larry Wall’s (or maybe Henry Spencer’s) fault for not understanding the theory in the first place.

If anything, this causes greater confusion because someone who’s curious about the underlying theory may go read up on it, and then not understand why their code doesn’t work the same way. This contributes to the illusion that the theory is impenetrable and irrelevant. It may also cause a programmer to charge off, confidently assuming that his or her usage of regular expressions is correct and guaranteed to run in time linear in the size of the string being parsed with bounded memory usage, only to find that the thing actually blows up.

A much more sensible approach would be to just call a spade a spade: Perl et al don’t use NFAs. They really don’t, despite what people keep insisting. So, simply say what they do do: they use backtracking, which is subject to exponential run times and memory consumption. There’s nothing wrong with that, as long as people know that’s what is happening and plan accordingly. So just tell them, so that they can be better prepared to make use of the tools that the language provides because they understand the potential consequences.

http://swtch.com/~rsc/regexp/regexp1.html is stupidely based on a cornercase that any formal regex analyser would solve in microseconds: (a?){20}a{20} is better written as a{20}(a?){20} (when the nfa meets the dfa …. ). Backtracking explosion is gone, There is also a load of formal negated logic which can be applied to reduce similar linear and quadratic explosions (even without using assertions). Identically, a lot of backscratching is prevented in engines which detect | as a xor, i.e. exclusive alternations, or can scan backwards when necessary.
Can even implement a few fail transitions in a NFA and then rebrand it more-deterministic-less-stupid-nfa. That would be a good compromize. Would any user care about it ?

Jeffrey, you are right against the theoricists and talebans of the FA classification as frozen in the 60′s (half a century ago).

[quote]
This document summarizes Apocalypse 5, which is about the new regex syntax. We now try to call them regex rather than “regular expressions” because they haven’t been regular expressions for a long time,
[/quote]

If you want to read more, see:
http://dev.perl.org/perl6/doc/design/syn/S05.html

Forum thread on which the documentation is mentioned:
http://forum.java.sun.com/thread.jspa?threadID=768745

I don’t know whether it is a good news or bad news.