re_format —
POSIX regular expressions
TABLE OF CONTENTS
Regular expressions (REs), as defined in
IEEE Std
1003.1-2004 (“POSIX.1”), come in two forms: basic regular
expressions (BREs) and extended regular expressions (EREs). Both forms of
regular expressions are supported by the interfaces described in
regex(3). Applications dealing
with regular expressions may use one or the other form (or indeed both). For
example,
ed(1) uses BREs, whilst
egrep(1) talks EREs. Consult the
manual page for the specific application to find out which it uses.
POSIX leaves some aspects of RE syntax and semantics open; ‘**’
marks decisions on these aspects that may not be fully portable to other POSIX
implementations.
This manual page first describes regular expressions in general, specifically
extended regular expressions, and then discusses differences between them and
basic regular expressions.
An ERE is one** or more non-empty**
branches,
separated by ‘|’. It matches anything that matches one of the
branches.
A branch is one** or more
pieces, concatenated. It
matches a match for the first, followed by a match for the second, etc.
A piece is an
atom possibly followed by a single**
‘*’, ‘+’, ‘?’, or
bound. An atom followed by ‘*’
matches a sequence of 0 or more matches of the atom. An atom followed by
‘+’ matches a sequence of 1 or more matches of the atom. An atom
followed by ‘?’ matches a sequence of 0 or 1 matches of the
atom.
A bound is ‘{’ followed by an unsigned decimal integer, possibly
followed by ‘,’ possibly followed by another unsigned decimal
integer, always followed by ‘}’. The integers must lie between 0
and
RE_DUP_MAX (255**) inclusive, and if
there are two of them, the first may not exceed the second. An atom followed
by a bound containing one integer
i and no
comma matches a sequence of exactly
i matches
of the atom. An atom followed by a bound containing one integer
i and a comma matches a sequence of
i or more matches of the atom. An atom
followed by a bound containing two integers
i
and
j matches a sequence of
i through
j
(inclusive) matches of the atom.
An atom is a regular expression enclosed in ‘()’ (matching a part
of the regular expression), an empty set of ‘()’ (matching the
null string)**, a
bracket expression (see below),
‘.’ (matching any single character), ‘^’ (matching
the null string at the beginning of a line), ‘$’ (matching the
null string at the end of a line), a ‘\’ followed by one of the
characters ‘^.[$()|*+?{\’ (matching that character taken as an
ordinary character), a ‘\’ followed by any other character**
(matching that character taken as an ordinary character, as if the
‘\’ had not been present**), or a single character with no other
significance (matching that character). A ‘{’ followed by a
character other than a digit is an ordinary character, not the beginning of a
bound**. It is illegal to end an RE with ‘\’.
A bracket expression is a list of characters enclosed in ‘[]’. It
normally matches any single character from the list (but see below). If the
list begins with ‘^’, it matches any single character
not from the rest of the list (but see below). If
two characters in the list are separated by ‘-’, this is
shorthand for the full
range of characters
between those two (inclusive) in the collating sequence, e.g.
‘[0-9]’ in ASCII matches any decimal digit. It is illegal** for
two ranges to share an endpoint, e.g. ‘a-c-e’. Ranges are very
collating-sequence-dependent, and portable programs should avoid relying on
them.
To include a literal ‘]’ in the list, make it the first character
(following a possible ‘^’). To include a literal
‘-’, make it the first or last character, or the second endpoint
of a range. To use a literal ‘-’ as the first endpoint of a
range, enclose it in ‘[.’ and ‘.]’ to make it a
collating element (see below). With the exception of these and some
combinations using ‘[’ (see next paragraphs), all other special
characters, including ‘\’, lose their special significance
within a bracket expression.
Within a bracket expression, a collating element (a character, a multi-character
sequence that collates as if it were a single character, or a
collating-sequence name for either) enclosed in ‘[.’ and
‘.]’ stands for the sequence of characters of that collating
element. The sequence is a single element of the bracket expression's list. A
bracket expression containing a multi-character collating element can thus
match more than one character, e.g. if the collating sequence includes a
‘ch’ collating element, then the RE ‘[[.ch.]]*c’
matches the first five characters of ‘chchcc’.
Within a bracket expression, a collating element enclosed in ‘[=’
and ‘=]’ is an equivalence class, standing for the sequences of
characters of all collating elements equivalent to that one, including itself.
(If there are no other equivalent collating elements, the treatment is as if
the enclosing delimiters were ‘[.’ and ‘.]’.) For
example, if ‘x’ and ‘y’ are the members of an
equivalence class, then ‘[[=x=]]’, ‘[[=y=]]’, and
‘[xy]’ are all synonymous. An equivalence class may not** be an
endpoint of a range.
Within a bracket expression, the name of a
character
class enclosed in ‘[:’ and ‘:]’ stands for the
list of all characters belonging to that class. Standard character class names
are:
alnum digit punct
alpha graph space
blank lower upper
cntrl print xdigit
These stand for the character classes defined in
isalnum(3),
isalpha(3), and so on. A
character class may not be used as an endpoint of a range.
There are two special cases** of bracket expressions: the bracket expressions
‘[[:<:]]’ and ‘[[:>:]]’ match the null
string at the beginning and end of a word, respectively. A word is defined as
a sequence of characters starting and ending with a word character which is
neither preceded nor followed by word characters. A word character is an
alnum character (as defined by
isalnum(3)) or an underscore.
This is an extension, compatible with but not specified by POSIX, and should
be used with caution in software intended to be portable to other systems. The
additional word delimiters ‘
\<’ and
‘
\>’ are provided to ease
compatibility with traditional SVR4 systems but are not portable and should be
avoided.
In the event that an RE could match more than one substring of a given string,
the RE matches the one starting earliest in the string. If the RE could match
more than one substring starting at that point, it matches the longest.
Subexpressions also match the longest possible substrings, subject to the
constraint that the whole match be as long as possible, with subexpressions
starting earlier in the RE taking priority over ones starting later. Note that
higher-level subexpressions thus take priority over their lower-level
component subexpressions.
Match lengths are measured in characters, not collating elements. A null string
is considered longer than no match at all. For example, ‘bb*’
matches the three middle characters of ‘abbbc’;
‘(wee|week)(knights|nights)’ matches all ten characters of
‘weeknights’; when ‘(.*).*’ is matched against
‘abc’, the parenthesized subexpression matches all three
characters; and when ‘(a*)*’ is matched against
‘bc’, both the whole RE and the parenthesized subexpression
match the null string.
If case-independent matching is specified, the effect is much as if all case
distinctions had vanished from the alphabet. When an alphabetic that exists in
multiple cases appears as an ordinary character outside a bracket expression,
it is effectively transformed into a bracket expression containing both cases,
e.g. ‘x’ becomes ‘[xX]’. When it appears inside a
bracket expression, all case counterparts of it are added to the bracket
expression, so that, for example, ‘[x]’ becomes
‘[xX]’ and ‘[^x]’ becomes ‘[^xX]’.
No particular limit is imposed on the length of REs**. Programs intended to be
portable should not employ REs longer than 256 bytes, as an implementation can
refuse to accept such REs and remain POSIX-compliant.
The following is a list of extended regular expressions:
-
-
- c
- Any character c not listed below matches
itself.
-
-
- \c
- Any backslash-escaped character c matches
itself.
-
-
- .
- Matches any single character that is not a newline
(‘\n’).
-
-
- [char-class]
- Matches any single character in
char-class. To include a
‘
]’ in
char-class, it must be the first
character. A range of characters may be specified by separating the end
characters of the range with a ‘-’;
e.g. a-z specifies the lower case
characters. The following literal expressions can also be used in
char-class to specify sets of characters:
[:alnum:] [:cntrl:] [:lower:] [:space:]
[:alpha:] [:digit:] [:print:] [:upper:]
[:blank:] [:graph:] [:punct:] [:xdigit:]
If ‘-’ appears as the first or last
character of char-class, then it matches
itself. All other characters in
char-class match themselves.
Patterns in char-class of the form
[.col-elm.] or
[=col-elm=], where
col-elm is a collating element, are
interpreted according to
setlocale(3) (not
currently supported).
-
-
- [^char-class]
- Matches any single character, other than newline, not in
char-class.
char-class is defined as above.
-
-
- ^
- If ‘^’ is the first character of a regular expression, then
it anchors the regular expression to the beginning of a line. Otherwise,
it matches itself.
-
-
- $
- If ‘$’ is the last character of a regular expression, it
anchors the regular expression to the end of a line. Otherwise, it matches
itself.
-
-
- [[:<:]]
- Anchors the single character regular expression or subexpression
immediately following it to the beginning of a word.
-
-
- [[:>:]]
- Anchors the single character regular expression or subexpression
immediately preceding it to the end of a word.
-
-
- (re)
- Defines a subexpression re. Any set of
characters enclosed in parentheses matches whatever the set of characters
without parentheses matches (that is a long-winded way of saying the
constructs ‘(re)’ and ‘re’ match
identically).
-
-
- *
- Matches the single character regular expression or subexpression
immediately preceding it zero or more times. If ‘*’ is the
first character of a regular expression or subexpression, then it matches
itself. The ‘*’ operator sometimes yields unexpected
results. For example, the regular expression
b* matches the beginning of the string
“abbb” (as opposed to the substring “bbb”),
since a null match is the only leftmost match.
-
-
- +
- Matches the singular character regular expression or subexpression
immediately preceding it one or more times.
-
-
- ?
- Matches the singular character regular expression or subexpression
immediately preceding it 0 or 1 times.
-
-
- {n,m} {n,} {n}
- Matches the single character regular expression or subexpression
immediately preceding it at least n and
at most m times. If
m is omitted, then it matches at least
n times. If the comma is also omitted,
then it matches exactly n times.
-
-
- |
- Used to separate patterns. For example, the pattern
‘cat|dog’ matches either ‘cat’ or
‘dog’.
Basic regular expressions differ in several respects:
- ‘|’, ‘+’, and ‘?’ are ordinary
characters and there is no equivalent for their functionality.
- The delimiters for bounds are ‘\{’ and ‘\}’,
with ‘{’ and ‘}’ by themselves ordinary
characters.
- The parentheses for nested subexpressions are ‘\(’ and
‘\)’, with ‘(’ and ‘)’ by
themselves ordinary characters.
- ‘^’ is an ordinary character except at the beginning of the
RE or** the beginning of a parenthesized subexpression.
- ‘$’ is an ordinary character except at the end of the RE
or** the end of a parenthesized subexpression.
- ‘*’ is an ordinary character if it appears at the beginning
of the RE or the beginning of a parenthesized subexpression (after a
possible leading ‘^’).
- Finally, there is one new type of atom, a
back-reference: ‘\’ followed by
a non-zero decimal digit d matches the
same sequence of characters matched by the
dth parenthesized subexpression
(numbering subexpressions by the positions of their opening parentheses,
left to right), so that, for example, ‘\([bc]\)\1’ matches
‘bb’ or ‘cc’ but not ‘bc’.
The following is a list of basic regular expressions:
-
-
- c
- Any character c not listed below matches
itself.
-
-
- \c
- Any backslash-escaped character c, except
for ‘{’, ‘}’, ‘(’, and
‘)’, matches itself.
-
-
- .
- Matches any single character that is not a newline
(‘\n’).
-
-
- [char-class]
- Matches any single character in
char-class. To include a
‘
]’ in
char-class, it must be the first
character. A range of characters may be specified by separating the end
characters of the range with a ‘-’;
e.g. a-z specifies the lower case
characters. The following literal expressions can also be used in
char-class to specify sets of characters:
[:alnum:] [:cntrl:] [:lower:] [:space:]
[:alpha:] [:digit:] [:print:] [:upper:]
[:blank:] [:graph:] [:punct:] [:xdigit:]
If ‘-’ appears as the first or last
character of char-class, then it matches
itself. All other characters in
char-class match themselves.
Patterns in char-class of the form
[.col-elm.] or
[=col-elm=], where
col-elm is a collating element, are
interpreted according to
setlocale(3) (not
currently supported).
-
-
- [^char-class]
- Matches any single character, other than newline, not in
char-class.
char-class is defined as above.
-
-
- ^
- If ‘^’ is the first character of a regular expression, then
it anchors the regular expression to the beginning of a line. Otherwise,
it matches itself.
-
-
- $
- If ‘$’ is the last character of a regular expression, it
anchors the regular expression to the end of a line. Otherwise, it matches
itself.
-
-
- [[:<:]]
- Anchors the single character regular expression or subexpression
immediately following it to the beginning of a word.
-
-
- [[:>:]]
- Anchors the single character regular expression or subexpression
immediately following it to the end of a word.
-
-
- \(re\)
- Defines a subexpression re.
Subexpressions may be nested. A subsequent backreference of the form
\n, where
n is a number in the range [1,9], expands
to the text matched by the nth
subexpression. For example, the regular expression
\(.*\)\1 matches any string consisting of
identical adjacent substrings. Subexpressions are ordered relative to
their left delimiter.
-
-
- *
- Matches the single character regular expression or subexpression
immediately preceding it zero or more times. If ‘*’ is the
first character of a regular expression or subexpression, then it matches
itself. The ‘*’ operator sometimes yields unexpected
results. For example, the regular expression
b* matches the beginning of the string
“abbb” (as opposed to the substring “bbb”),
since a null match is the only leftmost match.
-
-
- \{n,m\} \{n,\} \{n\}
- Matches the single character regular expression or subexpression
immediately preceding it at least n and
at most m times. If
m is omitted, then it matches at least
n times. If the comma is also omitted,
then it matches exactly n times.
regex(3)
IEEE Std 1003.1-2004
(“POSIX.1”): Base Definitions, Chapter 9 (Regular
Expressions).
Having two kinds of REs is a botch.
The current POSIX spec says that ‘)’ is an ordinary character in
the absence of an unmatched ‘(’; this was an unintentional
result of a wording error, and change is likely. Avoid relying on it.
Back-references are a dreadful botch, posing major problems for efficient
implementations. They are also somewhat vaguely defined (does
‘a\(\(b\)*\2\)*d’ match ‘abbbd’?). Avoid using
them.
POSIX's specification of case-independent matching is vague. The “one
case implies all cases” definition given above is the current consensus
among implementors as to the right interpretation.
The syntax for word boundaries is incredibly ugly.
![[OpenBSD]](/openbsd.gif){kind=small}