Go to the previous, next section.
Expressions are the basic building block of awk actions. An
expression evaluates to a value, which you can print, test, store in a
variable or pass to a function. But beyond that, an expression can assign a new value to a variable
or a field, with an assignment operator.
An expression can serve as a statement on its own. Most other kinds of
statements contain one or more expressions which specify data to be
operated on. As in other languages, expressions in awk include
variables, array references, constants, and function calls, as well as
combinations of these with various operators.
The simplest type of expression is the constant, which always has the same value. There are three types of constants: numeric constants, string constants, and regular expression constants.
A numeric constant stands for a number. This number can be an
integer, a decimal fraction, or a number in scientific (exponential)
notation. Note that all numeric values are represented within
awk in double-precision floating point. Here are some examples
of numeric constants, which all have the same value:
105 1.05e+2 1050e-1
A string constant consists of a sequence of characters enclosed in double-quote marks. For example:
"parrot"
represents the string whose contents are `parrot'. Strings in
gawk can be of any length and they can contain all the possible
8-bit ASCII characters including ASCII NUL. Other awk
implementations may have difficulty with some character codes.
Some characters cannot be included literally in a string constant. You represent them instead with escape sequences, which are character sequences beginning with a backslash (`\').
One use of an escape sequence is to include a double-quote character in
a string constant. Since a plain double-quote would end the string, you
must use `\"' to represent a single double-quote character as a
part of the string.
The
backslash character itself is another character that cannot be
included normally; you write `\\' to put one backslash in the
string. Thus, the string whose contents are the two characters
`"\' must be written "\"\\".
Another use of backslash is to represent unprintable characters such as newline. While there is nothing to stop you from writing most of these characters directly in a string constant, they may look ugly.
Here is a table of all the escape sequences used in awk:
\\
\a
\b
\f
\n
\r
\t
\v
\nnn
\xhh...
awk.)
A constant regexp is a regular expression description enclosed in
slashes, such as /^beginning and end$/. Most regexps used in
awk programs are constant, but the `~' and `!~'
operators can also match computed or "dynamic" regexps
(see section How to Use Regular Expressions).
Constant regexps may be used like simple expressions. When a constant regexp is not on the right hand side of the `~' or `!~' operators, it has the same meaning as if it appeared in a pattern, i.e. `($0 ~ /foo/)' (see section Expressions as Patterns). This means that the two code segments,
if ($0 ~ /barfly/ || $0 ~ /camelot/)
print "found"
and
if (/barfly/ || /camelot/)
print "found"
are exactly equivalent. One rather bizarre consequence of this rule is that the following boolean expression is legal, but does not do what the user intended:
if (/foo/ ~ $1) print "found foo"
This code is "obviously" testing $1 for a match against the regexp
/foo/. But in fact, the expression (/foo/ ~ $1) actually means
(($0 ~ /foo/) ~ $1). In other words, first match the input record
against the regexp /foo/. The result will be either a 0 or a 1,
depending upon the success or failure of the match. Then match that result
against the first field in the record.
Since it is unlikely that you would ever really wish to make this kind of
test, gawk will issue a warning when it sees this construct in
a program.
Another consequence of this rule is that the assignment statement
matches = /foo/
will assign either 0 or 1 to the variable matches, depending
upon the contents of the current input record.
Constant regular expressions are also used as the first argument for
the sub and gsub functions
(see section Built-in Functions for String Manipulation).
This feature of the language was never well documented until the POSIX specification.
You may be wondering, when is
$1 ~ /foo/ { ... }
preferable to
$1 ~ "foo" { ... }
Since the right-hand sides of both `~' operators are constants,
it is more efficient to use the `/foo/' form: awk can note
that you have supplied a regexp and store it internally in a form that
makes pattern matching more efficient. In the second form, awk
must first convert the string into this internal form, and then perform
the pattern matching. The first form is also better style; it shows
clearly that you intend a regexp match.
Variables let you give names to values and refer to them later. You have
already seen variables in many of the examples. The name of a variable
must be a sequence of letters, digits and underscores, but it may not begin
with a digit. Case is significant in variable names; a and A
are distinct variables.
A variable name is a valid expression by itself; it represents the variable's current value. Variables are given new values with assignment operators and increment operators. See section Assignment Expressions.
A few variables have special built-in meanings, such as FS, the
field separator, and NF, the number of fields in the current
input record. See section Built-in Variables, for a list of them. These
built-in variables can be used and assigned just like all other
variables, but their values are also used or changed automatically by
awk. Each built-in variable's name is made entirely of upper case
letters.
Variables in awk can be assigned either numeric or string
values. By default, variables are initialized to the null string, which
is effectively zero if converted to a number. There is no need to
"initialize" each variable explicitly in awk, the way you would in C or most other traditional languages.
You can set any awk variable by including a variable assignment
among the arguments on the command line when you invoke awk
(see section Invoking awk). Such an assignment has
this form:
variable=text
With it, you can set a variable either at the beginning of the
awk run or in between input files.
If you precede the assignment with the `-v' option, like this:
-v variable=text
then the variable is set at the very beginning, before even the
BEGIN rules are run. The `-v' option and its assignment
must precede all the file name arguments, as well as the program text.
Otherwise, the variable assignment is performed at a time determined by its position among the input file arguments: after the processing of the preceding input file argument. For example:
awk '{ print $n }' n=4 inventory-shipped n=2 BBS-list
prints the value of field number n for all input records. Before
the first file is read, the command line sets the variable n
equal to 4. This causes the fourth field to be printed in lines from
the file `inventory-shipped'. After the first file has finished,
but before the second file is started, n is set to 2, so that the
second field is printed in lines from `BBS-list'.
Command line arguments are made available for explicit examination by
the awk program in an array named ARGV
(see section Built-in Variables).
awk processes the values of command line assignments for escape
sequences (see section Constant Expressions).
The awk language uses the common arithmetic operators when
evaluating expressions. All of these arithmetic operators follow normal
precedence rules, and work as you would expect them to. This example
divides field three by field four, adds field two, stores the result
into field one, and prints the resulting altered input record:
awk '{ $1 = $2 + $3 / $4; print }' inventory-shipped
The arithmetic operators in awk are:
x + y
x - y
- x
+ x
x * y
x / y
awk are double-precision
floating point, the result is not rounded to an integer: 3 / 4
has the value 0.75.
x % y
b * int(a / b) + (a % b) == a
One possibly undesirable effect of this definition of remainder is that
x % y is negative if x is negative. Thus,
-17 % 8 = -1
In other awk implementations, the signedness of the remainder
may be machine dependent.
x ^ y
x ** y
2 ^ 3 has
the value 8. The character sequence `**' is equivalent to
`^'. (The POSIX standard only specifies the use of `^'
for exponentiation.)
There is only one string operation: concatenation. It does not have a specific operator to represent it. Instead, concatenation is performed by writing expressions next to one another, with no operator. For example:
awk '{ print "Field number one: " $1 }' BBS-list
produces, for the first record in `BBS-list':
Field number one: aardvark
Without the space in the string constant after the `:', the line would run together. For example:
awk '{ print "Field number one:" $1 }' BBS-list
produces, for the first record in `BBS-list':
Field number one:aardvark
Since string concatenation does not have an explicit operator, it is
often necessary to insure that it happens where you want it to by
enclosing the items to be concatenated in parentheses. For example, the
following code fragment does not concatenate file and name
as you might expect:
file = "file" name = "name" print "something meaningful" > file name
It is necessary to use the following:
print "something meaningful" > (file name)
We recommend you use parentheses around concatenation in all but the most common contexts (such as in the right-hand operand of `=').
Comparison expressions compare strings or numbers for relationships such as equality. They are written using relational operators, which are a superset of those in C. Here is a table of them:
x < y
x <= y
x > y
x >= y
x == y
x != y
x ~ y
x !~ y
subscript in array
Comparison expressions have the value 1 if true and 0 if false.
The rules gawk uses for performing comparisons are based on those
in draft 11.2 of the POSIX standard. The POSIX standard introduced
the concept of a numeric string, which is simply a string that looks
like a number, for example, " +2".
When performing a relational operation, gawk considers the type of an
operand to be the type it received on its last assignment, rather
than the type of its last use
(see section Numeric and String Values).
This type is unknown when the operand is from an "external" source:
field variables, command line arguments, array elements resulting from a
split operation, and the value of an ENVIRON element.
In this case only, if the operand is a numeric string, then it is
considered to be of both string type and numeric type. If at least one
operand of a comparison is of string type only, then a string
comparison is performed. Any numeric operand will be converted to a
string using the value of CONVFMT
(see section Conversion of Strings and Numbers).
If one operand of a comparison is numeric, and the other operand is
either numeric or both numeric and string, then gawk does a
numeric comparison. If both operands have both types, then the
comparison is numeric. Strings are compared
by comparing the first character of each, then the second character of each,
and so on. Thus "10" is less than "9". If there are two
strings where one is a prefix of the other, the shorter string is less than
the longer one. Thus "abc" is less than "abcd".
Here are some sample expressions, how gawk compares them, and what
the result of the comparison is.
1.5 <= 2.0
"abc" >= "xyz"
1.5 != " +2"
"1e2" < "3"
a = 2; b = "2"
a == b
echo 1e2 3 | awk '{ print ($1 < $2) ? "true" : "false" }'
prints `false' since both $1 and $2 are numeric
strings and thus have both string and numeric types, thus dictating
a numeric comparison.
The purpose of the comparison rules and the use of numeric strings is to attempt to produce the behavior that is "least surprising," while still "doing the right thing."
String comparisons and regular expression comparisons are very different. For example,
$1 == "foo"
has the value of 1, or is true, if the first field of the current input record is precisely `foo'. By contrast,
$1 ~ /foo/
has the value 1 if the first field contains `foo', such as `foobar'.
The right hand operand of the `~' and `!~' operators may be
either a constant regexp (/.../), or it may be an ordinary
expression, in which case the value of the expression as a string is a
dynamic regexp (see section How to Use Regular Expressions).
In very recent implementations of awk, a constant regular
expression in slashes by itself is also an expression. The regexp
/regexp/ is an abbreviation for this comparison expression:
$0 ~ /regexp/
In some contexts it may be necessary to write parentheses around the
regexp to avoid confusing the gawk parser. For example,
(/x/ - /y/) > threshold is not allowed, but ((/x/) - (/y/))
> threshold parses properly.
One special place where /foo/ is not an abbreviation for
$0 ~ /foo/ is when it is the right-hand operand of `~' or
`!~'! See section Constant Expressions, where this is
discussed in more detail.
A boolean expression is a combination of comparison expressions or matching expressions, using the boolean operators "or" (`||'), "and" (`&&'), and "not" (`!'), along with parentheses to control nesting. The truth of the boolean expression is computed by combining the truth values of the component expressions.
Boolean expressions can be used wherever comparison and matching
expressions can be used. They can be used in if, while
do and for statements. They have numeric values (1 if true,
0 if false), which come into play if the result of the boolean expression
is stored in a variable, or used in arithmetic.
In addition, every boolean expression is also a valid boolean pattern, so you can use it as a pattern to control the execution of rules.
Here are descriptions of the three boolean operators, with an example of each. It may be instructive to compare these examples with the analogous examples of boolean patterns (see section Boolean Operators and Patterns), which use the same boolean operators in patterns instead of expressions.
boolean1 && boolean2
if ($0 ~ /2400/ && $0 ~ /foo/) print
The subexpression boolean2 is evaluated only if boolean1
is true. This can make a difference when boolean2 contains
expressions that have side effects: in the case of $0 ~ /foo/ &&
($2 == bar++), the variable bar is not incremented if there is
no `foo' in the record.
boolean1 || boolean2
awk '{ if ($0 ~ /2400/ || $0 ~ /foo/) print }' BBS-list
The subexpression boolean2 is evaluated only if boolean1 is false. This can make a difference when boolean2 contains expressions that have side effects.
!boolean
awk '{ if (! ($0 ~ /foo/)) print }' BBS-list
An assignment is an expression that stores a new value into a
variable. For example, let's assign the value 1 to the variable
z:
z = 1
After this expression is executed, the variable z has the value 1.
Whatever old value z had before the assignment is forgotten.
Assignments can store string values also. For example, this would store
the value "this food is good" in the variable message:
thing = "food" predicate = "good" message = "this " thing " is " predicate
(This also illustrates concatenation of strings.)
The `=' sign is called an assignment operator. It is the simplest assignment operator because the value of the right-hand operand is stored unchanged.
Most operators (addition, concatenation, and so on) have no effect except to compute a value. If you ignore the value, you might as well not use the operator. An assignment operator is different; it does produce a value, but even if you ignore the value, the assignment still makes itself felt through the alteration of the variable. We call this a side effect.
The left-hand operand of an assignment need not be a variable
(see section Variables); it can also be a field
(see section Changing the Contents of a Field) or
an array element (see section Arrays in awk).
These are all called lvalues,
which means they can appear on the left-hand side of an assignment operator.
The right-hand operand may be any expression; it produces the new value
which the assignment stores in the specified variable, field or array
element.
It is important to note that variables do not have permanent types.
The type of a variable is simply the type of whatever value it happens
to hold at the moment. In the following program fragment, the variable
foo has a numeric value at first, and a string value later on:
foo = 1 print foo foo = "bar" print foo
When the second assignment gives foo a string value, the fact that
it previously had a numeric value is forgotten.
An assignment is an expression, so it has a value: the same value that
is assigned. Thus, z = 1 as an expression has the value 1.
One consequence of this is that you can write multiple assignments together:
x = y = z = 0
stores the value 0 in all three variables. It does this because the
value of z = 0, which is 0, is stored into y, and then
the value of y = z = 0, which is 0, is stored into x.
You can use an assignment anywhere an expression is called for. For
example, it is valid to write x != (y = 1) to set y to 1
and then test whether x equals 1. But this style tends to make
programs hard to read; except in a one-shot program, you should
rewrite it to get rid of such nesting of assignments. This is never very
hard.
Aside from `=', there are several other assignment operators that
do arithmetic with the old value of the variable. For example, the
operator `+=' computes a new value by adding the right-hand value
to the old value of the variable. Thus, the following assignment adds
5 to the value of foo:
foo += 5
This is precisely equivalent to the following:
foo = foo + 5
Use whichever one makes the meaning of your program clearer.
Here is a table of the arithmetic assignment operators. In each case, the right-hand operand is an expression whose value is converted to a number.
lvalue += increment
lvalue -= decrement
lvalue *= coefficient
lvalue /= quotient
lvalue %= modulus
lvalue ^= power
lvalue **= power
^= operator is specified by POSIX.)
Increment operators increase or decrease the value of a variable
by 1. You could do the same thing with an assignment operator, so
the increment operators add no power to the awk language; but they
are convenient abbreviations for something very common.
The operator to add 1 is written `++'. It can be used to increment a variable either before or after taking its value.
To pre-increment a variable v, write ++v. This adds
1 to the value of v and that new value is also the value of this
expression. The assignment expression v += 1 is completely
equivalent.
Writing the `++' after the variable specifies post-increment. This
increments the variable value just the same; the difference is that the
value of the increment expression itself is the variable's old
value. Thus, if foo has the value 4, then the expression foo++
has the value 4, but it changes the value of foo to 5.
The post-increment foo++ is nearly equivalent to writing (foo
+= 1) - 1. It is not perfectly equivalent because all numbers in
awk are floating point: in floating point, foo + 1 - 1 does
not necessarily equal foo. But the difference is minute as
long as you stick to numbers that are fairly small (less than a trillion).
Any lvalue can be incremented. Fields and array elements are incremented just like variables. (Use `$(i++)' when you wish to do a field reference and a variable increment at the same time. The parentheses are necessary because of the precedence of the field reference operator, `$'.)
The decrement operator `--' works just like `++' except that it subtracts 1 instead of adding. Like `++', it can be used before the lvalue to pre-decrement or after it to post-decrement.
Here is a summary of increment and decrement expressions.
++lvalue
lvalue++
--lvalue
++lvalue, but instead of adding, it subtracts. It
decrements lvalue and delivers the value that results.
lvalue--
lvalue++, but instead of adding, it subtracts. It
decrements lvalue. The value of the expression is the old
value of lvalue.
Strings are converted to numbers, and numbers to strings, if the context
of the awk program demands it. For example, if the value of
either foo or bar in the expression foo + bar
happens to be a string, it is converted to a number before the addition
is performed. If numeric values appear in string concatenation, they
are converted to strings. Consider this:
two = 2; three = 3 print (two three) + 4
This eventually prints the (numeric) value 27. The numeric values of
the variables two and three are converted to strings and
concatenated together, and the resulting string is converted back to the
number 23, to which 4 is then added.
If, for some reason, you need to force a number to be converted to a string, concatenate the null string with that number. To force a string to be converted to a number, add zero to that string.
A string is converted to a number by interpreting a numeric prefix
of the string as numerals:
"2.5" converts to 2.5, "1e3" converts to 1000, and "25fix"
has a numeric value of 25.
Strings that can't be interpreted as valid numbers are converted to
zero.
The exact manner in which numbers are converted into strings is controlled
by the awk built-in variable CONVFMT (see section Built-in Variables).
Numbers are converted using a special version of the sprintf function
(see section Built-in Functions) with CONVFMT as the format
specifier.
CONVFMT's default value is "%.6g", which prints a value with
at least six significant digits. For some applications you will want to
change it to specify more precision. Double precision on most modern
machines gives you 16 or 17 decimal digits of precision.
Strange results can happen if you set CONVFMT to a string that doesn't
tell sprintf how to format floating point numbers in a useful way.
For example, if you forget the `%' in the format, all numbers will be
converted to the same constant string.
As a special case, if a number is an integer, then the result of converting
it to a string is always an integer, no matter what the value of
CONVFMT may be. Given the following code fragment:
CONVFMT = "%2.2f" a = 12 b = a ""
b has the value "12", not "12.00".
Prior to the POSIX standard, awk specified that the value
of OFMT was used for converting numbers to strings. OFMT
specifies the output format to use when printing numbers with print.
CONVFMT was introduced in order to separate the semantics of
conversions from the semantics of printing. Both CONVFMT and
OFMT have the same default value: "%.6g". In the vast majority
of cases, old awk programs will not change their behavior.
However, this use of OFMT is something to keep in mind if you must
port your program to other implementations of awk; we recommend
that instead of changing your programs, you just port gawk itself!
Through most of this manual, we present awk values (such as constants,
fields, or variables) as either numbers or strings. This is
a convenient way to think about them, since typically they are used in only
one way, or the other.
In truth though, awk values can be both string and
numeric, at the same time. Internally, awk represents values
with a string, a (floating point) number, and an indication that one,
the other, or both representations of the value are valid.
Keeping track of both kinds of values is important for execution efficiency: a variable can acquire a string value the first time it is used as a string, and then that string value can be used until the variable is assigned a new value. Thus, if a variable with only a numeric value is used in several concatenations in a row, it only has to be given a string representation once. The numeric value remains valid, so that no conversion back to a number is necessary if the variable is later used in an arithmetic expression.
Tracking both kinds of values is also important for precise numerical calculations. Consider the following:
a = 123.321 CONVFMT = "%3.1f" b = a " is a number" c = a + 1.654
The variable a receives a string value in the concatenation and
assignment to b. The string value of a is "123.3".
If the numeric value was lost when it was converted to a string, then the
numeric use of a in the last statement would lose information.
c would be assigned the value 124.954 instead of 124.975.
Such errors accumulate rapidly, and very adversely affect numeric
computations.
Once a numeric value acquires a corresponding string value, it stays valid
until a new assignment is made. If CONVFMT
(see section Conversion of Strings and Numbers) changes in the
meantime, the old string value will still be used. For example:
BEGIN {
CONVFMT = "%2.2f"
a = 123.456
b = a "" # force `a' to have string value too
printf "a = %s\n", a
CONVFMT = "%.6g"
printf "a = %s\n", a
a += 0 # make `a' numeric only again
printf "a = %s\n", a # use `a' as string
}
This program prints `a = 123.46' twice, and then prints `a = 123.456'.
See section Conversion of Strings and Numbers, for the rules that specify how string values are made from numeric values.
A conditional expression is a special kind of expression with three operands. It allows you to use one expression's value to select one of two other expressions.
The conditional expression looks the same as in the C language:
selector ? if-true-exp : if-false-exp
There are three subexpressions. The first, selector, is always computed first. If it is "true" (not zero and not null) then if-true-exp is computed next and its value becomes the value of the whole expression. Otherwise, if-false-exp is computed next and its value becomes the value of the whole expression.
For example, this expression produces the absolute value of x:
x > 0 ? x : -x
Each time the conditional expression is computed, exactly one of
if-true-exp and if-false-exp is computed; the other is ignored.
This is important when the expressions contain side effects. For example,
this conditional expression examines element i of either array
a or array b, and increments i.
x == y ? a[i++] : b[i++]
This is guaranteed to increment i exactly once, because each time
one or the other of the two increment expressions is executed,
and the other is not.
A function is a name for a particular calculation. Because it has
a name, you can ask for it by name at any point in the program. For
example, the function sqrt computes the square root of a number.
A fixed set of functions are built-in, which means they are
available in every awk program. The sqrt function is one
of these. See section Built-in Functions, for a list of built-in
functions and their descriptions. In addition, you can define your own
functions in the program for use elsewhere in the same program.
See section User-defined Functions, for how to do this.
The way to use a function is with a function call expression, which consists of the function name followed by a list of arguments in parentheses. The arguments are expressions which give the raw materials for the calculation that the function will do. When there is more than one argument, they are separated by commas. If there are no arguments, write just `()' after the function name. Here are some examples:
sqrt(x^2 + y^2) # One argument atan2(y, x) # Two arguments rand() # No arguments
Do not put any space between the function name and the open-parenthesis! A user-defined function name looks just like the name of a variable, and space would make the expression look like concatenation of a variable with an expression inside parentheses. Space before the parenthesis is harmless with built-in functions, but it is best not to get into the habit of using space to avoid mistakes with user-defined functions.
Each function expects a particular number of arguments. For example, the
sqrt function must be called with a single argument, the number
to take the square root of:
sqrt(argument)
Some of the built-in functions allow you to omit the final argument. If you do so, they use a reasonable default. See section Built-in Functions, for full details. If arguments are omitted in calls to user-defined functions, then those arguments are treated as local variables, initialized to the null string (see section User-defined Functions).
Like every other expression, the function call has a value, which is
computed by the function based on the arguments you give it. In this
example, the value of sqrt(argument) is the square root of the
argument. A function can also have side effects, such as assigning the
values of certain variables or doing I/O.
Here is a command to read numbers, one number per line, and print the square root of each one:
awk '{ print "The square root of", $1, "is", sqrt($1) }'
Operator precedence determines how operators are grouped, when
different operators appear close by in one expression. For example,
`*' has higher precedence than `+'; thus, a + b * c
means to multiply b and c, and then add a to the
product (i.e., a + (b * c)).
You can overrule the precedence of the operators by using parentheses. You can think of the precedence rules as saying where the parentheses are assumed if you do not write parentheses yourself. In fact, it is wise to always use parentheses whenever you have an unusual combination of operators, because other people who read the program may not remember what the precedence is in this case. You might forget, too; then you could make a mistake. Explicit parentheses will help prevent any such mistake.
When operators of equal precedence are used together, the leftmost
operator groups first, except for the assignment, conditional and
exponentiation operators, which group in the opposite order.
Thus, a - b + c groups as (a - b) + c;
a = b = c groups as a = (b = c).
The precedence of prefix unary operators does not matter as long as only
unary operators are involved, because there is only one way to parse
them--innermost first. Thus, $++i means $(++i) and
++$x means ++($x). However, when another operator follows
the operand, then the precedence of the unary operators can matter.
Thus, $x^2 means ($x)^2, but -x^2 means
-(x^2), because `-' has lower precedence than `^'
while `$' has higher precedence.
Here is a table of the operators of awk, in order of increasing
precedence:
The relational operators are `<', `<=', `==', `!=', `>=' and `>'.
The I/O redirection operators are `<', `>', `>>' and `|'.
Note that I/O redirection operators in print and printf
statements belong to the statement level, not to expressions. The
redirection does not produce an expression which could be the operand of
another operator. As a result, it does not make sense to use a
redirection operator near another operator of lower precedence, without
parentheses. Such combinations, for example `print foo > a ? b :
c', result in syntax errors.
Go to the previous, next section.