awk utility is powerful data manipulation/scripting programming language (In fact based on the C programming Language). Use awk to handle complex task such as calculation, database handling, report creation etc. 

General Syntax of awk
awk  -f {awk program file}   filename

awk Program contains are something as follows

Pattern {
                   action 1
                   action 2
                   action N 
           }

awk reads the input from given file (or from stdin also) one line at a time, then each line is compared with pattern. If pattern is match for each line then given action is taken. Pattern can be regular expressions. Following is the summery of common awk metacharacters.

 Now consider following text database file
----------------------------------------
Sr.No  Product     Qty   Unit Price
======================
1.        Pen          5        20.00
2.        Pencil      10         2.00
3.        Rubber      3         3.50
4.        Cock         2        45.50
----------------------------------------

In above file fields are Sr.No,Product,Qty,Unit Price. Field is the smallest element of any record. Each fields has its own attributes. For e.g. Take Qty. field. Qty. fields attribute is its numerical (Can contain only numerical data). Collection of fields is know as record. So 
1.        Pen          5        20.00  ----> Is a Record.

Collection of record is know as database file. In above text database file each field is separated using space (or tab character) and record is separated using new-line character ( i.e. each record is finished at the end of line ). In the awk, fields are access using special variable. For e.g. In above database $1, $2, $3, $4 respectively represents Sr.No, Product, Qty, Unit Price fields. (Don't confuse $1,$2 etc with command line arguments of shell script) 

For this part of tutorial create text datafile inven (Shown as above). Now enter following simple awk program 
$ awk  '{ print $1 $2 "--> Rs." $3 * $4 }'  inven 
1.Pen--> Rs.100 
2.Pencil--> Rs.20 
3.Rubber--> Rs.10.5 
4.Cock--> Rs.91 

Above awk can be explained as follows

Note $1,$2 etc (in awk) also know as predefined variable and can assign any value found in field.

Type following awk program at shell prompt,

$ awk '{ print $2 }' inven 
Pen 
Pencil 
Rubber 
Cock 

Program will print second field from file. Same way if you want to print second and fourth field then give following,

$awk '{ print $2 $4}' inven 
Pen20.00 
Pencil2.00 
Rubber3.50 
Cock45.50 

Try the following

$ awk '{ print $0 }' inven 
1. Pen 5 20.00 
2. Pencil 10 2.00 
3. Rubber 3 3.50 
4. Cock 2 45.50 

Note that $0 is special variable of awk , which print entire record.

You can also create awk command (program) file as follows
$ cat > prn_pen 
/Pen/ { print $3 } 

(Save the program pressing CTRL + D)

And then you can execute or run above "prn_pen" awk command file as follows
$ awk -f prn_pen  inven 
5 
10
 
Here /Pen/ is the search pattern, if this pattern is found on line (or record) then print the third field of record. { print $3 } is called Action. 

On shell prompt , $ awk -f prn_pen  inven , -f option instruct awk, to read its command from given file, inven is the name of database file which is taken as input for awk.

Now create following awk program as follows
$cat > comp_inv 
3 > 5 {  print $0     } 
Run it as follows
$ awk -f comp_inv  inven 
2. Pencil 10 2.00 

Here third field of database is compared with 5, this the pattern. If this pattern found on any line database, then entire record is printed.
Our next example talks more about predefined variable of awk. Create file as follows
 
$cat > def_var 
{ 
  print "Printing Rec. #"  NR   "(" $0 "),And # of field for this record is "  NF 
}  

Run it as follows.
$awk -f def_var  inven 
Printing Rec. #1(1. Pen 5 20.00),And # of field for this record is 4 
Printing Rec. #2(2. Pencil 10 2.00),And # of field for this record is 4 
Printing Rec. #3(3. Rubber 3 3.50),And # of field for this record is 4 
Printing Rec. #4(4. Cock 2 45.50),And # of field for this record is 4 

NR and NF are predefined variables of awk which means Number of input Record, Number of Fields in input record respectively. In above example NR is changed as our input record changes, and NF is constant as there are only 4 field per record. Following table shows list of such built in awk variables.

You can easily, do the arithmetic with awk as follows
$ cat > math 
click here for code. 

Run the awk program as follows

$ awk -f math 
20 3 
20 + 3 = 23 
20 - 3 = 17 
20 / 3 = 6.66667 
20 x 3 = 60 
20 mod 3 = 2 
(Press CTRL + D to terminate)

In above program print $1 " + " $2 " = " $1 + $2, statement is used for addition purpose. Here $1 + $2, means add (+) first field with second field. Same way you can do - (subtraction ), * (Multiplication), / (Division), % (Modular use to find remainder of division operation).

You can also define your own variable in awk program, as follows 
$ cat > math1 
click here for code. 

Run the program as follows
$ awk -f math1 
1 5
1 + 5 = 6

In the above program,  no1, no2, ans all are user defined variables. Value of first and second field are assigned to no1, no2 variable respectively and the addition to ans variable. Value of variable can be print using print statement as, print no1 " + " no2 " = " ans. Note that print statement prints whatever enclosed in double quotes (" text ") as it is. If string is not enclosed in double quotes its treated as variable. Also above two program takes input from stdin (Keyboard) instead of file.

Now try the following awk program and note down its output.
$ cat > bill
click for code. 

Run it as
$ awk -f bill inven 
1.Pen Rs.100 
2.Pencil Rs.20 
3.Rubber Rs.10.5 
4.Cock Rs.91 

Here we are printing the total price of each product (By multiplying third field with fourth field). Following program prints total price of each product as well as the Grand total of all product in the bracket.
$ cat > bill1
click for code.  
$ awk -f bill1 inven 
1.Pen Rs.100 [Total Rs.100]
2.Pencil Rs.20 [Total Rs.120]
3.Rubber Rs.10.5 [Total Rs.130.5]
4.Cock Rs.91 [Total Rs.221.5] 

In this program, gtotal variable holds the grand total. It adds the total of each product as gtotal = gtotal + total. Finally this total is printed with each record in the bracket. But their is one problem with our script, Grand total mostly printed at the end of all record. To solve this problem we have to use special BEGIN and END Patterns of awk. First take the example, 

$ cat > bill2
click for code.

Run it as
$awk -f bill2 inven 
--------------------------- 
Bill for the 4-March-2001. 
By Vivek G Gite. 
--------------------------- 
1.Pen Rs.100 
2.Pencil Rs.20 
3.Rubber Rs.10.5 
4.Cock Rs.91 
--------------------------- 
Total Rs.221.5 
===============

Now the grand total is printed at the end. In above program BEGIN and END patters are used. BEGIN instruct awk, that perform BEGIN actions before the first line (Record) has been read from database file. Use BEGIN pattern to set value of variables, to print heading for report etc. General syntax of BEGIN is as follows

BEGIN {
            action 1
            action 2
            action N
}

END instruct awk, that perform END actions after reading all lines (RECORD) from the database file. General syntax of END is as follows

END {
            action 1
            action 2
            action N
}

In our example, BEGIN is used to print heading and END is used print grand total.

Next example shows the use of special printf statement 
$ cat > bill3  
click for code.

Run it as 
$ awk -f bill3 inven 
Bill for the 4-March-2001. 
By Vivek G Gite.
--------------------------- 
1 Pen Rs.100.000000 
2 Pencil Rs.20.000000 
3 Rubber Rs.10.500000 
4 Cock Rs.91.000000 
--------------------------- 
Total Rs. 221.500000 
=============== 

In above example printf statement is used to print formatted output of the variables or text. General syntax of printf as follows
printf   "format"   ,var1, var2, var N

If you just want to print any text using try printf as follows
printf "Hello"
printf "Hello World\n"

In last example \n is used to print new line. Its Part of escape sequence following may be also used
\t     for tab
\a    Alert or bell
\"     Print double quote etc
For e.g. printf "\nAn apple a day, keeps away\t\t\tDoctor\n\a\a"
It will print text on new line as 
An apple a day, keeps away            Doctor
(Notice that twice the sound of bell is produced by \a\a)

To print the value of decimal number use %d as format specification code followed by the variable name. For e.g.
printf "%d" , no1

It will print the value of no1. Following table shows such common format specification code

To run above example simply create any awk program file as follows
$ cat > p_demo
BEGIN {
   n = 10
   printf "%d", n
   printf "\nAn apple a day, keeps away\t\t\tDoctor\n\a\a"
}
(Save it by pressing CTRL + D)
Run it as 
$ awk -f p_demo
10
An apple a day, keeps away                Doctor

:-) Write awk program to test format specification code. According to your choice.

$ cat > bill4
click for code.

Run it as 
$ awk -f bill4 inven 
Bill for the 4-March-2001. 
By Vivek G Gite. 
--------------------------- 
1 Pen      Rs. 100.00 
2 Pencil   Rs.   20.00 
3 Rubber Rs.   10.50 
4 Cock    Rs.   91.00 
--------------------------- 
Total      Rs.  221.50 
=============== 

From the above output you can clearly see that printf can format the output. Let's try to understand formatting of printf statement. For e.g. %2d, number between % and d, tells the printf that assign 2 spaces for value. Same way if you write following awk program ,

$ cat > prf_demo
{
   na = $1
   printf "|%s|", na 
   printf "|%10s|", na 
   printf "|%-10s|", na 
}

Run it as follows
$ awk -f  prf_demo
God
|God|
|     God|
|God     |

(press CTRL + D to terminate) 

Same technique is used in our bill4 awk program to print formatted output. Also the statement like gtotal += total, which is equvalent to gtotal = gtotal + total. Here += is called assignment operator. You can use following assignment operator 

Now lets move to next program
$ awk > math2
click for code.

Run it as
$awk  -f math2
Welcome to MyAddtion calculation awk program v0.1 
(To Stop press CTRL+D) > 5 + 2 
5 + 2 = 7 
(To Stop press CTRL+D) > 3 - 1 
Opps!Error I only know how to add. 
Syntax: number1 + number2 
(To Stop press CTRL+D) > 
Goodbuy vivek 

In the above program various, new concept are introduce so lets try to understand them step by step

ENVIRON is the one of the predefined system variable that is array. Array is made up of different element. ENVIRON array is also made of elements. It allows you to access system variable (or variable in your environment). Give set command at shell prompt to see list of your environment variable. You can use variable name to reference any element in this array. For e.g. If you want to print your home directory you can write printf as follows: 
printf "%s is my sweet home", ENVIRON["HOME"]

General syntax of if 
if  ( condition )
{
    Statement 1
    Statement 2
    Statement N     
    if  condition is TRUE
}
else
{
    Statement 1
    Statement 2
    Statement N     
    if  condition is FALSE
}

For loop and while loop are used for looping purpose in awk. 
Syntax of for loop
for (expr1; condition; expr2)
{
    Statement 1
    Statement 2
    Statement N         
}

Statement(s) is executed repeatedly UNTIL the condition is true. BEFORE the first iteration, expr1is evaluated. This is usually used to initialize variables for the loop. AFTER each iteration of the loop, expr2 is evaluated. This is usually used to increment a loop counter.  

Example:
{
 sum = 0
 i = 1
 for (i=1; i<=10; i++)
 {
   sum += i; #  sum = sum + i
 }
printf "Sum for 1 to 10 numbers = %d ", sum

Now run the following program
$ cat > for_loop
click for code.

Run it as
$ awk -f for_loop
To test for loop
Press CTRL + C to Stop
Welcome to Linux!
Welcome vivek, 0 times.
Welcome vivek, 1 times.
Welcome vivek, 2 times.

Program uses for loop as follows

Here i++, is equivalent to i = i + 1 statement. ++ is increment operator which increase the value of variable by one and -- is decrement operator which decrease the value of variable by one. Don't try i+++, to increase the value of i by 2 (since +++ is not valid operator), instead try i+= 2.  

Following program shows the while loop
$ cat > while_loop
click for code.

Run it as
$awk -f while_loop 
654 
456
Next number please(CTRL+D to stop):587 
785
Next number please(CTRL+D to stop): 

Here user enters the number 654 which is printed in reverse order i.e. 456. Above program can be explained as follows

Our next example is quite interesting. Here I would like to read name of all files from the file and copy them to given destination file. For e.g. The file filelist.conf; looks something as follows 

/home/vivek/awks/temp/file1          /home/vivek/final
/home/vivek/awks/temp/file2          /home/vivek/final
/home/vivek/awks/temp/file3          /home/vivek/final
/home/vivek/awks/temp/file4          /home/vivek/final

In above file first field ($1) is the name of file that I would like to copy to the given destination ($2 - second field) i.e. copy /home/vivek/awks/temp/file1 file to        /home/vivek/final directory. For this purpose write the awk program as follows

$ cat > temp2final.awK
click for code.

Run it as 
$ awk -f   temp2final.awk   filelist.conf

Program can be explained as follows

system() function execute given command. For e.g. if you want to remove file using rm command of Linux, you can write system as follows

system("rm  foo") 

OR

dcmd = "rm " $1
system(dcmd)

The output of command is not available to program; but system() returns the exit code (error code) using which you can determine whether command is successful or not. For e.g. We want to see whether rm command is successful or not, you can write code as follows
$ cat > tryrmsys
{
dcmd = "rm " $1
if ( system(dcmd) != 0 )
    printf "rm command not successful\n"
else
    printf "rm command is successful and %s file is removed \n", $1
}

Run it as (assume that file foo exist and bar does not exist)

$ awk -f tryrmsys
foo
rm command is successful and foo file is removed 
bar
rm command not successful

(Press CTRL + D to terminate)

You can even take input from keyboard while running awk script, type the following script 
$ cat > testusrip
BEGIN  {
printf "Your name please:"
getline na < "-"
printf "%s your age please:",na
getline age < "-"
print "Hello " na, ", next year you will be " age + 1
}