Linux, programming, computers and life

I wrote about changing of certain columns in a file, but here let’s solve a simpler task. I’d like to add something, let’s say date, to every line of a file. I’ll emulate a file using `seq 1 10`.

The following is easy enough:
seq 1 10 | awk '{print "echo `date` "$0}' | bash

However, there’s simpler solution:
seq 1 10 | while read ; do echo `date` ${REPLY} ; done

It works because REPLY variable is set to the line of input read by the read builtin command when no arguments are supplied (from bash manpage).

I wanted to call this “advanced awk”, but it’s in fact simple idea with awk and bash integration. I got this idea from wonderful ArchLinux forums.
The problem is as follows. Given a file with several columns, i want to run a command on some columns and keep other columns as is.
For example, given a file with user names and some more information, i need to convert user names to user ids and keep other information unchanged. Example file is:

root a b c
ftp   d e f

The desired result is:

0   a b c
14 d e f

(of course, user id of ‘root’ is 0 and of ‘ftp’- 14)
Well, there’s a way to do this . First we’ll convert the file using awk or sed to ‘echo’ commands and then use bash to interpret those.Take a look at the output of the following command:
cat myFile | awk '{print "echo \"$(id -u "$1") " $2 " " $3"\""}'
And now, only thing left to do is to execute the commands awk created for us - i.e. pipe to bash
The final command is:
cat myFile | awk '{print "echo \"$(id -u "$1") " $2 " " $3"\""}' | bash

Enjoy!

Technorati Tags: bash, awk, cli, shell, script

One more awk trick . Let’s say you have a huge file. You only want to get all the lines of the file which are:

1. before certain string
2. after certain string

It’s easy to do it using any text editor, but it’ll take time and if the file is really huge you’ll have hard time finding a working editor. awk to the resque!

The solutions:

1. seq 1 30 | awk '/.*8.*/ { nextfile } { print $0 }'
2. seq 1 30 | awk '/.*8.*/ { pr = 1 } { if (pr == 1) print $0 }'
3. seq 1 30 | awk '/.*8.*/ { pr = 1; print $0 } { if (pr == 1) print $0 }'

In this example i use ‘<anything>8<anything>’ as the string which divides the file. The difference between (2) and (3) is whether the string itself is printed or not.

Short explanation: awk is instructed to look for regular expression (which is enclosed in ‘/’) and in (1) stops parsing the file before stopping each line is echoed as is. In (2) and (3) it sets pr variable to be 1 (it’s 0 by default and is short of “print”), and lines are echoed only if pr is 1.

Technorati Tags: awk, truncate file, cli, linux

I tried do do it using awk, but i did not find an easy way to determine the last line in file. So sed did the trick:

cat myFile | sed '1 {d}; $ {d}' > myNewFile

Technorati Tags: sed, awk, linux, cli

As i already mentioned, awk is great. More examples in this post.
Print every n’th line (n=15):
seq 1 100 | awk 'NR%15==0 {print $0}'
Print only n’th line (n = 10):
seq 1 100 | awk 'NR==10 {print $0}'
Print average value:
seq 1 10 | awk '{sum+=$1} END {print sum/NR}'

Feel free to try this at ~

Technorati Tags: awk, linux, cli, programming

awk is even better than i thought it is. It has arrays. Recently i needed to examine a log file, which includes mixed output from several threads, and to find 2 latest strings of each thread. I didn’t know how to do it, till a coworker mentioned to me about those arrays.
The script is extremely simple:

cat my_log_file.txt | awk '{arr[$2, 1] = arr[$2, 0]; arr[$2, 0]=$0} END { for (s in arr) print s" : "arr[s] }'

Some explanations:

• {arr[$2, 1] = arr[$2, 0]; arr[$2, 0]=$0}
- an operation which is performed on every line. Column 2 is the thread number. I keep in array with index [threadNumber,0] the last line and the line before last in index [threadNumber,1]
• END { for (s in arr) print s" : "arr[s] }'
  - the final operation - print all the array with it indexes

Technorati Tags: awk, programming, linux, array, shell, script

Recently i had a long text file (about 114K lines) which contained strictly formatted lines (output from some of our tools) which i wanted to sort by a certain columns.
While my first solution is usually command line based i thought “I will need a lot of different sorts - let’s do it in Excel - it’ll probably be faster over all”. So i went to my remote desktop, opened Excel and imported the file - which was very easy (i believe there’s File->Import and even File->Open worked). However there’s a limit of 64K lines in Excel - so i got less then 50% of the file. I didn’t give up. I remembered i have OO installed so i launched Calc and, after some time spent in menus browsing (well, there’s File->Export but no File->Import, i believe the solution was somewhere in Insert menu), i imported the file. Well. A surprise - OO also has line amount limitation.
I’m human after all, and i gave up. In order to find minimal and maximal values in 2nd column of the file i needed to do the following:

sort -un -k 2 MyFile | head -n 2

and

sort -unr -k 2 MyFile | head -n 2

Windows users will not even think of this solution…. And with sort, grep, awk and sed one can truly make wonders. Go read the man pages!

Technorati Tags: sed, awk, spreadsheet, sort, grep, excel, openoffice, linux