Chris's wiki | Main / SedAndAwk

sed replace apostrophe at end of line
$ needs to be in single quotes ' needs to be in double

sed s/'$'/"'"/ file

sed regex to reduce numeric precision by truncation - in this case finds 8 sig fig inc . followed by 3 number and replaces with the first 8 sig fig

sed -e 's/\(-*[.0-9]\{8\}\)\([0-9]\{1,3\}\)/\1/g' infile > outfile

awk to generate more awk to produce sql column=value from a list of columns in an sql table - suitable for an updaye type query - sounds worse than it is, run this on a list of columns and then run the output again on a tab delim file of data.

awk 'BEGIN { ct=1; sep=""; printf "nawk -F \"\\t\" '\''{ print \"" }
$1~/[A-Z][A-Z]+/ { printf "%s %s=|\"$%i\"|", sep, $1, ct++; sep="," }
END { printf "; \"} '\'' \n" } ' table-desc.txt

awk does unbiased rounding - so be careful when using to round numbers as it rounds to even - e.g. 1.5 rounds to 2 but 2.5 also rounds to 2. This awk checks the difference between two sets of values. If the vals are in two separate files then sdiff or diff --side-by-side to get the two files, side-by-side into one file.

$1~/[0-9][0-9]*\.[0-9]*/ {
        ++lnCt;
        for(i=1; i<11; ++i) {
                j=i+11;
                print lnCt " " $i " " $j;
                if (($i-$j) > 0.001 || ($i-$j) <  -0.001 ) {
                        bln = bln " " $i;
                        cur = cur " " $j;
                }
        }
        if (length(bln) > 1) {
                printf ("< %s\n", bln);
                printf ("> %s\n", cur);
        }
        cur = "";
        bln = "";
}

Monster awk script to do a context based grep - if your grep does not support -b and -a flags (modified from script posted by aigles (Jean-Pierre) from COMPUTING.NET)

To use enter something like
nawk -f ~/cgrep.awk -v context=0 -v before=25 -v after=0 pattern path/to/file


function PrintBeforeContext ( lindex, lfrom, lto) {
        lfrom = before_index - before
        if (lfrom < 0) lfrom = 0
        lto = before_index - 1

        print "\n --------- before context start ------------\n"

        for ( lindex = lfrom ; lindex <= lto ; lindex++ )
        print before_context[lindex % before]
        before_index = 0
}

BEGIN {
        context = context + 0
        before = before + 0
        after = after + 0

        printf ("\n\n>>>>>>>>>>  %s \n",ARGV[2])

        if (context > 0)
        {before = context
                after = context}
        before_index = 0
        before_context[0] = ""
        after_index = 0
        pattern = ARGV[1]
        ARGV[1] = ""
        if (ARGC <= 2) ARGV[ARGC++] = "-"
}

$0 ~ pattern {
        PrintBeforeContext()
        print $0
        after_index = after
        next
}

END {
        print "=======================\n\n"

}

after_index > 0 {
        print $0
        after_index--
        if (after_index == 0)    print "\n --------- after context end ------------\n"
        next
}

before > 0 {
        before_context[before_index % before] = $0
        before_index++
}

If you like to display the path in your prompt, but find it gets too long, this awk one-liner displays only the last 4 path elemets

pwd | awk -F/ '{ if(NF < 4 )print ; else {for (i=NF-3; i<=NF; i++) printf "%s/", $i}; }'

what it does is pipe pwd to awk which is told to use "/" as a separator (-F/) if the number of fields is less than 4 (NF < 4 ) it prints the path, otherwise it loops through the last 4 fields (path elements) and outputs them

These extract references from MySQL table definitions in a format suitable for input to my SVG ERD utility aSVERD (see http://asverd.sourceforge.net.

awk '/CREATE TABLE/{TBL=$3};$3~/FOREIGN/{printf("TABLE %s %s %s %s %s\n", TBL, $5, $6, $7, $8) }' *.ddl
awk '$7~/FOREIGN/{printf("TABLE %s %s %s %s %s\n", $3, $9, $10, $11, $12) }' *.ddl

Here's some awk to convert a table of information into a CREAT TABLE script. I used it to generate table scripts from table definitions in a Word document - the process was to copy the table info from the Word doc and paste into a text file then run this on it to produce a create table script.

BEGIN {split(ARGV[1], n, "."); print "CREATE TABLE " n[1] " (" }
{
        if($2=="A") {
                print $1 " VARCHAR2(" $3 ")," }
        else if ($2=="N") {
                print $1 " INTEGER," }
        else if ($2=="F") {
                print $1 " FLOAT," }
}
END {
        print "CREATION_ID VARCHAR2(30),"
        print "LAST_MOD_ID VARCHAR2(30),"
        print "CREATION_DATE DATE,"
        print "LAST_MOD_DATE DATE,"
        print "RECORD_STATUS VARCHAR2(1)"
        print ");"
}
=

and run it with something like

gawk -f gen-ddl.awk $1.txt | sed -e 's/\.0//g' > $1.DDL

The above line was in a shell script and called with a paramter of the name of the text file to be processed

Getting lots of script kiddies trying to crack your website, or spammers for that matter? Here is an awk script that extracts their IP from you access log and generates an IP tables rule. I append the output from this to a script file and then flush iptables and run the script.

awk '$7~"My" { print "iptables -I INPUT -s " $1 " -j DROP"  }' \
/var/log/httpd/access_log | sort | uniq >> baddies.sh

=

the example above looks for any request for any page containing "My" - because the script kiddies mainly seem to look for phpMyAdmin (you need to consider other variants and don't block yourself!), I also run this to find hits on my contact page (don't exclude legitimate users though).

Main.SideBar (edit)

SedAndAwk