Your favourite UNIX-pipe (summary)

[Andreas Lampen]

Hi folks,

several weeks ago, I asked for

	Your most favourite (sophisticated, cryptic, funny) pipe !

I received a lot of them. Thank you all. Here is a list

--------------------------------------------------------------------------

1) Matt Crawford (matt at oddjob.uchicago.edu)

Here's one csh alias I like, although the pipe is not the best part of it. 

alias	ptr	"(" echo set q=PTR ";" echo \!\$:e.\!\$:r:e.\!\$:r:r:e.\!\$:r:r:r.in-addr.arpa ")" "|" nslookup

Here's a script which is one long, fun pipe.  (I wrote it when I was new
to unix and didn't know that sh was better than csh for scripts.)

#! /bin/csh -f
# Script to notify heavy disk users of their sins
# Change the constant at the beginning of the awk command as needed.
#
(/usr/etc/quot -f /dev/rxy0e ; /usr/etc/quot -f /dev/rxy0f ; \
 /usr/etc/quot -f /dev/rxy1e ; /usr/etc/quot -f /dev/rxy1f) | \
 egrep -v 'root|/dev/' | \
 sort -nr | \
 awk '$1 > 8000 { printf "mail -s Disk-Usage %s << EOM\n", $3 ;\
    printf "Your %d files in your home directory are using %d KB.\\n",$2,$1 ;\
    printf "This makes you the number %d disk-hog.\n",NR ;\
    printf "EOM\n" ; }' | \
 csh -sf

2) Ken Yap (ken at cs.rochester.edu)

  One of my favourites for renaming all .pas files to .p (or similar) is

	ls *.pas | sed 's/\(.*)\.pas/mv & \1.p/' | sh

3) Andy Behrens (andyb at coat.com)

The next is part of a shell script that runs multiple copies of
compress in parallel, for rapidly compressing large directories.  (Our
host is a multiprocessor, so this really does save time).  The case
statement probably causes a subshell to be started up, so I should
rewrite it so it will run faster.

    PROGRAM=compress
    PARALLEL=10				# number of processors
    P=`expr $N / $PARALLEL + 1`		# each copy of compress gets $P files

    echo "$files" |
    xargs -n$P | 
    case "$trace" in
       -v)  sed "s/.*/$PROGRAM -v   & 2>\&1 | cat  \&/"; echo wait ;;
       *)   sed "s/.*/$PROGRAM      & 2>\&1        \&/"; echo wait ;;
    esac |
    sh |
    sed 's/-- replace.*//'

4) Frank P. Bresz (..!uunet!ittc!fbresz)

#! /bin/csh -f
#
#	Postport Postscript File Conversion.  TLM
#
#	4/10/89 - modified to make lpr use -s option.  TLM
#
if ($#argv == 0) then
	expand | sed -f /usr/local/bin/postsed | awk -f /usr/local/bin/postawk | lpr -s
else
	cat $argv | expand | sed -f /usr/local/bin/postsed | awk -f /usr/local/bin/postawk | lpr -s
endif

This is postsed
s/\\/\\\\/g
s/[(]/\\(/g
s/[)]/\\)/g

This is postawk
BEGIN { cpage = 0
	newline = 720
	newpage = 0
	print "%!\n%%Pages: (atend)\n%%DocumentFonts: Courier"
	print "%%EndComments\n/docstart save def"
	print "/Courier findfont 11 scalefont setfont\n%%EndProlog" }
{ if (newpage == 0) {
	cpage++
	newpage++
	print "%%Page: \"" cpage "\" " cpage "\n/pagestart save def" } }
{ if ( length($0) != 0 ) {
	{ if ( $0 != "\f" ) {
		printf "(%s) 25. %3d. moveto show\n", $0, newline }
	  else {
		{ if ( newline != 720 ) {
			newline = 64 }
		  else {
			newline += 12 } } } } } }
{ newline -= 12 }
{ if (newline <= 64) {
	newline = 720
	newpage--
	print "pagestart restore\nshowpage" } }
END { { if (newline != 720) {
	print "pagestart restore\nshowpage" } }
	{print "%%Trailer\ndocstart restore\n%%Pages: " cpage } }

5) John M. Blasik (john at mintaka.mlb.semi.harris.com)

	rsh other.sun screendump | screenload

[We had a lot of fun with that, Andy]

6) David Elliott (dce at smsc.sony.com)

There are two cases of pipes that I use a lot.

The first is backquotes, and I only mention it to keep you
honest.  I often do things like

	vi `grep -l pattern *.[ch]`

In general, I use the more typical pipes interactively (you don't
want to deal with sh program type pipes, do you?) in an iterative
way.  For example, a csh session might look like:

	% grep pattern1 *.[ch]
	<too much stuff>
	% !! | grep -v pattern2
	grep pattern1 *.[ch] | grep -v pattern2
	<still too much>
	% !! | grep pattern3
	...

until I finally get the set of lines I was looking for.  This
isn't efficient, but it's conceptually the easiest way to do this,
as having to use egrep with OR'ed patterns takes a lot more
typing.

7) Dick Dunn (rcd at ico.isc.com)

A short pipe sequence I use fairly often is
	whatever-stuff | sort | uniq -c | sort -nr | more
which produces a frequency count (in descending order) of lines produced by
"whatever-stuff".

8) Brian Rice (rice at dg-rtp.dg.com)

Wenn man mehreren Prozessen mit aehnlichen Namen (z.B., "biod") toeten
will, kann er benuetzen:

ps -e | grep biod | grep -v grep | awk '{print $2}' | xargs kill -9

(Mit Berkeley UNIX setzt man "-agx" an die Stelle von "-ef", und
auch "grep -v PID | " vor "xargs" einschaltet.)

9) boyd at necisa.ho.necisa.oz.au

It's not one you'd type on the command line, but it's pretty involved.

case "$1" in
8???????)
	;;

"")
	echo "usage: `basename $0` vaddr" 1>&2
	exit 1
	;;

*)
	echo "`basename $0` \"$1\" is not a valid virtual address." 1>&2
	exit 1
	;;
esac

(
	echo 8i
	(
		addr="`echo "$1" | tr '[a-f]' '[A-F]'`"
		echo "*pc* |0`echo 16i8o${addr}p | dc`|.text"
		nm -vo unix
	) | sed -e '/^etext/s/$/.text/' -e '/\.text$/!d' -e 's/|02//' -e 's/|.*//' -e 's/^/[/' -e 's/  */	]P/' -e 's/$/pc/'
) | dc | sort -n +1 | awk '
BEGIN		{
			addr[1] = 0
			addr[2] = 0
			pc = -1
			symbol = "zero"
		}

$1 == "*pc*"	{
			pc = $2
			next
		}

pc != -1 && $2 > pc	{
			addr[2] = $2
			exit
		}

		{
			symbol = $1
			addr[1] = $2
		}

END		{
			printf("0x%x 0x%x\t%s+0x%x (0x8%07x)\n", addr[1], addr[2], symbol, pc - addr[1], pc)
		}
'

10) Paul Davey (pd at ixi.co.uk)

alias tarcp " (cd \!:1 ; tar cf - . ) | ( cd \!:2 ; tar xf - )"

alias psg 'ps aux | sed -n -e  "/sed -n -e /d\\
				/\!$/p\\
				/TIME COMMAND/p"'

set lsfields = `/bin/ls -ld / | wc -w`
if ( "$lsfields" == "8" ) then
	setenv SYS BSD
else
	setenv SYS SYS5
endif
unset lsfields

11) Charles Geyer (charlie at milton.u.washington.edu)

#! /bin/sh
if test $1 = '-b'
then
  shift
  B='-b'
else
  B=''
fi
if test -f $1.spell
then  sed 's/\\[a-zA-Z]*//g' $1.tex | spell $B | sort | comm -23 - $1.spell
else sed 's/\\[a-zA-Z]*//g' $1.tex | spell $B | more

#! /bin/sh
CODA=">/dev/null 2>&1"          # This is the magic
rhost=$1
shift
options=$*
host=`hostname`.stat.washington.edu
DISP="-display $host":0.0
if test `xhost | grep $rhost | wc -l` -eq 0
then
  xhost $rhost
fi
echo "xterm -n $rhost -ls $options $DISP $CODA &" | rsh $rhost sh

12) Martin Weitzel (martin at mwtech.UUCP)

Consider the classic
example where all files with name *.c should be renamed to *.c.bak.

The first approach does it this way:

	for i in `ls *.c`
	do
		mv $i $i.bak
	done

Besides that it is difficult to invert if run-time performance is an
issue (renaming *.c.bak to *.c commonly requires execution of an
`expr ... : ...` in every cycle of the loop), it has more drawbacks:
The list of names generated by ls *.c can not exceed the limit
of several KByte (excact value depends, but is sufficiently low
to cause problems sometimes) and there are some hassles if you
must take the possibility of "no matching files" into account.

Now, compare this to

	ls | sed -n '/\.c$/s=.*=mv & &.bak=p' | sh

which has none of the above drawbacks. Doing the reverse is
not much more complicated:

	ls | sed -n 's=\(.*\.c\)\.bak$=mv & \1=p' | sh

Furthermore, *very* sophisticated things can be done by tailoring
some shell script "on the fly" (as above with "sed", with "awk",
....  or even with an "echo" from another shell) and piping this
to a shell. As shown above this technique at least will take the
burden of looping away from the shell, because the loop is embedded
now in the method that generates the data ("ls"ing a directory
"cat"ing the contents of a file or whatever).

13) Peter da Silva (peter at ficc.ferranti.com)

[Responding to Martin Weitzel (above)]

I would write

ls *.c | while read i; do mv $i $i.bak; done

Or,

ls *.c | sed 's/\(.*\).c$/& \1.bak/' | while read i j; do mv $i $j; done

This does have a bit of looping overhead, and is a big help when what you
want to do is a bit too complex to put into your sed command:

find . -name *.c -print | while read file
  do
    grep '^something:' $file | while read junk body
      do
	process $file -options $body
	and more stuff
      done
  done

If this can qualify as a single 'pipeline', then I have some real bottlers.

14) Jan Christiaan van Winkel (jc at atcmp.nl)

I once had three big files (just below 1 MB each) that when catenated 
made a uuencoded compressed TAR file. The problem was that on the system 
I work on, the ulimit is set to 1 Mb, i.e. I could not create files larger than 
1 Mb. Therefore I had to use a pipe to un-tar the file, for example something 
like

cat xaa xab xac | uudecode | compress -d | tar xvf -

However, uudecode INSISTS on writing to the file named in the first line.
In this case 'tarfil.Z'. This would create a file larger than
1 Mb. (Okay, I could have used the source of uudecode to change that, but I
wanted to do it quick 'n' dirty).
I used a named pipe with the name tarfil.Z to let uudecode write to.
Thus, the commands became:

cat xaa xab xac | uudecode &
compress -d < tarfil.Z | tar xvf -

Mayby not sophisticated, but it sure hepled me!

15) Mitchell Wyle (wyle at inf.ethz.ch)

tr -cs A-Za-z '\012' | tr A-Z a-z | sort | uniq -c | sort -rn

Try that one in a "high-level" language!

16) Joe Bush (bush at evax.arl.utexas.edu)

	Here is one I concocted to search for key words in include
files:

echo -n "key=";set kw=`line`;find /usr/include/. /sys/. -name \*.h -print | xargs hgrep "$kw" {} 

	I keep a file of such one-liners like the one above (file of
one line pipe-programs is named $HOME/.syscom) and have the following
line in my .cshrc:

alias g 'set j=`cat ${home}/.syscom|wc -l`;source -h ${home}/.syscom; history | tail -"$j"'

	Then when I enter "g" from the keyboard, my csh history
mechanism gets primed for easy execution. I find it quite handy...

17) Bjorn Engsig (bengsig at oracle.nl, bengsig at oracle.com)

Neither sophisticated, cryptic or funny, but often useful:
$ make foo
make: don't know how to make `foo'
$ make -n all | grep foo | sh -x

18) Thomas Truscott (trt at rti.rti.org)

Here is a shell script that has a pipeline in it somewhere.

#! /bin/sh

# you pick a word, e.g. "hangman".
# if antihang guesses "a", the new pattern is ".a...a.".
# if antihang then gueses "t", the new pattern is ".a...a." (no change).
# antihang fails on words not in /usr/dict/words

wordlist=/usr/dict/words
myguess=e	guesses=
while [ "$myguess" ]; do
	guesses=$guesses$myguess
	echo -n "I guess $myguess, new pattern: "
	read pattern
	srchpat=`echo "$pattern" | sed "s/\./[^$guesses]/g"` 
	myguess=`grep "^$srchpat$" $wordlist | \
		sed -e "s/[$guesses]//g" -e 's/\(.\)\(.*\)\1/\1\2/g' -e 's/./&\\
/g' | \
		sort |  uniq -c |  sort -nr | \
		sed -n -e 's/^.*\([^ ]\)$/\1/p' | sed 1q`
done

19) Heiner Marxen (heiner at specs.uucp)

Du sammelst doch Pfeifen :-), also hab' ich mal in meine aliase gesehen.
Unten ein Exerpt meines ".login" (fuer csh).  Interessant ist vielleicht
das alias "wot", es enthaelt eine pipe in `` als Teil einer pipe.
     
# Define macros for symbolic naming of directories
# and fast travelling between them.
     
a hier          set GO'\!:1'=\`/bin/pwd\`
a here          set GO'\!:1'='"`dirs | sed '"'"'s/ .*//'"'"'`"'
a go            echo cd \"\$GO'\!:1'\" \; cd \$GO'\!:1'
a gp            echo pd \"\$GO'\!:1'\" \; pd \$GO'\!:1'
a wo            set \| sed '"/^GO/\\!d;s/^GO//"'
a Wo            wo \| sort +1
a wg            wo \| grep
a wot   wo \| egrep '"  (`dirs|sed '"'"'s/ .*//'"'"'`|`pwd`)"'
a wo.   wo \| egrep '"  (`dirs|sed '"'"'s/ .*//'"'"'`|`pwd`)"'\\$
     
a unhere        unset GO'\!:1'
a hereset       set GO'\!:1'='\!:2'
     
a putgo         set \|  \
                sed "'"'/^GO/\\!d;s/$/"/;s/     /       "/;s/^GO/hereset /'"'"
 \
                \>\! ~/.gosave
a getgo         "if( -e ~/.gosave ) source ~/.gosave"
     
     
Dann noch ein gar nicht so untypischer Fall fuer Kommandos, die mit "sh"
implementiert werden.  Das folgende ist Teil eines lokalen scripts um
unseren Postscript-Laser zu beschicken:
     
     
case $# in
 0)     expand $tab | pr5 -w190 $two -f -l77 $prflags
        ;;
 *)     for f
        do
                t="$two"
                l=`sed -e 68q "$f" | wc -l | sed -e 's/^ *//' -e 's/ .*//'`
                case $l in
                 ? | [1-5]?)    t=""    ;;
                 6?)
                        case `expr $l \<= 67` in
                         1)     t=""    ;;
                        esac
                esac
                expand $tab "$f" | pr5 -w190 $t -f -l77 $prflags -h "$f"
        done
        ;;
esac    | lpscript -s7 -r -o1.0 | lpr -Pps

20) Phil Budne (budd at bu-it.bu.edu)

For seeing if anything of mine is stuck in the mqueue (I call it "mq");

	#!/bin/sh
	# display my items in the mail queue
	if [ $# -gt 0 ]; then
		who=$1
	else
		who=`whoami`
	fi
	echo Pending messages on `hostname` from user $who
	#
	mailq | \
	awk "	BEGIN		{ pr = items = found = 0; } \
		/^AA/		{ if( \$0 ~ /$who/ || \$0 ~ /$who\@\.\*/ ) { \
					pr = 1; found++; } \
				  else { pr = 0; } items++; } \
				{ if( pr ) print } \
		END		{ if( found > 0 ) fm = \"(\" found \" displayed)\";
				  print items \" total \" fm } "

A summation of rwho output (I call it "uz");
	#!/bin/sh
	if tty -s <&1;
	then
	    rwho $* | sed 's/[ 	].*//' | uniq | fmt
	else
	    rwho $* | sed 's/[ 	].*//' | uniq
	fi

A count of users as displayed by "uz" (I call it "uc");
	rwho $* | sed 's/[ 	].*//' | uniq | wc -l

For seeing what batched news is queued for a site (I call it "gq")

	#/bin/sh
	# NOTE! our /usr/spool/news is in /news
	SPOOLDIR=/news
	if [ "$1" -a -f $SPOOLDIR/batch/$1 ]; then
	    sed -e 's@/[0-9].*$@@' \
		-e "s@^$SPOOLDIR/@@" \
		-e 's@/@. at g' < /usr/spool/news/batch/$1 |\
	    sort | uniq -c | sort -rn
	fi

Show most frequently executed commands (I call it "lastfreq");

	lastcomm | sed 's/ .*//' | sort | uniq -c | sort -rn

-- 
----
  Andreas Lampen, Tech. Univ. Berlin
      andy at coma.cs.tu-berlin.de