How to make the tape streamer stream

[Martin Weitzel]

Some time ago Conor P. Cahil posted the following request:
>I am trying to collect data on the performance of the different tape
>backup systems.
[....]
>If you do run the test please send me the following info:
>
One file system on my machine produced the following results:

	CPU: 20 MHz 80386 (ACER 32/20 w 8 MB RAM)
	Tape drive: Archive 60MB 1/4" Cartridge (FT 60 & SC 422R)
	Disk Drive & controller: CDC WREN V (383H) & WD 1007V-SE2
	OS: 386/ix R 2.0.2
	Command: /bin time sh -c "find /usr -print | cpio -ocO /dev/tape"
	Time: Real: 9:08.5	User: 21.6	System: 2.38.3
	Size of file system: 68338 blocks (reported by du -s)
			     80066 blocks written by cpio

The times rate slightly above those that Conor P. Cahill found for
his (very similar) system - but I would like to comment a little on
mine. As I was never quite satisfied with the speed of my streamer
I played around in the last few days and have written some test
programs to see where the bottlenecks are. You are invited to
read on, though be warned: It's not a very short article.

Furthermore I must admit that I have not much background with respect
to the internals of streamers (and the QIC-02 standard?). I would
appreciate corrections from everybody with better technical background.

1) Problems with meassuring the time:
   -----------------
Concerning the method (proposed by the original poster) for meassuring
the time (/bin/time sh -c "find .... | cpio ...."), someone else
(Walter Mecky) has allready pointed out that only the "Real Time"
may be meaningful, as parent-child relationships in the shell usually
will not deliver the times of the *left* side of a pipeline. (If you
don't believe, try "factor 40000000003" - "4", nine "0", "3" - vs.
"factor 40000000003 | cat". In the latter case you will only see the
system/user times for cat, which are negligible.)

If 60 MB cartridge is constantly streaming, the write speed is ~83 KB/sec
or 5 MB/min. So if you take "start-up" and "turn-direction" time into
the calculation, the above times could not be much better. Further if
the tape is constantly streaming and stops only to turn direction, one
should be careful not to mis-interpret some timings: If it happens 
that the last (partially) written track was the 2nd, 4th, 6th or 8th,
timing results are not likely to change within about 7 MB of data
written to the tape, if you use the driver that rewinds the tape
finally. This can have some impact on the avarage result depending
on the number of blocks totally written (if it are at least 30 MB
as in the original posting suggested, the transfer rate in MB/min
may vary by 20% in the worst case as a quick calculation shows).

Note also that if the goal is to evaluate tape performance the timing
results should rather be based on the number of blocks written to the
tape, instead on the size of the file system (otherwise you include
the quality of the backup program, ie. by how much the data is expanded
by the backup program). Further taking the number of blocks written to
the tape as base of the evaluation makes it a little easier to compare
the results against the optimum of a constantly streaming tape.

2) Influence of buffer size.
   ------------------
After having made the above observations, I tried to find the maximum
speed with which cpio could work, if it had *not* to write to the tape.
In case of my "/usr"-file system, it was more than 250 KB/sec
(or 15 MB/min) so that it should be not much of a problem to keep
the tape streaming. But a little thought quickly showed that a large
buffer could rather tend to make the tape stop than a small buffer.

Why? Well, a test program(%1) has shown that writing to the tape
is handled slightly asynchroneous and it would be an advantage
if the buffers get filled rapidly. If eg. you specify buffer size
of around 250 KB, it would keep the tape streaming for three seconds,
then need one second to fill the buffer again. This will stop and
restart the tape. During most of the first three seconds cpio will
not continue to read the disk and when cpio reads the streamer will
wait. It may even happen that you reduce cpio-s effective throughput
below that of the streamer, though throughput would have been more
than sufficient with a small buffer! Practical experience emphasized
this theory and of course it seems clear that frequent stops of the
tape are performance killers.

After this I tried with various buffer sizes and observed that
the default - specifying non at all, neither "B" nor "Cnnn" -
would keep the tape better streaming than everything else. What
I'm not quite sure of is the influence of "cpio ... >/dev/tape"
vs. "cpio ... -O /dev/tape". IMHO the latter performs a bit
better, though I'm not quite sure. As it has other advantages
over output redirection, the second is clearly preferable anyway.

3) Backing up the spool directory with my news
   ------------------
Trying to do this (*lots* of small files) dramatically changed
figures. Here the throughput from "cpio ... >/dev/null" falls
below the streaming rate of the tape and goes as low as 45 KB/sec
(2.7 MB/min) in some cases(%2). As on this partition there are not
only the news but also some larger files, average throughput is
approx. 75 KB/sec - still not fast enough to keep up with the tape.

Could a large buffer at least help now by "smoothening" the throughput
a little, so that the tape does not stop so often? Can we get at
least somewhere near 75 KB/sec (currently it seems that caused by
the numereous stops of the tape throughput is even below 45 KB/sec)?
As backup typically is done when the load of the system is near to
zero, it would seem advisable to give huge amounts of memory to the
buffer - PROVIDED IT IS SIMULTANEOUSLY FILLED AND EMPTIED(%3).

The effects of buffering and especially the recommendable buffer size
depend much on the characteristics of the file system: Optimally
numereous small files would be alternatingly cpio-ed with some large
files, but in any case some really large files should come at the
beginning. It may be helpful to optimize that by meassuring the
"peaks" and the "over time distribution" of the transfer rate(%2),
but even then it may not be possible to achieve satisfactory results
in each and every case; ideally the buffering algorithms should be
somewhat "adaptive"(%3).

4) Conclusions
   ------------------
If your tape doesn't stream, you should first evaluate the
transfer rate which is requiered to keep the tape streaming.
Then take a measurement of the rate with which the data can be read
from disk. If both are very close, it might eventually be helpful
if you try to avoid the "find"-side of the pipe to cpio by running
find seperatly and storing the results in a file which you give
later as standard input to cpio. Note also that recent finds have
a "-cpio" option, which also may be helpful in reducing the overhead.
(Cpio-ing directly from find supports much less options than if
you execute cpio directly, but it may suffice for regular backups.)

If your meassurements show that disk reads are two to three times
faster than tape writes and that the latter are somewhat asynchroneous,
you are generally better off if you choose a small buffer. Especially
keep in mind that a large buffer alone seems never sufficient. Good
results would require simultaneous operation on different buffers. I've
heard that some cpio-s claim they do so, but at least for 386/ix
I doubt and it couldn't be confirmed from the manuals, though this
could be considered a detail of the implementation and would not
necessarily have to be mentioned. (Well, even important things would
not necessarily have to be mentioned as the current manuals show, but
that's quite a different story ... 1/2:-))

So you may either look for some PD backup program that does multi-
buffering or you may wait until I come up with a solution that can be
tacked to the end of a pipeline(%3). If there is sufficient interest
from the net and I find the time to write some documentation, I'll
make my programs available.

%1: A program that allows you to check the best streaming rate of
    your tape (and decide whether streaming is possible at all)
    and that lets you estimate the degree of "asynchronity" of
    the write requests to your tape driver.

%2: A program which shows you peak and average data rates with
    which the files can be read from the disk.
    
%3: A program that smoothens throughput on the way from the disk
    to the tape. (I allready have written one, but it's not quite
    what I finally want. Especially the buffering algorithms will
    have to be changed a little and made "intelligent".) This program
    will be usable in a pipe *after* cpio and do its best to get the
    streaming rate of the tape close to the rate with which the data
    is read from disk.
-- 
Martin Weitzel, email: martin at mwtech.UUCP, voice: 49-(0)6151-6 56 83