GNU Emacs, memory usage, releasing

[News system owner ID]

In article <129799 at sun.Eng.Sun.COM> jpayne at sun.UUCP (Jonathan Payne) writes:
< ...
< I will never use a linked list of lines approach again.

Amen.  (after hacking too much on mg:) If I never see a linked list
buffer again it will be too soon.

< ...
< Lessee.  You complained about the memory usage and the slowness of buffer
< gap.  First of all, I think the average file in UNIX is much less than
< 100k.  Actually I KNOW the average file is much less than that, but for
< the hacker types, I bet the average size of a source files is also less
< than 100k, more like 50k and below.  The point is, moving the gap around
< is not that big a deal.  The amount of copying done is directly related
< to how far the gap is moving, not how big the file is, and most of the
< time the gap does not move very far!  It just doesn't.

Actually, as I (perhaps mistakenly recall), the Emacs Cookbook says
that a buffer gap is _more_ efficient than a linked list for files up
to about 75K in size.

< >7) Most interestingly when the gap closes, because we have inserted that
< >much worth of data, the *entire* buffer contents is realloc()ated. If the
< >buffer is not the top one in virtual memory, or it is but you have a a
< >realloc() that will not attempt just extending a block, but simply allocates
< >a new block and copies the old into the new, you are in for extra overheads.
< >They are no longer proportional to the amount of work you do, but to the
< >total size of the file as well, which may be bad news.
< 
< >8) most interestingly, realloc()ating the buffer will leave large holes in
< >your virtual address space, that will expand forever, taking up if anything
< >swap space. The holes are also likely to get fragmented as your session
< >progresses (remember, GNU emacs has such high overheads that you are
< >supposed to start it up once as a server and then use emacsclient as the
< >"real" editor), and therefore the virtual memory profile of your session
< >will worsen steadily.
< 
< These are the main problems with buffer gap.

Actually, they look to me like problems with this _implementation_ of
buffer gap.  Consider making the global data layout something like

	<random C data structures>
	<used elisp heap space>
	<availiable elisp heap space (possibly zero)>
	<used buffers>
	<unused buffer space>

The lisp heap should be kept with a compacting GC (incremental would
be nice too).  New versions of malloc and friends that interact well
with the lisp heap would be required, since the extention of buffer
memory needs to be much more low-level.

The basic idea is that when a buffer needs to be expanded, all the
buffers would be moved to open up the space.  This is obviously a
trade-off: it would't leave holes, but it would do *even more* memory
copying.  A portable, efficient, memcpy would be required for those
systems that don't have a nice in-assembler version.

		-- Paul Placeway