GNU Emacs, memory usage, releasing

[Israel Pinkas]

In article <1558 at aber-cs.UUCP> pcg at aber-cs.UUCP (Piercarlo Grandi) writes:

> 7) Most interestingly when the gap closes, because we have inserted that
> much worth of data, the *entire* buffer contents is realloc()ated. If the
> buffer is not the top one in virtual memory, or it is but you have a a
> realloc() that will not attempt just extending a block, but simply allocates
> a new block and copies the old into the new, you are in for extra overheads.
> They are no longer proportional to the amount of work you do, but to the
> total size of the file as well, which may be bad news.

> 8) most interestingly, realloc()ating the buffer will leave large holes in
> your virtual address space, that will expand forever, taking up if anything
> swap space. The holes are also likely to get fragmented as your session
> progresses (remember, GNU emacs has such high overheads that you are
> supposed to start it up once as a server and then use emacsclient as the
> "real" editor), and therefore the virtual memory profile of your session
> will worsen steadily.

One thing to note is that in most malloc() packages (including the one in GNU
Emacs), malloc() actually returns a pointer to a block of memory that is
2^n words long.  (Actually, the pointer usually points one or two words
into the block, to allow the size of the block to be stored so that free
knows how big the block is.)

The man page for realloc() specifically states that the block may or may
not be moved.  If you ask for a smaller block, and cross the 2^n word
threshold, realloc can do one of three things:

	1) grab a smaller block off the free list, and copy the data
	2) split the block, if that is supported
	3) silently hand back the same block

Obviously, 2 is the fastest, but tends to fragment memory.  3 is the same
speed, but wastes memory.  Some versions will use combinations of the 3
schemes, depending of what the free list looks like and current memory
usage.

When realloc() is called to allocate a larger chunk of memory, it will
often be able to return back the same chunk.  Remember, if the current
buffer is 100,000 bytes long, the block actually has room for 131,068 bytes
(10^17 - 4).  calling realloc() with a size of 102,000 is a no-op.

As the buffer gets larger, realloc() has to move the block less frequently.
At a certain point, the overhead is negligible.  (With the 100,000 byte
buffer, we would have to run out of room 16 times to necessitate moving the
block.  We would then have room for 66 gaps (over 130k) before having to
move the block again.

-Israel
--
--------------------------------------
Disclaimer: The above are my personal opinions, and in no way represent
the opinions of Intel Corporation.  In no way should the above be taken
to be a statement of Intel.

UUCP:	{decwrl,hplabs,oliveb,pur-ee,qantel}!intelca!mipos3!st860!pinkas
ARPA:	pinkas%st860.intel.com at relay.cs.net
CSNET:	pinkas at st860.intel.com