Adding memory to VAXen: going beyond the 8M limit

[Chris Torek]

In article <1097 at trwrb.UUCP> simpson at trwrb.UUCP writes:
>A friend of mine has a VAX running 4.2 with 16 meg of memory; however,
>the VAX only recognizes 8 meg of memory.

Here is what I think must be done to a 4.1 or 4.2 BSD system to
make it handle more than 8M.  You should be aware that I have
never done this, and these instructions are therefore suspect.
But I can, at least, point out those things that are commonly
missed, and give sample changes (from BRL 4.2).

First, there is a loop in /sys/vax/locore.s that finds out how
much memory is available.  It looks like this:

	/* count up memory */
		clrl	r7
	1:	pushl	$4; pushl r7; calls $2,_badaddr; tstl r0; bneq 9f
		acbl	$8192*1024-1,$64*1024,r7,1b
	9:

(that is, probe at each 64K boundary until the probe causes a
machine check, but stop at 8192K, or 8M).  Obviously this 8192 must
be increased.  You can simply stick in a different constant if you
like, though this is not the best solution:  A better is defining
MAXMEM in cmap.h and using `acbl $MAXMEM*1024-1,$64*1024,r7,1b';
see below.

This is NOT sufficient.  There is also this thing called the
`core map', which has one entry for each `page' (4BSD likes to
deal in 1K pages, hence the quotes).  Included therein are
some `pointers' stored in bit fields.  These bit fields are
just large enough to map 8M, and therefore must be expanded.
So you will need to edit /sys/h/cmap.h.  The following excerpts
are from a cmap.h that is adapted for a 64M maximum (and more
mountable file systems):

	/*
	 * core map entry
	 *
	 * Limits imposed by this structure:
	 *
	 *		limit		     cur. size		fields
	 *	Physical memory+		64 Mb	c_next, c_prev, c_hlink
	 *	Mounted filesystems		255	c_mdev
	 *	size of a process segment	1 Gb	c_page
	 *	filesystem size			8 Gb	c_blkno
	 *	proc, text table size		64K	c_ndx
	 *
	 *	+ memory can be expanded by converting first three entries
	 *	to bit fields, shrinking c_ndx, and increasing MAXMEM below.
	 */
	#ifndef	LOCORE
	struct cmap
	{
	unsigned short 	c_next,		/* index of next free list entry */
			c_prev,		/* index of previous free list entry */
			c_hlink;	/* hash link for <blkno,mdev> */
	unsigned short	c_ndx;		/* index of owner proc or text */
	unsigned int	c_page:21,	/* virtual page number in segment */
			c_lock:1,	/* locked for raw i/o or pagein */
			c_want:1,	/* wanted */
			c_intrans:1,	/* intransit bit */
			c_free:1,	/* on the free list */
			c_gone:1,	/* associated page has been released */
			c_type:2,	/* type CSYS or CTEXT or CSTACK or CDATA */
			:4,		/* to longword boundary */
			c_blkno:24,	/* disk block this is a copy of */
			c_mdev:8;	/* which mounted dev this is from */
	};
	#else	LOCORE
	/*
	 * bit offsets of elements in cmap
	 */
	#define	C_INTRANS	87
	#define	C_FREE		88
	#define	SZ_CMAP		16		/* sizeof(struct cmap) */

	#define	MAXMEM		64*1024		/* maximum memory, in Kbytes */
	#endif	LOCORE
	...
	#ifndef	LOCORE
	#ifdef	KERNEL
	struct	cmap *cmap;
	...
	#define	cmtopg(x)	((((x)-1) * CLSIZE) + firstfree)
	#endif	LOCORE

Now then, what are all those `#ifndef LOCORE's doing in there?
Well, if you just make the above changes, and change the acbl
loop limit to `$64*1024*1024', your 4.1 machine will be perfectly
happy.  Your 4.2 machine, however, will run for a while, then
crash.  Why?  Because of `Fastreclaim', also in locore.s.  This
bit of code has hardwired into it the offsets of each of those
bit fields in /sys/h/cmap.h.

A good way to fix this is to add a `#include "../h/cmap.h"' at
the top of locore.s (along with the other `#include's), and use
those symbolic constants so carefully defined in BRL's cmap.h.
Then make Fastreclaim look more like this (also excerpted):

	/*
	 * Extracted and unrolled most common case of pagein (hopefully):
	 *	resident and not on free list (reclaim of page is purely
	 *	for the purpose of simulating a reference bit)
	 *
	 * Built in constants:
	 *	CLSIZE of 2, any bit fields in pte's
	 */
		.text
		.globl	Fastreclaim
	Fastreclaim:
	...
		subl2	_firstfree,r0
		ashl	$-1,r0,r0	
		incl	r0			# pgtocm(pte->pg_pfnum) 
		mull2	$SZ_CMAP,r0
		addl2	_cmap,r0		# &cmap[pgtocm(pte->pg_pfnum)] 
		tstl	r2
		jeql	2f			# if (type == CTEXT &&
		jbc	$C_INTRANS,(r0),2f	#     c_intrans)
		POPR; rsb			# 	let pagein handle it
	2:
		jbc	$C_FREE,(r0),2f		# if (c_free)
		POPR; rsb			# 	let pagein handle it 
	2:
		...

If you have enough memory, you may have to do one more thing:
increase SYSPTSIZE (probably in /sys/vax/vmparam.h).  If you do
not and your new kernel boots, fine; if it says `panic: sys pt too
small', this is the problem.  (As the comment says, SYSPTSIZE is
indeed silly; but it would be a bother to fix, so no one has.)

Of course, if all else fails you can simply get BRL's 4.2/SysV
distribution---though you will need a SysV source license---or
(in a few days perhaps, or maybe weeks) 4.3BSD. . . .
-- 
In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 1516)
UUCP:	seismo!umcp-cs!chris
CSNet:	chris at umcp-cs		ARPA:	chris at mimsy.umd.edu