Is there a device memory limit?

[Donald Schmitz]

We are working on a package to support multiple (non-Sun) realtime CPUs
residing on a SUN3(160 or 260) VME bus, similar to VxWorks.  Currently we
are running OS 3.5, with 4.0.X on the way.  We wrote a driver that maps
processors boards into the Sun memory space and supports interrupt driven
communication between the Sun and remote CPU.  

The problem is, the system dies in various horrible ways when more than one
CPU device is configured into the kernel. By juggling the kernel size, the
machine will die in different ways, during things like mount or vi or
suntools, suggesting kernel memory is being corrupted.  Of course the driver
code has been scrutinized for the last 2 weeks, and all the pointers seem to
be nailed down tight.  The machine runs ok with more than one CPU installed
but not included as a device, so it doesn't seem to be a hardware problem.
The one correlation we are seeing is that whenever the hardware devices
total more than 8M of memory (the add-on CPUs are 4M each), we see the
problems.  Since 8M is 1K of pages, this seems suspiciously like some sort
of hardware or software limit - however we can't find any mention of such a
limit in Sun's driver doc.

I'd appreciate hearing from anyone who can either confirm or deny this
guess, and if you've actually managed to support more than 8M of devices,
was there any special magic required? I'll sumarize any email responses
here.

Thanks, Don Schmitz (schmitz at fas.ri.cmu.edu)