Is there a device memory limit?

[Donald Schmitz]

We are working on a package to support multiple (non-Sun) realtime CPUs
residing on a SUN3(160 or 260) VME bus, similar to VxWorks.  Currently we
are running OS 3.5, with 4.0.X on the way.  We wrote a driver that maps
processors boards into the Sun memory space and supports interrupt driven
communication between the Sun and remote CPU.  

The problem is, the system dies in various horrible ways when more than
one CPU device is configured into the kernel. By juggling the kernel size,
the machine will die in different ways, during things like mount or vi or
suntools, suggesting kernel memory is being corrupted.  Of course the
driver code has been scrutinized for the last 2 weeks, and all the
pointers seem to be nailed down tight.  The machine runs ok with more than
one CPU installed but not included as a device, so it doesn't seem to be a
hardware problem.  The one correlation we are seeing is that whenever the
hardware devices total more than 8M of memory (the add-on CPUs are 4M
each), we see the problems.  Since 8M is 1K of pages, this seems
suspiciously like some sort of hardware or software limit - however we
can't find any mention of such a limit in Sun's driver doc.

I'd appreciate hearing from anyone who can either confirm or deny this
guess, and if you've actually managed to support more than 8M of devices,
was there any special magic required? I'll sumarize any email responses
here.

Thanks, Don Schmitz (schmitz at fas.ri.cmu.edu)