. the total amount of memory in the system didn't include the memory
used by the boot-time modules and some dynamic allocation by the
kernel at boot time (to map in VM). especially apparent on our
ARM board with 'only' 512MB of memory and a huge ramdisk.
. also: *add* the VM loaded module to the freelist after it has
been allocated for & mapped in instead of cutting it *out* of the
freelist! so we get a few more MB free..
Change-Id: If37ac32b21c9d38610830e21421264da4f20bc4f
. allow any number of pde's used for pagedir mapping
. allows >1024 NR_PROCS on x86, >64 on ARM
. allows NR_PROCS to be the same in both cases
. also cleanup: allocating spare PDE's is not necessary
throw that function out
Change-Id: Ibb8f8cf6e7db6a4d6384b6911d1a3f3f5e5d8256
if an exec() fails partway through reading in the sections, the target
process is already gone and a defunct process remains. sanity checking
the binary beforehand helps that.
test10 mutilates binaries and exec()s them on purpose; making an exec()
fail cleanly in such cases seems like acceptable behaviour.
fixes test10 on ARM.
Change-Id: I1ed9bb200ce469d4d349073cadccad5503b2fcb0
The 'polarity' of the RW bit is inversed on ARM, causing one
of the sanity check compensations to fail. ARM now runs basic
stuff with sanity checks passing.
Change-Id: Iee28ab63e430e759f204eeb204b24c301d5ea3c9
. make vm tell kernel virtual locations of mappings
. makes _minix_kerninfo feature work
. fix for mappings being larger than what 1 pde can address
(e.g. devices memory requested on arm)
. still requires a special case for devices memory for the
kernel, which has to switch to virtual addressing
Change-Id: I2e94090aa432346fa4da0edeba72f0b7406c2ad7
Due to the ABI we are using we have to use the earm architecture
moniker for the build system to behave correctly. This involves
then some headers to move around.
There is also a few related Makefile updates as well as minor
source code corrections.
Fix warnings about:
. Unused variables
. format mismatch in printf/scanf format string and arguments
. Missing parenthesis around assignment as truth values
. Clang warnings anout unknown GCC pragma
* Updating common/lib
* Updating lib/csu
* Updating lib/libc
* Updating libexec/ld.elf_so
* Corrected test on __minix in featuretest to actually follow the
meaning of the comment.
* Cleaned up _REENTRANT-related defintions.
* Disabled -D_REENTRANT for libfetch
* Removing some unneeded __NBSD_LIBC defines and tests
Change-Id: Ic1394baef74d11b9f86b312f5ff4bbc3cbf72ce2
This patch uses stricter locking for REQ_LINK, REQ_MKDIR, REQ_MKNOD,
REQ_RENAME, REQ_RMDIR, REQ_SLINK and REQ_UNLINK. For all requests, VFS
locks the directory in which we add or remove an inode with VNODE_WRITE.
I.e., the operations have exclusive access to that directory.
Furthermore, REQ_CHOWN, REQ_CHMOD, and REQ_FTRUNC now lock the vmnt
VMNT_READ; VMNT_WRITE was unnecessary.
Because pipes have no file position. VFS maintained (file) offsets into a
buffer internal to PFS and stored them in vnodes for simplicity, mixing
the responsibilities of filp and vnode objects.
With this patch PFS ignores the position field in REQ_READ and REQ_WRITE
requests making VFS' job a lot simpler.
.sync and fsync used unnecessarily restrictive locking type
.fsync violated locking order by obtaining a vmnt lock after a filp lock
.fsync contained a TOCTOU bug
.new_node violated locking rules (didn't upgrade lock upon file creation)
.do_pipe used unnecessarily restrictive locking type
.always lock pipes exclusively; even a read operation might require to do
a write on a vnode object (update pipe size)
.when opening a file with O_TRUNC, upgrade vnode lock when truncating
.utime used unnecessarily restrictive locking type
.path parsing:
.always acquire VMNT_WRITE or VMNT_EXCL on vmnt and downgrade to
VMNT_READ if that was what was actually requested. This prevents the
following deadlock scenario:
thread A:
lock_vmnt(vmp, TLL_READSER);
lock_vnode(vp, TLL_READSER);
upgrade_vmnt_lock(vmp, TLL_WRITE);
thread B:
lock_vmnt(vmp, TLL_READ);
lock_vnode(vp, TLL_READSER);
thread A will be stuck in upgrade_vmnt_lock and thread B is stuck in
lock_vnode. This happens when, for example, thread A tries create a
new node (open.c:new_node) and thread B tries to do eat_path to
change dir (stadir.c:do_chdir). When the path is being resolved, a
vnode is always locked with VNODE_OPCL (TLL_READSER) and then
downgraded to VNODE_READ if read-only is actually requested. Thread
A locks the vmnt with VMNT_WRITE (TLL_READSER) which still allows
VMNT_READ locks. Thread B can't acquire a lock on the vnode because
thread A has it; Thread A can't upgrade its vmnt lock to VMNT_WRITE
(TLL_WRITE) because thread B has a VMNT_READ lock on it.
By serializing vmnt locks during path parsing, thread B can only
acquire a lock on vmp when thread A has completely finished its
operation.
mount.c: In function 'mount_pfs':
mount.c:395:17: error: variable 'rfp' set but not used [-Werror=unused-but-set-variable]
Change-Id: I2f22590ab4e3a4a1678e9096626ebca53d2660e6
. make vm be able to use malloc() by overriding brk()
and minix_mmap() functions
. phys regions can then be malloc()ed and free()d instead
of being in an avl tree, which is slightly faster
. 'offset' field in phys_region can go too (offset is implied
by position in array) but leads to bigger code changes
new_node makes the assumption that when it does last_dir on a path, a
successive advance would not yield a lock on a vmnt, because last_dir
already locked the vmnt. This is true except when last_dir resolves
to a directory on the parent vmnt of the file that was the result of
advance. For example,
# cd /
# echo foo > home
where home is on a different (sub) partition than / is (default
install). last_dir would resolve to / and advance would resolve to
/home.
With this change, last_dir resolves to the root node on the /home
partition, making the assumption valid again.
. 'anonymous' cache blocks (retrieved with NO_DEV as dev
parameter) were used to implement read()s from holes in
inodes that should return zeroes
. this is an awkward special case in the cache code though
and there's a more direct way to implement the same functionality:
instead of copying from a new, anonymous, zero block, to
the user target buffer, simply sys_safememset the user target
buffer directly. as this was the only use of this feature,
this is all that's needed to simplify the cache code a little.
- CHOOSETRAP define makes impossible to use some common words
like send, receive and notify in any other context, for
instance as members or structures
- any reasonable compiler inlines the static inline functions so
no extra function call overhead is introduced by this change
- this gets us back to the situation before the SYSCALL/SYSENTER
change. It is not perfect, but it used to work and still does.
The tested targets are the followgin ones:
* tools
* distribution
* sets
* release
The remaining NetBSD targets have not been disabled nor tested
*at all*. Try them at your own risk, they may reboot the earth.
For all compliant Makefiles, objects and generated files are put in
MAKEOBJDIR, which means you can now keep objects between two branch
switching. Same for DESTDIR, please refer to build.sh options.
Regarding new or modifications of Makefiles a few things:
* Read share/mk/bsd.README
* If you add a subdirectory, add a Makefile in it, and have it called
by the parent through the SUBDIR variable.
* Do not add arbitrary inclusion which crosses to another branch of
the hierarchy; If you can't do without it, put a comment on why.
If possible, do not use inclusion at all.
* Use as much as possible the infrastructure, it is here to make
life easier, do not fight it.
Sets and package are now used to track files.
We have one set called "minix", composed of one package called "minix-sys"
The VFS/FS protocol does not require the file server to supply a
special device node number in response to a REQ_CREATE request, as
this call creates only regular files. Therefore, VFS should not
erroneously save this piece of information from the REQ_CREATE reply
either.
Upon reboot VFS semi-exits all processes and unmounts the file system.
However, upon unmount, exiting FUSE file systems might need service from
the file system (due to libc). As the FUSE process is halfway the exit
procedure, it doesn't have a valid root directory and working directory.
Trying to do system calls then triggers a sanity check in VFS.
This fix first exits normal processes which should then allow for
unmounting FUSE file systems. Then VFS exits all processes including
File Servers and unmounts the rest of the file system.
There is a deadlock vulnerability when there are no worker threads
available and all of them blocked on a worker thread that's waiting for a
reply from a driver or a reply from an FS that needs to make a back call. In
these cases the deadlock resolver thread should kick in, but didn't in all
cases. Moreover, POSIX calls from File Servers weren't handled properly
anymore, which also could lead to deadlocks.
. also make other out-of-memory conditions less fatal
. add a test case for a user program using all the memory
it can
. remove some diagnostic prints for situations that are normal
when running out of memory so running the test isn't noisy
Add primary cache management feature to libminixfs as mfs and ext2
currently do separately, remove cache code from mfs and ext2, and make
them use the libminixfs interface. This makes all fields of the buf
struct private to libminixfs and FS clients aren't supposed to access
them at all. Only the opaque 'void *data' field (the FS block contents,
used to be called bp) is to be accessed by the FS client.
The main purpose is to implement the interface to the 2ndary vm cache
just once, get rid of some code duplication, and add a little
abstraction to reduce the code inertia of the whole caching business.
Some minor sanity checking and prohibition done by mfs in this code
as removed from the generic primary cache code as a result:
- checking all inodes are not in use when allocating/resizing
the cache
- checking readonly filesystems aren't written to
- checking the superblock isn't written to on mounted filesystems
The minixfslib code relies on fs_blockstats() in the client filesystem to
return some FS usage information.
Introduce explicit abstractions for different mapping types,
handling the instantiation, forking, pagefaults and freeing of
anonymous memory, direct physical mappings, shared memory and
physically contiguous anonymous memory as separate types, making
region.c more generic.
Also some other genericification like merging the 3 munmap cases
into one.
COW and SMAP safemap code is still implicit in region.c.
The check_bsf() macro uses assert(mutex_trylock(&bsf_lock)) and
assumes bsf_lock is locked afterwards. This breaks when compiling
with NOASSERTS="yes". Also: macro to function transition.
. add cpufeature detection of both
. use it for both ipc and kernelcall traps, using a register
for call number
. SYSENTER/SYSCALL does not save any context, therefore userland
has to save it
. to accomodate multiple kernel entry/exit types, the entry
type is recorded in the process struct. hitherto all types
were interrupt (soft int, exception, hard int); now SYSENTER/SYSCALL
is new, with the difference that context is not fully restored
from proc struct when running the process again. this can't be
done as some information is missing.
. complication: cases in which the kernel has to fully change
process context (i.e. sigreturn). in that case the exit type
is changed from SYSENTER/SYSEXIT to soft-int (i.e. iret) and
context is fully restored from the proc struct. this does mean
the PC and SP must change, as the sysenter/sysexit userland code
will otherwise try to restore its own context. this is true in the
sigreturn case.
. override all usage by setting libc_ipc=1
. whenever this function is called, pm will expect
the process to be cleaned up
. so don't abort the process entirely on error
. fixes a later 'forking on top of in-use child' vfs panic
fixes an assert() firing when starting X. thanks to the report by pikpik.
. NO_MEM was 0, which is actually an existing piece
of physical memory. it can't be allocated because it's reserved
for bios data (by the kernel), but it can be mapped in (e.g.
by X), causing sanity check disaster.
. NONCONTIGUOUS is also obsolete as all allocations are single-page
now, i.e. NONCONTIGUOUS is really the default and only mode.
complete munmap implementation; single-page references made
a general munmap() implementation possible to write cleanly.
. memory: let the MIOCRAMSIZE ioctl set the imgrd device
size (but only to 0)
. let the ramdisk command set sizes to 0
. use this command to set /dev/imgrd to 0 after mounting /usr
in /etc/rc, so the boot time ramdisk is freed (about 4MB
currently)
. only reference single pages in process data structures
to simplify page faults, copy-on-write, etc.
. this breaks the secondary cache for objects that are
not one-page-sized; restored in a next commit
By decoupling synchronous drivers from VFS, we are a big step closer to
supporting driver crashes under all circumstances. That is, VFS can't
become stuck on IPC with a synchronous driver (e.g., INET) and can
recover from crashing block drivers during open/close/ioctl or during
communication with an FS.
In order to maintain serialized communication with a synchronous driver,
the communication is wrapped by a mutex on a per driver basis (not major
numbers as there can be multiple majors with identical endpoints). Majors
that share a driver endpoint point to a single mutex object.
In order to support crashes from block drivers, the file reopen tactic
had to be changed; first reopen files associated with the crashed
driver, then send the new driver endpoint to FSes. This solves a
deadlock between the FS and the block driver;
- VFS would send REQ_NEW_DRIVER to an FS, but he FS only receives it
after retrying the current request to the newly started driver.
- The block driver would refuse the retried request until all files
had been reopened.
- VFS would reopen files only after getting a reply from the initial
REQ_NEW_DRIVER.
When a character special driver crashes, all associated files have to
be marked invalid and closed (or reopened if flagged as such). However,
they can only be closed if a thread holds exclusive access to it. To
obtain exclusive access, the worker thread (which handles the new driver
endpoint event from DS) schedules a new job to garbage collect invalid
files. This way, we can signal the worker thread that was talking to the
crashed driver and will release exclusive access to a file associated
with the crashed driver and prevent the garbage collecting worker thread
from dead locking on that file.
Also, when a character special driver crashes, RS will unmap the driver
and remap it upon restart. During unmapping, associated files are marked
invalid instead of waiting for an endpoint up event from DS, as that
event might come later than new read/write/select requests and thus
cause confusion in the freshly started driver.
When locking a filp, the usage counters are no longer checked. The usage
counter can legally go down to zero during filp invalidation while there
are locks pending.
DS events are handled by a separate worker thread instead of the main
thread as reopening files could lead to another crash and a stuck thread.
An additional worker thread is then necessary to unlock it.
Finally, with everything asynchronous a race condition in do_select
surfaced. A select entry was only marked in use after succesfully sending
initial select requests to drivers and having to wait. When multiple
select() calls were handled there was opportunity that these entries
were overwritten. This had as effect that some select results were
ignored (and select() remained blocking instead if returning) or do_select
tried to access filps that were not present (because thrown away by
secondary select()). This bug manifested itself with sendrecs, but was
very hard to reproduce. However, it became awfully easy to trigger with
asynsends only.
Instead of using a loop to find a matching ipc (inter process
communication) system call type, the offset in the call table can be
simply calculated in constant time.
Also, when the interprocess communication server receives an ipc
system call from a process, ipc should tell VM to watch the process
only once. This patch fixes that also.
(Patch and commit message slightly edited by committer.)
. ld.so is linked at 0 but it can relocate itself; we
wish to load ld.so higher though to trap NULL dereferences.
if we know we have to execute ld.so, vfs tells libexec to put it
higher.
. done by RS to reduce/remove dependency on VM for recovery
. RS has the default stack size of 64MB since the nosegments
change, using a huge amount of unused memory to pre-allocate
. ignore these requests until actually required (i.e. being able
to survive VM crashes)
Thanks to pikpik for investigating why RS was so huge.
