This commit removes the secondary cache code implementation from
VM and its usage from libminixfs. It is to be replaced by a new
implementation.
Change-Id: I8fa3af06330e7604c7e0dd4cbe39d3ce353a05b1
. test70: regression test for m_out vfs race condition
The following tests use testcache.c to generate test i/o
patterns, generate random write data and verify the reads.
. test71: blackbox full-stack test of FS operation, testing
using the regular VFS interface crazy i/o patterns
with various working set sizes, triggering only
primary cache, also secondary cache, and finally
disk i/o and verifying contents all the time
. test72: unit test of libminixfs, implementing
functions it needs from -lsys and -lblockdriver
and the client in order to simulate a working
cache client and backend environment.
. test73: blackbox test of secondary vm cache in isolation
Change-Id: I1287e9753182b8719e634917ad158e3c1e079ceb
. vfs read_only() assumes vnode->v_vmnt is non-NULL, but it can
be NULL sometimes
. e.g. fchmod() on UDS triggered NULL deref; add a check and
add REQ_CHMOD to pfs so unix domain sockets can be fchmod()ded
. add to test56
Change-Id: I83c840f101b647516897cc99fcf472116d762012
m_out is shared between threads as the reply message, and it can happen
results get overwritten by another thread before the reply is sent. This
change
. makes m_out local to the message handling function,
declared on the stack of the caller
. forces callers of reply() to give it a message, or
declare the reply message has no significant fields except
for the return code by calling replycode()
Change-Id: Id06300083a63c72c00f34f86a5c7d96e4bbdf9f6
Variant of utime(2) with struct timespec (with ns precision)
instead of time_t values; also allows for tv_nsec members
the values UTIME_NOW (force update to current time) or
UTIME_OMIT (allow to set either atim or mtim independently.)
Provides a superset of utimes(2), futimes(2), lutimes(2),
and futimens(2).
Provides the same subset of utimensat(2) as does NetBSD 6.
Also import utimens() and lutimeNS() from NetBSD-current.
makes lwip use "unsigned int" instead of "unsigned" since this is
more obvious (i.e. type is not implied).
Change-Id: I852eb80484516e1235241d55be3e15174fa24109
. inet: silence message about exceptions not being implemented
for select(UDP)
This message generates a lot of noise with openntpd. Hide it unless DEBUG is
turned on.
Change-Id: I1527a9ca2583601d6087456062b4f675c80dd711
This also adds the sys_settime() kernel call which allows for the adjusting
of the clock named realtime in the kernel. The existing sys_stime()
function is still needed for a separate job (setting the boottime). The
boottime is set in the readclock driver. The sys_settime() interface is
meant to be flexible and will support both clock_settime() and adjtime()
when adjtime() is implemented later.
settimeofday() was adjusted to use the clock_settime() interface.
One side note discovered during testing: uptime(1) (part of the last(1)),
uses wtmp to determine boottime (not Minix's times(2)). This leads `uptime`
to report odd results when you set the time to a time prior to boottime.
This isn't a new bug introduced by my changes. It's been there for a while.
In order to make it more clear that ticks should be used for timers
and realtime should be used for timestamps / displaying the date/time,
getuptime() was renamed to getticks() and getuptime2() was renamed to
getuptime().
Servers, drivers, libraries, tests, etc that use getuptime()/getuptime2()
have been updated. In instances where a realtime was calculated, the
calculation was changed to use realtime.
System calls clock_getres() and clock_gettime() were added to PM/libc.
. data structure that automatically keeps a set
of pages in reserve, to replace sparepages and
possibly re-used in the future for similar situations,
e.g. if in-filesystem-cache block eviction is
implemented and FS asks for a new block
Change-Id: I149d46c14b9c8e75df16cb94e08907f008c339a6
When you provided a string with junk after the terminating nul to a
UNIX domain socket and used bind(2), the canonical path function would
not properly terminate the new string. This caused VFS to return
ENAMETOOLONG on an otherwise valid path name.
Test case is added to test56.
Change-Id: I883b6be23d9e4ea13c3cee28cbb3726343df037f
Do not hardcode warning and optimisation flags, otherwise the
main options (i.e. DBG, CPPFLAGS) will not work as expected.
You can still provide specific default by using DBG?=<value>.
Doing so leaves the opportunity to override the setting from the
commandline, while the default value from the build system is
then ignored for that particular package.
When crosscompiling, and using build.sh, adding -V DBG=<value> has
this same effect as make DBG=<value>.
Change-Id: Ic610e4d33b945acad64571e1431f1814291e2d84
REQ_PEEK behaves just like REQ_READ except that it does not copy
data anywhere, just obtains the blocks from the FS into the cache.
To be used by the future mmap implementation.
Change-Id: I1b56de304f0a7152b69a72c8962d04258adb44f9
Select(2)ing on UNIX domain sockets was not working properly because
connection state wasn't properly checked/propagated. So selecting for
a read descriptor and closing the write descriptor on the other end
didn't cause select to return. Similarly, read(2) kept blocking while
it should return an error when the other end closed the socket.
Change-Id: I3f5bb52af1a6b03313d508bf915fc838357ba450
. if there is no memory there, it's not writable; this
check bug by the shared memory's writable() method causes
pagefaults not to be handled at all in certain situations,
triggering an assert() in pt_writemap()
. added some assert()s to catch this and similar situations
in the future
Change-Id: Ife89bfab4f9a3aa7bf4e33dfb0b13b89dcd5bb94
The build system distinction between "bootprog" and "service" is
meaningless as boot programs are standard services.
As minix.service.mk simply imports minix.bootprog.mk, reduce confusion
by removing minix.bootprog.mk and placing the rules in minix.service.mk.
Change-Id: I4056b1e574bed59a8c890239b41b1a7c7cad63e8
Remove old versions of system calls and system calls that don't have
a libc api interface anymore (dup, dup2, creat).
VFS still contains support for old system call numbers for the new stat
system calls (i.e., 65, 66, 67) to keep supporting old binaries built for
MINIX 3.2.1 (prior to the release).
Change-Id: I721779b58a50c7eeae20669de24658d55d69b25b
When a service fails to initialize, RS exits the service. When injecting
faults this is undesired behavior. With this patch, we're going to assume
that when starting services with the -b flag (no binary exponential
offset), we don't want to exit the service but simply restart the
initialization.
Change-Id: Ie8b9c89e16fe4df8a89ec30ec678a216b4ec5fd0
libchardriver does not support DEV_REOPEN and will return ERESTART
when you do try it. This made VFS unhappy and concluded erroneously
that the driver was EDEADEPT.
. the total amount of memory in the system didn't include the memory
used by the boot-time modules and some dynamic allocation by the
kernel at boot time (to map in VM). especially apparent on our
ARM board with 'only' 512MB of memory and a huge ramdisk.
. also: *add* the VM loaded module to the freelist after it has
been allocated for & mapped in instead of cutting it *out* of the
freelist! so we get a few more MB free..
Change-Id: If37ac32b21c9d38610830e21421264da4f20bc4f
. allow any number of pde's used for pagedir mapping
. allows >1024 NR_PROCS on x86, >64 on ARM
. allows NR_PROCS to be the same in both cases
. also cleanup: allocating spare PDE's is not necessary
throw that function out
Change-Id: Ibb8f8cf6e7db6a4d6384b6911d1a3f3f5e5d8256
if an exec() fails partway through reading in the sections, the target
process is already gone and a defunct process remains. sanity checking
the binary beforehand helps that.
test10 mutilates binaries and exec()s them on purpose; making an exec()
fail cleanly in such cases seems like acceptable behaviour.
fixes test10 on ARM.
Change-Id: I1ed9bb200ce469d4d349073cadccad5503b2fcb0
The 'polarity' of the RW bit is inversed on ARM, causing one
of the sanity check compensations to fail. ARM now runs basic
stuff with sanity checks passing.
Change-Id: Iee28ab63e430e759f204eeb204b24c301d5ea3c9
. make vm tell kernel virtual locations of mappings
. makes _minix_kerninfo feature work
. fix for mappings being larger than what 1 pde can address
(e.g. devices memory requested on arm)
. still requires a special case for devices memory for the
kernel, which has to switch to virtual addressing
Change-Id: I2e94090aa432346fa4da0edeba72f0b7406c2ad7
Due to the ABI we are using we have to use the earm architecture
moniker for the build system to behave correctly. This involves
then some headers to move around.
There is also a few related Makefile updates as well as minor
source code corrections.
Fix warnings about:
. Unused variables
. format mismatch in printf/scanf format string and arguments
. Missing parenthesis around assignment as truth values
. Clang warnings anout unknown GCC pragma
* Updating common/lib
* Updating lib/csu
* Updating lib/libc
* Updating libexec/ld.elf_so
* Corrected test on __minix in featuretest to actually follow the
meaning of the comment.
* Cleaned up _REENTRANT-related defintions.
* Disabled -D_REENTRANT for libfetch
* Removing some unneeded __NBSD_LIBC defines and tests
Change-Id: Ic1394baef74d11b9f86b312f5ff4bbc3cbf72ce2
This patch uses stricter locking for REQ_LINK, REQ_MKDIR, REQ_MKNOD,
REQ_RENAME, REQ_RMDIR, REQ_SLINK and REQ_UNLINK. For all requests, VFS
locks the directory in which we add or remove an inode with VNODE_WRITE.
I.e., the operations have exclusive access to that directory.
Furthermore, REQ_CHOWN, REQ_CHMOD, and REQ_FTRUNC now lock the vmnt
VMNT_READ; VMNT_WRITE was unnecessary.
Because pipes have no file position. VFS maintained (file) offsets into a
buffer internal to PFS and stored them in vnodes for simplicity, mixing
the responsibilities of filp and vnode objects.
With this patch PFS ignores the position field in REQ_READ and REQ_WRITE
requests making VFS' job a lot simpler.
.sync and fsync used unnecessarily restrictive locking type
.fsync violated locking order by obtaining a vmnt lock after a filp lock
.fsync contained a TOCTOU bug
.new_node violated locking rules (didn't upgrade lock upon file creation)
.do_pipe used unnecessarily restrictive locking type
.always lock pipes exclusively; even a read operation might require to do
a write on a vnode object (update pipe size)
.when opening a file with O_TRUNC, upgrade vnode lock when truncating
.utime used unnecessarily restrictive locking type
.path parsing:
.always acquire VMNT_WRITE or VMNT_EXCL on vmnt and downgrade to
VMNT_READ if that was what was actually requested. This prevents the
following deadlock scenario:
thread A:
lock_vmnt(vmp, TLL_READSER);
lock_vnode(vp, TLL_READSER);
upgrade_vmnt_lock(vmp, TLL_WRITE);
thread B:
lock_vmnt(vmp, TLL_READ);
lock_vnode(vp, TLL_READSER);
thread A will be stuck in upgrade_vmnt_lock and thread B is stuck in
lock_vnode. This happens when, for example, thread A tries create a
new node (open.c:new_node) and thread B tries to do eat_path to
change dir (stadir.c:do_chdir). When the path is being resolved, a
vnode is always locked with VNODE_OPCL (TLL_READSER) and then
downgraded to VNODE_READ if read-only is actually requested. Thread
A locks the vmnt with VMNT_WRITE (TLL_READSER) which still allows
VMNT_READ locks. Thread B can't acquire a lock on the vnode because
thread A has it; Thread A can't upgrade its vmnt lock to VMNT_WRITE
(TLL_WRITE) because thread B has a VMNT_READ lock on it.
By serializing vmnt locks during path parsing, thread B can only
acquire a lock on vmp when thread A has completely finished its
operation.
mount.c: In function 'mount_pfs':
mount.c:395:17: error: variable 'rfp' set but not used [-Werror=unused-but-set-variable]
Change-Id: I2f22590ab4e3a4a1678e9096626ebca53d2660e6
. make vm be able to use malloc() by overriding brk()
and minix_mmap() functions
. phys regions can then be malloc()ed and free()d instead
of being in an avl tree, which is slightly faster
. 'offset' field in phys_region can go too (offset is implied
by position in array) but leads to bigger code changes
new_node makes the assumption that when it does last_dir on a path, a
successive advance would not yield a lock on a vmnt, because last_dir
already locked the vmnt. This is true except when last_dir resolves
to a directory on the parent vmnt of the file that was the result of
advance. For example,
# cd /
# echo foo > home
where home is on a different (sub) partition than / is (default
install). last_dir would resolve to / and advance would resolve to
/home.
With this change, last_dir resolves to the root node on the /home
partition, making the assumption valid again.
. 'anonymous' cache blocks (retrieved with NO_DEV as dev
parameter) were used to implement read()s from holes in
inodes that should return zeroes
. this is an awkward special case in the cache code though
and there's a more direct way to implement the same functionality:
instead of copying from a new, anonymous, zero block, to
the user target buffer, simply sys_safememset the user target
buffer directly. as this was the only use of this feature,
this is all that's needed to simplify the cache code a little.
- CHOOSETRAP define makes impossible to use some common words
like send, receive and notify in any other context, for
instance as members or structures
- any reasonable compiler inlines the static inline functions so
no extra function call overhead is introduced by this change
- this gets us back to the situation before the SYSCALL/SYSENTER
change. It is not perfect, but it used to work and still does.
The tested targets are the followgin ones:
* tools
* distribution
* sets
* release
The remaining NetBSD targets have not been disabled nor tested
*at all*. Try them at your own risk, they may reboot the earth.
For all compliant Makefiles, objects and generated files are put in
MAKEOBJDIR, which means you can now keep objects between two branch
switching. Same for DESTDIR, please refer to build.sh options.
Regarding new or modifications of Makefiles a few things:
* Read share/mk/bsd.README
* If you add a subdirectory, add a Makefile in it, and have it called
by the parent through the SUBDIR variable.
* Do not add arbitrary inclusion which crosses to another branch of
the hierarchy; If you can't do without it, put a comment on why.
If possible, do not use inclusion at all.
* Use as much as possible the infrastructure, it is here to make
life easier, do not fight it.
Sets and package are now used to track files.
We have one set called "minix", composed of one package called "minix-sys"
The VFS/FS protocol does not require the file server to supply a
special device node number in response to a REQ_CREATE request, as
this call creates only regular files. Therefore, VFS should not
erroneously save this piece of information from the REQ_CREATE reply
either.
Upon reboot VFS semi-exits all processes and unmounts the file system.
However, upon unmount, exiting FUSE file systems might need service from
the file system (due to libc). As the FUSE process is halfway the exit
procedure, it doesn't have a valid root directory and working directory.
Trying to do system calls then triggers a sanity check in VFS.
This fix first exits normal processes which should then allow for
unmounting FUSE file systems. Then VFS exits all processes including
File Servers and unmounts the rest of the file system.
There is a deadlock vulnerability when there are no worker threads
available and all of them blocked on a worker thread that's waiting for a
reply from a driver or a reply from an FS that needs to make a back call. In
these cases the deadlock resolver thread should kick in, but didn't in all
cases. Moreover, POSIX calls from File Servers weren't handled properly
anymore, which also could lead to deadlocks.
. also make other out-of-memory conditions less fatal
. add a test case for a user program using all the memory
it can
. remove some diagnostic prints for situations that are normal
when running out of memory so running the test isn't noisy
Add primary cache management feature to libminixfs as mfs and ext2
currently do separately, remove cache code from mfs and ext2, and make
them use the libminixfs interface. This makes all fields of the buf
struct private to libminixfs and FS clients aren't supposed to access
them at all. Only the opaque 'void *data' field (the FS block contents,
used to be called bp) is to be accessed by the FS client.
The main purpose is to implement the interface to the 2ndary vm cache
just once, get rid of some code duplication, and add a little
abstraction to reduce the code inertia of the whole caching business.
Some minor sanity checking and prohibition done by mfs in this code
as removed from the generic primary cache code as a result:
- checking all inodes are not in use when allocating/resizing
the cache
- checking readonly filesystems aren't written to
- checking the superblock isn't written to on mounted filesystems
The minixfslib code relies on fs_blockstats() in the client filesystem to
return some FS usage information.
Introduce explicit abstractions for different mapping types,
handling the instantiation, forking, pagefaults and freeing of
anonymous memory, direct physical mappings, shared memory and
physically contiguous anonymous memory as separate types, making
region.c more generic.
Also some other genericification like merging the 3 munmap cases
into one.
COW and SMAP safemap code is still implicit in region.c.
The check_bsf() macro uses assert(mutex_trylock(&bsf_lock)) and
assumes bsf_lock is locked afterwards. This breaks when compiling
with NOASSERTS="yes". Also: macro to function transition.
. add cpufeature detection of both
. use it for both ipc and kernelcall traps, using a register
for call number
. SYSENTER/SYSCALL does not save any context, therefore userland
has to save it
. to accomodate multiple kernel entry/exit types, the entry
type is recorded in the process struct. hitherto all types
were interrupt (soft int, exception, hard int); now SYSENTER/SYSCALL
is new, with the difference that context is not fully restored
from proc struct when running the process again. this can't be
done as some information is missing.
. complication: cases in which the kernel has to fully change
process context (i.e. sigreturn). in that case the exit type
is changed from SYSENTER/SYSEXIT to soft-int (i.e. iret) and
context is fully restored from the proc struct. this does mean
the PC and SP must change, as the sysenter/sysexit userland code
will otherwise try to restore its own context. this is true in the
sigreturn case.
. override all usage by setting libc_ipc=1
. whenever this function is called, pm will expect
the process to be cleaned up
. so don't abort the process entirely on error
. fixes a later 'forking on top of in-use child' vfs panic
fixes an assert() firing when starting X. thanks to the report by pikpik.
. NO_MEM was 0, which is actually an existing piece
of physical memory. it can't be allocated because it's reserved
for bios data (by the kernel), but it can be mapped in (e.g.
by X), causing sanity check disaster.
. NONCONTIGUOUS is also obsolete as all allocations are single-page
now, i.e. NONCONTIGUOUS is really the default and only mode.
complete munmap implementation; single-page references made
a general munmap() implementation possible to write cleanly.
. memory: let the MIOCRAMSIZE ioctl set the imgrd device
size (but only to 0)
. let the ramdisk command set sizes to 0
. use this command to set /dev/imgrd to 0 after mounting /usr
in /etc/rc, so the boot time ramdisk is freed (about 4MB
currently)
. only reference single pages in process data structures
to simplify page faults, copy-on-write, etc.
. this breaks the secondary cache for objects that are
not one-page-sized; restored in a next commit
By decoupling synchronous drivers from VFS, we are a big step closer to
supporting driver crashes under all circumstances. That is, VFS can't
become stuck on IPC with a synchronous driver (e.g., INET) and can
recover from crashing block drivers during open/close/ioctl or during
communication with an FS.
In order to maintain serialized communication with a synchronous driver,
the communication is wrapped by a mutex on a per driver basis (not major
numbers as there can be multiple majors with identical endpoints). Majors
that share a driver endpoint point to a single mutex object.
In order to support crashes from block drivers, the file reopen tactic
had to be changed; first reopen files associated with the crashed
driver, then send the new driver endpoint to FSes. This solves a
deadlock between the FS and the block driver;
- VFS would send REQ_NEW_DRIVER to an FS, but he FS only receives it
after retrying the current request to the newly started driver.
- The block driver would refuse the retried request until all files
had been reopened.
- VFS would reopen files only after getting a reply from the initial
REQ_NEW_DRIVER.
When a character special driver crashes, all associated files have to
be marked invalid and closed (or reopened if flagged as such). However,
they can only be closed if a thread holds exclusive access to it. To
obtain exclusive access, the worker thread (which handles the new driver
endpoint event from DS) schedules a new job to garbage collect invalid
files. This way, we can signal the worker thread that was talking to the
crashed driver and will release exclusive access to a file associated
with the crashed driver and prevent the garbage collecting worker thread
from dead locking on that file.
Also, when a character special driver crashes, RS will unmap the driver
and remap it upon restart. During unmapping, associated files are marked
invalid instead of waiting for an endpoint up event from DS, as that
event might come later than new read/write/select requests and thus
cause confusion in the freshly started driver.
When locking a filp, the usage counters are no longer checked. The usage
counter can legally go down to zero during filp invalidation while there
are locks pending.
DS events are handled by a separate worker thread instead of the main
thread as reopening files could lead to another crash and a stuck thread.
An additional worker thread is then necessary to unlock it.
Finally, with everything asynchronous a race condition in do_select
surfaced. A select entry was only marked in use after succesfully sending
initial select requests to drivers and having to wait. When multiple
select() calls were handled there was opportunity that these entries
were overwritten. This had as effect that some select results were
ignored (and select() remained blocking instead if returning) or do_select
tried to access filps that were not present (because thrown away by
secondary select()). This bug manifested itself with sendrecs, but was
very hard to reproduce. However, it became awfully easy to trigger with
asynsends only.
Instead of using a loop to find a matching ipc (inter process
communication) system call type, the offset in the call table can be
simply calculated in constant time.
Also, when the interprocess communication server receives an ipc
system call from a process, ipc should tell VM to watch the process
only once. This patch fixes that also.
(Patch and commit message slightly edited by committer.)
. ld.so is linked at 0 but it can relocate itself; we
wish to load ld.so higher though to trap NULL dereferences.
if we know we have to execute ld.so, vfs tells libexec to put it
higher.
. done by RS to reduce/remove dependency on VM for recovery
. RS has the default stack size of 64MB since the nosegments
change, using a huge amount of unused memory to pre-allocate
. ignore these requests until actually required (i.e. being able
to survive VM crashes)
Thanks to pikpik for investigating why RS was so huge.
When VFS runs out of vnodes after closing a vnode in opcl, common_open
will try to unlock a vnode through unlock_filp that has already been
unlocked in clone_opcl. By first obtaining and locking a new vnode this
situation is prevented; if there are no free vnodes, common_open will
unlock a still locked vnode.
.enable all compile time warnings and make them errors
.refactor functions with unused parameters
.fix null pointer dereference before checking for null
.proper variable initialization
.use safe string copy functions
.fix massive memory corruption bug in fs_getdents
