Primary purpose of change: to support the mmap implementation, VM must
know both (a) about some block metadata for FS cache blocks, i.e.
inode numbers and inode offsets where applicable; and (b) know about
*all* cache blocks, i.e. also of the FS primary caches and not just
the blocks that spill into the secondary one. This changes the
interface and VM data structures.
This change is only for the interface (libminixfs) and VM data
structures; the filesystem code is unmodified, so although the
secondary cache will be used as normal, blocks will not be annotated
with inode information until the FS is modified to provide this
information. Until it is modified, mmap of files will fail gracefully
on such filesystems.
This is indicated to VFS/VM by returning ENOSYS for REQ_PEEK.
Change-Id: I1d2df6c485e6c5e89eb28d9055076cc02629594e
Memory types in VM are described by methods. Each mapped region has
a type, and all pages instantiated get that type on creation.
Individual page types has to be able to change though. This commit
changes the code to use the memory types of the individual pages,
where appropriate, instead of just the higher-level region, in case
it has changed. This is needed to e.g. support future copy-on-write
MAP_PRIVATE mmap modes.
Change-Id: I5523db14ac036ec774a54392fb67f9acb8725731
Some (backwards-compatible) changes in mmap() call message fields
that allow for a 64-bit offset. minix_mmap() takes an off_t and
minix_mmap64() takes a u64_t. Some mmap() work in VM goes into a
separate function, using the new fields, so that that can be re-used
when files are to be mapped (future commit).
Change-Id: Ifb77a90b593dd3c33cf81b396068e4da1ec5fb1c
This commit removes the secondary cache code implementation from
VM and its usage from libminixfs. It is to be replaced by a new
implementation.
Change-Id: I8fa3af06330e7604c7e0dd4cbe39d3ce353a05b1
. test70: regression test for m_out vfs race condition
The following tests use testcache.c to generate test i/o
patterns, generate random write data and verify the reads.
. test71: blackbox full-stack test of FS operation, testing
using the regular VFS interface crazy i/o patterns
with various working set sizes, triggering only
primary cache, also secondary cache, and finally
disk i/o and verifying contents all the time
. test72: unit test of libminixfs, implementing
functions it needs from -lsys and -lblockdriver
and the client in order to simulate a working
cache client and backend environment.
. test73: blackbox test of secondary vm cache in isolation
Change-Id: I1287e9753182b8719e634917ad158e3c1e079ceb
In order to make it more clear that ticks should be used for timers
and realtime should be used for timestamps / displaying the date/time,
getuptime() was renamed to getticks() and getuptime2() was renamed to
getuptime().
Servers, drivers, libraries, tests, etc that use getuptime()/getuptime2()
have been updated. In instances where a realtime was calculated, the
calculation was changed to use realtime.
System calls clock_getres() and clock_gettime() were added to PM/libc.
. data structure that automatically keeps a set
of pages in reserve, to replace sparepages and
possibly re-used in the future for similar situations,
e.g. if in-filesystem-cache block eviction is
implemented and FS asks for a new block
Change-Id: I149d46c14b9c8e75df16cb94e08907f008c339a6
. if there is no memory there, it's not writable; this
check bug by the shared memory's writable() method causes
pagefaults not to be handled at all in certain situations,
triggering an assert() in pt_writemap()
. added some assert()s to catch this and similar situations
in the future
Change-Id: Ife89bfab4f9a3aa7bf4e33dfb0b13b89dcd5bb94
The build system distinction between "bootprog" and "service" is
meaningless as boot programs are standard services.
As minix.service.mk simply imports minix.bootprog.mk, reduce confusion
by removing minix.bootprog.mk and placing the rules in minix.service.mk.
Change-Id: I4056b1e574bed59a8c890239b41b1a7c7cad63e8
. the total amount of memory in the system didn't include the memory
used by the boot-time modules and some dynamic allocation by the
kernel at boot time (to map in VM). especially apparent on our
ARM board with 'only' 512MB of memory and a huge ramdisk.
. also: *add* the VM loaded module to the freelist after it has
been allocated for & mapped in instead of cutting it *out* of the
freelist! so we get a few more MB free..
Change-Id: If37ac32b21c9d38610830e21421264da4f20bc4f
. allow any number of pde's used for pagedir mapping
. allows >1024 NR_PROCS on x86, >64 on ARM
. allows NR_PROCS to be the same in both cases
. also cleanup: allocating spare PDE's is not necessary
throw that function out
Change-Id: Ibb8f8cf6e7db6a4d6384b6911d1a3f3f5e5d8256
if an exec() fails partway through reading in the sections, the target
process is already gone and a defunct process remains. sanity checking
the binary beforehand helps that.
test10 mutilates binaries and exec()s them on purpose; making an exec()
fail cleanly in such cases seems like acceptable behaviour.
fixes test10 on ARM.
Change-Id: I1ed9bb200ce469d4d349073cadccad5503b2fcb0
The 'polarity' of the RW bit is inversed on ARM, causing one
of the sanity check compensations to fail. ARM now runs basic
stuff with sanity checks passing.
Change-Id: Iee28ab63e430e759f204eeb204b24c301d5ea3c9
. make vm tell kernel virtual locations of mappings
. makes _minix_kerninfo feature work
. fix for mappings being larger than what 1 pde can address
(e.g. devices memory requested on arm)
. still requires a special case for devices memory for the
kernel, which has to switch to virtual addressing
Change-Id: I2e94090aa432346fa4da0edeba72f0b7406c2ad7
Due to the ABI we are using we have to use the earm architecture
moniker for the build system to behave correctly. This involves
then some headers to move around.
There is also a few related Makefile updates as well as minor
source code corrections.
Fix warnings about:
. Unused variables
. format mismatch in printf/scanf format string and arguments
. Missing parenthesis around assignment as truth values
. Clang warnings anout unknown GCC pragma
. make vm be able to use malloc() by overriding brk()
and minix_mmap() functions
. phys regions can then be malloc()ed and free()d instead
of being in an avl tree, which is slightly faster
. 'offset' field in phys_region can go too (offset is implied
by position in array) but leads to bigger code changes
. also make other out-of-memory conditions less fatal
. add a test case for a user program using all the memory
it can
. remove some diagnostic prints for situations that are normal
when running out of memory so running the test isn't noisy
Introduce explicit abstractions for different mapping types,
handling the instantiation, forking, pagefaults and freeing of
anonymous memory, direct physical mappings, shared memory and
physically contiguous anonymous memory as separate types, making
region.c more generic.
Also some other genericification like merging the 3 munmap cases
into one.
COW and SMAP safemap code is still implicit in region.c.
. add cpufeature detection of both
. use it for both ipc and kernelcall traps, using a register
for call number
. SYSENTER/SYSCALL does not save any context, therefore userland
has to save it
. to accomodate multiple kernel entry/exit types, the entry
type is recorded in the process struct. hitherto all types
were interrupt (soft int, exception, hard int); now SYSENTER/SYSCALL
is new, with the difference that context is not fully restored
from proc struct when running the process again. this can't be
done as some information is missing.
. complication: cases in which the kernel has to fully change
process context (i.e. sigreturn). in that case the exit type
is changed from SYSENTER/SYSEXIT to soft-int (i.e. iret) and
context is fully restored from the proc struct. this does mean
the PC and SP must change, as the sysenter/sysexit userland code
will otherwise try to restore its own context. this is true in the
sigreturn case.
. override all usage by setting libc_ipc=1
fixes an assert() firing when starting X. thanks to the report by pikpik.
. NO_MEM was 0, which is actually an existing piece
of physical memory. it can't be allocated because it's reserved
for bios data (by the kernel), but it can be mapped in (e.g.
by X), causing sanity check disaster.
. NONCONTIGUOUS is also obsolete as all allocations are single-page
now, i.e. NONCONTIGUOUS is really the default and only mode.
complete munmap implementation; single-page references made
a general munmap() implementation possible to write cleanly.
. memory: let the MIOCRAMSIZE ioctl set the imgrd device
size (but only to 0)
. let the ramdisk command set sizes to 0
. use this command to set /dev/imgrd to 0 after mounting /usr
in /etc/rc, so the boot time ramdisk is freed (about 4MB
currently)
. only reference single pages in process data structures
to simplify page faults, copy-on-write, etc.
. this breaks the secondary cache for objects that are
not one-page-sized; restored in a next commit
. done by RS to reduce/remove dependency on VM for recovery
. RS has the default stack size of 64MB since the nosegments
change, using a huge amount of unused memory to pre-allocate
. ignore these requests until actually required (i.e. being able
to survive VM crashes)
Thanks to pikpik for investigating why RS was so huge.
. map all objects named usermapped_*.o with globally visible
pages; usermapped_glo_*.o with the VM 'global' bit on, i.e.
permanently in tlb (very scarce resource!)
. added kinfo, machine, kmessages and loadinfo for a start
. modified log, tty to make use of the shared messages struct
. some strncpy/strcpy to strlcpy conversions
. new <minix/param.h> to avoid including other minix headers
that have colliding definitions with library and commands code,
causing parse warnings
. removed some dead code / assignments
This commit removes all traces of Minix segments (the text/data/stack
memory map abstraction in the kernel) and significance of Intel segments
(hardware segments like CS, DS that add offsets to all addressing before
page table translation). This ultimately simplifies the memory layout
and addressing and makes the same layout possible on non-Intel
architectures.
There are only two types of addresses in the world now: virtual
and physical; even the kernel and processes have the same virtual
address space. Kernel and user processes can be distinguished at a
glance as processes won't use 0xF0000000 and above.
No static pre-allocated memory sizes exist any more.
Changes to booting:
. The pre_init.c leaves the kernel and modules exactly as
they were left by the bootloader in physical memory
. The kernel starts running using physical addressing,
loaded at a fixed location given in its linker script by the
bootloader. All code and data in this phase are linked to
this fixed low location.
. It makes a bootstrap pagetable to map itself to a
fixed high location (also in linker script) and jumps to
the high address. All code and data then use this high addressing.
. All code/data symbols linked at the low addresses is prefixed by
an objcopy step with __k_unpaged_*, so that that code cannot
reference highly-linked symbols (which aren't valid yet) or vice
versa (symbols that aren't valid any more).
. The two addressing modes are separated in the linker script by
collecting the unpaged_*.o objects and linking them with low
addresses, and linking the rest high. Some objects are linked
twice, once low and once high.
. The bootstrap phase passes a lot of information (e.g. free memory
list, physical location of the modules, etc.) using the kinfo
struct.
. After this bootstrap the low-linked part is freed.
. The kernel maps in VM into the bootstrap page table so that VM can
begin executing. Its first job is to make page tables for all other
boot processes. So VM runs before RS, and RS gets a fully dynamic,
VM-managed address space. VM gets its privilege info from RS as usual
but that happens after RS starts running.
. Both the kernel loading VM and VM organizing boot processes happen
using the libexec logic. This removes the last reason for VM to
still know much about exec() and vm/exec.c is gone.
Further Implementation:
. All segments are based at 0 and have a 4 GB limit.
. The kernel is mapped in at the top of the virtual address
space so as not to constrain the user processes.
. Processes do not use segments from the LDT at all; there are
no segments in the LDT any more, so no LLDT is needed.
. The Minix segments T/D/S are gone and so none of the
user-space or in-kernel copy functions use them. The copy
functions use a process endpoint of NONE to realize it's
a physical address, virtual otherwise.
. The umap call only makes sense to translate a virtual address
to a physical address now.
. Segments-related calls like newmap and alloc_segments are gone.
. All segments-related translation in VM is gone (vir2map etc).
. Initialization in VM is simpler as no moving around is necessary.
. VM and all other boot processes can be linked wherever they wish
and will be mapped in at the right location by the kernel and VM
respectively.
Other changes:
. The multiboot code is less special: it does not use mb_print
for its diagnostics any more but uses printf() as normal, saving
the output into the diagnostics buffer, only printing to the
screen using the direct print functions if a panic() occurs.
. The multiboot code uses the flexible 'free memory map list'
style to receive the list of free memory if available.
. The kernel determines the memory layout of the processes to
a degree: it tells VM where the kernel starts and ends and
where the kernel wants the top of the process to be. VM then
uses this entire range, i.e. the stack is right at the top,
and mmap()ped bits of memory are placed below that downwards,
and the break grows upwards.
Other Consequences:
. Every process gets its own page table as address spaces
can't be separated any more by segments.
. As all segments are 0-based, there is no distinction between
virtual and linear addresses, nor between userspace and
kernel addresses.
. Less work is done when context switching, leading to a net
performance increase. (8% faster on my machine for 'make servers'.)
. The layout and configuration of the GDT makes sysenter and syscall
possible.
. all invocations were S or D, so can safely be dropped
to prepare for the segmentless world
. still assign D to the SCP_SEG field in the message
to make previous kernels usable
. new mode for sys_memset: include process so memset can be
done in physical or virtual address space.
. add a mode to mmap() that lets a process allocate uninitialized
memory.
. this allows an exec()er (RS, VFS, etc.) to request uninitialized
memory from VM and selectively clear the ranges that don't come
from a file, leaving no uninitialized memory left for the process
to see.
. use callbacks for clearing the process, clearing memory in the
process, and copying into the process; so that the libexec code
can be used from rs, vfs, and in the future, kernel (to load vm)
and vm (to load boot-time processes)
. make exec() callers (i.e. vfs and rs) determine the
memory layout by explicitly reserving regions using
mmap() calls on behalf of the exec()ing process,
i.e. handling all of the exec logic, thereby eliminating
all special exec() knowledge from VM.
. the new procedure is: clear the exec()ing process
first, then call third-party mmap()s to reserve memory, then
copy the executable file section contents in, all using callbacks
tailored to the caller's way of starting an executable
. i.e. no more explicit EXEC_NEWMEM-style calls in PM or VM
as with rigid 2-section arguments
. this naturally allows generalizing exec() by simply loading
all ELF sections
. drop/merge of lots of duplicate exec() code into libexec
. not copying the code sections to vfs and into the executable
again is a measurable performance improvement (about 3.3% faster
for 'make' in src/servers/)
these two functions will be used to support all exec() functionality
going into a single library shared by RS and VFS and exec() knowledge
leaving VM.
. third-party mmap: allow certain processes (VFS, RS) to
do mmap() on behalf of another process
. PROCCTL: used to free and clear a process' address space
use the user-supplied point to lookup which region to perform brk() on,
and if it's a reasonable one, do it, no matter what vm's notion of the
heap region is.
Previously, the mmap address (if given) was merely used as a lower
bound, and then possibly overriden with a hint. Now, the mapping is
first tried at the exact given address. If that fails, the start of
the mmap range is used as lower bound (which is then still overridden
by the hint for efficiency).
This allows two pages to be mapped in at predefined addresses, where
the second address is lower than the first. That was not possible.
remove some old minix-userland-specific stuff
. /etc/ttytab as a file, and minix-compat function (fftyslot()),
replaced by /etc/ttys and new libc functions
. also remove minix-specific nlist(), cuserid(), fttyslot(), v8 regex
functions and <compat/regex.h>
. and remaining minix-only utilities that use them
. also unused <compat/pwd.h> and <compat/syslog.h> and
redundant <sys/sigcontext.h>
. only use for single-page invalidations initially
. shows tiny but statistically significant performance
improvement; will be more helpful in certain VM debug
modes
. ipc wants to know about processes that get
signals, so that it can break blocking ipc operations
. doing it for every single signal is wasteful
and causes the annoying 'no slot for signals' message
. this fix tells vm on a per-process basis it (ipc)
wants to be notified, i.e. only when it does any ipc calls
. move ipc config to separate config file while we're at it
. MAP_SHARED was used to implement sysv shared memory
. used to signal shareable memory region to VM
. assumptions about this situation break when processes
use MAP_SHARED for its normal, standardised meaning
* VFS and installed MFSes must be in sync before and after this change *
Use struct stat from NetBSD. It requires adding new STAT, FSTAT and LSTAT
syscalls. Libc modification is both backward and forward compatible.
Also new struct stat uses modern field sizes to avoid ABI
incompatibility, when we update uid_t, gid_t and company.
Exceptions are ino_t and off_t in old libc (though paddings added).
3 sets of libraries are built now:
. ack: all libraries that ack can compile (/usr/lib/i386/)
. clang+elf: all libraries with minix headers (/usr/lib/)
. clang+elf: all libraries with netbsd headers (/usr/netbsd/)
Once everything can be compiled with netbsd libraries and headers, the
/usr/netbsd hierarchy will be obsolete and its libraries compiled with
netbsd headers will be installed in /usr/lib, and its headers
in /usr/include. (i.e. minix libc and current minix headers set
will be gone.)
To use the NetBSD libc system (libraries + headers) before
it is the default libc, see:
http://wiki.minix3.org/en/DevelopersGuide/UsingNetBSDCode
This wiki page also documents the maintenance of the patch
files of minix-specific changes to imported NetBSD code.
Changes in this commit:
. libsys: Add NBSD compilation and create a safe NBSD-based libc.
. Port rest of libraries (except libddekit) to new header system.
. Enable compilation of libddekit with new headers.
. Enable kernel compilation with new headers.
. Enable drivers compilation with new headers.
. Port legacy commands to new headers and libc.
. Port servers to new headers.
. Add <sys/sigcontext.h> in compat library.
. Remove dependency file in tree.
. Enable compilation of common/lib/libc/atomic in libsys
. Do not generate RCSID strings in libc.
. Temporarily disable zoneinfo as they are incompatible with NetBSD format
. obj-nbsd for .gitignore
. Procfs: use only integer arithmetic. (Antoine Leca)
. Increase ramdisk size to create NBSD-based images.
. Remove INCSYMLINKS handling hack.
. Add nbsd_include/sys/exec_elf.h
. Enable ELF compilation with NBSD libc.
. Add 'make nbsdsrc' in tools to download reference NetBSD sources.
. Automate minix-port.patch creation.
. Avoid using fstavfs() as it is *extremely* slow and unneeded.
. Set err() as PRIVATE to avoid name clash with libc.
. [NBSD] servers/vm: remove compilation warnings.
. u32 is not a long in NBSD headers.
. UPDATING info on netbsd hierarchy
. commands fixes for netbsd libc
