The memory-mapped files implementation (mmap() etc.) is implemented with
the help of the filesystems using the in-VM FS cache. Filesystems tell it
about all cached blocks and their metadata. Metadata is: device offset and,
if any (and known), inode number and in-inode offset. VM can then map in
requested memory-mapped file blocks, and request them if necessary.
A limitation of this system is that filesystem block sizes that are not
a multiple of the VM system (and VM hardware) page size are not possible;
we can't map blocks in partially. (We can copy, but then the benefits of
mapping and sharing the physical pages is gone.) So until before this
commit various pieces of caching code assumed page size multiple
blocksizes. This isn't strictly necessary as long as mmap() needn't be
supported on that FS.
This change allows the in-FS cache code (libminixfs) to allocate any-sized
blocks, and will not interact with the VM cache for non-pagesize-multiple
blocks. In that case it will also signal requestors, by failing 'peek'
requests, that mmap() should not be supported on this FS. VM and VFS
will then gracefully fail all file-mapping mmap() calls, and exec() will
fall back to copying executable blocks instead of mmap()ping executables.
As a result, 3 diagnostics that signal file-mapped mmap()s failing
(hitherto an unusual occurence) are disabled, as ld.so does file-mapped
mmap()s to map in objects it needs. On FSes not supporting it this situation
is legitimate and shouldn't cause so much noise. ld.so will revert to its own
minix-specific allocate+copy style of starting executables if mmap()s fail.
Change-Id: Iecb1c8090f5e0be28da8f5181bb35084eb18f67b
* Add _SC_LINE_MAX to unistd.h (value of 15 from NetBSD).
* Add _SC_LINE_MAX case to sysconf(3) in libc.
* usr.bin/nl itself required no Minix specific changes.
Change-Id: I10f056ccdf4f212beb1272f735f41303e5036c05
. initial workaround for assert() firing on iovec
size on ARM. likely due to alloc_contig() allocating
unusually mapped memory in STATICINIT.
. for the same reason use the regular cache i/o functions
to read the superblock in mfs - avoid the alloc_contig()
that STATICINIT does.
Change-Id: I3d8dc635b1cf2666e55b0393feae74cc25b8fed4
When compiling with -O3 gcc seems to be a lot stricter on its checks for
index out of bounds error.
Change-Id: I5a63e1db6aa2375223ff13c6352643a9804fa93d
Many i2c device drivers used similar code to access registers on
the ICs they drive. This commit implements that functionality in
libi2cdriver and updates the drivers to use the library instead of
their own register access functions. The net result is 375+ fewer
lines of code and less work for people developing new drivers.
The two exceptions were cat24c256 and parts of tda19988. They access
the bus in uncommon ways. It doesn't make sense at this time to
move their read/write functions into libi2cdriver.
Change-Id: Id8280b71af33b710a49944d7f20a7262be9f5988
* port libgpio to am335x.
* increase NR_IRQ to 16 to avoid too many IRQs requested error.
* move gpio binary to /sbin (that's where mount looks for drivers).
* fix bug that re-enabled banks that were disabled due to bad revision.
* fix padconf masks for dm37xx in gpio driver (should be 16-bit).
* fix a few typos in README.txt and comments.
Change-Id: I32374415a97135174c5224f76374a870909fa90c
On the AM335X, writes to the padconf registers must be done in privileged
mode. To allow userspace drivers to dynamically change the padconf at
runtime, a kernel call has been added.
Change-Id: I4b25d2879399b1785a360912faa0e90b5c258533
Apparently, qemu's sb16 doesn't reset the interrupt register
of the mixer and the driver ends up in an endless loop when
playing a "multi-fragment" sample.
Change-Id: I47633b99a0d79ba201aec785da69fb3950e026bf
Righ now, the DMA controller will access some random memory below 16 MB,
as the physical address returned by alloc_contig() is simply truncated
when given to the DMA controller.
Yes, it's bad...
. this is OK (although it wastes some memory) as
long as the VM interface isn't used, which has its
own checks in libsys
Change-Id: I28decd367b2cd5c01482bdc71615c65ab61c9a71
for clang, fix warnings in drivers/, lib/, servers/, sys/, common/.
by turning off fatal warnings (takes effect if the default is on),
fixing warnings or reducing the warning level.
Change-Id: Ia1b4bc877c879ba783158081b59aa6ebb021a50f
Previously, all incoming messages would be blocked before a DL_CONF
message arrives from the TCP/IP stack. This however makes it impossible
for a driver to process interrupts before the DL_CONF initialization.
This patch blocks only datalink messages before the initial DL_CONF,
and lets through all other messages to the driver.
Change-Id: I89988958c0bff9bb38e0379b66f6142491a67b61
The VM server now manages its call masks such that all user processes
share the same call mask. As a result, an update for the call mask of
any user process will apply to all user processes. This is similar to
the privilege infrastructure employed by the kernel, and may serve as
a template for similar fine-grained restrictions in other servers.
Concretely, this patch fixes the problem of "service edit init" not
applying the given VM call mask to user processes started from RC
scripts during system startup.
In addition, this patch makes RS set a proper VM call mask for each
recovery script it spawns.
Change-Id: I520a30d85a0d3f3502d2b158293a2258825358cf
To do so, a few dependencies have been imported:
* external/bsd/lutok
* external/mit/lua
* external/public-domain/sqlite
* external/public-domain/xz
The Kyua framework is the new generation of ATF (Automated Test
Framework), it is composed of:
* external/bsd/atf
* external/bsd/kyua-atf-compat
* external/bsd/kyua-cli
* external/bsd/kyua-tester
* tests
Kyua/ATF being written in C++, it depends on libstdc++ which is
provided by GCC. As this is not part of the sources, Kyua is only
compiled when the native GCC utils are installed.
To install Kyua do the following:
* In a cross-build enviromnent, add the following to the build.sh
commandline: -V MKBINUTILS=yes -V MKGCCCMDS=yes
WARNING:
At this point the import is still experimental, and not supported
on native builds (a.k.a make build).
Change-Id: I26aee23c5bbd2d64adcb7c1beb98fe0d479d7ada
-By adding MKGCC=yes and MKGCCCMDS=yes on the make commandline
it is now possible to compile and install GCC on the system.
Before doing this, if you are not using the build.sh script,
you will need to call the fetch scripts in order to retrieve
the sources of GCC and its dependencies.
-Reduce difference with NetBSD share/mk
Move Minix-specific parameters from bsd.gcc.mk to bsd.own.mk,
which is anyway patched, so that bsd.gcc.mk is now aligned
on the NetBSD version.
-Clean libraries dependencies, compiles stdc++ only if gcc is
also compiled (it is part of the gcc sources)
-Correct minix.h header sequence, cleanup spec headers.
-Fix cross-compilation from a 32bit host targeting MINIX/arm
Change-Id: I1b234af18eed4ab5675188244e931b2a2b7bd943
Implement getrusage.
These fields of struct rusage are not supported and always set to zero at this time
long ru_nswap; /* swaps */
long ru_inblock; /* block input operations */
long ru_oublock; /* block output operations */
long ru_msgsnd; /* messages sent */
long ru_msgrcv; /* messages received */
long ru_nvcsw; /* voluntary context switches */
long ru_nivcsw; /* involuntary context switches */
test75.c is the unit test for this new function
Change-Id: I3f1eb69de1fce90d087d76773b09021fc6106539
kernel:
. modules can be as big as the space (8MB) between them
instead of 4MB; memory is slightly bigger with DBG=-g
arm ucontext:
. r4 is clobbered by the restore function, as it's
used as a scratch register, causing problems for the
DBG=-g build
. r1-r3 are safe for scratch registers, as they are
caller-save, so use r3 instead; and don't bother
restoring r1-r3, but preserve r4
vfs:
. improve TLL pointer sanity check a bit
Change-Id: I0e3cfc367fdc14477e40d04b5e044f288ca4cc7d
Clock configuration will be needed by other/future subsystems such as i2c.
Extract the functionality from libgpio and put it into it's own library.
Change-Id: I6f6de0b3fb4d305ddfeac74123b78c983d1318dd
. by making the address and frequency of the
free running clock kinfo members, set at runtime
in the kernel, instead of compile time constants
in libsys
Change-Id: I4a8387302d4d3ffd47d2448525725683a74c9a4f
- setlogin() not available
- softfloat needed in -lminc for arm
- libminc: we have to set LIBCDIR for the included files,
so use that instead of LIBCSRCDIR
Change-Id: I7f92621ebbca9ce08dc377b3fa61dee089071757
. libc: add vfs_mmap, a way for vfs to initiate mmap()s.
This is a good special case to have as vfs is a slightly
different client from regular user processes. It doesn't do it
for itself, and has the dev & inode info already so the callback
to VFS for the lookup isn't necessary. So it has different info
to have to give to VM.
. libc: also add minix_mmap64() that accepts a 64-bit offset, even
though our off_t is still 32 bit now.
. On exec() time, try to mmap() in the executable if available.
(It is not yet available in this commit.)
. To support mmap(), add do_vm_call that allows VM to lookup
(to ino+dev), do i/o from and close FD's on behalf of other
processes.
Change-Id: I831551e45a6781c74313c450eb9c967a68505932
Omap timers remove hardcoded base address and add some initial
support for the beaglebone's timers. Frclock_util will need
refactoring to remain independent of the ARM flavour.
Change-Id: I2b5d04e930364262c81b5686de634c0a51796b23
kernel: stop gathering timestamps once the bin is full per interrupt
random: once seeded, retrieve new entropy at a lower rate
Change-Id: I4ce6081d39274728d82c6889686d1650cfd5fc2e
. add receive hooks in the kernel to print asynchronously
delivered messages
. do not rely on MF_REPLY_PEND to decide between calls and errors,
as that isn't reliable for asynchronous messages; try both instead
. add _sendcall() that extract-mfield.sh can then reliably recognize
the fields for messages that are sent with just send()
. add DEBUG_DUMPIPC_NAMES to restrict printed messages to
from/to given process names
Change-Id: Ia65eb02a69a2b58e73bf9f009987be06dda774a3
libminixfs may now be informed of changes to the block usage on the
filesystem. if the net change becomes big enough, libminixfs may
resize the cache based on the new usage.
. update the 2 FSes to provide this information to libminixfs
Change-Id: I158815a11da801fd5572a8de89c9e6c039b82650
This commit introduces a new request type called REQ_BPEEK. It
requests minor device blocks from the FS. Analogously to REQ_PEEK,
it requests the filesystem to get the requested blocks into its
cache, without actually copying the result anywhere.
Change-Id: If1d06645b0e17553a64b3167091e9d12efeb3d6f
In libexec, split the memory allocation method into cleared and
non-cleared. Cleared gives zeroed memory, non-cleared gives 'junk'
memory (that will be overwritten anyway, and so needn't be cleared)
that is faster to get.
Also introduce the 'memmap' method that can be used, if available,
to map code and data from executables into a process using the
third-party mmap() mode.
Change-Id: I26694fd3c21deb8b97e01ed675dfc14719b0672b
Primary purpose of change: to support the mmap implementation, VM must
know both (a) about some block metadata for FS cache blocks, i.e.
inode numbers and inode offsets where applicable; and (b) know about
*all* cache blocks, i.e. also of the FS primary caches and not just
the blocks that spill into the secondary one. This changes the
interface and VM data structures.
This change is only for the interface (libminixfs) and VM data
structures; the filesystem code is unmodified, so although the
secondary cache will be used as normal, blocks will not be annotated
with inode information until the FS is modified to provide this
information. Until it is modified, mmap of files will fail gracefully
on such filesystems.
This is indicated to VFS/VM by returning ENOSYS for REQ_PEEK.
Change-Id: I1d2df6c485e6c5e89eb28d9055076cc02629594e
Some (backwards-compatible) changes in mmap() call message fields
that allow for a 64-bit offset. minix_mmap() takes an off_t and
minix_mmap64() takes a u64_t. Some mmap() work in VM goes into a
separate function, using the new fields, so that that can be re-used
when files are to be mapped (future commit).
Change-Id: Ifb77a90b593dd3c33cf81b396068e4da1ec5fb1c
This commit removes the secondary cache code implementation from
VM and its usage from libminixfs. It is to be replaced by a new
implementation.
Change-Id: I8fa3af06330e7604c7e0dd4cbe39d3ce353a05b1
. test70: regression test for m_out vfs race condition
The following tests use testcache.c to generate test i/o
patterns, generate random write data and verify the reads.
. test71: blackbox full-stack test of FS operation, testing
using the regular VFS interface crazy i/o patterns
with various working set sizes, triggering only
primary cache, also secondary cache, and finally
disk i/o and verifying contents all the time
. test72: unit test of libminixfs, implementing
functions it needs from -lsys and -lblockdriver
and the client in order to simulate a working
cache client and backend environment.
. test73: blackbox test of secondary vm cache in isolation
Change-Id: I1287e9753182b8719e634917ad158e3c1e079ceb
Variant of utime(2) with struct timespec (with ns precision)
instead of time_t values; also allows for tv_nsec members
the values UTIME_NOW (force update to current time) or
UTIME_OMIT (allow to set either atim or mtim independently.)
Provides a superset of utimes(2), futimes(2), lutimes(2),
and futimens(2).
Provides the same subset of utimensat(2) as does NetBSD 6.
Also import utimens() and lutimeNS() from NetBSD-current.
This also adds the sys_settime() kernel call which allows for the adjusting
of the clock named realtime in the kernel. The existing sys_stime()
function is still needed for a separate job (setting the boottime). The
boottime is set in the readclock driver. The sys_settime() interface is
meant to be flexible and will support both clock_settime() and adjtime()
when adjtime() is implemented later.
settimeofday() was adjusted to use the clock_settime() interface.
One side note discovered during testing: uptime(1) (part of the last(1)),
uses wtmp to determine boottime (not Minix's times(2)). This leads `uptime`
to report odd results when you set the time to a time prior to boottime.
This isn't a new bug introduced by my changes. It's been there for a while.
In order to make it more clear that ticks should be used for timers
and realtime should be used for timestamps / displaying the date/time,
getuptime() was renamed to getticks() and getuptime2() was renamed to
getuptime().
Servers, drivers, libraries, tests, etc that use getuptime()/getuptime2()
have been updated. In instances where a realtime was calculated, the
calculation was changed to use realtime.
System calls clock_getres() and clock_gettime() were added to PM/libc.
Do not hardcode warning and optimisation flags, otherwise the
main options (i.e. DBG, CPPFLAGS) will not work as expected.
You can still provide specific default by using DBG?=<value>.
Doing so leaves the opportunity to override the setting from the
commandline, while the default value from the build system is
then ignored for that particular package.
When crosscompiling, and using build.sh, adding -V DBG=<value> has
this same effect as make DBG=<value>.
Change-Id: Ic610e4d33b945acad64571e1431f1814291e2d84
REQ_PEEK behaves just like REQ_READ except that it does not copy
data anywhere, just obtains the blocks from the FS into the cache.
To be used by the future mmap implementation.
Change-Id: I1b56de304f0a7152b69a72c8962d04258adb44f9
Remove old versions of system calls and system calls that don't have
a libc api interface anymore (dup, dup2, creat).
VFS still contains support for old system call numbers for the new stat
system calls (i.e., 65, 66, 67) to keep supporting old binaries built for
MINIX 3.2.1 (prior to the release).
Change-Id: I721779b58a50c7eeae20669de24658d55d69b25b
Make the frclock functions similar to the tsc utility functions. This
way, we can call frclock functions from the framebuffer driver which
will use frclock on ARM and tsc on X86.
Also, frclock_64_to_micros computed seconds, not microseconds
Change-Id: I6718ae0fb7db050794f6f032205923e1a32dc1ac
. ldivmod, uldivmod were passing the modulo argument pointer
in R4, which is bogus, as qdivrem expects it on the stack as
per the EABI, causing essentially 'random' memory to be trampled
by qdivrem. fix by pushing R4 before the call.
. also add these functions to -minc so -lminc clients can be
linked without -lgcc
Change-Id: I90b0b28b51a188c93da5de6afb108224749ea794
* Generalize GPIO handling.
* Add libs to configure gpio's clocks and pads
* Add Interrupt handling.
* Introduce mmio.h and log.h
Change-Id: I928e4c807d15031de2eede4b3ecff62df795f8ac
if an exec() fails partway through reading in the sections, the target
process is already gone and a defunct process remains. sanity checking
the binary beforehand helps that.
test10 mutilates binaries and exec()s them on purpose; making an exec()
fail cleanly in such cases seems like acceptable behaviour.
fixes test10 on ARM.
Change-Id: I1ed9bb200ce469d4d349073cadccad5503b2fcb0
. kernel: signal handler args for ARM
. kernel: sanity check return address (LSB indicates thumb mode)
. libc: properly retrieve signal mask for ARM
together fix test37 on ARM.
Change-Id: I4e00f754c50104ed85c7fdf8ec5ad54568f20a81
Also did some cleanup in ash sources, to make minix modifications
more obvious, as well as some simplifications (by removing code which
is never compiled)
Removed EDITLINE support, use libedit, which does the termcap/terminfo
handling.
Change-Id: I19f7f425ed6a61298844631f9d7f3173cf7f30c0
The Cycle CouNTer on ARM cannot be used reliably as it wraps around
rather quickly and can be altered by user space (on Minix). Furthermore,
it's buggy when wrapping and is not implemented at all on the Linaro
Beagleboard emulator.
This patch programs GPTIMER10 as a free running clock at 1.625 MHz (it
doesn't generate interrupts). It's memory mapped into every process,
which enables libsys to provide micro_delay().
Change-Id: Iba004c6c62976762fe154ea390d69e518eec1531
A few kernel and calling convention adjustments to make sigsend and
sigreturn work for arm.
. provide a arch_proc_setcontext for earm in kernel
. set LR in context of signal handler to provide a proper
return address (to __sigreturn)
. change __sigreturn to retrieve the sigcontext pointer
from the sigframe struct and pass it to _sigreturn() in r0
Change-Id: Icd135a70595382c79d11d8dd9876f6a6f1df41f8
. make vm tell kernel virtual locations of mappings
. makes _minix_kerninfo feature work
. fix for mappings being larger than what 1 pde can address
(e.g. devices memory requested on arm)
. still requires a special case for devices memory for the
kernel, which has to switch to virtual addressing
Change-Id: I2e94090aa432346fa4da0edeba72f0b7406c2ad7
On ARM we can't yet globally map pages into every process. So now that
we correctly receive the pointer to the globally mapped kern_info
struct, we have to ignore it on ARM because attempting to dereference
the pointer yields a segfault.
Due to the ABI we are using we have to use the earm architecture
moniker for the build system to behave correctly. This involves
then some headers to move around.
There is also a few related Makefile updates as well as minor
source code corrections.
Fix warnings about:
. Unused variables
. format mismatch in printf/scanf format string and arguments
. Missing parenthesis around assignment as truth values
. Clang warnings anout unknown GCC pragma
* Updating common/lib
* Updating lib/csu
* Updating lib/libc
* Updating libexec/ld.elf_so
* Corrected test on __minix in featuretest to actually follow the
meaning of the comment.
* Cleaned up _REENTRANT-related defintions.
* Disabled -D_REENTRANT for libfetch
* Removing some unneeded __NBSD_LIBC defines and tests
Change-Id: Ic1394baef74d11b9f86b312f5ff4bbc3cbf72ce2
. make vm be able to use malloc() by overriding brk()
and minix_mmap() functions
. phys regions can then be malloc()ed and free()d instead
of being in an avl tree, which is slightly faster
. 'offset' field in phys_region can go too (offset is implied
by position in array) but leads to bigger code changes
Also removing minix-config.inc and placing its content in
Makefile.inc, as the file is modified anyway. Simplifies diff.
Change-Id: I6262fdb790a5455b59e07acdccdc221fd5dd41f8
. 'anonymous' cache blocks (retrieved with NO_DEV as dev
parameter) were used to implement read()s from holes in
inodes that should return zeroes
. this is an awkward special case in the cache code though
and there's a more direct way to implement the same functionality:
instead of copying from a new, anonymous, zero block, to
the user target buffer, simply sys_safememset the user target
buffer directly. as this was the only use of this feature,
this is all that's needed to simplify the cache code a little.
upgrade to NetBSD CVS release from 2012/10/17 12:00:00 UTC
Makefiles updates to imporve portability
Made sure to be consistent in the usage of braces/parenthesis at
least on a per file basis. For variables, it is recommended to
continue to use braces.
The tested targets are the followgin ones:
* tools
* distribution
* sets
* release
The remaining NetBSD targets have not been disabled nor tested
*at all*. Try them at your own risk, they may reboot the earth.
For all compliant Makefiles, objects and generated files are put in
MAKEOBJDIR, which means you can now keep objects between two branch
switching. Same for DESTDIR, please refer to build.sh options.
Regarding new or modifications of Makefiles a few things:
* Read share/mk/bsd.README
* If you add a subdirectory, add a Makefile in it, and have it called
by the parent through the SUBDIR variable.
* Do not add arbitrary inclusion which crosses to another branch of
the hierarchy; If you can't do without it, put a comment on why.
If possible, do not use inclusion at all.
* Use as much as possible the infrastructure, it is here to make
life easier, do not fight it.
Sets and package are now used to track files.
We have one set called "minix", composed of one package called "minix-sys"
Bumping libc files for unsupported architectures, to simplify merging.
A bunch of small fixes:
* in libutil update
* the macro in endian.h
* some undefined types due to clear separation from host.
* Fix a warning for cdbr.c
Some modification which were required for the new build system:
* inclusion path for const.h in sconst, still hacky
* Removed default malloc.c which conflicts on some occasions.
Add primary cache management feature to libminixfs as mfs and ext2
currently do separately, remove cache code from mfs and ext2, and make
them use the libminixfs interface. This makes all fields of the buf
struct private to libminixfs and FS clients aren't supposed to access
them at all. Only the opaque 'void *data' field (the FS block contents,
used to be called bp) is to be accessed by the FS client.
The main purpose is to implement the interface to the 2ndary vm cache
just once, get rid of some code duplication, and add a little
abstraction to reduce the code inertia of the whole caching business.
Some minor sanity checking and prohibition done by mfs in this code
as removed from the generic primary cache code as a result:
- checking all inodes are not in use when allocating/resizing
the cache
- checking readonly filesystems aren't written to
- checking the superblock isn't written to on mounted filesystems
The minixfslib code relies on fs_blockstats() in the client filesystem to
return some FS usage information.
. add cpufeature detection of both
. use it for both ipc and kernelcall traps, using a register
for call number
. SYSENTER/SYSCALL does not save any context, therefore userland
has to save it
. to accomodate multiple kernel entry/exit types, the entry
type is recorded in the process struct. hitherto all types
were interrupt (soft int, exception, hard int); now SYSENTER/SYSCALL
is new, with the difference that context is not fully restored
from proc struct when running the process again. this can't be
done as some information is missing.
. complication: cases in which the kernel has to fully change
process context (i.e. sigreturn). in that case the exit type
is changed from SYSENTER/SYSEXIT to soft-int (i.e. iret) and
context is fully restored from the proc struct. this does mean
the PC and SP must change, as the sysenter/sysexit userland code
will otherwise try to restore its own context. this is true in the
sigreturn case.
. override all usage by setting libc_ipc=1
complete munmap implementation; single-page references made
a general munmap() implementation possible to write cleanly.
. memory: let the MIOCRAMSIZE ioctl set the imgrd device
size (but only to 0)
. let the ramdisk command set sizes to 0
. use this command to set /dev/imgrd to 0 after mounting /usr
in /etc/rc, so the boot time ramdisk is freed (about 4MB
currently)
. rename minix malloc sources to minix-* so Makefile
references aren't ambiguous
. throw out malloc source file copies in libminc
. make libminc use phkmalloc instead of minix malloc (slightly faster)
. ld.so is linked at 0 but it can relocate itself; we
wish to load ld.so higher though to trap NULL dereferences.
if we know we have to execute ld.so, vfs tells libexec to put it
higher.
. map all objects named usermapped_*.o with globally visible
pages; usermapped_glo_*.o with the VM 'global' bit on, i.e.
permanently in tlb (very scarce resource!)
. added kinfo, machine, kmessages and loadinfo for a start
. modified log, tty to make use of the shared messages struct
. some strncpy/strcpy to strlcpy conversions
. new <minix/param.h> to avoid including other minix headers
that have colliding definitions with library and commands code,
causing parse warnings
. removed some dead code / assignments
This commit removes all traces of Minix segments (the text/data/stack
memory map abstraction in the kernel) and significance of Intel segments
(hardware segments like CS, DS that add offsets to all addressing before
page table translation). This ultimately simplifies the memory layout
and addressing and makes the same layout possible on non-Intel
architectures.
There are only two types of addresses in the world now: virtual
and physical; even the kernel and processes have the same virtual
address space. Kernel and user processes can be distinguished at a
glance as processes won't use 0xF0000000 and above.
No static pre-allocated memory sizes exist any more.
Changes to booting:
. The pre_init.c leaves the kernel and modules exactly as
they were left by the bootloader in physical memory
. The kernel starts running using physical addressing,
loaded at a fixed location given in its linker script by the
bootloader. All code and data in this phase are linked to
this fixed low location.
. It makes a bootstrap pagetable to map itself to a
fixed high location (also in linker script) and jumps to
the high address. All code and data then use this high addressing.
. All code/data symbols linked at the low addresses is prefixed by
an objcopy step with __k_unpaged_*, so that that code cannot
reference highly-linked symbols (which aren't valid yet) or vice
versa (symbols that aren't valid any more).
. The two addressing modes are separated in the linker script by
collecting the unpaged_*.o objects and linking them with low
addresses, and linking the rest high. Some objects are linked
twice, once low and once high.
. The bootstrap phase passes a lot of information (e.g. free memory
list, physical location of the modules, etc.) using the kinfo
struct.
. After this bootstrap the low-linked part is freed.
. The kernel maps in VM into the bootstrap page table so that VM can
begin executing. Its first job is to make page tables for all other
boot processes. So VM runs before RS, and RS gets a fully dynamic,
VM-managed address space. VM gets its privilege info from RS as usual
but that happens after RS starts running.
. Both the kernel loading VM and VM organizing boot processes happen
using the libexec logic. This removes the last reason for VM to
still know much about exec() and vm/exec.c is gone.
Further Implementation:
. All segments are based at 0 and have a 4 GB limit.
. The kernel is mapped in at the top of the virtual address
space so as not to constrain the user processes.
. Processes do not use segments from the LDT at all; there are
no segments in the LDT any more, so no LLDT is needed.
. The Minix segments T/D/S are gone and so none of the
user-space or in-kernel copy functions use them. The copy
functions use a process endpoint of NONE to realize it's
a physical address, virtual otherwise.
. The umap call only makes sense to translate a virtual address
to a physical address now.
. Segments-related calls like newmap and alloc_segments are gone.
. All segments-related translation in VM is gone (vir2map etc).
. Initialization in VM is simpler as no moving around is necessary.
. VM and all other boot processes can be linked wherever they wish
and will be mapped in at the right location by the kernel and VM
respectively.
Other changes:
. The multiboot code is less special: it does not use mb_print
for its diagnostics any more but uses printf() as normal, saving
the output into the diagnostics buffer, only printing to the
screen using the direct print functions if a panic() occurs.
. The multiboot code uses the flexible 'free memory map list'
style to receive the list of free memory if available.
. The kernel determines the memory layout of the processes to
a degree: it tells VM where the kernel starts and ends and
where the kernel wants the top of the process to be. VM then
uses this entire range, i.e. the stack is right at the top,
and mmap()ped bits of memory are placed below that downwards,
and the break grows upwards.
Other Consequences:
. Every process gets its own page table as address spaces
can't be separated any more by segments.
. As all segments are 0-based, there is no distinction between
virtual and linear addresses, nor between userspace and
kernel addresses.
. Less work is done when context switching, leading to a net
performance increase. (8% faster on my machine for 'make servers'.)
. The layout and configuration of the GDT makes sysenter and syscall
possible.
When a file system is mounted some heuristics are used to define
a RS label for that system. This commit allows to specify the
label to use in an optional mount argument using either
mount -o rslabel=fs_myfs or as a mount option in fstab.
This can be used to start services that later also need to be
accessed directly.
make weak symbol references and namespace renames references
the renamed versions.
function renaming, weak symbol references and libc namespace.h
protection interact in hairy ways and causes weak symbol references
for renamed functions to be unresolved; e.g. vfork should be an
alias for _vfork but _vfork doesn't exist because __vfork14()
exists.
this is a problem for dynamically linked executables as all symbols
have to be resolved, used or not, at link time. it was masked by
clang-compiled base system libraries but is a problem when gcc does
it.
. sys_vircopy always uses D for both src and dst
. sys_physcopy uses PHYS_SEG if and only if corresponding
endpoint is NONE, so we can derive the mode (PHYS_SEG or D)
from the endpoint arg in the kernel, dropping the seg args
. fields in msg still filled in for backwards compatability,
using same NONE-logic in the library
. all invocations were S or D, so can safely be dropped
to prepare for the segmentless world
. still assign D to the SCP_SEG field in the message
to make previous kernels usable
. new mode for sys_memset: include process so memset can be
done in physical or virtual address space.
. add a mode to mmap() that lets a process allocate uninitialized
memory.
. this allows an exec()er (RS, VFS, etc.) to request uninitialized
memory from VM and selectively clear the ranges that don't come
from a file, leaving no uninitialized memory left for the process
to see.
. use callbacks for clearing the process, clearing memory in the
process, and copying into the process; so that the libexec code
can be used from rs, vfs, and in the future, kernel (to load vm)
and vm (to load boot-time processes)
. make exec() callers (i.e. vfs and rs) determine the
memory layout by explicitly reserving regions using
mmap() calls on behalf of the exec()ing process,
i.e. handling all of the exec logic, thereby eliminating
all special exec() knowledge from VM.
. the new procedure is: clear the exec()ing process
first, then call third-party mmap()s to reserve memory, then
copy the executable file section contents in, all using callbacks
tailored to the caller's way of starting an executable
. i.e. no more explicit EXEC_NEWMEM-style calls in PM or VM
as with rigid 2-section arguments
. this naturally allows generalizing exec() by simply loading
all ELF sections
. drop/merge of lots of duplicate exec() code into libexec
. not copying the code sections to vfs and into the executable
again is a measurable performance improvement (about 3.3% faster
for 'make' in src/servers/)
justification: soon we won't be able to execute sep I&D aouts at
all (because of the vanishing segments), which was the default mode
to generate them so most binaries will be sep I&D.
this makes the vfs/rs exec() unification work simpler.
after unification, common I&D aout could be added back quite simply.
these two functions will be used to support all exec() functionality
going into a single library shared by RS and VFS and exec() knowledge
leaving VM.
. third-party mmap: allow certain processes (VFS, RS) to
do mmap() on behalf of another process
. PROCCTL: used to free and clear a process' address space
WARNING: this will break existing dynamically linked binaries if they
exist. If you have any:
. re-build world statically first if necessary
. remove libraries from /lib and /usr/lib
. then build world
This change:
. avoids possible future dismay when interfacing other
systems' binaries; done until they are abi-compatible
Thanks to Antoine Leca for pointing this out.
This can be turned back on when the library is compiled with
-DMTHREAD_STRICT (which enables more sanity checks). However,
always performing this check shows up in system profiling results.
. vfs: pass execname in aux vectors
. ld.elf_so: use this to expand $ORIGIN
. this requires the executable to reserve more
space at exec() calling time
building defaults to off until clang is updated.
current clang does not handle -shared, necessary to change the ld
invocation to build shared libraries properly. a new clang should be
installed and MKPIC defaults to no unless the newer clang is detected.
changes:
. mainly small imports of a Makefile or two and small fixes
(turning things back on that were turned off in Makefiles)
. e.g.: dynamic librefuse now depends on dynamic
libpuffs, so libpuffs has to be built dynamically too
and a make dependency barrier is needed in lib/Makefile
. all library objects now have a PIC (for .so) and non-PIC
version, so everything is built twice.
. generate PIC versions of the compat (un-RENAMEd) jump files,
include function type annotation in generated assembly
. build progs with -static by default for now
. also build ld.elf_so
. also import NetBSD ldd
. generalize libexec slightly to get some more necessary information
from ELF files, e.g. the interpreter
. execute dynamically linked executables when exec()ed by VFS
. switch to netbsd variant of elf32.h exclusively, solves some
conflicting headers
. file- and functionality-compatible with previous situation
(FreeBSD csu) (with a crt1.o -> crt0.o symlink in /usr/lib)
. harmonizes source with netbsd
. harmonizes linker invocation (e.g. clang) with netbsd
. helpful to get some arm code in there for the arm port project
This Shared Folders File System library (libsffs) now contains all the
file system logic originally in HGFS. The actual HGFS server code is
now a stub that passes on all the work to libsffs. The libhgfs library
is changed accordingly.
. fixes e.g. ssh sessions not getting their own
controlling tty (causing ^C getting broadcast to too
many processes)
. previously (before -lutil) handled like this by bsd-openpty.c in
openssh
. reported by Andy Kosela, debugged by ThomasV
remove some old minix-userland-specific stuff
. /etc/ttytab as a file, and minix-compat function (fftyslot()),
replaced by /etc/ttys and new libc functions
. also remove minix-specific nlist(), cuserid(), fttyslot(), v8 regex
functions and <compat/regex.h>
. and remaining minix-only utilities that use them
. also unused <compat/pwd.h> and <compat/syslog.h> and
redundant <sys/sigcontext.h>
- add files needed for acpi, ahci, fbd, vfs to libminc
- remove "-lc" from their respective makefiles
- remove setenv from libminc (requires initialization)
- libnetsock - internal implementation of a socket on the lwip
server side. it encapsulates the asynchronous protocol
- lwip server - uses libnetsock to work with the asynchronous
protocol
- if an operation (R, W, IOCTL) is non blocking, a flag is set
and sent to the device.
- nothing changes for sync devices
- asyn devices should reply asap if an operation is non-blocking.
We must trust the devices, but we had to trust them anyway to
reply to CANCEL correctly
- we safe sending CANCEL commands to asyn devices. This greatly
simplifies the protocol. Asynchronous devices can always reply
when a reply is ready and do not need to deal with other
situations
- currently, none of our drivers use the flags since they drive
virtual devices which do not block
. harmonize bsd.lib.mk and bsd.man.mk with netbsd files
. throw out minix section 3 (library calls) manpages,
replaced by netbsd ones that are now installed
There is important information about booting non-ack images in
docs/UPDATING. ack/aout-format images can't be built any more, and
booting clang/ELF-format ones is a little different. Updating to the
new boot monitor is recommended.
Changes in this commit:
. drop boot monitor -> allowing dropping ack support
. facility to copy ELF boot files to /boot so that old boot monitor
can still boot fairly easily, see UPDATING
. no more ack-format libraries -> single-case libraries
. some cleanup of OBJECT_FMT, COMPILER_TYPE, etc cases
. drop several ack toolchain commands, but not all support
commands (e.g. aal is gone but acksize is not yet).
. a few libc files moved to netbsd libc dir
. new /bin/date as minix date used code in libc/
. test compile fix
. harmonize includes
. /usr/lib is no longer special: without ack, /usr/lib plays no
kind of special bootstrapping role any more and bootstrapping
is done exclusively through packages, so releases depend even
less on the state of the machine making them now.
. rename nbsd_lib* to lib*
. reduce mtree
Add guard pages to the top of the stack to catch overflow errors.
Moreover, fix a bug where libmthread would keep using a stack that was
just deallocated; a detached thread would deallocate its own stack after
it was finished running).
This driver can be loaded as an overlay on top of a real block
device, and can then be used to generate block-level failures for
certain transfer requests. Specifically, a rule-based system allows
the user to introduce (overt and silent) data corruption and errors.
It exposes itself through /dev/fbd, and a file system can be mounted
on top of it. The new fbdctl(8) tool can be used to control the
driver; see ``man fbdctl'' for details. It also comes with a test
set, located in test/fbdtest.
This removes a race condition when the block driver performs a
complete restart after a crash (the new default). If any user of
the driver finds out its new endpoint and sends a request to the
new driver instance before this instance has had the chance to
initialize, then its initialization would clear all IPC state and
thereby erroneously cancel the incoming request. Clearing IPC
state is only desired upon a stateful restart (where the driver's
endpoint is retained). This information is now passed to and used
by libblockdriver accordingly.
This patch provides basic protection against damage resulting from
differently compiled servers blindly copying tables to one another.
In every getsysinfo() call, the caller is provided with the expected
size of the requested data structure. The callee fails the call if
the expected size does not match the data structure's actual size.
Each block driver now gets to specify whether it is a disk block
driver, which implies it wants the library to handle getting and
setting partitions for it.
The NetBSD libc provides a mechanism to have versions of system calls.
By 'renaming' symbols to a new version, freshly compiled programs will
automatically use the new symbol iff they use the proper header files. The
old, not renamed, version of the symbol will still exist (after being moved
to the compat directory), so old programs can still link.
Since MINIX doesn't support dynamic linking, the whole rename mechanism
doesn't really work for us. However, removing it would create a huge diff
with the current NetBSD libc.
A lot of the compat code relies on things we don't (seem to) have, and
therefore does not get built and linked. This causes trouble for tools like
autoconf, which will fail to find the renamed symbols. For example,
currently select gets renamed to __select50 in libc. Autoconf looks for
'select' and doesn't find it and reports we don't have it. This is where
the compat.S stub comes into play: it generates the old symbols and jumps to
the new symbols. However, as this is done in one object file, all renamed
symbols get linked together, causing binaries to be huge. This patch fixes
that by generating an object file for each renamed symbol.
This patch also makes the MISSING_SYSCALLS more complete and marginally
reduces the diff with NetBSD.
The implementation is in libblockdriver, and works transparently for
all block drivers. The new btrace(8) tool can be used to control block
tracing; see ``man btrace'' for details.
This patch separates the character and block driver communication
protocols. The old character protocol remains the same, but a new
block protocol is introduced. The libdriver library is replaced by
two new libraries: libchardriver and libblockdriver. Their exposed
API, and drivers that use them, have been updated accordingly.
Together, libbdev and libblockdriver now completely abstract away
the message format used by the block protocol. As the memory driver
is both a character and a block device driver, it now implements its
own message loop.
The most important semantic change made to the block protocol is that
it is no longer possible to return both partial results and an error
for a single transfer. This simplifies the interaction between the
caller and the driver, as the I/O vector no longer needs to be copied
back. Also, drivers are now no longer supposed to decide based on the
layout of the I/O vector when a transfer should be cut short. Put
simply, transfers are now supposed to either succeed completely, or
result in an error.
After this patch, the state of the various pieces is as follows:
- block protocol: stable
- libbdev API: stable for synchronous communication
- libblockdriver API: needs slight revision (the drvlib/partition API
in particular; the threading API will also change shortly)
- character protocol: needs cleanup
- libchardriver API: needs cleanup accordingly
- driver restarts: largely unsupported until endpoint changes are
reintroduced
As a side effect, this patch eliminates several bugs, hacks, and gcc
-Wall and -W warnings all over the place. It probably introduces a
few new ones, too.
Update warning: this patch changes the protocol between MFS and disk
drivers, so in order to use old/new images, the MFS from the ramdisk
must be used to mount all file systems.
. clang-linked binaries were not calling global constructors, as the
code to do so wasn't in csu/ and linked
. it does work for gcc as it uses its self-supplied crt{begin,end} code
. this commit copies netbsd's crt{begin,end}.S, which contains
constructor/destructor calling code, called from .init and .fini
sections already accumulated by the linker. the _init function was already
called by the C startup code before calling main.
. based on work by Antoine Leca
Import libpuffs and our port of libpuffs. The port was done as part of
GSoC 2011 FUSE project, done by Evgeniy Ivanov. The librefuse import
did not require any porting efforts. Libpuffs has been modified to
understand our VFS-FS protocol and translate between that and PUFFS. As
an example that it works, fuse-ntfs-3g from pkgsrc can be compiled and
used to mount ntfs partitions:
mount -t ntfs-3g <device> <mountpoint>
FUSE only works with the asynchronous version of VFS. See <docs/UPDATING> on
how to run AVFS.
This patch further includes some changes to mount(1) and mount(2) so it's
possible to use file systems provided by pkgsrc (note: manual modifications
to /etc/system.conf are still needed. There has been made an exception for
fuse-ntfs-3g, so it already as an entry).
This patch fixes most of current reasons to generate compiler warnings.
The changes consist of:
- adding missing casts
- hiding or unhiding function declarations
- including headers where missing
- add __UNCONST when assigning a const char * to a char *
- adding missing return statements
- changing some types from unsigned to signed, as the code seems to want
signed ints
- converting old-style function definitions to current style (i.e.,
void func(param1, param2) short param1, param2; {...} to
void func (short param1, short param2) {...})
- making the compiler silent about signed vs unsigned comparisons. We
have too many of those in the new libc to fix.
A number of bugs in the test set were fixed. These bugs were never
triggered with our old libc. Consequently, these tests are now forced to
link with the new libc or they will generate errors (in particular tests 43
and 55).
Most changes in NetBSD libc are limited to moving aroudn "#ifndef __minix"
or stuff related to Minix-specific things (code in sys-minix or gen/minix).
The "bdev" library provides basic primitives for file systems to talk
to block device drivers, hiding the details of the underlying protocol
and interaction model.
This version of libbdev is rather basic. It is planned to support the
following features in the long run:
- asynchronous requests and replies;
- recovery support for underlying block drivers;
- retrying of failed I/O requests.
The commit also changes our block-based file systems (mfs, ext2, isofs)
to make use of libbdev.
This patch adds support for executing multiple concurrent requests on
different devices on the same AHCI controller. The libdriver library
has been extended to include a generic multithreading interface, and
the AHCI driver has been extended to make use of this interface.
The original version of this code has been written by Arne Welzel.
. add bsd-style MLINKS to minix man set, restoring aliases
(e.g. man add64 -> int64)
. update daily cron script to run makewhatis and restore makewhatis
in man Makefile (makedb), restores functionality of man -k
. netbsd imports of man, mdocml, makewhatis, libutil, apropos
. update man.conf with manpage locations, restoring man [-s] <section>
. throws out some obsolete manpages
. move cache size heuristic from mfs there
so mfs and ext2 can share it
. add vfs credentials retrieving function, with
backwards compatability from previous struct
format, to be used by both ext2 and mfs
. fix for ext2 - STATICINIT was fed no.
of bytes instead of no. of elements, overallocating
memory by a megabyte or two for the superblock
. default jemalloc is not too easy to compile without threads
libraries/types
. non-default malloc has odd virtual address space binge problem
. switch to ack/minix malloc in old libc for now
. ipc wants to know about processes that get
signals, so that it can break blocking ipc operations
. doing it for every single signal is wasteful
and causes the annoying 'no slot for signals' message
. this fix tells vm on a per-process basis it (ipc)
wants to be notified, i.e. only when it does any ipc calls
. move ipc config to separate config file while we're at it
- BSD-licensed Code gratefully taken from the project at
http://en.sourceforge.jp/projects/sfnet_vassertlinuxsdk/
- For more information on vmware VAssert, a powerful debugging
facility usable under vmware, see:
www.vmware.com/pdf/ws65_vassert_programming.pdf
. don't install minix <termcap.h> as libterminfo
has its own (but still install it in /usr/include.ack)
. forget minix termcap functions in -lcompat_minix
. make commands use -lterminfo in netbsd libc compile mode
Improves cache locality by grouping together dependency generation
with building for each program instead of doing a whole-tree dep
generation phase followed by a whole-tree build phase
. it's a good extra interface to have but doesn't
meet standardised functionality
. applications (in pkgsrc) find it and expect
full functionality the minix mmap doesn't offter
. on the whole probably better to hide these functions
(mmap and friends) until they are grown up; the base system
can use the new minix_* names
. strerror() assumes this
. remove generated libminc/errlist.c
. errno's in <sys/errno.h> have to be in sorted order
. filtering out some errno.h in Makefile lets us use near-stock
errlist.awk
* VFS and installed MFSes must be in sync before and after this change *
Use struct stat from NetBSD. It requires adding new STAT, FSTAT and LSTAT
syscalls. Libc modification is both backward and forward compatible.
Also new struct stat uses modern field sizes to avoid ABI
incompatibility, when we update uid_t, gid_t and company.
Exceptions are ino_t and off_t in old libc (though paddings added).
Now users can choose between libsys, libsys + libminc and
libsys + libc. E.g. PUFFS/FUSE servers need libsys + libc while
old servers can use libsys + libminc.
1. ack, a.out, minix headers (moved to /usr/include.ack),
minix libc
2. gcc/clang, elf, netbsd headers (moved to /usr/include),
netbsd libc (moved to /usr/lib)
So this obsoletes the /usr/netbsd hierarchy.
No special invocation for netbsd libc necessary - it's always used
for gcc/clang.
The opendir(3) function was setting errno to ENOTDIR even
when the directory existed and was opened successfully. This
caused git to falsely detect an error.
This change moves the errno assignment into the failure code
block. It also adds a test to test24 to check for errno
changing when opendir(3) returns success.
