Commit graph

171 commits

Author SHA1 Message Date
Thomas Veerman
c89bc85009 Don't repeat out-of-space messages
This patch makes PFS, EXT2 and MFS print only once that they're out of
space. After freeing up space and running out of space again, the message
will be printed again also.
2011-12-21 10:47:28 +00:00
David van Moolenbroek
db087efac4 VFS/FS: REQ_NEW_DRIVER now provides a label 2011-11-30 19:05:26 +01:00
David van Moolenbroek
b4d9ac91ba FS servers: fix dependencies broken in git-af01bda 2011-11-23 17:03:16 +01:00
David van Moolenbroek
b4d909d415 Split block/character protocols and libdriver
This patch separates the character and block driver communication
protocols. The old character protocol remains the same, but a new
block protocol is introduced. The libdriver library is replaced by
two new libraries: libchardriver and libblockdriver. Their exposed
API, and drivers that use them, have been updated accordingly.
Together, libbdev and libblockdriver now completely abstract away
the message format used by the block protocol. As the memory driver
is both a character and a block device driver, it now implements its
own message loop.

The most important semantic change made to the block protocol is that
it is no longer possible to return both partial results and an error
for a single transfer. This simplifies the interaction between the
caller and the driver, as the I/O vector no longer needs to be copied
back. Also, drivers are now no longer supposed to decide based on the
layout of the I/O vector when a transfer should be cut short. Put
simply, transfers are now supposed to either succeed completely, or
result in an error.

After this patch, the state of the various pieces is as follows:
- block protocol: stable
- libbdev API: stable for synchronous communication
- libblockdriver API: needs slight revision (the drvlib/partition API
  in particular; the threading API will also change shortly)
- character protocol: needs cleanup
- libchardriver API: needs cleanup accordingly
- driver restarts: largely unsupported until endpoint changes are
  reintroduced

As a side effect, this patch eliminates several bugs, hacks, and gcc
-Wall and -W warnings all over the place. It probably introduces a
few new ones, too.

Update warning: this patch changes the protocol between MFS and disk
drivers, so in order to use old/new images, the MFS from the ramdisk
must be used to mount all file systems.
2011-11-23 14:06:37 +01:00
David van Moolenbroek
af01bda509 libbdev: initial version
The "bdev" library provides basic primitives for file systems to talk
to block device drivers, hiding the details of the underlying protocol
and interaction model.

This version of libbdev is rather basic. It is planned to support the
following features in the long run:

 - asynchronous requests and replies;
 - recovery support for underlying block drivers;
 - retrying of failed I/O requests.

The commit also changes our block-based file systems (mfs, ext2, isofs)
to make use of libbdev.
2011-11-09 14:43:25 +01:00
Ben Gras
c83a90e2f1 mfs: ditch global readahead 2011-11-06 19:15:12 +01:00
Ben Gras
49532259e5 mfs: improve & simplify secondary cache logic
. fixes "!" errors when booting from cd
2011-09-23 15:02:20 +00:00
Thomas Veerman
203937456e Fix off-by-one errors and increase PATH_MAX to 1024
In some places it was assumed that PATH_MAX does not include a
terminating null character.

Increases PATH_MAX to 1024 to get in sync with NetBSD. Required some
rewriting in AVFS to keep memory usage low (the stack in use by a thread
is very small).
2011-09-12 09:00:24 +00:00
Ben Gras
4857d5d554 add -lminixfs with fs support routines
. move cache size heuristic from mfs there
	  so mfs and ext2 can share it
	. add vfs credentials retrieving function, with
	  backwards compatability from previous struct
	  format, to be used by both ext2 and mfs
	. fix for ext2 - STATICINIT was fed no.
	  of bytes instead of no. of elements, overallocating
	  memory by a megabyte or two for the superblock
2011-09-08 16:52:13 +00:00
Thomas Veerman
ce1a2793f9 Fix off-by-one error 2011-09-08 15:10:57 +00:00
Thomas Veerman
f78fb05676 Check group range for sanity 2011-09-08 12:23:03 +00:00
Thomas Veerman
8a266a478e Increase gid_t and uid_t to 32 bits
Increase gid_t and uid_t to 32 bits and provide backwards compatibility
where needed.
2011-09-05 13:56:14 +00:00
Thomas Veerman
8a73de51f1 Tell VFS how many requests an FS can handle concurrently 2011-08-26 12:47:43 +00:00
Ben Gras
f984dbba70 increase system-wide filename limit to 255
. move mfs-specific struct, constants to mfs/, so
	  mfs-specific, on-disk format structs and consts are
	  fully isolated from generic structs and functions
	. removes de and readfs utils
2011-08-17 16:00:01 +00:00
Evgeniy Ivanov
bc434254ea MFS: optimize dentry acllocation.
When search for free slot always cache position of last allocated slot,
so next time when need to allocate new slot, we can search from that
position.
2011-08-17 08:41:35 +00:00
Evgeniy Ivanov
426cddbb30 Add missing put_inode().
Fixes bug from a9a45cddc.
2011-08-15 16:20:06 +00:00
Evgeniy Ivanov
a9a45cddc4 Fix bugs found in MFS by clang static analyzer. 2011-08-10 20:47:16 +00:00
Thomas Veerman
7588db2691 Add support for VFS-FS transaction IDs 2011-07-27 15:49:42 +00:00
Evgeniy Ivanov
255ae85b1e Fix time modification on truncate()
POSIX truncate specification says "Upon successful completion, if
the *file size is changed*, this function shall mark for update the
st_ctime and st_mtime fields of the file." This patch prevents
changing of the date fields when the size stays the same.
2011-07-15 14:21:05 +00:00
Ben Gras
253c293213 mfs<->vfs stat fallback for old vfs 2011-07-12 18:09:52 +02:00
Evgeniy Ivanov
ef0a265086 New stat structure.
* VFS and installed MFSes must be in sync before and after this change *

Use struct stat from NetBSD. It requires adding new STAT, FSTAT and LSTAT
syscalls. Libc modification is both backward and forward compatible.

Also new struct stat uses modern field sizes to avoid ABI
incompatibility, when we update uid_t, gid_t and company.
Exceptions are ino_t and off_t in old libc (though paddings added).
2011-07-12 16:39:55 +02:00
Ben Gras
9c01ceb576 introduce sqrt_approx() in -lsys
. use this to avoid -lm dependency in mfs
2011-07-04 02:51:12 +02:00
Arun Thomas
93ae43f577 boot: Add multiboot support
Not yet fully spec-compliant; work in progress
2011-06-24 17:21:51 +02:00
Gianluca Guida
cc17b27a2b Build NetBSD libc library in world in ELF mode.
3 sets of libraries are built now:
  . ack: all libraries that ack can compile (/usr/lib/i386/)
  . clang+elf: all libraries with minix headers (/usr/lib/)
  . clang+elf: all libraries with netbsd headers (/usr/netbsd/)

Once everything can be compiled with netbsd libraries and headers, the
/usr/netbsd hierarchy will be obsolete and its libraries compiled with
netbsd headers will be installed in /usr/lib, and its headers
in /usr/include. (i.e. minix libc and current minix headers set
will be gone.)

To use the NetBSD libc system (libraries + headers) before
it is the default libc, see:
   http://wiki.minix3.org/en/DevelopersGuide/UsingNetBSDCode
This wiki page also documents the maintenance of the patch
files of minix-specific changes to imported NetBSD code.

Changes in this commit:
  . libsys: Add NBSD compilation and create a safe NBSD-based libc.
  . Port rest of libraries (except libddekit) to new header system.
  . Enable compilation of libddekit with new headers.
  . Enable kernel compilation with new headers.
  . Enable drivers compilation with new headers.
  . Port legacy commands to new headers and libc.
  . Port servers to new headers.
  . Add <sys/sigcontext.h> in compat library.
  . Remove dependency file in tree.
  . Enable compilation of common/lib/libc/atomic in libsys
  . Do not generate RCSID strings in libc.
  . Temporarily disable zoneinfo as they are incompatible with NetBSD format
  . obj-nbsd for .gitignore
  . Procfs: use only integer arithmetic. (Antoine Leca)
  . Increase ramdisk size to create NBSD-based images.
  . Remove INCSYMLINKS handling hack.
  . Add nbsd_include/sys/exec_elf.h
  . Enable ELF compilation with NBSD libc.
  . Add 'make nbsdsrc' in tools to download reference NetBSD sources.
  . Automate minix-port.patch creation.
  . Avoid using fstavfs() as it is *extremely* slow and unneeded.
  . Set err() as PRIVATE to avoid name clash with libc.
  . [NBSD] servers/vm: remove compilation warnings.
  . u32 is not a long in NBSD headers.
  . UPDATING info on netbsd hierarchy
  . commands fixes for netbsd libc
2011-06-24 11:46:30 +02:00
Ben Gras
a77c2973b3 fix clang warnings -R in kernel/ and servers/ 2011-06-09 16:09:13 +02:00
David van Moolenbroek
c51cd5fe91 Server/driver protocols: no longer allow third-party copies.
Before safecopies, the IO_ENDPT and DL_ENDPT message fields were needed
to know which actual process to copy data from/to, as that process may
not always be the caller. Now that we have full safecopy support, these
fields have become useless for that purpose: the owner of the grant is
*always* the caller. Allowing the caller to supply another endpoint is
in fact dangerous, because the callee may then end up using a grant
from a third party. One could call this a variant of the confused
deputy problem.

From now on, safecopy calls should always use the caller's endpoint as
grant owner. This fully obsoletes the DL_ENDPT field in the
inet/ethernet protocol. IO_ENDPT has other uses besides identifying the
grant owner though. This patch renames IO_ENDPT to USER_ENDPT, not only
because that is a more fitting name (it should never be used for I/O
after all), but also in order to intentionally break any old system
source code outside the base system. If this patch breaks your code,
fixing it is fairly simple:

- DL_ENDPT should be replaced with m_source;
- IO_ENDPT should be replaced with m_source when used for safecopies;
- IO_ENDPT should be replaced with USER_ENDPT for any other use, e.g.
  when setting REP_ENDPT, matching requests in CANCEL calls, getting
  DEV_SELECT flags, and retrieving of the real user process's endpoint
  in DEV_OPEN.

The changes in this patch are binary backward compatible.
2011-04-11 17:35:05 +00:00
Ben Gras
c90cdadcfb auto-tune mfs cache size based on FS usage and remaining system memory 2011-02-28 14:19:19 +00:00
Ben Gras
dc1cc91df1 <ansi.h> -> <minix/ansi.h> 2011-01-28 11:35:02 +00:00
David van Moolenbroek
5d8d5e0c3a change bitchunk_t from 16-bit to 32-bit 2010-12-21 10:44:45 +00:00
Dirk Vogt
9ed280d1ec decouple file system server start/termination from mount/umount 2010-11-23 19:34:56 +00:00
Ben Gras
b653422883 mfs - no more WRITE_IMMED 2010-10-11 15:35:43 +00:00
Thomas Veerman
af73996d51 Fix failing unmount attempt at shutdown after running test58 2010-09-17 09:34:36 +00:00
Thomas Veerman
0b00cf70b6 - Return ENOENT when trying to add files to removed (but open) directories.
- Add test58 to test this behavior.
2010-09-01 09:07:18 +00:00
Erik van der Kouwe
0fc71a1154 Increase NR_INODES in the FS servers to match NR_VNODES in VFS 2010-08-05 11:37:37 +00:00
Erik van der Kouwe
912b4e0cb7 Return ENOSPC when out of inodes 2010-08-05 11:37:06 +00:00
Thomas Veerman
34a2864e27 Fix a few compile time warnings 2010-07-02 12:41:19 +00:00
Arun Thomas
c0c8d25799 Rename mkfiles from minix.*.mk to bsd.*.mk
Makes things easier for pkgsrc
2010-06-25 18:29:09 +00:00
Ben Gras
33c9d427fd mfs - added put_block() to statvfs loop. 2010-06-24 10:56:45 +00:00
Erik van der Kouwe
498d7d8a4c Don't use kernel responses in servers 2010-06-24 07:37:26 +00:00
Ben Gras
491efeead9 mfs - revert part of previous commit; should not have been commited 2010-06-24 00:32:17 +00:00
Ben Gras
6cd2d1218e mfs - statvfs call, by Buccapatnam Tirumala, Gautam. 2010-06-23 23:58:16 +00:00
Ben Gras
13b5dd4a82 statvfs/fstatvfs entries. 2010-06-23 23:56:36 +00:00
Ben Gras
a89bcc465b mfs: minor cleanup of readahead: they can be private to read.c. 2010-06-21 18:25:04 +00:00
Thomas Veerman
be6490f4b3 Turn off debug message. 2010-06-09 11:05:16 +00:00
Thomas Veerman
a0eaaa5c9f Fix a bug in put_inode that causes corruption to the file system and another
bug that causes problems when files grow bigger than a certain threshold. Also
fix a few type and code inconsistencies.
2010-06-09 09:56:43 +00:00
Arun Thomas
4c10a31440 Remove legacy MM, FS, and FS_PROC_NR macros 2010-06-08 13:58:01 +00:00
Kees van Reeuwijk
ed0b81c25c Removed some unused variables and functions. 2010-06-02 19:41:38 +00:00
Thomas Veerman
6bbcab3ec4 Clean up MFS a bit:
- Remove unused includes.
 - Add include guards to headers.
 - Use unsigned variables in case they're never going to hold a negative
   value. This causes GCC's complaints to disappear and should make flexelint
   a lot happier, too.
 - Make functions private when they're used only within a module.
 - Remove unused variables.
 - Add casts where appropriate.
2010-06-01 12:35:33 +00:00
Arun Thomas
007104d60e GCC build fixes/updates
-Set stack sizes for boot image processes
-Increase RS stack size
-Reduce ramdisk size
-HARDWARE task should use kernel stack
-Minor asm tweaks for leading underscores
2010-05-26 18:45:55 +00:00
Kees van Reeuwijk
ac14a989b3 Fixed some inconsistent strict typing declarations.
Better strict typing.
2010-05-25 07:23:24 +00:00
Tomas Hruby
6e25ad8b0a Use of all NIL_* defines converted to NULL 2010-05-10 13:26:00 +00:00
Thomas Veerman
0aceb25535 Small cleanup of dead and/or redundant code. 2010-05-06 09:32:40 +00:00
Ben Gras
86e1b9d770 fsctl.h doesn't exist. 2010-05-05 11:49:41 +00:00
Ben Gras
f78d8e74fd secondary cache feature in vm.
A new call to vm lets processes yield a part of their memory to vm,
together with an id, getting newly allocated memory in return. vm is
allowed to forget about it if it runs out of memory. processes can ask
for it back using the same id. (These two operations are normally
combined in a single call.)

It can be used as a as-big-as-memory-will-allow block cache for
filesystems, which is how mfs now uses it.
2010-05-05 11:35:04 +00:00
David van Moolenbroek
cfb108afc7 fix mfs/isofs signal handling 2010-04-15 16:10:28 +00:00
Kees van Reeuwijk
bc314bda91 Remove the types Dev_t, _mnx_Gui, _mnx_Uid, and similar.
Use ANSI-style function declarations where necessary.
2010-04-13 10:58:41 +00:00
Cristiano Giuffrida
65ef539739 Driver mapping refactory.
VFS CHANGES:
- dmap table no longer statically initialized in VFS
- Dropped FSSIGNON svrctl call no longer used by INET

INET CHANGES:
- INET announces its presence to VFS just like any other driver

RS CHANGES:
- The boot image dev table contains all the data to initialize VFS' dmap table
- RS interface supports asynchronous up and update operations now
- RS interface extended to support driver style and flags
2010-04-09 21:56:44 +00:00
Cristiano Giuffrida
48c6bb79f4 Driver refactory for live update and crash recovery.
SYSLIB CHANGES:
- DS calls to publish / retrieve labels consider endpoints instead of u32_t.

VFS CHANGES:
- mapdriver() only adds an entry in the dmap table in VFS.
- dev_up() is only executed upon reception of a driver up event.

INET CHANGES:
- INET no longer searches for existing drivers instances at startup.
- A newtwork driver is (re)initialized upon reception of a driver up event.
- Networking startup is now race-free by design. No need to waste 5 seconds
at startup any more.

DRIVER CHANGES:
- Every driver publishes driver up events when starting for the first time or
in case of restart when recovery actions must be taken in the upper layers.
- Driver up events are published by drivers through DS. 
- For regular drivers, VFS is normally the only subscriber, but not necessarily.
For instance, when the filter driver is in use, it must subscribe to driver
up events to initiate recovery.
- For network drivers, inet is the only subscriber for now.
- Every VFS driver is statically linked with libdriver, every network driver
is statically linked with libnetdriver.

DRIVER LIBRARIES CHANGES:
- Libdriver is extended to provide generic receive() and ds_publish() interfaces
for VFS drivers.
- driver_receive() is a wrapper for sef_receive() also used in driver_task()
to discard spurious messages that were meant to be delivered to a previous
version of the driver.
- driver_receive_mq() is the same as driver_receive() but integrates support
for queued messages.
- driver_announce() publishes a driver up event for VFS drivers and marks
the driver as initialized and expecting a DEV_OPEN message.
- Libnetdriver is introduced to provide similar receive() and ds_publish()
interfaces for network drivers (netdriver_announce() and netdriver_receive()).
- Network drivers all support live update with no state transfer now.

KERNEL CHANGES:
- Added kernel call statectl for state management. Used by driver_announce() to
unblock eventual callers sendrecing to the driver.
2010-04-08 13:41:35 +00:00
Kees van Reeuwijk
94a81c840a Removed unused variables, added const where possible. 2010-04-07 11:25:51 +00:00
Kees van Reeuwijk
fc7dced1fa Fix printfs with too few or too many parms, remove unused vars, fix incorrect flag tests, other code cleanup. 2010-04-01 13:25:05 +00:00
Thomas Veerman
4d686f1616 Move allocation of temporary inodes for cloned character special devices from
MFS to PFS.
2010-03-30 15:00:09 +00:00
Kees van Reeuwijk
4865e3f4f9 More use of endpoint_t. Other code cleanup. 2010-03-30 14:07:15 +00:00
Arun Thomas
436d6012a3 Convert drivers/ and servers/ over to bsdmake
-Move libdriver to lib/
-Install all boot image services on filesystem to aid restartability
2010-03-22 21:25:22 +00:00
Cristiano Giuffrida
cb176df60f New RS and new signal handling for system processes.
UPDATING INFO:
20100317:
        /usr/src/etc/system.conf updated to ignore default kernel calls: copy
        it (or merge it) to /etc/system.conf.
        The hello driver (/dev/hello) added to the distribution:
        # cd /usr/src/commands/scripts && make clean install
        # cd /dev && MAKEDEV hello

KERNEL CHANGES:
- Generic signal handling support. The kernel no longer assumes PM as a signal
manager for every process. The signal manager of a given process can now be
specified in its privilege slot. When a signal has to be delivered, the kernel
performs the lookup and forwards the signal to the appropriate signal manager.
PM is the default signal manager for user processes, RS is the default signal
manager for system processes. To enable ptrace()ing for system processes, it
is sufficient to change the default signal manager to PM. This will temporarily
disable crash recovery, though.
- sys_exit() is now split into sys_exit() (i.e. exit() for system processes,
which generates a self-termination signal), and sys_clear() (i.e. used by PM
to ask the kernel to clear a process slot when a process exits).
- Added a new kernel call (i.e. sys_update()) to swap two process slots and
implement live update.

PM CHANGES:
- Posix signal handling is no longer allowed for system processes. System
signals are split into two fixed categories: termination and non-termination
signals. When a non-termination signaled is processed, PM transforms the signal
into an IPC message and delivers the message to the system process. When a
termination signal is processed, PM terminates the process.
- PM no longer assumes itself as the signal manager for system processes. It now
makes sure that every system signal goes through the kernel before being
actually processes. The kernel will then dispatch the signal to the appropriate
signal manager which may or may not be PM.

SYSLIB CHANGES:
- Simplified SEF init and LU callbacks.
- Added additional predefined SEF callbacks to debug crash recovery and
live update.
- Fixed a temporary ack in the SEF init protocol. SEF init reply is now
completely synchronous.
- Added SEF signal event type to provide a uniform interface for system
processes to deal with signals. A sef_cb_signal_handler() callback is
available for system processes to handle every received signal. A
sef_cb_signal_manager() callback is used by signal managers to process
system signals on behalf of the kernel.
- Fixed a few bugs with memory mapping and DS.

VM CHANGES:
- Page faults and memory requests coming from the kernel are now implemented
using signals.
- Added a new VM call to swap two process slots and implement live update.
- The call is used by RS at update time and in turn invokes the kernel call
sys_update().

RS CHANGES:
- RS has been reworked with a better functional decomposition.
- Better kernel call masks. com.h now defines the set of very basic kernel calls
every system service is allowed to use. This makes system.conf simpler and
easier to maintain. In addition, this guarantees a higher level of isolation
for system libraries that use one or more kernel calls internally (e.g. printf).
- RS is the default signal manager for system processes. By default, RS
intercepts every signal delivered to every system process. This makes crash
recovery possible before bringing PM and friends in the loop.
- RS now supports fast rollback when something goes wrong while initializing
the new version during a live update.
- Live update is now implemented by keeping the two versions side-by-side and
swapping the process slots when the old version is ready to update.
- Crash recovery is now implemented by keeping the two versions side-by-side
and cleaning up the old version only when the recovery process is complete.

DS CHANGES:
- Fixed a bug when the process doing ds_publish() or ds_delete() is not known
by DS.
- Fixed the completely broken support for strings. String publishing is now
implemented in the system library and simply wraps publishing of memory ranges.
Ideally, we should adopt a similar approach for other data types as well.
- Test suite fixed.

DRIVER CHANGES:
- The hello driver has been added to the Minix distribution to demonstrate basic
live update and crash recovery functionalities.
- Other drivers have been adapted to conform the new SEF interface.
2010-03-17 01:15:29 +00:00
Ben Gras
35a108b911 panic() cleanup.
this change
   - makes panic() variadic, doing full printf() formatting -
     no more NO_NUM, and no more separate printf() statements
     needed to print extra info (or something in hex) before panicing
   - unifies panic() - same panic() name and usage for everyone -
     vm, kernel and rest have different names/syntax currently
     in order to implement their own luxuries, but no longer
   - throws out the 1st argument, to make source less noisy.
     the panic() in syslib retrieves the server name from the kernel
     so it should be clear enough who is panicing; e.g.
         panic("sigaction failed: %d", errno);
     looks like:
         at_wini(73130): panic: sigaction failed: 0
         syslib:panic.c: stacktrace: 0x74dc 0x2025 0x100a
   - throws out report() - printf() is more convenient and powerful
   - harmonizes/fixes the use of panic() - there were a few places
     that used printf-style formatting (didn't work) and newlines
     (messes up the formatting) in panic()
   - throws out a few per-server panic() functions
   - cleans up a tie-in of tty with panic()

merging printf() and panic() statements to be done incrementally.
2010-03-05 15:05:11 +00:00
Ben Gras
adf0b6fb26 No more E{SRC,DST}DIED errno's, replaced by EDEADSRCDST.
The callers don't care about the difference and had to check 3 error
codes instead of one.
2010-03-03 15:47:16 +00:00
Kees van Reeuwijk
1597e701a0 Remove useless variables and the computations on them. 2010-02-19 10:00:32 +00:00
Arun Thomas
b706112487 Incorporate bsdmake into buildsystem and reorganize libs 2010-02-16 14:41:33 +00:00
David van Moolenbroek
bdd4f5857f Fixes for truncate system calls:
- VFS: check for negative sizes in all truncate calls
- VFS: update file size after truncating with fcntl(F_FREESP)
- VFS: move pos/len checks for F_FREESP with l_len!=0 from FS to VFS
- MFS: do not zero data block for small files when fully truncating
- MFS: do not write out freed indirect blocks after freeing space
- MFS: make truncate work correctly with differing zone/block sizes
- tests: add new test50 for truncate call family
2010-02-09 08:12:37 +00:00
Kees van Reeuwijk
2ba237cd4e Fixed a number of uses of uninitialized variables by adding assertions
or other sanity checks, code reshuffling, or fixing broken behavior.
2010-01-27 10:23:58 +00:00
Thomas Veerman
ca9280e097 - Fix dangling symlink regression
- Make open(2) more POSIX compliant
- Add a test case for dangling symlinks and open() syscall with O_CREAT and
  O_EXCL on a symlink.
- Update open(2) man page to reflect change.
2010-01-21 09:32:15 +00:00
Cristiano Giuffrida
c5b309ff07 Merge of Wu's GSOC 09 branch (src.20090525.r4372.wu)
Main changes:
- COW optimization for safecopy.
- safemap, a grant-based interface for sharing memory regions between processes.
- Integration with safemap and complete rework of DS, supporting new data types
  natively (labels, memory ranges, memory mapped ranges).
- For further information:
  http://wiki.minix3.org/en/SummerOfCode2009/MemoryGrants

Additional changes not included in the original Wu's branch:
- Fixed unhandled case in VM when using COW optimization for safecopy in case
  of a block that has already been shared as SMAP.
- Better interface and naming scheme for sys_saferevmap and ds_retrieve_map
  calls.
- Better input checking in syslib: check for page alignment when creating
  memory mapping grants.
- DS notifies subscribers when an entry is deleted.
- Documented the behavior of indirect grants in case of memory mapping.
- Test suite in /usr/src/test/safeperf|safecopy|safemap|ds/* reworked
  and extended.
- Minor fixes and general cleanup.
- TO-DO: Grant ids should be generated and managed the way endpoints are to make
sure grant slots are never misreused.
2010-01-14 15:24:16 +00:00
Cristiano Giuffrida
d1fd04e72a Initialization protocol for system services.
SYSLIB CHANGES:
- SEF framework now supports a new SEF Init request type from RS. 3 different
callbacks are available (init_fresh, init_lu, init_restart) to specify
initialization code when a service starts fresh, starts after a live update,
or restarts.

SYSTEM SERVICE CHANGES:
- Initialization code for system services is now enclosed in a callback SEF will
automatically call at init time. The return code of the callback will
tell RS whether the initialization completed successfully.
- Each init callback can access information passed by RS to initialize. As of
now, each system service has access to the public entries of RS's system process
table to gather all the information required to initialize. This design
eliminates many existing or potential races at boot time and provides a uniform
initialization interface to system services. The same interface will be reused
for the upcoming publish/subscribe model to handle dynamic 
registration / deregistration of system services.

VM CHANGES:
- Uniform privilege management for all system services. Every service uses the
same call mask format. For boot services, VM copies the call mask from init
data. For dynamic services, VM still receives the call mask via rs_set_priv
call that will be soon replaced by the upcoming publish/subscribe model.

RS CHANGES:
- The system process table has been reorganized and split into private entries
and public entries. Only the latter ones are exposed to system services.
- VM call masks are now entirely configured in rs/table.c
- RS has now its own slot in the system process table. Only kernel tasks and
user processes not included in the boot image are now left out from the system
process table.
- RS implements the initialization protocol for system services.
- For services in the boot image, RS blocks till initialization is complete and
panics when failure is reported back. Services are initialized in their order of
appearance in the boot image priv table and RS blocks to implements synchronous
initialization for every system service having the flag SF_SYNCH_BOOT set.
- For services started dynamically, the initialization protocol is implemented
as though it were the first ping for the service. In this case, if the
system service fails to report back (or reports failure), RS brings the service
down rather than trying to restart it.
2010-01-08 01:20:42 +00:00
David van Moolenbroek
c8f8d69204 Fix MFS ftruncate crash (Bug#370, reported by Aki Goto) 2009-12-24 23:43:16 +00:00
Cristiano Giuffrida
1f5841c8ed Basic System Event Framework (SEF) with ping and live update.
SYSLIB CHANGES:
- SEF must be used by every system process and is thereby part of the system
library.
- The framework provides a receive() interface (sef_receive) for system
processes to automatically catch known system even messages and process them.
- SEF provides a default behavior for each type of system event, but allows
system processes to register callbacks to override the default behavior.
- Custom (local to the process) or predefined (provided by SEF) callback
implementations can be registered to SEF.
- SEF currently includes support for 2 types of system events:
  1. SEF Ping. The event occurs every time RS sends a ping to figure out
  whether a system process is still alive. The default callback implementation
  provided by SEF is to notify RS back to let it know the process is alive
  and kicking.
  2. SEF Live update. The event occurs every time RS sends a prepare to update
  message to let a system process know an update is available and to prepare
  for it. The live update support is very basic for now. SEF only deals with
  verifying if the prepare state can be supported by the process, dumping the
  state for debugging purposes, and providing an event-driven programming
  model to the process to react to state changes check-in when ready to update.
- SEF should be extended in the future to integrate support for more types of
system events. Ideally, all the cross-cutting concerns should be integrated into
SEF to avoid duplicating code and ease extensibility. Examples include:
  * PM notify messages primarily used at shutdown.
  * SYSTEM notify messages primarily used for signals.
  * CLOCK notify messages used for system alarms.
  * Debug messages. IS could still be in charge of fkey handling but would
  forward the debug message to the target process (e.g. PM, if the user
  requested debug information about PM). SEF would then catch the message and
  do nothing unless the process has registered an appropriate callback to
  deal with the event. This simplifies the programming model to print debug
  information, avoids duplicating code, and reduces the effort to print
  debug information.

SYSTEM PROCESSES CHANGES:
- Every system process registers SEF callbacks it needs to override the default
system behavior and calls sef_startup() right after being started.
- sef_startup() does almost nothing now, but will be extended in the future to
support callbacks of its own to let RS control and synchronize with every
system process at initialization time.
- Every system process calls sef_receive() now rather than receive() directly,
to let SEF handle predefined system events.

RS CHANGES:
- RS supports a basic single-component live update protocol now, as follows:
  * When an update command is issued (via "service update *"), RS notifies the
  target system process to prepare for a specific update state.
  * If the process doesn't respond back in time, the update is aborted.
  * When the process responds back, RS kills it and marks it for refreshing.
  * The process is then automatically restarted as for a buggy process and can
  start running again.
  * Live update is currently prototyped as a controlled failure.
2009-12-21 14:12:21 +00:00
David van Moolenbroek
d5dee93bee Support for larger disks.
- MFS, df(1), fsck(1), badblocks(8), de(1x) now compute the
  superblock's s_firstdatazone value if the on-disk value is zero
- mkfs(1) sets s_firstdatazone in the superblock to zero if the
  on-disk field is too small to store the actual value
- more agressive mkfs(1) inode number heuristic, copied from r5261
2009-12-21 11:20:30 +00:00
Thomas Veerman
958b25be50 - Introduce support for sticky bit.
- Revise VFS-FS protocol and update VFS/MFS/ISOFS accordingly.
- Clean up MFS by removing old, dead code (backwards compatibility is broken by
  the new VFS-FS protocol, anyway) and rewrite other parts. Also, make sure all
  functions have proper banners and prototypes.
- VFS should always provide a (syntactically) valid path to the FS; no need for
  the FS to do sanity checks when leaving/entering mount points.
- Fix several bugs in MFS:
  - Several path lookup bugs in MFS.
  - A link can be too big for the path buffer.
  - A mountpoint can become inaccessible when the creation of a new inode
    fails, because the inode already exists and is a mountpoint.
- Introduce support for supplemental groups.
- Add test 46 to test supplemental group functionality (and removed obsolete
  suppl. tests from test 2).
- Clean up VFS (not everything is done yet).
- ISOFS now opens device read-only. This makes the -r flag in the mount command
  unnecessary (but will still report to be mounted read-write).
- Introduce PipeFS. PipeFS is a new FS that handles all anonymous and
  named pipes. However, named pipes still reside on the (M)FS, as they are part
  of the file system on disk. To make this work VFS now has a concept of
  'mapped' inodes, which causes read, write, truncate and stat requests to be
  redirected to the mapped FS, and all other requests to the original FS.
2009-12-20 20:27:14 +00:00
David van Moolenbroek
f197bcb435 Allow servers to run with fewer privileges:
- allow non-root processes to get their own endpoint
- make alloc_contig() call sys_umap() only when requested
2009-12-02 10:06:58 +00:00
Tomas Hruby
d2c10fb85e inodes - using types with known size
- fixes a problem in inodes truct definitions. The original definitions use
  posix types. These types don't have well defined size. Therefore when
  compiling mkfs on a different system natively the inodes sizes do not match.
  This patch replaces the posix types with interger types of the same size and
  signedness as the original types in use.
2009-11-06 08:55:07 +00:00
David van Moolenbroek
f89388c241 Kernel, servers: remove unused proto.h definitions 2009-10-31 14:11:50 +00:00
David van Moolenbroek
bd30f2a988 Ground work for larger file systems, and miscellaneous fixes:
- MFS and mkfs(1) now perform extra sanity checks
- fsck(1) can now deal with inode tables extending beyond the file
  system's first 4GB
- badblocks(8) no longer writes out the superblock for no reason
- mkfs(1) no longer crashes when given no parameters
- more(1) no longer crashes when standard output is redirected
2009-10-26 13:35:39 +00:00
Tomas Hruby
4dad30937b Removed macros that depend on NOTIFY_FROM from servers and drivers. They
determine the information defined by these macros from the m_source field of the
notify message.
2009-09-29 18:47:56 +00:00
Ben Gras
a0d8cc0765 - No maximum block size any more.
- If allocation of a new buffer fails, use an already-allocated
   unused buffer if available (low memory conditions)
 - Allocate buffers dynamically, so memory isn't wasted on wrong-sized
   buffers.
 - No more _MAX_BLOCK_SIZE.
2009-09-21 14:47:51 +00:00
David van Moolenbroek
5a370afc8e MFS put_inode issues (reported by Maurizio Lombardi) 2009-07-22 08:54:28 +00:00
David van Moolenbroek
f80aaae86a MFS: remove some redundant code (reported by Maurizio Lombardi) 2009-07-12 13:57:34 +00:00
David van Moolenbroek
9808816c14 MFS fixes:
- Don't dereference NULL dir inode in advance_* (reported by Maurizio Lombardi)
- Fix potential inode reference leak in fs_slink_*
2009-07-11 10:36:57 +00:00
David van Moolenbroek
f76d75a5ec Various VFS and MFS fixes to improve correctness, consistency and
POSIX compliance.

VFS changes:
* truncate() on a file system mounted read-only no longer panics MFS.
* ftruncate() and fcntl(F_FREESP) now check for write permission on
  the file descriptor instead of the file, write().
* utime(), chown() and fchown() now check for file system read-only
  status.

MFS changes:
* link() and rename() no longer return the internal EENTERMOUNT and
  ELEAVEMOUNT errors to the application as part of a check on the
  source path.
* rename() now treats EENTERMOUNT from the destination path check as
  an error, preventing file system corruption from renaming a normal
  directory to an existing mountpoint directory.
* mountpoints (mounted-on dirs) are hidden better during lookups:
  - if a lookup starts from a mountpoint, the first component has to
    be ".." (anything else being a VFS-FS protocol violation).
  - in that case, the permissions of the mountpoint are not checked.
  - in all other cases, visiting a mountpoint always results in
    EENTERMOUNT.
* a lookup on ".." from a mount root or chroot(2) root no longer
  succeeds if the caller does not have search permission on that
  directory.
* POSIX: getdents() now updates directory access times.
* POSIX: readlink() now returns partial results instead of ERANGE.

Miscellaneous changes:
* semaphore file handling bug (leading to hangs) fixed in test 32.

The VFS changes should now put the burden of checking for read-only
status of file systems entirely on VFS, and limit the access
permission checks that file systems have to perform, to checking
search permission on directories during lookups. From this point on,
any deviation from that spceification should be considered a bug.
Note that for legacy reasons, the root partition is assumed to be
mounted read-write.
2009-05-18 11:27:12 +00:00
David van Moolenbroek
0ac1aaccca Limited support for nested FS->VFS requests during VFS->FS call.
- Changed VFS-FS protocol to only store OK or negative error code in
  m_type field of reply messages.
- Changed VFS to treat nonzero positive replies from FS as requests.
- Added backwards compatibility to VFS and MFS.
No protection of global data structures is provided in VFS, so many
VFS calls cannot be made safely by FS servers during many FS calls.
Use with caution (or, preferably, not at all).
2009-05-11 10:02:28 +00:00
Ben Gras
a38287067a disable scary looking debug messages. 2009-05-07 09:58:16 +00:00
Ben Gras
b7e23b70e8 - delete unused .h files
- slight code cleanup
 - neater exit procedure: exit when unmount
   message received and kill signal (from RS 'down' or
   reboot/shutdown) received (speed up unmount, but don't
   confuse VFS by exiting before/during unmount msg)
2009-04-27 16:13:51 +00:00
Ben Gras
60e7602aad When we receive a PROC_EVENT message from PM, exit nicely; avoids
annoying graceful RS timeout on unmount.
2009-04-27 12:02:31 +00:00
Arun Thomas
e9e1ae1cfc Move queue.h to include/sys so that it can be used elsewhere. Pull in
FreeBSD's r179210 queue.h.
2009-04-22 20:02:39 +00:00
Ben Gras
2d1c884e35 suppress these noisy, alarming messages. 2009-03-26 15:56:08 +00:00
Ben Gras
3cc092ff06 . new kernel call sysctl for generic unprivileged system operations;
now used for printing diagnostic messages through the kernel message
   buffer. this lets processes print diagnostics without sending messages
   to tty and log directly, simplifying the message protocol a lot and
   reducing difficulties with deadlocks and other situations in which
   diagnostics are blackholed (e.g. grants don't work). this makes
   DIAGNOSTICS(_S), ASYN_DIAGNOSTICS and DIAG_REPL obsolete, although tty
   and log still accept the codes for 'old' binaries. This also simplifies
   diagnostics in several servers and drivers - only tty needs its own
   kputc() now.
 . simplifications in vfs, and some effort to get the vnode references
   right (consistent) even during shutdown. m_mounted_on is now NULL
   for root filesystems (!) (the original and new root), a less awkward
   special case than 'm_mounted_on == m_root_node'. root now has exactly
   one reference, to root, if no files are open, just like all other
   filesystems. m_driver_e is unused.
2009-01-26 17:43:59 +00:00
Ben Gras
e96f86ed8c throw out debugging code. 2008-12-11 14:44:10 +00:00
Ben Gras
51fdce1d36 minor fixes 2008-11-19 14:10:33 +00:00
Ben Gras
c078ec0331 Basic VM and other minor improvements.
Not complete, probably not fully debugged or optimized.
2008-11-19 12:26:10 +00:00
Philip Homburg
3f23bca404 Removed defines not needed by mfs (XPIPE, XLOCK, XPOPEN, XSELECT, DUP_MASK). 2008-02-21 16:22:36 +00:00
Ben Gras
2876d5c4ba Optimization in searching for new zones to allocate contributed
by Jens de Smit.
2008-02-06 15:05:57 +00:00
Ben Gras
e8aec69c7b tweak to panic functions of mfs and vfs.
. print newline
  . when recursive panic detected, don't simply return, confusing
    the caller, but print a diagnostic and exit
  . don't call sys_exit as this may confuse PM; it should be OK
    to call PM exit() nowadays.
2007-10-23 14:17:51 +00:00