This is an implementation of synchronous Send/Receive/Reply message passing, and is meant to be API compatible with QNX4 (probably a trademark) from QSSL, ( http://www.qnx.com ), one of the best and most interesting OSs available.
The current state is "Beta". Its been used to successfully run a fairly large set of QNX application under Linux, but hasn't been used extensively. I welcome any comments, and hope it will be useful for anybody porting code from QNX to Linux, or wanting to experiment with S/R/R under Linux, perhaps even with the intention of porting to QNX eventually.
Conceptually, this code should be portable to any Linux version, and even to other Un*x systems (potentially). However, current development is with Linux kernels in the 2.2.* series. Previous development was with the 2.0.37 kernel. The module might still run on the 2.0.* versions, but unless there's a lot of interest I don't expect to continue to try and test it on a kernel thats so different from the 2.2.* series.
Update: With the addition of the user-space interrupt functionality, we are getting more dependent on the latest kernel versions. Forget about trying to run on a 2.0 system, and please send patches if you need to modify anything for a 2.2 system.
Notes on installation are still a little brief. I'm not entirely sure how users would prefer this kind of software to be installed. See INSTALL for a summary.
Sam Roberts is the original author of this code. Andrew Thomas contributed parts prior to the first release. It has since been modified heavily by a number of people.
The current maintainer of this module is Andrew Thomas.
George Boudreau, george_boudreau@sympatico.ca
Charlie Brady
Milind Changire, milindc@telesoftnet.com
Brad Cowie, brad@comtech.co.nz
Sylvain Falardeau, sylvainf@xsoli.com
John Green, jgreen@ionus.com.au
Aditi Hatwalne, aditi.hatwalne@gmail.com
Len Meakin, lmeakin@actemium.co.uk
Sam Roberts, sroberts@uniserve.com
Richard Romano, raromano@ix.netcom.com
Frederic Rossi, frederic.rossi@savoirfairelinux.com
Alessandro Sala, alessandro.sala@gemmo-sistemi.it
Roland Stahn
Andrew Thomas, andrew@cogent.ca
Gerald P. Wilson, wilson@wilsonworks-eng.com
See CHANGES for attributions.
Because it's interesting, and a good way to learn more about Linux, QNX, and OS architecture. It has also been useful. Cogent uses this module to support all of our software under Linux.
This module has some relationships with the SIMPL project ( http://www.holoweb.net/~simpl ) and I subscribe to the simpl mailing list. FC Software is using shared memory and Unix pipes to implement S/R/R message passing. So far our focuses have been different. My interests are making the API as close as possible to QNX's (to simplify porting of code) and learning more about the QNX and Linux kernels. FC is interested in supporting their ongoing work under Linux. See TODO in the distribution for the directions I'm taking this code.
Some differences you may bump into are that with this module you send and
receive messages by process id, whereas with FC's you have to use the return
value of a qnx_name_attach()
(so Send(getppid(), ...) is not allowed).
The implementations seem similar in performance. I haven't ported my test app to their API yet, but it does run (without change) under both QNX and Linux (with this module). QNX is faster, of course, but not by an order of magnitude.
You can join the SIMPL mailing list by sending mail to:
simpl-mail-request@lists.sourceforge.net
with a subject line of "subscribe".
this serves the same function as QNX's _select_receive hack
just _PPF_INFORMED (notification of process death)
(POSIX.4 timers with signal or proxy notification)
(POSIX.4 clock functions)
qnx_vc_attach()
, getnid()
, etc.A number of other QNX APIs that relate to networking are implemented as stubs since in the absence of networking they degenerate to nothing, or APIs already listed as implemented above.
And finally, qnx_prefix_[attach/detach/locate] are possible, but without an
open()
that uses prefixes they are useless. You could re-write the open()
function in the glibc shared library like some of the Linux user-space
filesystems have... any takers? ;-)
The module is copyright (C) 2000 by Cogent Real-Time Systems Inc. It is distributed under the same conditions as the Linux Kernel, the GNU General Public License. The API library (srr_lib.c, srr.h) is also distributed under the GNU General Public License. See COPYING for the details of this licence.
In the past, the API library has been distributed under the LGPL, which allowed you to use the library in commercial software. Due to the rising cost and effort of maintaining this library, we have modified the library licensing starting with version 1.4.1 of the module to the GPL. This makes the license inappropriate for use in commercial software, since the GPL requires that the commercial software must therefore also be covered by the GPL. If you wish to use the SRRIPC module in a commercial product, please contact Cogent for alternate licensing terms.
Please do not interpret this change in license as a "money grab". The purpose of this change is to allow us to devote the time and resources to the SRR module that it deserves. If you use this module for non-commercial reasons, or make the source code available to anything developed with this module, then it remains free. If you are making money from this module, then you already understand the nature of the commercial food chain, and understand the cost of maintaining a high-quality product. We ask only that you license the fruit of our labour as you ask others to license yours.
The SRR Module software and documentation is offered "as is", without warranty of any kind. Neither Cogent Real-Time Systems, Inc. nor any related third parties shall be liable for the cost of any direct, indirect, incidental, or consequential damages whatsoever, such as, but not limited to, loss of anticipated benefits or profits resulting from the use of this software. The entire risk as to the results and performance of this software is assumed by the user.
The ideas behind this implementation of SRR under Linux are simple:
extend the kernel by writing a loadable kernel module,
register with that module by opening it's dev file "/dev/srr" (either
explicitly with SrrReg()
, or implicitly by making one of the library
calls),
map all (pseudo-)system calls to an ioctl()
to the driver -- like
other system calls the ioctl()
can block, not block, perform arbitrary
reads/writes into the process' address space, etc., everything that
the QNX S/R/R API appears to do.
The ioctl()/module implementation should look familiar to a QNX systems
programmer, it is essentially an inversion of a QNX IO resource manager. In
QNX, S/R/R is used to pass structures holding the arguments to Posix system
calls (read, write, ioctl!) to the IO manager, in Linux I use an ioctl()
to
pass structures holding the arguments to QNX system calls to the kernel
module.
There are a few wrinkles:
1) I don't know how, in Linux, to access the address space of anything but the currently running process. In fact, it is not generally possible: the process's memory may be swapped to disk. This forces me to buffer all messages in kernel space before causing a context switch to the process for which the message is destined. Unfortunately this means that all data must be copied from sender to kernel to receiver, whereas under QNX message passing results in a single copy, directly from the senders address space to the receivers. Oh well.
Update: It IS possible to access the address spaces of two user processes if you lock the sender's buffers into memory before the sender sleeps. We even have a patch of this against an old version of the SRR module, but it is too different from the current code to be easily integrated. Any takers?
2) There's a problem with pre-2.2 kernels involving the use of fork()
when
the parent exits, and the child never makes a SRR library call. This problem
does not exist in the 2.2.0 and later kernels.
In Unix, when a process forks its file descriptors are duplicated in
the child. In Linux this dup()
is done without the co-operation of
the driver to which the fd points, usually a good thing. However, the
only way my module can detect the death of a process is by that
process closing the fd it uses to communicate with me. I need to know
about process death to release any processes that may be blocked on
it. The close()
on process exit/ death is guaranteed when a single
process has a copy of the fd.
However, when a process forks, both the parent and the child have the
fd open, and it will not closed until both processes close it. This
can happen explicitly via close()
, or on exit. This can lead to
scenarios where the parent exits, but the child still holds the fd
open, so the kernel module never realizes a process has exited. I try
and compensate for this by detecting when a process uses any of the
APIs through a fd that it did not originally open, then forcing that
process to open its own fd. Unfortuneately, if the child never makes
any API calls, I will never know the parent died.
In Linux 2.2 closing a fd is indicated by flush being delivered to the module. If it is the last close the flush is followed by a release. This is sufficient information that the module operates correctly under all conditions. See tfork.c in the distribution for an example demonstrating the bug. The example also demonstrates that the bug does not exist when using the module under a 2.2 or later kernel.
3) Need to do some tests, QNX documentation is not quite complete enough to determine what error numbers should be returned for certain strange errors, sending to oneself, attaching a proxy to a proxy, etc. You'll get an error, but perhaps not the correct errno.
Note: I have done this now, and discovered some interesting things
about QNX. For instance, contrary to the docs, you can call
qnx_proxy_detach()
on <any> proxy, assuming you have sufficient
priviledges, not just on proxies attached to you. I still have to do a
reread of my code to make sure that I'm doing the correct things in
these cases.
You can find API documentation by going to
http://www.cogent.ca/Software/SRR.html and following the
documentation link. It's not exactly rich documentation, but if you
want more depth, take a look at the QNX4 documentation available
through www.qnx.com. Generally, SRRIPC works like QNX4, except that a
process doesn't exist for the purposes of sending to it or receiving
from it until it has registered with the module. Calling any API
function, or SrrReg()
, will cause the process to be registered with
the module. Exiting, or calling SrrUnreg()
witll cause the process to
be exited from the perspective of s/r/r, causing ESRCH for processes
blocked on the deregistering process.
See the 4 simple example files recv.c, send.c, reply.c, and wait.c for simple examples of how to call, and tserver.c for a more complex example.
Look in /proc/srr for info on names, task states, proxies, and timers.
The QNX documentation for Send/Receive/Reply is on-line, check out http://www.qnx.com/literature/qnx_sysarch/microkernel.html#KERNELIPC . It gives an excellent description of QNX IPC.
The SRRIPC module now includes an experimental interrupt handling mechanism that essentially allows you to handle interrupts in user-space. The user process registers for an interrupt with the SRRIPC module and provides a proxy that should be triggered when the interrupt occurs. The module triggers the proxy once per interrupt.
Most hardware cards need a small amount of processing performed when the interrupt occurs, and while interrupts are still disabled. This is particularly true of PCI cards where interrupts are level triggered. In order to accomodate this, the SRRIPC module contains a simple byte-code interpreter that allows you to create a program that will run at interrupt time. It allows only a linear flow of control, giving you the opportunity to read and write memory, read and write I/O ports, and perform simple mathematical operations on 1, 2 and 4 byte integers. If you need something more complex than that, you will have to extend the language. For most hardware cards, this is sufficient.
The intention of the interrupt handling mechanism is that only the most critical processing be done in the interrupt handler itself, where interrupts are disabled. All of the major work is performed when the proxy is delivered to the user-space process. Remember that a user process can still map device memory and interact with I/O ports. You can think of the delivery of the proxy as the start of the "bottom half handler" for your interrupt. From the perspective of real-time response, it is generally a good thing to have as much of your interrupt handling code be scheduled (by the process scheduler) rather than occur at interrupt time. This allows higher-priority tasks to run if necessary. Strictly speaking, since Linux does not use a real-time scheduler, and since there is no priority mechanism in the SRRIPC module, this reasoning is more forward-looking that actually valuable at this point.
In any case, user-space interrupt handlers are a huge boon to the average driver-writer, particularly those porting code from QNX4 (or even QNX6).
For documentation of the opcodes for the interrupt handler's bytecode interpreter, look in mod/bytecode.txt.