Chapter 8 Connection Management
This chapter describes how omniORB manages network connections.
8.1 Background
In CORBA, the ORB is the `middleware' that allows a client to invoke an operation on an object without regard to its implementation or location. In order to invoke an operation on an object, a client needs to `bind' to the object by acquiring its object reference. Such a reference may be obtained as the result of an operation on another object (such as a naming service) or by conversion from a stringified representation. If the object is in a different address space, the binding process involves the ORB building a proxy object in the client's address space. The ORB arranges for invocations on the proxy object to be transparently mapped to equivalent invocations on the implementation object.
For the sake of interoperability, CORBA mandates that all ORBs should support IIOP as the means to communicate remote invocations over a TCP/IP connection. IIOP is asymmetric with respect to the roles of the parties at the two ends of a connection. At one end is the client which can only initiate remote invocations. At the other end is the server which can only receive remote invocations.
Notice that in CORBA, as in most distributed systems, remote bindings are established implicitly without application intervention. This provides the illusion that all objects are local, a property known as `location transparency'. CORBA does not specify when such bindings should be established or how they should be multiplexed over the underlying network connections. Instead, ORBs are free to implement implicit binding by a variety of means.
The rest of this chapter describes how omniORB manages network connections and the programming interface to fine tune the management policy.
8.2 The Model
omniORB is designed from the ground up to be fully multi-threaded. The objective is to maximise the degree of concurrency and at the same time eliminate any unnecessary thread overhead. Another objective is to minimise the interference by the activities of other threads on the progress of a remote invocation. In other words, thread `cross-talk' should be minimised within the ORB. To achieve these objectives, the degree of multiplexing at every level is kept to a minimum.
On the client side of a connection, the thread that invokes on a proxy object drives the IIOP protocol directly and blocks on the connection to receive the reply. On the server side, a dedicated thread blocks on the connection. When it receives a request, it performs the up-call to the object and sends the reply when the up-call returns. There is no thread switching along the call chain.
With this design, there is at most one call in-flight at any time on a connection. If there is only one connection, concurrent invocations to the same remote address space would have to be serialised. To eliminate this limitation, omniORB implements a dynamic policy---multiple connections to the same remote address space are created on demand and cached when there are concurrent invocations in progress.
To be more precise, a network connection to another address space is only established when a remote invocation is about to be made. Therefore, there may be one or more object references in one address space that refer to objects in a different address space but unless the application invokes on these objects, no network connection is made. The maximum number of connections opened to another address space is 5 by default. Since 2.6.0, this parameter can be changed by setting the variable omniORB::maxTcpConnectionPerServer before calling ORB_init().
It is wasteful to leave a connection open when it has been left unused for a considerable time. Too many idle connections could block out new connections to a server when it runs out of spare communication channels. For example, most Unix platforms have a limit on the number of file handles a process can open. 64 is the usual default limit. The value can be increased to a maximum of a thousand or more by changing the `ulimit' in the shell.
8.3 Idle Connection Shutdown and Remote Call Timeout
Inside the ORB, a thread is dedicated to scan for idle connections. The thread looks after both the outgoing connections and the incoming connections.
When a connection is idle for a period of time, the connection is shutdown. Similarly, if a remote call has not completed within a defined period of time, the connection is shutdown and the ORB will return COMM_FAILURE to the client.
How often the internal thread scans the connections is determined by the value of the scan granularity. This value is defaulted to 5 seconds and can be changed using the command-line option -ORBscanGranularity or using the omniORB::scanGranularity call. Notice that this value determines the precision the ORB is able to keep to the value of the idle connection or remote call timeout.
How long the ORB will wait before it shuts down an idle connection is determined by the idleConnectionPeriods. There are separate values for incoming and outgoing connections. The default values are 180 and 120 seconds for incoming and outgoing connections respectively. These values can be changed using the command-line options -ORBinConScanPeriod and -ORBoutConScanPeriod. They can also be controlled by the omniORB::idleConnectionScanPeriod() call.
Similarly, how long the ORB will wait for a remote call to complete is determined by the parameter clientCallTimeOutPeriod for the client side and the serverCallTimeOutPeroid for the server side. By default calls will not timeout on either the client or server side.
The timeouts can be changed using the omniORB::callTimeOutPeriod() call, or with the command line options -ORBclientCallTimeOutPeriod and -ORBserverCallTimeOutPeriod.
The APIs are documented in include/omniORB3/omniORB.h.
class omniORB {
public:
static void scanGranularity(CORBA::ULong sec);
static CORBA::ULong scanGranularity();
enum idleConnType { idleIncoming, idleOutgoing };
static void idleConnectionScanPeriod(idleConnType direction, CORBA::ULong sec);
static CORBA::ULong idleConnectionScanPeriod(idleConnType direction);
enum callTimeOutType { clientSide, serverSide };
static void callTimeOutPeriod(callTimeOutType direction, CORBA::ULong sec);
static CORBA::ULong callTimeOutPeriod(callTimeOutType direction);
};
The scan can be disabled completely by setting the scan granularity to 0.
8.4 Interoperability Considerations
The IIOP specification allows both the client and the server to shutdown a connection unilaterally. When one end is about to shutdown a connection, it should send a closeConnection message to the other end. It should also make sure that the message will reach the other end before it proceeds to shutdown the connection.
The client should distinguish between an orderly and an abnormal connection shutdown. When a client receives a closeConnection message before the connection is closed, the condition is an orderly shutdown. If the message is not received, the condition is an abnormal shutdown. In an abnormal shutdown, the ORB should raise a COMM_FAILURE exception whereas in an orderly shutdown, the ORB should not raise an exception and should try to re-establish a new connection transparently.
omniORB implements these semantics completely. However, it is known that some ORBs are not (yet) able to distinguish between an orderly and an abnormal shutdown. Usually this is manifested as the client in these ORBs seeing a COMM_FAILURE occasionally when connected to an omniORB server. The work-around is either to catch the exception in the application code and retry, or to turn off the idle connection shutdown inside the omniORB server.
8.5 Connection Acceptance
omniORB provides the hook to implement a connection acceptance policy. Inside the ORB runtime, a thread is dedicated to receive new connections. When the thread is given the handle of a new connection by the operating system, it calls the policy module to decide if the connection can be accepted. If the answer is yes, the ORB will start serving requests coming in from that connection. Otherwise, the connection is shutdown immediately.
There can be a number of policy module implementations. The basic one is a dummy module which just accepts every connection.
In addition, a host-based access control module is available on Unix platforms. The module uses the IP address of the client to decide if the connection can be accepted. The module is implemented using tcp_wrappers 7.6. The access control policy can be defined as rules in two access control files: hosts.allow and hosts.deny. The syntax of the rules is described in the manual page hosts_access(5) which can be found in appendix A. The syntax defines a simple access control language that is based on client (host name/address, user name), and server (process name, host name/address) patterns. When searching for a match on the server process name, the ORB uses the value of omniORB::serverName. ORB_init() uses the argument argv[0] to set the default value of this variable. This can be overridden by the application with the -ORBserverName <string> command line argument
The default location of the access control files is /etc. This can be overridden by the extra options in omniORB.cfg. For instance:
# omniORB configuration file - extra options
GATEKEEPER_ALLOWFILE /project/omni/var/hosts.allow
GATEKEEPER_DENYFILE /project/omni/var/hosts.deny
As each policy module is implemented as a separate library, the choice of policy module is determined at program linkage time. For instance, if the host-based access control module is in use:
% eg1 -ORBtraceLevel 2
omniORB gateKeeper is tcpwrapGK 1.0 - based on tcp_wrappers_7.6
I said,"Hello!". The Object said,"Hello!"
Whereas if the dummy module is in use:
% eg1 -ORBtraceLevel 2
omniORB gateKeeper is not installed. All incoming are accepted.
I said,"Hello!". The Object said,"Hello!"