Study Notes

Kernel: fs/dlm/lowcomms-top.c
The patch adds a TCP based communications layer to the DLM which is compile time selectable. The existing SCTP layer gives the advantage of allowing multihoming, whereas the TCP layer has been heavily tested in precious versions of the DLM and is known to be robust and therefore can be used as a baseline for performancing testing. Internode communication using SCTP. This level is not aware of locks or resources or other dlm objects, only data buffers. These functions also batch(and extract) lots of small messages bound for one node into larger chunks.

Why current node use SCTP?
This is the low-level comms layer. It’s responsible for sending/receiving message from other nodes in the cluster.Cluster nodes are referred to by their nodeids, nodeids are simply 32bit numbers to the locking module. If they need to be expanded for the cluster infrastructure then that is its responsibility.It is this layers responsiblity to resolve there into IP addresses or whatever it needs for internode communications.

The comms level is two kernel threads that deal mainly with the receiving of messages from other nodes and passing them up to the mid-level comms layer(which understand the message format) for execution by locking core, and a send thread which does all the setting up of connections to remote nodes and the sending of data. Threads are not allowed to send their own data because it may cause them to wait in times of high load. Alao, this way, the sending thread can collect together message bound for one node and send them in one block.

From David Tergland: I don’t see any problem with the recv thread executing the locking code on behalf of remote processes as the locking code is short, efficient and nevel(well, hardly ever) waits.

In my way of thinking, you’re well on the way.

TCP Limitation:
TCP(rfc793) has performed immense service as the primary means of reliable data transfer in IP networks.However, an increasing number of recent applications have found TCP too limiting. The limitation which means have wished to bypass including the following:

1.TCP provides both reliable data transfer and strict order-of-transmission delivery of data. Some applications need reliable transfer without sequence maintenance, while other would be satisfied with partial ordering of the data. In both of these cases the head-of-line blocking offerred by TCP causes unnecessary delay.
2.The stream oriented nature of TCP is often an inconvenience. Applications must add their own record marking to delineate their messages, and must make explict use of the push facility to ensure that a complete message is transferred in a reasonable time.
3.The limitation scope of TCP sockets complicates the task of providing highly-available data transer capability using multi-homed hosts.
4.TCP is relatively vulenerable to denial of service attacks such as SYN attacks.(Dos).

With message framing(SCTP and UDP), the boundaries in which messages are communicated through a socket are preserved, this means that if a client sends 100 bytes to a server following by 50 bytes, the server will read 100 bytes and 50 bytes, respectively, for two reads.UDP also operaters in this way, which makes it advantage for message-oriented protocols.

In contrast, TCP operates in a byte-streams fashion, without framing, a peer may receive more or less than was sent(splitting up a write or aggregating multiple writes into a single read). This behavior requires that message-oriented protocols operating over TCP provide data-buffer and message framing within their application layer.

Message in SCTP are transferred reliably but not necessary in the desire order. TCP guarantees that data is delivered in order(which is good thing, considering TCP is a stream protocol). UDP guarantees no ordering. But you can also configure streams within SCTP to accept un-ordered message if desired.

Detail for SCTP: http://www.ibm.com/developerworks/linux/library/l-sctp