Documentation
Flush Spread is an extension to the Spread Wide Area Group
Communication System. Flush Spread and Spread are extremely similiar group communication
systems (GCSs), in the services that they provide, and in their general interface. Therefore,
reading Spread's documentation before
reading Flush Spread's documentation is highly recommended to get a general grasp of group
communication and to fill in any holes in this documentation :).
There are two main differences between Spread and Flush Spread:
(1) Spread provides extended virtual
synchrony GCS semantics while Flush Spread provides view synchrony GCS semantics, and (2) Spread
provides open-group semantics while Flush Spread only provides closed-group semantics.
Extended Virtual Synchrony (EVS) vs. View Synchrony Semantics (VS)
The main difference between these two models is how they handle view/membership changes. In an EVS
system, membership changes occur without a client's intervention, whereas in a VS system a client
must give its permission before a new view/membership can be installed. Of course, in "real-life" a
VS system cannot truly hold-off membership changes (as other clients can crash, network can
partition, etc.), but what it can do is insulate a client from these changes and not allow new
members (who either joined or merged) or their messages to be presented to the client without its
permission.
In a VS system, when an underlying membership change occurs (another member joins/leaves/disconnects,
network parititions or merges) the GCS generates a flush request message indicating that the current
view is out-of-date and requesting permission to install a new view. The client is still allowed to
send messages at this point, and they may be able to receive messages from surviving members. When
the client is ready, it flushes the group, which gives the GCS permission to install a new view.
After flushing, a client is not allowed to send any messages to the flushed group until they receive
the new view of the group.
In a EVS system, when an underlying membership change occurs the GCS figures out the new membership
and delivers a new view to the client in its normal stream of messages.
In both systems, using the safety guarantees of the different message types and the virtual synchrony
property (i.e. transitional sets) the client can infer some information about what other members have
seen without further communication.
Although the difference in these models seems slight, it can lead to quite astonishing differences
in performance and ease of programming. In general, the EVS model is faster and scales better with
respect to membership changes, especially for simple group changes such as joins and leaves. But
this performance comes at a price, namely that the client has less control over what is transpiring
and therefore in general it is harder to program algorithms under EVS than under VS. For example,
under EVS the sender doesn't know which other clients might receive its messages until the message
is delivered back to the sender. Also, membership changes can come fast and furious and your
algorithm must be tolerant to every combination of such changes. Under VS, the client knows exactly
which other clients might receive its message when it sends -- so the application doesn't
have to worry about some arbitrary client receiving its messages and mis-interpreting them, for
example. Also, membership changes only occur as fast as the applications want them to, although
several underlying memberships may be collapsed into one. So they are generally easier to handle
and track.
Open-Group vs. Closed-Group Semantics
Open-group semantics allows clients to perform group specific calls, such as multicasts, without
being a member of the group in question. Closed-group semantics requires a client to be a full
fledged member of a group (see below) in order to perform group specific calls. In general, a
closed-group GCS is much more picky than an open-group system about a client's state and the
operations they perform with respect to any particular group.
Under Spread, a client can call group specific calls such as join, leave, and *cast (multicast and
its variants) pretty much whenever they want regardless of their membership in that group. Also,
under Spread the guarantees of multicast messages span differerent groups. For example, if a sender
sends two FIFO messages to two different groups and a receiver receives both messages, they will
always receive the first and then the second FIFO message.
Under Flush Spread, a client can make group specific calls such as join, leave, flush, and *cast
only when in the proper membership state with respect to that group. For example, leave, flush, and
*cast calls can only be made when the client is a full-fledged member of the group. A full-fledged
member is a client that has joined a group, received a membership including them in the group and is
not in the process of leaving the group. Join calls can only be made when the client is not a
member of the group (either never joined it or whose most recent membership for that group was a
self-leave) and not already in the process of joining that group.
Also under Flush Spread, *cast and flush calls (i.e. FL_flush) are restricted depending on the group
membership state. Flush calls can only be made ONCE per view/membership in response to
receiving a flush request message. Once a client flushes a group they are not allowed to *cast to
that group until they receive the next view/membership for that group. Flush Spread also does not
support cross group message semantics. For example, if a sender sends two FIFO messages to two
different groups and a receiver receives both messages, they may receive the first and then the
second OR vice-versa.
C/C++ Documentation
The following documentation explains the user level C interface to Flush Spread as specified in the
header file fl.h. This documentation assumes a knowledge of the ideas,
discussed above, of view synchronous group communication.
- Message Types for Data and Membership Messages
- Message Type Access Functions
- Error Return Codes
- Miscellaneous Constants
- Flush Spread C Function Interface