FILE: CCF/CCFcctl/README

                          CCF
            Collaborative Computing Frameworks

                    Emory University
                    Atlanta, GA, USA
                       June 1999

    CCF is a software system that supports collaborative, 
distributed, computer-based problem solving in the natural 
sciences, business, government, and in educational environments.  
The goal is to evolve a virtual environment for distributed 
computation that supports integrated human AV communication, 
high performance heterogeneous computing and distributed data 
management facilities. CCF is a research project at Emory University
involving the Math/Computer Science and Chemistry departments.
Recently, the Computer Science department of the University of
Reading (UK) has joined the project.

DISCLAIMER:

    This is alpha release 2.00 of CCF -- Collaborative 
    Computing Frameworks.

    This software is provided as is with no warranty expressed 
    or implied. We hope you find it useful, but we won't be held 
    responsible for any damage that may occur from reading, 
    compiling, installing, using, or even thinking about it.


LICENSE:

    CCF is Copyright (C) 1996 by Emory University except for the 
    code in directories GSM, LPC, LPC10 in the CCFaudio directory 
    and is distributed under the terms of GNU General Public 
    License (GPL) and the GNU Library General Public License (LPGL). 
    The files COPYING and COPYING.LIB in each directory will tell 
    exact licensing restrictions. 

    This package is free software; you can redistribute it 
    and/or modify it under the terms of the GNU General Public 
    License as published by the Free Software Foundation; either 
    version 2 of the license, or (at your option) any later version.


CREDITS:

    CCF was created by Vaidy Sunderam, Injong Rhee, 
    Alan Krantz, Shun Yan Cheung, Julie Sult, 
    Soeren Olesen, Paul Gray, Phil Hutto, Sarah Chodrow, 
    Michael Hirsch, Ted Goddard, Mic Grigni, N. Balaguru,
    Jim Nettles, Luigi Marzilli, Sue Onuschak, Scott Childs, 
    Kevin Williams.

    The project is internationaly collaborating with Roger Loader
    and James Pascoe of the department of Computer Science at The 
    University of Reading (UK).
    
    The CCF project is sponsored by the U.S. National 
    Science Foundation under the multidisciplinary challenges 
    initiative.   

    CCF currently supports ten platforms: Mandrake Linux 6.5, 
    Red Hat 6.0 and 6.2, SuSe Linux 6.4, TurboLinux 6.0, SunOS-5.7, 
    SunOS-5.6, SunOS-5.5.1 and IRIX 6.2. The SunOS-5.7 version is the 
    most thoroughly tested.

OVERVIEW:

CCTL is the CCF multicast transport layer. It provides a reliable multicast
service offering a variety of qualities of service (totally ordered, reliable,
unreliable, etc.). CCTL is used by ccsm and ccfx and all CCF tools. 

COMMON PROBLEMS:

DEVELOPER NOTES:

%% edited 8/1/00 by James Pascoe

The new distribution of the CCF comes with a much improved version of
CCTL. In the first instance, fault tolerance has been incorporated which
handles arbitrary non session owner member crashes. Furthermore, the leave 
protocols have been improved and now session members are able to join and
leave sessions in a much cleaner fashion. Note also that cctlwp and ccfsns
are much more synchronised with the system and are themseleves much more
resiliant. We have also fixed a number of other bugs.

CCTL can not at present deal with a session owner crash (although the 
session owner is able to leav at any time). Furthermore, partitioning
and remerging is not supported. This is something we are currently working 
on.

For more information on the fault tolerance mechanisms in CCTL, see 
CCF/doc/README.ft.

%% edited 8/19/96 by Injong Rhee

The new CCTL API is in /aux/10/ccf/cctl/include/cctl.h. It contains 
some descriptions of each function call. Basically as of today,
it supports various type of QOS include ATOMIC multicast. However,
this version does not handle virtual synchrony yet so the dynamic
joining and leaving may give you some problem. I am debugging 
the version with virtual synchrony, but have not gotten around
to fix all the bugs yet. But it is expected to be released next
week or so. Right now, we need some documentation on how to use
all these calls, so I am going to jot down some basic stuffs here
and if you need to know more about them, please feel free to
see me and we can have a little chat.

The first thing you need to do is to set an environment
variable "CCTL_WP_HOST_NAME" if you would like to use your 
own white page server for your testing purpose, and then
start cctl/bin/cctlwp in the machine specified in CCTL_WP_HOST_NAME.  
Note that if someone is already running wp in the machine, then
you don't need to start your own wp.
If you don't set CCTL_WP_HOST_NAME, then it will defaultly
connect to wp in ccf.mathcs, which I will be crashing all the time :).

The good thing about this version of cctl is that you don't have to
mess up with cctl daemon which is completely invisible and comes_
and_goes_away as your session. Every thing is done automatically.

To use cctl, in your process (you should do this for every process
that joins a session), each process shoudl call cctl_init 
before it does any thing with cctl.

 cctl_init();
 
Next, you create or join a session using the following fn call.
Say you want to join a session called "red" -- vss's favorite 
session :).

 ret = cctl_create_session("red",0,NULL);

   ret will have zero if it is successful. If not, it will return
   a negative error code (you can look up the meaning of each 
   code in <cctl_ecode.h>.


This call automatically gives the process a default channel of which
id is 0, and qos is reliable. So you can not send a message to the channel
using the following fn.

To open another reliable message channel (other than the default channel)
called "test", you can now call 

 ret = cctl_create_channel("test", CCTL_RELIABLE,NULL);

This call create a new channel test if the channel does not exist in
the session, and joins existing channel called test. The qos can be
one of the following: CCTL_RELIABLE, CCTL_ATOMIC and CCTL_UNRELIABLE.

CCTL_RELIABLE guarantees that messages are delivered to every process
in FIFO order in the channel to which the messages are sent as long as the 
prcess is reachable and non-faulty. 

CCTL_ATOMIC guarantees that messages are gloabally ordered with respect
to every message sent by every process in the channel regardless of
who sent the messages. It also guarantees FIFO. But it does not
handle message fragmentation so you cannot send more than 8000 bytes
of information. 

CCTL_UNRELIABLE guarantees that messages are received in FIFO (not
necessary consecutive order). But the messages can be lost.

cctl_create_channel returns a (positive) number which we call channel 
id if the call is successful. If not, it returns an error.

To send a message "msg" of size "sz" to every proc in a channel with 
channel id chid, and the process wants to wait until a message is 
delivered, you call

 ret = cctl_send(chid, CCTL_ALL, sz, msg, CCTL_SYNC, 0); 

ret will return 0 if it is successful, and if not, it returns an error code. 
The actual API for send is the following:

 int cctl_send(int chid,int dest,int msg_size, char *msg, int mode, int tag)

chid is channel id.  Details about each arg can be found in <cctl.h>. 
Coversely,  to receive a message,

 ret = cctl_recv(chid, &srcid, sz, msg, CCTL_ASYNC,0);

ret will return the size of message received in "msg", and
sz is the size (number of bytes) of msg.  

Currently, I have not implemented "src" and "tag" filtering yet, which
I will next week. If you need filtering, you might want to wait until
next week or use own filtering scheme. 

If you need more information, please see me and I don't mind any question
you have about cctl.

Enjoy,
Injong

-------------------
Update (10/9/96):

   
  int cctl_recv(int chid,
                int *src,  /* sender id */
                char *msg, /* msg buffer */
                int bufLen,/* size of buffer in bytes */ 
                int *msgLen,/* size of the message returned in bytes */
                int flags,  /* CCTL_SYNC or CCTL_ASYNC */
                int srcTag, /* CCTL_ALL or  sender id */
                int tag);   /* tag */
 
Now cctl_recv returns either 0 or negative number. A negative
number indicates an error. The description about error code can
be found in cctl_ecode.h.
 
In the following calls, the msg buffer and the size of buffer are
switched their positions. 
 
  int cctl_send(int chid, int dest, char *msg, int len, int flags, int tag);
  int cctl_urecv(int chid, int *src, char *msg, int len, int flags);
  int cctl_usend(int chid, int dest, char *msg, int len, int flags);
 

---------------------
Update (10/23/96)

The api's for cctl_select, cctl_create_session and cctl_create_channel has
been changed. A new function call "cctl_get_ch_mbrship" is implemented to
return the number of mebers in a channel and membership bit vectors.

1. CCTL_SELECT

  int cctl_select(int nfds, /* max fd */
                fd_set *fdset, /* tcp or udp socket descriptor sets */
                int nchids,  /* max chid */
                cctl_chid_set *chids, /* channel id sets */
                struct timeval *timeout); /* timeout */
 

fdset and nfds are the same parameters being passed to a unix select call.  
To set fdset, you use FD_SET, FD_ZERO, FD_CLR, etc. "nfd" contains the 
maximum socket descriptor that you listen to those in fdset.

chids is the set of channel ids that you listen to.  chids can be set 
by CHID_SET, CHID_ZERO and CHID_CLR. nchids is the maximum channel id
in chids.

cctl_select returns the number of sockets and channels that
are currently having messages to be received.

When cctl_select returns, fdset and chids  contain the bit vectors
for the sockets and channels with messages to be read. To see whether
a socket has a message, use FD_ISSET. Likewise, to see whether
a channel has a message, use CHID_ISSET.


The following example program shows the usage of cctl_select.

        fd_set fdset;
        cctl_chid_set chidset;

        /* I want to wait for a std input */ 
        FD_SET(0,&fdset);

        /* assume that "number_of_channels" contains the number of
           channels  that you want to listen to, and "chids" is
           an array of those channels. */
        if ( number_of_channels > 0 ) printf( "waiting on channel(s):" );
        for ( i=0; i<number_of_channels; i++ ) {
             CHID_SET(chids[i],&chidset);
             printf( " %d", chids[i] );
             if (max <= chids[i])
                   max = chids[i]+1;
        }
        if ( number_of_channels > 0 ) printf( "\n" );
 
        result = cctl_select( STDIN_FILENO+1,&fdset,max,&chidset, NULL );
        printf( "after select result=%d\n", result );
 
        if ( number_of_channels > 0 ) printf( "getMsg in channel(s):" );
        for ( i=0; i<number_of_channels; i++ ) {
                if ( channel_list[i].listening ) {
                        if (CHID_ISSET(chids[i],&chidset))
                            printf( " %d", chids[i] );
                }
        }

2. CCTL_CREATE_SESSION and CCTL_CREATE_CHANNEL

   These apis now contain a flag field where you specify whether
you want to listen to yourself.


  int cctl_create_session(char *sname, int mode,int flags, void h(void *));
  int cctl_create_channel(char *chname,int qos,int flags, void h(void *));

  The default value for flags is 0 which is to receive from all 
  (including you). You can have flags set to CCTL_OTHERS_ONLY to
  indicate that you are not interested in receiving your own msgs.


3. CCTL_GET_CH_MBRSHIP

 int cctl_get_ch_mbrship(int chid, /* channel id */
                         int *mbrship_mask, /* membership vector */
                         int sz) /* size of membership vector*/

 This call returns the number of current members in the channel. 
 The following shows an example of the usage.

int
ShowChMbrship( int chid )
{
        static char fn[] = "ShowChMbrship";
        int result,numOfMbrs;
        int i;
        int mask;
 
        result = cctl_get_ch_mbrship(chid,&mask,1);
        numOfMbrs = result;
 
        printf("Channel %d's Membership\n",chid);
        fprintf( stdout, "Number of channel mbrs = %d\n", numOfMbrs );
 
        fprintf( stdout, "Mbrs: ");
        for(i=0; i<32; i++)
        {
           if ((mask & (1<<i)) != 0)
              fprintf(stdout," %d", i);
        }
        fprintf(stdout,"\n");
        fflush( stdout );
}

  
-----------------------
Update 10/23/96


cctl_select's api is changed now to incorporate Phil's comments.
 
    int cctl_select(int nfds,
                fd_set *readfds,  /* read sockets */
                fd_set *writefds, /* write sockets */
                fd_set *exceptionfds, /*exception sockets */
                int nchids,
                cctl_chid_set *chids,
                struct timeval *timeout)
 
In addition, cctl_join* are incorporated. Now you can use
cctl_join* in the same way as cctl_create*

------------------------
Update 1/2/97


1) new qos called CCTL_FAST_RELIABLE is implemented. It is intented to
replace current CCTL_RELIABLE, but since the new qos is implemented
so recently, it may not be stable (i.e., might contain some bugs). 
So CCTL_RELIABLE is going to stay for a while until we are sure 
that CCTL_FAST_RELIABLE is reasonally bug-free. 

2) In order to use it, you have to set CCTL_FAST_RELIABLE as channel
qos when a channel is created. This new qos supposedly gives you
faster transmission of data as it solves the nasty flow control
problem. 

3) It also features message packaging so that small messages
are packed together into a single packet until the packet gets
full or a timer expires (which is normally 100 ms). 

4) There is a slight modification on cctl_recv parameter list.
It now returns the tag of the message received.
   So the API for cctl_recv is changed to 

   int cctl_recv(int chid, int *src, char *msg, int bufLen, int *msgLen,
                 int flags, int srcTag, int tag,int *returnedTag);
                                                ^^^^^^^^^^^^^^^
                                                this field is added.   


5) A new stream interface on top of message interface is also 
supported.  When an application provides a smaller buffer
than the actual message, it returns the same amount of data as
the size of the buffer. And at the next time it tries to receive
the message, it returns the remaining part of the message up to
the size of the provided buffer.
But be warned that its semantics are a bit different from those of
TCP because cctl always returns at a message boundary 
as there is enough buffer space (whereas in TCP, there is no 
concept of messages). 

In order to use the stream interface, you need to set a flag 
to CCTL_STREAM and pass it to cctl_recv. The following gives
you one example.

   err=cctl_recv(chid,&src,buf,bufLen,&msgLen,
                 CCTL_ASYNC|CCTL_STREAM, CCTL_ALL,0,&tag);
                 ^^^^^^^^^^^^^^^^^^^^^
 
                 this can be CCTL_SYNC|CCTL_STREAM 

If you don't set CCTL_STREAM, it provides the message interface. 


To do:
   1. need to optimize unicast.
   2. optimize membership operations (channel create and leave).
   3. handle faults.
------------------------------
1/4/97

There are a few other features I forgot to mention
in my previous mail.
 
1) In cctl, messages passing within a process (i.e., messages sent to
itself) is now strictly by shared memory. So, it is very fast.
This has one significant implication in designing cctl applications with
threads.  One good example is our beloved ccsm/ccfx. Currently ccsm/ccfx 
are two separate processes. If you have ccsm and ccfx in the same  
process as separate threads, a lot of overhead in communication between 
ccsm and ccfx will be eliminated.
 
 
2) In the previous mail, I write that cctl handles message packaging 
with 100 ms timeout. There is a way to go around the timeout when 
you have an urgent data or time critical data to send. You can   
set CCTL_FLUSH  flag in cctl_send. This will cause the data 
immediately sent without any further delay.
 
 
3) About the cctl stream interface,  I would like to point out that
when you send large messages (something like 32 Kbytes, 20 Kbytes,
or an entire file as a single message), the stream interface runs
a lot faster than the default cctl interface (which is message-based). 
The reason is because under message-based interface, 
cctl_recv would not get a message out from cctl buffer until the 
message is completely received. Thus when it comes to a large message,
that message won't be available until cctl completely receives it 
although cctl currently has a part of that message in the buffer.
 
-------------------------------------------------------
2/2/97 --- 01:30 am

1. SPEED, SPEED and more SPEED.

I just finished optimizing various functions of cctl. The following
operations are optimized. Although some of them are not completely
tuned, they seem to me satisfactory at this moment.

   (1) cctl_create_channel
   (2) cctl_leave_channel 
   (3) cctl_send and cctl_recv (fast recv and send) 
   
create_channel and leave channel take about 100 ms. This can be
further optimized when I am finished with session operation --
which I will do in March (after my interview trip).  Most of the
time is spent on doing membership total ordering. I am planning
to devise or use some existing total ordering protocols to 
improve the timing. 

send and recv on reliable channels are really super faster 
even better than TCP! I reach the limit of ethernet at this
moment fully utilizing the network. I get about 8-9Mbits/sec
for point to point between chilidog and chorale. 
For multicast, it also stays in that
range pretty much though I have not measured it for more
than three machines.

2. Membership operations are ASYNCHRONOUS. 

Another thing to note here is channel join and leave operations
are now completely ASYNCHRONOUS which means the appl. does not block
on join and leave. This speeds up the application because while
cctl is waiting to synchronize with other group members for join
or leave ops, the appl can have the control back to do its 
much needed or more time critical operations. For example, ccsm
can bring up its window while joining a channel.

3. Local membership query operations:

This will be a newly added feature. The new channel membership
protocols allow channel membership infomation to be cached locally
so the quary is fast!

I have not implemented API for it though. But it would not take 
more than 20 minites to do so. But I am too sleepy now. 

4. Synchronous signals.
 
Last but most important thing you have to aware now. cctl now returns
synchronous signal message or notification (as refered in the cctl paper)
for member join and leave. This is virtually synchronous in the sense
that all members in a group receive the same set of messages between
any two consecutive membership change.  For each membership change,
cctl returns a notification message to cctl_recv. The message is
returned with tag (CCTL_MEMBER_JOIN and CCTL_MEMBER_LEAVE) with
src id corresponding to the joining or leaving member.  
You can choose not to receive this message. To do that, you have
to set a flag when you create the channel. The flag is CCTL_DATA_ONLY.

5. File descriptors: 
 
a constant number of  file descriptors (currently it uses 5 or 6
and that's it). Creating more channel does not open more file
descriptors. It uses only a fixed number of fds.
 
6. Known bug. cctl_leave_session. If you leave a session, it will
abort for now. If it does not hang the other members. I will
try to fix if I have time....
 
 
7. To do: optimize the session membership operations. I don't think I
will have time to do this Feb, but I will think about nice protocols
for this while I am on my interview trip.

------------------
4/10/97

New cctl lib is in ~ccf/DEV/CCFcctl/src. The race condition in session
join is fixed now. It also joins and leaves
a session so fast, even before you know it! It does not occupy 
much memory (its default memory consumption is ~4M), either.
No hidden process or cctl daemon is being spawned, either.
It does not leave any hanging processes behind if the processes
leave the session properly. Note it isn't yet geared up with 
fault detection, so it may leave some process and also hangs
the entire session, if even one session member does not 
respond or cores dump.
 
***** The old version that used to be ~ccf/DEV/CCFcctl/src is now
moved to ~ccf/SHIP/ccf-0.70/cctl.
 
***** Chat Tool users, READ THIS: 
Oh, one more thing: current version does not support atomic
multicast. I took it out from this version because it soon will 
be replaced with a new and fast atomic channel (Phil 
is working on it). This means bad news for the folks with chat 
tool, but don't worry Phil says it's coming soon to a 
theater (or cctl library) nearest to you.
 
-----------------
4/15/97

Now cctl_init takes integer as an argument which specifies the
debug level -- the level of the debugging print statements. 
Currently it supports 0 and 1, and more levels
will be added in the future. 0 means no debug print 
statements, and 1 means some debug print statement.
 
int cctl_init(int);