Protocol Stack Implementation using Shared Memory as IPC



Introduction

This article talks about implementing a generic protocol stack with shared memory as means of communication between main layer and its user and service layer(s). It comprises of four sections. The first section gives a brief overview of Generic protocol design based on FSM model. The second section introduces the concept of shared memory as a means of communication between individual protocol layers. The next section dwells into the world of Semaphores used for synchonization between individual protocol elements for common resources. The concluding section explains communication between two protocols using shared memory and semaphores.

NOTE : All system calls mentioned are conformant to System 5 Release 4. The target kernel on which the code is tested is kernel version 2.4.20. All software is released under GPL.

  1. Generic Protocols design based on FSM

    A set of functionally active protocol entites composes a stack. Each protocol entity is modelled as a layer serving a number of other upper level protocol entites called Users and using services from a number of loer level protocol protocol entities. This is shown in the following figure which depicts two users layers and one service layer for protocol X.

                  -------    ------
                 |User 1 |  |User 2|
                  -------    ------
                    |         |
                  --------------   
                 | Protocol  X  |
                  --------------
                        |                   
                   --------------
                  | Service Layer |    
                   --------------
    
                    Figure 1. Protocol Model
    
    
    A typical protocol stack is divided into two or more planes as per their functionalities and role. This division can be at both Vertical and Horizontal Direction or at a single orientation. The protocol stack will consists of a number of layers interacting in vertical manner (and or horizontal level also) and finally terminating at PHYSICAL Layer which will actually communicate to its PEER running at some other system. One such example is TCP IP stack which is shown in the following figure.
                       -----------------------
                      |Application Layer (FTP)|
                       -----------------------
                                |
                             -------   
                            | TCP   |
                             -------
                               |                   
                              ----
                             | IP |    
                              ----
                               |
                            ---------
                           |Ethernet |
                            ---------
                               |
                          --------------
                         |Physical Layer|
                          --------------
    
                    Figure 2. TCP IP protocol stack
    

    Protocol Designers usually model the system at an abstract level and then implement the model in an implementation language. Complex behavior can be obtained by combining simpler building blocks: receiving an event from outside, performing a computation in response to the event, and generating output events. This kind of interaction is done by modeling the problem as Finite State Machine (FSM)(usually using SDL (State Descriptive Languange ).

  2. Concept of Shared Memory

    Shared memory allows one or more processes to communicate with each other without going through kernel on each read/write as in the case of Message Queues or pipes. Processes communicate via memory that appears in their virtual address spaces. As with all System V IPC objects, access to shared memory areas is controlled via keys and access rights checking. Once the memory is being shared, system does not impose checks on how the processes are using it. Thus synchronization contructs like System V Semaphores are used to synchronize access to this memory.

    Fig 1. shmid_ds structure

    Each newly created shared memory area via shmget() is represented by a shmid_ds data structure. These are kept in the shm_segsvector. The shmid_ds data structure decribes how big the area of shared memory is, how many processes are using it, time of access and information about how that shared memory is mapped into their address spaces. Each process that wishes to share the memory must attach to that virtual memory via a system call shmat(). This creates a new vm_area_struct data structure describing the shared memory for this process.When processes no longer wish to share the virtual memory, they detach from it via shmdt(). So long as other processes are still using the memory the detach only affects the current process.When the last process sharing the memory detaches from it, the pages of the shared memory current in physical memory are freed, as is the shmid_ds data structure for this shared memory. The behaviour of shared memory segment can be controlled by systm call shmctl().

  3. Semaphore as synchronization objects

    Semaphores is not a communication construct, rather it is a synchronization contruct which help multiple processes to synchronize their operations. Their absence would result in extremer deadlocks and loss of data integrity. One of their main use is synchronize access to shared memory segments. Semaphore can be thought of as resource counter where its value denotes total number of resources available. Whenever a process needs access to a shared resource, it checks the current value of the Semaphore instance. If it is greater than 0 (means resource available), it decrements the count by one and locks that resource instance. Otherwise, it goes in a wait state (or do some otherwork as per its design). The system maintains the list of processes waiting for that resource. Whever the process is done with the reource instance, it frees it and increments the Semaphore count by 1. This event will trigger the processes waiting for that resource( which process comes alive first, is an implementation strategy).

    These two operations on Semaphore i.e., checking the value of Semaphore and decrementing it must be ATOMIC i.e., done as a single non interruptible exeution. Hence SVR4 Semaphores are implemented in Kernel. Semaphore is created or existing semaphore is accessed with the semget() system call. Semaphores can be binary or counting. The Semaphores are controlled by system call semctl().The increment/decrement operation on Semaphores is perfomed by semop().

  4. Generic protocol implementation using shared memory as IPC

    Once Finite State machine model of a protocol layer is developed, it can be coded in the following two ways :

    1. State Model
    2. Primitive/Event Model

    In the State Model, all distinct states of the protocol entity are identified and all external primitives exchanged with other protocol layers as well as internal events are identified. Suppose the Protocol layer X interacts with three different layers namely, L1,L2 and L3 and it has five states say, S1,S2,...,S5. Further X exchanges 12 pritimitives (external events) with these 3 layers and has 5 primitives(intrinsic events). In tihs method only those primitives are included under each state which are applicable to it.

    The main module of X is designed in this scheme as following

    /***********Main Include File***************/
    ..............
    ..............
    
    /*******Standard Config Parameters**********/
    ..............
    ..............
    
    XX_main (void)
    {
    /* Declare and Initialise the state machine and process control block */
    	..............
    	..............
    	XXX_Init(dsc);
    	..............
    	..............
    	while (TRUE)
    	{
    /* Receive the message aany of the 3 layers */
    /* Allocate the space for the message being received */
    		if ((msg = (Mesg_t *) malloc (sizeof (Mesg_t))) == 0)
    		{
    			printf ("error in malloc");
    			exit;
    		}
    /* Extract Layer identitifier from the incoming message */
    		switch (layerid)
    		{
    		  case L1:
    /* Catch primitive in the mesage */
     		   switch (primitiveid)
    		   {
                     	     case P1 :
    
    /* Call respective primitive/event handler as per primitive in the incoming message */
    			    P1_handler(state);
    			    .............	
     			    .............
    			    break;
                     	     case P2 :
    
    /* Call respective primitive/event handler as per primitive in the incoming message */
    			    P2_handler(state);
    			    .............	
     			    .............
    			    break;
    
    		     ..............................................	
    		     ..............................................
    		   }//end switch for primitive
    
      		  case L2:
    /* Catch primitive in the mesage */
     		   switch (primitiveid)
    		   {
                     	     case P3 :
    
    /* Call respective primitive/event handler as per primitive in the incoming message */
    			    P3_handler(state);
    			    .............	
     			    .............
    			    break;
                     	     case P4 :
    
    /* Call respective primitive/event handler as per primitive in the incoming message */
    			    P4_handler(state);
    			    .............	
     			    .............
    			    break;
    
    		     ..............................................	
    		     ..............................................
    		   }//end switch for primitive  		  
    		  case L3:
    		       ..............................................	
    		      /* same steps as in above */	
    		       ..............................................	
    		}//end switch for layer id
    	..............
    /* Some processing like memory deallocation and housekeeping activities */              
    	}//end while		  	
    /* Cleanup fuctions */	
    	XX_cleanup();
    }//end XX_main
    

    In the Primitive/Event model, only those states are included under each primitive/events, which are applicable to it. Considering the same example as in State module, protocol X is modeled is designed in this scheme as following.

    /***********Main Include File***************/
    ..............
    ..............
    
    /*******Standard Config Parameters**********/
    ..............
    ..............
    
    XX_main (void)
    {
    /* Declare and Initialise the state machine and process control block */
    	..............
    	..............
    	XXX_Init(dsc);
    	..............
    	..............
    	while (TRUE)
    	{
    /* Receive the message aany of the 3 layers */
    /* Allocate the space for the message being received */
    		if ((msg = (Mesg_t *) malloc (sizeof (Mesg_t))) == 0)
    		{
    			printf ("error in malloc");
    			exit;
    		}
    /* Extract Layer identitifier from the incoming message */
    		switch (layerid)
    		{
    		  case L1:
    /* Descriminate between the  various states applicable for this layer*/
     		   switch (present state)
    		   {
                     	     case S1 :
    
    /* Call respective primitive/event handler as per primitive in the incoming message */
    			    if (primitive==P1) P1_handler();
    			    if (primitive==P2) P2_handler();	
    			    .............	
     			    .............
    			    break;
                     	     case S2 :
    
    /* Call respective primitive/event handler as per primitive in the incoming message */
    			    if (primitive==P3) P3_handler();
    			    if (primitive==P4) P4_handler();	
    			    .............	
     			    .............
    			    break;
    		     ..............................................	
    		     ..............................................
    		   }//end switch for state
    
      		  case L2:
    /* Descriminate between the  various states applicable for this layer*/
     		   switch (present state)
    		   {
                     	     case S1 :
    		       ..............................................	
    		       ..............................................
    		       break;	
    
                        	     case S2 :
    		       ..............................................	
    		       ..............................................
    		       break;	
    		   }//end switch for state
    
      		  case L3:
    		       ..............................................	
    		      /* same steps as in above */	
    		       ..............................................	
    		}//end switch for layer id
    	..............
    /* Some processing like memory deallocation and housekeeping activities */              
    	}//end while		  	
    /* Cleanup fuctions */	
    	XX_cleanup();
    }//end XX_main
    

    The following code snippet gives an implementation of using shared memory as IPC mechanism between two processes which represents two protocols layers LAYER 1 and LAYER 2. This gives a minimal main funtion for LAYER 2. It uses two memory segments for communication with eact other and thus two semaphores to synchronize each. saallm_sysmgmt_msg shared memory segment is written by LAYER2 and read by LAYER 1 and is guarded by saallmsemid whereas sysmgmt_saallm_msg shared memory segment is written by LAYER 1 and read by LAYER 2 and is guarded by sysmgmtsemid. Most of the code is self explanatory. LAYER 2 first checks creates and initializes semaphore set and shared memory segments and attaches them. Then it checks sysmgmtsemid and locks it if free to read and process the mesage from LAYER 1. It then unlocks the semaphore to let LAYER 2 write a new message. Next if it has some message to send to LAYER 1, it checks and locks saallmsemid and then write a message in the common memory segment saallm_sysmgmt_msg. After this it unlocks the semaphore to enable LAYER 1 to read the mesage. All this is done in an infifnite loop as an eternal process.
    /*Include all System Headers */
    
    #define MAX_LENGTH	100
    
    int saallmsemid;
    int sysmgmtsemid;
    int saallmshmid;
    int sysmgmtshmid;
    
    typedef struct Message
    {
    	char Mesg_Data[MAX_LENGTH];
    	int Mesg_Length;
    }Mesg_t;
    
    Mesg_t* saallm_sysmgmt_msg;
    Mesg_t* sysmgmt_saallm_msg;
    
    struct shmid_ds *bufsaallm;
    struct shmid_ds *bufsysmgmt;
    
    main ()
    {
    	int i,j,k;
    	sem_init();
    	k=sizeof(Mesg_t)+2;	//to avoid any boundary conditions
    	shmem_init(k);
    	
    	saallm_sysmgmt_msg=(Mesg_t*)shmat(saallmshmid,(const void *)0,0);
    	sysmgmt_saallm_msg=(Mesg_t*)shmat(sysmgmtshmid,(const void *)0,0);
    	
    	if(saallm_sysmgmt_msg==NULL||sysmgmt_saallm_msg==NULL)
    	{
    		perror("\nShared Memory segments for saallm and sysmgmt cannot be attached to OS\n");
    		exit(-1);	
    	}
    
    	while(TRUE)
    	{	
    /* wait on the sysmgmtsemid to read data from LAYER 1 */
    		semop(sysmgmtsemid,&op_lock[0],2);
    		if (layerid=LAYER 1)
    		{
    /* Read the message and process */
    			........................
    			........................
    /* signal the sysmgmtsemid to enable Sysmgmt to write new data */
    			semop(sysmgmtsemid,&op_unlock[0],1);
    		}
    		else semop(sysmgmtsemid,&op_unlock[0],1); 
    
    /* wait on the saallmsemid to write new data for LAYER 1 */
    		semop(saallmsemid,&op_lock[0],2);
    
    		............................
    		............................
    		
    		memcpy(saallm_sysmgmt_msg->Mesg_Data, Data buffer in which u have the message);
    /* signal the saallmsemid to enable LAYER 1 to read this new data */
    		semop(saallmsemid,&op_unlock[0],1);
    	}//end while
    /* Free shared memory and semaphore resources*/
    	sem_cleanup();
    	shmem_cleanup();
    }//end main
    
    void sem_init(void)
    {
    	/* First create semaphores in not already created */
    	/* Initiailize both semaphores to 0 as done by default */ 
    	
     	saallmsemid=semget(SAALLMSEMKEY,1,PERMS|IPC_CREAT);
     	sysmgmtsemid=semget(SYSMGMTSEMKEY,1,PERMS|IPC_CREAT);
    	
    	if(saallmsemid<0||sysmgmtsemid<0)
    	{
    		perror("\nSemaphores for saallm and sysmgmt cannot be created\n");
    		exit(-1);	
    	}
    }
    
    void shmem_init(int k)
    {
    	saallmshmid=shmget(SAALLMMEMKEY,k,PERMS|IPC_CREAT);
    	sysmgmtshmid=shmget(SYSMGMTMEMKEY,k,PERMS|IPC_CREAT);
    	
    	if(saallmshmid<0||sysmgmtshmid<0)
    	{
    		perror("\nShared Memory segments for saallm and sysmgmt cannot be created\n");
    		exit(-1);	
    	}
    }
    
    void sem_cleanup(void)
    {
    	if(semctl(saallmsemid,0,IPC_RMID)<0) perror("\n Cannot delete SAALLM semaphore\n");
    	if(semctl(sysmgmtsemid,0,IPC_RMID)<0) perror("\n Cannot delete SYSMGMT semaphore\n");
    }
    
    void shmem_cleanup(void)
    {
    	if (shmdt(saallm_sysmgmt_msg)<0) perror("\n Cannot detach shared memory between saallm and sysmgmt\n");
    	if (shmdt(sysmgmt_saallm_msg)<0) perror("\n Cannot detach shared memory between sysmgmt and saallm\n");
    
    	if (shmctl(saallmshmid,IPC_RMID,bufsaallm)<0) perror("\n Cannot remove shared memory between saallm and  
    sysmgmt\n");
    	if (shmctl(sysmgmtshmid,IPC_RMID,bufsysmgmt)<0) perror("\n Cannot remove shared memory between sysmgmt and  
    saallm\n");
    }
    

I am Nikhil Bhargava, curently working on 3G mobile communication protocol stacks at C-DOT, India for past one year in capacity of Research Engineer. All comments, bug fixes, and suggestions will be highly appreciated.

1