tclmsgque

The TCL Application-Server Project

SYNOPSIS

package require TclMsgque

INDEX

OBJECT CREATION AND DELETION
OBJECT CONFIGURATION
LINK CREATION AND DELETION
ADDING SERVICES
SEND DATA
READ DATA
SLAVE OBJECT
BUFFER OBJECT
EXCEPTION OBJECT
CONTEXT LOCAL STORAGE

INTRODUCTION

The msgque project is an infrastructure to link software together to act like a single software. To link mean distributing work from one software to an other software an wait or not wait for an answer. The linking is done using unix or inet domain sockets and is based on packages send from one software to an other software and back. The msgque project is used to handle all the different aspects for setup and maintain the link and is responsible for:

• starting and stopping the server application
• starting and stopping the communication interface
• sending and receiving package data
• reading and writing data from or into a package
• setup and maintain the event-handling for an asynchronous transfer
• propagate the error messages from the server to the client

OBJECT CREATION AND DELETION

MqS

The object is the user visible part of a tclmsgque application link.

set ctx [tclmsgque MqS]

Create an object ready to be used with $ctx LinkCreate ... to setup an application link. Use the isConnected to check if a link is already available.

rename $ctx ""

Delete the entire object including the application link using $ctx LinkDelete. The object can not be reused.

OBJECT CONFIGURATION

The configuration of an object is done using the functions starting with MqConfig... or by using command-line arguments.

static: tclmsgque Init ...args

Initialize the process startup argument prefix. The startup-prefix have to be the name of the executable found in the PATH and additional arguments like a script name or required startup options. The startup-prefix is used for two different purpose:

• To start a new entity using the --spawn command-line option
• To replace the SELF argument after the "@" command-line argument

The startup-prefix is initially set by tclmsgque during package loading.

command-line: --buffersize

type: int, get: ConfigGetBuffersize , set: ConfigSetBuffersize

The send and receive buffersize in bytes (default: 4KB)

command-line: --debug

type: int, get: ConfigGetDebug , set: ConfigSetDebug

Debug a message-queue application. Valid values are 0 <= NUM <= 9 using 0 for "no debugging" and 9 for "maximum debugging". (default: 0)

command-line: --timeout

type: int, get: ConfigGetTimeout , set: ConfigSetTimeout

User defined timeout used for connection setup and data sending (default: 90 seconds)

command-line: --name

type: string, get: ConfigGetName , set: ConfigSetName

Use STRING as basename of the new message-queue object. The name shows up in the debugging output and is used as prefix for the new created command. (default: the name of the executable)

command-line: --srvname

type: string, get: ConfigGetSrvName , set: ConfigSetSrvName

Use STRING as a client specific prefix in the server debugging output. This is used to link debugging and error messages on the server to a specific client connection. (default: empty)

command-line: --silent

type: boolean, get: ConfigGetIsSilent , set: ConfigSetIsSilent

Write (no) or don't write (yes) any output to stdout or stderr. (default: write to stdout or stderr)

command-line: --string

type: boolean, get: ConfigGetIsString , set: ConfigSetIsString

Use (yes), as much as possible, strings in the data-package. Convert any native data-type, like integer or double, into a string (sending) and back to native (receiving). (default: use binary)

command-line: --thread --spawn --fork

type: integer, get: ConfigGetStartAs , set: ConfigSetStartAs startAs

Start a new server as thread or spawn a new process or fork a new process. This arguments are used with:

• a server-context to start a new instance after a client connection request
• on a server to create a worker with: $ctx SlaveWorker id ...
• on a client/server together with the "SELF" command-line argument: $ctx LinkCreate ...

Parameters:

[in]

startAs

0=default, 1=fork, 2=thread and 3=spawn

command-line: --daemon

type: string, get: ConfigGetStartAs , set: ConfigSetStartAs pidfile

Close all default IO (e.g stdout, stdin, stderr) and fork the server into the background. (default: no daemon)

Attention:

this option require the fork system-call and is not compatible with threads.

Parameters:

[in]

pidfile

the name of the file to write the PID into

command-line: --tcp --host --port --myhost --myport

type: string, get: ConfigGetIoTcpHost/Port/MyHost/MyPort , set: ConfigSetIoTcp host port myhost myport

Configure a tcp socket link.

Parameters:

	host	client: name of the remote host (default: localhost) server: name of the local interface (default: listen on all interfaces)
	port	client: name of the remote port server: name of the local port
	myhost	client: name of the local interface
	myport	client: name of the local port

Attention:

named ports are supported as well

command-line: --uds --file

type: string, get: ConfigGetIoUds , set: ConfigSetIoUds pidfile

Use a uds (http://en.wikipedia.org/wiki/Unix_domain_socket) socket link. The uds communication is usually 50% faster than a local tcp communication but only available on UNIX.

Parameters:

[in]

file

the name of the uds file

command-line: --pipe --socket

type: integer, get: ConfigGetIoPipe , set: ConfigSetIoPipe socket

Start a pipe server to listen on socket. This is the default mode to start a server. The socket option is special because it is used for internal purpose to submit the socket from the client to the server started as pipe by the client.

Parameters:

[in]

socket

the file-descriptor number. The only public usage for this option is to serve as interface for an existing tool like (x)inetd. The (x)inetd tool is a UNIX service to listen on a tcp/ip port and start for every connection the proper entry from the file /etc/(x)inetd.conf with the file-descriptor 0 as send/recv socket handle.

master

type: Command-Object, get: ConfigGetMaster , set: SLAVE OBJECT

return the master if ctx is a slave-context or NULL if ctx is not a slave-context. !Only a SLAVE has a master!

ctxId

type: int, get: ConfigGetCtxId , set: NO

return an identifier as integer and is unique per parent-context. The parent-context is always 0 and every new child-context get a new identifier by adding 1.

token

type: string, get: ConfigGetToken , set: NO

return the current TOKEN IDENTIFIER and is only useful in a service callback. This command is needed on the server to implement a generic service (A $ctx ServiceCreate token service with different TOKEN IDENTIFIER but with the same service callback).

isConnected

type: boolean, get: ConfigGetIsConnected , set: NO

Is the tclmsgque context connected ? A context is connected if the $ctx LinkCreate ... command was successful and a context is NOT connected if a) the object has just been created or b) the link was deleted with $ctx LinkDelete

isServer

type: boolean, get: ConfigGetIsServer , set: IServerSetup or IServerCleanup

True if object is a server object (default: no, be a client)

isParent

type: boolean, get: ConfigGetIsParent , set: NO

True if object is a parent object (default: yes, be a parent)

isSlave

type: boolean, get: ConfigGetIsSlave , set: NO

True if object is a slave object (default: no, not be a slave)

IServerSetup

type: Interface, get: NO , set: $ctx ConfigSetServerSetup callback

Interface to define a server tclmsgque object. This Interface define the ServerSetup callback called at $ctx LinkCreate ... to define the services or to initialize context specific variables. This interface set the isServer configuration value to yes.

IServerCleanup

type: Interface, get: NO , set: $ctx ConfigSetServerCleanup callback

Interface to define a server tclmsgque object. This Interface define the ServerCleanup callback called at $ctx LinkDelete to free context specific variables. This interface set the isServer configuration value to yes.

IFilterFTR

type: Interface, get: NO , set: $ctx ConfigSetFilterFTR callback

Interface required to define a filter data stream object. This Interface define the IFilterFTR callback. The callback is used to act on filter data rows. Every filter input data is a list of filter data rows and every row is a list of filter data columns. Every row is send from one filter-command to the following filter-command as FTR service request. This interface set the isServer configuration value to yes. (read more at: FILTER MODE)

IFilterEOF

type: Interface, get: NO , set: $ctx ConfigSetFilterEOF callback

Interface required to define a filter data stream object. This Interface define the IFilterEOF callback. The callback is used to act on End-Of-Filter data and is called after all IFilterFTR data was send. Sometimes the filter data can not be served as IFilterFTR data (example: sorting of the input rows need to read all rows before the data can be send to the next filter command) and the IFilterEOF callback is used to continue send IFilterFTR data rows. This interface set the isServer configuration value to yes. (read more at: FILTER MODE)

IFactory

type: Interface, get: NO , set: $ctx ConfigSetFactory

The factory pattern is used to create a new application object (read more at: MqSetupS::Factory) Without the Factory pattern only the initial startup context is available to serve incoming requests. In general every server need to provide a Factory pattern.

IBgError

type: Interface, get: NO , set: $ctx ConfigSetBgError callback

Define an asyncrone error handler. This handler is used for handle errors send with $ctx SendERROR. Use [lindex $errorCode 1] or $ctx ErrorGetNum and [lindex $errorCode 3] or $ctx ErrorGetText to access the error.

LINK CREATION AND DELETION

To create and to destroy a link is the main purpose of the tclmsgque library. A link is a client/server connection used to perform various tasks.

$ctx LinkCreate ...

Create a parent-context using the OBJECT CONFIGURATION. A parent-context is responsible to setup the client/server link:

• the client-parent-context create the server-parent-context
• the server-parent-context wait for a client-parent-context connection request

Parameters:

[in]	ctx	the object from OBJECT CREATION AND DELETION
[in]	args	command-line arguments including the "@" item for the --pipe setup

Attention:

if the first argument after the "@" item is the string "SELF" an independent server of the current server is started. This is not a SLAVE OBJECT. The "SELF" argument is replaced by an application default setting (if available) or by arguments set with static: tclmsgque Init ...args.

$ctx LinkCreateChild $parent ...

Create a child-context from a tclmsgque parent-context and command-line arguments. A child is using the same process or thread as the parent but a different namespace. With a different namespace a child is able to act on different services on the shared server.

Parameters:

[in]	ctx	the object from OBJECT CREATION AND DELETION
[in]	parent	the parent object
[in]	args	all command-line arguments including the "@" token

$ctx LinkDelete

Close the client/server link and delete the underlying communication object. All depending objects will be deleted as well like depending child-context and slave-context local and on the remote site of the link.

$ctx Exit

Shutdown the entire communication and exit the current process or thread.

ADDING SERVICES

To provide a service is the main purpose of a server. The $ctx ServiceCreate token service method is usually added in the ServerSetup method of the IServerSetup.

proc ServerSetup {ctx} {
  $ctx ServiceCreate "SRV1" serviceF
}

TOKEN IDENTIFIER

The token is a 4 byte string and identifies the service. The token is used in:

• $ctx SendEND token
• $ctx SendEND_AND_WAIT token ?timeout=MQ_TIMEOUT_USER?
• $ctx SendEND_AND_CALLBACK token callback
• $ctx ServiceCreate token service
• $ctx ServiceDelete token

SERVICE CALLBACK

A service is using a callback to act on an incoming service request. The callback is used on a server using $ctx ServiceCreate token service or on a client using $ctx SendEND_AND_CALLBACK token callback.

The callback is implemented as TCL procedure using Service {ctx}

$ctx ServiceCreate token service

Create a new service listen on TOKEN IDENTIFIER and start for every incoming request a service. (on error return an EXCEPTION OBJECT)

$ctx ServiceDelete token

Delete a new service listen on TOKEN IDENTIFIER . (on error return an EXCEPTION OBJECT)

$ctx ProcessEvent ?-timeout val? ?-wait (NO|ONCE|FOREVER)?

Start the Event-Loop and check for the next event. In client mode this command is usually used together with $ctx SendEND_AND_CALLBACK token callback to process the results.

Parameters:

[in]	timeout	don't wait more than TIMEOUT seconds to get an answer from the server. If no answer is available an error is raised. (default: command-line: --timeout)
[in]	MqC.WAIT	use flag value to define the action (default: NO) NO, don't wait for an event do just the check ONCE, use timeout seconds to wait for exact one event FOREVER, wait forever and only come back on error or on exit

SEND DATA

The Send style methods are used to build and send data using the tclmsgque send data package. The data is typed using the TYPE IDENTIFIER and the methods can wait or not for an answer. During waiting the the application is still able to work on events like other tclmsgque client/server links.

$ctx SendSTART
$ctx SendI $num
$ctx SendL_START
 $ctx SendD $balance
 $ctx SendC "name of the owner"
 $ctx SendB $signature
$ctx SendL_END
$ctx SendEND_AND_WAIT "SRV1"

$ctx SendSTART

Initialize a tclmsgque send data package. This method is required.

$ctx SendEND token

Send the package using TOKEN IDENTIFIER without expecting an answer (FireAndForget). (on error return an EXCEPTION OBJECT)

Attention:

if an error is raised on the server during service-processing the error is send back as asynchronous event and can be raised sometime in the future. To be sure to get the error immediately use $ctx SendEND_AND_WAIT token ?timeout=MQ_TIMEOUT_USER?

$ctx SendEND_AND_WAIT token ?timeout=MQ_TIMEOUT_USER?

Send the package using TOKEN IDENTIFIER and wait timeout seconds for an answer. (on error return an EXCEPTION OBJECT)

$ctx SendEND_AND_CALLBACK token callback

Send the package using TOKEN IDENTIFIER and don't wait for an answer. The answer will be handled by the SERVICE CALLBACK. (on error return an EXCEPTION OBJECT)

$ctx SendRETURN

Answer an incoming service-call started with $ctx SendEND_AND_WAIT token ?timeout=MQ_TIMEOUT_USER? or $ctx SendEND_AND_CALLBACK token callback. The return package is build-up like the previous service call package but with a $ctx SendRETURN as final command. (on error return an EXCEPTION OBJECT)

$ctx SendERROR

Send an asyncrone error to the link target. (on error return an EXCEPTION OBJECT)

$ctx SendFTR ?timeout=MQ_TIMEOUT_USER?

Send a filter stream data package using FILTER MODE. (on error return an EXCEPTION OBJECT)

$ctx SendEOF ?timeout=MQ_TIMEOUT_USER?

Send a filter end-of-file data package using FILTER MODE. (on error return an EXCEPTION OBJECT)

$ctx Send[TYPE] value

Add an item using TYPE IDENTIFIER to the tclmsgque send data package. (example: SendY)

$ctx SendL_START

Start to send an embedded list item.

$ctx SendL_END

Finish to send an embedded list item.

READ DATA

Read data from an incoming tclmsgque read data package:

1. on a server to serve an incoming service-call from the client
2. on a client to process the return-data from a previous service-call

Reading data is a passive task in opposite to sending data which is an active task. Passive mean that the read is triggered by an incoming data-package and not by the software-workflow.

set num [$ctx ReadI]
$ctx ReadL_START
 set balance [$ctx ReadD]
 set owner [$ctx ReadC]
 set signature [$ctx ReadB]
$ctx ReadL_END

$ctx Read[TYPE]

return an item using the TYPE IDENTIFIER from the tclmsgque read data package. (on error return an EXCEPTION OBJECT)

$ctx ReadU

return a temporary BUFFER OBJECT. (on error return an EXCEPTION OBJECT)

Attention:

The object is only valid until a new tclmsgque read data package has arrived.

$ctx ReadProxy otherCtx

Link two tclmsgque objects to direct pass a data item from one object to the other. (on error return an EXCEPTION OBJECT)

int: $ctx ReadGetNumItems

return the number of items left in the tclmsgque read data package.

bool: $ctx ReadItemExists

Check if are still items in the tclmsgque read data package.

$ctx ReadUndo()

Undo the last read operation. This is useful if an error was raised or if a buffer object (returned from $ctx ReadU) has not the expected type. (on error return an EXCEPTION OBJECT)

$ctx ReadL_START

$ctx ReadL_START buffer

Start to read an embedded list item. Use the current read package (buffer = NULL) or a buffer object. (on error return an EXCEPTION OBJECT)

$ctx ReadL_END

Finish to read an embedded list item.

SLAVE OBJECT

The master/slave link is used to create a mesh of nodes by linking different PARENT context objects together. The master control the slave.
The link is used to perform the following tasks:

• report error messages from the slave to the master
• to create a slave-child-context if a master-child-context is created
• to delete a slave-context if a master-context is deleted

In difference to the client/server link the master/slave link connect two independent msgque-context in the same process or thread (e.g. node). This leads to the restriction that only the master-msgque-context can be a server because only one server per node is possible.

$ctx SlaveWorker id ...

Create a worker as slave of the ctx object using the image of the ctx object self and started as an independent process or thread based on the command-line: --thread --spawn --fork argument.

Parameters:

[in]	ctx	the master tclmsgque object as SERVER-PARENT without a CHILD
[in]	id	an integer used as unique identifer for the master/slave link
[in]	args	command-line arguments passed to the worker-client or the worker-server. all arguments prior the first @ token are added to the worker-client and the other arguments to the worker-server.

example create a worker with id 7 as thread.

$ctx SlaveWorker 7 "--thread" "--silent" "@" "WORKER" "--name" "MyWorker"

$ctx SlaveCreate id slave

Create a master/slave link between the ctx object and the slave object. (on error return an EXCEPTION OBJECT)

Parameters:

[in]	ctx	the master tclmsgque object as PARENT without a CHILD
[in]	id	an integer used as unique identifer for the master/slave link
[in]	slave	the slave tclmsgque object as CLIENT-PARENT without a CHILD

$ctx SlaveDelete id

Delete a slave-context from a former $ctx SlaveCreate id slave command using the same identifier id. By default the slave-context object will be deleted if the master-context is deleted. In addition only the parent-slave-context can be deleted explicit. If id is invalid nothing will happen. (on error return an EXCEPTION OBJECT)

$ctx SlaveGet id

return the slave-context from the $ctx SlaveCreate id slave using the same identifier id or NULL if the id is not valid or if ctx is not a master.

BUFFER OBJECT

The class MqBufferC object is created by $ctx ReadU and id used to save a typeless temporary data item from the tclmsgque data package. The lifetime of the class MqBufferC is only the current callback up to the next read operation of the same context.

set buf [$ctx ReadU]
if {[$buf GetType] == 'C'} {
  puts [$buf GetC]
}

TYPE IDENTIFIER

A tclmsgque buffer data package is type safe, this mean that every item has a type prefix and every $ctx Read[TYPE] or $buffer Get[TYPE] have to match the previous $ctx Send[TYPE] value with the same TYPE. One exception, the cast from and to string (TYPE=C) is allowed. The following type identifier's are available:

• Y : 1 byte signed character
• O : 1 byte boolean character using yes for yes or true or no for no or wrong
• S : 2 byte signed short
• I : 4 byte signed integer
• W : 8 byte signed long long integer
• F : 4 byte float
• D : 8 byte double
• B : unsigned char array used for binary data
• C : string data using a \0 at the end
• L : list type able to hold a list of all items from above
• U : typeless buffer able to hold a single item from above

$buffer Get[TYPE]

return the data form the buffer using the TYPE IDENTIFIER. (on error return an EXCEPTION OBJECT)

$buffer GetType

return the type as single character of the item stored in the buffer object.

EXCEPTION OBJECT

The exception object is used to transport a tclmsgque error using the TCLMSGQUE errorCode.

if {[catch {
 i = $ctx ReadI();
}]} {
 $ctx ErrorSet 
 puts [$ctx ErrorGetText]
 $ctx ErrorReset
}

[lindex $errorCode 3] or $ctx ErrorGetText

return the error message from the error object.

[lindex $errorCode 1] or $ctx ErrorGetNum

return the error number from the error object. The number can be used as exit-code.

$ctx ErrorC prefix errnum message

create a tclmsgque error object but do not raise the EXCEPTION OBJECT

$ctx ErrorSet

convert a TCLMSGQUE errorCode into a tclmsgque conform error using the ctx object. This method is used to enable additional error processing capabilities from $ctx LinkDelete or $ctx Exit after the exception was caught and before the application exit or the object is deleted.

$ctx ErrorReset

clear the tclmsgque error object.

$ctx ErrorRaise

convert and raise an tclmsgque error object into a TCL EXCEPTION OBJECT.

$ctx ErrorPrint

print an tclmsgque error object to stderr and clear the tclmsgque error

CONTEXT LOCAL STORAGE

The CONTEXT LOCAL STORAGE is required to bind user data to the tclmsgque context

$ctx dict set key... data

Add data using key into the local storage and return the data afterwards.

$ctx dict get key...

return the data associated with key or nothing if key does not exists.

$ctx dict exists key...

return yes if key exists or no if not.

$ctx dict unset key...

Remove the key and the associated data from the local storage.

FILTER MODE

The filter mode is related to a special usage of the tclmsgque software called a command pipeline. A command pipeline is well known from the unix shell to link different commands together to act like a single application:

    command1 | command2 | command3

A command pipeline with tclmsgque queues is using the special token @ to link the commands together:

    msgcmd1 @ msgcmd2 @ msgcmd3

To create a tclmsgque filter use:

• IFilterFTR
• IFilterEOF

and to send data to the next filter command use:

• $ctx SendFTR ?timeout=MQ_TIMEOUT_USER?
• $ctx SendEOF ?timeout=MQ_TIMEOUT_USER?

PORTABILITY ISSUES

The library was tested on Linux, FreeBSD and WindowsXP using a x86_64, ppc processor or i686 processor from the VritualBox emulator.

EXAMPLES

1. create a server to multiplicate 2 double values and return the result

package require TclMsgque
proc MMUL {ctx} {
  $ctx SendSTART
  $ctx SendD [expr {[$ctx ReadD] * [$ctx ReadD]}]
  $ctx SendRETURN
}
proc ServerConfig {ctx} {
  $ctx ServiceCreate "MMUL" MMUL
}
tclmsgque Main {
  set srv [tclmsgque MqS]
  $srv ConfigSetName MyMulServer
  $srv ConfigSetFactory
  $srv ConfigSetServerSetup ServerConfig
  if {[catch {
    $srv LinkCreate {*}$argv
    $srv ProcessEvent -wait FOREVER
  }]} {
    $srv ErrorSet
  }
  $srv Exit
}

Start mulserver.tcl using TCP port 7777 and create a thread for every incoming connection

> tclsh mulserver.tcl --tcp --port 7777 --thread

2. in a client call the service from 1.

package require TclMsgque
set ctx [tclmsgque MqS]
$ctx LinkCreate --name "MyMul" {*}$argv
$ctx SendSTART
$ctx SendD 3.67 
$ctx SendD 22.3 
$ctx SendEND_AND_WAIT "MMUL" 
puts [$ctx ReadD]
$ctx Exit

Use mulclient.tcl to connect to mulserver.tcl using TCP port 7777:

> tclsh mulclient.tcl --tcp --port 7777

3. create a filter to wrap every column in a '<>' pair

package require TclMsgque
proc FTRcmd {ctx} {
  $ctx SendSTART
  while {[$ctx ReadItemExists]} {
    $ctx SendC "<[$ctx ReadC]>"
  }
  $ctx SendFTR
}
set srv [tclmsgque MqS]
$srv ConfigSetName ManFilter
$srv ConfigSetFilterFTR FTRcmd
if {[catch {
  $srv LinkCreate {*}$argv
  $srv ProcessEvent -wait FOREVER
}]} {
  $srv ErrorSet
}
$srv Exit

Use manfilter.tcl in a LibMsgque command pipeline:

> echo -e "1:2:3\na:b:c" | atool split -d : @ tclsh manfilter.tcl @ atool join -d :

SEE ALSO

TCL, C: libmsgque(n), C++: ccmsgque(n), C#: csmsgque(n), JAVA: javamsgque(n), PYTHON: pymsgque(n), TCL: tclmsgque(n), PERL: perlmsgque(n)

KEYWORDS

TCL, unix, socket, message, msgque