Connectors

Connectors provide the interface between Enterprise Messaging Gateway and other systems and applications. For example, one connector can connect to an operator’s SMSC and another connector can accept incoming messages via HTTP. Each connector can be available in one or more instances. The core functionality of EMG is to route messages from one connector to another.

When testing EMG or running benchmarks it can be useful to route message from an outgoing EMG connector to an incoming connector in the same server. We call this short-circuiting connectors.

Connector types

A connector can be one of two types: Incoming or outgoing. An incoming connector accepts incoming connections and an outgoing connector initiates connections. Please note however that whether a connector is incoming or outgoing does not tell anything about the direction of flow of the messages on the connector. Messages can be received on an outgoing connection, for example when connecting to a Nokia SMSC.

Both incoming and outgoing connectors can exist in 0 or more instances. However, normally an outgoing connector only needs one instance while an incoming connector needs several instances in order to be able to serve several simultaneous connections.

Connector modes

Connectors can be defined to be transmit-only, receive-only or both transmit and receive. This is not the same as incoming vs outgoing. For example when connecting to a SMSC an outgoing connector can poll for messages, which will then be received.

The connector mode is specified using the MODE keyword. By default connectors can both transmit and receive messages. Incoming messages on a transmit-only connector will be rejected.

In SMPP 3.3 however, sessions will be used either for transmit or receive depending on the bind operation used. This means that in order to be able to both send and receive messages at least two connectors must be defined.

Static vs non-static

An outgoing connector can be defined as static using the STATIC keyword. This indicates that the connector should connect to the remote entity immediately even if there are no messages to send. The connector will still respect any idle timeout specified. If IDLETIMEOUT is set to a non-zero value the connector will disconnect after being idle the specified number of seconds and the immediately reconnect since it is configured as static.

Usually you would use IDLETIMEOUT=90 and set KEEPALIVE to a slightly lower value, for example 60 (seconds). This will keep the connection open by sending keepalive operations periodically to ensure the connection up.

Connector states

A connector can be in exactly one of the following states at a specific time:

IDLE	Connector is just created or has been dynamically disconnected. Static connectors should never be in this state during normal operation.
CONNECTED	Connector is connected to remote endpoint but has not been authenticated.
BOUND	Connector is authenticated and allowed to transmit and receive messages.
TERMINATING	Connector will perform a graceful shutdown as soon as possible.
DEAD	Connector is shut down.
ERROR	Connector has failed a repeated number of times and is awaiting a new retry period.

On top of the above a connector can be put “on hold”, which means messages are accepted in the queue but no messages are transmitted. A message can be put on hold using the emgclient utility or another implementation using the MGP protocol.

Starting with EMG 6 it is also possible to “force close” a connector either using emgclient or via MGP API. The connector can either be bounced (forced to close but then released again to allow for immedia reconnect) or forced to close with this state persisted in which case the “force close” state will survive a server restart.

Instances

A connector is available in 0 or more instances. In order for the connector to be active and be able to send or receive messages it must be available in at least one instance.

Outgoing connectors usually only needs to be available in one instance. From EMGs point of view one instance is enough to handle hundreds of messages per second. However depending on delays imposed by the remote entity using more than one instance MAY increase performance.

Incoming connectors must be available in more than one instance in order to allow for several simultaneous connections. When an incoming request has been processed and the remote entity logs out and drops the connection the instance can be re-used. However, in some cases if the connection fails due to a network error for example it may take a while before this is detected and the session is cleared and available for use again. In this case it would be an advantage to have more instances than the expected number of simultaneous connections. If two incoming connections are expected maybe 5 instances would be suitable.

Message types

The default message type in EMG is SMS. This for example means that when a message is received on an SMTP connector it is automatically parsed and converted to a format that suits SMS and all attachments that are not text are discarded. In order to preserve the formatting for an incoming e-mail message it is therefore necessary to set the message type to e-mail by using DEFAULT_MSGTYPE=EMAIL. Then the conversion will not take place until the message is sent out over a connector of another message type. The message types in EMG are as shown below.

Type	MSGTYPE value	Description
SMS	1	Default message type
DLR	5	Delivery report
EMAIL	6	E-mail message formatted according to RFC 822 or RFC 1521 (MIME)
MMS	7	Multimedia message (MMS)
WAPPUSH	9	WAP push message

Mappings

In order to translate between different character sets, being able to send and receive specific symbols etc it may be necessary to apply mappings to the message data.

Mappings in EMG are defined per connector using the MAPPING keyword. The keyword specifies a filename which contains the mapping or translation table. EMG handles one-to-one, one-to-many, many-to-one and many-to-many.mappings and both text and binary data can be processed.

The format of a mapping file is the following (fields are tab-separated), left-hand column specifies a source data looked for and right-hand column the replacement:

# This is a comment in a sample mapping file.

# First we define a mapping applied when a message is received
# Both character string and hex codes can be used
# First line translates all "a"s into "b"s
IN <
"a"     "b"
"xyz"   "zyx"
03,"a"  "a",03,02
>

# Then a mapping which will be applied to outgoing messages
First line will have no effect (as long as ascii char 0x41 is "A"
OUT <
"a",41,"b"  "aAb"
01,02,03    20,20
>

Mappings are processed top to bottom and the following two examples will generate different result if applied to the string “aabacc”.

# Example 1 (will generate "ddbdcc")
IN <
"a"   "d"
"aa"  "xx"
>

# Example 2 (will generate "xxbdcc")
IN <
"aa"  "xx"
"a"   "d"
>

Incoming (IN) mappings are applied when an incoming message is parsed before it is logged and routed. Therefore the EMG server only sees the translated message data.

Outgoing (OUT) mappings are applied to the message just before they are sent and leave EMG. Therefore the translated message data is never seen in the EMG server.

Characters can be removed by using an empty replacement, “”.

Mappings are only supported for 7-bit text messages.

Address rewriting

Both source and destination addresses can be rewritten by EMG after a message has been received or before a message is sent. This is particularly useful to make sure that addresses comply with the format requirements of the receiving entity.

There are four keywords used for destination address rewriting applied in the order they appear below:

REMOVEPREFIX
REQUIREPREFIX
REPLACEPREFIX
REGEXP_DESTADDR

There are also four corresponding keywords for source address rewriting:

REMOVEPREFIX_SOURCEADDR
REQUIREPREFIX_SOURCEADDR
REPLACEPREFIX_SOURCEADDR
REGEXP_SOURCEADDR

These keywords handle outgoing messages via a specific connector while the corresponding keywords ending with “_IN” handle incoming messages, as in REPLACE_PREFIX_IN and REGEXP_SOURCEADDR_IN.

REPLACEPREFIX was introduced in EMG 2.5 and can do all that the REMOVE/REQUIRE-keywords can do and more. It takes as argument one or more comma-separated pattern/replacement pairs where each matched (prefix) pattern is replaced by the corresponding replacement. An empty pattern can be used to add a prefix and an empty replacement to remove a prefix. All pairs are applied sequentially in the order they appear.

An example:

REPLACEPREFIX=+/,00/,0/46

This example could be used in Sweden where we want to handle addresses received in the following formats:

+46123456	International format, + as prefix
46123456	International format, no prefix
0046123456	International format, 00 as prefix
070123456	National format

When messages are sent out the receiving SMSC expected format is usually an international format without any prefix. The above REPLACEPREFIX statement would perform the wanted rewrite. Once again, operations are performed in sequence from left to right.

First any prefix (`+’ or `00′) is removed if present, then if a national number was supplied the leading 0 is replaced by the country code (46). If instead a +-character was required for the connection only a small modification would be needed:

REPLACEPREFIX=+/,00/,0/46,/+

This example performs the same conversion as the earlier and in addition a “+” is added as the final step.

Masquerading

Address masquerading can be used to hide the real sender of a mobile originated (MO) message before forwarding the message to a third-party. This may be required by some operators in order to get permission to handle MO traffic.

The masquerading is performed using a simple but efficient obfuscation of the address and the algorithm used is reversible and unique for each address. An address with 1-9 digits will be translated into a 10-digit address. Addresses with 10-18 digits will be translated into a 20-digit address.

When using the masquerade keyword on a connector it will be applied to all messages sent and received over that connector.

See the connector keyword MASQUERADE for more information.

Source Address Translation (SAT)

Source Address Translation (SAT) is the procedure of replacing the source address of a message with another address and is most commonly used for implementing bidirectional e-mail to SMS services.

For an e-mail to SMS service an incoming e-mail will obviously have an e-mail address as source (sender) address and before forwarding the message to a mobile phone as SMS it is needed to replace the e-mail address with a valid GSM number. If the user should then be able to reply to the message by simply hitting the reply button the GSM number much be chosen so that the reply message can be routed back to EMG via the SMSC.

Further if the combination of source and destination address can be made unique for each message a specific mobile phone user receives it will be possible to map a specific reply to a message back to the original message sent to the user. We call this threaded messages.

All this functionality can be implemented using SAT pools in EMG where a SAT pool is a pool of source addresses from which EMG chooses an address for each message going out to a mobile phone via EMG.

For the connector where the e-mail is to be received the connector keyword SATPOOL_CREATE_IN is used to indicate that SAT entries should be created when messages are received. On the connector where a reply SMS arrives the SATPOOL_LOOKUP_IN keyword is used in order to perform the lookup for the previously created SAT entry and to rewrite the address back to the original e-mail address.

The SATPOOL_CREATE and SATPOOL_LOOKUP keywords works the same way but operates on messages being sent out via the connector where they are used.

Sample configuration (parts of configuration omitted):

SATPOOL sat1 <
ADDRESSRANGE=10010-10020
THREADED
>

CONNECTOR smtp-in1 <
PROTOCOL=SMTP
...
SATPOOL_CREATE_IN=sat1
# Keep source e-mail address intact
REGEXP_SOURCEADDR_IN=
ROUTE=smsc
...
>

CONNECTOR smsc <
PROTOCOL=SMPP
...
SATPOOL_LOOKUP_IN=sat1
ROUTE=smtp-out1
...
>

CONNECTOR smtp-out1 <
PROTOCOL=SMTP
INSTANCES=10
ADDRESS=#MX
DOMAIN=example.com
MAPPING=mappings/gsm-iso8859-out.map
DEFAULT_MSGTYPE=EMAIL
>

Inheritance and virtual connectors

When having a large configuration with many connectors there will be many connectors with similar configurations. In order to minimize configuration and make it more readable it is possible to let connectors inherit attributes from other connectors and also defining virtual connectors which are only used for inheritance and never instantiated themselves.

Sample configuration:

# Parent for all outgoing SMSC connectors
CONNECTOR smsc-template <
TYPE=OUTGOING
INSTANCES=1
LOGMESSAGE=160
LOGPDU
VIRTUAL
>

CONNECTOR smsc1 <
INHERIT=smsc-template
ADDRESS=10.0.0.1:5000
PROTOCOL=SMPP
USER=user1
PASSWORD=secret
>

CONNECTOR smsc2 <
INHERIT=smsc-template
ADDRESS=10.0.0.2:5000
PROTOCOL=UCP
USER=user1
PASSWORD=secret
>

A connector can only inherit from one other connector but it is possible to have “inheritance chains” where connectors inherit in multiple levels. If a keyword is specified on both a parent connector as well as the child connector the keyword on the child connector is used.

Limiting connector queue sizes

By using the general keyword MAXTOTALQUEUESIZE it is possible to limit the total number of messages in the connector queues server-wide. This is useful when EMG is one of several components through which messages will pass and it is preferable not to let queues build in EMG. When the queue size limit has been reached EMG will reject further incoming messages using a, protocol specific, temporary error code, indicating that the client should check back later for a new try.

It is possible to use the IGNOREMAXTOTALQUEUESIZE keyword for a connector in order for it to accept messages even after the limit has been reached. This is useful for testing purposes as well as for priority connectors.

MAXTOTALQUEUESIZE does not impose an exact limit. It may be exceeded by a smaller amount of messages depending on current server conditions.

In EMG 5.2 MAXTOTALQUEUESIZE_SOFT was introduced providing a way to throttle messages softly by imposing a delay of 0.1 seconds when receiving a message and the specified queue size has been exceeded.

Retry schemes

It is possible to define custom retry schemes through connector keyword RETRYSCHEME.

EMG distinguishes between connector and message errors. If an error occurs for a static connector and the connector is disconnected it will automically try to reconnect and if a number of reconnects fail it will go ERROR temporarily and then another set of reconnects will be performed.

For message errors the default is to terminate the message with status “failed” if an error is received in response to the send operation (for example SMPP operation “submit_sm”). This is not always the case, especially if the error code indicates a temporary error. One such error is SMPP error code ESME_RTHROTTLED (0x58) which indicates messages more frequently than allowed.

Sample retry scheme

Sample retry scheme entry to handle ESME_RTHROTTLED:

#Type (C=Connector, M=Message) and command (*=any)
#   Error code
#         Retrytime (applies to type C only)
#            Connects (applies to type C only)
#               Maxsleep (applies to type C only)
#                  Hold delay (in seconds)
#                     Flags (1:good message, 2:bad domain, 4:bad connector)
M*  0x58  0  0  0  5  1

The above would cause a message, sent over the connector with any message operation, that is rejected with an error code of 0x58 to be requeued and put on hold for 5 seconds before next send attempt. The flag indicates the message is “good”, ie that the error is not related to the specific message.

Sample configurations

Sample connector configurations.

Incoming MGP supporting up to 3 connections

CONNECTOR mgp-in1 <
TYPE=INCOMING
PROTOCOL=MGP
ADDRESS=localhost:7185
# Up to 3 connections
INSTANCES=3
# File where we find authentication info for users
USERS=mgp-users
>

Incoming SMPP supporting up to 10 connections

CONNECTOR smpp-in1 <
TYPE=INCOMING
PROTOCOL=SMPP
ADDRESS=localhost:9000
# Up to 10 connections
INSTANCES=10
# Remove prefixes and convert national swedish numbers
# to international format
REPLACEPREFIX_IN=+/,00/,0/46
REPLACEPREFIX_SOURCEADDR_IN=+/,00/,0/46
# File where we find authentication info for users
USERS=smpp-users
>

Incoming CIMD2 supporting 1 connection

CONNECTOR cimd2-in1 <
TYPE=INCOMING
PROTOCOL=CIMD2
ADDRESS=localhost:9000
# One connection only
INSTANCES=1
# File where we find authentication info for users
USERS=cimd2-users
# Force messages to be routed to connector ebe1
ROUTE=ebe1
>

Outgoing SMPP

CONNECTOR smsc1-smpp <
TYPE=OUTGOING
PROTOCOL=SMPP
# Connector to server 10.0.0.1 port 2775
ADDRESS=10.0.0.1:2775
# One instance is enough
INSTANCES=1
# Username/password to use when authenticating
USERNAME=smppuser
PASSWORD=secret
# Try to connect three times before considered connection failed
MAXFAILEDCONNECTS=3
# When connection failed sleep for 5 minutes before trying again
MAXFAILEDSLEEP=300
# If destination address does not start with "00", add it
REQUIREPREFIX=00
# IDLETIMEOUT defaults to 30 seconds
>

Outgoing UCP using authentication via operation 60

CONNECTOR smsc1-ucp <
TYPE=OUTGOING
PROTOCOL=UCP
# Connector to server 10.0.0.1 port 5000
ADDRESS=10.0.0.1:5000
# One instance is enough
INSTANCES=1
# Username/password to use when authenticating
USERNAME=ucpuser
PASSWORD=secret
# Type of number, short number alias (used by auth operation 60)
AUTHTON=6
# Number plan id, private (used by auth operation 60)
AUTHNPI=5
# Do not timeout when idle
IDLETIMEOUT=0
# Incoming messages should be routed to connector smpp-in1
ROUTE=smpp-in1
>

Outgoing UCP via modem

CONNECTOR smsc2-ucp <
TYPE=OUTGOING
PROTOCOL=UCP
# Phone number to operator's modem pool
ADDRESS=01122334456
# Modem is connected to /dev/ttyS0 (Linux?)
MODEM=ttyS0
# Init string to send before dialing
INIT_STRING=AT&F&K3
# One instance is all a tty can handle
INSTANCES=1
# Wait 2 seconds after connect
WAITDELAY=20
# Wait 2 seconds after message operation is sent
MSGDELAY=20
# Operator only allows us to send 2 messages per session
OPS_MAXPERSESSION=2
# If message does not contain a source address use this
DEFAULT_SOURCEADDR=7654321
>

Outgoing HTTP

CONNECTOR http1 <
TYPE=OUTGOING
PROTOCOL=HTTP
# We use full URL for HTTP
ADDRESS=http://www.myhost.com/cgi-bin/handlemessage.sh
# One instance is enough
INSTANCES=1
# Timeout after 5 seconds idle
IDLETIMEOUT=5
>

Outgoing EBE

CONNECTOR ebe1 <
TYPE=OUTGOING
PROTOCOL=EBE
# Program or script to execute
ADDRESS=/usr/local/bin/ebe-handlesms.sh
# One instance is enough
INSTANCES=1
>

Outgoing GSM

CONNECTOR gsm1 <
# GSM connectors are always outgoing
TYPE=OUTGOING
PROTOCOL=GSM
# TTY to which GSM device is attached
MODEM=cua/a
# Only one instance is allowed
INSTANCES=1
# Poll for incoming messages every 30 seconds
POLLRECEIVE=30
# No SCA in PDU, needed for some GSM devices
#GSMNOSCA
# Memory storage is used (Ericsson devices)
#GSMSTORE=ME
# Required for setting dest address of received messages
FORCE_DESTADDR_IN=4670123123
>