Release Notes for CRs 1310 and 1324

CR 1310:

Create a new eFax status called RECVD. This status will be the first action once the document has been received. Having this status code will allow us to accurately track all inbound document and locate any holes currently in the system.

Requirements

For each recipient of each message entering the Node Server, a log entry will be generated. Each log entry will contain the usual date, time, and processName[pid], followed by the email address of the message originator, then the email address of the message recipient. For example, if 2 messages are received at the Node Server, the first having 2 recipients and the second having 5 recipients, at total of 7 (=2+5) log entries (=7 lines) will be added to the log.

High-level Design

A new queue will be added to the Node Server, upstream of SMTP-in. Each message in this queue will be examined for the RFC 821 "Mail From:" and "RCPT TO:" information. This information will be used to create the log entries.

Since mhsqd will be managing the invocation of this queue's process, the process will generate entries into the /var/adm/MHS.log log file.

Detailed Design

The new queue will be called "SMTP-in_UNIFI". inetd.conf will be modified to invoke SMTPresponder with the "-I SMTP-in_UNIFI" option. A new entry will be added to the mhsqd.cf file to address this new queue. The new process will be called "LogReceivedMessages". LogReceivedMessages will be written in a common shell scripting language, or Perl, as appropriate.

Implementation

The new process name was changed to be called "IncomingMessageLogger". The solution was implemented using Java, since reuse of component developed during the solution to CR 1324, allowed this CR to be fully implemented in 20 lines of new code. Because of the Java solution, and its use of the com/unifi/ufc/Log class, logging is being done to a file separate from MHS.log.

Each recipient log entry takes up about 100 bytes, so Prod Ops needs to ensure that it has enough disk space on the Node Server.

Messages that are generated by processes on this machine, that do not communicate through port 25 (the SMTP port), but are placed directly into the SMTP-in queue, will not be logged by the solution to this CR.

CR 1324:

(GA) Improperly addressed eFaxes are not trapped and NDN'ed

Numerous improperly addressed eFaxes were received in the NOC. The improperly addressed eFaxes progressed through the system until they got stuck in various locations or caused peculiar failures. They should have been caught upon initial entry into the system and NDN'ed with appropriate messages.

Recent examples include:
- a recipient name with the 1st entry having >16 characters
- a recipient name containing a '('
- a '#' delimiter being used before a 'fax number only' eFax fax number
- a '.' being used instead of a '#' as a delimiter
- a recipient name with the first or last character being a '.'

Requirements

(NOTE: Only SMTP mail is being addressed by this CR. X.400 mail is specifically being excluded in the solution to this CR.)

‘delimiter’ refers to a ‘#’, ‘%’ or ‘|’, which separates UNIFI local-part fields

'local-part' refers to a the portion of an SMTP address to the left of the '@' sign.

'domain' refers to a the portion of an SMTP address to the right of the '@' sign.

'eFax recipient' refers to an SMTP address whose domain is 'fax.cc.unifi.net', where 'cc' is the country code. The country code follows the ISO 3166 Code 2 convention for country codes. See www.unicode.org/unicode/onlinedat/countries.html for details. Access Server will assume that if it receives mail addressed to ‘fax.cc.unifi.net’ with ‘cc’ being any 2 characters, that the mail is addressed to an eFax recipient.

For each SMTP email message received by the Access Server, each eFax recipient will be checked to see if the local-part is invalid, per the table below, which was constructed using "Detail Business Requirements - Release 2.0 for EFAX Access Server, Rendering, and Cover Page". As many addressing errors as possible will be checked for each recipient, e.g., as long as there is no local-part addressing error, Recipient Name and Fax Number addressing errors can be checked independently, and the existence of space(s) and parentheses is independent of each other.

For each error type found, make note of the error type, along with the offending eFax recipient(s).

eFax Recipient Addressing Errors: Local-part Addressing

Errors	Bad Address Examples	Error code, suggested message
1 delimiter in the local-part, but nothing to the left or right of the delimiter	#	Please supply either (1) a fax number or (2) a recipient name and fax number. Valid fax address formats include (1) [email protected] or (2) John.Smith#[email protected]. The fax number must include a country code. Please do not include any international access codes (e.g. 001, 00, 011).
1 delimiter in the local-part, but nothing to the left of the delimiter and something to the right	#19785517988	It is not necessary to include a "#" when supplying a fax number only. For example, for the fax number "12345678," [email protected] is the fax address format. Please remove the ":#" and resend.
1 delimiter in the local-part, but nothing to the right of the delimiter and something to the left	Murray.Hays#	A fax number is required to send a message. The fax address format is firstname.lastname#[email protected]. For example, John.Smith#[email protected]. The fax number must include a country code. Please do not include any international access codes (e.g. 001, 00, 144). Please supply a fax number and resend.
More than 1 delimiter in the local-part	Murray#Hays#19785517988 Murray#Hays%19785517988	The valid fax address format is firstname.lastname#[email protected]. For example, John.Smith#[email protected]. The fax number must include a country code. Please do not include any international access codes (e.g. 001, 00, 011). Please remove any additional symbols, such as "#,"and resend.

If any local-part addressing error was found…

No further checking can be done (since the Recipient Name and Fax Number can not be parsed out of the local-part). Exit this table.

Else…

Independently check Recipient Name and Fax Number addressing errors.

eFax Recipient Addressing Errors: Recipient Name Addressing

Errors	Bad Address Examples	Error code, suggested message
one or more spaces in the recipient name	Murray Hays	Valerie Johns: Recipient names should not contain spaces. Please remove the space(s). The valid fax address format is firstname.lastname#[email protected]. For example, John.Smith#[email protected]. You may separate the first and last name by using a period as illustrated in this example.
one or more ‘(‘ and/or ‘)’ in the recipient name	(Murray.Hays)	Fax addresses should not include parentheses, '(' and/or ')'. The valid fax address format is firstname.lastname#[email protected]. For example, John.Smith#[email protected]. Please remove the parenthesis and resend.
no period in the recipient name and >40 characters in the recipient name	MurrayHaysIsTheRecipientOfThisEfaxMessage	The recipient name can include up to 40 characters. The recipient name provided is greater than 40 characters. Please modify the recipient name and resend.
a period at the beginning and/or end of the recipient name	.Murray.Hays Murray.Hays.	Valid recipient name formats include firstname.lastname, such as John.Smith in the fax address John.Smith#[email protected]. Please remove any additional periods before (.John.Smith) or after (John.Smith.) the recipient name.
1-6 periods in the recipient name and >16 characters in the first part of the recipient name	MurrayHaysIsTheRecipient.O.f.ThisEfaxMessage	The first name within the full recipient name can include up to 16 characters. In the example, 'John.Jones#[email protected]', 'John' is the first name in the recipient name. The first name in the recipient name provided is greater than 16 characters. Please modify the recipient name and resend.
1-6 periods in the recipient name and >40 characters in the last part of the recipient name	MurrayHays.I.s.T.h.e.RecipientOfThisEfaxMessage ThatIsBeingSentUsingUNIFI	The last name within the full recipient name can include up to 40 characters. In the example, 'John.Jones#[email protected]', 'Jones' is the last name in the recipient name. The last name in the recipient name provided is greater than 40 characters. Please modify the recipient name and resend.
2-6 periods in the recipient name with any but the last period followed by more than 1 alpha character	MurrayHays.Is.T.h.e.RecipientOfThisEfaxMessage ThatIsBeingSentUsingUNIFI	The fax address may include up to 5 initials between the first name and last name in the recipient name. For example, James.T.U.V.W.X.Smith. The recipient name supplied includes a middle name entry. Please modify the recipient name entry and resend.
7 or more periods in the recipient name	MurrayHays.I.s.T.h.e.R.e.c.i.p.i.e.n.t.OfThisEfaxMessage	The recipient name in the fax address may include a first name, up to five middle initials, and last name in the recipient name. The total number of periods supported is 6. For example, James.T.U.V.W.X.Smith. The recipient name supplied includes more than 6 periods. Please modify and resend.

eFax Recipient Addressing Errors: Fax Number Addressing

Errors	Bad Address Examples	Error code, suggested message
one or more spaces in the fax number	1 508 551 7699	The fax number cannot contain spaces. Please remove the space(s)and resend.
one or more ‘(‘ and/or ‘)’ in the fax number	(15075517699)	The fax number should not contain parentheses, '('and/or ')'. Please remove all parentheses.
a ‘+’ as the first character in the fax number	+15085517699	The fax number should not contain the international access code symbol ('+'). A valid fax number for a fax destination in the United States is 19785517699. The fax number must include a country code. Please do not include any international access codes (e.g. 001, 00, 011). Please remove the international access symbol and resend.
an IAC as the first N characters in the fax number	0014969530699 01185229982999	International access codes (e.g. 001, 00, 011) or symbols (e.g.+) are not valid. A valid fax number for a fax destination in the United States is 19785517699. The fax number must include a country code. Please remove any international access codes and resend.
a ‘9’ as the first character in the fax number	95085517699	It appears that this fax number contains a PBX access code, such as an 8 or a 9. Please remove it. A valid fax number for a fax destination in the United States is 19785517699. The fax number must include a country code.
fax number contains anything but a DIGIT, ‘.’ or ‘-‘		The fax number should only contain numbers, hyphens or periods, and not any other symbols or characters. An example valid fax address format is John.Smith#[email protected]. The '#' is required to separate the recipient name and the fax number.
other problems with the fax number		The fax number is in a format that will not be accepted by UNIFI's Intelligent Delivery Network. A valid fax number includes the country code + area code + local exchange and does not include an international access code. An example for a US fax number would be 1-978-555-1212.

If any invalid recipients were found,

return one NDN to the originator of the email message, addressing each type of invalid recipient. Do NOT deliver the efax to any recipient.

Else,

continue with normal processing for the whole message.

eFax Domain Name Addressing Errors

Errors	Bad Address Examples	Error code, suggested message
More than one "@" in the address	Murray.Hays@[email protected]	Address must contain one and only one '@'.
domain name not in correct format	[email protected]	Domain name must be in the form 'fax.cc.unifi.net'
country code in domain name more or less than 2 characters	[email protected]	Domain name contains a country code that is not 2 characters long.

Ensure that each message communicated in an NDN is generated by Martha Tassinari, Valerie Johns, or her designee.

The NDN will be similar to the following example:

UNIFI Communications has received your UNIFI.eFax message, but cannot successfully send your message until you fix the following addressing errors. Please resend your UNIFI.eFax message once these addresses have been fixed. If you would like more information, consult our on-line service at http://www.unifi.com/support/quickframes.html.

Thank you for your patience.

Error 115 - The recipient name in the fax address may include a first name, up to five middle initials, and last name in the recipient name. The total number of periods supported is 6. For example, James.T.U.V.W.X.Smith. The recipient name supplied includes more than 6 periods. Please modify and resend. Your following recipient addresses contain this error and need to be fixed.

MurrayHays.I.s.T.h.e.R.e.c.i.p.i.e.n.t.OfThisEfaxMessage#[email protected]

Error 114 - The fax address may include up to 5 initials between the first name and last name in the recipient name. For example, James.T.U.V.W.X.Smith. The recipient name supplied includes a middle name entry. Please modify the recipient name entry and resend. Your following recipient addresses contain this error and need to be fixed.

.Murray.Hays#[email protected]

MurrayHays.I.s.T.h.e.R.e.c.i.p.i.e.n.t.OfThisEfaxMessage#[email protected]

Herr.Jurgen.Tepper#[email protected]

Mr._James.Brown#[email protected]

Mr.James.Brown.President#[email protected]

Mdme.Isabelle.LeBlanc#[email protected]

Mr.Joseph.I.Brown#[email protected]

Ian.DeClive.Lowe#[email protected]

Joseph.Brown.Vice.President#[email protected]

Joseph.L.Brown.Jr#[email protected]

Error 113 - The last name within the full recipient name can include up to 40 characters. In the example, 'John.Jones#[email protected]', 'Jones' is the last name in the recipient name. The last name in the recipient name provided is greater than 40 characters. Please modify the recipient name and resend. Your following recipient addresses contain this error and need to be fixed.

MurrayHays.I.s.T.h.e.RecipientOfThisEfaxMessageThatIsBeingSentUsingUNIFI#[email protected]

MurrayHays.Is.T.h.e.RecipientOfThisEfaxMessageThatIsBeingSentUsingUNIFI#[email protected]

Error 112 - The first name within the full recipient name can include up to 16 characters. In the example, 'John.Jones#[email protected]', 'John' is the first name in the recipient name. The first name in the recipient name provided is greater than 16 characters. Please modify the recipient name and resend. Your following recipient addresses contain this error and need to be fixed.

MurrayHaysIsTheRecipient.O.f.ThisEfaxMessage#[email protected]

Error 111 - Valid recipient name formats include firstname.lastname, such as John.Smith in the fax address John.Smith#[email protected]. Please remove any additional periods before (.John.Smith) or after (John.Smith.) the recipient name. Your following recipient addresses contain this error and need to be fixed.

.Murray.Hays#[email protected]

Murray.Hays.#[email protected]

Error 110 - The recipient name can include up to 40 characters. The recipient name provided is greater than 40 characters. Please modify the recipient name and resend. Your following recipient addresses contain this error and need to be fixed.

MurrayHaysIsTheRecipientOfThisEfaxMessage#[email protected]

Error 109 - Fax addresses should not include parentheses, '(' and/or ')'. The valid fax address format is firstname.lastname#[email protected]. For example, John.Smith#[email protected]. Please remove the parenthesis and resend. Your following recipient addresses contain this error and need to be fixed.

(Murray.Hays)#[email protected]

---------------- Your original message ----------------

Date: Wed Dec 09 13:46:33 EST 1998

From: [email protected]

To: (Murray.Hays)#[email protected], MurrayHaysIsTheRecipientOfThisEfaxMessage#[email protected], .Murray.Hays#[email protected], Murray.Hays.#[email protected], MurrayHaysIsTheRecipient.O.f.ThisEfaxMessage#[email protected], MurrayHays.I.s.T.h.e.RecipientOfThisEfaxMessageThatIsBeingSentUsingUNIFI#[email protected], MurrayHays.Is.T.h.e.RecipientOfThisEfaxMessageThatIsBeingSentUsingUNIFI#[email protected], MurrayHays.I.s.T.h.e.R.e.c.i.p.i.e.n.t.OfThisEfaxMessage#[email protected], Mr.Joseph.I.Brown#[email protected], Joseph.L.Brown.Jr#[email protected], Mr._James.Brown#[email protected], Mr.James.Brown.President#[email protected], Joseph.Brown.Vice.President#[email protected], Ian.DeClive.Lowe#[email protected], Herr.Jurgen.Tepper#[email protected], Mdme.Isabelle.LeBlanc#[email protected]

Subject: subject

[Image]

Logging

The eFax message validator will generate log entries into a log file using the existing com.unifi.ufc.Log class. The minimum information that will be logged is:

starting and stopping of the message validator
for each invalid eFax recipient, for each addressing error type: error number and complete eFax recipient address

High-level Design

On the DMZ machine, insert a queue before SMTP-in, in which SMTPresponder will insert the incoming message. Perform the above described processing in this queue. Normal processing will proceed on to the SMTP-in queue. An NDN'ed message will be sent to mailman.faxinter.com for delivery to the message originator.

NOTE: The choice of the above queue location for the address processing is based upon the fact that sendmail.turbo modifies the addresses in the message, and we need to examine the addresses prior to any modification occurring.

Detailed Design

Modify /etc/inetd.conf to kick off SMTPresponder with the "-I SMTP-in_UNIFI", which will cause all incoming messages to be deposited in the SMTP-in_UNIFI queue.

Modify /var/mhs/mhsqd.cf to add the following lines:

SMTP-in_UNIFI 1 /usr/java/bin/jre -classpath <needed .jar and .zip files> EfaxMessageValidator mailman.faxinter.com -d 1 -W 300000 -w 500 SMTP-in

NDN'ing – will be performed using the JavaMail MimeMessage and Transport classes.

The initial design was reviewed by Jim Giza, Bob Walter and Leo Holewa.

Implementation

The following classes were created in the com.unifi.apps.queue package:

SMTPMessage – an RFC 821/822 style message (CDS style)

SMTPMessageProcessor – an abstract class that can be started by mhsqd, will monitor a CDS queue for messages and process the messages using the abstract method processMessage() that is implemented by a derived class.

The following classes were created in the com.unifi.apps.adval package:

NDN – a class derived from MimeMessage that embodies a Non-Delivery Notice

EfaxMessageValidator – a class derived from SMTPMessageProcessor that NDN’s messages containing incorrectly addressed eFax recipients, or forwards correctly addressed messages to SMTP-in for further processing.

EfaxRecipient – a ‘static’ class (i.e., not intended to be instantiated; only static methods) used to validate eFax recipients in a message

EfaxRecipientError – a ‘static’ class containing recipient addressing error numbers and accompanying text

Leo Holewa reviewed the code, providing valuable feedback on exception handling strategies, which was incorporated into the final code.

Testing

I successfully ran unit tests on the:

SMTPMessage class that exercised all of its public methods

EfaxRecipient class that validated all of the good and bad addresses that I could find in the Access Server 2.0 Requirements document, applicable CR’s, any others of which I could think, for a total of 93 different eFax recipient addresses.

I also installed the solution on websqa02 (development’s DMZ machine) and successfully system tested it using Jon Campbell’s Datamail. I used a collection of messages that consisted of:

1 good message containing a To, Cc and Bcc
6 messages containing a single type of addressing error (including CR 1411’s problem address)
1 message containing a number of good and bad To, Cc and Bcc local-part addressing errors
2 messages containing a number of good and bad To, Cc and Bcc recipient name addressing errors
1 message containing a number of good and bad To, Cc and Bcc fax number addressing errors

NOTE: The good and bad addresses that were tested at the system level included only a subset of the ones tested at the unit test level, since, for example, the bad addresses with spaces in the local-part can’t actually get sent past the originator’s email message user agent (MUA), or email client.

I ran over 400 of the messages, in quick succession, through the new queue, without a problem. I received the correct number of NDN’s, and the good messages were correctly placed in the SMTP-in queue. This testing was done to exercise the capability of the new SMTPMessageProcessor and EfaxMessageValidator (a subclass of SMTPMessageProcessor) classes in acting as message queue processors.

Each message took about .3 seconds to pass through the queue when I artificially just put NDN’ed messages into the SMTP-out queue. When I, correctly, mailed the NDN’ed using mailman.faxinter.com as a Transport Agent (TA), each message took about .6 seconds to get through the queue. While this seems like a large increase in the amount of time, remember that the amount of NDN’ed messages is expected to be a small proportion of total messages.

Deployment (PROD OPS PLEASE NOTE)

When invoked, EfaxMessageValidator will process all eligible messages in the queue at the time of invocation. It will then optionally stay alive for a while, processing other messages that come into the queue, or become eligible to be processed.

While processing a message, the EfaxMessageValidator will set its eligibility time to 1 year from the time its processing starts. So, if you see a message stuck in the SMTP-in_UNIFI queue on the DMZ with an eligibility time almost a year in the future (which looks like 99.12.10 as an eligibility time), it might be a message that started to be processed, but didn’t get finished because of some unknown problem. One way to get the message processed would be to use xmhadm to re-schedule the message and hope that processing completes this go round.

To see message eligibility times, view the output of:
manxq –c status –b SMTP-in_UNIFI

To re-schedule a message from the command line, type:
manxq –c change –e 0 <message filename> SMTP-in_UNIFI

where <message filename> is the full filepath of the message that you got from doing the previous ‘manxq –c status’.

A log file will be generated, and automatically rolled over daily, as necessary. The log for the SMTP-in_UNIFI queue (EfaxMessageValidator process) on the DMZ will be found in /usr/var/adm/EfaxMessageValidatorXXXX.log, where XXXX is the date.
Each recipient addressing error will be logged. Each of these log entries takes up about 100 bytes, so be sure that enough disk space is available to handle this log. Also, older logs may need to be deleted manually, if disk space needs to be conserved.

/etc/inetd.conf will be modified
/usr/var/mhs/mhsqd.cf will be modified.
DO NOT modify the SMTP-in_UNIFI line to allow more than 1 EfaxMessageValidator process to be started up at one time.

The EfaxMessageValidator options "mailman.faxinter.com -d 1 -W 300000 -w 500" can be modified, if desired, as follows:

"mailman.faxinter.com" can be changed to any appropriate SMTP mail Transport Agent that is capable of delivering SMTP mail to the world.
"-d 1" can be changed to "-d 2" if a problem is occurring and more debug information is needed in the log. Please DO NOT remove "-d 1" or "-d 2" from the invocation of EfaxMessageValidator, as Product Support is counting on being able to look at the log file to get data to analyze the types of addressing errors that are being encountered most often.
"-W 300000" means that the EfaxMessageValidator will stay alive, looking for new, incoming messages (since the last processed message) for 300000 milliseconds (5 minutes) and processing them before dying and having to be re-invoked. If you find that it be more useful to have a different number in production, feel free to change it. (A negative number means that EfaxMessageValidator will NEVER die.)
"-w 500" means that the EfaxMessageValidator will look for new, incoming messages every 500 milliseconds (1/2 second) while it is staying alive. Change this message polling rate if you find it useful to do so.
Be aware that EfaxMessageValidator is ultimately dependent on the IAC_DATABASE.DAT file, which is sometimes updated (e.g., CR’s 1383 and 1480), and would need to be updated here if so.