PeopleTools 8.51 & Exchange 2010: Part 3, Testing and Conclusion

We downloaded Javamail 1.4.4, the latest version available.  It comes in the mail.jar file. This version tested out nice and speedy, but the SMTP Trace stopped working.  That trace is an App Server configuration setting that we had been using to identify where the slowdown was happening.  After that, we saw in the case notes that PS development was using Javamail 1.4.3.  We downloaded it, tested it, and now we are back to the slow email, but the SMTP Trace worked.  The other downside for the 1.4.4 version was that PS wouldn't recommend that option, and so wouldn't support it.

On the Exchange side, the admins were able to configure a connector just for the IP address of the PeopleSoft server.  And on that connector, the Tarpitinterval is set to zero.  Testing went through multiple iterations to get the Exchange server right.  The next test round had each email taking one second.  Better than five seconds, but not acceptable.  We had eliminated the 

         Tarpit for '0.00:00:05'

 but we still saw this

         Tarpit for '0.00:00:01' due to 'DelayedAck'

These were entries in the Exchange logs.  On the PeopleSoft side, we just saw a delay from one step to the next.  Found a reference to this type of delay here.  To get rid of the one second delay the Exchange server admins had to adjust the ConnectionTimeout and set MaxAcknowledgementDelay to zero.  After they made that change, it worked as quickly as it had with Exchange 2003.  

And we decided that for future development, we would batch the email processing.

PeopleTools 8.51 & Exchange 2010: Part 2, The Options

We opened a case with Oracle about the slowdown.  PeopleSoft Development said they had seen this five second per email issue before, but weren't able to get a customer to test it to provide "concrete evidence".  We sent them SMTP trace logs from the application servers and we sent them logs from the Exchange server.  They wanted us to test with Javamail 1.4.3, rather than the Javamail 1.3 that was supplied with PT 8.51.  Evidently PeopleSoft development was using 1.4.3.  The newer version of Javamail will not to send the NOOP command, but will send RSET command which might solve the problem.  Javamail 1.3 does not have this feature.  They wanted us to try it out to see if it worked, but they couldn't recommend that solution since it hadn't been fully tested.  It would be standard in some future version of PeopleTools.

The option that the PeopleSoft developers recommended was to have us to try setting the TarpitInterval to 0.  From the Microsoft Technet link, "The TarpitInterval parameter specifies the period of time to delay an SMTP response to a remote server that Exchange determines may be abusing the connection. Authenticated connections are never delayed in this manner. The default value is 5 seconds. ...When you set the value to 00:00:00, you disable the tarpitting interval."

The Exchange server admins were loath to disable the new security feature that they had just gained with their upgrade.  They didn't want to change a global setting on their servers for just one application.  They were not able to give us our own Exchange server.  I asked about setting the PeopleSoft application servers as "authenticated connections", but was told that it was too much work to pass credentials.

The other option was to rewrite the code in TAM that sent the emails to all the rejected applicants.  It is a custom process.  The delivered process sent snail-mail to the applicants, and as a cost saving measure we changed it to send email to all the applicants that had email addresses entered.  The easy solution would be to cut the PeopleCode out of the process and paste it into an Application Engine process.  Using the same code would be easy, but there would still be a five second per email delay later.  The other option would be to rewrite it in SQR.  The SQR would use the Unix sendmail, and so would run faster.  Either of those options would handle the emails in a batched mode.  The user would no longer need to wait for the email to be sent before moving on to the next screen.  While it may seem a bad practice to have the user wait while we send an email, you need to remember that with the previous Exchange server we could sent 100 emails in one second.  The downside for these solutions is that it doesn't address the five second per email delay for the email sent from the application server.  It will affect other parts of the system later.  We want to deploy more AWE (Workflow) processes either as part of or after the upgrade.

PeopleTools 8.51 & Exchange 2010: Part 1, The Problem

Our Exchange servers were upgraded last month from Exchange 2003 to Exchange 2010.  In TAM the recruiters have a process to close a job request, and it will send email to each of the applicants. This process can send out hundreds of emails.

As soon as they upgraded the mail server to Exchange 2010, the recruiters started seeing a big slowdown in their process. We traced it back to a five second delay for each email at the Exchange 2010 server. The Exchange server admins said the "PS application server is requesting the connection to stay open with a NOOP command. There is no response to close connection within 5 seconds so exchange closes that connection. 2003 ignores this basically due to tarpit functionality is not a feature on 2003 which noop is triggering on 2010."  Appropriate to its name, the tarpit is meant to slow down an automated process that a spammer might use.  You can read more on the Tarpitinterval here.

What does this issue have to do with PeopleTools 8.51?  If we were back on PeopleTools 8.49.10, we would not have this problem.  Starting in PT 8.49.12 the application servers now use Javamail to send out email.  Before that point, the application server used Sendmail, which didn't leave the connection open with a NOOP.  Javamail is part of the mail.jar file, and for this version of tools, the version of Javamail is 1.3.3 (this is important later).

BTW, the other PeopleSoft processes that send email have the same five second per email delay, but they don't send hundreds of emails.