"Processing" and cache

Right at go-live from 8.49 to 8.51 we had some people that complained about sitting and waiting for the "Processing".  At first I thought they were using the old term for the new spinning pinwheel.  Then I got the same thing.  On one page, I got that old flashing gif that symbolized "I'm working on it, please wait".

I deleted my browser cache and the problem went away.  A bunch of other problems went away too.  In particular, the eRecruit job search started working.  This happened at the live system checkout after we applied Tools 8.51.  We had all the testers delete their browser cache, and many issues cleared up.

We sent out communication to the users so that everyone who ran into issues would delete their browser cache.  Then we started getting complaints from external job applicants, who we also had delete their browser cache.

We needed a better solution.  We had never had this type of problem with a Tools upgrade before.  I suspect that Oracle rewrote many Javascripts this time, rather than replacing them.  After some research, we found the following to address the problem.

On the Web Profile Configuration  / Caching, go to the Directories section.  Change the name of all of the directories.  I added the Tools version to the end.  So for our current version of 8.51.13, it's set like this:

Remember to change it for all of your Web Profiles.  We have a separate one for external applicants.

Patching to PeopleTools 8.51.13

We applied a patch to PeopleTools 8.51 this last weekend.  It went fairly smoothly and according to schedule.  The patch brought us from PT 8.51.11 to 8.51.13.

The main reason for applying the patch was to get the session timeout to work.  At patch level 11 we had problems with the session timeout not working.  It would occasionally work, but there wasn't a pattern to when it would or wouldn't.  A session would sit there past the warning time and timeout.  Then the user would click on a link or tab, and then they would get a login screen.  This login lead to a different set of problems.  After this re-logging in, the users would either get an "Authorization error - Contact your security administrator", or they re-login and the breadcrumbs were missing.  We had this problem on 8.51.07 too.

Opening another new window

Okay, not on upgrading today ... more of an enhancement.

On most pages there's a link for opening a "New Window".  The newwin servlet can do the same thing, if you have the URL to open.  Over at PeopleSoft Wiki they have a function that takes your current URL and gives you a new URL that will open as a second window with the "_1".  The limitation with the listed code is that if you already have a second window open, it won't work.  So here's an improvement.

/******************************************************
AnotherWinUrl(&strUrl)
This function modifies a URL to return a new window URL. It allows a new state block to be generated in a separate browser instance.
http://servername:port/psp/ps/EMPLOYEE/HRMS/c/...
 or
http://servername:port/psp_1/ps/EMPLOYEE/HRMS/c/...    (or _2, etc)
 becomes
http://servername:port/psp/ps_newwin/EMPLOYEE/HRMS/c/...  
 which the web server turns into  
http://server:port/psp/ps_1/EMPLOYEE/HRMS/c/...        (or _3, etc)
Input parameters: &strUrl - a URL to manipulate Output parameters:
&strUrlModified - New URL with _newwin parameter
************************************************/
Function AnotherWinUrl(&strUrl As string) Returns string;
   Local string &sRegEx, &sReplace, &NewWinURL;
   /* Declare java object */
   Local JavaObject &jUrl;


   &sRegEx =
   "/(ps[cp])/([^\/_]*)?(_[1-9])?/([^\/]*)?/([^\/]*)?/([csehtqwnf]{1})/";
   &sReplace = "/$1/$2_newwin/$4/$5/$6/";

   /* Instantiate objects */
   &jUrl = CreateJavaObject("java.lang.String", &strUrl);
   &NewWinURL = &jUrl.replaceAll(&sRegex, &sReplace);

   /* Return modified URL */
   Return &NewWinURL;
End-Function;

The original second pattern ($2) said it was made of non-slash characters. I added that it should also not be an underscore. Then I added a pattern ($3) of an underscore followed by a number. Then this third pattern is excluded in the replace string.

The PeopleSoft Wiki had a link to this site, http://gskinner.com/RegExr/ which makes testing these much easier.

RTF Editor ( CKEditor ) Spellcheck

CKEditor is the RTF editor that comes with PeopleTools. If you check out the editor at its web site, it has a nice spell check built in. But PeopleSoft seems to have disabled it so we use their PeopleTools spell check. Our customers don't care for the PeopleTools spell check.

We've gone through how to turn on the spell check at the CKEditor web site, but it doesn't work in PeopleSoft. We can turn on other tool bar settings, but not the spell check.  To auto-start the SCAYT(Spell Check As You Type), you edit its configPsft.js and in the following line change the false to true:
     CKEDITOR.config.scayt_autoStartup = false;

The CKEDITOR.config.scayt_autoStartup suggestion worked if you are using Firefox, but not with Internet Explorer. Our users need to use IE. Other browsers are not supported by our Desktop group, and the users don't have the permissions to install software. Not only does it not auto-start in IE -- the spellcheck does not work at all.  But using the demo at the CKEdit site, SCAYT works just fine in both IE and Firefox.  Oracle has locked it in some way that we can't unlock.

Let me know in the comments if you have found a fix or workaround for this issue.

PeopleTools 8.51 & Exchange 2010: Part 3, Testing and Conclusion

We downloaded Javamail 1.4.4, the latest version available.  It comes in the mail.jar file. This version tested out nice and speedy, but the SMTP Trace stopped working.  That trace is an App Server configuration setting that we had been using to identify where the slowdown was happening.  After that, we saw in the case notes that PS development was using Javamail 1.4.3.  We downloaded it, tested it, and now we are back to the slow email, but the SMTP Trace worked.  The other downside for the 1.4.4 version was that PS wouldn't recommend that option, and so wouldn't support it.

On the Exchange side, the admins were able to configure a connector just for the IP address of the PeopleSoft server.  And on that connector, the Tarpitinterval is set to zero.  Testing went through multiple iterations to get the Exchange server right.  The next test round had each email taking one second.  Better than five seconds, but not acceptable.  We had eliminated the 

         Tarpit for '0.00:00:05'

 but we still saw this

         Tarpit for '0.00:00:01' due to 'DelayedAck'

These were entries in the Exchange logs.  On the PeopleSoft side, we just saw a delay from one step to the next.  Found a reference to this type of delay here.  To get rid of the one second delay the Exchange server admins had to adjust the ConnectionTimeout and set MaxAcknowledgementDelay to zero.  After they made that change, it worked as quickly as it had with Exchange 2003.  

And we decided that for future development, we would batch the email processing.

PeopleTools 8.51 & Exchange 2010: Part 2, The Options

We opened a case with Oracle about the slowdown.  PeopleSoft Development said they had seen this five second per email issue before, but weren't able to get a customer to test it to provide "concrete evidence".  We sent them SMTP trace logs from the application servers and we sent them logs from the Exchange server.  They wanted us to test with Javamail 1.4.3, rather than the Javamail 1.3 that was supplied with PT 8.51.  Evidently PeopleSoft development was using 1.4.3.  The newer version of Javamail will not to send the NOOP command, but will send RSET command which might solve the problem.  Javamail 1.3 does not have this feature.  They wanted us to try it out to see if it worked, but they couldn't recommend that solution since it hadn't been fully tested.  It would be standard in some future version of PeopleTools.

The option that the PeopleSoft developers recommended was to have us to try setting the TarpitInterval to 0.  From the Microsoft Technet link, "The TarpitInterval parameter specifies the period of time to delay an SMTP response to a remote server that Exchange determines may be abusing the connection. Authenticated connections are never delayed in this manner. The default value is 5 seconds. ...When you set the value to 00:00:00, you disable the tarpitting interval."

The Exchange server admins were loath to disable the new security feature that they had just gained with their upgrade.  They didn't want to change a global setting on their servers for just one application.  They were not able to give us our own Exchange server.  I asked about setting the PeopleSoft application servers as "authenticated connections", but was told that it was too much work to pass credentials.

The other option was to rewrite the code in TAM that sent the emails to all the rejected applicants.  It is a custom process.  The delivered process sent snail-mail to the applicants, and as a cost saving measure we changed it to send email to all the applicants that had email addresses entered.  The easy solution would be to cut the PeopleCode out of the process and paste it into an Application Engine process.  Using the same code would be easy, but there would still be a five second per email delay later.  The other option would be to rewrite it in SQR.  The SQR would use the Unix sendmail, and so would run faster.  Either of those options would handle the emails in a batched mode.  The user would no longer need to wait for the email to be sent before moving on to the next screen.  While it may seem a bad practice to have the user wait while we send an email, you need to remember that with the previous Exchange server we could sent 100 emails in one second.  The downside for these solutions is that it doesn't address the five second per email delay for the email sent from the application server.  It will affect other parts of the system later.  We want to deploy more AWE (Workflow) processes either as part of or after the upgrade.

PeopleTools 8.51 & Exchange 2010: Part 1, The Problem

Our Exchange servers were upgraded last month from Exchange 2003 to Exchange 2010.  In TAM the recruiters have a process to close a job request, and it will send email to each of the applicants. This process can send out hundreds of emails.

As soon as they upgraded the mail server to Exchange 2010, the recruiters started seeing a big slowdown in their process. We traced it back to a five second delay for each email at the Exchange 2010 server. The Exchange server admins said the "PS application server is requesting the connection to stay open with a NOOP command. There is no response to close connection within 5 seconds so exchange closes that connection. 2003 ignores this basically due to tarpit functionality is not a feature on 2003 which noop is triggering on 2010."  Appropriate to its name, the tarpit is meant to slow down an automated process that a spammer might use.  You can read more on the Tarpitinterval here.

What does this issue have to do with PeopleTools 8.51?  If we were back on PeopleTools 8.49.10, we would not have this problem.  Starting in PT 8.49.12 the application servers now use Javamail to send out email.  Before that point, the application server used Sendmail, which didn't leave the connection open with a NOOP.  Javamail is part of the mail.jar file, and for this version of tools, the version of Javamail is 1.3.3 (this is important later).

BTW, the other PeopleSoft processes that send email have the same five second per email delay, but they don't send hundreds of emails.  

Session Timeout

We had problems on PT 8.51.07 and again after we applied patch 11 with the session timeout.  It lead to a few different symptoms that users reported.  The problem was that the session would not produce the timeout warning or the Your Session Has Expired page.  You could leave the session there for a hour past the timeout window, and it would never look like it timed out.  But if you clicked on a tab or a menu, it would take you to the login screen.  After login it would sometimes take you to the page you were on before, but the breadcrumbs and header were missing.  Other times it would give you an Authorization Error.

We are testing 8.51.13 and now it's fixed.  It works like it did back in 8.49.  It was listed in patch 8.51.12, but we applied 13 too in case it would fix some of our other issues.  Maybe next time.


And PeopleTools 8.52 did come out by the end of October.  We need to continue with our application upgrade to 9.1.  We're on 8.51.11, patching to 8.51.13 soon.  We aren't considering 8.52 until after it the application upgrade is done.

Beginning with the UMRUG

Yesterday was the Upper Midwest Regional User Group meeting for PeopleSoft and JD Edwards.  I attended some good presentations on AWE and real time integration between PeopleSoft and the cloud.

AWE (Approval Workflow Engine) is more than a new name for Workflow.  It's a new process where the events are entered on-line instead of in the App Designer.  There are delivered processes for Time & Labor 9.1,  TAM, and for some of eProfile.  There's a new process for delegating approval work to proxies.  And in multi-step approvals, there is now a send-back to send it back one step.

During the technical break out session, I got the impression that PT8.52 will be out soon.  The PeopleSoft guy wouldn't say anything concrete, but he had info from the release notes.  Also during that session, I asked about a couple of our current issues, but no one else had run into these problems.  More on them later.