When you create a domain for a PeopleSoft application server, the default configuration for the Tuxedo listener is a variable called %PS_MACH% (which expands to the hostname of the machine). When the domain is booted, the hostname is resolved into an IP address and that is what the Tuxedo listener will bind to.

This is generally what you want for the most common scenarios, but there are cases where you want to change it.

Binding to localhost

I used to do lots of PeopleTools demos in the PeopleSoft corporate visit center when customers came to see what new things we were working on. That meant getting up from my office over in building F and walking across the campus over to the corporate visit center in building A. As a general rule, I always kept my laptop running when walking between the different buildings on the PeopleSoft campus ⁽¹⁾, partly because it would take so long to boot up an entire PeopleSoft demo environment.

Since I would end up changing IP addresses when I switched buildings, I would always set the Tuxedo listeners to listen on localhost. If I had left them on %PS_MACH%, the Tuxedo listener would still be bound to the IP address I had in my office in building F, not the new IP address I would receive when connecting in building A.

Multiple IP addresses

Binding to localhost worked for my scenario, but binding to localhost means that you can only connect webservers running on the same machine as the application server.

An alternate strategy is to use the special IP address 0.0.0.0 in your Tuxedo configuration. That tells Tuxedo to bind to all IP addresses on the machine. That is handy not just for our scenario of changing IP addresses, but also for situations where you have multiple IP addresses on your server. That could be from multiple NICs (network interface cards), virtual IP addresses or some other exotic configuration.

Once you do that, then your PeopleSoft appservers will survive what IP address reconfiguration that gets thrown at them without needing to restart your appserver domains.

1) I became semi well known around PeopleSoft headquarters as the only dork that kept working while walking around, but not as well known as the one PeopleTools developer that used to walk around wearing a black cape. I’m still not sure if that was supposed to be a magician’s cape or Count Dracula.

And no, I never dropped a laptop or tripped 🙂

Labels: 2010, psadmin, Sysadmin, Tuxedo

I had a question the other day about getting going with PeopleSoft Performance Monitor so I thought I’d post a quick roundup/summary of the various material out there.

For those that don’t know, Performance Monitor is included with PeopleTools since PeopleTools 8.44. There’s a little bit to learn on setting it up / understanding it, but it’s a great tool and well worth learning.

Documentation.

You’ll certainly want to have the Performance Monitor PeopleBook handy while going through setting up Performance Monitor.

Presentations.

Back in April Oracle had an Oracle Advisor webcast on the setup and configuration of Performance Monitor. Here is the recording of that (Customer Connection/Metalink/My Oracle Support login required).

Lorne Kaufman of System Efficiency gave a great session at Collaborate this year about setting up Performance Monitor. Unfortunately, I can’t seem to find a link to the actual presentation anywhere. We’ll have to hassle Lorne about that.

David Kurtz of Go-Faster Consultancy has a nice selection of Performance Monitor presentations on his website. He also has a few different blog entries with tips and techniques on Performance Monitor. Those include some performance tuning tips for Performance Monitor itself which he figured out by having Performance Monitor monitor itself 🙂 He also mentions some issues that they had in a large scale environment where Performance Monitor was handling 20-30 million rows of history data each week! Good stuff.

Performance Monitor enhancements.

Brent Martin of ERP Associates has a writeup on their blog about how to get Performance Monitor to send emails when certain conditions are triggered.

Performance Monitor incidents

Here are some quick searches for Performance Monitor incidents by PeopleTools release.

• 8.49
• 8.48
• 8.47
• 8.46

Labels: 2009, Performance, Sysadmin

I helped some folks the other day with an issue that had the potential to be very serious for them; exposure of production data without someone needing to login to the production system. Not good.

The Problem

For testing purposes they have copies of their production PeopleSoft databases; one for Financials and one for HCM. These copies are refreshed regularly and there are scripts run against them to handle some cleaning/sanitizing of the production data, so that the database can be used for testing.

They have some people that they allow to login to these test environments with any account so that they can confirm the results of basic processes. The accounts don’t have default passwords, so this access was limited to the people that they authorized to do testing, but testers with access to these cloned environments could login as anyone.

One of the testers (we’ll call him Fred) was logged in to the Financials test environment as some other user (we’ll call her Sally) to test some process that would normally be initiated by Sally in the production system.

While doing this testing, Fred wanted to enter his timesheet data so he clicked on a link to go to the production HCM system for time entry. Much to his surprise, he was logged in to HCM as Sally! So Fred goes back to the test Financials environment, logs in as a different user, clicks the link into production HCM and sure enough he is logged in as that user in production HCM. To Fred’s credit, Fred alerted the PeopleSoft support team there instead of snooping around in HCM.

How did that happen?

The problem comes from the fact that the production environments were configured to trust each other for PeopleSoft Single Signon, but the node names and node passwords were not changed as part of the environment cloning logic.

So when Fred signed on to the cloned Financials environment, the PS_TOKEN cookies generated are identical to what the production environment would generate (the details of PS_TOKEN cookie are documented in PeopleBooks, but the node name and node password are the important pieces here).

How could this be prevented?

There are several different ways of preventing this from happening. Let’s take the pragmatic ones first.

1) Change the node password as part of the refresh script.

This is the easiest, most expedient thing to do. It is the absolute minimum way to solve this problem.

Although this solves the security problem, it introduces a headache for security auditors because the app server logs in the production environment will be full of warnings about failed logins from bad node passwords. That means that testers accessing production while logged in to the test environment will be indistinguishable from someone trying to break in by generating their own PS_TOKEN cookies.

It’s not nice to annoy your security auditors by purposely creating log entries that look like break-in attempts, but are actually OK 🙂

2) Change the node name.

Changing the node name when you clone environments will also solve the problem (because then the production environments will no longer trust the cloned nodes since they don’t recognize the new names). This also solves the log problem mentioned in the previous item.

Changing the node name can have an impact on Integration Broker testing, but that just means that your integration test scripts need to be aware of this. Not a huge drawback in my opinion.

3) Use distinct passwords for each account in the test environment.

This is easy enough to do, it’s mainly a question of distributing the passwords to the testers so that they can get in and do their jobs. Depending on your organization, this may or may not be easy.

4) Don’t provide passwords to the testers, but allow them to reset them on demand.

Similar to the previous item, but gets around the distribution issue by making it an after the fact auditing issue. If Fred requests the CEO’s password in the test system, it’s very easy to audit for that and force Fred to explain why he felt that was necessary.

Computer security is typically focused on preventing people from doing things, but in some cases this model of auditing and disciplining after the fact may be OK.

This model reminds me of when we took our dog to a dog trainer a long time ago that recommended leaving a steak or something else on the kitchen counter for the dog to find. This was in conjunction with hiding around the corner so that at the moment the dog put his paws up to grab the steak, you would jump out and impress upon the dog that it wouldn’t be wise to do that again. A real-time security audit if you will 🙂

What are some of the less pragmatic options?

5) Have a new version of PeopleTools that can detect when a database is copied, and automatically scrub key information like node passwords, GUID, etc.

The implementation of this would be platform specific, but do-able. It would complicate things like having a hot database backup server though. I’m sure those sorts of issues could be addressed though.

6) Embed additional information beyond just node name and node password (like database name) in the PS_TOKEN cookie.

One issue that I can think of with this is that older PeopleTools versions would not be able to interoperate with this, but it could be a flag on the node definition as to whether interoperability with earlier versions of PeopleTools is required. If interoperability was required, then the existing PS_TOKEN format would be used, otherwise the newer format would be used.

Why didn’t they know about this?

PeopleSoft Single Signon is documented in PeopleBooks, but there aren’t any great writeups out there on the implications of it when cloning a PeopleSoft environment.

There are lots of writeups out there on “How to Clone a PeopleSoft Environment”, and some discuss the importance of getting the node configuration correct, but they typically come at from the perspective of just making sure the environment works; not the security implications of it.

PeopleSoft Single Signon is well documented in PeopleBooks, but it just doesn’t seem to jump out at people when thinking about cloning. I will give a shout-out here to Brent Martin of ERP Associates. Brent’s writeup on cloning a PeopleSoft database is the only one out there that I could find that mentions this issue at all.

Labels: 2009, Audit, Security, Sysadmin