21 January 2014

DSRA0302E: XAException. Error code: XAER_RMERR (-3). Exception:

It's been published in several places, I guess but for the sake of simplicity:

Websphere 8.5, Oracle 11.2, datasource and transactions. You start your WAS server and see this in the logs:

[20.01.14 13:36:43:670 CET] 0000004f WSRdbXaResour E   DSRA0304E:  Wystapil wyjatek XAException. Tresc i szczególy wyjatku XAException: The cause is               : null.
[20.01.14 13:36:43:671 CET] 0000004f WSRdbXaResour E   DSRA0302E:  Wystapil wyjatek XAException.  Kod bledu: XAER_RMERR (-3).  Wyjatek: <null>
[20.01.14 13:36:43:671 CET] 0000004f XARminst      E   WTRN0037W: Usluga transakcji napotkala blad w operacji xa_recover. Zasób: com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@c6a9f4fb. Kod bledu: XAER_RMERR. Dane sledzenia stosu wyjatku: javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709)
at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:1116)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:141)
at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:716)
at com.ibm.tx.jta.impl.PartnerLogTable.recover(PartnerLogTable.java:431)
at com.ibm.tx.jta.impl.RecoveryManager.resync(RecoveryManager.java:1534)
at com.ibm.tx.jta.impl.RecoveryManager.performResync(RecoveryManager.java:2267)
at com.ibm.ws.tx.jta.RecoveryManager.performResync(RecoveryManager.java:117)
at com.ibm.tx.jta.impl.RecoveryManager.run(RecoveryManager.java:2220)
at java.lang.Thread.run(Thread.java:772)

the key is:

at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709)

as there are some pending, unrecoverable transaction in the transaction log. there are two ways to deal with it:
- start your server, nevermind your business applications not working, go to console, Application servers > server1 > Transaction service, then to Runtime tab and there you can see some options to manually manage pending troublesome transactions:

- or alternate: stop server, go to <profileroot>/tranlog/(respective path)/tranlog & partnerlog, delete tranlog and partnerlog files. start server.

Now, the way no. 2 is advisable for development systems, where any data loss is irrelevant. For production, proceed with care and try way no.1 first. Should it fail, go hardcore and delete tranlog.
Good luck.

29 November 2013

TSPM 7.1 installation - ADFS0124 Error occurred during upload to: upload/cells/nullNode01Cell/rtss/security-services.xmi. Exception: java.net.UnknownHostException: null

So, this time something about powerful security tool from IBM - Tivoli Security Policy Manager 7.1. This software let's you manage your security policies (surprisingly :) on various servers from single point of authoring and distribution (PAP, PDP). It can manage multiple policy enforcement points (PEPs) on the online and offline basis (online means pushing policies to PEPs on-the-go, while offline means that you can author policies on PAP and distribute them asynchronously, or even over portable memory (when your PEP is in DMZ or something like that). Above features are very well described on IBM pages, so let's cut this here.

The problem I had with installation was a bit surprising (believe me, I overcame numerous problems with this product before, that included temporarily giving up on installing TSPM 7.1 on 64bit Linux - which is possible only after adding 32bit compatibility libraries), so when this time it went smooth, up to some point, I was very disappointed to see it failed again.

Quick look in the log and voila:

Nov 15, 2013 4:25:31 PM com.tivoli.am.fim.rte.config.impl.RuntimeConfigurationImpl putConfiguration
INFO: com.tivoli.am.fim.rte.config.exception.RuntimeConfigurationRepositoryException: +null
        at com.tivoli.am.fim.rte.config.impl.RuntimeConfigurationImpl.putConfiguration(RuntimeConfigurationImpl.java:491)
        at com.ibm.tspm.installer.platform.actions.ConfigReposUtils.putConfiguration(Unknown Source)
        at com.ibm.tspm.installer.platform.actions.ConfigReposUtils.putConfiguration(Unknown Source)
        at com.ibm.tspm.installer.platform.actions.ConfigReposUtils.pushFiles(Unknown Source)
        at com.ibm.tspm.installer.platform.actions.ConfigReposUtils.install(Unknown Source)
        at com.ibm.tspm.installer.platform.actions.ConfigReposUtils.run(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:618)
        at com.ibm.cic.agent.core.commonNativeInstallAdapter.Invoke$InvokeRunnable.run(Invoke.java:189)
        at java.lang.Thread.run(Thread.java:810)
Caused by: java.security.PrivilegedActionException: com.ibm.websphere.management.exception.RepositoryException: com.ibm.websphere.management.filetransfer.client.TransferFailedException: Error occurred during upload to: upload/cells/nullNode01Cell/rtss/security-services.xmi. Exception: java.net.UnknownHostException: null
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:4265)
        at com.tivoli.am.fim.rte.config.impl.RuntimeConfigurationImpl.putConfiguration(RuntimeConfigurationImpl.java:487)
        ... 11 more
Caused by: com.ibm.websphere.management.exception.RepositoryException: com.ibm.websphere.management.filetransfer.client.TransferFailedException: Error occurred during upload to: upload/cells/nullNode01Cell/rtss/security-services.xmi. Exception: java.net.UnknownHostException: null
        at com.ibm.ws.management.repository.client.JMXRemoteConfigRepositoryClient.upload(JMXRemoteConfigRepositoryClient.java:195)
        at com.ibm.ws.management.repository.client.JMXRemoteConfigRepositoryClient.create(JMXRemoteConfigRepositoryClient.java:204)
        at com.tivoli.am.fim.rte.config.impl.PrivilegedConfigCreateAction.run(PrivilegedConfigCreateAction.java:41)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:4253)
        ... 12 more
Caused by: com.ibm.websphere.management.filetransfer.client.TransferFailedException: Error occurred during upload to: upload/cells/nullNode01Cell/rtss/security-services.xmi. Exception: java.net.UnknownHostException: null
        at com.ibm.ws.management.filetransfer.client.FileTransferClientImpl.uploadFile(FileTransferClientImpl.java:275)
        at com.ibm.ws.management.repository.client.JMXRemoteConfigRepositoryClient.upload(JMXRemoteConfigRepositoryClient.java:189)
        ... 15 more
Why would it have problem with hostname when I'm installing locally? Of course! Quick glance on /etc/hosts revealed that only loopback address has been defined, so no hostname could be resolved to any ip address, except for localhost. What is this needed for? Behind the scenes of Installation Manager, there's some work done over wsadmin (WAS administrative client), which essentially needs network infrastructure working fine, that includes possibility to resolve system's own hostname. If that fails, wsadmin has no point to connect to. 

Adding machine's own ip and hostname (both fully qualified and short) to /etc/hosts solved the issue.

Now, one interesting thing to add here is that problem can be seen even by looking oin WebSphere cell name. It is:


and it is very wrong - it usually (by default, at least) should contain hostname - which in that case was missing and replaced by null. Of course you may fancy naming your machine "null" but that was not the case.

Ok, so good luck installing TSPM 7.1. 

05 November 2013

Android SDK - AVD virtual SDcard image for download (mksdcard Windows 7 x64)

I started some experiments with Android SDK recently, and came against the problem with creating virtual sdCard image for usage by the AVD (Android Virtal Device). Unfortunately mksdcard utility kept either returning errors or just doing nothing, I had to go some lengths and try to create sdcard image on linux 32bit. I did not find any relevant hits on google how to work around my issue (create virtual sd card on Windows 7 64bit with mksdcard).

Anyway, here it's - available for you to download, it has just 120Megs to be handy.


30 October 2013

CRIMA5096821AE during WebSphere 8 update/fixpack or other Installation Manager operations

You might happen to be updating your WAS 8 installation with a fixpack, and more, you may want to do it with iclm tool, instead of IM GUI.

First of all, you need to be sure which package you are choosing from the repo, and to be sure you need to read the content of the repository with:

./imcl listAvailablePackages -repositories source_repository1,source_repository2

also, you need to be sure what is the exact signature of the packages you have installed:

 ./imcl listInstalledPackages -features -long

After you make sure you pick the ones you have and need, you do this (exemplary packages):

./imcl install com.ibm.websphere.ND.v80_8.0.7.20130725_2248 -installationDirectory /opt/IBM/WebSphere8/AppServer -repositories /opt/was8inst –acceptLicense

And, you might run into that error (exact paths vary from your system):

CRIMA5096821AE ERROR: Error updating.

CRIMA5096821AE ERROR: Error getting file for installation: 'jar file com.ibm.was.backup.nsf_8.0.4.20120410_0000' not found in /var/tmp/IBM/IMShared..

CRIMA5096821AE 'jar file com.ibm.was.backup.nsf_8.0.4.20120410_0000' not found in /var/tmp/IBM/IMShared.

In that case, you might start to worry, because most probably somebody deleted IMShared directory from your filesystem. And this is very bad thing, since Installation Manager needs this to operate on already installed packages. It stores vital data for IM operation, such as sychronization repos, libraries etc.
DO NOT REMOVE IT or you won't be able to use your IM with existing installation (to be precise: particular software packages for which this particular IMShared directory was chosen, as this may vary between packages, using the same always is a default).

There's nothing but one thing you can do about it - if you happen to work in multiple machine environment, you might have different machine with identical software set installed (like prod/dev/test systems). In that case, you might try to copy IMShared directory from other machine with identical set and in MIGHT work. Not guaranteed, but sometimes it's a last resort.

Good luck!

Couple of links on the topic:

02 October 2013

WAS 8.5 Trace missing - SLF4J: Class path contains multiple SLF4J bindings

It's been quite long since I posted here. Yet I have one new tip for you today:

The problem was that WebSphere 8.5 server did not recognize components to trace after application deployment, and after finally appearing in the traceable components tree did not log all expected output to trace log. As a result we wasn't able to troubleshoot the installation, not to mention that some of the log output was required by defined use cases of the designed solution.

After some search, asking here and there, a colleague of mine (thanks, Agnieszka!) noticed the following entries in the SystemOut.log file of the server when the troublesome application was deployed:

10/2/13 13:42:10:331 CEST] 00000042 SystemErr     R   SLF4J: Class path contains multiple SLF4J bindings.
[10/2/13 13:42:10:331 CEST] 00000042 SystemErr     R   SLF4J: Found binding in [bundleresource://243.fwk-1319487885:1/org/slf4j/impl/StaticLoggerBinder.class]
[10/2/13 13:42:10:331 CEST] 00000042 SystemErr     R   SLF4J: Found binding in [wsjar:file:/opt/ibm/WebSphere/AppServer/profiles/AppSrv01/installedApps/localhostNode01Cell/xxxxx/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[10/2/13 13:42:10:332 CEST] 00000042 SystemErr     R   SLF4J: Found binding in [wsjar:file:/opt/ibm/WebSphere/AppServer/profiles/AppSrv01/installedApps/localhostNode01Cell/xxxxx/lib/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[10/2/13 13:42:10:332 CEST] 00000042 SystemErr     R   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
[10/2/13 13:42:10:357 CEST] 00000042 SystemErr     R   SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]

and apparently this was causing the server to make wrong class pickup the logging (and hence missing output as that class missed appropriate configuration).

I came up with the idea of removing one of the conflicting classes from ear with console option "Remove file" and selecting whole jar (lib/slf4j-log4j12-1.7.2.jar). After application restart logging started to work as expected!

Good luck!

Env: WebSphere ND, Linux rhel 6 64bit, application build with Maven (some predefined class includes defined).

29 May 2013

my technote has been published!

So, the single most popular post from my blog has been accepted as IBM Support technote.

You can ssee it here: User cannot log into console due to SRVE0260E error

Not a big thing, but still, nice :)

21 May 2013

CRRTC3505E: The following fetch destination cannot be deleted - Rational Build Engine

If you happen to have problem running your build in RTC and your builds end up with:

2013-05-21 10:29:15 [Jazz build engine] Deleting fetch destination "/db2data/BuildWorkspace" before fetching ...
com.ibm.team.build.common.TeamBuildException: CRRTC3505E: The following fetch destination cannot be deleted: "/db2data/BuildWorkspace". For more details, open the help system and search for CRRTC3505E.
at com.ibm.team.build.internal.engine.JazzScmPreBuildParticipant.preBuild(JazzScmPreBuildParticipant.java:218)
at com.ibm.team.build.internal.engine.BuildLoop.invokePreBuildParticipants(BuildLoop.java:881)
at com.ibm.team.build.internal.engine.BuildLoop$2.run(BuildLoop.java:685)
at java.lang.Thread.run(Thread.java:738)

try to kill existing engine process and restart it anew:

ps -ef | grep jbe

kill -9 <pid from above>

Good luck, leave a comment!

15 May 2013

libXm.so.3: cannot open shared object file & libXm.so.3: wrong ELF class: ELFCLASS64 (RRDI 2.0.1 installation)

Just today I wanted to install Rational Reporting for Development Intelligence (RRDI) 2.0.1 on 64bit SuSE 11. I got my repository, I got Installation Manager 1.6 (it's a must!) uznipped and started the ride.

The first error IM thrown at me was "Installation failed with code status =127" and investigation in the IM log files (/var/ibm/InstallationManager/logs/native/...) shown:

/opt/IBM/RRDI/install/cognos_bi/linuxi38664h/issetup: error while loading shared libraries: libXm.so.3: cannot open shared object file: No such file or directory

So, I looked google up and found that I'm missing some libraries for my system, namely:

Open Motif 2.2.4 Libraries (openmotif22-libs-2.2.4-189.1.i586.rpm)

After getting and installing it (for 64bit) I rerun the installation....and failed again. I found then some tip to link libraries from /usr/lib64 into /usr/lib and during next installation attempt I ran into:

/opt/IBM/RRDI/install/cognos_bi/linuxi38664h/issetup: error while loading shared libraries: libXm.so.3: wrong ELF class: ELFCLASS64

So, fine, some progress but different error :) Searched again, and found that this means wrong bitness of the libraries, so I went back to RPM search and got 32bit libraries, unlinked those 64bit from /usr/lib, installed the package and...this time success! I'm no Linux expert, but that occured a bit strange to me, that I eventually solved it by installing 32bit libs on 64bit system. Anyway, should you run into the same problem, perhaps this helps you.

Good luck, comments welcome!

15 March 2013

Google, don't take Reader away from us!

As you may know, Google wants to close Google Reader service, which is widely used and highly praised.

You may want to contribute to possible prolonging of this tool of choice of many internauts here.

05 March 2013

WAS 8.5 profile creation sample response file

I looked over the net for sample response files for profile creation in WAS 8.5, but I found none. Well, I didn't expect much of a revolution in comparison to V7 or V8, but still, better be safe than sorry :) 

So I ended up converting possible listed manageprofiles flags into response file entries, in a traditional manner. The result was as follows and you can safely use it as a starting point for your profile management automation.

#profile name and capabilities
#location and names
#starting port
#standard WAS keystore/truststore password
#admin security

good luck, and leave a comment if you made some use of this content. and don't forget to like this page on facebook :)

18 February 2013

http status 500: exception on request for [/web] : null - Rational Jazz/CLM 4.0.1

If you happen to deploy CLM 4.0.1 (formerly known as Jazz Team Server/Requirements Composer/Requirements Management etc.) and when you further try to customize it (in my case customization was attempt to deploy BuildForge Connect Adapter for CLM) you may run inot some CLM dashboard pages unresponsive. Particularly, admin.war application stops responding and seems to have some sort of authentication issues. In my case, when I tried to go to Home->Collaborative Lifecycle Project Management page, I received the follwing error in the browser:

Http status 500: exception on request for [/web] : null

and SystemOut.log contained a bunch of exceptions:

00000044 zazl E org.dojotoolkit.zazl.internal.RhinoDTLHandler runDTLScript Exception on request for [/projects/new] java.lang.NullPointerException

00000044 SystemOut     O 15:39:08,820 [WebContainer : 14] ERROR com.ibm.team.lpa.config - An uncaught Exception has been thrown com.ibm.team.jfs.app.http.HttpInternalServerErrorException: Exception on request for [/projects/new]

00000044 zazl E org.dojotoolkit.zazl.internal.RhinoDTLHandler runDTLScript Exception on request for [processTemplate]

0000004f zazl E org.dojotoolkit.zazl.internal.RhinoDTLHandler runDTLScript Exception on request for [/web]


I wondered  what to do about it, and finally I saw that all troublesome pages are in /admin context. As my CLM was deployed to WAS 8 server, I simply redeployed (by Update) admin.war application from the console. It worked as a wonder!
to do this go to Websphere applications -> tick admin.war, click Update and find original file in
<CLMInstallDir>/server/webapps, proceed through Nexts, and save configuration afterwards. Then restart application and it should be working.

This trick also works with jts.war application, if you happen to have any trouble with it (with similar errors). Of course, I assume that your previous configuration was in general ok, because war Update does not fix any database content issues and so on. (for db/config problems you should use repotools but that's another story)

Hope this helps, good luck and leave a comment!

22 January 2013

TAMeSSO 8.2 configuration SQL0613N SQLSTATE=54008

I have just ran into this problem today, when configuring my test instance of TAMeSSO (lately known as ISAM - IBM Security Access Manager for Enterprise Single Sign-on).

After installing WAS7 and IMS server package, I tried to configure my IMS server. But it failed with error SQL06013N on WAS side, when attempting to create DB tables (on the screen you can see below, procedure dropped with error at ~6%)

After a quick search and a conversation with a colleague I found solution: to increase pagesize in your database to at least 8K. Actually, it is listed as requirement here:

but there is a slight chance you might omit that :) (I did...)

So, be sure that your database have been created with pagesize of 8K. If not, drop it and recreate it with a proper setting. The reason for it being necessary is a requirement for an index tablespace to be large enough to house an indexes for a colums of a specified size. Here it was 1024 VARCHAR + 128 bit for index, which gives over 1024, and hence requires 8K Pagesize. Details can be found here

Good luck and leave a comment if it solved your problem.

02 January 2013

TAM 6.1 - HPDAC0457E The protected object name is invalid. (status 0x1005b1c9)

A colleague of mine run today into a problem while trying to attach ACL to a protected object seen in the objectspace tree. Although being "seen" by both WPM console and pdadmin client via 

object show /xxx/yyy/subcat

command, trying to use 

acl attach /xxx/yyy/subcat ACL_name

resulted in getting below error:

Error: HPDAC0457E   The protected object name is invalid. (status 0x1005b1c9)

At the same time attaching ACLs to parent and child objects was just fine.

We checked initial loading scripts, and we found that there were only entries for super- and sub- items, like:

/xxx/yyy/ and then for /xxx/yyy/subcat/subsubcat/ but not for the middle one

So what was happening was that /xxx/yyy/subcat wasn't really created before, but only was visible as a tree level, because TAM had to show something in that place. Then the fix was easy, we just created the missing object with

object create /xxx/yyy/subcat "" 10 ispolicyattachabe yes

and then we could successfuly attach ACL to this object.

Hope this helps, enjoy and leave a comment!

20 September 2012

commonDBUtility.ant Source file does not exist! during WebSphere ESB DMgr creation

When creating ESB DMgr profile you may encounter the following error in the log:

<date>Sep 20, 2012 10:47:53 AM</date>
    <message>/opt/websph/ESB/util/dbUtils/profileHelpers/commonDBUtility.ant:985: Source file does not exist!
at org.apache.tools.ant.taskdefs.SQLExec.execute(SQLExec.java:324)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
at org.apache.tools.ant.Task.perform(Task.java:364)
at org.apache.tools.ant.Target.execute(Target.java:341)


and resulting:

    <date>Sep 20, 2012 10:47:53 AM</date>
    <message>Returning with return code: INSTCONFFAILED</message>

This happened to my colleague, who was configuring ESB DMgr against Derby network server. We spent a while figuring out what's wrong. Closer look into logs couple of lines before first error showed

<message>Target stopped for: init.db - SUCCESS</message>

so the Derby DB was initialized fine. Right afterwards we found:

 <message>Execute SQL script with parameters: JDBCDriver='org.apache.derby.jdbc.ClientDriver' DB_URL='jdbc:derby://localhost:1527//opt/websph/ESB/profiles/Dmgr2/databases/WPRCSDB4' dbUserId='admin' sqlScriptPath='/opt/websph/ESB/profiles/Dmgr2/dbscripts/CommonDB/Derby/WPRCSDB4/createTable_CommonDB.sql' JDBC_DRIVER_FILE_STATIC='/opt/websph/ESB/derby/lib/derbyclient.jar'</message>

following this trail I asked my buddy to check dbScriptOutputDir content. Bingo! it contained the following structure:


with clearly abundant subtree portion. The reason for this was that in the manageprofiles dbScriptOutputDir  parameter contained unnecessary tail that is created automatically. Hence, ant script failed not finding sql files where expected. Changing

./manageprofiles.sh -create  (...) -dbOutputScriptDir '/opt/websph/ESB/profiles/Dmgr2/dbscripts/CommonDB/Derby/WPRCSDB2'


./manageprofiles.sh -create (...) -dbOutputScriptDir '/opt/websph/ESB/profiles/Dmgr2/dbscripts'

did the trick. Please note that InfoCenter says you need to do it wrongly! Look here (Table 2, -dbOutputScriptDir description. Thank you Rafał!)

Good luck!

30 July 2012

HPDAZ0602E Corrupted file: Insufficient information to contact a Policy Server (SECJ0391E)

Now, a few words about possible problems when configuring external JACC authorization provider for WebSphere App Server, namely - Tivoli Access Manager. The key issue here always lies in obtaining Policy Server's certificate for JRE to authenticate later on. This can be done in various ways (sslsrvcfg, pdconfig etc.) and obviously, the results may vary depending on the way used - at least it seems so.
Anyway, recently I was asked what may be the reason for the following error appearing in the logs:

[7/25/12 10:07:44:950 CEST] 00000013 AMWASConfigMe I   com.tivoli.pd.as.jacc.cfg.TAMConfigController execute() AWXJC0048E   An error occurred during the configuration. The details are: com.tivoli.pd.as.jacc.cfg.ConfigActionFailedException:
[java.lang.IllegalStateException: HPDAZ0602E   Corrupted file: Insufficient information to contact a Policy Server.

Wrappered Exception:
java.lang.IllegalStateException: HPDAZ0602E   Corrupted file: Insufficient information to contact a Policy Server..

This one happened during JACC configuration attempt from the WAS admin console, when console reported that action cannot be successfully completed. "Insufficient information" could only mean that JRE is trying to register with Policy Server, but fails to trust/present valid certificate. Unfortunately, we were stubborn, and hoped that action will complete during server restart. But after saving configuration and restarting server, it fails to start! In the logs we could see:

[7/25/12 10:19:49:404 CEST] 00000000 distSecurityC E   SECJ0391E: Error when setting the Policy object to the provider's policy implementation com.tivoli.pd.as.jacc.TAMPolicy. The exception is com.tivoli.pd.as.jacc.util.JACCException: AWXJR0006E   The file, /opt/ibm/WebSphere/AppServer/profiles/PSS1UPRB01DMgr/etc/tam/amwas.PSS1UPRB01DMgr_dmgr.amjacc.properties, was not found.
    at com.tivoli.pd.as.jacc.TAMPolicy.init(TAMPolicy.java:680)
    at com.tivoli.pd.as.jacc.TAMPolicy.<init>(TAMPolicy.java:97)
    at java.lang.J9VMInternals.newInstanceImpl(Native Method)
    at java.lang.Class.newInstance(Class.java:1345)

which meant that WAS has got JACC enabled, but in fact had failed to provide proper configuration file for amwas/jacc. We gave it some thinking and first, we had to recover from this fatal condition: by editing security.xml we disabled security and disabled using JACC in section:

<security:Security xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:orb.securityprotocol="http://www.ibm.com/websphere/appserver/schemas/5.0/orb.securityprotocol.xmi" xmlns:security="http://www.ibm.com/websphere/appserver/schemas/5.0/security.xmi" xmi:id="Security_1" useLocalSecurityServer="true" useDomainQualifiedUserNames="false" enabled="true" cacheTimeout="600" issuePermissionWarning="false" activeProtocol="BOTH" enforceJava2Security="false" enforceFineGrainedJCASecurity="false" appEnabled="true" dynamicallyUpdateSSLConfig="true" allowBasicAuth="true" activeAuthMechanism="LTPA_1" activeUserRegistry="WIMUserRegistry_1" defaultSSLSettings="SSLConfig_1"> <--- change bold to FALSE

<authConfig xmi:id="AuthorizationConfig_1" useJACCProvider="true">  <-- change bold to FALSE   

After that server went up again and we gave a bit thinking to the problem. It turned out to be awfully simple and...well, strange ? We had to add hostname of the WAS machine to policy server's /etc/hosts. Possibly this can be also solved by adding WAS hostname to DNS, anyway, pdmgrd must be able to resolve WAS's IP based on presented hostname. Then, we were able to complete JACC configuration successfully, turn security back on for the whole cell, save and restart and all worked like a wonder!

Why do I say it was strange? Because my suspicion is that different PD.jar packages (or more precisely: pd.* classes) responsible for connecting to Policy Server do this in a different way. I'm too weak a programmer to dig this up and resolve it 100%, but I just know that you may get different results when a) configuring JRE from pdconfig b) using sslsrvcfg and c) configuring JACC from WAS (WebSphere's embedded TAM) - eventually, you just need to see what works best for you.

Good luck, leave a comment if it helped!