Generation of deployment environment fails - Reason: CWLDB9014E and Reason: CWWBZ0058E

First of all, this is the task I failed numerous times on, until finally I managed to get past, getting the desired result at the same time. This is because I found some sort of workaround before, but the results were more or less doubtful. Well, this workaround is a bit messy as well, but at least you can benefit from creation of Deployment environment with the wizard, instead of putting DE parts together step by step (and probably missing some parts, which later cause you some, or a lot of, pain).

So, we're talking WPS 6.2.0.0 (a bit obsolete) and 7.0.0.4, deployed vs. DB2 database hosted on z/OS (which probably is the most troublesome circumstance, as you cannot create any objects automatically like in LUW edition). What we try to do is to generate Deployment environment (in any topology - single, distributed or "golden"), while defining databases on z/OS DB2. What we get is failure in this particular part:, where instead of:



you keep getting:

CWLDB9014E: The configuration of component WBI_BPCEventCollector failed.

To make it short, before, I could not find the reason. But recently, I think I found it by looking into FFDC error report, and saw there errors related to failing connection to DERBY database. I wondered why it tries to use Derby when I explicitly requested all databases to be hosted on z/OS, but I will never know (perhaps some buggy ant script, or other config xml issue). Nevertheless, what we need to do is to:

1. create "dummy" Derby database, just for sake of generation of DE:
   
   • go to /opt/ibm/WebSphere/AppServer/derby/bin/networkServer
   • run ./ij.sh
   • issue command: connect 'jdbc:derby:/opt/ibm/WebSphere/AppServer/profiles/PPD2UZLB01DMgr/databases/event/PD2UZL.Support/DerbyEventDB/BPD2CEDB;create=true';
   your database has been created and is ready to use, but first:
2.  add custom user to connect to Derby server
   
   • edit /opt/ibm/WebSphere/AppServer/derby/derby.properties by adding line
     

     derby.user."yourUser"=yourpwd

3. start Derby network server (/opt/ibm/WebSphere/AppServer/derby/bin/networkServer/startNetworkServer.sh)
   Note: default TCP/IP port for derby is 1527
4. now, get back to DE generation wizard and run it again, but now type in Derby database for your EventDB - server is localhost:1527, user is "yourUser" and password "yourpwd"
5. finish generation and you should get to successful end of operation
6. assuming you have already created all necessary database in your target DB (either DB2 on z/OS or other supplier), you now need to redefine jdbc datasource for EventDB to point to the datastore of your choice.

You may of course run into numerous problems before getting your DE working 100% fine but above instruction should at least get you through DE generation wizard.

Good luck, leave a comment or vote in the poll please!

WPM config - HPDCO1364E The specified domain does not exist. (0x1354a554)

While trying to configure WPM (TAM Web Portal Management), you may encounter this particular error:

HPDCO1364E   The specified domain does not exist. (0x1354a554)

You are most probably using non-default (not "Default") domain. Apparently, WPM has a problem with specifying different domain name than default, but this occurs in version 6.1.0.0. What you need to do is to upgrade to fp5 (or lower fp, but I tested it with fp5)

1. get 6.1.0-TIV-TAM-FP0005-LIN.tar.Z file (for Linux x86)
2.  gunzip and tar -xf it
3. install with: rpm -Uhv PDWPM-PD-6.1.0-5.i386.rpm
4. try pdconfig

after fixpacking it should suggest you (get from configured Java Runtime) correct domain name.

Good luck, leave a comment.

a new EEv10 is here!

Aside from this blog activity, I'm also active Experts Exchange contributor. They launched new version of their site recently, and are very proud of it. So proud, that they give iPads away :) Why should I not get one?

javax.management.JMRuntimeException: ADMN0022E

If you stumble upon the following error:

See nested exception; nested exception is: javax.management.JMRuntimeException: ADMN0022E: Access is denied for the getChain operation on TransportChannelService MBean because of insufficient or empty credentials.

that means you need to revise your application permissions settings. That is, your application most probably has a RunAs role defined, which determines identity that is being mapped to each appliacation call - WebSphere then "thinks" that each call to this EJB is being made by some specified user (RunAs).

To check this, go to:

Applications > Enterprise Applications > application_name 

and find  Map RunAs roles to users under Additional Properties section. There you will find RunAs roles defined for this particular EJB. What you need to do is to type in valid username and password (having necessary permission level if you are using any authorization system - either role-based or external, like calling PDContext for TAM authorization), tick roles you want to assign them to and use Aplly, then save, synchronize and restart server.

Detailed information can be found on IBM InfoCenter.

Good luck, leave a comment or vote in the poll, please :) 

humble anouncement

Dear Readers, just want to humbly (ok, not so :) anounce that this blog has just reached 1000 pageviews, mostly happened withinin the last two months.

Thank you all for visiting, I do hope you will return to my posting spot. Please leave comments and feedback in the poll.

The new Google's privacy policy - take notice!

Google is going to make it's new privacy policy effective soon. Among other things, one is particularly alarming - information given to google in all their services (60+) will be merged to create your unified online history. That includes your goolge search history, which is a clear footprint of your age, gender, preferences, medical concerns etc.

You may want to read more and before all - disable you Online history recording, as described in this EE article.

IBM HTTP Server not starting - http_plugin.log excess size

I just happened to come across the following error:

I'm using IBM HTTP Server 7 with WAS Plugin configured. the following behavior was observed - after attempting to start IHS with 

<IHS_INSTALL_ROOT>/bin/apachectl start

control returns to shell, as it should. But when I checked if IHS is up and running with

nestat -an | grep <portnumber>

it didn't show desired port listening, however when looking for httpd processes with:

ps -ef | grep httpd

showed all necessary httpd jobs (namely, 4 of them). That is a problem symptom.

This was the second time I came across that, so I knew what to do. The clue is http_plugin.log file size: when it grows over 2147483647 bytes (on Linux), OS is no longer able to write to it, and despite seemingly successful startup of the server. So, you just need to remove (or move to other place if you need it) this file, and try to start IHS again, it will recreate it and work fine. The log file is located in <PLUGIN_ROOT>/logs/<Servername>/http_plugin.log

If you happen to see this malfunction, review your WAS plugin LogLevel settings. Remeber that on Trace or Detail level it generates fairly lot of entries and log grows very quickly. If your system is working fine and you don't have any sophisticated file log monitoring implemented, just change it to Info or Warn - it will be sufficient and your log file won't clog up.

Good luck,  thanks for comments and feedback!

Installation Manager java.lang.UnsatisfiedLinkError: Could not load SWT library

IBM Installation Manager is more and more widely used to deploy number of IBM products, so it may be useful to know a little about possible problems when running this tool.
My today's accomplishment is that I overcame following issue with installation on a lightweight SuSE 11 distribution (eg. stripped from almost every non-necessary package).
First, when trying to install IM itself I was knocked by:


JVMDUMP010I Snap dump written to /tmp/was/IM/Snap.20120213.121315.11358.0003.trc
libgcc_s.so.1 must be installed for pthread_cancel to work

This one was tackled by adding libgcc43-32bit package to the system. After successful silent installation, I tried to actually run IM to install WAS 7. Shell showed nothing after issuing:

./install or ./IBMIM

It was all I got:

prep2def:/opt/ibm/InstallationManager/eclipse # ./launcher
prep2def:/opt/ibm/InstallationManager/eclipse # cd /tmp/ 
Looking into configuration/datestamp.log files I found this error:

!ENTRY org.eclipse.osgi 4 0 2012-02-13 13:32:15.756
!MESSAGE Application error
!STACK 1
java.lang.UnsatisfiedLinkError: Could not load SWT library. Reasons:
        /opt/ibm/InstallationManager/eclipse/configuration/org.eclipse.osgi/bundles/454/1/.cp/libswt-pi-gtk-3659.so (libgthread-2.0.so.0: cannot open shared object file: No such file or directory)
        swt-pi-gtk (Not found in java.library.path)
        /tmp/swtlib-32/libswt-pi-gtk-3659.so (libgthread-2.0.so.0: cannot open shared object file: No such file or directory)
        /tmp/swtlib-32/libswt-pi-gtk.so (/tmp/swtlib-32/liblibswt-pi-gtk.so.so: cannot open shared object file: No such file or directory)

And finally, installing:

DejaVu Truetype Fonts
 
and

libgthread-2_0-0-32bit

helped, and I was able to successfully run Installation Manager. Hope it helps. Good luck.

SECJ0053E and AWXJR0044E: PolicyConfiguration exists = false

If you are working on WAS with JACC configured against Tivoli Access Manager (or other external authorization provider, but I assume TAM is the most popular) in your application development cycle you may stumble upon the following error sequence:
[2/7/12 11:24:45:304 CET] 000000c8 SecurityColla A   SECJ0053E: Authorization failed for defaultRealm/username while invoking (Bean)XyEAR-2.2#XyzBean.jar#soapAction:5 JACC Authorization failed for bean: xyzvBean
[2/7/12 11:24:45:660 CET] 0000003d AMWASJACCMess I   com.tivoli.pd.as.jacc.TAMPolicy implies(ProtectionDomain, Permission): permission = perm.toString()
0x864297004
AWXJR0044E   The access decision for Permission, (javax.security.jacc.EJBMethodPermission xyz), was denied because either the PolicyConfiguration or RoleConfiguration objects did not get created successfully at application installation time.  RoleConfiguration exists = true, PolicyConfiguration exists = false.

It is more probable to occur right after application update (new version deployed, for instance).

Here are the things you may do to steer out of this trouble:

 

1. Ensure all necessary protected object exist in TAM policy database (use pdadmin or WPM - TAM console deployed on WAS)
2. Update role definition in your authorization provider with:
   

   Global security > External authorization providers

   choosing "Update with application names listed" and typing in your application name (as appears in Enterprise Applications tab), then Apply

3. If point 2. fails, follow this sequence: stop application, remove application from server, stop application server, clean temp directories for this server, start server and redeploy application. It should bind with TAM properly this time. After that, restart the server again.

 Good luck.

Java exception: org.omg.CORBA.BAD_INV_ORDER

It may happen that all of a sudden you will see the following exception in your SystemOut.log of your managed server (that is the one managed from deployment manager console via nodeagent process). That may occur during either server run or (more often) during server startup attempted from shell level.

[1/25/12 11:25:36:870 CET] 0000002c WSSecurityCon A   Error creating client_auth_token in initSecContext, reason: Major Code[15] Minor Code[0] Message[ create_gss_initial_context_token: Java exception: org.omg.CORBA.BAD_INV_ORDER: ORB has been shut down  vmcid: OMG  minor code: 4  completed: No]
[1/25/12 11:25:36:884 CET] 0000002c IPCConnectorC E   ADMC0076E: Failed to get security token due to Major Code[15] Minor Code[0] Message[ create_gss_initial_context_token: Java exception: org.omg.CORBA.BAD_INV_ORDER: ORB has been shut down  vmcid: OMG  minor code: 4  completed: No]
        at com.ibm.ISecurityUtilityImpl.CSIUtil.create_gss_initial_context_token(CSIUtil.java:979)
        at com.ibm.ISecurityLocalObjectTokenBaseImpl.WSSecurityContextLTPAImpl.initSecContext(WSSecurityContextLTPAImpl.java:179)
        at com.ibm.ISecurityLocalObjectTokenBaseImpl.WSSecurityContextLTPAImpl.initSecContext(WSSecurityContextLTPAImpl.java:133)
        at com.ibm.ws.management.connector.ipc.IPCConnectorClient.fillInCredential(IPCConnectorClient.java:307)
        at com.ibm.ws.management.connector.ipc.IPCConnectorClient.queryNames(IPCConnectorClient.java:792)
        at com.ibm.ws.management.AdminClientImpl.queryNames(AdminClientImpl.java:108)
        at com.ibm.ws.management.discovery.transport.MBeanDiscoveryHelper.discovery(MBeanDiscoveryHelper.java:507)
        at com.ibm.ws.management.discovery.transport.MulticastServer.rediscover(MulticastServer.java:406)
        at com.ibm.ws.management.discovery.transport.MulticastServer.run(MulticastServer.java:279)
        at java.lang.Thread.run(Thread.java:736)

It usually means that your server has lost communication with it's nodeagent, and the reasons for that may be numerous, but the most common are:

• nodeagent process crashed or was killed any other way - that is the easiest: just bring up back the nodeagent
• something went wrong with network interfaces (that's more applicable to startup problem) - in that case you need to try to bring up/down lo interface (read here) or check if in the meantime your hostname or ip address hadn't been changed (little chance, but you never know)
• if above fail, check if nodeagent NDS (node discovery service) port, among with other ports (soap etc.) are locally available with telnet. if not, you need to diagnose further system's ip sockets setup.

Good luck!

WebSphere JSSL0080E: javax.net.ssl.SSLHandshakeException - The client and server could not negotiate the desired level of security.

If you see the following exception while or after starting your server (nodeagent or app server): 

JSSL0080E: javax.net.ssl.SSLHandshakeException - The client and server could not negotiate the desired level of security. 

then big chance is that there's something wrong with internal certificates between WAS instances, either:
1. Hostname has been changed and hostname verification is turned on in SSL setup. you may check this here:
http://www.ibm.com/developerworks/websphere/techjournal/0612_birk/0612_birk.html
in paragraph:

The default trust manager, IbmX509, performs fundamental certificate validation, including the certificate signature validation (ensuring it has not been modified) and certificate expiration validation (ensuring it has not expired). This trust manager does not perform hostname verification by default, although you can set the com.ibm.ssl.performURLHostNameVerification=true property in the security custom properties to enable this function for URL connections only. If this is done, the trust manager will ensure that for URL connections, the hostname specified on the connection matches the SubjectDN in the certificate returned by the server (just as Web browsers do).
(recommend reading the whole article btw.)

2. Something went horribly wrong in terms of cell synchronization, so I'd suggest stopping servers and nodeagent on remote machine and running ./syncNode.sh from profile's bin directory. not sure it will work as probably wsadmin script client won't be able to connect to SOAP port due to root cause. hence, we're back at point 1. and I guess only manual copying of trust/keystores from dmgr machine to remote machine might help.

hope this helps, or at least points to the right direction.

WebSphere SOAP message trace (XML, JAX-WS)

Okay, there are numerous articles on how to trace XML SOAP messages sent to websphere server, but it seems all of them address some different scenario I am in. Eg. they involve custom development or code-changing in order to add handlers and so on.

What I needed was to simply dump content of SOAP messages incoming to my http inbound channel. However, anything less than wssecurity.*=all given no SOAP content, so I started to drill down to the very one I needed. Finally I found some least fraction of WAS framework to check if you want to get incoming XML content, out of WSSecurityHandler class. As XML parsing involves security processing, you simply need to use the following trace:

com.ibm.ws.wssecurity.handler.*=finest 

in addition to the one you already have set.

Good Luck, I wish you happy tracing!

300 and counting.

facebook and g+ shares pumped up visit count a bit. tempting as it is, I won't however use it to make my blog notable based on non-technical posts!

WebSphere: disable SSL between IHS and WAS (IHS SSL offload)

Ever wondered how to disable SSL between IHS and WebSphere App Server? In some environments, your IHS and WAS remain in secure zone, and you don't want to spend any CPU time for doing SSL between web server and backend server.
In case of IBM HTTP Server + WAS combination there's a simple way to do it:

• assuming you have Plugin already configured for IHS, you need to edit it's config (plugin-cfg.xml) with some editor eg. vi

• find section for your backend server cluster you want to SSL off-load:

<ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="clustername" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
      <Server ConnectTimeout="5" ExtendedHandshake="false" MaxConnections="-1" Name="PTS2CSIP01Node_STS2CSIP01S01" ServerIOTimeout="60" WaitForContinue="false">
         <Transport Hostname="backend_hostname" Port="4300" Protocol="http"/>
         <Transport Hostname="backend_hostname" Port="4343" Protocol="https">
            <Property Name="keyring" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.kdb"/>
            <Property Name="stashfile" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.sth"/>
         </Transport>
      </Server>
   </ServerCluster>

 and comment out second Transport definition to:

      <!--Transport Hostname="backend_hostname" Port="4343" Protocol="https">
            <Property Name="keyring" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.kdb"/>
            <Property Name="stashfile" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.sth"/>
      </Transport-->
 

• restart your IHS server and check in error_log if plugin has been loaded properly 

To check, you may turn on plugin tracing:

<Log LogLevel="Trace" Name="/opt/ibm/HTTPServer/Plugins/logs/plgcfgname/http_plugin.log"/>

Restart IHS once again and tail  http_plugin.log while sending request to your https port of IHS. You shall get trace like that:
TRACE: ws_common: websphereFindTransport: Finding the transport
DETAIL: ws_common: websphereFindTransport: Setting the transport(case 3): backend_hostname on port 4300 <--- PLEASE NOTE THIS IS HTTP PORT OF WAS!
TRACE: ws_common: websphereExecute: Executing the transaction with the app server reqInfo is OKuseExistingStream=0, client->stream=(nil)
DEBUG: ws_common: websphereGetStream: Getting the stream to the app server
TRACE: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
TRACE: ws_common: websphereGetStream: Have a connect timeout of 5; Setting socket to not block for the connect
TRACE: ws_common: websphereGetStream: Reseting socket to block
TRACE: ws_common: websphereGetStream: Setting socket to non-block for ServerIOTimeout over HTTP
DEBUG: ws_common: websphereGetStream: socket 13 connected to backend_hostname:4300
DEBUG: lib_stream: openStream: Opening the stream
DETAIL: ws_common: websphereGetStream: Created a new stream; queue was empty, socket = 13

Any questions? Good Luck!!! Feel free to comment&leave feedback.

200 views! done.

big thanks to you, dear visitors! my blog has just reached 200 views. not so much, I know, but it gives at least some satisfaction!
feel invited to drop in, leave comments and feedback. see ya!