Java exception: org.omg.CORBA.BAD_INV_ORDER

It may happen that all of a sudden you will see the following exception in your SystemOut.log of your managed server (that is the one managed from deployment manager console via nodeagent process). That may occur during either server run or (more often) during server startup attempted from shell level.

[1/25/12 11:25:36:870 CET] 0000002c WSSecurityCon A   Error creating client_auth_token in initSecContext, reason: Major Code[15] Minor Code[0] Message[ create_gss_initial_context_token: Java exception: org.omg.CORBA.BAD_INV_ORDER: ORB has been shut down  vmcid: OMG  minor code: 4  completed: No]
[1/25/12 11:25:36:884 CET] 0000002c IPCConnectorC E   ADMC0076E: Failed to get security token due to Major Code[15] Minor Code[0] Message[ create_gss_initial_context_token: Java exception: org.omg.CORBA.BAD_INV_ORDER: ORB has been shut down  vmcid: OMG  minor code: 4  completed: No]
        at com.ibm.ISecurityUtilityImpl.CSIUtil.create_gss_initial_context_token(CSIUtil.java:979)
        at com.ibm.ISecurityLocalObjectTokenBaseImpl.WSSecurityContextLTPAImpl.initSecContext(WSSecurityContextLTPAImpl.java:179)
        at com.ibm.ISecurityLocalObjectTokenBaseImpl.WSSecurityContextLTPAImpl.initSecContext(WSSecurityContextLTPAImpl.java:133)
        at com.ibm.ws.management.connector.ipc.IPCConnectorClient.fillInCredential(IPCConnectorClient.java:307)
        at com.ibm.ws.management.connector.ipc.IPCConnectorClient.queryNames(IPCConnectorClient.java:792)
        at com.ibm.ws.management.AdminClientImpl.queryNames(AdminClientImpl.java:108)
        at com.ibm.ws.management.discovery.transport.MBeanDiscoveryHelper.discovery(MBeanDiscoveryHelper.java:507)
        at com.ibm.ws.management.discovery.transport.MulticastServer.rediscover(MulticastServer.java:406)
        at com.ibm.ws.management.discovery.transport.MulticastServer.run(MulticastServer.java:279)
        at java.lang.Thread.run(Thread.java:736)

It usually means that your server has lost communication with it's nodeagent, and the reasons for that may be numerous, but the most common are:

• nodeagent process crashed or was killed any other way - that is the easiest: just bring up back the nodeagent
• something went wrong with network interfaces (that's more applicable to startup problem) - in that case you need to try to bring up/down lo interface (read here) or check if in the meantime your hostname or ip address hadn't been changed (little chance, but you never know)
• if above fail, check if nodeagent NDS (node discovery service) port, among with other ports (soap etc.) are locally available with telnet. if not, you need to diagnose further system's ip sockets setup.

Good luck!

WebSphere JSSL0080E: javax.net.ssl.SSLHandshakeException - The client and server could not negotiate the desired level of security.

If you see the following exception while or after starting your server (nodeagent or app server): 

JSSL0080E: javax.net.ssl.SSLHandshakeException - The client and server could not negotiate the desired level of security. 

then big chance is that there's something wrong with internal certificates between WAS instances, either:
1. Hostname has been changed and hostname verification is turned on in SSL setup. you may check this here:
http://www.ibm.com/developerworks/websphere/techjournal/0612_birk/0612_birk.html
in paragraph:

The default trust manager, IbmX509, performs fundamental certificate validation, including the certificate signature validation (ensuring it has not been modified) and certificate expiration validation (ensuring it has not expired). This trust manager does not perform hostname verification by default, although you can set the com.ibm.ssl.performURLHostNameVerification=true property in the security custom properties to enable this function for URL connections only. If this is done, the trust manager will ensure that for URL connections, the hostname specified on the connection matches the SubjectDN in the certificate returned by the server (just as Web browsers do).
(recommend reading the whole article btw.)

2. Something went horribly wrong in terms of cell synchronization, so I'd suggest stopping servers and nodeagent on remote machine and running ./syncNode.sh from profile's bin directory. not sure it will work as probably wsadmin script client won't be able to connect to SOAP port due to root cause. hence, we're back at point 1. and I guess only manual copying of trust/keystores from dmgr machine to remote machine might help.

hope this helps, or at least points to the right direction.

WebSphere SOAP message trace (XML, JAX-WS)

Okay, there are numerous articles on how to trace XML SOAP messages sent to websphere server, but it seems all of them address some different scenario I am in. Eg. they involve custom development or code-changing in order to add handlers and so on.

What I needed was to simply dump content of SOAP messages incoming to my http inbound channel. However, anything less than wssecurity.*=all given no SOAP content, so I started to drill down to the very one I needed. Finally I found some least fraction of WAS framework to check if you want to get incoming XML content, out of WSSecurityHandler class. As XML parsing involves security processing, you simply need to use the following trace:

com.ibm.ws.wssecurity.handler.*=finest 

in addition to the one you already have set.

Good Luck, I wish you happy tracing!

300 and counting.

facebook and g+ shares pumped up visit count a bit. tempting as it is, I won't however use it to make my blog notable based on non-technical posts!

WebSphere: disable SSL between IHS and WAS (IHS SSL offload)

Ever wondered how to disable SSL between IHS and WebSphere App Server? In some environments, your IHS and WAS remain in secure zone, and you don't want to spend any CPU time for doing SSL between web server and backend server.
In case of IBM HTTP Server + WAS combination there's a simple way to do it:

• assuming you have Plugin already configured for IHS, you need to edit it's config (plugin-cfg.xml) with some editor eg. vi

• find section for your backend server cluster you want to SSL off-load:

<ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="clustername" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
      <Server ConnectTimeout="5" ExtendedHandshake="false" MaxConnections="-1" Name="PTS2CSIP01Node_STS2CSIP01S01" ServerIOTimeout="60" WaitForContinue="false">
         <Transport Hostname="backend_hostname" Port="4300" Protocol="http"/>
         <Transport Hostname="backend_hostname" Port="4343" Protocol="https">
            <Property Name="keyring" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.kdb"/>
            <Property Name="stashfile" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.sth"/>
         </Transport>
      </Server>
   </ServerCluster>

 and comment out second Transport definition to:

      <!--Transport Hostname="backend_hostname" Port="4343" Protocol="https">
            <Property Name="keyring" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.kdb"/>
            <Property Name="stashfile" Value="/opt/ibm/HTTPServer/Plugins/config/plgcfgname/plugin-key.sth"/>
      </Transport-->
 

• restart your IHS server and check in error_log if plugin has been loaded properly 

To check, you may turn on plugin tracing:

<Log LogLevel="Trace" Name="/opt/ibm/HTTPServer/Plugins/logs/plgcfgname/http_plugin.log"/>

Restart IHS once again and tail  http_plugin.log while sending request to your https port of IHS. You shall get trace like that:
TRACE: ws_common: websphereFindTransport: Finding the transport
DETAIL: ws_common: websphereFindTransport: Setting the transport(case 3): backend_hostname on port 4300 <--- PLEASE NOTE THIS IS HTTP PORT OF WAS!
TRACE: ws_common: websphereExecute: Executing the transaction with the app server reqInfo is OKuseExistingStream=0, client->stream=(nil)
DEBUG: ws_common: websphereGetStream: Getting the stream to the app server
TRACE: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
TRACE: ws_common: websphereGetStream: Have a connect timeout of 5; Setting socket to not block for the connect
TRACE: ws_common: websphereGetStream: Reseting socket to block
TRACE: ws_common: websphereGetStream: Setting socket to non-block for ServerIOTimeout over HTTP
DEBUG: ws_common: websphereGetStream: socket 13 connected to backend_hostname:4300
DEBUG: lib_stream: openStream: Opening the stream
DETAIL: ws_common: websphereGetStream: Created a new stream; queue was empty, socket = 13

Any questions? Good Luck!!! Feel free to comment&leave feedback.

200 views! done.

big thanks to you, dear visitors! my blog has just reached 200 views. not so much, I know, but it gives at least some satisfaction!
feel invited to drop in, leave comments and feedback. see ya!

VMWare on mobile phone....oh gosh!

It's a bit off topic, but worth noting: LG is going to release virtualized mobile phone. What for ? They claim many people are tired of carrying around two phones - privateone  and the one from your employer so here's a thing: on new LG you'll be able to run two Android systems to provide for separate virtual devices for personal and professional use. VMWare folks did a good job. And well, it might as well work for me, although I'm rather cold for gadgets :)
You may rea more on ExpertsExchange. Let's see what more comes from CES 2012.

WebSphere WPS/ESB CWSIS1538I,CWSIS1545I,CWSIS1535E and CWSIS1546I

When working with WebSphere Process Server (or Enterprise Service Bus, which is a limited set of WPS functionality) you may run into a problem with using database for messaging engines. That appears as a series of errors in SystemOut.log file during particular server startup, while the errors are: CWSIS1538I,CWSIS1545I,CWSIS1535E and CWSIS1546I. That usually means that you are trying to re-use SIB tables in the database, that were previously used by another WebSphere instance. Good example of this problem is given here in infoCenter but to say short that is what you need to do:

• find database used by WPS/ESB
• check if <YOURSCHEMA>.SIBOWNER table exists and what are it's contents (simply: SELECT * FROM <YOURSCHEMA>.SIBOWNER
• you'll see that there's some ME UUID (messaging engine unique id) exists that is preventing WPS/ESB to start using this table exclusively (WPS needs to "know" that SIB tables are his and only his, so you need to ensure he can think so)
• clear SIBOWNER table with DELETE * FROM <YOURSCHEMA>.SIBOWNER
• restart the server and check if errors are gone

My recommendation (as well as support's) is to clear all tables used by SIB engine (which ones? check out in DB generation ddls, produced by sibDDLGenerator.sh scripts) to ensure no further data conflict will occur in future.

Good luck! Leave a comment!

Happy New Year!

I wish a Happy New Year to all readers and visitors! Let the newly arrived 2012 be prolific in both personal and professional success!

Tivoli Access Manager 6.1 account/password expiry notification

Ever wondered how to use password/account expiry notification in your TAM 6.1 deployment? As it is not supported out of the box, you need to either:

- enable appropriate stanzas in your pd.conf file:

[acnt-mgt]
enable-passwd-warn = yes
passwd-warn = passwd_warn.html
passwd-warn-failure = passwd_warn.html
account-inactivated = acct_locked.html

but this requires LDAP supporting password lifecycle options

- or use passwordexpiry.war application supplied by helpful guys from Australia.
I did this and even enhanced app's functionality, but I need some confirmation from the authors that it has been 
posted in the article.

if your not close with the LDAP underneath your TAM, go for the 2nd.

Good luck!

pdadmin login fail after migration

There are time that you want to do some unusual stuff with Tivoli Access Manager such as migrate whole TAM instance to a different machine/environment. I did that when trying to replicate KVM environment to a different location, with changing target KVMs IPs to exactly match original one and all the rest as well, including LDAP.

That last component replacement seemed to be a bit too harsh for TAM, and despite copying secAuthority suffix data 1:1, after starting KVM machine and trying to log into pdadmin, I failed to authenticate.

After some investigation it turned out that after copying security suffix data into LDAP, you then need to apply TAM acls to secAuthority objects with ivrgy tool, such as:

/opt/PolicyDirector/sbin/ivrgy_tool -h ldap_host -389 -D "<LDAP_admin_id>" -w <LDAP_admin_pwd> add-acls domain_name

where domain_name is usually Default, but you may also have something more fancy :)

good luck, feel free to comment.

WebSphere console error SRVE0260E

If when trying to log to a newly configured Deployment Manager in Websphere you see the following error with a stack dump:

Error Page Exception
SRVE0260E: The server cannot use the error page specified for your application to handle the Original Exception printed below.

Original Exception:
Error Message: java.lang.NullPointerException
Error Code: 500
Target Servlet:
Error Stack:
java.lang.NullPointerException
     at com.ibm.isclite.container.provider.PortletContainerContextProvider.addContainerService(PortletContainerContextProvider.java:101)


then you need to raise your system limit for number of open files. on linux you do it with ulimit command:

ulimit -f 8192

where number depends on the DMgr console configuration - for standard WebSphere 1024 is sufficient, whereas with WebSphere Process Server 7 you need to raise it more, to, say, 8192.

Using ulimit -f however sets ulimit for your shell session only, so next time when you restart dmgr in your shell you will hit the same problem again. You can either raise fopen parameter for your profile (in .profile) or for the whole system by inserting:

#<domain>      <type>  <item>         <value>

*               soft    nofile          8192

to /etc/security/limits.conf. You must reboot your Linux in order for this change to take effect.

But of course beware on the impact on your system and consider this change well.

Good luck! Please leave a comment if my hint helped you and feel free to contact me on G+ !

websphere addNode problem: ADMU0036E: The Deployment Manager cannot lookup by name host at address

That's exactly what hit me: I was trying to federate node to deployment manager when I got this error:

ADMU0036E: The Deployment Manager cannot lookup by name host .. at address 127.0.0.2

I checked local config: no 127.0.0.2 address was there.
I found this thread on developerWorks, and it helped me - I just needed to turn of loopback interface with:

/etc/init.d/network stop lo

After that addNode went smoothly.

Software version: WAS 7.0.0.19, SLES 11.1

strongly recommended for anyone at tech lost: The New Experts Exchange!

Ever been stuck in tech problem? Google is no use in your case? Get ready for the new Experts Exchange! Thousands of experts just waiting for your call!

Linux port forwarding (port 389, LDAP Proxy)

What I was trying to do this morning was to get my LDAP proxy going. By LDAP proxy I mean simple port 389 forwarding from one machine to the 389 port on actual LDAP server.

After intial attmpt to set up iptables to do what I wanted, I started to look through internet, as this posed to be non-trivial task, at least for me.

Eventually, after putting together instructions found on numerous web pages and forums I put together my Golden Instruction!!! TaDAM!

It looks like that:

echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -A INPUT -p tcp --dport 389 -j ACCEPT
iptables -t nat -A POSTROUTING -d ldap_ip_number -j MASQUERADE
iptables -t nat -A PREROUTING -i bond0 -d box_ip_number -p tcp --dport 389 -j DNAT --to ldap_ip_number:389

It does the following:

you connect on port 389 of the "box" machine and you can use LDAP running on "ldap" machine.

Simple as it is, but finally I got my satisfaction.

Hope this helps! Good luck.