Problem: Oozie fails smoke tests in secured cluster.
Workaround:
Download the following files attached to, https://issues.apache.org/jira/browse/AMBARI-2879:
check_oozie_status.sh
oozieSmoke.sh
Replace
/var/lib/ambari-agent/puppet/modules/hdp-nagios/files/check_oozie_status.shwith the downloaded file.On the Nagios Server host machine, restart Nagios using the following command:
service nagios start
Replace
/var/lib/ambari-agent/puppet/modules/hdp-oozie/files/oozieSmoke.shwith the downloaded file all the hosts in your cluster.Restart Oozie on the Oozie Server host machine using the following command:
sudo su -l $OOZIE_USER -c "cd $OOZIE_LOG_DIR/log; /usr/lib/oozie/bin/oozie-start.sh"
where:
$OOZIE_USERis the Oozie Service user. For example,oozie$OOZIE_LOG_DIRis the directory where Oozie log files are stored (for example:/var/log/oozie).
Problem:
TestBundleJobsFiltertest fails on RHEL v6.3, Oracle v6.3, and SUSE clusters with PostgreSQL.This issue is caused due to the strict typing of PostgreSQL which restricts the auto casting of
string integerto aninteger. The issue is reported when string representation of integer values is substituted into a query for PostgreSQL on the JPA layer.Problem: Delegation Token renewal exception in JobTracker logs.
The following exception is reported in the JobTracker log file when executing a long running job on Oozie in secure mode:
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:jt/hor1n22.gq1.ygridcore.net@HORTON.YGRIDCORE.NET cause:org.apache.hadoop.security.AccessContr olException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt 2013-04-25 15:09:41,543 ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 06 68 72 74 5f 71 61 02 6a 74 34 6f 6f 7a 69 65 2f 68 6f 72 31 6e 32 34 2e 67 71 31 2e 79 67 72 69 64 63 6f 72 65 2e 6e 65 74 40 48 4f 52 54 4f 4e 2e 59 47 52 49 44 43 4f 52 45 2e 4e 45 54 8a 01 3e 41 b9 67 b8 8a 01 3e 65 c5 eb b8 8f 88 8f 9c, Kind: HDFS_DELEGATION_TOKEN, Service: 68.142.244.41:8020. Not rescheduled org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:678) at org.apache.hadoop.security.token.Token.renew(Token.java:309) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:6280) at org.apache.hadoop.hdfs.server.namenode.NameNode.renewDelegationToken(NameNode.java:652) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1118) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy7.renewDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:676) ... 9 moreWorkaround: Any new job job on secure cluster that runs longer than the validity of the Kerberos ticket (typically 24 hours) will fail as the delegation token will not be renewed.

