Unable to connect to a Hadoop cluster with Kerberos security configured.

[shahabyunus]

Hi,

I am using the HBase UI tool 'H-Rider'. I was able to use it with a cluster which does not have Kerberos security configured but was getting 'failed to connect' errors when trying to connect to a cluster with Kerberos authentication. Is it possible to connect H-Rider with such a cluster? I couldn't find anything where I could set this settings or properties file. I am able to connect to this secured cluster through a Java HBase client so Kerberos on my machine is setup properly, I believe.

Thanks & Regards,
Shahab

[icher-g]

Hi,

I'm not an expert in security but I think that kerberos uses credentials of the user that running the process. So if you run h-rider with the user that have access to your cluster it should be ok. After all h-rider uses regular hbase client so if you could do it on your own the h-rider should work as well.

Unfortunately I don't have environment to check it.

Best Regards,
Igor Cher.

[shahabyunus]

Thanks for the reply. Yes, I configure the Kerberos ticket on my macine
independent of any application but having said that, my regular Java client
uses the hbase-site.xml in which I configure kerberos authentication (the
parameter itself and principal settings for Master and Region servers)
matching my target cluster. I had to do this as otherwise even that client
fails to connect.

In HRider's case, which hbase-site.xml it is using? The one in its own lib
folder within the hbase jar? Then it must be the default and without
Kerberos configuration, right? I can try to update that, what do you think?

I will try that and let you know.

Regards,
Shahab

On Mon, Jun 3, 2013 at 10:05 AM, Igor Cher notifications@github.com wrote:

Hi,

I'm not an expert in security but I think that kerberos uses credentials
of the user that running the process. So if you run h-rider with the user
that have access to your cluster it should be ok. After all h-rider uses
regular hbase client so if you could do it on your own the h-rider should
work as well.

Unfortunately I don't have environment to check it.

Best Regards,
Igor Cher.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42#issuecomment-18843252
.

[shahabyunus]

So I latest downloaded the code and built it and now when I try to run it on Linux, I get the following exception:

2013-06-03 15:41:00,832 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from jar '/Users/syunus/Downloads/hrider-master/Users/syunus/Downloads/hrider-master/target/h-rider-1.0.7.0.jar'
2013-06-03 15:41:00,833 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from folder '/Users/syunus/Downloads/hrider-master/converters/classes'
Exception in thread "AWT-EventQueue-0" java.lang.ExceptionInInitializerError
at hrider.data.ColumnQualifier.(ColumnQualifier.java:32)
at hrider.ui.views.DesignerView.initializeColumnsTable(DesignerView.java:1273)
at hrider.ui.views.DesignerView.(DesignerView.java:167)
at hrider.ui.forms.Window.loadView(Window.java:441)
at hrider.ui.forms.Window.loadViews(Window.java:423)
at hrider.ui.forms.Window.createAndShowGUI(Window.java:271)
at hrider.ui.forms.Window.access$1300(Window.java:59)
at hrider.ui.forms.Window$7.run(Window.java:256)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:708)
at java.awt.EventQueue.access$400(EventQueue.java:82)
at java.awt.EventQueue$2.run(EventQueue.java:669)
at java.awt.EventQueue$2.run(EventQueue.java:667)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:678)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Caused by: java.lang.IllegalArgumentException: converter cannot be null
at hrider.data.ColumnType.(ColumnType.java:60)
at hrider.data.ColumnType.(ColumnType.java:55)
at hrider.data.ColumnType.(ColumnType.java:33)

I get the similar exception when I download and run the packaged the 1.0.7 version from the site.

Thanks,
Shahab

[shahabyunus]

Just adding the exception that I get when I directly try to run the 1.0.7 version (the pre-packaged one downloaded from the main site.)

2013-06-03 16:16:45,256 INFO [AWT-EventQueue-0] RunnableAction : Action 'compare-versions' started.
2013-06-03 16:16:46,636 ERROR [compare-versions] Window : Failed to retrieve update information from the URL: https://raw.github.com/NiceSystems/hrider/master/update.properties
java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
at java.util.regex.Matcher.reset(Matcher.java:308)
at java.util.regex.Matcher.(Matcher.java:228)
at java.util.regex.Pattern.matcher(Pattern.java:1088)
at java.util.regex.Pattern.split(Pattern.java:1197)
at java.util.regex.Pattern.split(Pattern.java:1259)
at hrider.system.Version.(Version.java:43)
at hrider.system.Version.compare(Version.java:37)
at hrider.ui.forms.Window.compareVersions(Window.java:333)
at hrider.ui.forms.Window.access$1200(Window.java:59)
at hrider.ui.forms.Window$6.run(Window.java:241)
at hrider.actions.RunnableAction.run(RunnableAction.java:132)
at java.lang.Thread.run(Thread.java:722)
2013-06-03 16:16:46,638 INFO [compare-versions] RunnableAction : Action 'compare-versions' completed.
2013-06-03 16:17:00,525 INFO [AWT-EventQueue-0] RunnableAction : Action 'xxxx.corporate' started.
2013-06-03 16:17:03,244 INFO [tul1cdhdmn01.corporate.local] RunnableAction : Action 'xxxx.corporate' completed.
2013-06-03 16:17:03,635 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from package 'hrider.converters'
2013-06-03 16:17:03,636 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from jar '/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/h-rider-1.0.7.0.jar'
2013-06-03 16:17:03,637 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from folder '/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/converters/classes'

The following path does not exist. Why is it trying to go there?
'/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/converters/classes'

The older version that I am using 1.0.3, works fine with non-Kerberos clusters, as mentioned before.

Thanks,
Shahab

[icher-g]

Hi,

The h-rider doesn't use hbase-site.xml or any other XML. It simply configures Configuration object directly before connecting to hbase. Can you tell me how do you configure kerberos in hbase-site.xml?

In regards to the errors you get, are you sure you have the latest code? The jar still contains the bug (although it should not appear when the h-rider is executed from the jar) but the code is fixed. So running in debug or recompiling the code should work.

[shahabyunus]

First, just to clarify one thing, HBase first loads the hbase-default.xml and core-default.xml which comes bundled in the hadoop and hbase jars as dependencies and then loads these custom ones (*-site.xml) on top of that. Point being, the client does need some hbase/core config files on its classpath. In most of the cases it is the one packaged in the dependent libs which suffices with user only have to enter zookeeper.quorum. This is what is happening, I believe, with the H-rider too where you only get the zookeeper.quorom from the user.

Issue#1:
I downloaded the jar from http://bit.ly/19PgWvd. The May 13 release. When I try to run it in Mac, I see the following error in the log file, h-rider.log and it doesn't open. Note that I am connecting to a normal server without Kerberos which is working fine with 1.0.3:

2013-06-04 09:32:37,809 INFO [AWT-EventQueue-0] RunnableAction : Action 'compare-versions' started.
2013-06-04 09:32:39,226 ERROR [compare-versions] Window : Failed to retrieve update information from the URL: https://raw.github.com/NiceSystems/hrider/master/update.properties
java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
at java.util.regex.Matcher.reset(Matcher.java:308)
at java.util.regex.Matcher.(Matcher.java:228)
at java.util.regex.Pattern.matcher(Pattern.java:1088)
at java.util.regex.Pattern.split(Pattern.java:1197)
at java.util.regex.Pattern.split(Pattern.java:1259)
at hrider.system.Version.(Version.java:43)
at hrider.system.Version.compare(Version.java:37)
at hrider.ui.forms.Window.compareVersions(Window.java:333)
at hrider.ui.forms.Window.access$1200(Window.java:59)
at hrider.ui.forms.Window$6.run(Window.java:241)
at hrider.actions.RunnableAction.run(RunnableAction.java:132)
at java.lang.Thread.run(Thread.java:722)
2013-06-04 09:32:39,228 INFO [compare-versions] RunnableAction : Action 'compare-versions' completed.
2013-06-04 09:32:52,625 INFO [AWT-EventQueue-0] RunnableAction : Action 'xxxx.corporate' started.
2013-06-04 09:32:54,991 INFO [xxxx.corporate] RunnableAction : Action 'xxxx.corporate' completed.
2013-06-04 09:32:55,223 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from package 'hrider.converters'
2013-06-04 09:32:55,224 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from jar '/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/h-rider-1.0.7.0.jar'
2013-06-04 09:32:55,225 INFO [AWT-EventQueue-0] JavaPackage : Loading classes from folder '/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1/converters/classes'

I am running it from Users/syunus/Downloads/h-rider-1.0.7.0-hbase-0.94.1, so I don't get why it is appending the extra path. I then downloaded the code, built it myself (using maven profile 0.94.1) and when I run it, I see the same error. Of course, the directory paths are different but still incorrect.

As mentioned, the 1.0.3 version is working fine.

Issue# 2:
With Kerberos, we have to set the following properties in hbase-site.xml and core-site.xml as explained the official docs. It is a pretty standard config. Nothing custom.
http://hbase.apache.org/book/security.html

I have set those for thee xmls, in my regular Java client and it works. I have tried adding these configuration files on the class path of h-rider with both versions (1.0.3 and 1.0.7) and H-rider does not work. If I run h-rider, without any of these custom configuration files then I get 'Connection reset by peer' exceptions when I try to connect to a Kerberos configured cluster. With these files, I get NoClassFoundError (KerberosName) which is kind of strange as this class does exist in the included hadoop library. I am still looking into it in case I am adding thoes configuration files incorrectly in the class path. But in any case, as per my understanding (and experience with the regular Java client) we do need to have Kerberos-configured hbase/core-site.xml files on the class path which will be read by the HBaseConfiguration.create() which reads all files on the path with hbase-*.xml.

I added some debug statements in the code in the Connection.java and ConnectionDetails and that is how I got the messages above.

Let me know if any question.

[icher-g]

Hi,

In regards to the first issue, it fails because by some reason h-rider cannot read the jar version from its manifest file. I experienced this only when I ran it from the Intellij in debug mode. In regular execution it should work.

Regarding the path this is a known issue

I will check the security configuration required for kerberos.

[icher-g]

Hi,

Try to add the following code to the ConnectionDetails.createConfig() method:

config.set("hbase.security.authentication", "kerberos");

Let me know if it helped you.

Best Regards,

[shahabyunus]

Nope, that doesn't help. I have to copy my hbase/core-site.xml files in the
src/main/resources folder. The configuration object does pick up the
kerberos related configurations from there (hbase-site.xml has everything
needed for our particular cluster configuration) but now I get this. I am
using 1.0.3 code with 0.94.1 profile to build:
java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.ipc.SecureRpcEngine
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.ipc.SecureRpcEngine
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProtocolEngine(HBaseRPC.java:114)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:335)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:312)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:364)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:665)
at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:109)
at hrider.hbase.Connection.(Connection.java:83)
at hrider.hbase.ConnectionManager.create(ConnectionManager.java:23)
at
hrider.ui.forms.ConnectionDetailsDialog.onOK(ConnectionDetailsDialog.java:115)
at
hrider.ui.forms.ConnectionDetailsDialog.access$000(ConnectionDetailsDialog.java:32)
at
hrider.ui.forms.ConnectionDetailsDialog$1.actionPerformed(ConnectionDetailsDialog.java:56)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2028)
at
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2351)
at
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.AbstractButton.doClick(AbstractButton.java:389)
at
javax.swing.plaf.basic.BasicRootPaneUI$Actions.actionPerformed(BasicRootPaneUI.java:191)
at javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1645)
at javax.swing.JComponent.processKeyBinding(JComponent.java:2859)
at javax.swing.KeyboardManager.fireBinding(KeyboardManager.java:267)
at javax.swing.KeyboardManager.fireKeyboardAction(KeyboardManager.java:216)
at
javax.swing.JComponent.processKeyBindingsForAllComponents(JComponent.java:2936)
at javax.swing.JComponent.processKeyBindings(JComponent.java:2928)
at javax.swing.JComponent.processKeyEvent(JComponent.java:2822)
at java.awt.Component.processEvent(Component.java:6159)
at java.awt.Container.processEvent(Container.java:2083)
at java.awt.Component.dispatchEventImpl(Component.java:4744)
at java.awt.Container.dispatchEventImpl(Container.java:2141)
at java.awt.Component.dispatchEvent(Component.java:4572)
at
java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1856)
at
java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:722)
at
java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1000)
at
java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:865)
at
java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:686)
at java.awt.Component.dispatchEventImpl(Component.java:4616)
at java.awt.Container.dispatchEventImpl(Container.java:2141)
at java.awt.Window.dispatchEventImpl(Window.java:2489)
at java.awt.Component.dispatchEvent(Component.java:4572)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:710)
at java.awt.EventQueue.access$400(EventQueue.java:82)
at java.awt.EventQueue$2.run(EventQueue.java:669)
at java.awt.EventQueue$2.run(EventQueue.java:667)

On Tue, Jun 4, 2013 at 10:15 AM, Igor Cher notifications@github.com wrote:

Hi,

Try to add the following code to the ConnectionDetails.createConfig()
method:

config.set("hbase.security.authentication", "kerberos");

Let me know if it helped you.

Best Regards,

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42#issuecomment-18911658
.

[shahabyunus]

Ok, I made it work. So what I did:

1- Included those hbase-site and core-site XML files (in the src/main/resources) with correct information about our cluster's specific Kerberos configuration. These are exactly same what we use in other HBase clients.

2- Set the following System properties before loading the configuration (creating the Configuration object) for HBase (I did it in ConnectionDetails.java):
System.setProperty("java.security.krb5.realm", "YOUR.REALM");
System.setProperty("java.security.krb5.kdc", "COLON SEPARATED LIST OF YOUR KDC SERVERS");

Of course you need a kerberos ticket on your local machine but that is something independent of H-Rider and a general step required to make Kerberos work on your machine.

3- Rebuilt it with the cdh4 profile (not 0.94.1).

It now works. This is for 1.0.3. I am unable to run 1.0.7 in general, kerberos or not.

The thing to note is that this is of course hard-coded solution and not an ideal deployment/design. I mean that I have to include the hbase/core-site XMLs in the code and rebuild. So in future if any of the Kerberos settings change I have update the config files and build it again. Plus more importantly, this specific jar that I have generated will now ONLY work for this cluster and no where else.

But at least we know that how we can use Kerberos with H-Rider :)

Regards,
Shahab

[icher-g]

Good to know you handled it.
You are right I cannot add it as a general solution so I need to find another way to support kerberos security.

You probably cannot run 1.0.7 because it is compiled with 0.94.1 hbase. At some point I added to h-rider a feature which is supported by hbase only starting from 0.94 version so two other profiles are not supported any more.

Best Regards,

[damienclaveau]

Hi,

Instead of hardcoding System properties, you can also set it at launchtime
java -Djava.security.krb5.conf=/etc/krb5.conf -jar target/h-rider-1.0.9-SNAPSHOT.jar
Notice that hdfs-site.xml could be required if you have NN HA and an associated nameservice.
Works great for me.

[OhadR]

Thanks. Feel free to prepare a merge request, and we will add it! On Nov 10, 2017 15:22, "Damien Claveau" <notifications@github.com> wrote: Hi, Instead of hardcoding System properties, you can also set it at launchtime java -Djava.security.krb5.conf=/etc/krb5.conf -jar target/h-rider-1.0.9-SNAPSHOT.jar and add core-site.xml,hdfs-site.xml and hbase-site.xml in the classpath. Works great for me. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACSLZDeYgtfI6Nnvq-YdN-eQ40hnWyaeks5s1E4AgaJpZM4AtAqY> .