CreateIndex failed #423

soyarym · 2021-04-18T22:57:00Z

soyarym
Apr 18, 2021

Hello!
I use pyspark 2.4.7 and on-premise Hadoop cluster, and also Hyperspace v 0.4.0 by adding param "spark.jars.packages": {"com.microsoft.hyperspace:hyperspace-core_2.11": "0.3.0"} to the spark-session conf.
When I try to reproduce the example from the quick start guide:

sample_data = [(1, "name1"), (2, "name2")]

spark.createDataFrame(sample_data, ['id', 'name'])\
    .write\
    .mode("overwrite")\
    .parquet("warehouse/something_sb.db/hyperspace_test_1")

df = spark.read.parquet("warehouse/something_sb.db/hyperspace_test_1")

from hyperspace import Hyperspace

hs = Hyperspace(spark)

from hyperspace import IndexConfig

idxConfig = IndexConfig("test_index", ["id"], ["name"])
hs.createIndex(df, idxConfig)

I get this error:

Py4JJavaError: An error occurred while calling o324.createIndex.
: com.microsoft.hyperspace.HyperspaceException: Could not acquire proper state
	at com.microsoft.hyperspace.actions.Action$class.saveEntry(Action.scala:79)
	at com.microsoft.hyperspace.actions.Action$class.begin(Action.scala:53)
	at com.microsoft.hyperspace.actions.Action$class.run(Action.scala:90)
	at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:30)
	at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:43)
	at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77)
	at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:42)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

What did I do wrong?
Thank you!

Answered by sezruby

Apr 19, 2021

@soyarym Yes it's spark.sql.warehouse.dir by default and you can change the dir by setting this config: spark.hyperspace.system.path

View full answer

sezruby · 2021-04-19T04:03:47Z

sezruby
Apr 19, 2021

Hi,

Could you try again with "0.4.0" version?
Could you check the spark driver log if there's logError(s"Failed to write log with id = '$id'", ex)?

2 replies

soyarym Apr 19, 2021
Author

Yep, but I've got the same error.
Yeah, you're totally right:

21/04/19 14:53:21 ERROR IndexLogManagerImpl: Failed to write log with id = '0'
org.apache.hadoop.security.AccessControlException: Permission denied: user=xxxxxx, access=WRITE, inode="/warehouse/indexes/test_index/_hyperspace_log/temp4837c652-bb48-472c-84a3-6828d00439eb":hive:hdfs:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:427)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:277)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1653)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
        at com.microsoft.hyperspace.util.FileUtils$.createFile(FileUtils.scala:39)
        at com.microsoft.hyperspace.index.IndexLogManagerImpl.writeLog(IndexLogManager.scala:155)
        at com.microsoft.hyperspace.actions.Action$class.saveEntry(Action.scala:78)
        at com.microsoft.hyperspace.actions.Action$class.begin(Action.scala:53)
        at com.microsoft.hyperspace.actions.Action$class.run(Action.scala:90)
        at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:30)
        at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:47)
        at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:78)
        at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:43)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=xxxxxx, access=WRITE, inode="/warehouse/indexes/test_index/_hyperspace_log/temp4837c652-bb48-472c-84a3-6828d00439eb":hive:hdfs:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:427)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:277)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
        at org.apache.hadoop.ipc.Client.call(Client.java:1475)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy13.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy14.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
        ... 31 more

What does it mean?

soyarym Apr 19, 2021
Author

Oh, now I understand - it tries to write to the /warehouse/indexes/.. , to which I can't have a permission to write, but I don't understand, why it does that.
In my opinion it should write to the /warehouse/something_sb.db/hyperspace_test_1/indexes/.., isn't it?

soyarym · 2021-04-19T22:46:01Z

soyarym
Apr 19, 2021
Author

I think I find the source of my problem - it is here:
https://github.com/microsoft/hyperspace/blob/master/src/main/scala/com/microsoft/hyperspace/index/PathResolver.scala

Here, to be precise:

def systemPath: Path = {
    val defaultIndexesPath =
      new Path(conf.getConfString("spark.sql.warehouse.dir"), "indexes")
    new Path(conf.getConfString(IndexConstants.INDEX_SYSTEM_PATH, defaultIndexesPath.toString))
  }

I suppose we can fix it by getting additional param like "spark.sql.warehouse.index.dir" from spark.conf, but If it's empty let it be by default "spark.sql.warehouse.dir".
@sezruby What do you think?

0 replies

sezruby · 2021-04-19T23:17:35Z

sezruby
Apr 19, 2021

@soyarym Yes it's spark.sql.warehouse.dir by default and you can change the dir by setting this config: spark.hyperspace.system.path

1 reply

soyarym Apr 19, 2021
Author

Thanks, man, it works! :)
You can't imagine how stupid I'm feeling myself now.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CreateIndex failed #423

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

CreateIndex failed #423

Uh oh!

soyarym Apr 18, 2021

Replies: 3 comments · 3 replies

Uh oh!

sezruby Apr 19, 2021

Uh oh!

Uh oh!

soyarym Apr 19, 2021 Author

Uh oh!

soyarym Apr 19, 2021 Author

Uh oh!

Uh oh!

soyarym Apr 19, 2021 Author

Uh oh!

sezruby Apr 19, 2021

Uh oh!

soyarym Apr 19, 2021 Author

soyarym
Apr 18, 2021

Replies: 3 comments 3 replies

sezruby
Apr 19, 2021

soyarym Apr 19, 2021
Author

soyarym Apr 19, 2021
Author

soyarym
Apr 19, 2021
Author

sezruby
Apr 19, 2021

soyarym Apr 19, 2021
Author