Skip to content

[Bug] Some entities of spark uniffle ui cannot be deserialized in history server hybridStore #2709

@wForget

Description

@wForget

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

Some entities of spark uniffle ui cannot be deserialized in spark history server hybridStore.

com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "total" (class org.apache.uniffle.common.ShuffleReadTimes), not marked as ignorable (7 known properties: "backgroundDecompress", "decompress", "crc", "copy", "fetch", "backgroundFetch", "deserialize"])
 at [Source: (GZIPInputStream); line: 1, column: 123] (through reference chain: org.apache.spark.AggregatedShuffleReadTimesUIData["times"]->org.apache.uniffle.common.ShuffleReadTimes["total"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1138)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2224)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1709)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1687)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1409)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3809)
	at org.apache.spark.util.kvstore.KVStoreSerializer.deserialize(KVStoreSerializer.java:70)
	at org.apache.spark.status.protobuf.KVStoreProtobufSerializer.deserialize(KVStoreProtobufSerializer.scala:38)
	at org.apache.spark.util.kvstore.RocksDB.get(RocksDB.java:169)
	at org.apache.spark.util.kvstore.RocksDB.read(RocksDB.java:181)
	at org.apache.spark.UniffleStatusStore.shuffleReadTimes(UniffleStatusStore.scala:66)
	at org.apache.spark.ui.ShufflePage.doRender(ShufflePage.scala:165)
	at org.apache.spark.ui.ShufflePage.render(ShufflePage.scala:473)
com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.apache.spark.ShuffleType` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
 at [Source: (GZIPInputStream); line: 1, column: 16] (through reference chain: org.apache.spark.ShuffleTaskSummary["shuffleType"])
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1915)
	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1360)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserialize(AbstractDeserializer.java:274)
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1409)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3809)
	at org.apache.spark.util.kvstore.KVStoreSerializer.deserialize(KVStoreSerializer.java:70)
	at org.apache.spark.status.protobuf.KVStoreProtobufSerializer.deserialize(KVStoreProtobufSerializer.scala:38)
	at org.apache.spark.util.kvstore.RocksDB.get(RocksDB.java:169)
	at org.apache.spark.util.kvstore.RocksDB.read(RocksDB.java:181)

Affects Version(s)

0.10.0

Uniffle Server Log Output

Uniffle Engine Log Output

Uniffle Server Configurations

Uniffle Engine Configurations

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions