Skip to content

写入 rdd 索引到 hdfs后 重新reload 报错 麻烦帮忙看看 谢谢 #1

@lzp0070007

Description

@lzp0070007
    SparkConf conf = new SparkConf();
    conf.setAppName("GeoSpark03");
    conf.setMaster("local[*]");
    conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
    conf.set("spark.kryo.registrator", "org.datasyslab.geospark.serde.GeoSparkKryoRegistrator");
    JavaSparkContext sc = new JavaSparkContext(conf);

    SpatialRDD parkRdd = ShapefileReader.readToGeometryRDD(sc, "D:/bigdata/data/parks");
    // 构建索引
    boolean buildOnSpatialPartitionedRDD = false; // 如果只需要在做空间分析的时候构建索引,则设置为true
    // 临近查询(KNN)
    parkRdd.buildIndex(IndexType.RTREE, buildOnSpatialPartitionedRDD);  // QTREE不支持KNN查询
    //写入
    parkRdd.indexedRawRDD.saveAsObjectFile("hdfs://localhost:8020/parkrdd");

    //读取
    SpatialRDD savedRDD = new SpatialRDD<Geometry>();
    savedRDD.indexedRawRDD = sc.objectFile("hdfs://localhost:8020/parkrdd");
    GeometryFactory geometryFactory = new GeometryFactory();
    boolean usingIndex = true;  // 使用索引
    Point pointObject = geometryFactory.createPoint(new Coordinate(-123.1,49.2));
    int K = 5; // K Nearest Neighbors
    List<Geometry> result = KNNQuery.SpatialKnnQuery(savedRDD, pointObject, K, usingIndex);  这里报错
    // 输出结果
    System.out.println("距离点(-123.1,49.2)最近的五个公园是:");
    for (int i = 0; i < result.size(); i++){
        System.out.println(String.format("%d: %s",i+1, result.get(i).toString()));
    }
    sc.stop();

image

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost, executor driver): java.util.NoSuchElementException: next on empty iterator

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions