learning-pytorch-geometric-on-ipus/ notebooks have C++ endpoint not connected errors

[nmb-paperspace]

2_a_worked_example.ipynb cell 11 from tqdm import tqdm ... fails with

Error: In unknown:0: 'std::exception': boost::filesystem::directory_iterator::operator++: Transport endpoint is not connected: "/tmp/exe_cache/pyg-a-worked-example"
Error raised in:
  [0] popart::TrainingSession::createFromOnnxModel
  [1] Compiler::initSession
  [2] LowerToPopart::compile
  [3] compileWithManualTracing

3_small_graph_batching_with_padding.ipynb cell 14 fails with

Error: In unknown:0: 'std::exception': boost::filesystem::directory_iterator::operator++: Transport endpoint is not connected: "/tmp/exe_cache/pyg-padding"
Error raised in:
  [0] popart::TrainingSession::createFromOnnxModel
  [1] Compiler::initSession
  [2] LowerToPopart::compile
  [3] compileWithManualTracing

4_small_graph_batching_with_packing.ipynb cell 20 poptorch_model.train() ... fails with

Error: In unknown:0: 'std::exception': boost::filesystem::directory_iterator::operator++: Transport endpoint is not connected: "/tmp/exe_cache/pyg-packing"
Error raised in:
  [0] popart::TrainingSession::createFromOnnxModel
  [1] Compiler::initSession
  [2] LowerToPopart::compile
  [3] compileWithManualTracing

The remainder of these 3 notebooks are blocked from running.

[anjleeg-gcai]

Hi, I’m unable to reproduce these errors. Can you give me more information on the runtime details? Which image is your container pointing to? Which machine type are you running on?