Flatpak TeleSculptor

[kevinsmia1939]

Hello,

I am trying to build Telesculptor for Linux Flatpak.

I got the error below.
I build Kwiver 1.5.1 with cmake with the flags,

       -DCMAKE_BUILD_TYPE:STRING=Release
       -DKWIVER_ENABLE_SPROKIT=ON
       -DKWIVER_ENABLE_C_BINDINGS=ON
       -DKWIVER_ENABLE_TOOLS=ON

Build qtExtensions master branch with cmake with default flag. https://github.com/Kitware/qtextensions
Build VTK 9.0.1 with cmake with these flags,

     -DBUILD_SHARED_LIBS=ON
     -DOpenGL_GL_PREFERENCE=GLVND
     -DCMAKE_BUILD_TYPE=Release
     -DVTK_PYTHON_VERSION:STRING=3
     -DVTK_GROUP_ENABLE_Qt=YES
     -DVTK_MODULE_ENABLE_VTK_GUISupportQt=YES
     -DModule_vtkGUISupportQtOpenGL:BOOL=ON

And finally Telesculptor 1.1.1 with Cmake.

       -DTELESCULPTOR_SUPERBUILD=OFF
       -DCMAKE_BUILD_TYPE:STRING=Release
       -DTELESCULPTOR_ENABLE_MANUALS=OFF
       -DTELESCULPTOR_ENABLE_TESTING=OFF

Could this issue be relate to this?
Kitware/kwiver#517

I got this error.

-- The C compiler identification is GNU 10.2.0
-- The CXX compiler identification is GNU 10.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for kwiver in : 
-- Current path: '/app/lib/cmake/kwiver;/run/build/telesculptor/CMake'
-- Import CMake future from '/app/lib/cmake/kwiver/future'
-- Import CMake future '3.10'
-- Found Doxygen: /usr/bin/doxygen (found version "1.9.1") found components: doxygen missing components: dot
-- Could NOT find Sphinx (missing: SPHINX_EXECUTABLE) 
-- Performing Test has_compiler_flag--std=cplusplus11
-- Performing Test has_compiler_flag--std=cplusplus11 - Success
-- Performing Test has_compiler_flag--pthread
-- Performing Test has_compiler_flag--pthread - Success
-- Performing Test has_compiler_flag--fvisibility=hidden
-- Performing Test has_compiler_flag--fvisibility=hidden - Success
-- Performing Test has_compiler_flag--Wall
-- Performing Test has_compiler_flag--Wall - Success
-- Performing Test has_compiler_flag--Werror=return-type
-- Performing Test has_compiler_flag--Werror=return-type - Success
-- Performing Test has_compiler_flag--Werror=non-virtual-dtor
-- Performing Test has_compiler_flag--Werror=non-virtual-dtor - Success
-- Performing Test has_compiler_flag--Werror=narrowing
-- Performing Test has_compiler_flag--Werror=narrowing - Success
-- Performing Test has_compiler_flag--Werror=init-self
-- Performing Test has_compiler_flag--Werror=init-self - Success
-- Performing Test has_compiler_flag--Werror=reorder
-- Performing Test has_compiler_flag--Werror=reorder - Success
-- Performing Test has_compiler_flag--Werror=overloaded-virtual
-- Performing Test has_compiler_flag--Werror=overloaded-virtual - Success
-- Performing Test has_compiler_flag--Werror=cast-qual
-- Performing Test has_compiler_flag--Werror=cast-qual - Success
-- Performing Test has_compiler_flag--Werror=vla
-- Performing Test has_compiler_flag--Werror=vla - Success
-- Performing Test has_compiler_flag--Wno-unknown-pragmas
-- Performing Test has_compiler_flag--Wno-unknown-pragmas - Success
-- Performing Test has_compiler_flag--Wl,--no-undefined
-- Performing Test has_compiler_flag--Wl,--no-undefined - Success
-- Performing Test has_compiler_flag--Wl,--copy-dt-needed-entries
-- Performing Test has_compiler_flag--Wl,--copy-dt-needed-entries - Success
-- checking CPP_AUTO auto.cxx
-- checking CPP_CONSTEXPR constexpr.cxx
-- checking CPP_DEFAULT_CTOR default-ctor.cxx
-- checking CPP_FINAL final.cxx
-- checking CPP_NOEXCEPT throw-noexcept.cxx
-- checking CPP_RANGE_FOR range-for.cxx
-- checking STD_CHRONO std_chrono.cxx
-- checking STD_NULLPTR null_ptr.cxx
-- checking STD_REGEX std_regex.cxx
-- Found Git: /usr/bin/git (found version "2.30.0") 
-- [configure-version.h] Creating configure command
-- Making library "maptk"
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- [configure-gui_align] Creating configure command
-- [configure-gui_bundle_adjust] Creating configure command
-- [configure-gui_compute_depth] Creating configure command
-- [configure-gui_default_camera_intrinsics] Creating configure command
-- [configure-gui_filter_tracks] Creating configure command
-- [configure-gui_frame_image_writer] Creating configure command
-- [configure-gui_image_list_reader] Creating configure command
-- [configure-gui_image_video_reader] Creating configure command
-- [configure-gui_initialize] Creating configure command
-- [configure-gui_integrate_depth_maps] Creating configure command
-- [configure-gui_keyframe_image_writer] Creating configure command
-- [configure-gui_mask_list_reader] Creating configure command
-- [configure-gui_mask_video_reader] Creating configure command
-- [configure-gui_st_estimator] Creating configure command
-- [configure-gui_track_features] Creating configure command
-- [configure-gui_triangulate] Creating configure command
-- [configure-track_features_embedded_debug] Creating configure command
-- [configure-log4cplus] Creating configure command
CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkFiltersSources' component is 'FiltersSources'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkGUISupportQt' component is 'GUISupportQt'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkIOGeometry' component is 'IOGeometry'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkIOImage' component is 'IOImage'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkIOPLY' component is 'IOPLY'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkIOXML' component is 'IOXML'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkImagingCore' component is 'ImagingCore'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkInteractionStyle' component is 'InteractionStyle'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkInteractionWidgets' component is
  'InteractionWidgets'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkRenderingAnnotation' component is
  'RenderingAnnotation'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-config.cmake:64 (message):
  The new name for the 'vtkRenderingFreeType' component is
  'RenderingFreeType'
Call Stack (most recent call first):
  gui/CMakeLists.txt:17 (find_package)


-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found X11: /usr/include   
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found OpenGL: /usr/lib/x86_64-linux-gnu/libOpenGL.so  found components: OpenGL GLX 
CMake Deprecation Warning at /app/lib/cmake/vtk-9.0/vtk-use-file-deprecated.cmake:1 (message):
  The `VTK_USE_FILE` is no longer used starting with 8.90.
Call Stack (most recent call first):
  gui/CMakeLists.txt:35 (include)


-- File: 'ACKNOWLEDGMENTS'
-- File: 'BUILDINFO'
-- File: 'LICENSE'
-- [configure-telesculptor-config] Creating configure command
-- [configure-telesculptor-install-config] Creating configure command
-- Configuring done
CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "maptk_detect_and_describe" links to target
  "kwiver::kwiver_algo_core" but the target was not found.  Perhaps a
  find_package() call is missing for an IMPORTED target, or an ALIAS target
  is missing?
Call Stack (most recent call first):
  tools/CMakeLists.txt:8 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "maptk_track_features" links to target "kwiver::kwiver_algo_core"
  but the target was not found.  Perhaps a find_package() call is missing for
  an IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  tools/CMakeLists.txt:17 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "maptk_analyze_tracks" links to target "kwiver::kwiver_algo_core"
  but the target was not found.  Perhaps a find_package() call is missing for
  an IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  tools/CMakeLists.txt:56 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "maptk_match_matrix" links to target "kwiver::kwiver_algo_core" but
  the target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  tools/CMakeLists.txt:26 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "maptk_bundle_adjust_tracks" links to target
  "kwiver::kwiver_algo_core" but the target was not found.  Perhaps a
  find_package() call is missing for an IMPORTED target, or an ALIAS target
  is missing?
Call Stack (most recent call first):
  tools/CMakeLists.txt:40 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "maptk_apply_gcp" links to target "kwiver::kwiver_algo_core" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  tools/CMakeLists.txt:48 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "TeleSculptor" links to target "kwiver::kwiver_adapter" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  gui/CMakeLists.txt:199 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "TeleSculptor" links to target "kwiver::kwiver_algo_core" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  gui/CMakeLists.txt:199 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "TeleSculptor" links to target "Boost::chrono" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?
Call Stack (most recent call first):
  gui/CMakeLists.txt:199 (kwiver_add_executable)


CMake Error at /app/lib/cmake/kwiver/utils/kwiver-utils-targets.cmake:182 (add_executable):
  Target "TeleSculptor" links to target "Boost::system" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?
Call Stack (most recent call first):
  gui/CMakeLists.txt:199 (kwiver_add_executable)


-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
Error: module telesculptor: Child process exited with code 1

[mleotta]

The issue is that the current release branch of TeleSculptor does not support VTK 9. The master branch should support VTK 9, but it is not enabled by default in the superbuild. The superbuild option is the easiest way to build any version of TeleSculptor and all the dependencies that it needs. Even if you plan to manually build KWIVER and projects in Fletch you can look at the CMake to see how the superbuild configures them together.

If you want to build against VTK 9 you can change the 8.2 to 9.0 on this line:
https://github.com/Kitware/TeleSculptor/blob/master/CMake/telesculptor-external-fletch.cmake#L66
and build the superbuild. This should work on the master branch, but not release. The master branch of TeleSculptor requires the master branches of Fletch and KWIVER to make this all work, but this is handled by the superbuild.

[kevinsmia1939]

Thanks for reply.

I decide to go with VTK 8.2.0.
But I was hit with this bug when building VTK.
https://gitlab.kitware.com/vtk/vtk/-/issues/17774

[mleotta]

I see. I don't think and of our testing covers gcc 10 yet.

Again, you should be able to make it work with VTK 9 if you use the master branch. We are working on getting those changes out in a release, hopefully soon.

[kevinsmia1939]

I see. I don't think and of our testing covers gcc 10 yet.

Again, you should be able to make it work with VTK 9 if you use the master branch. We are working on getting those changes out in a release, hopefully soon.

Does superbuild work without network connection? I am trying to build a Flatpak for Telescuplter. So it do not allow external download, all dependencies will have to be included first.

Thanks.

[mleotta]

Fletch does require a network connection to download the source packages for the projects it builds. However, you can run that download step once and cache the downloads. It doesn't download if the files are found locally.

I don't know much about Flatpak, but isn't that a system for packaging binaries? Wouldn't you build the software locally and then package all the binaries into Flatpak? Note that the superbuild does something similar already but it creates a .tar.gz or self extracting archive of all the binaries using CMake/CPack. If we could extend this to make a flatpak that would be useful, because our current approach to packaging for linux is not great.

[kevinsmia1939]

Fletch does require a network connection to download the source packages for the projects it builds. However, you can run that download step once and cache the downloads. It doesn't download if the files are found locally.

I don't know much about Flatpak, but isn't that a system for packaging binaries? Wouldn't you build the software locally and then package all the binaries into Flatpak? Note that the superbuild does something similar already but it creates a .tar.gz or self extracting archive of all the binaries using CMake/CPack. If we could extend this to make a flatpak that would be useful, because our current approach to packaging for linux is not great.

Yes, Flatpak is the system for packaging binaries. Is can be build locally and in Github. There are Flatpak store called Flathub, which I am trying to submit too.
https://flathub.org/home

Step to start building Flatpak can be summed up:
Install flatpak-builder, most Linux distro include it already. For Ubuntu just do sudo apt install flatpak-builder
Create com.github.Kitware.TeleSculptor.yaml or .json
Build locally with flatpak-builder --install --user build com.github.Kitware.TeleSculptor.yaml --force-clean
If it is successful, run with flatpak run com.github.Kitware.TeleSculptor
If you want to submit to Flathub, create pull request to https://github.com/flathub/flathub, which I wanted to do.
https://github.com/flathub/flathub/wiki/App-Submission

This is what I worked on so far: https://github.com/kevinsmia1939/com.github.Kitware.TeleSculptor
If you want to try it, just clone the repo including shared-modules, and build with flatpak-builder --install --user build com.github.Kitware.TeleSculptor.yaml --force-clean

Inside the com.github.Kitware.TeleSculptor.yaml, you will see a lot of dependencies being downloaded (in source: section). The build process is from top to bottom, so kwiver and fletch must be on top of telescupltor.

The inconvenient thing is that it does not allow cmake or other stuff to download anything except flatpak-builder itself. I think most Linux package build system avoid network connection when building anyway, I think it is to ensure reproducible build. Which kinda make the superbuild not so applicable for flatpak. One of the solution is to link kwiver, fletch, etc as git submodules in Telesculptor (something like this https://github.com/OpenChemistry/tomviz/tree/master/thirdparty). So when Flatpak download Telesculptor with submodule, all deps will be downloaded too.

So maybe create another folder, called it something like 3rdparty or dependencies, and put kwiver and fletch submodule in there, and point cmake to that folder.

[mleotta]

The problem with git submodules is that not everything fletch builds comes from another git repository.

How do you normally specify how to build dependencies for a flatpak?

[kevinsmia1939]

The problem with git submodules is that not everything fletch builds comes from another git repository.

How do you normally specify how to build dependencies for a flatpak?

Just use URL with type archive or git.
Like,

    sources:
      - type: archive
        url: https://github.com/Kitware/fletch/archive/v1.4.0.tar.gz
        sha256: 7e784da627ccced0eae87418d16683d923c18052a17c730fe61f31b6237b145a    

or

    sources:
      - type: git
        url: https://github.com/Kitware/qtextensions
        commit: 38e70b32807c0317cc3804a6dad303f1b0ec2e13 

I'm in the process of hunting for these deps to add it in.

[mleotta]

Here is the file that documents/controls what archives Fletch downloads:
https://github.com/Kitware/fletch/blob/master/CMake/fletch-tarballs.cmake

Some of these have alternate versions depending on how you configure Fletch. Likewise some dependencies might not be needed if they are not enabled in Fletch. TeleSculptor does not turn everything on in Fletch. The packages enabled and versions used are documented here:
https://github.com/Kitware/TeleSculptor/blob/master/CMake/telesculptor-external-fletch.cmake

[kevinsmia1939]

Does the VTK build flag controlled by Fletch?

[mleotta]

TeleSculptor specifies a Fletch configuration that tells Fletch to build VTK and selects from a small number of version choices (e.g. 8.2 and 9.0). Then when Fletch is built it downloads the selected VTK source archive and provides more specific CMake configuration describing how to build VTK depending on which version was selected and what OS we are building on. That configuration is here:
https://github.com/Kitware/fletch/blob/master/CMake/External_VXL.cmake

Does that answer your question?

[kevinsmia1939]

Thanks, that clear it up, but that is some complicated dependencies.

[kevinsmia1939]

Hi,

Is there a way to make Fletch prefer system installed dependencies? I look through cmake files but it is not clear to me. I already have Eigen install for example, but Fletch attempt to download Eigen which it failed because there is no network connection inside Flatpak-builder.

[borovik135]

Kevin, I have recently gone through the procedure of building the TS <= Kwiver <= Fletch chain on Ubuntu 20.04 using Ubuntu's pre-built packages and left my comments in a shared document. I have found that using cmake-gui simplifies package selection via various flags of what to enable/disable in Fletch, Kwiver or TeleSculptor. One can disable most of the packages in Fletch and use Linux distro's pre-built binaries. However, I still had to build Proj4, log4cplus, VXL in Fletch and referenced them in Kwiver and TeleSculptor builds.

[mleotta]

In theory you should be able to turn on only the components you don't have as system packages in Fletch and then let CMake find system packages to fill in the rest. However, we don't test this approach regularly and it is likely that some of your system packages may not be compatible with KWIVER, TeleSculptor, or other packages in Fletch. The purpose of Fletch is to provide a consistent collection of third-party dependency versions that we can build across different platforms. It's hard to support package manger versions of dependencies because every OS packages different versions.

I think the easiest way to make this work (but not necessarily the best from a maintenance perspective) would be to transfer the URLs from the fletch-tarballs.cmake into archive sources in the flatpack file. Then you'll just need a way to move those downloaded archives into the fletch/Downloads directory after fletch is cloned.