absence of __complex__ attribute in h5 file

[dombrno]

Good morning,
I use ALPSCORE/CT-HYB for computing G1 and G2, and then post process the output data, using C++ code using the same ALPSCORE libraries. In my own C++ code, I read and write code from and to hdf5, using the same process as that used in ALPSCORE/CT-HYB. For example in this code snippet, I save a complex boost multiarray to hdf5:

https://github.com/dombrno/SC_Loop_ALPSCore/blob/eb14d7459b248b7a3d7f01a19c0c68fda22b59cc/src/greens_function.cpp#L207-L225

This seems to me the same exact process used to save, e.g. the G2_LEGENDRE data in ALPSCORE/CT-HYB:

https://github.com/dombrno/CT-HYB-1/blob/adbb7b81f16e85f5d86748649af5fb6bb06cb39a/src/postprocess.hpp#L297-L357

Nevertheless, in the case of my code, the hdf5 dataset ends up being equipped with the attribute complex = 1, while in the case of ALPSCORE/CT-HYB, this attribute is not created.

It is not a big problem, but that leads me to having to use two different procedures for reading data, depending on whether they were generating by ALPSCORE/CT-HYB, or by my own CT-QMC code (2-orbital version of CT-HYB-SEGMENT): if the data comes from ALPSCORE/CT-HYB, I need to recombine the real and imaginary parts into a complex boost::multiarray, while the reading of complex data works direct when coming from my other code (also using ALPSCORE libraries), thanks to the attribute being present.

https://github.com/dombrno/SC_Loop_ALPSCore/blob/eb14d7459b248b7a3d7f01a19c0c68fda22b59cc/src/greens_function.cpp#L244-L269

I am using ALPSCORE c6eddbb9326a82a970a5a73569bc61f82cc1b46f , and ALPSCORE/CT-HYB 26f6acf , i.e. relatively recent versions from end of March.

[dombrno]

I attach two screenshots to illustrate my point: they show my hdf5 file reader, with windows showing the properties of two "compatible" objects: G2_LEGENDRE from ALPSCORE/CT-HYB, and bubbles, produced by my code based on ALPSCORE. On can see that a small read "A" in the left panel indicates the presence of the attribute, and the dialog windows show the full details of the data structure/attributes. In the case of ALPSCORE/CT-HYB, the attribute is absent. The shape of the data is nevertheless identical. I have tried to add the attribute manually, and check if I could read the complex type directly, but it seems that I cannot add this exact kind of attribute manually through HDFView, so could not perform the test.

[egull]

Hi,
at this point we don't plan to change the way ALPS stores complex numbers (that's really an ALPS issue, not a CT-HYB issue). Complex storage in HDF5 was a mess last time we looked at it, with different codes doing it in a different (and incompatible) way.

I can see a few ways of how you could 'hack' this, but none of them are nice:

1. read in the ALPS information as real double precision arrays, with separate imaginary and real parts, and then rearrange (or cast to complex) in your code.

2. Have an intermediate step where you read the information into, say, h5py, and just add the attribute.

3. Change your routine to conform to ALPS, rather than ALPS to conform to you.

@shinaoka , do you have a better idea?

[dombrno]

Hello,
thanks for your answer. I am doing 1. at the moment for data coming from CT-HYB, and that's quite all right with me.

There is absolutely no issue with the way ALPS stores complex numbers. My point was rather that I am perplex as to how/why two different codes using the same procedure on the same hardware with the same version of the ALPSCore and h5 libraries can produce two different outputs as far as the h5 attributes are concerned, and if anyone had an idea how I could make the behavior identical in all codes (maybe some compiler option that anyone would know of or such, because really the main difference between the codes would be at the cmake level).

[egull]

@galexv, can you weigh in? Any idea where this comes from?

[galexv]

@dombrno Could you please send your CMakeCache.txt from:

1. ALPSCore build directory
2. CT-HYB build directory
3. Your code build directory

If you could send a minimal (or small!) code example demonstrating the behavior, it may help. For example, could you please send an input for CT-HYB that runs in a short time and produces the G2_LEGENDRE data without __complex__ attribute?

I will look at the problem meantime. (First wild guess: could G2_LEGENDRE be overwritten somewhere with real-valued data? Or could the attribute be inadvertently dropped by some other means?)

[dombrno]

@galexv Thank you for looking into this. As a start, you will find attached the files requested in 1. 2. 3. I will prepare a small code example demonstrating the behavior. please note that none of the results dropped by CTHYB at the root level of the h5 files possesses the complex attribute. Among them G1_LEGENDRE and G2_LEGENDRE. Conversely, the parameters object and its contents do possess a bunch of attributes. I cannot see at this point how the G2_LEGENDRE data could be overwritten with real valued data: the h5 file is handled by my post process directly as it comes out of CTHYB.

I will prepare a small example producing a G1_LEGENDRE and send it to you asap.

CMakeCache.mycode.txt
CMakeCache.CTHYB.txt
CMakeCache.ALPSCORE.txt

[shinaoka]

Sorry for being late.
I looked into sample data computed on my machine.
You're right, that attribute is missing in the output of the CT-HYB.

[dombrno]

Interesting - so this seems to be the normal behavior of CT-HYB, but somehow, my codes provide this attribute in their output, even though I used your code as my inspiration for dumping data :). The mystery deepens !

[shinaoka]

I got the point!

In the CT-HYB code, "ar" is an object of std::mapstd::string,boost::any.
We just stored an multi-array object into this map object.
https://github.com/dombrno/CT-HYB-1/blob/adbb7b81f16e85f5d86748649af5fb6bb06cb39a/src/postprocess.hpp#L396

After collecting all results of postprocess, this map object is dumped into a HDF file.
This was designed to enable us to pass multi-dimensional data to TRIQS libraries in C++ level without dumping data into a file.

The dumping is done through a function in boost_any.hpp.
We should set the correct flag at the following lines.
https://github.com/dombrno/CT-HYB-1/blob/adbb7b81f16e85f5d86748649af5fb6bb06cb39a/src/hdf5/boost_any.hpp#L67-L74

[dombrno]

@shinaoka oh very nice, thank you! I had never paid real attention to the fact that you used this boost::any type :) I remember being surprised at spotting it in the past, but had forgotten all about it since!. It all makes sense now. I might look into the fix in boost_any.hpp and offer a pull request? That could teach me some stuff about its inner workings --> actually, this is not very intuitive, I would need to be pointed to a code sample where such a property is added to see how it's done.

Thanks a lot!