Validate code against CCL predictions

[damonge]

We should validate the code in a realistic user scenario

• 3D and skewers was already validated in James' paper.
• LSST-like simulation and validate density, shear and magnification for a couple of distant bins.
• Intensity mapping (generate an SKA-like simulation and validate for a couple of frequency bins, both C_ells and Pk^1D - see e.g. https://arxiv.org/abs/1609.00019)

[cramirezpe]

Hi @damonge,

Following the last discussion, we have been working on the tests for the Cls using theoretical predictions made with CCL.

The script used for this (with some modifications) is cl_test_ccl.py.

We have some doubts/problems that we were not able to solve by ourselves and we wanted to share them with you in case you have any idea of what is happening.

The first thing that we noted is in the line 122 of the previous script:

cl_dd_t = np.array([ccl.angular_cl(cosmo, tr_d[p1], tr_d[p2], larr, p_of_k_a=pk2d_mm)
                    for p1, p2 in pairs])

Here it is used as power spectrum to project the pk2d_mm extracted from the CoLoRe output. The question we ask ourselves is whether we should use pk2d_dd instead since here we are working with the galaxy-galaxy Cls.

We understand that we should match the pk2d_mm with lensing in the same way as pk2d_dd is matched with galaxies. Is it right?

Another doubt related to the script is located in the line 114:

tr_d = [ccl.NumberCountsTracer(cosmo, False, (z_nz, nz_tot[i]), bias=(z_nz, np.ones_like(z_nz)))
        for i in range(nbins)]

Here it is used as bias a one_like array. We understand that this happens because we are using directly the galaxies as the objects and they already include the bias. Is it right?

Moving to the other quantities (dm,mm); we modified the script to allow computing them. However, the results do not look convincent to us:

Here 0,1 are the two redshift bins considered ((0,0.15) and (0.15,inf)).

The results for the galaxy-galaxy part are good (as they were in the original script), even with the line 122 changed (it does not affect this plot by much).

To compute the other quantities (dm,mm) the script was modified as follows:

Data

In the script cl_test_ccl.py data is computed as:

cl_dd_d = np.array([hp.anafast(dmap[p1], dmap[p2]) for p1, p2 in pairs])

We used healpy anafast in a similar way as it was used in the scrip cl_test.py as it allows us to compute all the needed quantities at the same time. For a given pair (p1,p2) (pair of bins):

d_values = 
hp.anafast(np.asarray([dmap[p1],e1map[p1],e2map[p1]]),np.asarray([dmap[p2],e1map[p2],e2map[p2]]), pol=True)

This numpy array d_values will be composed by the cls (following anafast documentation using T -> D): DD, EE, BB, DE, EB, DB.

For the plots we considered E to be the lensing part. That is we used the equivalence:

• dd -> DD
• dm -> DE
• mm-> EE

Theoretical predictions

In cl_test_ccl.py for the dd case it is computed as:

cl_dd_t = np.array([ccl.angular_cl(cosmo, tr_d[p1], tr_d[p2], larr, p_of_k_a=pk2d_mm)
                    for p1, p2 in pairs])

For our theoretical predictions we used:

cl_dd_t = np.array([ccl.angular_cl(cosmo, tr_d[p1], tr_d[p2], larr, p_of_k_a=pk2d_dd)
                    for p1, p2 in pairs])
cl_dm_t = np.array([ccl.angular_cl(cosmo, tr_d[p1], tr_l[p2], larr, p_of_k_a=pk2d_dm)
                    for p1, p2 in pairs])
cl_mm_t = np.array([ccl.angular_cl(cosmo, tr_l[p1], tr_l[p2], larr, p_of_k_a=pk2d_mm)
                    for p1, p2 in pairs])

Note again the change pk2d_mm -> pk2d_dd in the first line.
I also assumed here that the angular power spectrum coming from the WeakLensingTracer tr_l is the E-mode shear power spectrum; this is why I selected before the E mode for everything. Is it right?

Other computations

Different size

We tried for a larger simulation to check if a different behavior could tell us what is happening. The quantities that were changed are:

• z_max: 0.4 -> 1.5
• r_smooth: 5. -> 2.0
• n_grid: 512 -> 4096

These quantities are similar to the ones used in the LSST simulations (except for the z_max that was 2.5 for the LSST case). The bias model is changed to behave as a one_like array modifying the Bz_test.txt file accordingly.

This simulation was run with the New shear method and the result is the following:

The output was downsampled evenly across the sky and redshifts to save memory (10% downsampling). The results are quite confusing and I don't think they give much information. However, maybe they are showing something relevant that we are not understanding.

Thanks!

[fjaviersanchez]

Hi @cramirezpe! Thanks a lot for the detailed analysis. I think that you are right in that you should use pk2d_dd rather than pk2d_mm as you mention (since then you set the bias=1 in the prediction so that should be folded into the pk). I'm pretty surprised with the last results that you show in the post above. Did you change this https://github.com/damonge/CoLoRe/blob/master/examples/cl_test/cl_test_ccl.py#L58 to accommodate the larger redshift range now? (so instead of range=[0, 0.5] put [0, 1.6] -- and maybe you also want to change nz_h and increase it).

[cramirezpe]

Hi @fjaviersanchez ,

You are totally right, that's a silly error from my side. I corrected it while keeping the value of nz_h to 50.

Now the results look more sensible:

I'm playing with other values of nz_h and also setting the z_split at other regions. But I'm getting weird things so I save them for another update.

Thanks for your help!

[fjaviersanchez]

Thanks a lot! I'm still puzzled by the large difference between in the panels on the right. Yes, please keep us updated :)

[cramirezpe]

Hi @damonge , @fjaviersanchez

I continued the analysis with the big test adding more features. I use the following parameters:

• zbins:
  • 0: [0, 0.5)
  • 1: [0.5, 1)
  • 2: [1, inf)
• range=[0,2.5]
• nz_h = 100
• downsampling (to save computing time): 1 % (I expect it to only add noise).

I also included error bars estimated from the standard deviation in the rebinning (is there a way to obtain errors from pyccl?).

But the problem with the mm and dm plots can be seen clearly when going from the logarithmic plot to the linear one:

More specifically when computing the ratio between the data and the theoretical prediction:

There is a bias 2 affecting the lensing part (therefore 2 for the dm part and 4 for the mm part).

It can be corrected by hardcoding:

cl_mm_d = cl_mm_d/4
cl_md_d = cl_md_d/2

and then everything looks reasonable:

The problem is to know where is this bias coming from. I've change the input bias from CoLoRe and the ratio stills the same, so I'm quite sure the problem is not coming from there.

Summarizing we have two questions:

1. Is it possible to obtain errors from the CCL software (I was not able to find it in the documentation).
2. We see a bias 2 affecting the lensing part. Do you have an idea of where it is coming from?

[andreufont]

Hi @damonge - we had a factor of 2 issue in CoLoRe's lensing a few month ago, might this be related?

[fjaviersanchez]

Hi @cramirezpe! This looks beautiful! Great agreement (modulo the typical factors of 2 that we have to figure out).

1. CCL doesn't know about covariances but you can probably just use the mode-counting error-bars: Δ Cℓ = sqrt(2 / (2ℓ + 1)) *Cℓ
2. I'm with Andreu, it might be one of those factors of 2 that we were seeing a few months ago. Either that or some convention mismatch between the shear from CCL and the measurements from anafast (there are always these factors of 2 floating around with different conventions).

[andreufont]

Hi @cramirezpe - Could you redo the measurement using a larger fraction of galaxies? 10% should be enough for now, to reduce a bit the noise in the plots. Is the limitation speed, or memory?

It would also be good to try to use the mode-counting errorbar that @fjaviersanchez suggested, instead of the ones you have now.

@damonge - Would you be able to dig into the code to chase the missing factor of 2? Is there a test that we could run to help you identify where it is?

[cramirezpe]

Hi @andreufont.

I've cut the number of galaxies two times.

• In the CoLoRe simulation itself: Here I applied a downsampling of 10% directly to the nz file. (The original is the one coming from the LSST-like simulation). This number can be increased if I increase the resources. However for the LSST comparison between Old and New lensing I used at maximum this 10%.
• In the script to compute the plots for this thread: Here is just a matter of time. I run it once without downsampling and this are the results. The reduction in noise is low, maybe too low for a sample 100 times larger.

In this plot the error bars are computed following Javier's suggestion.

[andreufont]

Thinking about it, the equation from @fjaviersanchez is probably only valid in the absence of shot noise, i.e., when we are dominated by cosmic variance. Is this true, Javi? The density of galaxies does not appear in his equation.

They clearly underpredict the scatter in some plots, like in d1-d2, don't you think?

@cramirezpe , could you redo the plot using again your old errorbars? Also, do you mind correcting for the factors of 2,4 that we discussed?

[fjaviersanchez]

Yes, that's right you have to add the shot-noise term, that you already calculated here: https://github.com/damonge/CoLoRe/blob/master/examples/cl_test/cl_test_ccl.py#L131. So the total error would be sqrt((Delta C_ell)^2 + n_l)

[andreufont]

mmm... the cross-terms should not have shot-noise contributions, right? How does the low density of tracers affect the errorbars in a cross-correlation of galaxy densities at different bins? And how does it affect the lensing correlations, since these should not have shot-noise perse either, right? Sorry if these are basic questions, I'm not used to thinking about these tomographic measurements.

[cramirezpe]

Here are the plots with the error computed as the last update from Javier.

But as Andreu says, shootnoise is only applied in d0d0 d1d1 and d2d2 (and the effect is tiny). If it should be included also in the mm and dm part I should make some changes to the script.

[andreufont]

It looks to me that the shot-noise should not affect the signal in cross-bins like d1-d2, or d0-d2, but it should affect its errorbars...

We could always get the errorbars by looking at the scatter in 10 boxes or so...

[damonge]

Hi all,

sorry for the delay, you caught me coming back from holidays and catching up with things.

1. Regarding the factor of 2: I think this is fine, let me check with the previous plotting scripts, but this is just the convention of going from elipticities to shear (there's an equation in Dodleson). I'll get back to you.

2. Regarding errors: this is the formula you should use for the covariance of the power spectrum between tracers (ab) and the power spectrum of tracers (cd):
   
   so, if you want the error bar for C_ell^{ab}:
   

In this formula, the auto-correlations should contain shot noise (and that's how even cross-correlations are affected by shot noise in the uncertainties).