Line-of-Sight Structure-Enhanced Ray-Tracing Halos Rendering for External Convergence and Shear by trivialTZ · Pull Request #64 · LSST-strong-lensing/slsim

trivialTZ · 2023-09-14T03:04:13Z

Line-of-Sight Structure-Enhanced Ray-Tracing Halos Rendering for External Convergence and Shear

Description:

This pull request introduces an enhancement to the ray-tracing halos rendering mechanism. It incorporates both large-scale structures using GLASS and small-scale structures via the halo mass function.

Key Features:

Line-of-Sight Structure Rendering: This implementation utilizes the SkyPy halo module for the line-of-sight structure. It's worth noting that the currently used SkyPy halo module has not been merged into the main SkyPy branch, hence we're temporarily making use of Skypy_halos_duplicate.
Ray-Tracing Computations: For a given sky area, this feature generates halos within the corresponding light cone. It then employs ray-tracing techniques to compute the distribution of convergence ( $\kappa_{ext}$ ) and shear ( $\gamma_{tot}$ ).
Negative Mass Sheet Option: Users have the flexibility to generate a negative mass sheet which can be used for kappa correction.
Demonstrative Notebook: A notebook has been added to provide a hands-on demonstration of this feature.

Next Steps:

Testing and Documentation:
- Test Functions: Enhance and finalize the test functions to ensure the feature's robustness.
- Notebook: Refine the notebook to provide clearer demonstrations and examples.
- Documentation: Improve the documentation to ensure comprehensive understanding and ease of use for the users.
Incorporate $\kappa_{ext}$ Correction:
- We aim to introduce a correction factor for $\kappa_{ext}$:

$$ 1 - \kappa_{ext} = \frac{(1-\kappa_d)(1-\kappa_s)}{1-\kappa_{ds}} $$

This will ensure a more accurate representation and calculation in our rendering mechanism.

Acknowledgements:

Guidance and assistance from: @sibirrer .
Feedback and suggestions from: @ajshajib .
Additional help from: @nkhadka21 .

We kindly request a review of this feature, and are open to further feedback and suggestions.

Some bugs still need to be fixed

with documentation, jupyter sample, test function.

Create a utility function that handles default parameter and then modify the other functions to use this new utility function.

1. update Jupyter notebook with the computing gamma and kappa 2. change yml file to make it run quicker for debug(will change back) 3. add halos_len.py with Class HalosLens() in it and def concentration_from_mass() in it.

1. **halos_lens.py**: - Introduced `get_kappa_gamma_distib()` for computing the kappa and gamma_tot values from the list. - Implemented running time tracking for performance analysis. 2. **halos.py**: - Added running time metrics for enhanced performance tracking. 3. **Notebook Renaming**: - Renamed `get_halos.ipynb` to `halos_rendering.ipynb`. 4. **halos_rendering.ipynb Enhancements**: - Added `kappa_gamma_distribution` for visualizing the joint distribution of kappa-gamma_tot. - Introduced visualizations to showcase the distributions of the modified kappa and gamma values from halos using histograms. - Incorporated KDE functionality for generating random samples of `kappa_halos` and `gamma_halos`. 5. **Data Files**: - Added `kgdata.npy`: Weak-lensing maps sourced from GLASS with n=128 (z=5). (better try other way to upload) - Introduced `z5data.npy`: Weak-lensing maps from GLASS with n=2048 (z=5). 6. **halos_plus_glass.py**: - This script fuses several functions to produce the joint distribution of kappa and gamma values leveraging both halo and GLASS data. a. **Glass Data & Sample Generation**: Utilized `read_glass_data` to extract kappa and gamma values from the Glass data file. Implemented `skyarea_form_n` to deduce the corresponding sky area for each pixel. Employed `generate_samples_from_glass` to fit a Gaussian KDE to the joint distribution and generate a random sample. b. **Halo Data & Sample Generation**: Applied a Gaussian KDE to derive the joint distribution and presented a hexbin plot visualization. c. **Combining Halo and Glass Data**: Invoked `generate_halos_multiple_times` to repeatedly generate kappa and gamma values from halos. Subsequently, merged the halo and Glass data samples with the `halos_plus_glass` function. 7. **glass_convergencen_shear.ipynb**: - A dedicated notebook to import data from GLASS and facilitate its visualization. 8. **glass_plus_halos_rendering.ipynb**: - This notebook illustrates the process of deriving kappa and gamma values from both halo and GLASS sources. It also demonstrates the method to visualize the combined kappa and gamma values' joint distribution. ---------------------------------------------------------------------- If there are no issues with these codes and the underlying physics, I will generate weak lensing maps for add different values of z and add test function next.

1. debug: including debug about the position sample (of disk); halos mass's resolution and others. 2. improve: add/improve some methods and improve the docstring. 3. The results should be more realistic now.

1.make a duplicate of Skypy halos branch and make a read.me for credits statement 2. Optimize the project. （mulitprocess & numpy vector operator) (still some room for improve)

(optimized code)

For passing the test

@sibirrer

1. add test function for Halos 2. rename file name halos ---> Halos 3. move the jupyter notebooks to the notebook folder 4. debug of halos_plus_glass.py () 5. give proper attribution for the code copy from Skypy (as once it is in the main branch or release version of SkyPy, that this code becomes deprecated) @sibirrer

For tqdm

1. add def run_halos_without_kde_by_multiprocessing() as it can help improve halos rendering when n_iterations is large. 2. debug

1.Add test for new method 2.delete unnecessary duplicate from Skypy 3.move pipeline test file 4.minor change on the markdown of a jupyter notebook

Test function and documentation are on the way.

still need further look of what went wrong.

1.deg2_to_cone_angle debug 2.add a test function

1.add test for cone_radius_angle_to_physical_area() 2. will update documentation and complete test functions in the furture.

1, Make the negative mass in the mass sheet correction uniformly distributed within the light cone according to the halo mass function, rather than keeping it at the same redshift as the generated halos. 2,Address the issue in the code when the number of generated halos is too small, causing N to be equal to zero.

trivialTZ · 2024-04-24T06:12:38Z

@sibirrer good for code review

and and more test function

trivialTZ · 2024-05-25T09:45:54Z

@sibirrer the notebook is updated, good for further reviews.

sibirrer · 2024-05-28T21:11:19Z

@@ -2,8 +2,13 @@
 "cells": [


can you try not to submit this notebook as it adds to disk space

Reply via ReviewNB

sibirrer

Looks great @trivialTZ! I have one request to remove one notebook from the PR and then we can merge it

sibirrer · 2024-06-06T14:04:42Z

@nkhadka21 from my side, TZ's PR is ready to be merged (modulo the minor conflict that will be resolved). Can you look through the files and see whether there are any issues or requests? Thank you!

trivialTZ · 2024-06-09T11:34:39Z

@nkhadka21 this pull request is good for reviewing, thanks!

nkhadka21

Hi @trivialTZ , thank you very much for this wonderful work! Code is very well documented. I have a very minor comment that you can address. It looks good to me and I approve this PR.

nkhadka21 · 2024-06-10T13:13:42Z

+
+        path = os.path.dirname(slsim.__file__)
+        module_path, _ = os.path.split(path)
+        if skypy_config is None:


You have given default sky area below but It would be good to mention the default sky area here.

trivialTZ and others added 30 commits July 19, 2023 02:33

pipeline for halos

75f13fb

Some bugs still need to be fixed

Add Halos pipeline

965ff80

with documentation, jupyter sample, test function.

Create .name

9574b9d

Update halos.py

5b07bcb

Create a utility function that handles default parameter and then modify the other functions to use this new utility function.

Add Class HalosLens() foe computering lensing effect for halos

5f48fa5

1. update Jupyter notebook with the computing gamma and kappa 2. change yml file to make it run quicker for debug(will change back) 3. add halos_len.py with Class HalosLens() in it and def concentration_from_mass() in it.

Debug and improve code

9e2cda8

1. debug: including debug about the position sample (of disk); halos mass's resolution and others. 2. improve: add/improve some methods and improve the docstring. 3. The results should be more realistic now.

Short-term solve the SkyPy issue and optimize

6198272

1.make a duplicate of Skypy halos branch and make a read.me for credits statement 2. Optimize the project. （mulitprocess & numpy vector operator) (still some room for improve)

The missing upload code from last commit

e247e7b

(optimized code)

Make it past test

b78f638

Update abundance_matching.py

ef5ae32

For passing the test

update for passing test

de1c8a7

Update halo.yml for passing the test

980e166

Update halo.yml

45a0963

Update test_halos_pipeline.py

9a56df8

Delete glass_convergence_shear_map.ipynb

2fc7e9c

Delete halos_rendering.ipynb

00804ed

Delete glass_plus_halos_rendering.ipynb

e9006a8

Update test_requirements.txt

a1cc715

For tqdm

Improve code and debug

9d6db8f

1. add def run_halos_without_kde_by_multiprocessing() as it can help improve halos rendering when n_iterations is large. 2. debug

Add test for new method & delete unnecessary duplicate from Skypy

7b651b6

1.Add test for new method 2.delete unnecessary duplicate from Skypy 3.move pipeline test file 4.minor change on the markdown of a jupyter notebook

Add mass sheet correction

991a906

Test function and documentation are on the way.

debug on calculating mass expectation

fd4e9da

still need further look of what went wrong.

Incomplete debug

de80048

1.deg2_to_cone_angle debug 2.add a test function

debug for the mass-sheet correction

a7b0183

1.add test for cone_radius_angle_to_physical_area() 2. will update documentation and complete test functions in the furture.

Merge branch 'main' into main

64cc999

Merge branch 'sibirrer:main' into main

f0de76e

Merge branch 'sibirrer:main' into main

a98a321

meet codecov

376a2b2

trivialTZ added 4 commits April 24, 2024 11:34

Update external_convergence_shear_distributions_tutorial.ipynb

8a99c7a

pre commit

570f435

lens_model_list = []

efba31a

minor debug of z_source

cf11bce

and and more test function

sibirrer reviewed Apr 30, 2024

View reviewed changes

trivialTZ added 4 commits May 24, 2024 00:30

Create test_coolest_slsim_interface.py

8dd3f98

Merge branch 'LSST-strong-lensing:main' into main

0b9f030

Added more explanations for notebooks

65497db

pre-commit

5c76586

sibirrer reviewed May 28, 2024

View reviewed changes

sibirrer approved these changes May 28, 2024

View reviewed changes

Create galaxy_galaxy_lensing_tutorial.ipynb

cbf632f

sibirrer requested a review from nkhadka21 June 6, 2024 14:03

trivialTZ added 9 commits June 8, 2024 16:36

try Resolving conflicts

cda553a

try Resolving conflicts

3f6448b

Resolving conflicts

0c0dc82

Merge branch 'LSST-strong-lensing:main' into main

5a5752d

uodatting lens and lens_pop

5e3b746

Merge branch 'main' of https://github.com/trivialTZ/slsim

e7420c3

Update lens.py

28eeaef

independent from hmf

c0aaf02

pre-commit

30686fc

nkhadka21 approved these changes Jun 10, 2024

View reviewed changes

Update halos_pipeline.py

ebe6dc1

nkhadka21 merged commit 68adf90 into LSST-strong-lensing:main Jun 10, 2024

Navigation Menu

Line-of-Sight Structure-Enhanced Ray-Tracing Halos Rendering for External Convergence and Shear - #64

Line-of-Sight Structure-Enhanced Ray-Tracing Halos Rendering for External Convergence and Shear#64
nkhadka21 merged 179 commits into
LSST-strong-lensing:mainfrom
trivialTZ:main

trivialTZ commented Sep 14, 2023

Uh oh!

trivialTZ commented Apr 24, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trivialTZ commented May 25, 2024

Uh oh!

sibirrer May 28, 2024 •

edited

Loading

Uh oh!

sibirrer left a comment

Uh oh!

sibirrer commented Jun 6, 2024

Uh oh!

trivialTZ commented Jun 9, 2024

Uh oh!

nkhadka21 left a comment

Uh oh!

nkhadka21 Jun 10, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

trivialTZ commented Sep 14, 2023