Add multi-resolution TS map fitting

[Yong2Sheng]

This PR adds the method to do multi-resolution (also called multi-order coverage, MOC) TS map fitting.

Changes to FastTSMap

1. FastTSMap is the parent class of MOCTSMap

2. I need to make several changes to FastTSMap to make it work with MOCTSMap:

   • Add pixel_idx parameter for fast_ts_fit.
     • This parameter is the index of the pixels fitted by FastTSMap.
     • This is added since we only fit a portion of the multi-resolution map. So I need to record the indices of the fitted pixels to match the likelihood values.
     • Note it's index of the pixels in a multi-resolution map, not the unique numbers of the pixels.
   • Add a new function zip_comp to generate the inputs for multiprocessing.
     • I used itertools.product before. However, it will cause duplicated fittings in MOCTSMap.
     • So I use zip_comp to replace it.

New module MOCTSMap.

1. This is the new module responsible for multi-resolution TS map fitting.
2. It checks the likelihood during each iteration, and only fit the child pixels of the mother pixels that meets the criterion.

Changes to the TS map tutorial

1. I renamed the tutorial notebook file name.
2. I added Example 4, which fits the Crab with the multi-resolution map.

Please let me know if you have any questions and suggestions. Thank you!

[codecov]

Codecov Report

Attention: Patch coverage is 31.70732% with 84 lines in your changes missing coverage. Please review.

Project coverage is 79.88%. Comparing base (f25cc32) to head (c5e3419).
Report is 359 commits behind head on develop.

Files with missing lines	Patch %	Lines
cosipy/ts_map/moc_ts_fit.py	22.33%	80 Missing ⚠️
cosipy/ts_map/fast_ts_fit.py	76.47%	4 Missing ⚠️

❌ Your patch status has failed because the patch coverage (31.70%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines	Coverage Δ
cosipy/__init__.py	100.00% <100.00%> (ø)
cosipy/ts_map/__init__.py	100.00% <100.00%> (ø)
cosipy/ts_map/fast_ts_fit.py	55.78% <76.47%> (+1.55%)	⬆️
cosipy/ts_map/moc_ts_fit.py	22.33% <22.33%> (ø)

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ldigesu]

Hello Yong, I looked at this tutorial, including the MOC-TS map. It works fine for me, I have only very minor suggestions from the point of view of the user. Apart from those, I think I can complete the review.

i) MOC-TS map: maybe add a markdown reminding the user of the meaning of the parameters max_moc_order and top_number. I think also it would be useful to print on the screen the delta_likelihood between the top pixels (that will continue to be fitted and higher resolution) and the others. So that the user can have a feeling of how significant is the source the the procedure is finding.

ii) Example 3, when the data are downloaded and binned. Maybe you can add another if statement checking for the existence of the .gz version of the file. The .gz file can also be unzipped on the fly using: import subprocess and subprocess.run(["gunzip", crab_unbinned_zipped_path]). The "shutils" command that you use before works only with .zip files, not with .gz. I would also a note to remind the user that the .ymal files have to be edited by hand (updating the paths) before proceeding.

iii) There are a few typos, for instance, in example 2, 1st cell: excpt>>execEpt

iv) Maybe you could consider splitting the tutorial into two notebooks e.g. one with the Crab and one with the GRB, or one with the Fast TS map and one with the MOC TS map.

[Yong2Sheng]

Hi @ldigesu,

Thank you for the review. I updated the PR based on your comments.

Add a markdown reminding the user of the meaning of the parameters max_moc_order and top_number.

I added comments to the cell to explain max_moc_order, top_number, and energy_channel parameters.

It would be useful to print on the screen the delta_likelihood between the top pixels (that will continue to be fitted and higher resolution) and the others?

I printed the top ts values.

Example 3, when the data are downloaded and binned. Maybe you can add another if statement checking for the existence of the .gz version of the file. The .gz file can also be unzipped on the fly using: import subprocess and subprocess.run(["gunzip", crab_unbinned_zipped_path]). The "shutils" command that you use before works only with .zip files, not with .gz. I would also a note to remind the user that the .ymal files have to be edited by hand (updating the paths) before proceeding.

• The notebook avoids downloading the gz files when they already exist.
• The DataIO can read and bin the gz file directly, so we don't have to unzip them.

There are a few typos, for instance, in example 2, 1st cell: excpt>>execEpt

• I didn't see excpt after searching the jupyter lab. Sorry, could you please remind me with the whole sentence with the typo?

Maybe you could consider splitting the tutorial into two notebooks e.g. one with the Crab and one with the GRB, or one with the Fast TS map and one with the MOC TS map

• I think we can put them together because they are the same process, so we can reuse the codes without explaining them again in another notebook.
• What do you think @israelmcmc ?

---

• I also updated the orientation-related cells so that they are compatible with the orientation with spacecraft attitude information.
• I noticed a warning when fetching the files from Wasabil. I have never seen this before. Not sure if we need to fix it now since it doesn't cause any errors @israelmcmc

Run:

%%capture
crab_unbinned_path = data_dir/"Crab_DC2_3months_unbinned_data.fits.gz"

# download 3-month unbinned Crab data ~619.22 MB
if not crab_unbinned_path.exists():
    fetch_wasabi_file("COSI-SMEX/DC2/Data/Sources/Crab_DC2_3months_unbinned_data.fits.gz", crab_unbinned_path)

Warning output:

DEBUG:awscli.plugin:Importing plugin __builtin__: awscli.handlers
DEBUG:awscli.plugin:Initializing plugin __builtin__: <module 'awscli.handlers' from '/Users/shengyong/conda_envs/moc_ts_map/lib/python3.10/site-packages/awscli/handlers.py'>
DEBUG:awscli.clidriver:OrderedDict([('bucket', <awscli.arguments.CLIArgument object at 0x17a258370>), ('if-match', <awscli.arguments.CLIArgument object at 0x17a258eb0>), ('if-modified-since', <awscli.arguments.CLIArgument object at 0x17a259570>), ('if-none-match', <awscli.arguments.CLIArgument object at 0x17a25b2e0>), ('if-unmodified-since', <awscli.arguments.CLIArgument object at 0x17a259a20>), ('key', <awscli.arguments.CLIArgument object at 0x17a259210>), ('range', <awscli.arguments.CLIArgument object at 0x17a2595a0>), ('response-cache-control', <awscli.arguments.CLIArgument object at 0x17a25b2b0>), ('response-content-disposition', <awscli.arguments.CLIArgument object at 0x17a25b010>), ('response-content-encoding', <awscli.arguments.CLIArgument object at 0x17a25bd60>), ('response-content-language', <awscli.arguments.CLIArgument object at 0x17a259450>), ('response-content-type', <awscli.arguments.CLIArgument object at 0x17a25b190>), ('response-expires', <awscli.arguments.CLIArgument object at 0x17a259690>), ('version-id', <awscli.arguments.CLIArgument object at 0x17a2599f0>), ('sse-customer-algorithm', <awscli.arguments.CLIArgument object at 0x17a25b760>), ('sse-customer-key', <awscli.arguments.CLIArgument object at 0x17a25a1a0>), ('sse-customer-key-md5', <awscli.arguments.CLIArgument object at 0x17a25ae30>), ('request-payer', <awscli.arguments.CLIArgument object at 0x17a25add0>), ('part-number', <awscli.arguments.CLIArgument object at 0x17a25abc0>), ('expected-bucket-owner', <awscli.arguments.CLIArgument object at 0x17a25a800>), ('checksum-mode', <awscli.arguments.CLIArgument object at 0x17a258d90>)])
DEBUG:awscli.arguments:Unpacked value of 'cosi-pipeline-public' for parameter "bucket": 'cosi-pipeline-public'
DEBUG:awscli.arguments:Unpacked value of 'COSI-SMEX/DC2/Data/Sources/Crab_DC2_3months_unbinned_data.fits.gz' for parameter "key": 'COSI-SMEX/DC2/Data/Sources/Crab_DC2_3months_unbinned_data.fits.gz'
DEBUG:awscli.formatter:RequestId: E02D97B6F5DB2710:B

[israelmcmtest]

@Yong2Sheng I'm going back to this PR. I haven't finished, but here are a few things to start with:

• Can you please? add units test to your code to get more coverage? Speacially for moc_ts_fit.py
• Since you changed the tutorial name, can you please also update the index?
• Related, can you please also delete the previous tutorial file?
• About @ldigesu of splitting the notebook: I think it's fine for now keep it as one notebook, but I think it's a good suggestion and we should do it later. I think the actual tutorials (the one listed as such in the documentation) should be simple and just show how to get started, not every single functionality should be present. There's another section called "other examples" where we can put the rest.
• About the orientation and fetch_wasabi warning, this should be fixed with recent changes to develop. I'll give it a try at modifying your notebook to be compatible.
• I'll also try to rebase your change onto v0.3.x so we can make another release.

[israelmcmtest]

@Yong2Sheng It seems you did delete the old nb, but somehow it came back. Maybe during the sync with develop. Anyway, nevermind, I'll handle that during the rebase and adjustment to your nb based on recent changes in #334

[israelmcmc]

@Yong2Sheng I made another branch rebasing onto v0.3.x:
https://github.com/israelmcmc/cosipy/tree/PR248_rebased_onto_v0.3.x

This should have all of your commits. Please double check that it still looks good to you? If so, please branch out of this branch and open a new PR with v0.3.x as target.

I also made this other (auxiliary & temporary) branch where I merged it with the current develop. I verified that it's still running fine with the script described in #334
https://github.com/israelmcmc/cosipy/tree/PR248_rebased_onto_v0.3.x_merge_develop

I think this works well and (the rebased version) can be merged after:

• Adding the unit tests described above.
• I'm thinking it would be better to change the tutorial name to the previous one Parallel_TS_map_computation.ipynb. Otherwise we'll have to change a bunch of places since we already released this with another name. What do you think? We can change it later in develop.

[israelmcmc]

Closing and opening this to run the tests again.

[Yong2Sheng]

Hi @israelmcmc, thank you for reviewing it! I will address the comments this week.

[israelmcmc]

Thanks @Yong2Sheng.

Btw, it seems that kicking the tests worked. Everything is passing now, except the coverage, as expected.

[Yong2Sheng]

Hi @israelmcmc, I opened a new #350, could you please take a look?

[israelmcmc]

Thanks @Yong2Sheng! I'm closing this in favor of #350. I'll take a look.

[Yong2Sheng]

Thank you @israelmcmc !