Issue Template Matching Nucleosomes

[ashishsdalvi]

Hi pyTME developers,

I am having an issue with template matching nucleosomes using pyTME. I have a denoised and aligned tomogram as well as a template for a nucleosome (mrc file) that I ran a low pass filter over. The template was originally 700^3 pixels but I made sure to use the mask tool to reduce it to 200^3 pixels. It ran but the result was that it template matched things other than the nucleosomes (such as the nuclear membrane). Do you have any idea how I can fix this? The top image is the pyTME output and the bottom image is the original tomogram. The pixel size of the original tomogram is 3 angstrom and template is 40 angstrom after running the filter. Would this make a difference as well? Thank you for your help!

[maurerv]

Hi @ashishsdalvi,

When running match_template, it prints out a summary of input parameters, as shown below. Could you share this for your case?

From the images alone, it's hard to troubleshoot, but here are some additional things you could check

• Does the contrast of your template match the contrast of your tomogram? When creating templates from EMDB or PDB, you need to invert their contrast to match the tomogram contrast convention https://kosinskilab.github.io/pyTME/quickstart/matching/particle_picking.html#template-and-mask-generation
• If you are using a template mask, does it fit the template?
• If you are using a non-spherical template mask, you should use --score FLC

********************************************************************************
*                                 pytme v0.3.1                                 *
********************************************************************************

> Target
  - Initial Shape:       (140, 140, 140)
  - Sampling Rate:       (1.01, 1.01, 1.01)
  - Final Shape:         (140, 140, 140)

> Template
  - Initial Shape:       (140, 140, 140)
  - Sampling Rate:       (1.01, 1.01, 1.01)
  - Final Shape:         (140, 140, 140)

> Template Mask
  - Inital Shape:        (140, 140, 140)
  - Sampling Rate:       (1.01, 1.01, 1.01)
  - Final Shape:         (140, 140, 140)

--------------------------------------------------------------------------------

> Template Matching
  - Angular Sampling:      180.0 [1 rotations]
  - Center Template:       False
  - Scramble Template:     False
  - Invert Contrast:       False
  - Extend Target Edges:   False
  - Interpolation Order:   3
  - Setup Function:        <function 'tme.matching_scores.flcSphericalMask_setup'>
  - Scoring Function:      <function 'tme.matching_scores.corr_scoring'>

> Computation
  - Backend:               <class 'tme.backends.npfftw_backend.NumpyFFTWBackend'>
  - Compute Devices:       CPU [4], GPU [0]
  - Assigned Memory [MB]:  5229.0 [out of 6151.0]
  - Temporary Directory:   /var/folders/v8/4v7htwd96013v0bw23630ntc0000gp/T
  - Target Splits:         0:1, 1:1, 2:1 [N=1]

> Filters
  - Smooth Pass:           True
  - Pass Format:           sampling_rate
  - Wedge Axes:            (2, 0)
  - Reconstruction Filter: ramp

> Analyzer
  - Analyzer:              <class 'tme.analyzer.aggregation.MaxScoreOverRotations'>
  - Score Threshold:       0
  - Num Peaks:             1000
  - Min Distance:          46
  - Use Memmap:            False

--------------------------------------------------------------------------------

Distributing 1 split on 1 job each using 4 cores.
Running Template Matching. This might take a while ...

--------------------------------------------------------------------------------

Runtime real: 3.085s user: 12.339s.

[ashishsdalvi]

Hi @maurerv this is my result. I reran it and it takes a while to run so I didn't add the runtime section at the end but here is the output. Also its possible that the sampling rate is the issue so would you happen to know how to change the sampling rate of the template and mask? Is there a built in function for this in your library? Also is there a function in the library to build the mask because I built it using external tools. Thank you!

---

•                             pytme v0.3.1                                 *
  

---

/data/homezvol3/asdalvi/.conda/envs/pytme/lib/python3.11/site-packages/tme/density.py:295: UserWarning: Non standard MAPC, MAPR, MAPS, adapting data and origin.
warnings.warn("Non standard MAPC, MAPR, MAPS, adapting data and origin.")
/data/homezvol3/asdalvi/.conda/envs/pytme/bin/match_template.py:795: UserWarning: Target sampling rate is 1.0, which may indicate missing or incorrect metadata. Verify that your target file contains proper sampling rate information, as filters (CTF, BandPass) require accurate sampling rates to function correctly.
warnings.warn(
/data/homezvol3/asdalvi/.conda/envs/pytme/bin/match_template.py:806: UserWarning: Sampling rate mismatch detected: target=[1. 1. 1.] template=[0.40074998 0.40074998 0.40074998]. Proceeding with user-provided values. Make sure this is intentional.
warnings.warn(

Target

• Initial Shape: (682, 682, 232)
• Sampling Rate: (1.0, 1.0, 1.0)
• Final Shape: (682, 682, 232)

Template

• Initial Shape: (200, 200, 200)
• Sampling Rate: (0.4, 0.4, 0.4)
• Final Shape: (200, 200, 200)

Template Mask

• Inital Shape: (200, 200, 200)
• Sampling Rate: (0.4, 0.4, 0.4)
• Final Shape: (200, 200, 200)

---

Template Matching

• Angular Sampling: 8.0 [12648 rotations]
• Center Template: False
• Scramble Template: False
• Invert Contrast: False
• Extend Target Edges: False
• Interpolation Order: 3
• Setup Function: <function 'tme.matching_scores.flcSphericalMask_setup'>
• Scoring Function: <function 'tme.matching_scores.corr_scoring'>

Computation

• Backend: <class 'tme.backends.npfftw_backend.NumpyFFTWBackend'>
• Compute Devices: CPU [4], GPU [0]
• Assigned Memory [MB]: 164980.0 [out of 194094.0]
• Temporary Directory: /tmp/asdalvi/40647483
• Target Splits: 0:1, 1:1, 2:1 [N=1]

Filters

• Lowpass: 12.0
• Smooth Pass: True
• Pass Format: sampling_rate
• Wedge Axes: (2, 0)
• Reconstruction Filter: ramp

Analyzer

• Analyzer: <class 'tme.analyzer.aggregation.MaxScoreOverRotations'>
• Score Threshold: 0
• Num Peaks: 1000
• Min Distance: 66
• Use Memmap: False

---

Distributing 1 split on 1 job each using 4 cores.
Running Template Matching. This might take a while ...

[ashishsdalvi]

Also its possible that my tomogram itself is not great. You can see in the image I provided that there are clearly visible structures in the tomogram but perhaps the contrast is not great enough. Do you have some way to debug if this is the issue and if so how can I fix it. Apologies for so many questions at once and I appreciate your tool and your help!

[maurerv]

Thank you for sharing the log.

Is the sampling rate of your tomogram really 1 Angstrom voxel? From the size it looks 4x binned, so around 8-12. You can update the sampling rate information using tools like imod.

The template you pass to match_template needs to be at the same sampling rate than the target. You can follow this tutorial to see how to prepare the templates and masks with pytme https://kosinskilab.github.io/pyTME/quickstart/matching/particle_picking.html#template-and-mask-generation

[ashishsdalvi]

Hi I was told the pixel size was ~3 angstroms so is this the same as sampling rate? I will also go back change the template so the sampling rate is the same. Thank you and I will let you know how it goes!

[maurerv]

Hi,

Exactly, the sampling rate in pytme is the same as the pixel size. Just keep in mind that if the original tomogram had a pixel size of 3 angstrom and you binned that 4 times the pixel size of the output would be 12. The template you use to match on the binned tomogram would also need to be at 12

[ashishsdalvi]

Hi @maurerv I tried out some of the things you were saying. As a test I took the ribosome template from the tutorial and used preprocess.py to make it the same pixel size (18.25) as my denoised tomogram (is human so I used 80S ribosome template). I also drew a mask by hand in napari and made sure to only make it extended slightly over the boundaries of the ribosome. I made sure to set -s FLC and tried out two --lowpass values 40 and 25 (image below is 25). I did set the box size to 32 under the assumption the ribosome is 300 angstroms. However there still seem to be too many false positives. The left side of the original image (shown below in black/white) clearly has ribosomes but the right side (inside of the nucleus) is also predicted to have ribosomes after template matching which shouldn't be the case. Any ideas on what else I can try to fix this? Additionally my ultimate goal is to be able to template match for histones but I am worried I will not be able to do this if it doesn't work for ribosomes. Thank you for your help so far!

[maurerv]

Hi, you are on the right track!

I would suggest template matching on the reconstructions directly rather than the denoised ones. 18.25 is fairly coarse, I typically stay around 10, but it might work for ribosomes. Depending on the quality of your tomogram 18.25 might be too coarse for nucleosomes.

Not sure you are doing this already, but providing information on the CTF typically reduces false positives (see https://kosinskilab.github.io/pyTME/quickstart/matching/particle_picking.html#parameter-comparison). Even if you dont have per-tilt CTF estimates, providing an average for the reconstruction will go a long way.

The 80S template in the tutorial is S. pombe, using a human 80S template should work better. What kind of mask did you create? For 80S a simple sphere, does not need to be very tight, should suffice.

[ashishsdalvi]

Hi @maurerv thanks for the information. I am in the process of getting the CTF information and possibly the original unbinned tilt stack which I will try to reconstruct so I can identify smaller particles like histones. What do you think about this since 18.25 might be too coarse as you said? Also would it be a good idea to do CTF correction on these unbinned tomograms and is it possible to do CTF correction on the binned and reconstructed tomograms (what I currently have)? If so should I do phase flipping or is there something else that would work better for template matching for ribosomes and histones? Thank you so much once again!

[maurerv]

Happy to help :)

Phase-flipping is fine. You should already see an improvement for ribosomes at 18.25, then. Perhaps try to half that first to see whether it's picking up on nucleosomes, before going to lower binning.

[ashishsdalvi]

Thank you I will give it a shot once I get access to the tilt series. On another note I tried doing ctf correction like you advised but my template matching still have lots of false positives. Additionally many of the matches appear to be in one 2D plane. I am still waiting to hear back from the technician about the parameters so I used so estimates and ran the following command. I tried both inverted and non-inverted mask but both weren't great (the images below are from inverted 3D and 2D perspective in napari)

python scripts/match_template.py --target /pub/asdalvi/original_mrc/NNW_G4_42kx_ts_10_tomogram.mrc --template /pub/asdalvi/template_mrc/ribosome_resampled.mrc --template-mask /pub/asdalvi/mask_files/ribosome_mask.mrc --lowpass 25 --angular-sampling 8 --backend cupy -s FLC --defocus 30000 --amplitude-contrast 0.08 --acceleration-voltage 300 --spherical-aberration 27000000.0 --output /pub/asdalvi/results/CTF_original_ribosome_resampled_G4_10.pickle

I also did post processing which generated the point cloud shown below which further shows the matches all being in one plane. Any idea why this happens? This continues even if I filter out by high scores and if i do --mask-edges. Here is the command I ran:

python scripts/postprocess.py --input-file /pub/asdalvi/results/CTF_original_ribosome_resampled_inverted_G4_10.pickle --min-distance 10 --mask-edges --output-prefix /pub/asdalvi/results/orientations --output-format orientations

Here is pyTME output:

Parameters

• Angles Clockwise: False
• Invert Target Contrast: False
• Local Optimization: False
• Mask Edges: True
• Mask Subtomograms: False
• Min Boundary Distance: 0
• Min Distance: 10
• Min Score: 0.0
• Num Peaks: 1000
• Output Format: orientations
• Output Prefix: /pub/asdalvi/results/orientations
• Peak Caller: PeakCallerMaximumFilter
• Peak Oversampling: 1

---

Foreground Entities

• 0: /pub/asdalvi/results/CTF_original_ribosome_resampled_inverted_G4_10.pickle

Finished conversion - Now aggregating over entities.

Foreground mean 0.178, std 0.027, max 0.494.

Score statistics per entity

Entity 0: mean 0.178, std 0.027, max 0.494.

I am excited to get this to work and am open to trying anything. Thanks in advance and for the easy to read documentation.

-Ashish

[maurerv]

You are using non-denoised data now, right?

I tried both inverted and non-inverted mask

The template typically needs to be inverted, not the mask.

But my template matching still have lots of false positives.

Unfortunately, you will rarely recover only true positives with template matching, even with very clean data.

I also did post processing which generated the point cloud shown below which further shows the matches all being in one plane

It definitely looks a bit strange, but it's hard to say from the projection what is going on. Do these actually correspond to high-scoring particles or to some reconstruction artifact in your tomogram?

[ashishsdalvi]

Yes this is the non-denoised data. I misspoke so yes the template is inverted not the mask. Also you are right I think the tomogram just contains a reconstruction artifact. Tomorrow I will get access to the original tilt series so I will try again. I am optimistic this will improve the results so ill let you know how it goes. Cheers!