PSNR metric returns `all: 1` resulting in incorrect results

[vishaalagartha]

When I run compare -verbose -metric psnr test.png test.png output.png (comparison on the same file)` this is the output I get:

test.png PNG 3456x2234 3456x2234+0+0 8-bit TrueColorAlpha sRGB 1.03965MiB 0.060u 0:00.063
test.png PNG 3456x2234 3456x2234+0+0 8-bit TrueColorAlpha sRGB 1.03965MiB 0.050u 0:00.050
Image: test.png
  Channel distortion: PSNR
    red: 1
    green: 1
    blue: 1
    alpha: 0
    all: 1
writing raw profile: type=icc, length=4064
writing raw profile: type=xmp, length=450
test.png=>output.png PNG 3456x2234 8-bit sRGB 1.03965MiB 0.640u 0:00.417

It appears that imagemagick can also output 1 if the files are identical? Hence, shouldn't this line (https://github.com/bgeron/diff-pdf-visually/blob/16513817e5c116507b3e08e4cdca72b1d53e7463/diff_pdf_visually/diff.py#L79C1-L80C1)
all_num = INFINITY if (all_str == "0" or all_str == "1.#INF") else float(all_str)
be changed to
all_num = INFINITY if (all_str == "0" or all_str == "1.#INF" or all_str == "1") else float(all_str)
?

Thanks in advance! I manually changed this in my site_packages and things to be working fine now.

ImageMagick version: 7.1.2-0 Q16-HDRI aarch64 23234
diff_pdf_visually version: 1.3.1

[xljones]

Had the same issue. Setting the threshold explicitly to 1.00 in your call to pdf_similar() should also resolve the issue, as the DEFAULT_THRESHOLD of 100 is of course unreachable! This should allow an ImageMagick response of Inf or 1 to be valid as a perfect match.

[xljones]

@bgeron would you be open to a PR to fix this? Will take a look later this week 🙌

[bgeron]

Hey @xljones, for sure! Sorry I let this ticket slip when my life was being busy. I should be able to respond pretty quickly, in the unlikely case it does slip off my radar please do nag me about it.

Thanks for looking into this!

[Younith]

Had the same issue, the solution proposed above fixed it!

Although, switching to a different metric also worked, so it might be worth exploring an option to specify the metric instead of just PSNR.

[bgeron]

Looking into this, tests are failing but this seems related to ImageMagick instead. Expect a release with a fix tomorrow evening.

[bgeron]

Looked more into this, and the problem is unfortunately much bigger. I will try to get to this latest at the weekend. edit: making progress but didn't end up having enough time.

@vishaalagartha, with ImageMagick 7.1.2 (August 2025) you are running a new version. I assume @Younith is too.

Since ImageMagick 7.1.44 (March 2025), the reported PSNR values are completely opposite.

• Old ImageMagick: same images = PSNR infinity, bit different = PSNR 30dB, very different = PSNR 5db. More different is smaller PSNR.
• Modern ImageMagick: same images = PSNR 0, bit different = PSNR 0,000000001, very different = PSNR 0,1. More different is larger PSNR.

(I'm just making up some numbers for the example)

You can imagine the threshold computation is completely screwed up now. In the past, we reported "different" when PSNR < threshold. Now we should report "different" when PSNR > threshold.

The problem is even bigger: my local ImageMagick 7.1.1-47 reports PSNR=1 when comparing a black and a white image. When comparing the same image twice, I get PSNR=0. It would not really be cool if my tool says that a fully black and a fully white PDF image are the same. So I don't really know what's going on with ImageMagick 7.1.2-0. Maybe they inverted the logic a second time? Any hints will be most helpful.

@vishaalagartha Is your test.png completely transparent by any chance? If you have time to investigate, could you compare -verbose -metric PSNR white.png white.png diff.png and compare -verbose -metric PSNR white.png black.png diff.png (images attached) and report the results? I'm getting all: 0 when comparing black/black and white/white, and all: 1 when comparing black/white and white/black.

@xljones , do you also get all: 1 like @vishaalagartha when comparing an image with itself, or do you get all: 0 like me? And which version of ImageMagick do you have (compare -version)?

black.png
white.png

To be honest this needs a bunch of changes in multiple places. I have an early draft but it looks like a lot of work is needed to catch up with the last 9 months of ImageMagick.

Thanks for any info you can help with!

[bgeron]

I did some research, and different versions of ImageMagick seem to have these scales:

• 7.1.1-0…7.1.1-45: 0 = same → ∞ (-0) = completely different
• 7.1.1-46: broken: 0 can mean the same or completely different
• 7.1.1-47: 0 = same → 1 = completely different
• 7.1.2-0…7.1.2-8: 1 = same → 0 = completely different

I'm making a new version that detects the version of ImageMagick installed, and behaves right for that version.

[bgeron]

update: I calculated some metrics for examples between v7.1.1-45, v7.1.1-47, and v7.1.2-8, and I figured out some formula to make the metrics comparable, but it also seems that it's not simply a formula but an algorithm change. For instance, it seems like v7.1.1-45 doesn't care as much about layout shifts as higher versions. However, we can still bring the numbers on a similar scale to make them somewhat comparable.

[bgeron]

I hope I fixed the issue in new prerelease version 2.0.0.a1. Try it with:

uvx diff-pdf-visually@2.0.0a1

or using your package manager of choice.

This version detects the installed version of ImageMagick, and converts all metrics to the modern range: from 0 to 1, higher is more similar.

@vishaalagartha @xljones @Younith If you have time, could you try some of your use cases with this version, see if things are working or if you see anything surprising? I would like to release this as a new major version, but there's a lot of new code and I'd like to avoid accidentally breaking anything for others.

[xljones]

Hey @bgeron sorry for the slow response here. Ultimately I returned back to the original version of ImageMagick i was using before (for unrelated reasons to this!), but giving this a test out for you anyway using that version as a data point. tl;dr I think there may be an issues here for ImageMagick 7.0.11-14 the version we're currently using.

Sanity checking the versions of diff-pdf-visually

/app $ pip list | grep diff
diff-pdf-visually                2.0.0a1

and compare

/app $ compare --version
Version: ImageMagick 7.0.11-14 Q16 aarch64 2021-05-31 https://imagemagick.org
Copyright: (C) 1999-2021 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules 
Delegates (built-in): bzlib cairo fontconfig freetype gslib heic jng jpeg lcms ltdl lzma png ps rsvg tiff webp x xml zlib

Visually, I can see that these files are identical. However, I'm seeing an assertion failure with the following logs from the call to pdf_similar:

----------------------------------------- Captured stdout call ------------------------------------------
  Temporary directory: /tmp/diffpdf-wooqjk4w
  Converting each page of the PDFs to an image...
  PDFs have same number of pages. Checking each pair of converted images...
- Page 1: significance=0.0
Lowest similarity score found = 0.0, significant?=True. The PDFs are different. The most different pages are: page 1 (sgf. 0.0).
----------------------------------------- Captured stderr call ------------------------------------------
  Running: pdftocairo -png -r 300 /tmp/pdf-renders/file_generated_under_test.pdf /tmp/diffpdf-wooqjk4w/a
  Running: pdftocairo -png -r 300 /app/expected_to_match_this.pdf /tmp/diffpdf-wooqjk4w/b
  Running: magick compare -verbose -metric PSNR /tmp/diffpdf-wooqjk4w/a-1.png /tmp/diffpdf-wooqjk4w/b-1.png /tmp/diffpdf-wooqjk4w/diff-1.png

This test is called as follows

    assert pdf_similar(
        tmp_filepath,
        golden_filepath,
        verbosity=3,
        dpi=300,
    ), (
        f"Generated PDF does not match golden for template: {template}.\n"
        f"Golden: {golden_filepath}\n"
        f"Test:   {tmp_filepath}"
    )

If i now subtly change the expected PDF and re-run those tests; I see the following:

------------------------------------------------- Captured stdout call --------------------------------------------------
  Temporary directory: /tmp/diffpdf-6205rfhf
  Converting each page of the PDFs to an image...
  PDFs have same number of pages. Checking each pair of converted images...
- Page 1: significance=0.07099449082751179
Lowest similarity score found = 0.07099449082751179, significant?=True. The PDFs are different. The most different pages are: page 1 (sgf. 0.07099449082751179).
------------------------------------------------- Captured stderr call --------------------------------------------------
  Running: pdftocairo -png -r 300 /tmp/pdf-renders/file_generated_under_test.pdf  /tmp/diffpdf-6205rfhf/a
  Running: pdftocairo -png -r 300 /app/expected_to_match_this.pdf /tmp/diffpdf-6205rfhf/b
  Running: magick compare -verbose -metric PSNR /tmp/diffpdf-6205rfhf/a-1.png /tmp/diffpdf-6205rfhf/b-1.png /tmp/diffpdf-6205rfhf/diff-1.png

So it looks like this version of ImageMagick is using 0 = perfect match, but pdf_similar doesn't seem to be accounting for this.

If I now revert back to 1.8.1, rebuild the image with a further quick sanity check of versions

/app $ pip list | grep diff
diff-pdf-visually                1.8.1

/app $ compare --version
Version: ImageMagick 7.0.11-14 Q16 aarch64 2021-05-31 https://imagemagick.org
Copyright: (C) 1999-2021 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules 
Delegates (built-in): bzlib cairo fontconfig freetype gslib heic jng jpeg lcms ltdl lzma png ps rsvg tiff webp x xml zlib

I see the following when the PDFs are subtly different

------------------------------------------------- Captured stdout call --------------------------------------------------
  Temporary directory: /tmp/diffpdf-l6k3pfob
  Converting each page of the PDFs to an image...
  PDFs have same number of pages. Checking each pair of converted images...
- Page 1: significance=17.0523
Min sig = 17.0523, significant?=True. The PDFs are different. The most different pages are: page 1 (sgf. 17.0523).
------------------------------------------------- Captured stderr call --------------------------------------------------
  Running: pdftocairo -png -r 300 /tmp/pdf-renders/file_generated_under_test.pdf  /tmp/diffpdf-l6k3pfob/b
  Running: pdftocairo -png -r 300 /app/expected_to_match_this.pdf /tmp/diffpdf-l6k3pfob/a
  Running: compare -verbose -metric PSNR /tmp/diffpdf-l6k3pfob/a-1.png /tmp/diffpdf-l6k3pfob/b-1.png /tmp/diffpdf-l6k3pfob/diff-1.png

and the following when a perfect match occurs (assertion is True, test is passing as expected)

------------------------------------------------- Captured stdout call --------------------------------------------------
  Temporary directory: /tmp/diffpdf-h2kjdndc
  Converting each page of the PDFs to an image...
  PDFs have same number of pages. Checking each pair of converted images...
- Page 1: significance=inf
Min sig = inf, significant?=False. The PDFs are the same.
------------------------------------------------- Captured stderr call --------------------------------------------------
  Running: pdftocairo -png -r 300 /tmp/pdf-renders/file_generated_under_test.pdf /tmp/diffpdf-h2kjdndc/a
  Running: pdftocairo -png -r 300 /app/expected_to_match_this.pdf /tmp/diffpdf-h2kjdndc/b
  Running: compare -version
  Running: compare -verbose -metric PSNR /tmp/diffpdf-h2kjdndc/a-1.png /tmp/diffpdf-h2kjdndc/b-1.png /tmp/diffpdf-h2kjdndc/diff-1.png