sslyze results missing for some domains

[PaulSD]

Example: pivcheck1.max.gov and pivcheck2.max.gov are identical (those are two names for the same server/app), but pulse currently shows sslyze data for pivcheck1 but not pivcheck2.

Any idea what is going on there?

[konklone]

Here's what we saved from our SSLyze scan results for those two domains last night:

• https://s3-us-gov-west-1.amazonaws.com/cg-4adefb86-dadb-4ecf-be3e-f1c7b4f6d084/live/subdomains/scan/cache/sslyze/pivcheck1.max.gov.json
• https://s3-us-gov-west-1.amazonaws.com/cg-4adefb86-dadb-4ecf-be3e-f1c7b4f6d084/live/subdomains/scan/cache/sslyze/pivcheck2.max.gov.json

It looks like, for whatever reason, SSLyze was able to connect to pivcheck1 but not pivcheck2. When I try, just now, to scan pivcheck2 with SSLyze, it works fine.

I wonder if perhaps your firewalls dropped our connection to pivcheck2 because it was happening shortly after or concurrently with our connections to pivcheck1 from the same source. We run these in a pretty high-density manner (900 concurrent Lambda executions!) without rate limiting, relying on the general federated/disparate nature of federal infrastructure to avoid DDoSing servers. And the SSLyze negotiations for each scan are fairly intense, because it tries out each protocol version in turn.

We've seen the same behavior exhibited from some other federal services, and I suspect it's the same dynamic.

For the record - the results I got from sslyze'ing pivcheck2 just now:

{
  "certs": {
    "any_sha1_constructed": false,
    "any_sha1_served": false,
    "constructed_issuer": "Entrust Root Certification Authority - G2",
    "key_length": 2048,
    "key_type": "RSA",
    "leaf_signature": "sha256",
    "not_after": "2020-03-20T21:42:36",
    "not_before": "2017-11-13T21:12:38",
    "served_issuer": "Entrust Root Certification Authority"
  },
  "config": {
    "all_dhe": false,
    "all_rc4": false,
    "any_3des": false,
    "any_dhe": true,
    "any_rc4": false,
    "weakest_dh": 256
  },
  "errors": "",
  "hostname": "pivcheck2.max.gov",
  "protocols": {
    "sslv2": false,
    "sslv3": false,
    "tlsv1.0": true,
    "tlsv1.1": false,
    "tlsv1.2": true
  }
}

[konklone]

@PaulSD In last night's run, both pivcheck1 and pivcheck2 returned sslyze results. I didn't make any changes on this end, did you?

In an near term update, SSLyze is dropping the threads for cipher evaluation from 15 to 10: nabla-c0d3/sslyze@6fa2989 Which could help reduce this sort of churn (at the cost of slightly slower scans), if it is about just getting dropped for having too many open connections in too short a period of time.

[PaulSD]

I haven't changed anything...

[konklone]

I'm going to start capturing some more information going forward here:
#762

Since it's intermittent, I'll leave this open for a bit, and treat this as a reference issue for seeing if I can get more data about intermittent sslyze failures in general, whether they affect pivcheck2.max.gov or not.

[PaulSD]

Interesting to note:
On Friday, pivcheck1 and pivcheck3 had results but pivcheck2 did not. Today, pivcheck1 and pivcheck2 have results, but pivcheck3 does not. All three of those are the same.

I see no evidence on my side that this traffic tripped any rate limits.

[PaulSD]

Has #762 been deployed? Can you point me at the output from last night?

[konklone]

Yes it is -- here are the latest sslyze scan results:

https://s3-us-gov-west-1.amazonaws.com/cg-4adefb86-dadb-4ecf-be3e-f1c7b4f6d084/live/subdomains/scan/results/sslyze.csv

[PaulSD]

"Connection timeout while talking to Lambda. Scan returned nothing."

There are relatively few of these in the first 10000 or so scans, then tons of them for the next 5000 scans, then a modest number in the last 8000 scans. Only one of my domains failed last night, and it failed with this error.

Perhaps you are hitting rate limits between the system that is coordinating the scans and AWS?