On Tuesday October 14th, 2014, Google announced the POODLE vulnerability for SSLv3. Engineering teams scrambled to remove support for SSLv3 on their websites, including our own system operations team.
My team's data engineering system scrapes information from retail banking sites, which we have learned were heavily impacted by the SSL fix on Tuesday. It makes sense that banks would want to keep up-to-date with the latest security issues.
When we scraped pages on Wednesday morning, a large portion of the sites we collect from came up blank in the PhantomJS browser, but without any obvious errors in our log files or in our log analysis dashboard (Kibana).
We were puzzled at this behavior, and when we went to look at the URL's that failed to produce images, it was clear that they were all SSL-enabled and responding with a recent TLS protocol. It turns out that PhantomJS by default only uses the SSLv3 protocol, and so it does not support any of the TLS protocols without additional configuration.
The solution to our problem was to set the command line flag for preferred SSL protocol to any so PhantomJS is free to pick whatever protocol is supported by the site being scraped. While it is unintuitive for PhantomJS to select the least secure protocol as the default, at least we can override it easily.
My team's data engineering system scrapes information from retail banking sites, which we have learned were heavily impacted by the SSL fix on Tuesday. It makes sense that banks would want to keep up-to-date with the latest security issues.
When we scraped pages on Wednesday morning, a large portion of the sites we collect from came up blank in the PhantomJS browser, but without any obvious errors in our log files or in our log analysis dashboard (Kibana).
We were puzzled at this behavior, and when we went to look at the URL's that failed to produce images, it was clear that they were all SSL-enabled and responding with a recent TLS protocol. It turns out that PhantomJS by default only uses the SSLv3 protocol, and so it does not support any of the TLS protocols without additional configuration.
The solution to our problem was to set the command line flag for preferred SSL protocol to any so PhantomJS is free to pick whatever protocol is supported by the site being scraped. While it is unintuitive for PhantomJS to select the least secure protocol as the default, at least we can override it easily.
Comments
Post a Comment