Greg Leib on Software

We're using Databricks as our provider for Spark execution, and we've been struggling to get the Spark Cassandra connector to work outside of the local development environment. The connector was attempting to connect to 127.0.0.1 even though we were passing the new host information into the getOrCreate(..) call. After working with Ganesh at Databricks support, we figured it out. The realization is that in Databricks, calls to getOrCreate() from a fat jar don't create a new SparkContext object. Thus, the configuration passed in gets ignored. If you want to update the Cassandra host information for the connector, you must update it after  the call to getOrCreate() instead. Add the configuration directly to the context and you'll be good to go!

On Tuesday October 14th, 2014, Google announced the POODLE vulnerability for SSLv3. Engineering teams scrambled to remove support for SSLv3 on their websites, including our own system operations team. My team's data engineering system scrapes information from retail banking sites, which we have learned were heavily impacted by the SSL fix on Tuesday. It makes sense that banks would want to keep up-to-date with the latest security issues. When we scraped pages on Wednesday morning, a large portion of the sites we collect from came up blank in the PhantomJS browser, but without any obvious errors in our log files or in our log analysis dashboard ( Kibana ). We were puzzled at this behavior, and when we went to look at the URL's that failed to produce images, it was clear that they were all SSL-enabled and responding with a recent  TLS protocol. It turns out that PhantomJS by default only uses the SSLv3 protocol, and so it does not support any of the TLS protocols without ...

I attempted to run a Spring WebMVC application, and when starting up the application complained that it didn't know how to handle the MVC namespace in my XML configuration. The project runs JDK 7 and Spring 4.0.6 using Maven as the build system. The following is my XML configuration file: <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans"        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"        xmlns:mvc="http://www.springframework.org/schema/mvc"        xsi:schemaLocation="         http://www.springframework.org/schema/beans         http://www.springframework.org/schema/beans/spring-beans.xsd         http://www.springframework.org/schema/mvc         http://www.springframework.org/schema/mvc/spring-mvc.xsd">          <mvc:annotation-driven/> ...

I just worked through an interesting bug in a front-end single page application. The system has the ability to create new items, and one of the attributes of the item is an array of ids. When I created a new item after working with an existing one, ids from the older item were showing up in the new one. Similar issues were happening in several places in the codebase for that interface. What I eventually realized is that the default attributes are passed into instantiated objects directly -- they are not deep copies. So my code was simply modifying the existing array rather than overwriting it, and as a result my old and new instances were both using the same attribute array of ids. The takeaway lesson is that complex defaults, such as arrays or objects, should be set in the instantiation method so they are distinct in distinct instances.

For a long while now, my demo transit application hasn't been picking up new articles from the web - leaving the home page looking quite stale between deployments. Earlier this week I added extra logging to capture counts of different kinds of articles, and noticed this morning that new articles were only found the first time the parsers run. After reviewing the simple logic for determining which articles are new, I noticed the cache implementation problem. I had implemented a crude cache using a hash map, but apparently forgot to write the cache expiration logic. When the parsers went to look at news listing pages, the cache was returning a stale block of content rather than fetching the new content from the internet. I was so sure I wrote that expiration logic! So things are all fixed up and I'm going to be verifying the fix over the next few days. Nothing quite beats the feeling of finding a simple explanation for a pesky bug.