Greg Leib on Software

We're using Databricks as our provider for Spark execution, and we've been struggling to get the Spark Cassandra connector to work outside of the local development environment. The connector was attempting to connect to 127.0.0.1 even though we were passing the new host information into the getOrCreate(..) call. After working with Ganesh at Databricks support, we figured it out. The realization is that in Databricks, calls to getOrCreate() from a fat jar don't create a new SparkContext object. Thus, the configuration passed in gets ignored. If you want to update the Cassandra host information for the connector, you must update it after  the call to getOrCreate() instead. Add the configuration directly to the context and you'll be good to go!

This week I've started planning for the next version of our data collection system. The key realization for me is that I do not know all the questions we will need to answer in the future. Our current focus is on specific sequences of click events, but in the future we might want to look at browser versions or behavioral patterns related to IP addresses. If we don't capture user-agent, for example, we won't be able to answer questions about browser versions. If we don't capture IP then we cannot look for patterns in IP addresses. We should store data in a way that maximizes the range of questions we can address in the future. In the past few years, the cost of storing data have continued to fall. We use AWS extensively.  Amazon S3 costs are very reasonable and guarantees a high level of availability. Also, lower compute costs and open source tools like Hadoop that process large data volumes have greatly increased our ability to extract valuable insights from data. So s...

As a new technology executive, my biggest worry is whether I am focusing on the right issues. My prior roles primarily required technical work, but they mostly involved small problems and small teams. For my first executive position, I expected to focus on organizational issues more than technical ones. Two months in, I think my expectations and role reset strategy were about right. Here are some of the challenges I've been working on so far: Hiring. I'm shifting the organization from being mostly outsourced offshore to a more even onshore/offshore model. Sourcing and assessing engineers for senior roles has been challenging. I'm committed to building a great team. This is my top priority, and so far we've filled two of the four most senior roles. Our offshore people are solid, and I'm really excited about the new people we're hiring. Process.  We need a simple and unambiguous approach for software development. I've been pushing the team to use Jira a...

I recently received this marketing communication from LinkedIn, featuring a YouTube video . The video shows children talking about what they want to be when they grow up, and they are juxtaposed against images of successful professionals doing those things. The message is that you can follow your dreams and succeed. It's cute. LinkedIn marketing communication promoting http://imagine.linkedin.com The website accompanying the campaign goes on to explain how some of the professionals featured in the video use LinkedIn to do their current jobs. For example, a chief merchandising officer uses it to discover vendors for unique items. And that's it: end of communication. What I find interesting is that LinkedIn isn't set up very well for people trying to "follow their dreams". I've worked hard to grow my capabilities. I average two hours per day studying work-related topics to expand my reach. For my career, the signal-to-noise ratio on LinkedIn is mostly...