Skip to main content

Posts

Spark Cassandra Connector Tip

We're using Databricks as our provider for Spark execution, and we've been struggling to get the Spark Cassandra connector to work outside of the local development environment. The connector was attempting to connect to 127.0.0.1 even though we were passing the new host information into the getOrCreate(..) call. After working with Ganesh at Databricks support, we figured it out. The realization is that in Databricks, calls to getOrCreate() from a fat jar don't create a new SparkContext object. Thus, the configuration passed in gets ignored. If you want to update the Cassandra host information for the connector, you must update it after  the call to getOrCreate() instead. Add the configuration directly to the context and you'll be good to go!
Recent posts

Where's Greg?

It's been almost two years since I posted anything here. In late 2015, things got really busy for me at Recruitics and, to be totally candid, maybe I wasn't sure what this blog is for. What happened since late 2015? My department moved offices. Twice. We weathered a round of layoffs and rebuilt the teams that were affected. And we've made a ton of progress improving our applications and data processing. I joined Twitter . I became the solo developer of a new integrations platform that now handles over 30% of our data collection traffic and cost tracking. And in two months we launch a free analytics product that will change the competitive landscape in online job advertising. So ... there's been a lot going on for me. What's next for this blog? My original purpose for this blog was to contribute something that other software engineers might find useful. I posted a few technical commentaries on topics not well-covered elsewhere, and I wrote somewhat haphazar

ReactJS, NPM and Maven

I'm just starting to get into working with ReactJS, Facebook's open source rendering framework. My project uses SpringBoot for annotation-driven dependency injection and MVC. I thought it would be great if I could use a bit of ReactJS to enhance the application. If you're looking for a basic conceptual intro, I recommend ReactJS for Stupid People and of course the official documentation  is quite good. In full disclosure, I still have no idea how to do "flux" yet. As an experienced Java backend developer, I'm pretty decent at hacking Maven builds - which is precisely what this blog post is going to be about. First, a word about how React likes to be built. Like many front-end tools, there is a toolkit for the node package manager (NPM). From the command prompt, one might run npm install -g react-tools  which installs the jsx command. The  jsx  command provides the ability to transform JSX syntax into ordinary JavaScript, which is precisely what I want. O

Culture Matters

Yesterday morning, my software engineering team and I learned that we would be moving into subleased office space on the fourth floor. In the afternoon, with little fanfare and only one cardboard box among us, we moved all our stuff down two floors. In full disclosure, I did offer to buy some at Staples across the street, but nobody really felt it necessary. I think our move exemplifies the kind of culture I want to build as we grow the engineering team: unfussy, collaborative, empowered, pragmatic. The job market for software engineers in NYC is booming, so it is surprising to many candidates how much we care about team culture. We've declined to make an offer based on culture more here than anywhere else that I've conducted engineering interviews for. There's certainly a "no assholes" rule around here, but our considerations go deeper than that. Ultimately, we want to hire people aligned with our company's values: Entrepreneurial thinking and actio

Three Legs of GSD

Recruitics has a culture that celebrates exceptional results.  The expression "get shit done" or more commonly "GSD" has become something of a mantra around our office.  We have GSD stickers and t-shirts. We even have a #gsd message channel where we call out peers who go above and beyond.  When I first heard the phrase, my reaction was not positive. To me it seemed to encourage indiscriminate action and low-quality work. But as I have learned since then, there is a lot of nuance behind the concept - and it goes to the core of what makes Recruitics a unique workplace. In this post, I want to use GSD as a lens to explain a little about our engineering culture. Of course, within Recruitics d ifferent disciplines and teams manifest GSD in different ways: A salesperson who calls a prospect on a Saturday afternoon with a carefully-crafted value proposition.  An account manager who obtains an industry certification that helps her better serve clients. An executiv

Capture Everything

This week I've started planning for the next version of our data collection system. The key realization for me is that I do not know all the questions we will need to answer in the future. Our current focus is on specific sequences of click events, but in the future we might want to look at browser versions or behavioral patterns related to IP addresses. If we don't capture user-agent, for example, we won't be able to answer questions about browser versions. If we don't capture IP then we cannot look for patterns in IP addresses. We should store data in a way that maximizes the range of questions we can address in the future. In the past few years, the cost of storing data have continued to fall. We use AWS extensively.  Amazon S3 costs are very reasonable and guarantees a high level of availability. Also, lower compute costs and open source tools like Hadoop that process large data volumes have greatly increased our ability to extract valuable insights from data. So s

Two Months

As a new technology executive, my biggest worry is whether I am focusing on the right issues. My prior roles primarily required technical work, but they mostly involved small problems and small teams. For my first executive position, I expected to focus on organizational issues more than technical ones. Two months in, I think my expectations and role reset strategy were about right. Here are some of the challenges I've been working on so far: Hiring. I'm shifting the organization from being mostly outsourced offshore to a more even onshore/offshore model. Sourcing and assessing engineers for senior roles has been challenging. I'm committed to building a great team. This is my top priority, and so far we've filled two of the four most senior roles. Our offshore people are solid, and I'm really excited about the new people we're hiring. Process.  We need a simple and unambiguous approach for software development. I've been pushing the team to use Jira a