Greg Leib on Software

Recruitics has a culture that celebrates exceptional results.  The expression "get shit done" or more commonly "GSD" has become something of a mantra around our office.  We have GSD stickers and t-shirts. We even have a #gsd message channel where we call out peers who go above and beyond.  When I first heard the phrase, my reaction was not positive. To me it seemed to encourage indiscriminate action and low-quality work. But as I have learned since then, there is a lot of nuance behind the concept - and it goes to the core of what makes Recruitics a unique workplace. In this post, I want to use GSD as a lens to explain a little about our engineering culture. Of course, within Recruitics d ifferent disciplines and teams manifest GSD in different ways: A salesperson who calls a prospect on a Saturday afternoon with a carefully-crafted value proposition.  An account manager who obtains an industry certification that helps her better serve clients. An executiv...

This week I've started planning for the next version of our data collection system. The key realization for me is that I do not know all the questions we will need to answer in the future. Our current focus is on specific sequences of click events, but in the future we might want to look at browser versions or behavioral patterns related to IP addresses. If we don't capture user-agent, for example, we won't be able to answer questions about browser versions. If we don't capture IP then we cannot look for patterns in IP addresses. We should store data in a way that maximizes the range of questions we can address in the future. In the past few years, the cost of storing data have continued to fall. We use AWS extensively.  Amazon S3 costs are very reasonable and guarantees a high level of availability. Also, lower compute costs and open source tools like Hadoop that process large data volumes have greatly increased our ability to extract valuable insights from data. So s...

This weekend one of my choral ensembles had a lesson in dissonance. The two bass parts hold a (dissonant) whole tone interval, which eventually resolves down to a (harmonious) minor third. We were having trouble getting the intonation right, so our choir director had us go very slowly and hold the dissonant interval so our ears would be accustomed to the sound. Just like music, over time software systems cycle between dissonance and harmony. One significant way dissonance occurs is during migration. In a migration, multiple valid domain models partly overlap. The dissonance is created because they embody slightly different assumptions, structure and capability - yet they share elements in common. When the old model falls away, dissonance resolves to harmony, and the cycle renews. Most software teams minimize the duration of their migrations, perhaps regarding them as unpleasant or untidy. In my current work setting, my team cannot risk an abrupt migration because there are too many m...

I was among the thousands of attendees of yesterday's Amazon AWS Summit in New York. One of the key services announced was Zocalo , a file storage and sharing service for the enterprise. One aspect of the Zocalo service that wasn't clear is whether it will have an API. Part of the system I am currently working on provides a workflow for analysts to work with small sets of files. An API for Zocalo would allow automated process and analysts to jointly curate the files. Other AWS services have great API's: I've especially enjoyed working with the Java libraries for AWS services. Amazon usually takes API development very seriously and produces solid interfaces. Interoperating Zocalo with other AWS services would be the killer feature for my current use case, and an API with good libraries would facilitate this. Here are some ideas for what could be done combining Zocalo and other AWS services: Analyze file access and update logs using EMR or Kinesis. This could rev...

A big part of my current job is getting different systems to work together and sometimes to work in a way not entirely intended by the original authors. For example, getting a SSO server to share account data with a CRM platform, or getting any "enterprise" system to have a reasonable user interface (enterprise software is always ugly by default). One important consideration is how much I trust the system I am working to integrate: it's more work to be paranoid, but sometimes the software is  out to get you. I tend to trust popular open source libraries, such as those included in the Apache family like Lucene, Hadoop or Cassandra. Level of activity is an important indicator of a high quality open source project. I also tend to trust self-contained libraries more than external services, since many network and availability failure modes just don't apply when code runs in the same runtime as my own application logic. Conversely, I distrust closed vendor systems and ...

Software engineers often stick close to successful designs used in our past. We have favorite tools and techniques that we reach for without deliberation, and the more specialized our knowledge, the more this tendency dominates approaches to software design. How do we know the difference between the best solution and the one that we're most comfortable with? In a word, the answer is experience . The second system effect postulates that the second system a designer creates will tend to be over-complex for the problem it solves, but eventually experience will help the designer to distinguish between core technique and chance detail. Many online commentaries focus on the way features accumulate as systems mature. There is merit in this view, but I think the key observation is rather that acquired experience is what allows software designers to create more elegant solutions. Different software design projects build experience in differing amounts. Imagine solving very similar probl...