Greg Leib on Software

This week I've started planning for the next version of our data collection system. The key realization for me is that I do not know all the questions we will need to answer in the future. Our current focus is on specific sequences of click events, but in the future we might want to look at browser versions or behavioral patterns related to IP addresses. If we don't capture user-agent, for example, we won't be able to answer questions about browser versions. If we don't capture IP then we cannot look for patterns in IP addresses. We should store data in a way that maximizes the range of questions we can address in the future. In the past few years, the cost of storing data have continued to fall. We use AWS extensively.  Amazon S3 costs are very reasonable and guarantees a high level of availability. Also, lower compute costs and open source tools like Hadoop that process large data volumes have greatly increased our ability to extract valuable insights from data. So s...

I was among the thousands of attendees of yesterday's Amazon AWS Summit in New York. One of the key services announced was Zocalo , a file storage and sharing service for the enterprise. One aspect of the Zocalo service that wasn't clear is whether it will have an API. Part of the system I am currently working on provides a workflow for analysts to work with small sets of files. An API for Zocalo would allow automated process and analysts to jointly curate the files. Other AWS services have great API's: I've especially enjoyed working with the Java libraries for AWS services. Amazon usually takes API development very seriously and produces solid interfaces. Interoperating Zocalo with other AWS services would be the killer feature for my current use case, and an API with good libraries would facilitate this. Here are some ideas for what could be done combining Zocalo and other AWS services: Analyze file access and update logs using EMR or Kinesis. This could rev...

My main project at TheStreet is a web scraper, and it relies on PhantomJS browser to run user interactions, make DOM adjustments and take screenshots. Taking screenshots is an area where PhantomJS excels over the other Selenium WebDriver implementations. The thing that works so well for PhantomJS is that the viewport is not constrained to a screen, so it takes screen captures that are as long as the page content is. I just set the viewport to 1366x1 and let the content extend the height of the screen. One thing that doesn't work well is rendering fonts. Running on AWS the rendering of fonts is extremely unstable. The same site can render as a serif or sans-serif font, and it's not clear why. The problem with this is that it makes the screenshots very different even if the site isn't changed. I'm using pixel-level analysis of the screenshot to detect changes, so this sets off lots of false positives. PhantomJS uses the JavascriptExecutor interface to allow arbitrar...

A big part of my current job is getting different systems to work together and sometimes to work in a way not entirely intended by the original authors. For example, getting a SSO server to share account data with a CRM platform, or getting any "enterprise" system to have a reasonable user interface (enterprise software is always ugly by default). One important consideration is how much I trust the system I am working to integrate: it's more work to be paranoid, but sometimes the software is  out to get you. I tend to trust popular open source libraries, such as those included in the Apache family like Lucene, Hadoop or Cassandra. Level of activity is an important indicator of a high quality open source project. I also tend to trust self-contained libraries more than external services, since many network and availability failure modes just don't apply when code runs in the same runtime as my own application logic. Conversely, I distrust closed vendor systems and ...

Yesterday I took Amazon CloudFront for a spin. Creating the CloudFront distribution was pretty simple - the wizard process flowed nicely. I found myself relying on the help text in places, but the most surprising thing was how long it took for the distribution to become enabled. I didn't time it exactly, but I probably spent 45 minutes waiting for my new CloudFront distribution to change from "In Progress" to "Enabled" status. The performance is a bit confusing. Compared to the S3 bucket, I didn't see any improvement in performance in a few tries - in fact, the CloudFront CDN performance was worse than the S3 bucket on its own for my 217 KB image file. I decided to take a larger sample, loading the same image 30 times in Chrome and noting the timing data from the "network" tab in the developer tools. I'm located in Brooklyn, have CloudFront configured for the US/Europe with download mode configured. My S3 bucket is in the US Standard zone, w...

I am officially registered for AWS Summit 2013 | NYC on April 18th. I've been using Amazon AWS in various ways for the past 3 years. Kanban Solutions, my current employer, hosts its own virtual infrastructure on VMware and serves clients who pay for "high touch" support from RackSpace and others. I didn't expect they would have much interest in letting me attend an all-day conference on Amazon AWS. Surprisingly, not only are they sending me but our director of engineering and another engineer were already planning to attend. I need to decide which breakout session to attend. They have a few options that are pretty interesting to me. The use cases session is interesting because it will cover Amazon's new operations tool OpsWorks , which allows administrators to manage the configuration of servers and resources deployed in the AWS cloud using "Chef" recipes. I think I'm going to attend their architecture session, which address how to scale applica...