Charting some KPIs ... how hard can it be?

In the last couple of weeks I was working on an initiative to make all the Gilt Tech KPIs available in one place. There are lots of dashboard solutions available, but nothing really did what we wanted it to do or it was clumsy to use. At the end we decided to build a small Scala Play app to get what we want.

As part of that exercise we had to find a way to visualise the history of the KPI values (timestamp, value pairs) and I started to play around with charting frameworks and figured it makes sense to write up what I have found out. The list of solutions is kind of arbitrary and is by all-means not aiming to be a complete list of available charting frameworks, but gives you a glimpse of what is out there.

The source code is available on my Github account. Feel free to clone the repo and play around with it yourself. There are 4 tagged versions available ...

• v0.1 - Featuring charting based on sfreechart
• v0.2 - Featuring charting based on highchart
• v0.3 - Featuring charting based on D3
• v0.4 - Featuring charting based on chartjs

After you have cloned the repo you can checkout the version you are interested in and start the server as described in the README. Let's go through them one by one.

I started with a very simple server based charting solution called SFreeChart. It is Scala wrapper around JFreeChart. I was able to get that up and running very fast, but realised very quickly that it would not give me what I want, because I do not want to manage the files on the server side and the chart is not interactive.

 KPI charted with SFreeChart

Note: While I was looking into how to manage the files on the server side I found Scala automatic resource management, which is a very nice way to make sure that you always manage/close your resources (e.g. file descriptors, database connections, ...). Recommended. Take a look at it.

At Gilt we are using Gatling to do our load and performance testing. The Gatling charts are very nice and are based on Highcharts, a charting solution from Highsoft. Highcharts is free for non-commercial use. For commercial use, you have to get a license.

Highcharts is awesome! To get into it you can use the online documentation or you can read the book. The API online documentation is very well done. It has lots of examples and links to jsFiddle and lets you experiment with the framework. The book is very well structured and is a must read, if you want to get serious about Highcharts.

KPI charted with HighChart

The one thing that made me keep on looking was the license fee for commercial use.

Next I looked at D3 (Data-Driven Documents). D3 is not really a charting framework. It is much bigger, much more powerful. It allows you to visualise data in any way imaginable. It is unbelievable what you can do with D3!!! But ... the flexibility comes at a price. For me the best way to look at it is the way Git is structured: If Highcharts is the porcelain of charting then D3 is the pluming. D3 is raw. There is nothing you cannot do, but you have to do it (lots of pixel pushing).

KPI charted with D3

At the end I was looking for a client-side, license-free, easy-to-use charting framework and stumbled of the new kid on the blog: Chart.js. Chart.js is new and I was able to make it work in half an hour. For now it seems to fit the bill.

KPI charted with Chart.js

As I said this is by no means an all-inclusive review of all available charting solutions. All discussed solutions and approaches have their merits. It comes down to your requirements to decide what is the right one for you.

About broken windows and software gardening ... - Thoughts from The Pragmatic Programmer

These days I am writing more source code again (mostly Scala) and decided to (re)read a couple of books to refresh some thoughts and ideas and believes that I always had when it comes to writing source code.

One of the books I was (re)reading was Clean Code and there are lots of good ideas in it, e.g. after reading the book 15 years ago I got into the habit of refactoring, whenever I feel the urge to put a comment in the code, which is an indication that the code is not clear/self-documenting enough. The only comments I am writing these days are comments on why the code is doing what it is doing, not what it is doing (that should be apparent from the code itself).

Anyway ...

While I was reading Clean Code one of the Gilt engineers in Dublin recommended to also take a look at the Pragmatic Programmer and I just finished it two weeks ago. The book is more compact than Clean Code. If you have the time, read both. If not, just read the Pragmatic Programmer.

Two thoughts/pictures resonated with me ...

• Broken Windows and
• Software Gardening

Broken Windows is about fighting Software Entropy and puts a nice picture/story around the concept that you should always check-in the code a little bit cleaner than you found it. Be clean and if you find a broken window (that somebody else broke), fix it! That behavior can be omnipresent in your life. I live in Ireland and do a lot of hill-walking and the rule is whatever you carry up the hill, you will also carry down, but you can take that a step further and pick up cans and bottles that you find on the hill (my habit to pick up litter wherever I find it drives my wife nuts) and leave the hill cleaner than you found it.

Software Gardening puts our ambitions to construct software into perspective. If you construct something (e.g. a bridge) it is kind of static. The idea is that software engineering is more like gardening and less like construction. The software system has a life on its own. It grows. Initially you have an idea how the garden should look like and put the flowers and the bushes and the trees into place, but then over time you need to make sure you also spend some time on pruning and refactoring to keep the garden in shape, because it has a life on its own.

Here is a picture of me looking through a window into my garden.

Hi Ugo, thanks for the pointer ...

Sometimes small things make a big difference ... using Source Code Pro with Emacs

<outing alarm>I love my emacs.</outing alarm>

Sometimes small things can make a big difference. Like a dot. You can put it at the end of the sentence or in a number (1.001 vs. 100.1 - can make a "small" difference) or you put it into the zero to make it easily distinguishable from capital O.

Now you might say, who cares, but if you read a lot of source code (like I do) Source Code Pro is a real pleasure for your eyes. Give it a try. Highly recommended.

PS: Source Code Pro works with all kinds of IDEs, not only Emacs.

Teaching Scala @ TCD ... back to the roots

20 years ago I used to teach object-oriented programming (based on Eiffel) at my old university in Germany.

This week I got the chance to experience a deja-vu, because earlier this year I did run into Glenn Strong who is teaching Functional Programming in TCD in Dublin and we started to develop the idea that it would be useful to add a Scala module to his Haskell based lecture.

The main idea was that the module could complement the lecture by exposing the students to a second functional programming language and at the same time highlight why at the current point in time Scala is the programming language of choice when large corporations decide to adopt a more functional programming paradigm (Gilt being one of them).

In that context the module focused mainly on how well Scala co-exists with Java. The two real-world scenarios that got implemented in the lab exercises was to rewrite a piece of Java in Scala and to integrate Scala with existing Java source code. Feel free to clone the repo and play with it yourself. My stuff is in the roland.tritsch folder.

A starting point for ... Apache Felix, OSGi, Scala, Sbt

It is not a secret ...

I like OSGi and Scala ... and therefore it was only a question of time until I started to wonder, if there is anything that would prevent me from using OSGi from/with Scala. The short answer and the good news is ... nothing.

I just checked in my port of the original Apache Felix Tutorial to Scala. As mentioned in the README I do not want to claim that this boilerplate code, but it is a working development environment that you can use as a starting point for your own experiments.

Machine Learning at Gilt - "Maven-izing" a/the Apache Mahout tutorial

If you work for Gilt Tech than you WILL (sooner or later ... somehow) get involved in our Machine Learning and Recommendation efforts. Our requirements are extremely complex and challenging. We have over 5 million members (fast growing) and a product catalog which features products that are selling out between 10 secs and 3 days, means we need to make recommendations for products that have (almost) no history to members that might not be with us for a long time (yet). In that context we are constantly investing in making our personalized EMails and also the content of the WebSite more relevant to the member who is looking at it.

If you do/get this right, it is Mass-Customization (the art of cost-effectivly build instances of one (store)) at its best ...

“If I have 3 million customers on the Web, I should have 3 million stores on the Web.” – Jeff Bezos, CEO of Amazon.com

The difference between Amazon and Gilt is that Amazon got a "lot" of time to build these stores (hours, days, ...) . In our case we almost need to build these stores on the fly (seconds, minutes, ...). This requires new thinking, new approaches, new algorithms, ... it requires innovation. Got ideas? Let's talk!

Anyway ...

In that context I started to play with Apache Mahout and found this good (old) tutorial (still top of the list when you Google "Apache Mahout Tutorial") from Grant and decided to "maven-ize" it. Just install git and run ...

> git clone git://github.com/rolandtritsch/Apache-Mahout-Tutorial.git

... and then install maven and follow the instructions in the README. Enjoy!

Dublin Innovation Festival - Talking about Innovation, Gilt and Ireland ... and Evolution and Darwinism

A couple of weeks ago I had the chance to talk about three things I care about: Innovation, Gilt and Ireland. The occasion was the Dublin Chamber of Commerce Smart Series Event on Innovation (that was the day after Dundrum Shopping Center and half of Dublin got flooded). The objective was to share a couple of thoughts on what innovation is (or is not), what Gilt is doing to foster innovation and what Ireland has to offer to companies like Gilt.

To start ... Innovation is probably on all our minds. If you get it right as an individual or as a company or as a country, you create a differentiator for yourself. Everybody wants to be innovative. The main point of the talk was to point out the similarities between innovation and evolution.

There are lots of definitions available, but one way to think about innovation is in terms of the hard and the brain. The hard tells us that innovation is this wonderful thing that happens in the shower or in my case on the bike on the way up to Sally Gap, when you have this moment of brilliance for the next best, bigger business idea that will actually change the world. And then we look at individuals like Steve Jobs, who are obviously either showering a lot or who know the way up to Sally Gap like the back of their hands, because it seems they innovate everyday and we start to wonder what is wrong with us.

The short answer is actually: nothing. Nothing, because in my opinion it does not work this way. Even in the case of the iPod you could argue that the real innovation was the Sony Walkman and the iPod was a refinement on the original idea.

The more rational definition behind innovation is that innovation is not a thing or an event that "happens". It is the result of a process. I am personally a big fan of evolution and Darwinism. There is only so much you can plan and/or foresee. For the rest you have to try (a lot) and see what works.

That sounds easy, but it isn't, because it is counterintuitive. You have to embrace mutation and selection. You first have to create a "creative" environment that produces a lot of (hopefully) good, new ideas and then you have to filter them and needs to happen early, because being innovative does not mean to be just creative. It means to be successful and that means it is not about creating a lot of good, new ideas, it is also about finding the ones that make a difference for your business and for your customers and if your are lucky for your industry.

Not to a small extend it is this “bi-polar” nature of innovation that makes it difficult for us to implement innovation. Just think about it: On one hand you ask people to mutate (means to come up with a lot of good, new ideas) and to be passionate about these ideas and then you need to have strong selection in place to kill most of them. If you do it wrong it will frustrate people, because they might get the impression that nobody is listening and innovation will just stop to happen.

In a lot cases we fail to get good innovation because we either mutate too much and do not select enough or the other way around. Good innovative environments show both characteristics: A lot of mutation and strong selection. Innovation is the evolution of ideas. It is Darwinism of ideas. It is the survival of the fittest idea.

To create such an environment and to maybe avoid the frustration mentioned above, you should reward people equally for coming up with an idea and also for giving up on an idea. Failure must become an option and maybe even something that you should or could celebrate.

Let's talk about Gilt. Gilt is innovative. Let me talk about two examples that will illustrate how innovation based on strong selective pressure (means a well-defined niche that you need to adapt to) and/or strong mutation (means a good idea) can look like.

The first example I want to talk about is inventory management. Normal retail business models allow you to do inventory management based on thresholds, means as soon as your inventory goes beyond a certain threshold you reorder. When you reorder has something to do with your lead times and also with the quality of your inventory management, means you actually do not care, if there is a little bit more or little bit less in stock, as long as you never run out of stock.

The Gilt business model requires a totally different approach. Because in most cases we are selling excess stock, 80% of all products we have ever sold we have sold only once, means we never really can reorder and that means we are in the business of selling down to the very last unit. Always!!! And we need to do that while we maintain highest possible availability and scalability. To deliver on these business requirements we had to question conventional wisdom and build a very innovative inventory management runtime platform, based on highly available and highly scalable NoSQL databases that actually relax some of the transactional ACID properties, but still allows us to sell down to the very last unit. And we succeeded. That’s an example for innovation out of necessity to adapt to a niche.

The second example for an innovation at Gilt is the 360 degree "look-around" Jetsetter iPad app. The application allows you to take a 360 degree view of a room and explore literally every ankle of it. This is an example for a strong mutation.

Gilt is still a very young company, means we are still learning how to institutionalize innovation in the company, but so far I believe that the 3 pillars of innovation in Gilt are people, process and culture. You need people that are creative and intelligent. You need a process that is open and transparent. And you need a culture that equally celebrates success and failure.

In that context we are experimenting with "Unstructured Time" where we bring people together without an agenda. We also believe that a lot of good ideas will not come from us, but from outside the company and in that context we are currently building a public API that we will expose to partners and other interested parties, so that they can innovate independently from us. Earlier this year we did a hackathon and had the entire SW development organization on the hunt for the next better idea and also prove it (within 24 hours). Our color search feature (which allows you to take a picture of a color and search for similar products) and the public API were actually incubated in the hackthon. We are also doing a lot of A/B testing to measure the "goodness" of our ideas. This allows us to rationalize the conversation about the value of certain ideas.

But there is always more to do. Right now, we are thinking about a "follow-the-sun" hackathon were global teams will implement something over a period of three days, because we are suspicious that some good ideas, will get conceived, when people from different cultural backgrounds work together, means it is not only about creativity and intelligence, but also about heterogeneity.

Let me wrap up, by exploring that thought more. It is my believe that culturally diverse, culturally heterogeneous environments are more likely to be innovative than homogeneous environments. If you also believe in the value of heterogeneity than there is good news, because the Irish software engineering population is among the most heterogeneous and diverse in Europe, maybe even in the world, means Ireland allows you to build teams that will look at problems and possible solutions in different/unique ways. They all went to different schools and had different experiences before they came to Ireland. This creates an environment for innovation to happen.

Ireland is the only English speaking country in the Eurozone. This makes it easy for talent to relocate to Ireland and to work here. Building an innovation hub will probably to a certain extend rely on our ability to take advantage of the talent in the rest of Europe and to use the heterogeneity and diversity that comes with it as a driver to build creative, innovative teams here in Ireland.

Gilt is very excited to be in Ireland. We are looking to use our presence here to build a stronger, more scalable, even more innovative Gilt Technologie Organisation.

(Small) change to my social networking ...

First of all ... apologies!!! In the last 2 weeks my (private) tweeds (almost) completely died away. I was just madly busy with work :(.

Going forward, I will (continue to) send my "diary" updates to @rolandtritsch, but will send my more work related tweeds (e.g. Blogging about ..., Reading about ..., Looking at ...) to @innolocity. @innolocity is a public feed. Just register here to follow.

Cloud Computing - A practical guide for the common user

These days (and especially after the Amazon EC2 outage :)) everybody is talking about cloud computing. It seems you cannot be sexy, if you do not "live" in the cloud. But most of the more "common" users (and for the sake of the discussion, I am including myself in that group) probably think that cloud computing and using "the cloud" is something that enterprises use to (potentially) safe cost and also to (potentially) gain flexibility.

The first observation is that there is a fair amount of confusion about what cloud computing is. The reason is that everybody is latching on to the buzz and is using the word cloud to make the product or the service at hand look more trendy.

To bring order to the chaos you could probably say that there are three categories of cloud computing solutions:

• Data Clouds - Give access to a large amount of disk space and allow the disk space to be shared between a (potentially) large number of users (e.g. DropBox)
• Computing Clouds - Give access to large amount of computing (CPU) and hardware power in general (including main memory, disk space and in most cases a/the operating system) to run whatever application you want to run (e.g. Amazon EC2)
• Services/Application Clouds - Give access to ready-to-run/ready-to-use services/application to solve a specific problem (e.g. Google Mail (running a mail server for you), GitHub (running a source code repository for you), SalesForce.com (running a CRM for you))

So question becomes, why is this important or relevant to a/the common (means non-enterprise) user? For me personally I get two benefits out of it: I use "cloud computing" as a (you could call it) "poor-mans backup solution" and also as an "information sharing/syncing solution". Specifically I am using ...

• Google Mail/Calendar - to "host/store" all my email folders, contacts and my personal calendar
• DropBox - to store all my files (including picture and music libraries)
• GitHub - to store all my source code

Syncing the data is kind of automatic and there is always an offline version of the data available, which gives me very good protection against disk failures (but on top of that I am also running a TimeMachine backup (better safe than sorry :))), means in case of a disk failure, I can install an IMAP email client, a DropBox client and Git on a new piece of hardware and I am up and running again in no time.

The second value is information sharing. By now I have 3 devices (laptop, tablet, phone) and need to share/sync data between the devices and my wife got a laptop and a phone too and I also need to share/sync information with her (especially my source code :)). Having the data/information available in "the cloud" makes that syncing/sharing exercise straight forward (will publish a blog post on how to share/sync music using DropBox in due course).

Buttom line: "Cloud computing" is not only for enterprises :).

Adventure 2.0 - Building the Gilt Software Development Center in Dublin

A couple of weeks ago I got a call from the Gilt Groupe in NYC and they asked, if I would be interested in joining Gilt Groupe to build a software development center in Dublin. I went to NY and met some of the brightest engineers in the industry and was totally blown away by the enthusiasm and the commitment to build and run the best e-commerce shopping experience on the web.

And here I am today sitting in a room with the Irish Finance Minister Michael Noonan announcing that Gilt is establishing its International HQ and a software development center in Dublin. And I can't tell you how exited I am about being part of that adventure.

Gilt is the number one flash sales site in the US. Within 3 years they went from 0 to more than USD 400M in revenue. Flash sales is outlet shopping over the internet and allows large designer brands to get the best possible deal on their excess stock. And it is frantic. Gilt has close to 5M members and sends 3M emails every day (within 15 mins) to notify its members about upcoming sales events. The next day at noon(est) the sale goes live and it is first-come first-served on all kinds of goods from a wide variety of designer brands. The discounts are from 20 to (sometimes) over 70 percent.

That presents a "small" technical challenge, because at noon within seconds the traffic on the website goes from decent to 5000 pages/sec. At peak time the site processes over USD 1M in revenue within 30 mins (every day). And it serves 99.7% of all pages, fully dynamic and personalized, within 100 ms at all times. And it cannot go down, meaning scalability, performance and availability are at the top of the requirements list. Fortunately Gilt employs only the brightest and the best.

Let me know, if you are interest to join us. I am hiring.