Clojure: an outsider's investigation

Last week, this post on the Puppet Labs blog caught my eye. It announces a services framework called TrapperKeeper, which seems interesting. To be honest I haven't looked into what it does and how it does the things it does.

I did however spend a bit of time investigating clojure as well as the community response to this announcement. I'll share my thoughts here. I do have to warn that this is all found through creative surfing, so welcome to how my mind works when investigating a (to me) new piece of open source technology.

Clojure

I started by looking at Clojure. Not so much at what the language can do and it's syntax and all that, since a) my programming days are (sadly) mostly over and b) there are far smarter minds that can say sensible things about that.

I am however increasingly interested in the continuity of technologies, as this seems to be an important thing in order for enterprises to adopt them. This in turn helps me to decide wether we should look into offering training for those technologies. So, I dug into the information that is publically and easily accessible:

• The GitHub contributor stats page: as of this moment, the vast majority of the commits (1600+ vs 200+ by Stuart Halloway, the runner up) are done by Rich Hickey, the original author. In the past 3 months however, Alex Miller has the lead (stats here), indicating a possible shift of attention for Rich. Of course this is pure speculation and I don't claim to have any inside knowledge here, remember this is just an outsider's perspective.
• Let's dig into which companies are behind those top contributors: This gives me a decent feeling. There's not a lot of commits going on to the clojure core repository, but control doesn't seem to be resting with a single company. That said, the main contributor is a professional services company, so people can turn somewhere if they want support.
  • 4 out of the top 6 are from Cognitect, a company fully focussed on Clojure and Datomic. Doing a bit of reading, they seem to be the "good kind" of open source company. Minor downside: the company is quite young.
  • the other 2 top contributors are Toby Crawley who works for Red Hat and Andy Fingerhut who according to a quick LinkedIn search works for Cisco. Good, two major enterprises who at the very least have people working on this. Toby's site clearly states he works on this professionally, for Andy this is less clear.
• Going through profiles of the main contribs, I found an interesting blog post by Alex Miller. This blog summarizes the 2013 State of Clojure Survey. Inside it, we find some interesting nuggets:
  • Tooling is the biggest category of complaints. This is interesting, because it directly conflicts with what the Puppet Labs blog post lists as a good reason to go for the JVM. It seems like the culprit is "the relationship between Clojure code and bytecode is complex and not necessarily 1-to-1 – getting good s-expression level support is challenging". I don't pretend to know better then either party, but I am cautioned by this. Anyone who can shed a light on this is welcome to leave a comment.
  • The other big category of problems is documentation. That is the same thing we can read in the HackerNews discussion. Having spent a decent bit of time with half-assed documentation in the MySQL HA scene, I am not super thrilled when reading this.
• Comments on the Puppet Labs blog post itself:
  • 'Engineer' said: "It's sad that the level of technical ability is so low that people adopt languages like clojure because they think its "concurrent". The JVM is not concurrent, therefore clojure is not concurrent therefore you're just signing up for a world of hurt. The Erlang VM has been around longer than the JVM, it really is concurrent, and it really is battle tested."
    Sadly, I have no idea if this is true, and without much further investigation it's hard to verify. I just see it as a possible red flag that I'd want to dig into later on.
  • 'Jeff Dickey' said: "First clue: the "we're so awesome we have to build our own infrastructure even when we're probably complete n00bs in our new hipster language" syndrome. Most of the dings that were laid out as "justification" were *operational nice-to-haves*; if your new environment isn't mature enough to have rock-solid operational support (and anything on the JVM really *should*), then you are fundamentally misunderstanding something."
    This doesn't pertain to clojure so much, but to the fact that Puppet Labs created TrapperKeeper. While the language used here is not my favorite, I am concerned about the underlying point: This framework is obviously not Puppet Labs' core business. While important for their products, I wonder wether it's a great long-term plan to build this stuff in-house (and thus spend resources on this). I guess the longer term will have to determine, mostly by wether the project will get outside contributors/contributions. Irrelevant but ironical: discussing this issue in this specific case, given that puppet was created to counter an exactly similar problem of everyone creating their own tooling in-house :)

That's all for now. It's too early to tell what this all means in the grander scheme of the Puppet ecosystem and where this will all lead in the next few years. Personally I'm not happy with the JVM from an operational perspective as it's startup time and memory usage are a bit of a turnoff. That said, PuppetDB has been a major step forward since Stored Configurations in Puppet 3.x, so I'm just going to sit back and digest all my newly found knowledge while waiting to see where this is all going.