Fate has made me the “money guy” for OpenSSL so I’m going to talk about that for a bit.
As has been well reported in the news of late, the OpenSSL Software Foundation (OSF) is a legal entity created to hustle money in support of OpenSSL. By “hustle” I mean exactly that: raising revenue by any and all means[1]. OSF typically receives about US$2000 a year in outright donations and sells commercial software support contracts[2] and does both hourly rate and fixed price “work-for-hire” consulting as shown on the OSF web site. The media have noted that in the five years since it was created OSF has never taken in over $1 million in gross revenues annually.
Thanks to that publicity there has been an outpouring of grassroots support from the OpenSSL user community, roughly two hundred donations this past week[3] along with many messages of support and encouragement[4]. Most were for $5 or $10 and, judging from the E-mail addresses and names, were from all around the world. I haven’t finished entering all of them to get an exact total, but all those donations together come to about US$9,000. Even if those donations continue to arrive at the same rate indefinitely (they won’t), and even though every penny of those funds goes directly to OpenSSL team members, it is nowhere near enough to properly sustain the manpower levels needed to support such a complex and critical software product. While OpenSSL does “belong to the people” it is neither realistic nor appropriate to expect that a few hundred, or even a few thousand, individuals provide all the financial support. The ones who should be contributing real resources are the commercial companies[5] and governments[6] who use OpenSSL extensively and take it for granted.
Lacking any other significant source of revenue, we get most of ours the hard way: we earn it via commercial “work-for-hire” contracts[7]. The customer wants something related to OpenSSL, realizes that the people who wrote it are highly qualified to do it, and hires one or more of us to make it happen. For the OpenSSL team members not having any other employment or day job such contract work is their only non-trivial source of income.
Which gets me to the main point I want to make in this essay, about responsibility and pride. You can see right on the OSF web site that our consulting rate is US$250 an hour. Two hundred fifty dollars an hour; not high for a lawyer or doctor or even many professional tech jobs, but a living wage for sure. “These guys must be sitting pretty flush, eh?” Uh, no. “Ah, overpriced then, no takers.” Wrong again; I could sell more hours at that rate if only there were more hours to sell. At the moment OSF has about a hundred grand in open contracts — these are executed contracts with purchase orders, not just contracts in discussion or negotiation — that aren’t being worked because no one in this very small “workforce” of qualified OpenSSL developers is available to work on them. Even though they could make good money moonlighting they tend to their other responsibilities first: day job, family, OpenSSL itself. I’ve had prospective clients call me and beg for Stephen Henson to look at their problem. I have standing instructions from one client to please let them know if Andy Polyakov ever has any free time. I’ve had clients ask “would more money help”? Some queries I just turn down right away with “sorry, we’re unable to help”.
Even when we can staff a commercial contract, it can’t be rushed or skimped; these guys are just too used to taking pride in their work no matter what it is. Having worked for decades in industry and government I know that “good enough” and “quick and dirty” are the norm, so for some of the contract work I’ve tried encouraging a pragmatic “get ‘er done” attitude. They won’t do it; nothing less than the very best work they are capable of will do.
The team member without conventional full time outside employment is Dr. Stephen Henson. He’s a pretty private person[8] and he’ll probably be unhappy with me for what I’m writing here (sorry Steve). The creation of OSF was largely inspired by a revelation that was shocking to me at the time. I had been working with some of the OpenSSL team for several years when I learned how much income Steve was receiving (then as now he had no conventional employment). I was stunned to realize that my income, as one consultant of hundreds in one program of thousands in the U.S. military/industrial complex, was over five times his. Five. Times. 5X! This for a world class talent carrying an enormous burden, and when it comes to coding I’m not qualified to carry his keyboard. I had naively assumed that someone with his talent and experience would have a commensurate income, or at the very least be outearning run-of-the-mill hack programmers and consultants like me. Now that OSF is well established and has a growing roster of clients we have gone a long ways towards redressing that situation, but he could pull in a lot more commercial revenue if he didn’t steadfastly refuse to neglect OpenSSL.
These guys don’t work on OpenSSL for money. They don’t do it for fame (who outside of geek circles ever heard of them or OpenSSL until “heartbleed” hit the news?). They do it out of pride in craftsmanship[9] and the responsibility for something they believe in.
I stand in awe of their talent and dedication, that of Stephen Henson in particular. It takes nerves of steel to work for many years on hundreds of thousands of lines of very complex code, with every line of code you touch visible to the world, knowing that code is used by banks, firewalls, weapons systems, web sites, smart phones, industry, government, everywhere. Knowing that you’ll be ignored and unappreciated until something goes wrong. The combination of the personality to handle that kind of pressure with the relevant technical skills and experience to effectively work on such software is a rare commodity, and those who have it are likely to already be a valued, well-rewarded, and jealously guarded resource of some company or worthy cause. For those reasons OpenSSL will always be undermanned, but the present situation can and should be improved.
There should be at least a half dozen full time OpenSSL team members, not just one, able to concentrate on the care and feeding of OpenSSL without having to hustle commercial work. If you’re a corporate or government decision maker in a position to do something about it, give it some thought. Please. I’m getting old and weary and I’d like to retire someday.
_________
1 Any legal and moral means. Geeze, give me a break…
2 I said legal and moral; shameless still goes so here’s a plug for one of the most effective ways your corporation can not only support OpenSSL but also receive something of tangible value in return: a software support contract. We have a formal contract with the fine print that lawyers love, and your accounts payable people won’t be all flummoxed at the bizarre notion of giving money away as they’re used to paying for expensive commercial support contracts for proprietary software. Someday you may even encounter an issue with your mission critical use of OpenSSL that could benefit from direct and prompt attention from the people who wrote that code.
3 The accounting software into which each and every donation is manually entered doesn’t have an easy way of counting the number of transactions of a particular type.
4 One message in particular cheered me (and hopefully my colleagues) and I can’t resist quoting it here. It begins [edited for NSFW filters]: “Thank you … For doing something really f**king hard and making it free.”
5 I’m looking at you, Fortune 1000 companies. The ones who include OpenSSL in your firewall/appliance/cloud/financial/security products that you sell for profit, and/or who use it to secure your internal infrastructure and communications. The ones who don’t have to fund an in-house team of programmers to wrangle crypto code, and who then nag us for free consulting services when you can’t figure out how to use it. The ones who have never lifted a finger to contribute to the open source community that gave you this gift. You know who you are.
6 Multiple agencies of the U.S. Department of Defense (DoD) have provided substantial financial support over a decade for the OpenSSL FIPS Object Module series of open source based FIPS 140-2 validations, most recently DARPA. But, those validations essentially just distort and contort existing OpenSSL code to satisfy some peculiar and arbitrary requirements and do nothing to improve the overall quality of OpenSSL itself. Having consulted in that environment I know OpenSSL is very widely used throughout DoD, both directly and as repackaged by commercial vendors. Given the bazillions of dollars in DoD funding you’d think an investment in OpenSSL would be a no-brainer.
7 The commercial contracting work falls into four general categories:
- Annual software support contracts, mentioned above. Realistically speaking we’re usually going to address the kind of problems reported under these contracts anyway (though perhaps not as quickly), so these provide the most benefit overall.
- Adding/extending specific features of general interest, e.g. TLS 1.2, hardware specific optimizations. This kind of work is a win-win for everyone as the entire OpenSSL community typically benefits along with the sponsor of the work.
- FIPS 140-2 validation related work. This is of benefit to a much smaller segment of the user community, and has significant outsourced costs. It also arguably has a negative impact on the OpenSSL code base and diverts scarce manpower from improving OpenSSL proper.
- Consulting on issues unlikely to be of general interest, such as porting to specialized proprietary environments or assisting with customer modifications to OpenSSL.
With very few notable exceptions (Qualys, PSW Group) commercial contracts are tied to specific deliverables and do not fund work on fundamental maintenance and development activities like releases management, code review and refactoring, performance and security, etc.
8 He really is the private sort, even (perhaps especially) when it comes to maudlin sentiments as expressed here. He also has to deal with a large volume of technical correspondence. So please don’t contact him directly without a really good reason. I will be happy to collate and forward on a reasonably timely basis a digest of comments sent c/o marquess@opensslfoundation.com.
9 “Hey wait a minute — didn’t those bozos just make a dumb sloppy mistake and break the internet?” That’s really a topic for another essay, but all non-trivial software has bugs (the Apple “goto fail” and Debian PRNG bug come to mind). Given the widespread use of OpenSSL over many years it still has an excellent track record. The question that has been asked repeatedly and not often answered is why did this bug take so long to find? Well consider that:
- The code was written by someone with a proven track record who is a co-author of the heartbeat specification (RFC6520). It was reviewed by the OpenSSL team and no one spotted a problem.
- The code was visible all along to the entire OpenSSL community and no one saw it.
- OpenSSL is used by many multinational companies and major government agencies with huge resources who didn’t spot it (or at least did not report it, same difference).
- Many have called this “the worst security bug ever”, which is debatable but it is a very serious vulnerability. There are many security researchers in the world who have found problems in OpenSSL and reviewed the code with a fine tooth comb, as shown by all the academic papers which have been written over the years and the security advisories relating to them. Finding this bug would have been a feather in the cap of any one of those security researchers.
- Two years passed before Google with its impressive technical resources and talent (and shortly thereafter Codenomicon) found this issue.
So the mystery is not that a few overworked volunteers missed this bug; the mystery is why it hasn’t happened more often.