http://royal.pingdom.com/2010/06/18/the-software-behind-facebook/

What really impressed me is that, even though the underlying components are often very sophisticated, the overall architecture is very straightforward and adheres to the architectural premise of “simplify, simplify, simplify”…

Kudos to the Facebook team, and thanks to the folks at pingdom for compiling this information.

http://www.cio.com/article/645763/Cloud_Computing_2011_Predictions?page=1&taxonomyId=3112

“No cloud vendor, no matter how large or smart, can repeal the laws of physics, and migrating workloads and (especially) data between sites confronts the issue of “the skinny straw,” which is the fact that the connectivity between internal sites and public cloud providers is much lower than that within either of those environments.

Furthermore, supporting seamless migration requires a sophisticated IT infrastructure and operations capability, which translates into investment and skill building. While these factors can be procured or created, it’s not a trivial task to do so.

Both of these challenges will make IT executives realize that the all-moving hybrid cloud strategy is overly ambitious and needs to be scaled back. Trying to implement such a vision, for many organizations, will prove to be “A Bridge Too Far”.”

Cloud is powerful and should be considered in nearly any IT strategy, but it also falls into “trained experts at work, don’t try this at home”.  Lumenovis and others have years of experience in hosted / SaaS systems and can help with the design, planning, and implementation of SaaS solutions.

http://www.eweek.com/c/a/Application-Development/Java-C-C-Top-18-Programming-Languages-for-2011-480790/?kc=EWKNLBOE12242010FEA1

I’m excited to see the number of new platforms that aim to simplify software development.  Seriously, the biggest language advance over the last 20-30 years (since C/C++ days) may have been virtual machines with garbage collection.  Useful, but certainly not earth-shattering.  True, there have been major improvements in productivity now that app servers and open source libraries provide most of the “plumbing” for free (or close to it), but the act of writing a program using basic language syntax is still pretty primitive and prone to error.  I’m excited to explore the new languages, and I look forward to radical improvements in productivity and quality over the next decade.

It’s not easy, even when you’re aware of the typical vulnerabilities and diligent about securing against them.  Recently I was involved in a project in which an attack vector was accidentally left wide open through a one line oversight in a Grails application.  Grails has excellent built-in protections against attach vectors such as XSS (Cross-Site Scripting) and SQL injection- assuming you actually take advantage of them.  In this case, our team had enabled XSS protection on pages via the “render as HTML” global configuration- this setting forces all variables written to the HTML page via GSP to be encoded as HTML.  Unfortunately, we were also making heavy use of taglibs (which are not affected by the global setting), and in one taglib we failed to manually encodeAsHtml() a single parameter which was subsequently exploited when the site was penetration tested.

One hole.  Busted.

Moral of the story:

- Be very familiar with ALL common (and hopefully some of the uncommon) web vulnerabilities.  A good list can be found here:  http://www.owasp.org/index.php/Top_10_2007.

- Leverage best practices in software development: code reviews, TDD, vulnerability testing, etc.

- Think like a hacker: try to crack your own site.

- If you’re developing a site with critical financial or privacy data, spend the money to have high quality independant vulnerability tests performed.

I’ll be adding more security best practices and lessons learned over the coming months.

Test early.  Test often.  Be happy.

Getting to <5 minutes is not a trivial task; in this time, we need to:

1. DETECT the outage
2. ALERT personnel (assuming there is no automated recovery capability in place)
3. Allow time for personnel to DIAGNOSE the cause of the issue.
4. FIX the problem through some set of corrective actions.
5. VERIFY the system works properly.

Clearly there is no time to waste, so each step has to be traversed as efficiently as possible.  Here are some general best practices on how to reach this goal:

1)  DETECT:  the system needs to have both internal (push) and external (pull, polling) monitoring in place.  The internal checks need to be tuned to identify any condition (usually exceptions) that indicate that the system is sick or about to be sick.  The system needs a way to push this information to a monitoring console or alerting system.  Likewise, an independent, continuously polling/monitoring system should be in place that sends test events or transactions through the system every few seconds; this external monitor also needs a pay to post alerts.

Time: < 5 seconds after the issue

2) ALERT: when the system detects a potentially customer affecting issue, an alert must immediately be sent to support personnel.  In the cases where there is a 24×7 NOC, this should not be an issue (though some thought needs to be put into preventing alert-storms).  Where there is no NOC in place, paging, email, SMS, automated call, etc. should be explored, and additional tools will need to be developed for remote trouble resolution.

Time: 30-90 seconds

3) DIAGNOSE: this is usually the “make or break” step- the responding technician needs to have all of the relevant information “at their fingertips”, with perhaps a small amount of digging, in order to quickly identify the cause of the problem.  Logs typically don’t cut it… by the time the technician has identified the affected server, logged into the host, and reviewed (potentially millions of lines of) the logs, the five minute window will long have been history.  Don’t get me wrong- logs are great for debugging (when you have time), forensics, and as a last resort, but they should NOT be the first line of defense for time-critical outages.

The best tool I’ve used/experienced is a lightweight, extensible central monitoring console.  Typically a small webapp, the console will display the status of all running servers and applications using the appropriate APIs for the environment in question (for Unix/Java, SNMP + JMX is a good start).  These APIs can be used to pull information and effect changes as needed.  Besides environmental data, business data is often useful to have as well (e.g. number of “sessions” in progress, etc).  If you need to push information to console, consider using publish/subscribe (e.g. JMS).  Finally, the console can also provide access to recent logs, and should certainly provide access to recent errors.

Ideally, the technician should be able to look at the console and immediately (color, etc) see which host / app is in distress and then drill down into that host / app to view additional diagnostics that should quickly identify the root cause (which may, in some cases, be in a downstream component).

Time: 60-120 seconds

4)  FIX:  at this point the action(s) required should be clear; now the only trick is making sure the technicians have the tools and procedures required to take the actions.  More often than not, “fix” entails bouncing and app or a server.  In some cases it’s more complex- perhaps there’s data corruption that needs to be cleaned up, for example.  Regardless, wherever possible, have scripts that can be executed safely but easily, perhaps even from the monitoring console.  This increases speed and removes opportunities for mistakes.

A key success factor here is “defensive” component design.  If your system is multiple component executables (e.g. app server and database), make sure that (a) the failure of any one component doesn’t cascade through the entire system, and (b) that each component will automatically “reconnect” to the failed component once it’s fixed.  The last thing you want to do is have to shut down the entire system and execute a slow sequential startup just because a single component failed.

Time: 60-90 seconds

5) VERIFY: once the fix has been applied, verify the system now works as expected and that customers are not being impacted; if there is still a problem, go back to #3.

The list above is admittedly high-level- each system has it’s own challenges and complexities that need to be factored in.  There are also many commercial and open-source third party components that either should or must be taken into account.  But regardless of the final toolset the five above steps should provide a framework and acid-test for the integrated design.

A final note: effective outage resolution capabilities, like most RASS (Reliability Availability, Scalability and Security) elements, fall into the “don’t try this at home” category… there is no substitute for hard-won battle scars earned from developing and operating similar systems.  Do what you can to get someone like that on your team, at the very least in a coaching role.

If you’ve been in the technology field for any length of time, you witnessed some variation of this logic:

1. The company has a budget X for a certain project/product.
2. We need as many people as we can get for that budget.
3. Let’s avoid top talent and instead target the market average (or below) to get as many people as possible.
4. Let’s apply the latest hot methodology to produce a great product (a topic for another day).

In other words, I’m going to create a team of C players but expect A results.  C + C = A.

The major breakdown here is, of course, #2: the assumption that quantity begets productivity.  While that is true in some cases (you can’t build a massive, multi-million-lines-of-code enterprise system in a reasonable timeframe with 2-3 sharp developers), it is not true when you measure across quality levels.  Ten C-players will probably not out-produce 5 A-players.  In fact, they might not even outproduce 1 A-player.  If the problem is complex, the C-players may never deliver an acceptable solution.

I’ve seen many variations of these cases play out in the real world.  I’ve seen a large development team shed 1/3 of it’s (bottom ranking) staff and improve productivity by 50% or more.  I’ve seen a small group of A-players produce an order of magnitude (yes, 10X) more features in a given time period than a larger group of C-players.  But most of all, I’ve seen teams of C players produce lots and lots of C results, followed by inevitable surprise and confusion at the management level.

So if it’s clear that, in many or most cases, net productivity improves when you go up-talent, why don’t companies do so?  In my experience, the number one reason is concerns about cost (actual or perceived).  Paying 20-40% more for an A player (vs. a C player) seems prohibitive on a person-by-person basis, but once you look at the ratio of productivity to cost at the team level, it quickly becomes apparent that going up-talent is probably a net cost savings.  If I had a budget of X, I’d much rather hire 6 A players than 10 C players, knowing full well that I’ll get a 2-3X productivity boost even with fewer bodies.

But management (especially CFOs and financial types) generally can’t get past the cost of the individual resources.  Large companies are often driven by the CFOs, who are by and large focussed on raising the top line and reducing the bottom line on a quarterly or annual basis; in their quest to squeeze out all costs, they get a little gassy when they see high-dollar individuals on the payroll.  In short they see top-performers as an expense item, and not as an investment in future success.  Great for the short run; not so good if you want to be competitive 3-5 years from now.

A more reasonable excuse is finding that talent- there are not many A players on the market, and the companies that have them keep them.  But it’s not impossible to move up-talent over time.  Companies that are brutally honest in their evaluations and ones that refuse to keep low-talent can move up-talent over time.  I believe it’s possible to bump up a full grade level within a year or two with some firm commitment.  But again, this is tough.  Companies and made of people, and people live by relationships, and relationships make it sometimes hard to do what is best for the team.  Likewise companies (and their management) like to think they’ve done a great job building a great team (like the old adage about 70% of people thinking of themselves as above average)… deep down many of these managers are resistant to going up-talent.

Another reason is “thrashing”, which results when C players produce a faulty, leaky system, and then massive amounts of manpower have to be thrown at the problems to keep them at bay.  In this case you are caught between a rock and hard place- you can’t shed too many bodies since you simply need numbers to spread them out over all the problems.  Once you’re at this point, good luck.  I’m sure you’ll spend many painful sleepless nights wishing you (or your predecessor) had invested in A players and avoided the problems altogether.

I’m running out of time for this entry, but suffice it to say that there are more reasons for the perpetuation of this insanity, and I see no evidence that it will stop (in aggregate).  That said, there are plenty of exceptions to the rule- teams that have invested in A-talent and are reaping the benefit; teams that are working hard to move up-talent; etc.  Kudos to these teams.  And if you’re on the wrong side of the insanity, just know that it’s never too late start moving up-talent.

By the way, there is a variation of this myth known as outsourcing, but we’ll hit that another day.

William asked me to lunch to catch up and to discuss some challenges he was having in a new position; not surprisingly, his style was creating some friction.  He was perplexed because, for the most part, he was “in the right”.  The new organization needed some serious discipline and procedural house cleaning, and William was doing everything in his power to make the changes as quickly as possible.

William was caught in the classic trap- he was “right”, but he wasn’t being “effective.”  I see this all the time, and I’m guilty more often than I’d like to admit.  When you know you’re right, it is very seductive to take what appears to be the shortest route, which is ofter to simply announce that “this is the way it will be, because I’m right” (or some variation).

But the truth is that being right is not enough if you actually want to make a difference.  Sun Tzu said it well: “In war, then, let your great object be victory, not lengthy campaigns.”  You can be “right” over and over and never get out of the campaign.  Being effective requires you to expand your definition of “victory.”  Your “great object” (i.e. goal) is to bring about some change, not to achieve a state of moral righteousness.

So how do you go from being “right” to being “effective”.  That’s more art than science, but here’s a few guidelines I try to keep in mind:

-  When possible, take some time socializing the idea.  Change is scary to many people, so allow for some time.  Once a critical mass of the group believes its a good idea, then it’s a done deal.

-  Get buy-in from the leadership (peers or superiors).

-  Create a sense of ownership; make it everyone’s idea, not just yours.

-  Listen.  Let people voice their objections or concerns.  Address them with respect.  Use them to beter the idea.

-  Let others fight the battle.  If you can convince a few “thought leaders” of your idea, let them run with it.  Don’t hesitate to give them credit (or even let them take credit).

-  Stay focussed on the real victory conditions.  It’s easy to get bogged down in tactical conflicts, but if you stay focussed on the end-game, you’ll be in a great position to compromise or even back down in areas that are not critical to getting the idea done.  ”Lose the battle but win the war” is a great tool to have in the toolbox.

-  Be willing to compromise and be patient.  If you can get 50% of your idea implemented without much fuss today vs. 100% with major push-back, I’d take the 50% and then go after the remainder in the next round.

-  Use Guerilla tactics (carefully).  It’s sometimes helpful to throw an idea on the table that is so unpleasant that your original idea looks positively great by comparison.  This works best if someone else floats the idea.

-  Last, but perhaps most important, lead by example.  If you can show personally or within your organization that your idea not only works but gives you and your team a competitive advantage, others will be highly motivated to get on board.

Here’s an example.  I’m finishing on an app for a nationwide company that decided a couple of weeks before the portal was to go live that it needed to support co-branding for all of their clients- basically, display a custom logo depending on who was logged in.  The right answer should have been “No!” since we were short on time and had limited resources available, but I had a hunch that Grails would make it easy enough to take on despite the risk.  So I branched the code (just in case), started a timer (literally) and dove into it.  I developed a “Brand” table that would store name, styles, and logos.  I developed full featured CRUD pages for admins to add, change, and delete brands.  I added the Brand section to the Admin section of the portal.  I modified the portal pages to show the brand logos when available.  And I developed a servlet (“controller” in Grails) to serve up images out of the database.  Grand total:  35 minutes and maybe 50 lines of handwritten code.  (To be fair, I should also add another 30-60 minutes for automated test development and testing.)  My guess is that this would have been a couple of days in any other framework.

If you’re a Java web developer do yourself a favor and have a look.  I think you won’t be disappointed.