We Were Developers

06 November 2012

I’m a sucker for epic movies. I especially enjoy the epics where regular people are put in extraordinary circumstances. Heroes with no super powers, or unlimited resources, are forced to face off against seemingly insurmountable odds. And after a long struggle, the heroes emerge victorious. These stories inspire me and drive me to be more like the heroes. But how can someone who writes code for a living aspire be a hero or even resemble the people in these movies? Most people see coding as a docile and individual activity occurring in a tranquil setting. There’s nothing epic about typing words on a keyboard…or is there?

Many enterprise software projects run much like the epic war movie We Were Soldiers. A company tasks a few battle hardened veterans to lead a whole bunch of young, inexperienced, and often delusional privates into a dangerous high risk situation. There’s incomplete information, constant churn, and careers hanging in the balance. Enterprise software development is jungle warfare. And jungle warfare requires heroes.

So what does a hero look like on a software development project? The Hollywood depiction of a rock star developer is the guy pulling all-nighters to write thousands of lines of cryptic code that leaves mere mortals awe struck and dumbfounded. Other developers are afraid to even touch the code assuming they are even smart enough to comprehend it. Sound about right? Well I’m going to challenge that perspective. Developers like the one described above do exist in our world, but are they heroes? Well… yes, but they’re more like the guy who sacrifices himself to cover a grenade early in the movie. In general you want to avoid creating situations where this type of hero emerges. Gen. George Patton said it best when he said: “The object of war is not to die for your country but to make the other guy die for his.” The type of hero described above often allows the development project to hit the date, but creates an unmaintainable mess for everyone else. The survivors of the grenade attack are happy to be alive, but they’re fighting a man down for the rest of the movie. Shit happens and if a grenade rolls by it’s important to have people in the group that are willing to jump on it. But I wouldn’t want to create a culture out of it, because after a while you’ll run out of people.

So let’s talk about the heroes that I’d like to see more of: the hero that confronts danger head on, inspires greatness all around them, and does what’s right instead of what’s easy. I call them Hands-on heroes.

Most of my favorite epic movie characters can be classified as Hands-on heroes: William Wallace (Braveheart), Maximus (Gladiator), and Hal Moore (We Were Soldiers). None of these men are above the fray. They stand side by side with their men and face-to-face with whatever adversity is in front of them. To bring this back to the world of software development, these are the men Fred Brooks refers to as “Thinker Doers”, the rarest breed in the software world. These are results driven leaders that have abandoned the pursuit of the ivory tower. Instead they embed within the team actually doing the work and contribute directly to the end result. For these folks failure is not an option.

The thought of putting leaders in the trenches flies in the face of most organizational structures. The best are supposed to rise to the ivory towers and scale their efforts out. They’re too valuable to put in harm’s way. However, the ivory tower removes leaders from the situation on the ground which distorts their reality. There is a difference between moving figurines around a map and giving orders while bullets are whizzing by your head. Just like there is a difference between moving people around on a spreadsheet verses dealing with individuals face to face when allocating work. The argument could be made the other way that the ivory tower allows people to think more strategically by removing the person from the situation. It’s true that it can be harder to make rational decisions on the ground especially you’re close enough to the people for them to pull on your emotional strings. But is the purely rational decision always the best one?

Jonathan Haidt makes some very interesting points related to decision making in his book The Righteous Mind. Haidt cites extensive research demonstrating that strategic reasoning follows intuition. So the person in the ivory tower is still going to first follow their intuition, but without having a relationship with the people impacted by the decision there’s not going to be much weight given to the decision’s effect on the people involved. Therefore the decision appears to be reached more rationally. The problem with this mindset is that not weighing the emotional response to decision making (even in a purely technical one) carries some consequences. Why can’t we just consider the emotional response as another piece of data? It’s easy to try to distill technical problem solving down to a problem of reason. However, as Gerald Weinberg states in The Secrets of Consulting “it’s always a people problem”. So decision making without proper consideration of the “people problem” is likely not going to lead to an optimal decision. These problems are best understood by the Hands-on heroes since they’re on the ground floor. It’s on the Hands-on hero not to give too much weight to the personal reactions. The overall success of the project is still the top factor.

Another staple of epic movies is not only the hero stepping up their level, but inspiring people around them to go above and beyond what they believe possible. Organizations that are under constant pressure to create innovative technical products or services require technical expertise. And unless they’re willing to pay a lot of money to consultants and expert hires they need to figure out how to grow new technical people internally. Hands-on heroes don’t buy the people around them, they grow them. In Gladiator, Maximus is surrounded by a group of fellow slaves that function as a small army by the conclusion of the movie. Some people are self-driven and will grow themselves without direction. Others need a little push. Both types can benefit from the leadership of Hands-on heroes.

For many new developers, understanding where to start can be challenging. Pair-programming is a hands-on technique that can be used to allow a team member to absorb knowledge by watching how a more experienced person solves a problem. This also allows the hands-on leader to observe the other person’s work strategies to suggest improvement areas. On the other side self-motivated technical people can get stuck in the mentality that they have to learn everything that’s out there. This can lead to the creation of a technophile. Wikipedia defines this type of person slightly different than I do (http://en.wikipedia.org/wiki/Technophilia ). I define a technophile as someone who learns a bunch of technologies for the sake of knowing them, but never really connects the dots to build platforms. A hands-on leader can differentiate a platform built on buzz words from a platform built on complementary frameworks. Complementary frameworks allow developers to focus on the domain problem instead of technical problems; buzz word platforms allow people’s resumes to grow. Hands-on heroes inspire people to grow skills that complement each other. Skills must be learned to serve a purpose and those skills must be constantly improved. Teddy Roosevelt was another Hands-on hero, he articulated this idea quite well when he said, “Power undirected by high purpose spells calamity; and high purpose by itself is utterly useless if the power to put it into effect is lacking.”

Finally hands-on heroes believe in karma. They don’t do the easy thing; they do the right thing. Doing the right thing is often a learned behavior; many times it’s because they’ve seen where the easy path has led. It would have been easier for William Wallace to admit treason to the English and receive a quick death, but the legacy of his quest for independence would have been tainted. For developers, it’s easy to deliver a project without unit tests if you know you get to walk away at the end. It’s easy to write methods with cyclonic complexity over 20. It’s easy to Ctrl-c and Ctrl-v. But if you stick around your community long enough, it’s a matter of when (not if) it comes back to bite you. Code has karma, bad code perpetuates itself. But so does good code. Anyone that’s walked on to a project that has 80% code coverage knows the guilt you feel when you add a class without writing tests for it. Hands-on heroes know that the things they produce are meant to live on well past their time on the project. They understand that doing the right thing perpetuates others doing the right thing. They give others something to aspire to and leave a legacy worth remembering.

Delivering great software can be epic. As in all epics, heroes must emerge for the good guys to win. The current trend in business today is to remove the heroes from the fray and put them in ivory towers. The intent is to better scale out the hero’s talents, but often this only causes their efforts to be diluted. Battles can be won or lost in the trenches. It’s important to ensure that the right people are in these positions. Great software requires Hands-on heroes.

Full Contact Web Development

31 January 2012

As the super bowl nears you can’t avoid constant bombardment of advertising, predictions and analysis leading up to the game. You can either fight it or embrace it. I personally love the sport so today I am embracing it in the context of web development.

Football like web development is driven by a number of specialist chasing the same goal. Each specialist bring unique talents to the table to achieve success and weakness at any position or bad overall chemistry can lead to failure. So lets take a look at some of these positions and see how far we can take this analogy…​

First your engineers are your offensive line. Typical of offensive linemen they do a lot of the heavy lifting and often don’t receive a lot of attention. And for many that’s just fine. You might have some rockstars here or there but success is measured by how well the unit as a whole performs rather than individual performance. Case and point a rockstar programmer’s elegant but confusing design might weigh down the team instead of lifting it up. Most projects are too large for your single rockstar to handle themselves so unless others can work with the solutions they develop the team is often better off without them. Unfortunately engineers like linemen have difficulty tracing their work directly to outcomes. Often a job well done just enables another person to do there job upon which the project’s success hinges. However on the flip side a poorly done job can blow the whole project up. Having a solid engineering team is important to enable the rest of the team to do their job. Like the best units in the NFL good engineering teams are experienced, communicate, and they understand what the guy next to them is doing (cross trained).

Your User Experience and Front End Development teams are your wide receivers. These folks do work that is very visible and often takes a great deal of skill some of which is hard to quantify. Some people seem to have it and others don’t. Given their position they have the opportunity to make huge plays. They are given some freedom on the routes they take when the requirements change. But one of the issues of playing exclusively outside the hash is that they often don’t understand some of the other things going on in the game. There is sometimes conflict when they’re not getting thrown to enough. Sometimes the game plan calls for establishing an architecture that may prove some of their designs difficult to implement. However there are many receivers that do understand these things and are willing blockers and they work to understand all aspects of the game. There are even some UX/FED developers that dabble in engineering (and vice versa). These folks are the rare hybrids that I’d label a tight end. Like the best receivers good UX and FED enjoy the visibility of there work, they work to align to the strategy being employed in a given game and they almost never drop the ball.

Your Development Leads are your running backs. These are the folks that carry the load and run head first into problems all day long. There are some that do this with some flash and others that just grind it out. There is some glory at these positions but it’s also heavily criticized. A receiver might drop a few balls and still have a good game. However if a running back fumbles only once every few games it’s a black mark. This is also a position that can let success go to it’s head. Most of the great running backs understand that they are dependent on the rest of the team. It’s not surprising that the great Walter Payton was known to take teammates (especially his linemen) out to dinner after big games. Like great running backs the best leads spread the credit around.

Your PM and Delivery Managers are your Quarterbacks. These folks are calling the shots on the field and determining who needs to be where and when. They are judged on some statistics but at the end of the day it’s how many times did they win the big one. Although unlike football in development you often have more than one person in this position on a project. Just like in football this causes problems since there’s nothing like a good quarterback controversy to screw up team chemistry. One way to make this work is to separate the roles so that you’ve got quarterbacks by committee. But even that’s not a silver bullet. In the end the most successful approach is to get them to understand that they are all on the same team working towards the same set of goals. Easier said than done. The best quarterbacks are the ones that can spread the ball around and always end up with the win (even if it’s not pretty). Those are the types you want in those Project Manager and Delivery Manager roles.

Quality Analyst and Business Analysts comprise your defense. You might have the best offense group in the game but if you’re not delivering what your customers want you still end up losing. Great defenses are dynamic and can adapt to take away or contain different things that an offense may throw at it. On any given project there’s almost no way you can cover all the test cases or capture all the requirements but if you can determine what the most important ones are you’re usually going to be just fine. Good analysts need to be relentless, have great instincts, and sometimes even a little unorthodox. From my experiences there are a lot of places that have good serviceable analysts however very few that have truly great ones.

Finally Operations is your special teams unit. Unfortunately these are the guys everyone forgets about because it seems like their job is automatic…​ until something goes wrong. Operations teams need to be extremely discipline and need to be composed of unique individuals. You don’t find a lot of people willing to run full speed with a ball with 11 other guys running full speed at them. Just like you won’t find many people that are able to fix production defects at 3 AM while the rest of the development team is asleep. Ops teams often have very specialized skills that are not always found in other positions in development teams. Good operations folks are selfless, a little crazy, but when then time comes they deliver (almost automatically).

I hope you’ve enjoyed my ode to the super bowl. Unfortunately I will once again be watching a Bear-less Super Bowl. But I can always root against the Patriots. Go Giants!

Evil Software

01 December 2011

A shocking thing occurred to me the other day while I was reading Catching Fire by Suzanne Collins. I started thinking about ethics in an unexpected way. The book and it’s predecessor Hunger Games have themes that touch on a number of moral issues around killing and exploitation that are very troubling. As my mind drifted past those questions I asked myself could this really happen sometime in the future? I felt a gripping terror when I rationalized it as entirely possible. The most frightening aspect for me is the power of big brother watching your every action and the control they could have over everyday life. Much like in Nineteen Eighty-Four by George Orwell people with unprecedented access to your daily interactions have the ability to manipulate you and your peers. This type of control was never possible in the past. Kings could rule with an iron fist, people could be watched and controlled but only to a certain degree. Thanks to technology, mainly software, the rulers in the above stories are able monitor your every move. And right now as we speak in the real world, this type of technology is being developed and could soon become a reality. The potential impact of this technology is frightening.

As someone who writes software for a living I think it’s clear that we need to step back and think about these things. Gerald Weinberg writes about software ethics in the last chapter of his landmark book The Psychology of Computer Programming stating that software can be used for good and for evil. He mentions an example of how the computer has provided for great advancements in how we store, query, and represent information. However he points out that for all the good that has come from computers just think of how much more efficient the Nazis would be if they were able to use them to track people. Add a database with a couple table columns, an agile methodology, and they could track the velocity to wipe entire races of people off the map. Chilling…​

Today we have drone aircraft flying around all corners of the globe, facial recognition software, video cameras mounted in numerous places to do things from tracking motorist behavior to tracking terrorists. As free people we have to ask ourselves what controls are in place to protect us against these things if they fall into the wrong hands. What actions do we have to protect ourselves from misuse? What is the point where protection intrudes on our basic rights as people? Is my current project making things worse?

So what’s the point of all this? The point is that as a technologist you have a great responsibility to think of these things and ask yourself what your work is really amounting to. You have the power to change the world and become famous for it. But before you change the world have you asked yourself if the world you’re creating is one you’d like to live in?

Periodization for Developers

29 June 2011

Probably one of the last things a person may associate together is a software developer and a weightlifter (there are exceptions). However there are techniques in weight lifting that I believe can be applied to a software developer’s regime to enhance performance. I’m not talking about steroids although caffeine is probably the closest thing to steroids for a developer. I’m talking about periodization.

Periodization is a weight lifting system that’s purpose is to prevent a lifter’s strength gains from leveling off or plateauing. Plateauing often happens if a lifter uses the same training routine for an extended period of time (same exercises at the same targeted number of reps). The lifters muscles will adapt to the routine causing strength gains to level off. Therefore strength experts recommend periodic mixing the routine up every month or so to prevent adaptation and achieve peak performance. For example the book Complete Conditioning for Football by Michael J. Arthur and Bryan L. Bailey recommends alternating between routines focusing on increasing muscle size consisting of exercises with 10 reps for 3 sets with 1 min rest increments and muscle strengthening exercises consisting of 5 reps for 3 sets at higher weights followed by a peak phase of sets of 4, 3, and 2 reps each at very high weights just prior to the start of camp. The transitions between periods are often pretty rough as muscles need to adapt to the addition loads or intensities. So you can count on being pretty sore for the first week or so. However despite the discomfort this is type of program is backed by extensive research to enhance performance by allowing a lifter to experience more continuous improvements in strength. But could these same principals be applied to mental activities?

After a couple of days of doing any repetitive cognitive activity you are going to get bored and your desire to do that activity will decrease. Likewise so will your performance doing that activity. For a software developer this could be writing SQL queries, working on front end gui components, or name your favorite framework here that dictates that you code in a specific way. In time these activities become boring or we simply stop getting better at doing them… we plateau. Trying a new framework or area of the system to develop on is often painful at first as everything is new and you can’t make assumptions about how things work. However after some time you start to learn and things get easier. But we didn’t become a stronger developer we just increase our breadth of knowledge right? Well not always. Say you’re a hot shot SOA developer that can code up web services and database persistence in your sleep. However even though you’re good at what you do every assignment you get starts looking the same. But you look at the guy in the cube next door who does front end work which looks interesting. So you ask your boss if you can take a 3 – 4 month stint doing front end development with browser related technologies like HTML and JavaScript. You gain proficiency in the technology and soon you can code up a web page from scratch. Then you return to doing the SOA stuff. Well after doing all that front end coding you realize that the service that you’ve just started designing could be structured in a way that the front end developer could plug the data right into the GUI without having to manipulate it first. Well now due to your new knowledge of front end development you’ve actually become a better SOA developer. You have broken through a plateau.

So how can you implement such a program? First its sounds like a hard sell to take someone who’s a rock star at one thing and put them in foreign part of the system. Who’s going to cover for the SOA rock star while he’s out experimenting in GUI land? Sometimes it’s surprising to see who steps up when the resident rock star is out. In the end the organization benefits from improved performance from its people as well as some knowledge redundancy that can come in handy if the department experiences some churn. So in those terms it’s really a win-win. What are the appropriate period lengths? As a general rule it takes roughly 3 – 4 months working with a technology on a daily basis to get a sense of it. I’m talking in the general sense not in a mastery sort of way but clearly beyond the hello world stage. This is generally when most developers can begin to apply the technology without needing guidance from someone already competent in the technology. This is however the level of understanding that gets you the most bang for your buck. Having a general understanding of a technology allows you to be conversational with other developers and incorporate that knowledge into your designs of a complimentary technology effectively.

So be wary when you think you’re at your peak. You could just be standing on one of many plateaus. Change things up or you might find your development muscles have atrophied!

The Black Swans of Software Engineering

21 June 2011

I happen to be rereading The Black Swan by Nassim Taleb and started to thinking about the black swans that I run into in the world of software. Software embodies the essence of the black swan which consists of the following:

The event is a surprise (to the observer). The event has a major impact. After its first recording, the event is rationalized by hindsight, as if it could have been expected (e.g., the relevant data were available but not accounted for).

Bugs are always a surprise to a developer. Only an evil developer would purposefully introduce bugs to software that was knowingly going to be shipped. Because of this software must proceed through rigorous testing prior to shipping to attempt to eliminate these surprises. Often prior to such cycles a manager asks the developer "Is this code complete?" to which a confident developer will eventually reply "Yes". Subsequently testing commences and to the developers surprise the next morning a list of bugs appear in his/her inbox or worse the testing reveals the application is completely untestable. Often the later is caused by some minor configuration that was missed when deploying the application in the testing environment. This satifies the second requirement of the surprise event having a major impact. In software a missing charactor can devastate an entire program. Compiles and unit testing frameworks eliminate some of these more obvious bugs but there are still the bugs that only emerge in the harshest conditions such as a heavy load or concurrent access. These are difficult to test and often hard to fix. Additionally because software is so scalable (you can just keep selling the some piece of code over and over with very little cost) a bug found after shipping needs to be fixed perhaps on millions of individual machines. Finally after the bug is revealed or fixed how many times has a developer quickly rationalized that this event obviously occurred because of someone else’s work or an unexpected use case. If they had known these things the software would have been coded correctly. In fact often fixing one bug introduces another (Fred Brooks estimates this at 50% on the higher end) so these after the fact explanations often do not hold weight.

Software development is always striking a delicate balance between enhancing productivity and making systems understandable. AOP is one example of a programming technique that has significant power to append code to various parts of a system based on rules. This technique can provide productivity gains to an individual coder however it can also introduce black swans in your software. One of the top uses of AOP is to adding logging throughout the system. This is successful because logging does not in anyway modify the state of the system so appending code that just records what has happened is relatively harmless. But what if a clever program devises AOP rules that modify the state of the system. For example updating a required field to all objects of a certain type when called by a method that starts with "update". From on aspect the design is elegant and clean since the code is "doing something for free" for other developers. The problem is other developers may not understand these rules or even know that these things are happening. So if a developer goes to update that field within there method and upon exiting the method is has not changed they may not have the understanding of the system to be able to debug. Magic has happened which is a very uncomfortable feeling in the mind of a developer.

A second example is Java is concurrency. Java’s memory model is implemented to optimize the speed at which concurrent events can happen on say a website like eBay. The problem is developers tend to assume that concurrency is handled for them so when 2 users bid on the same item at the same time the JVM knows what to do. And it does but for the sake of speed Java sometimes has multiple copies of an object in memory and unless you tell it to publish the result to all of those copies as is necessary for concurrent programs to work "weird things" start to happen. Another black swan.

Finally there is the case of the JDK upgrade in which there is potential for the foundations of the language to change. Suddenly performance tricks implemented in the old version no longer work. Memory model changes may expose lurking race conditions that were always there but just never came up. And on the flip side certain bugs and performance issues may even vanish due to fixes to the underlying core code. All of these likely make software development equal if not more black swan prone that even financial markets. Some of these are unavoidable but others can be mitigated. Fred Brooks has so far been proven correct in that order of magnitude increases in software development productivity is hard to come by. Though he is really talking about the process of creating software and not exclusively the tools we use. I think we need to take inventory of the tools we use and ask our selves the question: "Where am I creating black swans in my code?"

Working With Black Boxes

31 March 2011

I hate black boxes. For folks that don’t know what I’m talking about imagine a gumball machine made entirely of steal (no glass). The first time you put a quarter in and the first time you get a gumball. Wanting 2 more gumballs you put in 2 quarters in and out pops a hairball. Frustated the next time you put a dime in, kick it for good measure and out spouts a cheeseball. Blackboxes are frustrating and risky in software development for the same reasons. You input a value in order to obtain an expected outcome then when it doesn’t work or even worse works sporatically you’re left scratching your head to what to do next. You read the documentation but the thing is not working as you expect it to.

For a business this is huge risk since you never really know if you’ve covered all the scenerios. Producing production grade software is like walking through a mine field. Untested code fragments are mines. With closed source you really have no choice but to send folks in with blind folds on. You have no idea if you’re going to prod some live rounds in the ground. With open source it is possible to use code coverage tools to measure how much of the code you’ve run thorugh. So in that case you’re still in a mine field but you’ve got a metal detector to guide you along the way. But even with open source there are always cases were the environment is interfering or perhaps something is going on with the hardware (plastic mines!). Even stuff we think we know is often built on top of black boxes.

I’ve had a couple extremely frustrating experiances working with black boxes. The first was trying to build a message board with Microsoft Frontpage back in 2000. They had this great interface that allowed you to generate a message board with a wizard and embed it into a page. I added it to my fraternity’s site and quickly decided I wanted to tweak the thing a bit. I was looking through the code tinkering with config parameters for hours. Finially i discovered that all this "stuff" was going into an executable that completely mystified me. I was completely stuck. That’s probably what drove me away from Microsoft to Java when it came to web development. When I started out a lot of libraries were still black boxes to me. I hadn’t learned enough about the language yet. As I became a more seasoned developer I learned that if a couple quick searches on message boards didn’t solve the problem then next best thing was to just pull down the source or even use a decompiler to see what made the library tick. It’s a huge security blanket dealing with open source software since you can always pull back the curtain and see what the Wizard (pun intended) is doing. And that got me pretty comfortable for a while until I met a new black box …​ SiteMinder.

SiteMinder and I had our first encounter in 2009 but it wasn’t until I got into consulting last year that I saw how much of black box the product really is. SiteMinder is a security product that protects websites from unwanted visitors by blocking pages that user should not have access to. The package comes with it’s own templating file type known as FCC’s that allows a programmer to create custom pages to collect login credentials and change passwords. It also comes with an SDK that allows you to send commands to it using Java. And though it uses open source web servers (Apache) to render the FCC files and contains a published JavaDoc for the sdk the core program is all proprietary and poorly documented. Black box. The company that I was consulting at for the job hired another consultant that specialized in the software. This is a common strategy. If you have a black box hire somone who’s used it before. For common tasks this worked quite well. Things that this consultant had done previously happend very quickly with very few problems. The issues came when we started doing things the consultant had not previously encountered. His first reaction "Hey the samples that came with the black box worked so it must be because you’ve deviated from it" or "Since you’re doing it slightly different that I’ve done it on previous projects that must be the issue". When I pressed for an explaination why most of the time I could not get a straight answer. This led to a great deal of frustration and finger pointing. In some cases I found another way that was closer to the consultant’s previous experiances to get things to work. Other times I complied a number of experiments to defend my team’s code and a few times he discovered some new secret configuration option that solved the problem instantly.

So what did I learn from all this? The only way to understand a black box is either to find a way to open it or through rigorous experimentation. Decompilers and open source software are great things to combat black boxes. However more than likely at some point in your career you’ll run into them. And frankly get used to it. The world is just one more black box waiting to be openned.

A couple of parting words of advice dealing with black boxes:

Start with your Intuition

Experiance trumps Intuition

Experimentation trumps Experiance


November 2012

January 2012

December 2011

June 2011

March 2011

Older posts are available in the archive.