Developing in Fits and Starts

In biology, the punctuated equilibrium theory suggests that evolution happens rapidly, followed by long periods of stagnation.  Organisms change and speciate, and come to an equilibrium for a very long time.

Today, coding made me think that development is a punctuated equilibrium, too.

I was working in a fairly large codebase with some deep dark corners.  Nothing far off what we all do; nothing special at all.  I was having trouble getting all the details into my meatspace, but was managing to get things done, code was progressing.

And then I hit a wall of "what the what?"

Happens to me from time to time, when a little detail will stymie me for five minutes, ten minutes, a half hour. Where you twist your head sideways to look at the code and wonder what the heck the code is doing.

I ran into a few of these today.  I figured them all out, but they each shut me down for longer than I would want to admit to anyone:

• Where the heck is that css parameter getting set?  I can see it in the browser's "inspect element", but I can't see where it's coming from in the code.  Not in the HTML, certainly, and not in the CSS.  Oh, there it is, right there in the TypeScript file!
• Why the heck is this object serializing to JSON differently than to string?  Does the ServiceStack.Text call the ToString() method under the scenes when serializing objects?  Is the "override ToString" code section here followed by ToJson() example a typo, or is that how it is supposed to work (My legacy object has a ToString() and a ToJson() method on it, and the service stack uses the latter.  Makes sense.
• Where is the bug in my KnockoutJs code?  I'm sure I've forgotten a set of parenthesis somewhere, but I can't for the life of me find it.  Oh there's the little bugger.  Oh yeah, when I want to do a visible check against an observable property, I can check it directly, but when negating, I have to invoke it as a function.  A known property of the library, but darnit if I don't forget that sometimes.

Coding along happily, then boom, stuck in the mud.  Trudge, trudge, trudge, finally get out and get back in the rhythm, then bam, back in the mud.

Is my solution evolving?  Is that why developing happens in fits and starts?

Paying Down Your Technical Debt

Technical debt is bothering me lately.  Spurred on by this article, I was doing a lot of thinking about how to pay back some of my technical debt.

A little scenario background, as I observe it in Corporate IT shops.  In the type of shop I'm thinking of, IT builds applications for the business, and doesn't always get to set the deadlines, creating a scenario in which potentially weak-willed IT managers cut corners to meet increasing business manager pressure to "get things done."  What that results in is a lot of manual maintenance; for example, data updates to production systems by IT where a user interface would put both the responsibility and the ability in the hands of the users.

And this goes on for years, until someone looks out over the IT group and proclaims that the IT group is too big for the organization, or that it’s not producing enough value for the business units.  And both may actually be true.  Because over the years, IT keeps building applications, and each of those applications has a little bit of this manual maintenance associated.  Eventually the maintenance cost - which is the interest payment on the technical debt – becomes all the IT department can do, since the business wants to remain flat in staffing.

When they get to this point, many businesses look for the big kill.  “Outsource the lot of it,” they might say.  “Let’s spin up a program to rewrite everything that's currently wrong and costing us maintenance work,” is another approach.  They spin up a big expensive effort to fix the mess they find themselves in.  It’s a form of declaring technical bankruptcy, or at best an attempt to pay off the largest pieces of technical debt first (in the guise of “bang for the buck”).

I've think that both these approaches are bad.  They are often big and risky.  The big efforts often fail, collapsing under their own weight, or they might involve a huge, expensive bandwidth increase in the form of consultants who don’t know the company history, why the bad decisions were made, and what dark puddles of sticky ichor lie in wait in the legacy codebase(s).

A while back I paid off a lot of my personal financial debt using the snowball method of debt reduction.  That method suggests you pay off your smallest debts first while paying only the interest on the other debts.  If you have a couple credit cards, a couple student loans, a couple car payments, and a mortgage, don’t start by paying off your mortgage first.  Pay that $500 credit card bill off first.  That will give you some momentum and cut out some of the debt as well. 

In the case of IT projects then, this method would suggest that you stop development to the extent you can on all but the smallest creator of maintenance noise.  Do something (give the user the power to fix the errors, automate something manual, perform automated cross-checks) that eliminates that noise.  Then look for the next smallest piece of maintenance noise that can be silenced, and roll the resources from not doing the manual maintenance into fixing the next noisy thing. 

Soon, your whole team will be working on paying down some seriously large debt.  Especially since you’ll be able to see more clearly through the noise (if you have ten noisy applications, it’s hard to work on one, but if you have three noisy applications, it’s much easier to concentrate without loss due to context-switching).

Another mechanism for paying back real financial bills is to pay off the highest interest obligations first.  That is sound advice, and in the IT world, that means work first on the highest maintenance/smallest effort work to get better bang for your buck.  That’s a fine approach as well.  Often, however, you can’t always get permission to work on this type of effort.  Or naysayers say, “Well, we should just add in a little more effort, and then we’ll quiet down things more.” And the scope creeps and creeps and then the project gets too big to get done (remember, we need to work on little things first because we only have staff to pay down very minor debts).

In general, though, just don’t go for the biggest project because it will allow you to quiet the most things down (don’t pay the mortgage first).  Do whatever it takes to get momentum.  Refuse to participate in manual processes.  Be adamant about the need for change. 

Be adamant.  Be the change.

Software Archaeology

What is a Software Archaeologist?

Most of us that work in corporate Information Technology shops are Software Archaeologists. Every organization I’ve seen and many I’ve talked to has a built-up layer of legacy applications that are old and in disrepair. While many of these applications are used daily, it may have been years since they were opened, rebuilt, refactored. The source code may be missing, and the original authors may be long gone from the organization, taking their knowledge of the application with them.

It’s this organizational brain-drain that turns many of us in corporate IT into Software Archaeologists. Every time a developer has to don his leather fedora and dive into an ancient codebase with little to guide him but a few cryptic comments littered in the code tomb, and a document that some long-ago intern put together in a half hour to meet a letter-of-the-law requirement in the then-current SDLC, that developer becomes an archaeologist.

Being an archaeologist means digging through all the various eras of coding.  Being able to parse the Scatter-Gather patterns in Visual FoxPro, say, or to unwind the data calls from the business objects that were once part of the popular CSLA methodology - those become part of your everyday toolkit.  What about that legacy web services architecture that passes around typed data sets everywhere?  What about that .NET 1.1 Remoting service layer you built?  You need to know it all, and to hop back and forth through the disparate layers, gathering golden icons and dodging rolling boulders all the way.

Even those developers who only develop new applications need to be versed in Software Archaeology, because nowadays many internally developed applications are on end-of-life platforms or hardware, being scoped for moving to the cloud, or are being revisited as part of Business Process Reengineering.  These developers need to get back into recently rewritten code and re-envision.  You have to know how to dig.

As a Software Archaeologist it pays to know not only today’s architecture, but those of the past, especially those that were popular in the organization around the time the code was written. But not limited to the standard architectures of the day. Organizations hire people who come in to try their hand at something new, or bring in some flavor-of-the-day architecture, tool, library. As a result, one-off applications are going to be unearthed. As architectures become archeological layers in the binary geological strata of an organization, so do architects become archaeologists.

I have a deep appreciation for all my colleagues who perform software archaeology on a daily basis.  Those whose jobs are just to maintain those applications that time has forgotten.

I respect the dig.