CodeBetter.Com
CodeBetter.Com
RSS 2.0 via Feedburner
           Do you Twitter? Follow us @CodeBetter

Jeremy D. Miller -- The Shade Tree Developer

Under the hood and working with .Net, TDD, Software Design, and Agile Stuff

June 2005 - Posts

  • Create new entry points to enable unit testing of Legacy code

    If you've been reading the Shade Tree Developer this week you'll know that I'm fighting with a legacy component. I was trying to get a piece of code into NUnit to test my new functionality. A class in the stack works by giving it a key value to a record in the database. I tried for awhile to use an existing stored procedure to push in test data. Guess what I found? I had relational integrity issues with setting up the test data, but so what, that's just what a database is. I wasted a bit of time pushing past the referential integrity by getting the correct data and stumble right into security authorization being performed inside the f***ing stored procedure! It was about this time I remembered what I had been reading in the Feathers book on legacy code. Sometimes you have to change the code just to get it into the test harness. A quick "CTRL-ALT-M" action with ReSharper, and voila, an entry point into the business logic that bypasses the database and security checks. Add some ObjectMother magic to construct the fugly data structures, and I've got a working unit-ish test. Moral of the story? Don't be afraid to change the code first before writing a test (well, maybe afraid but not terrified). Remembering to backup and look for an easier way to write a test was probably a good idea too.
  • Hulk Angry When He No Have ReSharper

    I'm a ReSharper junkie, and I get angry when I don't have it. I've spent the majority of my .Net career working with converted Java developers. To a man, they all complained about the crudeness of Visual Studio.Net compared to IntelliJ or even Eclipse. I wrote them off as whiners because I was used to working in either VB6 or Visual InterDev (I still have a soft spot in my heart for doing what's now called Ajax development in VI) and VS.Net was clearly more polished (except for that little code and continue goodness). Then the ReSharper 1.0 beta builds dropped. My project team of ex-IntelliJ jockeys jumped at it immediately. Even with the sometimes spectacular instability of the ReSharper beta, it made an obvious improvement in coding efficiency. I know this because several times I started to code on the flight home without a working copy of ReSharper and became frustrated quickly when the code navigation and refactoring was missing. ReSharper quickly improved with each passing build, and the phrase "I'm having a ReSharper moment" left the project vocabulary. Why am I complaining about this today? I'm wrestling with a legacy C# codebase this month and it's not entirely "ReSharper-able." One of Michael Feather's pieces of advice in his book Working Effectively with Legacy Code is to lean on the compiler when making changes to legacy code. Going farther, I really want to be able to use ReSharper's "Find Usages" capabilities to trace code in an attempt to understand interdependencies between classes and trace through code. In order to insert any kind of unit tests I'm having to extract methods, classes, and interfaces. The safest and quickest way to do this by far is to utilize the automated refactorings in ReSharper. Out of the box the code could not be traversed by the static code analysis of ReSharper. The reasons? The solution files aren't using project references between projects. They are compiling all assemblies to a common hard-coded folder and making all references to this folder, creating a force field impervious to ReSharper code navigation and refactoring. This was largely done to get around problems they had with circular references between class libraries. A lot of classes are loaded and executed by reflection. Part of the problem is that the class library projects are much more fine-grained than they really need to be. I'm guessing that the assemblies were subdivided so granularly to apportion out the work between developers and avoid checkout conflicts with Visual SourceSafe's pessimistic locking, but I don't know that. So far we're only tackling a small portion of the legacy code at first. The long range steps we're trying first are:
    1. Get to project references immediately. Enable both ReSharper refactoring across projects and the almighty task in NAnt. It also solves some reference conflicts
    2. Collapse the assembly structure to mitigate the circular dependencies and simplify deployment and build
    3. Eliminate as much of the load by reflection instantiations as possible. Minus automated unit testing, strong typing is absolutely essential. Since we don't have an automated NUnit test suite to speak of, we have to have the compiler enforce strong typing as a first line of defense when it's time to modify the code
    4. Get rid of VSS, move to Subversion, and get some automated builds put together for continuous integration
    5. Automate the migration of build products (think shared libraries, frameworks, and staged builds) between our different codebases. This is a soft spot in our infrastructure at the moment that is slowing us down. We're certainly not the first team to struggle with this by any means.

    Another thing I highly recommend; my partner is an uber-buildmaster and one of the first adopters of the original CruiseControl. Fellows like him are awfully nice to have around when you're in this sort of situation. I have played with IntelliJ briefly when I thought I was getting a chance to go onto a J2EE web project. For the agile style of incremental development (red bar, green bar, refactor), IntelliJ is a coder's paradise. I think this largely due to JetBrains dogfooding their own tool using an XP process. I know pockets of MS use agile processes (and I'm dubious about counting MSF Agile in that column), but I don't think the VS team really takes it seriously. VSTS is clearly aimed at the heavyweight Rational Unified Process product. VS 2005 with VSTS is a definite improvement for agile development, but I think it still falls short of what I've observed in IntelliJ or Eclipse. I'm not entirely sure that some of the enhancements like unit testing integration are actually as good as the existing open source equivalents, and the refactoring in VS 2005 is inferior to ReSharper. I'm honestly more excited for the upcoming ReSharper 2.0 release than I am for the improvements with VSTS.

  • Legacy Code and the Life and Death of an Application

    My colleague and I are making minor extensions to some legacy code (in C#) for the next month or so (and that generally implies that the Shade Tree Developer will feature quite a bit of ranting in the near future). After that my team will be making some large scale architectural changes to an existing product. My job for the rest of the year is going to revolve around legacy code. We very frequently make customer driven modifications to our application. We also have to keep the product ahead of the competition, and that means being agile (in the literal sense) with our development practices. Our existing codebase has some definite weak spots. The development organization has the beginnings of a long term "Grand Unified Theory" to transmogrify our existing systems and development environment to extend the company's investment in the current products and prevent the downward spiral into "legacy code." Rewriting the applications is completely out of the question. So first, here's the symptoms we're trying to prevent and eliminate. As we get farther into the "Grand Unified Theory," I'll blog about our strategies for getting out of the legacy code trap.

    Life and Death of an Application

    In my finite experience and research, applications (or services if you're of the SOA persuasion) reach the end of their useful lifespan when the cost of altering or extending an application is greater or riskier than throwing it away and starting over. I've seen a lot of people make a parallel to the concept of entropy. The natural state of any system will tend towards entropy (in software terms, the big ball of mud). You have to exert energy into an application to reverse or stave off application rot (entropy). Most applications undergo a twilight period where the previously shiny, well-factored application descends into a morass of spaghetti code that becomes more and more difficult to work with. The applications are still providing business value, but they are starting to be more of an opportunity cost rather than an enabler because of the increasing friction incurred by further modifying the application. This twilight slide into retirement is what most people mean when they say “Legacy Code.” The code works, but no one wants to touch it for fear of bringing the down the whole thing.

    What is Legacy Code?

    Here's a couple of takes: "Before we get started, do you know what 'Legacy Code' means? It’s code that works" -- via Roy Osherove. Yeah, but this is true only if you leave it alone. Michael Feathers described legacy code as "code without tests." What I think he’s driving for is code that can be safely modified and rapidly verified.

    I'm going to add a corollary to Mr. Feather's definition; legacy code without tests is code that is difficult to test. Twice in the last year I’ve transitioned from greenfield development projects that were written with TDD to working with brownfield code that had not been written with TDD. In almost startling contrast, the test-first code was vastly easier to extend with new unit tests than the code written test-last. To paraphrase the noted development sage Ferris Bueller, “the legacy code was so tightly coupled that if you put a chunk of coal between the classes you would get a diamond.” What made the legacy code made writing unit tests difficult?

    • Overuse of static methods. Static methods result in tight coupling. I ranted about this here. Keeping state in static fields makes it even worse.
    • Tight coupling to external configuration. It’s often impossible to run any piece of the code without an external configuration file. Dealing with configuration slows down unit testing. I’ll blog more on this soon, specifically ways to avoid the coupling.
    • Poorly factored code and tight coupling between concerns. Database, business logic, and presentation logic all mixed together.
    • Blobs

    Legacy Code isn’t Necessarily Old

    Of course, many applications aren't well-factored to begin with -- instant legacy code. Entropy also creeps in with any gap in communication or participation between the designer’s vision and the development team. I’m dealing with one of those right now. The original vision and design was thoughtful and innovative, but it was thrown over the wall to a team of developers who had not participated in the original vision. It’s particularly bad when the original prototype is coded by a skilled architect and then handed off to the developers with instructions to “just finish it off.” One of the advantages to continuous design is that the rough edges are smoothed down over the course of development. Remove the designer from the coding and those rough edges will only become sharper over time. Shamefully, I think I’ve been responsible for a bit of this myself. One way or another the technical vision needs to be socialized and shared throughout the development team if you want to create a consistently structured application. Ideally, the development team should be creating the technical vision themselves.

    The Development Environment Matters Too

    Since much, if not most, of the cost of modifying an existing application is verification and regression testing, the ability to quickly migrate the application to a testing environment is paramount. Repeatable automated testing is such a great long term investment in an application that it’s a shame so few projects invest upfront in test automation. At this point in my career, I would consider development infrastructure like build scripts, database scripts, and continuous integration servers to be a vital part of the application itself. If that infrastructure is lacking or nonexistent, the application is that much harder to modify. I'm working with legacy code right now, and the lack of an automated development build with unit tests is making me feel, well, like I'm naked in a hailstorm. The worst case I know of was a very large VB6 application that had been accreted around a nice tidy core over the course of several years. Simply setting up a test environment manually could take up to 10 business days, and even then they were never very confidant in the installation. That application was a core piece of the manufacturing system and changed often. Not only was the migration slow and hard, but the haphazard nature of the environmental control allowed several pieces of bad code to get through testing and into production, leading to factory downtime (and take it from me, factory downtime is a bad, bad thing. When the billionaire tycoon founder himself comes looking for someone’s head in IT, run away and get some plausible deniability). The real gallows humor was that the application was customized somewhat for each factory and business line and therefore had to be regression tested for each and every configuration.

  • Taming the Legacy Beast - Dose It With Some NAnt

    Yesterday I griped about legacy code. I carped about VSS, whined about the tight coupling, groused about the arcane configuration setup, and spewed invective about the stupid manual post compilation steps it took just to make the code work. Today it's time to start cleaning things up and applying some ibuprofen to the legacy headache.

    Even though the work we're doing is supposed to be throwaway in a couple of months, we decided to "NAnt-ify" the development environment. First we're skimming the VSS project and setting up a new Subversion repository. The next step is to create a bare bones NAnt build to automate the application setup. The last step is a simple CruiseControl.Net configuration to get it into our existing continuous integration server (this is sooooo much easier than it was just a couple of years ago. Hat's off to the CC.NET guys).

    If you can't easily make a repeatable build and get the application working on your own box in short order, you're not gonna go anywhere fast. That whole NAnt/CC.NET helps keep the code clean in source control so your coworkers aren't idle while you go hunt down the new class file you forgot to add. And one last time, VSS is a piece of crap. Using the far superior clients for Subversion (or almost anything else) that are actually intelligent enough to determine what files have been changed, added, or deleted makes for a smoother source control experience. And no, we're not going to wait until November for the promise of Hatteras in VSTS when Subversion is here now and free.
  • You know it's a bad day when...

    You're typing "iisreset" or "net stop iisadmin /y" a lot.
  • "#region/#endregion" Tags Smell Bad

    Ranting ahead...

    I'm unilaterally declaring overuse of the "#region / #endregion" blocks in VS.NET a code smell. When I first started playing with VS.NET in 2001 I thought this feature was going to be the best thing since sliced bread. Boy, was I wrong. It seems like every time I expand an innocuous looking region out spills a big bag of entirely unrelated code.

    Much like the excessive comments code smell, the existence of a large code region generally implies that the code in the region really should be in a different method or even a different class. The region block is no substitute for good factoring. The region block is often used as just a bit of perfume sprayed over smelly code.

    If you want what I would call a bad example, go take a look at the source code of the original application blocks from Microsoft Patterns and Practices. Instead of factoring functionality, they wrote more or less procedural code segregated by region blocks that were nested 4-5 layers deep. From what I understand, the code style was mandated by their internal QA code standards. The result is thoroughly unreadable code. Finding the code that does anything is like playing a game of hide and go seek in a thick forest.

    The .Net community's tendency to take anything written or published from MS as the gospel without any kind of critical analysis is a huge pet peeve of mine. The original application blocks were a good example. The code was rotten and they were reportedly buggy in spots -- but I saw nary a word of criticism anywhere.
  • Flimsy evidence that agile delivers the goods...

    Does Agile development actually work in real life? Here's anecdotal proof in the affirmative (since the XP Refactored book sold a bunch of copies mostly based on negative anecdotes I figure I can get away with this too).

    Production Moves in Ceremonial Waterfall

    Production moves in big IT waterfall land are living nightmares of pure desperation and panic just waiting for the other shoe to drop. We usually had most of the team work an entire weekend in shifts just to be on hand as code and data was manually migrated. It's not unusual to see the cots come out in project warrooms. I don't think that's what the Godfather gang have in mind when they "take it to the mattresses," and the mafia seemed to have more fun with it than we did. As I've mentioned before, teams often made last minute coding changes right before go live because integration problems weren't discovered until making the migration and everybody was afraid to miss the installation window.

    Agile Deployment

    Shifting attention back to the shiny agile world of Test Driven Development and Continuous Integration, we made a final push into a production staging environment with our new product yesterday (with no known defects I might add). After yesterday's move, I left work on time and took my toddler to the YMCA pool with nary a cell phone at hand because I was perfectly confident everything was fine (I'll probably pay for that this morning).

    On a previous project last year for a previous employer we made our second release one Tuesday morning. I was quite happily working on a user story in the mainline code branch for the following release when the project manager came up to tell us the new release had been put into production without a hitch. The developers had completely forgotten about the production move!

    Where I Think Agile Development Helps

    So why do I think the production moves went so much better with agile development than my earlier projects in waterfall world (to be fair, the earlier projects were COM based with a fair amount of middleware and the later projects were .NET systems)? The waterfall shop had a admirably organized (but radically inconsistent) ceremonial change management process, but pitiful build management. I think most of the reasons I list below are applicable for any disciplined iterative and incremental process (I'm thinking RUP here), but agile processes put more emphasis on early and frequent feedback over almost anything else.

    1. Earlier, more frequent, and repeatable testing. Test Driven Development is really a design technique as much as anything, but it certainly creates a large body of automated unit tests that provide a safety net to keep new bugs from emerging. Add any kind of automated acceptance or regression test suite and that safety net gets much tighter.
    2. Continuous Integration. CI can be thought of as a lot of trial runs at automated code installation. Much the same way that TDD forces you to write code that is testable, maintaining an automated CI build forces you (if your thinking cap is on straight) to think hard about making your application or service easier to deploy and version. Some people will excoriate me for this statement, but I think a NAnt build is the best documentation for system installation because it unambiguously specifies what has to happen for the application to function (and it tends to stay synchronized with reality easier than a document). I'd still supplement that with a Wiki page though.
    3. Defects can be fixed faster and stay fixed. CI allows a team to push code fixes to testing much faster without sacrificing environmental control. In ceremonial waterfall land, code pushes required filling out forms and convincing an external group to make my code move first, often up to a six hour process if other projects got there first. By automating builds under the control of the project team, I think the control over the environment is much better even without a controlling environment authority outside the development team, and certainly much faster. When you do write an automated test for a defect as you fix it, the defect stays fixed. Lost time due to regression failures can be mitigated by automated testing.
    4. Project teams are, or should be, more self-reliant and self-organizing. What I mean is that a project team should be able to carry out its own configuration management without waiting for a sluggish external group. In agile projects and organizations I've consistently had fewer dependencies on external IT service teams. I think that goes a long way towards reducing project friction. It does require carrying a broader skillset on a project team and people willing to step outside their normal skill silo. I think that might be one of the largest barriers to agile adoption in big shops. Flame away on point number 4, I probably deserve it.
    Our organization has started down a path of using the WiX tool to create installers as part of our NAnt/CruiseControl.Net build. Using this, our testers will now be able to work off the automated installation in testing environments at all times, in effect testing the application and installation at the same time. I know several other organizations in the blogosphere are trying this, too. I'm only peripherally involved, but I'm excited for the potential. More updates when I know more.
  • I detest Visual SourceSafe

    Ranting ahead, obviously...

    All our new projects are in bright, shiny Subversion repositories and humming along just fine. I've got about a month of detour through existing products in Visual SourceSafe and I'm already aggravated on the first day. Don't get me wrong, every source control system has some sort of problem, but VSS is the absolute worst.

    The client is a piece of crap; it falls over and corrupts in a swift gale, it's slow, and hangs during CC.NET builds almost as often as not. Branching and merging with VSS is a joke. In a world where Subversion and CVS are free and reliable with multiple clients and integration into IDE's, and SourceGear Vault is inexpensive, why is VSS still so popular in .Net shops?

    I've played quite a bit with the vss2svn perl script from Tigris.org. I have no idea if it works or not because VSS keeps crapping out on corrupted history trails. VSS won't even behave long enough to get rid of it.

    One thing I'd really like to see from the .Net community as a whole is to stop taking everything from Microsoft without question. There is a whole world of development tools and knowledge out there that doesn't originate in Redmond.

    How's this for scary though, in a former job we used VSS under the dark of night on the side because the officially mandated source control system (CC/Harvest ) was even worse.
  • Thinking about Developer and Tester Interaction

    The following is essentially a lecture and reminder to myself and my team, I apologize in advance for being preachy.

    We're wrapping up a release candidate build this morning and quite naturally the subject of testing and bug fixing is on my mind. Because of resource constraints (i.e. not enough testers) we were not able to do much testing (acceptance testing by real testers, not just TDD) within iterations. Doing the old-fashioned (and stupid) big bang, waterfall testing cycle at the end under schedule duress is always tricksome.

    It is absolutely imperative that the developer/tester interaction be as smooth as possible. Everybody needs to be focused on a common goal of finding and removing defects from the code, and that means paying attention to what the other half of the equation needs. Verbal abuse or general incivility from either side needs to be smacked down.

    If you're going to do any kind of Agile development, one of the first and hardest lessons to learn is that the role boundaries must be blurred, or at least have much more transparency between project roles. You can't bury your head in coding and designing, you have to actively cooperate with testing on test automation and code migration between development and testing environments.

    What Developers Owe Testers

    1. Don't waste the tester's time with buggy code. At a minimum, unit testing and either pairing rotation or code review should stop the majority of the problems from reaching the testers. If you can get acceptance tests to the developers before coding is "complete," all the better still.
    2. When you fix a bug, try hard to write an automated test that validates the exact conditions of the defect. At a bare minimum, exercise the code end-to-end to ensure the bug is really fixed. Don't assume the defect is really fixed because you dealt with the root cause (I'm currently wearing some egg on my face from this one).
    3. If anything is blocking the testers, stop what you're doing and get them working again. The application or even a user story cannot be considered complete until it is tested. If the testers are sitting on their hands because a testing environment is invalid or the application is down, your project is down. Again, a reminder to my team and I because we dropped the ball on this one day a week or so back.

    What Testers Owe Developers

    The singular thing we developers really need from the testers is just communication. When a defect is found, we have to understand a couple of things.
    1. A description of exactly what the error is
    2. Exactly how to reproduce the error
    3. The desired outcome of an operation in clear, unambiguous language
    4. Contextual information if it exists. Stack traces, audit trails, input data
    5. Severity of the defect. Is the defect stopping the tester from testing any farther?

    The worst thing a tester can do is just say "It doesn't work" without any context or adequate description of the desired outcome (or my colleague's ultimate pet peeve, "it says NullReferenceException"). I had an incident a couple years ago with an analyst slash tester. She had given us the original requirements for a data exchange between two systems that wasn't much more than a big SQL statement in a stored procedure (mildly modified from the previous system we were replacing). When she was testing the exchange, she kept coming back to us and telling us the results were wrong. Her SQL script she was using to test the exchange was giving different results. I think at some point I said something like "!~@#* it, let's just use your SQL" so we could move on.

    My favorite tester used to show me all defects at the tester workstation and walk me through the problems before his team would log a defect. That helped dramatically. It obviously helps if the testing team is colocated with the developers.

    Testers are Pigs

    Made you look, didn't I? All I mean is that the testers should be an active participant in the standup meetings and the project as a whole. One of the agile practices I like is to have a bug report discussed every single morning in the standup, even if it's just to say there are no bugs. It's a great way to ensure the developers are getting the testers what they need and vice versa. Bug tracking tools are fine and dandy (except ours is terrible), but nothing beats face to face communication.

    Here's the original story about the derivation of Pigs and Chickens if you've never heard the phrase.

  • Steve Hebert article on Code-Talking

    Code Talking: The key to great software?

    A great article. I admit that I don't particularly enjoy pair programming, but Steve Hebert describes one of the best benefits of pairing. Collective code ownership is an awfully good thing in many ways. I've certainly observed that my stress level is generally less on projects that have some degree of collective ownership.
  • A classic article from Alistair Cockburn and horror stories just for fun

    http://alistair.cockburn.us/crystal/articles/cpanfocisd/characterizingpeopleasnonlinear.html

    This one has been around awhile, but it's still a worthwhile (re-)read. It's full of radical ideas like:
    • Individuals are important
    • People aren't "Plug'n Play" components
    • Productivity can be maximized by making it easy for the project team to communicate
    • Pay attention to the team's environment

    If you're in anything like the environment I describe below, you can try using Cockburn's writing in an effort to reform your workplace. If you have any influence over your workplace, heed Mr. Cockburn's writing now rather than learning the hard way.

    This Stuff Matters

    A couple years ago I got to jump straight from pure insanity to a high caliber agile shop. An oranges to oranges comparison of effectiveness between the Microsoft Solutions Framework process at the first shop and Extreme Programming in the second is useless because too many other factors were different (stronger teams, more supportive organizations, less customer involvement), but the simple ideas of:

    1. Sitting near all of my teamates in an open room
    2. Everybody focused on only the one project
    3. Developers, testers, and analysts on the project at the same time!

    were an enormous improvement in my mind.

    Project Team Insanity

    I used Cockburn's chart on the effectiveness of various communication channels in a presentation several years ago trying to convince my then management to move towards iterative and more collaborative processes. At the time the entire organization was in a "matrix" organization structured by functional group first, and project teams second. Cubicle assignment was strictly by discipline (development, architects, analysts, testers, etc.) because management was tired of reorganizing teams and paying moving costs to colocate teams. Everybody was working on multiple projects at a time so that all projects had equal access to specialized knowledge and skillsets. I had the distinctly unpleasant experience of simultaneously working on the rump end of a death march project as a developer (later cancelled), and trying to lead the development of a separate project.

    The first project was an early Web Services project using a custom in-house SOAP-like framework developed by the internal integration architecture team (if you ever hear a sentence like that, run away!). The lead developer/project manager was in his first lead role and had no prior lead or architecture experience. There were five distinct methods for the service, and five developers, so naturally each developer was assigned a method and told to go off and code it. I failed in my attempt to escape the assignment and began to code my assignment (throwing out the unworkable design that was handed to me. Win32 API and COM calls from within XSLT transforms? No thank you).

    Early on, I asked the lead where the development database was located and was told each developer was just using their own box. Come to find out, every single developer had created or reused a different database schema structure to hold the same logical data. Two developers were using an existing Oracle schema structure (the database was overworked and we had an opportunity to move some load off), and the other three had made their own Sql Server databases. Two of us had created fully functional, specialized rules engines that did roughly the same thing, but radically different in implementation. My first action as a newly minted Systems Architect (make finger quotes for the full effect) was to blow the whistle on the technical weaknesses to my new boss. The project was fortunately abandoned and replaced by a different effort (2+ years in MSF Planning!). Integration testing at the very end was apparently pretty interesting;) I sincerely think the project could have succeeded if the developers had been working in a common location instead of in a complete vacuum. Just accidentally overhearing conversations would have tipped us off that our code could be shared between service methods and certainly would have got us on a common database early. Of course, the fact that every developer was actively plotting to get off the project and the organization went through a massive reorganization at the same time didn't help either.

    The second project was an extremely complex system with three other developers, all of whom sat in different wings of the floor, and were working on other projects at the same time. At no time during the course of the project was the requirements analyst present on the team at the same time I was. All communication was through a requirements document that was prepared when I was on the other project (that turned out to be incomplete and/or flatout wrong). Come to find out late in the game, there was a second requirements specification no one in development knew about. We pulled it off by working way too many hours, but I'll never do a project like that again. The saving grace was very good involvement from the business partners, otherwise it would have bombed. The system is undeniably successful in production (one of my friends won an award from the business for it), but it was a long time before I enjoyed software development again. I'm pretty sure the CMM crowd would call this an unrepeatable process.

    Of course it didn't help that I over-engineered the application a bit, but I'm making fun of other people here, dammit! I read somewhere that the worst designer is a fellow on his second project trying to use every idea he had from the first system (editor's note, it's from Fred Brooks in the "Mythical Man Month."  Thanks to Darrell for the head's up). Guilty as charged. I'd also just read the GoF book for the first time and got a little patterns happy. They should put a warning label on that thing.

    My former employer has since quietly removed the worst of the organizational madness and is simultaneously working toward climbing the CMM ladder and tentatively introducing some basic best practices like automated builds and better unit testing. All I can say is good luck to you guys, and it'll be worth the effort (maybe not the CMM part, though).

    I think it's fun to laugh at the absurdity of past projects, as long as you don't repeat them. Bad examples can be more illustrative than good examples sometimes.

  • The Worst Possible Way to use a Stored Procedure

    Rant time...

    Repeat after me please,

    I will not put any business logic into a stored procedure.
    I will not put any business logic into a stored procedure.
    I will not put any business logic into a stored procedure.
    I will not put any business logic into a stored procedure.
    I will not put any business logic into a stored procedure.

    Okay, if your business logic can be expressed easiest through a declarative SQL WHERE clause, a SPROC can be a cool way to go. Set-based logic is almost always simpler to do with SQL than mucking through with procedural C#, but that's a different rant for another day, and I'm definitely talking about procedural code within sproc's here.

    Question: How do I know if I might be doing something in a sproc that I shouldn't be doing? Answer: If there is *anything* other than CRUD in your sproc (IF/ELSE/ENDIF, LOOP/END LOOP, etc.).

    The all time dumbest thing to do is to split your business logic between a SPROC and the consuming middle tier code. Twice now in the past 12 months I've bumped into cases where business logic is performed inside a sproc, then interpreted by middle tier C# or VB6 code. Splitting the logic into multiple spots like this makes a system harder to understand. It also makes a system brittle to change because some logic is duplicated in both places. A change in the sproc or C# can, and usually does, break the other. This is a case of the Don't Repeat Yourself principle the Pragmatic Programmer guys talk about.

    I had an interesting experience as a consultant one time. I was fresh into the project working with some legacy C# code (yes, there is already legacy .NET code). I had just stumbled on the fact that most of the business logic was really in T-SQL procedures. In many cases, the C# code undid some of the T-SQL transformations to filter the results further. The client architect was giving us a bit of a tongue-lashing ("you consultants better not write crappy code, and there better be tests for everything you write") while I was looking at a 5,000 line stored procedure, with his name all over the comments, trying to decide if some erroneous data was coming out of the database or being transformed in the C# code. This particular system is the subject of a case study/testimonial on MSDN as an example of all the wonderful things about .NET;)

    "Thank you for listening, I feel better now." -- Roy Moore
  • Learning Lessons -- Can You Make Mistakes at Work?

    Of course you make mistakes, that's not really what I meant. Can you comfortably admit any type of fault in your workplace without undue fear of reprisal? Do you trust your teammates and management enough to discuss problems openly or do you sweep problems under the rug? After all, we know that admitting a problem is the first step to recovery.

    A Healthy Environment

    Last week my team did a project retrospective on our last four months effort. Half of our team is new to the company and our team is distributed between Austin and Houston, so there were definitely some rough spots to sand down. We came up with a list of problem areas with our execution and some lost opportunities. It was actually a very positive experience, because everyone involved was focused on improving our execution instead of pointing fingers or deflecting blame. Never once did I feel defensive or sense any kind of defensiveness in anyone else in the room.

    We've already started to implement some improvements in our iteration and project management. Of course we still have to follow through and correct the problems, but at least the team got the issues out into the open and everybody is aware. I've heard the phrase "Lessons Observed" in the past to describe teams that don't act upon their lessons.

    The Blame Culture

    Much of my IT career has been in a shop where too much energy was wasted on blame avoidance and "CYA". We didn't learn any lessons or ever make any corrections because no one ever wanted to admit any kind of failing. It was more or less a waterfall shop (and still is) and management stressed a predictable project date and resource count over everything else. The worst sin was missing a deadline. The result? Sacrifice quality or outrageously pad estimates. I routinely saw projects that were coding right up to the production install because everybody was afraid to miss the arbitrary date. Very scary.

    External compliance teams are the worst. They aren't involved in your project at all, but they get the last say on the success of the project by putting out a metrics laden scorecard. One of the primary measurements is a bug count read out loud in a nasty meeting at the end of the project. The typical response was to hide bugs. Developers would quietly negotiate with testers to fix bugs without reporting them. That's an awful practice, but it seemed more like survival at the time.

    Now they're climbing up the CMM ladder (I think they're about to be level 3). It turns out CMM is a great work avoidance tool. You need what? Did you fill out the correct paperwork? Sorry, I can't do that until I have the TPS form signed and approved.

    Avoid the Embarrassment Meeting

    Be careful how you use metrics. Metrics reveal information and trends about your effort, but they're never the entire story. Avoid the "embarrassment Meeting" described by Steve Maguire. Misuse of metrics to punish people leads to stupid behavior that doesn't contribute to the project.

    An agile process alone isn't enough. I worked on an XP-ish project with an absolutely miserable jerk of a project manager that routinely abused the development team with metrics. The developers varied wildly in skillset, and we were struggling. Every iteration close meeting featured a full review of exactly how the development team missed their estimates on a story by story basis. One time he went away to a conference for several days. Coincidentally the developers had what we all considered to be our best iteration. We completed stories on time, we got a lot of refactoring done that had been put off, and we even managed to make some overdue improvements to the NAnt build. The project manager came back from his conference, looked at *his* metrics, and loudly declared that we still weren't doing any better. I wanted to punch him.

    An iteration later, I excitedly showed him a story that was finished ahead of the estimate. His response? We had sandbagged the estimate in the first place.

    After some external intervention from higher management, we made some adjustments to our estimating and planning process that lowered the misery on the project considerably. It wasn't pleasant, but the mistakes were recognized and corrected. A healthier team would have made the correction faster though. I sincerely wish I had done something to alleviate the problem instead of fuming, but I didn't feel comfortable in the environment and didn't feel like I had a voice in the project.

    To state the obvious, that was a dysfunctional environment. Martin Fowler has some pointed comments on the usage of metrics here.

    Does Agile Development Help?

    I want to say yes, but the real answer is "it depends." I think this is really more of a people issue than a process or organization problem. An iterative process is definitely advantageous because it gives you so many more chances to reflect on your project and make adjustments mid-stream. However, if you don't trust your team or feel comfortable making yourself heard this opportunity will be wasted. A team with poor chemistry is probably going to fail no matter what else is in place.

    One of the tenets of agile development is the self-organizing team. It is everybody's responsibility to continuously improve the team's work, not just the project manager. Being empowered to make changes in your process means you're responsible for the process. Don't ever assume anyone else knows something you don't. If you think your team is doing something wrong, say so, no matter what position you have on the team.

    Collective code ownership does help in this regard by removing some psychological barriers to code inspection. Classic code reviews can be nasty affairs. A row of accusers sitting across the table from a solo defendant is the image that comes to mind. With true collective ownership, the team can calmly talk knowledgeably about improving "our" code instead of pointing fingers to "his" code.
  • Things to Learn

    "Life moves pretty fast. If you don't stop and look around once in a while, you could miss it." -- Ferris Bueller

    The art of software development moves fast. We're still a young discipline compared to engineering and we're still learning new and better ways to create software. Last year I sharpened my saw by adding mock objects, dependency injection/inversion of control, WinForms, and "BuildMaster" skills to my toolset. I spent a ludicrous amount of time on StructureMap though and let other new skill development and learning slide.

    Here's the things I want to learn more about this year:


    • Ruby and Ruby on Rails. I'm still a little dubious about the dynamic typing, but too many people are effusive about Ruby's potential to ignore this very much longer. If nothing else, Ruby should grow into the language of choice for scripts and what I call "NAnt-ey" tasks.
    • Ajax. I did a lot of IE specific Ajax development several years ago (before Google named the concept) and had quite a bit of success with it (a mission critical supply chain application and a dispute resolution workflow system, both still running). At the time it wasn't really a good platform choice because of the lack of developer familiarity with the techniques (plus IE5 had *huge* memory leaks with heavy DHTML manipulation). Because of the new hype from the Google apps and cross browser testing tools for DHTML like JSUnit and Selenium, Ajax development is looking like a viable choice now. I'll blog more on my experiences with this now that it is actually relevant.
    • Acceptance testing tools like NFit and Selenium. I love the concepts, but I think the NFit implementation could be better.
    • Finally finish reading Domain Driven Development
    • NHibernate. When .NET first came out in beta in 2002, I had dreams of creating the ultimate O/R mapper for the platform (use emitting for the mapping and use parameterized sql for better performance than NHibernate's purely reflective approach). I even wrote an article for C# Today on using reflection for O/R mapping. Alas, I didn't have the skillset then and I don't have the ambition today. Especially when NHibernate is there with an active development community and poised to become the de facto standard for O/R mapping in .NET. I know they have their proponents, but I don't count persistence generation tools like Neo, LLBGen, or Codus as O/R mapping.
    • Aspect Oriented Programming. I'd love to add some runtime AOP support to StructureMap.

    My StructureMap tool actually started its life as an O/R and data access layer tool. What's actually there on SourceForge is just a refinement of what was supposed to be the configuration framework for the rest.

More Posts

This Blog

Syndication

News

All opinions expressed here constitute my (Jeremy D. Miller's) personal opinion, and do not necessarily represent the opinion of any other organization or person, including (but not limited to) my fellow employees, my employer, its clients or their agents.

About Me

"Best Of" Compendium

StructureMap (Dependency Injection for .Net)

StoryTeller (Supercharged Fit)

Build your own Cab

TestDriven

MVP