Last week at DevTeach I did a talk entitled "How does Design get done on an Agile Project?" In the talk I tried to explain and defend the way that we design software on Agile/Lean projects. Among the issues that I talked about were:
- Return on investment. How to wrest the maximum business value out of the development effort spent in delivery.
- Doing design continuously is more responsible than big upfront design.
- Delaying technical complexity and decision making until the proper time.
- Enabling the future by creating maintainable code rather than trying to anticipate the future
- You aren't gonna need it -- YAGNI! Why YAGNI is important and valid. How to call YAGNI today without eliminating tomorrow's efforts.
- How does continuous design as practiced by Agile teams affect the composition and roles of the development team?
- How the desire for continuous design affects our design choices
In order to jump straight into these subjects I used the following scenario taken from real life at one of my former employers shortly after I left.
A software team is building a new ASP.NET application with substantial business logic and processing. Do you:
- Create a web service for the business logic and make the web application call this web service - even if the web service is running in process with the web application.
- Build and deploy the business logic in process with the web application as a simple set of library calls.
The architect (a friend of mine) with oversight responsibilities was adamant that the team chose option #1. I wasn't there, but I would have vehemently argued for option #2 (partially just to argue with the architect truth be told). I do have a pretty good idea what that architect was thinking and the rationale behind his decision making. Let's take a look at his reasoning first, then talk about why I think option #1 is wrong and option #2 is the more responsible choice.
Why the Web Service?
The web application was the one and only consumer of the business logic and processing exposed by the web service today, but what if some other project needed that functionality tomorrow? This enterprise had any number of large scale enterprise projects get effectively derailed by the difficulty of integration with many of their legacy systems. In particular, the inventory system was notoriously hard for integration. Because the company was a large manufacturing company, almost every significant business automation flow had to touch the inventory system, so the difficulty with integration was a large drag across the enterprise.
The problem with the old inventory system was that all the business logic was tightly embedded into the user screens written in a proprietary language. When we did projects that required automated interaction with the inventory system we had two basic choices:
- Use the screen emulation software from the vendor to drive the UI automatically
- Rewrite the functionality of the screens in some other language, in this case PL/SQL stored procedures
Neither choice is very desirable. You could say that everybody involved with creating enterprise solutions there had some scar tissue is an understatement. To prevent this problem from happening with the new ASP.NET application, the architecture team was demanding that the business layer be exposed and called by web services that would be open to the rest of the world.
Some of the desire for web services was SOA utopia run amok. The central architecture team was making a very large bet on SOA throughout the enterprise. Their philosophy was to expose all business functionality and processing via SOAP services until they reached a critical mass where Business Processing Management and Business Activity Management would be possible. The infamous "build it (services), and they will come" SOA strategy. The SOA vision was compelling and set the architect's heads a-spinning with visions of grandeur and colored their thinking. But was it really paying off? Would the new services really be useful down the road?*
Why I think my old Architect Buddy was wrong
I understand the rationale behind my old architect buddy's reasoning, but I'll put forth a couple reasons why I think the web service approach was dead wrong.
It was more expensive. The simplest way to deliver the ASP.NET system was to develop the business logic as libraries that ran in process with the ASP.NET application. Think about that for a minute. If you built a web service you'd have:
- An extra set of build scripts for the second web application, and the build scripts would take longer to run. That's friction.
- Additional security infrastructure between the UI and the backend, especially if you want the web service to be publicly open
- Automated and manual testing between the UI and the web services would take longer to run than the corresponding in process solution. Don't discount how much drag that would introduce into development productivity.
- Extra code complexity with the web service contract and web service proxies
- More complicated deployment scenarios
In addition, I'd say that the usage of a web service would make continuous design harder. If I'm developing both layers in process I can handily evolve the interaction between the UI and the backend by using automated refactoring's with ReSharper. If I have that kind of "reversibility," I can shave off development time by making simpler assumptions now and add complexity if I absolutely have to later. With the web service I have to spend more time upfront getting the web service contract right because making evolutionary changes to the web service is harder than using ReSharper on the in process model. If the web services are really going to be publicly exposed for reuse, I have to spend even more time polishing the service contracts because once it's published I can't easily change my mind. My advice is to err on the side of reversibility as much as possible. In this particular case that means bypassing web services in favor of an in process model.
Reversibility is a real issue. Because a mistake in the web service contract is so much more difficult and costly to change once it's being used, the rightful tendency is to do more rigorous analysis upfront. That analysis is going to cost us time. If I know I can back out or modify a design later, I can do the simple thing to deliver value sooner. In the case of the web service with only once consumer, an Extreme Programmer would "call YAGNI" to permanently table the construction of the web service until it was absolutely necessary.
The Economic Rationale
All arguments have to be grounded in some sort of economic rationale, and this case is no different. Going with the in process model is more economically valuable for two reasons:
- Opportunity cost. The extra mechanical cost of building the web service can be applied instead to creating other features that will provide business value now. Delivering the web service now creates no immediate value.
- It potentially gets the business feature into production earlier. There isn't any return on investment until something is deployed. By deploying some features earlier we also start the ROI meter earlier.
- The extra complexity would slow down feature delivery throughout the project
One of the ways that an Agile team strives to deliver a good return on the investment is by eliminating waste. We don't need a web service so we don't build it. As simple as that. The way that I think about it is an analogy from manufacturing. Exercise "pull" design and architecture instead of "push" architecture. Don't build infrastructure because you think it'll be valuable later. Build only the infrastructure that you need for the business functionality on the table. When we try to get clever with upfront frameworks, abstractions, and infrastructure we can often miss. We're speculating on what's going to be valuable instead of working off of hard facts. Remember that no matter how good you are in predicting what you'll need, you're requirements can easily get changed underneath you. Sometimes the speculation will pay off and other times it'll make things worse. In the formulation of Extreme Programming I think that Kent Beck made an implicit assumption that the percentages are against speculative design paying off most of the time, so you should practice simplicity every time until forced to do more complex things as a way to be more efficient.
Tomorrow comes
You did things the way that I said to and built the business logic as simple libraries running in process with the ASP.NET application. Tomorrow is here and it turns out that some new initiative does indeed need access to our business logic and a web service seems to be the logical choice. The fears of our enterprise architect have been fully realized - or have they? Is the business logic hopelessly coupled to the ASP.NET application like he feared would happen, or is it perfectly economic to just expose a new web service on top of the existing business logic and use it as is? It depends on how we built the system.
Agile design isn't all dessert and no vegetables. We saved time and effort on the first project by forgoing the web service we didn't need then, but tomorrow always comes. The key to working iteratively and incrementally is following good design principles all the time. If we built the ASP.NET with a good separation of concerns so that the business logic is orthogonal to the user interface, we should be able to put a new headless web service right on top of the existing business logic. In effect, we just need to build a new remote facade for the business logic and expose this to the outside world.
But it's cheaper to build it now!
As a consultant in decidedly non-Agile clients I'm constantly fighting off the notion that it'll be cheaper to just "build it now while we're in there." It's a tempting thought and it's partially caused by people with overwhelming memories of bad code and painful regression testing experiences. I'm making the argument that the cost of building the web service should remain constant no matter when we choose to build it assuming that we're following good design principles (separation of concerns, orthogonality, Single Responsibility Principle, etc.).
Team Composition
As I've tried to convey, my old architect buddy was acting quite rationally according to what he knew and thought. He wouldn't have bought my argument about the web service being inexpensive to add later because he flat out didn't trust the development team to build the ASP.NET application with a good architecture. He probably had the design chops to get it done himself, but he wasn't on the project himself. By forcing the business logic into a web service he thought that he could mandate to the development team some semblance of separation of concerns. The actual development team needed some developers or architects dedicated to the project that had the ability to create a good design. It's not good enough to have a supervisory architect drop by for a couple hours at the beginning of the project. Technical design strength should be applied at the point of attack.
It would have been wrong anyway
There's one last reason not to build the web service upfront. It would have been wrong anyway. In the absence of any other concrete use cases for the web service we would have been either tailoring it to the ASP.NET client or making assumptions about how future callers would be using the web service. In my opinion, the chances of getting it wrong are far too high to make building the web service upfront all that valuable. The future projects that need to consume this business logic may need an entirely different usage than the original application. We're better off waiting until the actual needs arise in order to create the exact web service that we need. Building the web service now is acting far in advance of the Last Responsible Moment. Assuming a good design, we haven't eliminated the possibility of building the web service later and we'll know much more about the proper shape of the web service at a later date. There's no justifiable reason to build the web service now.
So how about we just do more analysis upfront so that we can make some educated guesses on the future usage? Now we're allowing ourselves to be vulnerable to analysis/paralysis. That extra analysis is going to delay the delivery of the initial functionality. And it'll still be wrong.
One other last thought about the Web Service
Let's say there was some perfectly valid reason for building the business logic into a web service from the very beginning. We've got a reusable web service! Cool! Except it isn't exactly what the next project needs. Part of the functionality is usable, but other parts need to be different. If the actual business logic is built granularly with the Single Responsibility Principle we can reuse the pieces that we need and write all new code for the new functionality. If the business logic was created as one big chunk and tightly coupled to the service contract of the web service, we're screwed. My only point here is that the design of the innards of the web service needs to follow the same design principles as we would in endeavoring to create a solid application structure. I think the SOA enthusiasts too often overlook this issue or somehow believe that a design is "loosely coupled" just because there's a well defined service boundary in there somewhere. To really be Agile in your design and delivery efforts, almost all production code needs to be built with solid design principles.
Anyway, that's my take on the scenario. What do you think? What did I miss? Wanna argue?
* Part of the web service craze was caused by the fact that all of us architect types had a provision in our yearly performance plan that we would be responsible for creating at least one new web service each year.