Get it in Gear - Google Gears

Scott has a nice piece on Google Gears - Google's recent move to bring local-storage and offline capabilities to web applications.

"Stunning move by Google today in the Rich Internet Application space. While most of us (myself included) are off debating Flash vs. Silverlight vs. Apollo vs. Whatever, Google introduces Google Gears...at technology all of the above (or none of the above) can utilize...

This is a huge move and is quite brilliant. In one seemingly innocuous move (and one tiny 700k (yes, 700K) download) Google is well positioned to get Google Docs, including Writely, Spreadsheet and Presentation, along with who knows what else, enabled for offline use. And the whole thing is Open Sourced via the New BSD License."

This is gonna get good.

Google Is On The Street

I could probably come up with some flimsy rationalization about how this post is dripping with insight into software architecture and how if you study it carefully and pay close attention, the secrets to unencumbered enterprise delivery will be unlocked.

But, I won't.

I'm just having fun with a recent addition to Google Maps that I stumbled into tonight. Turns out Maps has added "Street Views", a feature that allows you to view photographic representations of locations visited on the map. As of tonight, views are available for Denver, Las Vegas, Miami, New York, and San Francisco.

Click on the "Street View" button next to "Traffic", "Map", "Hybrid", etc, and a new blue outline appears on the streets where "Street View" is available. Click anywhere on an outlined street, and an icon is positioned at that spot on the map. A green arrow of sorts shows you the direction you're facing. For your viewing enjoyment, a widget pops up and shows a wide-angle image of the immediate area. Want to take in all the sights? Simply click and drag the image to rotate a full 360 degrees. Want to meander up and down the street? No problem. Just click the overlaid directional arrows.

Look really close, and you may find someone you know. I'm pretty sure I found a coworker in this picture of Times Square... Hey! I thought he was out sick today?!?

Of course, Google claims to take the potential privacy consideration very seriously. A link within the help screens for "Street View" encourages reporting images that are a "security" or "privacy" concern, along with those that may be inappropriate.

This could be cool. Suppose these images will be extensible? Available via the Google API? How about interactive images where useful info would pop up when you hover over certain properties in the image? For example, maybe a phone number and a coupon would be revealed when you hover over the Paris hotel in Vegas? What ideas can we come up with?

Real-World Enterprise Architecture

A couple good insights in this entry from Neil Ward-Dutton at IT-Director: Real-world Enterprise Architecture part I: journey vs destination

Focus on EA as a journey, rather than as a destination

"It's all too easy to focus on the technical nature of EA outputs; which bits of the Zachman Framework should we complete? Should we mandate that all our models use UML?… and so on. Now don't get me wrong, it's important to get a handle on the scope of your efforts, and try and create some consistency in what gets done—but these things are means to an end, not the end in itself."

Establish business commitment and understanding and buy-in (after all, it's not about technical pursuits)

"Where I see organisations spending a lot of time worrying about the format and scope of EA outputs and artefacts, often, perversely, it comes about because there's a lack of organisational ambition regarding the role and contribution of EA as a practice. The hole left by a lack of ambition here is often filled by huge technical ambition: 'let's model the world'. We all know what happens if you follow that road too far."

Spend your time with the stakeholders - not other architects

"For EA practice to have a valuable contribution, it has to be prepared to prioritise conversations with business people (and less so with other IT people) over conversations with other architects. Although that's not within the comfort zone of every architect, it's critical. Real architecture has to involve real stakeholder engagement, otherwise architecture is just design with a corner office."

Become bilingual in the languages of business and technology

"In the context of 21st Century business, the critical EA competency is the ability to drive shared language and multiparty understanding—and conversations."

Neil describes a perfectly rational approach to architecture. One that takes into account the often under-represented breadth of the architect's responsibilities and the underestimated importance of engaging the business in active and pragmatic dialog.

Hidden in Plain Sight...

Sanjay Dalal at Creativity and Innovation Driving Business speaks with Erich Joachimsthaler, author of the new book Hidden in Plain Sight: How to Find and Execute Your Company's Next Big Growth Strategy from Harvard Business School Press.

The book makes a case for the increasing importance of innovation in today's global economy and explains a number of barriers to innovation. It introduces the "Demand-first Innovation and Growth” (DIG) model that encourages an organized and deliberate approach to innovation.

In this interview, the author describes the DIG model like this:

The DIG model is a systematic, systemic and repeatable process to identify and execute innovation and growth strategies. It replaces the existing model of SAV or screwing around vigorously, sometimes also called the fuzzy front end of existing innovation models. In the fuzzy front end, one searches wildly for ideas that then can be put through the classic stage-gate process of new product development. In the DIG model, the focus is not on the product, it is on finding ways of creating a transformational change in consumers’ everyday life.

Also in this interview, the author offers the following essential guidance for companies looking to improve their innovation and growth strategies:

1. Innovation and growth is not a fuzzy process of screwing around vigorously (SAV) but can be a systematic process,
2. Innovation and growth is not something that happens in a department like R&D or product development – innovation and growth is a company-wide activity and only if you have a process can you also engage the entire organization,
3. Innovation and growth is not about products or solutions – it is about creating a transformational change in the way people live, work and play – and in order to achieve that, the innovation can be a product, a solution, a technology and new business model like at Netflix or no product at all. It could even just be a management innovation like brand management at BMW or a better supply chain management process.

This is intriguing to me from the standpoint of the impact the architect has in the innovation process. I just ordered the book, so more to come...

Service Design Principles (The Contract Story Continues)

Circling back to the theme of service definition and contracts, let's look at the principles that are important to keep in mind when designing "good" services. As with most design pursuits, there is rarely an "ideal" model or an absolute truth. It's frequently necessary to balance influences to achieve the most appropriate and valuable design. Striking the right balance requires good experience and the ability to apply effective judgment. Fortunately, there are a number of well-established principles and patterns that help encourage good decisions about services, interfaces, and relationships. Leveraging this shared perspective leads to better designs.

The principles discussed here build on the most fundamental characteristic of a quality service definition: explicit boundaries. Our service definitions must precisely define the interface syntax and semantics - what it looks like and how it behaves. The definition forms a "contract" - a commitment on the part of the service - that the specified interface and behavior will continue to be honored until which time the "contract" terms are explicitly modified.

The remaining design quality attributes build upon this contract foundation and make solutions easier to build, support, and enhance. We'll briefly define several of these attributes and consider their impact:

1. Coupling
2. Encapsulation
3. Autonomy
4. Orthogonality
5. Idempotency
6. Cohesion

Coupling

The Loose Coupling design goal encourages the design of an interface that makes as few assumptions as possible about the service consumer or other factors external to the component exposing the interface.

The objective is to encourage modular and flexible design by reducing the interdependencies among elements and reducing the risk that changes in one element will necessitate changes in another. Adhering to this principle reduces the overall complexity of the system and significantly enhances flexibility.

Encapsulation

The encapsulation principle motivates us to hide behavioral characteristics and data constructs behind the elements interface. In a well-encapsulated service, the implementation is clearly distinguished from the service contract and hidden from view (or concern). Strong encapsulation is an important tool for creating loosely-coupled designs.

As described, encapsulation is strongly aligned with "information hiding". Some have argued that these are distinct characteristics, emphasizing the role of encapsulation as one of keeping the data packaged with the operations that can be performed on that data. There is a fair argument to be made, but I tend to think of it as stated and emphasize the latter concern as a distinct principle called Autonomy (see below).

While applied in somewhat of a different context, network protocols and the OSI model provide an easy way to think about encapsulation. From Wikipedia:

In computer networking, "encapsulation" is to include data from an upper layer protocol into a lower layer protocol... This is a method of abstraction for networking by allowing different layers to add features/functionality.

The motivation in service design is the analogous to that described here for network protocol design. Allow a service (or any type of design element) to add features and functionality, replace or modify implementation of existing functionality, or in other ways make changes to the implementation without interfering with other dependent elements.

Autonomy

The American Heritage Dictionary defines the word autonomous with expressions such as the following:

• Self-government or the right of self-government; self-determination.
• Not controlled by others or by outside forces; independent:
• Self-directed

As it applies to service design, this principle is closely aligned with "separation of concerns" and emphasizes the responsibility and right of a service to assume full control and authority over the capabilities it delivers. This principle encourages a design whereby two factors are assumed and emphasized by the producer of a service:

• There is full authority to change any internal aspect of the service implementation, so long as the external contract is strictly maintained
• There is no authority to change the responsibilities of another service

In this environment, where each participating service is autonomous and each is able to evolve independently of the other, we realize a much greater ability to build, comprehend, and manage complex interdependent systems.

Orthogonality

A service exhibits good orthogonality when it's interface provides:

• one straightforward way to accomplish a given task
• no redundancy
• no side effects

Wikipedia describes it like this:

Orthogonality is a system design property facilitating feasibility and compactness of complex designs. Orthogonality guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system. The emergent behavior of a system consisting of components should be controlled strictly by formal definitions of its logic and not by side effects resulting from poor integration, i.e. non-orthogonal design of modules and interfaces. Orthogonality reduces testing and development time because it is easier to verify designs that neither cause side effects nor depend on them.

For example, a car has orthogonal components and controls (e.g. accelerating the vehicle does not influence anything else but the components involved exclusively with the acceleration function). On the other hand, a non-orthogonal design might have its steering influence its braking (e.g. Electronic Stability Control), or its speed tweak its suspension.

There's an amusing and effective description of orthogonality in Pragmatic Programmer.¹ I wont' go into the whole story, but Dave Thomas describes how the controls on a helicopter are anything but orthogonal (unlike the car mentioned in the Wikipedia description). If you adjust one of the controls to influence, say, the pitch, you must also adjust others to change things like the tail rotor and the throttle, or you won't achieve the desired results. Essentially, a change in one characteristic of the helicopter's flight has side-effects on others, and those side-effects must be anticipated and accounted for to obtain the desired behavior.

In designing systems, it's essential that we provide orthogonal controls that can be easily understood - by the consumers who will use the service and by the developers who must make future changes to its implementation. Orthogonal designs lead to significantly reduced development and test time, greater flexibility, and enhanced opportunities for reuse.

Idempotency

An idempotent design is one that has the same effect if an operation is used multiple times as it does if used only once. This is especially important in the types of highly-distributed systems we frequently build these days, where interdependent design elements may be isolated on separate networks, potentially positioned in different geographical locations (increasing the chances of communication failures).

Cohesion

There's a common maxim in organization (at least I hear it a lot on those home improvement shows):

"A place for everything and everything's in its place"

I may be taking a bit too much liberty with the principle, but I think this is a reasonable way to describe high cohesion. When a service exhibits high cohesion, its responsibilities are intuitive, unambiguous and focused. They're also well-organized. The service offers the "right" set of related capabilities and stays focused - not trying to become all things.

High cohesion reduces complexity and increases reuse, in part because of the ease of understanding the role and purpose of the element.

Understanding these principles and energetically applying them when designing software systems will lead to products that are delivered more quickly with higher quality, exhibit greater flexibility and adaptability, encourage greater levels of reuse, hold up more reliably under the pressures of time and influences of evolution, and overall add greater value to the business.

¹Hunt, Andrew, and David Thomas. The Pragmatic Programmer: From Journeyman to Master. Addison-Wesley, 1999.

The State of the IT-Business Relationship

Alastair Bathgate at WorkForceInABox wrote a compelling piece titled IT vs The Business - the Final Countdown? in which he asserts the following.

"IT functions have now implicitly turned into two types:

a) the “can’t do” department, focussing only on operations and not strategy (I mean business strategy not IT strategy) and who buy blade servers and occasionally put out a tender for some outsourced development to compete against an SI proposal.

b) the “can do” department that focuses on business strategy, acts as an interpreter between the business and the IT world, and perhaps most importantly focuses on solving business problems. Yes, of course the operations, governance, security and compliance still needs looking after - but that’s all overhead. The exciting bits are the bits that drive the business forwards and deliver value, not cost. And if you’re not at the centre of the business you are on the periphery - and that is no exciting, rewarding, or secure place to be…"

Let's be sure we're the "can do" department that focuses on business strategy. Indeed, Let's be the bilingual partner, actually participating in the definition of the business strategy, translating between business and IT perspectives, and establishing an IT strategy that supports the business. It's a challenging role, but I bet we're up to it. Ready? Set? Go!

Contracts Are The Thing

For as long as I can remember, I've been obsessed with the importance of contracts - deliberate definition/specification of component interface boundaries. In fact, I take a fair share of grief for climbing atop my soapbox on this topic. I suppose that's the price to pay for championing the cause.

The Risk

I was recently skimming through Software Systems Architecture: Working With Stakeholders Using Viewpoints and Perspectives and came across the author’s assertion that that the most important consideration in the Functional Viewpoint (logical model) is poor interface definitions:

"Many architects define their elements, responsibilities, and interelement relationships well, yet totally neglect their interface definitions. Defining interelement interfaces clearly can often be something of a chore. However, it is on of the most important tasks you can perform for the system. Without good interface definitions, major misunderstandings will occur between subsystem development teams, leading to a range of problems from obviously incorrect behavior to subtle, occasional system unreliability."

What’s a Contract?

Bertand Meyer, the designer of the Eiffel programming language, was one of the early proponents of the importance of contracts. In “A Conversation with Bertrand Meyer, Part II - Design by Contract” (http://www.artima.com/intv/contracts.html), he puts it this way, "The main purpose of contracts is to help us build better software by organizing the communication between software elements through specifying, as precisely as possible, the mutual obligations and benefits that are involved in those communications."

Taking it somewhat deeper, an effective contract specification provides an unambiguous description of the interface syntax and semantics - what it looks like and how it behaves. The contract defines the operations offered by the interface, the inputs, the outputs, the effects, and the nature or style of communication (web services, messaging, rpc, etc).

Contracts should exhibit well-known design quality attributes such as separation of concerns, high-cohesion, loose-coupling, granularity and degree of element interactions and others. We’ll come back to those in detail another day.

For today, let’s just emphasize one: the separation of concerns which engenders the ability to organize the development effort around the architecture. Effective contract definition allows individuals or teams to be assigned responsibility for delivery of specific subsystems or components, understanding how that functionality is integrated to the overall system. In other words, contracts help us to divide and conquer.

Who’s on the Hook?

It is squarely the responsibility of the Architect to lead the definition of contracts, and as mentioned above, it’s a common execution weakness. It’s essential that the Architect have ultimate responsibility and accountability for this aspect of the architecture. But the Architect is not a solo act and relies on the engagement of many team members or even multiple teams to achieve results. It’s important for the team members to understand not only “what”, but “why”. In other words, the rationale is also important. As such, successful execution of contract definition will typically require a command of the technical solution, awareness of good design attributes, and ability to orchestrate communication among interested parties - and facilitate negotiation when necessary.

As an aside, you can see a couple of important characteristics of an effective Architect in this description. In particular, a successful Architect is a strong leader and a good communicator. It is not enough to make great decisions and define great contracts. The Architect is not successful unless the contracts are well-understood and agreed to by the relevant stakeholders.

Turning back to “Software Systems Architecture”, I’ll close with the following checklist to help guide our effort and make sure we deliver on this important responsibility:

• Define your interfaces clearly and as early as possible
• Review them frequently to ensure that they are clearly understood
• Do not consider element definition complete until interfaces have been designed
• Make sure that interface definitions include the operations, their semantics, and the examples where possible.

¹Rozanski, Nick; Woods, Eoin. Software Systems Architecture. Upper Saddle River, NJ.: Addison Wesley, 2005

Kruchten: Antipatterns

Picking up on the theme of antipatterns, let's look at a list provided by Philippe Kruchten in his whitepaper "What do software architects do?". This paper, by the way, is another good source of perspective on the emerging discipline of Software Architecture.

Antipattern: Creating a perfect architecture, for the wrong system.

A software architect that is not communicating regularly with the customer, the end users, or whomever represent them (the product manager) is likely to miss the target, in particular as the target is moving, or rather, as the target is only gradually understood.

Antipattern: Creating a perfect architecture, but too hard to implement.

A software architect who does not understand the (maybe limited) skills, capability and experience of the implementation team(s) that will continue and finish the work will create enormous level of stress and frustration, and likely not deliver a quality product in time. The architectural effort has turned into a computer science research project.

Antipattern: Ivory tower

The worse combination is the architecture team that lives isolated in some other part of the organization - another floor, another building, another country - and who comes up after some months with a complete architecture, out of the blue. To their complete surprise, they will experience rejection: an apparent misfit on both front, functional and implementation. This is especially the case if the developers (the non architects) had a few months to make some progress and they have in some ways made some architectural decisions, under some other name. A special case of this antipattern is the architecture group that only scouts technologies and provides recommendations to other groups, but is not making design decisions and is not accountable, as I have witnessed in two large telecommunication companies, the architecture watch..

There is another issue that cannot be completely ignored; it has to do with whom you have chosen to be the architects. It is very likely that you have in this role some of your most talented staff: good at manipulating abstractions, wide experience of a range of systems and technologies, good communication skills, domain knowledge, etc. and you may want to use some of these skills for other tasks than just building architectural views. You want them to speak to the new prospective customers, to show off the organization technical expertise, you want them to help this or that team that experiences a difficult technical issue, you want them to review the architecture of another project, to take part of a due diligence process to acquire a company, to present papers a conference to strut your stuff, or merely extinguishing some nasty fire. But if you are not careful, this leads to another "antipattern:"

Antipattern: The absent architects

No or little architecture design progress is made: the architects are always away, doing fascinating things, or fighting fires. It is very easy to slip in this mode, especially after some initial good progress and early successes, which brought some fame on the architects.

Have you seen examples of these anti-patterns in action? What were the results?

Enterprise Modeling Anti-Patterns

Scott Ambler's Agile Modeling site has a good article on Anti-Patterns for Enterprise Modeling.

Enterprise Modeling Anti-Patterns: "A process anti-pattern is a common strategy which sounds good in theory but in practice proves to be harmful, if not outright disastrous. This article overviews a collection of anti-patterns pertaining to enterprise architecture efforts within an IT organization. These anti-patterns are:

* 30,000 Feet and Climbing
* Bleeding Edge
* Brain Trust Parking Lot
* Buzzword-Driven Architecture
* Detailed Enterprise Modeling
* Enterprise Parking Lot
* Goldplating
* Ivory Tower Architecture
* Modeling for Modeling's Sake
* One Truth Above All Else
* Real-World Disconnect
* Strive for Perfection
* Stuck in the Weeds
* Technology Above All
* Tomorrow Suffers From Today
* Ungrounded Future
* Yesterday's Enterprise Model"

Each of these Anti-Patterns is well worth taking a look, but I'd like to highlight a couple that seem to pop up quite often.

30,000 Feet and Climbing - The enterprise model is so high-level that it is of limited or no practical use to application teams.

Solution: Develop reference architectures which provide working examples of individual aspects of your architecture. Roll up your sleeves and get actively involved with the development teams. Act on feedback from the development teams to evolve your model(s).

Ivory Tower Architecture - Your enterprise architecture model(s) reflect a wishful, perfect world scenario instead of the realities of your actual environment.

Solution: See the solution to 30,000 Feet and Climbing.

Stuck in the Weeds - You are too far into the details, attempting to do all the work for application teams.

Solution: Mentor developers in the architecture, work with them as equal partners on the development teams, and, after they've gotten going, focus on providing them with guidance and advice as needed.

Brain Trust Parking Lot - Your enterprise modeling group is composed of a lot of very smart people who don't fit in well anywhere else within IT but you don't want to lose their knowledge.

Solution: Find a way to make them useful to existing project teams, perhaps as a modeling mentor or as a subject matter expert (SME) in the business domain.

Keep Your Tail Short

Doing some research this evening, I came across a post related to Test Driven Development (TDD) by Jeremy Miller. This was a nice way to describe the importance of keeping the responsibilities of a component very specific and its coupling precisely controlled.

Keep Your Tail Short. The best description for this general concept I've ever heard comes from Stuart Holloway - "When you pull a class off the shelf, what else is coming with it?." Think of it this way, if I want to test my business rules, or my display logic for that matter, what other areas of the code do I have to pull as well? By and large you want to answer that question with "not much." Writing test code just to placate my database or web server when I'm testing business logic is purely overhead -- both in terms of the mechanics of creating the test and the intellectual overhead of understanding the test.

While this refers to an Object-Oriented context, the principle applies perfectly well in just about any development environment, and we see evidence in this scenario of the fact that good design is essential to good testing.

Let's make sure we're able to take just what we need off the shelf... no strings attached.

Who Do You Think Created The Chaos? :-)

Let's have a little fun this morning. No matter how many times I hear this, it still cracks me up.

A physician, a civil engineer, and a computer scientist were arguing about what was the oldest profession in the world. The physician remarked, "Well, in the Bible, it says that God created Eve from a rib taken out of Adam. This clearly required surgery, and so I can rightly claim that mine is the oldest profession in the world." The civil engineer interrupted, and said, "But even earlier in the book of Genesis, it states that God created the order of the heavens and the earth from out of the chaos. This was the first and certainly the most spectacular application of civil engineering. Therefore, fair doctor, you are wrong: mine is the oldest profession in the world." The computer scientist leaned back in her chair, smiled, and then said confidently, "Ah, but who do you think created the chaos?"

As is almost always the case, this tickles our funny bone because there's at least a shred of truth in it. Hmmm... Who's job is it to navigate into clear waters and circumvent the creation of chaos? ( hint)

By the way, I stumbled into this today as I scanned through Booch's recent release of the 3rd Edition of "Object-Oriented Analysis and Design with Applications" and compared it to my dusty copy of the 14 year old 2nd Edition. More on that another day.

The Architect Is Accountable!

This post could be subtitled, "The Architect Makes Decisions... and is ultimately responsible"

We have previously identified this responsibility for decision-making as one of the most important roles of the Architect:

"The software Architect has overall responsibility for driving the major technical decisions, expressed as the software architecture. This typically includes identifying and documenting the architecturally significant aspects of the system, including requirements, design, implementation, and deployment "views" of the system."

In this capacity, the Software Architect has ultimate accountability for the results of the project. When things go south, people should turn to the Architect to recover... and explain. All too often, we see cases where an Architect goes along with a project until the "architecture is done" and then moves on. When the project gets into trouble, you hear "Those developers didn't follow instructions" or worse. It's the responsibility of the Architect to avoid even this potential, something that can simply be accomplished by remaining a fully engaged member of the team - operating as "we" and "our's" instead of "them" and "their's".

But the Architect's responsibility doesn't end there. The measure of success for the Architect is based on more than just the project results. He is also accountable for the Business and IT Strategy. It isn't enough to for the single project to successfully meet it's objectives, but the results must also contribute to these strategic considerations. Factors that are by definition more valuable and significant.

Being successful in this capacity will almost always require an ability to balance competing stakeholder needs. The Architect rarely finds himself facing simple black-and-white decisions, but rather the need to carefully select just the right shade of gray, and to do so under intense pressure and without enough time and not enough information. That's the life of the Architect. Philippe Kruchten summed it up like this: "The life of a Software Architect is a long and rapid succession of suboptimal design decisions taken partly in the dark."

But like most things in the endless balancing act of the Architect, we have to be careful not to fall into the deadly cycle of becoming the decision authority for every decision. While the Architect is accountable, he simply can't own every decision. Even if it made sense for some reason, it's simply not practical. The Architect is out-numbered by a factor of, perhaps, 10 or 20 to one on any given project, and those other 10 or 20 engineers aren't just drones - they're experienced professionals who's input is essential.

So where do you draw the line? There are many factors at play there, but we'll consider one important one. In "What is Architecture", we talked about the need to organize a team around the architecture. A key result of this practice is the establishment of a natural and effective demarcation of decision-making responsibility. In their article titled Guiding Principles for Enterprise Architects, Bredemeyer Consulting talk about the "Minimalist Architecture Principle" which essentially says “if a decision can reasonably be made by someone with a more narrow scope of responsibility, defer the decision to that person or group.”

This practice keeps the Architect focused on the most significant decisions - those of the greatest value (and consequence) and of the type that require the broadest perspective and authority. But we have to be careful. This is essentially a form of delegation. As is always the case with delegation, we can transfer the responsibility but not the accountability. The Architect remains accountable for the overall results, and he must put practices in place to effectively assess those results throughout every stage of the project.

A few of us were discussing this concept of accountability recently, and someone tossed out the idea that the Architect is responsible from "cradle to grave." That's a good way to look at it. The Architect is accountable from the first stages of ideation all the way through to the production release and beyond as this new capability helps to realize the overall Business and IT Strategy. We got to thinking, though, and decided maybe "grave" wasn't a good thought to associate with this accountability. How about if we just say the Architect is accountable "From Cradle to Glory."

Good Architecture Encourages Good Estimates

When it comes to project estimation, much has been written about the importance of historical contexts and correlations, and it only stands to reason that relevant historical data helps lead to more accurate estimates. As Steve McConnell points out in Software Estimation: Demystifying the Black Art (Best Practices (Microsoft))¹, historical data helps to work around common sources of inaccuracies in estimation because it accounts for organizational influences, avoids subjectivity and unfounded optimism, and reduces estimation politics.

But do we get the biggest bang for the buck out of our historical data? While common approaches to estimation successfully leverage this data, there's still more we can do. Historical data is even more valuable when used in the context of effective architectural practices. In situations where it is possible to leverage consistent architectural patterns and frameworks, historical data can be captured and put to use at a lower level of granularity, resulting it much more reliable estimates earlier in the project.

Let's look at an example. In my experience, the most common practices have us maintain and leverage experience at the project level. In this approach, a database of project performance attributes is built up over time, supporting the establishment of prediction models for similar projects. For example, if an organization has completed 10 "green field", web-based, Java/J2EE projects for the Health Care industry using Oracle and MQ Series in a b2b services delivery model, and these projects have taken on average 47,600 man-hours to complete (.6 std dev), then it's reasonable to predict (for planning purposes) that a future project of similar parameters will also require 47,600 hours of effort. This is a common approach and certainly provides value, especially at the early stages of a project.

As a project progresses and the architecture is better understood and becomes more stable, the same principles can be applied a lower level of granularity. For sake of example (albeit perhaps contrived), let's imagine we get towards the end of Elaboration (assuming RUP for the moment), and we are satisfied that our architecture essentially consists of a collection of typical patterns like this:

A. Web Page (JSF)
B. Service Facade (stateless session beans)
C. Business Object (pojo)
D. Data Transfer (Value Objects)
E. Persistence (Data Access Objects)

Now let's imagine a straightforward business application with, say, 20 Use Cases such as "Create Account", "View Order History", "Make Payment", etc. If the team applies the architectural patterns consistently, it is completely reasonable to look at the project as a situation where the engineers will produce 1 (or n) of each of these above "components" for each use case. In other words, for each use case, a new Web Page will be created to talk to a new Service Facade using a new Value object for data transfer. The Service Facade component uses a new Business Object that handles the business logic. The new Business Object depends on a new Data Access Object (DAO) for data persistence.

If we have good historical data for what it typically takes to build a Service Facade or a Data Access Object (in this architecture), then the estimate nearly becomes a mathematical exercise... 20 Web Pages at 40 hours each plus 20 Service Facades at 15 hours each plus 20 Business Objects at 52 hours each plus... and so on.

While this is no estimation panacea, and it's only one part of the overall estimation process (for example, let's talk about the value of architecture on "impact analysis" another day), I've been able to apply this approach for a long time with very positive results. Of course, the key is designing and applying a consistent architecture and capturing the historicals at the relevant level of detail.


¹Software Estimation: Demystifying the Black Art. Redmond, Wa.: Microsoft Press, 2006

What's All the Fuss About Architecture?

Software Architecture is all the buzz in the IT world right now. What's up with that? I think of it like this. Over the past decade, the decentralization pendulum gradually swung full-tilt towards distribution. To a large degree, individual divisions or departments were chartered with building out independent systems, absent of any unifying IT strategy. In fact, in some cases, software was being put into production completely within a business unit with no IT involvement at all! New applications sprung up like dandelions, as businesses scrambled to respond to competitive pressures and leveraged the climate to wrestle "control" away from the IT "Borg."

While for some time, this approach yielded favorable results in terms of flexibility and speed to market, it was a course of divergence that could only lead to one place - the place we find ourselves today. Disparate systems, duplication of resources, closets full of unknown capabilities, inability to leverage corporate assets, etc, etc. Meanwhile, the "science" of Software Architecture and enterprise technologies have continued to mature to the point where intelligent IT strategies promote more enterprise-scale thinking and yield tremendous competitive advantages.

It'll be our job to ensure that the pendulum swings back into balance so we position IT to be the "go to" guys and gals for the business... a partnership of mutual objectives and ambitions... jointly pursuing the overall goals of the business.

Cheers.

What is Software Architecture?

While the science associated with Software Architecture is an emerging discipline in the software industry, there is little debate about the nature of its importance. The following quote from the Software Engineering Institute (SEI) makes the case loud and clear (http://www.sei.cmu.edu/architecture/index.html):

1. Software architecture forms the backbone for any successful software-intensive system.


2. An architecture is the primary carrier of a software system's quality attributes such as performance or reliability.


3. The right architecture - correctly designed to meet its quality attribute requirements, clearly documented, and conscientiously evaluated - is the linchpin for software project success.


4. The wrong one is a recipe for guaranteed disaster.

What is Software Architecture?

But what exactly is meant by "Architecture" in the software field? The concept has been around for a couple of decades at this point, and absent a formal definition, I think we can rally around the central themes embodied in the following "authoritative" sources:

IEEE

“Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.” [IEEE 1471]

IEEE Computer Society, IEEE Recommended Practice for Architectural Description of Software-Intensive Systems: IEEE Std 1472000. 2000.

Grady Booch, Philip Krutchen, Kurt Bittner, and Rich Reitman

Software Architecture encompasses the set of significant decisions about the organization of a software system:

• Selection of the structural elements and their interfaces, by which a system is composed
• Behavior as specified in collaborations among those elements
• Composition of these structural and behavioral elements into larger subsystems
• Architectural style that guides this organization

G. Booch, P. Krutchen, K. Bittner and R. Reitman. The Rational Unified Process — AnIntroduction. 1999. Definition derived from Mary Shaw’s definition presented in 1995 at theFirst International Workshop on Architectures for Software Systems.

Len Bass (SEI)

“The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.”

Bass, Len; Clements, Paul; & Kazman, Rick. Software Architecture in Practice, Second Edition. Boston, MA: Addison-Wesley, 2003.

Studying each of these definitions, we see several themes echoed repeatedely. The architecture of a software-intensive system consists of three cooperative elements: 1) The Structure - a static dimension illustrating the partitioning of responsibilities among subsystems, 2) The Behavior - a dynamic dimension representing the relationships and interactions among those structures, and 3) The Style - the principles and guidelines that motivated the structure - and should continue to do so.

What Does Architecture Consist Of?

Let's take a very high-level look at each of these elements and consider a very simple example. For additional insight, I'd encourage digging into one or more of the many excellent publications on the topic. I've listed a few of my favorites at the end of this post.

The Structure

The structural elements of the architecture capture the static set of subsystems/components that make up the system as well as the responsibilities of these subsytems and their organization into larger subsystems. An important consideration in this dimension is the dependencies between components and on other/external systems. The organization of the architecture at this level and from this perspective is also important from the standpoint of coordinating work effort and planning a project. It is frequently the case that a project team is organized around this structure - different teams responsible for delivery of different subsystems. This consistency between product architecture and "project team architecture" provides considerable value in planning and managing a development effort.
The diagram above captures at a grossly high level part of the static dimension of a theoretical e-commerce application that accepts credit cards. Capturing and articulating the architecture from this point of view helps to establish the context of the system and provide a frame of reference for decision making.

The Behavior

The behavioral dimension represents the ways in which the subsystems or components interact to fulfill the system requirements. A diagram such as the following represents the "messages" flowing between the components of the same hypothetical e-commerce system referenced above.

In this scenario, we can follow the system in operation as the Shopping Cart collects credit card data from the Consumer and uses the Customer Management System to get additional information from the consumers profile. With that data, a payment instruction is dispatched to the Card Gateway that leverages the Third Party Processor to execute the payment. After the payment transaction is complete, the Shopping Cart records the payment information in the Payment Management System.

The Style

Overall, the style dimension captures the principles and patterns that motivated the structure and behavior of the architecture. In some ways, the style is analogous to that of architectural style in building architecture (e.g., Roman, Gothic, Victorian, etc). This includes concepts such as architectural patterns, technical principles such as rpc and idempotency mechanisms, and values such as simplicity of design.

The style dimension effectively sets forth a vocabulary and a collection of guidelines and constraints - boundaries, if you will - that encourage effective decision making and clarify communication about the architecture. It's essential this dimension be explicity represented, as future evolution of the product depends on understanding the factors that influenced the existing architecture.

How is Architecture Communicated?

There are many ways to capture these key dimensions. The "4+1 Views" promoted by the Rational Unified Process and Phillipe Krutchen (http://www.win.tue.nl/~mchaudro/sa2004/Kruchten4+1.pdf) is a common approach. More on that another day.

It's important to note that in whatever way the architecture is conceived and communicated, it is the responsibility of the architect (What is A Software Architect) or a team of architects to drive the towards conclusion, constantly navigating the murky waters, balancing the many competing stakeholder needs, and rallying the team around the "right" decisions. The future of the product depends on successful accomplishment of this mission.

In an influencial work a decade ago, Mary Shaw and David Garlan put it as follows, and I think many of us can attest to the truth of this statement:

"The software architecture you marry in haste will be there for you to repent at liesure."

Let's do what we can to get it right :-)

Suggested Reading

1. Kruchten, Phillipe B., The 4+1 View Model of Architecture, IEEE Software, vol. 12, no. 6, November 1995


2. Bass, Len; Clements, Paul; & Kazman, Rick. Software Architecture in Practice, Second Edition. Boston, MA: Addison-Wesley, 2003. (Software Architecture in Practice, Second Edition)


3. Clements, Paul; Bachmann, Felix; Bass, Len; Garlan, David; Ivers, James;; Little, Reed; Nord, Robert; Stafford, Judith. Documenting Software Architectures: Views and Beyond. Boston, MA: Addison-Wesley, 2002. (Documenting Software Architectures: Views and Beyond)


4. Fowler, Martin. Patterns of Enterprise Application Architecture. Boston, MA: Addison-Wesley, 2002. (Patterns of Enterprise Application Architecture)


5. Shaw, Mary; Garlan, David. Software Architecture; Perspectives on an Emerging Discipline. New Jersey: Prentice-Hall, 1996. (Software Architecture: Perspectives on an Emerging Discipline)

What is a Software Architect?

Characteristics of the Software Architect

"The software architect must be well-rounded, posses maturity, vision, and a depth of experience that allows for grasping issues quickly and making educated, critical judgment in the absence of complete information." (Rational Unified Process)

Role of the Software Architect

• The software architect has overall responsibility for driving the major technical decisions, expressed as the software architecture. This typically includes identifying and documenting the architecturally significant aspects of the system, including requirements, design, implementation, and deployment "views" of the system.


• The architect is also responsible for providing rationale for these decisions, balancing the concerns of the various stakeholders, driving down technical risks, and ensuring that decisions are effectively communicated, validated, and adhered to.

Skills of the Software Architect

• Experience in both the problem domain, through a thorough understanding of the requirements, and the software engineering domain. The software architect must provide the global vision for the project.


• Leadership in order to drive the technical effort across the various teams, and to make critical decisions under pressure and make those decisions stick. To be effective, the software architect and the project manager must work closely together, with the software architect leading the technical issues and the project manager leading the administrative issues. The software architect must have the authority to make technical decisions.


• Communication to earn trust, to persuade, to motivate, and to mentor. The software architect cannot lead by decree, only by the consent of the rest of the project. In order to be effective, the software architect must earn the respect of the project team, the project manager, the customer, and the user community, as well as the management team.


• Goal-orientation and Pro-activity with a relentless focus on results. The software architect is the technical driving force behind the project, not a visionary or dreamer. The career of a successful software architect is a long series of sub-optimal decisions made in uncertainty and under pressure. Only those who can focus on doing what needs to be done will be successful.

Many Hats of the Software Architect

• Visionary – Discovers innovative and novel approaches, translating highly-abstract concerns into tangible opportunities… and results


• Manager - Coordinates with all stakeholders to formulate and communicate a practical blueprint


• Salesman - Sells the ideas and appropriateness of the solution


• Developer – Helps develop POC or pilot to validate solution, patterns, practices, and principles


• Leader - Shepherd's from ambiguous and abstract to concrete and practical


• Coach – Mentors and coaches others on effective application of industry best practices


• Governor – Helps to establish architectural standards and guidelines

Responsibilities of the Software Architect

• Abstracts the complexity of a system into manageable models that describes the essence of a system by exposing important details and significant constraints.


• Creates and communicates tailored views of the software architecture to appropriate stakeholders at appropriate intervals.


• Provides leadership on critical decisions that designate a specific direction for a system in terms of implementation, operations, and maintenance. These decisions usually result in tradeoffs that define key characteristics of the solution, and they must be well documented and easily understood.


• Establishes quantifiable objectives that encapsulate quality attributes based upon which the fitness of the architecture is measured.


• Proactively leads the implementation during all iterations and monitors the adherence of the implementation to the chosen architecture.


• Collaborates with the Project Manager and other key disciplines to prioritize use cases and organize work to drive down risk and encourage strong project control structures.

Architect as Advocate of the Business

Architecture is “an instrument whose central function is to intervene in man’s favor”

James M Fitch, 1972

As an architect, one of our most important responsibilities is working with the business stakeholders to understand their objectives and help ensure an effective solution. What's more, we typically need to understand the problem domain (or learn it very quickly) so we can help define those requirements and develop the solution strategy.

The following quote from WWISA does a good job of capturing this responsibility:

"Architects spend the lion’s share of their time up front: listening to clients, understanding the totality of their needs and resources, scrutinizing feasibility, forming a practical vision of a structure, and creating a blueprint. As the structure is built, the architect intervenes in the client’s favor, ensuring compliance to the plan and guiding the vision through the tempest of design changes, crises and ambiguities. Client advocacy is the cornerstone of the architect’s role." (http://www.wwisa.org/wwisamain/role.htm)

The role of advocacy is exhibited in many activities (negotiation, scope management, corporate politics, risk management, prioritization, etc.), but let's quickly highlight just two that revolve around the principle of understanding the problem domain.

Product Definition

The architect's awareness and appreciation of the business domain is essential to the best definition of a software-systems capabilities. The architect is able to understand likely requirements and encourage penetrating assessment of key product areas. I like to think about this in terms of how understanding the "solution space" has influence over the understanding of the "problem space". It's true we want to maintain separation of our models and abstractions between these domains, but it's critical that the solution side have a deliberate and calculated influence on the problem side.

In my classes, I like to use the example of buying a new car. While this is a low-level example, I think it illustrates the topic well (and it's easy to personally relate to). With a few essential "market needs" in mind (e.g., 4 door because it'll be too hard to get the car seat in and out of the coupe when the new baby is born, 6 cylinder because this 4-banger isn't cutting it in Atlanta traffic, and candy-apple red because, well, just because), we head out into the "solution space" (wandering lots, reading brochures, taking test drives, talking to friends, reading Consumer Reports, etc) and begin to evaluate our options. The available options immediately begin to influence our understanding of the most suitable feature set - the best balance of needs and solutions - and we ultimately arrive at the best "product."

The architect is most effective at helping to arrive at this "perfect balance" when he or she has a practical understanding of the business domain - the "problem space".

Product Evolution

The architecture of a system is subject to the tides of change within the business environment being served, reinforcing that the architect should be familiar with the business domain. In this regard, the architect needs to anticipate future changes to the architecture, and integrate styles that lead to the most valuable types of flexibility, a challenge more readily met with an appreciation of the business domain. For example, architects of today's payments-related systems should be keenly aware of the industry trends towards payments convergence - the collapsing of disparate payments channels into fewer products with greater compatibility and higher-value capabilities. Likewise, regulatory changes in the financial services space are shifting the landscape and encouraging the overlap of banking, insurance, and health care services. Business factors such as these industry and government trends must be taken into account in the design of effective architectures.

Let's sum this up with a quote from Wikipedia. This is actually in reference to structural architects, but it seems fitting nonetheless:

"The practice of architecture is a business, in which technical knowledge, management skills, and an understanding of good business practice are as important as creative design." (http://en.wikipedia.org/wiki/Architect)

And I'll leave you with the following WWISA story reinforcing the close relationship between the business and the Architect. :-)

"The architect and client meet when it is all over and reminisce about the trials and triumphs. They hold a big party at a Mexican restaurant, complete with Mariachi Band, for all the builders, employees, and customers involved with the project. Those nay-sayers who whined incessantly and said it couldn't be done now stand mute, sipping their margaritas. (http://www.wwisa.org/wwisamain/role.htm)"

Anyone hear the music?

Ridiculous Simplicity (#1)

"Simplicity--the art of maximizing the amount of work not done--is essential." (http://agilemanifesto.org/principles.html)

When I was a kid and would start to make something into a big deal, I remember my mom accusing me of making a "mountain out of a mole hill." That may be an expression that only makes sense here in the southern part of the US, but it seems to be applicable to projects I see all over the world. The systems we undertake today, and the technologies we have at our disposal, certainly allow us to make mountains. Of course, we all know mountains tend to be rigid and inflexible - immovable. In response, a deliberate practice - a core value - of delivering simplicity has become even more important in today's IT climate. What I'm thinking about here spans both the problem space and the solution space... we must simplify the problems and simplify the solutions. But what does that mean?

Ridiculous Simplicity in the Problem Space

I bet we all can think of plenty of examples where we allowed the business problem to get out of hand. Allowed it to grow nearly unbounded. I've witnessed a couple of projects over my career where "we" fell into the "boil the ocean" trap in the problem space, imagining that a product could not go to market unless it had nearly every conceivable feature. We allow the project to get bogged down in "requirements gathering" and don't demonstrate the collective discipline (and confidence) necessary to establish direction and make hard decisions on trade-offs in the market. I'm thinking about things like which features are really required and which are not. It's true that there are market situations where a product should not be released without all the "bells and whistles" or "this and that", but those scenarios are rare. Generally speaking, we do our business partners a tremendous favor by encouraging an environment that releases "right-sized" product and builds on it over subsequent releases.

This reminds me of a story... About 14 years ago, when I first left the US Air Force, I was working with a team building an internal account payables system. Users of the system choose the appropriate cost center upon login, and about 10% of the users had rights to access more than one cost center. When the user was ready to operate on a different cost center, they would have to logout and then log back in, this time selecting the alternate cost center. I made a case that we should change the screen design to allow the user to choose a different cost center without logging out. This alternative design would simply place a drop down list or something to that effect in the product. The user would select the appropriate cost center from the list.

My boss at that time argued against this approach and helped reinforce the importance of managing scope on the project - and on the product. It's not just the 15 minutes that it would take to add the feature (and that's really all it would have taken), but it was also the longer term consideration such as "care and feeding" of this capability, the constraints it introduced on the future evolution of the architecture and design, etc. It was a 15 minute change request in this one project, but it had hours and hours of impact over the life of the product. My boss insisted, "one of my principle responsibilities is to control this scope." I learned a lot from this guy. I think we all can. Today, many of us can think of projects where we refused to let the "15 minute feature" get in, but also the dozens of other features, functions, etc, that ultimately lead to the death of a product or project - crushed under it's own weight. Where successfully managed, these products remain lean and responsive to change, able to keep up with evolving demands over a healthy and long life.

Ridiculous Simplicity in the Solution Space

We can see similar characteristics on the solution side. Most developers I know really care about the "right-ness" of the design and implementation of the product. Their heart is in it, and they want it to be "perfect." And high-quality, precise work is always right and proper. We should never settle for mediocrity. The trouble we seem to run into, however, is in the definition of "right" or of "mediocrity." How do we assess the "right" design and the "best" implementation? The tendency seems to slant towards more or bigger or complex, when we should evaluate a solution based on the degree to which it satisfies the requirement and on its simplicity - the elegance of its simplicity. Without direct energy applied, a team frequently migrates towards what I like to refer to as "someday maybe" design. The design begins to get tweaked in sometimes subtle ways to account for the idea that "someday maybe" we'll need to be able to do X, Y, or Z. The "right" design satisfies today's requirements without trying to predict the future, recognizing that the chances of predicting correctly have been shown to be very slim, AND, when the future comes and we know what it holds, we can change the design. At that point, we'll be in the advantageous position of knowing precisely what's needed (based on actual data and experience as opposed to predictions), and we'll be able to get the design "right". I'll assert then, that the best design is the one that satisfies today's requirements while leaving options open for tomorrow's.

Just one brief example while we're on the topic of simplicity in the solution space. I recently witnessed a project where there was a fairly straightforward b2b web service interface that precisely satisfied the market needs. Falling directly into the "someday maybe" trap, the team added additional attributes to this interface - attributes that were not used in any fashion today - just in case they may be valuable to the business in the future. Compounding the situation, the team built a separate database to store the service response and create reports to demonstrate whether or not the b2b partners were populating the unused fields. The cost to the business is significant as a result of design by prediction rather than data, architectural inflexibility, additional scenarios and code paths to test and design around in future releases, client implementations and documentation confusion, etc.

Wrapping It Up

On my teams, we incessantly strive to find the most straight-forward solutions to problems and execute those solutions with pinpoint accuracy. You’ll hear team members and managers echo the phrase “do the simplest thing that could possibly work” and challenging one another with "is that really the simplest thing?"

This is a key principle of an agile, nimble, and responsive development team. And interestingly, this principle manifests itself in the products produced by such team. The products themselves exhibit agile, nimble, and adaptable characteristics, essentially mirroring the nature of the team that created them.

I'll wrap up with this quote from Dr. Paul MacCready:

"Treat every problem as if it can be solved with ridiculous simplicity. The time you save on the 98% of problems for which this is true will give you ridiculous resources to apply to the other 2%."

I've consistently found this to be overwhelmingly true. Without the precise data to back it up, I'll assert that a focus on simplicity reduces the overall work effort on a project by up to 50% and enables the Business/IT partnership to more quickly deliver higher quality software that satisfies the market needs. These same products then respond to the pressure of market change and evolution in a much more resilient manner, ultimately leading to higher profitability and better performance in the marketplace.

One last thing... I would expect these thoughts to stir questions on topics such as the following, each of which we'll take some time to explore in the near future:

1. Cost of Change curve
2. The necessity of automated testing
3. The YAGNI (you ain't gonna need it) principle
4. Refactoring