Release frequently and deploy as often as feasible, so that most of
development occurs within a maintenance context.

Suppose your strategy were to sort features in business priority, and work
on the most important ones first. They might not lead immediately to a
deployable solution. They might require a few more iterations of
lower-priority work until

Suppose you address that issue by buying a legacy system that comes close to
covering your features. Now you must retrofit tests on it.

Suppose instead you address that issue by writing an entire tool from
scratch (test-first, natch).

The first and last option raise quality, because in-house test-first is
always better than OTS. (Jack of all trades, master of none, etc.)

The middle option lowers quality.

All three options delay feedback regarding external quality. How well does
our deployment channel work?

You are asking how to be a Software Engineer... ;-)

Unexpected and untested things. Tests should cover every detail from the
programmer's keyboard to the user's platform.

For example, consider these two user stories:

- the appliance shall use a dual-booting RAID

- install the appliance, turn it off, yank either RAID
    cable, turn it on, and expect it to boot and work

Now suppose the first story goes long, because Fry's does not carry the RAID
card that supports our appliance and OS. That bumps the second story out of
the iteration.

Now suppose you fly to the customer's site, install the appliance, turn it
on, plant your hands on your hips with a big satisfied grin, and the RAID
cable falls out. The system doesn't boot, and you have to fly home, fix
everything, and fly back.

That's a true story: The one thing they didn't test was the first thing that
happened to them at deployment time.

Now consider this user story (also part of that true story):

- a form on a Web page backs up the server's database

To get that to pass, write a test that claws its way thru all these steps:
Create a sample database, put sample data in it, call production code to
back it up, zilch the database, restore it from the backup, and look for
that sample data.

On the customer site (the second time around), demonstrate the backup
system. Then accidentally erase the production database. You are saved: The
backup is tested, and all you need to do is restore it. (Then quickly
disable the backdoor you used to erase the database!)

By trying to force them to surface. Engineers seek ways to cause the next
problem they plan to solve.

I have heard of the acronym ROI, but folks use it somewhat mystifyingly...

Emergently. Follow simple rules that constrain decisions, and repeat all
aspects of a project over and over again. This cycle converges on an
optimum, without over-control.

 Hello Andrew,

 I would hope that as VP of Engineering, you would include as part of
your responsibilities, a periodic evaluation of your overall costs,
productivity, and conformance/deviations to prescribed standards.
This would help identify your costs, how they fluctuate, and which
measurements might be tunable.

 While your responsibility would be very high level, the figures
gathered from teams or groups under you may have varied organizations.
You must also be careful to make changes that will achieve your goals
while not adversely affecting other parameters.

 What are your goals?  What is an acceptable cost for all the
development?  This should include design, implementation, verification,
validation, release preparation, and training.  What are your post
delivery costs?  These may include rework time, emergency customer
support, irate customers, and perhaps unexpected impact on
subsequent development or procedures.  When must you deliver products?
What classes of service do you offer?  For instance, many companies
provide "bronze", "gold", and "platinum" service to customers.
What preparations have you and will you make to accomplish those
service agreements?

 Are there particular failures within your organization today?
If not, look closer.  I could name perhaps a dozen problems
that could be factors in a large organization.  The top 3-5
problems for your hypothetical organization must be determined
so that you will address the proper issues.

 There are more risks than can easily be mentioned.  How good is
your current process and what are you and your teams/departments
doing to improve your overall goals?

 I'll hazard a guess that in many organizations your top problem
is that you were given the goal to unilaterally reduce costs in the
first place.  Your second problem would be the unexpected results
of changes that you actually can make.  Your third problem would be
the political ramifications and tradeoffs you had to make to accomplish
your goals.  (e.g. Reducing documentation effort under your group
may adversely affect another department who has similar or perhaps
conflicting goals.)

 Any optimal configuration is likely to be for the moment and will
change.  Your next set of goals as VP may be different, but affect what
you just (hopefully) accomplished.

 David

I think the first thing is to get an economic model, and since I'm
answering this from an XP perspective, it's going to be the simplest
economic model that could possibly be informative.

Post-deployment costs of defects start with lost productivity
due to the defects, which in turn cascades into lost sales, etc.
Those, in turn, go back to engineering costs in finding and fixing
the defects, which in turn displaces other activities that could
be used to enhance revenue.

In other words, I find it hard to believe that there is such a
thing as a field-reported defect where the economic loss in
the field is less than the economic cost of not letting it out in
the first place. Other people may differ on this, however.

The second limb of this is that code quality directly affects
the ease of adding additional features (and of repairing
defects) in an existing code base. Remember that I'm answering
this from an agile perspective: most of what is delivered will
consist of code that has been extensively modified in the
process of developing the product.

The metric here is velocity: how does the ease (that is to say,
the cost) of adding a feature change over time. If it's stable, then
you've got a code base whose quality is "good enough."

These two measures of quality are related, but not identical.
For may business applications, the second (quality as seen
by the developers) may exceed the quality as needed by
the end users. For life, safety or mission critical applications,
the reverse may be true.

XP teams can deliver substantial projects that have less than
one field reported defect per month. See, for example,

http://martinfowler.com/bliki/VeryLowDefectProject.html

By can I don't mean that they manage to do so as a routine
consequence of using XP! I mean that there are some real
world examples of projects that do it. Regardless of some
proponent's rhetoric, XP is not a magic bullet that cures
every software development ill under the sun. It is, combined
with experianced project management and a dedicated
development staff (and possibly a secret sauce) capable
of getting the job done.

There are several basic processes needed.

1. Executable (meaning repeatable) acceptance tests.
These tests must be accessable to non-technical people
on the customer side (including analysts, QA technicians,
and documentation specialists.) That means that they
can't be written in xUnit or, even worse, some variety
of GUI automation scripting language.

2. Every line of code the developers write must be
integrated that night. The integration and build system
must run all of the passing acceptance tests to date,
and all of the unit tests must run at 100%. The build
must finish with a properly packaged deployable. The
build system may run analysis code like code coverage
metrics; if so, unit test statement and branch coverage
must be at 100%, and acceptance tests coverage and
branch metrics must be fairly high.

3. QA must be done statistically. That is, it is the job
of QA to insure that the product meets quality standards,
not to discover specific defects. A failure to meet
quality standards means that the process needs to be
amended until it can produce a product that meets
quality standards.

The interesting thing about this list is that it's not completely
specific to XP. XP mandates the first two and strongly suggests
the third. Most shops doing standard software engineering
could put in 1 and 3 without seriously affecting their
work flow. Item 2 is not possible unless the shop shifts
to TDD rather than sequential design, code, test and
integrate phases on each module.

The other thing about this is that it doesn't mandate
a lot of expensive tools. The current practice on 1
involves the FIT and FitNesse packages, both of which
are GPL.

Code analyzers (like coverage analyzers) in item 2
may well cost a bit, but they're worth every cent both
from the standpoint of knowing where you're at and
from the standpoint of controlling cowboy coders.

Likewise, some tool support of item 3 may well
be appropriate.

The top risk is not paying enough attention to keeping
the code base sufficiently maintainable that you get a
flat velocity curve. Crap code costs money. So does
deteriorating design.

At this point, there are projects that succeed with just about
every imaginable process, and projects that fail with the
exact same processses.

The only thing I know of that's helpful across the board is
to keep believable progress indicators, and the only way
I know of doing that is to keep WIP (work in progress)
to an absolute minimum. WIP is anything that's somewhere
between "detailed requirements analysis started" and
"successfully integrated, all acceptance tests pass."

The biggest factor with an XP team is that it's
capable of putting out a fairly predictable amount
of function per iteration. Once a team has geled,
it's not all that easy to add or subtract people, or
change process. Increasing the amount of function
per iteration is a slow process that requires a lot
of attention.

The lever that can be controlled is the actual functionality
that's worked on in any iteration going toward a formal
release. Pulling all of the stakeholders together for a
project is a problem that's fully worthy of a VP's time.

The other issue is that there will be times when the
product needs major restructuring work to maintain
design integrity, and that may not correspond to
actual deliverables. This can be deferred for a while,
but it cannot be ignored or the ability of the team
to deliver at a constant velocity will suffer.

I've never figured out how a marketing department comes
up with their estimates.

You're welcome.

John Roth

Alarm bells start ringing here - "absolute lowest possible total cost"
emphasizes low cost twice.  This is means _cheap_.  That is ok - a first
crack proof of concept is usually better done really cheaply, because it
is going to be completely redone before delivery.  Two poorly written,
but working, demos are often better than one quality demo.

That said, if the folks at the top REALLY mean absolute bottom quality,
you may have some frights ahead.

The tradeoff, with vague terms.  Get the stakeholders to commit to a
level of quality and a cost, and then you might have a prayer.  For
example, ask them how many weeks of downtime a month they want, then
when they go ballistic, point out that they likely want fairly high
quality code, and thus it will take appropriate resources to create.  
Get a commitment for the number of bugs (statistically) they are willing
to accept in the code base, and some idea of budget.  Then, you can
start planning.  Also ask which one is more flexible - budget, time, or
bug count.

Also, if lowest cost is a priority, they are unlikely to pay for top
tier talent.  Again, if the project does not justify it, you have
nothing to worry about, but if it does, then you have made a business
case for better employees.

Beware of vague specs and people who refuse to comment on intermediate
builds.  You do not always need an "on site customer", but you do need
someone who has approval and rejection authority for features, and you
need the key stakeholders to agree that said person can do that.  
Nothing sucks more than hitting the end of a project and being told "the
VP of finance really wanted feature foo, and is thus unhappy with the
product and the high cost", when you gave them the choice of feature foo
or feature bar, and they chose foo.

Finally, lowest cost often implies cheaping out on the physical plant.  
Get your developers in offices with doors that close, and get them out
of meetings.  This increases productivity and lowers costs over the long
run, though it may be hard to run by the hierarchy police.

Again, these are not always necessary to accomplish great things, but
they help, and thus they reduce the risk of problems.

Provable?  Don't go down that road, as that boosts your cost by orders
of magnitude.  For reasonable cost, you get statistical bug counts.  
This is not a bad state, as you can adopt unit tests and continuous
integration to try and get a handle on bug counts, fix rates, and
reversion prevention.

If your bug counts are higher than you expect, or your feature set
changes after you have built your acceptence criteria and unit test
harness, you may well be shafted.

Some risks happen - that is why they are risks.  By taking control, you
can manage risk, but then you have higher cost as a result.  Look at
DeMarco's work, and the people he references to see how you can  do some
of that management.  (Hint: you can drive the odds of a given failure
arbitrarily close to zero by spending enough and doing enough testing,
but cost goes up exponentially, because these techniques

The biggest single decision is who gets to work on the project, and how
you intend to run it.  Get buy-in on your high level decisions, and you
are more likely to make the proper low level decisions.

Scott

1: Institutional blindness: a typical problem in many institutions
   is the inability to see actual differences between what we
   -say- we do and what we actually do. Most companies
   that make crap have a motto that says something like
   "quality is job 1"
2: Unmeasurable goals. You have one above. "Lowest possible
   cost, but high enough quality to keep post release costs low"
   is unmeasurable. Accept that that is a -wish-, not a goal,
   and use it as a guideline in setting measurable goals.
   Measure current results, decide on an acceptable metric,
   and measure future progress against that metric.
3: Authority and responsibility resting in separate hands.
   Whoever is charged with making things work has to have
   the authority to: (a) change the plan, (b) reject requests from
   business people (c) go directly to highest management levels
   at need (d) decide spending [how much may be up to
   others; how it is spent must be up to the guy who has to
   make things work] (e) the authority to slip the release
   date; this will be the one you probably won't get.

No.

No.

No tools can solve the biggest problems in developing
quality sotware cheap. The biggest issues are -always-
people issues, never technology issues. For every one
project that fails due to a technology issue, there are
probably hundreds that fail due to people management
issues.

The absolute best thing any tool or mehodology can do
for you is let you see the people problems quickly
enough to step in.

Many of the things Agile mthodologies recommend are
directed at this problem. One value of peer programming
is that it identifies coder problems that show up in the
code more quickly than otherwise.

An advantage to frequent unit testing and frequent
releases is that they identify communication issues
quickly. They find cases where the code fulfils the
requirements but not the user needs.

1: slow reaction time to a problem. Problems will occur;
the earlier they are identified and addressed, the cheaper
it is, and the better the code quality.

No.

No.

Tools don't solve problems. People solve problems. The
good tools help people solve problems faster, easier, cheaper.

Ultimately, though, computers don't run the business;
people do. You can have the best technology, and the
company can still end up in the dumpster.

1: quality vs time to market. The best thing is to try to
   accurately measure the cost vs. benefit. A typcial
   problem is that a decision is made prior to the CB
   analysis, and the CB analysis is subtly skewed toward
   supporting the prior decision. Logic is used to justify
   the decision, rather than to make the decision.

2: support costs vs time to market. See above. Put
   numbers on time to market.

Mathematically. See a basic accounting textbook.
Typical examples used are the book publishing examples
for return on investment. Don't forget returns, exchanges,
competitive upgrades, other non-obvious support costs.

My apologies if my answers seem terse. But your
questions are geared toward finding a technological
solution, and there ain't no such animal. Ultimately
the best technology we have can only -support-
good management, never make up for its absence.