Adam White

•  All P1 and P2 bugs are fixed
•  P1s and P2s are issues that need to be fixed before the release. P1 requires an immediate fix 
•  P2 is important for the release but does not prevent the testers from doing their job.
• Customer issues usually get marked as P2 or sometimes P1 depending on the criticality of the problem. 
• The processs of keeping track of these issues is managed in combination by support , product management, project management, development and test engineering teams.
• If customer issue 
•  Product management assess the impact with input from support 
•  Evaluate the value having the specific issue fixed against the list of other P1 and P2 bugs.
• If it doesn’t need to be fixed move it to the next realese.
• If interal issue
• Evaluate it’s impact on the project schedule
•  If high – then fix it now
• If Low – then move it to the next release
• Frequency of meetings
• IIn the beginning it was every day. This was effective in managing the list when there were only 3 or 4 people.
• As the number of contributing members grew this became difficult 
• We scaled it back to twice a week
• Near the end of the project we will met more frequently to make sure nothing slips through the cracks.
• Risk matrix 
• The risk matrix is a simple spreedsheet that contains each feature listed in the left column and each build number across the top. Cells get updated on a daily basis with 3 status codes – Green, Red and Grey. Green means everything is good with that feature. Red means there is a problem and includes the bug number in the cell. Grey means that there is more testing or investigation to be done before a reliable status can be done.
• With the risk matrix the question then becomes can we live with the risk that feature X, Y and Z havne’t been thoroughly tested. 
• Is it worth slipping the schedule to have these features "fully" tested? 
•  We provide this document to the project stakeholders at varying frequencies during the project life cycle.  The information is always available but we don’t want information overload. Near the beginning of the project the interested in our ability to ship the software is generally low so we don’t send the document out. The week or two before release is a different story. There is a very high interest in this information in the last stages of the product so we provide the document on a daily basis.
• Doing this achives buy-in at all levels of project involvement. 
• This helps us stay away from becoming the "Quality gatekeepers".
•  All automation passes
•  Nightly automation passes
• We use our own product to test our product.
•  Requirements verified
• Walk through the requirements and make sure we have test case coverage for each item
• Test cases aren’t very detailed. (More on this later.
• 95% test case coverage
• This is useless metric even when used in context of the other heursitics.
• but, It makes those who don’t understand what we do feel really good
• Metrics 
•  Bug rates
• Double cross over graph
• High severity issues outstanding
• Ask the team!
• Ask the team "Can we ship?"
• Very good way to figure out if you are done testing.  
• You will be able to tell if they are comfortable with their own work.

Why are we this way.

• We are market driven and IP intensive
• Our goal is to be first to market with new products and maintain market leadership with current products.
• We deal with any missed bugs/features  afterwards.
• We need to respond quickly to customer concerns
• Lightweight approach to test documentation because product changes are very dynamic. 
• We don’t document every step required to run a test case because tomorrow – some of those steps won’t be around
• We lean towards writing test cases that get the point across but not at the expense of going into detail.

• All signs can point to go but we overlook something.
• We’ve done enough releases to feel comfortable that this list will cover all items to a suffecient depth
• It’s a trade off between breadth vs depth
• Confidence with this heuristic is still touchy when we release new products

A co-worker of mine – Kenny Kerr – has written a series of articles on MSIL (http://weblogs.asp.net/kennykerr/archive/2004/09/07/226232.aspx) over the past few months. The series intrigued me to learn more about MSIL but haven’t had/made much time to play around with it.

This weekend while I trying to figure how to obsfuscate PlateSpin’s code during the build process I noticed that Dotfuscator was working with MSIL and temporarily outputting a file with a .IL extension. Dotfuscator takes a compiled .Net application and changes the variable/method names etc for you so that when you view the MSIL with a tool like ILDASM it’s really hard to understand what is going on.

Kenny didn’t go into much detail on using ILDASM to view IL and conversly how to compile his IL examples into executable code you can play with. I’m going to provide some instructions on how you can disassemble an EXE, save it as IL, then compile the IL into an EXE so you can play with Kenny’s examples yourself.

There are two approaches.

1) Write your own code, disassemble it into IL, use an assembler to tranform IL into an EXE or;
2) Copy some of Kenny’s code and use the assembler to create an exe.

I’ll use option 1 to show you how to disassemble a simple HelloWorld C# command line program. I’ll use Option 2 to show you how to compile IL.

Disassembling a .Net Framework EXE

By taking this approach you will be able to read through Kenny’s posts and see some of the topics he was discussing and see them in use in your own program.

1) Write and compile a C# program with one method. A simple hello world program will do.

using System;

namespace HelloWorld
{
 class EntryPoint
 {
  [STAThread]
  static void Main(string[] args)
  {
   SayHello();
  }

  private static void SayHello()
  {
   Console.WriteLine("Hello");
  }
 }
}

2) Open the Ildasm UI
 It is stored here on my machine -> C:\Program Files\Microsoft Visual Studio .NET 2003\SDK\v1.1\Bin\Ildasm.exe

3) Choose File->Open – Select your HelloWorld executable

4) What you will see is in the ILDASM UI is tree view of your program in its MSIL form.

5) Navigate the tree until you find your method and double click on it. Here you will see the MSIL for your method – this should look familiar to those that have read Kenny’s articles on MSIL.

.method private hidebysig static void  SayHello() cil managed
{
  // Code size       11 (0xb)
  .maxstack  1
  IL_0000:  ldstr      "Hello"
  IL_0005:  call       void [mscorlib]System.Console::WriteLine(string)
  IL_000a:  ret
} // end of method EntryPoint::SayHello

If it doesn’t look familiar then you haven’t read Kenny’s articles and you should go here http://weblogs.asp.net/kennykerr/archive/2004/09/07/226232.aspx

6) Saving your EXE as IL
- In the Ildasm tool Choose File -> Dump
- You will be prompted with a screen that has options for file encoding and other parameters.
- For the purpose of the example we will leave everything at it’s default.
- Find a location and save your IL file.

7) Now you can view all the MSIL that the compiler made from your HelloWorld example.

For a 17 line program in C# I got 103 lines of MSIL. 106 if you count the comments at the end of the file.

You should now open up this file and inspect it. You will be able to see some of the concepts Kenny explains in this file.

Creating a EXE from IL – My Experimental Approach

My instructions here are steps with my thought process thrown in where it wasn’t clear what I should do next.

1) Read Kenny’s introductory article on MSIL http://weblogs.asp.net/kennykerr/archive/2004/09/07/226300.aspx

2) Copy the MSIL code from the article – It looks like this

.method static void main()
{
    .entrypoint
    .maxstack 1

    ldstr "Hello world!"
    call void [mscorlib]System.Console::WriteLine(string)

    ret
}

3) Save it to a file with a .IL extension –> C:\Example.IL

4) Run the ILASM tool (note the missing D)
ILASM /exe C:\Example.IL
 The ILASM tool is in a different place than the ILDASM tool. On my computer it’s here -> C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Ilasm.exe

The /exe switch tells ILASM to output an EXE. You can also change this to be a DLL.

5) Run the resulting exe from a dos prompt.

Wait a minute – it doesn’t work. I got a VS Just-In-Time debug prompt. I chose No to not debug. The dos prompt shows the following output.

Unhandled Exception: System.IO.FileLoadException: The module ‘KennysExample.EXE’ was expected to contain an assembly manifest.
File name: "KennysExample.EXE"

Hmmm….something fishy here – Kenny is leaving out some details for the purpose of his example.

From Kenny’s first post – "Since I doubt anybody reading this will actually go and start writing code in MSIL, I will skim over some of the uninteresting details and focus on things like defining types, writing methods, calling instructions and handling exceptions."

Ahh – Kenny my friend – some of us will attempt to write MSIL and want to run your examples :

So now to figure out what is missing.
The error message says that the EXE was supposed to contain an assembly manifest. What is that? Let’s look at the MSIL from our original HelloWorld Example.

The MSIL at the top of the file mentions something about an assembly.

//  Microsoft (R) .NET Framework IL Disassembler.  Version 1.1.4322.573
//  Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                         // .z\V.4..
  .ver 1:0:5000:0
}
.assembly HelloWorld
{
  .custom instance void [mscorlib]System.Reflection.AssemblyCopyrightAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyKeyFileAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyDelaySignAttribute::.ctor(bool) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyTrademarkAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyKeyNameAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyProductAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyCompanyAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyConfigurationAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyDescriptionAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyTitleAttribute::.ctor(string) = ( 01 00 00 00 00 )
  // — The following custom attribute is added automatically, do not uncomment ——-
  //  .custom instance void [mscorlib]System.Diagnostics.DebuggableAttribute::.ctor(bool,
  //                                                                               
bool) = ( 01 00 01 01 00 00 )
  .hash algorithm 0×00008004
  .ver 1:0:1982:25786
}
.module HelloWorld.exe
// MVID: {FAB29BDB-0D01-4B46-9F87-4DA3716CC18B}
.imagebase 0×00400000
.subsystem 0×00000003
.file alignment 4096
.corflags 0×00000001
// Image base: 0x06fc0000
//
// ============== CLASS STRUCTURE DECLARATION ==================

6) Copy and past this at the top of your Example.il file (C:\Example.IL) before the code that was already in there.
7) Change the references to HelloWorld to Example
Repeat Steps 4 and 5

The Exe should now execute and you can continue trying out Kenny’s Examples.

Some feedback from Kenny regarding my Article before I posted –

"All the lines of code in step 5 can be replaced with the following two lines.

.assembly extern mscorlib {}
.assembly Example {}

The rest the assembler can deduce based on defaults."

Have fun with MSIL!