FinlogiK Statistics Collector for FxCop

Project License Changed

Mon, 15 Feb 2010 16:03:22 GMT

Copyright (c) 2007, FinlogiK Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of FinlogiK Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Updated Wiki: Home

dgouin — Tue, 10 Nov 2009 19:45:12 GMT

The FinlogiK statistics collector tool for FxCop is intended to help development teams to track the progress of an FxCop backlog cleanup effort. It is capable of counting total remaining and TODO rule violations in an FxCop 1.35 project.

How does it work?

The tool is intended to count two types of FxCop violations:

The number of "TODO" violations that are flagged for short-term clean-up as part of an immediate backlog (i.e.: violations of rules that have already been activated as decribed here, and
The total number of violations left to fix before the entire backlog has been addressed for all existing FxCop rules.

This counting is accomplished by running FxCop multiple times and evaluating the results of the report produced by FxCop for each execution. In particular, the following operations are carried out by the tool:

The tool removes all TODO exclusions (if applicable) from the target FxCop project file.
The tool runs FxCopCmd.exe to count the TODO violations. Any active violation found in the resulting FxCop report is assumed to represent a TODO task.
The tool modifies the FxCop project file to activate all rules.
The tool re-runs FxCopCmd.exe to count the total violations. Any active violation found in the resulting FxCop report is counted.

N.B.: Exclusions created via SuppressMessageAttribute instances in source code are never counted as violations by the tool.

Installation

Download and install FinlogiK.FxCop.Statistics.Setup.msi. (Installation should be performed under a local administrator account.)
After installation on at least one machine, run the CreateDatabase.sql SQL script from the install directory against a SQL Server 2005 instance in order to create the statistics database.
After database creation, add any users who should be able to collect and view statistics to the public database role.

Client configuration

Launch the FinlogiK.FxCop.Statistics.Windows.exe application.
Use the database configuration dialog to specify the connection string to the statistics database. (If you cannot use the connection string builder, you should still be able to provide a connection string in the following format (without the quotes): "Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=FxCopStatistics;Data Source=SomeSql2005Instance".)

Adding target applications

Before using the tool, you will need to provide a list of applications/projects for which you wish to collect FxCop statistics. To do so, follow these steps:

Launch the FinlogiK.FxCop.Statistics.Windows.exe application.
Click the "Application Profiles" button in the application toolbar.
Enter a row for each application/project for which you wish to be able to collect statistics:
1. Use the "Name" column to specify the name you wish to use to identify the application within the statistics tool.
2. Use the "Active" column to indicate whether one should be able to provide new statistics for the application using the tool. This will usually be set to true for a new application.
3. Use the "Treat all exclusions as TODOs" column to indicate whether all exclusions found the in FxCop project file should be assumed to be backlog cleanup tasks (as opposed to "real" violation suppressions). This will usually be true iff you are using SuppressMessageAttribute for your real exclusions and the FxCop project for your TODO exclusions.
4. The "FxCop project path" column is a user- and machine-specific path to the FxCop project path for the application. It is an optional value that is only used if you perform a statistics collection on that particular client machine.
5. The "FxCopCmd.exe path" column is a user- and machine-specific path to FxCopCmd.exe (usually found in C:\Program Files\Microsoft FxCop 1.35\FxCopCmd.exe). It is an optional value that is only used if you perform a statistics collection on that particular client machine.

Specifying TODO prefixes

Unless all the applications for which you will be using the statistics tool are flagged to treat all exclusions found in the FxCop project file as TODOs, you will need to help the tool identify which of the in-project exclusions are TODOs as opposed to real exclusions. This is done by specifying a list of prefixes that the tool will attempt to match in the last note recorded against any given in-project exclusion.
To provide a prefix list, follow these steps:

Launch the FinlogiK.FxCop.Statistics.Windows.exe application.
Click the "TODO prefixes" button in the application toolbar.
Enter a row for each prefix (case-insensitive) that should identify a TODO exclusion.

For example, the current TODO prefix list in use at FinlogiK is the following:

?
pas le temps
temporary
to check
to do
todo
too many problems
too many to fix
will be automatically converted
will be fixed

Since the appropriate prefix list may evolve as developers add new exclusions to your FxCop projects, you should consider periodically reviewing your in-project exclusions in order to identify any new TODO prefixes that may have crept in.

Running an analysis using the tool UI

Unless you are already using automated builds for your target projects, you will probably end up running statistics collection analyses from the tool UI. To launch an analysis run from the UI, follow these steps:

Launch the FinlogiK.FxCop.Statistics.Windows.exe application.
Click the "Run Analysis..." button in the application toolbar.
Select the application/project for which you wish to perform the analysis.
Click the "Start" button to initiate the analysis run.
Once the analysis run has completed, a message box will be shown indicating whether the analysis was successful and inviting you to view the resulting statistics should you wish to do so immediately.

Running an analysis from an automated build

The tool may be launched in command line mode by providing the following arguments (in this order):

The project name (required),
The statistics database connection string (optional),
The full path to FxCopCmd.exe (optional), and
The full path to the FxCop project file to be used in the analysis (optional).

The command line arguments are read by position, so you should provide values for all optional arguments if you provide values for any of them. If you do not provide the optional values, they will be read from the tool configuration for the build machine and user.

In practice, you probably won't want to perform a statistics collection at every automated build, particularly if you're using continuous integration. At FinlogiK, we've set things up so that we only run FxCop stats collection once per day for any given project.

Viewing analysis results

To view analysis results, open the tool UI and click the "View statistics" button in the toolbar. Select the application and statistics run for which you wish to view statistics. (The analysis runs are sorted in descending chronological order, and the timestamp shown is UTC.)

The four tabs represent the summary and detailed results for each of the "total" and "TODO" violation categories. The summary results show violation counts by rule category, and the detailed results show violation counts by rule.

If you wish to modify the category mappings (for example, to merge multiple categories of custom rules into one category) and/or sorting for the summary results, this can be done by manually editing the CategoryGroups and RuleCategories tables in the statistics database. The current version of the statistics tool does not include a UI for editing this data.

Known issues

This tool was developed for internal use at FinlogiK, and its functionality set is pretty close to the absolute minimum that we need for tracking our FxCop backlog cleanup efforts. There's quite a bit we would have liked to change before distributing a version here on CodePlex, but that would probably have delayed the first public release more of less indefinitely. So... Instead of no release at all, what you see here is exactly the same beastie we're using internally, warts and all.

We've added some of those those warts to the Issue Tracker for this project. Unfortunately, given limited resources, it's unlikely that we'll be addressing these issues any time soon, but at least you'll know that they bug us too... ;) If there's some change that you absolutely can't live without, please feel free to download the source code and modify it as necessary for your internal use.

Created Release: 1.0.4.0 (Nov 10, 2009)

dgouin — Tue, 10 Nov 2009 19:40:33 GMT

Application re-branded with the company new name.

Released: 1.0.4.0 (Nov 10, 2009)

Tue, 10 Nov 2009 19:40:33 GMT

Application re-branded with the company new name.

Source code checked in, #46219

dgouin — Tue, 10 Nov 2009 19:37:17 GMT

Rebranded the application

CREATED FEATURE: Richer database security model

calinoiu — Fri, 15 Feb 2008 16:07:53 GMT

Increased granularity of permissions via at least splitting of editor and reader roles would be advantageous.

CREATED FEATURE: UI for rule category editing and mapping

calinoiu — Fri, 15 Feb 2008 16:06:24 GMT

Rule categories must currently be edited directly in the database tables since the application does not include a UI for this purpose.

CREATED FEATURE: Command line argument syntax

calinoiu — Fri, 15 Feb 2008 16:05:28 GMT

It would be more convenient if command line arguments were named rather than posititional.

CREATED FEATURE: Storage of FxCop reports

calinoiu — Fri, 15 Feb 2008 16:04:46 GMT

In order to facilitate backward application of any statistics to be added in the future, it would be convenient if the FxCop reports for both TODO and total violations were stored in the database at the time of analysis.

CREATED ISSUE: Exception explanations

calinoiu — Fri, 15 Feb 2008 16:03:19 GMT

Exceptions are not currently mapped to explanations of how the user might address a problem. For the moment, it is expected that users are developers and are capable of examining/debugging the source code in order to figure out what they need to do to address any problems, including configuration issues.

CREATED ISSUE: Backend database platform

calinoiu — Fri, 15 Feb 2008 16:01:42 GMT

Only SQL Server is supported as a backend database.

UPDATED WIKI: Home

calinoiu — Fri, 15 Feb 2008 16:00:25 GMT

The FinRad statistics collector tool for FxCop is intended to help development teams to track the progress of an FxCop backlog cleanup effort. It is capable of counting total remaining and TODO rule violations in an FxCop 1.35 project.

How does it work?

The tool is intended to count two types of FxCop violations:

The number of "TODO" violations that are flagged for short-term clean-up as part of an immediate backlog (i.e.: violations of rules that have already been activated as decribed here, and
The total number of violations left to fix before the entire backlog has been addressed for all existing FxCop rules.

The tool removes all TODO exclusions (if applicable) from the target FxCop project file.
The tool runs FxCopCmd.exe to count the TODO violations. Any active violation found in the resulting FxCop report is assumed to represent a TODO task.
The tool modifies the FxCop project file to activate all rules.
The tool re-runs FxCopCmd.exe to count the total violations. Any active violation found in the resulting FxCop report is counted.

N.B.: Exclusions created via SuppressMessageAttribute instances in source code are never counted as violations by the tool.

Installation

Download and install FinRad.FxCop.Statistics.Setup.msi. (Installation should be performed under a local administrator account.)
After installation on at least one machine, run the CreateDatabase.sql SQL script from the install directory against a SQL Server 2005 instance in order to create the statistics database.
After database creation, add any users who should be able to collect and view statistics to the public database role.

Client configuration

Launch the FinRad.FxCop.Statistics.Windows.exe application.
Use the database configuration dialog to specify the connection string to the statistics database. (If you cannot use the connection string builder, you should still be able to provide a connection string in the following format (without the quotes): "Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=FxCopStatistics;Data Source=SomeSql2005Instance".)

Adding target applications

Before using the tool, you will need to provide a list of applications/projects for which you wish to collect FxCop statistics. To do so, follow these steps:

Launch the FinRad.FxCop.Statistics.Windows.exe application.
Click the "Application Profiles" button in the application toolbar.
Enter a row for each application/project for which you wish to be able to collect statistics:

Use the "Name" column to specify the name you wish to use to identify the application within the statistics tool.
Use the "Active" column to indicate whether one should be able to provide new statistics for the application using the tool. This will usually be set to true for a new application.
Use the "Treat all exclusions as TODOs" column to indicate whether all exclusions found the in FxCop project file should be assumed to be backlog cleanup tasks (as opposed to "real" violation suppressions). This will usually be true iff you are using SuppressMessageAttribute for your real exclusions and the FxCop project for your TODO exclusions.
The "FxCop project path" column is a user- and machine-specific path to the FxCop project path for the application. It is an optional value that is only used if you perform a statistics collection on that particular client machine.
The "FxCopCmd.exe path" column is a user- and machine-specific path to FxCopCmd.exe (usually found in C:\Program Files\Microsoft FxCop 1.35\FxCopCmd.exe). It is an optional value that is only used if you perform a statistics collection on that particular client machine.

Specifying TODO prefixes

Launch the FinRad.FxCop.Statistics.Windows.exe application.
Click the "TODO prefixes" button in the application toolbar.
Enter a row for each prefix (case-insensitive) that should identify a TODO exclusion.

For example, the current TODO prefix list in use at FinRad is the following:

?
pas le temps
temporary
to check
to do
todo
too many problems
too many to fix
will be automatically converted
will be fixed

Running an analysis using the tool UI

Launch the FinRad.FxCop.Statistics.Windows.exe application.
Click the "Run Analysis..." button in the application toolbar.
Select the application/project for which you wish to perform the analysis.
Click the "Start" button to initiate the analysis run.
Once the analysis run has completed, a message box will be shown indicating whether the analysis was successful and inviting you to view the resulting statistics should you wish to do so immediately.

Running an analysis from an automated build

The tool may be launched in command line mode by providing the following arguments (in this order):

The project name (required),
The statistics database connection string (optional),
The full path to FxCopCmd.exe (optional), and
The full path to the FxCop project file to be used in the analysis (optional).

The command line arguments are read by position, so you should provide values for all optional arguments if you provide values for any of them. If you do not provide the optional values, they will be read from the tool configuration for the build machine and user.

In practice, you probably won't want to perform a statistics collection at every automated build, particularly if you're using continuous integration. At FinRad, we've set things up so that we only run FxCop stats collection once per day for any given project.

Viewing analysis results

Known issues

This tool was developed for internal use at FinRad, and its functionality set is pretty close to the absolute minimum that we need for tracking our FxCop backlog cleanup efforts. There's quite a bit we would have liked to change before distributing a version here on CodePlex, but that would probably have delayed the first public release more of less indefinitely. So... Instead of no release at all, what you see here is exactly the same beastie we're using internally, warts and all.

We've added some of those those warts to the Issue Tracker for this project. Unfortunately, given limited resources, it's unlikely that we'll be addressing these issues any time soon, but at least you'll know that they bug us too... ;) If there's some change that you absolutely can't live without, please feel free to download the source code and modify it as necessary for your internal use.

Source code checked in

calinoiu — Thu, 14 Feb 2008 16:24:56 GMT

Initial commit

UPDATED RELEASE: 1.0.2.0 (Feb 14, 2008)

Thu, 14 Feb 2008 16:03:48 GMT

Initial release

UPDATED WIKI: Home

calinoiu — Thu, 14 Feb 2008 15:42:32 GMT