Marv: Mutations Analysis, Review and Visualisation

Marv is a visualization and review tool for mutation testing. It provides a standardized results format and visualization across all supported frameworks.

Marv displays the results of multiple frameworks simultaneously, allowing for review of results across many frameworks or even languages in one go.

Table of Contents

• Supported Frameworks
• Installation
  • Build from source
  • Libraries
• Usage
• Gallery
• Export Format
  • Mutations Format
  • Reviews Format
• Other

Supported Frameworks

A list of mutation testing frameworks that either are currently supported or will be supported in the future.

• 🏆 Supported out of the box
• ✅️ Supported with some configuration
• ⚠️ Experimental support
• 🚧 In development
• 🚫 Not currently supported

Framework	language	Support	Marv Version	Notes
Mull	C/C++	🏆	1.2.0	Supported through MTE schema
Dextool Mutate	C/C++	🚫
stryker-net	C#	🏆	1.2.0	Supported through MTE schema
go-mutesting	Go	🏆	1.2.1
hcoles/pitest	Java	✅️	1.0.0	See Pitest configuration
Major	Java	🚫
stryker-js	JavaScript	🏆	1.2.0	Supported through MTE schema
mutaml	OCaml	🚫
infection	PHP	🏆	1.2.0	Supported through MTE schema, additionally see infection readme
Cosmic Ray	Python	🚫
MutPy	Python	🚫
mutant	Ruby	🚫
mutest-rs	Rust	🏆	1.0.0
strkyer4s	Scala	🏆	1.2.0	Supported through MTE schema

Installation

Marv can be installed with the go tool. To use the installed executable, add the GOPATH environment variable to your system path. For more information run go help install.

go install github.com/SecretSheppy/marv@latest

Build from source

Builds exactly as a normal go project would. See the go.dev tutorial for more information. The target file to build is cmd/marv/main.go.

go build cmd/marv/main.go -o [output]

Libraries

If using a framework that requires external libraries, they will need to be set with the MARV_LIB_PATH environment variable. The alternative to this is to put the library into the local directory where the Marv tool is being run.

Usage

A simple guide of how to run Marv on a project for the first time. If at any point you need more information about one of the Marv commands, try using the help command.

marv help [command]

1. The first step is to ensure that Marv is correctly installed. If the Marv executable is correctly installed, running the Marv version command will output a version number. If an error is printed, then it likely means you need to add the Marv executable install location to your system path.

marv --version

2. Run the list command to see a list of all the frameworks that your installed version of Marv supports.

marv list

3. Then navigate to your project location and run the Marv init command with the list of frameworks you are using (Marv framework names are case-sensitive, so make sure to copy them correctly from the output of the list command). This will create a .marv.yml file in the directory that Marv was run in. The file will contain the default Marv configuration as well as a blank configuration for each framework that was listed.

marv init -f [framework] -f [framework] ...

4. Now fill in the configurations for each framework. Where frameworks require paths, using paths relative to a repository will allow you to safely commit the .marv.yml file for others to use. When finished with the configurations, simply run marv.

marv

5. If you have correctly configured the frameworks then that is it! Provided you keep the .marv.yml configuration file then all you have to do in future is simply run marv.

Gallery

Screenshots of the Marv user interface showing results from various frameworks.

Marv Results Overview: Showing results from stryker-net run on itself	Marv Pitest Results: Showing hcoles/pitest mutants inline with a file from guava
Marv mutest-rs Results: Showing mutest-rs mutants inline with a file from alacritty	Marv Infection PHP Mutant: Showing an isolated Infection mutant inline with a file from its own source

Export Format

Marv exports both the mutations and reviews as a .json marshal of its internal mutations format for all frameworks. By using the -m or --merge flags, the results from all frameworks are merged into one large .json file.

Mutations Format

The mutations format follows the internal structures defined in internal/mutations. The basic structure is file path > conflict region > mutation. Marv uses conflict regions or internally called mutations.Conflict to wrap all mutations that would conflict with each other if just rendered inline due to overlaps.

Any ID field is a UUID created by Marv. Where frameworks create mutant identifiers, they are stored against the mutant as FrameworkMutantID.

{
    "path/file.lang": [
        {
            "ID": "00000000-0000-0000-0000-000000000000",
            "StartLine": 94,
            "EndLine": 94,
            "Mutations": [
                {
                    "ID": "00000000-0000-0000-0000-000000000000",
                    "FrameworkMutantID": "",
                    "Description": "some description here",
                    "Operation": "some operator here",
                    "Start": {
                        "Line": 94,
                        "Char": 11
                    },
                    "End": {
                        "Line": 94,
                        "Char": 32
                    },
                    "Status": "SURVIVED",
                    "Replacement": "some.Replacement.String.Here()"
                }
            ]
        }
    ]
}

Reviews Format

Reviews are exported against the corresponding mutations Marv Mutation ID and their Framework Mutation ID (if applicable). The review structure is defined in internal/review.

[
    {
        "MutationID": "00000000-0000-0000-0000-000000000000",
        "FrameworkMutationID": "",
        "Framework": "mock-fw",
        "Review": "An example review"
    }
]

Other

Icon Licenses (icons.md)