configure deduplication algorithm for API-imported findings

[fuzzah]

Hello!
I import findings via the findings API endpoint with good values in the "unique id from tool" field.
DD seems to ignore this field as it marks my duplicates as just similar findings.
In my case actual duplicates have the same unique id from tool, but almost always their description differs slightly from each other. Title and other fields are the same.
Is there a way to configure deduplication algorithm for API-imported findings so that DD would either ignore the description field or rely solely on the unique id from tool field?

[fuzzah]

I found a way to do it!
The following instructions should work for the docker-compose installation method.
So, let's say we have a custom test type named "Some Test".

Edit the file docker-compose.yml to include the following env variable to the sections uwsgi, celerybeat, and celeryworker:

DD_DEDUPLICATION_ALGORITHM_PER_PARSER: '{"Some Test": "unique_id_from_tool"}'

Available algorithms are defined in the file dojo/settings/settings.dist.py.

Then find the following code in the file dojo/settings/settings.dist.py:

# Override the hardcoded settings here via the env var
if len(env('DD_DEDUPLICATION_ALGORITHM_PER_PARSER')) > 0:
    env_dedup_algorithm_per_parser = json.loads(env('DD_DEDUPLICATION_ALGORITHM_PER_PARSER'))
    for key, value in env_dedup_algorithm_per_parser.items():
        if key in DEDUPLICATION_ALGORITHM_PER_PARSER:
            print("Replacing {} with value {} from env var DD_DEDUPLICATION_ALGORITHM_PER_PARSER".format(key, value))
            DEDUPLICATION_ALGORITHM_PER_PARSER[key] = value

and remove this check in the for loop:

if key in DEDUPLICATION_ALGORITHM_PER_PARSER:

This step is important, as our custom test is not listed in this dictionary, meaning that by default our env variable will get ignored.

Rebuild DD's images and restart the containers with docker-compose.

Now all new findings for test type "Some Test" are going to be deduplicated using the unique id from tool field.

If there's a need to deduplicate previously created findings of that test type, run the manage.py dedupe script:

docker exec -it django-defectdojo-uwsgi-1 ./manage.py dedupe --parser "Some Test"

After a minute or two the duplicates will be marked and deleted according to the settings in the "System Settings" tab of the web app.

Of course, when updating DD, we'll need to make sure our changes in these files persist, so I'd use git stash like so:

git stash
git pull
git stash pop