Data Cleaning

The Odoo Data Cleaning app keeps the database neat and organized throughout the use of its provided features:

• Deduplicates: merges or removes duplicate entries to ensure data is unique.

• Recycles: identifies outdated records to either archive or delete them.

• Merges: combines multiple similar entries into one streamlined record.

• Formats: standardizes text data by finding and replacing it according to specified needs.

Through customizable automated rules (or manual praxis), individual records and accompanying text data in the database will remain up-to-date and consistently formatted, tailored to company specifications.

Install modules

The Data Cleaning application consists of several modules. Install the following to access all available features:

Name	Technical name	Description
Data Recycle	data_recycle	Base module to enable the recycle feature, available on Odoo Community edition.
Data Cleaning	data_cleaning	Enables field cleaning feature to format text data across multiple records, available only on Odoo Enterprise edition.
Data Cleaning (merge)	data_merge	Enables the deduplication feature to find similar (or duplicate) records, and merge them, available only on Odoo Enterprise edition.

Additionally, several app-specific modules are available

CRM Deduplication	data_merge_crm	Enables the deduplication feature on the CRM app, and uses the CRM default merging feature.
Helpdesk Merge action	data_merge_helpdesk	Enables the merge feature for the Helpdesk app.
Project Merge action	data_merge_project	Enables the merge feature for the Projects app.
UTM Deduplication	data_merge_utm	Enables the merge feature for the UTM Tracker app.
WMS Accounting Merge	data_merge_stock_account	Creates a warning in cases of products merging that may affect inventory valuation, if the Inventory app is installed.

Deduplication

On the Duplicates dashboard (Data Cleaning app ‣ Deduplication), Odoo suggests groups of similar records to be merged by matching conditions within the records set by the deduplication rules.

The RULE sidebar lists each of the active deduplication rules, and displays the total number of duplicates detected beside each rule.

By default, the All rule is selected. Displayed records are grouped by their rule, with a Similarity rating (out of 100%) in the list view, with the following columns:

• Created On: the date and time the original record was created.

• Name: the name or title of the original record.

• Field Values: the original record’s values for the fields used to detect duplicates.

• Used In: lists other models referencing the original record.

• ID: the original record’s unique ID.

• Is Master: the duplicates are merged into the master record. There can be one master record in a grouping of similar records.

Select a specific rule in the RULE sidebar to filter the duplicate records.

Merge duplicate records

To merge records, first choose a master record within the grouping of similar records. The master record acts as the base, at which any additional information from similar records are merged into.

Optionally, no master record can be set, leaving Odoo to choose a record at random to merge into.

Next, click the Merge button at the top of the similar records grouping. Then, click Ok to confirm the merge.

Once a record is merged, a message is logged in the chatter of the master record, describing the merge. Certain records, like Project tasks, are logged in the chatter with a link to the old record for a convenient reference.

Tip

Discard groupings by clicking the DISCARD button. Upon doing so, the grouping is hidden from the list and archived.

View discarded groupings by selecting the Discarded filter from the Search… bar.

Deduplication rules

The Deduplication Rules page (Data Cleaning app ‣ Configuration ‣ Rules: Deduplication) is where the conditions for records to be detected as duplicates can be set.

These rules can be configured for each model in the database, and with varying levels of specificity.

Tip

The deduplication rules run once every day, by default, as part of a scheduled action chron (Data Merge: Find Duplicate Records). However, each rule can be ran manually anytime.

Modify a deduplication rule

Select a default rule to edit, or create a new rule by clicking on the New button, on the Deduplication Rules page (Data Cleaning app ‣ Configuration ‣ Rules: Deduplication).

First, choose a Model for this rule to target. Selecting a model updates the rule title to the chosen model.

Optionally, configure a Domain to specify the records eligible for this rule. The number of eligible records is shown in the # record(s) link.

Depending on the selected Model, the Duplicate Removal field appears. Choose whether to Archive or Delete merged records.

Next, select a Merge Mode:

• Manual: requires each duplicate grouping to be manually merged, also enables the Notify Users field.

• Automatic: automatically merges duplicate groupings, without notifying users, based on the records with a similarity percentage above the threshold set in the Similarity Threshold field.

Enable the Active toggle to start capturing duplicates with this rule as soon as it is saved.

Lastly, create at least one deduplication rule in the Deduplication Rules field, by clicking Add a line, under the Unique ID Field column.

• Select a field in the model from the Unique ID Field drop-down menu. This field is referenced for similar records.

• Select a matching condition in the Match If field to apply the deduplication rule, depending on the text in the Unique ID Field:

  • Exact Match: the characters in the text match exactly.

  • Case/Accent Insensitive Match: the characters in the text match, regardless of casing and language-specific accent differences.

Important

The rule does not capture duplicates without at least one deduplication rule set in the Deduplication Rules field.

Tip

A few more fields are available for an advanced configuration.

If on a multi-company database, the Cross-Company field is available. When enabled, duplicates across different companies are suggested.

Activate Developer mode (debug mode) to display the Suggestion Threshold field. Duplicates with a similarity below the threshold set in this field are not suggested.

With the rule’s configuration complete, either close the rule form, or run the rule manually to instantly capture duplicate records.

Manually run a deduplication rule

To manually run a specific deduplication rule at any time, navigate to Data Cleaning app ‣ Configuration ‣ Rules: Deduplication, and select the rule to run.

Then, on the rule form, select the Deduplicate button on the top-left. Upon doing so, the Duplicates smart button displays the number of duplicates captured.

Click on the Duplicates smart button to manage these records.

Recycle records

Use the Recycle Records feature to rid the database of old and outdated records.

On the Field Recycle Records dashboard (Data Cleaning app ‣ Recycle Records), Odoo detects records that can be archived or deleted, by matching conditions within the records set by the recycle record’s rules.

The RECYCLE RULES sidebar lists each of the active recycle record rules, and displays the total number of records detected beside each rule.

By default, the All option is selected. Records are displayed in the list view, with the following columns:

• Record ID: the ID of the original record.

• Record Name: the name or title of the original record.

Select a specific rule in the RECYCLE RULES sidebar to filter the duplicate records.

To recycle records, click the Validate button on the row of the record.

Upon doing so, the record is recycled, depending on how the rule is configured, to be either archived or deleted from the database.

Tip

Discard groupings by clicking the Discard button. Upon doing so, the record is hidden from the list, and is not detected by the recycle rule again in the future.

View discarded records by selecting the Discarded filter from the search bar drop-down menu.

Recycle record rules

The Recycle Records Rules page (Data Cleaning app ‣ Configuration ‣ Rules: Recycle Records) is where the conditions for records to be recycled can be set.

These rules can be configured for each model in the database, and with varying levels of specificity.

Tip

The recycle rules run once a day, by default, as part of a scheduled action chron (Data Recycle: Clean Records). However, each rule can be run manually anytime.

By default, no recycle record rules exist. Click the New button to create a new rule.

On the recycle record rule form, first choose a Model for this rule to target. Selecting a model updates the rule title to the chosen model.

Optionally, configure a Filter to specify the records eligible for this rule. The number of eligible records is shown in the # record(s) link.

Next, configure the field and time range for how the rule detects the records to recycle:

• Time Field: select a field from the model to base the time (Delta).

• Delta: type the length of time, which must be a whole number (e.g. 7).

• Delta Unit: select the unit of time (Days, Weeks, Months, or Years).

Then, select a Recycle Mode:

• Manual: requires each detected record to be manually recycled, and enables the Notify Users field.

• Automatic: automatically merges recycled groupings, without notifying users.

Lastly, select a Recycle Action to either Archive or Delete records. If Delete is selected, choose whether or not to Include Archived records in the rule.

With the rule’s configuration complete, either close the rule form, or run the rule manually to instantly capture records to recycle.

Example

A recycle rule can be configured to delete archived leads and opportunities that were last updated a year ago, and with a specific lost reason, by using the following configuration:

• Model: Lead/Opportunity

• Filter:

  • Active is not set

  • Lost Reason is in Too expensive

• Time Field: Last Updated on (Lead/Opportunity)

• Delta: 1

• Delta Unit: Years

• Recycle Mode: Automatic

• Recycle Action: Delete

• Include Archived:

Manually run a recycle rule

To manually run a specific recycle rule at any time, navigate to Data Cleaning app ‣ Configuration ‣ Rules: Recycle Records, and select the rule to run.

Then, on the rule form, click the Run Now button on the top-left. Upon doing so, the Records smart button displays the number of records captured.

Click the Records smart button to manage these records.

Field cleaning

On the Field Cleaning Records dashboard (Data Cleaning app ‣ Field Cleaning), Odoo suggests formatting changes to data in fields of a record, to follow a convention set by the field cleaning rules.

Merge action manager

The Merge Action Manager (Data Cleaning app ‣ Configuration ‣ Merge Action Manager) enables or disables the Merge action available in the Actions menu for models in the database.