Data cleaning
Data cleaning is the process of identifying and correcting errors, inconsistencies, and quality issues in a dataset before it is used for modeling. It is f...
Turning Messy Reality into Reliable Training Data
Data cleaning is the process of identifying and correcting errors, inconsistencies, and quality issues in a dataset before it is used for modeling. It is frequently the most time-consuming phase of a machine learning project — industry surveys consistently find that practitioners spend 60 to 80 percent of their time on data preparation rather than modeling.
Common data quality problems include duplicate records, inconsistent formatting, entry errors, and structural issues. A customer database might have the same person listed under "John Smith," "J. Smith," and "JOHN SMITH" as three separate records. Duplicate rows inflate counts and skew statistics. Structural problems like a numeric column stored as text prevent algorithms from processing the data correctly.
Inconsistent categories are subtle but damaging. A country field might contain "US," "USA," "United States," and "united states" all meaning the same thing. A model trained on this data might treat them as four different countries, creating noise in its predictions. Standardizing categories — mapping all variants to a canonical form — is a key cleaning step.
Incorrect values require domain knowledge to detect. An age of 200 or a temperature of minus 500 degrees Celsius are impossible values that signal data entry errors. A salary of zero for an employed person may indicate a missing value incorrectly recorded. Validating values against realistic ranges and cross-checking related fields helps surface these issues.
A concrete example: a healthcare dataset includes patient weights ranging from 2 kg to 400 kg. Values below 2.5 kg likely represent newborns recorded on an adult patient form, and values above 250 kg are probable entry errors. Both should be flagged and investigated before training.
Exam tip: be ready to name at least four types of data quality issues and describe a specific technique for addressing each. Showing awareness that cleaning is iterative and domain-driven impresses interviewers.