Package Design vignette for {cleanepi}

Modules in {cleanepi}

In this section, we provide a detailed description of the way that every module is built.

1. Standardization of column names

This module is designed to standardize the style and format of column names within the target dataset, offering users the flexibility to specify a subset of:

• focal columns to preserve in their original format, and

• columns to be renamed i.e. given a new name chosen by the user.

• Main function: standardize_column_names()

• Input:

  • A data.frame or linelist object.
  • Optionally, a vector of focal column names and a vector of column names to be renamed in the form of new_name = "old_name". If not provided, all columns will undergo standardization.

• Output:

  • The input object with standardized column names.

• Report:

  • A two-column table displaying the initial and current column names for each updated column in the original dataset.

• Mode:

  • Explicit

By incorporating the standardize_column_names() function, {cleanepi} streamlines the process of ensuring consistency and clarity in column naming conventions, thereby enhancing the overall organization and readability of the dataset.

2. Removal of empty rows and columns and constant columns

This module aims at eliminating irrelevant and redundant rows and columns, including empty rows and columns as well as constant columns. The main function was initially (i.e. up to version 1.0.2) built based on the {janitor} R package. We used janitor::remove_empty() and janitor::remove_conatant() to remove empty rows and columns and constant columns respectively. In janitor::remove_empty(), the empty rows are removed first, then the empty columns. This maximizes the chance of keeping more columns after this operation. As we noticed that the removal of the constant data might still result in a dataset with some empty row and/or columns and constant columns, we introduced the concept of iterative constant data removal in more recent versions of the package (> v.1.0.2). This means that the process of removing constant data is performed iteratively until there is no constant data. The report made from this operation informs about what rows and columns were removed at every iteration.

• Main function: remove_constants()
• Input: Accepts a data.frame or linelist object, along with:
  • A cut-off that determines the percent of missing values beyond which a row or column should be deleted.
• Output: Returns the input object after applying the specified operations.
• Report:
  • A data frame with four columns showcasing the removed empty and columns and constant columns at every iteration.
• Mode:
  • explicit

3. Detection and removal of duplicates

This module is designed to identify and eliminate duplicated rows.

• Main functions: find_duplicates(), remove_duplicates()
• Input: Accepts a data.frame or linelist object, along with optional parameters:
  • Vector of target columns (default is to consider all columns; possible to use linelist_tags to consider tagged variables only when the input is a linelist object).
• Output: Returns the input object after applying the specified operations.
• Report:
  • Two table(s) showcasing the found and removed duplicates.
  • A character vector with the columns used during duplicates removal.
• Mode:
  • explicit

Through the remove_duplicates() function, users can streamline their dataset by eliminating redundant rows, thus enhancing data integrity and analysis efficiency.

4. Replacement of missing values with NA

This module aims to standardize and unify the representation of missing values within the dataset.

• Main function: replace_missing_values()
• Input: Accepts a data.frame or linelist object, along with:
  • A vector of column names (if not provided, the operation is performed across all columns)
  • A string or a vector of strings representing the current missing values (default value is cleanepi::common_na_strings)
• Output: Returns the input object with all missing values represented as NA.
• Report: Generates a three-column table featuring index, column, and value for each missing item in the dataset.
• Mode:
  • explicit

By utilizing the replace_missing_char() function, users can ensure consistency in handling missing values across their dataset, facilitating accurate analysis and interpretation of the data.

5. Standardization of date values

This module is dedicated to convert date values in character columns into ISO8601 Date format, and ensuring that all dates fall within the expected user-provided timeframe.

• Main function: standardize_dates()
• Input: Accepts a data.frame or linelist object, along with:
  • A vector of targeted date columns (automatically determined if not provided)
  • Tolerance threshold that defines the % of missing (out of range values converted to NA) values to be allowed in a converted column. When % missing values exceeds or is equal to it, the original values are returned (default value is 40%)
  • format (default value is NULL)
  • timeframe (default value is null)
• Output: Returns the input object with standardized date values formatted as %Y-%m-%d.
• Report:
  • A two-column table listing the columns where date values were standardized.
  • A three-column table featuring index, column name, and values that fall outside the specified timeframe.
  • A data frame featuring date values that can comply with more than one specified format. Users can update the standardized date values with the correct ones when it’s appropriated.
• Mode:
  • explicit

By employing the standardize_dates() function, users can ensure uniformity and coherence in date formats across their dataset, while also validating the temporal integrity of the data within the defined timeframe.

6. Standardization of subject IDs

This module is tailored to verify whether the values in the column uniquely identifying subjects adhere to a consistent format. It also offers a functionality that allows users to correct the inconsistent subject ids.

• Main function: check_subject_ids()
• Input: Accepts a data.frame or linelist object, along with:
  • The name of the ID column
  • Strings for prefix, suffix,
  • A numerical range within the IDs
  • An integer representing the expected number of characters in a subject id.
• Output: Returns the input object with standardized subject IDs.
• Report: Generates a two-column table featuring index and value of each subject ID that deviates from the expected format.
• Mode:
  • explicit

The correct_subject_ids() function can be used to correct the identified incorrect subject ids. In addition to the input data, it expects a data frame with two columns from and to containing the wrong and the correct ids respectively.

By utilizing the functions in this module, users can ensure uniformity in the format of subject ids, facilitating accurate tracking and analysis of individual subjects within the dataset.

7. Dictionary based substitution

This module facilitates dictionary-based substitution, which involves replacing existing values with predefined ones. It replaces entries in a specific columns to certain values, such as substituting 1 with “male” and 2 with “female” in a gender column. It also interoperates seamlessly with the get_meta_data() function from {readepi} R package.

Note that the clean_using_dictionary() function will return a warning when it detects unexpected values in the target columns from the data dictionary. These unexpected values can be added to the data dictionary using the add_to_dictionary() function.

• Main function: clean_using_dictionary()
• Input: Accepts a data.frame or linelist object, along with a data dictionary featuring the following column names: options, values, and order.
• Output: Returns the input object where the specified options are replaced by their corresponding values.
• Report: Generates a three-column table with index, column, and value for each unexpected value encountered in a targeted column.
• Mode:
  • explicit

By leveraging the clean_using_dictionary() function, users can streamline and standardize the values within specific columns based on predefined mappings, enhancing consistency and accuracy in the dataset.

8. Conversion of values when necessary

This module is designed to convert numbers written in letters to numerical values, ensuring interoperability with the {numberize} package.

• Main function: convert_to_numeric()
• Input: Accepts a data.frame or linelist object, along with:
  • A vector of column names to be converted into numeric. When not provided, the target columns are those returned by the scan_data() function
  • A string that denotes the language used in the text. The current version supports English, French and Spanish.
• Output: Returns the input object with values in the target columns converted into numeric format.
• Report: Generates a three-column table with index, column, and value for each unrecognized value in the dataset (strings that could not be converted into numeric).
• Mode:
  • explicit

By employing the convert_to_numeric() function, users can seamlessly transform numeric representations written in letters into numerical values, ensuring compatibility with the {numberize} package and promoting accuracy in numerical analysis.

9. Verification of the sequence of date-events

This module provides functions to verify whether the sequence of date events aligns with expectations. For instance, it can flag rows where the date of admission to the hospital precedes the individual’s date of birth.

• Main function: check_date_sequence()
• Input: Accepts a data.frame or linelist object, along with:
  • A vector of date column names to be considered
• Output: Returns the input object and sends a message to inform about the presence of rows where the expected sequence of dates is not respected.
• Report: Generates a table listing the incorrect rows from the specified columns.
• Mode:
  • explicit

By using the check_date_sequence() function, users can systematically validate and ensure the coherence of date sequences within their dataset, promoting accuracy and reliability in subsequent analyses.

10. Transformation of selected columns

This module is dedicated to performing various specialized operations related to epidemiological data analytics. In the current version of the package, this module includes the following functions:

• Main function: timespan()
• Input: Accepts a data.frame or linelist object, along with:
  • The name of the column of interest
  • The reference date value or column
  • The time and remainder units (possible values are days, weeks, months, or years)
  • The names of the newly generated columns
• Output: Returns the input object with clean data and one or two additional columns with values in the specified time unit.
• Report: None.
• Mode:
  • explicit

By leveraging the timespan() function, users can efficiently compute and integrate time span information into their epidemiological dataset based on user-defined parameters, enhancing the analytics capabilities of the dataset.

Surrogate functions

1. scan_data(): This function is designed to generate a quick summary of the dataset, offering insights into the composition of each character column. It calculates the percentage of values belonging to different data types such as character, numeric, missing, logical, and date. This summary can help analysts and data scientists understand the structure and content of the dataset at a glance.

2. print_report(): This function is used for displaying the report detailing the result of the cleaning operations executed on the dataset. It likely presents information about the data cleaning processes performed, such as handling missing values, correcting data types, removing duplicates, and any other transformations applied to ensure data quality and integrity.

These surrogate functions play crucial roles in the data analysis and cleaning workflow, providing valuable information and documentation about the dataset characteristics and the steps taken to prepare it for analysis or modelling.