Data Quality
Duplicated Dose Readings
Detection of Data Duplication in Historion is an important Data Quality Management (DQM) tool and data integrity safeguard. Duplication of data can happen at multiple levels and in multiple circumstances if left unchecked. Data Duplication checking in Historion is strict and thorough.
Duplicated is highly undesirable in most circumstances and Historion includes safeguards against duplication rigorously. Dose Duplication is common, especially when importing large volumes of Dose Readings that have been collected in Provider files over a long period of time and where those files have been kept in different places and managed by different people.
The important task for Historion is to make the presence of Duplicate Readings visible to the RSO, so that the RSO can investigate and assess the legitimacy of suspected or potential Duplicate Dose Readings and whether records appearing in the data are unusual and need further investigation.
Frequent causes of Dose Duplication include;
Re-issued or “versioned” Dose Readings from the Dosimetry Provider
Provider Laboratory Error; process fault or failure
Provider Data Error; software fault or failure
Provider Human error in manual Dose Report changes
Client Organisation Human error in manual Dose Report changes
Double Badging (wearing multiple dosimeters)
Special purpose testing and research
Pregnancy Monitoring
Extremity Monitoring with left hand and right hand side readings
Client Organisation badge identification or administration or returns processing errors
Dosimetry Data Duplication scenarios detected by Historion include;
Dose Report External File Name Duplication.
Dose Report File Content duplication.
Identical Dose Readings within the same Dose Report.
The same Dose Reading present in multiple Dose Reports.
Closely matching Dose Readings within the same Dose Report, for the same wearer and wear period, indicating potential duplication within the Dose Report.
Closely matching Dose Readings present in multiple Dose Reports, for the same Wearer and Wearing Period, indicating potential duplication across multiple Dose Reports.
Dose Reading Duplication Checking Levels
The following table explains the Dose Duplication checking levels in Historion. Each level has an example and further explanation below. Each level is tested as Dose Reports are imported.
Level
|
Duplicate Check
|
Definition and Considerations
|
1
|
Dose Report
Full File Path Duplication
|
Historion will not import a Dose Report file in either Single File Import Mode or Bulk File Import Mode if the full path of the file already exists in the list of Dose Reports imported. Historion will not even try to open the file. Full Dose Report File Path duplication is not allowed.
|
2
|
Dose Report CRC Value Duplication
|
Historion will not import a Dose Report in any mode if the Cyclic Redundancy Check (CRC) value for the Dose Report already exists in the list of Dose Reports imported. The CRC value is effectively a digital signature for the file. It is derived using all of the content inside the file. The CRC value check ensures that the exact content, (the same electronic version of a Dose Report file) regardless of any folder paths or names or any changed file names, cannot be imported twice.
|
3
|
Row Level Duplication Inside New Dose Report
|
If a Dose Report passes level 1 and 2 duplication checks, above, the Dose Readings inside the Dose Report are checked for duplication within the Dose Report being imported using row level CRC values derived for each Dose Reading row being imported.
A Duplicate Readings system setting (under the Admin tab / Settings / Configuration tab / then the Duplicate Readings setting) determines how Historion will respond to Duplicated Readings inside a Dose Report as the Dose Report is being imported.
There are three value options recognized by Historion for this setting; Ignore, Allow and Disallow, these options and their effects determine whether a Dose Report with duplicated Dose Readings inside it is allowed to be imported into the Historion database. The options for this setting are further described below as three separate scenarios.
All Dose Readings are subject to the level 4 and 5 checks during import, outlined below, if the Dose Report is allowed to be imported.
|
4
|
Compare Incoming
Dose Readings to Imported Dose Readings
|
The incoming Dose Readings in a new Dose Report are compared to the existing Dose Readings in the Historion database. The Dose Duplication setting is checked again for handling duplicates detected.
If the setting is “Ignore” any detected duplicates are removed from the inbound rows. If the setting is “Allow” the Dose Readings are imported and Duplicates will be detected in the final stage 5 check, below. If the setting is “Disallow” and any duplicates will result the import is denied.
|
5
|
Check ALL Dose Readings Imported
|
The final Duplication Detection that occurs toward the end of the Dose Report import process is a scan across the entire Historion database for Dose Readings that look like Duplicates. The criteria for Dose Readings that look like Duplicates includes: The Wearer, Wearing Date Range and one or more Dose Reading values match. Any Dose Readings detected as meeting these criteria and not already flagged in the system as Allowed are reported as Not Allowed duplicates. The number detected is shown as a message to the user after importing.
|
By default the Bulk Dose Report Import tool makes level 5 checks after all Bulk Import Processing.
Duplicate Check Level 1 – Dose Report File Path Duplication
If a Dose Report has already been imported and an attempt is made to import the Dose Report again using the Individual Dose Report Import option, from the exact same folder location and file name, the following message is displayed:
.png)
The Dose Report Single File Import page is displayed (behind the above message box) and can be accessed as soon as the User clicks OK as shown below.
Note that the originally imported Dose Report can be accessed by clicking the Imported Dose Report link;
.png)
Note that at this stage and the type of check involved, Historion is only rejecting the selected file on the basis of the full path (folder and sub folder location + file name) already existing in the Historion database, regardless of whether the file’s content is the same or not.
Only one instance of a full Dose Report file path (folder and sub folder location + file name) can exist in Historion at a time.
Bulk Dose Reporting Importing and Duplication
If a Dose Report has already been imported and an attempt is made to import the Dose Report again using the Bulk Dose Report Import tool and the file is still in its original folder location and has its original file name, the Dose Report will not be imported, as highlighted in the Bulk Import Log below;
.png)
Already Imported Log Result
The Bulk Dose Report importer is designed to attempt importing of all potential Dose Report files in a folder and sub folder structure. If errors or duplication are found for a Dose Report file, the Bulk Dose Report utility logs the issue as shown above and moves onto the next file.
This behaviour is by design.
Duplicate Check Level 2 – Dose Report CRC Duplication
If the same file in the Check 1 example above is moved to a different folder location or if the same file is renamed, or both, Historion will open the file for import and execute the next level, CRC value check for the file. If the exact digital content of the file already exists in Historion, which is what the derived CRC value represents, Historion will reject the file on due to CRC value duplication as shown below.
.png)
After selecting Import the Dose Report CRC duplication check executes:
.png)
If a single character, comma or space is altered in a Dose Report the CRC value of the file changes and it can be imported. If already imported, before modification, this will result in multiple Duplicate Dose Readings being detected. Never modify Dose Reports from their original Provider supplied files.
Duplicate Check Level 3 Scenarios
Historion settings allow for different preferences and circumstances surrounding the handling of duplicated Dose Readings detected within a Dose Report that may or may not be imported. Note that these options are only applicable to the single Dose Report import feature.
The Bulk Data Transfer tool and the Online Data Transfer feature are designed to keep importing large numbers of Dose Reports in an uninterrupted process and accordingly they ignore and do not import Dose Reports having duplicated Dose Readings within the Dose Report, then log the issue for the User to review.
There are three options for handling Dose Report duplication detected within a Dose Report;
Option 1 – Allow (default and recommended for ongoing use)
Option 2 – Disallow (some RSOs prefer this level of rigor, see below)
Option 3 – Ignore (use with caution, this setting is for a limited set of data migration scenarios)
Options 1 and 3 result in the following prompt, shown over the Single Dose Report Import Page;
.png)
The User can continue or cancel when options 1 or 3 are used. Option 2 stops the import process.
All these scenarios are demonstrated below.
Duplicate Check Level 3 - Option 1 “Allow” Setting (Default and Recommended)
The default Duplicate Dose Readings setting is “Allow”. This means Historion will allow the Duplicates to import into the database, as part of the import process, and then detect them in the final import stage at check 5.
This setting is not to be interpreted as just allow Duplicated Readings to come in and then leave them there. The setting is intended to allow subsequent review and follow up by exposing duplicated doses.
.png)
Double-click to change setting text
The “Allow” setting gives the RSO the opportunity to identify and handle duplicates, usually in association with the Dosimetry Provider. By importing Dose Readings identified as duplicates the RSO can view and work further with the Dose Readings and make and record decisions as required.
If the “Allow” setting is used, if a Dose Report file has Duplicate Readings inside it the Duplicate Check level 3 step pauses the import process and a User must allow Historion to continue or cancel the import process:
.png)
Duplicate Check Level 3 - Option 2 “Disallow” Setting
The Disallow setting is intended to prevent duplicate Dose Readings from being imported into the Historion database and it requires duplication to be resolved outside of the Historion database.
Some RSOs prefer this strict option, however as there are convenience tools provided with Historion for finding, grouping and managing Dose Reading duplicates, the “Disallow” setting may create extra work for the RSO in managing duplication outside of Historion using external tools such as Excel.
.png)
Double-click to change setting text
With the “Disallow” setting, if a Dose Report file has Duplicate Readings inside it the Duplicate Check level 3 step stops the import process and the User does not have the option to continue or cancel;
.png)
Duplicate Check Level 3 - Option 3 “Ignore” Setting (use with caution! Not for day-to-day use!)
The “Ignore” Dose Reading Duplication setting is used in very limited circumstances.
This setting instructs Historion to invisibly ignore any duplicated readings detected during import, in either stage 3 or stage 4, and instruct Historion to not import any detected duplicated Dose Readings into the database. This setting means Ignore (don’t import) Duplicate Readings, it doesn’t mean import all duplicates and then just ignore them.
Duplicated Dose Readings detected inside an incoming dose Report file or detected between the file and existing Dose Readings in the Historion database are discarded and not imported with other Readings, as though they did not exist.
.png)
.png)
The “Ignore” setting is used in a limited set of circumstances such as bulk importing discrete records into a parallel database, for purposes of cross-checking Dose Reading numbers in migrated data. “Ignore” is not recommended for day-to-day use in Historion implementations.
Dose Reading Duplication and Zero Value Readings
Historion includes a setting that instructs the duplication scanning algorithm whether or not Dose Reading values of zero will be included or ignored in scans for duplication:
.png)
The default value for this setting is 1, which means ignore Readings with zero values for checking for Dose Duplication. The alternative value is 0 (zero) which means include Readings with zero values.
Historically a zero for a Dose Reading outcome has been used inconsistently amongst providers. Providers have used a blank, have used “<MRD”, use a zero or an “M” to report Dose Readings that are less than the minimum reportable dose. Over time the meaning of a zero has become ambiguous.
There are reasons the default is set the way that it is:
Analysing and interpreting these outcomes, given the above, across thousands of Dose Readings is very time consuming. Historion ignores duplicate zero readings for convenience.
When detected during dose importing, duplication interrupts the import process for the RSO. The default setting ensures that the import process is not being interrupted for duplicated doses that have no measurable exposure and would be unnecessarily delaying the import.
Including zero value Dose Readings can significantly inflate duplicate numbers as shown in red in the data quality panel of the Home dashboard. RSOs find inflated data quality numbers overwhelming and may be less likely to follow these numbers to full data quality resolution.
High volumes of zero value Dose Reading duplicates will tend to hide more important Dose Reading Duplicates amongst less problematic duplicates.
Underreporting is not occurring in duplicated doses with zero exposure, so the default setting is not in conflict with the ALARA principle nor nonconservative.
There may be issues for averaging in custom created reports where numbers of Readings are higher than expected for duplicated zero result outcomes. Custom reports can be developed to examine and ignore zero Dose Readings with the Is Duplicate Dose Reading flag.
The RSO can change Historion's setting to include zero based readings on duplicate scans. If this is to be done and duplication rescanned a trial in a test copy of the Historion database is recommended to observe any increase in duplicate numbers.
The admin script Reset Cross Referenced Data must be executed after duplicate scanning to refresh duplicate numbers detected against the new setting.
Dose Reading Duplication and Blank (No Value) Readings
Blank readings have been used inconsistently amongst providers. A blank reading could mean there was no result, the dose was not reportable or the badge was not read. Blank readings have also been used as "spacers" to vertically pad the dose report results for display or grouping purposes, even though date range values are present against the Dose Reading.
The following points are important in understanding Historion’s handling of blank Dose Readings:
Historion's reports ignore blank readings as there is nothing to report or compare.
Any dose readings that are blank and may be duplicated based on Wearer and Date Range are not flagged as duplicated Dose Readings for follow up, by Historion.
In the case of blank readings used to stylise or pad Dose Readings within the Dose Report, if detected as duplicates these Dose Readings would stop a dose report from being imported.
Blank Dose Readings are still imported into Historion because they affect the calculation of the parent Dose Report’s CRC, but they are not scanned for determining duplication.
If blank readings were included in the duplicate detection list the number for the RSO to examine would be significantly inflated and important duplicates hidden amongst frequently meaningless duplication.
Dose Duplication Check Algorithm
The duplicate detection algorithm does not include the badge number. This means the dose outcome for a wearer being doubled-up is being reported regardless of the badge numbers involved. This will (and many times has) detect end user or provider data errors resulting in the duplicating.
If the badge numbers were included in the check the duplication would not be detected because the badge numbers would make the Dose Readings appear unique.
This also highlights double badging where coincidentally matched exposure outcomes occur, although this is less frequent.
Bulk Dose Report Importer Response to Reading Duplication
If a Dose Report has Duplicate Readings inside it and an attempt is made to import the Dose Report with using the Bulk Dose Report Import tool regardless of the stage 3 settings described above the Dose Report will not be imported as highlighted in the Bulk Import Log, shown below:
.png)
Duplicate Readings in Report Result
This behaviour is by design. Not importing problem Dose Reports ensures that the Individual Dose Report import feature will be used instead.
The individual Dose Report feature has many additional diagnostic routines built into it and it is designed to provide more verbose and more accurate troubleshooting detail, on a per Dose Report basis.
Duplicate Check Level 4 – Check and Flag Dose Reading Duplication after Import
Duplicate Check Level 4 is the most important Dose Readings Duplication check in Historion. Check Level 4 tests imported Dose Reading for a potential duplicate amongst the other Dose Readings imported in the Historion database for the same wearing period. The following Check Level 4 process steps are executed. Check Level 4 runs each time Dose Reports are imported or deleted. Level 4 has 9 steps as described below;
Step
|
Definition and Considerations
|
1
|
Load or reload Dose Readings Data Cache into memory
|
2
|
Clear internal table listing previously identified Dose Reading Duplicates from the last scan. This table records the internal Dose Reading IDs, instance and Group Numbers.
|
3
|
Analyse Dose Readings against each other, by wearer, and group Dose Reading Duplicates detected into Duplicate Groups then assign Instance Numbers within respective Groups
|
4
|
Repopulate listing of detected Dose Reading Duplicates from the current scan
This step records a Duplicate Group and Instance within the Group for Duplicate Readings
|
5
|
Compare Dose Readings previously flagged as Reading Duplicates to the new list detected from in the current scan and Set IsDuplicate flag = False for Dose Readings that are no longer Duplicates (due to deleted Dose Reports and/or Readings)
|
6
|
Compare the IsDuplicate flag of Dose Readings detected as Duplicates in the current scan against their existing flags in the Historion database. Set IsDuplicate flag = True for Dose Readings newly detected as Duplicates, such as after a new import
|
7
|
Reload Dose Readings Data Cache into memory
|
8
|
Update Dose Record Status Indicator icons where currently displayed in open pages
|
9
|
Report numbers of allowed verses not allowed Duplicates detected back to page for user
|
.png)
In Figure 130 above the Duplicate Level 5 check has been run. The last task is applying Dose Application Rules.
Handling Duplicate Readings
There are three steps in handling Dose Reading Duplication, described further below;
1.Identify Dose Reading Duplication occurrences
2.Decide on a course of action (4 scenarios are described here)
3.Replace, Allow, Delete or Ignore Dose Reading Duplicates in future we will include void here
Identifying Dose Reading Duplication Occurrences
Identifying Dose Reading Duplication is straightforward. In the first instance Historion notifies the User when Dose Reading Duplication has been detected during import. Dose Reading Duplication may be checked manually by selecting the Duplicates features in the Data Quality Panel of the Dosimetry tab then clicking Re-Scan. This can be a time-consuming process. An example of detecting duplicated Dose Readings is shown below;
Duplicates DQM Feature
.png)
Managing Dose Reading Duplication requires either System Admin or Centre Admin permissions.
Duplicates are listed and managed in the Duplicate Readings listing page, accessed through the Dosimetry Tab, Data Quality Panel, Duplicates DQM feature icon.
Duplicate Groups and Instances
The rows in the Duplicate Readings Listing Page represent individual Dose Readings that Historion has detected as being duplicated. The Dose Reading Duplicates are sorted and presented in sets in the list known as Duplicate Groups.
The Duplicate Readings List includes a mixture of Dose Reading columns and some columns normally associated with Dose Reports, such as the [Dose Report Readings] column which shows the number of Dose Readings in the parent Dose Report that each duplicated Dose Reading belongs to.
The layout of the Duplicate Readings List and the default columns displayed were selected to assist in identifying groups of Dose Reading Duplicates and to quickly identify whether different Dose Reports are involved with the duplication. Historion highlights rows belonging to alternate Duplicate Groups in different Colour Bands for clarity.
Duplicate Group “Set” Grey/White/Grey Colour Banding
.png)
Duplicate Instance No. of Readings in Dose Report
The Grey-White-Grey colour banding makes the sets of repeated Dose Readings or Duplicate Groups easier to see on the page. The alternating bands are shown with the green arrows above.
Each Duplicate Group is assigned a non-permanent Group Number when Historion runs its Duplicate Detection process. The Duplicate Group number is a simple numeric sequence and a new number may be assigned to a group if Dose Reports are imported or deleted and Duplicate Detection re-run. The Duplicate Group 2 is emphasized in red in the example above.
Each Dose Reading that is a Detected Duplicate is assigned a simple numeric sequence within its parent Duplicate Group termed Duplicate Instance. The Duplicate Instance number is a simple numeric sequence and a new number may be assigned to a Dose Reading Duplicate if Dose Reports are imported or deleted and Duplicate Detection re-run.
The highest Duplicate Instance within a Duplicate Group indicates the number of times the same Dose Reading or a suspected same Dose Reading for the same Wearer has been detected. In the example above the blue checked line emphasizes a set of two overlapping Dose Reading instances.
Dose Reading Overlaps may appear in the same Dose Report or across multiple Dose Reports. This is important information when trying to decide how to handle duplicated Dose Readings because it will lead to further identification of the cause of the duplication.
The Dose Readings column shows the number of Dose Readings present in the parent Dose Reports and is useful as a quick check to see if the Overlapping Dose Readings are from the same or different Dose Reports.
This is important information when trying to decide how to handle duplicated Dose Readings because it will lead to further identification of the cause of the duplication.
This highlights the importance of the Dose Report Readings column in identifying where different Dose Reports are involved and the circumstances around the duplication of a Dose Reading. Similar comparison information can be gained from the Date/Time imported columns.
When trying to understand the circumstances around detected Duplicate Dose Readings observe the Duplicate Grouping and Instance details and the numbers of Dose Readings column, to determine whether multiple Dose Reports contributed to any duplication.
Duplicate Indicator Symbol
Historion’s visual communication standard provides for immediate recognition of Dose Readings flagged as Duplicates, as shown in figure 132 below.
.png)
Duplication indicator symbol
Dose Reading Duplicate Information Columns
In addition to the Duplicate Indicator Symbol outlined above, Historion provides a set of Duplicate Information columns for each Dose Reading (in Dose Reports as distinct from the Duplicate lists);
.png)
Duplicate Group and Instance values
None of the column values shown above, described below, can be changed directly in their respective cells within the Dose Report Details page. These values show the results of calculations and actions in managing Duplicate Readings in the Duplicate lists described above.
Sorting these columns within a large Dose Report can be helpful to show all duplicates together. By default, the Duplicate Groups and Instance values will appear ordered by their numeric sequence.
The definitions of these columns are as follows;
Is Duplicate
This column provides an additional visual indication as to whether the Dose Reading has been detected as a Duplicate Reading.
Allowed Duplicate
This column provides a visual indication as to the whether the Dose Reading Duplicate has been classified by the RSO as an allowed Duplicate Reading.
Duplicate Group
This column provides a visual indication as to the numeric group number assigned to a set of Dose Duplicate detected as further described above.
Duplicate Instance
This column provides a visual indication as to the numeric instance or sequence number assigned to a specific Dose Duplicate within a set of Dose Duplicates detected as further described above.
Dose Reading Duplicate CRC Values
Every Dose Reading in Historion is assigned a CRC value. The Dose Reading CRC Value is like an electronic signature for the row and it is calculated based on the maximum set of the Dosimetry Provider’s field values that can be used for each Dose Report Type to ensure Dose Readings are unique Provider rows. Historion derived values (such as Import Date) are not used to calculate CRCs.
Observing the CRC Values for Dose Reading Duplicates will quickly lead to understanding the type of duplicate that has been detected as outline below. Dose Reading Duplicates are either Row Duplicates or Suspected Duplicates. CRCs are convenient as they can be read from one field, used to represent the whole Dose Reading row, instead of a difficult cross checking of multiple column values.
Row Duplicates
.png)
Row Duplicates have matching CRCs
Rows with a matching CRC can only be imported with the Allow duplicate setting and are known as Row Duplicates. The matching CRCs mean that the rows are the same, the entire whole row is duplicated. Specially:
All values of the Dose Reading: (the same Wearer, all date values, numeric measurements etcetera), supplied by the Provider, exactly match all the same value positions of one or more other Dose Readings in the Historion database.
Row Duplicates are a Data Quality problem. Retaining Row Duplicates longer term is not recommended, and they should be investigated with Provider assistance.
Suspected Duplicates
Dose Reading Duplicates with CRCs that do not match are Suspected Duplicates flagged for further investigation. Sometimes Suspected Duplicates are not actually Dose Reading Duplicates and are more likely to be classified as allowed.
.png)
Rows that have different CRCs but the same Wearing Period, the same Wearer and one or more measurement values that match and are not Row Duplicates are Suspected Duplicates. Specifically, the following fields match the corresponding fields in one or more other Dose Readings present:
Wearer, Wearing Period From, Wearing Period to, Dosimetry Report Type/Provider all match
One or more not all, of the measurement field values is the same: Hp(10), Hp(0.07) or Hp(0.3)
In addition, the following conditions apply:
The Dose Reading doesn’t meet the definition of a Duplicate Row, above, so there are one or more, but not all, measurement value differences or different Badge IDs (below) are present.
The Badge ID (the Monitor Number or Dosimeter Serial Number) may be the same or different
The Dose Reading may be a result of deliberate Double Badging of a wearer and it may be a legitimate additional Dose Reading for the same Wearer for the same Wearing Period.
The Dose Reading may be an uncovered duplication error, such as a reversion/replacement Dose Reading included in more than one Dose Report or duplicated within one Dose Report.
Deciding on a course of action (4 scenarios)
Replace
Before replacing a Dose Report, ensure a new version of the Dose Report has been received from the Dosimetry Provider. A Dose Reports can then be replaced by first deleting it from Historion and then importing the new version.
Allow
Before allowing a Duplicate Dose Reading it is strongly recommended to review and where required change the Dose Reading Classification for the Duplicate Dose Reading to be allowed. Duplicate Dose Readings can be set to Allowed (kept) in Historion if the RSO has determined that the Dose Reading duplication is legitimate, given the circumstances of monitoring and results data involved.
An Allowed Duplicate Dose Reading is moved to the Allowed Duplicate Dose Readings list and is then hidden from view by default. This is a basic Historion Workflow concept intended to assist in managing large volumes of Dose Readings been flagged for further assessment and management by the RSO.
Duplicate Readings can be Allowed in two ways; by ticking the [Is Allowed Duplicate] cell for a single selected Duplicate Dose Reading in the Duplicate Dose Readings list as shown below or as a group by selecting multiple Dose Duplicate Dose Readings in the list and clicking the Allow button at the lower right corner of the listing page as shown next page.
Box to tick to allow a selected Duplicate Dose Reading
.png)
After selecting [Yes] when prompted for confirmation the selected Duplicate Dose Reading is filtered out of the list. Allowed Duplicate Readings can still be viewed, or the allowed flag removed, by using the Show All or the Allowed feature options displayed at the lower left corner of the Duplicate Dose Readings list page.
.png)
Allowed Duplicate Reading Visibility Options
Multiple Duplicate Dose Readings can be allowed (with caution!) by selecting multiple rows in the Duplicate Dose Readings list and using the bulk [Allow] feature (lower right corner of the listing page).
A prompt is shown for confirmation indicating the number of Duplicate Dose Readings that have been selected to be allowed as shown below;
Six Duplicate Dose Readings have been selected
.png)
Bulk Allow Confirmation prompt
Allowing Duplicate Dose Readings will not change anything related to the Dose Reading itself and has no implications for the Dose Reading other than hiding the duplication record from the default view in the Duplicate Dose Readings list as a basic Historion Work flow concept.
In order to change the effect of the Duplicate Dose Reading in reporting areas of Historion the Dose Reading will need its classification details changed as described under Dose Reading Classification.
Deleting Duplicate Dose Readings
Allowing Duplicate Dose Readings will not change anything related to the Dose Reading itself and has no implications for the Dose Reading other than hiding the duplication record from the default view in the Duplicate Dose Readings list as a basic Historion Work flow concept.
In order to change the effect of the Duplicate Dose Reading in reporting areas of Historion the Dose Reading will need its classification details changed as described under Dose Reading Classification.
Deleting Duplicate Dose Readings
There are very limited circumstances under which any Dose Readings should be deleted. It is always preferable to seek a replacement Dose Report from the Provider, remove the imported version and import the corrected Dose Report as an official, legal Provider originated record.
The following circumstances would seem to justify deleting Duplicate Dose Readings, however in each case this is a local decision and policy of the Historion users and their management.
These are not recommendations but scenarios in which removing duplicates may be understandable;
A data migration exercise, from a legacy system involving large numbers of scripted Dose Reading transactions (added to the Historion database by an IT Department or a contractor) has resulted in data duplication that is identified and can be conveniently managed in the Duplicates Data Quality page.
In this instance there may be no Original Dose Reports available and data cleansing and duplicate removal is desirable and is done as a once-off step, as part of the data migration.
A Duplicate Dose Reading which is present and determined to be a Data Quality or other record error and the Provider of the result may be no longer available or unwilling to change and reissue the Dose Report as a new version, to address the Duplicate Dose Reading result.
The Historion User’s organisation has decided to, and takes complete responsibility for, (and has documented the fact of) removing the Duplicate Dose Reading.
A comment recording the removal of the Duplicate Dose Reading is made in the Notes field of the applicable Dose Report, explaining the difference between the Stored Original and the Dose Readings available for reporting that are still contained in the transacted Dose Report’s Readings appearing as rows in Historion. A File Note is registered against the applicable Wearer’s details explaining the circumstances and any record action(s) taken.
A Legacy Body or Legacy Extremity Dose Report Type has been imported from a manually authored (typed) Dose Report and the User is the author of and is directly responsible for generating the content of the Dose Report produced.
An external Provider is not involved or represented. Duplicated Dose Readings are present due to human error and removal is permissible and more convenient than generating a new Dose Report version.
In this case the new Dose Report version, to replace the one with duplicates is still preferable, so that stored Original Dose Report content matches Dose Readings retained for reporting.
Be certain to only delete one Dose Reading Duplicate at a time for any Duplicate Group. The Groups include Dose Readings that are to be kept; the whole set it shown. Any accidental removal will require deleting and reimporting the parent Dose Report and starting again with Duplicate detection and removal.
To delete a Duplicate Dose Reading only select the row representing the reading to be removed then click the Delete icon at the lower left corner or the Duplicates Listing Page or press <Delete>;
.png)
Delete Duplicate Reading(s)
A confirmation prompt requires a Yes response to continue with the Duplicate Dose Reading delete and a dialogue is displayed after the delete, showing the number of Duplicate Dose Readings remaining;
.png)
.png)
After deleting Duplicate Dose Reading(s) Historion prompts for a rescan against the Dose Readings that remain in the database;
.png)
Re-scanning after deleting a Duplicate Dose Reading is recommended. Rescanning recreates the Duplicate Groups and removes the IsDuplicate flags from Dose Readings that are no longer in a state of duplication as a result of the deleting one or more Dose Readings in a duplicate set.
.png)
At any stage after rescanning for Duplicate Dose Readings Historion will provide an indication on completion as to the number of Duplicate Dose Readings retained and how many have been flagged as Allowed Duplicate Dose Readings.
Ignore
Ignoring Duplicate Dose Readings is not recommended. Ignoring Duplicate Dose Readings means not managing or reclassifying duplications at all and this exposes the organisation to data related risk and problems as described under Risks Associated with Poor Data Quality.
Even zero value, less than detectable or less than minimum reportable dose (MRD) Duplicate Dose Readings should be investigated as they may represent assignment of the Dose Reading to the wrong Wearer, the wrong wearing period or other data recording or reporting failure.
|