Everybody, who came into touch with Risk-based Monitoring (RbM), met at least once such terms as Key Risk Indicators (KRIs), Key Performance Indicators (KPIs) and even Key Quality Indicators(KQI). And I am sure that you feel the difference among them, although the explanation is not obvious. Let me try to clarify these terms.
For the sake of simplicity, let us take an example: imagine you sit into a car and observe your well-known dashboard in front of each driver. Here you can see some KRIs and KPIs.
For instance, a speedometer is a simple performance indicator, it tells you how fast you are driving. It does not tell you if you should drive 50 km/h or 80 km/h in this area, this information a driver should know.
Risk indicator is a bit more complicated construct, in our example it would be a fuel gauge. It consists of a performance indicator – amount of liters in a tank – and additional thresholds, which warns a driver (in two stages) to tank in a near future. If she/he does not take any actions, a complication (read: risk) will happen, e.g. you will stay without fuel in the middle of a road.
Thus, this would mean a performance indicator contains only measurement of one metric, and a KRI incorporates additional information about thresholds and certain risk, which may happen if no further action is undertaken.
Typical KPIs within RbM are:
- Number of enrolled patients
- Number of adverse events (AEs) per site
- Number of patient weeks (i.e. how many weeks patients are in a clinical trial)
Now if would add additional thresholds, or better to say, information from experience, we can covert the KPIs into KRIs, here you are:
- Number of enrolled patients is less than it was planned
- Number of AEs is too high per site comparing to other sites in the same clinical trial
- Average number of patient weeks per month is decreasing, e.g. due to high drop-out rates
Now you can monitor the KRIs and see their dynamics as shown on the graph above.
Until now it was fairy simple, now, how about the KQIs? What can these “monsters” indicate? As it is clear from the name “Key Quality Indicators” measures Quality. And quality is a degree of excellence. And it depends on how it was planed.
Despite the significant amount of research, operational cost, and complexity in conducting clinical trials, at the end of a trial is represented by its Dataset. The Datasets are expected to be in full compliance with protocol specifications while also carrying high data quality standards. Therefore, we will speak here mostly about KQIs for clinical data.
For the KQI calculation researchers use certain data categories:
- Data which should not change (longitudinal)
- Data which should not be cleaned
- Data which should not be missing (e.g. recorded by devices)
- Data which is derived from other data
- E.g. best values in Spirometry
- Datasets which should correlate with each other (so called “data integrity”)
- E.g. questions describing the same physical condition
- E.g. values based on measurements (e.g. in Spirometry or ECG)
- Data which has defined value ranges
- E.g. questionnaires with max possible answers
- E.g. Spiro and ECG parameters
- E.g. birthdays not older than 120 years, not in the future
- E.g. height and weight limitations
- E.g. patient visits on unusual dates or times (e.g. on Sundays in a country, where Sunday belongs to weekend)
- Data with different “encoding”/formatting, sanity checks
- E.g. Wrong entries from person (initials with three characters)
- E.g. Decimal formatting
- Data which has time or duration constraints
- E.g. several visits for one patient on the same day
- E.g. visits of different patients in unusual short intervals
- Artifacts in data
- E.g. wrong Spirometry measurements (either analysed by ourselves or based on overreads)
- E.g. impedance of ECG electrodes
- E.g. artifacts in ECG measurements
- E.g. missing pages or whole questionnaires
- E.g. calibration
- Data which should not change
- Inclusion criteria
- Raw data
- Questionnaire data
- Data which should change based on other changes
These quality parameters can be measured by KQIs separately or united into joined indicators, where several checks can be combined together.
Above on the picture you can see a number of joined KQIs. E.g. Vital signs score applies a number of sanity checks on vital signs such as body temperature, pulse rate, blood pressure.
Another well-known data quality indicator, which is applicable for the whole trial, is “confidence intervals” of the measured data. A threshold used in this context – “statistical significance”, the p-value, which in clinical trials must be less than 5%. Different significance levels (e.g., 1%) may be applied depending on the field of study
Another typical KQI in RbM would be “normality test”, which indicates whether data is normally distributed or skewed. The skewed data is often influenced by a systematic error, which shifts the bell-shape distribution to one side or another (see below).
The advantage of KPIs, and KQIs is their comparability among trials and retrospectively. The KRIs are more useful for their predictability of the dangerous situations as taking the right actions in the right time can prevent unwanted consequences.
CAPA on practice
Summing up, all discussed instruments like KPIs, KRIs, KQIs in RbM aim at identifying, assessing, controlling, communicating and reviewing risks associated with your clinical trial during its lifecycle, i.e. make your trial safer for the patients, give your trial management team full control over a trial and inform about dangerous situation approaching. What you will do in these situations, is up to you to decide.
 Sproull, Natalie L. (2002). “Hypothesis testing”. Handbook of Research Methods: A Guide for Practitioners and Students in the Social Science (2nd ed.). Lanham, MD: Scarecrow Press, Inc. pp. 49–64. ISBN 0-810-84486-9.
Interested in learning more about Cyntegrity and our services?