What is the formula of clinical trial success?

Performance is hard to sustain and risks are emerging unexpectedly. The goal of this article is to define a robust methodology for managing performance and risk in clinical trials.

by Dimitri Stamatiadis, PhD, MBA

Why measure performance?

You may be telling yourself as you start reading this article: I am so busy trying to carry out my daily work, how much work can I devote to measuring my performance? And to what avail? Is it worth the extra effort? Will it bring any additional benefit or is it just there to please management and make them feel they are in command?

Well let’s think about why we measure performance. Why we measure anything for that matter? Lord Kelvin in the late 18th century said the following:

“When you can measure what you are speaking about, and express it in numbers, you know something about it, otherwise your knowledge is of a meager and unsatisfactory kind…”

No doubt he had temperature in mind.

In business we measure performance because there is competition (external and internal), because there are mandatory rules and regulations to be followed, because resources are not infinite and because we have plans for the future. In other words, because we need to make decisions.

Clinical studies are necessary to obtain market authorization for new drugs and new indications and they cost an awful lot. The high price of new medicines is largely driven by R&D costs and clinical development eats up more than half of that. Clinical trials are lengthy and risky. Therefore, good performance in clinical trials is key to the success and often to the very survival of a pharmaceutical company.

What are the keys to success then? How do we define and measure performance and how does risk play a role? How do we process the wealth of information that we have today, turn it into performance indicators and use it to support decision-making? Finally, how can KPIs help us align the organization?

Making decisions

We make decisions every day at work and elsewhere but how do we do that? Most of the time intuitively and because we know our job, our intuition combines information, training and experience to go straight to the right conclusions

[1]. After all, an athlete does not need measuring devices and big data computing to perform a double flip and land on his toes; just training and skills. Why would a manager operate differently?

Sometimes though, our intuition can be tricked or overwhelmed by the reality of our work; the athlete can trip and fall. Data, if used wisely, can support intuition, strengthen our confidence; refine our judgment. Airplane pilots may be screened for their skills, sound judgment and physical health; they still rely on checklists and an array of sophisticated instruments to fly their planes because they have the responsibility of success.

Well, what is success?

Well, let’s start at the beginning. What is success? How can we know that we are successful and when can we say that he have succeeded in something? One of the scholar definitions is the following:

A good strategy implies setting goals, planning and executing the plan. According to the Balanced Scorecard methodology[2], strategy must cover at least four dimensions: People, Processes, Customers and Finance. Goals, plans, targets, thresholds should be set along these for dimensions to ensure a holistic strategic view. Performance is defined as obtaining the right things, in the right way and at the optimal cost. Finally risk is a potential damage resulting from the combination of a threat (external) with a vulnerability (internal).

So in short, what you want is to set your strategic goals intelligently, plan and execute your strategy. Along the way you want to make sure that you obtain the results that you planned for while respecting rules and regulations and sticking to your budget. You also need to regularly assess the external threats and minimize your vulnerability accordingly.

Does your company have a clear clinical strategy? Has it been communicated to you in a way where you understand it and you know your part in it? In industry surveys the vast majority of companies have a clearly communicated strategy (or so does management think) but only 14% of the employees declare knowing it. This is because there is a lack of adequate tracking tools. Business intelligence has been widely developed in recent years but it is limited to gathering information and displaying it in graphs and figures. BI tells you nothing about your strategy and how it is playing out.

Turning data into KPIs

In order to support decision-making, we need to turn information into key performance indicators (KPIs). What this means is that our indicators must describe the three dimensions of performance: results, know-how and means. They must be “3D”.

Results are of course whatever you are trying to achieve. Know-how can be described as qualification, experience and processes. And means can be divided in time, running cost (what we frequently refer as op-ex) and fixed assets cost (what we usually call investment or cap-ex).

Assessing and using risk

Risk is also a major component for our success. We should not fear risk and we should not shy away from studying and understanding it. Along the way, there can be damage to our performance because of external threats (which we cannot control) but only if we are vulnerable to those threats. Of course we cannot take all the possible measures up-front as this would be too costly. Therefore it is important to monitor risk at the appropriate frequency that allows for timely action. One good rule of thumb is to monitor at the immediately lower level from the perceived risk. For example if a risk is quarterly, monitor monthly. If it is monthly, monitor weekly. Finally use risk to determine the relative weight of a KPI.

There are many definitions of risk and many methods of evaluation.
The ISO definition of risk is: “effect of uncertainty on objectives”

The risk-based approach of GCPA [3] is based on the following criteria[4]:

Critical process and data identification
Risk identification
Risk evaluation
Risk control (mitigation actions)
Risk communication
Periodical risk review
Risk reporting

The analytic part of risk evaluation occurs with the help of the three main risk properties:

Likelihood that a risk or failure mode materializes
Impact on subject’s safety, rights and data integrity and reliability
Detectability – extent to which such threats or errors are detectable

In any case, risk can be used to define what to measure and how often, evaluate the relative weight of each measure, define the appropriate thresholds and corrective actions.

In clinical trial we now have excellent tools for evaluating and monitoring different specific types of risk.

Using the data

We have seen how information can be used to support intuition in decision-making. Key performance indicators are useful for the correct management of conflicting urgencies as well. We constantly deal with urgent situations and we do not always have the capacity to handle all of them at the same time. While the urgency is manifest, in the absence of data it is not always easy to prioritize our actions.

But how should we approach the data that we are able to collect in order to make the best use of them? Business intelligence (BI) tools such as the clinical trial tracking systems (CTTS) and the Risk Monitoring tools are able to collect a very large amount of data. The trouble with these data is that we cannot devote the same amount of work to cleaning them as we do for clinical data. Therefore they are usually incomplete and are met with suspicion.

Small data / Big data

There are two possible paths for making good use of the available data: I will call the first “small data”. This is the traditional approach. Select a limited set of relevant indicators and make sure that all the data is complete and accurate. Ensure full ownership of the information by the relevant department. Data can be easily reviewed in a regular basis and decision made accordingly. Software can be useful to manage this approach but is not necessary. The dashboards and graphs produced by the BI tool are usually sufficient.

I will call the second method “big data” as it is based on a methodology of big data analytics[5]. This will need a dedicated software to be managed and is based on large volumes of data. Information from as many sources as possible are collected and assessed for quality (completeness, consistency with historical values etc.) BI tools are a good source but financial (e.g. SAP) and other databases are also very useful. As long as quality is at an acceptable level (e.g. 70-80%) it can be trusted for the purpose of this method. The exact value of the indicator is irrelevant. Data are assessed using thresholds (to determine what is considered low/medium/high/very high) and tendencies (up or down, crossing a threshold or not). The advantage of this methodology is that it can process data in the same way (by thresholds) at any magnitude (from patient level all the way up to a clinical program) as categories apply similarly at all levels.

Alignment

Finally, aligning the organization has been a long-standing challenge of any strategy implementation[6]. It is paramount to make sure that all parts of a clinical research unit at all levels are working towards the strategic goals in one or many of the balanced scorecard categories, i.e. people and learning, process excellence, customer satisfaction and financials. In all cases, KPIs must be set in a consistent way throughout the organization. Either manually in the case of a “small data” approach or programmatically in the case of a “big data” solution.

Conclusion

In conclusion, turning data into key performance indicators allows to support intuition in decision making. A robust methodology for defining success, strategy, performance and risk and a multi-dimensional approach are necessary to ensure the efficiency of any performance management system and finally, whether you use a “small data” or a “big data” approach, aligning the organization is paramount to the success of your clinical strategy.

Bibliography

[1] Jack Welch, John A. Byrne. Jack: Straight from the gut. August 28th 2005, Warner Books

[2] D. Stamatiadis. Measuring Success: Balanced Scorecard, International Clinical Trials, Samedan publishers, 2011

[3] How does the New GCP Addendum Influence Clinical Monitoring? October 6, 2015 By Artem Andrianov

[4] ICH, “Integrated addendum to ICH E6(R1): Guideline for Good Clinical Practice E6(R2),” 11-Jun-2015. [Online]. Available: http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R2__Addendum_Step2.pdf. [Accessed: 16-Jun-2015].

[6] Aligning Organizations Through Measurement, Basili, V., Trendowicz, A., Kowalczyk, M., Heidrich, J., Seaman, C., Münch, J., Rombach, D, Springer International Publishing, 2014.