The Goldilocks Zone | A useful perspective on performing a Systematic Literature Search

Manufacturers and regulators that work towards device compliance under the EU IVDR have encountered many new and challenging obstacles during the transition from the previous legislation. Among the key obstacles is with regards to clinical evaluations and performance evaluations which highlight the importance of providing a greater body of supporting clinical and performance evidence. It is expected that this will include a scientific literature review to acquire relevant data that relates to the devices claim(s) and intended purpose.

Due to this new regulatory requirement, it is not surprising how much uncertainty there is relating to the level of detail needed and what steps should be taken to perform a systematic literature search. This blog will strive to present a useful perspective on performing a systematic scientific literature review.  Many of the issues encountered are best avoided through the appropriate level of planning and a suitably flexible approach.

Literature reviews are applicable to newly marketed devices and should be continued throughout its lifecycle, this raises numerous questions. The first being:

• What research questions need to be answered.
• How much detail is expected during the planning of a literature search?
• How many results are considered enough?
• What key terms and filters are going to be most appropriate?
• What can be done if a search is not working?

What constitutes a systematic scientific literature review and how does it compare to more traditional approaches? Under the MDR and IVDR, the “systematic” process is best described in annex XIV and annex XIII, respectively that specifies a clear record needs to be made that starts with a literature search protocol (LSP) and ends with a literature search report (LSR) that details the findings and the means of execution. It is expected that the search methodology will evolve once exploratory searches are performed. The search protocol has to adapt to the way that papers have been indexed.

IVDR Annex VII 4.5.4 shows that the methodology to generate the literature search are a critical component of the review. It is not enough to simply provide relevant clinical data without justification whilst, the search and methodology has to be supplied to the notified body with the technical file. The notified body reviews these documents to ensure there is no bias and cherry-picking of articles.

Accountability plays a major part with identifying both positive and negative studies through the use of a logical search string assembled from relevant search terms. The design of an effective search string often relies on organising the search terms based around the research questions being asked. Methods such as PICO analysis can help direct the operator to think about key elements needed to reach, a focused group of studies or papers.

Consider an example for a generic coagulation analyser. What terms should be used to perform a search for scientific validity wherein the analyte (such as time until clot formation) is related to the clinical condition/physiological state (such as patients dosed with coumarin derivatives). The table below shows how this can be spilt into categories and may provide a better direction of the terms that could be selected to answer particular questions. Many of the terms listed are relevant and potentially useful, however it is only after a few preliminary searches that it will become apparent which ones will yield the most promising results. It is important to note that a given search string can be complete and logical but not yield good quality literature. Essentially trial and error to experiment with varying search terms is the name of the game, this improves with practice and experience.

Generally speaking, the steps taken for a literature review can appear to have rather subjective qualities to them as it is mostly reliant on an operator’s perspective to recognise a good or poor output and as a regulatory professional this can feel unscientific; however, the indexing of papers can be idiosyncratic, so it is a necessary part of the process. You could argue that there is an art to recognising the balance of quality and quantity of results which, by extension; would mean that being able to recognise this balance is an essential skill for a literature searcher. Considering that not everyone will have the same inclination to start a search of their own, it may be useful to run with the following thought experiment as a way of keeping track on where the search needs to go to reach adequate results.

Enter Goldilocks. A commonly known character from a children’s fairy tale that can be used as a rather helpful analogy when developing a literature search. Being able to visualise a given literature search can have its merits in finding what can be called, “The Goldilocks Zone”.

As a reminder, one of the most common lines remembered from the story brings us to Goldilocks trying the three bowls of porridge one after the other to which she exclaims: “the first one was too hot, the second too cold but the third was just right”. Sticking to the context of this statement and without getting too distracted with the other elements presented in the story, a fair comparison can be made between Goldilocks sampling porridge and the operator of a literature search sampling (or testing out) search strings in a database of their choosing, only to come upon unsatisfactory results after running a few trials. This is all too often the case and can be quite a time-consuming process before finding the search that is “just right”.

To further defend the idea of a purposeful comparison, an argument could be made that a search being too “hot” has not applied enough focus to the topic of interest and may need additional tools or search terms to better define the search query whilst, a search that is too “cold” has filtered out too many useful results to the point that the topic of interest has been lost in translation. Much like Goldilocks, the aim of a literature search is to find an optimum output. In our case this will be a search entry that is not too focused so as to be biased and yet unfocused at the same time otherwise, to be sufficiently broad. The repercussions may lead towards poor quality and/or quantity of results, both outcomes costing time that is not so affordable with meeting reporting timelines.

It should be noted that a catch-all-phrase called “The Goldilocks Principle” is already used by many other fields of study to describe a similar state of balancing values or traits. A rather useful comparison can be directly borrowed from the specific term for “Goldilocks Zone” which comes from the field of astronomy. Specifically, this term is used to describe the optimum distance a planet must be situated from its host star in order to maintain liquid water on its surface without freezing or boiling away (hence being a useful factor for finding potentially habitable worlds that may foster life). This can be depicted as shown below in which the coloured bands (red = too hot, blue = too cold and green = just right) can define the zones and how they change depending on the type of star otherwise, representing the core literature subject at its centre.

Applying this figure of a star-system can be a useful qualitative measure of an ongoing literature search in the hopes of finding better results. The idea is that it reminds the operator to consider whether the results are “too hot” or “too cold” before retrying a search query using more subtle or significant changes. Think of the centre of the star system as the core literature to support the search question(s).

For example, consider a device that is concerned with the screening of markers for breast cancer. The broad topic in question is going to be very well researched therefore, it is realistic that an extensive amount of literature needs to be filtered to produce results of relative quality and quantity. Without the Goldilocks approach, the operator will end up having to “boil the ocean” during appraisal of results which, is of course a very time-consuming process that may not yield enough data or completely cover all research questions. Continuing this example, HER2 is one of the most established markers and a thoroughly well researched topic. As a result, the search has to be focused to the application in question to combine how HER2 can be used, for example; to predict the current state of breast cancer or for companion diagnostic applications. A strategy will need to be devised that aims to filter out those studies that are not considering HER2 and are not associated with the application in question. It is at this point that the star-system model can be quite helpful. This particular example could be seen as a large (hotter than average) star that forces the Goldilocks Zone further out from the core literature. It is further influenced based on the research question such as the application that focuses the output.

Compared to the first and second examples (generic coagulation analyser and breast cancer assessment panel), it would stand to reason that a more novel device would have less literature available and so needing less refinement in the search to create specificity which in turn, narrows down papers that have potential for appraisal. In other words, a dwarf (colder than average) star will have its Goldilocks Zone closer to the core of the subject. In these cases, the manufacturer may need to rely primarily on their own feasibility data to support the scientific validity or the state of the art. The literature search can then be used to verify that this is consistent with the literature and is considered to be state of the art.

It is not uncommon for a subject of interest to be given minimal coverage amongst the available scientific literature that can then lead to difficulties acquiring adequate results that no amount of trial and error can solve. This is particularly possible as an early preview of search results will not reveal as much compared to an in-depth appraisal. It is for this reason that returning to perform additional searches after a first run can be justified as part of the systematic process. Of course, learning to recognise when literature is unsuitable for a given search objective depends entirely on the knowledge/experience of the operator as well as the depth in which these articles are assessed.

When faced with more difficult searches, an operator can look for individual articles that are the “gold standard” in relation to the focus of the literature review. Consider reading through the paper to find useful terms and find out how it is referenced to better identify terms or combinations that have not been considered as viable inputs for search strings. This alone can be a great way of getting a search started. Also consider tips and tricks such as, making a list of equivalent terms that help to widen the net on available literature to ensure all relevant papers are captured.  This can be important where the names of analytes have changed with time or different spellings, or words used in different countries. Switching to alternative databases may also be necessary, it is always advantageous to present more than one source and each database will be indexed in a different way. Guidelines can be particularly tricky to identify in a search and may benefit from these approaches.

A solid literature search needs to be followed by suitable analysis to a predefined protocol defining inclusion and exclusion criteria. It is important to include why papers that were included or excluded. The amount of effort that has to be allied at this stage can be significantly reduced by carefully defining the literature search criteria.

How to find the Goldilocks Zone?
The best advice is to keep trying new strategies and new ideas until its “just right”.

As Albert Einstein once said:

How do you find the Goldilocks Zone?
Join our experts for virtual training sessions on literature search for medical devices (29th March, 2022) or ivds (5th April, 2022)

Virtual training sessions: Literature search for medical devices and in vitro diagnostics