We use cookies to ensure that we give you the best experience on our website. You can change your cookie settings at any time. Otherwise, we'll assume you're OK to continue.

Durham University

Research & business

View Profile

Publication details

Kitchenham, Barbara, Brereton, Pearl, Turner, Mark, Niazi, Mahmood, Linkman, Stephen, Pretorius, Rialette & Budgen, David (2009), The Impact of Limited Search Procedures for Systematic Literature Reviews: A Participant-Observer Case Study, Third International Symposium on Empirical Software Engineering and Measurement. Florida, USA, IEEE Computer Society Press, Florida, 336-345.

Author(s) from Durham


This study aims to compare the use of targeted manual searches with broad automated searches, and to assess the importance of grey literature and breadth of search on the outcomes of SLRs. We used a participant-observer multi-case embedded case study. Our two cases were a tertiary study of systematic literature reviews published between January 2004 and June 2007 based on a manual search of selected journals and conferences and a replication of that study based on a broad automated search. Broad searches find more papers than restricted searches, but the papers may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers; if publication bias is not an issue; or if they are assessing research trends in research methodologies.