From 987acf19f5f7f53324841a5f40471676b35ffcd5 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Tue, 14 Nov 2023 23:28:31 +0100 Subject: [PATCH] Remove duplication in about and methods pages --- qmd/about.qmd | 42 +++++-------------- qmd/assets/phes-odm.scss | 4 ++ qmd/delphi.qmd | 30 ++++++++----- ... process - website.svg => dev_process.svg} | 0 qmd/index.qmd | 2 +- qmd/methods.qmd | 30 ++++++------- qmd/references.bib | 28 +++++++++++++ 7 files changed, 78 insertions(+), 58 deletions(-) rename qmd/images/{PHES-EF development process - website.svg => dev_process.svg} (100%) diff --git a/qmd/about.qmd b/qmd/about.qmd index e43664d..b01a5c7 100644 --- a/qmd/about.qmd +++ b/qmd/about.qmd @@ -1,37 +1,15 @@ --- title: "About" -format: html + bibliography: references.bib --- +This project will use a multidisciplinary and multinational consensus approach to develop an evaluation framework for public health environmental surveillance (Public Health Environmental Surveillance Evaluation Framework, PHES-EF) using wastewater surveillance as an example. -## What is PHES-EF? -This project will use a multidisciplinary and multinational consensus approach to develop an evaluation framework for public health environmental surveillance (Public Health Environmental Surveillance Evaluation Framework, *PHES-EF*) using wastewater surveillance as an example. - -An evaluation framework provides structured guidance to facilitate a systematic approach to the evaluation of the program (14). While there is variation in the terminology used to describe frameworks (15), our approach will be as follows: present a clear list of items that should appear in an evaluation; provide an organisation or structure for the list of items; define how the list and organisation were developed; and, potentially, include details about how to measure the evaluation items. - -## Why is PHES-EF Needed? -Public health environmental surveillance has evolved, especially during the coronavirus disease (COVID-19) pandemic, with wastewater-based surveillance being a prominent example. As surveillance methods diversify and expand, there is a need to evaluate these systems to improve surveillance program accountability, transparency, and trust. - -It may even be argued that over the past few years only wastewater-based monitoring has increased rather than surveillance, as the data were not used to inform public health decision-making. This issue exemplifies the need for *PHES-EF*, as an evaluation framework for public health environmental surveillance systems is required to increase their reliability and credibility, to inform public health decision-making. In turn, this will also promote their value and sustainability as a public health tool. - -There is also a need for such an evaluation framework as the recent and abrupt adoption of wastewater-based monitoring has resulted in poor standardization of analytic techniques, with ideal methods for reporting testing data still remaining unclear (1). Ultimately, *PHES-EF* aims to address this issue as well, as the framework is intended to be adopted by researchers to determine surveillance system performance and create analogous results for comparison within the field. - - -## Our Objectives -Specific objectives for the framework development process are as follows: - -1. Engage a multidisciplinary executive group of international experts to inform the development of the evaluation framework - -2. Review the literature to identify current guidance on surveillance evaluation items - -3. Prioritize items for an evaluation framework for environmental surveillance using an international electronic-Delphi (e-Delphi) process - -4. Conduct a consensus meeting that engages experts across disciplines to create an evaluation framework for public health environmental surveillance based on feedback from the international e-Delphi process +An evaluation framework provides structured guidance to facilitate a systematic approach to the evaluation of the program [@fynn2020]. While there is variation in the terminology used to describe frameworks [@calba2015], our approach will be as follows: present a clear list of items that should appear in an evaluation; provide an organisation or structure for the list of items; define how the list and organisation were developed; and, potentially, include details about how to measure the evaluation items. -5. Develop a comprehensive dissemination plan for the evaluation framework -## Our Approach -### Open Science +## Approach +### Open science The project will follow an open-science approach and make available the data for all project stages, including the search strategy, search findings, initial curation of evaluation measures, summary of executive group discussions, e-Delphi processes, and agreement procedures. @@ -39,13 +17,13 @@ The project will follow an open-science approach and make available the data for Anonymous aggregate data will be available on OSF, where the study protocol is registered. Individual responses to e-Delphi rounds will be anonymous at the source level by the platform used. -### Equity, Diversity, and Inclusion (EDI) +### Equity, diversity, and inclusion (EDI) -"Equity is defined as the removal of systemic barriers and biases enabling all individuals to have equal opportunity to access and benefit from the program"[@schmalz2021]. To achieve this, members of the research team are committed to developing a strong understanding of the systemic barriers faced by individuals from underrepresented groups. "Diversity is defined as differences in race, colour, place of origin, religion, immigrant and newcomer status, ethnic origin, ability, sex, sexual orientation, gender identity, gender expression and age". Panellists will be asked to provide characteristic information (e.g., country of residence, gender, primary discipline) to assess and help ensure (through more targeted recruitment) a large diversity in perspectives. "Inclusion is defined as the practice of ensuring that all individuals are valued and respected for their contributions and are equally supported". To help ensure that all research team members are integrated and supported, we will have a code of conduct statement in the Executive Group Terms of Reference. +"Equity is defined as the removal of systemic barriers and biases enabling all individuals to have equal opportunity to access and benefit from the program" [@schmalz2021]. To achieve this, members of the research team are committed to developing a strong understanding of the systemic barriers faced by individuals from underrepresented groups. "Diversity is defined as differences in race, colour, place of origin, religion, immigrant and newcomer status, ethnic origin, ability, sex, sexual orientation, gender identity, gender expression and age". Panellists will be asked to provide characteristic information (e.g., country of residence, gender, primary discipline) to assess and help ensure (through more targeted recruitment) a large diversity in perspectives. "Inclusion is defined as the practice of ensuring that all individuals are valued and respected for their contributions and are equally supported". To help ensure that all research team members are integrated and supported, we will have a code of conduct statement in the Executive Group Terms of Reference. All study working group members will complete EDI-related Canadian Institutes for Health Research (CIHR) Training, and the First Nations principles of Ownership, Control, Access and Possession (OCAP©) training for the research [@ocap2023]. The executive group will receive an overview of EDI and OCAP and will be encouraged and supported to complete such training. Furthermore, staff trained in EDI and OCAP principles will review the framework from an EDI perspective to ensure appropriate language. -### Knowledge User and Public Involvement +### Knowledge user and public involvement To ensure accurate and transparent reporting of knowledge users and public involvement throughout the study, we will refer to the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) checklist [@staniszewska2017]. We will document the methods used to engage knowledge users, report the impacts and outcomes of their engagement, and report on lessons learned from the experience. @@ -54,7 +32,7 @@ To ensure accurate and transparent reporting of knowledge users and public invol This project is funded by the Canadian Institutes for Health Research-funded network, CoVaRR-Net (Coronavirus Variants Rapid Response Network. FRN: 175622), and Health Canada (through the Safe Restart Agreement Contribution Program. Arrangement #: 2223-HQ-000098). The Canadian Institutes for Health Research and Health Canada have not been involved in the design or conduct or the study and the views expressed herein do not necessarily represent the views of either funding organization. -### Ethics Approval and Consent to Participate +### Ethics approval and consent to participate Ethical approval will be obtained from the Ottawa Health Science Network Research Ethics Board (OHSN-REB) before commencing the study Implied consent will be obtained from all panellists prior to participating in the e-Delphi survey. Information about the survey and consent practices will be provided on the first page of the survey. Panellists will be informed that by providing their demographic information and proceeding to the next page, they are providing their implied consent to participate in the study. Only anonymous panellist responses will be used for analyses. @@ -62,7 +40,7 @@ OHSN-REB Number: 20230428-01H If you have questions about ethical issues related to this study, you can talk to someone who is not involved in the study at the Ottawa Health Science Network Research Ethics Board, Chairperson at 1-613-798-5555 extension 16719. -### Competing Interests +### Competing interests All participants (i.e., study working group, executive group, and e-Delphi panellists) will be asked to declare any financial or personal relationships that may influence either the conduct or presentation of this research. diff --git a/qmd/assets/phes-odm.scss b/qmd/assets/phes-odm.scss index 3624dbc..fcc3898 100644 --- a/qmd/assets/phes-odm.scss +++ b/qmd/assets/phes-odm.scss @@ -76,6 +76,10 @@ a { font-size: 1.25em; } +.caption { + font-size: .8em; +} + mark { background-color: $background_color_1; padding: 0.2rem; diff --git a/qmd/delphi.qmd b/qmd/delphi.qmd index b680eb7..e0f08d3 100644 --- a/qmd/delphi.qmd +++ b/qmd/delphi.qmd @@ -1,16 +1,18 @@ --- title: "e-Delphi Study" -format: html bibliography: references.bib --- + ## Recruitment of Panellists -We will recruit a multinational, multidisciplinary panel of experts with knowledge or interest in wastewater surveillance to complete the e-Delphi survey. Experts will be defined as adult individuals with a high degree of knowledge in their respective disciplines. Panellists will be selected to capture the multiple perspectives of those that influence the design, implementation, evaluation, use, and reporting of wastewater surveillance activities including the following disciplines: mathematical sciences, social sciences, environmental and physical sciences, and public health and medicine. Targeted recruitment will also be conducted to identify knowledge users representing public interest. We will use purposive sampling to identify potentially eligible panellists by examining relevant author lists, professional networks, and internet-based searching, among other potential methods. Snowball sampling may also be employed by executive group members and potential panellists to maximize recruitment. -We will conduct an all-rounds e-Delphi invitation approach, meaning that panellists will be invited to every round independent of whether they participated in the previous round (1). We will aim to recruit at least 50 panellists, preferably, with at least 8-10 people per discipline group. The study working group will monitor the distribution of registered panellists based on their demographic information and will try to have appropriate distribution across disciplines and other demographic markers. +We will recruit a multinational, multidisciplinary panel of experts with knowledge or interest in wastewater surveillance to complete the e-Delphi survey. Experts will be defined as adult individuals with a high degree of knowledge in their respective disciplines. Panellists will be selected to capture the multiple perspectives of those that influence the design, implementation, evaluation, use, and reporting of wastewater surveillance activities including the following disciplines: mathematical sciences, social sciences, environmental and physical sciences, and public health and medicine. Targeted recruitment will also be conducted to identify knowledge users representing public interest. We will use purposive sampling to identify potentially eligible panellists by examining relevant author lists, professional networks, and internet-based searching, among other potential methods. Snowball sampling may also be employed by executive group members and potential panellists to maximize recruitment. + +We will conduct an all-rounds e-Delphi invitation approach, meaning that panellists will be invited to every round independent of whether they participated in the previous round (1). We will aim to recruit at least 50 panellists, preferably, with at least 8-10 people per discipline group. The study working group will monitor the distribution of registered panellists based on their demographic information and will try to have appropriate distribution across disciplines and other demographic markers. -Registration to complete the e-Delphi will open approximately four weeks before the first round and e-Delphi round survey. Each e-Delphi survey round will be active for roughly 4-6 weeks and will take approximately one hour to complete. +Registration to complete the e-Delphi will open approximately four weeks before the first round and e-Delphi round survey. Each e-Delphi survey round will be active for roughly 4-6 weeks and will take approximately one hour to complete. ### Eligibility Criteria for e-Delphi Survey Panellists + ```{r echo=FALSE, warning=FALSE} # Load the gt package @@ -63,28 +65,34 @@ gt(data, id = "delphi_table") %>% ``` + **Definitions** -Professional experience: Paid employment in current disciplines (current or former) post-training experience, 3+ graduate training or professional practice -Graduate or professional degree: Master, PhD, Engineering, Law, Medicine, Public Health +Professional experience: Paid employment in current disciplines (current or former) post-training experience, 3+ graduate training or professional practice Graduate or professional degree: Master, PhD, Engineering, Law, Medicine, Public Health ## Procedure + We will conduct a two-round e-Delphi survey to generate consensus on evaluation criteria. The full survey will be pre-tested and validated prior to administration. Summaries of Round 1 will be compiled for the subsequent round. Custom survey pathways will be generated for each stakeholder group (i.e., panellists from different stakeholder groups will be shown a different collection of candidate items). Within each custom stakeholder survey pathway, panellists will also have the option to skip items or self-declare that they are not qualified to assess certain candidate items. ### Round 1 -Panellists will be invited to rate their level of agreement with candidate items generated from scoping review results and consultation with the study executive group. Free-text boxes will be included for panellists to provide feedback or identify additional candidate items to be included in the next e-Delphi survey round. + +Panellists will be invited to rate their level of agreement with candidate items generated from scoping review results and consultation with the study executive group. Free-text boxes will be included for panellists to provide feedback or identify additional candidate items to be included in the next e-Delphi survey round. ### Round 2 + Regardless of whether they participated in the previous round, panellists will be invited to participate in Round 2 of the e-Delphi survey. They will be invited to rate their level of agreement with newly identified candidate items from the first e-Delphi round and to re-rate their level of agreement for items that did not reach consensus during the previous round. When re-rating their level of agreement, panellists will be presented with their previous round scores alongside the aggregate group results. Anonymous feedback from Round 1 panellists will also be compiled and presented during Round 2. Any newly suggested items during Round 2 will be deliberated on during the consensus meeting. ### Defining Consensus -We will use a 9-point Likert scale (1 = extremely irrelevant to 9 = extremely relevant). Panellist responses will be categorised as irrelevant (1–3), equivocal (4–6), or relevant (7–9). For each item, consensus will be reached if ≥ 80% of the panellist votes fall within the same category (1–3, 4–6, or 7–9) (26). The survey will include a free-text box for panellist to provide comments or suggest new items. Newly suggested items, as well as non-consensus items, will be listed in the second round. + +We will use a 9-point Likert scale (1 = extremely irrelevant to 9 = extremely relevant). Panellist responses will be categorised as irrelevant (1--3), equivocal (4--6), or relevant (7--9). For each item, consensus will be reached if ≥ 80% of the panellist votes fall within the same category (1--3, 4--6, or 7--9) (26). The survey will include a free-text box for panellist to provide comments or suggest new items. Newly suggested items, as well as non-consensus items, will be listed in the second round. ### Stopping Rules -When ≥ 80% of the panellist votes fall within the same category (1–3, 4–6, or 7–9) for a given item, no subsequent rounds of ranking for that item will be performed. Consensus items that are considered equivocal will be further deliberated during the consensus meeting. -## ![](images/Delphi_Consensus_Process - website.svg){fig-align="center" width="1000"} -**Figure 1.** E-Delphi and consensus meeting methodology, methods to achieve consensus on core items for the Public Health Environmental Surveillance Evaluation Framework (PHES-EF). +When ≥ 80% of the panellist votes fall within the same category (1--3, 4--6, or 7--9) for a given item, no subsequent rounds of ranking for that item will be performed. Consensus items that are considered equivocal will be further deliberated during the consensus meeting. + +## ![](images/Delphi_Consensus_Process%20-%20website.svg){fig-align="center" width="1000"} + +[**Figure 2.** E-Delphi and consensus meeting methodology, methods to achieve consensus on core items for the Public Health Environmental Surveillance Evaluation Framework (PHES-EF).]{.caption} ::: {#refs} ::: diff --git a/qmd/images/PHES-EF development process - website.svg b/qmd/images/dev_process.svg similarity index 100% rename from qmd/images/PHES-EF development process - website.svg rename to qmd/images/dev_process.svg diff --git a/qmd/index.qmd b/qmd/index.qmd index a93f3ef..0b39595 100644 --- a/qmd/index.qmd +++ b/qmd/index.qmd @@ -17,7 +17,7 @@ ``` -| [![](images/icon_data-reporting-71.svg){fig-align="center" width="75"}](about.qmd) | [![](images/icon_Laboratory.svg){fig-align="center" width="75"}](methods.qmd) | [![](images/icon_clipboard.svg){fig-align="center" width="75"}](delphi.qmd) | [![](images/icon_researcher.svg){fig-align="center" width="75"}](team.qmd) | [![](images/icon_Documentation.svg){fig-align="center" width="75"}](publications.qmd) | [![](images/icon_international.svg){fig-align="center" width="75"}](contact.qmd) | [![](images/icon_interpretation.svg){fig-align="center" width="75"}](faq.qmd) | +| [![](images/icon_data-reporting-71.svg){fig-align="center" width="75"}](about.qmd) | [![](images/icon_Laboratory.svg){fig-align="center" width="75"}](methods.qmd) | [![](images/icon_clipboard.svg){fig-align="center" width="75"}](delphi.qmd) | [![](images/icon_researcher.svg){fig-align="center" width="75"}](team.qmd) | [![](images/icon_Documentation.svg){fig-align="center" width="75"}](documents.qmd) | [![](images/icon_international.svg){fig-align="center" width="75"}](contact.qmd) | [![](images/icon_interpretation.svg){fig-align="center" width="75"}](faq.qmd) | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | **About** | **Methods**| **e-Delphi Study** | **Team** | **Documents** | **Contact Us** | **FAQ** | diff --git a/qmd/methods.qmd b/qmd/methods.qmd index 3082f8d..9c23562 100644 --- a/qmd/methods.qmd +++ b/qmd/methods.qmd @@ -2,33 +2,36 @@ title: "Methods" --- -The Public Health Environmental Surveillance Evaluation Framework (PHES-EF) will be developed in five steps. In Step 1, a multinational and multidisciplinary executive group will be formed to guide the framework development process. In Step 2, candidate items for Round 1 of the e-Delphi will be generated by conducting relevant scoping reviews and consultation with the study executive group. In Step 3, an international e-Delphi will be conducted over two rounds to develop consensus on items for the framework. In Step 4, the executive group will reconvene to finalize the evaluation framework, discuss standout items, and determine the dissemination strategies. Lastly, Step 5 will focus on disseminating the evaluation framework to all potential stakeholders using traditional and public-oriented methods. +The Public Health Environmental Surveillance Evaluation Framework (PHES-EF) will be developed in five steps. ## Study Steps -::: centered-content -## ![](images/PHES-EF development process - website.svg){fig-align="center" width="700"} -::: -Figure 1. Overview of the development process for the Public Health Environmental Surveillance Evaluation Framework (PHES-EF) + +![](images/dev_process.svg){fig-align="center" width="700"} + +[**Figure 1.** Overview of the development process for the Public Health Environmental Surveillance Evaluation Framework (PHES-EF)]{.caption} ### Step 1: Establishment of the Executive Group + A multinational and multidisciplinary executive group of 12 members will oversee the development of the evaluation framework. The group will bring expertise from the following fields: mathematical sciences, social sciences, environmental and physical sciences, public health and medicine, and public interest/knowledge users. -The executive group will be responsible for providing feedback on the framework development process, connecting the study to other related projects, recruiting e-Delphi panellists, participating in the consensus meeting, disseminating the evaluation framework, advising and/or participating in post-publication activities, and promoting the adoption of the evaluation framework among their professional networks. +The executive group will be responsible for providing feedback on the framework development process, connecting the study to other related projects, recruiting e-Delphi panellists, participating in the consensus meeting, disseminating the evaluation framework, advising and/or participating in post-publication activities, and promoting the adoption of the evaluation framework among their professional networks. You can learn more about the study executive group members [here.](about.qmd) ### Step 2: Generation of Candidate Items -We will generate a list of PHES-EF e-Delphi candidate items by conducting relevant scoping reviews(s) and consulting the study’s expert executive group. -Relevant scoping review registrations and publications can be accessed [here.](publications.qmd) +We will generate a list of PHES-EF e-Delphi candidate items by conducting relevant scoping reviews(s) and consulting the study's expert executive group. + +Relevant scoping review registrations and publications can be accessed [here.](documents.qmd) ### Step 3: International e-Delphi Survey -We will use an electronic Delphi (e-Delphi) technique to develop an internationally accepted evaluation framework for public health environmental surveillance. The Delphi method is an iterative multi-round approach that uses a series of sequential surveys, interspersed by controlled feedback, to elicit consensus among a group of individuals while maintaining anonymity (19,20). We will use the e-Delphi method to overcome geographic barriers and allow us to engage panellists internationally across various time zones. -## ![](images/Delphi_Consensus_Process - website.svg){fig-align="center" width="1000"} -Figure 2. E-Delphi and consensus meeting methodology, methods to achieve consensus on core items for the Public Health Environmental Surveillance Evaluation Framework (PHES-EF). +We will use an electronic Delphi (e-Delphi) technique to develop an internationally accepted evaluation framework for public health environmental surveillance. The Delphi method is an iterative multi-round approach that uses a series of sequential surveys, interspersed by controlled feedback, to elicit consensus among a group of individuals while maintaining anonymity (19,20). We will use the e-Delphi method to overcome geographic barriers and allow us to engage panellists internationally across various time zones. + +For more details, see the [Delphi page.](delphi.qmd) ### Step 4: Consensus Meeting + In preparation for the consensus meeting, the study working group will categorize the remaining candidate items for consideration into a preliminary evaluation framework. This will include merging and/or modifying the remaining items based on gathered feedback from the e-Delphi rounds. An in-person or hybrid consensus meeting will be held after the results of the e-Delphi have been compiled and analysed. The primary objective will be to achieve expert consensus on the final list of items for PHES-EF, through review and discussion of salient items. This process will be guided by the empirical evidence that was identified during the scoping review(s), and the opinions gathered during the e-Delphi process. The secondary objective will be to discuss publication and dissemination strategies for the final evaluation framework. @@ -36,8 +39,7 @@ An in-person or hybrid consensus meeting will be held after the results of the e Steps to produce the final list of items are as follows: (i) present the results of e-Delphi exercise (name, rationale, and score of each item); (ii) discuss the rationale and relevance for including the items in the framework; and (iii) vote on equivocal and non-consensual items. PHES-EF will be developed based on the final list of items that receive consensus during the executive group consensus meeting. ### Step 5: Dissemination of Evaluation Framework -The dissemination of this evaluation framework will start with the publication of the PHES-EF protocol. Active dissemination approaches will include presenting at relevant scientific conferences, holding webinars, and conducting workshops. Additional dissemination practices will be determined during the consensus meeting. - -The development of the evaluation framework will be reported in a statement document that will include the rationale and a brief description of the meeting and the panellists involved. The evaluation framework will be made available as a preprint, prior to publication in an open-access peer-reviewed journal. PHES-EF will be made available using an open license (CC-BY-SA-4.0 license). +The dissemination of this evaluation framework will start with the publication of the PHES-EF protocol. Active dissemination approaches will include presenting at relevant scientific conferences, holding webinars, and conducting workshops. Additional dissemination practices will be determined during the consensus meeting. +The development of the evaluation framework will be reported in a statement document that will include the rationale and a brief description of the meeting and the panellists involved. The evaluation framework will be made available as a preprint, prior to publication in an open-access peer-reviewed journal. PHES-EF will be made available using an open license (CC-BY-SA-4.0 license). diff --git a/qmd/references.bib b/qmd/references.bib index cc1c650..7d5acd8 100644 --- a/qmd/references.bib +++ b/qmd/references.bib @@ -73,3 +73,31 @@ @misc{ocap2023 note = {Citation Key: ocap2023}, langid = {canadian} } + +@article{fynn2020, + title = {A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation}, + author = {Fynn, Judith F. and Hardeman, Wendy and Milton, Karen and Jones, Andy P.}, + year = {2020}, + month = {06}, + date = {2020-06-26}, + journal = {BMC Public Health}, + volume = {20}, + number = {1}, + doi = {10.1186/s12889-020-09062-0}, + url = {http://dx.doi.org/10.1186/s12889-020-09062-0}, + langid = {en} +} + +@article{calba2015, + title = {Surveillance systems evaluation: a systematic review of the existing approaches}, + author = {Calba, Clementine and Goutard, Flavie L and Hoinville, Linda and Hendrikx, Pascal and Lindberg, Ann and Saegerman, Claude and Peyre, Marisa}, + year = {2015}, + month = {05}, + date = {2015-05-01}, + journal = {BMC Public Health}, + volume = {15}, + number = {1}, + doi = {10.1186/s12889-015-1791-5}, + url = {http://dx.doi.org/10.1186/S12889-015-1791-5}, + langid = {en} +}