index.Rmd

---
title: "Module 7: Evaluation"
subtitle: "EME5601: Introduction to Instructional Systems"
author: "Dr. Bret Staudt Willet"
date: "November 29, 2023"
output:
  xaringan::moon_reader:
    lib_dir: libs
    seal: true
    css: [default, 'custom.css', 'custom-fonts.css']
    nature:
      titleSlideClass: [left, bottom]
      highlightStyle: github
      highlightLines: true
      countIncrementalSlides: false
---

```{r setup, include=FALSE}
usethis::use_git_ignore(c("*.csv", "*.rds"))
options(htmltools.dir.version = FALSE)

library(knitr)
library(tidyverse)
library(xaringan)
library(fontawesome)
```

class: inverse, center, middle

# `r fa("fas fa-images", fill = "#fff")`

**View the slides:** 

[bretsw.com/eme5601-fs23-module7](https://bretsw.com/eme5601-fs23-module7)

---

class: inverse, center, middle

# `r fa("fas fa-mug-hot", fill = "#fff")` <br><br> Module 6 <br> Recap

---

# `r fa("fas fa-mug-hot", fill = "#fff")` Design & Development

Design & Development are **late steps** in the iterative design cycle:

```{r, out.width = "480px", echo = FALSE, fig.align = "center"}
include_graphics("img/Rothwell-fig2-1.png")
```

<div class="caption">
Figure 2.1 from Rothwell et al. (2016, p. 20)
<br><br>
</div>

---

# `r fa("fas fa-mug-hot", fill = "#fff")` Design & Development

```{r, out.width = "480px", echo = FALSE, fig.align = "center"}
include_graphics("img/design-book.jpg")
```

---

# `r fa("fas fa-mug-hot", fill = "#fff")` Design & Development

```{r, out.width = "720px", echo = FALSE, fig.align = "center"}
include_graphics("img/teapot3.png")
```

**Practice last week:**

- Team A designed a *Gagne-Style Nine Events Model for Self-Directed Learning*
- Team B designed a solution for working-from-home vs. return-to-office

---

class: inverse, center, middle

# `r fa("fas fa-utensils", fill = "#fff")` <br><br> Module 5 <br> Review

---

# `r fa("fas fa-utensils", fill = "#fff")` Work Analysis

```{r, out.width = "720px", echo = FALSE, fig.align = "center"}
include_graphics("img/5-chef-work.png")
```

- **Structure** the analysis in a variety of ways: headings, table, diagrams
- **Breadth and depth** of responsibilities/duties, tasks, subtasks, KSAs

--

<hr>

`r fa("far fa-comments", fill = "#782F40")` **Exercise:** Work Analysis of Project Manager

---

class: inverse, center, middle

#  `r fa("fas fa-leaf", fill = "#fff")` <br><br> Module 7 <br> Evaluation

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluation

```{r, out.width = "100%", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-leaf.png")
```

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluation

```{r, out.width = "100%", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-elephant.png")
```

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluation

```{r, out.width = "600px", echo = FALSE, fig.align = "center"}
include_graphics("img/blind_monks_examining_an_elephant.jpg")
```

**"Blind Monks Examining an Elephant" by Itcho Hanabusa (1652–1724).** 

Image is in the public domain, hosted by [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Blind_monks_examining_an_elephant.jpg)

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluation

```{r, out.width = "420px", echo = FALSE, fig.align = "center"}
include_graphics("img/blind_monks_examining_an_elephant.jpg")
```

[Blind Men and An Elephant](https://en.wikipedia.org/wiki/Blind_men_and_an_elephant)

> "The parable of the blind men and an elephant is a story of a group of blind men who have never come across an elephant before and who learn and imagine what the elephant is like by touching it. Each blind man feels a different part of the elephant's body, but only one part, such as the side or the tusk. They then describe the elephant based on their limited experience and their descriptions of the elephant are different from each other."

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluation

```{r, out.width = "600px", echo = FALSE, fig.align = "center"}
include_graphics("img/blind_monks_examining_an_elephant.jpg")
```

### Exercise:

--

`r fa("fas fa-person-walking", fill = "#782F40")` Walk around and notice the elephants (the things tricky to evaluate)





---

class: inverse, center, middle

#  `r fa("fas fa-leaf", fill = "#fff")` <br><br> Module 7 <br> Evaluation

**Part One: Evaluation of Learning**

---

class: inverse, center, middle

# `r fa("fas fa-leaf", fill = "#fff")` <br><br> Designing Learning Assessments <br> (Ch. 13)

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "360px", echo = FALSE, fig.align = "center"}
include_graphics("img/otter.png")
```

--

### Performance Objectives `r fa("fas fa-arrow-right", fill = "#782F40")` Performance Metrics

--

- **Why?** Accountability for showing results

--

- **What?** Monitoring learner achievement

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "360px", echo = FALSE, fig.align = "center"}
include_graphics("img/otter.png")
```

### Performance Objectives `r fa("fas fa-arrow-right", fill = "#782F40")` Performance Metrics

<hr>

### `r fa("far fa-comments", fill = "#782F40")` Discuss in groups:

--

What is the best learning assessment or evaluation you have experienced? Why?

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "360px", echo = FALSE, fig.align = "center"}
include_graphics("img/teaching-to-the-test.png")
```

### Performance Metrics are benchmarks that guide ISD

--

<hr>

### `r fa("far fa-comments", fill = "#782F40")` Discuss in groups:

--

What about issues of "teaching to the test"?

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "560px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-elephant.png")
```

### **What** should be measured?

--

- **Kirkpatrick's Four Levels:**
  - Participant reaction (enjoyment)
  - Participant learning (meeting performance objectives)
  - Transfer (on-the-job performance change)
  - Organizational impact

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "560px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-elephant.png")
```

### **What** sources of information for performance metrics?

--

- Performance objectives

--

- Learner/worker actual performance

--

- Stakeholder preferences

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "480px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-elephant.png")
```

### **What** issues affect performance metrics? (pp. 218-220)

--

- Objectives and omissions

--

- Content and method of instruction

--

- Amount of learning, length of instruction, place of instruction

--

- Relevance, learning transfer, and application of learning

--

- Skill of instructor

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "480px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-elephant.png")
```

### **How** should performance metrics be measured?

--

- Learner involvement

--

- Manager and stakeholder involvement

--

- Time required

--

- Cost

--

- Relevant quantifiable data

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "480px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-elephant.png")
```

### **How** should performance metrics be measured?

--

- Questionnaires, interview guides, observation forms, simulations

--

- Criterion-referenced tests: Essay, fill-in-the-blank, completion, multiple-choice, true-false, matching

--

- Others: Advisory committee, external assessment center, attitude survey, group discussion, exit interview

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "560px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-leaf.png")
```

--

### Trustworthiness (pp. 228-229)

- **Reliability:** consistency of measurement (inter-rater, test-retest, inter-method, internal consistency)

--

- **Validity:** accuracy of measurement (construct, convergent, discriminant, content, representation, face, criterion, concurrent, predictive)

--

- **Credibility:** belief in measurement (method, tool)

---

# `r fa("fas fa-leaf", fill = "#fff")` Learning Assessments

```{r, out.width = "560px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-leaf.png")
```

### Questions for Formative Evaluation (p. 230)

--

- How well are assessments aligned with the organization’s strategic goals?

--

- How well are assessments aligned with the desired learning outcomes?

--

- How well are assessments aligned with instructional strategies?

--

- What can be done to improve alignment? 






---

class: inverse, center, middle

#  `r fa("fas fa-leaf", fill = "#fff")` <br><br> Module 7 <br> Evaluation

**Part Two: Evaluation of Design**

---

class: inverse, center, middle

# `r fa("fas fa-leaf", fill = "#fff")` <br><br> Evaluating Instructional and Noninstructional Interventions <br> (Ch. 14)

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### Key Questions

```{r, out.width = "560px", echo = FALSE, fig.align = "center"}
include_graphics("img/Rothwell-table14-1.png")
```

<div class="caption">
Table 14.1 from Rothwell et al. (2016, p. 235)
<br><br>
</div>

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### Types of Data

```{r, out.width = "420px", echo = FALSE, fig.align = "center"}
include_graphics("img/data.jpg")
```

--

- Quantitative vs. Qualitative

--

- Self-Report, Observational, Extant/Naturalistic

--

- Small vs. Big (Human vs. Computational)

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### Methods of Data Collection

```{r, out.width = "420px", echo = FALSE, fig.align = "center"}
include_graphics("img/data-collection-medium.jpg")
```

--

- Interviews and Focus Groups

--

- Observation

--

- Surveys and Questionnaires

--

- Tests

--

- Data Mining and Analytics

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### **Formative** Evaluation of Design

--

- Formative **Product** Evaluation

--

- Formative **Process** Evaluation

--

<hr>

**Planning for formative evaluation:**

--

- Step 1: Determining purpose, obejctives, audience, and subject

--

- Step 2: Assessing information needs

--

- Step 3: Considering proper protocol

--

- Step 4: Describing the population to be studied and selecting subjects

--

- Step 5: Identifying other variables of importance

--

- Step 6: Formulating a study design

--

- Step 7: Formulating a management plan to guide the study

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### **Formative** Evaluation of Design

**Planning for formative evaluation:**

--

- Expert reviews

--

- Management or executive rehearsals

--

- Individualized pretests and pilot tests

--

- Group pretests and pilot tests

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### **Summative** Evaluation of Design

--

**Kirkpatrick's Four Levels:**

--

- **Level 1:** Learner satisfaction (did they like it?)

--

- **Level 2:** Knowledge or skill acquisition (did they learn it?)

--

- **Level 3:** Learner transfer (did they apply it on the job?)

--

- **Level 4:** Organizational impact (did it make a difference?)

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### **Summative** Evaluation of Design

**Phillips' (2011) ROI Model:**

--

- **Level 1:** Learner satisfaction (did they like it?)

- **Level 2:** Knowledge or skill acquisition (did they learn it?)

- **Level 3:** Learner transfer (did they apply it on the job?)

- **Level 4:** Organizational impact (did it make a difference?)

--

- **Level 5:** Return-on-investment (financial impact of Level 4)

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

### **Summative** Evaluation of Design

**Brinkerhoff's (2010) Success Case Method (SCM):**

--

Holistic and systemic approach that answers: 

--

- How well is an organization using learning to improve performance?

--

- What organizational processes/resources are in place to support performance improvement?

--

- What needs to be improved?

--

- What organizational barriers stand in the way of performance improvement? 

---

# `r fa("fas fa-leaf", fill = "#fff")` Evaluating Interventions

```{r, out.width = "720px", echo = FALSE, fig.align = "center"}
include_graphics("img/girl-elephant.png")
```

### `r fa("far fa-comments", fill = "#782F40")` Discuss in groups:

--

Develop a formative or summative approach to evaluating interventions...





---

class: inverse, center, middle

# `r fa("fas fa-leaf", fill = "#fff")` <br><br> Revising Instructional and Noninstructional Solutions Based on Data <br> (Ch. 15)

---

# `r fa("fas fa-leaf", fill = "#fff")` Revising Interventions

--

**Revise content:**

--

- Key messages, narrative, and substantive information in the product or program

--

**Revise delivery:**

--

- Exercises, activities, methods

--

**Evaluate effectiveness:**

--

- Did the changes lead to improved outcomes?

--

**Rapid Prototyping:**

--

- Efficiency vs. Effectiveness?

---

# `r fa("fas fa-leaf", fill = "#fff")` Revising Interventions

### Stakholder Support

```{r, echo = FALSE, out.width = "48%", fig.align = "default", fig.show = "hold"}
include_graphics(c("img/Rothwell-exhibit15-1a.png", "img/Rothwell-exhibit15-1b.png"))
```

<div class="caption">
Exhibit 15.1 from Rothwell et al. (2016, pp. 268-269)
<br><br>
</div>





---

class: inverse, center, middle

# `r fa("fas fa-leaf", fill = "#fff")` <br><br> Implementing Instructional and Noninstructional Interventions <br> (Ch. 16)

---

# `r fa("fas fa-leaf", fill = "#fff")` Implementing Interventions

### Implementation Plan

**Begin with the end in mind**

```{r, out.width = "600px", echo = FALSE, fig.align = "center"}
include_graphics("img/Rothwell-exhibit16-1.png")
```

<div class="caption">
Exhibit 16.1 from Rothwell et al. (2016, p. 273)
<br><br>
</div>

---

# `r fa("fas fa-leaf", fill = "#fff")` Implementing Interventions

### Implementation Plan

**Align** learning and performance goals with organizational goals

```{r, out.width = "720px", echo = FALSE, fig.align = "center"}
include_graphics("img/Rothwell-figure16-1.png")
```

<div class="caption">
Figure 16.1 from Rothwell et al. (2016, p. 274)
<br><br>
</div>

---

# `r fa("fas fa-leaf", fill = "#fff")` Implementing Interventions

### Implementation Plan

**Engage stakeholders:** (Table 16.1, pp. 279-280)

--

- Subject matter experts

--

- Technology experts

--

- Instructional designers

--

- Graphic designers

--

- Facilitators

--

- Communication specialists

--

- Project managers

--

- Business leaders


---

# `r fa("fas fa-leaf", fill = "#fff")` Implementing Interventions

### Implementation Dissemination

--

- Self-directed learning

--

- Self-paced interventions

--

- In-person interventions

--

- Blended learning

--

- Train-the-trainer

--

- Monitoring quality (back to performance metrics)

--

<hr>

### Implementation Diffusion

--

Focus on the speed, depth, and quality of **adoption**





---

class: inverse, center, middle

# `r fa("fas fa-toolbox", fill = "#fff")` <br><br> Expanding the <br> ISD Analysis Toolbox

---

# `r fa("fas fa-toolbox", fill = "#fff")` ISD Analysis Toolbox

```{r, out.width = "280px", echo = FALSE, fig.align = "center"}
include_graphics("img/toolbench.jpg")
```

- Systems Analysis
- Needs Assessment / Needs Analysis
- Training Requirements Analysis
- Root Cause Analysis
- Competency Assessment
- Performance Analysis
- Learner Assessment
- Setting Analysis
- Developmental Setting Assessment
- Job Analysis
- Task Analysis
- Content (Subject Matter) Analysis

---

# `r fa("fas fa-toolbox", fill = "#fff")` ISD Analysis Toolbox

```{r, out.width = "280px", echo = FALSE, fig.align = "center"}
include_graphics("img/toolbench.jpg")
```

- Goal Analysis
- Learning Task Analysis
- Hierarchical Analysis
- Cluster Analysis
- Procedural Analysis
- Materials Assessment / Materials Analysis
- Cost-Benefit Analysis

---

# `r fa("fas fa-toolbox", fill = "#fff")` ISD Analysis Toolbox

```{r, out.width = "280px", echo = FALSE, fig.align = "center"}
include_graphics("img/toolbench.jpg")
```

- **Performance Metrics**
- **Formative Product Evaluation**
- **Formative Process Evaluation**
- **Kirkpatrick's Four Levels Evaluation**
- **ROI Five Levels Evaluation**
- **Success Case Evaluation**

---

# `r fa("fas fa-toolbox", fill = "#fff")` Evaluation

```{r, out.width = "480px", echo = FALSE, fig.align = "center"}
include_graphics("img/Rothwell-fig2-1.png")
```

<div class="caption">
Figure 2.1 from Rothwell et al. (2016, p. 20)
<br><br>
</div>





---

class: inverse, center, middle

# `r fa("fas fa-clipboard-list", fill = "#fff")` <br><br> Course Evaluations





---

class: inverse, center, middle

# `r fa("fas fa-binoculars", fill = "#fff")` <br><br> Looking ahead

---

# `r fa("fas fa-calendar-day", fill = "#fff")` Semester Schedule

```{r, out.width = "480px", echo = FALSE, fig.align = "center"}
include_graphics("img/across-time.jpg")
```

- **Module 1:** Introduction to Instructional Systems Design

- **Module 2:** Systems Analysis

- **Module 3:** ISD & HPT

- **Module 4:** Needs Assessment

- **Module 5:** Work Analysis

- **Module 6:** Design & Development

- **Module 7: Evaluation**

---

# `r fa("far fa-keyboard", fill = "#fff")` Major Assignments

```{r, out.width = "320px", echo = FALSE, fig.align = "center"}
include_graphics("img/build.jpg")
```

- **Assignments** (70%)

  - Module 2 Assignment: Systems Analysis paper (150 points)

  - Module 3 Assignment: Annotated Bibliography 1 (50 points)

  - Module 4 Assignment: Needs Assessment paper (150 points)

  - Module 5 Assignment: Annotated Bibliography 2 (50 points)

  - Module 6 Assignment: Work Analysis paper (150 points)

  - **Module 7 Assignment: ISD Process Model paper (150 points)**

---

class: inverse, center, middle

# `r fa("fas fa-question", fill = "#fff")` <br><br> Questions

<hr>

**What questions can I answer for you now?**

**How can I support you this week?**

<hr>

`r fa("fas fa-envelope", fill = "#fff")` [bret.staudtwillet@fsu.edu](mailto:bret.staudtwillet@fsu.edu) | `r fa("fas fa-globe", fill = "#fff")` [bretsw.com](https://bretsw.com) | `r fa("fab fa-github", fill = "#fff")` [GitHub](https://github.com/bretsw/)