Health information technology – evaluation plan project – evaluation

STUCK with your assignment? When is it due? Hire our professional essay experts who are available online 24/7 for an essay paper written to a high standard at a reasonable price.


Order a Similar Paper Order a Different Paper

  

Because of the great differences between HIT systems and different goals of an evaluation, there is no one-size-fits-all evaluation plan. Different technologies require different evaluation methods. Consequently, in this part of the Evaluation Plan Project, there is a need to conduct research on how system implementations prevent errors, enhance the quality, and efficiency of health systems. Select one research goal and viewpoint to use in the evaluation.

In a 4-page documentation:

· Identify which of the cases from the text, will be evaluated in this project, summarize the case, and explain why the selected system featured in the case.

· Next, summarize your three research findings on similar HIT implementations. Include the models, evaluation methods, findings, and plans for reevaluation in each article.

· Critique the HIT implementations: Identify successful and unsuccessful elements of the implementations. Explain the reasons for the successful elements. Identify areas for improvement and explain why (or how) they could be improved.

· Create an evaluation goal and identify the viewpoint related to the goal that will guide your own evaluation plan. Provide a rationale for choosing that goal and viewpoint.

Reference

Abbasi-moghaddam, M. A., Zarei, E., Bagherzadeh, R., Dargahi, H., & Farrokhi, P. (2019). Evaluation of service quality from patients’ viewpoint. BMC Health Services Research, 19(1), 1–7. https://doi:10.1186/s12913-019-3998-0

Aven, T. (2016). Risk assessment and risk management: Review of recent advances on their foundation. European Journal of Operational Research, 253(1), 1–13. https://doi.org/10.1016/j.ejor.2015.12.023

Brender, J. (2006). Handbook of evaluation methods for health informatics. Elsevier Academic Press

Carayon, P., Smith, P., Hundt, A., Kuruchittham, V., & Li, Q. (2009). Implementation of an electronic health records system in a small clinic: the viewpoint of clinic staff. Behaviour & Information Technology, 28(1), 5–20. https://doi.org/10.1080/01449290701628178

Cho, H. Yen, P. Dowding, D., Merril, J., & Schnall, R. (2018, October). A multi-level usability evaluation of mobile health applications: A case study. Journal of Biomedical Informatics, 86, 79–89. https://doi.org/10.1016/j.jbi.2018.08.012

Hamann, D. J., & Bezboruah, K. C. (2020). Outcomes of health information technology utilization in nursing homes: Do implementation processes matter? Health Informatics Journal, 26(3), 2249–2264. https://doi.org/10.1177/1460458219899556

Huang, Y.-H., & Gramopadhye, A. K. (2016). Recommendations for health information technology implementation in rural hospitals. International Journal of Health Care Quality Assurance (09526862), 29(4), 454–474. https://doi.org/10.1108/IJHCQA-09-2015-0115

Recommendations for health
information technology

implementation in rural hospitals
Yuan-Han Huang

Department of Industrial Engineering, The Pennsylvania State University,
Erie, Pennsylvania, USA, and
Anand K. Gramopadhye

Department of Industrial Engineering, Clemson University,
Clemson, South Carolina, USA

Abstract
Purpose – The purpose of this paper is to investigate violations against work standards associated
with using a new health information technology (HIT) system. Relevant recommendations for
implementing HIT in rural hospitals are provided and discussed to achieve meaningful use.
Design/methodology/approach – An observational study is conducted to map medication
administration process while using a HIT system in a rural hospital. Follow-up focus groups are held
to determine and verify potential adverse factors related to using the HIT system while passing
drugs to patients.
Findings – A detailed task analysis demonstrated several violations, such as only relying on the
barcode scanning system to match up with patient and drugs could potentially result in the medical
staff forgetting to provide drug information verbally before administering drugs. There was also a lack
of regulated and clear work procedure in using the new HIT system. In addition, the computer system
controls and displays could not be adjusted so as to satisfy the users’ expectations. Nurses prepared
medications and documentation in an environment that was prone to interruptions.
Originality/value – Recommendations for implementing a HIT system in rural healthcare facilities
can be categorized into five areas: people, tasks, tools, environment, and organization. Detailed
remedial measures are provided for achieving continuous process improvements at resource-limited
healthcare facilities in rural areas.
Keywords Process improvement, Electronic health record, Health information technology,
Medication administration process, Workflow
Paper type Research paper

1. Introduction
Health information technology (HIT) systems have been recognized as a solution for
reducing medication errors and improving the quality of care in the healthcare field
(Bates et al., 1998; Poon et al., 2006, 2010; Jaana et al., 2012; El-Kareh et al., 2013;
Patterson, 2012). Since the late 1990s, HIT systems have been promoted in the
healthcare sector not only because they are compatible with the commonly used clinical
documentation system and the electronic health record (EHR) system (Khoury, 1997;
Krall, 1995; Berg et al., 1998), but also because they include a computerized physician
order entry (CPOE) system, a barcode scanning system, electronic medication
administration records (eMAR, a formal computer-based record of the drugs
administered to a patient), decision support tools, etc. (Wakefield et al., 2010;
McCartney, 2006, 2011; Goldschmidt, 2005). The aforementioned systems cannot be
considered separately if the US healthcare providers wish to achieve meaningful use of
the EHR which was introduced by the US administration in a law known as the Health

International Journal of Health
Care Quality Assurance
Vol. 29 No. 4, 2016
pp. 454-474
© Emerald Group Publishing Limited
0952-6862
DOI 10.1108/IJHCQA-09-2015-0115

Received 22 September 2015
Revised 21 January 2016
18 February 2016
26 February 2016
Accepted 29 February 2016

The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/0952-6862.htm

454

IJHCQA
29,4

Information Technology for Economic and Clinical Health Act in 2009 (Blumenthal,
2010; Jha, 2010; Casey et al., 2013).

For monitoring the progress of implementing the “EHR” system (please note that the
EHR system does not just describe the electronic clinical documentation system specifically,
but more appropriately represents the whole package of HIT systems used to achieve
quality of care), the Healthcare Information Management System Society (HIMSS) uses an
eight-stage electronic medical record adoption model (EMRAM) to track the progress of
EHR implementation in various categories of healthcare facilities (Garets and Davis, 2006;
Hersh and Wright, 2008). The EMRAM model ranges from stage 0, which indicates a HIT
system that cannot communicate between ancillary departments, to stage 7, which
describes the establishment of a fully paperless clinical environment, data warehousing,
and a mechanism for data exchange among facilities (HIMSS Analytics, 2014b).

Table I shows detailed EHR adoption model scores for nine categories of healthcare
facilities, in which teaching hospitals have the highest adoption score of 5.18; the
critical access hospitals and rural hospitals have the lowest scores, 3.60 and 3.59,
respectively out of 7.00, (HIMSS Analytics, 2014a). In the USA, rural hospitals are the
hospitals located at the rural areas, defined by US Census Bureau. Rural hospitals in
the USA have relatively small amount of beds and physicians due to less population
resides in the rural areas; but rural hospitals are usually defined by geographic
location instead of size of the hospitals. In the USA, there are around 25 percent US
population live in the rural areas, but only about 10 percent of physicians practice in
these areas. Due to less medical practices in the rural areas, the rural healthcare
facilities are relatively smaller than hospitals in the urban, and the rural hospitals
are usually supported by the government (National Rural Health Association, 2015).

In addition, in the first quarter of 2014 the HIMSS EHR adoption model scores by
number of bed sizes showed that hospitals with fewer than 100 beds have the lowest
score, 3.65 out of 7.00. Hospitals with 501-600 beds have the highest adoption score of
5.21 (HIMSS Analytics, 2014a).

Although the government provides funding incentives and has clear expectations
for the meaningful use and widespread adoption of HIT (Bahensky et al., 2008;
Abramson et al., 2012), small-scale rural hospitals still lag behind other classes of
healthcare facilities (Table I) (HIMSS Analytics, 2014a; Jaana et al., 2012; American
Hospital Association, 2011; Yeager et al., 2010). This study mainly focusses on
investigating the impact of using the new HIT system on the workflow in a rural
hospital specifically on the medication administration process.

Hospital type category Mean Number

Academic/teaching hospitals 5.18 209
General medical/surgical hospitals 4.48 3,177
Urban hospitals 4.27 4,273
Integrated healthcare delivery systems (IDS) 4.18 3,624
Non-academic hospitals 4.08 5,240
Independent hospitals 4.00 1,807
Others 3.62 2,272
Critical access hospitals (CAH) 3.60 1,341
Rural hospitals 3.59 1,176
Source: HIMSS Analytics (2014a)

Table I.
HIMSS EHR

adoption model
scores by hospital

type in the first
quarter, 2014

455

HIT
implementation

in rural
hospitals

2. Methods
In this study, an observational analysis was used to map the workflow of the
medication administration process while using the new HIT system in a rural hospital
in the USA. The medication administration process was described by hierarchical task
analysis (HTA) diagrams. Follow-up focus groups with the medical staff were held to
review the medication administration process HTA diagram and explore violation
against work standards associated with the HIT system. This study focusses on
investigating the misconduct of work standards that are related to the HIT in the
medication administration process. The process of medication administration is
considered to be a series of complex tasks that involve multidisciplinary interactions
between the medical staff, including physicians, pharmacists, and nurses (Fraind et al.,
2002; Grigg et al., 2011; Sittig and Singh, 2010; Wetterneck et al., 2012). Most HIT
systems (e.g. EHR, CPOE, or the barcode scanning system) are involved in the five
main phases of the medication administration process: prescribing, documenting,
transporting, administering, and monitoring (Agrawal, 2009; Agrawal and Glasser,
2009; Aspden et al., 2007). These HIT systems were developed primarily to prevent
adverse events such as medication errors in drug ordering, transcription, dispensing,
and administration (Lisby et al., 2005; Boockvar et al., 2010; Abramson et al., 2014).

2.1 Observational study settings and design
After two months of implementing the new EHR, CPOC, and barcode scanning systems
in a rural South Carolina hospital (which has 55 beds), an observational study was
conducted to investigate the changes made to the workflow with regards to the
medication administration process in the medical/surgical unit between May and August
of 2012. In the facility being studied, the medical/surgical unit was typically staffed with
one attending physician, three nurses, and one nursing assistant per eight hour shift. The
lab and pharmacy staffs worked from 8:00 a.m. to 5:00 p.m., Monday-Friday.

Data collection took place only during the first shift (starting at 7:00 a.m.) due to the
earlier shift not being able to collect detailed medication administration task
information from the nurses, physicians, patients, and the patients’ families on the
floor. Also, the relevant tasks for the lab and the pharmacy, such as ordering lab tests
or restocking medications by the pharmacy technician, could only take place during
the morning shift.

The study participants were recruited and informed about the study via
announcements in their staff meetings. All participants voluntarily scheduled their
observation times with the research team in advance. In this observational study,
nurses were shadowed by trained research assistants while they administered drugs to
patients during the first shift. The following information was documented in real time
during the observations: time to perform a specific task; type and number of drugs for
the patient; interruptions that occurred during the task; descriptions of the nursing
task; and descriptions of the documentation task.

All of the observational data were verified by the research advisors to ensure the
quality of the data. Verified observational data were transcribed into visualized HTA)
diagrams for review by the focus groups. The HTA diagram has been demonstrated to
be a comprehensive tool for describing activities and locating violations or deviations
in a series of tasks (e.g. in the fields of healthcare or aviation maintenance) (Drury, 2008;
Lane et al., 2006). The detailed methodology and application of HTA diagrams to
medication administration processes can be found in Huang and Gramopadhye’s (2014)
previous studies.

456

IJHCQA
29,4

2.2 Focus group settings and design
The medication administration process was observed and mapped by an industrial
engineering research team. However, none of the research team members had medical
training to detect potential violations in the process. Thus, focus groups were used to
identify the failures to follow of work standards relating to EHRs in the medication
administration process when using an EHR system. Two focus group interviews
involving medical staff were held at the hospital under study two weeks after the first
phase of the observational study. The focus group participants were recruited via
internal staff meetings and all of them volunteered to participate in the study.

A trained research assistant served as the moderator when conducting focus group
interviews via a set of probe questions, while another research assistant transcribed the
discussion verbatim during each focus group. The probe questions were designed to be
based on the cluster of tasks from the HTA diagram (results of the observational study).
Those probes were used to address potential issues in each medication administration
task. The detailed focus group probe questions are shown in the following list:

(1) How do nurses access the medication administration records?
(2) How does the new system change the way records are reviewed?
(3) How does the new system communicate with the medication cart?
(4) How does the new system communicate with the medication station?

(5) Does the new system impact the requirement of having a witness while
preparing certain medications?

(6) How do nurses check out medical supplies using the new system?

(7) If supplies brought to the patient room are not used, how are they handled?

(8) How do nurses perform charting with the new system in the patient room?

(9) If the patient is not ready for medications (e.g. patient is sleeping), how does the
nurse handle this with the new system?

The focus group study mainly concentrated on the medical staff’s concerns about how
the new HIT system would impact the medication administration workflow. Consensus
about misconducts in the medication administration process was reached through open
discussion. During the focus group interviews, the moderator introduced each task in
the order in which it is embedded in the medication administration process. The main
responsibility of the moderator was to ensure that the participants had a shared
understanding and agreement about the potential violations or deviations that took
place when performing a particular task.

All the participants in the observational study and the focus groups were required to
sign a consent form that specified that there were no adverse consequences for
withdrawing from the study. No personally identifiable information about the medical staff
or patients was recorded. All research data were kept confidential. All the study protocols
were reviewed and approved by the Clemson University Institutional Review Board.

3. Results
For this study, seven observational periods (based on the number of patients) were
conducted during the morning medication administration rounds. Each observational
period was approximately three hours. Observational data primarily included the
nursing tasks performed while passing drugs to the patients. All the tasks were

457

HIT
implementation

in rural
hospitals

converted into a hierarchical structure and presented in an HTA diagram. Figure 1
shows the HTA diagram for administering medications when using the new HIT system.

In the two focus group sessions, there were five registered nurses (RNs) from the
medical/surgical unit and three RNs from the critical care unit/emergency room. All of
the participants were female. In the focus group interviews, the HTA diagram
(Figure 1) was presented to all the participants, and consensus about potential
violations or deviations during a clinical work standard relating to the new HIT system
were generated from the discussion. There were 15 adverse factors that could deviate
from the normative clinical workflow, potentially leading to medication errors (Adverse
Factors a-o in Table II), and these adverse factors can be classified into six categories:
barcode scanning tasks, computer interface or operating issues, unfamiliar procedures,
omitted procedures, logistic issues, and miscellaneous others. Table II shows the 15
potential adverse factors divided by category.

3.1 Potential adverse factors in barcode scanning tasks
In general, the barcode scanning system is used to verify the medication (type and
dose) and patient identification while administering medication. However, since
introducing the new barcode scanning system, it has been noted that nurses would just
scan the patient’s barcode on the wristband rather than interacting with the patient in
person (Adverse Factor a). Nurses should have a simple conversation with the patient
(e.g. greeting them) not only to verify the patient’s name and date of birth, but also to
evaluate the patient’s cognitive status.

In addition, the current study discovered that nurses often possess extra patient
barcodes for convenience (Adverse Factor b). Nurses would scan the wristband before
or after administering the drugs to the patient, but not at the time the medication was
being given. The main reason for possessing a patient’s barcode is that the barcoded
wristband can be smudged and hard to scan when the patient stays in the hospital too
long without the wristband being replaced.

3.2 Computer interface or operating issues
During the observations and the discussions in the focus groups, participants
mentioned that the information in the EHR system is not easy to read. The size of the
icons and fonts on the screen were too small to read and they could not be adjusted
(Adverse Factor c). Sometimes, nurses would select the wrong function or not realize
that they had made typos while charting in the system.

Another issue regarding computer operation was that the mouse speed is not
adjustable. Thus, users would have problems tracking the movement of the mouse
cursor (Adverse Factor d). A lack of appropriate feedback was another issue while
operating the new HIT system (Adverse Factor e). For example, the new barcode
scanning system makes a beeping sound as feedback, confirming that the barcode has
been read by the system. However, the audio feedback cannot distinguish whether or
not the correct patient/medication information was obtained.

3.3 Unfamiliar Procedures
Before the new barcode scanning system was implemented, the nurses usually
collected the barcode stickers from the package of medical supplies and then
manually input the barcode information into the computer system for tracking the
patient’s consumption of supplies. Although a new barcode scanning system for

458

IJHCQA
29,4

(P
la

n
2

.3
:
d
o
2

.3
.1

o
r/

a
n
d
2

.3
.2

o
r/

a
n
d

2
.3

.3
o

r/
a
n
d
2

.3
.4

t
h
e
n
2

.3
.5

a
cc

o
rd

in
g
t
o
p

a
tie

n
t’s

e
M

A
R

)

1
R

e
vi

e
w

e
le

ct
ro

n
ic

m
e
d
ic

a
tio

n
re

co
rd

s
(e

M
A

R
)

2
M

e
d
ic

a
tio

n
p
re

p
a
ra

tio
n

4
F

in
a
liz

e
d
o
cu

m
e
n
ta

tio
n

1
.6

.1
R

e
a
d
p

a
tie

n
t

n
a
m

e

1
.6

.2
R

e
a
d
d

ru
g

n
a
m

e

1
.6

.3
R

e
a
d
d

ru
g

fo
rm

1
.6

.4
R

e
a
d
d

ru
g

d
o
se

1
.6

.5
R

e
a
d
d

ru
g

a
d
m

in
is

tr
a
tio

n
tim

e

(P
la

n
1

:
d
o
1

.1
-1

.6
in

o
rd

e
r)

(P
la

n
1

.6
:

d
o
1

.6
.1

-1
.6

.5
t

h
e
n
e

xi
t

)

2
.3

O
b
ta

in
m

e
d
ic

a
tio

n
s

2
.1

G
e
t

to
o
ls

f
o
r

d
ru

g

p
re

p
a
ra

tio
n

2
.1

.1
H

e
a
d
t

o
t

h
e

n
u
rs

e
s

ta
tio

n

2
.1

.2
L
o
ca

te
t

o
o
ls

fo
r

m
e
d
ic

a
tio

n
p
re

p
a
ra

tio
n

2
.1

.3
G

a
th

e
r

e
ss

e
n
tia

l t
o
o
ls

2
.2

G
e
t

m
e
d
ic

a
l

su
p
p
lie

s
fo

r
p
a
tie

n
t

2
.2

.1
H

e
a
d
t
o
t
h
e

su
p
p
ly

r
o
o
m

2
.2

.2
S

e
a
rc

h
f
o
r

re
le

va
n
t

su
p
p
lie

s

2
.2

.3
G

a
th

e
r

e
ss

e
n
tia

l
su

p
p
lie

s

2
.3

.1
G

e
t

m
e
d
ic

a
tio

n
fr

o
m

t
h
e

ca

rt

2
.3

.2
G

e
t

m
e
d
ic

a
tio

n
fr

o
m

t
h

e
st

a
tio

n

2
.3

.5
P

re
p
a
re

m
e
d
ic

a
tio

n
(r

e
fe

r
to

n
e
xt

p
a
g
e
)

2
.3

.1
.1

L
o
g
in

to
t
h
e

ca
rt

a
n
d

u
n
lo

ck
t
h
e

d
ra

w
e
r

2
.3

.1
.2

L
o
ca

te
t
h
e

co
rr

e
ct

d
ra

w
e
r

(P
la

n
2

.2
:
d
o
2

.2
.1

-2
.2

.4
in

o
rd

e
r)

2
.3

.2
.1

L
o
g
in

to
t
h
e

st
a
tio

n
a

n
d

u
n
lo

ck
t
h
e

d
ra

w
e
r

2
.3

.2
.2

L
o
ca

te
t

h
e

co
rr

e
ct

m
e
d
ic

a
tio

n
ch

a
m

b
e
r

2
.3

.2
.3

O
p
e
n
t

h
e

ch
a
m

b
e
r

(P
la

n
2

.3
.1

:
d
o
2

.3
.1

.1
-2

.3
.1

.2
,

2
.3

.1
.3

in
o

rd
e
r)

(P
la

n
2

.3
.2

:
d

o
2

.3
.2

.1
-2

.3
.2

.2
,

2
.3

.2
.3

in
o

rd
e

r)

(P
la

n
2

.1
:

d
o
2

.1
.1

-2
.1

.2
,

2
.1

.3
in

o
rd

e
r)

(P
la

n
0

:
d
o
1

,
2
,

3
,

4
in

o
rd

e
r)

0

A
d
m

in
is

te
r

m
e
d
ic

a
tio

n
t

o

p
a
tie

n
t

4
.1

C
o
n
fir

m
a

ll
m

e
d
ic

a
tio

n
s

a
re

p
a
ss

e
d
in

e
M

A
R

4
.2

W
a
sh

h
a
n
d
s

P
la

n
2

:

N
e

e
d

t
o

o
ls

?

Y
e
s

N
o

2
.1

N
e

e
d

su

p
p

lie
s?

2

2
.2

2
.3

Y
e
s

N
o

3
P

a
ss

m
e
d
ic

a
tio

n
s

(r
e
fe

r
to

n
e
xt

p
a
g
e
)

2
.2

.4
S

ca
n
b

a
rc

o
d
e

o
n
t
h
e
s

u
p
p
ly

p
a
ck

a
g
e

1
.1

W
a
sh

h
a
n
d
s

1
.4

L
o
g
in

to
t
h
e

e
M

A
R

s
ys

te
m

1
.5

S
ca

n
b

a
rc

o
d
e

fr
o
m

p
a
tie

n
t’s

w
ri
st

b
a
n
d

1
.6

R
e
a
d
e

M
A

R
1
.2

G
re

e
t
th

e
p
a
tie

n
t

1
.3

E
va

lu
a
te

p
a
tie

n
t’s

st
a
tu

s
w

h
ile

p
a
ss

in
g

m
e
d
ic

a
tio

n

(P
la

n
4

:
d

o
4

.1
a

n
d

4
.2

in
o

rd
e

r)

2
.3

.3
G

e
t

m
e
d
ic

a
tio

n
fr

o
m

t
h
e

co
n
tr

o
lle

d
ca

b
in

e
t

2
.3

.3
.2

U
n
lo

ck
t

h
e

co
n
tr

o
lle

d
ca

b
in

e
t

2
.3

.1
.3

O
p
e
n
t
h
e

d
ra

w
e
r

2
.3

.3
.1

G
e
t

ke
ys

f
o
r

th
e
c

o
n
tr

o
lle

d
ca

b
in

e
t(P

la
n
2

.3
.3

:
d
o
2

.3
.3

.1
a

n
d
2

.3
.3

.2
)

2
.3

.4
C

a
ll

su
p
e
rv

is
o
r

a
b
o
u
t

m
is

si
n
g

m
e
d
ic

a
tio

n

Figure 1.
A hierarchical task

analysis (HTA)
diagram for the

medication
administration

process using the
new HIT system

(c
on
ti
nu

ed
)

459

HIT
implementation

in rural
hospitals

2
.3

.5
P

re
p

a
re

m
e

d
ic

a
tio

n

2
.3

.5
.3

P
u
ll

o
u
t

in
tr

a
ve

n
o
u
s

(I
V

)
flu

id
m

e
d
ic

a
tio

n
s

(P
la

n
2

.3
.5

:
d
o
2

.3
.5

.1
o

r/
a
n
d
2

.3
.5

.2
o

r/
a
n
d
2

.3
.5

.3
a

cc
o
rd

in
g
t
o
p

a
tie

n
t’s

e
M

A
R

)

2
.3

.5
.1

.1
T

a
ke

t
h

e
m

e
d

ic
a

tio
n

fr
o

m
t

h
e

co
n

ta
in

e
r

2
.3

.5
.1

.2
S

ca
n

b
a

rc
o

d
e

o
n

t
h

e
d

ru
g

s
p

a
ck

a
g

e

2
.3

.5
.1

.3
V

is
u

a
lly

ch
e

ck
t

h
e

sy
st

e
m

h
a

s
sc

a
n

n
e

d
t

h
e

ri
g

h
t

ite
m

2
.3

.5
.1

.4
C

u
t/

se
p

a
ra

te
/

m
e

a
su

re
u

p
th

e
m

e
d

ic
a

tio
n

s

2
.3

.5
.1

.5
L

o
ck

u
p

t
h

e
co

n
ta

in
e

r/
d

ra
w

e
r/

st
a

tio
n

(P
la

n
2

.3
.5

.1
:
d
o
2

.3
.5

.1
.1

-2
.3

.5
.1

.5
in

o
rd

e
r

)

2
.3

.5
.2

.1
T

a
ke

t
h

e
m

e
d

ic
a

tio
n

fr
o

m
t

h
e

g
la

ss
v

ia
l

2
.3

.5
.2

.2
S

ca
n

b
a

rc
o

d
e

o
n

t
h

e
v

ia
l

2
.3

.5
.2

.3
V

is
u

a
lly

ch
e

ck
t

h
e

sy
st

e
m

h
a

s
sc

a
n

n
e

d
t

h
e

ri
g

h
t

ite
m

2
.3

.5
.2

.4
A

cc
e

ss
n

e
e

d
le

a
n

d
s

yr
in

g
e

s

2
.3

.5
.2

.5
M

e
a

su
re

u
p

th
e

m
e

d
ic

a
tio

n

2
.3

.5
.2

.6
D

ra
w

u
p

so
lu

tio
n

f
ro

m
th

e
v

ia
l

2
.3

.5
.2

.7
F

in
d
a

n
o
th

e
r

n
u

rs
e

t
o

co
n

fir
m

t
h

e
m

e
d
ic

a
tio

n
(t

yp
e

a
n

d
d

o
se

)
fo

r
a

p
a
rt

ic
u
la

r
p
a
tie

n
t

2
.3

.5
.2

.8
A

n
o

th
e

r
n

u
rs

e
si

g
n
in

to
t

h
e

sy
st

e
m

f
o

r
w

itn
e

ss
in

g

2
.3

.5
.2

.1
0

L
o
ck

u
p
t

h
e

co
n
ta

in
e
r/

d
ra

w
e

r/
st

a
tio

n

2
.3

.5
.1

P
u

ll
o

u
t

g
e

n
e

ra
l

m
e

d
ic

a
tio

n
s

(o
ra

l p
ill

,
p

a
tc

h
,

cr
e

a
m

,
o

r
liq

u
id

)

2
.3

.5
.2

P
u
ll

o
u
t

in
je

ct
io

n
m

e
d

ic
a

tio
n

s

2
.3

.5
.3

.2
S

ca
n
b

a
rc

o
d
e

o
n
t

h
e

p
a
ck

a
g
e

2
.3

.5
.3

.3
V

is
u
a
lly

ch
e

ck
t

h
e

sy
st

e
m

h
a

s
sc

a
n
n
e
d
t

h
e

ri
g

h
t

ite
m

2
.3

.5
.3

.4
L
o
ck

u
p
t

h
e

co
n

ta
in

e
r/

d
ra

w
e

r/
st

a
tio

n

(P
la

n
2

.3
.5

.3
:
d
o
2

.3
.5

.3
.1

-2
.3

.5
.3

.4
in

o
rd

e
r)

2
.3

.5
.3

.1

T
a

ke
o

u
t

IV
flu

id
b

a
g

P
la

n
2

.3
.5

.2
:

d
o
2

.3
.5

.2
.1

to
2

.3
.5

.2
.6

Is
it

a
co

n
tr

o
lle

d
m

e
d
ic

a
tio

n
(n

a
rc

o
tic

s
o
r

in
su

lin
)? N
o

Y
e
s

d
o
2

.3
.5

.2
.9

th
e
n
e

xi
t

N
e
e
d
a

w
itn

e
ss

p
ro

ce
d
u
re

in
th

e
f

a
ci

lit
y?

Y
e
s

d
o
2

.3
.5

.2
.7

to
2

.3
.5

.2
.9

th
e
n
e

xi
t

N
o

2
.3

.5
.2

.9
R

e
tu

rn
t

h
e

vi
a

l (
w

ith
re

m
a
in

in
g

so
lu

tio
n

)
to

th
e

s
ta

tio
n

Figure 1.

(c
on
ti
nu

ed
)

460

IJHCQA
29,4

3
.2

P
a

ss
I

V
f

lu
id

m
e

d
ic

a
tio

n
s

3
.3

P
a

ss
in

je
ct

io
n

m
e

d
ic

a
tio

n
s

(P
la

n
3

:
d
o
3

.1
o

r/
a
n
d
3

.2
o

r/
a
n
d
3

.3
o

r/
a
n
d
3

.4
a

cc
o
rd

in
g
t
o
p

a
tie

n
t’s

e
M

A
R

)

3
P

a
ss

m
e

d
ic

a
tio

n
s

3
.1

.1
P

re
p

a
re

w
a

te
r

o
r

ju
ic

e

3
.1

.2
T

e
ll

m
e

d
ic

a
tio

n
n

a
m

e
s

a
n

d
p

u
rp

o
se

s
to

th
e

p
a

tie
n

t

3
.1

.3
P

re
se

n
t

p
ill

s
a

n
d

w
a

te
r

to
th

e
p

a
tie

n
t

3
.1

.4
H

e
lp

/o
b

se
rv

e
p

a
tie

n
t

ta
ki

n
g

m
e

d
ic

a
tio

n
s

3
.1

.5
C

h
e

ck
t

h
a

t
p

a
tie

n
t

h
a

s
a

p
p

lie
d

m
e

d
ic

a
tio

n
s

p
ro

p
e

rl
y

3
.2

.1
T

e
ll

m
e

d
ic

a
tio

n
n

a
m

e
s

a
n

d
p

u
rp

o
se

s
to

th
e

p
a

tie
n

t

3
.2

.2
H

a
n

g
u

p
t

h
e

IV
f

lu
id

b
a

g

3
.2

.3
A

tt
a

ch
t

h
e

tu

b
e

3
.2

.4
A

ss
e

m
b

le
t

h
e

in
fu

si
o

n
p

u
m

p

3
.2

.5
P

ro
g

ra
m

t
h

e
in

fu
si

o
n

p
u

m
p

3
.2

.6
C

h
e

ck
/

m
o

n
ito

r
IV

flo
w

3
.3

.2
P

re
p

a
re

p
a

tie
n

t
fo

r
th

e
in

je
ct

io
n

3
.3

.3
T

e
ll

m
e

d
ic

a
tio

n
n

a
m

e
s

a
n

d
p

u
rp

o
se

s
to

th
e

p
a

tie
n

t

3
.3

.1
C

h
e
ck

re
so

lu
tio

n
in

sy
ri
n

g
e

a
n

d
re

m
o

ve
a

ir
fr

o
m

t
h

e
sy

ri
n

g
e

3
.3

.4
In

je
ct

m
e

d
ic

a
tio

n
in

to
ve

in
/m

u
sc

le
/

su
b

cu
ta

n
e

o
u

sl
y

3
.4

P
a

ss
o

th
e

r
m

e
d

ic
a

tio
n

s
(c

re
a

m
,

p
a

tc
h

,
o

r
liq

u
id

)

3
.4

.1
T

e
ll

m
e

d
ic

a
tio

n
n

a
m

e
s

a
n

d
p

u
rp

o
se

s
to

th
e

p
a

tie
n

t

3
.4

.2
P

re
se

n
t

m
e

d
ic

a
tio

n
s

to
t

h
e

p
a

tie
n

t

3
.4

.3
H

e
lp

/o
b

se
rv

e
p

a
tie

n
t

a
p

p
ly

in
g

m
e

d
ic

a
tio

n
s

3
.4

.4
C

h
e

ck
t

h
a

t
p

a
tie

n
t

h
a

s
a

p
p

lie
d

m
e

d
ic

a
tio

n
s

p
ro

p
e

rl
y

(P
la

n
3

.1
:
d
o
3

.1
.1

-3
.1

.5
in

o
rd

e
r)

(P
la

n
3

.2
:
d
o
3

.2
.1

-3
.2

.6
in

o
rd

e
r)

(P
la

n
3

.3
:
d
o
3

.3
.1

-3
.3

.4
in

o
rd

e
r)

(P
la

n
3

.4
:
d
o
3

.4
.1

-3
.4

.4
in

o
rd

e
r)

3
.1

P
a

ss
o

ra
l p

ill
s

Figure 1.

461

HIT
implementation

in rural
hospitals

checking out medical supplies was installed in the supply room, the nurses did not
use it to check out the supplies (Adverse Factor f). Instead, they brought the supplies
to the patient’s room and checked them out using the barcode scanning system
attached to the medication cart. Yet, the system on the medication cart does not
communicate with the medical supply inventory system. Participants in the focus
groups also indicated that the procedure for checking out medical supplies from the
supply room is confusing.

Another unfamiliar procedure with regard to the new system was also mentioned in
the discussion. When a patient was not ready for their medications, the nurses did not
know how to override the system to return unpassed and “checked out” drugs back into
the storage system (Adverse Factor g). In fact, the participants were not sure that the
system even allows them to put the unpassed drugs back into the medication cart/station.
This issue could cause the unpassed medications to be stored in a risky manner.

3.4 Omitted procedures
Sometimes, nurses omit critical procedures that could impact the patient’s safety.
For example, nurses fail to return any remaining medication immediately after its
preparation (Adverse Factor h). The exposure of the medications could affect the
quality of the drug or result in it being accessed by unauthorized people. Additionally,
the study also observed that nurses would temporarily leave the medication
preparation site to seek tools, supplies, or help. However, some nurses did not lock the

Category Potential adverse factors

Barcode scanning tasks a. Scanning the patient’s identification barcode from the wristband
without checking the patient’s cognitive status (e.g. without greeting
the patient)

b. Possessing extra patient barcodes aside due to a failure to scan old or
smudged barcoded wristbands

Computer interface or
operating issues

c. Text and icons on the computer screen are too small to read
d. The mouse cursor is moving too fast to follow
e. The barcode scanning system failed to provide the appropriate
feedback to identify the correct patient or medication information

Unfamiliar procedures f. Do not know how to check out supplies with the barcode scanning
system

g. Do not know how to override the system to return unpassed drugs
back into the system

Omitted procedures h. Failure to return remaining medication immediately
i. Failure to lock the medication cart/station securely and leaving it alone
with the patient

Logistic issues j. Failure to pass drugs on time (the pharmacy did not restock the
inventory)

k. Cannot locate the correct place to fill inventory
l. Medical supplies are out of stock (the system does not count the
inventory accurately)

Others m. Ambiguity or a lack of work procedures
n. Environmental lighting is insufficient to prepare the medication and

conduct the charting tasks
o. Unnecessary idleness and interruptions while accessing drugs from
the medication station

Table II.
Potential adverse
factors in the
medication
administration
process with a new
HIT system

462

IJHCQA
29,4

medication cart securely before leaving the preparation site (Adverse Factor i). It was
found that semi-prepared medications had been left alone with a patient in the room.

3.5 Logistic problems
Sometimes, drugs could not be administered to patients on time because the pharmacy
did not restock the inventory (Adverse Factor j). In general, the pharmacy would send
the next morning’s medications to the floor before 5:00 p.m. Physicians usually need to
make their last orders before 3:00 p.m. using the CPOE system. However, if the
physician put in the orders after 3:00 p.m., these will be delayed and processed the next
morning at 8:00 a.m. Further, it takes time to verify, prepare, and deliver the
medications from the pharmacy. Thus, some patients could miss their scheduled
medications (usually starting at 9:00 a.m.). It was also common to see nurses call the
pharmacy to track the status of a medication that was being prepared.

Incomplete supplies of medications would be sent to the floor while the medication
administration process was taking place in the morning. When the nurses push the
medication cart from room to room to pass drugs, the pharmacy technician would have
difficulty locating the cart to fill the inventory (Adverse Factor k). This causes another
delay in restocking the medical inventory.

Another logistical issue was caused by the barcode scanning system not working
effectively for accurately counting the inventory of medical supplies. It was observed
that medical supplies in the supply room would be out of stock without any notice
(Adverse Factor l). Thus, the nurses spent a great deal of time accessing and searching
for supplies from the warehouse.

3.6 Other adverse factor
After implementing the new HIT system, the clinical workflow evolved and some new
tasks were revealed. These new tasks usually lacked standard procedures for how to
handle them (Adverse Factor m). Even floor managers and experienced nurses did not
know how to correctly deal with certain tasks related to the new HIT system. An
example of this is that when a patient is not ready for their medication, there is no clear
procedure for telling the nurses how to check the unpassed medications back into the
system (the medication station or cart). Additionally, no policy had been established to
tell nurses when they should recheck a patient’s status to see if they are ready for
drugs. Some nurses mentioned that the new EHR system would pop up with an alarm
to remind them to recheck the patient’s status every ten minutes. Some of them
preferred to recheck the patient after passing drugs to all the other patients.

Furthermore, the new EHR and barcode scanning systems are mounted on the
medication cart. Nurses need to push the cart around the floor while passing
medications. The nurses claimed that they felt eyestrain when charting tasks for long
periods of time on the computer screen (Adverse Factor n). Indeed, the dim light sources
in each patient’s room also impacted the medication preparation tasks (e.g. made it hard
to measure medications precisely).

Interruptions and idleness were other issues that appeared while implementing the
new HIT system in the hospital (Adverse Factor o) (Weigl et al., 2011, 2012). Some drugs
are stored in the medication station in the hallway. The nurses usually needed to spend
some time waiting in line to access the medications from the medication station.
At the same time, the nurses in line would chat with the other nurse who was preparing
the medications. Medication errors could be caused by these interruptions during the
preparation of the medication (Westbrook et al., 2010).

463

HIT
implementation

in rural
hospitals

3.7 Reason’s Swiss cheese model and adverse factors
However, not all potential adverse factors would cause medication errors by
themselves. Based on Reason’s “Swiss Cheese Model” of system failure, one potential
adverse factor represents a hole in a layer of Swiss cheese and systematic errors are
usually caused by the aligned holes throughout all of the Swiss cheese slices (Reason,
1990, 2000). For example, for Adverse Factor i, the failure to securely lock the
medication cart/station and leaving it alone with the patient does not indicate that
Adverse Factor i must necessarily lead to any obvious error. The error is only apparent
when the patient takes the exposed drugs without the care provider’s verification.

Furthermore, the root causes of the adverse factors should be analyzed in depth. It has
been realized that some adverse factors have a cause-and-effect relationship with each
other. For instance, the omitted procedure, Adverse Factor i (failure to lock the medication
cart securely and leaving it alone with the patient) could be caused by the facility lacking
clear standard procedures (Adverse Factor m) for securely storing the drugs. It could also
be caused by unnecessary interruptions and in turn leads to the nurse leaving the
medication preparation site without securely storing the drugs (Adverse Factor o).

Thus, the potential adverse factors related to using the HIT during medication
administration processes cannot simply be considered individually. The root causes of
some factors should be explored in depth and be related to a comprehensive
investigation in other healthcare scenarios and settings.

4. Discussion
This study is based on a series of detailed task analyses and focus group interviews
that were used to demonstrate violations of work standards during the medication
administration process while working with a new HIT system in a rural hospital. The
results show that the potential adverse factors are interconnected and revealed the
complexity of the clinical interactions among the medical staff, patients, technology,
facility logistics, and the workflow. These findings suggest that the assessment of
clinical tasks when using a new HIT system is necessary for adapting to the evolving
workflow. These continuous process improvements to the clinical workflow should be
introduced before, during, and after HIT system implementation (Carayon et al., 2014).

4.1 Potential adverse factor found in other healthcare settings
Previous studies have reported rich findings with regarding to potential adverse
factors that are related to using a new HIT system in other healthcare settings (e.g.
teaching hospitals clinics, medical centers, etc.). Table III summarizes nine potential
adverse factors (Adverse Factors p-x) related to HIT systems that were not discussed in
this study so as to address potential hazards that could impact patient safety.

Previous studies (Table III) showed that medical staff could suffer from some
unexpected IT issues when using the new HIT (Adverse Factors p-r), such as lost
connectivity with the network and printers, or with troubleshooting hardware
problems (Carayon et al., 2007; Koppel et al., 2008; Patterson et al., 2004). Global system
issues were also mentioned in several previous studies. Those issues usually could not
be solved by a single staff member, but would need coordination among departments.
For example, not all barcoded drugs could be registered in the medication
administration system in a timely manner (Adverse Factor s). Thus, the nurses and the
pharmacy would need to effectively communicate regarding how to correctly document
uncategorized drugs in the system (Bramble et al., 2013; Linsky and Simon, 2013).

464

IJHCQA
29,4

Similar issues would arise when the system did not recognize when a partial dose
of a drug was administered (Adverse Factor t) (Grigg et al., 2011). The drug dose
information in the inventory system would then be inconsistent with the physician’s
order (Ranji et al., 2014). Even when the pharmacists, nurses, and physicians agreed to
pass a partial dose of a drug to a patient, there would be difficulties in properly
documenting the drug administration history into the system.

Some potential adverse factors related to the HIT could be categorized as training
issues. These problems can be dealt with by proper training and practice. For example,
proficient typing skills would decrease the amount of time spent on computer entry
(Adverse Factor u) (Koppel et al., 2008; Unertl et al., 2009; Carayon et al., 2009).
In addition, users who are familiar with HIT system operations and the interface would

Category Potential adverse factors in using HIT Studied settings

Troubleshooting
skills

a. Staff experienced some IT technical problems
(e.g. lost of network or printer connectivity)

Two teaching hospitals (Koppel
et al., 2008) (470 and 929 beds)

b. Unexpected hardware
downtime

Simulations in a laboratory
(Patterson et al., 2004)

c. Computer screen needed alignment or had
inadequate contrast

One teaching hospital (Carayon
et al., 2007) (472 beds)

Global system
issues

d. Medication identifying numbers (barcode
numbers) had not yet been categorized in
the system. Nurses needed to override the
records manually when a particular drug
was administered

Two teaching hospitals (Koppel
et al., 2008) (470 and 929 beds)
Simulations in a laboratory
(Patterson et al., 2004)
Pediatric Oncology Department in
an academic medical center (Kim
et al., 2006)

e. When a partial dose of drug was
administered, the system did not recognize
this. Instead, the system recorded a
complete dose of the drug

Two teaching hospitals (Koppel
et al., 2008) (470 and 929 beds)

Ability
improvement
issues

f. Dramatic increase in the amount of time
spent on computer entry. This may slow the
medication administration process during
an emergency

Two teaching hospitals (Koppel
et al., 2008) (470 and 929 beds)
Fifteen chronic disease care clinics
(Unertl et al., 2009)
One family medicine residency
clinic (Carayon et al., 2009)

g. Failure to document drugs or patient
information correctly

One teaching hospital (Carayon
et al., 2007) (472 beds)

h. Medical staff did not know how to retrieve
the appropriate information when facing
certain scenarios (e.g. how to respond to an
allergy notification)

Two teaching hospitals (Koppel
et al., 2008) (470 and 929 beds)
Simulations in a laboratory
(Patterson et al., 2004)
Three chronic clinics (Unertl et al.,
2009)
One teaching hospital (Horsky et al.,
2005)

Other i. New HIT impacted the patient- provider
relationship because of fewer interactions
with the patient or a difficulty focussing on
patient communication

One academic medical center
(Makoul et al., 2001)
Four primary care offices (Ventres
et al., 2006)
One family clinics (Asan et al., 2014)

Table III.
Potential adverse

factors in using HIT
in different

healthcare settings

465

HIT
implementation

in rural
hospitals

make fewer mistakes when performing clinical documentation or retrieving
information (Adverse Factors v and w) (Koppel et al., 2008; Patterson et al., 2004;
Carayon et al., 2007, 2009; Unertl et al., 2009; Horsky et al., 2005).

It was also mentioned that the new HIT would impact the patient-provider
relationship because of fewer face-to-face communications and less eye contact between
the patient and their providers (Adverse Factor x) (Makoul et al., 2001; Ventres et al.,
2006; Asan et al., 2014). Nurses would be too focussed on a computer screen, and as a
result they would neglect to interact with patients while providing care.

4.2 Work system model
To consolidate potential adverse factors that occur while working with the new
HIT, the work system model was used to demonstrate the taxonomy of HIT
implementation recommendations. The work system model was developed in 1989, and
it is defined as individuals who usually perform tasks using certain tools or
technologies in their physical working environments under organizational conditions
(Carayon et al., 2006, 2007, 2009; Carayon and Smith, 2000; Smith and Carayon-Sainfort,
1989). Figure 2 shows the interrelationship among five elements in the work system
model. In the current study, this work system model is adopted to further provide
recommendations for rural hospitals when implementing a HIT system, as described in
the following section.

4.3 Recommendations for rural hospitals
In total, 15 recommendations that are divided into five categories are put forward,
based on: the study results regarding to the failures to meet work standards (Adverse
Factors a-o in Table II) in using HIT in a rural hospital; and potential adverse factors in
using HIT found in other healthcare settings (Adverse Factors p-x in Table III).
The recommendations for rural hospitals for HIT implementation are summarized in
Table IV, and each recommendation is matched with the corresponding adverse
factors. Most of the recommendations are related to each other because of the
complexity of the workflow in the healthcare system. Thus, some recommendations
should embed within another one, especially from an organizational aspect. In this
section is a discussion of some major common issues in rural healthcare with 15
recommendations, which include topics relating to communication, interruptions,
computer literacy, facility configurations, etc.

Technology Organization

Tasks Environment

People

Source: Carayon et al. (2006)

Figure 2.
The framework
of the work
system model

466

IJHCQA
29,4

4.3.1 Communication and interruptions. Due to the implementation of the new HIT,
some essential communication between patients and care providers could be neglected.
For example, the current barcode scanning system supports part of the medication
administration tasks by verifying a patient’s identification and drugs. The system
ensures safety during the process while minimizing conversation between patients and
their care providers. However, oral verification of drug information with the patient not
only helps to establish a positive patient-provider relationship, but is also an important
procedure for assessing the patient’s cognitive status through the greeting and other
basic communicative interaction with patients. Therefore, Recommendation 1 suggests
that the process of communicating with patients regarding medication information
should be a non-neglected verification step for medication safety.

On the other hand, some communications could potentially cause interruptions in a
healthcare setting. For example, unnecessary synchronous communication (e.g. face-to-face

Category Recommendations
Corresponding
adverse factors

People 1. Greeting the patient and giving elaborate drug information
to the patient not only establishes a positive patient-
provider relationship, but can also assess the patient’s
physical and cognitive status at the same time

(a), (x)

2. Interruptions should be avoided during the preparation of
medication

(o)

Tasks 3. Lock/store medications securely at all times (i)
4. Confirm the patient and medication information before
passing drugs; do not just rely on the feedback from the HIT
system alone

(a), (e), (x)

5. Return tools and remaining medications to fixed locations
immediately after finishing the task

(h), (i), (o)

Tools 6. Change barcoded wristbands regularly to prevent invalid
scanning

(b)

7. Provide staff with a computer operation troubleshooting
manual (e.g. how to connect peripherals, adjust the
computer screen, or setup the mouse sensitivity)

(c), (d), (f), (p), (r)

8. Exclude/distinguish confusing signals from the HIT system;
information from the HIT system should be received so as to
prevent errors

(e)

Environment 9. Provide sufficient lighting for preparing drugs or using the
computer

(n), (v)

10. A quiet and private workspace is preferred so that
medication preparation and documentation tasks will not be
interrupted

(o)

Organization 11. Procedures and sequences of tasks should be established/
regulated clearly using proper standards

(a), (f), (k),( l), (m), (o),
(s)

12. Enforce safety procedures regularly within the facility (a), (h), (i), (m)
13. Provide IT training and maintain the hardware regularly (c), (d), (f), (g), (l), (p),

(q), (v), (u), (w)
14. Communication among the staff, managers, IT support, and

system vendors should be freely available so as to solve
global system issues

(e), (f), (g), (j), (s), (t), (u)

15. Collaborate with an outside 24 hour pharmacy to avoid
logistical gaps

(j), (k)

Table IV.
Recommendations

for HIT
implementation

467

HIT
implementation

in rural
hospitals

and phone call) among care providers could result in interruptions during drug
preparations and potentially lead to certain subsequent medication errors. It has been
suggested that introducing some asynchronous communication modalities can help to
reduce interruption in healthcare settings (Recommendation 2) (Huang and Garrett,
2012; Fairbanks et al., 2007). For instance, Bardram and Hansen (2004) suggested
an asynchronous communication-based context-mediated social awareness model (e.g.
send text message via the Awarephone), which allows providers to be aware of the
current situation and avoid interrupting other staff. The context-mediated social
awareness model mechanism (2004) supports decisions about when and how to
appropriately contact staff and is aimed at minimizing interruptions to staff who are
engaging in high-risk clinical tasks. However, entirely eliminating all of the
interruptions among care providers is impossible due to of the fact that some
interruptions in a healthcare setting could actually advance quality of care and achieve
safer healthcare (Rivera-Rodriguez and Karsh, 2010) when proper communication
modalities are adopted.

4.3.2 Computer literacy in rural healthcare settings. The shortage of health
professionals in rural areas is critical, and the situation will become worse in the next 20
years due to aging care providers reaching retirement age (National Rural Health
Association, 2012). The American Organization of Nurse Executives (2010) reported
that nearly 40 percent of nurses in rural healthcare facilities are at least 50 years of age.
Those aging care providers also tend to have inadequate computer literacy compared
to their younger peers (Huang et al., 2012). Providing computer and IT training to care
providers is one of the solutions that is regularly suggested so as to improve computer
literacy, but this training could increase the financial burden, especially in resource-
limited rural hospitals (Recommendation 13). Additionally, due to the shortage of
health workforces in rural areas, supplemental part-time care providers are generally
employed across different facilities. This alone increases the difficulties for part-time
employees to become familiar with diverse HITs across multiple hospitals. Thus,
providing computer operation manuals for basic HIT troubleshooting would be a
feasible solution for those without adequate computer skills (Recommendation 7).

4.3.3 HIT design and IT support. Pursuing a transparent HIT system is the most
realistic and practical solution to overcome the usability problems or other computer
troubleshooting issues for care providers. However, rural healthcare facilities with
limited resources usually are unable to acquire an optimal HIT system due to logistical,
financial, and operational constraints. Therefore, it was observed that some HITs were
not being utilized well with respect to current clinical practices in rural hospitals
because the system was originally designed for a large-scale healthcare system. The
HIT being used in rural healthcare facilities could not be easily reconciled human/
computer interaction design or to the necessary computer literacy. The HIT design for
rural healthcare should be systematically reviewed to ensure that a detailed workflow
can achieve quality of care. For example, it had been observed that not all HIT systems
provide the authority or functions to allow staff to override any unexpected data, such
as documenting partial doses of drug in the eMAR. Thus, Recommendation 14
suggests that communication among the staff, IT support, and HIT vendors is essential
to maintain the flexibility of clinical documentation via an appropriately customized
functional HIT system. Gathering feedback from users about the system operation
would be another excellent way to understand how to better design a workflow-
orientated HIT to support clinical tasks (Huang et al., 2012). In addition, IT support

468

IJHCQA
29,4

should be provided in a timely fashion within the facility so as to maintain a seamless
clinical workflow.

4.3.4 Organizational regulated procedures, safety policies, and resilient work.
Recommendations 11 and 12 suggest that establishing clear work procedures and
enforcing safety regulations are the highest priorities for maintaining patient safety. In
practice, care providers constantly assess situations and locate potential resources to
prioritize their tasks when accommodating resilient clinical works in a dynamic
healthcare work system (Nemeth et al., 2008; Jeffcott et al., 2009). Thus, implementing a
series of rigidly regulated procedures and a safety culture within the facility need not
imply that providers are failing to develop resilient strategies when dealing with
unexpected and complex disturbances. By appealing to the organizational regulation
and safety guidelines, it not only gives care providers a strong groundwork when
practicing resilient clinical work, but can also improve situational awareness and
responses in high-priority situations. For example, an unexpected event interrupts
medication preparation, and the care provider must terminate the current task to
provide assistance to others. In this scenario, the first response should not be to leave
the current task to accommodate the immediate need. Instead, the care provider should
store the drugs securely before taking any action. Therefore, with a resilient clinical
practice, it not only reduces on-going temporary disturbances, but can also prevent any
chaos from arising in the near future as well (Woods, 2006).

With a well-ground safety culture and regulated policies in place in healthcare
facilities, resilience engineering concepts in the healthcare system would provide better
performance for understanding situations occurring at any particular moment
selecting correct responses, anticipating accurate projecting consequences, and gaining
insights from concurrent medical practices (Fairbanks et al., 2014). Clear, simple, and
high-level organizational policies would help to develop a continuous process
improvement culture in a facility. The care delivery team should review the clinical
practices regularly to ensure that the components in the work system well support the
tasks in an effective, efficient, and safe manner.

4.4 Study limitations
In the observational study, the participants’ practices could also be indirectly affected
due to the researchers were shadowing nurses while performing the clinical activities; it
is also known as the Hawthorne effect (Campbell et al., 1995). In this study, some
strategies were applied to reduce the impact of the Hawthorne effect on the study
results. For example, all the participants had been informed in advance that the
study results will not be associated with any format of performance evaluations.
Second, the research team did not collect any demographic information about the
participants from either the observational study or the focus group. Thus, the results
are only presented the systematic workflow information without recording
participants’ age, gender, education, experience data, etc.

5. Conclusions
Rural hospitals usually have fewer resources for obtaining an ideal HIT system that
operates flawlessly within their preferred workflow (American Hospital Association,
2011; McCullough et al., 2011). Thus, it is important to introduce the concept of
continuous process improvement into rural hospitals so as to clearly define a
“context-appropriate” workflow and develop proper accommodations regarding the

469

HIT
implementation

in rural
hospitals

evolving clinical practice caused by implementing a new HIT (Huang and
Gramopadhye, 2014). This study provides a roadmap for investigating potential
adverse factors while using a new HIT in a rural hospital. In total, 15
recommendations for implementing HIT are provided to help facilities make the
transition to a new HIT system. These recommendations can also be guidelines for
any other facilities which are adopting a new HIT. The study methodology can also
be applied to other clinical practices to investigate violations, deviations, or adverse
factors affecting a clinical work standard.

References

Abramson, E.L., Kern, L.M., Brenner, S., Hufstader, M., Patel, V. and Kaushal, R. (2014), “Expert
panel evaluation of health information technology effects on adverse events”, Journal of
Evaluation in Clinical Practice, Vol. 20 No. 4, pp. 375-382.

Abramson, E.L., McGinnis, S., Edwards, A., Maniccia, D.M., Moore, J. and Kaushal, R. (2012),
“Electronic health record adoption and health information exchange among
hospitals in New York State”, Journal of Evaluation in Clinical Practice, Vol. 18 No. 6,
pp. 1156-1162.

Agrawal, A. (2009), “Medication errors: prevention using information technology systems”,
British Journal of Clinical Pharmacology, Vol. 67 No. 6, pp. 681-686.

Agrawal, A. and Glasser, A.R. (2009), “Barcode medication. Administration implementation in an
acute care hospital and lessons learned”, Journal of Healthcare Information Management,
Vol. 23 No. 4, pp. 24-29.

American Hospital Association (2011), “The opportunities and challenges for rural hospitals in an
era of health reform”, American Hospital Association, Chicago, IL, available at: www.
raconline.org/publications/documents/7785 (accessed April 2014).

American Organization of Nurse Executives (2010), “AONE guiding principles for the aging
workforce”, available at: www.aone.org/resources/PDFs/AONE_GP_Aging_Workforce.pdf
(acccessed June 2015).

Asan, O., Smith, P.D. and Montague, E. (2014), “More screen time, less face time – implications for
EHR design”, Journal of Evaluation in Clinical Practice, Vol. 20 No. 6, pp. 896-901.

Aspden, P., Wolcott, J., Bootman, J.L. and Cronenwett, L.R. (2007), Preventing Medication Errors,
National Academies Press, Washington, DC.

Bahensky, J.A., Jaana, M. and Ward, M.M. (2008), “Health care information technology in rural
America: electronic medical record adoption status in meeting the national agenda”,
The Journal of Rural Health, Vol. 24 No. 2, pp. 101-105.

Bardram, J.E. and Hansen, T.R. (2004), “The AWARE architecture: supporting context-mediated
social awareness in mobile cooperation”, Proceedings of the 2004 ACM Conference on
Computer Supported Cooperative Work, pp. 192-201.

Bates, D.W., Leape, L.L., Cullen, D.J., Laird, N., Petersen, L.A., Teich, J.M., Burdick, E., Hickey, M.,
Kleefield, S. and Shea, B. (1998), “Effect of computerized physician order entry and a team
intervention on prevention of serious medication errors”, The Journal of the American
Medical Association, Vol. 280 No. 15, pp. 1311-1316.

Berg, M., Langenberg, C., vd Berg, I. and Kwakkernaat, J. (1998), “Considerations for
sociotechnical design: experiences with an electronic patient record in a clinical context”,
International Journal of Medical Informatics, Vol. 52 No. 1, pp. 243-251.

Blumenthal, D. (2010), “Launching HITECH”, New England Journal of Medicine, Vol. 362 No. 5,
pp. 382-385.

470

IJHCQA
29,4

Boockvar, K.S., Livote, E.E., Goldstein, N., Nebeker, J.R., Siu, A. and Fried, T. (2010), “Electronic
health records and adverse drug events after patient transfer”, Quality and Safety in Health
Care, Vol. 19 No. 5, pp. 1-5.

Bramble, J.D., Abbott, A.A., Fuji, K.T., Paschal, K.A., Siracuse, M.V. and Galt, K. (2013),
“Patient safety perspectives of providers and nurses: the experience of a rural ambulatory
care practice using an EHR with E‐prescribing”, The Journal of Rural Health, Vol. 29 No. 4,
pp. 383-391.

Campbell, J.P., Maxey, V.A. and Watson, W.A. (1995), “Hawthorne effect: implications for
prehospital research”, Annals of Emergency Medicine, Vol. 26 No. 5, pp. 590-594.

Carayon, P. and Smith, M.J. (2000), “Work organization and ergonomics”, Applied Ergonomics,
Vol. 31 No. 6, pp. 649-662.

Carayon, P., Smith, P., Hundt, A.S., Kuruchittham, V. and Li, Q. (2009), “Implementation of an
electronic health records system in a small clinic: the viewpoint of clinic staff”, Behaviour &
Information Technology, Vol. 28 No. 1, pp. 5-20.

Carayon, P., Schoofs Hundt, A., Karsh, B.T., Gurses, A.P., Alvarado, C.J., Smith, M. and
Flatley Brennan, P. (2006), “Work system design for patient safety: the SEIPS model”,
Quality & Safety in Health Care, Vol. 15 No. 1, pp. i50-i58.

Carayon, P., Wetterneck, T.B., Hundt, A.S., Ozkaynak, M., DeSilvey, J., Ludwig, B., Ram, P. and
Rough, S.S. (2007), “Evaluation of nurse interaction with bar code medication administration
technology in the work environment”, Journal of Patient Safety, Vol. 3 No. 1, pp. 34-42.

Carayon, P., Wetterneck, T.B., Cartmill, R., Blosky, M.A., Brown, R., Kim, R., Kukreja, S., Johnson, M.,
Paris, B. and Wood, K.E. (2014), “Characterising the complexity of medication safety using
a human factors approach: an observational study in two intensive care units”,
BMJ Quality & Safety, Vol. 23 No. 1, pp. 56-65.

Casey, M.M., Moscovice, I. and McCullough, J. (2013), “Rural primary care practices and
meaningful use of electronic health records: the role of regional extension centers”,
The Journal of Rural Health, Vol. 30 No. 3, pp. 244-251.

Drury, C.G. (2008), “Procedures and technical documentation”, in FAA (Ed.), Human Factors
Guide for Aviation Maintenance and Inspection, FAA, Washington, DC, available at: www.
hf.faa.gov/hfguide/index.html (accessed December 2014).

El-Kareh, R., Hasan, O. and Schiff, G.D. (2013), “Use of health information technology to reduce
diagnostic errors”, BMJ Quality & Safety, Vol. 22 No. 2, pp. ii40-ii51.

Fairbanks, R.J., Bisantz, A.M. and Sunm, M. (2007), “Emergency department communication links
and patterns”, Annals of Emergency Medicine, Vol. 50 No. 4, pp. 396-406.

Fairbanks, R.J., Wears, R.L., Woods, D.D., Hollnagel, E., Plsek, P. and Cook, R.I. (2014), “Resilience
and resilience engineering in health care”, The Joint Commission Journal on Quality and
Patient Safety, Vol. 40 No. 8, pp. 376-383.

Fraind, D.B., Slagle, J.M., Tubbesing, V.A., Hughes, S.A. and Weinger, M.B. (2002), “Reengineering
intravenous drug and fluid administration processes in the operating room: step one: task
analysis of existing processes”, Anesthesiology, Vol. 97 No. 1, pp. 139-147.

Garets, D. and Davis, M. (2006), “Electronic medical records vs electronic health records: yes,
there is a difference”, HIMSS Analytics, Chicago, IL, available at: http://s3.amazonaws.com/
rdcms-himss/files/production/public/HIMSSorg/Content/files/WP_EMR_EHR.pdf

Goldschmidt, P.G. (2005), “HIT and MIS: implications of health information technology and
medical information systems”, Communications of the ACM, Vol. 48 No. 10, pp. 68-74.

Grigg, S.J., Garrett, S.K. and Craig, J.B. (2011), “A process centered analysis of medication
administration: identifying current methods and potential for improvement”, International
Journal of Industrial Ergonomics, Vol. 41 No. 4, pp. 380-388.

471

HIT
implementation

in rural
hospitals

Hersh, W. and Wright, A. (2008), “What workforce is needed to implement the health information
technology agenda? Analysis from the HIMSS analytics™ database”, American Medical
Informatics Association 2008 Annual Symposium Proceedings: American Medical
Informatics Association, pp. 303-307.

HIMSS Analytics (2014a), “EMR adoption model score, 1st Quarter 2014”, HIMSS Analytics”,
Chicago, IL, available at: www.himssanalytics.org/emram/structure.aspx (acccessed March).

HIMSS Analytics (2014b), “Structure and stage detail of EMR adoption model”, HIMSS Analytics,
Chicago, IL, available at: www.himssanalytics.org/emram/structure.aspx (accessed March).

Horsky, J., Kuperman, G.J. and Patel, V.L. (2005), “Comprehensive analysis of a medication dosing
error related to CPOE”, Journal of the American Medical Informatics Association, Vol. 12
No. 4, pp. 377-382.

Huang, Y.H. and Garrett, S.K. (2012), “Defining characteristics of communication quality in
culture-changed long-term healthcare facilities”, Journal of Communication in Healthcare,
Vol. 5 No. 4, pp. 227-238.

Huang, Y.H. and Gramopadhye, A.K. (2014), “Systematic engineering tools for describing and
improving medication administration processes at rural healthcare facilities”, Applied
Ergonomics, Vol. 45 No. 6, pp. 1712-1724.

Huang, Y.H., Garrett, K.S., Taaffe, M.K. and Gramopadhye, K.A. (2012), “Are staff in rural
healthcare facilities ready for EHRs?”, Proceedings of the 2012 Industrial and Systems
Engineering Research Conference, Orlando, FL, June 30.

Jaana, M., Ward, M.M. and Bahensky, J.A. (2012), “EMRs and clinical IS implementation in
hospitals: a statewide survey”, The Journal of Rural Health, Vol. 28 No. 1, pp. 34-43.

Jeffcott, S., Ibrahim, J. and Cameron, P. (2009), “Resilience in healthcare and clinical handover”,
Quality and Safety in Health Care, Vol. 18 No. 4, pp. 256-260.

Jha, A.K. (2010), “Meaningful use of electronic health records the road ahead”, Journal of the
American Medical Association, Vol. 304 No. 15, pp. 1709-1710.

Khoury, A. (1997), “Finding value in EMRs (electronic medical records)”, Health Management
Technology, Vol. 18 No. 8, pp. 34-36.

Kim, G.R., Chen, A.R., Arceci, R.J., Mitchell, S.H., Kokoszka, K.M., Daniel, D. and Lehmann, C.U.
(2006), “Error reduction in pediatric chemotherapy: computerized order entry and failure
modes and effects analysis”, Archives of Pediatrics & Adolescent Medicine, Vol. 160 No. 5,
pp. 495-498.

Koppel, R., Wetterneck, T., Telles, J.L. and Karsh, B.T. (2008), “Workarounds to barcode
medication administration systems: their occurrences, causes, and threats to patient
safety”, Journal of the American Medical Informatics Association, Vol. 15 No. 4, pp. 408-423.

Krall, M. (1995), “Acceptance and performance by clinicians using an ambulatory electronic
medical record in an HMO”, Proceedings of the Annual Symposium on Computer
Application in Medical Care: American Medical Informatics Association, pp. 708-711.

Lane, R., Stanton, N.A. and Harrison, D. (2006), “Applying hierarchical task analysis to
medication administration errors”, Applied Ergonomics, Vol. 37 No. 5, pp. 669-679.

Linsky, A. and Simon, S.R. (2013), “Medication discrepancies in integrated electronic health
records”, BMJ Quality & Safety, Vol. 22 No. 2, pp. 103-109.

Lisby, M., Nielsen, L.P. and Mainz, J. (2005), “Errors in the medication process: frequency, type,
and potential clinical consequences”, International Journal for Quality in Health Care,
Vol. 17 No. 1, pp. 15-22.

McCartney, P.R. (2006), “Using technology to promote perinatal patient safety”, Journal of
Obstetric, Gynecologic, & Neonatal Nursing, Vol. 35 No. 3, pp. 424-431.

472

IJHCQA
29,4

McCartney, P.R. (2011), “Meaningful use and certified electronic health records”, MCN-The
American Journal of Maternal-Child Nursing, Vol. 36 No. 2, pp. 137-137.

McCullough, J., Casey, M., Moscovice, I. and Burlew, M. (2011), “Meaningful use of health
information technology by rural hospitals”, The Journal of Rural Health, Vol. 27 No. 3,
pp. 329-337.

Makoul, G., Curry, R.H. and Tang, P.C. (2001), “The use of electronic medical records:
communication patterns in outpatient encounters”, Journal of the American Medical
Informatics Association, Vol. 8 No. 6, pp. 610-615.

National Rural Health Association (2012), “Health care workforce distribution and shortage issues
in rural America”, available at: www.ruralhealthweb.org/go/left/policy-and-advocacy/policy-
documents-and-statements/official-nrha-policy-positions (accessed June 2015).

National Rural Health Association (2015), “What’s different about rural health care?”, available at:
www.ruralhealthweb.org/go/left/about-rural-health (accessed December).

Nemeth, C., Wears, R., Woods, D., Hollnagel, E. and Cook, R. (2008), “Minding the gaps: creating
resilience in healthcare”, Advances in Patient Safety: New Directions and Alternative
Approaches, Vol. 3, pp. 1-13.

Patterson, E.S. (2012), “Technology support of the handover: promoting observability, flexibility
and efficiency”, BMJ Quality & Safety, Vol. 21 No. 1, pp. i19-i21.

Patterson, E.S., Rogers, M.L. and Render, M.L. (2004), “Fifteen best practice recommendations for
bar-code medication administration in the Veterans Health Administration”, Joint
Commission Journal on Quality and Patient Safety, Vol. 30 No. 7, pp. 355-365.

Poon, E.G., Keohane, C.A., Featherstone, E., Hays, B.S., Dervan, A., Woolf, S., Hayes, J., Bane, A.,
Newmark, L.P. and Gandhi, T.K. (2006), “Impact of barcode medication administration
technology on how nurses spend time on clinical care”, AMIA Annual Symposium
Proceedings, p. 1065.

Poon, E.G., Keohane, C.A., Yoon, C.S., Ditmore, M., Bane, A., Levtzion-Korach, O., Moniz, T.,
Rothschild, J.M., Kachalia, A.B. and Hayes, J. (2010), “Effect of bar-code technology on the
safety of medication administration”, New England Journal of Medicine, Vol. 362 No. 18,
pp. 1698-1707.

Ranji, S.R., Rennke, S. and Wachter, R.M. (2014), “Computerised provider order entry combined
with clinical decision support systems to improve medication safety: a narrative review”,
BMJ Quality & Safety, Vol. 23, pp. 773-780.

Reason, J. (1990), Human Error, Cambridge University Press, Cambridge.

Reason, J. (2000), “Human error: models and management”, British Medical Journal, Vol. 320
No. 7237, pp. 768-770.

Rivera-Rodriguez, A. and Karsh, B.T. (2010), “Interruptions and distractions in healthcare:
review and reappraisal”, Quality and Safety in Health Care, Vol. 19 No. 4,
pp. 304-312.

Sittig, D.F. and Singh, H. (2010), “A new sociotechnical model for studying health information
technology in complex adaptive healthcare systems”, Quality and Safety in Health Care,
Vol. 19 No. 3, pp. i68-i74.

Smith, M.J. and Carayon-Sainfort, P.C. (1989), “A balance theory of job design for stress
reduction”, International Journal of Industrial Ergonomics, Vol. 4 No. 1, pp. 67-79.

Unertl, K.M., Weinger, M.B., Johnson, K.B. and Lorenzi, N.M. (2009), “Describing and modeling
workflow and information flow in chronic disease care”, Journal of the American Medical
Informatics Association, Vol. 16 No. 6, pp. 826-836.

473

HIT
implementation

in rural
hospitals

Ventres, W., Kooienga, S., Vuckovic, N., Marlin, R., Nygren, P. and Stewart, V. (2006), “Physicians,
patients, and the electronic health record: an ethnographic analysis”, The Annals of Family
Medicine, Vol. 4 No. 2, pp. 124-131.

Wakefield, D.S., Ward, M.M., Loes, J.L. and O”Brien, J. (2010), “A network collaboration
implementing technology to improve medication dispensing and administration in critical
access hospitals”, Journal of the American Medical Informatics Association, Vol. 17 No. 5,
pp. 584-587.

Weigl, M., Müller, A., Vincent, C., Angerer, P. and Sevdalis, N. (2012), “The association of
workflow interruptions and hospital doctors’ workload: a prospective observational
study”, BMJ Quality & Safety, Vol. 21 No. 5, pp. 399-407.

Weigl, M., Müller, A., Zupanc, A., Glaser, J. and Angerer, P. (2011), “Hospital doctors’ workflow
interruptions and activities: an observation study”, BMJ Quality & Safety, Vol. 20 No. 6,
pp. 491-497.

Westbrook, J.I., Woods, A., Rob, M.I., Dunsmuir, W.T. and Day, R.O. (2010), “Association of
interruptions with an increased risk and severity of medication administration errors”,
Archives of Internal Medicine, Vol. 170 No. 8, pp. 683-690.

Wetterneck, T.B., Lapin, J.A., Krueger, D.J., Holman, G.T., Beasley, J.W. and Karsh, B.-T. (2012),
“Development of a primary care physician task list to evaluate clinic visit workflow”,
BMJ Quality & Safety, Vol. 21 No. 1, pp. 47-53.

Woods, D.D. (2006), “Essential characteristics of residence”, in Hollnagel, E., Woods, D.D. and
Leveson, N. (Eds), Resilience Engineering: Concepts and Precepts, Ashgate, Aldershot.

Yeager, V.A., Menachemi, N. and Brooks, R.G. (2010), “EHR adoption among doctors who treat
the elderly”, Journal of Evaluation in Clinical Practice, Vol. 16 No. 6, pp. 1103-1107.

For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: [email protected]

474

IJHCQA
29,4

Implementation of an electronic health records system in a small clinic: the viewpoint of clinic staff

Pascale Carayon
a,b
*, Paul Smith

c,d
, Ann Schoofs Hundt

a
, Vipat Kuruchittham

e
and Qian Li

f

a
Center for Quality and Productivity Improvement, University of Wisconsin-Madison, Madison, WI, USA;

b
Department of

Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI, USA;
c
Department of Family Medicine,

University of Wisconsin Medical School, WI, USA;
d
University of Wisconsin Medical Foundation, WI, USA;

e
College of Public

Health, Chulalongkorn University, Bangkok, Thailand;
f
Center for Quality and Productivity Improvement, University of Wisconsin-
Madison, WI, USA

(Received November 2005; final version received August 2007)

In this study, we examined the implementation of an electronic health records (EHR) system in a small family
practice clinic. We used three data collection instruments to evaluate user experience, work pattern changes, and
organisational changes related to the implementation and use of the EHR system: (1) an EHR user survey, (2)
interviews with key personnel involved in the EHR implementation project, and (3) a work analysis of clinic staff. A
longitudinal design with two data-collection rounds was employed: data were collected prior to EHR
implementation and after EHR implementation. Both quantitative and qualitative data were collected and
analysed. Employees of the small clinic perceived few changes in their work after the implementation of the EHR
system, except for increased dependency on computers and a small increase in perceived workload. The work
analysis showed a dramatic increase in the amount of time spent on computers by the various job categories. The
EHR implementation did not change the amount of time spent by physicians with patients. On the other hand, the
work of clinical and office staff changed significantly, and included decreases in time spent distributing charts,
transcription and other clerical tasks. The interviews provided important contextual information regarding EHR
implementation, and showed some positive elements (e.g., planning of training), but also some negative elements
(e.g., unclear structure of the project) that would have deserved additional attention.

Keywords: technology implementation; healthcare; electronic health records system (EHR)

1. Introduction

The importance of implementing and using health
information technology (HIT) to improve the delivery
of health care has been increasingly recognised
(Institute of Medicine 2000, 2001, Thompson and
Brailer 2004, Ash and Bates 2005, Berner et al. 2005,
Middleton et al. 2005). The Institute of Medicine
(2001) highlighted the central role of HIT in the
redesign of the health care system: ‘‘Automation of
clinical, financial, and administrative transactions
(through information technology) is essential to
improving quality, preventing errors, enhancing con-
sumer confidence in the health system, and improving
efficiency’’ (p. 16). In the United States, federal and
regional efforts are under way to accelerate the
adoption and use of electronic health records as a
means of facilitating clinical data sharing, protect
health information privacy and security, and quickly
identify emerging public health threats (Thompson and
Brailer 2004, Overhage et al. 2005).

Driven by the needs to facilitate clinical and
administrative processes, to reduce medical errors,

and to reduce healthcare costs, many healthcare
institutions are deciding to implement electronic health
records (EHR) systems to allow clinical information
gathering and access at the point of patient care. An
EHR system can access progress notes or procedures
data, and may support other functions such as CPOE
(computerised provider order entry) and CDSS (clin-
ical decision support systems). Tools to support
administrative procedures, such as billing and schedul-
ing, are also becoming common EHR features. The use
of EHR can facilitate clinical decision-making and
minimise the potential for mistakes due to the
inaccuracy and incompleteness of paper records
(Institute of Medicine 2001, Thompson and Brailer
2004, Kawamoto et al. 2005, Ohsfeldt et al. 2005).
However, the effects of EHR use on quality of care are
not necessarily automatic (Linder et al. 2007); they
very much depend on the specific characteristics of the
EHR system and its impact on the work of healthcare
providers and other staff.

Recently, the need to adopt and adapt methods and
techniques to understanding human factors and

*Corresponding author. Email: [email protected]

Behaviour & Information Technology

Vol. 28, No. 1, January–February 2009, 5–20

ISSN 0144-929X print/ISSN 1362-3001 online

� 2009 Taylor & Francis
DOI: 10.1080/01449290701628178

http://www.informaworld.com

organisational issues of the technology implementation
process has been increasingly recognised (Smith and
Carayon 1995, Carayon and Karsh 2000, Carayon and
Haims 2001, Karsh 2004). Regarding EHR implemen-
tation, these include poor usability of EHR user
interfaces, clinicians’ resistance to EHR acceptance,
and patients’ reaction to EHR (Ash and Bates, 2005).
The key to a successful EHR implementation project is
how well the technology is implemented and how the
technology can be used to improve clinician perfor-
mance and produce positive individual and organisa-
tional outcomes (Smith and Carayon 1995, Berner
et al. 2005). Increased efficiency in healthcare delivery
and improvements in patient information collection,
administrative processing, working conditions, and
user acceptance should lead to improvements in safety,
efficiency, and quality. Without a comprehensive
understanding of end user experience and the organi-
sational changes produced by the EHR technology, we
are missing opportunities to develop better approaches
to designing and implementing EHR technology.

According to the Center for Disease Control and
Prevention, general and family practices represent
about 24% of all physician office visits (Centers for
Disease Control and Prevention 2000). It is therefore
important to understand the barriers to effective and
successful implementation of EHR technology in
family practice clinics as a substantial portion of
ambulatory health care occurs in these settings. EHR
has been estimated to be used by about 24% of
physicians in ambulatory settings in the United States
in 2005 (Jha et al. 2006). Challenges in dealing with
human and organisational factors can partially explain
why the majority of small family practice clinics are
still unwilling or unable to consider the use of EHR in
their patient care. In addition, as compared to large
hospitals, small clinics face further challenges due to
limited financial and human resources (Middleton
et al. 2005). Healthcare professionals and adminis-
trative staff of small clinics frequently have to share job
responsibilities and cover for their coworkers because
of high workload, patient emergencies, and staffing
issues such as employee vacation and illness.

In this study, we evaluated the implementation of
Practice Partner Patient Records, by Physician’s
Microsystems, Inc., in a small family practice clinic;
before the EHR implementation, health records were
completely in paper records. This EHR system is a
vendor software intended to replace paper-based
patient health records. We evaluated the organisa-
tional aspects of the EHR implementation process and
the human factors issues resulting from the EHR
implementation. A systematic evaluation approach was
employed: both quantitative and qualitative data were
collected. This allowed us to evaluate how employees

in the clinic perceived their work as it related to the
EHR technology and the changes in work patterns due
to the EHR implementation. The direct impact of
EHR technology on clinical performance and patient
care (e.g., quality and safety of patient care) was not
examined in this study.

2. Conceptual framework

The most common reason for failure of technology
implementation is that the implementation process is
treated as a technological problem, and the human
factors and organisational issues are not fully ad-
dressed (Eason 1988). In reaction to this problem,
Carayon and Karsh (2000) have proposed a conceptual
model that specifies the human and organisational
issues related to technology implementation (see
Figure 1).

The introduction of a new technology is likely to
change jobs and work processes. It can create both
positive and negative impacts on job characteristics
(Carayon-Sainfort 1992); therefore, it is important to
understand the impact of the technology on multiple
dimensions of the jobs and work processes. Technol-
ogy characteristics can also impact job characteristics
and quality of working life in both positive and
negative manners (Carayon-Sainfort 1992). For in-
stance, a technology with usability deficiencies can
increase the workload of the users, and affect their
frustration at work and other attitudes toward their
organisation. This conceptual framework is used as the
basis for selecting measures to assess EHR implemen-
tation in a small family practice clinic.

3. Study design

In this study, three data collection instruments were
used to assess user experience and organisational
changes related to the implementation and use of
EHR: a user survey, interviews with key personnel
involved in the EHR project, and a work analysis. A
longitudinal design with two data collection rounds
before and after the EHR implementation was
employed.

The study site is a University of Wisconsin family
medicine residency clinic in a small community with a
population of about 1800, located 18 miles southwest
of Madison, Wisconsin. At the time of study, it had 6
family medicine faculty, 7 resident physicians, and 12
medical support and office staff. It had approximately
11 000 patient visits annually. Participation in the
study by the clinic personnel was voluntary.

Each data collection is described separately. The
results of each data collection are reported after the
description of the data collection method.

6 P. Carayon et al.

4. Survey of EHR users

4.1. Pre- and post-implementation survey

Based on the conceptual framework (see Figure 1) of
Carayon and Karsh (2000), the pre-implementation
survey examined the following human and organisa-
tional factors:

(a) Job information: job position (e.g., office staff,
nurse, doctor), job experience, and computer
experience

(b) Job characteristics: role ambiguity (Caplan
et al. 1975), quantitative workload (Caplan
et al. 1975), uncertainty (Seashore et al. 1983),
challenge (Seashore et al. 1983), task control
(McLaney and Hurrell 1988, Greenberger et al.
1989), decision control (McLaney and Hurrell
1988, Greenberger et al. 1989), resource control
(McLaney and Hurrell 1988, Greenberger et al.
1989), and general job control (McLaney and
Hurrell 1988, Greenberger et al. 1989)

(c) Quality of working life: organisational identi-
fication (Cook and Wall 1980), organisational
involvement (Cook and Wall 1980), daily life
stress (Reeder et al. 1973), job satisfaction
(Quinn et al. 1971), musculoskeletal discomfort
(Sainfort and Carayon 1994), and anxiety
(Sainfort and Carayon 1994)

(d) Technology characteristics: dependency on
computers (Carayon 1994), information
received about EHR system (Bailey and
Pearson 1983), input regarding design and

implementation of the EHR system (Bailey and
Pearson 1983), attitude toward EHR system
(Bailey and Pearson 1983), EHR effect on
performance (Davis 1989), overall user accep-
tance, learning, and EHR system capabilities
(Chin et al. 1988)

(e) Self-rated performance (Carayon 1994)
(f) Demographics: gender, age, educational level,

and marital status.

The first five sections (a, b, c, d, and e) of the pre-
implementation survey were also included in the post-
implementation survey. Twelve questions on technol-
ogy characteristics were added to the post-implemen-
tation survey. These questions were derived from the
Questionnaire for User Interface Satisfaction (QUIS)
(Chin et al. 1988). As a usability evaluation tool, QUIS
(Chin et al. 1988) consists of five categories of
questions on user experience with software user inter-
face: overall reactions to the software, learning, system
capabilities, terminology and system information, and
screen. We used the first three sections (reactions to the
software, learning, and system capabilities) in the post-
implementation survey. The demographics section
(Section f) was excluded from the post-implementation
survey based on a recommendation by the University
Institutional Review Board.

4.2. Participants

Twenty-one out of 25 clinic employees completed
the pre-implementation survey, while 20 out of

Figure 1. Impact of EHR technology on quality of working life and performance (adapted from Carayon and Karsh 2000).

Behaviour & Information Technology 7

25 employees completed the post-implementation
survey. Response rates were 84% and 80%,
respectively.

4.3. Procedures

The pre-implementation survey, along with the consent
form, was distributed to all 25 clinic employees in the
spring of 2000, six months before the EHR imple-
mentation. Clinic employees who agreed to participate
in the study signed the consent form and then
completed the survey. They left the signed consent
forms and completed surveys in a secured mailbox
that was accessed only by the researchers. In February
2002, 15 months after the implementation, the
post-implementation survey was administered using
the same procedure as in the pre-implementation
survey.

4.4. Data analysis

Data from the survey were manually entered into an
SPSS database and double-checked by another
researcher for quality control. The first step of the
data analysis produced normalised scores, from 0
(low) to 100 (high), for each measure included in the
following four sections of the survey: job character-
istics, quality of working life, technology issues, and
self-rated performance. Descriptive statistics were
calculated. Survey data were collected at two
different points in time: before and after the EHR
implementation; however, because of the small
sample size and the threat to anonymity, individual
responses were not tracked over time. Therefore,
Mann-Whitney tests were performed to compare the
group responses of the pre- and the post-implemen-
tation surveys.

The section on quality of working life consisted of
twenty-two four-point health questions with answers
ranging from 1 (‘‘never’’) to 4 (‘‘constantly’’) and
concerning three dimensions: (1) back, neck, shoulder
discomfort, (2) other musculoskeletal discomfort, and
(3) anxiety (Sainfort and Carayon 1994). Responses
to each of the 22 questions were grouped into
‘‘never,’’ ‘‘occasionally,’’ and ‘‘frequently and con-
stantly’’ in order to examine participants with no,
some, and a lot of perceived discomfort and anxiety.
Kruskal Wallis tests were performed to compare the
results of the pre- and the post-implementation
surveys.

4.5. Results

Descriptive statistics of job characteristics, quality
of working life, technology issues, and self-rated

performance, along with the results of Mann-Whitney
tests, are reported in Table 1. Two measures, resource
control and dependence on computer, were significantly
different between the pre- and the post-implementa-
tion surveys (p 5 0.05). Participants reported less
resource control and more dependence on computers
after EHR implementation. Perceived quantitative
workload increased slightly after EHR implementa-
tion, compared to before EHR implementation
(p 5 0.10).

Descriptive statistics of the 22 health questions as
well as the results of Kruskal Wallis tests can be
found in Table 2. The measure of tight feeling in
stomach was found to be significantly different
between the pre- and the post-implementation
surveys (p 5 0.05). Fewer participants reported tight
stomach feeling after EHR implementation. There
was a slight increase in the percentage of participants
who reported back pain and pain or stiffness in arms
or legs (p 5 0.10), and a slight decrease in terms of
swollen or painful muscles and joints (p 5 0.10), after
EHR implementation.

5. Interviews of key personnel involved in EHR

implementation project

5.1. Pre- and post-implementation interview guide

Structured interviews were conducted using an inter-
view guide based on the IT project management
interview guide (Korunka et al. 1997, Korunka
and Carayon 1999). The timeframe of the questions
was modified to reflect the pre- and post-implementa-
tion times. For example, a question on training
activities planned for the clinic staff was asked in
the pre-implementation interview, while a question on
the actual training activities that had taken place was
asked in the post-implementation interview. The
interview guide was structured as follows:

(a) Implementation background
(b) Project identity
(c) Project team
(d) Project manager
(e) Steering committee
(f) Implementation process, including goals,

processes, schedule, budget, information diffu-
sion, evaluation, problems/difficulties, project
crises, feedback/complaints (only in the
post-implementation interviews), and user
participation

(g) Training
(h) EHR support
(i) Changes in the working environment
(j) Interviewee profile.

8 P. Carayon et al.

5.2. Interviewees

Four key personnel who were directly involved in the
implementation process of the EHR system were
interviewed: the project director, the project manager,
the clinic manager, and the information system manager.

5.3. Procedures

The face-to-face pre-implementation interviews were
conducted with the four interviewees in April 2000.
They were provided with a copy of the interview guide
before the interviews. An interviewer asked questions
one by one following the structure of the interview
guide. In addition to answering the questions, inter-
viewees were encouraged to give feedback about the
questions and to provide additional information that
may be helpful to understand the EHR implementa-
tion process. The individual interviews lasted 60 –
120 min. The post-implementation interviews were
again conducted with the same four people by
telephone between March and April 2001.

5.4. Data analysis

Data collected during the interviews were entered into
an Access database. Descriptive information about the
EHR implementation process is provided in the next
section.

5.5. Results

5.5.1. Implementation background

The primary factors driving EHR implementation were
the need for improving medical care and the trend in the
industry. Secondary factors included the desire for work
reduction, adjustment to market demands, and the
reduction of employee workload. These resulted in the
introduction of an EHR system to replace an existing
paper-based medical record system. It took four months
to complete the actual implementation. The workstation
selection was based on vendor recommendations.
Selection criteria for the software included capability,
serviceability, user friendliness, popularity, and recom-
mendations of the product. Decisions on project scope

Table 1. Survey of EHR users: descriptive statistics and results of Mann-Whitney tests.

Pre
{
(normalised

scores: 0 – 100)
Post

{
(normalised

scores: 0 – 100)
Mann-Whitney
tests (p values)Mean S.D. Mean S.D.

Job characteristics
Role ambiguity 13 13.9 14 13.4 0.666
Quantitative workload 72 12.1 77 15.6 0.091*
Uncertainty 70 17.3 64 19.6 0.341
Challenge 76 17.3 81 15.5 0.440
Task control 50 18.8 47 17.0 0.801
Decision control 32 26.9 27 23.5 0.713
Resource control 47 25.6 29 25.8 0.028**
General job control 44 18.2 38 16.3 0.325

Quality of working life
Organisational identification 87 12.0 83 14.6 0.597
Organisational involvement 88 12.5 85 12.5 0.477
Daily life stress 44 11.6 41 11.3 0.247
Job satisfaction 72 25.6 78 24.3 0.436
Musculoskeletal discomfort 15 12.0 17 12.1 0.530
Back, neck, shoulder 24 23.8 29 21.2 0.159
Other musculoskeletal 11 9.3 11 9.9 0.916
Anxiety 18 14.1 15 11.3 0.453
Self-reported performance 77 12.4 76 11.3 0.800

Technology characteristics
Dependence on computers 73 27.5 89 26.8 0.018**
Information received about system 53 27.6 63 25.1 0.295
Design input 46 33.2 54 28.2 0.271
Implementation input 49 32.8 56 27.4 0.569
Effect on performance 67 23.9 53 25.7 0.117
Attitude toward system 68 24.0 66 24.5 0.790
Overall reactions N/A N/A 54 16.9 N/A
Learning N/A N/A 51 17.4 N/A
System capabilities N/A N/A 63 17.1 N/A

{
Pre-implementation survey.

{
Post-implementation survey.

*p 5 0.10; **p 5 0.05.

Behaviour & Information Technology 9

and hardware/software selection were made jointly by
the steering committee, the project team, expert end
users, and the information systems department.

5.5.2. Project identity

During the pre-implementation interviews, all four
interviewees said that the project was given a special
identity using the project name. Three interviewees
reported that the project had no special identity when
interviewed after the implementation.

5.5.3. Project team

All four interviewees agreed that the project team had
an informally defined scope, authority, and responsi-
bility. However, their understanding of the project
team composition diverged. For example, their an-
swers to the question on how many expert end users
were on the project team varied from 1 to 5. The four
interviewees agreed that the project team members
were chosen based on professional expertise and EHR
knowledge by top management. Besides their regular
job duties, team members were granted time to work

on project-related activities, including 30 weekly
project meetings throughout the EHR implementation
process. The four interviewees agreed that the overall
attitude of the project team was good.

5.5.4. Project manager

The project manager was hired externally and tem-
porarily for the EHR implementation project and
reported to the project director. Top management
made the hiring decision based on criteria such as
experience as project manager, professional expertise,
and personality. The project manager did not receive
extra training on project management. It was unclear
whether the project manager had authority to make
decisions in cases of diverging opinions: two inter-
viewees said that the project manager did not have this
authority, while the other two considered that the
project manager had informal authority.

5.5.5. Steering committee

There was not an officially designated steering com-
mittee specifically for this project. The project team

Table 2. Survey of EHR Users – Descriptive Statistics of Health Questions and Results of Kruskal-Wallis Tests

Pre
{

Post
{

Kruskal-Wallis
test (p values)N

x
O
{

F/C
{{

N
x

O
{

F/C
{{

Back, neck, shoulder discomfort
1. Back pain 7 12 2 3 12 5 0.095*
2. Pain or stiffness in your neck and shoulders 4 14 2 3 12 5 0.291
3. Feeling of pressure in the neck 13 4 4 8 7 5 0.231
4. Shoulder soreness 11 7 3 9 9 2 0.796
5. Neck pain that radiates into shoulders, arms or hands 15 3 3 13 3 4 0.634

Other musculoskeletal discomfort
6. Swollen or painful muscles and joints 3 15 3 7 12 1 0.092*
7. Pain or stiffness in your arms or legs 12 8 1 6 12 2 0.085*
8. Persistent numbness or tingling in any part of your body 18 3 0 15 4 1 0.363
9. Pain down your arms 16 5 0 16 3 1 0.842
10. Leg cramps 16 4 1 15 5 0 1.000
11. Difficulty with feet and legs when standing for prolonged periods 12 8 1 14 5 1 0.431
12. Loss of feeling in the fingers or wrists 17 3 1 16 3 1 0.940
13. Cramps in hands/fingers relieved only when not working 18 3 0 17 3 0 0.949
14. Loss of strength in arms or hands 18 3 0 18 2 0 0.679
15. Stiff or sore wrists 17 4 0 16 3 1 0.880

Anxiety
16. Occasions of easy irritability 7 12 2 3 17 0 0.481
17. Difficulty sleeping 11 7 3 8 12 0 0.816
18. Periods of depression 9 10 2 12 8 0 0.194
19. Times of severe fatigue or exhaustion 10 10 1 10 10 0 0.766
20. Tight feeling in stomach 15 6 0 20 0 0 0.011**
21. Periods of extreme anxiety 15 6 0 17 3 0 0.300
22. High levels of tension 9 11 1 8 12 0 1.000

{
Pre-implementation survey.

{
Post-implementation survey.
x
Never.
{
Occasionally.

{{
Frequently and constantly.

*p 5 0.10; **p 5 0.05.

10 P. Carayon et al.

and the project director reported to the department’s
standing executive committee.

5.5.6. Implementation process

The goals of EHR implementation were to enhance
healthcare quality and patient safety, to improve work
quality and reliability, to improve information sharing
and communication, and to reduce work steps and
errors. These goals were formulated by the project team
and local top management through preliminary work
done before implementation, including goal setting,
cost-benefit analyses, and risk assessment activities.
Two interviewees indicated that technical difficulties
during EHR implementation were significant, the other
two reported noticeable but slight difficulties. Critical
issues included how the EHR system could interface
with billing functions. Problems with the vendor were
reported, such as corrupted configuration with lab data
(took six weeks to get it corrected), and system upgrade
crash before going live (lost days of data). One
interviewee rated the problems as significant, one as
noticeable, and the other two as slight. Underestimation
of the amount of work required for EHR implementa-
tion was another major difficulty reported by three
interviewees, in addition to the concern regarding the
authority of the project manager, the lack of interest
and resistance from end users, the disagreement within
the project team, the resistance from middle manage-
ment, and the lack of priority for the project. According
to the interviewees, end users complained of an increase
of work due to the implementation, technical interrup-
tion, and time pressure during EHR implementation.
The project manager complained about software bugs,
while local top management was concerned with
decreased productivity during the implementation and
the cost. Patients were reported to have concern
regarding privacy of their medical data. User acceptance
of the EHR was evaluated through informal discussion.

5.5.7. Training

The four interviewees considered that local top
management had been very positive towards training.
The amount of training that users received was decided
jointly by the project team and the EHR vendor. The
training scheduling was established by the project
team. All clinic employees were informed that they
would need to be trained. Training schedules and
training materials were provided. Groups of users with
similar needs were trained together through hands-on
practice. Expert users were trained for 8 h, while others
were trained for 4 h. The training consisted of two
sessions on basic Windows and the EHR system. When
a user was attending a training session, his (or her)

regular job duties were covered by other employees.
No extra work hours were explicitly needed.

5.5.8. EHR support

There were support staff present from the EHR vendor
on the day the EHR system went live. In the following
two weeks, at least one expert end user was present at
the clinic. The software maintenance was done
internally. In addition, there were plans for improving
the EHR system by correcting software bugs, adding
new applications, upgrading new releases, as well as
upgrading hardware components.

5.5.9. Changes in working environment

All 4 interviewees agreed that clinic employees
experienced changes in skills and work flow, and
increased workload due to the implementation and use
of EHR. The use of EHR did not result in reduction of
personnel. There was an increase in time spent using
the computer, although it varied depending on the job
category (e.g., nursing staff and physicians experienced
more changes than others). Two interviewees observed
a slight change in social climate as a result of the
implementation, while the other two observed no
change or did not know. In general, all interviewees
agreed that the climate of the entire clinic was positive
after EHR implementation.

6. Work analysis of clinic staff

6.1. Work analysis form

Pre- and post-implementation work analyses were
conducted using the multidimensional work sampling
technique (Sittig 1993, Murray et al. 1999). The
multidimensional work sampling technique was used
to determine time spent on a variety of predefined
activities (‘‘activity’’), the purpose of the activity
(‘‘function’’), and with whom the person was in contact
while performing the activity (‘‘contact’’). The work
analysis form and the definitions for the activities,
functions, and contacts were first created using
information from the position descriptions provided
by the clinic manager. After creating the form and the
definitions, the researchers met with the medical
director of the clinic and the clinic manager to discuss
and revise the data collection form. The frequency,
duration, and timing of the work analysis were also
discussed. The same form was used in both the pre-
and post-implementation studies. It included 13
activities, 22 functions, and 14 contacts. For each
entry on the form, study participants could also record
comments when they were unsure what activity,
function, or contact to record (see Appendix).

Behaviour & Information Technology 11

6.2. Participants

All clinic employees were invited to participate in the
work analysis in the pre- and post-implementation
phases. Twenty-seven clinic employees participated in
the pre-implementation study, while 26 employees
participated in the post-implementation study. Unlike
the employee survey, where primarily full time employ-
ees were recruited to participate, the work analysis
recruitment included everyone who worked at the
clinic, be they part-time, full-time, permanent, or
temporary. For that reason there were more partici-
pants in the work analysis than there were in the survey
questionnaire.

6.3. Procedures

The pre-implementation work analysis was conducted
in April 2000 for a period of 10 working days. It began
on a Tuesday due to the hectic nature of Mondays
following a weekend. It was believed that staff would
have more time to adjust and be familiar with the work
analysis tool by beginning on Tuesday. The work
analysis forms were distributed to all employees,
including the medical staff, at the clinic. Participants
indicated their position and the beginning and ending
times of their workday. During each day, participants
were asked to record activities, functions, and contacts
every 30 min. They were encouraged to write down
comments when they were uncertain about what to
record. An announcement via overhead speaker was
made approximately every 30 min to remind partici-
pants to complete the form. At the end of each day,
participants dropped their form in a locked mailbox,
which only researchers had access to. The post-
implementation work analysis was conducted in June
and July 2002 using the same process as in the pre-
implementation study, except for a slight variation in
the recording time. An observation was made by the
medical director that some staff, anticipating the
recording time, remained at their desk rather than
leave or initiate a different task when the 30-min
interval approached. Thus, in the post-implementation
study, an announcement was not made at the exact 30-
min interval, but rather at an approximate time (e.g.,
plus or minus five minutes of the half-hour). Partici-
pants again dropped their form in a locked mailbox at
the end of each day.

6.4. Data analysis

One hundred and forty-five forms were collected with
1960 entries for the pre-implementation study and 122
forms with 1825 entries for the post-implementation
study. Data from the work analysis were entered into

Excel worksheets. In case there were two activities,
functions, or contacts recorded for a time period, a
research scientist (A.S.H.) working on the study chose
what appeared to be the most appropriate code based on
comments provided by the respondent or the patterns of
other entries or both. This judgment was made by the
research scientist to ensure consistency in coding. Once
all data were entered, frequencies were computed for the
activities, functions and contacts for each of the three job
categories: physicians, nonphysician clinical staff (e.g.,
nurses, laboratory technicians, radiology technicians),
and office staff. We identified some confusion the
participants experienced in choosing the activity, func-
tion, or contact to record. For instance, computer entry
(A3) should be recorded when entering information into
the computer, and not typing/writing/signing (A12). To
address the confusion, a standard procedure for recod-
ing was developed and all data of the pre- and the post-
implementation studies were reviewed by the same
research scientist, again, to ensure consistency in the
recoding. After completing the recoding, all frequencies
were recomputed. Frequencies for each activity, func-
tion, contact, and their task combination (activity/
function/contact) were calculated. Comparisons of the
pre- and the post-implementation work analysis data
were performed to examine changes in the distribution
of time spent on various activities, functions, and
contacts for each of the three job categories (see Tables
4–6). w2 tests were run to compare the distribution of
frequencies for each of the three job categories
separately; this same analysis was done for the data on
activities, function, and contact. Because some of the
percentages were small, we combined the data for
the categories of activity, function, or contact whenever
the ‘‘pre’’ and the ‘‘post’’ percentages were smaller than
5% (see Figures 2–4).

6.5. Results

The numbers of tasks and entries are provided in
Table 3. Physicians had the least number of tasks and

Table 3. Work analysis of EHR users: number of tasks and
entries.

Number of tasks
Number of
entriesOriginal Recoded

Pre
{

Post
{

Pre
{

Post
{

Pre
{

Post
{

Physician 101 94 79 81 545 458
Clinical staff 138 83 122 76 576 393
Office staff 204 210 186 173 839 974
All 361 326 305 266 1960 1825

{
Pre-implementation survey.

{
Post-implementation survey.

12 P. Carayon et al.

office staff had the most number of tasks. The former
handled about 80 tasks, while the latter handled
approximately 180 tasks. The number of tasks
performed by physicians and office staff did not change
much in the post-implementation study as compared
with that in the pre-implementation study. There was
a 38% decrease in the number of tasks performed by
the clinical staff: from 122 tasks in the pre-
implementation study to 76 tasks in the post-
implementation study.

6.5.1. Activity

Frequencies of activity comparing pre- and post-
implementation are shown in Table 4 and Figure 2.
Physicians spent about half of their time caring for
patients in both the pre- and post-implementation
studies. The EHR implementation did not affect the
amount of time physicians spent with patients, but
increased the amount of time spent by physicians on
computer entry and decreased time spent on dictation,
phone, and typing/writing/signing. For clinical staff,
the main differences between the pre- and the post-
EHR implementation were the following: increases in
time spent on patient care, computer entry, phone and
preparing, and decreases in time spent in meeting,
performing lab work, and typing/writing/signing. The
difference between the pre- and post-EHR frequencies
of activity for the office staff was not statistically
significant.

6.5.2. Function

Frequencies of function comparing the pre- and post-
implementation are shown in Table 5 and Figure 3.

Physicians spent almost half of their time examining or
treating patients before and after the EHR implemen-
tation (from 42.4% to 48.7%). The frequencies of
function for the physicians did not significantly change
after the EHR implementation. Clinical staff spent
more time on the following functions: accompanying
patients (from 17.5% to 22.9%), examining patient
(from 4.3% to 12.7%), and maintaining medical
information system (from 5.6% to 10.9%).
Clinical staff spent less time on distributing chart/
master file/mail (from 6.6% to 0.3%), and performing
tests (from 18.6% to 11.2%). The functions of
office staff also changed significantly: they spent
about one-half less time for distributing chart/master
file/mail (from 5.6% to 2.5%), general clerical assis-
tance/office tasks (from 27.8% to 13.1%), and
transcription (from 13.3% to 7.5%), but more time
on maintaining the medical information system (0%
to 19.8%).

6.5.3. Contact

Frequencies of contact can be found in Table 6 and
Figure 4. Physicians had about the same distribution
of contact before and after the EHR implementation.
Clinical staff spent less time in contact with doctors
and nurses, but more time with patients and patient
representatives, and doing tasks by themselves. On the
contrary, office staff spent more time with nurses, but
less time doing tasks on their own.

6.5.4. Task combination

Task combinations represent combinations of an
activity, a function, and a contact. Some of the task

Table 4. Work analysis of EHR users: frequencies of activity (%).

Activity

Physicians Clinical staff Office staff All

Pre
{

Post
{

Pre
{

Post
{

Pre
{

Post
{

Pre
{

Post
{

A1 Absent 0 0 0.7 0 1.0 0.1 0.6 0.1
A2 Caring for patient 49.7 50.7 27.8 39.2 7.2 4.0 25.1 23.3
A3 Computer entry 1.7 21.2 4.7 10.4 19.0 30.3 9.9 23.7
A4 Dictation 9.7 1.5 0.7 0 0 0 2.9 0.4
A5 Meeting 1þ 4.8 3.3 11.1 1.0 5.6 4.6 7.0 3.5
A6 Meeting 3þ 0.7 1.1 1.0 3.1 1.9 2.3 1.3 2.1
A7 Performing lab work 0.2 0.2 18.4 11.7 0 0.1 5.5 2.6
A8 Phone 11.4 5.5 9.0 13.0 16.2 16.9 12.8 13.2
A9 Preparing 2.4 2.0 14.1 17.8 22.6 14.7 14.5 12.2
A10 Reviewing check 5.0 4.6 3.1 2.0 1.4 0.9 2.9 2.1
A11 Supervising 4.0 4.4 0.9 0 0.5 0.5 1.6 1.4
A12 Typing/writing/signing 9.0 3.5 4.5 1.0 16.2 18.3 10.8 10.8
A13 Other 1.5 2.2 4.0 0.8 8.3 7.2 5.2 4.5

{
Pre-implementation survey.

{
Post-implementation survey.

Behaviour & Information Technology 13

combinations performed by physicians are listed
below:

. ‘‘Computer entry/maintaining medical informa-
tion system/self’’ increased from 0.2% to 15.7%

. ‘‘Caring for patient/examination or treatment
of patient/patient and patient representative’’
slightly increased from 40.4% to 45.85%

. ‘‘Dictation/maintaining medical information sys-
tem/self’’ decreased from 9.7% to 1.5%

. ‘‘Typing, writing, signing/maintaining medical
information system/self’’ decreased from 7.7% to
1.5%

. ‘‘Phone/providing instruction, information/pa-
tient and patient representative’’ decreased
from 7.3% to 1.5%.

Some of the task combinations performed by clinical
staff are listed below.

. ‘‘Caring for patient/examination or treatment of
patient/patient and patient representative’’ in-
creased from 3.5% to 11.7%

. ‘‘Performing lab work/performing tests/doctor’’
decreased from 10.4% to 1%

. ‘‘Meeting 1þ/training/nurse’’ dropped out com-
pletely from 7% to 0%.

Some of the task combinations performed by office
staff are listed below.

. ‘‘Computer entry/maintaining medical informa-
tion system/self’’ increased from 0% to 7.2%

. ‘‘Typing, writing, signing/transcription/self’’ de-
creased from 11.7% to 6.5%.

Overall, the most significant change was found in the
combination of ‘‘Computer entry/maintaining medical
information system/self’’ with an increase from 0.6%
in the pre-implementation study to 9.1% in the post-
implementation study.

7. Discussion

7.1. Survey of EHR users

Overall, clinic employees experienced low role
ambiguity, high workload, high uncertainty, high
challenge, moderate task control, and low decision
control. They felt that they had high organisational
identification and involvement, moderate daily
life stress, low musculoskeletal discomfort, low anxi-
ety, high job satisfaction, and high self-rated
performance.

As expected, dependency on computers signifi-
cantly increased with EHR implementation. Unexpect-
edly, clinic employees felt that they had less resource
control after EHR implementation. The medical
director of the clinic provided a possible explanation
to this unexpected finding: the clinic employees were
under budgetary control at the time of the post-
implementation survey. Therefore, they may have
reported deceased control over resources such as
supplies and materials. Other interesting results of
the survey included increases in perceived workload,
back pain and pain/stiffness in arms/legs, and

Figure 2. Work analysis: comparison of pre-EHR and post-
EHR frequencies of activity. (a) Physicians (w2 ¼ 27.22;
df ¼ 6; p 5 0.001); (b) clinical staff (w2 ¼ 18.21; df ¼ 6;
p 5 0.01); (c) office staff (w2 ¼ 5.3; df ¼ 5; not significant).

14 P. Carayon et al.

decreases in swollen/painful muscles and joints and
report of tight feeling in stomach.

Before EHR implementation, clinic employees
reported that they received moderate information
about the EHR system, had moderate design input
and moderate input into the implementation process.
After EHR implementation, their perceptions on these
issues were more positive: they reported that they
received better information and that their inputs were
more widely considered.

In general, clinic employees’ attitude toward EHR
system was positive. They reported that the EHR
system had some positive effect on their performance.
They felt that the EHR system was moderately easy to
learn and that the EHR system had moderate
capabilities in terms of technical performance.

7.2. EHR implementation process: interviews

The four interviewees were the key personnel involved
in the EHR implementation process. They agreed that
the overall attitude of the project team was good and
the climate of the entire clinic during the implementa-
tion was positive. On the other hand, they reported
that clinic employees complained of the increased
workload due to the technical problems and time
pressure associated with EHR implementation.

Technical difficulties, user resistance, and other pro-
blems are relatively common with this type of
technology implementation project (Ash and Bates
2005). The project team was able to identify those
problems and correct them during weekly meetings. In
addition, the interviewees reported that user training
was thoroughly planned and delivered and technical
support was available to end users as they needed.

With regard to some issues such as the project
identity, the authority of project manager, and the
severity of technical difficulties experienced during the
implementation, the four interviewees’ perceptions
varied. This variation in perceptions highlights the
need for clarifying and specifying the structure of the
EHR implementation process.

7.3. Work analysis

Overall, the work analysis showed many differences in
the work of clinical staff and office staff, and few
changes for the work of physicians. There was no
difference in physician time spent caring for patients
before and after EHR implementation. Physicians
spent about half of their time on the activity of patient
care and about 55% of their time on the two functions
of ‘‘examination or treatment of patient’’ and ‘‘provid-
ing instruction-information.’’ This result is similar to

Table 5. Work analysis of EHR users: frequencies of function (%).

Function

Physicians
Clinical
staff Office staff All

Pre
{

Post
{

Pre
{

Post
{

Pre
{

Post
{

Pre
{

Post
{

F1 Accompanying patients 0.2 1.1 17.5 22.9 0.5 1.1 5.4 5.8
F2 Assisting physician, doctors or medical technician 0.4 0.9 1.9 3.1 0.1 0 0.7 0.9
F3 Billing activities 0.4 0.4 0 0.3 16.2 16.8 7.0 9.2
F4 Checking message 0.9 3.5 0.7 0.5 1.9 2.2 1.3 2.1
F5 Data review and retrieval 3.9 3.3 4.0 4.1 3.1 1.6 3.6 2.6
F6 Distributing chart/master file/mail 0 0.2 6.6 0.3 5.6 2.5 4.3 1.4
F7 Examination or treatment of patient 42.4 48.7 4.3 12.7 0 0.4 13.1 15.2
F8 General clerical assistance/office task 0.4 0.2 3.5 1.5 27.8 13.1 13.0 7.4
F9 Maintaining equipments, instruments, supplies and medications 0.2 0 3.0 7.1 0.5 0.3 1.1 1.7
F10 Maintaining lab reports 0.2 0.2 4.9 6.4 0 0.1 1.5 1.5
F11 Maintaining medical information system 23.3 24.7 5.6 10.9 0 19.8 8.1 19.1
F12 Maintaining patient’s information record 0.4 0.7 1.0 0.3 2.1 2.1 1.3 1.3
F13 Participating in resident and student education 7.3 5.5 0.7 0 0.5 0 2.4 1.4
F14 Performing tests 0 0.2 18.6 11.2 0.1 0.2 5.5 2.6
F15 Preparing for examinations and surgical procedures 0.4 0 1.4 2.0 0 0.3 0.5 0.6
F16 Providing instruction/information 13.0 4.8 5.6 6.6 1.0 1.2 5.7 3.3
F17 Purchasing and making inventory arrangement 0 0 0.3 0.5 0.2 0.2 0.2 0.2
F18 Reporting problem 0.4 0.9 0.9 0.3 0.2 1.2 0.5 0.9
F19 Scheduling 0.2 0.2 4.9 3.8 13.8 14.9 7.4 8.8
F20 Training 0.6 0.9 6.9 0.5 1.0 1.8 2.6 1.3
F21 Transcription 0 0 0.2 0 13.3 7.5 5.8 4.0
F22 Other 5.7 3.7 7.6 5.1 11.9 12.3 8.9 8.6

{
Pre-implementation survey.

{
Post-implementation survey.

Behaviour & Information Technology 15

those of other studies. For example, a study of
physicians in an outpatient oncology clinic found
that physicians spend about 43% of their time in
patient care (Fontaine et al. 2000). As expected,
computer entry activity by physicians increased in
place of dictation, phone, and typing/writing/signing
activities, which decreased.

Clinical staff spent more time caring for patients
after EHR implementation. A possible explanation to
this might be that there was less lab work to be
performed on patients (summer appointments – as

occurred during the post-implementation study –
frequently tend to include more well-patient work-
ups and physical exams that do not require lab work).
Meetings with one or two persons dropped from 11%
to 1%, probably because of use of the EHR internal
messaging system. In addition, clinical staff spent more
time in maintaining the medical information system
instead of distributing chart/master file/mail.

Office staff used EHR by spending more time on
computer entry and less time on preparing activities
(e.g., filling, retrieving, and distributing charts). After

Figure 3. Work analysis: comparison of pre-EHR and post-EHR frequencies of function. (a) Physicians (w2 ¼ 4.68; df ¼ 4; not
significant); (b) clinical staff (w2 ¼ 22.29; df ¼ 9; p 5 0.01); (c) office staff (w2 ¼ 28.03; df ¼ 6; p 5 0.001).

16 P. Carayon et al.

EHR implementation, more of their time was spent in
maintaining the medical information system rather
than on a general clerical assistance/office task. The
office staff spent more time with nurses and less time
doing tasks on their own after EHR implementation.

7.4. Implementation process

The three data collection methods provided comple-
mentary information on EHR implementation and its
impact on the clinic staff and their work. According to
the questionnaire survey, staff reported increased
dependency on the computer, which was confirmed
by the increased amount of time spent using the
computer in the work analysis. Perceptions of the staff
regarding the EHR implementation (i.e. information
received about the EHR implementation and input
into the implementation process) improved after the
EHR implementation. This information was supported
by reports by interviewees of a number of activities for
involving end users (e.g., planning of training, inquiry
by the project team regarding problems experienced
by the end users). The questionnaire data analysis
showed a slight increase in workload. This may have
been due to technical problems and time pressure
associated with the EHR implementation, issues that
were described by the key project members in the
interviews.

A number of interesting results emerge from this
case study. First, the EHR implementation had some
impact on the perceived work content of clinic staff as
measured by the survey, especially regarding increased
dependency on computers that was related to increas-
ing use of computers for various tasks. Second, the

amount of time spent by physicians on patient care
(about 50% of their time) did not change with the
EHR implementation. A recent study of physician time
use before and after implementation of an EHR system
provides a similar finding (Pizziferri et al. 2005).
Pizziferri and colleagues (2005) found that the mean
overall time spent by physicians per patient did not
significantly change from before implementation to
after EHR implementation. Third, there were major
changes in the work of clinical staff and office staff
following the EHR implementation. Clinical staff
spent more time in computer entry and maintaining
the medical information system; office staff also spent
more time in computer entry and maintaining the
medical information system, and less time in distribut-
ing chart and transcription; however, these changes did
not induce an increase in time spent doing tasks on
their own. On the contrary, office staff spent more time
in contact with nurses after the EHR implementation.

In this case study, the EHR implementation went
relatively smoothly, probably because of a positive
climate existing in the clinic. A few implementation
issues could have been improved (e.g., clarifying the
structure of the EHR implementation organisation).
However, it seemed that the project implementation
process was designed to identify emerging issues (e.g.,
reasons for resistance to change) and to provide
solutions ‘‘just-in-time.’’

7.5. Study limitations and future research

The data reported in this paper are based on only one
small family medicine residence clinic, and therefore
cannot be generalisable to other clinics. However, it

Table 6. Work analysis of EHR users: frequencies of contact (%).

Contact

Physicians Clinical staff Office staff All

Pre
{

Post
{

Pre
{

Post
{

Pre
{

Post
{

Pre
{

Post
{

C1 Billing coordinator 0 0 0 0 1.0 0.7 0.4 0.4
C2 Doctor 2.6 2.6 14.6 9.2 2.4 1.2 6.0 3.3
C3 Manager 0.7 0 0.5 0.3 0.8 0.7 0.7 0.4
C4 Medical student 1.5 0.2 0.2 0 0 0 0.5 0.1
C5 Medical technician 0.9 0.4 0.9 0 2.6 0.1 1.6 0.2
C6 Nurse 0.2 0.9 9.5 1.0 0.5 7.5 3.1 4.4
C7 Office staff 0 1.3 1.2 0.3 6.2 4.3 3.0 2.7
C8 Other student 0 0 0 0 0.4 0.2 0.2 0.1
C9 Patient and patient representative 56.1 53.7 29.2 40.7 18.7 19.2 32.2 32.5
C10 Resident 4.8 5.2 1.0 0.5 0.8 0.2 2.0 1.5
C11 Self 28.6 31.2 33.9 41.0 55.8 48.6 41.8 42.6
C12 Supervisor 0.7 1.1 0.3 0 1.2 2.0 0.8 1.3
C13 Other 3.9 3.3 7.8 7.1 9.3 14.9 7.3 10.3
C14 Radiographer 0 0 0.9 0 0.2 0.2 0.4 0.1

{
Pre-implementation survey.

{
Post-implementation survey.

Behaviour & Information Technology 17

provides important lessons regarding EHR implemen-
tation and its evaluation. First, an EHR implementa-
tion should be considered as a project and should
therefore utilise project management concepts and
methods (e.g., project structure, roles, timeline). This
can help with the process itself, such as monitoring the
implementation and being aware of problems with the
implementation. Second, attention to the EHR im-
plementation as a project can help anticipate the

impact of the technology on the work of providers and
clinic staff and provide important information for
training. In the process of change, we came to
understand several keys to a successful EHR imple-
mentation project:

. Importance of analysing needs and preferences
of medical providers and key administrators

. A strong physician leader to champion the
project

. Hiring a project manager with dedicated time to
lead the project

. Forming a project leadership team of key
personnel from clinical, office, and information
system staff

. Gathering needs of other users early in the
planning process

. Obtaining buy-in by clinicians and office staff
early in the process.

Our study clearly shows the importance of using
multiple data collection methods in order to fully
appreciate the range of human and organisational
factors involved in technology implementation. The
questionnaire survey provided information on the
EHR implementation from the viewpoint of the clinic
staff; the interviews with key project personnel allowed
a better understanding of the EHR implementation
process and its characteristics; the work analysis
allowed an in-depth evaluation of the impact of the
EHR technology on the work of different job
categories. We would like to recommend that future
research on the impact of EHR technology implemen-
tation use multiple data collection methods, including
both qualitative and quantitative approaches.

8. Conclusion

In this paper, we described a case study of the
implementation of an EHR system in a small family
practice clinic. Quantitative and qualitative data
collection methods provided complementary informa-
tion on how employees of the small clinic perceived
their work and the implementation of the EHR system.
The data showed few changes in work patterns of
physicians due to the use of EHR, except for the
increased computer entry. On the other hand, there
were major changes in the work of clinical staff and
office staff. A comprehensive examination of the
human and organisational factors as to EHR imple-
mentation was reported in this case study. This can
provide valuable inputs for a successful implementa-
tion of EHR in small clinic settings.

The results of our study highlight the need
to consider EHR implementation as a major

Figure 4. Work analysis: comparison of pre-EHR and post-
EHR frequencies of contact. (a) Physicians (w2 ¼ 0.20;
df ¼ 3; not significant); (b) clinical staff (w2 ¼ 11.73;
df ¼ 4; p 5 0.05); (c) office staff (w2 ¼ 7.04; df ¼ 4;
p 5 0.05).

18 P. Carayon et al.

sociotechnical change project. Once a healthcare
organisation has decided to purchase an EHR system,
principles of project management and technological
change need to be applied to ensure rapid and efficient
uptake by end users and to minimise disruptions to
work flow (Smith and Carayon 1995, Korunka and
Carayon 1999).

References

Ash, J.S. and Bates, D.W., 2005. Factors and forces affecting
EHR system adoption: report of a 2004 ACMI discus-
sion. Journal of the American Medical Informatics
Association, 12, 8–12.

Bailey, J.E. and Pearson, S.W., 1983. Development of a tool
for measuring and analyzing computer user satisfaction.
Management Science, 29, 530–545.

Berner, E.S., Detmer, D.E., and Simborg, D., 2005. Will the
wave finally break? A brief view of the adoption
of electronic medical records in the United States.
Journal of the American Medical Informatics Association,
12, 3–7.

Caplan, R.D., Cobb, S., French, J.R.P., Harrison, R.V., and
Pinneau, S.R., 1975. Job demands and worker health.
Washington, D.C.: U.S. Government Printing Office.

Carayon, P., 1994. Research on prevention strategies in
automated offices. In: G.E. Bradley and H.W. Hendrick,
eds. Human Factors in Organizational Design and
Management IV. Amsterdam: Elsevier, 707–712.

Carayon, P. and Haims, M.C., 2001. Information &
communication technology and work organization:
achieving a balanced system. In: G. Bradley, ed. Humans
on the Net-Information & Communication Technology
(ICT), Work Organization and Human Beings. Stock-
holm: Prevent, 119–138.

Carayon, P. and Karsh, B., 2000. Sociotechnical issues in the
implementation of imaging technology. Behaviour and
Information Technology, 19, 247–262.

Carayon-Sainfort, P., 1992. The use of computers in offices:
impact on task characteristics and worker stress. Inter-
national Journal of Human Computer Interaction, 4, 245–
261.

Centers for Disease Control and Prevention (CDC), 2000.
Healthy people 2010: US Department of Health and
Human Services. Available online at http://www.health.
gov/healthy people/Document/default.htm.

Chin, J.P., Diehl, V.A., and Norman, K.L., 1988. Develop-
ment of an instrument measuring user satisfaction of the
human-computer interface. In: Proceedings of ACM
SIGCHI (pp. 213–218), New York: ACM/SIGCHI.

Cook, J. and Wall, T.D., 1980. New work attitudes measures
of trust, organizational commitment, and personal need
non-fulfillment. Journal of Organizational Psychology,
53, 39–52.

Davis, F.D., 1989. Perceived usefulness, perceived ease of
use, and user acceptance of information technology. MIS
Quarterly, 13, 319–340.

Eason, K., 1988. Information Technology and Organizational
Change. London: Taylor & Francis.

Fontaine, B.R., Speedie, S., Abelson, D., and Wold, C.,
2000. A work-sampling tool to measure the effect of
electronic medical record implementation on health care
workers. The Journal of Ambulatory Care Management,
23, 71–85.

Greenberger, D.B., Strasser, S., Cummings, L.L., and
Dunham, R.B., 1989. The impact of personal control
on performance and satisfaction. Organizational Behavior
and Human Decision Processes, 43, 29–51.

Institute of Medicine, 2000. To Err Is Human: Building a
Safety Health System. Washington, D.C.: National
Academy Press.

Institute of Medicine, 2001. Crossing the Quality Chasm: A
New Health System for the 21st Century. Washington,
D.C.: National Academy Press.

Jha, A.K., Ferris, T.G., Donelan, K., DesRoches, C.,
Shields, A., Rosenbaum, S., and Blumenthal, D., 2006.
How common are electronic health records in the United
States? A summary of the evidence. Health Affairs, 25,
w496–w507.

Karsh, B., 2004. Beyond usability: designing effective
technology implementation systems to promote
patient safety. Quality and Safety in Health Care, 13,
388–394.

Kawamoto, K., Houlihan, C.A., Balas, E.A., and Lobach,
D.F., 2005. Improving clinical practice using clinical
decision support systems: a systematic review of trials to
identify features critical to success. British Medical
Journal, 330, 765.

Korunka, C. and Carayon, P., 1999. Continuous
implementation of information technology: the develop-
ment of an interview guide and a cross-national
comparison of Austrian and American organizations.
Human Factors and Ergonomics in Manufacturing, 9,
165–183.

Korunka, C., Weiss, A., and Zauchner, S., 1997. An
interview study of ‘continuous’ implementations of
information technology. Behaviour & Information Tech-
nology, 16, 3–16.

Linder, J.A., Ma, J., Bates, D.W., Middleton, B., and
Stafford, R.S., 2007. Electronic health record use and the
quality of ambulatory care in the United States. Archives
of Internal Medicine, 167, 1400–1405.

McLaney, M.A. and Hurrell, J.J.J., 1988. Control, stress,
and job satisfaction in Canadian nurses. Work and
Stress, 2, 217–224.

Middleton, B., Hammond, W.E., Brennan, P.F., and
Cooper, G.F., 2005. Accelerating U. S. EHR adoption:
how to get there from here. Recommendations based on
the 2004 ACMI retreat. Journal of the American Medical
Informatics Association, 12, 13–19.

Murray, M.D., Loos, B., Tu, W., Eckert, G.J., Zhou, X.H.,
and Tierney, W.M., 1999. Work patterns of ambulatory
care pharmacists with access to electronic guideline-
based treatment suggestions. American Journal of Health-
System Pharmacy, 56, 225–232.

Ohsfeldt, R.L., Ward, M.M., Schneider, J.E., Jaana, M.,
Miller, T.R., Lei, Y., et al., 2005. Implementation
of hospital computerized physician order entry systems
in a rural state: feasibility and financial impact. Journal
of the American Medical Informatics Association, 12, 20–
27.

Overhage, J.M., Evans, L., and Marchibroda, J., 2005.
Communities’ readiness for health information exchange:
the national landscape in 2004. Journal of the American
Medical Informatics Association, 12, 107–112.

Pizziferri, L., Kittler, A.F., Volk, L.A., Honour, M.M.,
Gupta, S., Wang, S., et al., 2005. Primary care physician
time utilization before and after implementation of an
eletronic health record: a time-motion study. Journal of
Biomedical Informatics, 38, 176–188.

Behaviour & Information Technology 19

Quinn, R., Seashore, S., Kahn, R., Mangion, T., Cambell,
D., Staines, G., et al., 1971. Survey of working
conditions: final report on univariate and bivariate
tables. Washington, D.C.: U.S. Government Printing
Office. Document No. 2916-0001.

Reeder, L.G., Schrama, P.G., and Dirken, J.M., 1973. Stress
and cardiovascular health: an international cooperative
study I. Social Science and Medicine, 7, 573–584.

Sainfort, F. and Carayon, P., 1994. Self-assessment of VDT
operator health: hierarchical structure and validity
analysis of a health checklist. International Journal of
Human Computer Interaction, 6, 235–252.

Seashore, S.E., Lawler, E.E., Mirvis, P., and Cammann, C..
eds. 1983. Observing and Measuring Organizational
Change: A Guide to Field Practice. New York: John Wiley.

Sittig, D.F., 1993. Work-sampling: a statistical approach to
evaluation of the effects of computers on work patterns
in healthcare. Methods of Information in Medicine, 32,
167–174.

Smith, M.J. and Carayon, P., 1995. New technology,
automation, and work organization – stress problems
and improved technology implementation strategies.
International Journal of Human Factors in Manufactur-
ing, 5, 99–116.

Thompson, T.G. and Brailer, D.J., 2004. The decade of
health information technology: delivering consumer-
centric and information-rich health care. Washington,
D.C.: U.S. Department of Health & Human Services.

Appendix

Activity Function Contact

A1, absent F1, accompanying patients C1, billing coordinator
A2, caring for patient F2, assisting physician, doctors or medical technician C2, doctor
A3, computer entry F3, billing activities C3, manager
A4, dictation F4, checking message C4, medical student
A5, meeting 1þ F5, data review and retrieval C5, medical technician
A6, meeting 3þ F6, distributing chart/master file/mail C6, nurse
A7, performing lab work F7, examination or treatment of patient C7, office staff
A8, phone F8, general clerical assistance/office task C8, other student
A9, preparing F9, maintaining equipments, instruments, supplies and medications C9, patient and patient representative
A10, reviewing check F10, maintaining lab reports C10, resident
A11, supervising F11, maintaining medical information system C11, self
A12, typing/writing/signing F12, maintaining patient’s information record C12, supervisor
A13, other F13, participating in resident and student education C13, other

F14, performing tests C14, radiographer
F15, Preparing for examinations and surgical procedures
F16, providing instruction/information
F17, purchasing and making inventory arrangement
F18, reporting problem
F19, scheduling
F20, training
F21, transcription
F22, other

Time Activity Function Contact Notes

13:30
14:00
14:30
15:00
15:30
16:00
16:30
17:00
17:30
18:00
18:30
19:00
19:30
20:00

Time leave:_________________________________________________

20 P. Carayon et al.

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

journal homepage: www.elsevier.com/locate/yjbin

A multi-level usability evaluation of mobile health applications: A case study

Hwayoung Choa,⁎, Po-Yin Yenb,c, Dawn Dowdingd, Jacqueline A. Merrilla,e, Rebecca Schnalla

a School of Nursing, Columbia University, New York, NY 10032, United States
b Institute for Informatics, Washington University School of Medicine in St. Louis, St. Louis, MO 63108, United States
c Goldfarb School of Nursing, BJC HealthCare, St. Louis, MO 63108, United States
d Division of Nursing, Midwifery and Social Work, University of Manchester, Manchester, United Kingdom
e Department of Biomedical Informatics, Columbia University, New York, NY 10032, United States

A R T I C L E I N F O

Keywords:
Mobile applications
Mobile Health
Usability evaluation
Health information technology
Information systems
Case study

A B S T R A C T

Objective: To report a methodological approach for the development of a usable mHealth application (app).
Materials and methods: This work was guided by a 3-level stratified view of health information technology (IT)
usability evaluation framework. We first describe a number of methodologies for operationalizing each level of
the framework. Following the description of each methodology, we present a case study which illustrates the use
of our preferred methodologies for the development of a mHealth app. At level 1 (user-task), we applied a card
sorting technique to guide the information architecture of a mobile HIV symptom self-management app, entitled
mVIP. At level 2 (user-task-system), we conducted a usability evaluation of mVIP in a laboratory setting through
end-user usability testing and heuristic evaluation with informatics experts. At level 3 (user-task-system-en-
vironment), usability of mVIP was evaluated in a real-world setting following the use of the app during a 3-
month trial.
Results: The 3-level usability evaluation guided our work exploring in-depth interactions between the user, task,
system, and environment. Integral to the findings from the 3-level usability evaluation, we iteratively refined the
app’s content, functionality, and interface to meet the needs of our intended end-users.
Discussion and conclusion: The stratified view of the health IT usability evaluation framework is a useful
methodological approach for the design, development, and evaluation of mHealth apps. The methodological
recommendations for using the theoretical framework can inform future usability studies of mHealth apps.

1. Introduction

Approximately two-thirds (66%) of Americans use mobile applica-
tions (apps) to manage their health [1], and nearly 165,000 mobile
health (mHealth) apps are now available in the Apple iTunes and An-
droid app stores in the US [2]. More than a few mHealth apps are de-
signed/developed with minimal end-user feedback, and continue to
proliferate with little evidence supporting user engagement of the app
[3,4]. For example, only 4% of mHealth apps providing breast-feeding
support had any evidence ensuring their use [3]. Apps are frequently
produced with poor design and inadequate consideration of the needs
of end-users [5]. Apps of poor quality that are disseminated can be
difficult to use, or misused or underutilized, and will ultimately fail to
accomplish their goals [4,6–9]. Thus, it is essential for apps to provide
the required functionality and also to ensure quality [10,11]. This

highlights the importance of usability evaluation of mHealth apps
throughout the development process – before and after prototyping
takes place [12–14]. More than 95% of mHealth apps have not been
tested [15] and existing studies have evaluated apps’ usability at a
certain single stage (mainly at an early stage) of development, and/or
without using a solid theoretical framework [16–18]. A usability as-
sessment which focuses on a single measure (e.g., time efficiency or
user acceptance) at a single stage (e.g., prototype) cannot capture the
complete usability of the system. The purpose of this paper is to de-
scribe a theoretically driven methodological approach for the devel-
opment of a mHealth app. The multi-level usability evaluation mapped
each methodology to a specific level in the conceptual framework.
Following our description of the framework and the relevant meth-
odologies, we present a case study to illustrate the operationalization of
the framework for the development of a mobile app.

https://doi.org/10.1016/j.jbi.2018.08.012
Received 28 December 2017; Received in revised form 2 July 2018; Accepted 22 August 2018

⁎ Corresponding author at: Columbia University, School of Nursing, 560 West 168th Street, New York, NY 10032, United States.
E-mail address: [email protected] (H. Cho).

Journal of Biomedical Informatics 86 (2018) 79–89

Available online 23 August 2018
1532-0464/ © 2018 Elsevier Inc. All rights reserved.

T

2. Methods/results

2.1. Theoretical framework: a stratified view of health information
technology (IT) usability evaluation

The Stratified View of Health Information Technology (IT) Usability
Evaluation Framework [19] provides a categorization of study ap-
proaches of usability assessments by three levels: user-task, user-task-
system, and user-task-system-environment (Fig. 1.1).

2.2. Methods for operationalizing framework

In the evaluation framework, level 1 targets system specification to
understand user-task interaction for system development. Level 2 ex-
amines the task performance to assess system validation and human-
computer interaction in a laboratory setting. Level 3 aims to in-
corporate environmental factors to identify system impact in a real-
world setting. It is of great importance that mHealth app developers be
not only aware of various usability methods, but also able to determine
which method is best suited to each level of the development of mobile
apps. Table 1 presents potential methods for usability evaluations and
the selected methods that are described in a case study.

Study context: This project designed, developed, and evaluated a
mHealth app to support symptom self-management for persons living
with HIV (PLWH). It was conducted as part of an Agency for Healthcare
Research and Quality initiative to incorporate the use of patient-cen-
tered outcomes research (PCOR) onto a mobile platform [17]. We de-
veloped mVIP (mobile Video Information Provider) by incorporating
evidence-based self-care strategies which had been previously dis-
seminated through a paper-based manual [18,19] into a mobile tool.

Level 1 (User-Task): To develop an effective mHealth app, it is ne-
cessary to start by incorporating users’ requirements into the app’s
design. The early feedback of potential users can improve the quality of
the system [20]. In our case study, a card sorting technique [21] was
selected for level 1 usability assessment since we had existing health
information (i.e., HIV-related symptoms and self-management strate-
gies from a paper-based HIV/AIDS symptom management manual)
[22]. Given that the card sorting technique is classified as ‘open’,
‘closed’, and ‘reverse’ card sorting, either online or in person, we chose
a reverse card sorting since it is well-suited for testing an existing
structure of categories (i.e., HIV-related symptoms) and sub-categories
(i.e., self-care strategies) [21,23]. Physical index cards were used in
person for the reverse card sort since with an online card sorting study,
researchers may miss out on the insights and additional comments end-
users can provide in person [24]. A card sorting technique is one of the

most effective methods for acquiring categorical and hierarchical data
about existing domains [25].

Level 2 (User-Task-System): Usability evaluations in a laboratory
setting are foundational to the success of achieving systems that meet
human-computer interaction principles. In our case study, two usability
evaluations: think-aloud protocols by end-users and heuristic evalua-
tion by experts [26,27], were chosen since they are two methods most
frequently used to guide system modification [28]. Moreover, eye-
tracking data were integrated and synthesized in the think-aloud pro-
tocols [29] since the use of eye-tracking has the potential to improve
usability of health IT by providing complementary objective data (i.e.,
responses provided by end-user during the think-aloud protocols) [30].

Level 3 (User-Task-System-Environment): System usability is closely
linked to the interaction of users performing tasks in the system within
a specified environment. Change in any of the components of user, task,
system, and environment may change the entire interaction and influ-
ence the usability of the system [31]. In our case study, an interview
with a survey [32] was selected since an interview is well-suited for
exploring and gaining an in-depth understanding of end-users’ experi-
ences, opinions, expectations, wishes, and concerns, especially after
they have used the technologies in a real-world setting.

2.3. Case study: a multi-level usability evaluation

To illustrate the operationalization of this framework, we present a
case study employing a 3-level usability evaluation for the development
of the mHealth app. Fig. 1.2 provides an outline of the 3-level usability
evaluation of mVIP.

2.3.1. Level 1 (User-Task): user-centered design
Sample: We recruited 20 PLWH from an academic medical center

and 4 community-based organizations in New York City between
December 2015 and May 2016. Inclusion criteria were: adults
(> 18 years of age) who were diagnosed with HIV; English-speaking;

Fig. 1.1. A stratified view of health information technology (IT) usability
evaluation.

Table 1
Usability evaluation methods by three levels.

Potential methods Selected methods

Level 1 • Interview
• Focus group/expert panel

meeting

• Questionnaire
• Use-case analysis/modeling
• Design task identification
• Card sorting technique

• Card sorting technique

Level 2 o End-user-based:

• Think-aloud protocols
+ other techniques/

technologies providing objective
cues

(e.g., video recording,
mouse-clicks, facial expression
coding, galvanic skin response,
electroencephalography, etc.)

• Field observation
• Time-and-motion study
• Task analysis
• Cognitive Task Analysis
• Interview
• Focus group
• Questionnaire

o Expert-based:

• heuristic evaluation
• Cognitive Walkthrough

o End-user usability testing:

• Think-aloud
protocols + eye-tracking

o Heuristic evaluation

Level 3 • Interview
• Focus group
• Questionnaire
• Survey

• Interview
• Survey

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

80

having at least 2 of 13 HIV-related symptoms in the past week; and a
cognitive state with acceptable responses for 6 items on a shortened
version of the Mini-Mental State Examination (MMSE) [33].

Procedures: A reverse in-person card sorting exercise [21] guided
the information architecture of mVIP. Users were presented with a pile
of cards; each card contained a symptom and self-management strategy
for ameliorating the symptom. There were a total of 154 self-manage-
ment strategies for 13 symptoms derived from the paper-based HIV/
AIDS symptom management manual [22]. Participants were first pro-
vided 13 index cards of symptoms and asked to select the index cards of
symptoms they experienced in the past week. Participants were then
provided with index cards of strategies for each symptom they chose.
The participants were asked to place the index cards of strategies in
order of individual priority applicable to the selected symptom. They
were allowed to place cards on an ‘irrelevant/unhelpful’ pile for stra-
tegies they thought were not relevant to the symptom or were unwilling
to try. Finally, participants were asked to add comments on a blank
index card if appropriate. At the end, all index cards were photo-
graphed (Fig. 2).

Data analysis: A hierarchy analysis established the rank order of
symptoms and strategies. Mean scores of the ordinal numbers of the
cards for strategies were calculated for each of the symptoms. A lower
mean score indicated a higher priority order of strategies for each
symptom.

Results: Rank order of the symptoms and self-management strate-
gies was established and reported elsewhere [34]. 85% (N = 17) of
participants reported fatigue and 60% (N = 12) difficulty sleeping in
the prior 7 days. 3 self-management strategies were excluded in re-
sponse to participants identifying as ‘irrelevant/unhelpful’. Findings
were incorporated into the information architecture of mVIP. The rank
order of the 13 symptoms and 151 self-management strategies de-
termined the order of appearance to end-users of the mVIP app, with
higher-ranked symptoms and strategies appearing first (Fig. 3).

2.3.2. Level 2 (User-Task-System): usability evaluation in a laboratory
setting

mVIP was designed and implemented by software developers at
Northwestern University based on findings from the card sorting study.
We conducted two types of usability evaluations of the mVIP alpha
version. The first end-user usability testing examined task performance
by PLWH, and the second heuristic evaluation assessed the user inter-
face by usability experts.

2.3.2.1. End-user usability testing. We conducted end-user usability
testing with an eye-tracking and retrospective think-aloud method [29].

Sample: We recruited 20 PLWH (10 Android and 10 iPhone users)
from June to July 2016. Participants met the inclusion criteria for the
level 1 study and also self-identified as a heavy smartphone user, de-
fined as a smartphone user for more than 1 year who also used mobile
apps more than 3 h/day on average [35]. Heavy smartphone use was
necessary to ensure that usability issues identified while using mVIP
were not related to lack of technology skills, but from actual short-
comings related to its usability. Participants who wore bifocal/pro-
gressive glasses were excluded since these types of glasses affect the
precision of the gaze estimation while collecting participants’ eye-
tracking data [36].

Procedures: End-user usability testing was comprised of the fol-
lowing 2 processes: eye-tracking while using the app and a retro-
spective think-aloud protocol [29]. First, participants were provided
with a use case scenario designed to determine usability of mVIP and
asked to complete 2 app sessions using mVIP (Table 2). While per-
forming the tasks, participant’s eye movements and smartphone screen
were video-recorded using Tobii X2-60 with a mobile device stand with
embedded camera and microphone (Fig. 4) [36]. After completing the
tasks, participants were asked to watch a recording of their task per-
formance that depicted their eye movements overlaid on the app
screen. They were encouraged to retrospectively think-aloud and asked
to verbalize their thoughts about the tasks they completed while
watching the recording. All verbalizations were audio-recorded.

Data analysis: Data analysis was based on the Tobii audio/video-
recordings. Gaze plots were created from screen-recordings synchro-
nized with eye movements. Participants’ vocalizations from the audio-
recordings were transcribed verbatim. Notes of critical incidents,
characterized by comments, silence, repetitive actions, and error mes-
sages were compiled from the recordings. Task performance time ana-
lysis: The mean performance time of each task was calculated and
compared among participants with/without trouble using a two-sample
t-test (α = 0.05). Participants with trouble in this study were defined as
those who received error messages during the app testing and self-re-
ported difficulties during the think-aloud protocol. Eye movement ana-
lysis: Gaze plots depicting participants’ eye movements were reviewed
in conjunction with notes of critical incidents. The gaze plots were
compared among participants with/without trouble. Content analysis:
Free text was excerpted from the transcripts and coded based on 9
concepts of the Health IT Usability Evaluation Model (Health-ITUEM)

Level 1 (user-task):
user-centered design

card sorting technique

Level 2 (user-task-system):
usability evaluation in a laboratory setting

end-user usability testing
heuristic evaluation

Level 3 (user-task-system-environment):
usability evaluation in a real-world setting

survey
in-depth interview mVIP Beta version

mVIP Alpha version

Fig. 1.2. An outline of studies: 3-level usability evaluation.

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

81

[31], and the 9 codes were broken into positive, negative, and neutral
codes.

Results: In task 1 at the first session, 50% of the participants
(N = 10) had trouble and mean time for completion of the task was
241 s, whereas the mean time for those without trouble was 49.5 s
(p = 0.002). The mean time to perform each task during the second
session was much lower than during the first session, suggesting that
the app is highly learnable.

Based on eye-tracking data of a larger red circle or longer red lines
for participants with trouble, compared to those shown while a parti-
cipant was performing a task without trouble (Fig. 5.1), we identified
several participants who encountered challenges navigating the app.
These challenges were specific to ambiguity regarding the ‘Continue’
and ‘Log-in’ button, which showed as an extremely long eye fixation
and distractive eye scan path up and down the smartphone screen
(Fig. 5.2).

Sample excerpts for each code of the 9 Health-ITUEM concepts are
presented in Table 3. Based on end-users’ recommendations and in-
tegration of the results, the content, functionality, and interface of

mVIP were refined. For example, the error message ‘Please check your
credentials’ was changed to ‘The email address or password you entered
is not valid’ since the term ‘credentials’ was unfamiliar to several par-
ticipants. We added instructions for ‘how our app works’ on the home
page. Several participants suggested changing the main logo of mVIP to
look more informative and professional (Fig. 6).

2.3.2.2. Heuristic evaluation. Sample: 5 informatics experts
participated in the heuristic evaluation of the web version of mVIP.
Inclusion criterion were: a minimum of a master degree in the field of
informatics and training in human computer interaction [37].

Procedures: Experts were provided with the same use case scenario
employed during the end-user usability testing. Each expert was en-
couraged to explore the user interface of mVIP at least twice and to
think-aloud while they performed the evaluation. The process was re-
corded using Morae™ software [38]. Following completion of the tasks,
evaluators rated the severity of the violations using a paper-based
heuristic evaluation form based on Nielsen’s heuristics [39]. Usability
problems were rated into 5 categories: no problem (0), cosmetic

Fig. 2. Sample picture of card sorting activities.

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

82

problem only (1), minor problem (2), major problem (3), and usability
catastrophe (4).

Data analysis: Content analysis of the transcripts of the experts’
vocalizations during the think-aloud was organized by the usability
factors of the 10 heuristics. The mean severity scores of the identified
heuristic violations were calculated for each principle.

Results: Mean scores and sample comments were organized into
Nielsen’s 10 usability heuristics [26,27]. Findings are reported else-
where [40]. The content, functionality, and interface of mVIP were
refined based on the evaluators’ recommendations. For example, the
term ‘Dashboard’ was changed to ‘VIP Home,’ which was a more fa-
miliar term to our end-users. To make the app’s functionality more
generalizable, we added a response option of ‘Didn’t try’ in addition to
the ‘Yes/No’ options when assessing helpfulness of previously suggested
strategies.

The 143 videos created using the GoAnimate™ software [41] to
present each of the 143 self-management strategies, were inserted into

Fig. 3. mVIP alpha version.

Table 2
Task included in a use case scenario.

Session_1 Session_2

Task 1 Log-in Task 1 Log-in
Task 2 Update a password Task 2 N/A
Task 3 Start a session – Get strategies

for the first two symptoms,
fatigue and difficulty sleeping

Task 3 Start a session – Get
strategies for one symptom,
difficulty sleeping

Task 4 Review the recommended
strategies

Task 4 Review the recommended
strategies

Fig. 4. Sample picture of eye-tracking app testing (taken of a research team
member with permission).

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

83

the refined mVIP with the existing text strategies (i.e., 63 strategies
were reworded and 8 were removed). While mVIP was initially de-
signed as a native app for mobile devices, the refined mVIP (i.e., mVIP
Beta version) was developed as a mobile web-app, due to different
capabilities between Android and iOS platforms (Fig. 7).

2.3.3. Level 3 (User-Task-System-Environment): usability evaluation in a
real-world setting

After mVIP was refined based on the second level usability eva-
luation, we implemented a 3-month RCT to test the feasibility of using
mVIP for improving symptoms in PLWH. The intervention group was
provided with self-management strategies for self-reported symptoms
through the app, while the control group self-reported symptoms but
was not provided with any strategies. A survey and interview were
conducted to evaluate the usability of the app by identifying user-task-
system-environment interaction.

Sample: Eligibility criteria were the same as that in the level 1
study. 80 PLWH who own a smartphone/tablet were randomized (i.e.,
40 in the intervention and 40 in the control) and 76 completed the 3-
month RCT between December 2016 and June 2017. Of the 76 PLWH,
10 PLWH were recruited for interviews.

Procedures: All participants in the RCT were asked to complete the
Health IT Usability Evaluation Scale (Health-ITUES) [42] survey ad-
ministered via Qualtrics® [43] at baseline and 3-month follow-up. At
the end of the RCT, a research coordinator facilitated 10 one-on-one
interviews using a semi-structured interview guide designed based on
the Health-ITUEM [31]. Participants were encouraged to talk about
their experiences, perceptions, and satisfaction of their app use. The
interviews were audio-recorded. Data collection continued until sa-
turation of themes was reached.

Data analysis: The Health-ITUES consists of 20 items rated on a 5-
point Likert scale from strongly disagree (1) to strongly agree (5) [42].
The overall Health-ITUES scores between baseline and follow-up visit
were analyzed using a linear mixed model controlled for age, sex, race,
education, and CD4 count (α = 0.05). We used thematic analysis to
explore patterns and themes that emerged across interviews through
NVivo™ software [44].

Results: Overall, participants in both intervention and control
groups rated the usability of the mVIP app as being high. There was no
significant difference in the overall Health-ITUES scores across the
groups at baseline and follow-up. The mean scores of the overall
Health-ITUES were more than 4.19 (SD = .50) of 5.00 (i.e., where the
higher the response, the higher the subject’s usability satisfaction with
the app), indicating a high user satisfaction of the mVIP app.

A total of 15 themes were identified from interviews. Of the sub-
jective constructs of Health-ITUEM, 9 themes identified in the inter-
vention group related to Perceived usefulness and 6 themes identified in
the control group related to Perceived ease of use (Table 4). Findings
from the interviews showed that first, mVIP is useful for HIV-related
symptom self-management and has the potential for being used as a
communication tool with healthcare providers; second, mVIP is easy to
use to monitor symptom experience over time. At the same time, par-
ticipants suggested mVIP be more sensitively tailored based on years
from initial diagnosis of HIV, an individuals’ age, and conditions.

3. Discussion

In order to ensure usability throughout the system development
process, we employed a 3-level usability evaluation of a mHealth app
(mVIP), exploring potential interactions between the user, task, system,
and environment that was guided by the Stratified View of Health IT
Usability Evaluation Framework [19]. Methods included a card sorting
technique, eye-tracking and retrospective think-aloud, a heuristic eva-
luation, a survey, and in-depth interview. Integral to the findings from
the 3-level usability evaluation, we iteratively refined the app’s content,
functionality, and interface, and we found the app was rated as highly
usable by our end-users. Findings from this study are unique in that the
mVIP app was developed by delivering evidence-based health in-
formation through a mobile health platform and assessed its usability at

Fig. 5.1. Sample gaze plot for a participant without trouble.

Fig. 5.2. Sample gaze plots for a participant with trouble.

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

84

every level throughout the development process.
Suboptimal usability is a major obstacle to technology adoption and

poor usability is a primary reason for discontinuing the use of mHealth
technology [45]; therefore, usability factors must be considered before
and after prototyping takes place to support the quality of the tech-
nology and the end-user experience [12,13]. Despite its importance,
most mHealth apps are released to the public without sufficient scien-
tific effort devoted to their design, development, and evaluation [3,46].
At the same time, there has been no clear methodological approach to
using a theoretical framework in apps’ usability studies [16,17,19]. To
overcome this challenge, we used several usability evaluation meth-
odologies at each stage of the development process to operationalize
the Stratified View of Health IT Usability Evaluation Framework. It is

Table 3
Health-ITUEM concepts, codes, and representative quotes.

Concepts Description and representative quotes

Error prevention mVIP offers error management, such as error messages as feedback, error correction through undo function, or error prevention, such as
instructions or reminders, to assist users in performing tasks.

+ Error prevention “It was very nice that it told me that I made a mistake instead of going right into it even if I did make a mistake. That alerted me that I put in the wrong
email. The error message was pretty clear.”

− Error prevention “I didn’t understand what ‘please check your credentials’ meant. I didn’t understand that at all. What exactly did it mean?” (unclear error messages)
“It tells me if the password failed, and it tells me it was updated, so that’s wrong. Because it’s giving me two different messages. That’s pretty confusing.”
(contradictory information in error messages)
“It might be beneficial to have a back feature, because you know, you never know and you might be just rushing through, or something like that.” (back
button)
“Give it a back button on certain pages. Maybe give it a menu screen. maybe like three lines right here that you can hit the menu and maybe it will bring up
dashboard” (home menu)
“Well, for those who don’t know how to use apps, they would need more instructions. They would need more instructions, more simple instructions for them
to adapt to. For them not to frustrated, but some way of making it fun, to where they will enjoy using the app, or enjoy getting into the app. You know,
instead of saying, Okay, alright, I’ve been in this app. What am I supposed to do?” (instruction for how the app works)

Completeness mVIP is able to assist users to successfully complete tasks.
+ Completeness Task success rate: 80% (N = 16)
− Completeness Task failure rate: 20% (N = 4)
Memorability Users can remember easily how to perform tasks in mVIP.
+ Memorability “I remember the steps. I logged in and created a new password after I logged in. And then, I had started a session, which asked me about various symptoms

that I may have experienced in the past week.”
− Memorability None
Information needs The information content offered by the mVIP for basic task performance, or to improve task performance.
+ Information needs “You can use the app to gain a lot of information. If you’re not feeling too well and you have certain symptoms that match with the ones that are on this

app, then it would give you helpful information right then and there.”
− Information needs “The one thing I had to question is, these 13 strategies are related to what particular? They don’t tell you what category. Is it fatigue or…what?”
Flexibility/Customizability mVIP provides more than one way to accomplish tasks, which allows users to operate system as preferred.
+ Flexibility/Customizability “There are different kinds of expression on the avatar’s face. So I also look at that as well. It is meaningful. The reason being because if you’ve a man and

thinking of yourself, it’s good to remember okay, this is me, so they’re talking about me.”
− Flexibility/Customizability “I would like to save it there…at least the ID, then the password. I can have my own password, but at least the ID can be saved there so I don’t have to be

typing the ID and then the password all the time.” (option of saving ID&PW)
“I don’t see any here that represent me, but you at least have a choice. I don’t think any of them. You should be able to create your own avatar, make it look
like you. Because I have an app and I made it look like me.” (option of choosing an avatar)
“If I can review this information before I fax it off or send it off to someone else, email it to someone else.” (option of Fax/email)

Learnability Users are able to easily learn how to operate mVIP.
+ Learnability “It was very easy to follow after the first use. I found the questions easy. They were precise and they were straight to the point. Once I answered the

question, the suggestions they gave were easy. They were easy to follow, so I think I can self-manage myself quite well with this app.”
− Learnability “I didn’t see the continue button. I thought this system was automatic, when I checked ‘yes,’ I thought it would move on.” (continue button)

“Well, the way I saw it was that it was checked already, so I guessed it continued, leave it as is because there is no change. Unless they were both blank, then
there is ‘no’ and ‘yes,’ and then I’ll check. But since it was checked for ‘yes,’ I left it at that. So then I hit the Continue. But I didn’t think to hit it because it
was already checked. If it had been blank, then I would have checked it. But I didn’t see it unchecked to put a check in. I left it as it was and hit Continue.”
(checkmark)

Performance speed Users are able use mVIP efficiently.
+ Performance speed “It was very short so it was real quick. I know I had to go with ‘yes’ for most of them and ‘no’ with everything else.”
− Performance speed None
Competency Users are confident in their ability to perform tasks using mVIP, based on social cognitive theory.
+ Competency “I would feel very confident, very, very confident, because I mean it gives you pretty much straightforward strategies to try. Like I said, trial and error, so

whichever ones do work, then do, if none of them work, then it’s time to see the doctor. But chances are if everything else is going the way it’s supposed to go,
as far as you taking care of yourself, then you know, your symptoms should be able to be relieved with these strategies.”

− Competency “And I think I was done. I wasn’t sure if there was anything else to be done that didn’t get done. So that’s why I didn’t understand that question. Was it
complete, or is there something else I have to follow through with? But I guessed I was done. So I was confused as to whether there was something else I
needed to do.”

Other outcomes Other mVIP-specific expected outcomes representing higher level of expectations (uses of non-phone app technology (i.e., phone, books), non-
mobile resources (i.e., parents, friends, siblings), other health related entities not directly related to the usability of mHealth (outside of study
protocol))

+ Other outcomes “I mean, not just mine, but I mean everybody that the virus, it could change their quality of life, giving them a better quality of life, you know, it’s kind of
like being your own doctor these days, without having to go to the doctor. And, being able to take better care of yourself through the app.”

− Other outcomes “I don’t think the app can very much change my life.”

Fig. 6. Change of mVIP main logo.

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

85

critical to select the most appropriate evaluation techniques that best
meet the study aims at every level of the system development process
and ultimately achieve the goals of the system. While the methods that
we selected are just a few of many possible methods, this mapping can
serve as a methodological guide for the multi-level usability evaluation
for development of a user-facing mHealth technology.

Card sorting technique: This technique is well-suited for in-
corporating existing health information into a mobile app, particularly
for the evidence-based information design phase since it generates an
overall information structure and suggestions for navigation, menus,
and possible taxonomies of the system [47]. In our case the technique
helped us identify the information priorities of our intended end-users
and establish the app’s components before prototyping. Despite these
benefits, there have been no usability studies using card sorting activ-
ities for mHealth app development. Good information architecture is
key to developing, easy to use, and intuitive technology [48–50]. Ap-
plication of the card sorting technique to the app design process is user-
centered and innovative. We recommend that it be integrated into the
mHealth app development process.

End-user usability testing and heuristic evaluation: In our case study,
two usability evaluations were conducted in a laboratory setting to
capture different usability perspectives from end-users and experts.
First, end-user usability testing showed significant differences in task
performance duration between participants who experienced difficul-
ties and those that did not. These usability problems were identified by
using both eye-tracking and think-aloud protocols. When a participant
had trouble with a task, (e.g., finding a certain button) long eye fixation
or distractive eye movements were found upon replay of the screen-
recordings. We learned the reason for their unusual eye movements
using the retrospective think-aloud protocol. This combination provides
valuable information that surpasses usability problems reported
without a cue, which are often biased [51,52]. A stand-alone think-
aloud protocol may fail to identify additional objective cues that

provide insight into participants’ expectations about where information
should be located and their level of confidence about information found
[53]. The usefulness we have documented leads us to recommend in-
corporating these methods in combination for future mHealth app de-
velopment. Second, while our heuristic evaluators identified similar
usability issues to those identified by our end-users, they were more
likely to focus on ‘making things work’ in a natural and logical order
[28,54]. For example, the experts identified the usability factor ‘match
between the system and the real world’ regarding the helpfulness assess-
ment question for each of the strategies suggested in the previous ses-
sion. They suggested that an additional response option for end-users
who did not try that particular strategy be included. They suggested
that no strategies be offered for symptoms the user marked ‘not both-
ersome’. To identify usability problems of different sorts and scope, we
recommend combining usability evaluations from both end-user and
expert perspectives as an effective and thorough approach for evalu-
ating mHealth apps prior to real-world deployment.

Survey and interview in a real-world setting: Numerous studies have
evaluated usability only in a laboratory setting [55]. Conducting us-
ability testing in a laboratory setting presents threats to external va-
lidity specifically overlooking usability issues related to the interaction
between user, task, system, and environment. In our work, inclusion of
interviews following testing of the app during participants’ everyday
lives allowed us to measure users’ actual experience with the mVIP app
in daily life. The overall usability scores as measured by the Health-
ITUES were high at baseline, and the app was perceived as highly
usable over time. Eventually, the app was efficacious at improving
outcomes in a 3-month feasibility trial [56]. These results are an im-
portant strength of our study.

It is vital to take into consideration specific aspects of a mobile
device when selecting evaluation methods during the app development
process. Identification of usability evaluation methods are unique for
mobile devices because they have small screens. For example, a

Fig. 7. mVIP beta version.

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

86

particular usability problem related to the screen size identified by eye
movements collected during the participant’s app testing process and
verbalizations collected during the retrospective think-aloud protocol
was the placement of the ‘Continue’ button. Since the button was placed
under the response options because of a small mobile screen, partici-
pants were required to scroll down to find the button. The usability
problem was identified as a result of using the eye-tracking method
with a mobile device stand, which captured the smartphone screen and
the user interactions. To resolve the usability issue related to the
‘Continue’ button being hidden, we transitioned the mVIP app from a
native app to a web-app optimized for mobile devices.

Concerns specific to the privacy and confidentiality of transmission
and storage of health information via a mobile device emerged during
our evaluation. Several participants expressed preferences for a func-
tion of saving their password if it is stored, whereas a few participants
raised privacy concerns about the security of the password. This was
specific to our study population where stigma associated with an HIV
diagnosis persists [57]. Findings from this study demonstrate the value

of rigorous usability evaluations with inclusion of end-users at every
stage and support our recommendations for this approach.

4. Limitations

We faced several challenges when collecting eye-tracking data. To
avoid losing participants’ eye movements, we dimmed the lights and
asked the participants to maintain the same position. These specifica-
tions may be tiring to participants and time-consuming for researchers.
Since there has been no rigorous standard of measuring what is con-
sidered a good eye pattern, it was hard to set standards for a given
interface [58]. Despite these limitations, eye-tracking proved to be a
valuable tool to explore usability issues in conjunction with a think-
aloud protocol as opposed to a stand-alone method.

5. Conclusions

In this paper, we presented methodologies for operationalizing the

Table 4
Themes and quotes of content analysis from the interviews.

Intervention Group:
Usefulness of mVIP and additional user expectations

Theme I-1. Usefulness for information needs – symptom problem solving
“It was very helpful (for reducing my symptoms), for different situations.
It (app) gave me a suggestion and I tried it and got better.”

Theme I-2. Usefulness for interaction needs with healthcare providers
“Sometimes you go to the doctor and forget the symptoms you have been going through. If you bring this app to the doctor and go to the review part (in the app), this app could show the stuff,
everything, you’ve been going through.”

Theme I-3. Usefulness of mobile app format as a perceived facilitator
“It’s very convenient because I can use it almost anywhere. While I’m in public transportation, on the buses, at the clinic, at home…everywhere.”

Theme I-4. Additional preference of strategy design (text & video with sounds)
“What the video is saying can give you a little more insight on how to do things and how to go about them and whatever, but if you don’t have the audio it’s like, ‘Just let me read this, click
and answer and just go on to the next one.’ So, the audio would help.”

Theme I-5. Additional preference of available language
“Because my community, the Latino community in NYC is very big and increasing in HIV. The Spanish community… It’s very important the Latino community can comprehend the app. You
can get two options, in English or in Spanish.”

Theme I-6. Intrinsic motivation of the frequency of app use (for enjoyment)
“I use it (app) 3 times a day sometimes. Just playing around.
Just to play a little game. I’m playing a game and I’m tired of the game, I just start to do the app instead. I don’t have anything else to do. I don’t think more (app use) would be helpful. Once a
week is right on point.”

Theme I-7. More symptoms and strategies needs according to years of diagnosis
“I think it (strategies) will help people that are newly diagnosed that they are dealing with something new that they’re not familiar with, something that they didn’t expect to have and, as a
newly diagnosed person, they go through a lot of confusion, a lot of questions in their mind. We need something geared towards where we are right now because we have much more issues
than the app is talking about for us.”

Theme I-8. More individually-sensitive self-tailoring symptom management
“Everyone is different when it comes to their health. It (strategies) was less personal. We may not have the same status. We all should have our own things that we’re dealing with in
(personal) life…”

Theme I-9. More communication needs with social groups
“I think you should create a network where we can network among each other within the app. (So) you can respond to someone and you can say, ‘I’m feeling the exactly same way today (like
you).’ Or, if someone is not feeling well you can send a message back like, ‘(tell me). Let me see how you’re feeling. We can share the feelings (because we are all HIV+).’”

Control Group:
Ease of mVIP use to track symptoms but also acknowledged its deficits

Theme C-1. Easy app as a regular tool to facilitate self-awareness of symptoms
“I like to listen to my body and it made me more aware of what’s going on with me. I like the app because it’s simple and easy to use and there were something that were listed in the app that I
had no idea that were related to my HIV. So, it caused me to listen more closely to what’s going on with me.”

Theme C-2. Lack of action planning of symptom self-management
“It was easy (to use the app) but I thought there could be another portion that would deal with stress (symptoms).”

Theme C-3. Lack of symptom summary to share with healthcare providers
“In regards to answering yes or no, and at the end. A short summary of what our symptoms were… when you see your doctor it just totals the graphs (charts) down and then he can see what’s
going on me (symptoms)… We can build that kind of provider relationship (using symptom report summary).”

Theme C-4. Tedious experience of the repeated questions
“The app, it kept repeating itself over and over again. It was like the same thing (questions) over and over, so it got kind of boring for me.”

Theme C-5. Extrinsic motivation of the frequency of app use (for rewards)
“Once a week is good. (But I used the app) At least twice a week because I want to build up my chances for being accepted for the research study next time. For the research study…”

Theme C-6. Appraisal of ease of use vs. security of the password
“9 out of 10 times, I forgot the password. The app had a little button that says: remember me. Therefore, I didn’t have to remember my password. It was very easy to use.”
“For me, I never do that. Like, speaking to the gentleman who had his phone stolen – I have the experience. If you have that ‘remember me’ and somebody accesses your phone…let’s say you
have it on your bank account (because we usually use the same passwords). They can immediately see what your bank account level is, they have access to your HIV.”

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

87

Stratified View of Health IT Usability Evaluation Framework [19] and a
case study to illustrate the use of these methods for the development of
a mHealth app, specifically focusing on HIV symptom self-management.
The operationalization of the Stratified View of Health IT Usability Eva-
luation Framework [19] in the context of mHealth technology can pro-
vide a methodological approach for future mHealth apps’ development.
This scientific and systematic approach to usability evaluations may
encourage other researchers to use an evidence-based evaluation fra-
mework when planning and designing their usability studies for the
development of mHealth apps and to choose the best evaluation
methods applicable to the goal of the technology.

Conflict of interest

The authors declare that they have no conflicts of interest in the
research.

Acknowledgements

This research was supported by the Agency for Healthcare Research
and Quality under award number R21HS023963 (PI: Schnall). The
content is solely the responsibility of the authors and does not ne-
cessarily represent the official views of the Agency for Healthcare
Research and Quality.

References

[1] Makovsky, Fifth Annual “Pulse of Online Health” Survey Finds 66% of Americans
Eager To Leverage Digital Tools To Manage Personal Health, 2015 (cited). Available
from: <http://www.prnewswire.com/news-releases/fifth-annual-pulse-of-online-
health-survey-finds-66-of-americans-eager-to-leverage-digital-tools-to-manage-
personal-health-300039986.html>.

[2] IMS Institute for Healthcare Informatics IIfHI, Patient Options Expand as Mobile
Healthcare Apps Address Wellness and Chronic Disease Treatment Needs, 2015
(cited). Available from: <http://www.imshealth.com:90/en/thought-leadership/
ims-institute/reports/patient-options-expand-as-mobile-healthcare-apps-address-
wellness-and-chronic-disease-treatment-needs>.

[3] A. Roess, The promise, growth, and reality of mobile health — another data-free
zone, N. Engl. J. Med. 377 (21) (2017) 2010–2011.

[4] M. Maguire, Methods to support human-centred design, Int. J. Hum. Comput. Stud.
55(4) (2001) 587–634 2001/10/01.

[5] T. McCurdie, S. Taneva, M. Casselman, et al., mHealth consumer apps: the case for
user-centered design, Biomed. Instrum. Technol. Fall (Suppl) (2012) 49–56.

[6] S. Hamine, E. Gerth-Guyette, D. Faulx, B.B. Green, A.S. Ginsburg, Impact of
mHealth chronic disease management on treatment adherence and patient out-
comes: a systematic review, J. Med. Int. Res. 17 (2) (2015) e52.

[7] M.L. Meuter, A.L. Ostrom, R.I. Roundtree, M.J. Bitner, Self-service technologies:
understanding customer satisfaction with technology-based service encounters, J.
Market. 64 (3) (2000) 50–64.

[8] S. Kumar, W.J. Nilsen, A. Abernethy, et al., Mobile health technology evaluation:
the mHealth evidence workshop, Am. J. Prevent. Med. 45 (2) (2013) 228–236
2013/08/01/.

[9] R. Schnall, J.P. Mosley, S.J. Iribarren, S. Bakken, A. Carballo-Diéguez, W. Brown III,
Comparison of a user-centered design, self-management app to existing mHealth
apps for persons living with HIV, JMIR mHealth uHealth 3 (3) (2015) e91.

[10] E. Folmer, J. Bosch, Architecting for usability: a survey, J. Syst. Softw. 70 (1–2)
(2004) 61–78.

[11] P. Bengtsson, N. Lassing, J. Bosch, Vliet Hv, Analyzing software architectures for
modifiability, 2000.

[12] A. Holzinger, Usability engineering methods for software developers, Commun.
ACM 48 (1) (2005) 71–74.

[13] B. Shackel, Usability—context, framework, definition, design and evaluation, in:
S. Brian, S.J. Richardson (Eds.), Human Factors for Informatics Usability,
Cambridge University Press, 1991, pp. 21–37.

[14] R. Schnall, M. Rojas, S. Bakken, et al., A user-centered model for designing con-
sumer mobile health (mHealth) applications (apps), J. Biomed. Inform. 60 (Apr)
(2016) 243–251.

[15] B. Furlow, mHealth Apps May Make Chronic Disease Management Easier, Retrieved
January, 2012, vol. 10, 2013.

[16] Y. Park, A pedagogical framework for mobile learning: categorizing educational
applications of mobile technologies into four types, Int. Rev. Res. Open Distrib.
Learn. 12 (2) (2011) 78–102.

[17] Z. Ping, R.V. Small, G.M.V. Dran, S. Barcellos, Websites that satisfy users: a theo-
retical framework for Web user interface design and evaluation, in: Proceedings of
the 32nd Annual Hawaii International Conference on Systems Sciences 1999 HICSS-

32 Abstracts and CD-ROM of Full Papers; 1999 5-8 Jan. 1999, 1999, 8pp.
[18] J. Horsky, K. McColgan, J.E. Pang, et al., Complementary methods of system us-

ability evaluation: surveys and observations during software design and develop-
ment cycles, J. Biomed. Inform. 43(5) (2010) 782–790 2010/10/01/.

[19] P.-Y. Yen, S. Bakken, Review of health information technology usability study
methodologies, J. Am. Med. Inform. Assoc.: JAMIA 19 (3) (2012) 413–422.

[20] B. Peischl, M. Ferk, A. Holzinger, The fine art of user-centered software develop-
ment, Software Qual. J. 23 (3) (2015) 509–536.

[21] J. Nielsen, Card Sorting to Discover the Users’ Model of the Information Space,
1995. Obtained from: <http://www useit com/papers/sun/cardsort>.

[22] University of California SFSoN, Symptom Management Strategies: A Manual for
People Living with HIV/AIDS, 2004.

[23] C.L. Paul, A modified Delphi approach to a new card sorting methodology, J.
Usability Stud. 4 (1) (2008) 7–30.

[24] M. Hawley, Extending Card-Sorting Techniques to Inform the Design of Web Site
Hierarchies, 2008 (cited). Available from: <http://www.uxmatters.com/mt/
archives/2008/10/extending-card-sorting-techniques-to-inform-the-design-of-web-
site-hierarchies.php#sthash.CAWNpz9Y.dpuf>.

[25] N.A.M. Maiden, G. Rugg, ACRE: selecting methods for requirements acquisition,
Software Eng. J. 11 (3) (1996) 183–192.

[26] J. Nielsen, Finding usability problems through heuristic evaluation, Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems, ACM, Monterey,
California, USA, 1992, pp. 373–380.

[27] J. Nielsen, Heuristic evaluation, Usab. Inspect. Meth. 17 (1) (1994) 25–62.
[28] T.-Y. Lai, Iterative refinement of a tailored system for self-care management of

depressive symptoms in people living with HIV/AIDS through heuristic evaluation
and end user testing, Int. J. Med. Inf. 76 (Supplement 2) (2007) s317–s324.

[29] Z. Guan, S. Lee, E. Cuddihy, J. Ramey, The validity of the stimulated retrospective
think-aloud method as measured by eye tracking, in: Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, ACM, Montal, Qubec,
Canada, 2006, pp. 1253–1262.

[30] O. Asan, Y. Yang, Using eye trackers for usability evaluation of health information
technology: a systematic literature review, JMIR Human Factors 2015 Jan-Jun 04/
14 11/20/received 12/11/rev-request 12/23/revised 03/01/accepted;2(1):e5.

[31] P.-Y. Yen, Health Information Technology Usability Evaluation: Methods, Models,
and Measures, 2010.

[32] N. Mays, C. Pope, Qualitative Research in Health Care, Wiley Online Library, 1996.
[33] M.F. Folstein, S.E. Folstein, P.R. McHugh, “Mini-mental state”. A practical method

for grading the cognitive state of patients for the clinician, J. Psychiatr. Res. 12 (3)
(1975) 189–198.

[34] Hwayoung Cho, Lena Milian, Rebecca Schnall, Card sorting of symptom self-man-
agement strategies to inform the development of a mHealth App in underserved
Persons Living with HIV, in: AMIA 2016 Annual Symposium; 2016, 2016.

[35] D. Cáliz, X. Alamán, Usability evaluation method for mobile applications for the
elderly: a methodological proposal, in: L. Pecchia, L.L. Chen, C. Nugent, J. Bravo
(Eds.), Ambient Assisted Living and Daily Activities: 6th International Work-
Conference, IWAAL 2014, Belfast, UK, December 2–5, 2014 Proceedings, Springer
International Publishing, Cham, 2014, pp. 252–260.

[36] Tobii Technology I. Tobii Technology, Stockholm, Sweden, 2016 (cited). Available
from: <http://www.tobiipro.com/product-listing/tobii-pro-x2-60/>.

[37] S. Po, S. Howard, F. Vetere, M.B. Skov, Heuristic evaluation and mobile usability:
bridging the realism gap, in: S. Brewster, M. Dunlop (Eds.), Mobile Human-
Computer Interaction – MobileHCI 2004: 6th International Symposium, MobileHCI,
Glasgow, UK, September 13–16, 2004 Proceedings, Springer, Berlin, Heidelberg,
2004, pp. 49–60.

[38] TechSmith, Morae Usability and Web Site Testing: TechSmith Corporation, 1995
(cited). Available from: <http://www.techsmith.com/products/morae/default.
asp>.

[39] T.J. Bright, S. Bakken, S.B. Johnson, Heuristic evaluation of eNote: an electronic
notes system, in: AMIA Annual Symposium proceedings AMIA Symposium, 2006,
pp. 864.

[40] H. Cho, H. Rojas, C. Fulmer, R. Schnall, Usability evaluation of a prototype mHealth
app for symptom self-management in underserved persons living with HIV, in:
AMIA 2017 Annual Symposium; 2017, 2017.

[41] GoAnimate, Corporation, GoAnimate Corporation, 2007.
[42] P.Y. Yen, D. Wantland, S. Bakken, Development of a customizable health IT us-

ability evaluation scale, in: AMIA Annual Symposium proceedings AMIA
Symposium, 2010 Nov 13, 2010, pp. 917–921.

[43] Qualtrics, Provo, Utah, USA, 2005.
[44] QSR International Pty Ltd. V, Australia, NVivo qualitative data analysis Software,

Version 11, 2015.
[45] A.M. Vuong, J.C. Huber Jr., J.N. Bolin, et al., Factors affecting acceptability and

usability of technological approaches to diabetes self-management: a case study,
Diabetes Technol. Ther. 14 (12) (2012 Dec) 1178–1182.

[46] W. Nilsen, S. Kumar, A. Shar, et al., Advancing the science of mHealth, J. Health
Commun. 17(sup1) (2012) 5–10 2012/05/02.

[47] D. Spencer, Card sorting: a definitive guide, 2004 (cited). Available from: <http://
boxesandarrows.com/card-sorting-a-definitive-guide/>.

[48] D.E. Zimmerman, C. Akerelrea, A group card sorting methodology for developing
informational Web sites, in: Proceedings IEEE International Professional
Communication Conference, 2002, pp. 437–445.

[49] J. Fuccella, Using user centered design methods to create and design usable Web
sites, 1997, pp. 69–77.

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

88

[50] M. Whang, Card-sorting usability tests of the WMU libraries’ web site, J. Web
Librarian. 2 (2–3) (2008) 205–218 2008/09/03.

[51] M. Manhartsberger, N. Zellhofer, Eye tracking in usability research: what users
really see, in: Usability Symposium, vol. 198, 2005, pp. 141–152.

[52] M. Schiessl, S. Duda, A. Thölke, R. Fischer, Eye tracking and its application in us-
ability and media research, MMI-interaktiv J. 6 (2003) 41–50.

[53] L. Cooke, E. Cuddihy, Using eye tracking to address limitations in think-aloud
protocol, in: IPCC 2005 Proceedings International Professional Communication
Conference; 2005 10–13 July 2005; pp. 653–658.

[54] C.E. Lathan, M.M. Sebrechts, D.J. Newman, C.R. Doarn, Heuristic evaluation of a
web-based interface for internet telemedicine, Telemed. J.: Off. J. Am. Telemed.
Assoc 5 (2) (1999) 177–185 Summer.

[55] J. Kjeldskov, C. Graham, A review of mobile HCI research methods, International
Conference on Mobile Human-Computer Interaction, Springer, 2003, pp. 317–335.

[56] R. Schnall, H. Cho, M. Alexander, A. Pichon, H. Jia, Mobile health technology for
improving symptom management in low income persons living with HIV, AIDS
Behav. Jan (2018) 3.

[57] R. Schnall, T. Higgins, W. Brown, A. Carballo-Dieguez, S. Bakken, Trust, perceived
risk, perceived ease of use and perceived usefulness as factors related to mHealth
technology use, Stud. Health Technol. Inform. 216 (2015) 467–471.

[58] L. Cowen, L.J.S. Ball, J. Delin, An eye movement analysis of web page usability, in:
X. Faulkner, J. Finlay, F. Détienne (Eds.), People and Computers XVI – Memorable
Yet Invisible: Proceedings of HCI 2002, Springer, London, 2002, pp. 317–335.

H. Cho et al. Journal of Biomedical Informatics 86 (2018) 79–89

89

  • A multi-level usability evaluation of mobile health applications: A case study
    • Introduction
    • Methods/results
      • Theoretical framework: a stratified view of health information technology (IT) usability evaluation
      • Methods for operationalizing framework
      • Case study: a multi-level usability evaluation
        • Level 1 (User-Task): user-centered design
        • Level 2 (User-Task-System): usability evaluation in a laboratory setting
        • End-user usability testing
        • Heuristic evaluation
        • Level 3 (User-Task-System-Environment): usability evaluation in a real-world setting
    • Discussion
    • Limitations
    • Conclusions
    • Conflict of interest
    • Acknowledgements
    • References

I-IANDE~OOK OF: EVALUATION METIdODS

10. Approach to Identification of
Pitfalls and Perils

A b i a s – “the arrival at a conclusion that differs sys tematically f r o m the t r u t h ”
(Jaeschke and Sackett 1989; Altman et al. 2001) – is the result of one or more
experimental flaws within the design, accomplishment, or interpretation of the
assessment study.

“What is r e q u i r e d . . , is the recognition o f potential confounders, followed by the
incorporation o f specific research design (or “architectural”) strategies to reduce
their impact and thus avoid bias.”

(Jaeschke and Sackett 1989).

“Bias is usually defined as a prejudice or partiality, whether conscious or not. This is
in contrast to systematic error where no prejudice exists “.

(DeMets 1997)

In this paper, emphasis is on addressing all kinds of (unintentional) biases a n d
systematic errors – together called ‘pitfalls and perils’

In order to achieve a candidate list of experimental flaws at assessment of IT-
based solutions in healthcare, a synthesis and an abstraction was made on
literature’s description of paradigmatic experimental perils and pitfalls within
medical science, natural science, and social science. Sources of inspiration are
numerous; nevertheless, the main ones remain the following review papers:
(Jaeschke and Sackett 1989; Wyatt and Spiegelhalter 1991; Fraser and Smith
1992; Sher and Trull 1996; Friedman and Wyatt 1997; and Coolican 1999). These
reviews address different experimental approaches like field studies, cohort
studies and case studies. Wyatt and Spiegelhalter (1991) explicitly and only
address the experimental pitfalls and perils at the assessment of knowledge-based
decision-support systems, comprising a specific subset of IT-based systems. The
detailed interpretation of the meaning of each pitfall and peril within the original
context as well as further examples are to be found in the original references.

The rationale of the basic approach has been to make a structured framework to
enable and structure a fair and comparable analysis, based on scientific principles
and documented knowledge. The structure was developed as an analogue to the
SWOT technique by inclusion of a large number of experimental perils and
pitfalls reported in scientific literature.

Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science &
Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691.
Created from waldenu on 2022-03-25 22:00:29.

C
o
p
yr

ig
h
t
©

2
0
0
6
.
E

ls
e
vi

e
r

S
ci

e
n
ce

&
T

e
ch

n
o
lo

g
y.

A
ll

ri
g
h
ts

r
e

se
rv

e
d
.

HANDt~OOI< OF: ~VALUATION M~TIdODS

The sole purpose of the framework is that of structuring the information seeking
and analysis to secure an objective analysis. The framework is inspired by and
structured as analogous to a SWOT analysis, while a transformation of the four
main concepts of a SWOT analysis has been tumed into similar concepts of
particular relevance for assessment studies. The framework was applied for
verification purposes on a number of assessment studies within the CANTOR
(HC4003) Healthcare Telematics Project under the EU Commission’s Fourth
Framework Programme, based on artificial experimental setups. Following the
application in CANTOR, the framework was refined to accommodate the
experiences achieved.

Finally, a review of case studies reported in the literature has been completed to
add published examples to the identified types of flaws, thereby making it
plausible whether analogies exist between the known types of experimental flaws
and those observed in the literature on real-life assessment of IT-based solutions
in healthcare. Within this process the contents of the framework was gradually
refined and incrementally extended to accommodate new types of flaws
identified.

An extensive search of methods and methodologies as well as case studies on
assessment of IT-based system/solutions was accomplished previously; see
(Brender 1997a, 1998, and 1999). The strategy then was to search for all
combinations of concepts of ‘assessment’ (with synonyms: technology
assessment, evaluation, verification, and validation), ‘computer system’ (with
synonyms: IT, IT system, computer, computer-based system), ‘methods’ and
‘methodologies’, and ‘case studies’. This search strategy was supplemented by
search on a number of key players of the domain. The literature databases
searched were PubMed and INSPECT, plus relevant databases of the local
research libraries. Also included in the original literature study were two extensive
databases of relevant literature that were acquired specifically for the assessment
of knowledge-based systems and decision-support systems; see (O’Moore et al.
1990b; and Talmon and van der Loo 1995).

The above literature study was repeated to find newer cases and methods, with
extensive use of PubMed’s facility ‘related references’ for key papers, explicit
search on authors of notable papers, and elicitation of key references from the
notable papers. In case of difficulty in retrieving a paper, little effort was put into
pursuing short publications (i.e., 2-4 pages), like proceeding papers. The reason
being a general experience of lack of sufficient detail of the method and
description of the results for the present purpose.

By application of this repeated and multiapproach strategy, the likelihood of the
completeness of the framework being rather exhaustive with respect to types of

?50

Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science &
Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691.
Created from waldenu on 2022-03-25 22:00:29.

C
o
p
yr

ig
h
t
©

2
0
0
6
.
E

ls
e
vi

e
r

S
ci

e
n
ce

&
T

e
ch

n
o
lo

g
y.

A
ll

ri
g
h
ts

r
e

se
rv

e
d
.

~ANICSOOK Og EVALUATION ME]TIdODS

experimental errors is rather high and depends mainly on the author’s ability to
identify all relevant (types of) flaws.

However, please note that reporting within this publication only includes
examples of the best cases and bad examples to learn from or otherwise
particularly relevant or paradigmatic examples out of the bulk of publications
acquired. Thus, this publication does not reference the entirety of literature on
assessment studies.

751

Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science &
Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691.
Created from waldenu on 2022-03-25 22:00:29.

C
o
p
yr

ig
h
t
©

2
0
0
6
.
E

ls
e
vi

e
r

S
ci

e
n
ce

&
T

e
ch

n
o
lo

g
y.

A
ll

ri
g
h
ts

r
e

se
rv

e
d
.

This Page Intentionally Left Blank

Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science &
Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691.
Created from waldenu on 2022-03-25 22:00:29.

C
o
p
yr

ig
h
t
©

2
0
0
6
.
E

ls
e
vi

e
r

S
ci

e
n
ce

&
T

e
ch

n
o
lo

g
y.

A
ll

ri
g
h
ts

r
e

se
rv

e
d
.

https://doi.org/10.1177/1460458219899556

Health Informatics Journal
2020, Vol. 26(3) 2249 –2264

© The Author(s) 2020
Article reuse guidelines:

sagepub.com/journals-permissions
DOI: 10.1177/1460458219899556

journals.sagepub.com/home/jhi

Creative Commons Non Commercial CC BY-NC: This article is distributed under the terms of the Creative
Commons Attribution-NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which

permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is
attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).

Outcomes of health information
technology utilization in nursing
homes: Do implementation
processes matter?

Darla J Hamann
St. Cloud State University, USA

Karabi C Bezboruah
The University of Texas at Arlington, USA

Abstract
We examined several outcomes of health information technology utilization in nursing homes and how the
processes used to implement health information technology affected these outcomes. We hypothesized
that one type of health information technology, electronic medical records, will improve efficiency and
quality-related outcomes, and that the use of effective implementation processes and change leadership
strategies will improve these outcomes. We tested these hypotheses by creating an original survey based
on the case study literature, which we sent to the top executives of nursing homes in seven US states. The
administrators reported that electronic medical record adoption led to moderately positive efficiency and
quality outcomes, but its adoption was unrelated to objective quality indicators obtained from regulatory
agencies. Improved electronic medical record implementation processes, however, were positively related
to administrator-reported efficiency and quality outcomes and to decreased deficiency citations at the next
regulatory visit to the nursing home. Change leadership processes did not matter as much as technological
implementation processes.

Keywords
electronic medical records, health information technology, long-term care, nursing homes, quality of care

Corresponding author:
Darla J Hamann, Master of Public Administration, St. Cloud State University, 720 4th Avenue South, St. Cloud, MN
56301, USA.
Email: [email protected]

899556 JHI0010.1177/1460458219899556Health Informatics JournalHamann and Bezboruah
research-article2020

Original Article

2250 Health Informatics Journal 26(3)

Introduction

The US federal government has successfully used financial incentives for increasing health infor-
mation technology (health IT) implementation in some industries, and some researchers have
suggested broadening their use to include nursing homes.1 Such policy is based on the premise
that health IT substantially impacts quality and cost-related outcomes in nursing homes, which
given the mixed and meager outcomes in the research literature2–6 may be an optimistic assump-
tion. In this study, we examine this assumption by systematically building an electronic medical
record (EMR) outcomes scale based on the quotations of hundreds of clinicians from four multi-
case studies7–9 (Alexander et al., 2007). We therefore examine the extent to which the most impor-
tant proximal outcomes of EMR implementation, as reported by case study interviewees, are
experienced at a larger cross section of nursing homes. Perhaps more importantly, however, our
approach allows us to answer the growing cry from researchers to consider how implementation
strategies affect the efficacy of health IT.7,10–12 We use the same studies to build a scale of effec-
tive implementation processes. We argue that ineffective implementation processes and poor
organizational change leadership in nursing homes are limiting the potential of one type of health
IT, EMRs, to positively impact quality and efficiency outcomes in some nursing homes, leading
to the uneven and often meager results seen in the case study literature. We suggest that with
heightened attention paid to implementation processes, health IT outcomes will improve.

Literature review and hypotheses development

EMR implementation outcomes

First, EMRs may improve the quality and efficiency of care in nursing homes for several rea-
sons. EMRs allow for increased accuracy, completeness, and legibility of and access to medical
information. Multiple staff can view a file simultaneously and files are not misplaced, and with
instantaneous and convenient access to each resident’s file, documentation accuracy and com-
pleteness should be improved. Indeed, these outcomes were reported by the clinicians inter-
viewed by Rantz et al.,5 Kramer et al.,8 Cherry et al.,13 Degenholtz et al.,14 Meehan,15 Munyisia
et al.,16 and Zhang et al.17 Accurate and timely documentation allows different clinicians to
understand the changes in resident condition and in medications given in real time, leading to
improvements in the quality of care and reductions in medical errors. Medical error reductions
were found by Qian et al.,4 and Rantz et al.5 found that activities of daily living, range of motion,
and high-risk pressure sores of residents improved. A second way EMRs can improve the quality
of care is by prompting clinicians to follow evidence-based medical recommendations based on
conditions in the resident’s chart, and charts may be programmed with reminders regarding the
residents’ care plans. These prompts have led to slight improvements in clinician action18 and to
increased clinician compliance.4 These mechanisms capitalize on the use of better data for clini-
cians to make better care-related decisions, which should ultimately improve the efficiency and
quality of care (Kruse et al., 2017).19 In addition, the data in aggregate form can help managerial
staff supply data to regulators, apply for grants and thus become better funded, and make better
managerial decisions.17,20

A third way that EMRs can improve the efficiency and quality of care is through staff oversight
and productivity improvements.7–9 With EMRs, it is more observable which cares were completed
on each resident each day as electronic charts are searchable while paper charts are not. With more
oversight, employees may do more in order to appear better to their supervisors, and supervisors
may better understand what employees do and may make better scheduling, delegating, and

Hamann and Bezboruah 2251

coaching decisions. In addition, EMRs increase the efficiency of documentation,17 for example, by
pre-filling common fields with default values that only rarely need to be changed, which may allow
nursing staff to spend more time on direct care.21

The impact of EMRs on the documentation time is not clear in the literature. While Mei et al.21
found that documentation time fell for a simple fall-reporting software and Bezboruah et al.7 found
that documentation time rose in some nursing homes that implemented EMRs at least initially, it is
not clear whether EMRs will generate staff time-savings. Qian et al.4 compared the EMR adminis-
tration records and paper-based records and found no significant differences in the nursing time
spent on various activities such as medication administration, documentation, and communication.
Munyisia et al.22 used a longitudinal case study design to evaluate whether nursing time with patients
improved after employees had time to adjust to EMRs, but even 23 months after implementation, no
difference in direct care time was found.22 Klinger and White23 and Cherry et al.13 found that some
employees spent more time with residents, while others spent less after EMR implementation. If
increasing nursing time with residents is a goal of EMR adoption, it may be important to include
training or incentives regarding it because it does not appear to happen automatically.

A fourth way that EMRs could improve the efficiency and quality of care is by increasing the
speed of service provision.7,8,13 Electronic alerts can be used to ensure timely responses to every-
thing from resident call lights (used by residents to signal an urgent need for assistance) to medica-
tion administration to family member inquiries.

Hypothesis 1. The use of EMRs is positively associated with quality and efficiency outcomes in
nursing homes.

Implementation processes

While in theory EMRs should positively impact efficiency and quality outcomes, many studies,
especially larger studies, have found meager or mixed findings. Hitt and Tambe2 used a more
sophisticated methodological design that compared the same organization’s performance before
and after EMR adoption and found only marginal improvements in employee productivity and
efficiency. Pillemer et al.3 in a well-done quasi-experimental design, found EMRs had virtually no
impact on resident health and quality of life. Pillemer et al.3 did find that EMRs were negatively
associated with poor resident behaviors and that residents viewed the technology as generally posi-
tive; however, in a qualitative study, Cherry et al.13 found a range of resident and family overall
reactions to EMRs, with residents lamenting their caregivers’ “playing” on computers. Klinger and
White23 and Bezboruah et al.7 reported that nursing staff generally did not believe they were more
efficient or documented more accurately, and Cherry et al.13 reported nursing staff did not think
EMRs improved the quality of care.

Large longitudinal studies of Medicare data and the Department of Veterans’ Administration
found that the benefits of health IT use were unexpectedly small.24,25 A few studies4,7,26 found that
EMRs sometimes even had adverse effects on the quality of care. With a strong theoretical ration-
ale for improved outcomes, researchers have begun to consider why so many studies, especially
the best studies, report uninspiring outcomes. Several have conjectured that poor implementation
processes are hindering the effectiveness of health IT (Kellerman and Jones, 2013).7,11,27,28,29

Several qualitative studies present evidence of implementation processes we believe to be subop-
timal. One area of concern was in the involvement and treatment of employees prior to and during the
EMR implementation process. Many case studies report staff resistances to using technology and
learning problems, technical and connectivity concerns, documentation duplication, unnecessarily
complex information management including wasting time clicking through screens, and difficulties

2252 Health Informatics Journal 26(3)

in delivering care or the quantity or quality of time spent with residents.7,23,26,30,31 Yet, nursing homes
with positive employee relations,9,23 those that included end-users in systematic planning,7 and those
that used employee empowerment practices2 usually had better outcomes. When asked by research-
ers, nursing staff had innovative ideas for improving the software to improve efficiency and quality
outcomes and desired meetings with the vendors for software customization.13,15

In every study reviewed, staff reported not receiving enough training,7,11,13,30 and implementa-
tion processes that emphasized training led to better outcomes.9,32 Several staff members also
expressed that off-site technological assistance was insufficient and used computers and screens
unlike those the employees would use, and on-site training and assistance was more helpful.7,11,30
After training, staff reported that workload initially increased due to learning curves and chart-
ing twice (electronically and in writing),7,30 and employees who experienced incremental roll-
outs reported better outcomes than those who were introduced to all new technical features
simultaneously.7,30

Based on these results, we conjecture that nursing homes that include the employees who will
use the EMRs during the planning and customized design phase of the EMR project will not only
face less resistance to EMRs but will also end up with EMR systems that save employees time, as
unnecessary screens can be eliminated and default answers can be appropriately set. We also
believe that additional training time, delivered on-site using the exact screens that employees will
be using, will result in greater meaningful use of EMR systems and their electronic alerts enabling
greater increases in the quality of care. In addition, including additional staff on busy shifts as
employees adjust to EMRs, and incremental EMR roll-outs, will allow employees the time to form
new habits that take advantage of the technological features and customizable prompts that they
learned in training and lead to fewer counterproductive EMR work-arounds.

Another area of concern was in the area of change leadership. While reviewing hundreds of
quotations in qualitative studies, we saw plenty of evidence of organizations either following or not
following the advice given in the change leadership literature. Change leadership touts the impor-
tance of organizational characteristics such as learning, innovation, experimentation, and question-
ing while preparing the organizations for change. In this change process, there is constant seeking
for new perspectives and encouragement of participation throughout the organization.
Organizational leadership are more aware of trends, crises, and evolutions in their organizational
environments, which prepare them to be resilient and adaptive to changes.33 Research on nursing
homes34 suggests that the presence of change leadership and an organizational climate of innova-
tion are likely to assist organizations preparing for change.

Studies of EMR implementation processes found that administrators faced challenges with
obtaining adequate information about the costs and benefits of health IT, or they did not prioritize
information gathering or learning about health IT systems.7,11,35,36 Song et al.36 found that pre-
implementation business case analyses lacked the rigor associated with comparable capital acqui-
sition projects, Bezboruah et al.7 found little systematic planning for EMR implementation, and Ko
et al.11 found limited involvement of nurses in EMR planning and little planning overall. Rationales
given for EMR implementation were often remarks about what other institutions were doing, lack-
ing the rigor of benchmarking studies, or a belief by the management team that EMR investment
was inevitable.7,36 A lack of clinical leadership and early communication with employees and other
important stakeholders for buy-in were also common (Bezboruah et al., 2014).11 Many nursing
homes failed to plan how the EMRs would impact staff workflow and consequently implemented
cumbersome systems that were not optimally utilized since employees found work-arounds.11
Often, the administrators, who made decisions in the absence of clinical staff, did not understand
the software or the needs of clinical staff and were too reliant on the biased views of vendors for
information and advice.11,37

Hamann and Bezboruah 2253

We gathered information from the study by Szydlowski et al.29 on nursing and suggest the fol-
lowing attributes would indicate a more ideal change leadership process: leadership of the EMR
acquisition team (1) developed a clear vision, (2) clarified the necessity of EMR implementation,
(3) proactively built a broad coalition to support implementation, (4) empowered employees to
implement EMRs, (5) monitored and communicated progress of EMR implementation, and (6)
gave individualized attention to employees struggling with EMR implementation. Even if manage-
ment did not enact the best practice implementation processes that we uncovered, managers who
were skilled at leading change initiatives should still improve the quality and efficiency of their
nursing homes as they lead their organizations through technological change.

Hypothesis 2. Better EMR implementation processes are positively associated with quality and
efficiency outcomes in nursing homes.

Hypothesis 3. Effective change leadership strategies are positively associated with quality and
efficiency outcomes in nursing homes.

Method

Data

The purpose of this study was to (1) estimate the impact of EMRs on nursing home quality and
efficiency outcomes (hereafter “outcomes”) and (2) determine whether good implementation pro-
cesses and change leadership strategies improve nursing home outcomes (hereafter “processes”
and “change leadership”). We began by designing an original survey using an inductive approach,
by examining the detailed practitioner responses38 recorded in several qualitative case studies
(including Alexander, 2007).7–9 We selected all the case studies available at the time that included
detailed reporting of raw data, which we obtained from a government report and conference paper
drafts available online. Following a process advised by Bowen and Gao,39 two researchers and two
graduate assistants independently reviewed all the case studies. We independently categorized
quotations into themes and then met to discuss the results. Outcomes or processes quotations that
were mentioned in only one case study were dropped. Quotation themes were turned into survey
items. By group consensus, similar survey items were combined and succinct language for the
items was developed. The draft survey was reviewed by three researchers and two nursing home
administrators and pilot tested. Changes to the survey were made based on the feedback received.

Electronic mail addresses were obtained from state Departments of Health/Aging or nonprofit
advocacy organizations for all nursing home administrators in Illinois, South Carolina, Kansas,
Texas, Georgia, Minnesota, and Missouri. The survey was conducted online between July and
October 2011. Three emails containing cover letters and a survey link were sent in 1-week incre-
ments to 2747 nursing home executives; 284 surveys were received (over three-fourths were the
chief executive officer or administrator, with the remainder being the chief information officer or
the director of nursing).

Survey measures. We grouped outcome items into three positive and one negative theme that
emerged from the data when we conducted exploratory factor analysis. All items were measured
on a 5-point Likert-type scale with the lead question, “To what extent has your EMR implementa-
tion resulted in. . ..” The first positive outcome was improved data access. It was measured with
the following items: (1) improved census reporting, (2) improved completeness of resident infor-
mation, (3) increased availability of resident information, (4) enhanced information sharing with
other health care providers, and (5) higher quality and more complete reports. The second positive

2254 Health Informatics Journal 26(3)

outcome was that the speed of service provision increased, which was measured with the following
items: (1) faster delivery of services to new residents, (2) improved on-call response time, and (3)
quicker response to family member inquiries. The third positive outcome was staff oversight and
productivity improvements, which were measured with the following items: (1) improved staff
productivity, (2) reduced administrative staff needed, and (3) improved managerial oversight of
employees. The negative outcome was staff resistance to EMRs (Alexander, 2007).7,8 It was meas-
ured by asking the level of resistance to EMR implementation expressed by each employee group
and by asking about employee turnover.

Seven items were included in the processes scale. They were measured on a 5-point scale,
and all questions began with “To what extent did the EMR implementation process include. . .” The
items were (1) staff training, (2) early communication with key stakeholders, (3) use of additional
staff in early adoption stages, (4) Incremental roll-out, (5) on-site technological assistance, (6) nurs-
ing managers in implementation decisions, and (7) nursing staff in implementation decisions.

“Change leadership” was measured by a six-item scale adapted from Harold et al.40 These scale
consists of the following items: (1) developed a clear vision, (2) made it clear why EMR imple-
mentation was necessary, (3) built broad coalition up-front to support implementation, (4) empow-
ered employees to implement EMRs, (5) monitored and communicated progress of EMR
implementation, and (6) gave individualized attention to those having trouble with EMR imple-
mentation. Although drawn from the management literature, these items were similar to the health
IT leadership change themes identified by Szydlowski and Smith29 in their study of nursing.

EMR adoption was measured by a single variable “Does your organization use electronic medi-
cal records?” Because nursing homes were in the early stage of health IT adoption and EMRs were
often the first systems adopted, a binary variable satisfactorily captures EMR use.2 The survey also
asked for the date of EMR implementation to calculate the length of time the facility had the sys-
tem. EMRs can have initial negative effects, since it takes several months for employees to adapt
to them.35

Regulator data measures. Regulator data were obtained from the Centers for Medicare and Medic-
aid Services (CMS), containing the size, ownership, chain status, occupancy rates, funding, quality
ratings (5-point scales), and regulatory citations for all certified facilities. We use the facility’s
overall rating, the quality indicator (containing nine items, such as the percentage of residents with
pressure sores, falls, or other injuries), and the count of regulatory citations received as distal qual-
ity outcomes.

Statistical strategy

Scale validation. Since our survey items were not previously validated, we followed Lance and
Vandenberg’s41 and Acock’s42 instructions to conduct confirmatory factor analysis (CFA) to gen-
erate scales. We used maximum likelihood estimation with imputation to adjust for missing data.

Response bias checks. We tested for survey response bias, which involves comparing survey
responders with non-responders.43 Responding nursing homes had on average 99 beds compared
to 104 beds among non-responders (p > 0.10). In all, 46 percent of the responders were for-profit,
47 percent were nonprofit, and 5 percent were government-owned, compared to 71, 24, and 3 per-
cent, respectively, of non-responders (p < 0.01). In total, 6.5 percent of the responders and non-
responders were located within a hospital (p > 0.10). Fifty percent of responders, versus 58 percent
of non-responders, were part of a regional or national chain (p < 0.01). In total, 3.5 percent of

Hamann and Bezboruah 2255

responders versus 5.2 percent of non-responders did not care for Medicaid residents (p > 0.10), and
3.5 percent of responders versus 6.0 percent of non-responders did not care for Medicare residents
(p < 0.10).

Regressions. We conducted ordered probit regressions to test Hypotheses 1 and 2. All measures
were adjusted for resident health conditions (case mix). We used the Heckman44 sample selection
correction method, a weighting technique, which can minimize survey response bias.45

Results

Descriptive statistics

Table 1 reports descriptive statistics for our survey. As expected, the availability and complete-
ness of health data increased a moderate to large amount at most nursing homes. Similarly, the
speed at which resident concerns were addressed increased. The administrators reported that

Table 1. Outcomes of EMR implementation as reported by nursing home administrators.

Number of
responses

Mean Standard
deviation

Minimum Maximum

Improved data
Improved census reporting 109 2.991 1.213 1 5
Improved completeness of resident

information
111 3.423 1.066 1 5

Increased availability of resident
information

110 3.782 0.961 1 5

Enhanced information sharing with other
health care providers

111 2.865 1.132 1 5

Higher quality and more complete reports 109 3.459 1.085 1 5
Improved speed of service provision
Faster delivery of services to new

residents
110 2.764 1.116 1 5

Improved on-call response time 100 2.220 1.177 1 5
Quicker response to family member

inquiries
110 2.964 1.188 1 5

Staff resistance to health IT
CNAs’ resistance to IT 101 2.307 0.821 1 5
Nurses’ resistance to IT 108 2.435 0.899 1 5
Nursing managers’ resistance to IT 108 2.102 0.937 1 5
Increased turnover 106 1.528 0.783 1 5
Oversight and productivity improvements
Improved staff productivity 110 2.873 1.101 1 5
Reduced administrative staff needed 110 1.573 0.784 1 4
Improved managerial oversight of

employees
111 3.243 1.223 1 5

IT: information technology; CNA: certified nursing assistant.
Sample sizes vary because questions marked “unsure” are excluded. All variables measured on the following Likert-type
scale: 1 (not at all), 2 (some), 3 (moderate), 4 (large), and 5 (extreme).

2256 Health Informatics Journal 26(3)

new residents received needed care, call lights (for urgent needs) were answered, and family
needs were addressed more quickly after the EMRs were introduced. In addition, the productiv-
ity of nursing and administrative staff improved slightly. Most administrators detected a moder-
ate to large increase in the level of oversight that they had over employees after EMR
implementation, and they believed that employee productivity had increased moderately.
However, over 50 percent of administrators responded that there had not been a reduction in
staff due to productivity gains. Our results also indicated that staff turnover was usually not
impacted by the introduction of the EMRs, and staff resistance to EMRs was relatively low in
the administrator’s view.

Table 2 reports descriptive statistics for our survey respondents who had implemented EMRs.
Consider first the processes scale. The administrators reported that they provided a large amount of
training and on-site technical help to employees and communicated with stakeholders to a moder-
ate extent. Administrators were less likely to provide additional staff to assist employees during
implementation, but most implemented the technology incrementally. Administrators included
staff in implementation decision-making, especially supervisory staff.

Finally, we consider the change leadership. This six-item scale had internal consistency reliabil-
ity (α) of 0.84 and a mean of 4.03 (95% CI = 3.9–4.2). Administrators reported following these
effective change leadership practices to a large extent.

Table 2. Implementation processes reported by nursing home administrators.

Variable Observations Mean Standard
deviation

Minimum Maximum

Process variables
Include staff training 109 3.917 0.873 1 5
Include early communication with key

stakeholders
103 3.340 1.053 1 5

Use additional staff in early adoption stages 107 2.925 1.052 1 5
Incremental roll-out 107 3.477 1.093 1 5
Include on-site technical assistance 110 3.473 1.171 1 5
Inclusion in decision-making
Include nurse managers in implementation

decisions
109 3.349 1.117 1 5

Include nursing staff in implementation
decisions

110 3.009 1.128 1 6

Change leadership
Developed a clear vision 108 4.009 0.743 1 5
Made it clear why EMR implementation was

necessary
109 4.174 0.768 1 5

Built a broad coalition up-front to support
implementation

107 3.766 0.875 1 5

Empowered employees to implement EMRs 109 3.899 0.816 1 5
Monitored and communicated progress of

EMR implementation
109 3.899 0.816 1 5

Gave individualized attention to those
having trouble with EMR implementation

108 4.148 0.783 1 5

EMR: electronic medical record.
Sample sizes vary because questions marked “unsure” are excluded. All variables measured on the following Likert-type
scale: 1 (not at all), 2 (some), 3 (moderate), 4 (large), and 5 (extreme).

Hamann and Bezboruah 2257

Tests of the factor structure of survey items

We modeled a CFA using four outcome variables and two implementation variables (implementa-
tion processes and participation in decision-making). It yielded insignificant fit, so we modified
our results along theoretical and statistical lines.42 We dropped two items that had factor loadings
below 0.40 (“increased turnover” and “reductions in administrative staff”).

The modified six-factor model fit our data well (χ2(df = 152) = 188.3, p < 0.05; comparative fit
index (CFI) = 0.97; root mean square error of approximation (RMSEA) = 0.05, 90% CI = 0.02–
0.07). These values are better than the fit cut-offs of CFI = 0.95 and RMSEA = 0.08.42,46 We com-
bined data and staff productivity outcomes into a single scale because the covariance between these
two variables was over 0.90, suggesting they measured the same latent variable. The resulting
five-factor model was a good fit for our data (χ2(df = 157) = 189.3, p < 0.05; CFI = 0.97;
RMSEA = 0.04, 90% CI = 0.01–0.06). Our best model compares favorably to a CFA where all sur-
vey measures load upon one latent variable (χ2(df = 170) = 491.9, p < 0.01; CFI = 0.70;
RMSEA = 0.13, 90% CI = 0.12–0.14) and to a three-factor model of positive outcomes, negative
outcomes, and implementation processes (χ2(df = 165) = 230.8, p < .01; CFI = 0.94; RMSEA = 0.06,
90% CI = 0.04–0.08). Since our variables do not load on a single factor, we also pass the Harman
single-factor test, suggesting little common method variance.47

Means and zero-order correlations

Given our CFA results, we created three scales to measure the survey outcomes of EMR implemen-
tation. We combined the data and staff productivity and oversight items into a scale for efficiency,
which had internal consistency reliability (α) of 0.90 and a mean of 3.23 (95% CI = 3.06–3.39). We
created a second scale measuring the speed at which residents’ needs were met (α = 0.85), which
had a mean of 2.66 (95% CI = 2.47–2.85). We created a third scale for employee resistance to
EMRs (α = 0.74), which had a mean of 2.08 (95% CI = 1.96–2.20).

We created two scales for health IT implementation processes. We labeled the first scale imple-
mentation processes (α = 0.76), which had a mean of 3.43 (95% CI = 3.29–3.57). The second scale
measured employee participation in implementation decision-making (α = 0.88), which had a mean
of 3.18 (95% CI = 2.98–3.38).

Table 3 also reports correlations between our processes and outcomes variables. We found posi-
tive associations between employee productivity and implementation processes (r = 0.44, p < 0.01)
and participation in decision-making (r = 0.43, p < 0.01). We also find positive associations between
the speed of service provision and implementation processes (r = 0.34, p < 0.01) and participation
in decision-making (r = 0.41, p < 0.01). Smaller associations were detected between effective
change leadership and employee productivity (r = 0.34, p < 0.01) and speed of service provision
(r = 0.22, p < 0.01).

Hypothesis testing: multivariate analysis

Table 4 has the regression results analyzing the impact of EMR utilization on regulatory data qual-
ity outcomes. Using EMRs was not related to the nursing home’s overall regulatory score, quality
index score, or deficiency citations. Recognizing that EMR implementation might not have posi-
tive results initially since staff face a learning curve and may resist technologies, we also tested
whether the length of time since EMR implementation was related to the quality of care. Again, we
found no such relationship. Administrators reported positive outcomes from using EMRs, but
EMR use was unrelated to distal objective measures, providing partial support for Hypothesis 1.

2258 Health Informatics Journal 26(3)

Table 5’s regression results corroborated our correlational analysis; the implementation pro-
cesses were positively related to our efficiency and speed of service outcomes, and unrelated to
employee resistance to EMRs. While the overall and quality star rating regressions were

Table 3. Means, standard deviations, and correlations among variables.

Only adopters of EMRs Mean Standard
deviation

1 2 3 4 5

1. Implementation processes 3.43 0.75
2. Participation in decision-making 3.18 1.06 0.52***
3. Employee productivity 3.23 0.88 0.44*** 0.43***
4. Speed of service provision 2.66 1.02 0.34*** 0.41*** 0.78***
5. Resistance to change 2.08 0.66 0.25*** 0.06*** 0.08 0.10
6. Change leadership 4.03 0.63 0.44*** 0.42*** 0.22** 0.22** −0.24**
All respondents and/or OSCAR data 7 8
7. Uses EMR 0.44 0.50
8. Citations 2.31 3.96 −0.03
9. Overall quality rating 2.84 1.28 0.06 −0.14***

EMR: electronic medical records; OSCAR: Online Survey, Certification, and Reporting.
*, **, and *** indicate statistical significance at the 10, 5, and 1 percent levels, respectively, in two-tailed tests.

Table 4. The impact of the use of electronic medical records, and the length of their use, on regulator
quality data and staffing levels.

Overall star rating Quality star rating Deficiency citations

Uses EMR 0.185
[0.179]

0.090
[0.163]

0.126
[0.209]

Total years EMR in place 0.014
[0.026]

0.023
[0.024]

−0.046
[0.035]

Nonprofit ownership (includes public) 0.284
[0.214]

−0.368**
[0.153]

−0.383
[0.240]

Number of certified beds −0.003**
[0.001]

−0.000
[0.001]

0.001
[0.002]

Chain status −0.034
[0.145]

0.130
[0.137]

−0.278
[0.174]

Located within a hospital −0.139
[0.299]

−0.168
[0.268]

0.518
[0.346]

Number of Medicaid patients 0.247
[0.371]

0.173
[0.322]

−0.061
[0.416]

Number of Medicare patients 1.072**
[0.440]

0.833**
[0.412]

−0.739
[0.554]

Observations 222 219 222
Log-likelihood −1092.5 −1073.4 −1103.4
χ2 17.86 17.09 9.29

Coefficients presented with standard errors in brackets. The Heckman sample selection corrections and ordered probit
were utilized in estimation. Sample selection equation included ownership, chain status, location within a hospital, num-
ber of residents, percentage of occupied beds, Medicaid or Medicare payment, retirement community status, and state.
Sample selection equation had 3725 observations.
**Statistical significance at the 5 percent level, in two-tailed tests.

Hamann and Bezboruah 2259

T
ab

le
5

.
T

he
im

pa
ct

o
f
EM

R
im

pl
em

en
ta

ti
o
n

pr
o
ce

ss
es

o
n

re
gu

la
to

r
qu

al
it

y
da

ta
a

nd
s

ta
ffi

ng
le

ve
ls

.

Ef
fic

ie
nc

y
o
ut

co
m

es
Sp

ee
d

o
f

se
rv

ic
e

pr
o
vi

si
o
n

Em
pl

o
ye

e
re

si
st

an
ce

to

E
M

R
s

O
ve

ra
ll

st
ar

ra

ti
ng

Q
ua

lit
y

m
ea

su
re

s
st

ar
r

at
in

g

D
ef

ic
ie

nc
y

ci
ta

ti
o
ns

R
N

s
ta

ffi
ng

le

ve
ls

Im
pl

em
en

ta
ti

o
n

pr
o
ce

ss
es

0.
46

9*
**

[0
.1

10
]

0.
44

7*
**

[0
.1

29
]

0.
11

9
[0

.1
12

]

0.
00

5
[0

.1
66

]
0.

01
4

[0
.1

31
]


0.

32
3*

*
[0

.1
52

]

0.
00

6
[0

.1
25

]
N

um
be

r
o
f
ce

rt
ifi

ed
b

ed
s

0.
00

4*
**

[0
.0

01
]

0.
00

4*
**

[0
.0

02
]


0.

00
0

[0
.0

01
]


0.

00
5*

*
[0

.0
02

]

0.
00

1
[0

.0
03

]
0.

00
1

[0
.0

02
]


0.

00
4

[0
.0

03
]

N
o
t

fo
r-

pr
o
fit

(
in

cl
ud

es
p

ub
lic

)

0.
17

0
[0

.2
35

]

0.
53

1*
*

[0
.2

64
]

0.
16

4
[0

.2
56

]
0.

67
8*

[0
.3

63
]


0.

39
2

[0
.2

46
]


0.

21
2

[0
.3

80
]

0.
02

3
[0

.4
64

]
C

ha
in

s
ta

tu
s

0.
07

9
[0

.1
65

]
0.

06
7

[0
.1

97
]


0.

24
5*

[0
.1

41
]


0.

08
0

[0
.2

41
]

0.
12

5
[0

.2
13

]

0.
09

5
[0

.2
77

]
0.

03
3

[0
.1

88
]

Lo
ca

te
d

w
it

hi
n

a
ho

sp
it

al

0.
14

8
[0

.2
60

]

0.
17

9
[0

.1
97

]
0.

10
6

[0
.2

34
]


0.

30
0

[0
.3

77
]

0.
01

0
[0

.3
03

]
0.

44
7

[0
.3

99
]

0.
26

4
[0

.3
53

]
N

um
be

r
o
f
M

ed
ic

ai
d

pa
ti

en
ts

0.
40

1
[0

.3
51

]
0.

89
5

[0
.4

23
]

0.
33

8
[0

.3
00

]
0.

58
8

[0
.5

60
]


0.

14
8

[0
.3

96
]


0.

25
6

[0
.6

06
]

0.
33

5
[0

.5
16

]
N

um
be

r
o
f
M

ed
ic

ar
e

pa
ti

en
ts


0.

59
6

[0
.8

26
]

0.
18

2
[0

.9
55

]

1.
21

7
[0

.7
90

]

0.
19

3
[1

.2
23

]
7.

12
2

[1
60

33
7]


6.

43
4

[1
60

80
]

0.
25

6
[1

.5
79

]
C

o
ns

ta
nt

2.
31

9*
**

[0
.6

93
]

2.
47

8*
**

[0
.7

53
]

0.
60

3
0.

75
2


2.

41
6*

**
[0

.2
88

]

2.
38

1*
**

[0
.2

89
]


2.

42
0*

**
[0

.2
87

]

2.
39

5*
**

[0
.2

81
]

O
bs

er
va

ti
o
ns

92
92

92
95

95
95

95
Lo

g-
lik

el
ih

o
o
d


46

5.
48


47

9.
86


45

0.
23


50

9.
2


51

0.
9


51

4.
7


50

5.
7

χ
2

37
.3

3
30

.3
1

12
.3

5
11

.1
8

4.
97

5
6.

29
9

11
.9

4

EM
R

: e
le

ct
ro

ni
c

m
ed

ic
al

r
ec

o
rd

s.
B

et
a

co
ef

fic
ie

nt
s

re
po

rt
ed

w
it

h
st

an
da

rd
e

rr
o
rs

in
b

ra
ck

et
s.

T
he

H
ec

km
an

s
am

pl
e

se
le

ct
io

n
co

rr
ec

ti
o
ns

a
nd

o
rd

er
ed

p
ro

bi
t

w
er

e
ut

ili
ze

d
in

e
st

im
at

io
n.

D
ue

t
o
a

la
rg

er
n

um
be

r
o
f
va

lu
es

t
ak

en
in

e
ffi

ci
en

cy
a

nd
s

pe
ed

s
ca

le
s

(o
ve

r
15

r
at

he
r

th
an

5
),
s

ta
nd

ar
d

H
ec

km
an

m
o
de

ls
a

re
u

se
d.

S
am

pl
e

se
le

ct
io

n
eq

ua
ti

o
n

in
cl

ud
ed

o
w

ne
rs

hi
p,

c
ha

in
s

ta
tu

s,
lo

ca
ti

o
n

w
it

hi
n

a
ho

sp
it

al
, n

um
be

r
o
f
re

si
de

nt
s,

p
er

ce
nt

ag
e

o
f
o
cc

up
ie

d
be

ds
, M

ed
ic

ai
d

o
r

M
ed

ic
ar

e
pa

ym
en

t,
r

et
ir

em
en

t
co

m
m

un
it

y
st

at
us

, a
nd

s
ta

te
. O

nl
y

su
rv

ey
r

es
po

nd
en

ts
w

ho

im
pl

em
en

te
d

EM
R

s
w

er
e

in
cl

ud
ed

in
e

st
im

at
io

n.
I
n

al
l,

35
98

o
bs

er
va

ti
o
ns

w
er

e
us

ed
in

t
he

s
el

ec
ti

o
n

eq
ua

ti
o
ns

.
*,

*
*,

a
nd

*
**

in
di

ca
te

s
ta

ti
st

ic
al

s
ig

ni
fic

an
ce

a
t

th
e

10
, 5

, a
nd

1
p

er
ce

nt
le

ve
ls

, r
es

pe
ct

iv
el

y,
in

t
w

o
-t

ai
le

d
te

st
s.

2260 Health Informatics Journal 26(3)

insignificantly related to our implementation processes, we found that these processes decreased
deficiency citations. Participation in decision-making and change leadership were not significantly
related to any outcomes and were therefore excluded from Table 5.

Discussion

Outcomes of EMR implementation

We synthesized the efficiency and quality outcomes of EMR use reported in several case studies and
created a survey to analyze them in a broader cross section of nursing homes. Overall, we found
similar efficiency and speed of service outcomes as those reported in qualitative studies, with the
exceptions that staff resistance to EMRs and turnover were lower than expected. It is possible that
administrators underestimate staff resistance to EMRs, as found by Bezboruah et al.,7 but Ko et al.11
found staff reports of health IT resistance to be relatively low as well. Ko et al.11 did suggest that
perhaps the staff who participated in the employee focus groups were not random. We encourage
future research to survey a broader cross section of nursing staff to see whether our finding is due to
low levels of staff resistance to EMR or low recognition of such resistance by administrators.

Our finding that administrator reports of EMR outcomes were quite positive should be viewed
in light of research that shows that managerial staff are more likely than non-managerial staff to
report favorable outcomes of health IT6,48 and our finding that EMR adoption did not improve
regulator quality indices or deficiency citations. However, outcomes noticed by administrators
may not result in objective quality outcomes because they impact different outcomes, such as resi-
dents’ quality of life rather than counts of bed sores or falls (components of the regulator quality
index). Existing research is mixed on the extent to which health IT improves communication
between staff and residents, and little is known about other factors related to residents’ quality of
life, such as timeliness of the delivery of messages from family or pain medication. Klinger and
White23 and Hitt and Tambe2 suggested that communication increases after health IT implementa-
tion only in nursing homes with positive employee relations. Pillemer et al.3 found no differences
in resident-reported mood outcomes as a result of health IT. It may be that EMRs impact proximal
outcomes, such as the speed of service delivery and the quality of clinical data, but these outcomes
do not matter enough to affect distal quality measures such as regulator five-star ratings because
something else matters more, or they could impact different dimensions of quality (like quality of
life instead of health). It could also be that EMR’s impact on the proximal outcomes is overstated
due to measurement artifacts such as common method bias. Future research will need to explore
these relationships using sources other than the administrator for information.

Implementation processes

We also estimated the extent to which certain implementation practices and general change leader-
ship practices led to positive distal (regulator) outcomes, as this has been cited as a reason for the
lackluster performance of health IT (Kellerman and Jones, 2013).7,27,28 While effective change
leadership practices are important, the implementation processes we uncovered and the scale we
developed in this article were more important predictors of the quality of care.

This article makes another contribution to the nursing home literature. The implementation
processes scale that we created was drawn from the qualitative literature and was more strongly
correlated with the efficiency and quality administrator-reported outcomes than the existing change
leadership scale. In addition, it predicted deficiency citations, a distal measure of nursing home
quality, while the change leadership scale did not.

Hamann and Bezboruah 2261

Limitations

A limitation of our study was that our response rate was low, but it is similar to other surveys of
health IT.27 Since many administrators use email filters, it is plausible that many administrators
never saw our survey, reducing the response rate but not in a way that would be expected to create
bias. Although web-based surveys tend to have lower response rates than paper surveys,49,50 their
results tend to be similar.51 The most significant problem with a low response rate is non-response
bias, which we checked for using the procedures recommended by Werner et al.43 Due to the exist-
ence of a national regulator dataset, we were able to test for differences between the sample and the
population on observable variables. It is important to note that not only do we compare the sample
and population for differences in observables, we also statistically correct for these differences using
the Heckman44 correction, a two-stage regression model that first models the probability of becom-
ing a survey participant and then adjusts the results of the final regression to mimic the population.

One problem with a low response rate is that although we statistically control for observable
variables, we cannot know whether administrators who were more comfortable with technology
and more knowledgeable about its implementation were more likely to respond to the survey. The
statistical problem that would result would be restriction of range, as the potential respondents who
did poorly on technology implementation would not respond. Restriction of range makes it more
difficult to detect a true relationship that exists since the observations on one side of the distribu-
tion are omitted. If this is true, the actual importance of technological implementation procedures
is greater than we have found, but the absolute value of the means of our statistics is inflated. This
does not call into question the core finding of this article.

Another potential problem with survey research of our type is common method bias. We test for
this with the Harman single-factor test, which suggested little common method variance.47 Because
our finding that nursing homes that used better implementation processes had higher quality of
care outcomes was replicated when we used objective regulator data, and that those results could
not be due to common method bias, we have more confidence in our results regarding the impact
of implementation processes on quality outcomes in nursing homes than in our results about the
direct relationship between EMR implementation and nursing home outcomes, which relied upon
same-source data.

Conclusion

We analyzed the extent to which EMR implementation impacts quality and efficiency outcomes in
nursing homes and examined the extent to which implementation processes impacted these out-
comes. We linked an original survey of executives in seven states to objective regulator data. We
used qualitative case studies and CFA to generate the scales used in the analysis and regressions
controlling for some types of sample selection to test our hypotheses. We found that EMR use led
to positive administrator-reported efficiency and quality outcomes, but was unrelated to regulatory
quality outcomes. We did, however, find that the following implementation processes improved
efficiency-related outcomes, increased the speed at which residents received care, and decreased
regulatory citations in nursing homes: (1) early communication with key stakeholders, (2) using
additional staff during early adoption stages, (3) high levels of staff training, (4) incremental EMR
roll-outs, and (5) ensuring staff access to on-site technical assistance. Our literature review and
analysis also suggest that change leadership practices that value innovation and employee empow-
erment can result in nursing homes being adaptive to implementing health IT and make the imple-
mentation process more streamlined. We encourage practitioners to utilize these practices and
researchers to examine whether our findings generalize to other industries or technologies.

2262 Health Informatics Journal 26(3)

Acknowledgements

We are grateful to Jason Smith, David Coursey, Jason Lambert, and Elena Radeva. IRB approval was obtained
from University of Texas at Arlington IRB Protocol: 2011-0532 “A comparative study of the implementation
of health information technology,” 11 June 2011.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publi-
cation of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Darla J Hamann https://orcid.org/0000-0001-6492-0672

References

1. Abramson EL, McGinnis S, Moore J, et al. A statewide assessment of electronic health record adoption
and health information exchange among nursing homes. Health Serv Res 2014; 49(1 pt. 2): 361–372.

2. Hitt LM and Tambe P. Health care information technology, work organization, and nursing home per-
formance. Ind Labor Relat Rev 2016; 69(4): 834–859.

3. Pillemer K, Meador RH, Teresi JA, et al. Effects of electronic health information technology implemen-
tation on nursing home resident outcomes. J Aging Health 2012; 24(1): 92–112.

4. Qian S, Yu P and Hailey DM. The impact of electronic medication administration records in a residential
aged care home. Int J Med Inform 2015; 84(11): 966–973.

5. Rantz MJ, Hicks L, Petroski GF, et al. Cost, staffing and quality impact of bedside electronic medical
record (EMR) in nursing homes. J Am Med Dir Assoc 2010; 11(7): 485–493.

6. Weech-Maldonado R, Davlyatov G and Lord J. EHR implementation among nursing homes: is it associ-
ated with better financial performance? Innov Aging 2018; 2(Suppl_1): 599–600.

7. Bezboruah K, Hamann DJ and Smith J. Management attitudes and technology adoption in long-term care
facilities. J Health Organ Manag 2014; 28(3): 344–365.

8. Kramer A, Richard A, Epstein A, et al. Understanding the costs and benefits of health information
technology in nursing homes and home health agencies: case study findings. Denver, CO: University of
Colorado, Denver; U. S. Department of Health and Human Services, 2009, http://aspe.hhs.gov/daltcp/
reports/2009/HITcsf.pdf

9. Lipsky DB, Avgar AC and Lamare JR. Organizational strategies for the adoption of electronic medical
records: Toward an understanding of outcome variation in nursing homes. In: Proceedings of the 59th
annual meeting of the labor and employment relations association, San Francisco, CA, 3–6 January
2009, pp. 73–85. Champaign, IL: Labor and Employment Relations Association.

10. Buntin MB, Burke MF, Hoaglin MC, et al. The benefits of health information technology: a review of
the recent literature shows predominantly positive results. Health Aff 2011; 30(3): 464–471.

11. Ko M, Wagner L and Spetz J. Nursing home implementation of health information technology: Review
of the literature finds inadequate investment in preparation, infrastructure, and training. Inquiry 2018;
55: 778902.

12. Wang T and Biedermann S. Adoption and utilization of electronic health record systems by long-term
care facilities in Texas. Perspect Health Inf Manag 2012; 9: 1.

13. Cherry BJ, Ford EW and Peterson LT. Experiences with electronic health records: Early adopters in
long-term care facilities. Health Care Manage Rev 2011; 36: 265–274.

14. Degenholtz HB, Resnick A, Lin M, et al. Development of an applied framework for understanding health
information technology in nursing homes. J Am Med Dir Assoc 2016; 17(5): 434–440.

Hamann and Bezboruah 2263

15. Meehan M. Electronic health records in long-term care: staff perspectives. J Appl Gerontol 2015; 36:
1175–1196.

16. Munyisia EN, Yu P and Hailey D. The changes in caregivers’ perceptions about the quality of informa-
tion and benefits of nursing documentation associated with the introduction of an electronic documenta-
tion system in a nursing home. Int J Med Inform 2011; 80: 116–126.

17. Zhang Y, Yu P and Shen J. The benefits of introducing electronic health records in residential aged care
facilities: a multiple case study. Int J Med Inform 2012; 81(10): 690–697.

18. Judge J, Field TS, DeFlorio M, et al. Prescribers’ responses to alerts during medication ordering in the
long-term care setting. J Am Med Inform Assoc 2006; 13(4): 385–390.

19. Kruse CS, Mileski M, Vijaykumar AG, et al. Impact of electronic health records on long-term care facili-
ties: systematic review. JMIR Med Inform 2017; 5(3): e35.

20. Jiang T, Yu P, Hailey D, Ma J, et al. The impact of electronic health records on risk management of
information systems in Australian residential aged care homes. J Med Syst 2016; 40: 204.

21. Mei YY, Marquarda J, Jacelon C, et al. Designing and evaluating an electronic patient falls reporting
system: perspectives for the implementation of health information technology in long-term residential
care facilities. Int J Med Inform 2013; 8(2): e294–e306.

22. Munyisia E, Yu P and Hailey D. The effect of an electronic health record system on nursing staff time in
a nursing home: a longitudinal study. Australas Med J 2014; 7(7): 285–293.

23. Klinger S and White S. Lessons from a health information technology demonstration in New York
nursing homes. The Commonwealth Fund, 2010, https://www.commonwealthfund.org/publications/
case-study/2010/apr/lessons-health-information-technology-demonstration-new-york (accessed 25
September 2013).

24. Agha L. The effects of health information technology on the costs and quality of medical care. J Health
Econ 2014; 34: 19–30.

25. Spetz J, Burgess JF and Phibbs CS. The effect of health information technology implementation in
Veterans Health Administration hospitals on patient outcomes. Healthc 2014; 2(1): 40–47.

26. Yu P, Zhang Y, Gong Y, et al. Unintended adverse consequences of introducing electronic health records
in residential aged care homes. Int J Med Inform 2013; 82(9): 772–788.

27. Penoyer DA, Cortelyou-Ward KH, Noblin AM, et al. Use of electronic health record documentation by
healthcare workers in an acute care hospital system. J Healthc Manag 2014; 59(2): 130–144.

28. Kellermann AL and Jones SS. What it will take to achieve the as-yet-unfulfilled promises of health
information technology. Health Aff 2013; 32(1): 63–68.

29. Szydlowski S and Smith C. Perspectives from nurse leaders and chief information officers on health
information technology implementation. Hosp Top 2011; 87(1): 3–9.

30. Alexander GL, Rantz M, Flesner FM, et al. Clinical information systems in nursing homes: an evaluation
of initial implementation strategies. Comput Inform Nurs 2007; 25(4): 189–197.

31. Brandeis G, Hogan M, Murphy M, et al. Electronic health record implementation in community nursing
homes. J Am Med Dir Assoc 2007; 8: 31–34.

32. Zarowitz BJ, Resnick B and Ouslander JG. Quality clinical care in nursing facilities. J Am Med Dir
Assoc 2018; 19(10): 833–839.

33. Dumas C and Beinecke RH. Change leadership in the 21st century. J Organ Change Manag 2018; 31(4):
867–876.

34. von Treuer K, Karantzas G, McCabe M, et al. Organizational factors associated with readiness for
change in residential aged care settings. BMC Health Serv Res 2018; 18(1): 77.

35. Chaudhry B, Wang J, Wu S, et al. Systematic review: Impact of health information technology on qual-
ity, efficiency, and costs of medical care. Ann Intern Med 2006; 144: E12–E22.

36. Song PH, McAlearney AS, Robbins J, et al. Exploring the business case for ambulatory electronic health
record system adoption. J Healthc Manag 2011; 56(3): 169–180.

37. Bezboruah K and Hamann DJ. Health IT adoption in nursing homes: the role of IT vendors. Int J Innov
Tech Manag 2018; 15(1): 185001.

38. Hinkin TR. A brief tutorial on the development of measures for use in survey questionnaires. Organ Res
Methods 1998; 1(1): 104–121.

2264 Health Informatics Journal 26(3)

39. Bowen N and Gao S. Structural equation modeling (pocket guides to social work research). New York:
Oxford University Press, 2011.

40. Harold DM, Fedor DB, Caldwell S, et al. The effects of transformational and change leadership on
employees’ commitment to a change: a multilevel study. J Appl Psychol 2008; 93(2): 346–357.

41. Lance CE and Vandenberg RJ. Confirmatory factor analysis. In: Drasgow F and Schmitt N (eds)
Measuring and analyzing behavior in organizations: advances in measurement & data analysis. San
Francisco, CA: Jossey-Bass, 2002, pp. 221–256.

42. Acock AC. Discovering structural equation modeling using stata. College Station, TX: The Stata
Press, 2013.

43. Werner S, Praxedes M and Kim H. The reporting of nonresponse analyses in survey research. Organ Res
Methods, (2007). 10, 287–295.

44. Heckman J. Sample selection bias as a specification error. Econometrica 1979; 47: 153–161.
45. Cuddeback G, Wilson E, Orme JG, et al. Detecting and statistically correcting sample selection bias. J

Soc Serv Res 2004; 30(3): 19–33.
46. Kline WC. Principles and practice of structural equation modeling. 2nd ed. New York: The Guilford

Press, 2005.
47. Podsakoff PM and Organ DW. Self-reports in organizational research: problems and prospects. J Manag

1986; 12: 531–544.
48. De Veer AGE and Francke AL. Attitudes of nursing staff towards electronic patient records: a question-

naire survey. Int J Nurs Stud 2010; 47: 846–854.
49. Lin W and Van Ryzin GG. Web and mail surveys: an experimental comparison of methods for nonprofit

research. Nonprof Volunt Sec Q 2012; 41(6): 1014–1028.
50. Manfreda KL, Bosnjak M, Berzelak J, et al. Web surveys versus other survey modes: a meta-analysis

comparing response rates. Int J Market Res 2008; 50(1): 79–104.
51. Teo T. Online and paper-based survey data: are they equivalent? Brit J Educ Technol 2013; 44(6):

E196–E198.

RESEARCH ARTICLE Open Access

Evaluation of service quality from patients’
viewpoint
Mohammad Ali Abbasi-Moghaddam1, Ehsan Zarei2, Rafat Bagherzadeh3, Hossein Dargahi4 and Pouria Farrokhi3*

Abstract

Background: Measuring patients’ perception from health service quality as an important element in the assessment of
service quality has attracted much attention in recent years. Therefore, this study was conducted to find out how the
patients evaluated service quality of clinics at teaching hospitals affiliated with Tehran University of Medical
Sciences in Iran.

Methods: This cross-sectional study was conducted in Tehran in 2017 and 400 patients were randomly selected from
four hospitals. Data were collected using a questionnaire, the validity and reliability of which were confirmed in previous
study. In order to analyze the data, T-test, ANOVA, and Pearson correlation coefficient were calculated using SPSS 23.

Results: The results indicated that among eight dimensions of health service quality, the patients were more satisfied
with physician consultation, services costs and admission process. The highest and lowest mean scores were related to
physician consultation (Mean = 4.17), and waiting time (Mean = 2.64), in that order. The total mean score of service quality
was 3.73 (± 0.51) out of 5. Outpatient services were assessed as good, moderate and weak by 57.5, 40 and 2.5% of the
patients, respectively. There was a significant relationship between the positive perception of service quality and reason
for admission, source of recommendation, gender, education level, health status, and waiting time in the clinics (p < 0.05).

Conclusion: The majority of the patients had a positive experience with visiting clinics and perceived service provision as
good. In fact, patients’ perceptions of physician consultation, provision of information to patients and the environment of
delivering services, are the most important determinants of service quality in clinics.

Keywords: Patient perception, Service quality, Outpatient services, Quality assessment

Background
The provision of high quality services is a prerequisite
for the success of service organizations since service
quality influences patients’ perceived value, their satis-
faction and faithfulness [1]; therefore, the improvement
of service quality has been on management agenda [2].
Growth in demand for healthcare, increased costs, lim-
ited resources, and the variety of clinical interventions
have led many health systems in the world to focus on
measuring and improving the quality of services. The
first step to this end is to define the concept of quality
that has long been a topic of much controversy [3, 4].
Service quality is a unique and abstract concept which

is difficult to define and measure. Researchers have pro-
vided different definitions [5]. It has been described as

the judgment or overall attitudes of customers towards
the provided services and refers to the differences and
mismatches between customers’ expectations and their
perceptions of service performance [3, 4]. Quality in
health services includes technical (clinical) quality and
functional (non-clinical) quality. The former focuses on
the skills, accuracy of procedures and medical diagnosis
while the latter refers to the way that health services are
provided to the patients [6].
Constant monitoring of health services is very import-

ant, thus measuring patient perception of health care
quality, as a key element in quality assessment, has
gained much attention in recent years. Monitoring pro-
vides important information about service quality which
cannot be obtained through traditional means for per-
formance evaluation [7].
In the past, the process of clinical quality assessment

was conducted without considering the viewpoints and

© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

* Correspondence: [email protected]
3School of Health Management and Information Sciences, Iran University of
Medical Sciences, Tehran, Iran
Full list of author information is available at the end of the article

Abbasi-Moghaddam et al. BMC Health Services Research (2019) 19:170
https://doi.org/10.1186/s12913-019-3998-0

feedback of patients; however, nowadays, emphasis is
placed on the importance of patients’ views in assessing
the quality of services, and mere reliance on clinical
effectiveness is not much supported [8]. The feedback
and opinions of patients or the voice of clients affect
the quality improvement and provides an opportunity
for organizational learning [9]. Patients’ perspective of
healthcare quality is important for several reasons.
First, the high quality of services offered by hospitals
is associated with issues, such as patient satisfaction,
willingness to re-use services in the future, compliance
with doctor’s order, and so on.
Second, patient feedback and perceptions are important

requirements for many accreditation and monitoring pro-
grams for hospital services. Third, high patient-perceived
quality is effectively and positively related to financial per-
formance and profitability of healthcare institutions [10].
Therefore, it can be said that the assessment of service
quality helps service providers recognize the specific and
often unmet needs of patients and problems in the deliv-
ery of services. Moreover, it helps hospital managers de-
sign problem-solving and quality-improvement programs
[11] and allocate resources more effectively and guarantee
high patient satisfaction.
Hospital clinics are one of the most important sources

of patients for inpatient departments; consequently, the
provision of services in this area affects patients’ overall
perception and choice of hospital [12]. Besides, ambula-
tory (outpatient) care is growing at a faster rate than
hospitals, and it is predicted that their revenues would
be equivalent or even surpass inpatient revenues in the
near future [9]. Most studies in Iran have focused on the
quality assessment of primary health care, inpatient ser-
vice quality and patient satisfaction [13], yet outpatient
services have been neglected during the assessment of
hospital services. Therefore, this study aimed to answer
the following research question: how do the patients
assess the quality of services provided by clinics at teach-
ing hospitals affiliated with Tehran University of Medical
Sciences?

Methods
Study design and sample
This was a cross-sectional study conducted on a random
sample of 400 patients who referred to outpatient depart-
ments (clinics) in teaching hospitals affiliated with Tehran
University of Medical Sciences (TUMS) in Tehran during
the first half of 2017. The patients were selected by multi-
stage systematic random sampling, but due to limited time
and resources, only four hospitals (two general and two
specialized) among 16 were randomly chosen, and each
hospital’s share was allocated based on its size (number of
beds). Outpatient departments work six days a week, thus
in order to increase the likelihood of patient participation

in the study, a systematic sampling technique was used to
select patients every day from Saturday to Thursday. The
patients were then asked to complete a questionnaire
before leaving the clinic and following the physician’s
consultation. The individual’s consent was a requirement,
and the patients who declined to participate in the study
(N = 14) were substituted by other patients. Since the per-
ception of quality is a subjective judgment, in order to
have an accurate yet close-to-reality evaluation, only pa-
tients of at least 18 years old and willing to participate
were included in the study.

Instrument
Data were collected using a questionnaire which was
designed and validated in a previous study [14]. The reli-
ability of the instrument, in this study, was assessed
using Cronbach’s Alpha Coefficient, which ranged from
0.6 to 0.9 for service quality dimensions and 0.92 for
overall service quality, indicating the sufficient level of
reliability. The questionnaire consisted of two sections;
the first part included 13 items on demographic and
socio-economic variables, and the second part contained
37 items about hospital’s outpatient services quality; acces-
sibility (three items), appointment (two items), waiting
time (two items), admission process (three items), physical
environment (six items), physician services (eleven items),
disclosure of information to patient (seven items) and cost
of services (three items).
The items were measured on a five-point Likert scale

ranging from 1 (strongly disagree) to 5 (strongly agree).

Data analysis
Data were analyzed by SPSS 23 using T-test, ANOVA
and Pearson correlation to compare service quality in
terms of patients’ demographic variables and assess the
relationship between quality dimensions. In addition to
the main tests, Friedman and Turkey tests were also
used. Regarding the mean score, the overall service qual-
ity was divided into three levels; poor (< 2.5), moderate
(2. 6–3.75) and good (> 3.75) [14].

Results
According to the findings, 221 (55.3%) of the patients
were male and 290 (72.5%) were married. In terms of
education, only 2.3% of the participants were illiterate
and most of participants lived in city (86%).Concerning
income distribution, the results showed that the majority
(63%) of the patients had reported their income as mod-
erate. About 33% of the patients visited hospital clinics
once whereas 25% of the patients visited hospital clinics
more than 5 times. The results also indicated a postop-
erative follow-up for 35% of the visits. Most patients
(44%) were referred to clinics by their physicians, and
the majority of them (about 77%) reported their health

Abbasi-Moghaddam et al. BMC Health Services Research (2019) 19:170 Page 2 of 7

status as good or moderate (Table 1). It was also found
that the minimum, average and maximum waiting time
were 10 min, three, and eight hours, respectively. Fur-
thermore, the lowest, average and the highest service
cost were 0.1, 2.25, and 15.5 USD, respectively (Table 2).
The findings on service quality dimensions indicated the

highest mean score related to physician’s consultation
(4.17) and the lowest to patient waiting time (2.64). Ser-
vice quality dimensions, according to Friedman’s test, were
ranked as follows; physician’s consultation, perceived ser-
vice costs, admission process, disclosure of information to
patient, physical environment, appointment, accessibility
and perceived waiting time (Table 3).
Based on the findings, 2.5% of the respondents assessed

the quality of outpatient services as poor, 40% as moderate
and 57.5% as good. Concerning to service quality dimen-
sions, the patients were mostly satisfied with physician’s
consultation (78.3%), cost of the services (76.5%) and ad-
mission process (62.5%). The patients were least satisfied
with waiting time which was evaluated as poor by 58% of
the patients (Table 4).
Furthermore, a significant correlation was found between

overall service quality and its dimensions, specifically physi-
cian’s consultation (r = 0.766) which was followed by other
dimensions, such as providing information to patient, phys-
ical environment, accessibility, appointment, perceived ser-
vice costs and waiting time (Table 5).
Comparison of mean scores of service quality in terms

of demographic variables showed that the highest quality
score was achieved by female patients, the patients who
referred to clinics due to new disease and those who
were familiar with clinics through media. Service quality
was improved by increasing education level and health
status and reducing waiting time at clinics. No signifi-
cant relationship was found between other variables and
service quality score (Tables 1 and 3).

Discussion
This study aimed to evaluate clinics service quality of
teaching hospitals in Iran from the patients’ viewpoint
and results showed that the overall services quality was
assessed as good by 57.5% of the patients while 2.5% of
the patients defined it as poor. The findings of the study
indicated a better status of service quality compared
with the service quality in Shiraz teaching hospitals
clinics where about 37% of the patients were satisfied
with service quality [12]. In a study conducted by Mpin-
ganjira, the patients reported status of service quality as
good [8]. In another study at cancer clinics in Canada
[15], the quality score was reported above average (3.66)
which is consistent with our result.
The findings demonstrated that the highest score of

service quality was attributed to the physician’s consultation.
Patients often lack sufficient information and knowledge to

Table 1 The relationship between demographic characteristics
and service quality score (N = 400)

Variables N % Mean (±SD) Test results

Gender

Male 221 55.3 3.66 (0.56) T = −2.99
P = 0.003

Female 179 44.8 3.81 (0.42)

Education level

No schooling 9 2.3 3.04 (0.17) F = 11.90
P < 0.001

Primary and Secondary school 162 40.5 3.82 (0.48)

University 229 57.3 3.69 (0.51)

Residence Area

Urban 344 86 3.72 (0.51) T = 0.13
P = 0.89

Rural 56 14 3.73 (0.47)

Marital status

Married 290 72.5 3.70 (0.54) F = 1.71
P = 0.14

Single 88 22 3.78 (0.52)

Widowed 10 2.5 4.04 (0.15)

Divorced 12 3 3.64 (0.19)

Economic status

Excellent 2 0.5 4.14 (0.01) F = 1.46
P = 0.22

Good 62 15.5 3.80 (0.56)

Average 250 62.5 3.69 (0.50)

Low 86 21.5 3.77 (0.50)

Rate of clinic visit

First 130 32.5 3.70 (0.51) F = 3.52
P = 0.08

Second 78 19.5 3.81 (0.54)

Third 60 15 3.78 (0.44)

Fourth 34 8.5 3.91 (0.45)

Fifth or more 98 24.5 3.59 (0.51)

Reason for admission

New disease 136 34 3.81 (0.49) F = 5.50
P = 0.04

Postoperative follow-up 139 34.8 3.62 (0.49)

Previous disease 125 31.2 3.75 (0.53)

Source of recommendation

Doctors 176 44 3.76 (0.55) F = 2.33
P = 0.04

Family 66 16.5 3.63 (0.36)

Friends or Relatives 110 27.5 3.69 (0.52)

Media 15 6.3 3.89 (0.50)

Other patients 23 5.8 3.73 (0.43)

Health status

Excellent 27 6.8 3.89 (0.46) F = 2.67
P = 0.04

Good 125 31.3 3.75 (0.54)

Fair 182 45.5 3.73 (0.51)

poor 66 16.5 3.59 (0.43)

Abbasi-Moghaddam et al. BMC Health Services Research (2019) 19:170 Page 3 of 7

assess the medical staff, and perhaps this is the reason why
they tend to assess them positively [16]. It should also be
noted that in the process of health service delivery, patients
are more sensitive to care provided by physicians and nurses
[17, 18]; in fact, human elements are more important com-
pared with non-human elements in patient perception of
care quality [19]. Doctor-patient interpersonal relationship
also plays a key role in shaping service quality judgments
[20]. Personal relationships greatly affect the service quality
perception since the services are intangible and inseparable
from consumers [21]. The findings of studies in Greece,
Norway, France and Finland, also indicated that the highest
mean score was related to the quality of physician’s consult-
ation [22–25].
Service costs and admission process ranked as the sec-

ond and third highest dimensions of outpatient services
quality. A study in Iran also showed that patients were
satisfied with the cost of outpatient services which is
similar to our findings [14]. According to the health in-
surance law in Iran, the amount of patient copayment
for outpatient services is 30% of the services cost [26]
and in public hospitals, outpatient services such as phy-
sician’s consultation are fully covered by health insur-
ance plans. Therefore, patients pay a small amount for
the outpatient services and are expected to be satisfied
with this dimension of service quality.
The provision of information to patients which had a

high correlation with service quality, took the fourth
rank in this study. This is in contrast with the findings

of other studies in which the patients did not give a high
score to the quality of information; consequently, this di-
mension was not included in the highest ranked dimen-
sions [8, 14, 16, 27].
The appointment process, which ranked fifth, was per-

ceived as moderate and good by approximately 72% of
the patients. The negative perception could be attributed
to bureaucratic processes, lack of proper appointment
systems, or inappropriate staff behavior. The results are
in line with those the findings of studies conducted in
Greece and Norway where the patients also perceived
the quality of appointment process as good and moder-
ate [16, 22, 23].
The sixth rank was related to the clinic environment

where the most important reason for dissatisfaction
seemed to be due to poor hygiene and insufficient num-
bers of seats. This is in accord with the findings of other
studies in which the quality of facilities and physical en-
vironment ranked four among five items [28, 29]. Al-
though the quality of clinic environment does not stand
in a good position in the overall ranking, the majority of
the patients had positively perceived it as moderate and
good (about 83%). This is also in line with the findings
of a study in Johannesburg private clinics, South Africa
[8] as well as a study in outpatient cancer clinics in
Canada [15] where the patients had a positive perception
of the physical environment.
The least positive perception of service quality was

related to waiting time and accessibility to outpatient
services. Long waiting time is the most important reason
for dissatisfaction and decreases patients’ positive per-
ception of services quality [30]. Previous studies have
also indicated that long waiting time at the clinic and
inaccessibility to hospital outpatient services, affect
patients’ dissatisfaction with service quality [25]. It has also
been found that patients had the least positive perception
of waiting time for visiting the physician [14, 15, 31].
Furthermore, the results indicated a significant relation-

ship between gender, education level, reason for admission,

Table 2 The Relationship between age, waiting time and
patient payment with service quality score

Mean SD Correlation
coefficient

p-value

Age 39.9 14.4 −0.017 0.730

Waiting time (min.) 185 99 −0.469 < 0.001

Out of pocket payment (USD) 2 2 −0.090 0.072

Table 3 Mean and standard deviations of service quality dimensions

Dimensions Mean SD Min score Max score Average rating (Friedman test)

Accessibility 3.23 0.82 1 5 4.06

Appointment 3.32 1.18 1 5 4.79

Waiting time 2.64 1 1 5 2.57

Admission process 3.94 0.76 2 5 6.89

Physical environment 3.33 0.78 1 5 4.16

Physician’s consultation 4.17 0.60 2.55 5 7.84

Information provision to patient 3.74 0.83 1.43 5 5.71

Service costs 4.15 0.84 1 5 7.79

Service quality 3.73 0.51 2.24 5 5.55

Abbasi-Moghaddam et al. BMC Health Services Research (2019) 19:170 Page 4 of 7

source of recommendation, health status and waiting time
in the clinic, and service quality. In this study, unlike the
previous studies, the male patients had higher expectations
compared with the female patients and were dissatisfied
with service quality [6, 32]. There was a statistically
significant difference between the patients’ perceptions
of quality and their education, meaning that less edu-
cated patients had the least positive perception of ser-
vice quality. It seems that lower education leads to
more illogical expectations, and this is in contrast
with the results of other studies [6].
Those patients, who referred to clinics due to new

health problems, had a more positive perception in
comparison with the other patients. This could be at-
tributed to some factors, such as recovery from their
previous illnesses, hoping for recovery in the selected
clinic, or lack of familiarity with the details and short-
comings in the service delivery processes. The findings
showed that the patients who got familiar with clinics
through media, gave higher scores to service quality,
this could be due to the fact they might have received

the same services. It was also found that the patients
with better health status had lower expectations and
more positive perceptions. This was consistent with
other studies in which health status was confirmed to
be one of the determinants of patient satisfaction with
service quality [6, 24, 32, 33].
There was no significant relationship between ser-

vice costs and age with service quality; however, they
were negatively correlated with the perception of
service quality, meaning that higher cost and older
age led to less positive perception of quality. Waiting
time in clinics had a significant inverse relationship
with the perception of service quality which has
been expected. It means that long waiting time was
associated with lower positive perceptions of service
quality. The same relationship was found in other
studies [12, 31].
Delays in the provision of hospital services are one of

the key issues in care quality and can lead to a negative
perception of the provided service quality if considered
as unreasonable and unnecessary by patients [34].
Therefore, hospitals should design patient-oriented ser-
vice processes rather than personnel-oriented and im-
prove quality of service delivery through education and
system design [35].

Study limitations
As any other study, this research has some limitations.
Healthcare quality is a broad concept that is affected by
several factors and cannot be adequately explored
through quantitative studies. However, a triangulation of
key informant interviews and focus group discussion
with patients and service providers would provide more
insight into this area. Therefore, it is suggested that po-
tential researchers use the triangulation design to assess
the quality of services.

Table 4 Clinics service quality status from patient’s perspective

Dimensions Good Moderate Poor

N % N % N %

Accessibility 93 23.3 226 56.5 81 20.3

Waiting time 67 16.8 101 25.3 232 58

Admission process 250 62.5 128 32 22 5.5

Physical environment 125 31.3 209 52.3 66 16.5

Physician’s consultation 313 78.3 87 21.8 – –

Information provision to patient 221 55.3 146 36.5 33 8.3

Service costs 306 76.5 74 18.5 20 5

Appointment 190 47.5 97 24.3 113 28.3

Service quality (Total) 230 57.5 160 40 10 2.5

Table 5 Correlation between service quality and its dimensions

Information
provision
to patient

Physician’s
consultation

Admission
process

Accessibility Appointment Waiting
time

Physical
environment

Service
costs

Service
quality

Information provision to patient 1

Physician’s consultation 0.584 1

Admission process 0.163 0.234 1

Accessibility 0.264 0.309 0.336 1

Appointment 0.199 0.176 0.289 0.465 1

Waiting time 0.225 0.271 0.334 0.317 0.331 1

Physical environment 0.313 0.349 0.274 0.343 0.410 0.445 1

Service costs 0.275 0.231 0.346 0.318 0.301 0.219 0.377 1

Service quality 0.729 0.766 0.520 0.579 0.557 0.533 0.693 0.537 1

All correlation was significant at the 0.01 level (2-tailed)

Abbasi-Moghaddam et al. BMC Health Services Research (2019) 19:170 Page 5 of 7

Conclusions
According to the findings, the majority of the patients
had a positive experience with visiting clinics at teaching
hospitals and perceived the service quality as good
(approximately 58%). The most positive perceptions
of the patients were related to the quality of physician
consultation, service costs, admission processes, and
information provision to patient. Also, physician consult-
ation and providing information to patient were two fac-
tors determining clinic’s service quality. For that reason, it
is suggested to improve the ‘disclosure of information to
patients’ which is one of the most important factors in
service quality, and use web based appointment system to
reduce waiting time for physician appointment. It is also
recommended that clinics improve their physical environ-
ment to increase their patient’s positive perceptions. The
findings could be valuable for healthcare managers/pro-
viders and provide them with useful information about
the special needs of their patients and the existing prob-
lems. In this case, they can channel their efforts to satisfy
their patients’ demands and eliminate the weak points.

Abbreviations
ANOVA: Analysis of variance; TUMS: Tehran University of Medical Sciences

Acknowledgments
This research has been supported by Tehran University of Medical Sciences.
The authors would like to thank the individuals and organizations that
contributed to this study, especially the patients and the personnel of the
hospitals under study.

Funding
Not applicable. The project has not received any financial support or grant
from any research or academic institutes.

Availability of data and materials
The data that support the findings of this study are available from the
corresponding author.

Authors’ contributions
MA, EZ, HD and PF contributed substantially to the conception and the
design of the study. PF carried out data collection and statistical analysis. EZ,
PF and RB interpreted the data. MA, EZ, RB and PF drafted and revised the
manuscript. All authors reviewed and approved the final manuscript.

Ethics approval and consent to participate
Ethical approval of the current study was obtained from the Deputy of
Research Affairs, The school of Allied Medical Sciences, Tehran University of
Medical Sciences. The permission to conduct the research was obtained
from the authorities in the study settings. All participants were informed of
the aims of the study and their participation was on voluntary basis. Verbal
informed consents were secured from each participant since according
ethical principles of Iran no written consent is needed for studies including
no invasive clinical techniques. As for the confidentiality of the information,
the participants were not required to write their names, phone numbers,
and their address in the questionnaire. The participants had the right to
refuse participation or withdraw from the study.

Consent for publication
Not applicable.

Competing interest
The authors declare that they have no competing interests.

Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.

Author details
1Department of Health Care Management, School of Allied Medical Sciences,
Tehran University of Medical Sciences, Tehran, Iran. 2Department of Health
Service Management, School of Management and Medical Education, Shahid
Beheshti University of Medical Sciences, Tehran, Iran. 3School of Health
Management and Information Sciences, Iran University of Medical Sciences,
Tehran, Iran. 4Health Information Management Research Center, Tehran
University of Medical Sciences, Tehran, Iran.

Received: 10 April 2018 Accepted: 7 March 2019

References
1. Izadi A, Jahani Y, Rafiei S, Masoud A, Vali L. Evaluating health service quality:

using importance performance analysis. Int. J. Health Care Qual. Assur.
2017;30(7):656–63.

2. Sahney S, Banwet D, Karunes S. An integrated framework for quality in
education: application of quality function deployment, interpretive structural
modelling and path analysis. Total Qual Manag Bus Excell. 2006;17(2):265–85.

3. Camilleri D, O’Callaghan M. Comparing public and private hospital care
service quality. Int. J. Health Care Qual. Assur. 1998;11(4):127–33.

4. Pantoja T, Beltrán M, Moreno G. Patients’ perspective in Chilean primary care: a
questionnaire validation study. Int J Qual Health Care. 2008;21(1):51–7.

5. Cronin JJ Jr, Taylor SA. Measuring service quality: a reexamination and
extension. J Mark. 1992:55–68.

6. Alhassan RK, Duku SO, Janssens W, Nketiah-Amponsah E, Spieker N, van
Ostenberg P, et al. Comparison of perceived and technical healthcare
quality in primary health facilities: implications for a sustainable National
Health Insurance Scheme in Ghana. PLoS One. 2015;10(10):e0140109.

7. Labarere J, Francois P, Auquier P, Robert C, Fourny M. Development of a French
inpatient satisfaction questionnaire. Int J Qual Health Care. 2001;13(2):99–108.

8. Mpinganjira M. Understanding service quality and patient satisfaction in
private medical practice: a case study. Afr J Bus Manag. 2011;5(9):3690.

9. Carlucci D, Renna P, Schiuma G. Evaluating service quality dimensions as
antecedents to outpatient satisfaction using back propagation neural
network. Health care manag sci. 2013;16(1):37–44.

10. De Man S, Gemmel P, Vlerick P, Van Rijk P, Dierckx R. Patients’ and
personnel’s perceptions of service quality and patient satisfaction in nuclear
medicine. Eur J Nucl Med Mol Imaging. 2002;29(9):1109–17.

11. Alrubaiee L, Alkaa’ida F. The mediating effect of patient satisfaction in the
patients’ perceptions of healthcare quality–patient trust relationship. Int J
Mark Stud. 2011;3(1):103.

12. Keshtkaran A, Heydari AR, Keshtkaran V, Taft V, Hashiani A. A. Outpatients
satisfaction level of teaching hospitals clinics in shiraz. J Monit. 2012;11(4):
459–65. (In Persian).

13. Moosazadeh M, Nekoei-moghadam M, Amiresmaili M. Determining the
level of hospitalized patients’ satisfaction of hospitals: a systematic review
and meta-analysis. J Hospital. 2013;12(1):77–87.

14. Zarei E. Service quality of hospital outpatient departments: patients’
perspective. Int. J. Health Care Qual. Assur. 2015;28(8):778–90.

15. Roberge D, Tremblay D, Turgeon M-È, Berbiche D. Patients’ and
professionals’ evaluations of quality of care in oncology outpatient clinics.
Support Care Cancer. 2013;21(11):2983–90.

16. Ekaterina G, Stavros K, Anca M, Lambrini K. Measurement of patient
satisfaction as a quality Indicator of hospital health services: the case of
outpatient clinics in general hospital. Science. 2017;5(2):128–35.

17. Dagger TS, Sweeney JC, Johnson LW. A hierarchical model of health service
quality: scale development and investigation of an integrated model. J Serv
Res. 2007;10(2):123–42.

18. Narang R. Measuring perceived quality of health care services in India. Int. J.
Health Care Qual. Assur. 2010;23(2):171–86.

19. Suki NM, Lian JCC, Suki NM. A comparison of human elements and
nonhuman elements in private health care settings: customers’ perceptions
and expectations. J. Hosp. Mark. Public Relations. 2009;19(2):113–28.

20. Padma P, Rajendran C, Sai Lokachari P. Service quality and its impact on
customer satisfaction in Indian hospitals: perspectives of patients and their
attendants. BIJ. 2010;17(6):807–41.

Abbasi-Moghaddam et al. BMC Health Services Research (2019) 19:170 Page 6 of 7

21. Brady MK, Cronin JJ Jr. Some new thoughts on conceptualizing perceived
service quality: a hierarchical approach. J Mark. 2001;65(3):34–49.

22. Aletras VH, Papadopoulos EA, Niakas DA. Development and preliminary
validation of a Greek-language outpatient satisfaction questionnaire with
principal components and multi-trait analyses. BMC Health Serv Res. 2006;6(1):66.

23. Danielsen K, Bjertnaes OA, Garratt A, Forland O, Iversen HH, Hunskaar S. The
association between demographic factors, user reported experiences and
user satisfaction: results from three casualty clinics in Norway. BMC Fam
Pract. 2010;11(1):73.

24. Gasquet I, Villeminot S, Estaquio C, Durieux P, Ravaud P, Falissard B.
Construction of a questionnaire measuring outpatients’ opinion of quality of
hospital consultation departments. Health Qual Life Outcomes. 2004;2(1):43.

25. Säilä T, Mattila E, Kaila M, Aalto P, Kaunonen M. Measuring patient
assessments of the quality of outpatient care: a systematic review. J Eval
Clin Pract. 2008;14(1):148–54.

26. Davari M, Haycox A, Walley T. The Iranian health insurance system; past
experiences, present challenges and future strategies. Iran J Public Health.
2012;41(9):1–9.

27. Arab M, Tajvar M, Akbari F. Selection an appropriate leadership style to
direct hospital manpower. Iran J Public Health. 2006;35(3):64–9.

28. Chakravarty A. Evaluation of service quality of hospital outpatient
department services. Medical Journal Armed Forces India. 2011;67(3):221–4.

29. Kaya SD, Maimaiti N, Gorkemli H. Assessing patient satisfaction with
obstetrics and gynaecology clinics/outpatient department in university
hospital Konya, Turkey. Int J Res Med Sci. 2017;5(9):3794–7.

30. McMullen M, Netland PA. Wait time as a driver of overall patient satisfaction in
an ophthalmology clinic. Clinical ophthalmology (Auckland, NZ). 2013;7:1655.

31. Nabbuye-Sekandi J, Makumbi FE, Kasangaki A, Kizza IB, Tugumisirize J,
Nshimye E, et al. Patient satisfaction with services in outpatient clinics at
Mulago hospital, Uganda. Int J Qual Health Care. 2011;23(5):516–23.

32. Rahmqvist M. Patient satisfaction in relation to age, health status and other
background factors: a model for comparisons of care units. Int J Qual
Health Care. 2001;13(5):385–90.

33. Cohen G. Age and health status in a patient satisfaction survey. Soc Sci
Med. 1996;42(7):1085–93.

34. Duggirala M, Rajendran C, Anantharaman R. Patient-perceived dimensions
of total quality service in healthcare. BIJ. 2008;15(5):560–83.

35. Kim Y-K, Cho C-H, Ahn S-K, Goh I-H, Kim H-J. A study on medical services quality
and its influence upon value of care and patient satisfaction–focusing upon
outpatients in a large-sized hospital. Total Qual Manag. 2008;19(11):1155–71.

Abbasi-Moghaddam et al. BMC Health Services Research (2019) 19:170 Page 7 of 7

BioMed Central publishes under the Creative Commons Attribution License (CCAL). Under
the CCAL, authors retain copyright to the article but users are allowed to download, reprint,
distribute and /or copy articles in BioMed Central journals, as long as the original work is
properly cited.

  • Abstract
    • Background
    • Methods
    • Results
    • Conclusion
  • Background
  • Methods
    • Study design and sample
    • Instrument
    • Data analysis
  • Results
  • Discussion
    • Study limitations
  • Conclusions
  • Abbreviations
  • Acknowledgments
  • Funding
  • Availability of data and materials
  • Authors’ contributions
  • Ethics approval and consent to participate
  • Consent for publication
  • Competing interest
  • Publisher’s Note
  • Author details
  • References

2/24/22, 12:48 PM Rubric Detail – Blackboard Learn

https://class.waldenu.edu/webapps/bbgs-deep-links-BBLEARN/app/course/rubric?course_id=_16936340_1&rubric_id=_2920716_1 1/4

Rubric Detail
Select Grid View or List View to change the rubric’s layout.

  Excellent Good Fair Poor

Conduct research
on how system
implementations
similar to the one
you select have
been previously
evaluated. After
exploring similar
system
implementations,
select one
research goal and
viewpoint to use
in the evaluation.

23 (23%) – 25
(25%)

Three
appropriate
articles are
researched, and
one research
goal and one
viewpoint are
identi�ed clearly
with speci�c
detail regarding
how system
implementation
evaluations
researched are
similar to the
selected model.

20 (20%) – 22
(22%)

Three
appropriate
articles are
researched, and
one research
goal and one
viewpoint are
identi�ed with
some detail
regarding how
system
implementation
evaluations
researched are
similar to the
selected model.

18 (18%) – 19
(19%)

Three
appropriate
articles are
researched, and
one research
goal and one
viewpoint are
identi�ed with
details regarding
how system
implementation
evaluations
researched are
similar to the
selected model
that are vague,
inaccurate, or
omitted.

0 (0%) – 17 (17%)
Less than three
articles are
researched,
and/or articles
are
inappropriate.
Research goals,
viewpoints, or
details regarding
how system
implementation
evaluations
researched are
similar to the
selected are
vague,
incomplete, or
missing.

In a 3- to 4-page
paper:

Identify which of
the cases from
the text you will
be evaluating in
this project,
summarize the
case, and explain
why you selected
the system
featured in the
case.

23 (23%) – 25
(25%)

The response
clearly,
accurately, and
with speci�c
detail
summarizes the
selected case
study and clearly
justi�es the
selection.

20 (20%) – 22
(22%)

The response
accurately
summarizes the
selected case
study and
justi�es the
selection.

18 (18%) – 19
(19%)

The response
summarizes the
selected case
study with a few
vague and/or
inaccurate
details, and/or
justi�es the
selection with
vague and/or
inaccurate
details.

0 (0%) – 17 (17%)
The response
summarizes the
selected case
study with many
vague and/or
inaccurate
details, and/or
justi�es the
selection with
vague,
inaccurate, or
missing details.

Name: NURS_6541_Week5_Assignment_Rubric EXIT

Grid View List View

2/24/22, 12:48 PM Rubric Detail – Blackboard Learn

https://class.waldenu.edu/webapps/bbgs-deep-links-BBLEARN/app/course/rubric?course_id=_16936340_1&rubric_id=_2920716_1 2/4

  Excellent Good Fair Poor

Critique the HIT
implementations
in your research:
Identify
successful and
unsuccessful
elements of the
implementations.
Explain the
reasons for the
successful
elements.
Identify areas for
improvement and
explain why or
how they could
be improved.

23 (23%) – 25
(25%)

The response
clearly and with
speci�c detail
critiques the HIT
implementations
by clearly
identifying
successful and
unsuccessful
elements of the
implementations
researched,
explaining in
detail the
reasons for the
successful
elements, and
clearly
identifying areas
for improvement
with
explanations of
why or how they
could be
improved.

20 (20%) – 22
(22%)

The response
clearly critiques
the HIT
implementations
by identifying
successful and
unsuccessful
elements of the
implementations
researched,
explaining the
reasons for the
successful
elements, and
identifying areas
for improvement
with
explanations of
why or how they
could be
improved.

18 (18%) – 19
(19%)

The response
critiques the HIT
implementations
with a few vague
and/or
incomplete
details regarding
successful and
unsuccessful
elements of the
implementations
researched,
reasons for the
successful
elements, and
areas for
improvement
with
explanations of
why or how they
could be
improved.

0 (0%) – 17 (17%)
The response
critiques the HIT
implementations
with several
vague,
incomplete,
and/or missing
details regarding
successful and
unsuccessful
elements of the
implementations
researched,
reasons for the
successful
elements, and
areas for
improvement
with
explanations of
why or how they
could be
improved.

Create an
evaluation goal
and identify the
viewpoint related
to the goal that
will guide your
own evaluation
plan. Provide
your rationale for
choosing that
goal and
viewpoint.

9 (9%) – 10 (10%)
The respone
clearly and with
speci�c detail
describes an
evaluation goal
and identi�es
the viewpoint
related to the
goal that will
guide the
evaluation plan,
providing
rationale for
choosing that
goal and
viewpoint.

8 (8%) – 8 (8%)
The respone
describes an
evaluation goal
and identi�es
the viewpoint
related to the
goal that will
guide the
evaluation plan,
providing
rationale for
choosing that
goal and
viewpoint.

7 (7%) – 7 (7%)
The respone
describes an
evaluation goal
with a few vague
and/or
incomplete
details regarding
identi�cation of
the viewpoint
related to the
goal that will
guide the
evaluation plan
and rationale for
choosing that
goal and
viewpoint.

0 (0%) – 6 (6%)
The respone
describes an
evaluation goal
with several
vague,
incomplete,
and/or missing
details regarding
identi�cation of
the viewpoint
related to the
goal that will
guide the
evaluation plan
and rationale for
choosing that
goal and
viewpoint.

2/24/22, 12:48 PM Rubric Detail – Blackboard Learn

https://class.waldenu.edu/webapps/bbgs-deep-links-BBLEARN/app/course/rubric?course_id=_16936340_1&rubric_id=_2920716_1 3/4

  Excellent Good Fair Poor

Written
Expression and
Formatting —
Paragraph
Development and
Organization:

Paragraphs make
clear points that
support well-
developed ideas,
�ow logically, and
demonstrate
continuity of
ideas. Sentences
are carefully
focused—neither
long and
rambling nor
short and lacking
substance. A
clear and
comprehensive
purpose
statement and
introduction are
provided that
delineate all
required criteria.

5 (5%) – 5 (5%)
Paragraphs and
sentences follow
writing
standards for
�ow, continuity,
and clarity.

A clear and
comprehensive
purpose
statement,
introduction,
and conclusion
are provided
that delineate all
required criteria.

4 (4%) – 4 (4%)
Paragraphs and
sentences follow
writing
standards for
�ow, continuity,
and clarity 80%
of the time.

Purpose,
introduction,
and conclusion
of the
assignment are
stated, yet are
brief and not
descriptive.

3 (3%) – 3 (3%)
Paragraphs and
sentences follow
writing
standards for
�ow, continuity,
and clarity 60%–
79% of the time.
Purpose,
introduction,
and conclusion
of the
assignment are
vague or o�
topic.

0 (0%) – 2 (2%)
Paragraphs and
sentences follow
writing
standards for
�ow, continuity,
and clarity < 60%
of the time.

No purpose
statement,
introduction, or
conclusion were
provided.

Written
Expression and
Formatting —
English Writing
Standards:

Correct grammar,
mechanics, and
proper
punctuation

5 (5%) – 5 (5%)
Uses correct
grammar,
spelling, and
punctuation with
no errors.

4 (4%) – 4 (4%)
Contains a few
(1 or 2)
grammar,
spelling, and
punctuation
errors.

3 (3%) – 3 (3%)
Contains several
(3 or 4)
grammar,
spelling, and
punctuation
errors.

0 (0%) – 2 (2%)
Contains many
(≥ 5) grammar,
spelling, and
punctuation
errors that
interfere with
the reader’s
understanding.

2/24/22, 12:48 PM Rubric Detail – Blackboard Learn

https://class.waldenu.edu/webapps/bbgs-deep-links-BBLEARN/app/course/rubric?course_id=_16936340_1&rubric_id=_2920716_1 4/4

  Excellent Good Fair Poor

Written
Expression and
Formatting —
The paper follows
correct APA
format for title
page, headings,
font,
spacing, margins,
indentations,
page numbers,
running heads,
parenthetical/in-
text citations,
and reference
list.

5 (5%) – 5 (5%)
Uses correct APA
format with no
errors.

4 (4%) – 4 (4%)
Contains a few
(1 or 2) APA
format errors.

3 (3%) – 3 (3%)
Contains several
(3 or 4) APA
format errors.

0 (0%) – 2 (2%)
Contains many
(≥ 5) APA format
errors.

Total Points: 100

Name: NURS_6541_Week5_Assignment_Rubric

EXIT

3/28/22, 4:12 PM Originality Report

https://class.waldenu.edu/webapps/mdb-sa-BBLEARN/originalityReport/ultra?attemptId=11a513ab-977e-46ba-8546-e604d6a2f8c8&course_id=_1693… 1/7

%71

%4

%2

SafeAssign Originality Report
NURS-6451A-1/NURS-6451C-1/NURS-6451S-1/NURS-6451-1-Eval Met… • SafeAssign Drafts

%77Total Score: High riskHetal Patel
Submission UUID: d96fc70d-088a-c3fd-2bc1-5b4b63b435c1

Total Number of Reports

1
Highest Match

77 %
WK5Assgn_Patel_H.docx

Average Match

77 %
Submitted on

03/27/22
06:23 PM PDT

Average Word Count

1,427
Highest: WK5Assgn_Patel_H.docx

%77Attachment 1

Institutional database (11)

Student paper Student paper Student paper

Student paper Student paper Student paper

Student paper Student paper Student paper

Student paper Student paper

Global database (3)

Student paper Student paper Student paper

Internet (1)

mq

Top sources (3)

Excluded sources (0)

View Originality Report – Old Design

Word Count: 1,427
WK5Assgn_Patel_H.docx

2 6 1

4 12 9

15 5 14

3 8

13 11 7

10

2 Student paper 6 Student paper 13 Student paper

1 7

Health Information Technology – Evaluation Plan Project Hetal Patel

College of Nursing, Walden University

NURS 6451: Evaluation Methods for Health Information Technology

Dr. Oscar Lee

March 28, 2022

Health Information Technology – Evaluation Plan Project It was in the 1960s that health information technology (HIT) was initially introduced to the healthcare in-

dustry, with the goal of assisting mainframes in the processing of financial transactions during business hours. As a result of HIT’s contributions, there has been a
mixed response, with a number of studies looking at this link as a result of its contributions. This paper investigates the impact of computerized incident report-

ing as a health information technology (HIT) system on medical outcomes in a somewhat complex healthcare ecosystem that necessitates the consideration of multi-
ple factors in order to ensure maximum efficiency in a highly delicate practice where human lives are at stake. Legendary Health Systems is the subject of this paper
(LHS). Performing a thorough review of health information technology systems is essential because, by constructing an organized analytical framework, healthcare
practitioners will be able to objectively uncover problems, hazards, and inadequacies without the effect of prejudice or systematic errors. Because of its rele-

1

2

2

2

patelhh0
Highlight
patelhh0
Highlight
patelhh0
Highlight
patelhh0
Highlight
patelhh0
Highlight
patelhh0
Highlight

3/28/22, 4:12 PM Originality Report

https://class.waldenu.edu/webapps/mdb-sa-BBLEARN/originalityReport/ultra?attemptId=11a513ab-977e-46ba-8546-e604d6a2f8c8&course_id=_1693… 2/7

Source Matches (36)

vance as a tool for continuous quality improvement, the computerized incident reporting system (CIRS) is one of the most widely utilized health information technol-
ogy (HIT) systems, as demonstrated in LHS (CQI) (Abbasi-Moghaddam, Zarei, Bagherzadeh, Dargahi & Farrokhi, 2019). Through the use of a cohort study, we were able
to undertake an objective and empirical evaluation of the adoption of CIRS in the workplace. In order to conduct this evaluation, a cohort would be established,

which would consist of identifying several healthcare organizations with a similar structure and output as LHS, and conducting an objective analysis of the implemen-
tation process of CIRS, as well as the impact of this HIT system in identifying and rectifying errors within these ecosystems, would be established. Not least among
these goals is continual improvement in care quality through the identification and correction of systemic problems and flaws that are frequent in primary health care
delivery operations, which is the goal of the integration of information technology and healthcare delivery. This is performed through the use of information technol-
ogy (IT) to identify and track the occurrence of errors, with the goal of reducing the severity and frequency of errors. Incident reporting (IR) is the core premise on
which CIRS is constructed (Ramirez et al., 2018). When correctly implemented, health information technology (HIT) systems have the potential to have a hugely

positive impact on the delivery of primary care (Kruse, & Beane, 2018). Any faults in the design and/or execution of health information technology (HIT) systems, on
the other hand, might add another layer of complexity to an already complex healthcare delivery environment. A variety of negative repercussions are projected for
the primary care delivery process, including delayed and staggered therapy administration as a result of poor human-computer interactions and/or data loss, as well
as prescription and dosage problems. This is why an evaluation of health information technology (HIT) systems is critical in order to determine the viability and impact
of such systems on healthcare delivery. Following an increase in HIT usage across the United States, a diverse range of HIT innovations have been implemented that
have been instrumental in improving primary care delivery. These innovations have been implemented as a result of patient-centered implementation efforts

that cater to the dynamic needs of healthcare delivery in the United States (Yen et al., 2017). Improving healthcare dependability requires preventing errors, identify-
ing and correcting errors, and taking appropriate steps to ensure that optimum quality standards are maintained in the future (Georgiou et al., 2019). A system of
electronic health records (EHRs) is one of the more complex health information technology (HIT) systems that has been widely adopted throughout the world and has
proven to be one of the most beneficial and innovative technological breakthroughs. The extensive database on electronic health records (EHR) that has been

compiled through years of inquiry and research, as well as the evaluation of EHR as a health information technology system through the use of case studies, provides

an objective perspective on the implementation process and the impact of EHR on the delivery of healthcare services to patients (Carayon, Smith, Hundt,
Kuruchittham, & Li, 2009). After conducting an objective analysis of multiple case studies on EHR utilization, it has been shown that this practice is valid as a result of
significant performance improvement. Even though it is an expensive endeavor, designing EHR systems in-house allows healthcare facilities to work with more exten-
sive data effectively and profitably through health analytics, despite the fact that it is an expensive task.

2

2

2

2

Enabling healthcare institutions to work with more extensive data reliably and effectively using health analytics, has the potential to greatly improve primary health
care delivery. Another key HIT method that has acquired widespread acceptance in recent years all around the world is e-prescribing, which allows for the administra-
tion of medications from a distance without the use of a physical prescription. Technology breakthroughs in information technology have resulted in the development
of highly adaptive systems that cater to patient support and caregiver engagement in order to enhance patient outcomes in recent years, thanks to advances in infor-
mation technology (Tayal, Gunasekaran, Singh, Dubey, & Papadopoulos, 2017). Researchers conducted a study to determine the influence of e-prescribing on health-
care delivery and discovered a mixed bag of results. More advances in HIT technology advancements are required, such as speeding the accuracy and sensitivity of
prescriptions, as well as making the human-computer connection more comfortable. When this system fails, wrong prescriptions or dosages are given, which can
have a potentially harmful effect that can even be lethal in some cases if not handled correctly. Through the integration of health and wellness data streams, analytics,
and trend visualization, health information technology has opened up a plethora of new pathways in biological research and electronic health data, resulting in a slew
of new healthcare delivery options (Georgiou, Thomas, Dahm & Westbrook, 2019). These health information technology systems are used to provide data on risk

estimation, wellness trend projections, and knowledge of complex disease predictions, allowing for a data-driven precise practice that ultimately improves the effi-
ciency and quality of healthcare delivery (Zayas-Cabán, & Wald, 2020). The goal of evaluating these HIT systems is to objectively analyze the implementation and im-
pact of these technological advancements on patient outcomes, as poor implementation or systemic flaws constitute a substantial threat to patient safety. Errors in

prescriptions or dosages can frequently result in death. The loss of patient data as a result of system failures can have a negative impact on the treatment

process and patient outcomes. As a result, better execution of HIT systems, which are complex healthcare delivery interventions with the potential to improve or
harm patient outcomes almost in equal measure, is made possible through evaluation. References

Abbasi-Moghaddam, M. A., Zarei, E., Bagherzadeh, R., Dargahi, H., & Farrokhi, P. (2019). Evaluation of service quality from patients’ viewpoint.

BMC Health Services Research, 19(1), 1-7. https://doi.org/10.1186/s12913-019-3998-0

Carayon, P., Smith, P., Hundt, A. S., Kuruchittham, V., & Li, Q. (2009). Implementation of an electronic health records system in a small clinic: the view-

point of clinic staff. Behaviour & Information Technology, 28(1), 5-20. https://doi.org/10.1080/01449290701628178

Georgiou, A., Li, J., Thomas, J., Dahm, M. R., & Westbrook, J. I. (2019). The impact of health information technology on the management and follow-up of

test results–a systematic review. Journal of the American Medical Informatics Association, 26(7), 678-688. https://doi.org/10.1093/jamia/ocz032

Kruse, C. S., & Beane, A. (2018). Health information technology continues to show positive effect on medical outcomes: systematic review. Journal of

medical Internet research, 20(2), e8793. https://doi.org/10.2196/jmir.8793

Tayal, A., Gunasekaran, A., Singh, S. P., Dubey, R., & Papadopoulos, T. (2017). Formulating and solving sustainable stochastic dynamic facility layout problem: A

key to sustainable operations. Annals of Operations Research, 253(1), 621-655. https://doi.org/10.1007/s10479-016-2351-9

Yen, P. Y., McAlearney, A. S., Sieck, C. J., Hefner, J. L., & Huerta, T. R. (2017). Health information technology (HIT) adaptation: refocusing on the journey to suc-

cessful HIT implementation. JMIR medical informatics, 5(3), e7476. https://doi.org/10.2196/medinform.7476

Zayas-Cabán, T., & Wald, J. S. (2020). Opportunities for the use of health information technology to support research. JAMIA open, 3(3), 321-325.

https://doi.org/10.1093/jamiaopen/ooaa037

2

2

3 4 2 5

1

6 6 7

6 8

9 10 2

10 11

2 2 2

12

13

2 2

12

14 2 15

3/28/22, 4:12 PM Originality Report

https://class.waldenu.edu/webapps/mdb-sa-BBLEARN/originalityReport/ultra?attemptId=11a513ab-977e-46ba-8546-e604d6a2f8c8&course_id=_1693… 3/7

Student paper 100%

Student paper 65%

Student paper 64%

Student paper 70%

Student paper 79%

Student paper 67%

1

Student paper

Evaluation Methods for Health Information Technology

Original source

Evaluation Methods for Health Information Technology

2

Student paper

Health Information Technology – Evaluation Plan Project It was in the 1960s that health
information technology (HIT) was initially introduced to the healthcare industry, with the
goal of assisting mainframes in the processing of financial transactions during business
hours.

Original source

In the 1960s, health information technology (HIT) was introduced to the healthcare indus-
try to help mainframes in operations with financial transactions

2

Student paper

This paper investigates the impact of computerized incident reporting as a health infor-
mation technology (HIT) system on medical outcomes in a somewhat complex healthcare
ecosystem that necessitates the consideration of multiple factors in order to ensure max-
imum efficiency in a highly delicate practice where human lives are at stake.

Original source

Improving healthcare involves improving reliability by preventing errors, identifying these
errors, and taking action to maintain optimal quality levels (Georgiou et al., 2019)This pa-
per looks into the impact of computerized incident reporting as a HIT system on medical
outcomes within a somewhat complex healthcare ecosystem that requires the factoring
in of multiple elements towards ensuring maximum efficiency in a highly delicate practice
where human lives are at stake by analyzing computerized incident-reporting systems as
utilized by Legendary Health Systems (LHS)

2

Student paper

Because of its relevance as a tool for continuous quality improvement, the computerized
incident reporting system (CIRS) is one of the most widely utilized health information
technology (HIT) systems, as demonstrated in LHS (CQI) (Abbasi-Moghaddam, Zarei,
Bagherzadeh, Dargahi & Farrokhi, 2019).

Original source

Computerized incident reporting system (CIRS) is one of the most widely implemented
HIT systems due to its practicality as a tool for continuous quality improvement (CQI), as
exemplified in LHS

2

Student paper

In order to conduct this evaluation, a cohort would be established, which would consist of
identifying several healthcare organizations with a similar structure and output as LHS,
and conducting an objective analysis of the implementation process of CIRS, as well as
the impact of this HIT system in identifying and rectifying errors within these ecosystems,
would be established.

Original source

A cohort study involves identifying several healthcare organizations with a similar struc-
ture and output as LHS and carrying out an objective analysis of the implementation
process of CIRS and the impact of this HIT system in identifying and rectifying errors
within these ecosystems

2

Student paper

When correctly implemented, health information technology (HIT) systems have the po-
tential to have a hugely positive impact on the delivery of primary care (Kruse, & Beane,
2018). Any faults in the design and/or execution of health information technology (HIT)
systems, on the other hand, might add another layer of complexity to an already complex
healthcare delivery environment. A variety of negative repercussions are projected for
the primary care delivery process, including delayed and staggered therapy administra-
tion as a result of poor human-computer interactions and/or data loss, as well as pre-
scription and dosage problems.

Original source

HIT systems, when implemented appropriately, hold the potential to realize an im-
mensely positive impact on primary care delivery (Kruse, & Beane, 2018) However, any
flaws in the design and or implementation of HIT systems can add a dimension of com-
plexity to an already complex healthcare delivery ecosystem This pronounces various ad-
verse effects on the process of primary care delivery, such as delayed and staggered de-
livery of treatment as a result of poor human-computer interactions and or data loss and
prescription and dosage errors (Yen et al., 2017)

3/28/22, 4:12 PM Originality Report

https://class.waldenu.edu/webapps/mdb-sa-BBLEARN/originalityReport/ultra?attemptId=11a513ab-977e-46ba-8546-e604d6a2f8c8&course_id=_1693… 4/7

Student paper 68%

Student paper 68%

Student paper 86%

Student paper 75%

Student paper 71%

Student paper 84%

Student paper 100%

Student paper 100%

2

Student paper

These innovations have been implemented as a result of patient-centered implementa-
tion efforts that cater to the dynamic needs of healthcare delivery in the United States
(Yen et al., 2017).

Original source

In the recent past, increased HIT usage across the United States has seen the implemen-
tation of an array of HIT innovations that have been instrumental in improving primary
care in delivery due to patient-centered implementation efforts that cater to the dynamic
needs of healthcare delivery (Yen et al., 2017)

2

Student paper

The extensive database on electronic health records (EHR) that has been compiled
through years of inquiry and research, as well as the evaluation of EHR as a health infor-
mation technology system through the use of case studies, provides an objective per-
spective on the implementation process and the impact of EHR on the delivery of health-
care services to patients (Carayon, Smith, Hundt, Kuruchittham, & Li, 2009).

Original source

The extensive database on EHR arrived at through years of inquiry and research, and
evaluation of EHR as a HIT system by utilizing case studies offers an objective outlook on
the implementation process and impact of EHR on delivery of healthcare

2

Student paper

These health information technology systems are used to provide data on risk estimation,
wellness trend projections, and knowledge of complex disease predictions, allowing for a
data-driven precise practice that ultimately improves the efficiency and quality of health-
care delivery (Zayas-Cabán, & Wald, 2020). The goal of evaluating these HIT systems is to
objectively analyze the implementation and impact of these technological advancements
on patient outcomes, as poor implementation or systemic flaws constitute a substantial
threat to patient safety.

Original source

These HIT systems are used to provide data on risk estimation, wellness trend projec-
tions, and knowledge of complex disease predictions allowing for a data-driven precise
practice that ultimately improves efficiency and quality of healthcare delivery (Zayas-
Cabán, & Wald, 2020) Evaluating these HIT systems aims to objectively assess the imple-
mentation and impact of these technological innovations on patient outcomes as poor
implementation or systemic errors pose a significant threat to patient safety

2

Student paper

The loss of patient data as a result of system failures can have a negative impact on the
treatment process and patient outcomes. As a result, better execution of HIT systems,
which are complex healthcare delivery interventions with the potential to improve or
harm patient outcomes almost in equal measure, is made possible through evaluation.

Original source

Loss of patient data due to system failures can offset the treatment process and ad-
versely affect patient outcomes Evaluation, therefore, allows for better execution of HIT
systems, which are complex healthcare delivery interventions that have the potential to
improve or harm patient outcomes almost in equal measure significantly

3

Student paper

Abbasi-Moghaddam, M.

Original source

Abbasi-Moghaddam, M.A., Zarei, E

4

Student paper

A., Zarei, E., Bagherzadeh, R., Dargahi, H., & Farrokhi, P.

Original source

Abbasi-moghaddam, M.A., Zarei, E., Bagherzadeh, R., Dargahi, H., & Farrokhi, P

2

Student paper

Evaluation of service quality from patients’ viewpoint.

Original source

Evaluation of service quality from patients’ viewpoint

5

Student paper

BMC Health Services Research, 19(1), 1-7.

Original source

BMC Health Services Research, 19(1), 1–7

3/28/22, 4:12 PM Originality Report

https://class.waldenu.edu/webapps/mdb-sa-BBLEARN/originalityReport/ultra?attemptId=11a513ab-977e-46ba-8546-e604d6a2f8c8&course_id=_1693… 5/7

Student paper 100%

Student paper 100%

Student paper 100%

Student paper 100%

Student paper 100%

Student paper 78%

Student paper 87%

mq 100%

Student paper 100%

mq 100%

1

Student paper

https://doi.org/10.1186/s12913-019-3998-0

Original source

https://doi.org/10.1186/s12913-019-3998-0

6

Student paper

Carayon, P., Smith, P., Hundt, A. S., Kuruchittham, V., & Li, Q.

Original source

Carayon, P., Smith, P., Hundt, A S., Kuruchittham, V., & Li, Q

6

Student paper

Implementation of an electronic health records system in a small clinic:

Original source

Implementation of an electronic health records system in a small clinic

7

Student paper

the viewpoint of clinic staff.

Original source

The Viewpoint of Clinic Staff

6

Student paper

Behaviour & Information Technology, 28(1), 5-20.

Original source

Behaviour & Information Technology, 28(1), 5-20

8

Student paper

https://doi.org/10.1080/01449290701628178

Original source

https://doi.org/10

9

Student paper

Georgiou, A., Li, J., Thomas, J., Dahm, M.

Original source

Georgiou, A., Li, J., Thomas, J., Dahm, M., & Westbrook, J

10

Student paper

R., & Westbrook, J.

Original source

R., & Westbrook, J

2

Student paper

The impact of health information technology on the management and follow-up of test
results–a systematic review.

Original source

The impact of health information technology on the management and follow-up of test
results – A systematic review

10

Student paper

Journal of the American Medical Informatics Association, 26(7), 678-688.

Original source

Journal of the American Medical Informatics Association, 26(7), 678-688

3/28/22, 4:12 PM Originality Report

https://class.waldenu.edu/webapps/mdb-sa-BBLEARN/originalityReport/ultra?attemptId=11a513ab-977e-46ba-8546-e604d6a2f8c8&course_id=_1693… 6/7

Student paper 93%

Student paper 100%

Student paper 100%

Student paper 90%

Student paper 100%

Student paper 94%

Student paper 100%

Student paper 97%

Student paper 76%

Student paper 100%

11

Student paper

https://doi.org/10.1093/jamia/ocz032

Original source

doi:https://doi.org/10.1093/jamia/ocz032

2

Student paper

S., & Beane, A.

Original source

S., & Beane, A

2

Student paper

Health information technology continues to show positive effect on medical outcomes:

Original source

Health information technology continues to show positive effect on medical outcomes

2

Student paper

Journal of medical Internet research, 20(2), e8793.

Original source

Journal of Medical Internet Research, 20(2)

12

Student paper

https://doi.org/10.2196/jmir.8793

Original source

https://doi.org/10.2196/jmir.8793

13

Student paper

Formulating and solving sustainable stochastic dynamic facility layout problem: A key to
sustainable operations. Annals of Operations Research, 253(1), 621-655.

Original source

Formulating and solving sustainable stochastic dynamic facility layout problem A key to
sustainable operations Annals of Operations Research, 621-655

2

Student paper

Y., McAlearney, A. S., Sieck, C. J., Hefner, J. L., & Huerta, T.

Original source

Y., McAlearney, A S., Sieck, C J., Hefner, J L., & Huerta, T

2

Student paper

Health information technology (HIT) adaptation: refocusing on the journey to successful
HIT implementation. JMIR medical informatics, 5(3), e7476.

Original source

Health Information Technology (HIT) adaptation Refocusing on the journey to successful
HIT implementation JMIR Medical Informatics, 5(3)

12

Student paper

https://doi.org/10.2196/medinform.7476

Original source

https://doi.org/10.2196/10426

14

Student paper

Zayas-Cabán, T., & Wald, J.

Original source

Zayas-Cabán, T., & Wald, J

3/28/22, 4:12 PM Originality Report

https://class.waldenu.edu/webapps/mdb-sa-BBLEARN/originalityReport/ultra?attemptId=11a513ab-977e-46ba-8546-e604d6a2f8c8&course_id=_1693… 7/7

Student paper 100%

Student paper 100%

2

Student paper

Opportunities for the use of health information technology to support research. JAMIA
open, 3(3), 321-325.

Original source

Opportunities for the use of health information technology to support research Jamia
Open, 3(3), 321–325

15

Student paper

https://doi.org/10.1093/jamiaopen/ooaa037

Original source

https://doi.org/10.1093/jamiaopen/ooaa037

Writerbay.net

Everyone needs a little help with academic work from time to time. Hire the best essay writing professionals working for us today!

Get a 15% discount for your first order


Order a Similar Paper Order a Different Paper