admin
Thu, 03/28/2024 - 19:09
Edited Text
Development of Civic Learning and Engagement Outcomes
Alyssa Hilliard, Mathematics and Statistics, anh1018@sru.edu
Introduction
Results
In community-engaged learning, there is a lack of comprehensive assessment
tools that assess all major areas. The Office for Community-Engaged Learning
has developed a set of sixteen civic learning and engagement outcomes (CLEOs)
to determine the quality of community-engaged learning efforts on campus. Each
outcome falls in the category of civic knowledge, skills, values, or action and
includes 2 – 5 quantitative and 1 – 3 qualitative questions. Two CLEOs are
selected per service-learning course, and a pre-assessment and post-assessment
are sent to students in the course.
Results for Academic Service-Learning Courses
To test the validity of the survey questions, cognitive interviewing was used for a
total of four outcomes and the process will continue until every outcome has been
tested.
Outcome
Civic Knowledge
Outcome 1**
Civic Knowledge
Outcome 2**
Civic Skills Outcome 1*
Civic Skills Outcome 4*
* Significant at α=0.05
Materials & Methods
To assess the outcomes, pre-assessments and post-assessments were
administered at the beginning and end of the semester. A paired t-test was then
used to analyze the data and decide if there was a true difference in the answers.
CK04
CK04
Wording for 1st Round
(06/29/2021)
Likert
I am knowledgeable
about the demographic
composition of the
Slippery Rock
community outside of
the University campus.
Likert
I know how to obtain
information about the
demographic, social,
cultural, life-style, and
religious composition of
a local community.
Notes
S1 – did not
understand
demographic
S2 – did not
understand
demographic
S1 – did not
understand
demographic
S2 – did not
understand
demographic
3.676
4.176
0.500
<0.0001
3.591
4.109
3.970
0.379
0.0401
4.324
0.215
0.0160
** Significant at α=0.01
The CLEOs have been used for three courses in Fall 2021 and are being used
for four courses in Spring 2022.
The Bonner Leader Program will also be assessed using combinations of the
CLEOs for each year.
The courses in the fall semester implemented a total of four CLEOs, and each
class had statistically significant results for at least one CLEO each.
Overall, two CLEOs had p-values less than α=0.01. The other two CLEOs
were significant at α=0.05.
•
•
Limitations
CLEOs 2, 4, 8, and 12 were tested using cognitive interviewing
After three rounds of testing for CLEO 4 and CLEO 8, all questions except one
were understood by participants
After two rounds of testing for CLEO 2 and CLEO 12, all questions were
validated, and participants understood what was being asked
Students misunderstood words such as “marginalized” and “demographics”
The question in CLEO 4 that did not improve after changing the wording two
times has been tabled for consideration.
Type of
Question
0.0041
•
Results for Cognitive Interviewing
Learning
Outcome
0.526
Many issues were found in the wording of the four CLEOs that were tested
Cognitive interviewing must be done on survey questions to determine if
students understand what is being asked.
Results
•
•
4.053
P-value
•
•
•
•
3.526
Difference
(post-pre)
Discussion
After creating the assessments, cognitive interviewing was used to test a total of
four outcomes and will be used to test the remaining twelve outcomes. There are
two main methods of cognitive interviewing: think aloud and verbal probing. The
think aloud technique was used in combination with verbal probing for the CLEOs.
•
•
PrePostassessment assessment
Average
Average
Future Work
•
•
•
Cognitive testing occurred over Zoom; little is known about the effects of
video-based cognitive interviewing.
The RockServe platform currently does not have an automated way to assign
specific pre- and post-assessment questions to specific courses; all coding of
the pre- and post-questionnaires was completed course by course.
Qualitative questions are more difficult to assess and interpret and are not
included in this poster.
Wording for 2nd Round
(06/30/2021)
I am knowledgeable about
the distribution of age,
gender, race, and other
demographic characteristics
of the Slippery Rock
community outside of the
University campus.
I know how to obtain
information about the age,
gender, race, and other
characteristics of a local
community.
Wording for 3rd Round
Notes
(07/02/2021)
Notes
S3/S4 - did not I am knowledgeable about the 3rd round volunteers were still
think of
percentage of people of
confused as to what is means
Slippery Rock
by “Slippery Rock Community
different ages, genders,
community. Di
outside of campus.” One
races, ethnic origins, and
d not
thought we meant the offsexualities within the Slippery
understand
Rock community outside of
campus student housing.
distribution
the University campus.
Tabled for consideration
I know how to obtain
information about the
S4 – did not
percentage of people of
understand
different ages, genders, races,
No issues
ethnic origins, and
“information”
sexualities within a
community.
•
Future work will include testing of the remaining twelve CLEOs and
development of a rating scale for the qualitative questions.
References
1. Association of American Colleges and Universities (AAC&U). (2009). Civic Engagement Value
Rubric.
2. Bringle, R. G. (2017). Hybrid high-impact pedagogies: Integrating service-learning with three other
high-impact pedagogies.
3. Bringle, Robert G, Hatcher, Julie A, & Hahn, Thomas W. (2016). Research on student civic outcomes
in service learning: conceptual frameworks and methods (Vol. 3, IUPUI series on service learning
research). Stylus Publishing, LLC. Large Guide
4. Clayton, P., Bringle, R., & Hatcher, J. (2013). Research on service learning. Volume 2A, Students and
faculty: Conceptual frameworks and assessment (IUPUI series on service learning research ; Volume
2A). Large guide.
5.Doolittle, A., & Faul, A. C. (2013). Civic engagement scale: A validation study. Sage Open, 3(3),
2158244013495542.
6. Einfeld, A., & Collins, D. (2008). The relationships between service-learning, social justice,
multicultural competence, and civic engagement. Journal of College Student Development, 49(2), 95109.
7. Felten, P., & Clayton, P. H. (2011). Service‐learning. New directions for teaching and learning,
2011(128), 75-84.
8. Franke, R., Ruiz, S., Sharkness, J., DeAngelo, L., & Pryor, J. (2010). Findings from the 2009
administration of the College Senior Survey (CSS): National aggregates. Higher Education Research
Institute at the University of California Los Angeles, Los Angeles, CA.
9. Gelmon, S. (2018). Assessing Service-Learning and Civic Engagement: Principles and Techniques.
Campus Compact.
10. Gottlieb, K., & Robinson, G. (Eds.). (2006). A practical guide for integrating civic responsibility into
the curriculum. Amer. Assn. of Community Col.
11. Hatcher, J. A. (2011). Assessing Civic Knowledge and Engagement. New Directions for Institutional
Research, 2011(149), 81-92.
12. Hébert, Ali, & Hauf, Petra. (2015). Student learning through service learning: Effects on academic
development, civic responsibility, interpersonal skills and practical skills. Active Learning in Higher
Education, 16(1), 37-49.
13.Higher Education Research Institute. (2021). College Senior Survey. HERI.
14. Keen, C. (2010). New Efforts to Assess Civic Outcomes. Journal of College and Character, 10(7), 18.
15. Lau, K. H., & Snell, R. S. (2021). Confirmatory factor analysis for a service-learning outcomes
measurement scale (S-LOMS). Metropolitan Universities, 32(1), 3-34.
16. Levesque-Bristol, C., & Richards, K. A. R. (2014). Evaluating civic learning in service-learning
programs: Creation and validation of the Public Affairs Scale–Short Survey (PAS-SS). Journal of Public
Affairs Education, 20(3), 413-428.
17. Locklin, Reid B. (2012). Civic Engagement in Higher Education: Concepts and Practices - By
Barbara Jacoby and Associates. Teaching Theology & Religion, 15(2), 196-197.
18. McIlrath, L., & Lyons, A. (Eds.). (2012). Higher education and civic engagement: Comparative
perspectives. Springer.
19. Moely, B. E., McFarland, M., Miron, D., Mercer, S., & Ilustre, V. (2002). Changes in college students'
attitudes and intentions for civic involvement as a function of service-learning experiences. Michigan
Journal of Community Service Learning, 9(1).
20. Molee, L. M., Henry, M. E., Sessa, V. I., & McKinney-Prupis, E. R. (2011). Assessing learning in
service-learning courses through critical reflection. Journal of Experiential Education, 33(3), 239-257.
21. Prentice, Mary. (2007). Social Justice Through Service Learning: Community Colleges as Ground
Zero. Equity & Excellence in Education, 40(3), 266-273.
22. Reiff, John. (2014). Civic Learning. Massachusetts Department of Higher Education
23. Terkla, Dawn Geronimo, & O'Leary, Lisa S. (2015). Assessing Civic Engagement (J-B IR Single
Issue Institutional Research). Somerset: John Wiley & Sons, Incorporated.
24. Torney-Purta, Judith, Cabrera, Julio C, Roohr, Katrina Crotts, Liu, Ou Lydia, & Rios, Joseph A.
(2015). Assessing Civic Competency and Engagement in Higher Education: Research Background,
Frameworks, and Directions for Next-Generation Assessment. ETS Research Report Series, 2015(2), 148.
Acknowledgements
•
•
•
Jeffrey Rathlef
Dr. Jana Asher
The Office for Community-Engaged Learning
Alyssa Hilliard, Mathematics and Statistics, anh1018@sru.edu
Introduction
Results
In community-engaged learning, there is a lack of comprehensive assessment
tools that assess all major areas. The Office for Community-Engaged Learning
has developed a set of sixteen civic learning and engagement outcomes (CLEOs)
to determine the quality of community-engaged learning efforts on campus. Each
outcome falls in the category of civic knowledge, skills, values, or action and
includes 2 – 5 quantitative and 1 – 3 qualitative questions. Two CLEOs are
selected per service-learning course, and a pre-assessment and post-assessment
are sent to students in the course.
Results for Academic Service-Learning Courses
To test the validity of the survey questions, cognitive interviewing was used for a
total of four outcomes and the process will continue until every outcome has been
tested.
Outcome
Civic Knowledge
Outcome 1**
Civic Knowledge
Outcome 2**
Civic Skills Outcome 1*
Civic Skills Outcome 4*
* Significant at α=0.05
Materials & Methods
To assess the outcomes, pre-assessments and post-assessments were
administered at the beginning and end of the semester. A paired t-test was then
used to analyze the data and decide if there was a true difference in the answers.
CK04
CK04
Wording for 1st Round
(06/29/2021)
Likert
I am knowledgeable
about the demographic
composition of the
Slippery Rock
community outside of
the University campus.
Likert
I know how to obtain
information about the
demographic, social,
cultural, life-style, and
religious composition of
a local community.
Notes
S1 – did not
understand
demographic
S2 – did not
understand
demographic
S1 – did not
understand
demographic
S2 – did not
understand
demographic
3.676
4.176
0.500
<0.0001
3.591
4.109
3.970
0.379
0.0401
4.324
0.215
0.0160
** Significant at α=0.01
The CLEOs have been used for three courses in Fall 2021 and are being used
for four courses in Spring 2022.
The Bonner Leader Program will also be assessed using combinations of the
CLEOs for each year.
The courses in the fall semester implemented a total of four CLEOs, and each
class had statistically significant results for at least one CLEO each.
Overall, two CLEOs had p-values less than α=0.01. The other two CLEOs
were significant at α=0.05.
•
•
Limitations
CLEOs 2, 4, 8, and 12 were tested using cognitive interviewing
After three rounds of testing for CLEO 4 and CLEO 8, all questions except one
were understood by participants
After two rounds of testing for CLEO 2 and CLEO 12, all questions were
validated, and participants understood what was being asked
Students misunderstood words such as “marginalized” and “demographics”
The question in CLEO 4 that did not improve after changing the wording two
times has been tabled for consideration.
Type of
Question
0.0041
•
Results for Cognitive Interviewing
Learning
Outcome
0.526
Many issues were found in the wording of the four CLEOs that were tested
Cognitive interviewing must be done on survey questions to determine if
students understand what is being asked.
Results
•
•
4.053
P-value
•
•
•
•
3.526
Difference
(post-pre)
Discussion
After creating the assessments, cognitive interviewing was used to test a total of
four outcomes and will be used to test the remaining twelve outcomes. There are
two main methods of cognitive interviewing: think aloud and verbal probing. The
think aloud technique was used in combination with verbal probing for the CLEOs.
•
•
PrePostassessment assessment
Average
Average
Future Work
•
•
•
Cognitive testing occurred over Zoom; little is known about the effects of
video-based cognitive interviewing.
The RockServe platform currently does not have an automated way to assign
specific pre- and post-assessment questions to specific courses; all coding of
the pre- and post-questionnaires was completed course by course.
Qualitative questions are more difficult to assess and interpret and are not
included in this poster.
Wording for 2nd Round
(06/30/2021)
I am knowledgeable about
the distribution of age,
gender, race, and other
demographic characteristics
of the Slippery Rock
community outside of the
University campus.
I know how to obtain
information about the age,
gender, race, and other
characteristics of a local
community.
Wording for 3rd Round
Notes
(07/02/2021)
Notes
S3/S4 - did not I am knowledgeable about the 3rd round volunteers were still
think of
percentage of people of
confused as to what is means
Slippery Rock
by “Slippery Rock Community
different ages, genders,
community. Di
outside of campus.” One
races, ethnic origins, and
d not
thought we meant the offsexualities within the Slippery
understand
Rock community outside of
campus student housing.
distribution
the University campus.
Tabled for consideration
I know how to obtain
information about the
S4 – did not
percentage of people of
understand
different ages, genders, races,
No issues
ethnic origins, and
“information”
sexualities within a
community.
•
Future work will include testing of the remaining twelve CLEOs and
development of a rating scale for the qualitative questions.
References
1. Association of American Colleges and Universities (AAC&U). (2009). Civic Engagement Value
Rubric.
2. Bringle, R. G. (2017). Hybrid high-impact pedagogies: Integrating service-learning with three other
high-impact pedagogies.
3. Bringle, Robert G, Hatcher, Julie A, & Hahn, Thomas W. (2016). Research on student civic outcomes
in service learning: conceptual frameworks and methods (Vol. 3, IUPUI series on service learning
research). Stylus Publishing, LLC. Large Guide
4. Clayton, P., Bringle, R., & Hatcher, J. (2013). Research on service learning. Volume 2A, Students and
faculty: Conceptual frameworks and assessment (IUPUI series on service learning research ; Volume
2A). Large guide.
5.Doolittle, A., & Faul, A. C. (2013). Civic engagement scale: A validation study. Sage Open, 3(3),
2158244013495542.
6. Einfeld, A., & Collins, D. (2008). The relationships between service-learning, social justice,
multicultural competence, and civic engagement. Journal of College Student Development, 49(2), 95109.
7. Felten, P., & Clayton, P. H. (2011). Service‐learning. New directions for teaching and learning,
2011(128), 75-84.
8. Franke, R., Ruiz, S., Sharkness, J., DeAngelo, L., & Pryor, J. (2010). Findings from the 2009
administration of the College Senior Survey (CSS): National aggregates. Higher Education Research
Institute at the University of California Los Angeles, Los Angeles, CA.
9. Gelmon, S. (2018). Assessing Service-Learning and Civic Engagement: Principles and Techniques.
Campus Compact.
10. Gottlieb, K., & Robinson, G. (Eds.). (2006). A practical guide for integrating civic responsibility into
the curriculum. Amer. Assn. of Community Col.
11. Hatcher, J. A. (2011). Assessing Civic Knowledge and Engagement. New Directions for Institutional
Research, 2011(149), 81-92.
12. Hébert, Ali, & Hauf, Petra. (2015). Student learning through service learning: Effects on academic
development, civic responsibility, interpersonal skills and practical skills. Active Learning in Higher
Education, 16(1), 37-49.
13.Higher Education Research Institute. (2021). College Senior Survey. HERI.
14. Keen, C. (2010). New Efforts to Assess Civic Outcomes. Journal of College and Character, 10(7), 18.
15. Lau, K. H., & Snell, R. S. (2021). Confirmatory factor analysis for a service-learning outcomes
measurement scale (S-LOMS). Metropolitan Universities, 32(1), 3-34.
16. Levesque-Bristol, C., & Richards, K. A. R. (2014). Evaluating civic learning in service-learning
programs: Creation and validation of the Public Affairs Scale–Short Survey (PAS-SS). Journal of Public
Affairs Education, 20(3), 413-428.
17. Locklin, Reid B. (2012). Civic Engagement in Higher Education: Concepts and Practices - By
Barbara Jacoby and Associates. Teaching Theology & Religion, 15(2), 196-197.
18. McIlrath, L., & Lyons, A. (Eds.). (2012). Higher education and civic engagement: Comparative
perspectives. Springer.
19. Moely, B. E., McFarland, M., Miron, D., Mercer, S., & Ilustre, V. (2002). Changes in college students'
attitudes and intentions for civic involvement as a function of service-learning experiences. Michigan
Journal of Community Service Learning, 9(1).
20. Molee, L. M., Henry, M. E., Sessa, V. I., & McKinney-Prupis, E. R. (2011). Assessing learning in
service-learning courses through critical reflection. Journal of Experiential Education, 33(3), 239-257.
21. Prentice, Mary. (2007). Social Justice Through Service Learning: Community Colleges as Ground
Zero. Equity & Excellence in Education, 40(3), 266-273.
22. Reiff, John. (2014). Civic Learning. Massachusetts Department of Higher Education
23. Terkla, Dawn Geronimo, & O'Leary, Lisa S. (2015). Assessing Civic Engagement (J-B IR Single
Issue Institutional Research). Somerset: John Wiley & Sons, Incorporated.
24. Torney-Purta, Judith, Cabrera, Julio C, Roohr, Katrina Crotts, Liu, Ou Lydia, & Rios, Joseph A.
(2015). Assessing Civic Competency and Engagement in Higher Education: Research Background,
Frameworks, and Directions for Next-Generation Assessment. ETS Research Report Series, 2015(2), 148.
Acknowledgements
•
•
•
Jeffrey Rathlef
Dr. Jana Asher
The Office for Community-Engaged Learning