Defining Youth Outcomes for STEM Learning in Afterschool (Part 1)

By Sam Piha

Sam Piha

Recently, we learned that the Afterschool Alliance has a staff person dedicated to the STEM movement in afterschool programming. We also learned that she was a leader in a recent study, which worked with afterschool stakeholders to identify honest youth outcomes of STEM afterschool and summer programs. What we found most interesting about her study was her findings about which outcomes afterschool leaders were most/least confident they could demonstrate. Using this "weighted" approach to discussing the proposed youth outcomes was very useful in raising issues and questions worth exploring.

Dr. Anita Krishnamurthi

Below is an interview with Dr. Anita Krishnamurthi, Director of STEM Policy at the Afterschool Alliance. In this interview, Dr. Krishnamurthi discusses her new study and the relevance of the Learning in Afterschool & Summer learning principles. 

Q: How do you view the role of afterschool and summer programs in promoting STEM learning?

A: Afterschool and summer programs that offer STEM learning opportunities are critical for young people, especially those who may not have the opportunities to enroll in enrichment activities that their more affluent peers might.  We know that children from populations that are traditionally under-represented in STEM fields participate in afterschool programs in large numbers.  Afterschool programs provide an opportunity to engage the very children we need to reach out to the most, offering hands-on learning experiences that can get them excited about STEM topics and careers.  Afterschool and summer programs also provide a flexible and personalized learning environment, allowing all children to engage in STEM fields on their own terms and see how they can make it “fit” for themselves.


Q: Do these informal youth programs have unique advantages over in-school STEM learning? If so, what would you point to? 


A: We need to shift our mindset about education away from treating school and out-of-school learning as completely separate things and instead move to one where we are thinking about providing an ecosystem for learning.  In such a system, school and out-of-school learning are partners with the end goal of supporting children who are engaged and excited about learning and gaining the holistic set of skills and proficiencies they need to become successful adults.  It is very difficult for schools to do this all by themselves and it is also not fair to expect schools to do this alone when there are many willing and capable partners.  

Informal youth programs, such as afterschool programs, have some unique traits that are really well-suited for STEM learning—the informal nature of such programs allows children to play around with STEM topics without a fear of failure and to explore topics in a manner that helps to bring the school-day learning to life.  This is a really important piece that doesn’t often get talked about—proficiency is not the only thing that will persuade young people to pursue these fields.  

What is more important is that they associate STEM fields with having fun while doing intellectually interesting and challenging things, and see careers in these fields as personally fulfilling as well as contributing meaningfully to their communities and the larger world.  Interest, engagement, and proficiency are part of a cycle—they feed off of each other and we need all of these components to get to the end goal of a young person who makes a decision to engage and stick with STEM fields.


Q: In your work at the Afterschool Alliance, you recently led a study of afterschool and STEM. Why did you conduct this? 


A: We found that a lot of people are very supportive of afterschool as a concept and were very willing to consider that it could be a good partner in STEM education.  But they wanted more clarity about exactly what afterschool programs can contribute to this national priority of reforming and improving STEM education.  

The afterschool field has generally always said that it can get children excited about STEM, however, we found that this was seen as a necessary but completely insufficient deliverable by the wider STEM education community. We knew that getting children excited about STEM was a foundational contribution of afterschool programs but that children were also learning many valuable skills and proficiencies in afterschool STEM programs. At the same time, there has been increasing interest in defining assessments for informal science education in a variety of settings including afterschool programs.  

We wanted to make sure that the afterschool field had a say in this discussion and could define appropriate and feasible youth outcomes for STEM learning in afterschool.  Without input from the afterschool field, we would be saddled with measures that didn’t quite fit the setting.  We know that test scores don’t necessarily capture all the impacts of afterschool STEM programs, but we need to get specific about what else afterschool programs are able to deliver.  It is not sufficient to reject test scores as a measure of the afterschool field’s impact on STEM learning without offering other measureable outcomes that are more reflective of our impact in this area.  

We know that afterschool STEM programs have some amazing results, so our study was an attempt to define and articulate those outcomes.  We hoped this would provide the field with a set of achievable goals and also provide some common language around how to describe them.


Q: In your study, you settle on a few STEM outcomes that afterschool programs should claim. What are those? 

A: There are three major developmental outcomes for children in afterschool STEM programs that the participants in our study agreed on.  
Through afterschool STEM programs, children and youth:

• Develop interest in pursuing STEM learning activities. (“I like to do this.”)
• Develop capacities to productively engage in STEM learning activities. (“I can do this.”)
• Develop expanded value for and commitment to pursuing STEM learning activities and pathways. (“This is important to me.”)

In addition, the participants in our study identified and ranked a set of six indicators of learning, which are concrete ways that young people demonstrate progress toward the intended program outcomes with respect to STEM learning.  These start to get more specific and include items like “Active participation in STEM learning opportunities,” “Ability to productively engage in STEM processes of investigation,” and “Ability to exercise STEM-relevant life and career skills.” 

We then got even more specific and asked them about “sub-indicators,” which represent specific, measureable dimensions of the indicators. The study results indicate that afterschool providers and supporters have a high level of confidence that afterschool programs are well positioned to support and expand those sub-indicators that stress the doing of science and developing “21st century skills” such as forming questions, problem solving and working in teams. They expressed a medium level of confidence in sub-indicators that addressed larger systems level issues that they may not have control over (such as pursuing other school-based or out-of-school STEM learning opportunities) or delved into specific issues such as understanding how to pursue STEM careers.  

But we think that this set of outcomes, indicators, and sub-indicators provides a good framework for how afterschool providers can think of their youth outcomes for STEM learning. It captures what is happening but provides specific language that the larger STEM education community will recognize and value.  I also want to stress that the outcomes framework we present is not intended to represent a set of mandatory goals for all afterschool STEM programs.  We know the afterschool STEM field is very diverse and the impacts are entirely dependent on the particular circumstances of each program (such as age of participants, resources, goals, community context, etc.). You can download the study report to learn more!



_________________________
Dr. Anita Krishnamurthi is the Director of STEM Policy at the Afterschool Alliance. She leads efforts to advance policies, research and partnerships so children and youth can have rich STEM education experiences in their afterschool programs. An astronomer by training, Anita received her PhD from The Ohio State University, conducting her postdoctoral work at the University of Colorado at Boulder.  Over the past decade, she has been deeply involved in science education and outreach through a range of roles at the National Academy of Sciences, NASA and the American Astronomical Society.  Anita strongly believes that afterschool programs play a critical role in STEM education reform and must be treated as strategic partners.

The Importance of Social-Emotional Learning

By Sam Piha

Joseph Durlak

Joseph Durlak and his team have produced some seminal studies that examine the impact of in-school and afterschool programs that promote social-emotional learning. Two of his most important studies offered a meta-analysis, examining high-quality evaluations of the programs and their many effects on youth outcomes. We urge the readers to review these studies when reflecting on the implementation of their own programs. Readers should also visit the Collaborative for Academic, Social, and Emotional Learning (CASEL) website. Below we offer a brief interview with Dr. Durlak. 

Q: In partnership with a number of your colleagues, you conducted two meta-analyses regarding the effectiveness in school and after school efforts to promote social-emotional learning. Can you briefly explain what is meant by social-emotional learning and competencies? 

A: In brief, social and emotional learning (or SEL) focuses on the systematic development of a core set of five interrelated social, cognitive, emotional, and behavioral competencies that help children more effectively handle life challenges and thrive in both their learning and their social environments. These skills fall into five areas:   self-awareness, self-management, social awareness, relationship skills, and responsible decision making. Representative competencies in these areas include the abilities to recognize and manage emotions, set and achieve positive goals, appreciate the perspectives of others, establish and maintain positive relationships, make responsible decisions, and handle interpersonal situations constructively (Collaborative for Academic, Social, and Emotional Learning, 2013).

Q: In your research you were interested in whether youth gained social-emotional competencies and whether this had an impact on school behavior and academic success. What did you learn?

A: Overall, we found that the 213 programs that we evaluated were successful in improving students functioning in all six of the outcome areas we examined.  

In perhaps the most important finding, programs were able to improve students’ academic performance (grades and test scores) to a degree that corresponds to an 11% percentile gain in academic performance.  This type of gain corresponds to what many educational interventions have been able to do based on other evaluative research.  SEL Programs also significantly improved students’ self-perceptions, their social and emotional skills, and their prosocial attitudes (e.g., toward school ) and significantly reduced levels of conduct problems and emotional distress (e.g., symptoms of anxiety and depression).   

Q: In these studies, the effectiveness of these efforts were conditional on the quality of the implementation. You identified some key characteristics of implementation that were vital to the program’s success. Can you briefly describe these characteristics? 

A: Extensive research on both children and adults suggests that skills training is most effective when certain principles are followed. Many of these principles are consistent with educational research on effective teaching practices. 

We found that four characteristics representing four of these principles were associated with programs that were more effective than others.  We constructed the acronym, SAFE, to capture these features. ‘S’ stands for Sequential, meaning that programs had a carefully sequenced plan that was designed to develop SEL skills.   This was usually accomplished by a specific curriculum or protocol that built competencies in a coordinated, step-by-step fashion.  ‘A’ stands for Active and means that active forms of learning involving practice and feedback were used to develop skills. Hands-on learning followed by constructive feedback leads to skills acquisition. ‘F’ stands for Focused, meaning that sufficient time was set aside exclusively for skill training, ‘E’ stands for Explicit and means that the skills targeted in each program were clearly explained to students so they knew what was expected of them. 

Aside from these four program characteristics, we also found that programs that were associated with better implementation were more effective than those that experienced problems with implementation.  Implementation involves several dimensions but basically refers to the extent to which intended programs are conducted as planned. 

In other words, effective SEL programs are both well-designed from the start and well executed when they are put into action.   

Q: There is growing discussion of the importance of non-cognitive skills. How are these non-cognitive skills related to the SEL skills that are promoted by the Collaborative for Academic, Social, and Emotional Learning (CASEL)?

A: In general, there is now growing evidence that success in school and later life is not dependent solely on one’s cognitive or intellectual abilities.  The different domains of functioning are not independent. We have shown that fostering students’ personal and social development via SEL programs can also improve their academic development.   

Q: The LIAS project is promoting 5 key principles that we should apply in our afterschool and summer programs: learning that is active, collaborative, meaningful, supports mastery, and expands horizons. Can you say a bit about the relevance of these principles and how they overlap, or not with social-emotional learning?

A: There is considerable overlap and sometimes only the terms are different. For example, learning that is active and that supports mastery relates to both the ‘A’ and ‘F’ of our acronym in that students need to practice new skill and we must devote sufficient time and attention to foster student mastery. 

The LIAS principles of collaboration, meaningfulness, and expanded horizons are each consistent with the types of skills that compose SEL such as skills relating to managing one’s emotions, developing and maintaining satisfactory relationships with others, and enhancing self-awareness.  In general, the LIAS principles and the five SEL domains allows flexibility and adaptations to occur for work with different types of youth at different developmental stages, and with different needs and interests.

___________________________

Joseph Durlak is Emeritus Professor of Clinical Psychology at Loyola University, Chicago. He currently lives in Santa Fe. New Mexico, where he remains active writing, editing, consulting, and reviewing. His primary interests are in prevention programs for children and adolescents, meta-analysis, community psychology, and social and emotional learning programs. He has published major reviews on prevention programs for youth, after-school programs, program implementation, and school-based social and emotional learning programs.  He is currently editing contributions for the Handbook of Social and Emotional Learning, to be published by Guilford in late 2013.