Hi Jeanne - great question. The external research and evaluation team at New Knowledge (NKO) is exploring differences in STEM learning, self-efficacy and interest in STEM fields between the control group (students and teachers in 20 regular SRL classes) and 20 experimental groups. NKO is using quantitative and qualitative data from multiple stakeholders: students, teachers, mentors and members of student social networks. 

Specifically, NewKnowledge developed a survey administered to students at the beginning and end of the school year. This survey and the quasi-experimental research design aimed to measure progress toward project goals, including changes in student science and media literacy. Each student randomly received either the STEM identity or STEM valuation module. NKO also does six site visits each year and interviews teachers and mentors. 

So far, NKO is finding that the curriculum is substantial, yet accessible, and engages students – many of whom did not have strong STEM identities – in thorough research of science and community issues. Fieldwork and interaction with mentors is especially rewarding, with students and teachers noting that speaking with experts helped them to feel more connected to their topics and confident in the quality of their stories. In addition to increasing confidence, NKO also found that the SRL program supports students in developing STEM and media literacy, including effective research and communication skills. Moreover, students who worked on STEM stories identified more strongly with STEM topics after the program than those who did not work on the STEM stories.

I've seen it first hand with students gaining confidence, communication skills and becoming much more interested in STEM topics such as health, the environment and infrastructure issues. It's really remarkable to see the students in all different communities and classroom environments learn to interview experts and produce compelling STEM-focused stories!

Thanks for the detailed response! I love how tangible their products are! What an experience that would be to have a story produced and shared on PBS--so cool! No wonder they feel connected and more confident. Wow! 

Jena, would it be possible to share this tool? I know a lot of folks who would love to see your work! 

Also, I wonder if you've found (or intend to study) impact on future course-taking behaviors and post-secondary degree/career paths. I hope so and would love to hear more about the long-term impact of this work. 

Hi Levi,

We're working on a publication right now. Once it's been through peer review, we'd be happy to share.

As for your second question, we'd absolutely love to get follow-up funding to study these kids longitudinally for the 8 or so years it would take to see their post-secondary career paths. That's always the challenge!

Jena

Hi Levi - Always appreciate wonderings :) I'll have to turn to our research experts at NKO for this one. But I will say that we have seen many different students blossom in the program - the ones who have prior academic achievement AND those who very specifically have not succeeded in traditional assessments - tests, written reports, presentations etc.

I would also add that while we are interested in STEM disciplinary readiness, I believe that STEM literacy is a very important part of civic participation and community engagement no matter what field these adults-to-be pursue.

Since Leah tagged NKO in, we'll take a stab at Levi's question -- it's a big one!

We are seeing increases in identity, literacy, and interest in experimental-group students vis-a-vis control students at a disciplinary level. (Design is quasi-experimental, with classes applying to the STEM program that are then assigned to one group or the other.)

But impact on specific math, science, or CS learning is, frankly, a little hard to say. On students' specific story topics, the results are pretty clear that students are learning a lot. But student ability to abstract to the broader discipline may not be developmentally appropriate to expect. That is, they may not realize until later that "hey, I learned a bunch of CS vocabulary after interviewing this person" can lead to more overall understanding of the field.

The other thing we'd add is that teachers and students alike tell us that student exposure to working adult scientists and engineers (through both interviews and mentorship) is hugely impactful -- it gives students a sense of what those disciplines look like outside the classroom that can be hard to come by at that age. Many can imagine how it might translate into specific career plans or having a plan at all, that awareness may well bear fruit down the line!  Given how much we now anticipate careers will change over a life-course, maybe we're a bit overstating specific career interest when we really need flexibility of mind and awareness to handle uncertain futures with STEM.

Yikes! Oddly, I didn't marry my boss - that was supposed to say Jena Barchas-Lichtenstein and John Fraser, and somehow it decided to only show the very first and very last word as my name. (We hope our significant others laugh.)

Haha! Too funny!

Jena, 

Thanks for the response. I certainly see Leah's point about STEM literacy having an impact on civic participation, and really appreciate that point. I really find STEM literacy to be a challenge to me, but mostly in the sense that I have heard it argued that STEM literacy and STEM are equivalent. I think that's incomplete especially after learning more about it from your perspective. I have a sense that STEM literacy is NECESSARY, but NOT SUFFICIENT (sorry for all-capping you there). That's fine and useful to me. I really feel like I've previously had a sense that STEM literacy was not equivalent to STEM and, therefore, wasn't a part of STEM! Yikes! I'm learning aways!

Hi Levi, It's Bill Swift and I run the STEM and Health Student Reporting Labs, most of our questions have been so technical, it is a joy to have New Knowledge as our partner. 

i just wanted to add my perspective to what you are mulling in terms of STEM Literacy versus STEM.  For one, they are very broad terms, but my hands-on interpretation, or what I strive for in terms of STEM literacy vs STEM is to have my students engage with an aspect of STEM, learn what it really means and become more "fluent" in its concepts and applications.  In my mind, that is what builds "STEM Literacy".  To me STEM literacy is your ability work in and have an all around understanding of how to meaningfully engage in STEM topics.  STEM is just a set of subject matter: Science, Technology, Engineering and Math, there is no saying that you are (think of it like a language) fluent in Science, Technology Engineering or Math, just because you work on a STEM project or STEM Class, but Ideally it is the understanding you develop as a result of working in those disciplines that defines a fluency in STEM terminology, concepts and applications, that in my mind builds "STEM Literacy".  i may be way off base compared to a text book, but that is what I strive for, that comfort and fluency of being able to grapple with STEM in a way that is satisfying and rewarding for students.

My favorite part of what I do is helping to teach students that STEM is fun and engaging, it does not have to painful, difficult and induce brain sweating.  STEM is all around us, STEM is life and nature and rain and sunshine and rainbows, but you have to open your mind to it and see that our world around us is really made from the elements of Science, Technology, Engineering and Math.  Once students get tuned to that, it opens their eyes, and they see STEM everywhere. 

I had a student want to do a story on the conductor of the school orchestra, but they didn't think that was a STEM story.  The crowd management and psychology of how you manage a student orchestra alone is plenty of STEM to work with, but the notes, the scales, the idea that if you blow twice as hard on a trumpet, the note goes up and octave or that music pervades every known culture in the world were some fun concepts I was able to talk with my student about in order to open their eyes that STEM permeates almost every spect of the job of being a school orchestra conductor.

I'm not sure if that helps, but it gives you a little more of a hands on idea of what my days involve.  My goal is not just great videos on STEM subjects, but to help students enjoy and have fun with STEM and not be afraid or shy away from it.  I believe Magic is STEM, and the world is a better place with a little magic in it.