Jamie.. you definitely bring up some challenging issues and we have learned a great deal.  While developing the proposal for WGG we worked with several clubs and began to learn how the culture of different BGCs could vary widely.  Additionally, the level of STEM expertise and interest varied.  Once funded and we expanded the number of participating clubs we purposefully included clubs that were struggling, and clubs that were efficiently run.   We knew to achieve wide dissemination we needed to look broadly at potential uses and our solutions would require a robust design.  We also knew we had a great deal to learn and to accomplish that we had members of the management staff serve as liaisons to different clubs.  They worked with the facilitators during the first two years to learn what we needed to provide in terms of support.  We also used an iterative design process for activities, the WISEngineering activities and software.  In the end we developed facilitator resources that could be readily accessible (actually on a smart phone) and understandable to facilitators with no STEM background.  We created activities that could be completed by middle age youth and included embedded content so that no real teaching was required of the facilitator (who might have no background in STEM).  In terms of evaluation strategies we used a variety of approaches and collected data about and from multiple perspectives.  As youth engaged in the activities, WISEngineering collected data through assessments embedded in the activities.  We also collected feedback from facilitators both within the WISEngineering platform and through surveys and interviews collected outside the activity.  The liaisons and evaluators observed club activity sessions, documenting youth and facilitator engagement, confusions and designs.  This past year we developed an assessment activity, Shark Tank, that challenges youth to create engineering design activities, along with a rubric so they can self assess.   The value of these designs and youth’s self-assessment for evaluating youth learning is currently being studied. 

Lorraine, Thanks to feedback from you and other Boys & Girls Clubs we made modifications.  For example, we streamlined the introductions to be concise without losing the key messages.  We moved to shorter career videos with more diversity.  We simplified the process for taking pictures and posting to the design wall.  We added an award certificate and enabled it for kids to email it to themselves and others.  Each  small change adds up to better outcomes! 

Kalie, glad you are intrigued by the tagline at the end.  We wanted to have a project that could scale and sustain itself after NSF funding and I think we found a method to do so.  This is a work in progress, we are putting pieces together.  First, we  had to have a way to do professional development virtually and the PD videos, plus text material, seem to be doing the job.  This year we had 50% new Facilitators leading activities, successfully, and we had no in-person PD.  We are learning more about how to create the videos inexpensively and have them be easily accessible.  Second, to sustain a project, we need to have some source of revenue, however modest, to support personnel in providing the service.  We have collaborated for many years with Brookhaven National Laboratory and they created a model from an earlier project that works and we are using that method.  Basically, for a modest fee we will provide the activities, the server support, register users, and they will have access to all the virtual PD materials.  We hope by early fall to have the STEMgineering Academy up and running.  Because we do not require in-person PD, we can work with organizations anywhere.  

Thanks for the reply, David. It's great that you are thinking about sustainability so early in the process. I hope in your dissemination strategy, you are able to share more about the model you're developing with Brookhaven with other PIs--speaking from a revenue-generating institution, I think that's something the field definitely needs!

Tiffany, we would very much like to do that.  On earlier projects we developed materials for elementary school aged children (I teach a course in Children's Engineering for elementary school teachers) and we have developed materials for high school and beyond.  I do believe in engineering design as a problem solving strategy across curricular areas, so using engineering design in a variety of content areas is something we are exploring.  If you go to www.hofstraCSR.org, there are a variety of resources and going to www.wiseguysandgals.com will lead you to activity resources for the current project.  

Preeti/Jamie—nothing like the Video Showcase during final exam week to make life more interesting. 

One of the innovations of WISEngineering is that we can capture youth responses directly and provide feedback to them.  For instance, we query them about design specifications and constraints, a key idea in the engineering design process, and provide immediate feedback to them re the accuracy of their responses.  Similarly, at different points in the design process, let’s say during testing and evaluation of their designs (the projects have design evaluation criteria linked to specifications) they can self evaluate the performance of their design solutions.  We are also seeking reflection on their design solution, how they might improve upon it based on the design evaluation.  This year is the first year that we have moved from piloting to collecting research data, and there is a huge amount we can draw upon from time on task, to changes in responses over time for the same person.  Coupled with this we have activity reflection responses from the Boys & Girls Club facilitators, their view of the youth participating and self-reflection on their part.  Of course, our work builds on the work of others, and the work we have done re informed engineering design over the past 20 years.  One of the compelling aspects of our efforts is the expansion of knowledge integration in the informal domain.  One of the consultants on our project is keenly interested in this.  We are also trying to tease out youth gaining design thinking as a problem solving strategy, and the longitudinal analysis of youth response we anticipate will provide us insight regarding this.  Regarding gender and individual identity, we do not have that information, we track participants by login information, which are letters and numbers.  We have general whole group demographics from the facilitators.