EVALUATE, SHARE, GROW

This evaluation was completed by a team of science teachers and education researchers based at the School of Education, UCD. Each of us on the evaluation team are familiar with relevant issues within science education as well as the Irish educational context. Each of us were, to varying degrees, embedded within the I-LOFAR education initiative and worked closely with those on the I-LOFAR education team. This report has been written with regard to SFI’s reporting template.

Karen Pillion (Physics teacher & researcher at School of Education, UCD & Our Lady’s Bower Secondary School, Athlone),

Philip Cullen (Chemistry teacher & researcher at School of Education, UCD),

Dr Rosario Carroll (Biology Teacher & researcher at School of Education, UCD & St Joseph of Cluny Secondary School, Killiney).

Dr Shane Bergin (Physicist and Assistant Prof in Science Education at School of Education, UCD),

How did you evaluate your project?

In order to evaluate the I-LOFAR Astronomical Midlands (AM) project, we chose to focus on certain aspects of the initiative that are representative of the entire project. Specifically, we focused on two week-long summer camps that were led by the I-LOFAR education team. The Junior Camp worked with children and young people aged 10-14. The Senior Camp worked with young people aged 16-18. While planned to run in person, due to COVID-19 constraints, all activities were held online. 

Each component of AM was evaluated against their associated learning objectives.

The learning objectives of the Junior Camp were to: 

• Increase enthusiasm and understanding of STEM and its role in everyday life; 
• Raise awareness of science as a creative discipline, and the need to collaborate across many disciplines.

The  learning objectives of the Senior Camp were to:

• Increase uptake of STEM subjects at LC level and at 3rd level, specifically among underrepresented groups in the midlands through sense of ownership of I-LOFAR; 
• Broaden teenagers concept of science as a creative discipline and awareness of the need for interdisciplinary collaboration. 

The evaluation team worked with the I-LOFAR education team to unpack these broad learning objectives and connect them to the tasks planned within each camp. An education team intern (who comes from a science education background) produced more detailed learning objectives for each day of the camps, facilitating a more structured programme that could facilitate student learning.

The camps’ learning objectives were evaluated using i) direct observations of the online summer camps by the evaluation team (all of whom are second-level science teachers); ii) field notes and prompted group reflections with I-LOFAR team members; and iii) surveys completed by parents/ guardians and students.

An adapted version of the ‘Teaching for Robust Understanding’ (TRU) framework (Schoenfeld, 2016) was used for the direct observations of the online camps. The TRU framework is used to characterise learning environments and is a useful tool for observers to focus their attention upon key disciplinary ideas and how they should link to certain aspects of the learning environment. We adapted this in order to frame an informal learning environment.

The modified TRU framework asked the following specific questions:

1. Are the scientific/astrophysics ideas, outlined in the I-LOFAR camp curricula, present and embodied in instruction? 
2. Do the camps create an environment of productive intellectual challenge, conducive to every participant’s deepening understanding of scientific content and practices?
3. Do the camps invite and support the meaningful engagement with core content by all participants,  supporting the diverse range of learners in engaging meaningfully.
4. Do the camps provide participants opportunities to explore, conjecture, reason, explain, and build on emerging ideas, contributing to the development of agency and ownership over the content?
5. Do camp activities elicit all participants’ thinking by building on productive beginnings or by addressing emerging misunderstandings?

Camp activities and resources were evaluated under the following headings (linked to the five questions posed above): (i) Content & Appropriate Content Level, (ii) Equitable Access to Content, (iii) Agency, Ownership and Identity and (iv) Formative Assessment. The framework was refined to match each of the activities and associated learning objectives under observation.

As well as using the adapted TRU framework to guide observations of camp activities, the evaluation team kept field notes. After each camp, the field notes promoted group reflections between the i-LOFAR education team and the evaluation team. This afforded the education team the opportunity to discuss how they felt the camps were going. Their remarks helped the evaluation team ‘make sense’ of the observed camps and incorporate learnings in ‘real time’. It is important to note the evaluation and education teams spoke openly, and that the evaluation team did not adopt the position of ‘silent observer’, modelling best educational practices (Etkina, 2017). 

Online questionnaires were made available to students after the camps, enabling them to share their experiences of the camps, resources, activities and any prior knowledge or experience they had. These were designed these questionnaires with the individual learning objectives of each camp in mind. This enabled us to arrive at a clearer picture of learner engagement with, and understanding of, the broader learning objectives.

The data from direct observations, group reflections, and the online questionnaire were combined to evaluate the learning objectives of the junior and senior space camps.

Items within I-LOFAR that we observed	What we sought to assess	The assessment tools that we utilised
Junior Space Camp	Learning Outcomes: Increase enthusiasm and understanding of STEM and its role in everyday life.  Raise awareness of science as a creative discipline, and the need to collaborate across many disciplines.	Surveys, TRU framework
Senior Space Camp	Learning Outcomes: Increase uptake of STEM subjects at LC level and at 3rd level, specifically among underrepresented groups in the midlands through a sense of ownership of I-LOFAR.  Broaden teens concept of science as a creative discipline and awareness of the need for interdisciplinary collaboration.	Surveys, TRU framework

Outline the findings from your evaluation

Findings from the evaluation of the space camps are presented under the following headings. The headings are based on the adapted TRU framework used for observations, field notes and prompted group reflections, and online questionnaires.

(i)               Content & Appropriate Level

At both camps, each day began with a discussion of the key ideas. Materials for the class centred on the key ideas being explored that day, e.g. origami models were used to show how space agencies pack important pieces of satellites or telescopes that can then be unfolded in space. Survey results from the Junior Camp indicated that students felt the materials and activities were closely aligned with the learning outcomes and had a successful effect. For example, in the survey responses, students unanimously responded that building models of the solar system helped them to comprehend the different sizes of celestial objects. 

Survey results from both camps suggest that the learners were meeting the specific learning outcomes of the camps through engagement with the specific activities. Ideas started off simple and were built on during the camps. Discussion was frequently used to allow the students to engage with the topic and check to see what their current understanding of the topic was. The surveys indicated that the different activities helped students to build on their thoughts and gain a better understanding of a given topic. During post-camp discussions, the i-LOFAR team frequently reflected on whether or not the participants were achieving the learning outcomes, what was working well with respect to content and activities, and what could be improved upon.

The education team were aware that students had come with differing amounts of prior knowledge. However, with guest speakers, the language used when presenting material was sometimes not age appropriate. This was noted particularly during the Senior Camp. This was discussed by the education team in the reflective discussion. 

Things to do in future:

• Checking for learning before and after the camp.
• Prior to the commencement of a lesson, activities such as ‘Think, Pair, Share’ or ‘Mind Maps’, could be utilised to check what students know and want to learn, provide the opportunity for peer teaching and sharing of ideas, and help facilitate discussion around the topic.

(ii)              Equitable Access to Content

During the camps, gender balance was evident in the education team and guest speakers at I-LOFAR. It was clear to us from this representation, discussions and feedback that there was an emphasis on inclusion of all students present. A parent sent in a comment from her young daughter after the camp:

“Mom, I’m going to be an astrophysicist, there’s lots of girls who are”.

Throughout the online camps, students were able to contribute vocally or through the chat box. The chat box was beneficial as it allowed students who might be shy the opportunity to contribute without having to speak. Students engaged with guest speakers and education team by asking questions to help improve their understanding of the content. During activities, students were more vocal and had less trouble asking for help or further instruction. 

A parent mentioned how accommodating the camp was for their child with additional educational needs and commended the team on the inclusive nature of the camp.

“I would like to say a huge thank you for allowing ___ to take part in the astro camp, he really enjoyed it and learned loads. He really enjoyed the week so thank you again. With his conditions it is hard to get him to go out sometimes or to get him to interact with other children his age so this was a fantastic opportunity for him to do something new and learn at the same time.”

While the I-LOFAR programme is mindful of equality and diversity in its curriculum and delivery, it is important to note that while young women made up 44% of the participants on the Junior Camp (n=18), only 21% of the participants at the Senior Camp (n=14) were female.

Things to do in future:

• Employ strategies to improve gender balance of Senior Camp.
• Work with local teachers and young people to talk about the associated issues and ways I-LOFAR could be more inclusive. 

(iii)            Agency, Ownership and Identity

The responses received from the Junior Camp survey show that prior to the camp, almost half of students were unaware of any astronomical sites in Ireland or of any space-linked research carried out in Ireland. In contrast, after the camp, all of the students answered that they now think that astronomy research that takes place in Ireland or has in the past, is important to the people of Ireland. 

Activities throughout the camps relied on students doing science, rather than listening to presentations from others who’d done it. Students appreciated this and indicated that feeling part of a community is important, for the experience to be meaningful. 

Some comments from the students suggest that they want to do more concerning space and astronomy in school:

“We enjoyed cosmo last year when we went to the Midlands astronomy, we got to do experiments and see different telescopes but that was only once and we were only allowed go for one day so there’s not really much.”

“Astronomy should be taught in primary school.

Students at the Senior Camp seemed to have strong science identities. Survey results suggest this may, in part, be attributed to the older students tending to have more of a pre-existing interest in astronomy than the younger students, and having attended more events such as this one in the past. The Senior Camp aligned with the students’ interests and positively contributed to their science identities and sense of community. 

Things to do in future:

• Consider strategies to sustain the communities formed/reinforced at I-LOFAR. 

(iv)            Formative Assessment

The main challenge with issuing feedback to students was the online nature of the activities. Students could show their work over the video chat or send pictures in through the chat feature. The education team were then able to give feedback to the students to help them build on what they had done. After students had completed certain activities, they were encouraged to experiment with the resources to see what else they could come up with, and were provided with feedback again. 

Things to do in future:

• Include ‘looking for learning’ strategies in the planning process.

The table below shows a summary of the main findings for both Junior and Senior Camps.

TRU Framework Categories	Junior Camp (ages 10-12)	Senior Camp (ages 13-16)
(i) Content & Appropriate Level	Materials were noted as being particularly effective in supporting students’ learning. Content was appropriate to students level.	The language was noted as being at too high a level at times. Content was, in general, appropriate to students level. Reference to more complicated issues was  identified as a way of piquing potential interest for further research.
(ii) Equitable Access to content	Strong emphasis on inclusion of all participants. Good gender balance noted among students.  The camps were particularly inclusive of children with additional educational needs.	Strong emphasis on inclusion of all participants. Lower participation from girls at this camp. This could possibly be improved by targeting specific underrepresented groups, as set out in the learning outcomes.
(iii) Agency Ownership & Identity	Mixed responses from this group. They would like to feel more involved in astronomy research. It was noted that this cohort felt ownership over the content while they were taking part in projects such as the camp, but initiatives like these would need to be more ongoing for younger students to feel a part of the astronomy community.	Strong sense of ownership over content and knowledge on how they could be involved.
(iv) Formative Assessment	Positive formative assessment took place throughout the week. However, this is a relatively short period of time to accurately assess this category.

 

Describe what you have learned from your findings

Here we will discuss what we have learned relative to the five questions associated with the adapted TRU framework

• Are the scientific/ astrophysics ideas, outlined in the I-LOFAR camp curricula, present and embodied in instruction?  Do the sessions create an environment of productive intellectual challenge, conducive to every participant’s deepening understanding of scientific content and practices?

Finding: Strong educational focus

The findings suggest that there was a strong educational focus within i-LOFAR. A clear commitment from the education team to create a meaningful learning environment resulted in a learner-centered, multifaceted, active learning environment. These are well understood to have positive impacts on students’ perceptions of science, the likelihood they will study science subjects in school, and the nature of their engagement with science (Mintzes, 2020)  There are parallels between the pedagogy used at the I-LOFAR camps and those used when teaching and learning Junior Cycle Science. During the camps, it was expected that students used logic to interpret and explain what they had just learned in terms of new content and share their opinions on scientific phenomena with other students. Within the new Junior Cycle Framework, students should “plan, draft and present scientific arguments, express opinions supported by evidence, and explain and describe scientific phenomena” (NCCA, 2014). There were divergences between the Leaving Certificate curriculum and the Senior Camp. With far more emphasis on ‘rich tasks’, creative thinking and team work, students who took part were, arguably, having a more authentic experience of science. This, in turn, may have given them insight into studying physics, mathematics and astronomy at third level and assistance in helping them make the decision to pursue these STEM courses.

• Do the sessions invite and support the meaningful engagement with core content by all participants,  supporting the diverse range of learners in engaging meaningfully. 

Finding: Workshops appeal to students who previously had a strong interest in the area.

The findings suggest that those attending the camps felt ‘part of the team’ while on the camp and due to the student-centred approach enjoyed ownership of the science they were doing. They also had the opportunity to learn from a diverse education team. While there was very good participant gender balance in the Junior Camp (44% of the attendees were young women). This balance was not present for the Senior Camp (21% were young women). This variation is, unfortunately, a familiar trend. The UN have noted that “gender differences in STEM education participation are more apparent […] usually in upper secondary education” (UNESCO, 2017). We would recommend that a more diverse range of learners are invited to engage with the I-LOFAR programmes in order to meet this learning objective in the future. Perhaps more targeted approaches are required to encourage greater diversity of the student population. 

UNESCO research (2017) also outlines that factors including teaching quality, having female STEM teachers and role-models, curricula, and teaching strategies that take their interests into account all influence girls performance and future engagement with STEM subjects and careers. Conversely, learning outcomes for women and girls can be compromised when teachers hold stereotypical beliefs about sex-based STEM ability or treat boys and girls unequally” (p. 12). Given the I-LOFAR camps are so aware of these issues (in their design and execution), we would hope more targeted invitations to young women will see greater participation in future Senior Camps. 

• Do the sessions provide participants opportunities to explore, conjecture, reason, explain, and build on emerging ideas, contributing to the development of agency and ownership over the content?

Finding: Students (particularly those in the Senior Camp) felt part of the astrophysics community while taking part in the camps.

It is clear that the camps made significant progress in highlighting the local and national astronomy research that is taking place. After the camps, students also had a greater sense of appreciation of the importance of this research to the general public.

The Senior Camp students, in particular, felt part of the astrophysics community. This is to be welcomed. Research shows that when students have a strong physics identity, they are more likely to continue with physics into the future (Fracchiolla, Prefontaine, & Hinko, 2020). However, that research also shows that in order for students to develop a real sense of agency and ownership over the research, there must be continued engagement. This need to sustain engagement emphasises the importance of teacher CPD events, such as those that i-LOFAR facilitate, in enabling teachers to connect the students’ time at I-LOFAR to STEM subjects in school. The team have done significant work ensuring that these CPD resources and workshops are clearly linked to the curriculum, particularly the JC curriculum. This aligns with a report by the Department of Education (2016), which specifically outlined improvements in STEM CPD as an action for priority implementation, as well as developing a comprehensive suite of STEM CPD programmes for primary and post-primary teachers and upskilling programmes in the sciences. 

• Do classroom activities elicit all participants’ thinking by building on productive beginnings or by addressing emerging misunderstandings?

Answering this question was very important, however focusing only on the camps, we were not able to acquire sufficient data to really address this. The camps are only a snapshot in time of the programme as a whole. Research carried out by Bell (2014, p. 24) remarks that interventions of longer duration “were considered to be successful because of the extended influence they had on science education more widely” and that “success criteria might be staged and applied sequentially to reflect the effectiveness at different phases during the intervention and then after its completion”. Such an approach “could help provide a more holistic way of judging success.” (p. 24)

The work of Bell (2014) further demonstrates the importance of programmes such as i-LOFAR being connected to local school communities. When the learning in these programmes is specifically connected to school curricula, there are opportunities for multiple synergies that see community form around space science. For example, data generated by students working at I-LOFAR could serve as ‘rich tasks’ for their schools to work with. At i-LOFAR, there was an emphasis placed on linking activities to the Irish primary and post-primary science curricula, as was highlighted in the material and training provided at the continual professional development (CPD) events for teachers.  We see the beginnings of a wonderful engagement between midlands schools and the education team and I-LOFAR – one that should be supported and encouraged.

Outline how these findings could be addressed in any future similar projects

(i)               Checking for Learning

Prior to the commencement of a lesson, activities could be designed to check what students know about a topic. Activities such as ‘Think, Pair, Share’ or ‘Mind Maps’, could be utilised to check what students know and want to learn, provide the opportunity for peer teaching and sharing of ideas, and help facilitate discussion around the topic. 

(ii)              Materials

While the materials used in the camp were excellent in that all were accessible, things could be improved by using materials that allow students to experiment further and ‘go beyond’ what was covered in the camps. Significant supports on differentiated materials are available from teacher colleagues in local schools.

(iii)            Guest Speakers

Many of the guest speakers ‘spoke above’ the students. A possible way of helping this would be for the education team to work with the guest speakers ahead of their talk, discussing the ages the students, their prior learning, and the objectives of the space camps. This might allow the guest speakers to better tailor the content to the class, through simpler language or examples that link to things the students may be interested in. It may also be helpful to try and facilitate further discussion between students and guest speakers so students can ask them about anything that they are unsure of. This would also help checking if the students are following and understanding the content of the lesson.

(iv)            Communication

Students having the ability to both communicate verbally and non-verbally in Microsoft Teams is extremely helpful as it gives a voice to students who might be nervous to talk. Having someone keeping a closer eye on the chat in Microsoft Teams and reminding students that they are looking at the chat may help prevent students veering off topic in the chat and prevent interesting questions or discussion ideas being missed. 

Conclusion

The individual elements that were evaluated at i-LOFAR give great insight into the project as a whole. The ethos of the programme and attitudes of the team came across clearly in the areas we focused on. They portrayed a team dedicated not only to scientific research, but strongly committed to engaging as many people as possible from the wider public in their community.

References:

Bell, D. (2014). The perceived success of interventions in science education : a report for the Wellcome Trust (https://wellcomecollection.org/works/dzvz7kea).

Department of Education. (2016). STEM Education in the Irish school system. Retrieved from the Department of Education website (https://www.education.ie/en/Publications/Education-Reports/STEM-Education-in-the-Irish-School-System.pdf).

Etkina, E., Gregorcic, B., & Vokos, S. (2017). Organizing physics teacher professional education around productive habit development: a way to meet reform challenges. Physical Review Physics Education Research, 13(1), 010107. 

Fracchiolla, C., Prefontaine, B., & Hinko, K. (2020). Community of practice approach for understanding identity development within informal physics programs. Physical Review Physics Education Research, 16(2), 020115. 

Mintzes, J.J., Walter, E. (2020). Active learning in college science – The case for evidenced-based practice. Springer International Publishing. ISBN 978-3-030-33599-1.

National Council for Curriculum and Assessment (NCCA). (2014). Science and key skills. ( www.curriculumonline.ie/Junior-cycle/Junior-Cycle-Subjects/Science/Science-and-Key-Skills).

Schoenfeld, A. H. (2016). The Teaching for robust understanding (TRU) observation guide for mathematics: a tool for teachers, coaches, administrators, and professional learning communities. Berkeley, CA: Graduate School of Education, University of California, Berkeley. 

The United Nations Educational, Scientific and Cultural Organization (UNESCO). (2017). Cracking the Code: girls’ and women’s education in science, technology, engineering and mathematics. (euagenda.eu/upload/publications/untitled-137226-ea.pdf).