Program Evaluation: Skype in the Classroom
By Carla van Vliet, GDPI 802, Queen’s University
1. Select and describe program context
Identify a social program that you would like to evaluate. Describe the focus of the program, size of staff, goals, resources, community demographics, and other details important to conducting a program evaluation.
The program I am evaluating is Skype in the Classroom – https://education.microsoft.com/skypeintheclassroom
Focus of the program:
The program is four fold.
- To connect classrooms with other classrooms.
- To play a geography, 20 questions game with a class with an unknown location in a ‘Mystery Skype’ chat.
- To provide the opportunity for classrooms and learners to connect with Guest Speakers
- To provide virtual Field Trips
Size of staff:
There are 1.5 million educators worldwide using Skype in the Classroom.
There were 19,802 available guest speakers on January 15th 2015.
There were 40 virtual field trips available as of January 15th 2015.
To connect learners with educators in a seamless and easy to access platform, enriching education worldwide and providing a culturally enriching experience.
About Skype in the Classroom:
Turtles in the Classroom:
Connecting with a classroom:
Virtual Field Trips:
World Read Aloud Day:
Victoria Government Teacher Support for Skype:
Live Online Education via Skype – Aslan Highschool for homeschooled kids
50 Awesome ways to use Skype in the Classroom: Teaching Degree Blog
Cool Ways to Use Skype in the Classroom: Teach Hub.com
50 Powerful ways to use Skype in the classroom:
6 Creative ways to use Skype in the Classroom:
A Window of the World: Using Skype in the Classroom, by Cindy Phthistic
Enable 21t Century learning with Microsoft Educational programs:
Skype is used all over the world for countless reasons including but not limited to science experts, language learning, authors and storyteller literacy guidance, classroom to classroom challenges and learning games.
There are 1.5 million educators worldwide using Skype in the Classroom.
There were 19,802 available guest speakers on January 15th 2015.
There were 40 virtual field trips available as of January 15th 2015.
Other important details:
Skype in the Classroom is completely free and volunteer based.
2. Identify purpose for evaluation and specify evaluation questions
Describe the purpose of your evaluation and identify specific evaluation questions to help guide your evaluation design.
How effective is Skype in the Classroom and is learning enriched through its use?
Is this program providing tools that advance specific curriculum goals?
To what extent is this program a replacement for field trips?
3. Construct a program theory
Construct a program theory: theory of action and theory of change.
|Purpose of Skype in the Classroom Experts
To provide a platform where curious groups of students can reach beyond walls and experience extraordinary learning through virtual interactions with knowledgeable people around the world, for free.
|Program Intensions||Program Activities and Resources (Inputs)||Short-term Outcomes: Indicators (Outputs)||Longer-term Outcomes: Indicators (Outputs)|
|o To provide a platform for limitless possibilities for learning
o To bring the world together through a free software program
o To connect available educators with students around the world
o To provide a unique learning opportunity for students all over the world
o To foster global connections, moulding students into global citizens
o To provide teachers with the tools they need to make learning more accessible and engaging
o To increase enthusiasm in learning, globally
o To establish a worldwide network of experts who can provide hands-on knowledge about their current and unique professions
o Enable 21st century learning
|o Participant computer access
o Participant computer projector access for large classrooms
o Knowledgeable experts willing to volunteer time
o Participants need Skype accounts, email and basic computer skills to access the service
o Participants need the Skype program installed on a video-calling device.
o Experts and learners need to communicate by email
o Learners need to prepare questions in advance of the Skype call
o Learners should research effective Skype call strategies and be well prepared
o Experts should have a variety of tools prepared for different age groups of learners
o Teachers in charge of the learners should have an awareness of curriculum targets, and inform the expert about them in advance of the video-call
|o After making arrangements for a Skype-date, students will learn more about the subject area as they research more and prepare applicable questions for the expert (Ask: What did you learn from your research on this topic? Were you eager to learn more about the topic before your Skype date?)
o Students will feel their learning is valuable (Ask: Did you learn something important during your Skype date?)
o Students will want to learn (Ask: Are you eager to see your Skype expert? Are you eager to learn more about the topic in your Skype date?)
o Students will be engaged, enthusiastic and motivated to learn (Ask: Are you excited to learn more? Did your classmates pay attention when you were learning about this topic? Has Skype in the Classroom helped you get excited about the topic?)
o Students will be eager to learn more about the subject because they have been exposed to real life learning (Ask: Do you want the job of the expert? Why/why not.)
o Students will be more knowledgeable about the subject studied (Ask: What facts do you know about the topic? (KWL chart))
o Through a reflection process, students will be more aware of their own learning (Ask: What did you learn?)
o Participants will form more positive attitudes towards learning and the subject (Ask: Was it fun to learn about the topic?)
o Meets curriculum teaching targets (Ask teacher: Which curriculum targets has this met?)
o Teachers can use Skype as a substitute for field trips when certain restraints are present for in-person trips. (Ask: Was this like a field trip? Did you enjoy learning as much as a field trip?)
|o Students become aware of the world and responsible global citizens (Ask: What have you learned about the world through Skype?)
o Students learn by example and use technology to better their lives in the future (Ask: How can you use technology to learn, how do you use technology in your life right now/future?)
o Students will understand more diverse opportunities for them to explore in their search for a life path and career (Ask: Where can you work in the future? What ideas to you have for jobs in the future?)
o Both students and experts have a rich experience that perpetuates the growing community of online learning (Ask: How have you used technology to learn/teach after having used Skype in the Classroom?)
o Students learn how to use the software independently and search our their own Skype learning tools (Ask: How have you used/plan to use Skype in your own independent learning?)
o Cultures are more aware and accepting of each other, worldwide (Ask: How do you feel about other cultures worldwide; What do you know about cultures around the world?)
o Students have a higher satisfaction with their schooling (Ask: Are you happy with your education? Does technology help with your education? How has Skype enriched your learning?)
o Students have a deep understanding and a passion to learn more about concepts and topics covered (Ask: What further research have you done on the topics from using Skype in the Classroom?)
|This program is aimed at enriching the educational environment for schooled learners through the direction of a teacher who connects the classroom to an expert in another part of the world to enhance the target curriculum or learning goals.|
Added: Evaluation Questions in Logic Model Above
4. Identify, describe, and rationalize your evaluation approach
Based on your learning in Modules 1 and 2, review your program theory and evaluation questions to identify an evaluation approach. Provide a description of how that approach will operate in your specific program context. Also provide a rationale for why your approach is a good fit with your program context.
An impact evaluation approach will be the best choice for me in my specific context as it assess the changes that occur from the learning tool on the actual learning and students. An impact evaluation will allow for a comparison between traditional schooling and Skype in the Classroom as a new, online and global approach to learning. I will use the data from personal recount videos on YouTube, as well as the information posted on the Skype Education Big Blog http://blogs.skype.com/?s=Skype+in+the+classroom&language=en . Impact evaluations look at what people do differently as a result of the program and focuses on increasing participant satisfaction. This will be an effective outlook for the evaluation because the premise for Skype in the Classroom is to change the way we look at education. The outcomes will be centred around social, economic, distance, education, and environmental impacts on people, communities and the world.
I am currently an unemployed teacher so completing a program evaluation remotely is mandatory. In my last year of teaching I used Skype in the Classroom many times for many different topics and I feel that knowing more about this program and its uses will be very helpful for my future; sharing this information with the world online on my blog will open new opportunities perpetuate the learning and from this amazing tool. Assessing and learning more about 21st century learning will better prepare me for the next step in my career, and hopefully my reflections will assist others along a similar path as me.
5. Identify data collection methods and analysis strategies
Based on the first four components of your evaluation plan, identify specific methods to collect information that will respond to your evaluation questions. Also think about and articulate how you might analyze the data you collect.
Ideally an evaluation of a program of this magnitude and breadth would incorporate a variety of strategies to encompass a more well-rounded and unbiased view of the program. Data collection methods such as online or paper surveys to teachers and students using the Likert scale would be beneficial as well as in-depth interviews are obvious choices. An even more efficient way to evaluate this would be to choose two very similar classes, perhaps in the same school or school location with teachers of similar teaching philosophies and student demographics, and cross examine the results from the standard class doing a research project and the stakeholder class doing the same research project but with the addition of Skype in the Classroom. Using a method like this with two similar classes in comparison, over a wide rage of locations would provide much more comprehensive data to analyse. This idea for implementation would be even more enriched if the classes and teachers were followed over a longer duration of time to see the longer effects of Skype in the Classroom such as the overall student use of technology and global awareness in the longer-term outcome indicators. If possible, anecdotal notes could also be taken into consideration in this case. Data based on identical subject matter test scores could be also analyzed. Direct classroom observation would also be beneficial in for this study. In such a closed and highly sensitive educational environment, information is always highly sensitive, so an external evaluation without personal or professional connections would be very difficult.
Of course, in my case, I’ll try other methods. Instead, the analysis will be based on testimonials on the Skype in the Classroom big blog. This blog is primarily written by the Skype team or the Experts in the Classroom, as a reflection of an ongoing ‘field trip’ option for Skype in the Classroom dates with classrooms. They have a variety of different ideas and points of views. A few examples are listed below. I will read the blog entries and extract any information that relates to the short term or long-term outcome indicators. Many referenced quotes will be saved.
Data collection will be primarily qualitative. After the data has been analysed, I feel that the amalgamated resources, testimonials and anecdotes can be used to better educate teachers about the use of this type of technology in the classroom, and if it’s a ‘right fit’ for their own classroom. Researching Skype in the Classroom has been very beneficial for me as a teacher, but I have realized that there is a serious lack of resources about it, and hopefully an in-depth evaluation of the content currently on the internet will provide more of a clear picture for interested participants.
Kaytee Smith, Coordinator of e-Learning at the North Carolina Museum of Natural Sciences offers a reflection about Meet the Huggables – all about meeting feared animals from around the world: https://education.microsoft.com/story/virtualfieldtrip?token=0ebd3 She likes to think of her programs as ‘curiosity catylists’. It’s a great way for people to visit the museum who are otherwise unable to attend. More information at: http://blogs.skype.com/2016/01/22/the-north-carolina-museum-of-natural-sciences-brings-science-to-life-with-skype/
Josh Plotnik, Founder and Executive Director of U.S.-based Think Elephants International, founded the non-profit in 2011 and brings live elephants into classrooms, far beyond the borders of Thailand. More information about his goals here: http://blogs.skype.com/2015/10/14/video-calling-with-elephants/
Jean Pennycook, educator and Antarctica researcher, brings about 4 classrooms a day during her peak season, from around the globe. They have had the chance to go on virtual field trips to Antarctica to see the impact of climate change on Adélie penguins. A parent’s reflection is included in this blog entry: http://blogs.skype.com/2016/01/25/antarctica-calling/
Expedition Educator Jamie Buchanan-Dunlop talks from Svalbard (between Norway and the north pole) about his expeditions in the ‘land of the ice bears’ and how he will be looking at how Arctic warming is affecting polar bears. The information can be found here: http://blogs.skype.com/2016/02/27/travel-with-skype-to-the-land-of-the-ice-bears/ and the Skype in the Classroom session is fully booked here: https://education.microsoft.com/arcticlive This is a youtube video related to the location:
This blog post follows a teacher on the Iditarod trail with sled dogs through Alaska, and her experiences with bringing it to the classrom. http://blogs.skype.com/2016/02/29/teacher-on-the-trail-reporting-from-the-iditarod-with-the-help-of-skype/
This blog entry talks about how Skype is helping 17,566 teachers teach: http://blogs.skype.com/2011/10/28/classroom-technology-with-skyp/
Professional diver Tom Daley was on a Skype call with several British Classrooms to celebrate his new book. http://blogs.skype.com/2012/07/03/classrooms-to-meet-professiona/
Blake Mycoskie, Chief Shoe Giver at Toms shoes, Skypes with classrooms to help them understand about building a charitable business. http://blogs.skype.com/2015/03/18/skype-in-the-classroom-partnering-with-toms/ This is another teacher anecdote about Toms shoes and Skyping with Tom Daley http://blogs.skype.com/2014/05/29/skype-in-the-classroom-and-the-extraordinary-lesson/
Celebrities motivate children and promote literacy in classrooms by reading their favourite stories to classrooms on World Read Aloud Day. http://blogs.skype.com/2013/04/24/celebrities-take-to-skype-to-promote-reading-and-literacy-to-classrooms/
Visit a rainforest in Sri Lanka with this classroom to classroom expert chat. http://blogs.skype.com/2013/04/30/skype-enables-unique-learning-experiences-beyond-the-classroom-sri-lanka/
Teacher Arin Kress from Ohio reflects about her past year and her experiences with Skype in the Classroom http://blogs.skype.com/2014/08/13/dive-in-to-skype-in-the-classroom/
Teacher Eric McFarland and 207 8th graders were able to tour and talk to three scientists living in a simulated Martain habitat at the University of North Dakota as a part of their astronaut studies in the local curriculum. http://blogs.skype.com/2014/11/06/skype-in-the-classroom-teacher-takes-students-on-an-inter-planetary-field-trip/
Roald Dahl Museum and Story Centre’s Education team welcome 30 schools and libraries from across Canada, Ireland and the USA through Skype in the classroom in 2014. Education Manager, Isy Mead, led them on a tour of the museum and allowed for lively question and answer sessions about the Roald Dahl’s place of work. http://blogs.skype.com/2014/12/17/68342/
A teacher-librarian reflects about Skype in the Classroom : http://blogs.skype.com/2015/02/12/what-skype-in-the-classroom-really-means-to-kids/
Dr Mikki McComb-Kobza, Ocean First’s Executive Director has completed 240 Skype sessions with schools across the globe [ one being my classroom in China! ] — educating students on the importance of the ocean and marine life. http://blogs.skype.com/2016/02/02/skype-in-the-classroom-learns-the-truth-about-sharks/
Volunteer firefighter Dayna Hilton and Molly the Fire Safety Dog share ways to stay safe, and you get to see Molly do some tricks too! http://blogs.skype.com/2015/03/06/69275/
6. Describe approach to enhance evaluation use
Write a statement on how you will ensure that your evaluation will have impact on the program. Stipulate reporting strategies and methods for enhancing evaluation use
I will post my information on my evaluation blog for the public to see online. It will be available through Google searches about Skype in the Classroom. I will also post the link to my professional social media accounts such as my twitter account. I will also suggest that it be posted on some reputable teacher blogs like the one I have asked to be a guest blogger on: An Ontario Teachers’ Blog https://ontarioteachers.wordpress.com/ . I will also attempt to contact the Skype in the Classroom team to see if they want to post the collaborative findings on the Big Blog to attract and inform future teachers and experts, as well as inquiring if they can provide any additional information to enhance the evaluation.
The purpose of the evaluation is to assess weather Skype in the Classroom enhances, and has an impact on student learning, so the people who would benefit most from a study of this nature would be the teachers. Sharing this information with all of the teachers I know, making it easily accessible on my blog, and asking them to spread the word is perhaps the best I can do, personally to share this.
Carol H. Weiss states that “instrumental use is common under three conditions: 1) if the implications of the findings are relatively non-controversial, neither provoking rifts in the organization nor running into conflicting interests, 2) if the changes that are implied within the program’s existing repertoire and are relatively small scale, and 3) if the environment of the program is relatively stable, without big changes in leadership, budget, , types of clients served, or public support.” (p. 23)
1) The results of the Skype in the Classroom evaluation are non-controversial, they promote collaboration between a corporation, teachers, students and experts on a fun and easy to access platform.
2) The changes that would be made as a result of this evaluation would be on the teacher side – teachers would be motivated to learn more about Skype in the Classroom, and Microsoft would be asked to continue to grow and move in the right direction towards making more international educational programs and cross-cultural connections.
3) Skype was acquired by Microsoft a few years ago, but since that time, Skype in the Classroom and other Skype resources have been consistently improving, so the environment of the program is stable.
Knowing these facts leads me to believe that there is a high likelihood that if a stakeholder (teachers/classrooms/Skype in the Classroom blog leaders/experts/other educators) comes into contact with this evaluation, there is a high likelihood of further use.
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19, 21-33.
7. Commitment to Standards of Practice
Write a statement that describes how your evaluation plan adheres to the Standards for Program Evaluation.
“The utility standards are intended to ensure that an evaluation will serve the information needs of intended users.”
The stakeholders in this evaluation have been identified as the teachers and students, and their needs are being addressed as best as possible in the remote type of online evaluation setting. The evaluators will be educators and sensitive to the needs of the stakeholders. The report has clarity and describes the intent and the evaluation report targets impact for future teacher/student program use.
“The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.”
The online approach to this evaluation makes it a feasible evaluation, because interviews, and direct contact isn’t part of the plan. The cost is minimal because the resources are readily available and there are no real resources to be expended.
“The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.”
The evaluation is designed to assist all of its participants gain insight on their program impact and how to better use and develop connections within it. This program honours human rights and protects children because they have not been questioned directly. The evaluator will do her best to evaluate without bias but the goal is mainly to prove Skype’s success in the Classroom, so there is room for misuse.
“The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.”
The program plan has been documented clearly and appropriately, and the evaluation will continue to do so. The information collected is sourced from the published words of users, so it is legitimate and reliable. The information that is extracted from the testimonials from the Skype blog will be quoted and categorized in an organized and systematic fashion.
Joint Committee on Standards for Educational Evaluation, James R. Sanders, Chair (1994). The program evaluation standards. Retrieved from http://www.oecd.org/dev/pgd/38406354.pdf