A comparison of flipped learning with traditional learning (face-to-face) in large calculus courses: The effects on students’ achievement and cognitive engagement

A comparison of


INTRODUCTION
In recent years, careers in STEM (science, technology, engineering, and mathematics) fields have been growing quickly.Among the four subject disciplines, mathematics can be particularly frustrating (Moliner & Alegre, 2020) and can prevent students from pursuing their STEM major (Adams & Dove, 2016, p. 154-164).Dove and Dove (2015) cautioned that negative learning experiences can lead to students avoiding mathematics and can even result in mathematics anxiety.
Many classes in higher education institutes use blended learning, whereby students learn in part at a supervised face-to-face location on campus, and in part through the Internet with some elements of student choice over place and pace (Horn & Staker, 2011).Of the many different models of blended learning in practice, the use of the flipped classroom (FC) approach has become increasingly common (Giannako et al., 2014;Karabulut-Ilgu et al., 2017;O'Flaherty et al., 2015).Various health professions have adopted FC approach into their curricula.Many reviews of learner perceptions of FCs in health education (Ramnanan & Pound, 2017) found an overwhelmingly positive response from students who attended flipped courses.
To avoid negative experiences with mathematics, some mathematics instructors (e.g., Cronhjort et al., 2018;Lo & Hew, 2021;Wilson, 2013) have redesigned their traditional lecture-based courses using FC approach.Using this instructional approach, "events that have traditionally taken place inside the classroom now take place outside the classroom and vice versa" (Lage et al., 2000).In some mathematics FCs, instructors present basic materials before class using instructional videos (Lo et al., 2017;Yang et al., 2019).Class time can thus be freed up for more instructor-student and student-student interactions (Bond, 2020;Erbil, 2020;Lo et al., 2017).
Edutainment is a word that states a mixture of education with entertainment (Aksalal, 2015).Interactive learning is the key element of edutainment.A significant component is that entertainment should not reduce the value of education and still meet the learning objectives.Any FC is based on the ideologies of edutainment as it makes the classroom an active learning environment.
This research paper reflects an ongoing effort to improve undergraduate teaching experiences in undergraduate calculus courses, where students' learning experience through a combination of FC with a plethora of Prep materials, small-group Collaboration, student Presentations, TopHat Clickers, and Engaged labs (PCPCE) was the focus rather than on assessing individual mathematical domain knowledge.Combined FC-PCPCE supports authentic learning in terms of authentic context, and multiple perspectives through teamwork and collaboration.
In this paper, we first describe the environment of FC-PCPCE format and the interactions of undergraduate students in a firstyear calculus course in this environment.Next, we discuss the students' responses to survey questions related to learning gain, relatedness, learner-related factors, mathematics self-concept in terms of ability, mathematics self-concept in terms of interest, and challenges faced.We conclude with a summary of the lessons learned and some advice for fellow instructors considering adopting a similar learning environment.Although FC and problem-based learning (PBL) pedagogies have been researched extensively, no research papers have investigated our combined FC and PCPCE approach and have assessed the entertainment and educational benefits from the student's perception as this research aims to do.In addition, we compare the perceptions of FC-PCPCE students with those in the traditional in-person format, where all the other aspects of the course are the same.Therefore, the overarching goal of this paper is to make suggestions for future practice of FC and our PCPCE approach in mathematics education and guide researchers who wish to pursue further studies on this topic.

LITERATURE REVIEW Flipped Classroom
The concept of FC has existed since the late 1990s, when educators such as Dr. Erik Mazur were determined to find ways to provide a student-centered but interactive learning experience (Gopalan et al., 2022).However, the credit for its invention is often given to Jonathan Bergmann and Aaron Sams, both high school chemistry teachers in 2007, who uploaded recorded lectures online for their students to watch before convening in the classroom to deepen their understanding of the concepts (Bergmann & Sams, 2012;Gopalan et al., 2022).
The main idea of an FC is that the order of the common instructional method is reversed.Activities that are classically done in class are now done for homework, while activities classically done for homework are now completed in class.The principal goal of flipping is to make the classroom an active learning environment.Cai et al. (2022) applied FC with a case-based learning approach to a third-year undergraduate pathology course.A total of 117 students were enrolled and assigned to FC group (n=59) or LBC group (n=58) with demographic matches.Students in FC format studied materials from two chapters of their pathology textbook before class before coming together to present PowerPoints and discuss medical case studies based on their learning.Results revealed that students in FC performed better on quizzes, showed higher cognitive abilities, exhibited higher motivation and teamwork skills, and understood the course concepts at a higher level than traditional students did.However, these results may be difficult to generalize to a large classroom setting.
Spotts and Gutierrez de Blume (2020) performed a study on the effects of FC on students' learning of matrices and vectors in a high school pre-calculus course.The results of the post-test taken by students indicated that although both groups improved their grades from the pre-test, the flipped class had improved significantly more for matrices compared to the traditional class.Limitations to this study include the varying difficulties of the two topics taught, a short duration, and the smaller sample size for each class.Using larger sample sizes and extending the intervention length to cover more course content could help this study become more generalizable.Price and Walker (2019) applied FC format to an introductory business statistics course.Overall, it was found that students in FCs expressed more interest and had less difficulty with the course material, but that differences in exam scores, attendance, and online engagement between the two formats were not significant.Moreover, females and international students in FC format were found to struggle more with the course content, as well as students in the Management program who generally have a less advanced mathematical background than Accounting and Finance students.Cronhjort et al. (2017) conducted research on the effect of FCs on the overall engagement and learning of students enrolled in an undergraduate calculus course.To analyze these effects, a calculus baseline test (using a pre-and post-test), a student survey, and final exam grades were compared.The results of the study suggested that the students in FC had a better understanding of the content, higher levels of engagement with an emphasis on teacher-student interaction, and a reduction of exam failure rates within each program of study.Turra et al. (2019) applied the interactive FC format to three mathematics courses taken by engineering students at a Chilean university.A pre-and post-test questionnaire was used to analyze students' confidence, perception of the content's usefulness, perceptions of mathematics as a predominantly male domain, and teacher's attitudes.Overall, significant increases were found for each item's rating in the post-test, though the significance of each effect varied.Heuett (2017) explored the effects of FC in an introductory statistics course.A major finding was that students in the flipped format performed better than those in the traditional format on exams, especially on questions involving step-by-step problemsolving.Students in the flipped sections were also significantly more confident in their understanding of the course content due to the interactive in-person lectures.Amstelveen (2018) applied FC to a first-year introductory mathematics course.It was found that video lectures helped FC students learn the course content in more depth, and that these students enjoyed the course more and scored marginally higher on the final exam.However, this study had many limitations such as a smaller sample size of 77 students, a lack of randomization for each class, incentives for study participation, and a multiple-choice final exam.The researcher found that the flipped format may be more beneficial to upper-year students since freshmen in an introductory course have varying mathematical backgrounds.
van Alten et al. ( 2019) completed a meta-analysis of 114 studies that compared the effects of flipped and traditional classrooms on assessed learning outcomes, perceived learning outcomes, and student satisfaction.It was found that FC students tended to outperform traditionally taught students in meeting assessed learning outcomes, but there was no significant increase in perceived learning outcomes or satisfaction for these students.However, confounding variables such as demographics and student motivation were not accounted for in this analysis and are worth exploring to ensure a more accurate result.Chis et al. (2018) embarked on a research case study to compare the quality of students' education and edutainment between traditional learning with FCs, problem-based learning, and a combination of flipped and PBL in a programming course.The overall conclusion was that the combined flipped and problem-solving approach worked best in engaging students and helping them succeed compared to all the other individual teaching methods.Lo and Hew (2021) used a meta-analytic approach to explore the effects of flipped undergraduate and secondary school mathematics classrooms on student engagement.It was concluded that FCs increased student interaction, participation, and attention, but not student effort.In addition, students in FCs had higher satisfaction rates, but the results were mixed between the formats regarding students' feelings and interest in mathematics.Moreover, results were mixed for self-efficacy, although students in FCs were more likely to enjoy challenges and have a stronger grasp of the mathematic concepts taught.
FC is one of many ways to incorporate active learning into courses.McDonald and Smith (2013) consider FC as a pedagogical model that includes interactive and group learning activities within the classroom.

Interactive Learning
The term "interactive" appears in two distinct strands of educational research discourse: one concerning pedagogy and the other concerning new technologies in education (Pradono et al., 2013).The term "interactive learning" is becoming increasingly common in a recent educational setting.Interactive learning is a method for lessons and teaching that includes the use of computing and social networking.The core idea behind this teaching method is about ensuring that students completely understand information rather than just memorizing and regurgitating it.Many of the benefits to interactive learning include engaging content, flexible learning, the ability to save time and cost, as well as fun and enjoyment (Aljaloud et al., 2019).Classes involving interactive technologies may be found to be less monotonous than classes utilizing a traditional learning environment (Kosterelioglu, 2016).
The interactive nature of technology in the classroom elevates the level of engagement that students show towards lessons.Research suggests that student engagement is a key contributor to academic success (Fredricks et al., 2004;Skinner et al., 2008).Recent scholarship continues to affirm the efficacy of engaged learning pedagogies such as undergraduate research, learning communities, and service learning (Lloyd, 2019).The implementation of collaborative practices in classrooms is however a challenge (Wise & Schwarz, 2017).The natural setting for enacting collaboration is the small group.Guidance, which is adaptive to the needs of the learners, is necessary for collaboration to occur (Rummel et al., 2016).
Research has shown that interactive learning in classrooms benefits students academically and improves their engagement.In a study (Rivadeneira & Inga, 2023), where peer instruction was used as an interactive method of learning in a flipped undergraduate engineering course, it was found that interactivity improves students' satisfaction, motivation and interest to learn, and understanding of the course material.Similarly, an interactive problem-posing guiding strategy was used in an elementary school science course and was found to have improved students' self-efficacy and approaches to learning (Ye et al., 2018).These results are indicative of interactive learning being beneficial for a wide range of students spanning different ages and areas of study.For all the above-mentioned reasons, we used the interactive learning method with MATH*1080-Elements of Calculus I in Fall 2022 with the two formats of the course: FC-PCPCE and traditional.We used two different interactive models; the first, during class time using the technology Top Hat as group learning activities within the classroom; the second, during the labs through "engaged mathematics labs."

Engaged Mathematics Lab
Collaborative learning is an umbrella term for a variety of educational approaches involving joint intellectual effort by students, or students and teachers together.Usually, students work in groups of two or more, mutually searching for understanding, solutions, or meanings, or creating a product ( Goodsell et al., 1992).There are many advantages to collaborative learning for the student, whether for the impact on relationships, mental or emotional health, or later in the individual's social and professional life.Abd Algani (2018Algani ( , 2019) ) highlights the importance of interactivity among students and teachers, which is very significant for educational environments.The collaboration of students in a classroom environment has been shown to not only increase engagement levels, but also improve teamwork and social skills (Wentzel, 2009).
Recent literature on small-group learning (VanRyzin & Roseth, 2018, 2019) suggests that when students are given a greater opportunity to interact with a range of peers in class, students become more likely to share resources with one another and to support each other socially and emotionally.Students participating in labs enjoy learning about interesting mathematical applications, attempting challenging problems, and being able to clarify concepts in a more intimate setting, which is not always possible in a traditional lecture environment (Krause et al., 2020).
Recent studies have shown that it is important that the lab component of the course is designed to maximize the engagement and performance of all students involved.Students feel the need to have a purpose for participating in lab activities, which involves academic incentives such as graded assignments or quizzes.Without a clear motivation to complete lab activities, students' emotional and cognitive engagement could suffer (Krause et al., 2020).Furthermore, students should be provided ample time to think through and complete activities to prevent the feeling of being rushed (Krause et al., 2020;Mohammad et al., 2023).
To supplement the learning done in traditional or flipped lectures, mathematics labs introduce a collaborative environment, where students can apply the concepts they learn in lecture to real-life situations (Krause et al., 2020).In this study, "engaged mathematics labs" (Mohammad et al., 2023) were implemented using three of the weekly labs, which were each 50 minutes long.Students formed their own groups of two to four when they arrived.If the students did not have a group to work with which is very common for first-year students, instructors and teaching assistants (TAs) assisted them in forming their groups.During the lab time, students were given a set of questions to work on in their group, but at the end of the period, each student submitted their own individual lab assignment paper.TAs were given sets of lab questions prior to the lab times to be prepared to help students.Instructors and TAs assisted students in completing their assignments.At the end of the lab time, TAs collected all the lab assignment papers from each individual student, marked them, and returned them to the students with their comments.To encourage the students to work on these lab assignments and to help them focus on learning the concepts rather than just achieving high grades, they were each awarded one free point out of five just to participate in the lab assignment.

Top Hat
Top Hat is a student response, web-based system.Instructors can use Top Hat to poll students, ask discussion prompts, present lecture material and track attendance.Students can respond to Top Hat questions and prompts using the devices they already own.Each student's ability to learn and interact with educators is influenced by their personality, family backgrounds, mental processes, learning styles, priorities, maturity levels, and academic ambitions (Tucker, 2021).Social constructivist theories suggest that learning is effective when students feel they are part of a community of positively interacting peers who learn from and with each other (Lin et al., 2021).Discussion with peers is critical to student learning and improves comprehension and engagement in various settings (Cazden, 2001).Through peer discussion in the classroom, students have the opportunity to jointly use their interpersonal and intrapersonal thinking skills to improve collective and individual learning (Lin et al., 2021).Effective classroom discussion provides important opportunities for students to practice social and emotional skills, including communicating effectively and disagreeing respectfully.Discussions in the form of peer instruction give students an avenue to discuss and defend their ideas relating to the topic at hand while questioning the accuracy of their thought processes.In the context of an FC, this allows students to better understand the material they learn outside of in-person instruction time (Rivadeneira & Inga, 2023).In addition to an enhanced understanding of academic concepts, increased opportunities for in-class peer discussions can promote healthy communication, improvement of social and teamwork skills, and encourage class participation.Students also develop stronger connections with their peers, which allow them to share more resources with each other and provide support (Lin et al., 2021).
As society progressively makes technological strides, incorporating technology into the classroom setting has become a necessary reality.The enhanced use of smartphones in the classroom for educational purposes by using interactive apps, such as clickers, is an example of this.To facilitate discussion and enhance student learning in and out of class, we use an educational software called Top Hat.By using interactive technology such as Top Hat in the classroom, students experience many benefits.These include the free expression of answers without peer pressure and increased levels of collaborative learning.Lessons can also be tailored to the students' needs as instructors can easily identify areas of weakness (Aljaloud et al., 2019;Vana et al., 2011).Past research has also shown that similar clicker applications have increased student interest, engagement, and understanding of course content (Pollozi et al., 2019;Salemi, 2009).However, it is important to consider that digital distractions remain possible and can detract from student engagement during the lecture period (Aljaloud et al., 2019;Pistilli & Cain, 2016, p. 247).
To incentivize learning and participation, students were required to attend the lecture section they were registered in to receive Top Hat marks.For each Top Hat question asked during class, students were awarded two marks: one for participation (regardless of whether the answer was correct), and another for a correct response.For the in-person section, only the best 70% of the Top Hat marks were used to determine students' final Top Hat grades since they had three hours of in-person class each week.For FC-PCPCE sections, only the best 85% of the Top Hat marks were used to determine students' final Top Hat grades since they had only 1.5 hours of in-person class each week.There were no alternate dates nor make-up for missing any Top Hat questions.Dropping the lowest 30% or 15% of Top Hat marks was meant to take any absences into account.

METHODOLOGY
For this study, an overarching objective was to measure the impact that course structure, learner interaction, instructor presence, and student engagement during classes and labs have on student perceptions of their satisfaction and learning.Levels of satisfaction were self-reported by students using responses to end-of-semester survey questions.These self-reports can include open-ended responses, checklists, and summative rating scales.This study was conducted in the researcher's own classroom and based on a quantitative analysis of a closed and open questionnaire addressing undergraduate students' perceptions and experience of learning in the flipped and traditional (i.e., non-flipped) classroom in a large calculus course.The course was implemented during Fall 2022 in one of the largest universities in Canada.

FC-PCPCE
A large variety of definitions of FC can be found (Abeysekera & Dawson, 2015).We define an FC as students utilizing a plethora of Preparation materials before class by watching instructor-recorded lecture videos, reading from PowerPoint slides, and doing assigned practice questions following a schedule prepared by the instructor.When students attended the one-hour and twentyminute class each week, they benefited from the interactive learning approach.Class time allows for more interactions between students and the professor in the hopes of increasing engagement.The students were given the opportunity to apply the instructional material that they learn outside the class by working on problem solving using small-group Collaboration, student Presentations, and Top Hat Clickers.The students in both formats took the same two midterm tests, submitted the same lab assignments in their "Engaged mathematics labs" and took an online final exam.This format of FC has not been used before.

Research Design
Three sections of a large first-year mathematics course were investigated in this study.Elements of calculus I (MATH*1080) teaches the principles of single-variable calculus with an emphasis on mathematical modeling in the biological sciences.In the Fall 2022 semester, two sections were delivered in FC-PCPCE format, and three sections used the traditional in-person format.Both FC-PCPCE sections and one traditional in-person section were taught by the same instructor, who was also the primary researcher.The scopes of teaching for each format are summarized in Table 1.
Each week before coming to the one-hour and twenty-minute lecture, the students in FC-PCPCE format watched instructorrecorded lecture videos and completed assigned practice questions and practice applications in life science following a schedule prepared by the instructor.However, for the traditional in-person section, the instructor lectured for three 50-minute classes each week.The course content of both formats was the same, as were the labs, engaged assignments, and tests.In addition to the recorded videos, the students in FC-PCPCE format were provided with complete PowerPoint slides.However, the students in the traditional in-person format were provided with fillable course notes.
In FC-PCPCE and traditional in-person formats, an online classroom response system ("Top Hat") was used to actively engage students in problem solving during lectures and to give an opportunity for feedback on student progress.In FC-PCPCE format, the students could think individually before discussing their thought processes with peers in small groups to solve practice problems using Top Hat during their class time.If the instructor noticed during class that the students did not understand a concept, she would explain the concept and give the students another try for the same question, or a new question related to the same concept.In the traditional in-person format, the students could think individually, then discuss their thought processes with peers in small groups to solve problems using Top Hat for only one or two questions during class time.In both formats, the students were provided with opportunities to present their work in front of the class as "the instructor of the day."As a result, FC-PCPCE students had more in-class time to be engaged and complete practice problems than the traditional in-person students.
For all students in this course, there were five different one-hour lab sections that were taught by TAs.Students could select any lab section at the beginning of the term based on their schedules.Each of the lab sections contained students from both formats of the course.Three lab times were assigned as "engaged mathematics labs", where the students submitted their assignments at the end of the lab time.
In both formats, online virtual office hours were provided by TAs and the instructor.Questions were also answered by the instructor via email.Moreover, the students were assessed using the same criteria and the same questions.Assessments included two in-person midterm tests, three in-person lab assignments, in-class Top Hat questions, and an online final exam.Midterm tests and assignments included both computational calculations, application-based questions and multiple-choice questions.At the end of the semester, all students in the three sections under study were invited to complete a survey to provide feedback and express their feelings regarding the set-up and their experiences of the course format they were enrolled in.Survey questions addressed students' learning gain, relatedness, learner-related factors, mathematics self-concept in terms of ability, mathematics self-concept in terms of interest, and challenges faced.Furthermore, students were able to express commentary on what they felt were the biggest advantages and disadvantages of their course format.Using the quantitative responses to the survey questions, the effectiveness of FC-PCPCE format in comparison to the traditional in-person lecture format was statistically analyzed.The results aided in identifying the changes that can improve FC-PCPCE approach to maximize the overall satisfaction and performance of future students.

Participants
The participants of the study were 354 first-year students in two different formats of a large mathematics class with 2200 students at the University of Guelph.This sample consists of 282 students who voluntarily chose to answer survey questions out of 950 students who were invited to participate from FC-PCPCE format.From the traditional in-person format, 72 students voluntarily chose to answer survey questions out of the 270 students who were invited to participate.All students enrolled in the two different formats for the same instructor were invited to participate in this research project.The end-of-semester survey was brief, requiring less than ten minutes per student to complete.The survey consisted of multiple choice, multi-select options, and open-ended response questions.The survey was open for students to complete in the last three weeks of the Fall 2022 semester.All information was kept confidential, and the investigator had access to the information only after all final grades were submitted to the registrar's office.No compensation or incentives were offered to the subjects, nor did the subjects incur any costs in participating.The study was approved by the research ethics board prior to the distribution of surveys.There were no known risks to the students.

Context of the Study
This aims of this study are the following: 1.To investigate the best pedagogical practices to use in teaching mathematics to students; these environments include traditional in-person lecturing and FC-PCPCE.
2. To determine to what degree that preparation ahead of classes increase learning gain relative to the traditional in-person lecture group.
3. To improve our understanding of the student perspective related to their feelings of belonging in their class format.
4. To examine potential forms of educational and methodological support that students perceive as helpful as they study mathematics in each of the involved learning environments.
5. To improve our understanding of the student perspective and the efficacy of FC-PCPCE compared to the other existing formats of teaching such as traditional in-person lecturing.
6. To find out the advantages and disadvantages of traditional in-person and FC-PCPCE formats of teaching from students' perspectives.
7. To find out whether students in an FC-PCPCE classroom outperform students in a traditional in-person classroom on the tests that were written over the semester.

Data Source & Analysis of the Data
A total of 282 students (30%) out of the 950 enrolled in FC-PCPCE sections and a total of 72 (27%) students out of the 270 enrolled in the in-person section of MATH*1080 participated in an online survey at the end of the semester.The survey was anonymous without any names or identification used and consisted of questions pertaining to different aspects of the students' experience in their course section.These questions consisted of Likert-type items, multiple choice responses, and open-ended written responses.The data were analyzed in the statistical software R.
The differences between FC-PCPCE and traditional in-person sections were analyzed using the Wilcoxon-Mann-Whitney test for Likert-type questions and Chi-square tests.A p-value of <0.05 was considered statistically significant.The open-ended student responses were organized based on the similarities between groups.

End-of-Semester Survey
The questions from the end-of-semester survey were divided into categories: learning gain, relatedness, learner-related factors, mathematics self-concept in terms of ability, mathematics self-concept in terms of interest, and challenges faced.The questions were further grouped within each category based on the nature of the response formats.

Learning gain
The questions regarding learning gain assessed students' overall satisfaction with their course format, their overall gain of content, their overall interest in the content, how well their format stimulated their interest in the course, whether they liked the setup of their course format, whether they liked the recorded videos (FC-PCPCE) or the fillable notes (traditional in-person), and if they found a personal space for their format.
Significant differences between FC-PCPCE and traditional in-person lecture groups were found for overall satisfaction (p=1.307e-06).However, no significant differences were found between groups for content gain (p=0.1363) or content interest (p=0.4765).Figure 1 and Table 2 show the percentage of student responses and descriptive statistics, respectively.
Additionally, significant differences between groups were found for format-stimulated interest (p=5.761e-07),satisfaction of the course setup (p=0.0004678), and satisfaction of the provided course resources (p=0.005798).Figure 2 and Table 3 show the percentage of student responses and descriptive statistics, respectively.In Table 3, the ranking for format-stimulated interest is set as 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, and 5=strongly agree, while the rankings for liking course setup and liking course resources are set as 1=no and 2=yes.No significant differences were found between groups for personal space (p=0.3189).Figure 3 and Table 4 show the percentage of student responses and descriptive statistics, respectively.

Relatedness
The questions regarding relatedness assessed students' feelings of belonging in their format, students' feelings of a friendly and relaxed atmosphere, and feelings of comfort amongst fellow students.Significant differences between FC-PCPCE and traditional in-person lecture groups were found for feelings of belonging (p=6.255e-06).However, no significant differences were found between groups for atmosphere (p=0.1292) or comfort (p=0.5029).The overall results of the questions, including the percentage of student responses and descriptive statistics are summarized in Figure 4 and Table 5.

Learner-related factors
Students were asked about their general feelings regarding mathematics courses before and after taking MATH*1080, their perceptions of the course format they were enrolled in, and their mathematics learning autonomy.No significant differences were found between FC-PCPCE and traditional in-person lecture groups about their feelings regarding mathematics courses before (p=0.8395) or after (p=0.3082)taking their format of the course, or for their level of mathematics learning autonomy (p=0.4657).6 show the percentage of student responses and descriptive statistics, respectively.In Table 6, the ranking for feelings about mathematics courses is set as 1=easy, 2=okay, and 3=hard, while the ranking for mathematics autonomy is set as 1=can understand lesson with minimal help, and 2=usually rely on outside help.
Additionally, significant differences were found between groups for their perceptions of their course formats (p=0.0004979).Figure 7 and Table 7 show the percentage of student responses and descriptive statistics, respectively.

Mathematics self-concept in terms of ability
The questions regarding mathematics self-concept in terms of ability assessed students' ease of understanding, ease of problem-solving, perceived achievement levels, love of attending class, ease of assignment completion, ease of helping classmates with assignments, ease of content recall, course performance relative to the other section, and ability to pass the course from attending their section.Significant differences between FC-PCPCE and traditional in-person lecture groups were found for content recall (p=0.002326),course performance relative to the other section (p=0.001649), and ability to pass the course (p=0.0002501).However, no significant differences were found between groups for understanding of course concepts (p=0.09141),problem-solving (p=0.3847),overall performance (p=0.06989),love of class attendance (p=0.08635),ease of assignment completion (p=0.1777), and ease of helping classmates with assignments (p=0.06016).
The overall results of the questions, including the percentage of student responses and descriptive statistics are summarized in Figure 8 and Table 8.

Mathematics self-concept in terms of interest
Students were asked if attending their course section helped improve their interest in learning and if their attendance helped them to enjoy the lecture more than they would in the other course format.More specifically, FC-PCPCE students were asked if they preferred recorded lectures over face-to-face lectures and traditional in-person students were asked if they preferred faceto-face lectures over videos.
Significant differences between FC-PCPCE and traditional in-person lecture groups were found for both interest in learning (p=4.929e-06) and lecture enjoyment relative to the other format (p=5.551e-05).
The overall results of the questions, including the percentage of student responses and descriptive statistics are summarized in Figure 9 and Table 9.

Challenges faced by respondents
The questions regarding challenges faced assessed students' perceptions of their focus, procrastination levels, time management, self-confidence in calculus, reduction of test anxiety, the course format they wish they enrolled in, the number of times students attended a different format to improve their understanding of course concepts, and whether they recommend the course format they were enrolled in.
The overall results of the questions, including the percentage of student responses and descriptive statistics are summarized in Figure 10 and Table 10.In addition, significant differences between groups were found for course format wishes (p=1.399e-05),number of times the other format was used (p=0.005416),and course format recommendations (p=8.317e-05).The overall results of the questions, including the percentage of student responses and descriptive statistics are summarized in Figure 11 and Table 11.
In Table 11, the ranking for course format wish and course format recommendation is set as 1=no, and 2=yes, while the ranking for number of times other format was used is set as 1=0 times, 2=1-5 times, 3=6-10 times, 4=10+ times.

Exam Grades
Average grades of all students in each course format on each exam were calculated and shown in Figure 12 and Table 12.
On average, students enrolled in the in-person course performed 3.61% higher than FC-PCPCE students on test #1, 4.054% higher on test #2, and 1.64% higher on the online final exam.

Advantages of each course format
Table 13 shows the advantages of each course format.

Disadvantages of each course format
Table 14 shows the disadvantages of each course format.

DISCUSSION
The purpose of this study was to investigate whether students perceived FC-PCPCE course format to be more effective than the traditional in-person lecture format in a large first-year undergraduate calculus course.In this context, FC-PCPCE provided students with the chance to learn autonomously and later convene with classmates to work together and discuss course concepts.This was hypothesized to improve students' interpersonal skills and sense of academic self-sufficiency.While FC-PCPCE course met these expectations for many students, the overwhelming consensus from students' perspectives was that the traditional inperson lecture format best met their needs.This study provides evidence that first-year undergraduate students in traditional inperson calculus lectures have greater overall satisfaction, recall more content, higher learning motivation, and stronger timemanagement skills than those in FC-PCPCE format.This may be attributable to the variety of different mathematics backgrounds and studying habits that first-year students have (Amstelveen, 2018).The results found from this study contrast with those of many similar studies that found FC to be more beneficial to students cognitively and academically.However, the findings in this study related to those of Lo and Hew (2021), where students in FC were not found to have improved their self-sufficiency skills or their feelings and interest towards mathematics.Another study yielded comparable findings in perceived learning outcomes and satisfaction, with FC students not sensing an improvement compared to in-person students despite their outperformance (van Alten et al., 2019).As such, this study can provide more insight into the potential disadvantages of FCs and help educators in similar contexts decide whether this format is appropriate for their courses.

CONCLUSIONS
Many notable results were seen in this study that can be used to guide conclusions towards the effectiveness of FC-PCPCE and traditional in-person lecture formats.The number of students who responded with Excellent, Very Good, or Good overall satisfaction was 12% higher in the traditional in-person section than in FC-PCPCE sections (Figure 2).Furthermore, despite being able to converse with peers during the weekly in-person lecture, the sense of belonging for traditional in-person students was 30% higher than FC-PCPCE students (Figure 4).This suggests that the more consistent contact with peers and the course instructor in the traditional in-person lecture section may lead to stronger camaraderie amongst classmates (Lin & Chang, 2022).Student comments from the traditional in-person section expressed their enjoyment of engaging, in-person lectures and the ease of obtaining support when faced with challenges or questions (Table 13), while many students in FC-PCPCE section expressed their struggles with the absence of these features (Table 14).
A desire for belonging extends into students' course performance, as feeling supported and welcomed is a great motivator of success (Zumbrunn et al., 2014).Students in the traditional in-person lecture section achieved higher exam grades by an average  20 students mentioned issues with needing to miss lectures for any reason and/or going back to review content as neither completed course notes nor lecture videos were posted on course website.4 students mentioned that fillable course notes were inconvenient for students who did not have access to a printer or did not own a tablet or touchscreen laptop for notetaking. of 3.10% in the course (Table 12) and 26% more of these students attributed their course format to the reason they passed compared to students in FC-PCPCE sections (Figure 8).In a similar fashion, learning interest for traditional in-person students was higher than FC-PCPCE format by 33% (Figure 9).
Overall, challenges appeared to be faced more often in flipped learning than in traditional, in-person lectures.49% more FC-PCPCE students experienced procrastination, and time management was a struggle for 40% more FC-PCPCE students (Figure 10).Furthermore, 27% more traditional in-person lecture students felt increased confidence in their calculus abilities after completing the course than students enrolled in FC-PCPCE (Figure 12).These results may be attributable to the fact that many university students, especially in their first semester of undergraduate studies, may not be acclimated to their new environment (Kumar, 2015).This can in turn make learning autonomously a challenge due to the traditional nature of most secondary school courses (Macaskill & Taylor, 2010).
Research has shown that FC format has a plethora of benefits for learners.These include motivating learning and interest, encouraging positive attitudes toward mathematics, catering to different students' learning needs, active engagement in the learning process, and the improvement of problem-solving skills.FC is a useful tool for giving students opportunities to improve their learning autonomy and teamwork skills during their studies, however this study sheds light on the fact that interactive methods must be used to ensure high student satisfaction and strong academic performance.

Limitations
There are some limitations of this study that should be noted, as well as some suggestions for future studies.First, this study was conducted using three sections of the same large, first-year undergraduate calculus course during Fall 2022.This course is aimed at students studying the biological sciences who typically do not pursue upper-year mathematics courses.Therefore, the results obtained may not be generalizable to students in different academic programs.Second, this study was conducted for students in a first-year calculus course, which could cause results to differ for upper-year courses; therefore, the findings cannot be generalized to other levels without conducting further studies.However, we tried to control many confounder variables in this study such as: instructor, tests, assignments, labs, and material since they were the same in both FC-PCPCE and traditional inperson formats.It is important to note that this was an observational study, where the students responded voluntarily to the survey questions, instead of a randomized experiment.Therefore, results should be only generalized to populations that are similar to the target population from this study (Graziano & Raulin, 2013).

Recommendations and Future Directions
The analysis and conclusions of this study allow for the proposal of some recommendations for future improvements of this type of study.
First, it is important to test the effectiveness of FC-PCPCE format across multiple course disciplines, different enrolment numbers and class sizes, as well as in upper-year courses.Different types and levels of courses require distinct styles of thought (Entwistle, 2005), which could make this format work better in some courses than others.
Additionally, an analysis done by linking students' grades with their survey responses would provide more insight into the format they are in.This would allow researchers the ability to discover correlations between students' reported characteristics and course grades.
Another important analysis that can be conducted is one that adjusts for explanatory variables such as gender, ethnicity, high school mathematics background, or academic program.Significant differences between these groups can then be assessed.Furthermore, in the future, an experiment can be used instead of an observational study.An experiment can reduce the bias present by ensuring that samples are randomized and reflective of the general student population.An example of a potential experiment that can be implemented is randomly assigning students from one section of MATH*1080 to either flipped or traditional in-person learning for a set period, and later observing their course performance and responses to the survey questions.However, this would require further student and ethics considerations approval.

Figure 1 .
Figure 1.Percentage of student responses to questions regarding learning gain in our FC vs. IP MATH*1080-Elements of calculus I (Source: Authors' own elaboration)

Figure 2 .
Figure 2. Percentage of student responses regarding whether format stimulated their interest in our FC vs. IP MATH*1080-Elements of calculus I (a), percentage of student responses regarding whether they liked setup of course in FC vs. IP MATH*1080-Elements of calculus I (b); & percentage of student responses regarding whether they liked course resources FC vs. IP MATH*1080-Elements of calculus I (c) (Source: Authors' own elaboration)

Figure 3 .Figure 4 .
Figure 3. Percentage of student responses regarding whether there was a personal space for class in our FC vs. IP MATH*1080-Elements of calculus I (Source: Authors' own elaboration)Table 4. Descriptive statistics for personal space with ranking set as 1=disagree, 2=neither agree nor disagree, & 3=agree Median Mode Inter-quartile range FC IP FC IP FC IP Q.Personal space 3 3 3 3 1 1

Figure 5 ,
Figure 5, Figure 6, and Table6show the percentage of student responses and descriptive statistics, respectively.In Table6, the ranking for feelings about mathematics courses is set as 1=easy, 2=okay, and 3=hard, while the ranking for mathematics autonomy is set as 1=can understand lesson with minimal help, and 2=usually rely on outside help.

Table 5 .Figure 5 .Figure 6 .
Figure 5. Percentage of student responses to questions regarding their feelings about mathematics courses in FC vs. IP MATH*1080-Elements of calculus I (Source: Authors' own elaboration)

Figure 7 .Figure 8 .
Figure 7. Percentage of student responses regarding their perception of course compared to other format in FC vs. IP MATH*1080-Elements of calculus I (Source: Authors' own elaboration)

Figure 9 .
Figure 9. Percentage of student responses to questions regarding their mathematics self-concept in terms of interest FC vs. IP MATH*1080-Elements of calculus I (Source: Authors' own elaboration)

Figure 10 .
Figure 10.Percentage of student responses to questions regarding their challenges faced FC vs. IP MATH*1080-Elements of calculus I (Source: Authors' own elaboration)

Figure 11 .
Figure 11.Percentage of student responses regarding whether they wish they took other course format in FC vs. IP MATH*1080-Elements of calculus I (a), percentage of student responses regarding how many times they used other format's resources in FC vs. IP MATH*1080-Elements of calculus I (b), & percentage of student responses regarding whether they recommend their course format to other students in FC vs. IP MATH*1080-Elements of calculus I (c) (Source: Authors' own elaboration)

Table 1 .
A summary of learning modalities across two different formats of course design

Table 3 .
Descriptive statistics for format stimulated interest, liking course setup, & liking course resources

Table 6 .
Descriptive statistics for feelings about mathematics courses & autonomy

Table 8 .
Descriptive statistics for mathematics self-concept in terms of ability with ranking set as 1=disagree, 2=neither agree nor disagree, & 3=agree

Table 9 .
Descriptive statistics for mathematics self-concept in terms of interest with ranking set as 1=disagree, 2=neither agree nor disagree, & 3=agree

Table 10 .
Descriptive statistics for challenges faced with ranking set as 1=disagree, 2=neither agree nor disagree, & 3=agree

Table 11 .
Descriptive statistics for course format wish, number of times other format was used, & course format recommendations Figure 12.Average exam scores in FC vs. IP MATH*1080-Elements of calculus I (Source: Authors' own elaboration)

Table 12 .
Average grades of all students in two different formats of course

Table 13 .
Student comments about advantages of FC vs. IP

Table 14 .
Student comments about disadvantages of FC vs. IP