MA ELT Dissertation

An Investigation into Learners’ Disposition and Perceived Ability to Learn Independently in the Centre for Independent Language Learning

3 The Questionnaire

3.1 Introduction

A survey was chosen as the research instrument because of the nature of the desired information. Dispositions and perceived abilities can best be investigated by surveying the holders of these dispositions and perceptions. The responses of the subjects need to be generalisable to the entire population of CILL students who are studying English, and it was necessary to "identify attributes of a population from a small group of individuals." (Creswell 1994: 118).

Other types of research into this area were considered and rejected for the following reasons. An experimental research method was not considered suitable because it would not be possible to modify a variable such as disposition or perceived ability. A case study research method would not have produced generalisable results that could inform decisions about LT programmes in CILL. There was insufficient time for ethnographic research, because only a period of about two months was available between the start of term and the closing of CILL for the move to new premises. Although the research was longitudinal in that students’ dispositions and perceived abilities were compared with the duration of the students’ attendance in CILL, it was cross-sectional in that no pre- and post-duration surveys were done. Although the investigation is part of the on-going evaluation of CILL it is not a piece of program evaluation research in itself because this investigation has a very limited scope, and program evaluations have a broader scope and take much longer.

3.2 Research Design

The questionnaire was used to investigate the first two aims and the sub-aim of the investigation. These were, firstly to identify the disposition towards independent language learning of CILL students. The second aim was to identify the perceived ability in independent language learning of these students. This was done by using an adaptation (see Section 3.4 below) of Sheerin’s (1997) model. (See Section 2.8 above.)

The sub-aim, which follows on from the first and second aims, is to investigate whether the amount of CILL experience of the students changes their dispositions and perceptions of their abilities systematically. This was to be done by comparing the results of the questions on disposition and perceived ability with the duration of the students’ attendance in CILL measured by the total number of minutes the student had spent in CILL. This measurement was chosen because students can stay in CILL for as long as they like, so measuring by number of student entries into CILL would not give a balanced picture of total attendance. This is because the wide differences in duration of a single entry ( a difference of 104 minutes between the average minimum and maximum durations) would have made this an inaccurate measure.

However, it is not a perfect measure, because students may not spend all their time in CILL studying, for example they may be using the e-mail facilities to chat with friends.

The accuracy of the results for the sub-aim of this investigation is also limited by inaccuracies in data collection in the CILL student user program. CILL students are supposed to collect their student diaries as soon as they arrive in CILL, which is when the technical staff enter the students’ entry times into the computer. However, there are frequent cases when students either fail to collect their diaries, and thus their presence is unrecorded, or when CILL tutors remind students to get their diaries, and thus the duration recorded in the computer system is less than the students’ actual attendance. However, this limitation is partly countered by the technical staff only giving questionnaires to students who were collecting their diaries, so students who did not collect their diaries did not get a questionnaire.

3.3 Sample Characteristics

The sample size in the survey was 30. The sampling method selected only from the 508 CILL students who had attended CILL during the autumn term. Individuals were selected to be questionnaired by using convenience sampling. This was done by asking the technical staff to hand out questionnaires to all CILL students collecting their diaries who were not French, German or Japanese students, and who had a hardback diary. Hardback diaries are given to students to replace their initial paperback four-page diaries. Thus a student gets a hardback diary after they have made enough notes to use up four pages. Students usually fill in one page per attendance, although they may fill in more. The data from the survey showed that one student had attended only once, and one student was on her 90^th attendance. The average number of entries when the students filled in the questionnaire was 18, as measured by the CILL attendance statistics computer system.

The reason for using convenience sampling was the limited duration of the investigation. The large number of CILL members and the small number of active English-studying members would have meant that simple random sampling, systematic, stratified or cluster sampling may not have included a significant number of active members, resulting in a less-than-significant survey sample size.

The result of this sampling system was that all frequently-attending students (mostly students with over 4 attendances) who came to CILL during the questionnaire administration should have received a questionnaire. This is assuming that they were correctly identified and given a questionnaire by CILL’s technical staff. This process was checked by the researcher, and if an unquestionnaired student was found, they were asked to complete a questionnaire. In conclusion, although only 6% of active members were questionnaired, 100% of frequently-attending members were questionnaired over a one-month period. There was a 100% response rate, as 100% of students given a questionnaire completed it.

3.4 Research Instrument

Contact was made with Sheerin by e-mail, and her response was that she had only used her model informally in a classroom setting, and had not published papers based on it. Therefore there were no existing measures of validity or reliability for the model. She welcomed research based upon the model, and requested to be kept informed.

The statements in the questionnaire were altered slightly from Sheerin’s (1997: 57 - 59) models. There were four main reasons for this. Firstly, my previous experience in doing questionnaires showed that where the respondents were asked to choose between two opposite statements there were frequent problems because many respondents either didn’t know, disagreed with both, wrote comments on the question form, or refused to make a choice. Therefore I decided to use a scale to try to reduce these problems. The scale is 1 to 4, to encourage respondents to make a choice between agreeing and disagreeing. The ‘Don’t Know’ option is physically separated and given letters instead of numbers to try to mentally separate it. This results in a four-point scale, chosen to discourage the tendency for respondents to choose a central option.

Secondly, in the first set of questions, based on Sheerin’s (1997: 59) second model and which are about students’ attitudes to learning independently, some of the statements are not exactly opposite to each other, and therefore the students might have agreed with both sides. For example, in the first question, statement (a) says, " I think it’s the teacher’s job to correct all my mistakes." And statement (b) is, "It’s good for me to find out my own mistakes whenever possible." Students’ strategies may be to proof-read their own work first, and then ask the teacher for corrections, thus using the strategies suggested in both statements. Question Four was similarly altered because Sheerin’s statement (a) says, "… the teacher isn’t listening to my group all the time", but statement (b) does not include "all the time". (See Appendix One.)

Thirdly, to eliminate having multiple factors in the same item, some of the questions based on Sheerin’s (1997: 57) first model were changed. The first statement, "Analyse one’s own strengths / weaknesses, language needs" has three factors, which I have reduced to one in ‘I am satisfied with my ability to analyse what English I need to learn.’ ‘Strengths and weaknesses’ were eliminated to reduce the number of questions and because they are the bases for analysing needs, and therefore are superfluous.

Fourthly the statements were altered to use terms that are commonly used in CILL and that the students are familiar with. So in Questions Two and Three of the second section of the questionnaire, ‘achievable targets and overall objectives’ and ‘the objectives set’ have been replaced by ‘aims’ which is the terminology used in the students’ learner diaries.

The questionnaire was then piloted with 13 students. (See Appendix One for the Pilot Questionnaire). In the light of students’ responses and feedback from the presentation of progress on this investigation to peers, supervisors and academics, a second set of modifications were made. These were firstly that the position of the independent and dependent statements in the disposition section were swapped so that throughout the questionnaire all the pro-independence options were on the left side of the page and the options favouring dependence were on the right. This was done to reduce the chance of confusion about which side of the scale represented more or less independence.

Secondly the words "in CILL" were added to each of the statements about perceived ability to distinguish learning in CILL from learning in other English classes.

Thirdly, the pro-independence option of the last item in the disposition statements was changed from "Tests can’t tell you everything. You know yourself if you’ve been learning well." to "I can test myself. I know myself if I’ve been learning well." This changes the emphasis from saying that tests are unhelpful to saying that it is possible for students to test themselves, rather than the test being imposed by a teacher. Thus the item became more about the student taking control of their own learning, rather than about the value of testing.

The questionnaire was re-administered to check its reliability and to test the altered items. Only five responses were received because the other students did not return to CILL during the investigation period. There was a positive but low correlation between the first and second pilot scores. The full comparison is in Appendix Two. This low correlation in test stability might be because the interval between administration and re-administration varied. Some students did the second pilot immediately after the one week interval. Others did not return for two weeks. This could cause instability in their answers because in that time they may have had new experiences in IL that modified their dispositions and perceived ability.

The data from the first pilot questionnaire was not analysed statistically because the questions were changed for the second pilot. The results of the second pilot were not analysed statistically because of the low sample size of five students.

The low number of responses to the second pilot may threaten the reliability of the questionnaire. However, it was impossible to delay the main questionnaire any longer without disrupting the research schedule. It was therefore decided to administer the main questionnaire in the same form as the second pilot. (See Figure 3.1 below.).

Survey Form

The aim of this survey is to investigate CILL students’ attitudes and abilities to learn English independently. The results of the survey will help CILL staff understand what independent language learning training CILL students want.

Please write your CILL membership number here: __________. Your name will be kept confidential.

Please circle 4 if you agree with statement (a) and not (b), circle 3 if you agree with (a) more than (b), circle 2 if you agree with (b) more than (a), and circle 1 if you agree with (b) and not (a). If you don’t know, circle DK in the box on the right.

Please circle a number to show how much you agree with the following statements:

4-Strongly Agree; 3-Agree; 2-Disagree; 1-Strongly Disagree; DK - Don’t Know

Thank you for your co-operation.

Figure 3.1 The Main Questionnaire

3.5 Variables

The first set of five statements in the table of the questionnaire is designed to measure the disposition towards independent language learning of the student in the areas of responsibility for correction, learning study skills, choosing activities, practising and self-evaluation.

The second set of six statements in the lower part of the questionnaire is designed to measure the students’ perceived ability in independent language learning. Individual items follow a syllabus design system from needs analysis to evaluation.

3.6 Data Analysis

The data from the questionnaire are available in Section Six and Appendix Four.

To test Hypothesis Three, that students’ disposition towards independent language learning and their perceived ability to learn independently would increase with the duration of their total attendance in CILL, the degree of correlation between them was analysed by using Pearson’s Product-moment Correlation Coefficient. This statistical method was chosen following the guidelines in Brown (1988: 130 - 153). There are four assumptions involved in choosing this method.

Firstly, it is assumed that each pair of scores is independent from all other pairs. In the questionnaire results, the pairs of duration and mean total disposition and perceived ability are independent because each pair is from a different student.

Secondly, the two variables, in this case firstly the total duration of the student’s attendance in CILL measured in minutes and secondly the mean total disposition and perceived ability, should be normally distributed, i.e. not skewed. For the duration figures the range of the times contains over four standard deviations, although the mean is only about one standard deviation from the minimum due to the cluster of students with only a few visits and therefore shorter durations. For the mean total disposition and perceived ability, 5.3 standard deviations can be fitted into the range, with the mean close to the middle of the range. Therefore it can be concluded that the skewedness of the figures is low and the distribution is normal.

The third assumption is interval scales. Interval scales represent the ordering of things and assume the equal distance between the intervals (Brown 1988: 22). The scale for duration fits this description, but it could be said that the scales for disposition and perceived ability are of at least ordinal status, and are possibly of interval status.

The fourth and final assumption is a linear rather than curvilinear relationship between the variables. The graph of the figures in Appendix Three demonstrates this.

3.7 Findings and Results of the Questionnaires

The full findings of the questionnaire are in Appendix Four: Questionnaire Statistics, and are discussed together in Section Six with the findings from the interviews and the diary research.

3.8 Limitations of the Questionnaire

The use of convenience sampling could be said to introduce bias into the results in that only active CILL students were selected, and therefore rarely-attending or inactive students were not surveyed. It is possible that these latter students attended infrequently or not at all because of their unfavourable disposition towards independent language learning or low perceived ability to learn independently. It could also be said that decisions on changes in LT programs in CILL designed to improve the students’ disposition and perceived ability should be based on a sample of students that includes infrequent or non-attenders. However, it was impractical to find these students and questionnaire and interview them due to problems of identifying and finding them.

Statistical analysis shows that the first part of the questionnaire, which measures students’ disposition towards IL is an unreliable instrument. Using SPSS, the alpha for reliability was 0.2603, when ideally an alpha of over 0.7 is needed before an instrument can be said to be reliable enough for serious discussion. (See Appendix Four: Questionnaire Statistics.)

This lack of reliability was investigated using factor analysis in SPSS. Varimax rotation analysis gave three factors, which would account for the lack of reliability as only one factor, disposition, was expected. The first factor involved the statements on correction of mistakes and finding out one’s needs, and although they were related, they were opposites, as the coefficient for mistakes was +0.82, and for finding out one’s needs it was -0.76. The second factor related choosing exercises and testing oneself. The third factor related to speaking activities without the presence of a teacher. It was not possible to say which of these factors most closely relates to disposition. As Pemberton (1997: 3) says, autonomy can vary with time and task, so multi-factor disposition has a basis in the literature. (See Section 2.7 above.)

The second section of the questionnaire, which is about perceived ability, has a greater degree of reliability, with an alpha of 0.699. Factor analysis reveals two factors, one encompassing ability to analyse needs, set aims and plan work, and the other involving the ability to select materials and work alone.

Therefore disposition data from the questionnaire were discounted when testing Hypothesis Three, and the disposition in the hypothesis was judged from the students’ statements in the interviews and from the diary research. As favourable disposition was seen as an important prerequisite for any additional LT provision in CILL, it was seen as useful to continue to investigate it, even if it is a multi-factor construct.

The reliability of the questionnaire can be also be questioned because of the correlation between the first and second pilot questionnaire results not being a value of one. Unfortunately it was not possible to administer a third pilot to test the test stability of the second pilot. Some of the reason for the differences in results in between the first and second pilots may have been due to the altered items, which is supported by the low correlation between the altered item on testing of -0.16667. This would have reduced the correlation for both the mean disposition score and the mean total disposition and ability score. The questionnaire findings are detailed in Section Six - Findings, below.

Back to Main Page of this Dissertation | Next Page

Link to Part: Intro, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, Bib, Appx

Last revised on Tuesday, September 24, 2002

If you have any comments, please contact Andy Morrall at ecandym@polyu.edu.hk