Indicator description

This indicator measures the self-declared climate change knowledge of 15-year-olds based on a selection of items from the 2018 Program for International Student Assessment (PISA). A total of three items were utilized to create a composite index and countries were assessed based on the percentage of students who scored above a threshold on this index.

The three items used for the indicator were: “How easy do you think it would be for you to perform the following tasks on your own?”,

a) Explain how carbon-dioxide emissions affect global climate change,

b) Explain why some countries suffer more from global climate change than others

and “How informed are you about the following topics?”,

a) “Climate change and global warming.”

Dataset(s) the indicator is based on

This indicator is based on the the Organization for Economic Cooperation Cooperation and Development (OECD)’s Programme for International Student Assessment (PISA). The student questionnaire is a core part of the triennial PISA which includes demographic items such as gender, socioeconomic and cultural status of the student based on parental education and occupation, as well as other questions that vary from wave to wave. In 2018 several items were included which covered the self-perception of the participants’ own knowledge of climate change.

Temporal scope

• Time period(s) collected data is available: PISA data is available from 2000 to 2021, with data collection rounds every 3 years, for a total of 7 rounds. Nevertheless, the Global Competence survey, from which the data for this indicator derives, was tested for the first (and so far only) time in PISA 2018.
• Frequency of data collection in the past: Every 3 years (but the items utilized for the indicator were only included in the 2018 PISA).
• Frequency of data collection in the future: Every 3 years (but the items utilized for the indicator were only included in the 2018 PISA).
• Change over time: The items utilized for the indicator were only included in the 2018 PISA.

Sample size, characteristics, and geographical coverage

In 2018, 612,003 students – in 66 countries and territories participated in the Global Competence part of the study. Among UNFCCC members 60 countries are represented.

Country coverage (out of 198 UNFCCC parties) by SDG grouping. Percentage in brackets corresponds to total number of countries per SDG grouping.

Data Preparations

The PISA 2018 data was downloaded and all data besides the relevant items and pertinent demographic information were removed. The answer options were recoded from 0 to 2 for each item, in which for question 196: 0 = not enough ability to explain what is asked (containing original options 1 and 2), 1 = some ability to explain what is asked (option 3 in original question), and 2 = sufficient ability explain what is asked (option 4 in the original). Similarly for question 197 original values were recoded reflecting  0 = not enough information (recoded from originals 1 and 2), 1 = some information (recoded from original option 3), and 2 = sufficient information (recoded from 4 in the original). 

The scores for each individual were summed up to create an index from 0 to 6. A cut-off score of 3 was determined to be the threshold of an individual having adequate climate change knowledge. Individuals exceeding the threshold were assigned the provided  variable “FINAL TRIMMED NONRESPONSE ADJUSTED STUDENT WEIGHT”, which is created by PISA to assure that the overall results are nationally representative across a variety of dimensions. This weight is specifically calculated for each country and assessment round. For each country the weighted total of individuals exceeding the threshold was divided by the weighted total population to calculate the percentage of individuals in a country who have adequate self-declared climate change knowledge.

Indicator Calculations

The percentage of weighted individuals who exceeded a threshold of self-declared climate change knowledge was used to determine the assigned level along the following criteria:
1: Up to 20%
2:Up to 40%
3: Up to 60%
4:Up to 80%
5: Up to 100%

Limitations

Besides the usual concerns with statistical population sampling and weighting, PISA has its own particular potential sources of error which include (but is not limited to): dealing with expatriate students, excluding of relevant populations along several variables, translations across languages, assumptions on enrollment of 15-year-olds, and general assessment design concerns.

The main conceptual weakness is relying on self-declared statements of knowledge rather than measuring actual knowledge. Using PISA itself for indicators has general weaknesses including country coverage and cultural differences in responses (especially an issue with things like self-declarations of knowledge) among other.

The face validity for the indicator is strong as PISA is designed to be used for research purposes (and is widely used thus). The content validity for this indicator is acceptable though the items are self-declared rather than measuring actual knowledge, they are right on topic and with the targeted group.

No geographical coverage of Sub-Saharan Africa and Oceania (excepting AU and NZ). There are also territories included in PISA which are not nationally representative samples (e.g. Hong Kong) and these have been excluded.

Acknowledgements

The MECCE Project would like to thank the OECD for making their data open access.

OECD (2019), PISA 2018 Results (Volume I): What Students Know and Can Do, PISA, OECD Publishing, Paris, https://doi.org/10.1787/5f07c754-en.