A tool to assess sex-gender when selecting health research projects

Abstract

Objective

To validate the questionnaire “Gender Perspective in Health Research” (GPIHR) to assess the inclusion of gender perspective in research projects.

Design

Validation study in two stages. Feasibility was analysed in the first, and reliability, internal consistence and validity in the second.

Where

Aragón Institute of Health Science, Aragón, Spain.

Participants

GPIHR was applied to 118 research projects funded in national and international competitive tenders from 2003 to 2012.

Main measurements

Analysis of inter- and intra-observer reliability with Kappa index and internal consistency with Cronbach's alpha. Content validity analysed through literature review and construct validity with an exploratory factor analysis.

Results

Validated GPIHR has 10 questions: 3 in the introduction, 1 for objectives, 3 for methodology and 3 for research purpose. Average time of application was 13 min Inter-observer reliability (Kappa) varied between 0.35 and 0.94 and intra-observer between 0.40 and 0.94. Theoretical construct is supported in the literature. Factor analysis identifies three levels of GP inclusion: “difference by sex”, “gender sensitive” and “feminist research” with an internal consistency of 0.64, 0.87 and 0.81, respectively, which explain 74.78% of variance.

Conclusions

GPIHR questionnaire is a valid tool to assess GP and useful for those researchers who would like to include GP in their projects.

Resumen

Objetivo

Validar el cuestionario “Perspectiva de género en la investigación en salud”(PEGEIN), para evaluar la inclusión de la perspectiva de género en los proyectos de investigación.

Diseño

Estudio de validación en dos fases. En la primera se analizó la viabilidad, y en la segunda, la fiabilidad, la consistencia interna y la validez.

Emplazamiento

Instituto Aragonés de Ciencias de la Salud, Aragón, España.

Participantes

Se aplicó el cuestionario PEGEIN a 118 proyectos de investigación financiados en convocatorias nacionales e internacionales competitivas en el periodo 2003-2012.

Mediciones principales

Análisis de la fiabilidad inter e intra-observador con el índice de Kappa y consistencia interna con el coeficiente alfa de Cronbach. Se analizó la validez de contenido a través de revisión bibliográfica y la de constructo con un análisis factorial exploratorio.

Resultados

El PEGEIN validado consta de un total de diez cuestiones: tres para introducción, una para objetivos, tres para metodología y tres para finalidad de la investigación. El tiempo medio de aplicación fue 13 minutos. La fiabilidad (kappa) interobservador osciló entre 0.35 y 0.94 y la intra-observadora entre 0.40 y 0.94. El constructo teórico está sustentado en la bibliografía. El análisis factorial identifica tres niveles de incorporación de la PG “diferencia por sexo”, “sensible al género” e “investigación feminista” con una consistencia interna de 0.64, 0.87 y 0.81 respectivamente, que explican un 74.78% de la varianza.

Conclusiones

El cuestionario PEGEIN es un instrumento válido para evaluar la PG y útil para aquellas investigadoras e investigadores que quieran incluir la PG en sus proyectos.

Introduction

Gender as a social determinant causes inequalities in the health status, in the access to services and in people's well-being.^1,2 National and international organisations have developed relevant reports that have emphasised the need to include gender perspective (GP) in health research to go further in the not biased scientific knowledge and in equality and equity in people's health. The effort has been focused on pointing out gender biases that weigh on research, health programmes and clinical practice, and on offering recommendations and guidelines to make easier the introduction of this approach by the largest number of professionals.^3–7

For some decades now, the theorisation about the advantages of including gender perspective in health research has been increasing. Its application, however, is still little.⁸ Since scientific knowledge strengthens the need to include GP in order to produce knowledge without errors, and since there is enough information about methodology to apply this perspective, it may be convenient to work actively and to apply as a selection criteria, both in scientific journals and in research tenders, that data according to sex are disaggregated and that a gender perspective is included, and failing to do so they should justify why. It is beyond doubt that project funding must be directed to those with a greater methodological, scientific and equity rigour and that ignoring the introduction of GP in research conditions the advance of science.

Health and gender is a priority line of research in Spain found in the Strategic Action in Health within the framework of the State Plan of Scientific, Technical and Innovation Research 2013–2016. Nevertheless, it is not a decisive factor yet in the global objectives of R+D research in Spain, since gender perspective as a quality criteria is not included in project assessment.⁹

Internationally, European tenders already include GP in the evaluation of the suitability of research projects and it is considered an added value if the project aims at promotion of gender equality and equal opportunities.¹⁰ From December 2010 Canada and the Canadian Institute of Health Research (CIHR) specifically state that the notion of gender and sex is integrated in the research project and make all the candidates for grants answer some questions about the place sex/gender has in their research plan.¹¹

With the objective of contributing to the systematic assessment of GP, a methodological instrument with dual purpose was created: to make easier the assessment of GP in projects from competitive tenders financed with public resources, and to guide researchers about the aspects to be assessed in their projects. The questionnaire “Gender Perspective In Health Research” (GPIHR) was drawn up applying a qualitative design with two techniques: a discussion group and a group meeting of experts. This was organised around the relevant questions to be asked in each of the stages of a research project and includes twelve items: four to apply in the introduction; two in the formulation of objectives and hypotheses; three in methodology and three in the purpose of the research.¹² This questionnaire is in tune with the concept of “gendered innovations”, which considers the application of sex and gender throughout all the research project.¹³

The objective of this study is to validate the “Gender Perspective in Health Research” questionnaire analysing its feasibility, reliability and validity after applying it in research projects from competitive tenders, aiming at its use as an assessment tool in research projects.

Methods

To validate the questionnaire different health research projects were selected, projects that had been managed by the Aragón Institute of Health Science (IHS) from 2003 to 2012 and funded in competitive tenders. The study was approved by the Ethics Committee of the IHS. We requested the consent of the main researchers. The GPIHR questionnaire was applied in 118 research projects, so the ratio research projects and questions in the questionnaire was 9.91/1, very close to the 10/1 recommended for this type of studies.¹⁴

The validation study was done in two stages. Feasibility was analysed in the first stage and reliability, internal consistence and validity in the second stage (Fig. 1).

Figure 1.

Validation of the Gender Perspectives in Health Research questionnaire. Application to 118 research projects. Feasibility, intra- and inter-observer reliability and validity studies. Production of a questionnaire that discriminates the sex-gender consideration in research projects.

To analyse the feasibility of the questionnaire a pilot study was performed on 30 projects. All 12 questions were applied with three possible answers: yes, no and not applicable, including in this case the corresponding reason. Two specialists in Public Health applied the questionnaire, a man and a woman, without specific training in research with GP. Different issues were studied: application time, clarity of the questions and their completion, together with the no-answer percentage of every item.

To analyse the reliability and internal consistence of the instrument in the second stage the resulting questionnaire was applied to all 118 projects with only two possible answers (yes/no). Two professionals were selected, a psychologist and an anthropologist with specific training in GP and health. Both applied the questionnaire to analyse the inter-observer reliability and 15 days later one of them performed the assessment again to study intra-observer reliability.

Reliability assessment was performed with Kappa index for categorical variables. The values were interpreted according to the six levels of strength of agreement proposed by Landis and Koch: <0 no agreement, 0–0.2 insignificant, 0.2–0.4 low, 0.4–0.6 moderate, 0.6–0.8 substantial and 0.8–1 very good agreement.¹⁵

Cronbach's alpha was estimated to evaluate internal consistency. Values of 0.70 or above are accepted as an indicator of good consistency.¹⁶

To analyse the validity of the questionnaire both the content validity and the construct validity were assessed. Content validity was established by literature review, considering whether the content includes the most relevant concepts of the gender theory. To study the construct validity, a factor analysis of the main components (AMC) was performed. Several criteria were taken into account before, to assess whether it was relevant to perform a factor analysis of the correlation matrix: the correlation matrix determinant, with a value of 0.001, Bartlett's test of sphericity was significant and the KMO test measuring sampling adequacy with a value >0.80.¹⁷

This analysis developed here starts from the analysis of the structures of the correlations between the variables; and, through data reduction, it will determine a small number of new factors that will be a lineal combination of the initial variables. Due to the dichotomous nature of the variables, tetrachoric correlation matrixes were used. When we carried it out, we assumed the dichotomous variables have an underlying continuous feature.¹⁸ The selected factors were those with eigenvalues above 1, and Varimax rotation was applied to improve the interpretation of the factors obtained originally; the internal consistency of each one of the factors was assessed with Cronbach's alpha. Once the factor analysis was done, the research team confirmed the plausibility of the factors drawn with the scientific knowledge of the application of gender perspective.

The statistical package SPSS 15.0 was used.

Results

Feasibility analysis

A pilot study was performed on 30 research projects, with a range between 10 and 15 min for the completion of the questionnaire.

In 65% of the studies, the evaluators were not able to answer the item “If the health subject being researched has scientific knowledge with gender perspective, have these references been included?”, arguing they ignored if there was scientific knowledge with GP or not. Thus, it was rephrased to “Have references to existence or non-existence of scientific knowledge with gender perspective been included in the health subject being researched?” since both possibilities indicate that gender has been taken into account as health factor in the literature review done.

Research projects do not include key words, so the item, “Is the term ‘gender’ included in the key words?”, was eliminated.

The item “Has the sample been stratified by sex and age group?”, was divided into two items for their independent assessment.

It was observed that professionals who performed this assessment included many commentaries in the section “not applicable”, which brought out the difficulties. This is attributed to lack of training, so professionals with training in GP research and dichotomous answers to every question were part of the next stage.

Reliability and internal consistency analysis

The questionnaire validation was performed on the 12 items resulting from the previous stage. Inter-observer and intra-observer reliability are presented in Table 1. Inter-observer reliability (Kappa) varied from 0.350 and 0.937, and intra-observer reliability between 0.399 and 0.943. The former presents values inferior to the latter in most of the variables; three of these variables present reliability values below 0.4 and the rest Kappa indexes very appropriate.

Table 1.

Reliability and Internal Consistency of the GPIHR Questionnaire.

	Kappa inter-observer	Kappa intra-observer
1. In the introduction, have references to existence or non-existence of scientific knowledge with gender perspective been included?	0.416	0.678
2. In the introduction, is there any reference to the magnitude of the problem in women and men?	0.725	0.924
3. Does the introduction take into account the gender category as a health determinant?	0.697	0.810
4. Through the objectives/hypotheses formulated, does it seek the association between the health issue studied and any gender determinant?	0.784	0.668
5. Are there any biases or gender stereotypes in the formulation of objectives, questions and/or hypotheses?	0.485	0.714
6. In the methodology, has the sample been stratified by sex?	0.937	0.943
7. In the methodology, has the sample been stratified by age group?	0.833	0.755
8. In the methodology, are gender biases distinguished in the criteria for inclusion or exclusion in the sample?	0.375	0.399
9. In the methodology, do the variables used highlight the existing relationship between the health issue studied and any of the gender factor(s): characteristics dependent on the social role, attitudes, beliefs, sex division of work, sexual identity, family role, life cycle?	0.602	0.713
10. In the purpose of the project, does the project help bring out the differences or inequalities between men and women in the health issue studied?	0.563	0.579
11. Does it aim at helping increase the knowledge of women and men's health and diversity in it expression?	0.350	0.651
12. Does it aim at helping point out changes in the gender structure that may affect on equality or equity between men and women in health?	0.350	0.604
Cronbach's alpha in the questionnaire: 0.740

The internal consistency of the questionnaire is good (Cronbach's alpha = 0.740).

Validity analysis

Content validity: the items in the questionnaire include very relevant concepts from the theory supporting GP in health.

Gender is a health factor. Sex and gender are related but are not synonyms.^19–23

Health can be the same and/or different between men and women. If this is not taken into account, it can cause bias because of generalising results to both sexes when only one of them has been studied. When this biased knowledge is applied it can cause inequality in health care. Samples must be stratified by sex.^{3,5–7,24–26}

As a category of analysis, gender informs us about how cultural construction of sex affects health, and points out the necessary changes to avoid inequality situations among people. Gender omission (gender blindness) leads to biased scientific knowledge.^27–32

Construct validity is analysed through a factor analysis on the 12 items of the questionnaire. Application conditions of the analysis are met (KMO = 0.81 and Bartlett's sphericity test is significant, p < 0.005).

In the first analysis the two factors with an eigenvalue equal to or above 1 were selected and they explained the 63.25% of the variance. The Cronbach's alpha is 0.909 for the first factor and −0.294 for the second one, which means it does not meet the assumptions of the reliability model. Therefore, it is decided to cancel items “Are there gender or stereotype biases in the formulation of the hypotheses/objectives?” and “Are possible gender biases identified in the inclusion or exclusion criteria in the sample?”, which condition this situation, and which did not have optimal values in the reliability analysis.

A second factor analysis was done on the remaining 10 items and three factors explaining the 74.78% of the variance were selected. Application conditions were met: KMO 0.793 and Bartlett's sphericity test is significant, p < 0.005. A Varimax rotation was performed. Three factors were selected, two with eigenvalue above 1 and the third one very close to 1, because it widens the interpretation and allows the gradual explanation of the inclusion of gender perspective. The internal consistency of this questionnaire was 0.899.

Table 2 presents the correlation matrix with the significant coefficients and Table 3 presents the variance values and the internal consistency.

Table 2.

Correlation Matrix. Analysis of Main Components.

	Factor
	1	2	3
1. In the introduction, have references to existence or non-existence of scientific knowledge with gender perspective been included?	0.862
2. In the introduction, is there any reference to the magnitude of the problem in women and men?	0.881
3. Does the introduction take into account the gender category as a health determinant?	0.313	0.802
4. Through the objectives/hypotheses formulated, does it seek the association between the health issue studied and any gender determinant?	0.638	0.532	0.319
5. In the methodology, has the sample been stratified by sex?		0.462	0.685
6. In the methodology, has the sample been stratified by age group?			0.893
7. In the methodology, do the variables used highlight the existing relationship between the health issue studied and any of the gender factor(s): characteristics dependent on the social role, attitudes, beliefs, sex division of work, sexual identity, family role, life cycle?	0.617	0.412
8. Does the project help bring out the differences or inequalities between men and women in the health issue studied?	0.522	0.323	0.528
9. Does it aim at helping increase the knowledge of women and men's health and diversity in it expression?	0.463	0.601	0.426
10. Does it aim at helping point out changes in the gender structure that may affect on equality or equity between men and women in health?	0.327	0.772

Extraction method: analysis of main components.

Rotation method: Varimax with Kaiser normalisation.

Table 3.

Factor Analysis and Internal Consistency of Factors. GPIHR Questionnaire.

Factor	Item	% Variance	Cronbach's alpha
Gender sensitive	1. In the introduction, have references to existence or non-existence of scientific knowledge with gender perspective been included?	51.92	0.872
	2. In the introduction, is there any reference to the magnitude of the problem in women and men?
	4. Through the objectives/hypotheses formulated, does it seek the association between the health issue studied and any gender determinant?
	7. In the methodology, do the variables used highlight the existing relationship between the health issue studied and any of the gender factor(s): characteristics dependent on the social role, attitudes, beliefs, sex division of work, sexual identity, family role, life cycle?
Feminist research	3. Does the introduction take into account the gender category as a health determinant?	14.98	0.814
	9. Does it aim at helping increase the knowledge of women and men's health and diversity in it expression?
	10. Does it aim at helping point out changes in the gender structure that may affect on equality or equity between men and women in health?
Sex difference	5. In the methodology, has the sample been stratified by sex?	7.77	0.643
	6. In the methodology, has the sample been stratified by age group?
	8. ¿Does the project help bring out the differences or inequalities between men and women in the health issue studied?

Factor 1 is named “gender sensitivity”. It refers to differences in health between men and women and to the relationship between the gender factors and the health issue approached in the research project.

Factor 2 is named “feminist research”. It gathers all the conditions required for a research to have gender perspective and feminist purpose, that is, it investigates the causes for inequality to try and change it.

Factor 3 can be defined as “difference by sex”. It reflects disaggregation of the data by sexes and age group, which will make possible the identification of differences in health.

The three selected factors reveal different levels of GP inclusion in the research projects.

Discussion

The validation of our GPIHR questionnaire has meant a modification of the original, removing three items, rewording a fourth and dividing a fifth. The application time of the questionnaire is appropriate thanks to these modifications, and questionnaire simplicity, clarity and ease of application have improved. It presents good reliability with the best matching values in the intra-observer reliability and a good internal consistency. Furthermore, content validity is sufficiently supported in the literature and in the guides for the inclusion of GP in health research.^{3,5–7,19–30,32–34}

The main contribution of our questionnaire, as the performed validation shows, is that it is an instrument that allows to assess three levels of GP inclusion.

If the project design includes both men and women in different moments of their life cycles, which translates into an appropriate methodology, then the project could establish the “difference by sex”, highlighting the existing differences between men and women in the health subject being researched.

A second level adds “gender sensitivity”. The project takes into account that there may be differences or similarities in the health of men and women and that these differences are not only biological. Nevertheless, it also assumes that culture determines differences in their way of living and the self-perception of men and women. These are projects that wonder about the relationship between health and life conditions of men and women both as individuals and as a social group.

Finally, the determinant that implies a commitment for the identification of causes socially constructed, that takes into account feminist epistemology and would enable the possible modification of the gender structures supporting them is known as “feminist research”.

With the application of the validated GPIHR questionnaire we can, therefore, answer the question “Does the research project include gender perspective?” in three possible ways: it establishes “difference by sex”, it is “gender sensitive” or it is “a feminist research”.

These factors or dimensions obtained from the analysis of main components have a parallelism with the theoretical models of gender inequalities and gender determinants.³¹ It is also in line with the “gendered innovations” approach that wants to develop the analysis of sex-gender in all the stages of the research, both basic and applied aiming at excellence in research.¹³

The main limitations of this questionnaire are that specific training in gender perspective in health is needed in order to use the questionnaire as an assessment tool, as the pilot study highlighted. Moreover, the type of research needs to be taken into account while the questionnaire is being applied, whether it is basic or applied research. Projects of basic research will only take into account items containing factor 3 “difference by sex”. However, if these projects have a stage of applied research as well, then it will be relevant to include variables sensitive to gender. Then, these could be projects with “gender sensitivity”.^5,13,35–37

This questionnaire is essentially destined to be applied by research project and agency evaluators to discriminate about the inclusion of gender perspective in the projects. We consider it appropriate not only to make this assessment exclusively but also to add to other project quality assessment questionnaires.

The questionnaire can also guide all researchers to include GP in their projects as well as health professionals to evaluate themselves whenever they apply for research projects in competitive tenders.

Ignoring the inclusion of gender perspective in health research worsens health and the quality of life of women and men, it limits their access to resources and slows the progress of science.

What we know about the subject

• •

  The need to study separately men and women is well documented, as well as to include gender perspective in health research.

• •

  There are guides to include gender perspective in research but there are no tools to assess whether projects include it.

• •

  The questionnaire must be validated to discriminate its inclusion.

What this study brings

• •

  The possibility of using a valid tool answering the question: does the research project include gender perspective?; and discriminating three levels: difference by sex, gender sensitivity and feminist analysis.

• •

  It may also be useful for those who would like to include perspective in their projects.

Funding

This paper has been financed by Fondo de Investigación Sanitaria, Instituto de Salud Carlos III (PS0901380).

Conflict of interest

The authors declare no conflict of interest.

Acknowledgement

This piece of work is dedicated to the memory of Dolores Ariño-Martín, our beloved friend and colleague, who with her passion and enthusiasm accompanied us in this task of thinking how to improve research and to reduce inequalities. You will always be with us.