SE-07: Education and Academia

ICO Std 2002 — Extension SE-07

This extension module specifies the methodology requirements for ranking and evaluating educational institutions, academic programs, and scholarly output. Educational rankings possess outsized influence on institutional strategy, resource allocation, student welfare, and public policy. This module therefore imposes additional requirements for methodological rigor, fairness across institutional contexts, and the safeguarding of academic integrity. It extends the core layer (Chapters 4–10) with domain-specific indicators, peer review mechanisms, and provisions for educational equity.

SE-7.1 Scope and Applicable Objects

SE-7.1.1 Scope

This module shall apply to the design, execution, and governance of ranking systems that evaluate:

a) higher education institutions (universities, colleges, institutes of technology);

b) academic departments and discipline-specific programs;

c) individual scholars and research outputs;

d) secondary and primary education institutions, where methodology adaptation is documented;

e) vocational and continuing education providers.

SE-7.1.2 Normative References

The following references shall inform the design of education ranking methodologies:

SE-7.1.3 Relationship to Core Layer

This module shall conform to the core layer as follows:

a) Chapter 4 (Principles): The principle of fairness shall be interpreted to require explicit treatment of contextual differences among institutions (size, mission, resource endowment, geographic context).

b) Chapter 5 (Indicator Design): Indicators shall be validated for cross-context comparability before inclusion.

c) Chapter 6 (Weighting): Weighting decisions shall account for institutional mission diversity and shall not systematically favor research-intensive over teaching-focused institutions unless the ranking explicitly declares a research-only scope.

d) Chapter 9 (Publication): Results shall be published with methodology, limitations, and contextual notes that enable informed interpretation.

SE-7.2 Indicator System

SE-7.2.1 Indicator Hierarchy

The indicator system for education evaluation shall comprise five dimensions:

Code Dimension Weight Range  
SE-7.2.1.1 Teaching Quality 20 %–35 %  
SE-7.2.1.2 Research Performance 20 %–35 %  
SE-7.2.1.3 Faculty Quality 10 %–20 %  
SE-7.2.1.4 Student Development & Outcomes 10 %–25 %  
SE-7.2.1.5 Social Contribution & Impact 5 %–15 %  

The ranking entity shall document the rationale for its specific weighting choices and shall disclose whether the weighting favors research-intensive or teaching-focused institutional models.

SE-7.2.2 Teaching Quality (SE-7.2.1.1)

a) Teaching quality indicators shall include both direct measures and proxy measures:

Direct measures 1) Student satisfaction surveys with validated instruments and documented response rates (minimum 30 % response rate for inclusion); 2) Teaching evaluation scores from structured peer observation; 3) Learning gain assessments (value-added measures) where available.

Proxy measures 4) Student-to-faculty ratio (with adjustment for disciplinary norms); 5) Completion and graduation rates (with adjustment for student intake selectivity and socio-economic profile); 6) Teaching awards and professional development investment per faculty member.

b) Proxy measures shall be clearly labeled as such and shall not be presented as direct measures of teaching quality.

c) The ranking entity should prioritize direct measures where available and should invest in or advocate for the development of learning-gain metrics that enable cross-institutional comparison.

SE-7.2.3 Research Performance (SE-7.2.1.2)

a) Research performance indicators shall be subject to the following requirements:

1) Bibliometric indicators — Publications, citations, h-index, and field-normalized indicators (e.g., Field-Weighted Citation Impact) shall be used with acknowledged limitations. Bibliometric data shall be sourced from at least two independent databases to mitigate database bias.

2) Field normalization — All citation-based indicators shall be field-normalized. Cross-field comparison without normalization shall not be used for ranking purposes.

3) Self-citation control — The ranking entity shall apply a self-citation exclusion or cap (maximum 25 % self-citations per article) to prevent inflation.

4) Output volume vs. impact — Research performance shall balance quantity and quality. Indicators that reward only publication count without quality adjustment shall not be used.

b) The ranking entity shall disclose the bibliometric databases used, the normalization method, and the self-citation treatment.

SE-7.2.4 Faculty Quality (SE-7.2.1.3)

a) Faculty quality shall be assessed through:

1) Proportion of faculty with terminal degrees in their field; 2) Faculty diversity (international, interdisciplinary, industry-experienced composition); 3) Faculty retention and career progression rates; 4) Ratio of full-time permanent faculty to total instructional staff (adjunctification indicator); 5) Research output per faculty member, field-normalized.

b) The ranking entity should consider the proportion of instruction delivered by full-time permanent faculty as a positive quality signal, and the over-reliance on contingent/adjunct faculty as a negative signal, without prescribing a specific ratio.

/ adjunct

SE-7.2.5 Student Development & Outcomes (SE-7.2.1.4)

a) Student outcome indicators shall include:

1) Employment outcomes (employment rate, time-to-employment, salary levels) — with adjustment for macroeconomic conditions and field of study; 2) Further study progression rates; 3) Alumni achievement indicators (with appropriate time lags — minimum 10 years post-graduation for career achievement measures); 4) Student satisfaction and engagement metrics; 5) Diversity of student intake (socio-economic, geographic, demographic).

b) Employment and salary data shall be adjusted for geographic cost-of-living differences and macroeconomic conditions at the time of measurement.

c) Alumni achievement indicators shall use appropriate time lags. Career achievement measures shall not be applied to recent graduates (within 10 years of graduation).

SE-7.2.6 Social Contribution & Impact (SE-7.2.1.5)

a) Social contribution indicators may include:

1) Knowledge transfer and industry collaboration (patents, spin-offs, industry-funded research); 2) Community engagement and public service; 3) Policy influence (citations in policy documents, government advisory roles); 4) Contribution to Sustainable Development Goals (SDGs), as measured by THE Impact Rankings methodology or equivalent; 5) Cultural and artistic contribution.

b) Social contribution indicators should be weighted to reflect the institutional mission. Institutions with explicit public-service mandates should have higher weighting for these indicators.

SE-7.3 Benchmarking Against Established Rankings

SE-7.3.1 QS World University Rankings

The ranking entity shall document its relationship to the QS methodology as follows:

a) QS indicators (Academic Reputation 30 %, Employer Reputation 15 %, Faculty/Student Ratio 10 %, Citations per Faculty 20 %, International Faculty 5 %, International Students 5 %, International Research Network 5 %, Employment Outcomes 5 %, Sustainability 5 %) shall be referenced, and deviations shall be justified.

b) The ranking entity shall note that QS relies heavily on reputational surveys (45 % combined weight), which are subject to known biases including halo effects, regional bias, and response bias. Where the ranking entity uses reputational data, it shall implement mitigations as specified in SE-7.3.4.

SE-7.3.2 THE World University Rankings

a) THE indicators (Teaching 30 %, Research Environment 30 %, Research Quality 30 %, International Outlook 5 %, Industry 5 %) shall be referenced. The ranking entity shall note THE’s use of bibliometric normalization and reputational survey methodology.

b) The ranking entity should consider THE’s approach to research quality indicators (field-weighted citation impact, research excellence, research environment) as a reference for field-normalized bibliometric design.

SE-7.3.3 ARWU (Shanghai Ranking)

a) ARWU indicators (Alumni Nobel/Fields 10 %, Staff Nobel/Fields 20 %, Highly Cited Researchers 20 %, N&S Articles 20 %, International Papers 20 %, Per Capita Performance 10 %) shall be referenced.

b) The ranking entity shall note ARWU’s known limitations: heavy reliance on award-based indicators (30 %), English-language publication bias, and emphasis on STEM and biomedical fields. These limitations shall be documented if the ranking entity adopts similar indicators.

SE-7.3.4 Reputational Survey Methodology Requirements

Where the ranking entity uses reputational survey data:

a) The survey instrument shall be validated for reliability (Cronbach’s α ≥ 0.80) and shall be administered with documented procedures for sample selection, invitation, and response tracking.

Cronbach’s α ≥ 0.80

b) Response rates shall be disclosed. A minimum response rate of 15 % shall be required for reputational data to be included in the ranking.

c) The ranking entity shall implement the following bias mitigations: 1) quota sampling by geographic region and discipline to ensure representativeness; 2) removal or downweighting of self-nominations and reciprocal nominations; 3) year-over-year stability analysis — reputational scores that change by more than 10 % in a single year shall be investigated before inclusion.

SE-7.4 Academic Integrity and Anti-Misconduct Mechanisms

SE-7.4.1 Data Integrity Verification

a) The ranking entity shall implement procedures to detect and address data manipulation by ranked institutions, including:

1) Outlier detection — Institutional data points that deviate by more than 2 standard deviations from the expected range shall be flagged for verification.

2) Consistency checks — Data submitted by institutions shall be cross-verified against independent sources (government databases, bibliometric databases, accreditation records).

3) Time-series analysis — Sudden, unexplained improvements in key indicators shall trigger verification requests.

b) Institutions found to have deliberately submitted falsified data shall be excluded from the ranking for a minimum of three (3) evaluation cycles and the exclusion shall be publicly disclosed.

SE-7.4.2 Citation Manipulation Detection

a) The ranking entity shall implement citation integrity screening, including:

1) excessive self-citation clusters (beyond the cap specified in SE-7.2.3); 2) citation cartels — mutual citation patterns among a small group of authors or institutions that are disproportionate to the cited work’s influence; 3) coercive citation — citations added under editorial pressure rather than scholarly merit.

b) Detected citation manipulation shall result in adjustment of the affected bibliometric indicators, with the adjustment method documented.

SE-7.4.3 Publication Strategy Distortion

a) The ranking entity shall monitor for and mitigate ranking-induced publication strategy distortions, including:

1) paper milling — increasing publication volume at the expense of quality; 2) salami slicing — dividing one substantial study into multiple minimal publishable units; 3) journal targeting — selecting journals for impact factor rather than disciplinary fit.

b) Indicators that reward only publication count shall not be used. All volume-based indicators shall be paired with quality-adjusted counterparts.

SE-7.5 Educational Equity Considerations

SE-7.5.1 Contextual Fairness

a) The ranking methodology shall account for the following contextual differences that affect institutional performance:

1) Resource endowment — Institutions with lower per-student funding shall not be directly compared to well-endowed institutions without adjustment. The ranking entity should implement a resource-adjusted scoring model or provide contextual banding.

2) Geographic context — Institutions in developing regions face systemic disadvantages in infrastructure, funding, and international visibility. The ranking entity should provide regional sub-rankings or contextual adjustment factors.

3) Institutional mission — Research-intensive, teaching-focused, and comprehensive institutions serve different missions. The ranking entity shall either restrict its scope to a single mission type or implement mission-adjusted weighting.

4) Student intake selectivity — Outcomes of highly selective institutions shall be adjusted for intake quality to isolate the institution’s value-added contribution.

SE-7.5.2 Inclusion and Access

a) The ranking entity should include indicators that reward educational access and inclusion:

1) proportion of students from underrepresented or disadvantaged backgrounds; 2) financial aid and scholarship coverage rates; 3) first-generation student participation rates; 4) accessibility provisions for students with disabilities.

b) These indicators should not be punitive — lower access due to structural constraints shall not penalize institutions in contexts where the constraints are beyond institutional control.

SE-7.5.3 Language and Regional Bias Mitigation

a) The ranking entity shall implement measures to mitigate English-language publication bias, which systematically disadvantages institutions in non-Anglophone regions:

1) inclusion of regional-language publication databases where available; 2) field-normalization that accounts for citation practices in different language communities; 3) explicit disclosure of the percentage of English-language-only sources in the data collection.

b) Regional bias in reputational surveys shall be mitigated through balanced geographic sampling and, where possible, regional calibration of reputational scores.

SE-7.6 Peer Review Mechanism

SE-7.6.1 Expert Panel Composition

a) The ranking entity shall convene an expert advisory panel for methodology design and validation, comprising:

1) senior academics from diverse disciplines and regions; 2) educational assessment and measurement specialists; 3) institutional research professionals; 4) student and alumni representatives; 5) policy and governance experts.

b) The panel shall include no fewer than 15 members, with representation from at least 5 countries and 5 disciplinary clusters.

SE-7.6.2 Methodology Review Process

a) The ranking methodology shall undergo periodic peer review at intervals not exceeding three (3) years. The review shall assess:

1) validity and reliability of indicators; 2) weighting rationale and stability; 3) fairness and equity provisions; 4) data quality and verification procedures; 5) alignment with evolving educational research and policy.

b) Review findings shall be documented and published, and methodological changes shall be implemented in response to review recommendations with a documented response to each recommendation.

SE-7.6.3 Stakeholder Consultation

a) The ranking entity should consult affected institutions and stakeholders before implementing major methodological changes. Consultation shall include:

1) a public consultation period of no less than 60 days; 2) documented consideration of all substantive feedback received; 3) publication of a summary of feedback received and the ranking entity’s response.

SE-7.7 Data Collection Standards

SE-7.7.1 Data Sources

a) Institutional data shall be collected from the following source hierarchy, in order of preference:

1) Audited institutional data — data verified by institutional auditors, government audits, or accreditation bodies; 2) Government statistical databases — national education statistics, funding records, enrollment data; 3) Bibliometric databases — with cross-database verification as specified in SE-7.2.3; 4) Institutional self-reported data — accepted only with verification as specified in SE-7.4.1; 5) Survey data — with documented methodology, sample composition, and response rates.

b) The ranking entity shall disclose the proportion of data from each source type for each indicator.

SE-7.7.2 Data Timeliness

a) Data used in the ranking shall be no older than three (3) academic years prior to the publication date, except for alumni achievement indicators which may use longer time windows as specified in SE-7.2.5.

b) The ranking entity shall disclose the reference year for each indicator and shall note any lag between the reference year and the publication year.

SE-7.8 Publication and Transparency

SE-7.8.1 Methodology Disclosure

a) The ranking entity shall publish a complete methodology document, including:

1) all indicators with definitions, data sources, and calculation methods; 2) indicator weights with rationale; 3) scoring and normalization procedures; 4) data verification procedures; 5) treatment of missing data; 6) institutional classification and grouping criteria.

b) The methodology document shall be publicly accessible at no cost and shall be versioned with change logs.

SE-7.8.2 Results Presentation

a) Published results shall include, for each ranked institution:

1) overall score and rank; 2) scores for each dimension and, where applicable, individual indicators; 3) data completeness indicators — the percentage of indicators for which data was available; 4) contextual notes — institutional size category, mission type, geographic region, and resource band.

b) The ranking entity should provide tools or formats that enable users to re-weight indicators and observe the impact on rankings, thereby promoting informed and critical use of the results.

Note: This extension module is part of ICO Std 2002 (Tianji Ranking Methodology Standard). Education rankings carry significant institutional and public-policy consequences. The requirements in this module are designed to ensure that such influence is exercised with methodological rigor, contextual fairness, and transparency. Implementation of this module shall be validated by the SE-07 Working Group with participation from educational assessment specialists, academic governance experts, and institutional research professionals.