IS EFFECTIVENESS OF STUDIES A SIGNIFICANT CRITERION IN ASSESSMENT OF STUDY PROGRAMMES?

 

Dr. Gintautė Žibėnienė

Vilnius Pedagogical University, senior expert of study and teaching quality evaluation

zibeniene@gmail.com

Professor, dr. habil. Vilija Targamadzė
Klaipėda university Vice-rector for international relations

vilija.targamadze@ku.lt

Inga Juknytė-Petreikienė

Klaipėda College of Social ScienceDirector of Vilnius filial

ingajp@klsmk.lt

 

Summary

The article analyses the concept of effectiveness of studies and economic utility. It discusses theoretical aspects (including effectiveness of studies) of assessment of quality in study programmes. Our assumption is that assessment of effectiveness of studies is a significant criterion in studies’ assessment. This criterion is predetermined by the market economy and information society. The article also reviews application of effectiveness of studies as a criterion of assessment or a theoretical principle in assessment of quality in study programmes in Lithuania.

 

Key words:  effectiveness of studies, criteria, study programme.

 

 

Introduction

The information society, market economy and a focus on harmony of science, studies and labour market prompt a continuous concern for quality of studies, in particular quality of the studies that are being realised at present. Assessment of quality of studies comprises a very wide spectrum of assessment, i.e. activities of an institution, study programmes or syllabi (including the study process). Therefore, it is of utmost importance to select appropriate points of departure, namely the theoretical aspects, on which we base assessment of quality of studies. The analysis of literature on the subject, documents of organisations conducting assessment of quality, and certain website pages has revealed a changing approach to quality itself and assessment of quality in study programmes for that matter.

Scientific novelity. Analysing the practice in other countries it is possible to envisage various assessment methods, assessment levels and different periodicity of assessment. However, the author of this research was unable to find enough scientific articles where the study effectiveness as criterion in assessment of quality of study programmes is presented as one of the most significant components. There is a lack of empirically and theoretically grounded research in the assessment in this aspect.

 

The aim of the article is to provide grounds for importance of effectiveness of studies in assessing the study programmes.

The research object is assessment of quality of study programmes in terms of effectiveness.

The research method applied is the analysis of theoretical literature, documents and website pages.

 

Concept of effectiveness of studies

H.M. Levin mentioned in academic literature already in 1975 that the study programme had to be primarily evaluated taking into consideration its costs and economic utility, and in 1983 this idea was developed to a consistent research of measuring economic utility and deep analysis in a monograph of the author in question. According to M. W.  Mclaughlin and D.C. Phillips (1991) the ideas of H.M. Levin can be treated as a separate model of assessment, which promoted theoretical underpinnings for the important role of studies’ effectiveness in assessment of study programmes. A relatively varying concept of the studies’ effectiveness can be observed in the academic literature. The English word ‘effectiveness’ in English – Lithuanian dictionaries is explained as efficacy, efficiency, providing of impression, and in the websites of institutions organising assessment of quality of studies it is described as rational use, efficiency and harmony of human and material resources, other inputs and achieved results. Some of the researches of study programmes maintain that this concept comprises the meaning of the term ‘efficiency’ (which in English – Lithuanian dictionaries is explained as efficacy, effectiveness, productivity, capacity, working capacity, usefulness, or/and economic utility. P.H. Rossi, M.W. Lipsey, H.E. Freeman (2003) use the term ‘efficiency’ and defines it as justification of costs and correspondence between costs and the achived result. Other authors claim that in assessing a programme the focus should be on the analysis of programme effectiveness, provision, procedures, products, and processes (Hatry, Winnie, Fish, 1981).  P.H. Rossi, M.W. Lipsey, H.E. Freeman (2003) is also noted that the general effectiveness and effectiveness in individual areas can be assessed. L. Rutman, G. Mowbray (1989) also acknowledge significance of effectiveness and indicate that effectiveness determines the nature of development of studies, funding of their creation and satisfaction of customers of the study programme, also if the results were worth the investment. Such explanation and the analysis of the surveyed literature enable us to claim that two concepts can be observed in the sources:

1) study effectiveness is a broad concept comprising efficacy of realisation of studies, customer satisfaction and economic utility;

2) study effectiveness and economic efficiency are two different concepts, the first of which is oriented to a general (versatile) usefulness of studies and the second is oriented to a consistent research of measuring economic utility.

It is also noteworthy that according to the analysis of the documents of assessment organisations and the web pages, provided a choice can be made whether to apply effectiveness of studies as a criterion in assessing study programmes, it is applied both in the internal and external assessment. The effectiveness of the study programme is the assumption for the assurance of quality of study and is the essential component of the system of assessment of the study quality.

 

Theoretical aspects of assessment of quality of study programmes comprising effectiveness of studies

The analysis of the literature on the subject has disclosed various theoretical aspects, which serve as a basis for assessment of quality of study programmes, and the instances where the importance of assessing the studies’ effectiveness is stressed. P.H. Rossi, M.W. Lipsey, and  H.E. Freeman (2003) maintain that in measuring the quality of a study programme the key issue is a periodical diagnosis of quality in one or more areas of the following five: 1) need for the programme,  2) drafting and conceptualisation of the programme, 3) implementation of the study programme, 4) input and results, 5) efficiency. E. Seyfried (1998)[1] notes that evaluation can be oriented towards assessment of the product quality or establishment of profit and emphasises different experience of various countries. E. Seyfried (1998) distinguishes three most popular theoretical principles of assessment oriented towards: 1) the context, input (contribution), process and product; 2) process (a systemic analysis, effect of the assessment results on the process, research of action); 3) output (the analysis of results, consequence, and efficiency of costs). In summarising the analysis the researcher distinguishes phases of assessment of a study programme: 1) assessment of internal conditions for implementation of the programme (the analysis of the context, input, measurable values, and feasibility); 2) assessment is aimed at the process of studies (the analytical analysis of the process, review of the prepared self-assessment, efficacy of the feedback in the process); 3) assessment of effectiveness of the study programme and quality of the specialists trained (assessment of teachers and target groups, assessment of  the level of competences and skills development, the level of the graduates’ employment, assessment of costs for achieving the result). A. Harrell et al. (1997) determine four most popular aspects of assessment: 1) effect (influence); 2) achievements; 3) process; 4) costs. In this case the above mentioned author lays more emphasis on the significance of economic utility and recoupment.  Having surveyed the literature on assessment of quality of study programmes, N. Saugėnienė (2003) distinguished three groups: 1) programme, provision, procedures, products, and processes; 2) effectiveness of the programme including utility, economy; 3) contexts, input, processes and the result.

The analysis of the website pages of INQAAHE[2], ENQA[3], CEE[4] has disclosed the most common aspects, on the basis of which the quality of study programmes is being assessed: 1) input (level of people entering the institution to study, criteria of their selection and grouping of students, qualification of teachers); 2) process (effectiveness of realisation of the aims of studies, organisation of the study, implementation of the study content); 3) effect (the analysis of statistical numbers of graduates, drop-outs, and those with successful employment and the context). On the grounds of the analysis of INQAAHE it can be noted that in some countries economic efficiency serves as a significant characteristic in assessing study programmes. As a separate criterion economic utility of studies is identified in Albania, Argentina, Denmark, Honk Kong, India, Indonesia, Jamaica, Japan, the United States of America (hereinafter ‘America”), Mexico, Holland, Oman, Trinidad and Tobago, and Germany.

The assessment of study programme quality is important and multiplex research with many criteria’s. The overview of the literature on the subject has revealed the following: the common feature is the fact that the authors though differing in terms they apply, still indicate phases-based nature of assessment, and describe the process with an indispensable periodical measurement - assessment of quality” G. Žibėnienė (2005) propose “The study quality assessment is an educational phenomenon; the assessment as a permanent process is grounded by the theoretical level of curriculum, and by the on-going renewal and the monitoring of the study programme” (p. 8, 2005). This predetermines a consistent assessment of the study programme, oriented to the subsequent key principles: 1) input (conditions for implementation of the study programme, level of entrants and grouping) 2) drafting (designing) the study content, 3) process (the process of studies, realisation of the study content), 4) effect (results, consequences), and 5) effectiveness (rational use of human and material resources and other inputs; their efficacy and harmony with the achieved results).

 

Applying effectiveness of studies as a theoretical principal or assessment criterion in Lithuania for assessment of the quality of study programmes. Lithuanian education system is integrating into the European framework of education and the priority must be given to developing the system of ensuring the quality of study the most important component of which is the assessment of study programmes[5]. In assessing the quality of study programmes in Lithuania, effectiveness of studies has not received adequate attention. Žibėnienė (2005) in research predicate that the methodology of the assessment is orientated towards these theoretical aspects of assessment: 1) input (the conditions for the implementation of the study programme, the educational background of the study applicants, and student selection), 2) the projection of the study subject-matter, 3) the process (study process, the implementation of the study subject-matter and the realization of the study objectives); however it is not enough orientated sufficiently towards the aspects of impact and efficiency. The research proved the lack of and the need for the study programmes assessment grounded on theoretical assumptions. Also the necessity for improvement of the system of the assessment of the non-university study programmes in Lithuania because of the absence of clear assessment conception and inefficient functioning of its components is revealed in the research as this conforms with the survey of persons involved in education policy decision making and highly-skilled experts.

In consideration of the fact that there is a number of independent education institutions in Lithuania, it is of major importance for the drafters of study programmes to undertake a deeper analysis of the aspects, which assist assessment of expediency of the study programme at the primary level of planning already. Let us consider the following aspects as an example: providing grounds for need of the study programme or the potential and actual situation of the ratio of readiness and possibilities to implement the study programme.  Such aspects are relevant both in Lithuania and abroad. America and many European countries devote special attention in order to determine whether an institution of education is capable, whether it fully serves the purpose to begin realisation of a new study programme, what is the relation of the programme in question with the currently implemented ones, and whether it is not a potential for a new specialisation (Royse, Thyer, 1996). It is quite common in America to assess the intended effectiveness of the study programme by estimating the necessary input and assessing future prospects and possible profit (Hatry,  Winnie, Fish, 1981). However, it is not being done in Lithuania or only very tentatively. During accreditation of a study programme, effectiveness of the programme is also an important criterion of assessment (according to experience of other foreign countries). However, the analysis of the documents related to assessment of studies’ quality has disclosed that there is lack of attention to assessment of rational use of human and material resources and other inputs (contributions) and comparison of the input with the achieved results in Lithuania and some other European countries.

With the aim of effectiveness of studies, the preparation should be undertaken in advance. An institution of studies intending to introduce a new study programme apart from such important steps as research of occupations, planning and assessing the study content, analysis of human and material resources, should also consider other significant aspects, namely those contributing to evaluation of the need for a new study programme and its further effectiveness.  

1) The data and associated surveys should be conducted and analysed with a due care, evaluating a possible demand for the intended programme not only in the upcoming years, but also in the subsequent year. The potential for employment of graduates should be evaluated. For instance, there are cases where a study programme exists for a year, but next year it does not attract enough students and either vegetates, or disappears.

2) The essential, important and less important material, technical and methodological means necessary for realisation of the study programme should be planned and identified in detail by drafting a plan for their procurement.

3) A draft of the detail plan of financing should be compiled by planning expenses to create, and update the new basis, invite new teachers, indicating actual sources of funding.

4) The calculations of evaluation the total cost of the study programme, as well as the time frame of recoupment should be as exact as possible.

5) Planning of the new study programme should be begun from evaluation of the need of the study programme/s and not from drafting the programme content and planning of how to use the existing teaching staff.

 

Conclusions

The literature on the subject distinguishes two concepts associated to effectiveness of studies in assessment of study programmes:

1) effectiveness is a broad concept comprising efficacy of realisation of studies, customer satisfaction and economic utility;

2) effectiveness and economic utility are two different concepts, the first of which is oriented to a general (versatile) usefulness of studies and the second is oriented to a consistent research of measuring economic utility.

Assessment of effectiveness of studies is a significant criterion in evaluation of the study programmes and is predetermined by the market economy and information society. Assessment based on such criterion would assist a better evaluation of advantages and disadvantages of a study programme, contribute to disclosing a more actual need for it, relevance of the study programme to the institution of education and would improve the quality of the study process in general.

 

Literature

1.                   HARRELL A. et al. Evaluation strategies for human Services Programs. From: A Guide for Policymakers and Provider. USA: The Urban Institute, 1997, p. 2-30.

2.                   HATRY, H., P.; WINNIE, R., E.; FISH, D., M. Practical program evaluation for state and local governments. Washington: the Urban Institute Press, 1981. 123 p. ISBN 87766-296-7.

3.                   MCLAUGHLIN, M., W.; PHILLIPS, D.C. Evaluation and education: at quarter century. Illinois: the University of Chicago Press, 1991. 296 p.

4.                   ROSSI, P., H.; LIPSEY, M.; W., FREEMAN, H. E. Evaluation. (7^th ed.). Thousand Oaks-London-New Delhi: SAGE Publications, 2003. 470 p. ISBN 0-7619-0894-3.

5.                   RUTMAN, L.; MOWBRAY, G. Understanding program evaluation. (3^rd ed.). Newbury Park-London-New Delhi: SAGE Publications, The Publishers of Professional Social Science, 1989. 110 p. ISBN 0-8039-2093-8.

6.                   SAUGĖNIENĖ, N. (2003). Ugdymo programų planavimas ir realizavimas. Kaunas: Technologija, 2003. 50 p. ISBN 9955-09-490-7

7.                   SEYFRIED, E. Evaluation of quality aspects in vocational training programmes. Luxembourg: Office for Official Publications of the European Communities, 1998. 43 p. ISBN 92-828-5161-3.

8.                  ZIBENIENE, G. Theoretical and empirical substantiation of the assessment of non-university study programmes quality. Vilnius: Leidykla. 32 p. UZSAK 05-088.