Home / Assessment Methodology
The Faculty of Engineering at the Islamic University, Madinah Munawarrah has agreed to select the ABET outcomes from ‘a’ to ‘k’for its various specializations. The refinement, revision and evaluation of the student outcomes will take place on regular basis in consultation with all constituencies.
Outcomes assessment is a process which gauges a program's educational quality and provides for continual teaching and learning improvement. The focus is on providing a meaningful and relevant student learning experience. The process is generally correlated to an institutional mission.
Figure 4.1 gives a general flow of the key elements of a quality assurance outcomes assessment process.
Our program intends to engage in a meaningful, outcomes-based educational practice. The goal is to improve the teaching and learning experience, which ensures a high-quality educational program. The flow cycle for engineering programs is adopted from Gloria Rogers' work and has been modified to suit our needs.
i) Level 1: PEO Assessment
Here the program educational objectives are the statements that describe the expected accomplishments of graduates during the first few years after graduation. The student outcomes are statements that describe what students are expected to know and be able to do by the time of graduation. From the program perspective, the SO is what can be measured and should be used as a guide to steer students towards the PEO after graduation. After the SO is properly measured and assessed, the action items generated should then be used to further assess if the PEOs are really being achieved. As indicated in the flow cycle, this level of assessment is termed Level 1: PEO assessment. Since the PEO looks at a time frame after students graduate the assessment cycle for Level 1 is expected to be conducted once every three years. This cycle is sufficient to gauge the PEO relevancy to the needs of the program's constituents. Since action items are the result of outcomes assessment, which involves a different cycle of assessment, it is clear that the outcomes assessment for addressing the PEOs is eventually met.
ii) Level 2: Student Outcomes Assessment Cycle
To assess student outcomes meaningfully, we must have performance criteria to guide us. Performance Criteria are specific measureable statements identifying the performances required to meet the outcomes with confirmable, thorough evidence.
Educational practices and strategies may mean how you would implement various combinations of assessment methods and the frequency and interval of those assessment methods. The Faculty of Engineering decided on a process of assessment based on FCAR whereby each constituent of direct assessment will be assessed in each term.
Assessments are the processes that identify, collect, analyze, and report data that can be used to evaluate achievement. In this case, achievement is the student outcomes or course outcomes. Assessment is generally a time-consuming data collection process and an analysis of evidence for meaningful evaluation, the next logical step.
Evaluation is a process of reviewing the results of data collection and analysis and making a determination of the value of findings and actions to be taken. In view of this definition of evaluation, evaluation is a critical step to ensure the quality improvement process. Without proper evaluation, the assessment of data collection is rendered meaningless and eventually leads to failure and frustration for the parties involved.
With evaluation come action items. Perhaps the most meaningful step in the overall flow cycle of assessment is the action items that allow closing the loop for real quality assurance. With action items as the feedback loop to change or improve student outcomes, performance criteria, and educational practices, the cycle of outcomes assessment is repetitive or a work-in-progress. This form the Level 2: SO Assessment.
If the educational practices in Level 2 SO Assessment use embedded course assessment, then this gives rise to Level 3: CO Assessment. That is, outcomes of core courses in the curriculum are being assessed to see if student learning meets the course outcomes. Course outcomes assessment involves the same cycle of assessment as student outcomes. That is, performance criteria for outcomes have to be defined, the assessment process of collecting and analysis of evidence is conducted, and the evaluation process of interpretation of evidence needs to be performed, which eventually leads to action items for improvement at a course level. This level of assessment is the most time consuming and rewarding.
In conclusion, the overall assessment cycle consists of three levels of interacting assessment cycles: (1) the PEO assessment (3 year time frame); (2) the SO Assessment (every semester); and (3) the CO Assessment (every semester). These three levels of the assessment cycle needs different toolsets since their assessment time frames are different.
b) Three Levels of Assessment
The three levels of assessment cycles are:
i) Level 1: PEO Assessment
Figure below depicts the Level 1: PEO Assessment cycle from the perspective of an operational process. The constituents of the program are the various bodies who define or refine PEOs. The faculty members plan the assessment details and performance criteria. After the relevant objective evidence for PEO are collected, a committee consisting of faculty and External Affairs Committee (EAC) members analyzes evidence and generates action items for closure. The tools relevant at this level are alumni survey, employer survey, EAC council review minutes.
ii) Level 2: Student Outcomes (SO) Assessment
A similar operational flow process for level 2 assessment is given in Figure 4.1. The faculty members of the department/program define/refine the student outcomes that gauge the movement of graduates towards the PEOs. With student outcomes defined, the faculty again is the main body who defines performance criteria and implements the educational practices to collect relevant objective evidence on student learning. Depending on the assessment strategies, committee(s) shall analyze the evidence and generate action items accordingly for closure. If there were action items from the last review cycle, decisions have to be made to improve curriculum or to refine the existing student outcomes. The toolsets relevant for Level 2 are the senior-exit survey, the curriculum outcomes matrix, the student outcomes assessment matrix, and the student outcomes action items matrix.
Level 2: SOs assessment
iii) Level 3: Course Outcomes (CO) Assessment
Level 3 assessment is the core assessment of the courses that your program offers. It deals directly with the student learning experience as well as the teaching. Students who graduate from the program may not know student outcomes at level 2 or even the PEOs from level 1, but the experiences they have in class will last them a lifetime. So assessment at this level is critical to the long lasting success of the program. The assessment cycle for level 3 is shown in the figure below.
Level 3: COs assessment
For each course, the responsible faculty defines/refines course outcomes that students should master by the end of the semester. Once evidence on student learning is collected, the responsible faculty will analyze data for any action items. The suggested action items will then be reviewed for appropriateness and will be implemented the next time the course is offered. This is the most time-consuming level of assessment, not only because of the amount of data needing collection, but also in the time spent on compilation and the analysis of data.
In reality, the actual flow of data in the assessment process starts at Level 3, flows to Level 2, and then to Level 1. Level 2 and 3 assessment activities should take place every semester. The data collected from Level 3 are then used for Level 2 assessment. The student outcomes assessment values are collected over a 2-year period. Level 2 and 3 assessments are later averaged and utilized for the Level 1 assessment of PEOs at the end of the two year period.