ERC PISA Computer Based Assessment Project 2015

on

Introduction:

The Programme for International Student Assessment (PISA) is an international assessment of ‎the skills and knowledge of 15-year-olds. PISA ‎assesses students’ performance on ‘real-life’ tasks that are considered relevant for effective ‎participation in adult society and for lifelong learning.

PISA is implemented every three years, ‎starting in 2000, and 2015 sees the sixth implementation of the study. The number of countries ‎participating in PISA has increased from 32 in 2000 to about 70 in 2015, making it the largest study of ‎its kind‎.

PISA is a project of the Organisation for Economic Co-operation and Development (OECD). In ‎Ireland, the project is managed by the Educational Research Centre on the behalf of the ‎Department of Education and Skills.

Hybrid Technology Partners won the tender requested by the ERC to provide the technical support and logistical administration in delivering the PISA assessment, which was to be carried out for the first time using IT technology.

Challenge:

The dates of the PISA computer based assessment to 168 secondary schools around the Republic of Ireland was already predetermined before Hybrid TP became involved. The time frame of one month meant that there was a lot of overlap between schools on particular dates. To ensure that the schools were covered with competent Technical Support people and minimise the amount of laptops needed to cover the tests, called for a carefully managed solution to ensure the test was delivered.

This involved the recruitment of suitably qualified technicians to provide technical support on the schools assigned day, and the logistical movement of the IT hardware on which the assessment was being carried out.

This test was the first PISA test to be carried out using IT and the chosen form of delivery was software running on a Windows Laptop. Given the nature of the new technology and its ability to be used to access external networks, which could compromise the validity of the test, extra care was needed to ensure the test was taking place in a closed network with no access to the internet.

The test information then had to be taken from each laptop and uploaded to the ERC’s server where an assessment on its validity could be ascertained.

The administering of the test for the first time using laptops presented a few unique challenges.

  • The covering of schools in the same hub (the project firstly divided the schools into hubs at various locations around the country) with both batches of laptops and Technical Support personnel.
  • The provisioning of an adequate amount of laptops to cover each school in the Hub areas
  • Setting up the laptops and loading the PISA software in adequate time.
  • Health and Safety – the tidying of cables etc.
  • The collecting and uploading of the exam results.
  • Logistically moving batches of laptops from one designated school to another.
  • The movement of batches from one hub to another to maximise efficient use.
  • The hiring of competent Technical Support for each area with the required business insurance

Solution:

Project Planning:

The project firstly divided the schools into hubs at various locations around the country. The project then looked at the dates for the schools in each hub and checked to see how many overlapped. The logistical planning of maximising the distribution batches of laptops most efficiently was then calculated using an algorithm that correlated postal addresses within the hubs and the dates the assessments were taking place. This was done to keep the number of laptop batches to a minimum (a single batch consisted of 43 laptops).

The next stage was the recruiting of Technical Support which was done by profiling them to be sure they had the proper insurance and business transport and interviewing them to qualify their suitability to the job. Then each TS, for each assigned hub, were allocated their assigned schools.

Training:

The training given beforehand stressed the importance of being on site an hour before the exam started and incorporating the help of Teaching Assistants and school staff to help in the set-up of both rooms for the test. The Technical Support (TS) then set-up their assigned batch of laptops in the rooms provided. All power strips came with 5-metre rubber cable covers. TS’s were shown how to log any issues as soon as they happened. The TS’s were then shown how to collect the exam data from each laptop via a batch file on a usb stick.

Technical Challenges:

The development of a batch file stored on a USB stick and distributed to Technical Support to automate the collecting of assessment reports. This would allow for quicker collection and upload to the ERC giving them the chance to make a qualitative decision on the data and whether enough was being collected to be statistically usable.

Results:

The project was a huge success with the ERC collecting a large enough data pool from the schools to give an accurate indication of the educational level of 15 year old students throughout the country.