Trac Talent

The MS Word document (9 pages) attached to this post discusses different Assessment Standard setting processes.

Click the link below to download and read the document.

http://tractalent.com/tmp/Setting%20Assessment%20Standards.doc

This post defines Standards and lists the uses of a Standard. This is a preamble to a detailed discussion on Assessment Standards.

What Is A Standard?

A Standard is a published document that contains a technical specification or other precise criteria designed to be used consistently as a rule, guideline, or definition. Standards help to make life simpler and to increase the reliability and the effectiveness of many goods and services we use. Standards are created by bringing together the experience and expertise of all interested parties such as the producers, sellers, buyers, users and regulators of a particular material, product, process or service.

Standards are designed for voluntary use and do not impose any regulations. Any Standard is a collective work. Committees of manufacturers, users, research organizations, government departments and consumers work together to draw up Standards that evolve to meet the demands of society and technology.

What Are The Benefits Of Standards?

Standards are a powerful tool for organizations of all sizes, supporting innovation and increasing productivity. Effective Standardization promotes forceful competition and enhances profitability, enabling a business to take a leading role in shaping the industry itself. Standards allow a company to:

• Attract and assure customers
• Demonstrate market leadership
• Create competitive advantage
• Develop and maintain best practice.

Read the rest of this entry »

This post is to announce the release of Dr. V. Natarajan’s ebook on IRT titled “The Basic Principles of Item Response Theory”.  

Dr. V. Natarajan is one of the world’s leading experts on Item Response Theory. He has worked extensively on theory and applications of IRT. He is passionate about IRT and believes it is one of the greatest inventions of the 20th century. Through this ebook, he wishes to make IRT accessible and interesting to young learners of assessment theory. He has made this ebook available for free downloads since he wishes it to reach as many students of assessment as possible.

Dr. V. Natarajan, is Professor Emeritus at MeritTrac Services. He is an engineer and a D.Litt in Educational Evaluation and Administration, guides the Test Development and Research teams in MeritTrac. The author of 68 books and over 60 international papers on assessment, he is a pioneer in the area of education assessments in India. He is a visiting faculty at ETS, Princeton, USA, and has been a member of the Association of Indian Universities for over three decades. He had the privilege of being taught the advanced principles of IRT from Dr. Fred Lord of ETS.

You can click the link below to download a copy of the ebook.  

http://tractalent.com/tmp/Basic%20Principles%20of%20Item%20Response%20Theory.pdf

Checking duplicates while processing results on an Excel sheet is a basic requirement. This post provides step by step instructions on doing this.

Instructions 
1) Checking duplicate records in an Excel sheet is a standard requirement in most of Result Processing  
2) Duplicate records are identified on the basis of a unique field. For instance “email id” or “application reference number” could be the fields which have to be unique for each of the records in the Excel sheet.  
3) The unique field will be used to check duplicate records in an Excel sheet  
4) The best way to check for duplicates records in an Excel sheet using the unique field is to use the Countif formula. 

Steps to Use the Countif Field 
1) Setup a new column next to the Unique field. Call this column as “Duplicate Check” 
1) Enter the countif formula in the Duplicate Check column in the first row. CountIf formula takes two inputs - Range and Criteria.  
2) Set the entire column of the unique field as Range. Set the entry in the Range to absolute reference.  
3) Set the top most unique field record as the criteria for the first field.  
4) Drag down the formula to all rows in the “Duplicate Check” column.  
5) Setup a filter on the “Duplicate Check” column. The rows which have duplicates will have an entry greater than 1 on it. Rows with no duplicates will show a value of 1. 
6) This is illustrated in the attached Excel sheet with an example below where Appln Reference Number is the unique entry. Click the link below titled “Duplicate Checking Guide” to download the Excel sheet

Duplicate Checking Guide

This post has a link to download the ebook ”A Beginner’s Guide To Recrutiment” written by Mohan Kannegal with Karthik Vijayaraghavan. Click the link below to download the ebook.

http://www.docstoc.com/docs/2719862/A-Beginners-Guide-To-Recruitment

The key note address was by Prof. Robert J Mislevy (Prof of Statistics and Measurement - University of Maryland http://www.education.umd.edu/EDMS/mislevy/). He has done research on characteristics that seperate an expert from a novice and used the learning to design assessments that are able to better identify experts and novices. The PPT is available at (http://www.iaea2008.cambridgeassessment.org.uk/ca/digitalAssets/135859_Mislevy.ppt).

A quick snapshot … Humans have two limitations while solving problems

1) Processing Limitations (Limited Attention & Working Memory).
2) Knowledge Limitations (not knowing what info is relevant, not being able to integrate info, not knowing what to expect, lack of production profiency)

An “EXPERT” in any field is one who can OVERCOME these limitations by enhanching his/her capabilties. (Capabilities are reasoning with patterns, improvement with practice, think about thinking, benefiting from process, methods, symbols etc).

The factor that differentiates Experts in any fields from novices are

- Experts organize knowledge more effectively
- Experts interaction with a situation through “percieve - understand - act” cycles. They dont jump to act. 
- Experts use knowledge representations - like symbols, nomenclature, tech jargon etc

So if you are designing an assessment to seperate Experts from Beginners keep the following in mind

- Ask these questions while desinging the assessment - What needs to be measured? What behavior or performance reveals what needs to be measured? What tasks draw out those behaviors or performance.

- In actual experiments what has been found is that experts and novices both succeed in achieving an outcome in a given task. The difference is that an Expert spends more time planning and less time executing so an expert ends up doing less revision involving rework. Experts also assume implicit constraints that novices do not consider leading to sub-optimal solutions.

- Some definitions of Expert versus Novice in the context of dentists but equally extendable to generic use are given in the tables below.

Actual examples of tests where the Professor has implemented this are Architectural Registration Examination, DISC Simulator and CISCO NetPASS.

Quite a few people have been asking me about the International Association for Education Assessments (IAEA 2008) Seminar at Cambridge that I attended this month. So here is a series of blog posts on some of the things I learnt (I am motivated to write all this since I do not want people to – rather rightly – suspect that I was holidaying in a quaint English town )

This post contrasts the Education Outcome Standards of England versus those in India.

In England, a regulator for education sets assessment standards which define what the outcome of education should be for different levels of schooling and college. The regulator, Ofqual (http://ofqual.gov.uk) ensures that all exams across different boards that award degrees follow standards while conducting examinations. A very interesting document to read would be the National Curriculum Assessments Regulatory Framework (http://www.ofqual.gov.uk/files/qca-06-2827_regulatory_framework.pdf) published by ofqual which highlights assessments standards. For instance, Page 12 of this document mentions what a Reading Test for target year group 2 should encompass. This ensures that students get fair exams that accurately reflect the quality of education imparted and imbibed.

Also, the research effort on education in general and assessments in particular to ensure that assessments are fair, accessible and of good quality is very high. The introspection on whether assessments are appropriate is also very high in England. This was noticeable in all presentations made by English representatives at the Seminar. An example of this is seen at http://ofqual.gov.uk/files/Ear_Chart03.pdf where a flow chart for appeal in case a student suspects that results are not correct is presented. While educational institutions and Universities in India also have similar processes for education regulation in general, I have noticed that there are no outcome/assessment standards defined. Further, the level of introspection on whether assessments are fair is lower. And most importantly assessment research in Education seems limited.

This is an area we in India can possibly do more work on.

I was at the International Association for Education Assessments 2008 Seminar at Cambridge, UK. This is the best seminar I have ever attended. The seminar had sessions covering some very exciting new areas in assessments. And the fact that it was at Cambridge magnified the effect.

I landed in London on 7th September and took the train from Heathrow to King’s Cross and from there on to Cambridge. The seminar was conducted at Robinson College and we stayed in student accommodation at Robinson College itself.

Cambridge is a quiet University town with little traffic, gigantic medieval buildings and several colleges. Since the town is on the river Cam and each college has its own private bridge across the across the river Cam, its called Cambridge.

The River Cam

For those interested in trivia … there are 31 college under Cambridge University. The oldest is Peterhouse founded in 1284 and the newest is Robinson College founded in 1977. The richest is Trinity college with 700 million pounds in fixed assets. Incidentally Jawaharlal Nehru, Rajeev Gandhi and Amartya Sen studied at Trinity College. And it is featured in Harry Potter. The picture most often shown of Cambridge is the picture of Kings College and its chapel.

The key note address for the IAEA 2008 was by a Professor Mislevy from the US who spoke about the importance of defining what an “expert” and a “novice” in any given field are. This definition serves as the start point to create assessments and to interpret scores from an assessment. He went on to describe definitions arrived at for Architecture and Dentistry from his research. This research is of great use to test creators since it can help in creation of highly valid tests. You can read more about the key note address at
http://tractalent.com/blog/2008/09/22/iaea-2008-learning-2-assessments-to-seperate-experts-from-novices/
and download the presentation at
http://www.iaea2008.cambridgeassessment.org.uk/ca/digitalAssets/135859_Mislevy.ppt
 

Several other presentations were very interesting. There is a new e-marking software application that has become popular in the UK over the last couple of years. University and School examinations are corrected on computers. Once students complete an examination, the answer scripts are scanned using huge scanners at a central processing location. The images of the answer scripts are then uploaded into a workflow based software application. The answer script images are then assigned to teachers and freelance markers for corrections. With a software application similar to Outlook, teachers and freelancer markers can download the answer scripts onto their computers sitting at home or at work and pull up the answer script images on the computer and correct it. All scores are entered into the application which automatically validates and totals the scores and sends it back to the central server. This dramatically increases efficiency of correction by removing the need to physically ship answer script papers to teachers doing the corrections and by making some of the steps such as score totaling automatic. Further freelance markers can easily be hired and used for marking. Apparently a lot of corrections of UK student responses are done out of India and Australia.

Another interesting company is Vantage Laboratories (www.vantage.com) which offers a software solution for automatic correction of essay type answers. This application is being explored by the Product Development Team and we hope to see an integration into OnTrac soon.
 

On the tests side, there were very interesting presentations on whether abilities tests actually test aptitude, on the taxonomy of Critical Thinking Tests and the validity of different methods of deciding cutoffs. All the papers presented are available at
http://www.iaea2008.cambridgeassessment.org.uk/
 

My own presentation went very well. I opened with how overwhelmed I was to be presenting at Cambridge considering 2 or 3 of our Prime Ministers have studied at Cambridge … a few smiles. I also said ‘Cambridge is a metaphor in India for what is unachievable … for instance if you speak English well in school, people would say “Dont speak like you studied at Cambridge’” … this was greeted with a lot of smiles. I then presented on how we have setup Pariksha centers and how they are enabling us to deliver large volume online tests. People had several questions about scalability of the model, whether we would be interested in extending this model to Africa, whether the use of biometric was legal in India (it is illegal in the UK to collect biometric data) and so on. All in all, the audience was impressed with what we had done. (email me for a copy of the PPT)
Madan’s presentation too went well. Madan presented on the NACTech program that has helped in creating more employability in India. There were questions on whether India had a regulatory body and a standards definition body. People from the UK were glad that since there were no such bodies, more innovation seems to have happened.

There was also a gala dinner organized at King’s College. Incidentally you cannot walk on the lawns of King College if you are a visitor. But we all were allowed to this as a special gesture. Also there is a private bridge for each college across the Cam where visitors are not allowed. We crossed this bridge too. Dinner was in a huge hall. 4 courses followed by coffee and brandy. There was also after dinner speeches. It was the real British experience. I had a senior person in Research at Cambridge Assessments to my right and Ugandans and Dutch around us. Truly international. You can see mine and Madan’s head in the photo below - bottom right - second column of tables from the right. Click on the picture to see a full sized image.

On the lighter side, I and Madan stayed on the third (top) floor of a student hostel at Robinson College which for some reason has decided not to have a lift. So we climbed six flights of stairs every day! The rooms were comfortable. We also strayed into a rock music place one of the evenings at Cambridge. We had to leave after two songs … realizing that we weren’t as young as we used to be. And this was the first conference I attended where Madan did not know any body. 

This post has a link to download the ebook ”A Beginner’s Guide To Recrutiment” written by Mohan Kannegal with Karthik Vijayaraghavan. There is also a short introduction to the ebook.

Click the link below to download the ebook.

http://www.docstoc.com/docs/2719862/A-Beginners-Guide-To-Recruitment

About “A Beginner’s Guide To Recruitment”

The success of companies hinges on the ability to recruit talent. Correspondingly, there is a need for the recruitment process to be run efficiently and productively. This book is intended to address this need by serving as a ready reference guide for Recruiters to enable them to hire better and for students interested in joining the recruitment function. The book explains the steps of the recruitment process that a Recruiter is involved in and suggests good practices for each step. In the chapters that follow, we have examined each step of the recruitment process in detail and have illustrated points with forms, tables and case studies where applicable.

This book is intended for reading by Recruiters and students who want to pursue a career in recruitment.

The book focuses on the set of recruitment activities in which Recruiters are involved. The start point of the recruitment process for a Recruiter is requisitioning people and the finish point is rolling out offer letters to selected candidates. In line with this, the book has a chapter each on

1. Requisitioning People
2. Job Opening Communication
3. Receiving CVs
4. CV Screening
5. Interview Scheduling & Coordination
6. Offer Making & On-boarding
7. Managing Recruitment

The recruitment process in companies also includes other aspects of recruitment such as manpower planning on the basis of business forecasts, managing different sources of CVs, using a variety of interviewing techniques and so on. This book does not cover these additional aspects of recruitment since these aspects of recruitment are usually handled by recruitment managers. There is a chapter on the use of pre-recruitment tests which is included since companies in India are increasingly using pre-recruitment tests and Recruiters are involved in the buying and day-to-day use of pre-recruitment tests. There are also a few new ideas on recruitment towards the end of book. 

Click the link below to download the ebook.

http://www.docstoc.com/docs/2719862/A-Beginners-Guide-To-Recruitment

.

MeritTrac follows a standard methodology to arrive at a Cut (passing) Score. A passing score is the minimum score a test taker should requires to score in a test in order to be eligible to go to the next level, in a hiring scenario.

The methods of setting the standards are based on the concept of considering responses from the “borderline” test taker.  This test taker is someone whose knowledge and skills are on the borderline area between the upper group and the lower group.  These methods are based on the idea that, once the test takers, who belong to the upper group actually start scoring higher than those who belong to the lower group, the passing score should be that which is expected from a person whose skills are just on the borderline.  The judgments these methods require are made in terms of the specific questions on the test…..

Click on the link below to download the complete article.

Cut Scores