http://tractalent.com/tmp/A%20Beginners%20Guide%20To%20Recruitment.pdf

The key note address was by Prof. Robert J Mislevy (Prof of Statistics and Measurement - University of Maryland http://www.education.umd.edu/EDMS/mislevy/). He has done research on characteristics that seperate an expert from a novice and used the learning to design assessments that are able to better identify experts and novices. The PPT is available at (http://www.iaea2008.cambridgeassessment.org.uk/ca/digitalAssets/135859_Mislevy.ppt).

A quick snapshot … Humans have two limitations while solving problems

1) Processing Limitations (Limited Attention & Working Memory).
2) Knowledge Limitations (not knowing what info is relevant, not being able to integrate info, not knowing what to expect, lack of production profiency)

An “EXPERT” in any field is one who can OVERCOME these limitations by enhanching his/her capabilties. (Capabilities are reasoning with patterns, improvement with practice, think about thinking, benefiting from process, methods, symbols etc).

The factor that differentiates Experts in any fields from novices are

- Experts organize knowledge more effectively
- Experts interaction with a situation through “percieve - understand - act” cycles. They dont jump to act. 
- Experts use knowledge representations - like symbols, nomenclature, tech jargon etc

So if you are designing an assessment to seperate Experts from Beginners keep the following in mind

- Ask these questions while desinging the assessment - What needs to be measured? What behavior or performance reveals what needs to be measured? What tasks draw out those behaviors or performance.

- In actual experiments what has been found is that experts and novices both succeed in achieving an outcome in a given task. The difference is that an Expert spends more time planning and less time executing so an expert ends up doing less revision involving rework. Experts also assume implicit constraints that novices do not consider leading to sub-optimal solutions.

- Some definitions of Expert versus Novice in the context of dentists but equally extendable to generic use are given in the tables below.

Actual examples of tests where the Professor has implemented this are Architectural Registration Examination, DISC Simulator and CISCO NetPASS.

Quite a few people have been asking me about the International Association for Education Assessments (IAEA 2008) Seminar at Cambridge that I attended this month. So here is a series of blog posts on some of the things I learnt (I am motivated to write all this since I do not want people to – rather rightly – suspect that I was holidaying in a quaint English town )

This post contrasts the Education Outcome Standards of England versus those in India.

In England, a regulator for education sets assessment standards which define what the outcome of education should be for different levels of schooling and college. The regulator, Ofqual (http://ofqual.gov.uk) ensures that all exams across different boards that award degrees follow standards while conducting examinations. A very interesting document to read would be the National Curriculum Assessments Regulatory Framework (http://www.ofqual.gov.uk/files/qca-06-2827_regulatory_framework.pdf) published by ofqual which highlights assessments standards. For instance, Page 12 of this document mentions what a Reading Test for target year group 2 should encompass. This ensures that students get fair exams that accurately reflect the quality of education imparted and imbibed.

Also, the research effort on education in general and assessments in particular to ensure that assessments are fair, accessible and of good quality is very high. The introspection on whether assessments are appropriate is also very high in England. This was noticeable in all presentations made by English representatives at the Seminar. An example of this is seen at http://ofqual.gov.uk/files/Ear_Chart03.pdf where a flow chart for appeal in case a student suspects that results are not correct is presented. While educational institutions and Universities in India also have similar processes for education regulation in general, I have noticed that there are no outcome/assessment standards defined. Further, the level of introspection on whether assessments are fair is lower. And most importantly assessment research in Education seems limited.

This is an area we in India can possibly do more work on.

I landed in London on 7th September and took the train from Heathrow to King’s Cross and from there on to Cambridge. The seminar was conducted at Robinson College and we stayed in student accommodation at Robinson College itself.

Cambridge is a quiet University town with little traffic, gigantic medieval buildings and several colleges. Since the town is on the river Cam and each college has its own private bridge across the across the river Cam, its called Cambridge.

The River Cam

For those interested in trivia … there are 31 college under Cambridge University. The oldest is Peterhouse founded in 1284 and the newest is Robinson College founded in 1977. The richest is Trinity college with 700 million pounds in fixed assets. Incidentally Jawaharlal Nehru, Rajeev Gandhi and Amartya Sen studied at Trinity College. And it is featured in Harry Potter. The picture most often shown of Cambridge is the picture of Kings College and its chapel.

The key note address for the IAEA 2008 was by a Professor Mislevy from the US who spoke about the importance of defining what an “expert” and a “novice” in any given field are. This definition serves as the start point to create assessments and to interpret scores from an assessment. He went on to describe definitions arrived at for Architecture and Dentistry from his research. This research is of great use to test creators since it can help in creation of highly valid tests. You can read more about the key note address at
http://tractalent.com/blog/2008/09/22/iaea-2008-learning-2-assessments-to-seperate-experts-from-novices/
and download the presentation at
http://www.iaea2008.cambridgeassessment.org.uk/ca/digitalAssets/135859_Mislevy.ppt
 

Several other presentations were very interesting. There is a new e-marking software application that has become popular in the UK over the last couple of years. University and School examinations are corrected on computers. Once students complete an examination, the answer scripts are scanned using huge scanners at a central processing location. The images of the answer scripts are then uploaded into a workflow based software application. The answer script images are then assigned to teachers and freelance markers for corrections. With a software application similar to Outlook, teachers and freelancer markers can download the answer scripts onto their computers sitting at home or at work and pull up the answer script images on the computer and correct it. All scores are entered into the application which automatically validates and totals the scores and sends it back to the central server. This dramatically increases efficiency of correction by removing the need to physically ship answer script papers to teachers doing the corrections and by making some of the steps such as score totaling automatic. Further freelance markers can easily be hired and used for marking. Apparently a lot of corrections of UK student responses are done out of India and Australia.

Another interesting company is Vantage Laboratories (www.vantage.com) which offers a software solution for automatic correction of essay type answers. This application is being explored by the Product Development Team and we hope to see an integration into OnTrac soon.
 

On the tests side, there were very interesting presentations on whether abilities tests actually test aptitude, on the taxonomy of Critical Thinking Tests and the validity of different methods of deciding cutoffs. All the papers presented are available at
http://www.iaea2008.cambridgeassessment.org.uk/
 

My own presentation went very well. I opened with how overwhelmed I was to be presenting at Cambridge considering 2 or 3 of our Prime Ministers have studied at Cambridge … a few smiles. I also said ‘Cambridge is a metaphor in India for what is unachievable … for instance if you speak English well in school, people would say “Dont speak like you studied at Cambridge’” … this was greeted with a lot of smiles. I then presented on how we have setup Pariksha centers and how they are enabling us to deliver large volume online tests. People had several questions about scalability of the model, whether we would be interested in extending this model to Africa, whether the use of biometric was legal in India (it is illegal in the UK to collect biometric data) and so on. All in all, the audience was impressed with what we had done. (email me for a copy of the PPT)
Madan’s presentation too went well. Madan presented on the NACTech program that has helped in creating more employability in India. There were questions on whether India had a regulatory body and a standards definition body. People from the UK were glad that since there were no such bodies, more innovation seems to have happened.

There was also a gala dinner organized at King’s College. Incidentally you cannot walk on the lawns of King College if you are a visitor. But we all were allowed to this as a special gesture. Also there is a private bridge for each college across the Cam where visitors are not allowed. We crossed this bridge too. Dinner was in a huge hall. 4 courses followed by coffee and brandy. There was also after dinner speeches. It was the real British experience. I had a senior person in Research at Cambridge Assessments to my right and Ugandans and Dutch around us. Truly international. You can see mine and Madan’s head in the photo below - bottom right - second column of tables from the right. Click on the picture to see a full sized image.

On the lighter side, I and Madan stayed on the third (top) floor of a student hostel at Robinson College which for some reason has decided not to have a lift. So we climbed six flights of stairs every day! The rooms were comfortable. We also strayed into a rock music place one of the evenings at Cambridge. We had to leave after two songs … realizing that we weren’t as young as we used to be. And this was the first conference I attended where Madan did not know any body. 

This post has a link to download the ebook ”A Beginner’s Guide To Recrutiment” written by Mohan Kannegal with Karthik Vijayaraghavan. There is also a short introduction to the ebook.

Click here to download the ebook (1.35MB) .

About “A Beginner’s Guide To Recruitment”

The success of companies hinges on the ability to recruit talent. Correspondingly, there is a need for the recruitment process to be run efficiently and productively. This book is intended to address this need by serving as a ready reference guide for Recruiters to enable them to hire better and for students interested in joining the recruitment function. The book explains the steps of the recruitment process that a Recruiter is involved in and suggests good practices for each step. In the chapters that follow, we have examined each step of the recruitment process in detail and have illustrated points with forms, tables and case studies where applicable.

This book is intended for reading by Recruiters and students who want to pursue a career in recruitment.

The book focuses on the set of recruitment activities in which Recruiters are involved. The start point of the recruitment process for a Recruiter is requisitioning people and the finish point is rolling out offer letters to selected candidates. In line with this, the book has a chapter each on

1. Requisitioning People
2. Job Opening Communication
3. Receiving CVs
4. CV Screening
5. Interview Scheduling & Coordination
6. Offer Making & On-boarding
7. Managing Recruitment

The recruitment process in companies also includes other aspects of recruitment such as manpower planning on the basis of business forecasts, managing different sources of CVs, using a variety of interviewing techniques and so on. This book does not cover these additional aspects of recruitment since these aspects of recruitment are usually handled by recruitment managers. There is a chapter on the use of pre-recruitment tests which is included since companies in India are increasingly using pre-recruitment tests and Recruiters are involved in the buying and day-to-day use of pre-recruitment tests. There are also a few new ideas on recruitment towards the end of book. 

Click here to download the ebook (1.35MB).

.

The methods of setting the standards are based on the concept of considering responses from the “borderline” test taker.  This test taker is someone whose knowledge and skills are on the borderline area between the upper group and the lower group.  These methods are based on the idea that, once the test takers, who belong to the upper group actually start scoring higher than those who belong to the lower group, the passing score should be that which is expected from a person whose skills are just on the borderline.  The judgments these methods require are made in terms of the specific questions on the test…..

Click on the link below to download the complete article.

Cut Scores

The Challenge Of Individualized CVs

The most popular mode of applying for a job is by typing out a CV in a format that appeals to the applicant (individualized), saving it as an MS Word or PDF or Text file and then emailing this CV to an employer. 60 – 90 percent of all applications a company receives are individualized CVs.

The problem with individualized CVs is that each CV presents information in different ways – some CVs start with contact information followed by education followed by employment history while others may miss out education information. Some CVs have information in chronological order while others may have it in reverse chronological order. Vital details might be missed out or may be incorrect – in our experience over 20% of all applicants either forget to mention their phone number or give a wrong phone number.

Further, screening individualized CVs is tedious and error prone. For instance, if a company’s selection criterion says, “should have a total of 2 years of experience on Java”, the CV screener has to add up time spent on each individual project in which Java was used and check if it adds up to 2 years. Moreover, since its a manual process and requires reading so much information screeners tend to make mistakes.

Since the CV screening process is time consuming, tech teams don’t do CV screening though it would be ideal if they did. It’s usually the recruitment team that does it. Since recruitment teams would not have a first hand feel of what is required it leads to wastage. For instance, a recruitment team sourcing Oracle DBAs might reject all SQL Server DBAs. Whereas the tech team may know that a few SQL Server DBAs are also required in the team since the project requires transitioning from SQL Server to Oracle databases. 

Finally, considering the number of CVs screened each month – approximately 200,000 software developer CVs screened in Bangalore alone – the process needs automation. But it is impossible to use automation to accurately parse individualized CVs owing to the fact that information is presented in so many different ways. Even the most “intelligent” parser is inaccurate.

Given these issues– multiple CV formats, incomplete information, tech teams not doing CV screening, automation unfriendly CVs – there is a need for a solution which will make the process of CV screening accurate and convenient.
 

The Software Developer Application Form

One solution is a web application form, which an applicant fills. The advantage here is that all CVs are in the same format making it easy to search and shortlist. The disadvantage is that it requires a candidate to fill out an application form each time he or she applies to a company. And filling out web forms is cumbersome. Companies who have offered applicants the option of filling out a web form or sending an email attachment have found that only 10– 20 % of applicants fill out web forms while others send email attachments. Even companies who say “web form or no way” have to make allowances for Placement consultants and Job sites since these sources cannot be easily integrated into web forms.
 
A simpler solution could be for IT companies to agree on a standard Software Developer Application Form, which is acceptable to all companies. Our own experience screening over 500,000 CVs for different software development companies has shown that all companies look for the same information on a CV. Across companies, employers are interested in experience years, technologies and projects worked on, current employer and education. Given this, it is possible to adopt a “Software Developer Application Form” that is acceptable to all companies – an Industry wide standard for software developer CVs. A sample Application Form with a some of the proposed 55 fields is shown in the table. This field list can be discussed and debated till a widely accepted standard is arrived at.

Table : Software Developer Application Form

:Source :Job Ad
:ApplyingFor:Java Developer
:Name:Ashish Ram
:Email:ashish@hotmail.com
:PassportNumber:T12040
:DateofBirth:2-Oct-73
:DayPhone(withSTDCode) :08026583827
:EveningPhone(withSTDCode) :08026584557
:City(Currentresidence) :Bangalore
:WillingtoRelocateto(CityNames) :Bangalore, Hyderabad, Chennai
:CurrentSalary(Annual) :3.5 Lakhs
:ExpectedSalary(Annual) :4.0 Lakhs
:Skills:Core Java, J2EE
:HighestQualification: Engineering
:HighestQualificationInstitution:REC Surat
:HighestQualification(%orCGPA) :65%
:TotalSoftwareDevelopmentExperience(months) :36
:Skill1Name:Java
:Skill1Expereince(months) :4

Some rules for the Application Form would be – do NOT change field names, field names are demarcated by separator like a “:”, Application Forms are submitted in plain text format (can be opened on any OS) and the Application Form name is always + + .

With a standard Application Form, the process of CV screening becomes accurate since CV screening becomes a “process” rather than an “art” – the rules are defined and easy to explain to anyone.  Further, it would possible to break up CV screening into two activities – the first part would be screening applicants on the basis of “hard” facts –age, educational background, willing to relocate etc which can be done by almost anyone. The second part would be to present the short-listed applicants’ information in a tabular format to Tech teams for them to choose candidates based on projects executed. This way, tech teams can participate in the CV screening process.

For applicants, the big benefit is “write once, apply several times”. A Job applicant would create a standard Application Form at the beginning of their career. As he or she goes along, they add or edit information on the Application Form. Each time they apply for a job, they open the standard Application Form, double check information, attach it and email it. And everyone - employers, placement companies and job sites - accepts the Application Form.

Finally, it is easy to write software to read the Application Form and upload the information into a database from where it is easy to search and retrieve information. And since all sources of CVs use the Application Form it is possible to put CVs from different sources into a single database – a big challenge currently.

Acceptance of such an Application Form has to be driven by the top IT services companies. These companies will have to agree on a standard Application Form and insist that everyone applying to them use it. With time the industry could move towards a “Only standard Application Forms accepted” model. To popularize the Application Form and to educate job applicants the Form could be made available on websites of employers, assessment companies, placement companies, HR portals, and emailed across to college campuses.

Scenario 1: As the manager of an internet book store, explain to a customer as to why there was a delay in the shipment of a book.

Let me clear my stand that, we had the books till the customer who bought before you and then it went out of stock.

It was due to some problem in the ship engine. The ship was laid down in the nearby island.

The books were already loaded to ship and the ship was about leave we got a red alert signal from the whetter forecast dept that there may be a storm, so you please pull your ship if there are leaving the shores. So because of whetter reasons we had to held the ships back.

The ship was stuck in the middle of the Pacific Ocean.

The ship was hijacked by sea robbers in the middle of the Pacific Ocean.

Scenario 2: As the manager of Reader’s Digest, send a mail to existing customers requesting them to renew their subscriptions.
Please subscribe today because tomorrow never dies.

The digest has various advantages which helps in development of mental health of customer.

Sooner you are, better you get.

I kindly oblique you.

Its being a problem for the reader’s reading Readers Digest.

It would be of great help if you could renew yourself as soon as possible.

Finally, I have a Career Objective taken from a resume that I found really interesting…the Monarch of Jargon!
Career objective:

Manipulating the tedious into salubrious to catch up with the palpable dividends with a relentless rapt for extracting the best from the good remains the hall mark in the scale of notching the apex.

Hope you had as much fun reading these as I had in compiling them. It would be great if all the evaluators could share their experiences with all of us!!

A ‘sense of right and wrong’ is pretty apparent a rationale, I know, but a fine combinational basis of deep-set assessment parameters furnished by MeritTrac and a sagacious ability to promptly identify or distinguish the best from out of the sizeable (sometimes sundry) restless lot seemed totally different a challenge for me. To judiciously recognize and assess the employable resources for our diverse list of clients is bound to remain a lucky inclination for me.

Basing upon my personal experiential knowledge, I tend to hypothesize on the lines that few of the job aspirants or test takers who take our wide-ranging tests seem not to realize the fact that almost every step of test taking is ‘measurable’ by standards (often defined standards), or is at once scalable based on the scores of assorted parameters relating to which we test administrators get on to score them.

Conversely, even if they (test takers) are superficially aware or are quite clearly knowledgeable about the aforementioned aspects, then they seem to take up all into their stride and score well, eventually ‘coming out’ with ‘flying colours’ and, then, an impending selection process awaits them. But, by sheer happenstance, may be, or like each cloud is said to have a silver lining, some likely selects most often than not happen to flounder. Well, this too can be reasonably explained as to why they happen to do so as they do! Several reasons have been propounded but lets not for the life of this article delve into that reality. That may be for some other time with another cup of tea.

I have had my “laughter challenges” during and after many English assessment procedures I carried out in the duration of the last three months now. I can’t help but think pretty often and habitually share them with other English Evaluators in my team, and I do so only for the sake of an agreeable experiential know-how though! There are so many unheard of bloopers, slip-ups, lapses, gaffes, and so many meaningless howlers that my personal list has already gone bust and went out of order, all smacking of some inconsequentially misused and abused English Language terrorism! But there is going to be an article written soon on that subject too in this blogosphere. So, puhleese! Hasten to fasten your seat belts!

There was as if a significant yet compelling itch to hilariously relate a few innocent offenses made by some fresh-off-the-boat aspiring candidates who were lined up for our English assessments, especially for the Spoken English first round, with my equally industrious associates. And I, eventually, could not help but surrender laughing at the often splitting or even atrociously funny, comical bloopers experienced first-hand! God is with me! Innocent mistakes are so sweet to know and almost therapeutically rejoicing! Likewise, there have been some silly peccadilloes and during that time amusing within our selves became the first point of release of our stresses and strains of our jobs. In the present times, it is more of been-there-done-that way of thinking that generally prevails upon us, without letting it concede on our deductive capacity. Bloopers or no bloopers, they matter a little less now.

The most significant facet of the E.E. exercise on a more epithelial level – depending upon the assessment criterion, for instance, whether it be Spoken, Written or Persuasive - is the elemental detection of ‘plausibility’ of the resource in question, besides which - hardly considered necessary to mention - there are other auxiliary technical traits like articulation (for Spoken - SET), sentence formation (both for Spoken and Written - WET), Indianisms, writing abilities (clearly for Written), persuasive abilities (for Persuasive Communication - PCT), etc. (all these based on defined set of parameters, normally on a scale of 1 to 5). ‘1’ is ‘poor’ and ‘5’ being ‘remarkable.’

On a personal level, I almost always tend to believe that any candidate or test-taker must have to be free from claustrophobic ideas because the answer to the given question should have to be only in accordance to the question asked and not fumbling into anything else unwanted that the interviewer never wishes to give all ears to. Because, if your answer has to be most suitable in its desired effect in that spur of the moment, then it should really have to come within the purview of the question asked and not getting “uncomfortably imbalanced” (only an opinion!) with the asker of the fine question! If not, the candidates lose marks and their general gravity found flaking off the scoreboard! The million-dollar question is how do we do it? Clear the mind from unwanted pounding that distracts. I think one needs a good night sleep after which in the morning a nice cool shower and a light breakfast in the belly! This thought is surely unconventional but trust the good Lord it helps the most.

‘Events’ the most popular action-oriented word in Operations department and in my English Evaluation dept. as well, is quite a weighty source for a good first-hand learning experience. I dub it as ‘action-oriented’ because it is really so based upon a display of pursuits and activities, what with all its direct surprises and learning expectations from interactive sources and, in fact, be as it may, the opportunity to represent MeritTrac at the most fundamental level and that which is happening right at the client’s place is most opportune a moment for MeritTracers everywhere. For any individual who is looking out for some added bonuses in terms of intermingling with a lot of business customers, then this is it: the right channel for us to represent as MeritTrac professionals our assessment competencies and know-how and to a progressive extent, as assessment executives, our kitty full of comprehensive skills assessment portfolios as well.

I have had my sojourns assessing for different customers where I noticed people performing distinguished work solely on the basis of opportune ideas that get generated while at work. At the helm of affairs were fairly consummate opinions and thoughts in rapid exchange with each other. These ideas were those, which were related to the exercises we all were primarily put out for. The stake of it all was hinged upon our very basic sense of business acumen. There has been a superior sense of belonging to the tasks at hand, by all accounts, to the good ideas and their continual consequences: mainly those which were simply generated at the spur of the moment. These ideas and exchanges most often than not look very logically bare and almost always are unaccounted for merely because they happen every day on a daily basis. But to a generic observer of ideas and thoughts of all kinds, wherever they might exist, in whatever form or substance, they eventually do form a chiseled gem quietly embedded in the chain of a wholesome experience.

I end this critique with a bated breath and, therefore, basking in the glory of the English Evaluation sunshine I shall never allow myself to sail my humble boat to the last sunset! The legend of good English language speakers does exist, but the associative contraband myth that it doesn’t, if there was one, has gone outdoors forever!

Needless to say, Security has been an issue for any assessment process. The primary target, it seems, is always the question papers used. Test users using assessments in their recruitment process is concerned about the security and would want assessments to be unique and different from others to ensure a fair recruitment process. This results in a great demand for creating different forms for a Test. So, to ensure security, Test creators need to build different forms of the tests. For example: if a Test creator has created a test on Electronics, having components A, B and C, he/she is expected to create different tests using the same components. This would mean that the test takers are not aware of the content of the tests before they take them.

In MeritTrac, the demand for unique sets exists because the Clients need unique sets for different events so that strict confidentiality is maintained, and questions do not get leaked. This is where Parallel Forms plays a major role.

Definition - Parallel forms:

Two or more forms of a test are considered parallel when they have been developed to be as similar to one another as possible in terms of the test specifications and statistical criteria.

Importance of Equivalence in Parallel forms:

If the parallel forms are not equal in terms of content or statistical criteria (p-value and discrimination index), it would impact the scores and hence any decision based on these scores will be inaccurate. Hence, we should take proper care before we release parallel forms to ensure we are fair to the candidates taking the test.

Let me explain this:

Suppose, we have a test of 5 items (questions) and we have created 2 forms (sets/papers) using 5 items each, for each paper. If the items are designed to check Prepositions, then, all these items should check only for Prepositions and nothing else. That is, the test taker does need to apply knowledge of any other “parts of speech” to answer these questions on Prepositions. All he/she needs to know is Prepositions. Each item will have to be created carefully so that it retains its validity measure. Otherwise, it will affect the test takers.

Similarly, if the DL (Difficulty Level) for a given question is not set after statistically verifying it, but set at random, based on the Item creator’s assumptions, the performance of the test takers will vary. For example, a typical assumption while setting a DL could be :  Out of 100 test-takers who attempt this question, 40 would answer it correctly.  But in reality, it could actually be 60. If a Statistical process were stringently followed to derive the DL, it would involve exposing this item to a similar group of test takers. That is, expose this item only to College Freshers or Developers with 2 years of experience, but never to a mixed audience, which comprises of both. However, while conducting such sampling exercise, the candidate should not be marked/scored on this item. That is answering this item should not affect his scores in any way.  Considering our shortlist criteria to be fixed, we cannot have discrepancies in our items in terms of content and DLs, because it would impact the throughput and also the test takers.

Equivalence of test score means, variances, and errors are interpreted as evidence that tests are parallel

Considering the business demands, we might not be able to check the P and DI value before we use items in our tests. But we should create a process to monitor each item’s performance and tag the question accordingly in terms of statistical criteria. Creating a good Question Bank having all the necessary statistical criteria will enable us to create parallel forms. 

Key points to remember while building Parallel forms:

• When items are created for a particular content area, multiple checks have to be conducted to ensure that the items belong to the field desired. That is, while building content for Pointers in C, we should not have questions related to Arrays.
• P Values (Facility values) & Discrimination Index value (The index of discrimination is a useful measure of item quality whenever the purpose of a test is to produce a spread of scores, reflecting differences in test takers’ achievement, so that distinctions may be made among the performances of test takers) for each item should be computed.
• Once we decide on the Blueprint, we need to select items, which satisfy content and cognitive specifications (problem solving, application, knowledge) & statistical criteria (P value & DI value).