As widely acknowledged, in order for a certain impairment to be recognized as a disability, it must constrain a person in a way that substantially limits the activity, especially in relation to employment or education. In other words, a person with same impairment could be disabled or non-disabled, depending on time and space where he or she lives in. For example, substantial leg impairment could be fatal in a pre-modern society, but it is now much less critical in modern society where a variety of transportation tools are available. Also under the strong emphasis of the social value on facial appearance, some big permanent scars on face could lead to disability. Disability is the combined product of impairment and diverse social environments, including technology, social norms, and cultural values. Because of this social process of disability, we need to consider and even incorporate social norms and values of the current society when we evaluate disability.

Furthermore, social environment where impairment is embedded is not a single constant entity. Social values are constantly changing and also vary across societies or even groups within one society. Thus, social surroundings of impairment must be recognized as an entity with fractured and diverse aspects rather than one simple universe. Considering this social characteristic of disability, the best strategy would be to incorporate diverse and even conflicting values and pressures from the beginning of the evaluation rather than treat them as a separate entity from the evaluation. The KAMS guide tried to adopt a series of strategies based on this viewpoint. Before we introduce those strategies, however, let me continue the debate on the social process of the evaluation procedure itself.

Disability evaluation in practice consists of a series of evaluations and re-evaluations. First, it should start from a small number of specialists. KAMS started with about ten people in each specialty. Let's call this group of medical specialist as core experts. But in many cases, this first step alone is not enough for successful evaluation procedure for two reasons. For any specialty field, there usually exist some vague grey areas that call for judgment call. Thus, the best approach might be to bring those unresolved issues to a bigger number of specialists: expanded experts. Also, expanded experts are necessary to obtain stronger legitimacy. Even if core experts could produce objective and scientific criteria to the full extent, as we discussed above, criteria themselves are social product, and thus, they need to be shared as many social members as possible to gain strong legitimacy and widely adopted by many experts later.

Second step would be, as we discussed above, to form a team of expanded experts. Depending on the magnitude of unresolved grey areas among core experts, the size of expanded experts should be determined. However, more critical issue here is how to recruit the members of expanded experts. The goal of forming core experts is not to obtain a representative sample of experts, but to recruit top experts to build up evaluation criterion. However, the reason we need expanded experts is different. We need a representative sample of experts as much as possible, because unresolved issues among core experts are not of technique but of social value or customs in many cases. Especially, considering legitimacy issue that could be critical once the evaluation criteria announced, the representativeness of the expanded experts is essential. Usually, there is no simple universal way of obtaining representative sample of medical experts and we will discuss this more later. The last step could be to probe and incorporate public opinions in various active ways. Since this last step usually is not considered as a part of evaluation process, we will discuss this later in detail.

The important issue regarding core experts is not the recruitment. Usually, it could be done by recommendation of top experts or appointment by an official national medical association. Once recruited, however, the consensus is the key to the successful evaluation procedure. Because of the nature of the evaluation, if possible, consensus instead of majority rule would be much more desirable.

Reaching a consensus poses two difficult challenges. First, it usually requires a moderator. However, it is hard to find a perfect moderator who can balance between every different or even conflicting views. Especially, expert committees like this also need an expert of the same field as a moderator, which makes him or her uneasy to act in a neutral way. Second, reaching a consensus needs more time and efforts than majority voting. Although consensus usually gives more satisfaction among participants, we sometimes have to rely on voting due to the lack of time.

If time permits, small number of experts like ten or so could reach a consensus. But, we do not usually have enough time for consensus regarding every issue. Here we suggest a way as an example to reach a consensus with the minimum activity of moderator in a relatively short time. It is called Multiperson Decision Making Problem (MDMP) model (1-5). This consists of the following six steps. 1) Opinion Expression: each expert expresses his or her own preference on the issue. Based on each opinion, the magnitude of consensus is measured. There exist several alternatives for this measure including variance that was used in Economics (5-8). Whatever measure we use, if it exceeds some pre-determined criterion, we can claim that we reach a consensus and all the following procedures are unnecessary. 2) Proximity measure: If the amount of consensus is not satisfactory, we measure proximity of each opinion from the whole opinion. This in principle measures how far each opinion is away from the whole committee. 3) Feedback: based on proximity measure, people with proximity score that are furthest from the collective score are recommended to change their opinion so that their opinions become closer to the collective one. 4) Changing the opinions: based on the feedback, some people change their opinion. 5) Now, new consensus score is calculated, and it is compared to the criterion. 6) If it exceeds the criterion, satisfactory level of consensus is reached and no more action is needed. If not, we will repeat from step 2) again until we will have satisfactory amount of consensus.

As we discussed above, the critical issue here is to obtain a representative sample of experts in specific medical field to resolve legitimacy issue. The most widely used method was a snow-ball sampling based on core experts as seeds. We ask core experts to enumerate other experts in the same field. Once we contact those additional experts, we ask the same favor: to enumerate other experts. We repeat this snowballing until we hit the desired number of expanded experts. Although this recruitment strategy is convenient and natural way to collect a small number of target people, it usually ends up with strongly biased sample. For example, if we start with male experts, the resultant sample would be male-biased.

Another popular strategy is to use a membership directory. Usually, medical experts must be registered and Korea is not an exception. If we can contact experts in a random way from the directory and do a survey with them, it would be a perfect representative sample. However, medical experts, especially in Korea, are busiest men due to tight schedule of caring patients. Thus, the response rate for survey is usually extremely low, and as result, the resultant final sample is also strongly biased.

One possible alternative which we plan to implement is to adopt a respondent driven sampling (RDS) method that adjusts biasedness introduced during snowball sampling (9-11). This method is handy in the sense that it is really similar to traditional snowball sampling except we collect some network information of respondents so that we can adjust biasedness of each respondent later. This method is widely adopted since its debut in the 1990s, especially among the studies that targeted for hidden small population such as jazz singers, drug users, men who have sex with men (MSM), etc.

We conducted a new survey of 400 experts (doctors) on their opinion on physical impairment rate for 16 different impairments via faxes and obtained 195 completed cases (48.8% response rate) form September 2008 to January 2009. The Table 1 summarized the result. Although in general experts showed very consistent opinions, some impairments such as 'gastrointestinal fistulas' and 'complex regional pain syndrome' exposed substantial variance (or standard errors) among expert opinions. For example, response rate for 'complex regional pain syndrome' ranges from 0% to 70% and also inter-quartile range is 30%, which is relatively large. This finding strongly suggests the necessity of developing strategies for successful consensus making among experts since simple mean could be ineffective and even illegitimate when there exist wide variations of opinions.

We need to examine opinion of the public beyond expert group for two reasons. First, we need to weigh impairments across field. For example, we need to decide relative importance of eyes vs. legs. In this case, it is not recommendable to ask opinions of experts only. We need to examine public opinion when our decision requires more than one medical special field. Second, we also need to examine public opinion to gain stronger legitimacy. Even if the evaluation criteria and procedure are scientific and objective, the legitimacy of the criteria could be weak unless they are accepted by the public. Especially, considering sharp conflicts between interest groups with regard to disability evaluation, it could be critical to secure public legitimacy.

There could be two alternatives to probe public opinion. The first one is to conduct a survey of a representative sample of the public. The second alternative is to combine MDEMP model with a traditional survey. The first approach is most widely used. Also if we obtain a majority preference from this survey, it is the best strategy available. But what if we confirm the existence of sharp division, for example 52% vs. 48%? It would be hard to draw conclusions from this type of neck to neck competing and sometimes even conflicting opinions. This is actually what we found from a public opinion survey. This survey is conducted in February 2009 on a 840 representative sample of Korean adults through mobile phone text messaging. A survey research company named MBIZONE keeps the list of more than one million people who registered and agreed to be respondents for the surveys conducted by the company in the exchange of small amount of money. MBIZONE selected 840 representative adults out of registered one million people for this study. Table 2 summarized the result. The table summarized a series of questions to compare two impairments. For example, the first question reads, "Who is more disabled in Korea, 'person who lost both arms' or 'person who lost both legs'?" We coded 1 if a respondent chose the first one and coded 3 if he or she chose the latter one. We coded 2 if a respondent said, 'almost identical'.

We obtained two important implications from Table 2. First, we can find some impairments had the mean value around 2. The closest one is 1.94 for the comparison between 'amputation of one arm' and 'multi-segmental fusion of the lumbar spine'. Public survey produced little information about the relative importance of this type of impairments. Second, public opinion does not always consistent with experts' view. For example, Table 1 clearly showed that Korean experts believed amputation of arms was more critical than amputation of legs (95% confidence intervals are '73.89-76.73' vs. '63.71-65.95'). Additional table (not shown) revealed that only 4.5% of experts believed that amputation of legs was more critical while 70% believed that amputation of arms was more critical. However the order is reversed in public opinion. In Table 2, the mean value of 'amputation of arms' vs. 'amputation of legs' was 2.15, which means that public believed that amputation of legs was more critical. According to additional analysis (not shown), the proportion of people who believed that amputation of legs was more critical was 44% while 30% of the respondent believed that amputation of arms was more serious. Both findings robustly implicated the need of developing efficient and legitimate procedure of consensus formation.

The second alternative to investigate public opinion is to integrate MDMP model with public survey. Because MDMP model includes feedback and changing opinions, it is not possible to apply this to huge number of people in person. Instead, it is possible to apply this to about one thousand people via mobile phone. Once we select a representative sample from MBIZONE, we can give feedbacks and obtain new changing opinions through its system. Considering the fact that mobile phone survey only takes about one day for about 1,000 people, this is a plausible strategy in the future.