Moneylife » Life » Public Interest » How UIDAI goofed up pilot test results to press forward with UID scheme
How UIDAI goofed up pilot test results to press forward with UID scheme
- 34 comments
- + COMMENT
| 18/03/2011 03:59 PM |
Follow @mldigital
According to test results of UIDAI’s biometrics-based Aadhaar project, there could be up to 15,000 false positives for every Indian resident. Moreover, this figure is just for identification and not for verification
The Indian government and its de-facto tagging institution, the Unique Identification Authority of India (UIDAI), have not only ignored privacy concerns but also ignored sample test results of its pilot project. Both the government and UIDAI have been in such a hurry that they have neglected the basic principle of pilot testing and size of sample. For over 1.2 billion UID numbers, they have used data from just 20,000 people, in pairs, as the sample and have on the basis of the results gone ahead with the UID number through the 'Aadhaar' project.
UIDAI conducted a proof of the concept trial of the Aadhaar project between March and June 2010. In the results, it said, "The matching analysis was done on two sets of 20,000 biometrics, for a total of 40,000. However, the number of comparisons was several orders of magnitude more than 40,000, since each set of fingerprints would be matched against every other set of fingerprints in the data set".
On the false positive identification rate (FPIR), the authority said, "We will look at the point where the FPIR (i.e. the possibility that a person is mistaken to be a different person) is 0.0025%". This means, for every 1 lakh comparisons, there would be two and a half false positives. On a large scale, it means for a population of over 120 crore, there would be 18 lakh crore false positives, or, for every single Indian resident there would be 15,000 false positives! (Click to see the calculations)
David Moss, who spent eight years campaigning against the UK's National ID (NID) card scheme, has questioned the logic of the UIDAI and the government to depending on biometrics to produce the UID number. In a report titled, "India's ID card scheme-drowning in a sea of false positives", Mr Moss said, "those (the FPIR) conclusions do not follow from the evidence reported. Nothing in UIDAI's surprisingly low quality report suggests that it would be feasible to prove that each electronic identity on the Central ID Repository (CIDR) is unique. Not with a billion plus people on the database. Far from it, India can be confident, from the figures quoted in UIDAI's proof of concept trial report, that de-duplication could never be achieved."
Speaking about the UK's NID scheme, Mr Moss said, "There were many problems with the UK scheme. Not just biometrics. But biometrics is the easiest problem to understand and to discuss objectively and on which to reach an agreed decision, as it's quantifiable, there are no difficult value judgements to make and it's just technology. But it's not a very good technology, for, whenever there is a large-scale field trial, mass consumer biometrics prove to be too unreliable for the ID card schemes that depend on them, as opposed to the mere computer modelling exercises favoured by the US National Institute of Standards and Technology (NIST)."
In addition, there are issues like the reliability of biometric identification for a large population like in India. For the record, no one has ever issued IDs to such a huge population anywhere in the world. And whoever has tried to issue biometrics-based Ids, even for a small size, had to abandon or discard the idea altogether. Like the UK government abolished its NID scheme citing higher costs, impracticality and ungovernable breaches of privacy as reasons for cancelling the NID project.
The UK government spent around £250 million on developing the national ID programme over eight years. However, its abolition means that the government will avoid spending another £800 million over a decade. The NID was launched in July 2002 and as of February 2010, its total costs rose to an estimated £4.5 billion.
For the biometrics-based ID cards, there was one study done at Seoul in Korea. The study was done for ID cards issued for driver licences. It was designed in such a way that by swiping fingers, the drivers were able to access services like paying parking charges and redeeming a ticket. However, after one year, it was found that 5% to 13% users could not use the system. The tests were conducted with four different manufacturers, with drivers being white collar workers and housewives in acceptable quality criteria. In the end the study recommended frequent re-enrolment of users.
According to JT D'Souza, who analysed the pilot study conducted by the UIDAI, given the well-known lacunae in our infrastructure and massive demographics, biometrics as an ID will be a guaranteed failure and result in denial of service. He said, "The sum of false acceptance rate and false rejection rate (EER) reveals only part of the problem, which is rejection or acceptance within a short duration of enrolment. The bigger problem is ageing, including health and environment factors, which causes sufficient change to make biometrics completely unusable and requires very frequent re-enrolment."
The International Biometric Group (IBG) testing also shows that performance can vary drastically within technologies-some fingerprint solutions, for example, had next to no errors during testing, while others rejected nearly 1/3rd of enrolled users. "Most interestingly, the testing shows that over time, many biometric systems are prone to incorrectly rejecting a substantial percentage of users. Verifying a user immediately after enrolment is not highly challenging to biometric systems. However, after six weeks, testing shows that some systems' error rates increase ten-fold," according to the research, consulting and integration firm, which works closely with the biometric industry. The report is titled "Real-World Performance Testing".
Despite all the issues, the UIDAI and the Indian government are pressing hard to implement the UID number scheme across the country. While maintaining that the UID number is not compulsory, both of them are making efforts to make it mandatory using backdoor methods. Nobody is even ready to pause and think about the possible consequences of the failure to identify some poor person from a remote place. It may be a technical glitch for the authorities, but could be a question of life and death for the 'aam admi', who would be denied food and other benefits due to the failure.
"By the time the stillborn (NID) scheme was finally cancelled, the UK's Home Office had lost all credibility, it was totally demoralised and it is now excluded from discussions of the new, and still unspecified, Digital Delivery Identity Assurance project. Having given their unsolicited testimonials to the biometrics industry and its unreliable products, UIDAI will be left to clean up the expensive mess left in India as best they can when 'Aadhaar' is cancelled, while the biometrics industry road-show moves on to the next country and repeats the trick," Mr Moss concluded.
You may want to read:
- UK scraps National ID project; Will India's UID face the same fate?
- UID: Enrolment troubles for helpless residents
- Fat profit institutions continue to board UID bandwagon
- Why is UID number being made compulsory through the backdoor
- UID = more 'consumers' admits Nilekani
- Now SEBI jumps on the UID bandwagon
- Right to privacy and biometrics of the UID
- How Aadhaar or the UID project can get you into deep trouble
- UID Issue: Numbers Game -1
- No card only a number despite Rs45000 crore being spent on the UID
- Even Mahatma Gandhi was against ID cards
More in Moneylife
VIDEOS
Keep your Money Safe: Avoid money traps and MLM
LATEST COMMENT
MORE
Govt approves restructuring of I-T dept; creates 20,751 posts
Info Commission directs Railways to share service record of a TC
|
|
|
- Phaneesh Murthy: Let off by Infosys, sacked by iGate over sexual harassment charges
- Phaneesh Murthy saga: Why insurers should refuse to cover serial offenders of sexual harassment
- Sensex Rally: Winners and Losers as the index challenges the high of 2010
- Vinod Rai demits office: A CAG that India will miss
- Stock Guru scam: ED to attach properties of accused couple
- Additional Home Secretary on the edge to complete probe into tampering of 26/11 call log records
- Sahara pulls out from IPL after failing to pay franchisee fee
- IPL spot fixing: Vindoo Dara Singh arrested by Mumbai police
- Government reduces import tariff value of gold to $440/10g
- The draconian LBT: Local Body Tax explained
- How much longer can the FM, RBI ignore HSBC in India?
- Aadhaar: Private ownership of UID data- Part I
- Aadhaar: Who owns the UID database? –Part II
- Did HSBC Bank resort to toxic churning and illegitimate transactions to earn commissions?
- PNB Metlife refunds Rs25,000 to the correct policyholder: another Moneylife victory
- Cobrapost exposes money-laundering racket in 23 entities including SBI and LIC
- Do we need a regulator for ‘unclaimed’ deposits?
- Confusion between yield and rate of interest reflects financial illiteracy among business journalists
- The draconian LBT: Local Body Tax explained
- Mass mis-selling: 59,000 investors in Kolhapur are alleged to have lost money in LIC ULIPs
- Vinod Rai demits office: A CAG that India will miss
- Sensex Rally: Winners and Losers as the index challenges the high of 2010
- Phaneesh Murthy saga: Why insurers should refuse to cover serial offenders of sexual harassment
- Additional Home Secretary on the edge to complete probe into tampering of 26/11 call log records
- Sunlight: The ‘be all and end all’ of human health
- Phaneesh Murthy: Let off by Infosys, sacked by iGate over sexual harassment charges
- RTI exposes a revenue loss of Rs25,000 crore in Maharashtra
What's your say?
What you said
Thanks for casting your votes! View Previous Polls
Join 22, 000 Others
Membership Benefits
- Daily & Weekly newsletters
- Access to www.moneylife.in to comment, create alerts
- Your own profile in Moneylife.in
- All special mailers
- Basic membership to MSSN, our new initiative
- Free ebooks
- Invitation to events
- Invitation to round-table meets
- Access to Insurance helpline
- Access to counselling sessions
- Access to Reading room in Mumbai
Foundation Events
Comment
LOKESH SHARMA 1 year ago
LOKESH
David Moss 2 years ago
The first version of the sea-of-false-positives review of UIDAI's proof of concept trial report was posted on a website called Planet Biometrics on 16 February 2011.
37 days later, a response was received from one Murali Chirala, who may or may not be associated with UIDAI.
Please see http://www.planetbiometrics.com/article-...
Murali Chirala's is a substantial response. I have today tried twice to post my initial answer on Planet Biometrics but there seems to be a problem with the website.
I have been recording events on the No2ID website, please see http://forum.no2id.net/viewtopic.php?f=5...
... and also here on moneylife.in. Just to keep things up to date, here it is, the first of many responses to Murali Chirala:
1 According to Murali Chirala, “Mr. David Moss makes two fundamental errors in ...” and “Mr. Moss confuses FPIR ... with FMR ...” and “Mr. Moss has misinterpreted FPIR and FNIR for FMR and FNMR ...” and there is “... a massive discrepancy in Mr. Moss’ calculations” and “Mr. Moss incorrectly assumes that ...”
2 He’s pretty stupid, this man Moss.
3 We can safely ignore him, he’s irrelevant.
4 Even so, the question remains, for an intelligent third party, are all the records on the Central ID Repository (CIDR) being built by UIDAI biometrically unique? Are they? Or aren’t they? Yes? Or no?
5 Let’s call this intelligent third party “Churali Mirala”, or “CM” for short. CM wants to understand exactly what Murali Chirala is saying, because this question of uniqueness is important. After all, if the Unique Identification Authority of India isn’t offering unique identification, what is it offering?
6 CM takes a good look at Murali Chirala’s second paragraph, where there is talk of “two different galleries”, each with 20,000 records.
7 A gallery here, CM assumes, is a mini CIDR, a set or database of biometric records.
8 What is in the records, CM asks? In UIDAI’s proof of concept trial, each record identified one Indian using 12 items, viz. ten fingerprints and two irisprints. Since that is what we are talking about, UIDAI’s proof of concept trial report, CM assumes that by a “record”, Murali Chirala means 12 fields/columns containing ten fingerprints and two irisprints.
9 Murali Chirala talks of “40,000 searches” on each gallery. Why 40,000? Murali Chirala has a rather elliptical way of writing. We have to “unpack” his words, so to speak. Where did 40,000 come from? CM makes a guess. He assumes that the ten fingerprints are treated as one search term/probe and the two irisprints are treated as a second probe. (CM has been doing some background reading and understands that when you do a matching exercise in the academic field of biometrics you “probe” the “gallery”, which is also sometimes called the “background”.) Good. So that makes two searches for each Indian in the gallery. 40,000 searches in all. CM would like Murali Chirala to confirm that this guess is correct. He doesn’t want to be thought to be as stupid as David Moss. But if the guess isn’t correct, then CM can’t understand what the elliptical Murali Chirala is talking about with his 40,000 searches.
10 And what are these searches, CM asks? There’s not much point searching a gallery against itself, CM thinks, because then the false negative identification rate would have to be zero. And it isn’t. Then CM remembers that the participants in the proof of concept trial had to make two visits to be registered. CM assumes that the gallery was built from the biometrics registered at the first visit and that it is the biometrics registered at the second visit which are being used as the probes. Again, CM would like Murali Chirala to confirm that this guess is correct. He still doesn’t want to be thought to be as stupid as David Moss. But if the guess isn’t correct, then CM still can’t understand what the elliptical Murali Chirala is talking about with his 40,000 searches.
11 Obviously Murali Chirala’s answers would be wasted on David Moss. No question. But Murali Chirala still nevertheless owes answers to any intelligent third party who is interested to know whether records on the CIDR are unique.
12 More questions for Murali Chirala. 60,000 participants attended the second registration session. And yet the gallery comprises only 20,000 Indians. Why, CM wants to know, why doesn’t the gallery have 60,000 people in it? How were the 20,000 chosen? What was wrong with the other 40,000?
13 Also, Murali Chirala, please confirm CM’s assumption that the 20,000 probes chosen from the second registration session used to match against the gallery created from the first registration session were chosen because UIDAI thought they were the same 20,000 people, they’re not 20,000 probes chosen at random, say, from the 60,000 available.
14 We’re still in the second paragraph of Murali Chirala’s post. And there’s another question which requires his attention. He really is the most elliptical of writers.
15 There are two galleries. One of them is “seeded” and the other one isn’t. What is the difference? It seems from what Murali Chirala says that the seeded gallery is required to calculate the false negative identification rate, you can’t do that with an unseeded gallery. CM makes a guess. If Murali Chirala, say, is not matched and the operation returns a negative, the only way UIDAI can know that that is a false negative is if they know in advance that Murali Chirala is in the gallery. So seeding the gallery must involve checking it first, before matching operations begin, to see who’s in it. Perhaps Murali Chirala could confirm.
16 There is an equivalent problem with false positives. Suppose UIDAI probe the unseeded gallery with Murali Chirala’s biometrics and get three matches. Perhaps Murali Chirala is registered on the CIDR three times. They could be legitimate matches. How do UIDAI know that at least two of the positives are false? Only if they already know that all the records in the gallery are unique.
17 But hang on a minute. Isn’t that where we started? That’s exactly what we need to know. Is each record on the CIDR unique? The answer from Murali Chirala so far is that, yes, each record is unique if each record is unique. It seems that whatever Murali Chirala is talking about, the correct way to measure false positive and negative identification rates, it doesn’t help us to establish whether these blessed records are or are not unique.
18 We need answers to these questions raised by Murali Chirala’s second paragraph before we can move on to his third paragraph, which we look forward to doing, soon.
19 Your comments would be appreciated Murali Chirala. It took you 37 days, from 16 February to 25 March to respond last time. That’s fine as long as we’re only dealing with the halfwit David Moss. But an intelligent third party, perhaps an Indian taxpayer wondering if his or her tax money is being wasted, will look forward to an answer from you, Murali Chirala, much more quickly than that, please.
interested observer 2 years ago in reply to David Moss
dear mr moss,
i was trying to decipher mr muralis explanation and as far as i understand what he finally says is that the number of false positives and false negatives on attempting to confirm the uniqueness of each id being issued can be kept within manageable limits by lowering the tolerances with which they are compared. As the sample size goes up, u look at each id less carefully so that u don't get too many false positive matches from the ids already in the database!!! I hope i am just misunderstanding him.Please tell me if i am missing something here.I can just imagine the chaos when 'mr.common man from the poor and marginalised parts of society' turns up with his uid card and the machine tells him he is not himself. And all these are problems inherent in the system, not to think of the deliberate misdeeds possible..
David Moss 2 years ago in reply to David Moss
20 Let’s say that CM, the intelligent third party, while waiting for Murali Chirala’s response, reads ahead and finds this: “Mr. Moss incorrectly assumes that the decision threshold on the 1-to-1 comparison score is kept constant independently of the gallery size. In fact this threshold is adjusted in the real operational system to keep FPIR constant independently of the gallery size G. This fact alone removes the possibility of the sea of false positives”.
21 Is Murali Chirala right when he says that the possibility of a sea of false positives has been removed?
22 Probably.
23 After all, this sea of false positives argument is devised by David Moss, already acknowledged as the biometrics village idiot.
24 Or is it? CM checks and, lo, he finds this*, talking about the UK ID card scheme: “academic John Daugman, a former member of the Biometrics Assurance Group (BAG) which reviewed the scheme, says its reliance on fingerprints and facial photos to verify a person's identity will cause the system to collapse under the weight of mismatched identifications”. And this: “Daugman, an expert on iris recognition, says fingerprints and facial photos are not distinctive enough to be able to tell the UK's 45-million-strong adult population apart”. And this: “Daugman said that even if the error rate was as low as one in a million, the 10 to the power of 15 comparisons needed to verify the IDs of 45 million people would result in one billion false matches”. And this: “He told silicon.com: ‘The use of fingerprints will cause deduplication to drown in false matches’ ...”.
25 And who, pray, is John Daugman?
26 Only the king of biometrics based on the iris, the inventor and patentholder of the iris-matching algorithm and a professor at the Cambridge Computer Lab, that’s who**. Search for “john daugman” “biometrics” and Google gets you 7,930 hits.
27 Professor Daugman is not going to be stupid. Not like David Moss. Clearly Murali Chirala disagrees with him. But then, if you search for “Murali Chirala” “biometrics”, Google gets you one hit.
28 It’s hardly a slam dunk argument, you can’t resolve biometrics problems by Google hits, but CM thinks he’s got to make up his own mind about the sea of false positives and not just take Murali Chirala’s word for it that there is not even possibly a drowning problem. Once again, he hopes that Murali Chirala will soon answer.
----------
* http://www.silicon.com/management/publi ... -39294213/
** http://www.cl.cam.ac.uk/~jgd1000/
David Moss 2 years ago in reply to David Moss
The link to the silicon.com article at para.24 above about Professor Daugman wasn't parsed properly by the moneylife.in editor. Let's try this alternative and this one instead. It's an important article and well worth reading. And it's only short.
David Moss 2 years ago in reply to David Moss
29 Considering paragraphs 20 to 28 above, CM, the intelligent Indian taxpayer investigating whether UIDAI is wasting public money, wonders why, if there is a sea of false positives, no explorer has discovered it and charted it.
30 One potential answer, considering paragraphs 1 to 19 above, is that no explorer has looked for it. If the sea of false positives is there, it would be revealed the minute anyone tried to prove that all records on the CIDR are biometrically unique. But they don’t try to prove that. Instead, if Murali Chirala is right, they try to calculate the false positive and negative identification rates. And that is a different job.
31 You’d think that CM would now test the truth of this hypothesis.
32 But no, you would be wrong.
33 CM’s eye has been caught by something else. Something else explosive. What looks like a fundamental error in Murali Chirala’s argument. An error that would bring down the whole house of cards. At least it would if UIDAI have made the same error. A hole at the heart of UIDAI’s case. A problem so glaring that it has even occurred to the troglodytic David Moss (in his own dim and benighted way, at least).
34 Every other reader of paragraph 20 above will also have spotted it. It’s a hand grenade with the pin removed. It’s a time-bomb, ticking down fast to zero.
35 Let’s take this slowly, just to check our logic as we go.
36 We’re going to look at matching, comparing someone’s biometrics against the template stored on the CIDR, we’re going to look at the “decision threshold” and the “comparison score” mentioned by Murali Chirala. About time too! That’s what biometrics is all about -- comparing/matching and verifying that Murali Chirala is Murali Chirala. How come it’s taken until paragraph 36 to get onto the subject? Never mind, here we are at last.
37 What we will find is that Murali Chirala is nearly right when he says “... this fact alone removes the possibility of the sea of false positives”. To be quite correct, Murali Chirala should have said “... this fact alone removes the possibility of FINDING the sea of false positives”.
38 We will find that UIDAI and NIST (see below) and others are using a peculiar concept of identity nothing like what normal people understand by the word and muddying the waters by confusing the two usages.
39 And, finally, we will discover that when it comes to biometrics NIST (see below) are arguably a busted flush and so is any organisation like UIDAI that follows NIST’s prescriptions.
40 Heady stuff. Hold on to your hat.
41 That’s what we will find and we’re going to start with NIST, a US organisation, the National Institute of Standards and Technology, and their May 2004 report ‘Matching Performance for the US-VISIT IDENT System Using Flat Fingerprints’, a report some of us have read at least a dozen times, please see http://dematerialisedid.com/PDFs/ir_7110...
42 Murali Chirala has great confidence in NIST. He says: “... while the [UIDAI proof of concept trial] does not provide sufficient number of identification searches to estimate [FPIR] and FNIR in a gallery of size of 1.2B directly, and extrapolating too far out has its own dangers in terms of statistical significance, methodology is available to extrapolate the [UIDAI proof of concept trial] results using trends established earlier in other large well established biometric performance evaluations performed by NIST and other agencies”.
43 Lots of other people share that confidence, including UIDAI, who were advised by NIST in the early days, please see UIDAI’s paper ‘Biometrics Design Standards For UID Applications’ at http://uidai.gov.in/UID_PDF/Committees/B...
44 That confidence is misplaced.
Ram Das 2 years ago
Are you referring to response at http://www.planetbiometrics.com/article-... ?
How did you conclude that the response is from UIDAI?
David Moss 2 years ago in reply to Ram Das
Good question.
The first version of my sea-of-false-positives argument was posted on the Planet Biometrics website, a sort of biometrics fanzine, please see http://www.planetbiometrics.com/article-...
That was on 16 February, about six weeks ago.
I was told a week later, 23 February, to expect a response from UIDAI.
A month later, on 25 March, I was told that the response I was expecting had now been posted on Planet Biometrics.
I agree that there is nothing to associate Murali Chirala with UIDAI anywhere on the web.
All I am going on is what I was told, by someone who ought to know.
David Moss 2 years ago
A response has now been received from UIDAI, posted on the PlanetBiometrics.com website.
There is homework to be done, getting to grips with this substantial response of UIDAI's.
More, next week ...
David Moss 2 years ago
Vivek Gupta 5 hours agoThis is what i call discussion without facts or rather distortion of facts. In any matching algorithm there are multiple inputs applied and not just one. Thus likelihood of falsepositive rate can be significantly eliminated by taking a segmented algorithm approach. I have personally worked on many such algorithms for credit bureaus so understand them well. This false positive rate has no relevance without understanding the full algorithm.
Moneylife is only publishing negative comments and distorting facts. Why? Are you afraid that corruption can be stopped using UIDAI and some people are propping you?
“In any matching algorithm there are multiple inputs applied and not just one”, says VivekGupta, very confidently, based on his experience of credit rating bureaux.
UIDAI is not a credit rating bureau, and they say “biometrics features are selected to be the primary mechanism for ensuring uniqueness” and their mission is “to design biometrics system that enables India to achieve uniqueness in the national registry” and “while certain demographical information is also provided, UIDAI provides no assurance of its accuracy ... demographic information shall not be used for filtering during the de-duplication process”.
There we have a nice example of the problem of induction. VivekGupta assumed that all organisations apply multiple inputs in their matching algorithms, his hypothesis was tested, and UIDAI prove it to be wrong. UIDAI are relying on biometrics only.
There is nothing wrong with moneylife.in publishing VivekGupta’s comments. He’s wrong. But that is not a reason to suppress his comments or to assume that some corrupt person is paying him to distort the facts. Rather, moneylife.in should be praised for hosting an open debate.
Which takes us back to JOABNarayan’s comment three days ago, “I wonder whats the reaction of UID authorities to David Moss' point?” -- UIDAI are strangely silent, aren’t they?
==========
Test area only
Italic
Bold
Link
Test area only
==========