The Association submitted feedback to the National Institute of Standards and Technology (NIST) in the USA on their plans to offer testing for facial age estimation.
We welcomed the decision to provide test protocols for Age Estimation Technologies, and made the following suggestions to improve the approach:
- The current proposal will yield raw false positive/false negative percentages, but the mean average error alone does not give any indication of the distribution of the error. Letting a few 17 year olds pass as 18 is less concerning than the same number of 8 year olds. Regulators are, in our experience, keen to understand how far wrong a solution may be when it is wrong; not just whether it is wrong per se.
- It appears to us impossible to find an “estimated age which, if used as a threshold, would give a zero-error rate on determining whether the subject is below all age of interest.” A zero-error rate is theoretically impossible. At a given age, the error rate will tend towards zero – that may be a better approach.
- It would be more relevant to test at the age of 13 not 12 (you currently plan to consider 12, 18, 21, and 70) and to add 16 give the US Senate is considering extending COPPA from 13 to 16.
- Their testing of a range of demographics should report any statistically significant variance by skin-tone (Fitzpatrick dermatological scale) and gender – solutions which discriminate significantly against protected characteristics should not be given a seal of approval by NIST, so an acceptable variance must be defined in context of likely use-cases (again, logically this will never be zero).
- Any statistical difference in facial expressions – particularly smiling or not smiling – should be reported. (See https://www.nature.com/articles/s41598-022-27009-w)
- There is no obvious commercial value in comparing the results from two images of the same person.
- It would be very helpful for NIST to share some of the 6 million images which you have for testing could be made available as training data. Access to training data is a huge barrier to entry in this field.
- It would be helpful to publish the test protocols in full – as we expect you will – so these tests can be replicated by ILAC approved auditors in other jurisdictions
- Please be careful always to refer to facial estimation or facial analysis, not to confuse this with “recognition” or “verification”. There data required to estimate age should be insufficient to allow for any form of individual recognition.
- Many providers train an estimate to the nearest month and year, not just the age in whole years. Also, some algorithms are designed to output only pass or fail for an age threshold. So results need to be carefully described e.g. “when applying a minimum age of 17 by testing that faces were estimated to appear 19 or older, the error rate was…
- We recommend work done on measuring and standardizing levels of age assurance by the Age Check Certification Scheme, to which our members contributed, for the UK Information Commissioner’s Office and Office for Communications. The first report can be found at Measurement of Age Assurance Technologies (ico.org.uk) and our second report is due to be published shortly which will propose particular tolerance thresholds e.g. 90%+, 99%+, 99.9%+ and 99.99%+ for Zero, Basic, Standard, Enhanced and Strict levels of age assurance
- We also recommend you consider, if you have no yet done so, the output of an EU funded project in 2021-22 euCONSENT – Trust Services for Children in Europe which developed interoperable age assurance, and parental consent mechanisms. As this requires a trust framework across all providers which join the network, two certification schemes were developed: https://euconsent.eu/download/certification-requirements-for-age-verification/ and https://euconsent.eu/download/certification-requirements-for-parental-consent/.