> but the plaintiff shows it was trained on data that allows for discrimination
That's all data; there would be no need to show anything.
There was a paper a while ago by a team of doctors who wanted to use classifiers on X-ray images and FREAKED OUT when they realized that the first thing the classifier did was categorize every image by the race of the patient. As they note, it will always do this regardless of whether the race of the patient is present in the input data. (Because obviously, the race of the patient is part of the information conveyed by the structure of their body, which is what an X-ray shows.)
That's all data; there would be no need to show anything.
There was a paper a while ago by a team of doctors who wanted to use classifiers on X-ray images and FREAKED OUT when they realized that the first thing the classifier did was categorize every image by the race of the patient. As they note, it will always do this regardless of whether the race of the patient is present in the input data. (Because obviously, the race of the patient is part of the information conveyed by the structure of their body, which is what an X-ray shows.)