Horrible misuse of AI. Should be spying on Uyghurs instead.
92% success rate. Cf 89% for CT scans.
1 Like
This is a very small study at 176 CT images. You would be brave to conclude anything off this small a sample size.
They conspicuously don’t mention how the detection fared in the control group, specifically if there were any false positives.
3 Likes
The thing about medical AI studies is that there are loads of normal image data, but just a handful of positive target data for training. How to made those few images count is the key and there are many, many methodologies to do so.
Having said that, 92 seems low. Also, success rate is a vague term. Do they mean total accuracy? Or precision (specificity)? Or recall (sensitivity)?
One way that medical professionals in the AI business seem to do all the time is divide up all images into 100 or more smaller images, and have at least 3 doctors label each one. All medical image have extremely high resolution, so doing so doesn’t degrade the quality of the images and at the same time makes the images more manageable for the GPU servers.
However, having 3 or more domain experts labeling thousands if not millions of images day in and day out is a luxury that most other businesses cannot afford. So while the medical industry feels they are doing things the more reliable way, having more resources to go around also limits their motivation to explore semi-supervised or self-supervised training methods.
3 Likes
10 years of images would be a start, so really roughly at an incidence rate of 6 per 100,000, 12000 images. I don’t know what the real incidence rate is I just picked the first one I could find. This is the sort of number any serious study would require.
Yes, AI training regimes for other diseases i have seen have numbered in the many 1000s of images, both negative and positive. 176 is very small. CT is not the best way of imaging pancreatic cancer (although it is external and can be done without surgery).
ultrasound images are too fuzzy, but MRI can be better than CT. However, MRI is way more expensive than CT.
1 Like
The problem also often comes down to the domain experts. Like you’ve mentioned, CT images might not be the best way to capture pancreatic cancer, and as a result it is even difficult for the doctors labeling the images to always get it right.
They would often resort to majority rule to determine the final label, but that is not concrete assurance for being the right answer. Having a bunch of conflicting labels in your supervised training will definitely keep your final accuracy down. If your training data is only 95% accurate, your model can do no better than that.
There are many ways to try to get rid of questionable training data, but at the end of the day, unless you are willing to let go of uncertain data, there’s no way to really get around this problem.
I think with screening pancreatic cancer the the pancreas is actually hidden behind other organs. So the problem they have is not so much imaging and classification, but a more inherent one in that the getting at the organ is difficult. AI isn’t going to solve physical constraints of the CT, or even perhaps the MRI.
I’m just looking back at the data the rise in this cancer in this country is truly frightening. A large part must be due to an increase of capturing and recording it accurately as pancreatic cancer I suppose.
How do you conclude 176 is small?
In cognitive science when they do brain scans they usually test 20 people.
It’s very small for a clinical trial anyway. For biomarker validation you’d be looking at thousands to tens of thousands of samples to verify the association with a disease.
Biomarker validation
1st phase hundreds of samples do identify putative biomarkers identificstions from millions of molecules to associate with a given disease
2bd phase tens to hundreds of samples to check your now putative small biomarker panel robustness
3rd phase large scale trial with thousands of samples from patients to nail down accuracy and association with disease in the population and become an accepted clinical test.
4th phase continuous monitoring of clinical test (true positives, false positives ， false negatives ) to check it’s effectiveness across population
As far I can see they are just at stage one or at most startiing phase two with that study. The issue they have with pancreatic cancer detection is indeed probably that the image quality is poor and pancreatic cancer isn’t one kind of cancer either. Still the current diagnosis rates are shockingly low so may still be an improvement .
1 Like
How do they actually follow up if a high risk of stroke is detected? Curious what the medical intervention is in this scenario.
Since every public university in Taiwan begins with an “N” and ends with a “U,” this moderator edit decreases the amount of information conveyed in the title.
At least Taiwan University would be better.