why is a two-tier system so bad?
So far as the big (43 system) paper is concerned Freestyle then Accuchek come out significantly more accurate than the rest - something like +/-10% for Accuchek, a bit better for Freestyle rather than +/-20% or, in some cases, smaller accuracy problems combined with a significant (approach 10% or more) systematic error.
One of the problems is that they only did 200 individual tests with each system, and that makes it difficult to be certain of the results. I think we can say with 95% certaininty that 19 of the systems pass the European standard, with 3 or fewer failures, and 6 of them failed, with 17 or more failures.
The remaining 8 systems are running close to the borderline. It looks like they are having problems maintaining standards to meet the quality requirements, or maybe they are maintaining accuracy at a level where they only just meet the CE requirements.
We cannot make certain statements about which systems would fail the USA standards; the authors of the paper could, because they have the original data, but not enough of it is in the paper to do the required math. We can *guess* that three of the systems would fail the US standard, because the number of failed samples above 75mg/dl is more than 5% of the total (more than 10).
There is a clear correlation with manufacturer; a manufacturer producing one system which passes with flying colors doesn't produce another than fails. I don't think this should be surprising. It also seems to be the case that, with the exception of TaiDoc, manufacturers who produce failing systems produce only one or two systems. Five of TaiDoc's systems were tested, the paper reports three as failing, however one of those is above my 95% confidence limit (it only had 13 failures).
There are five separate companies producing strips that failed the test performed. Two are Taiwanese, two are German and one is Austrian.
The authors of the paper state that "the products were from different countries". I take this to mean that the products were *obtained* in different countries, since country of manufacture was not stated and I know that some of the strips from US manufacturers were not obtained in the US (because I know that the strips sold in the US don't have the CE mark.)
Country of purchase is relevant because some countries may not give any standing to the CE mark.
John Bowler
John
Thanks for that summary, nice work and well said.
This at the very least a complex issue and I appreciate the effort you put into reading and helping sort out the complexities.
I share your concern over the sampling size of 200. Large samples with lot variations would help identify those with tighter manufacturing quality control. I would also like to see source of strips sampled. Some may be more tolerant of the environmental variances of the distribution process than others but there is no way to know.
I would love to know. I would love us all to know.
Your time and effort in bringing some light to issue is greatly appreciated.
All the best.
Bennet
I have been off the grid the last week at CWD's Friends for Life conference. Sorry I am late to respond to some comments here. Much of this was written standing in lines at Disney in my iPhone.
Some have asked for information on specific meters. I recommend jbowler's comments, written on Friday, above as a jumping off point. More can be found in citations linked here:
http://www.stripsafely.com/?page_id=75 - Specifically the Freckmann papers.
As I learn more about copyrights (one of the areas of law specifically mentioned in the Constitution) I am becoming increasingly careful about how I share material that is the intellectual property of others. Some have been critical of papers behind pay firewalls. I believe that valid information is not always free to copy. I am privileges to have some access to goo libraries. I have no doubt that lacking access to other material diminishes my perspective. I have tried to respect the rights of others in the citations used by limiting my fair use to small, cited quotes used in an educational context.
This issue is complex and certainly there are pitfalls in trying to communicate it in common language. That I use folksy, snarky, double entendre is not a mistake but a calculated means to try to engage people with diabetes to exercise their first amendment right to address the government for a redress of grievance. I am not a reporter. I am an advocate for my kids and others with diabetes.
Frankly I would prefer a independent third party doing post market evaluation but none exists. Nor does, as far as I can tell, does authority for the FDA to require post market evaluation of meter accuracy.
My concern is heightened in part as medical companies are legally prohibited from superiority claims in marketing. Brand x can't say we are better than brand y. This leave us with fairly slow academic process. See the citations linked above.
Thank all for you concerns. I apologize for an unfortunate lack of clarity. I have attempted be non partisan in my effort to support safe, accurate measure of blood glucose by the 4 million Americans who use insulin. That said, it does bother me that US manufacturers are held to a different regulatory standard than overseas manufacturers can be.
Specifically to the comments about Asian manufacturers. I struggled with that point as it does seem to create a valid criticism on ground of race. The fact is that the source documents specifically call out some Asian manufacturers. The FDA does acknowledge that they can not inspect these facilities unannounced as they do with manufacturers in the USA. There apparently are grounds to think that the documentation they do review when they inspect these facilities, with forewarning, may be too perfect.
Let me once again say I could care less who makes the strips and meters. I want a level playing field that includes, similar inspection processes and post market evaluation of all makers of strips in the real world, bought from the actual retail supply chain. I want to know that medical device reports (MDR) are being completed in consistent fashions by all players.
With that we can StripSafely.
My kindest regard fo all who challenge the accuracy of my calls for accuracy, I try to communicate in an engaging, slightly provocative but well researched way.
The conversation is an opportunity to be clearer.
Bennet
p.s. For Disney freaks like me, the new Fantasy Land exceeded my expectations.
Kelly Close from Close Concerns (Diatribe) shared this opinion piece today, by former Chief Scientific and Medical Officer of the American Diabetes Association, Dr. Richard Kahn:
Competitive Bidding Puts Diabetes Patients at Risk | Commentary
Thanks for that link bowler. It seems that Dr. Kahn has had a change of heart over the last 6 years.
Blood glucose test strips are at the center of diabetes life. Many of patients in the diabetes community are concerned about compromised test strips. I doubt that anything we can say is a clearer illustration of our community's need for highly accurate glucose testing than this FDA Warning Letter which in part says:
Complaints 618, 779, 728, and 818 describe events in which patients exhibited symptoms of low blood sugar levels after receiving normal (70-110 mg/dL) blood glucose reading results when using your firm's device. For each of the complaints, emergency technicians confirmed that the patient had low levels (<40mg/dL) of blood glucose. As a result of the low blood glucose levels, the patients required medical intervention (i.e. glucose liquid gels, (b)(4), sugar, IV glucose, juice, glucose water, and jelly sandwich) to preclude permanent impairment of a body function or permanent damage to a body structure. An MDR should have been submitted for each complaint.
People with diabetes are at risk from inaccurate strips. Let's change that.
Join us in the StripSafely campaign and share your feelings about accurate diabetes testing with policy makers.
Diabetes Daily just published a piece about meter accuracy, which includes a chart comparing accuracy in different meters. Here's the piece:
A major study found that almost half of meters do not meet the minimum required standards:
- For blood sugars over 75 mg (4.2 mmol): Accurate within 20%. For example, if your blood sugar is 200 mg (11 mmol), the meter must read between 160 (8.8 mmol) and 240 (13.3 mmol) at least 95% of the time.
- For blood sugars under 75 mg (4.2 mmol): Accurate within 15%. For example, if your blood sugar is 60 mg (3.3 mmol), the meter must read between 51 (2.8 mmol) and 69 (3.8 mmol) at least 95% of the time.
There is a new proposal that would require all results to be within 15%.
But how do you know if your meter is meeting this standard? Today, there is no systematic verification of meter accuracy after it gets approved for sale. And as you will see below, many meters are sub-standard.
The Diabetes Daily piece is a much better piece; it makes the very important point that the first paper only used one lot.
There is one other point that really should be made - the meters tested were *European* ones; they have the CE mark and they were tested to the ISO standard. These meters are, so far as I can see, *NOT* available in the US. None of the four US meters I have (ActiveChek, FreeStyle, Bayer, Agamatrix) have the CE mark. I believe the US standard is less stringent, but I haven't checked this.
On the "only one lot" issue; what would happen if, say the apparently accurate ActiveChek and FreeStyle meters were tested again using different lots of strips?
The answer is in the second paper (which I have yet to review to my satisfaction). One result that seems clear is that FreeStyle lucked out. The second paper tested four lots. The ActiveChek results seem consistent with the first paper because they don't show much inter-lot variation. The FreeStyle results show large varations with, apparently, the lot from the first trial being more accurate than the others.
John Bowler
I have heard a FDA representative say the current FDA standard is ISO 15197 2003, same as EU. FDA can choose to follow ISO but is not required to follow ISO. In the case of 15197 2003 they choose to. They have not yet made a public choice on ISO 15197 2013, they have said publicly they wanted tighter than +/- 15% and as a result did not vote for the the 2013 ISO standard.
Curious. I thought that the FDA standard was +/-20% above 100mg/dl (i.e. not 75mg/dl) and therefore +/-20mg/dl below 100mg/dl. This is from memory and is based on stuff posted on tudiabets.org. So far I haven't found any find any authoritative links.
John Bowler
Ok, I understand - it was a misunderstanding on my part. I got 100mg/dl from the Hasslacher paper (Google for "ISO 15197" and look for the PDF file 120710-Hasslacher.pdf), but it was the proposal for the *new* standard. The relevant text is:
"According to the current ISO 15197:2003 standard guidelines (Table 1), the minimum acceptable system accuracy for SMBG devices requires that ≥ 95% of the individual glucose results must fall within ± 15 mg/dL of the results of the reference method at glucose concentration < 75 mg/dL and within ± 20% at glucose concentrations ≥ 75 mg/dL [1]. A new standard with more stringent criteria (prEN ISO 15197:2011; Table 1) is being discussed and planned requirements are that ≥ 95% of the glucose meter (GM) results must fall with-in ± 15 mg/dL of the results of the reference method at glucose concentration < 100 mg/dL and within ± 15% at glucose concentrations ≥ 100 mg/dL [2-4]."
So the changes are only to blood samples >75mg/dl; the current percentage accuracy range, where the permitted error gradually drops to 15% at 100mg/dl.
Hum... Here is a better proposal, at least from the Type 1 viewpoint (posted by me on diabetesdaily.com too):
The standard should separate out readings for samples below 80mg/dl and readings above 250mg/dl from the rest, and should have requirements for a small systematic error for readings in the range 100mg/dl-200mg/dl. That's because us Type 1's need to know:
1) Is my blood sugar high (is that why I'm cranky?) At least 99.6% of tests should report a blood sugar of >250mg/dl as being >200mg/dl.
2) Is my blood sugar low (glucose time?) At least 99.6% of tests should report a blood sugar of <80mg/dl as being <100mg/dl and be within 20mg/dl of the actual number. 3) For blood glucose figures of between 80mg/dl and 250mg/dl 99.6% of readings should be within 20% of the actual value and the average over a period of an hour of at least 5 regularly spaced readings should be within 10% of the actual figure 99.6% of the time. In other words, systematic errors are +/-10% in the range 80-250mg/dl and actual errors are changed roughly as proposed in the 2011 standard within that range. For *lower* blood sugars accuracy is increased roughly in line with 2011, however the low range must pass independently of the other ranges. For *higher* blood sugars accuracy is relaxed slightly over 2003, but the high readings must pass these new requirements independently. John Bowler