A new study was published this week that attracted some attention.

“Low carbohydrate diets may increase risk of neural tube defects“

Zoe Harcombe wrote a blog post about it detailing criticisms of the study that did not appear particularly valid from my reading of the study. As these criticisms were being shared on social media today I thought I would write a quick response. Quoted sections are from the linked blog post.

“A study was published in the journal “Birth Defects Research” on 25th January 2018 (Ref 1). A press release accompanied it (Ref 2) claiming that “women with low carbohydrate intake are 30 percent more likely to have babies with neural tube defects, when compared with women who do not restrict their carbohydrate intake.” Newspapers worldwide reported the story (Ref 3), reiterating the scary headline that “Women on low carb diets may be at 30 percent greater risk of having a baby with a spinal and neurological birth defect, according to a new study.”

One of my lovely subscribers, Belinda Fettke, alerted me to the article and asked me to take a look at it as my weekly newsletter. So I did. It was a case control study. It was fundamentally flawed in a number of ways:

1) The characteristics table compared the control with the control, not the control with the cases.”

Exclusion of NTD cases is deliberate and is detailed in a footnote to Table 1. Table 1 compares the folate intakes and other characteristics of mothers eating lower-carb diet and those eating more carbs. This shows that estimated folate intakes were lower even when mothers with NTD cases were excluded in both groups. 

(…)

“2) The ‘correct’ characteristics table was available, as a Supplemental, but neither table included important data related to the study – not least carbohydrate and folate/folic acid intake.”

Table 1 does include estimated daily folate/folic acid intakes and does include the categories of carbohydrate intake in the table footnotes, although it does not detail the average intake of carbohydrate in each category.

(…)

This indicates that even when excluding cases of neural tube defects women in the restricted carbohydrate group (<95g) were estimated to be consuming less than half the folate of the nonrestricted group.

“3) Consequently the study did not adjust for material differences between the control and case groups.”

Incorrect, it was adjusted as is detailed in the methods section.

“4) The study could not make the conclusion that it did.”

Yes, it could.

“I sent the email below to the authors on 30th January and sent a copy to the journal editor. I was minded to wait until I received a reply before posting this openly, but I have been asked to post this by a number of people who know that I found retraction-level errors because the headlines generated by this article need to be countered immediately. Pregnancy is worrying enough for women and men without fabrications like this trying to scare the life out of them.

Dear Dr Desrosiers,

I am currently reviewing your very interesting paper “Low carbohydrate diets may increase risk of neural tube defects.”

The article abstract reported “To assess the association between carbohydrate intake and NTDs [Neural tube defects], we analyzed data from the National Birth Defects Prevention Study from 1,740 mothers of infants, stillbirths, and terminations with anencephaly or spina bifida (cases), and 9,545 mothers of live born infants without a birth defect (controls) conceived between 1998 and 2011.” The article reported that these numbers became 1,559 (cases) and 9,543 (controls) with valid exclusions. That’s fine.”

Table 1 reported the characteristics of the control group only. Table 1 should show the differences between the cases and the controls, not least so that it is known what to adjust for. Supplemental Table 1 reported this data.”

Incorrect, this is a case-control study looking at differences between exposure to different categories of dietary carbohydrate intake, and by inference folate intake, so this is the correct table.

“Q1) Please can you explain why Table 1 is in the main paper and not Supplemental Table 1?”

See above.

“Q2) Please can you add the carbohydrate intake data to Supplemental Table 1 and please can you add the folate/folic acid intake data to Supplemental Table 1? These are standard inclusions in the characteristics table, so that readers can review, prima facie, the hypothesis being tested.”

Readers would be unlikely to be able to review the hypothesis being tested. The numbers of women with low-carbohydrate intakes were small in the total sample numbers, therefore any differences in average carbohydrate intake would be lost in the large numbers of subjects.

“Q3) Please can you add the calorie intake data to Supplemental Table 1, as this has been adjusted for, but not reported anywhere?”

Perhaps a valid point, although average calorie intakes could be added they would not add much to the interpretation of the study.

“Q4) Please can you provide the p values for Supplemental Table 1?””

These could be added but this would not add much to the study as these are not the main subject of this study and these comparisons have been made in many other research papers.

“Q5) Working on the assumption that case/control ratios outside 0.9-1.1 are likely to be statistically significant, Supplemental Table 1 suggests that adjustments should have been made for (I have used a (Y) and a (X) to indicate what was/wasn’t adjusted for): maternal race/ethnicity (Y); maternal birthplace (X); education (Y); household income (X); BMI (X); smoking (X); alcohol use (Y); folic acid antagonist medication use (X); and study centre (Y).

Please can you explain why the factors marked with an (X) weren’t adjusted for?”

The answer to this question is detailed in the methods section of the paper.

“Q6) If I interpret Table 2 correctly, it means that of the 1,559 cases, 93 restricted carbohydrate (6%) and 1,466 (94%) didn’t and it means that 479 controls (5%) restricted carbohydrate and 9,064 (95%) didn’t.

The study “hypothesised that some women who restrict carbohydrates may have suboptimal folate status and subsequently may be at higher risk of having an NTD-affected pregnancy.”

Notwithstanding that you set out to prove a hypothesis (and not to disprove the null), please can you confirm that you failed to prove this hypothesis? Table 2 could conclude that “94% of NTD-affected pregnancies occurred in women not restricting carbohydrate.” Table 2 could also conclude that “of the women who had an NTD-affected pregnancy, fractionally more (1 in 100) restricted carbohydrate.” Table 2 cannot conclude the other way round – that those who restrict carbohydrates may be at higher risk of having an NTD-affected pregnancy.”

This question seems to result from a misunderstanding of odds ratios, which are tricky things to understand. Many explanations can be found online including this one.

Also this one: Explaining Odds Ratios.

Odds ratios are calculated like this:

These are some of the results of the study:

An odds ratio of 1.20, in this case, indicates that for every 5 children with any neural tube defect in the nonrestricted group there were 6 children with any neural tube defect in the restricted group.

After adjusting for the other factors listed in the footnotes the odds ratio is 1.30

Although this difference is small it could add up to many more cases of neural tube defects if the association held true across the population as a whole.

“The study “hypothesised that some women who restrict carbohydrates may have suboptimal folate status and subsequently may be at higher risk of having an NTD-affected pregnancy.”

Notwithstanding that you set out to prove a hypothesis (and not to disprove the null)…”

This appears to be a misunderstanding of the difference between a research hypothesis, that poses a specific hypothesis to be tested, and statistical hypothesis testing.

One of Zoe Harcome’s own papers appears to contain just such a research hypothesis that she seems to have set out to prove “Evidence from prospective cohort studies did not support the introduction of dietary fat guidelines in 1977 and 1983: a systematic review“.

“Q6) Having added carbohydrate intake, folate/folic acid intake, calorie intake and p values to Supplemental Table 1, please can you adjust for all the differences between cases and controls and then re-calculate the odds ratios accordingly?”

Adjusting for all possible variables when there is no rationale for doing so is not good science (it is stated in the methods why some of these were not adjusted for).

“Please can you then revise and reverse the directionality of the press release and correct the newspaper articles world-wide, which reported that: “women with low carbohydrate intake are 30 percent more likely to have babies with neural tube defects, when compared with women who do not restrict their carbohydrate intake.”

Many thanks

Kind regards – Zoë”

Comments

Almost every study has minor errors or additional things that could potentially be added. They type of study this is means that the authors have to rely on food frequency questionnaire and other questionnaires. While these are imperfect they are about the only way carry out a preliminary study into questions like this. If they find associations studies like this can then form the basis for further research.

However, from my reading of the study, the criticisms detailed by Zeo Harcombe do not appear to be valid and I do not think these are “retraction-level errors”.

While the results of this study are only epidemiological associations, together with the reduced estimated folate/folic acid intake reported, it makes a plausible link that should be investigated further.

Low-carb diets are not intrinsically high-or-low in folate. But as many people now rely upon fortified flour for their dietary folate, if this is removed from the diet and not replaced with other sources then folate intakes will decline.

Rather than trying to discredit the study, this could be used as a point to discuss supplementation or the inclusion of more low-carb foods that are naturally high in folate for women eating low-carb diets who may become pregnant.