Questions for Jake Hoffman on his 5/16 talk on "An Illusion of Predictability in Scientific Results"

[jamesallenevans]

Pose your questions here for Jake Hoffman regarding his 5/16 talk on
An illusion of predictability in scientific results. In many fields there has been a long-standing emphasis on inference (obtaining unbiased estimates of individual effects) over prediction (forecasting future outcomes), perhaps because the latter can be quite difficult, especially when compared with the former. Here we show that this focus on inference over prediction can mislead readers into thinking that the results of scientific studies are more definitive than they actually are. Through a series of randomized experiments, we demonstrate that this confusion arises for one of the most basic ways of presenting statistical findings and affects even experts whose jobs involve producing and interpreting such results. In contrast, we show that communicating both inferential and predictive information side by side provides a simple and effective alternative, leading to calibrated interpretations of scientific results. We conclude with a more general discussion about integrative modeling, where prediction and inference are combined to complement, rather than compete with, each other. Contributing papers one and two.

[yiang-li]

How might this emphasis on integrating predictive information influence future research practices in fields that traditionally focus heavily on inference, like sociology where inference allows for testing the theoretical frameworks?

[XiaotongCui]

Thanks for sharing! You mentioned that even experts might confuse inference with prediction. How should we assist them in distinguishing between the two more effectively during the process of education and training?

[zcyou018]

Thanks for sharing! my question is how can the integration of explanatory and predictive models in computational social science improve the robustness and reliability of conclusions drawn from large-scale data analyses, and what are the potential pitfalls or limitations of such integrative modeling?

[zimoma0819]

Thanks for sharing! The study highlights a crucial gap in scientific communication by demonstrating that the exclusive focus on inferential uncertainty often leads to overestimations of treatment effects by experts. Incorporating both inferential uncertainty and outcome variability in data visualizations could help foster a more accurate understanding of scientific results among professionals. This approach could potentially revolutionize the interpretation of research findings, especially in fields where precise data interpretation is critical. My questions are whether this approach could become a standard practice and what we need to do to persuade people to adopt this approach to foster a good environment to present more accurate understanding of scientific results.

[bhavyapan]

Thanks for sharing your work! I am wondering about the implications of this awareness when it comes to the perception of research conclusions -- would experts be more likely to find certain observations or conclusions from research "believeable" if certain practices in communicating them are followed -- essentially could such approaches be possibly used as signalling tools in academic? How prevalent could this issue be across different scientific fields? In terms of policy, how could this illusion of predictability affect the manner in which academic research supports decision-making or feeds into legislative action?

[Dededon]

Thank you for the sharing! I’m curious in your insights about combining machine learning with causal inference research. How do you think of the theoretical frameworks as CausalBERT, that seems to sacrifice interpretability towards better estimation?

[Adrianne-Li]

Hello Dr. Hoffman,

Thank you for your insightful exploration of the illusion of predictability in scientific results. Your discussion raises important questions about the perception of research conclusions. Could the strategies you've proposed for communicating both inferential and predictive information serve as signaling tools within the academic community to enhance the believability of research findings? Additionally, how widespread do you believe this issue might be across different scientific disciplines? From a policy perspective, what impact might this illusion of predictability have on the way academic research is utilized in decision-making processes or influences legislative actions?

Looking forward to your thoughts,
Adrianne(zhuyin) Li
(CNetID: zhuyinl)

[kiddosso]

Thanks for sharing Dr. Hoffman! Your research is really inspirational. I wonder if you have pedagogical advice for teachers and professors in schools based on your paper An illusion of predictability in scientific results? Such a bias may be prevented in school and college education.

[oliang2000]

Hi Dr. Hoffman! In a class, we read a quote by John Rogers Searle that said, 'Prediction and explanation are exactly symmetrical. Explanations are, in effect, predictions about what has happened; predictions are explanations about what’s going to happen.' In your paper, you attempt to distinguish between these concepts. I'm wondering, would you say they are continuous/symmetric concepts or how do you think they relate to each other on a high level?

[C-y22]

Thank you for sharing! You discuss the potential for computational social science to advance by integrating predictive and explanatory modeling. However, one of the major challenges you highlight is the cultural and methodological differences between fields that traditionally focus on explanation and those that prioritize prediction (like computer science). Given this context, how do you envision overcoming these differences to foster collaboration between these disciplines? Specifically, what practical steps or policies could be implemented to encourage integrative modeling in research institutions or academic programs?

[66Alexa]

Thanks for sharing! What areas of future research do you believe are crucial to further understanding and addressing the illusion of predictability in scientific results?

[yuy123337]

Hi Dr. Hoffman! I am wondering why is prediction important in scientific research, particularly in the context of computational social science? Based on the discussed integration of predictive and explanatory modeling in computational social science, why is there an increasing emphasis on predictive modeling? What are the strengths of predictive approaches, and how do they complement traditional explanatory methods?

[Zihannah11]

Thank you for sharing. How do you think the emphasis on inference over prediction in various fields has influenced the way scientific studies are conducted, reported, and interpreted by both experts and the general public?

[natashacarpcast]

Hi! Thank you for this interesting research. I'm wondering which implications does this have for previous science? Would you say you're skeptical of everything that's already done? Or are there any areas where these problematics wouldn't be so harmful?

[iefis]

Thanks for sharing your research! Could you provide some more concrete examples that apply the integrative modelling approach in social science research?