Making DGP of methods with ETOM

[hyunjimoon]

Goal

• short term with ETOM crew: better learning experience for future ETOM class students. I really love this class but lack of structure was gave me hard time learning. mail to toni and kris feedback by the end of summer is one measurable outcome.

• long term with Stan crew: can push dynamic calibration idea started in this paper using simulation-based calibration.

• long term alone: include my observation on andrew's love for synthetic data and his tendency to explain many things with EM algorithm into my prior

Belief

confident:1, not confident:0

type1 confidence (value of action):

1. (0) organizing different methods from ML, endogeneity bias, instrumental variables, field and lab experiments, simulations, diff in diff, regression discontinuity, matching, synthetic controls, propensity score matching, structural estimation would help learning

2. (1) methods with different labels from each domain (statistics, econometrics) like quantile regression, rank statistics, synthetic control, weight learning, matching, may share core ideas from nonparametrics, importance sampling, robust statistics and understanding the root by drawing method generation tree (i.e. method DGP) can help us catch up with newly generated methods like doubly robust means

3. (0) cause of 1,2: as movement is perceived by the gap of two movements, it is better to learn two concepts together (build hierarchy for pooled learning) dynamically concepts is more effective than separately learning static version of two concepts.

type2 confidence (dgp that affects my action):

5. (0) Paul will share his prior for goal1,2
6. (1) Michelle, Toni, Kris will share their prior for goal 2
7. (.5) Andrew will share his prior for goal3,4

Action

• Toni's verification on (Both methods rely on pre-treatment data to create a counterfactual scenario (what would have happened without the treatmen)t; but I didn't understand his comment about the two's distinction on before vs after when I showed "They differ in how they construct this counterfactual: propensity score matching does this by creating pairs or groups based on individual unit characteristics, while synthetic control does it by constructing an aggregate entity that represents a weighted combination of control units" so I followed up for further input.

• asking Paul's prior on
  -- idea behind doubly robust mean and reference (e.g. whether this is ok)
  -- hypothesis 5

• sharing my prior with Paul on stability : last paragraph of andrew's bayesian cringe

• asking Michelle's prior based on posterior above

• make example with synthetic control and propensity score matching to argue belief1,2,3 to lower uncertainty in belief 6,7

• asking Toni and Kris's prior based on posterior above

• asking Andrew's prior based on posterior above

---

• slide on type 1 vs 2 confidence (willingness to bet vs question)

[hyunjimoon]

unprocessed raw material

• ETOM class transcript here and here especially Toni's comment on doubly robust mean and its connection? to propensity score matching

Toni highlighted the concept of "doubly robust" methods, illustrating how different methodologies, like propensity score matching and inverse probability weighting, converge on a fundamental principle. In this context, doubly robust refers to the resilience of the analysis when either the propensity score model or the treatment outcome model is correctly specified, but not necessarily both. This robustness underscores a shared core idea in statistical methodologies, where different approaches can yield reliable results under certain conditions. Toni's discussion extends beyond the technicalities of doubly robust methods, advocating for a broader understanding of methodological evolution. By emphasizing the importance of aligning research questions with appropriate methodologies, Toni suggests that the evolution of statistical methods is not just about technical refinement but also about strategic selection and application of these methods in research. This perspective aligns with the hypothesis that understanding the genesis and evolution of methodological processes can help in navigating and leveraging the commonalities across diverse analytical techniques, such as quantile regression, rank statistics, and others, which may all share underlying principles like nonparametrics and importance sampling.

---

Mar.29

While they are methodologically distinct, both methods aim to estimate causal effects in observational settings by creating comparable groups or scenarios to mimic a randomized experiment. The formal connection lies in their shared goal of reducing selection bias and confounding to isolate the causal effect of an intervention.

• Both methods rely on pre-treatment data to create a counterfactual scenario (what would have happened without the treatment).

• They differ in how they construct this counterfactual: propensity score matching does this by creating pairs or groups based on individual unit characteristics, while synthetic control does it by constructing an aggregate entity that represents a weighted combination of control units.

Mar.3

connection with Synthetic Control and Propensity Score Matching

1. Synthetic Control:

   • Methodology: Synthetic control methods (SCM) aim to estimate the effect of an intervention by creating a synthetic control group that closely resembles the treated group. This method constructs a weighted combination of control units to mimic a randomized experiment.
   • Goal: SCM reduces selection bias and confounding, isolating the causal effect of an intervention.
   • Data Requirement: SCM relies on pre-treatment data to create a counterfactual scenario.
   • Construction of Counterfactual: In SCM, a synthetic control is created by selecting control units with similar pre-treatment characteristics to the treated unit.
   • Use Cases: SCM is useful when experimental randomization is not feasible due to legal, business, or platform reasons. For instance, it can be applied to assess the impact of changes (e.g., removing payment options) without running an actual experiment.
   • Example: Uber used SCM to test whether providing driver contact details before a ride increases customer satisfaction.
   • Advantages: Transparent, interpretable, and applicable in various practical scenarios.

2. Propensity Score Matching (PSM):

   • Methodology: PSM estimates treatment effects by accounting for covariates that predict receiving the treatment. It aims to balance the treatment and control groups on confounding factors.
   • Goal: PSM reduces bias due to confounding variables by creating comparable groups.
   • Data Requirement: PSM requires information on covariates that predict treatment assignment.
   • Construction of Counterfactual: PSM matches treated and control units based on their propensity scores (likelihood of being treated given covariate values).
   • Use Cases: PSM is suitable when few comparable units exist in the non-treatment group and high-dimensional covariates need consideration.
   • Example: Suppose we want to assess the impact of smoking (an unethical treatment to assign randomly). PSM helps control for biases by making treatment and non-treatment groups comparable.
   • Advantages: Useful for causal inference in non-experimental settings.

In summary, both methods serve the common purpose of estimating causal effects, but they differ in how they construct the counterfactual scenario. Synthetic control aggregates control units, while propensity score matching focuses on individual unit characteristics. Researchers choose the method based on practical feasibility and data availability¹⁶⁸.

Source: Conversation with Bing, 3/30/2024
(1) Introduction to Synthetic Control Using Propensity Score Matching. https://www.analyticsvidhya.com/blog/[2](https://en.wikipedia.org/wiki/Synthetic_control_method)022/12/introduction-to-synthetic-control-using-propensity-score-matching/.
(2) Synthetic control method - Wikipedia. https://en.wikipedia.org/wiki/Synthetic_control_method.
(3) Propensity score matching - Wikipedia. https://en.wikipedia.org/wiki/Propensity_score_matching.
(4) Synthetic Control Methods for the Evaluation of Single-Unit .... https://academic.oup.com/aje/article/190/12/2700/6336907.
(5) Combining Matching and Synthetic Control to TradeofBiases From .... https://a-torgovitsky.github.io/sc.pdf.
(6) Synthetic and External Controls in Clinical Trials – A Primer for .... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7218288/.
(7) Using Synthetic Controls: Feasibility, Data Requirements, and .... https://pubs.aeaweb.org/doi/pdfplus/10.1257/jel.20191450.
(8) A new tool for case studies in Epidemiology - the Synthetic Control .... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5986610/.
(9) Propensity Score Matching: A Guide to Causal Inference - Built In. https://builtin.com/data-science/propensity-score-matching.
(10) Propensity Score Matching: A Statistical Method - PubMed. https://pubmed.ncbi.nlm.nih.gov/31913173/.
(11) undefined. https://doi.org/10.1257/jel.20191450.

connection with missing data imputation

1. Synthetic Control:

   • Imbalance Problem:
     • In synthetic control, we aim to estimate the causal effect of an intervention (e.g., policy change) on a treated unit (e.g., a region or country).
     • The challenge arises when we lack a true control group due to practical constraints or ethical reasons.
     • Imbalance occurs when the treated unit's characteristics differ significantly from any available control units.
   • Imputation Approach:
     • Synthetic control constructs a synthetic control group by combining weighted elements from potential control units.
     • These weights are determined based on pre-treatment characteristics (covariates) to ensure similarity with the treated unit.
     • Imputation plays a crucial role in estimating missing values for these covariates.
     • By imputing missing data, synthetic control aims to create a balanced synthetic control group that closely resembles the treated unit.
     • The imputed covariates contribute to reducing bias and improving the reliability of causal effect estimates.

2. Propensity Score Matching (PSM):

   • Imbalance Problem:
     • PSM is used to mitigate confounding in observational studies.
     • When covariates used to estimate the propensity score (PS) are only partially observed, imbalance can occur.
     • Missing covariates lead to biased PS estimation and subsequent matching.
   • Imputation Approach:
     • Multiple imputation (MI) is a potential solution for handling missing covariates in PSM.
     • MI generates multiple datasets with imputed values for missing covariates.
     • Propensity scores are then estimated for each imputed dataset.
     • The Within approach combines propensity scores individually from each imputation to obtain exposure effect estimates.
     • By imputing missing covariates, PSM aims to create balanced comparison groups, reducing bias and enhancing precision in causal effect estimation.

In summary, both synthetic control and propensity score matching leverage imputation techniques to address imbalance issues, ensuring more reliable causal effect estimates in observational studies.

Source: Conversation with Bing, 3/30/2024
(1) A Design-Based Perspective on Synthetic Control Methods. https://gsb-faculty.stanford.edu/jann-spiess/files/2021/01/designsynth.pdf.
(2) The balancing trick: Optimized sampling of imbalanced datasets—A brief .... https://onlinelibrary.wiley.com/doi/full/10.1002/eng2.12298.
(3) A Factor Based Multiple Imputation Approach to Handle Class Imbalance .... https://dl.acm.org/doi/abs/10.1016/j.procs.2022.12.406.
(4) Avoiding pitfalls when combining multiple imputation and propensity scores. https://onlinelibrary.wiley.com/doi/full/10.1002/sim.8355.
(5) "How to Apply Multiple Imputation in Propensity Score Matching with .... https://digitalcommons.wayne.edu/jmasm/vol19/iss1/1/.
(6) Multiple imputation analysis for propensity score matching with missing .... https://journals.sagepub.com/doi/full/10.1177/09622802211037075.
(7) Double-adjustment in propensity score matching analysis: choosing a .... https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-017-0338-0.
(8) Considerations for Using Multiple Imputation in Propensity Score .... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9272848/.
(9) undefined. https://doi.org/10.1002/sim.8355.