SA
Skip to main content

Project Padme

Build an interaction effects calculator for Manakin.

  • Give our PMs a tool to answer:

For experiments AA and BB running at the same time, is there an interaction effect relative to some metric XX, where presence in experiment AA's group G1G1 influences the effect size between control and treatment in experiment BB? If so, what is the size and confidence?

Technologies and Languages

  • Java, Spring Boot
  • TypeScript, React
  • SparkSQL, Scala, Python

Midpoint Milestone

  • Evaluated a Technical Design and confirmed it applies to all metric templates
    • Accounted for any potential refactoring work needed to support this project without excessive tech debt
    • Reviewed the design for future refactoring of our metric structure
  • A way to submit a job that takes:
    • two experiment names
    • metric definition
    • range of dates
  • A table set up in Databricks with the proper schema to record the results of each job run
    • should consider multiple runs with the same parameters
    • should consider how the data displays in our UI for efficient layout
  • A library that combines given parameters to start a computation job on Databricks
  • Limitations:
    • This could be limited to only one, but not all templates
    • The way to submit the job might be pretty manual

Desired Outcome

  • A workflow that allows a user of our experimentation framework to request calculating interactions between two experiments on one metric
  • Interactions calculated in the past are displayed to all users in an easy-to-find place

Further Improvements

  • A user-friendly UI that reflects all past and current pending runs.
  • Automated set of jobs that get scheduled for high-impact experiments and a subset of metrics

Dependencies

  • Primarily decoupled from any other projects on the team, however
    • The technical design needs to align with our long-term vision
    • UI design needs to be done by the intern with limited support from Pasha Krasilnikov

Links to This Note