Research Design, Conduct, and Data Management
Research Design and Planning
- Hypothesis formulation – Formulate clear, testable, and falsifiable hypotheses prior to data collection and align them with study design and analysis plans. For studies where falsifiability of hypotheses is not feasible, ensure the studies are structured for rigorous inference through clear research questions, predefined methods, transparent assumptions, evaluation of alternative explanations, and clear communication of uncertainty and limitations.
- Clear operational definitions and methodological transparency – Fully specify methods, controls, and analysis decisions in study protocols and publications, including defining all variables, measurements, procedures, and data analysis plans clearly enough to enable independent replication.
- Data Management Plans (DMPs) – Develop and follow a Data Management Plan, or Data Sharing and Management Plan, that aligns with institutional and funder requirements.
- Research involving emerging technologies (dual-use, Super Intelligence, etc.) – Assess risks, limitations, ethical implications, and potential misuse of emerging technologies and develop and document safeguards. Consult appropriate institutional officials for additional guidance, requirements, and/or approvals.
Recordkeeping and Data Management
- Accurate and concurrent documentation – Maintain complete, accurate, and time-stamped records of all research activities, methods, and decisions.
- Version control – Use systematic version control for data, code, and documents to track changes and ensure reproducibility.
- Chain of custody for data and metadata standards – Document data provenance and handling from collection through analysis. Apply appropriate metadata standards to ensure data are interpretable and reusable.
- Adherence to retention requirements – Retain data and records in accordance with applicable laws, regulations, institutional, and funder policies.
- Stewardship of federally funded data – Manage data as a public asset, ensuring accessibility, integrity, and long-term usability.
Reproducibility, Replicability, and Transparency
- Open methods, data, and code – Share methods, data, and code to the extent possible, consistent with legal and ethical constraints and institutional and funder requirements.
- Transparent uncertainty characterization – Quantify and clearly communicate uncertainty, assumptions, and limitations.
- Distinction between reproducibility and replicability – Clearly distinguish and communicate whether results are reproducible (same data/code) or replicable (independent data).
- Long-term data archiving and data availability statements – Deposit data and materials in appropriate repositories for long-term access. Provide clear statements describing how and where data can be accessed.
- Reporting negative or null results – Report all relevant findings, including null or negative results in research communications.
Questionable or Detrimental Research Practices (To avoid)
- Conducting research without clearly defined hypotheses or presenting vague, unfalsifiable claims.
- Presenting post hoc hypotheses as if they were defined a priori, also known as HARKing (Hypothesizing After Results are Known)
- Using ambiguous definitions or omitting key methodological details, or using inadequate controls or inappropriate comparators.
- Iteratively manipulating analyses, selectively reporting results, or stopping data collection once statistical significance for a particular pattern is achieved, also known as p-hacking and data dredging.
- Selectively including or excluding data without justification. Suppressing results that do not support hypotheses.
- Prematurely discarding or failing to preserve required records.
References
