Learning to do good science

Sorry to break it to you, but a PhD might not be enough.

I wrote a somewhat ranty post (aren’t they all?) on LinkedIn about why the PhD is broken, and how we can fix it. But the problem’s a lot broader. I’m just going to throw down an idealised list of how good science should be done, and why the PhD isn’t always up to the task.

Establishing a good question

All good research starts with what we know. In a PhD, that’s usually a literature review - but these are often haphazard, poorly targeted and not systematic, or really only written towards the end of a candidature - when, more often than we’d like, PhD students are in the unpleasant position of hunting around for a unifying theme for our thesis. More meta-analyses and systematic reviews are definitely A Good Thing and make the perfect start point. A critical literature review, ideally with a quantitative element, which interrogates the existing theory and evidence, and results in a Good Question (that’s the hard bit), should be a requirement before embarking on your major thesis work (which isn’t to say you shouldn’t begin developing hands-on research skills in parallel).

Designing the program of research

OK, let’s say you’ve found A Good Question. How do you go about preparing to answer that? Well, in theory, PhD students carefully develop a theoretical framework which makes quantitative predictions (i.e. hypotheses), designs appropriate statistical tests for these hypotheses, and plans a data-collection program to satisfy the statistical tests. There’s also perhaps some preliminary pilot/feasibility investigations before committing to The Big Study, so any technical challenges can be dealt with or the program can pivot to pursue surprising results. In practice, the time constraints of a PhD usually make for a batten-down-the-hatches alert where students dive right into a long-term project and figure out the pesky details of hypothesis-testing later. This is obviously bad. See below for a post-mortem.

Collecting & analysing data

Alright, so you’re carrying out your carefully-designed program of research, and because of your very sensible feasibility/pilot projects, there are no nasty technical glitches (Ha! Ha! Ha!). Suddenly, you have all this data. Lovely. But… where does it go? How should you organize it? Label it and record meta-data? Quality control? Statistical and other software (or even, ye Gods, coding)? Visualisations? Frameworks for exploratory analyses that avoid multiple-comparisons and data mining? Reality-checking your results? Our “data science” skills, frankly, are usually not up to scratch. That’s even before we get into life skills and project management. Budgeting, time management, prioritization, task and goal setting, and just… staying organized are not skills we all have, especially when a project goes sideways (as they often do). On top of this, don’t forget to eat well, sleep, develop positive relationships, satisfy your other (e.g. family) commitments or obligations, work on (or, rather, panic about) career development, and feed your spirit with other interests!

What does it mean?

Alright, back to science. From statistical results to knowledge is still a long journey. We need to carefully make inferences about exactly what our models and data are, or are not, telling us. Then we need to think about our expectations - go back to that knowledge base and contextualise your findings. What existing knowledge has been supported, what needs revising, and what has been added? Does it suggest future research directions, through tweaks to the hypotheses or question? If the previous steps have been well carried-out, this will be slightly easier, but it’s a very creative endeavour - and our training is not very conducive to creativity. Recipe-book lab classes were poor preparation for this.

Sharing your findings

You now know more than anyone else about this particular topic. Cool. Probably better get it out there, though, in case you get hit by a bus or something. Publishing a paper, sure - in which journal? How should it be written? Who should you suggest for editors or reviewers? How do you use your digital tools to produce a nice-looking manuscript that facilitates understanding? What about other media - social media, interviews, engaging with people who are stakeholders in your research, conservation or other policy? Conferences - how do you find out about them, choose one, get funding, get a poster or a talk, network and dialogue effectively about your research, and revise your views based on other perspectives? Is there a translational opportunity for your research? How can you make sure this research, your hard work, has as much impact on the world as possible?

Academic-level difficulty

Now find a job (HA! HA! HA!) and do all of that at the same time, with multiple projects in parallel and running on different timelines, juggling funding applications and sources, teaching hundreds of demanding students, supporting your own struggling PhD students, contributing to your department and field pro-bono as reviewer or committee member, maybe moving countries a few times and raising a family, and doing lots of admin. Welcome to the wonderful world of the research academic.

So why don’t we do that?

A PhD usually lacks structured training in most of these skills. It’s ad-hoc, learning is conducted by yourself with much frustration, we lack context and direction, and mentorship is pretty lacking since everyone’s busy. The reality is, we need much more intensive and structure training in all this stuff both before, and in parallel with, our lab time. But we almost never get it. We labour in isolation, inefficient and without sufficient guidance. Why?

First: everyone’s busy. Dedicated staff, peer support groups, and more structure could overcome this. It’s a tricky one. I can only comment that there are massive inefficiencies at play which we can identify and reduce, and that changing our education and support will certainly mean more productive research for both students and supervisors.

Second: culture. It’s hard for research students to say “I feel lost and helpless. I don’t understand. I’m not working effectively. I’m disenchanted and disengaged, and I feel like crap”, because they feel they will be blamed. “Pull your finger out. Just do it. Just work harder. Maybe you’re not cut out for this. Do you really want to do this PhD?” I’ve heard all of that. What I needed was “We know you are committed, and that you have potential. So let’s identify what is missing from the project, from your environment, from your skill-set, and fill those gaps.”

Third: “you should know this already”. Let’s look at a list of things I want training in:

  • Critically interrogating the literature to establish a knowledge base
  • Identifying good questions
  • Developing questions into theoretical frameworks which generate hypotheses
  • Designing statistical tests for these hypotheses
  • Planning data-collection undertakings to feed these statistical tests
  • Actually planning and running a project - budgeting, time, task, and goal management
  • Data & analysis skills & tools
  • Inference, interpretation, contextualisation and extension
  • Communication, dialogue and revision
  • Proffessional context: funding, career development etc.

Yes. we specifically did a lot of this in Undergrad. But - and I want you to listen carefully - REAL PROJECTS ARE DIFFERENT. Sorry for shouting. The current educational model is like so:

Education: “OK, how about you master all this abstract theory?”
Student: “But… why? What’s it for?”
Education (amazed that not all students are perversely interested in esoteric knowledge for its own sake): “Don’t worry, it’ll come in handy. You’ll see.”
Student: “…alright, I guess?” (studies theory with varying effectiveness, nothing really gets cemented)
Education: “Well done! You followed instructions, wrote some stuff, and passed some tests! Here’s a degree!”
Student: “Thanks, that’s very exciting. Can I do a real project now?”
Education (hands multi-year original research project many orders of magnitude more complicated than anything previously attempted): “Oh, sure! Here you go!”
Student: “Um, that's… this is… what am I supposed to do with this?”
Education: “Just do what you did before, but… more. You remember all that stuff, right?”
Student: “I… not really, I mean it was kind of different…”
Education: “Better get a move on and watch that deadline! OK byeeeeee…”

Occasionally, Education will turn up again to offer some piecemeal advice, almost taunting, when we are most out of our depth and our need is greatest.

My ultimate message is this: theory and application together, forever. A PhD is the perfect time to cement deep learning and independence, but we need more support. Because too many students drop out, or get graduated without having developed these skills. How many graduates are career-ready - any career? Not enough. We need more structure to our researcher training.