Free-flowing project management for data science
Moving quickly and informally without much project management structure
In this issue, I’m writing my thoughts on how I approach project management for data science that I learned over the years through trial and error, experimentation, and iteration.
Data science is still considered a new field compared to engineering and product management, and when it comes to data science teams, we should at least question the more traditional ways of thinking. Project management is no exception. What works best for other functional teams doesn’t necessarily work best for data science, and here I describe the story of an unpopular opinion from my personal experience.
Atlassian defines project management as the art of making a plan and then executing it. Many project management frameworks like Scrum, Kanban, and Waterfall can be effective methods to promote efficiency. By submitting requests in the form of a ticket into a tool like Jira and reporting on project status, nothing should fall between the cracks and business partners1 would have complete visibility into the status of any given project and transparency over the project details.
While project management tools and frameworks can work wonders for projects with clearly defined milestones that need to be executed in a certain order where we know exactly what we need to build and how to build it, they can be more harmful than beneficial for data science. The life cycle of a typical unstructured data science project is complex and non-linear. We learn as we go with data science. The data science team just cannot anticipate so many twists and turns and won’t be able to plan for them ahead of time. The research and development component of data science work can’t be timeboxed, and their cost and level of effort can’t be confidently estimated upfront to merit proper resourcing. Models, hyperparameters, and feature transformations, for example, are all the elements of a data science project that need to be learned using data through trial and error, experimentation, and iteration, not designed upfront.
Trying to put too much structure and process around data science projects hinders the on-the-go learning of data scientists.
The beginning
At a startup that was eventually acquired, I was the first data scientist in the product and engineering org that used the scrum2 framework with a two-week sprint cycle consisting of daily stand-ups, sprint planning, demos and retros. Everything was according to the book. Data scientists, just like software engineers, would receive new requests through an off-the-shelf intake tool like Jira and with a product manager3, acting as a scrum master, would groom those piling backlog requests. Oftentimes, data science requests lacked context and reasoning because entering them through the tool was all too easy and it did nothing to push back or ask questions. Only after the grooming session did subsequent communication with business partners begin in the tool to clarify requests and report on their status, so data scientists found it difficult to get to the root cause right away which was a huge waste of time.
Because so much is unknown for data science, data scientists couldn’t commit to a short, strict two-week sprint timeline, and more often than not, new research findings in the middle of a sprint would completely derail the rest of the planned work. There were a bunch of unfinished parallel work at the end of the sprint that got carried over to the next sprint. Setting a reasonable goal for the sprint, estimating how much effort and how long it would take to get the work done, was nearly impossible. More importantly, data scientists were heads down too busy tackling day-to-day requests that they had little time to invest in research and development for learning new business capabilities that had a significant potential long-term value for the company.
At a daily stand-up meeting lasting 10-15 minutes, the status update of a data scientist would go something like this: “Yesterday, I started doing research on forecasting models. Today, I’ll continue doing that research.” This is an extreme but real example, and you get the idea. This kind of quick sharing of updates was unhelpful and useless—data scientists needed a venue to discuss their projects in depth with fellow data scientists that required longer dedicated time. Stand-ups were just not the right place to share ideas and thoughts and provide meaningful feedback.
All these ended up creating so much coordination and communication overhead. We’ve tried several improvements and modifications to address the issues, but fundamental problems still persisted. Data scientists were becoming mere order-takers from top down. Innovation was lacking, morale tanking, accountability low, and their precious time being wasted.
The transition
Shortly after I joined the data science team at a late-stage startup, I successfully helped transition the team to the kanban4 framework from scrum as a way to promote greater flexibility for the research and development component of data science. Kanban is less prescriptive, less rigorous, and less stressful than scrum. We created a Kanban board whose columns resembled the stages of a typical data science project workflow (e.g. research, data analysis, modeling, testing, review, and deployment) where data scientists could move their tasks around however they see fit. Data scientists need not worry about having to hit constant deadlines. They created flexible project milestones that were more spread out rather than laying out all deliverables upfront. Planning was done lightly, not tightly. Having learned from my experience before, I took the liberty to overhaul the existing project management structure to enable data scientists to do what’s right for them.
We decided to remove the grooming session, stand-up, and sprint review. Instead, we replaced daily stand-ups with weekly hourly sit-downs to promote deep technical discussions, demo projects, and share valuable concerns about what’s working and not. Because the team worked with so many different business partners, it made sense for individual data scientists to share their progress and results directly to their business partners in their own project meetings. The ownership of the backlog items was limited to individuals, not the entire team, because the cost of context switching and knowledge transfer was much higher for data scientists due to the nature of research and discovery and the level of expertise required in different projects.
Naturally, data scientists started doing PM activities for their own projects (e.g. writing and updating tasks in the tool, breaking down complex problems into items in the backlog, setting priorities and roadmap for their projects, running a project kick-off meeting, reporting on project status, writing documentation, and coordinating and communicating directly with business partners and PMs across the org to understand deeply the customer and the context behind their requests). Data scientists were given the accountability and autonomy they needed to make decisions for themselves. They were responsible for aligning with business partners on “what” and owning “how.”
At some point in time, we created a custom intake tool that asked a series of questions like a flow chart instead of using an off-the-shelf intake tool to help avoid working on someone else’s toy project or stale problems that never got anywhere5. As the principal data scientist, I leveraged my experience to support fellow data scientists along their project lifecycle to ensure successful project delivery6, but ultimately, individual data scientists were owning their projects.
Over time, however, data scientists doing PM activities quickly grew out of hand. With many layers of management and complex org structure and ways of working, some business partners would get frustrated when a more compliant data scientist would fail to communicate effectively or not properly manage the project. So the tempting remedy seemed like hiring a dedicated data science PM, also referred to as data product manager. Data science PMs were great for certain data science projects that involved coordinating and navigating through the large, complex cross-functional interests; that built on top of an already well-established non-ML product where data science would be a small part of it; and that required access to specialized knowledge not readily available to data scientists. They were also useful when the team needed to be intentional about data product improvements beyond data and models e.g. user experience. The company was very much a product-led organization, so almost every team, including the platform engineering team that data science was part of, had a PM and most of the core work happened when PMs were leading.
The problem was that the data science team ended up leveraging data science PMs even when the conditions didn’t warrant it. PMs cannot be a panacea for every problem! This created unnecessary “meetings for meetings” with flurry of documents and overly redundant communication between data science PMs and data scientists that held them up. PMs unduly prescribing upfront what will be done also prevented data scientists from moving quickly and informally. Some less experienced data science PMs didn’t fully grasp what it meant to build a data product and needed a shift in habits and mentality. Data scientists were the closest to the data and had great ideas, but data science PMs were dictating, ideating, and representing the team. The team needed to be explicit about when to leverage data science PMs so they were not taking ownership away from data scientists again. Data scientists needed to be provided with the autonomy beyond flexibility, along with the accountability and influence over their work for the bottom-up innovation.
Towards free-flowing
Finding a project management framework that’s appropriate to the research and development component of data science work is challenging. At my current company, we don’t use a formal project management tool or framework. I would describe what we do as something closer to open-source project practices described in Eric S. Raymond’s The Cathedral and the Bazaar7—by intentionally having small teams and hands-on managers, each manager responsible for a team8 of 3-5 people or even fewer, and each team dedicated to a business capability like recommender systems and demand forecasting, the team doesn’t need much project management structure. They can move quickly and informally, taking full ownership of their data science project and decision-making all the way to the customer.
[B]y intentionally having small teams and hands-on managers, … the team doesn’t need much project management structure.
It’s important to note that people will communicate naturally in the absence of project management structure. Inevitably, business partners would have to sufficiently explain the problem to a data scientist to ensure it gets solved. Rather than people simply passing half-baked ideas and strongly guarding their beliefs and opinions, it promotes a deep dialogue among people. Only the most pressing requests and crucial projects will end up surviving. In this generalist9 model, data scientists would communicate directly with business partners instead of leveraging data science PMs as an intermediary. If needed, data science managers should be more helpful than data science PMs.
In this world of free-flowing project management, the role of data science managers is to create circumstances where data scientists can do good work and then get out of the way. From time to time, they might need to step in and cut down a little to make room for new streams of work, but they're just trying to provide data scientists with the autonomy they need to learn along the way and innovate, with minimal intervention.
Of course, this approach is not without the downsides. We need to hire generalist data scientists who are superb communicators and proactive partners with cross-functional teams.
Outro
It was quite an uphill battle to push through the changes that seemed unorthodox to many folks outside of data science, for the existing project management tools and frameworks worked well for them already. It’s all too convenient to simply apply what’s been working well for others out there when things are unclear. Instead, we should at least take a step back to question the more traditional ways of thinking when it comes to data science.
This free-flowing approach has been extraordinary and very distinct in my view from how most data science companies are run today, which are much more top-down than bottom-up. In other top-down companies, somebody outside of the data science team dictates or ideates, and then data scientists are assigned to execute the idea10. However, data scientists are the ones closest to the data, and the best ideas are going to come from them. They see meaningful patterns and anomalies in data that the rest of us do not.
Ending note
A huge thanks to Rachel Moon on Data Science Doodles for reviewing drafts of this post. Curated archive of my newsletter can be found here.
About this newsletter
I’m Seong Hyun Hwang, and I’m a data scientist, machine learning engineer, and software engineer. Casual Inference is a newsletter about data science, machine learning, tech startups, management and career, with occasional excursions into other fun topics that come to my mind. All opinions in this newsletter are my own.
I intentionally use the term (business) partners instead of stakeholders throughout the post, for the meaning of the latter tends to imply an unequal partnership between business partners such as marketing and operations teams that become customers and data scientists that more often than not become mere order-takers.
Many companies use scrum for data science. I’m not saying they are wrong; in fact, the very downsides of scrum I describe can be the upsides depending on how you look at them.
In this post, I conveniently refer to the product manager, project manager, and program manager as PM. In large, complex organizations, these complementary roles perform slightly different duties, but at startups, the differences are usually trivial.
A couple of Medium articles on Scrum and Kanban for data science you may find useful:
Over time, however, business partners rarely used the custom intake tool ironically due to a high barrier of submission. Upon knowing they were required to write details in it, they felt more compelled to communicate directly with the data science team instead. In the end, the tool merely served as a formal record keeping that flowed into the team’s kanban board.
My responsibilities included assessing the viability of the project, pushing back and prioritizing projects, writing design docs, identifying obscure production issues and fixing them, and driving roadmap creation and execution. Managers were to support data scientists, not manage them.
If a team consists of fewer than 3-5 people, would it make sense to call it a “team?” 🤔
For a deeper treatise of why data scientists should be more generalists than specialists, check out this article from StitchFix.
I wrote in my previous post how this applies to software engineering.