Does Agile Methodology Apply To Data Projects?
When looking at best working practices and frameworks for overseeing data projects, many companies consider agile. What is agile? Well, agile methodology is a set of principles and practices for software development that prioritises flexibility, collaboration, and customer satisfaction.
This framework is often adopted due to the similarities in delivering a piece of software and delivering a data product. Both go through a similar lifecycle of ideation, work and deliverables. But how well do the four key principles of Agile apply to data projects?
Individuals And Interactions Over Processes And Tools.
The point of this principle is to emphasise the importance of communication and collaboration. Understanding the needs and expectations of end-users is key to any data project being successful and seeing an ROI for your organisation.
Sadly, the opposite is often true, and we’ve seen a lot of data professionals who are stuck to a particular way of working, a particular set of tools or process they have to follow.
When we consider the spirit of this principle, perhaps the lesson is that sometimes it is better to think about delivering something that works today, and then spend the time to ensure its robustness tomorrow. For example if a stakeholder wants to turn a spreadsheet into a dashboard, it might be better to create a dashboard directly from the CSV in a matter of hour, then spend dats modelling each of the data points, even if that is more scalable in the long-run.
This is not to say though that processes and tools are not important to the overall success of data initiatives, and this is perhaps where this principle is difficult to uphold. It is not always possible to be flexible or find faster ways of doing things when thinking about transforming large datasets. Data teams are typically bound to a small number of tools within their data stack, which will naturally come with some limitations or ways of working.
The challenge then in applying this principle is delivering data products that serve the ultimate goal, keep stakeholders engaged whilst still adhering wherever possible to best practice.
Working Software Over Comprehensive Documentation
We’ve always placed an emphasis on creating value from data. On measuring the success of a data initiative through the business impact. And this should ALWAYS be your priority as a data team rather than documentation, but it’s also true that documentation is increasingly important.
As companies look to scale their data and pave the road for AI, well structured and documented models and taxonomy are absolutely key. While no one enjoys documentation, it can easily be forgotten about and so while the agile principle of working software (or in our case data product) still holds true, this should not be an excuse to skip the paperwork!
Customer Collaboration Over Contract Negotiation
Perhaps this applies more to agencies and clients, but the point to emphasise in this principle is that traditional contracts will specify all requirements upfront, whereas the agile approach in data projects involves ongoing collaboration to adapt to changing business needs.
The problem for data projects in engaging with internal stakeholders is often a barrier of understanding. Effective collaboration requires a shared understanding of data capabilities and limitations, and so before projects can be scoped it is often wise to prioritise some basic level of data literacy training to explain the pipeline process, and why some tasks (like modelling) take longer than others.
If end-users or business stakeholders do not have a clear understanding of the data processes or the insights that can be derived, then its quite likely a gap will form between their requirements and what can be delivered, adding friction to the project. After all, if your stakeholders don’t understand the process, they are likely to become frustrated waiting months to see results.
This is where we always recommend an iterative approach to projects. Smaller steps give you the best of both worlds. A clear idea of the scope of the project, a smaller timeframe, and so less likelihood of needs changing, which relates nicely to our next point.
Responding To Change Over Following A Plan
While some flexibility is often needed in data projects, if you’re taking an iterative approach then there shouldn’t be big changes in direction from your stakeholders. Making changes on the fly can be resource-intensive and may disrupt your workflow. This principal may not directly apply to data projects were the needs of end customers are more limited then most software applications.
Of course, change can also be a positive catalyst to project success. The world of data is fast-moving and it seems as if new technologies and new ways of working emerge every two or three years. This may mean that there are faster and cheaper ways of achieving results which may not be possible before and so all data teams need to constantly be keeping in touch with industry, understanding new vendors and new tools and re-assessing best practice.
Conclusion
While the principles of the Agile methodology have proven invaluable in the realm of software development, the nature of data projects is different and means a direct application of this framework may fail. We instead advocate that under a robust framework and clear process, small interactive data projects which drive meaningful change, a light version of scrum.
Scrum has become the de-facto Agile framework, so much so they are often synonymous with each other. Scrum involves breaking down a project into smaller time-boxed sprints, typically lasting two to four weeks. 173tech favours this approach as it allows us to show clear outcomes and deliver tangible, incremental deliverables to stakeholders.
No framework is without its pros and cons, and the choice should be influenced by your organisation’s size, culture, and the nature of the data project at hand.