Before discussing about agile methodology in data projects, first let’s briefly discuss about the nature of data projects. In almost all data projects, there are mainly three steps, popularly known as ETL,: Extraction (E), Transformation (T), and Loading (L). At Deerwalk, the three main steps of data projects are data import, data mapping, and application processing, and an analogy can be drawn between these three steps with ETL.
Traditionally, most software development has been done in an ad-hoc manner: in a code and fix style, and without a clear and consistent set of rules to drive the development effort. The ad-hoc manner has brought about chaos in the development process, leading to many problems that plague the software industry today. There have been countless cases of software failures that have sometimes proved to be fatal – largely attributable to a chaotic style of development. Owing to the lack of a consistent process, budget overruns and schedule overruns have been the norm in many software firms.