In 2013, research firm Gartner predicted that by 2016, 75% of large enterprises will have more than four diverse automation technologies within their IT management portfolio. And robotics and automation were front and center of Gartner's top-10 list of strategic predictions for last fall, including the prediction that by 2020, autonomous software agents outside of human control will participate in 5% of all economic transactions.

Welcoming our robot overloads

data_automation_robots.jpgI’m personally passionate about automation, mostly because I believe the human mind should be freed from tedious and mundane tasks. Why waste human energy on something that can be effectively performed by a robot, freeing the human to engage in more creative endeavors, such as data-driven problem-solving?

I know the prospect of increased automation incites fear in many, who believe that automation might spell the end of their careers. And let's be honest: that is sometimes the case! After all, no one is hired to be a "calculator" any more (yes, that was a real job title), and vending machines are so commonplace that we often forget that selling snacks and beverages used to be a uniquely human endeavor.

But more often, what actually happens is that robots work withhumans and take over the time-consuming, mundane work that most of us don't particularly enjoy, anyway: driving, counting pharmaceuticals or making change. Now those humans are free to engage in more sophisticated and creative pursuits that no robot could possibly perform. After all, who resents a dishwasher for taking away that chore?

In the IT world, I do believe automation is the only way to achieve a truly agile and flexible data management system. As we mentioned in an earlier post, most data warehouse solutions are still living in the 90s, relying on star schema and selective data that is often out of date. But automation can modernize the data warehouse while offering the following benefits:

  • Reduced risk Automatically updated data is cleaner, which reduces the risk of introducing errors.
  • Reduced development time Once automation is established, development time required for data warehousing is minimal, thereby reducing cost and increasing speed.
  • Increased focus on mission-critical tasks Developers that aren't spending their hours maintaining the data warehouse are free to invest their skills in more mission-critical work generating business value.

ebook: Five Keys to Ending the Battle Between Business and IT

Optimize data scientist time

data scientist BIIt’s been estimated that 50 to 80 percent of an analyst’s time is spent on data preparation, the process of preparing data to be warehoused so it can be used for reporting at a later date. That's over half of that valued team member's brain's time being used for something that could easily be performed by a robot!

I think this is tragic when tools exist to automate the process and free data scientists to do what they love and what they are paid for: analyzing the data and recommending a data-driven course of action. If you want to speed up delivery times, lower the cost of development, and make better business decisions sooner, you should consider adding automation to your data warehouse. Or, if you have the luxury of building a data warehouse from scratch, consider building an agile Data Discovery Hub instead. 

The secret sauce: DWA

Data Warehouse Automation, or DWA, refers to the process of accelerating and automating everything that can be automated in the data warehouse lifecycle. In the latest report on data warehouse automation tools, industry analyst Wayne Eckerson describes data warehouse automation:

Data Warehouse Automation (DWA) tools eliminate the manual effort required to design, deploy and operate a data warehouse. By providing an integrated development environment, DWA tools enable developers and business users to collaborate around designs and iteratively create data warehouses and data marts. This turns data warehouse development from a laborious, time consuming exercise into an agile one.

Eckerson Report: Data Warehouse Automation Tools

In the above report, Eckerson argues that both model-driven and data-driven proponents can benefit from DWA. In our experience, we've seen great benefits from automating:

  • data exploration
  • warehouse data models
  • ETL generation
  • test automation
  • metadata management
  • managed deployment
  • scheduling
  • change impact analysis
  • maintenance and modification of the data 

How DWA improves self-service analytics

Many of my customers have already invested in self-service BI tools such as Qlik or Tableau, which provide obvious value to the business. However, users often discover that, if the original data wasn't prepared with excruciating foresight and attention to detail, much of the beautiful data visualization loses its value due to outdated, inaccurate or missing data.

How self-service BI is driving data democratization

When paired with Data Warehouse Automation, however, self-service BI tools readily deliver on their promises. With automation in place to categorize, clean and deliver the right data at the right time, those data visualizations are fully actionable. So why not welcome our robot overlords so we can focus on doing what we as humans do best: making data-driven decisions to build our business.