Market trends work against data pros
Market trends are working against data professionals whose jobs are becoming increasingly more difficult.
That was the word from Todd Goldman, VP and GM of enterprise data integration and data quality at software development company, Informatica, speaking during the Informatica Day event in Johannesburg yesterday.
"Great data isn't an accident. It happens by design. For organisations to ensure they have clean, safe and connected data, they need to power confident decisions and effective business processes, which is not an easy task," he said.
According to Goldman, there is more data, in more places, and moving faster than ever before. About 90% of the data in the world today has been created in the last two years alone, he revealed.
"The volume, velocity and variety of data is increasing at an unprecedented pace. The amount of data generated in the world today is doubling every two years. In 2009, there was about 0.8 zettabytes of data and in 2020, it will rise to 35.2 zettabytes," Goldman said.
He also pointed out there are a lot of technologies data professionals now have to master in order to deal with the large volumes of information.
For example, he said, technologies like Hadoop allow organisations to affordably store vast amounts of data. The power of mainframe computing now fits in the palm of our hand, he noted, adding RFID devices and sensors are also being attached to almost everything. "The volume, variety and velocity of data will only continue to increase."
Adding to the challenge data professionals are facing, Goldman noted although data is now everywhere, its quality is questionable. "Data is scattered everywhere. It's in all the old places, and all the new ones. Both on-premises and in the cloud."
The other concern is organisations are now demanding self-service regarding their data, he said, explaining this is as a result of the emergence of the empowered consumer who uses tools like apps, mobility and social media.
"Businesses do not want to wait for IT to deliver great data. They want to do it on their own," he noted.
Thus, he is of the view, faced with all these new demands, a new way of thinking is needed.
"The problem, however, is that people still think about their business applications as a single, monolithic thing, and that's where they are wrong. The new reality is we are faced with many interconnected applications," said Goldman.
Today, solving tough business problems depends on data from more than one system, he said.
"Every data project is like this - it is always harder than anyone thinks and the number of moving parts is only increasing. To overcome this challenge, you have to design great data into your business processes. You have to invest in competency and efficiency in data integration as well."