Great Expectations, the leading open-source platform for data quality, today announced the results of a survey highlighting top pain points and consequences of poor data quality within organizations. Insights from 500 data practitioners (engineers, analysts, and scientists) showed that 77% have data quality issues and 91% said it’s impacting their company’s performance.
“Poor data quality and pipeline debt create organizational friction between stakeholders, with consequences like degraded confidence,” said Abe Gong, CEO and Cofounder of Superconductive, the company that makes Great Expectations. “This survey made it clear that data quality issues are prevalent and they’re harming business outcomes.”
Data quality issues can make it difficult or impossible to see a “single view” of an end-user or service, lower productivity, obscure reliable performance metrics, and overwhelm development teams and budgets with data migration tasks. Data practitioners blamed poor data quality on lack of documentation (31%), lack of tooling (27%), and teams not understanding each other (22%). They said data scientists spent too much time preparing data for analysts, end-users complained about gaps in their data (such as lost transactions), and production teams were mired with delays.
Data confidence is critical for organizations to make informed business decisions. Fewer than half of respondents expressed high trust in their organization’s data, and 13% had low trust in data quality, which stemmed from broken apps or dashboards, decisions based on unreliable or bad data, teams having no shared understanding of metrics, and siloed or conflicting departments. Additional issues impacting data trust included alert fatigue, misalignment on certain metrics, and friction between teams.
“Data quality is critical to facilitate the making of decisions with confidence across the organization, enabling a singular understanding of what that data means and what it’s being used for. That’s why support for data quality efforts should be found at every level of an organization, from data scientists and engineers to the C-suite and board who have confidence in outcomes for decision-making,” Gong said.
89% of respondents said their leadership was supportive of data quality efforts, and 52% believed leadership regards data quality with high trust. When asked about their company’s current approach to data quality, 75% said they validated data. Only 11% do not think they have a data quality issue. Data quality efforts included having a data quality plan scoped and budgeted (22%), using a specific data quality tool (19%), checking data manually (14%), and building their own systems (15%).
This survey was conducted in May 2022 by Pollfish, an independent research platform, leveraging responses from 500 information services and data professionals based in the United States (57% men and 43% women aged 18-54). 60% of respondents were employed at companies with 250 or more employees.
