[Opinion] A strategy for building user-centered big data utilization

Many companies are building big data analytics platforms to use increasingly diverse data. Beginning with a data lake to hold enterprise data, they collect varied formats and pursue advanced analysis to deliver high-quality insights to decision-makers.

Yet many organizations remain skeptical of their platforms or struggle to operate them. Common concerns include return on investment, differentiation from existing information and analytics systems and ways to increase resource use. These questions need to be addressed in platform design and operation.

<Securing return on investment from existing data platforms >

Exploding storage requirements are increasing operating costs for information systems. Companies are responding with lifecycle management, compression and other measures.

Expensive appliances create budget constraints. Dividing storage into hot and cold data can free resources and reduce storage and expansion costs: Hadoop handles large, long-retained cold datasets, while information systems analyze recent hot data.

<Differentiating from existing information and analytics systems >

Big data professionals recognize the need for differentiation. Traditional analytics mostly uses structured legacy data; big data platforms also accept unstructured and semi-structured sources, with Kafka, Flink and Spark Streaming enabling real-time ingestion pipelines.

Unified document search can also cover varied documents and images used in everyday work. Text conversion and OCR extract, preprocess, collect and store text, broadening its use.

<Increasing use of big data resources and platforms >

Platforms must improve usability for both users and analysts. Essential foundations include data classification, lineage, ownership and stewardship, metadata, structured and unstructured search, and resource management, helping people find and use data effectively.

Traditional metadata tools for structured data are insufficient. Many companies focus on ingestion infrastructure without investing enough in practical use and supporting environments, leaving platforms less usable than existing systems. These services are essential: self-service analytics only becomes possible with a well-supported foundation.

<Securing data and services in big data environments >

Greater access must be accompanied by security. Legacy systems traditionally grant access through ITSM permissions based on data and service categories. Now organizations need extended classifications and new controls for Hadoop data access, including security and auditing.

Platforms such as Cloudera CDP use Atlas and Ranger to manage Hadoop classifications and permissions through web interfaces and APIs. The aim is to let analysts and users request access easily, classify sensitive data appropriately and ensure security through masking and access controls.

Related coverage: Electronic Times