Originally posted on Forbes Technology Council
As organizations increasingly embark on their digital transformation journey, IT is turning into a profit center, rather than a cost center. CIOs (chief information officers) are more than often referred to as chief innovation officers. New roles like chief data officer and chief analytics officer are rising to prominence. AI and data are at the center of this transformation, as CxOs are faced with daunting challenges in:
- Reducing their time to market.
- Reducing risk and improving productivity.
- Increasing the top line by introducing new business models and improving net promoter scores (NPS).
- Improving the bottom line by improving the total cost of ownership.
Organizations on their digital transformation journey are facing increasing pressures due to the pandemic, remote workspaces and increasingly distributed applications. IT’s ability to rapidly adapt to changing market needs is paramount to a successful digital transformation journey.
AI and data-driven analytics are the centers of this transformation, as decision support systems are evolving with their journey from rule-based systems to descriptive, diagnostic, predictive, prescriptive and eventually cognitive analytics, where the goal is to become fully autonomous. The success of this transformation is very much tied to underlying infrastructure technologies, which need to be real-time, agile and scalable and resilient.
In my role, as I speak to many of these CxOs, they say that adopting AI-first and cloud-first strategies to support these transformations is vital. These strategies leverage infrastructure technologies like the following to build their cloud-native data pipelines:
- 5G, IoT, hybrid, multi-cloud and edge computing.
- Microservices, serverless computing, containers and service meshes.
- Accelerated computing and NVMe flash-based storage services.
Building real-time operational data intelligence involves adopting new disciplines like MLOps, DataOps and AIOps. Each of these approaches borrows principles of collaboration from the success of DevOps, where development and IT engineering work in tandem in an agile manner. Similarly, these new disciplines have a line of business collaborating with IT engineering for operationalizing machine learning, data management and artificial intelligence.
MLOps, machine learning operations, primarily focuses on model cataloging, version control, compute orchestration and scheduling and involves feature engineering, hyperparameter optimization, inference and finally the deployment of a model for inference. It involves version control of an experiment, which contains the model and the training data, for easy reproducibility.
MLOps is increasingly adopting responsible AI, which encompasses explainability, transparency, security and reproducibility of experiments to incorporate ethics and eliminate biases. MLOps use cases range from statistical machine learning and computer vision to conversational AI and recommendation systems.
MLOps platforms need continuous data access to massive data-sets, as model learning is a virtuous process and neural net prediction and accuracy are only as good as its data and training.
DataOps, data management in the AI era, brings the same agility principles to data, which has now become the most strategic asset and is referred to as the new source code. DataOps involves two practices:
- Data transformation and enrichment, to ensure actionable insights from the data are ingested from a myriad of structured, semi-structured and unstructured data sources.
- Operationalizing the data, which involves the consolidation to a single source of truth, data orchestration and data governance (which involves security, cataloging, data protection, disaster recovery and lifecycle management) from the edge to the cloud.
DataOps needs high-performance and scalable data lakes, which can handle mixed workloads, different data types — audio, video, text and data from sensors — and that have the performance capabilities needed to keep the compute layer fully utilized.
AIOps is one of the disciplines where the use of AI has become prominent in IT automation and operational management. AIOps involves full-stack observability, where historical and real-time data from solutions is leveraged to co-relate, predict and prescribe actions. AIOps is used to augment ITOM (IT operational management) and ITSM (IT service management) practices to provide more holistic and causal analysis, contextualizing all incidents.
AIOps can span business processes, user experiences, application performance management, infrastructure management, network management and security. In this case, MELT (metrics, events, logs and traces) is built into the stack without the need for discrete monitoring agents.
AIOps data pipelines typically consist of five dimensions:
- Dataset selection (the ability to prioritize issues).
- Pattern discovery (the ability to deal with supervised or unsupervised datasets for event correlation or topology analytics).
- Inference (real-time, what-if or causality analysis).
- Communication (visual or textual).
- Automation (the ability to course-correct in flight).
All five dimensions are important for a successful AIOps implementation. Many vendors excel in one or the other. For example, Splunk has been traditionally strong with its log analytics offering, while others like CloudFabrix have shown more innovation moving to AIOps 2.0, promising faster time to value, with asset intelligence and data preparation and integration.
CxOs who are tasked to implement AI-first and cloud-first strategies should study these new disciplines and vendor offerings to make intelligent choices when it comes to streamlining their digital transformation journey in these transformational times. They should identify the business problem, identify KPIs (key performance indicators) to quantify progress, identify data sources and their veracity, run analytics to derive actionable intelligence and then operationalize it. These new disciplines certainly have the promise of operationalizing AI across several verticals.