Cover photo: Hao Wang on Unsplash

Given that Artificial Intelligence (AI) and machine learning (ML) are considered new in comparison to other fields of research and technology, it is not surprising there is a lot of misconception by companies as well.

Recently, I was interviewed for a Lead ML Engineer position. As usual, the job posting was somewhat vague. During the interview, it was clear that the company was new to AI/ML. When asked, the interviewer said that they only needed someone to help in deployment since their Data Scientists were doing model selection, feature engineering, and up-to-this-point deployment. Therefore, they just wanted an AI engineer to automate the deployment process using MLOps (which does not require an advanced degree). Trying to inplement MLOps in that environment is doomed to fail and/or be wrought with technical problems.

Background

Let us quickly review the definition of the fields of Data Science and AI Engineering:

• Data Science explores how large quantities of data can be efficiently stored, managed, queried, and summarized, and how massive data sets can be used for making predictions.

• AI Engineering focuses on the design and implementation of intelligent systems that perceive, act, and learn in response to their environment.

Obviously, the core field of study for an MSDS degree is not AI/ML. Thus, Data Scientists really should not be performing ML tasks such as feature engineering, model selection, hypertuning, deployment, etc.

AI engineering is an emergent discipline focused on developing tools, systems, and processes to enable the application of artificial intelligence in real-world contexts — SEI.

In a nutshell, AI engineering is the application of software engineering principles and best practices to AI.

There is a lot of work to be done on a commercial AI project, so if the roles are not clearly defined and/or there is overlap the entire project will be problematic. In fact, if the project involves MLOps then it is crucial for the AI engineer to have responsibility for the entire SDLC.

Since a majority of AI projects fail, it would be better for companies to focus on academic credentials (degree, certification, etc.) rather than “real-world” experience.

Conclusion

Since AI/ML is still considered a new field, it is important for companies to understand the job roles. The fact that software engineering tech interviews are broken only adds to the confusion and potential failure of AI projects.

References

[1] T. Shin, “4 Reasons Why You Shouldn’t Use Machine Learning,” Towards Data Science, Oct. 5, 2021.

[2] L. Visengeriyeva, A. Kammer, I. Bär, A. Kniesz, and M. Plöd, “MLOps Principles,” mlops.org, 2022.

[3] P. P. Ippolito, “Design Patterns in Machine Learning for MLOps,” Towards Data Science, Jan. 12, 2022.

[4] Artificial Intelligence Engineering, Software Engineering Institute (SEI), 2022.