Hey Guys,

Welcome back from the summer, I hope you had a rejuvenating time of it. Today I want to look at some Newsletters related to our spectrum of data science, data engineering and the more technical sides of ML. In this piece we’ll look at some key Newsletters and a few of their key posts.

Subscribe now

I really like compilations and I hope you find some value out of this.

1. ByteByteGo

ByteByteGo Newsletter

Explain complex systems with simple terms, from the authors of the best-selling system design book series. Get 158 page System Design PDF, FREE

By Alex Xu

What to look for?

Visual snippets.

Top Posts

1. Understanding Database Types

2. Password, Session, Cookie, Token, JWT, SSO, OAuth - Authentication Explained - Part 1

3. Database Indexing Strategies

2. Latent Space

Latent Space

The AI Engineer newsletter + Top 10 US Tech podcast. Exploring AI UX, Agents, Devtools, Infra, Open Source Models, and more. See https://latent.space/about for highlights from Andrej Karpathy, George Hotz, Simon Willison, Emad Mostaque, and more!

By swyx

What to look for?

In-depth explanations and good references.

Top Posts

1. The Rise of the AI Engineer

2. Reverse Prompt Engineering for Fun and (no) Profit

3. The Anatomy of Autonomy: Why Agents are the next AI Killer App after ChatGPT

3. The A.I. Edge Newsletter

The AiEdge Newsletter

A newsletter for continuous learning about Machine Learning applications, Machine Learning System Design, MLOps, the latest techniques and news. Subscribe and receive a free Machine Learning book PDF!

By Damien Benveniste

What to look for?

Tutorials.

Top Posts

1. Deep Dive: How to Build a Smart Chatbot in 10 mins with LangChain

2. Deep Dive: How do Vector Databases Work

3. Deep Dive: How to structure your code for Machine Learning Development

If you enjoyed these quick-links let me know and I will continue the series, I really like the idea of celebrating others who are creating immense value in the ecosystem.

Leave a comment

Hey Guys,

I feel as if the no-code era really is taking shape with how GitHub Copilot, Google PaLM and things like ChatGPT code interpreter are changing how developers work.

Subscribe now

So Tweet threads like this one really catch my attention.

See the Tweet Thread

I’m seeing a lot of the same observations on YouTube as well who sound bullish about what the ChatGPT Code Interpreter Plugin can do.

So some folks are hyping this Code Interpreter as a no-code tool for example, for marketing or product folk who don’t know how to code. Take a look:

The latest ChatGPT code interpreter is a plugin that allows users to run Python code within a chat session with ChatGPT. It also supports file uploads and downloads, and several Python modules. It is an experimental feature that is available to a limited number of users and developers.

“This Changes Everything”

Even for Data scientists I think this plugin could have some utility.

ChatGPT plugins of course, have a lot of potential.

The Code Interpreter plugin appears to be helpful for those interested in web development. OpenAI introduces plugin for ChatGPT that unlocks ability to access web content.

The ability to have ChatGPT complete tasks with information from the Internet is a feature that users have long wanted because the current model contains data that is current only up to 2021.

Ethan Mollick of is someone that appears especially bullish on ChatGPT's programming and web capabilities. He makes some lucid points at the rate these things are evolving at. While “prompt engineering” hacks are a cool thing in early 2023, they soon might not exist that is LLMs might be paired together in that Auto GPT and GPT agent trend that makes it a redundant skill.

Some a claiming that Code Interpreter may be the “most useful” ChatGPT plugin of them all so far:

Share

The newest ChatGPT plugin is the most powerful feature OpenAI has released since GPT-4.

Many believe that GPT-4 is the breakthrough moment for A.I. becoming more useful in terms of mass adoption, whether that’s with ChatGPT or something it leads to.

Whatever the case may be people like Ethan are hyping or are enthusiastic about the possibilities here.

In Aakash Gupta’s thread he goes over many potential utilities of how we could use Code Interpreter. Things like:

• Segment your customers in seconds

• Decompose seasonality simply via text

• Linear regression by itself

• Easy Geo Charts

• Basic descriptive charts in seconds

• Graph public data without input

• Automatic Radar Charts

• Heatmaps With Ease

• Output the log chart automatically, and others.

These sound vaguely like they might be useful for Data visualization and some tasks done by data scientists.

I think the full potential of the Code Interpreter Plugin might become known in the weeks ahead. But, I just wanted to let you know in case it’s something you might have an interest in playing around with (when it becomes more widely available).

A lot of the Twitter and LinkedIn A.I. grifters appear to have gotten the same memo.

GPT-4 is the new data scientist

With the new OpenAI code interpreter model, you just have to upload the data and provide instructions in simple english. The model does everything from cleaning data to generating insightful visualizations on autopilot!!

Usually I wouldn’t be intrigued, but this does sound potentially useful at scale. Source. I’m not usually a big fan of the ChatGPT prompting guides and profiteers.

A simple search on Twitter revealed that Akash and Ethan weren’t the only ones, there’s a whole army of people talking about this topic right now.

Share Datascience Learning Center

Another Twitter character said Code Interpreter could do the following:

1. Analyze your data with ease

2. Visualize your CSV data

3. Basic Video Editing

4. Music Market Segmentation

5. Color Palette Extractor & Image Compressor

6. Create Animated GIFs Easily

I’m not sure how ‘useful’ any of these things are but the PR marketing campaign for ChatGPT is 24/7 on Twitter these days.

View 'Code Interpreter' on Twitter

Thanks for reading!

Welcome back!

I’m hugely passionate about the space of data science and all things software engineering. Recently I’ve noticed a tangible shift to more data engineering type advice and Newsletters and I wanted you to be aware of what’s available here.

Data Engineering is one of the fastest growing topics on Substack’s Technology category now as of April, 2023.

To see the long list go here.

Top AI Newsletters Macro Overview

I did an initial survey of this and asked a number of people in the area for their lists of favorites. This is by no means a complete list, so take it with a grain of salt.

While the names and trends might change, the demand for educational info on these things does not. As a reader you need to subscribe to a combination of people and Newsletters to get good information in my humble opinion.

I believe in 2023 Substack is becoming a decent educational platform for these topics. This wasn’t always the case.

So let’s get into the preliminary list. And let me know your own favorites.

Leave a comment

I’m sure I missed a ton of the good ones: (this list is in no particular order)

Top Data Engineering Related Newsletter on Substack April, 2023.

I hope this list is valuable to someone out there.

1. SeattleDataGuy’s Newsletter

Really experiential actionable tips also has a YouTube.

SeattleDataGuy’s Newsletter

Learn About End-To-End Data Flows (Data Engineering, MLOps, and Data Science)

2. Data Analysis Journal

Specializing in analytics.

Data Analysis Journal

Where product, data science, and analytics intersect. Trusted by thousands of analysts and data scientists around the world

By Olga Berezovsky

3. Data Products

Data Products

A newsletter by Chad Sanderson - Discussing data product development, semantic layers, data APIs, and modern data modeling.

By Chad Sanderson

4. Data Engineering Central

Data Engineering Central

Long Live the Data Engineer. No holds barred.

By Daniel Beach

5. The Analytics Engineering Roundup

The Analytics Engineering Roundup

The internet's most useful articles on analytics engineering and its adjacent ecosystem. Curated with ❤️ by Tristan Handy and Anna Filippova.

By Tristan Handy

6. Data Engineering Weekly

Data Engineering Weekly

The Weekly Data Engineering Newsletter

By Ananth Packkildurai

7. Hysterical

Hysterical

Sharing insights into Technology topics and Tech Careers with the hope to add value to people's lives. Thoughts shared here are personal and not affiliated with any other entity. All Rights Reserved.

By Neelesh Salian

8. Mehdio’s Tech (Data) Corner

Mehdio's Tech (Data) Corner

data engineering, tech, career and coding ‍💻

By mehdio

9. Data Engineering on GCP

Data Engineering on GCP

Thoughts and learnings from building a shift-left, multi-modal data platform on Google Cloud Platform. Disclosure, I’m a Google Cloud Champion Innovator

By Robert Sahlin

10. Alt Data Weekly

Alt Data Weekly

all that is happening in the alternative data industry

By John Farrall - 90 West Data

11. Scaling Data Ops Newsletter

Scaling DataOps Newsletter

Every week I share insights from top data leaders on the challenges of scaling data infrastructure. In addition, I aim to make this newsletter feel like you are talking to my guests one-on-one, as each question response includes the audio from my guest.

By On the Mark Data

12. High ROI Data Science

Not always related to Data Engineering, but one of the pioneers here.

High ROI Data Science

I provide insights from over a decade in Data Science for technical and non-technical audiences. Covering Data & AI Strategy, Leadership, Data Product Management, Data & Model Literacy, Applied ML Research, & Causal Methods.

By Vin Vashishta

13. Latent Space

Latent Space

Productized AI/ML, Software 3.0, and other notes from Latent Space

By swyx

14. Data Creation

15. Swirl A.I.

SwirlAI Newsletter

Learn Data Engineering, Machine Learning, MLOps and Data Science.

By Aurimas Griciūnas

16. The Developing Dev

The Developing Dev

Filling the gaps for engineers who don’t have dedicated mentorship in 5 minutes or less. Written by a Staff Software Engineer @ Instagram

By Ryan Peterman

If you found this list useful, why not Restack it on Notes. It’s that share button thing at the bottom.

Hey Everyone,

I recently started a Talent Collective for software engineering and A.I. jobs, you can see the jobs here. You can join the collective if you see it here. I’m noticing a lot more SRE type jobs, which stands for site reliability engineer.

As the Modern Data Stack (MDS) evolves, new roles are increasingly being defined in data science and software engineering. How the SRE, Platform Engineering and DevOps roles are maturing is fairly interesting.

According to AWS:

What is site reliability engineering?

Site reliability engineering (SRE) is the practice of using software tools to automate IT infrastructure tasks such as system management and application monitoring. Organizations use SRE to ensure their software applications remain reliable amidst frequent updates from development teams. SRE especially improves the reliability of scalable software systems because managing a large system using software is more sustainable than manually managing hundreds of machines. 

But in a data-first world, increasingly even smaller firms need SREs. The SRE role is common in large enterprises, but smaller businesses need it, too.

As data grows and travels across clouds and to the edge and back, more experts are saying that organizations need an SRE-like role for data.

DevOps gained popularity in order to combat siloed workflows, decreased collaboration and a lack of visibility across the software development lifecycle. Now in 2023, it’s the era of a demand for SREs.

DevOps teams don’t necessarily have someone specifically dedicated to developing systems that increase site reliability and performance.

That’s where a site reliability engineer (SRE) comes into the picture.

Site reliability engineering was originally developed by Google. In the words of Ben Treynor, SRE is “what happens when you ask a software engineer to design an operations function.”

So what does that actually look like in terms of roles?

SRE is kind of like a more proactive form of quality assurance (QA). Site reliability engineers will be dedicated full-time to creating software that improves the reliability of systems in production, including:

• Fixing issues

• Responding to incidents

• Usually taking on-call responsibilities

Aside from its growing role today, SRE’s biggest claim to fame might be the four golden signals of monitoring:

• Latency

• Traffic

• Errors

• Saturation

Some argue, The data reliability engineer — often called the site reliability engineer (SRE) for data or database reliability engineer (DBRE) — could be the missing role needed to create clarity in the ever-more complicated stack. Roles are becoming increasingly specialized as the Modern Data Stack becomes more seamless and yet more complex at the same time.

Whatever your predictions for the Modern Data Stack in 2023, the importance of SREs and DBREs is on the rise.

Even though the site reliability engineer (SRE) role has become prevalent in recent years, many people—even in the software industry—don't know what it is or does. Site Reliability Engineering: How Google Runs Production Systems, written by a group of Google engineers, is considered the definitive book on site reliability engineering. Google vice president of engineering Ben Treynor Sloss coined the term back in the early 2000s.

Subscribe now

Read more

Hey Guys,

I’m not a software engineer but the debate of Data science vs. Data Engineer rages on. How would you summarize this debate? I wanted to sort of answer some of the FAQs on this topic. I hope this summary is helpful to someone out there reading this.

Share

But think about it, careers within the field of data science have in recent years seen soaring demand, with the Bureau of Labor Statistics forecasting a 22% increase in job growth from 2020-2030—much higher than the average growth of other occupations

Leave a comment

Which is better data science or data engineer?

Simply put, the data scientist can interpret data only after receiving it in an appropriate format. The data engineer's job is to get the data to the data scientist. Thus, as of now, data engineers are more in demand than data scientists because tools cannot perform the tasks of a data engineer.

In the recent past, the general belief in the industry was that as more and more advanced automation tools are developed, the need for pure data scientists would erode. But that hasn’t played out (yet) and may not.

Data Engineers Earn More

What pays more data engineer or data scientist?

Data engineering does not garner the same amount of media attention when compared to data scientists, yet their average salary tends to be higher than the data scientist average: $137,000 (data engineer) vs. $121,000 (data scientist).

You do the math, over a career that’s a significant difference.

Data science is easier to learn than data engineering.

Why? Well there's simply more resources available for data science, and there are a number of tools and libraries that have been built to make data science easier.

It’s all a bit confusing as these titles are different at different organizations, for instance:

Can a data scientist become a data engineer?

At some organizations, data scientists are tasked with doing things that data engineers should. While data scientists aren't equipped with the skills to become data engineers, they can acquire the skills. On the other hand, it's far less common when data engineers begin doing data science.

Job Descriptions are Different

Today, the main difference between these two data professionals is that data engineers build and maintain the systems and structures that store, extract, and organize data, while data scientists analyze that data to predict trends, glean business insights, and answer questions that are relevant to the organization.

• Builders vs. Storytelling

That is, Data scientists build and train predictive models using data after it’s been cleaned, and then they communicate their analysis to managers and executives. Data engineers build and maintain the systems that allow data scientists to access and interpret data. The role generally involves creating data models, building data pipelines and overseeing ETL (extract, transform, load).

• Engineering vs. Communication

That’s not to say that data scientists aren’t technical, they just aren’t only working on Engineering.

Read Datascience Learning Center in the Substack app

Available for iOS and Android

Get the app

Why data engineer is better than data scientist?

Data Engineers collect relevant Data. They move and transform this Data into “pipelines” for the Data Science team. They could use programming languages such as Java, Scala, C++ or Python depending on their task. Data Scientists analyze, test, aggregate, optimize the data and present it for the company.

So it all depends in the workflow where you prefer to be.

The science part of Datascience might not appeal to everyone:

As part of their job, they conduct online experiments, develop hypotheses, and use their knowledge of statistics, data analytics, data visualization, and machine learning algorithms to identify trends and create forecasts for the business.

While data engineers are really knee deep in the nitty gritty.

Does data science require coding?

All jobs in Data Science require some degree of coding and experience with technical tools and technologies. To summarize: Data Engineer: Moderate amount of Python, more knowledge of SQL and optional but preferable is knowledge on a Cloud Platform.

The past five years we’ve been trying to decode the difference between Data Science and Data Engineering and it may still in 2022 depend on the company, industry and the needs of the moment.

Many data engineers and data scientists hold a bachelor’s degree in computer science or a related field such as mathematics, statistics, economics, or information technology.

But think about it, with the increasing integration of AI and machine learning in data analytics platforms, the data scientist of tomorrow may no longer need to have degrees in quantitative fields or to develop algorithms from scratch. What do you think?

Leave a comment

Data Science Still Out earns MBAs

Who earns more MBA or data scientist?

The recent placement data from Symbiosis Pune reflects that a postgraduate program in Data Science when compared to a general MBA degree has better placement opportunities in terms of average salary and highest package offered.

Data Engineering is still a tough sport and is considered a stressful job:

Is data engineering stressful?

Many factors force data engineers to work long, irregular schedules that take a toll on their well-being. In fact, 78% of survey respondents wish their job came with a therapist to help manage work-related stress.

A Dataquest blog explains that the data engineer usually lays the groundwork for the data scientist to “analyze and visualize data.” Some of the initial tasks performed by the data engineer may include managing data sources, managing databases, and launching tools to make the data scientist’s job easy. So, strictly speaking, the data engineer handles all the back-end tasks of data analytics that lay hidden from the public eye.

Different Types of Data Engineers

Substack’s own Seattle Data Guy has a YouTube that can be entertaining. He reminds me of a gamer YouTube:

Hard-Coded Data Engineeers

What Are the Requirements To Become a Data Engineer?

Data engineers usually hail from a software engineering background and are proficient in programming languages like Java, Python, SQL, and Scala. Alternatively, they might have a degree in mathematics or statistics that helps them apply different analytical approaches to solve business problems.

Types of Data Engineers

You could even consider different types of Data Engineers basically:

Having an idea of which one you want to be might help you learn the right skills from day one:

1.The Big Data Engineer

2. The Data Warehousing Engineer

3. The Data Analyst

4. The Data Architect

5. The Cloud Engineer

6. The Data Scientist

7. The Machine Learning Engineer

8. The Software Engineer

9. The Data Programmer

There’s a certain amount of data engineering taking place in all of these different roles.

Why are such technical distinctions important, even to those not working directly with data? Because few business professionals — and even fewer business leaders — can’t afford to not know the difference.

A Data Engineer is a person who works with data to build better software, but there are different types of data engineers that are more specialized in certain areas. Also a Data Engineering role from one company or one industry to another, may be relatively different.

Some believe Data Engineers Will Be More Important Than Data Scientists suggesting that data chiefs in modern enterprises are realizing that advanced and automated tools alone cannot deliver results, which are expected to be both superfast and at scale.

In reality, many data engineers take advantage of roles such as data architect, solutions architect, and database developer to perfect their data engineering skills, develop a deeper knowledge of data processing and cloud computing, and gain experience with ETL and data layers. Some may also work in data analytics to bolster their knowledge of what data analysts and data scientists need before transitioning into data engineering. This also continually changes depending on the Modern Data Stack and software being used.

Can I move from data engineering to data science?

Yes, with some further training, data engineers can become data scientists and vice versa. Because of the overlap in abilities, from programming languages to data pipelines, individuals of both professions have the foundational understanding and terminology to make a relatively smooth job shift.

Is data engineering a lot of math?

Mathematics is necessary for programming or data engineering but it's not mandatory to have an academic degree or course on mathematics. But you have to be an expert on numerical analysis, statistics, probability and logistic analysis.

Is Python enough for data science?

Python is a high-level, general-purpose programming language known for its intuitive syntax that mimics natural language. You can use Python code for a wide variety of tasks, but three popular applications include: Data science and data analysis. Web application development.

Should I learn Python before data science?

To work in data science, you'll need to learn at least one of two languages — Python or R. If you already have some experience with R, then it's best to go through with it before starting with another language. On the other hand, if you're new, start with Python due to its versatility.

What Are the Requirements To Become a Data Engineer?

Data engineers usually hail from a software engineering background and are proficient in programming languages like Java, Python, SQL, and Scala. Alternatively, they might have a degree in mathematics or statistics that helps them apply different analytical approaches to solve business problems.

• So even on a programming level, the two roles are fairly different in knowledge levels and programming languages to focus on.

Is engineering harder than data science?

Is data science harder than software engineering? No, data science is not harder than software engineering. Like with most disciplines, data science comes easier to some people than others. If you enjoy statistics and analytical thinking, you may find data science easier than software engineering.

Do data engineers use C++?

C++ is one of the essential programming languages that can be used by Data Engineers. C++ can be used for computing large data sets along with processing around 1GB of data in a second. Through this, Data Engineers can retrain the data and maintain consistency with records.

Should I learn SQL or Python first?

One thing to remember is that SQL is a big first step to some more complex languages (Python, R, JavaScript, etc.). Once you understand how a computer thinks, it is easy to learn a new programming language to analyze your data.

Finally there was a YouTube video I found somewhat amusing on the topic of Data scientists vs. Data Engineering here that may still be relevant today:

Thanks for reading!

If you found this valuable considering supporting the channel and my efforts. You also get access to premium articles which are occasional but whose frequently are likely to increase.

Subscribe now

Hey Guys,

Data Engineering terms are very interesting especially for prospective data scientists.

So If you’re reading this, you’ve probably heard the term “ETL” thrown around in relation to data, data warehousing, and analytics. It stands for “Extract, Transform, and Load.”

I was curious about its history.

ETL has roots in the 1970s and the rise of centralized data repositories. But it wasn’t until the late 1980s and early 1990s, when data warehouses took center stage, that we saw the creation of purpose-built tools to help load data into these new warehouses.

ETL is a process that extracts, transforms, and loads data from multiple sources to a data warehouse or other unified data repository. For more terms you can go here.

A typical ETL process collects and refines different types of data, then delivers the data to a data lake or data warehouse such as Redshift, Azure, or BigQuery.

Top ETL Tools in 2022

1. IBM DataStage

2. Integrate.io

3. Fivetran

4. Oracle Data Integrator

5. SAS Data Management

6. Talend Open Studio

7. Azure Data Factory

8. Pentaho Data Integration

9. Hadoop

10. Stitch

11. Dataddo

12. Singer

13. AWS Glue

14. Google Cloud Dataflow

15. Informatica PowerCenter

16. Skyvia

Data Warehousing Tools 

The idea behind the data warehousing technique is to collect and manage data from varied sources to provide meaningful business insights to the user. A large amount of information is stored electronically by a business that, instead of transaction processing, is designed for query and analysis. Data warehousing is a process of transforming data into useful information and making it available to users for analysis. 

1. Amazon Redshift

2. Microsoft Azure

3. Google BigQuery 

4. MySQL

5. Snowflake 

Data Acquisition and Data Cleansing Tools 

1. Talend 

2. Mozenda

3. Octoparse

4. IBM Datacamp 

5. Go Spot Check

6. OnBase

Gartner Magic Quadrant

This Gartner report helps decision makers understand what’s needed to integrate complex data ecosystems and select data integration vendors that meet their specific needs.

ETL tools also makes it possible to migrate data between a variety of sources, destinations, and analysis tools. As a result, the ETL process plays a critical role in producing business intelligence and executing broader data management strategies.

Volumes of data can be extracted from a wide range of data sources, including:

• Existing databases and legacy systems

• Cloud, hybrid, and on-premises environments

• Sales and marketing applications

• Mobile devices and apps

• CRM systems

• Data storage platforms

• Data warehouses

• Analytics tools

WHY IS ETL IMPORTANT?

A lot of people ask, now that we’re in the cloud, why do we need ETL? Is it still important?

1. Productivity: It improves productivity with repeatable processes

2. Context: ETL helps businesses gain deep historical reference and context with data.

3. Accuracy: ETL improves data accuracy and audit capabilities for regulations and standards.

4. Consolidation: ETL provides a consolidated view of data for easier and improved analysis and reporting.

ETL History Continued

In the late 1980s and early 1990s, data warehouses came onto the scene. A distinct type of database, data warehouses provided integrated access to data from multiple systems – mainframe computers, minicomputers, personal computers and spreadsheets. But different departments often chose different ETL tools to use with different data warehouses. Coupled with mergers and acquisitions, many organizations wound up with several different ETL solutions that were not integrated.

Today in 2022, Core ETL and ELT tools work in tandem with other data integration tools, and with various other aspects of data management – such as data quality, data governance, virtualization and metadata.

ETL is closely related to a number of other data integration functions, processes and techniques. Understanding these provides a clearer view of how ETL works. 

Think about it:

1. SQL

2. Scripts  

3. Data mapping

4. Data quality   

5. Scheduling and processing  

6. Batch processing

7. Web services

8. Master data management 

9. Data virtualization

10. Event stream processing and ETL

Image Credit: Aaron Zhu

Don’t confuse ETL with a tool, it’s a concept of data movement which can be established with the help of various tools. Those tools keep changing as the MDS keeps pivoting to better ways of handling data science and business objectives.

I’ve even heard of ETL roles. An ETL Developer is usually a Software Engineer that handles the Extraction, Transformation, and Loading data processes by developing infrastructures to do this efficiently.

ETL is the Backbone of Business Intelligence

Finding ways and methods to harness information is the duty of the Data Engineering team and specifically the ETL Engineer who is also known as the ETL Developer.

Data Scientists, Data Analysts, and Data Engineers are employed to unearth this valuable information in the advent of the Cloud, Big Data and Data science as a whole.

This process is known as Extract, Transform, and Load (ETL) which is the backbone of Business Intelligence (BI) as the data can not be used in its raw format to get actionable information.

This data may be obtained from transactional applications such as Customer Relational Management (CRM), Enterprise Resource Planning (ERP), Relational Database, XML, JSON, third party, and others.

Defined standards and models are used in Cleansing, Mapping, and Augmenting the data to prevent bad and non-matching data in the designated repository.

The final stage of the ETL process is Loading, which is uploading the refined data into the repository location such as Data Warehouses where they are secured, shared across users and departments both within or outside the organization.

The Rise of Data Engineering

Data Engineering is the broad spectrum of the team issued with the responsibilities of obtaining raw data, developing infrastructures, building and testing the Data Pipelines to optimize a system for Analytical Purposes.

Many polls I’ve seen show that Data Engineers are more in demand than data scientists and other related terms and positions. That being said new terms and positions are constantly being refined and re-defined. Generally speaking:

The Data Engineering team is usually large and may comprise all or some of the following depending on the scope of the project to be executed by an enterprise:

• Data Architect

• Database/Warehouse Developer

• Database Administrator

• Data Scientists

• Business Intelligence Developer

• ETL Engineer

Anyways guys I hope you found this basic run-through somewhat helpful even if it’s mostly re-hashing what you already know. I'm curious about many educational aspects around data science and how the Cloud and the Modern Data Stack have changed how we do things.

If Data is the new oil in the age of A.I., what comes next?

I’ll leave you with an interesting way of looking at the world.

Hierarchy of Needs from the perspective of Data Science

Source: Monica Rogati

Leave a comment

Subscribe now

Hey Guys,

If the Modern Data Stack (MDS) is reshaping Data Engineering, Platform Engineering is becoming a newer and valuable role.

How do you become a platform engineer?

Platform engineers need a bachelor's degree in computer science or an associate's degree and relevant work experience.
...
The most similar professions to Platform Engineer are:

1. Software Engineer.

2. Software Developer.

3. Devops Engineer.

4. Cloud Engineer.

A platform engineer is responsible for the design, implementation and maintenance of a company’s technology infrastructure. They are often tasked with ensuring that all systems are running smoothly and resolving any issues that may arise.

In a sense this is why being a Data Engineering is becoming a better definition of a success role than being a Data scientist itself. Data infrastructure as a service is quickly evolving.

The Modern Data Stack is itself evolving as well to suit the supply-demands of organizations. A modern data stack is a solution that can help an organization save time, effort, and money. It is faster, more scalable, and more accessible than the traditional data stack.

Digital Transformation in the Data Age

One way to gain a competitive advantage is to have an efficient mechanism for deploying & operating applications in production that enables teams to provide features to end-users faster.

To achieve that, many companies realized that it helps to have a dedicated team that builds & maintains a common platform and constantly looks to innovate how applications are deployed and operated in Production. This practice of creating a common platform as a product is called Platform Engineering.

Platform Engineering is thus said to be that process of selecting frameworks -- i.e. whether to use third-party frameworks, rather than making your own, while also showing how to tie the frameworks together to build a platform that applications can be built on.

The rise of microservices, container orchestration, and the like have introduced novel engineering challenges. Platform engineering teams have formed at a number of organizations to shoulder these responsibilities.

At the same time, cloud-native organizations have radically changed how they’re organized, moving from large departments (development, QA, operations, release) to smaller, independent development teams.

Rise of Site Reliability Engineering (SRE) and Platform Engineering

SRE and platform engineering are spiritual successor of traditional operations teams, and bring the discipline of software engineering to different aspects of operations.

Site Reliability Engineering and Platform Engineering

Platform engineering teams apply software engineering principles to accelerate software delivery. Platform engineers ensure application development teams are productive in all aspects of the software delivery lifecycle.

Site reliability engineering teams apply software engineering principles to improve reliability. Site reliability engineers minimize the frequency and impact of failures that can impact the overall reliability of a cloud application.

There is truth in noting that the title, “Platform Engineer” is nothing but a new title. However, a number of factors are, and continue to, cause the traditional responsibilities of a Site Reliability Engineer (SRE) to shift.

New Principles:

• Platform as product.

• Self-service platforms.

• A constant focus on eliminating toil. As defined in the Google SRE book, toil is manual, repetitive, automatable, tactical work. The best SRE and platform teams identify toil, and work to eliminate it.

Therefore the evolution of the Modern Data Stack entails news ways software development engineers, data engineers and data science teams work together.

So, the role, “Platform Engineer” is considered by some to just be a different title for a job that has been traditionally performed by an infrastructure team.

Just as data engineers and data scientists may have a different skill set and slightly different roles, so too might a platform engineer with your average software engineer:

• Software engineer
  Specialist regarding developing software
  Often also called “software developer”

• Platform engineer
  Specialist regarding a certain platform, stack, etc…

As a Platform Engineer you will help to deploy, manage, fix and reinvent the tools, services and components that the network engineers rely on to automate our network and keep it operational.

• Deployment and maintenance

• Migration and automation

Though what a Platform Engineer is also somewhat subjective. Asking ten engineers this same question would likely yield ten different answers.

Think about it, Platform Engineers are applied specialists. Platform engineers commonly work in an IT department or as part of a larger engineering team. Their job requires them to have a deep understanding of computer hardware and software—they must be able to identify and resolve problems quickly and efficiently.

Roles and Job Duties of the Platform Engineer

Platform engineers have a wide range of responsibilities, which can include:

• Designing and implementing an automation framework for all aspects of the application lifecycle, including build, test, and deployment

• Performing operations tasks such as monitoring application performance over time or troubleshooting issues with production applications

• Monitoring network performance to ensure that the platform can handle increased traffic from new applications or services

• Designing, building, and maintaining the systems underlying an application platform

• Developing new technologies to support existing applications or creating new applications using new technologies

• Participating in meetings with executives to discuss technical issues and propose solutions

• Reviewing code written by other engineers to ensure it meets standards for quality, security, and performance

• Collaborating with other members of the engineering team to design new features or improve existing ones

• Consuming and analyzing large amounts of data in order to identify patterns and make predictions about future trends

Compensation

Source: Glassdoor

Platform Engineer Salary & Outlook

The salary of a platform engineer can vary depending on their level of education, years of experience, and the company they work for. They may also earn additional compensation in the form of bonuses or commissions.

• Median Annual Salary: $125,000 ($60.1/hour)

• Top 10% Annual Salary: $195,000 ($93.75/hour)

The employment of platform engineers is expected to grow much faster than average over the next decade.

• Many aspiring platform engineers will receive on-the-job training from their new employer after they are hired.

Share

Core Platform Engineer Skills

Platform engineers need the following skills in order to be successful:

Coding:

Coding is the process of creating software. As a platform engineer, you may be responsible for creating the software that supports the company’s cloud infrastructure. Coding skills can help you develop the software that keeps the company’s cloud infrastructure running smoothly.

Debugging:

Debugging is the process of finding and resolving errors in computer code. As a platform engineer, you may be responsible for identifying and resolving errors in the software of a company’s website or an app. Knowing how to debug effectively can help you save time and money.

Hardware knowledge:

Hardware knowledge is the ability to understand the components of a device and how they work together. This is an important skill for a platform engineer because it allows them to troubleshoot and repair computer systems. It’s also important for them to understand how software interacts with hardware.

Networking:

As an engineer, you may need to collaborate with other professionals in your field. Having strong networking skills can help you form connections with others in your industry. You can use your networking skills to find potential job opportunities, find other professionals to ask for advice or find other people who can help you learn more about your field.

Project management:

Project management skills can be helpful for engineers who work on large projects. This can include planning, organizing and overseeing the completion of a project. Project management skills can help you delegate tasks, set deadlines and track the progress of a project.

Platform development such as frameworks, operating system/kernel, Programming languages, Compilers etc demands considerable software skill. I hope this quick overview gives you a more realistic idea what Platform Engineering is and what it entails.

An organization’s platform engineers tailor infrastructure as code tools to the needs of the organization’s application developers.

Infrastructure as code

Infrastructure as code is one of a number of factors that have helped boost the notion of a platform engineer within the collective conscience.

Platform Engineering

Platform engineers constantly examine the entire software development lifecycle from source to production. From this introspective process, they build a workflow that enables application developers to rapidly code and ship software.

Site Reliability Engineering

Site reliability engineers create and evolve systems to automatically run applications, reliably. The concept of site reliability engineering originated at Google, and is documented in detail in the Google SRE Book.

Furthermore both DevOps and GitOps are a loosely codified set of principles of how to manage different aspects of infrastructure. The core principles of both of these philosophies -- automation, infrastructure as code, application of software engineering -- are very similar.

If you think of ROI ultimately Platform engineering is an emerging technology approach that can accelerate the delivery of applications and the pace at which they produce business value.

When we study software development, we might not have an inkling we might end up as a “Platform Engineer”. Yet times are changing. Platform engineering is an emerging trend intended to modernize enterprise software delivery, particularly for digital transformation. The engineering platform is created and maintained by a dedicated product team, designed to support the needs of software developers and others by providing common, reusable tools and capabilities, and interfacing to complex infrastructure. 

Read: What is the Modern Data Stack?

Thanks for reading! Give this post a like or a quick comment if you found the topic useful. I’m slowing getting into more data science topics on this Newsletter.

Subscribe now

Hey Guys,

I enjoy the definitions we use to try to explain how digital transformation occurs with our evolving use of data and machine learning that is occurring. There has been a lot of hype around “the modern stack” in recent years, so I thought I’d try to cover the topic.

The modern data stack (MDS) is a suite of tools used for data integration. These tools include, in order of how the data flows:

• a fully managed ELT data pipeline

• a cloud-based columnar warehouse or data lake as a destination

• a data transformation tool

• a business intelligence or data visualization platform.

Rapid innovation in cloud data technology and exponential growth in the number of new products and companies in the data and analytics space is likely changing the future of MDS.

The term modern data stack seems to have entered terminology in late 2019 and since the beginning of 2020 the term “the modern data stack” has gone more mainstream.

TL;DR: It refers to the new, best-of-breed data architecture, which is centered around a cloud data warehouse (or cloud lakes) and built to cope with massive amounts of data.

Ease of Use

The most important difference between a modern data stack and a legacy data stack is that the modern data stack is hosted in the cloud and requires little technical configuration by the user.

The Modern Data Stack (MDS) has been popularized for a couple of years but only recently has there been convergence on its definition. In short, if you use any of the following, you are likely to have the foundational piece of a Modern Data Stack: 

• Snowflake

• Redshift (AWS)

• Big Query (GCP)

• Databricks

• Synapse (Azure)

You can visualize the MDS as such, see above.

Designed for Scalability

Massive scalability: Another key aspect of modern tools is that they are designed for massive scale. Partly  due to the elastic nature of the cloud and partly because most of these modern systems are designed with distributed systems principles that allow for horizontal scalability instead of old desktop and single server applications (e.g. Snowflake, Databricks, ThoughtSpot.)

Composable Data Stack

Composable data stack: The idea behind a composable data stack is that each product behaves more like a configurable component into a larger architecture, rather than its own island.

MDS is Replacing TDS

The traditional data stack (TDS) is failing to deliver and keep up with the data demands of any modern organization. To maintain a competitive advantage, organizations need data that they can act on at the right time as well as being flexible enough to adapt to changes. A TDS typically refers to on-premise Hadoop (ecosystem) and SQL warehouses that are both logically coupled and complex.

The MDS is elegant and it enables new solutions and a more sophisticated DataOps future.

The traditional data stack (TDS) is dying mainly due to:

1. Long turn-around time to untangle and set up infrastructure

2. Slow response to new information

3. Expensive journey to insights

So what is the main result of the MDS replacing the TDS?

Automation

Automation: Once user workflows are represented in code, it is fairly easy to automate them. This allows the tool to be leveraged at a much bigger scale than previously possible.

Innovation Through Agility

You can tell how innovative a company is by measuring how long it takes them to go from an idea to validation (or invalidation.) If you look at the difference between the companies that are doing fast-paced innovation and their stagnant counterparts, it is not so much that the latter lacks people with great ideas. It’s that the former has a much better environment for supporting innovation.

It is thought by advocates then that the MDS augments innovation and therefore profitability.

The Narrative of DataOps entering the MDS Era

So if Data science and DataOps were a story the 2020s are quite interesting in that driven by demand from businesses seeking greater value and more control over their customer data, a new approach to data analysis and activation has emerged and become the “go-to” approach for building a data architecture. 

The Modern Data Stack (MDS) is centered around an ecosystem of tools businesses use to collect, move, store, transform, analyze, and operationalize their data. 

I think MDS is a precursor to more unified DataOps platforms that can do more with less (i.e. more no code interoperability).

How Fast is the MDS Growing in Investment and Adoption?

Rise of CDPs

A customer data platform is a collection of software which creates a persistent, unified customer database that is accessible to other systems.

Some believe that in the evolution of the Cloud, CDPs were the bridge to the MDS. Think about it, CDPs are often a foundational piece of an MDS. CDPs make integrations, one of the most challenging parts of building a traditional data stack, easy. This easy integration has allowed tools that run on customer data to flourish and an ecosystem of data tools to form.

More areas of the MDS will integrate with all-in-one solutions. For instance in marketing we have all-in-one marketing platforms – such as Salesforce Marketing Cloud and HubSpot Marketing Hub – have become popular over the last 10-15 years. But what will be the all-in-one ML and DataOps winners of the future? I don’t know, but I find it quite interesting.

Last Thoughts on Benefits of MDS Adoption

• Speed and scale of data. Since MDS tools are almost always managed SaaS, they can scale almost infinitely.

• Easier and faster implementation. It doesn’t take weeks or months to add a new data system anymore. You don’t have to provision infrastructure or deploy software.

• Reduced cost and maintenance. MDS tools generally have consumption-based pricing. You only pay for what you use.

• Easier to build on top of. MDS tools are built to plugin with other tools. They frequently offer out-of-the-box integrations with popular cloud services and platforms, so stitching together low-or-no-code systems is easier.

• Solve bleeding edge use cases. MDS tools give your digital business a competitive advantage by giving you faster and greater value out of your customer data.

In short I think it’s obvious that a modern data stack (MDS) is a solution that can help an organization save time, effort, and money. It is faster, more scalable, and more accessible than the traditional data stack.

What do you think?

Leave a comment

Read Datascience Learning Center in the Substack app

Available for iOS and Android

Get the app

Subscribe now

Did you like this exploration of the basic concepts? Give it a like so I know and I’ll write more posts like this.

Image source: Incorta

Hey Guys,

This is Data Science Learning Premium,

Benjamin Rogojan, the “Seattle Data Guy” is becoming increasingly a good source of information for data engineers and data scientists, he also has a stellar YouTube channel.

In a recent LinkedIn post he really perked my interest.

On the Modern Data Stack

This is a direct quote:

Piecing together a “modern data stack'' is far from the only option.

The “all-in-one data stack” is starting to come back into vogue. I have discussed some of these tools both with clients as well as VCs who have been paying attention to the recent customers who are being acquired by these new all-in-one data solutions.

In my newsletter this week I will be diving a little deeper in some of these options including Rivery, Keebola, Mozart Data, Nexla and Incorta.

Is the Era of All-In-One Data Stack Coming?

He cites companies such as Rivery, Keebola, Mozart Data, Nexla and Incorta. He even created a graph about them:

I’m very interested in the new data platforms that are coming. I’m forever thinking of how Snowflake, Databricks, Qubole, Vertica Systems, Datadog and so many others will evolve.

So what about the new breed of all-in-one data stack companies?

The Modern Data Stack (MDS) is centered around an ecosystem of tools businesses use to collect, move, store, transform, analyze, and operationalize their data. 

Ben in his LinkedIn post goes on:

The modern data stack was all the rage in 2020 and 2021, but in the late parts of 2021 and most of 2022 people started to ask questions.

Is the modern data stack even modern?

Isn’t it just a piecemeal of components from solutions we have known forever like SAP or Informatica.

Isn’t it just an unbundled version of Airflow?

Clearly there is a sense that how companies use data and do machine learning will keep transforming even as digital transformation, software and A.I. continue to impact society, businesses and how we become a data-based civilization.

Let’s quickly review the companies Ben has identified ordered by their current funding levels:

Incorta

Incorta provides a unified data and analytics platform that makes it quick and easy to unlock the full potential of data from multiple complex source systems. By eliminating traditional data transformation, modeling and aggregation steps, it makes 100% of data instantly ready for analysis. Pre-packaged data apps for common business applications make it easy to prepare data for analysis.

It’s a startup founded by former Oracle executives who want to change the way we process large amounts of data.

Founded: 2013

CEO: Osama Elkady

A typical data project involves ETL (extract, transform, load). It’s a process that takes data out of one database, changes the data to make it compatible with the target database and adds it to the target database. A lead venture capitalist said: Incorta is poised to upend the data warehousing market with innovative technology that will end 30 years of archaic and slow data warehouse infrastructure.

Rivery

Unify your ELT pipelines, workflow orchestration, & data operations with Rivery's complete SaaS platform.

• Whether you’re building out your data stack or transitioning to the cloud, managing your data workflows to analyze your business can be a real challenge.

• Developing an in-house solution requires valuable resources and upkeep, while integrating several tools adds new layers of complexity.

• Rivery’s SaaS platform provides a fully-managed solution for data ingestion, data transformation, data orchestration, reverse ETL and more, with built-in support for your data operations development and deployment lifecycles.

• Designed to be nimble for non-technical users and with advanced capabilities for experts, Rivery enables you to manage data workflows as the foundation of a modern data stack.

Founded: 2018 (New York with strong Israeli workforce presence)

CEO: Itamar Ben Hemo

Discover how Rivery enables data innovators from different industries to save time, automate processes, and unlock greater value from their data.

Rivery considers itself a DataOps Platform. Rivery is a SaaS DataOps platform that gives companies control over their organizational data through the ingestion, transformation, and orchestration of data processes. It was named “one to watch” by Snowflake’s modern marketing data stack.

Mozart Data

The modern data platform empowering anyone to easily centralize, organize, & analyze their data without engineering.

Mozart Data is the fastest way to set up scalable, reliable data infrastructure that doesn’t need to be maintained by you. Mozart Data’s all-in-one modern data platform empowers anyone to easily centralize, organize, and analyze their data without engineering resources.

Setting up a stack of data tools is a difficult exercise for any startup to undertake, involving picking and choosing among a wide variety of tools and approaches. Mozart Data’s founders have a background in data wrangling and have built a platform of tools that makes the decisions for you, especially designed for startups.

Being data-driven has never been easier

Your full-service modern data platform for centralizing, organizing,
and analyzing your data. Get set up in an hour and start working with
our data analysts.

• Mozart has built the pieces into the stack, while in others it recommends what it considers best-in-class as it has done by selecting Snowflake as the default choice for a data lake on the Mozart platform of tools.

• I’m a bit excited by Mozart Data’s no-code side of things. Say goodbye to exporting CSVs of data and building countless pivot tables. Connect your data sources and pull data automatically with reliable no-code integrations.

Founded: 2020

CEO: Peter Fisherman

Their unique value proposition seems really useful. The idea is to have a data stack in a box of sorts that removes the complexity for startups that just want to get going quickly with a data platform.

Nexla

A company in Unified Data Ops. Nexla is a leader in unified data operations and a 2021 Gartner Cool Vendor.

Their platform makes it simple for anyone to create scalable data flows. Teams working with data get a no/low-code unified experience to integrate, transform, provision, and monitor data for any use case. Data users with varying skill levels work collaboratively to create ready to use data products. Organizations get zero-friction, governed, and agile data operations.

Nexla works with larger companies that work with data like Doordash. Nexla’s initial customers have been larger companies. The goal with this money is actually to expand the market by offering it to smaller organizations.

Founded: 2016

CEO: Saket Saurabh

DATA ENGINEERING AUTOMATION

Ready-to-Use Data for Everyone

Reliable, consistent, and trusted data products in the hands of every user, in the applications they use.

It’s never been easier to create and run high-performance, automated data operations for your company. With their Data Product approach anyone can discover, integrate, transform, deliver, and monitor all data, regardless of format and velocity –  batch, streaming, real-time.

• Unifying integration, preparation, and monitoring whether you need ETL, ELT, Reverse ETL, API Integration, or Streaming. CLI, SDK, and API for developers. No-code for everyone else.

• Automated connectors and data products. No code, no wait. We start at connectors that auto-generate from configuration, and end with ready-to-use data products that bring data into the application you use in a format you need.

The market particularly in machine learning, has been heating up and the company has been seeing traction with customers like DoorDash, Instacart, Poshmark and Freshworks, among others.

Keebola

They are a data stack as a service company.

Keboola is a Prague based cloud-based data platform that helps clients combine, enhance and publish crucial information for their internal analytics projects and data products in a quick and easy fashion.

Operating and servicing clients for 8 years, they help a wide array of businesses from financial, travel, hospitality, retail and gaming industries, we help them significantly reduce or eliminate:

Time spent on repetitive maintenance tasks

• Adoption time and learning curves needed for outdated systems

• Drawn-out menial responsibilities which detract from efficiency Building on knowledge of programming that is available in the market such as SQL, R, Python etc…., we allow clients achieve unparalleled time to value ratio with all Keboola Connection implementations. Majority of our customers are completely self-serving from the inception of their project. We partner with professional services companies to build the right solutions for our clients. Our Developer partners build apps that seamlessly integrate their services and algorithms into our structure, making functions like predictive analytics and machine learning instantly available to our customers with no integration work required. We cast technology alliances with platforms that help our clients consume the data and insights in the ways most suitable for their particular use cases.

Founded: 2008

CEO: Pavel Doležal

Automate and analyze

Keboola enables data engineers, data analysts, and analytics engineers to collaborate on analytics and automation as a team, from extraction, transformation, data management, and pipeline orchestration to reverse ETL.

Anyways guys I hope you enjoyed the brief summary, I’m going to start digging into these kinds of companies more.

Subscribe now

Hey Guys,

You have stumbled on Graph Neural Networks somehow and now you’re interested in using it to solve a problem?

Here is an Email resource page on this. This is not a sponsored post, but I wanted to really highlight what’s going on here that’s relevant to the future of graph neural networks and how they may become more mainstream.

PyG (PyTorch Geometric) is a library built upon PyTorch to easily write and train Graph Neural Networks (GNNs) for a wide range of applications related to structured data.

The Latest release PyG 2.1.0 Graph is out!

PyG is the most popular library for Graph Machine Learning. The new functionality includes principled aggregations, link-level and temporal samplers, data pipe support, and many more features!

www.pyg.org

See on GitHub

PyG is a library built upon PyTorch to easily write and train Graph Neural Networks for a wide range of applications related to structured data.

PyG is both friendly to machine learning researchers and first-time users of machine learning toolkits.

It consists of various methods for deep learning on graphs and other irregular structures, also known as geometric deep learning, from a variety of published papers.

In addition, it consists of easy-to-use mini-batch loaders for operating on many small and single giant graphs, multi GPU-support, DataPipe support, distributed graph learning via Quiver, a large number of common benchmark datasets (based on simple interfaces to create your own), the GraphGym experiment manager, and helpful transforms, both for learning on arbitrary graphs as well as on 3D meshes or point clouds. Click here to join our Slack community!

1. PyG utilizes a tensor-centric API and keeps design principles close to vanilla PyTorch. If you are already familiar with PyTorch, utilizing PyG is straightforward.

2. Spend less time worrying about the low-level mechanics of implementing and working with Graph Neural Networks. All it takes is 10-20 lines of code to get started training a custom GNN model.

3. PyG covers a large number of state-of-the-art GNN architectures and training and scalability procedures, and can easily be extended to fit your specific use-case or conduct your own GNN research.

4. The new GraphGym API in PyG can help users easily reproduce GNN experiments, launching and analyzing thousands of GNN experiments, and registering customized modules to the GNN learning pipeline.

And as well:

Check out the PYG documentation here.

PyG is a library built upon PyTorch to easily write and train Graph Neural Networks for a wide range of applications related to structured data. PyG is both friendly to machine learning researchers and first-time users of machine learning toolkits.

At its core, PyG provides the following main features:

• Data Handling of Graphs

• Common Benchmark Datasets

• Mini-batches

• Data Transforms

• Learning Methods on Graphs

• Exercises

See the Discussions on PyG on GitHub here.

The 2.0 update was in September, 2021.

Now just 11 months later we have 2.1.0

Latest release PyG 2.1.0 is out! The new functionality includes principled aggregations, link-level and temporal samplers, data pipe support, and many more features!

There’s over 2,000 people on the Slack channel. The team appears to have a major Stanford connection. PyG is available for Python 3.7 to Python 3.10.

I believe some of the founders of PyG are also involved in Kumo.AI. The backbone of the founding engineering team at Kumo.AI is from Stanford.

Some of us believe graph neutral networks could become more important. In brief, a Graph neural network is a class of artificial neural networks for processing data that can be represented as graphs.

According to Wikipedia, in the more general subject of "Geometric Deep Learning", existing neural network architectures can be interpreted as GNNs operating on suitably defined graphs.

Thanks for reading!

If you want to support the channel and allow me to continue to write Newsletters feel free to get access to more content.

Subscribe now

MON AUGUST 15TH, 2022 11:40 AM MONTREAL, CANADA

Hey Guys,

Just as there is Databricks vs. Snowflake, there is DevOps vs. MlOps. While I’m not a technical person, I often find myself thinking about this.

For software developers this is already rather intuitive:

DevOps methodology helps improve communication between your developers and ops working on projects. It best serves the following purposes:

• you can launch new features faster

• increases the customer’s satisfaction and of developers too at the same time.

• feedback loops help better communication

Key principles of DevOps:

• Automation

• Iteration

• Self-service

• Continuous improvement

• Continuous testing

• Collaboration

Machine Learning Operations (MLOps)

If you think of how all this plays out in the real world, there appears to be a lack of a good bridge between DevOps and MLOps. Correct me if I am wrong?

Leave a comment

AI has been heralded as the new “brains” for software applications, a role long held by databases. Think about it, ML models depend on specific combinations of hardware and software infrastructure. Without the right infrastructure, the models either cannot perform well enough to be viable or, in some cases, become prohibitively costly.

According to Databricks, MLOps stands for Machine Learning Operations. MLOps is a core function of Machine Learning engineering, focused on streamlining the process of taking machine learning models to production, and then maintaining and monitoring them. MLOps is a collaborative function, often comprising data scientists, devOps engineers, and IT.

How DevOps and MLOps operates together seems to be a bit lacking. There’s a lot of wasted inefficiency.

Today there is no efficient bridge between the creation of ML models and the process of getting them into production. To illustrate this: The average time to production for ML models is 12 weeks. That’s 4 months, it’s not ideal.

The MLOps loop can be complicated with some bottlenecks along the way: data collection, data processing, feature engineering, data labeling, model building, training, optimizing, deploying, risk monitoring, and retraining. And in each organization, different people and teams may own one or more steps.

Why AI Falls Flat

What’s worse, nearly half of the models are shelved for performance or cost reasons, which makes AI less transformational than many hoped. Organizations have to think better about how to integrate DevOps and MLOps, and what tools can help?

I’m sometimes reading SeattleDataguy maybe one of the best Substack’s on data science right now in 2022:

SeattleDataGuy’s Newsletter

Learn About End-To-End Data Flows (Data Engineering, MLOps, and Data Science)

This is more his realm of expertise.

Clearly in the real world reasons why A.I. isn’t so transformative have to be dealt with head one. If AI is to be the “brains” of applications, a world where ML models are heavily specialized, requiring unique and customized workflows and tools is problematic.

Companies like Snowflake and Databricks are looking to create easier access to applications, machine learning models, and dashboards through their data marketplaces. They want to be your data platform, not your data warehouse or lakehouse. - Seattle Data Guy

One of the reasons I like Seattle Data guy is because he’s also often a guest on YouTube podcasts, I find this supplements his Substack and LinkedIn posts well. In case you are wondering who this guy really is, it’s Benjamin Rogojan.

Ben on what is Data Science

Ben Rogojan is a data engineering solutions architect with expertise in data architecture and statistics. He focuses on developing end-to-end data solutions that help take data from raw format into data products and analytics.

Ben has nearly 50k followers on Medium. I believe he does consulting as well. I view him as definately a pioneer of Substack’s data science community as well. On this LinkedIn, he says he talks about #bigdata, #datainfra, #datascience, #dataengineering, and #datawarehousing. LinkedIn has an incredible data science community (check out my list). I recommend you super-follow (tap on the notification bell) all of the people on this list.

MLOps Cycle

For developing machine learning solutions the standard lifecycle goes like this:

• Requirement gathering

• Exploratory data analysis

• Feature engineering

• Feature selection

• Model creation

• Model hyperparameter tuning

• Model deployment

• Retraining, if needed

The fact is once an ML model is trained and ready, we should be able to work with it as we do with any other software module because it is just code and data.

The theory goes that since DevOps came first, MLops has to integrate better with it and its loop cycle. It still seems to lack a good bridge.

As you know, MLOps originated as a term to refer to a set of best practices to design, build, deploy and maintain machine-learning models in production. As it evolves, however, the scope has expanded to the whole of ML lifecycle management.

It’s no surprise the Blog of Databricks often mentions MLOps.

So the current reality is sub-optimal at most organizations. Siloed teams of data engineers, data scientists, IT ops professionals, auditors, business domain experts, and ML engineering teams operate in a patchwork arrangement that bogs down the process. It’s not good. This means A.I. isn’t being implemented properly.

According to some ML Engineers, when model creation and model deployment are forced together into one mega-process, however, it usually limits flexibility and choice in a way that creates obstacles. Organizations clearly need to re-vamp how they integrate their DevOps, MLOps vis-a-viz model creation as distinct from model deployment. I don’t know what the answer is, but these problems are unique to each organization and to the field as a whole.

Databricks vs. Snowflake

In some sense I view the Databricks vs. Snowflake debate also as symbolic. Snowflake is a relational database management system and analytics data warehouse for structured and semi-structured data.

Again, I’m not an engineer. Both are incredible companies. With enterprises large and small racing to build out their data infrastructure, one foundational piece these enterprise companies all need is an easy place to store their data.

Databricks, has auto-scaling of clusters but is supposedly not so user friendly. The UI is more complex as it is aimed at a technical audience. It requires more manual input when it comes to things like resizing clusters, updating configurations, or switching options. There is a steeper learning curve to overcome.

Databricks, which innovated what is called a data lake, a place where you can dump all of your data – no matter the format. This is super convenient.

Some Terms

• A data warehouse is the database of choice for general-purpose analytics, including reporting, dashboards, ad hoc, and any other high-performance analytics.

• A data lake is a data store (only) for any raw structured, semi-structured, and unstructured data that makes data easily accessible to anyone. You can use it as a batch source for a data warehouse or any other workload.

• A data lakehouse is often described as a new, open data management architecture that combines the best of a data lake with a data warehouse. The goal is to implement the best of a data lake and a data warehouse, and to reduce complexity by moving more analytics directly against the data lake, thereby eliminating the need for multiple query engines.

In reality in 2022, I think many companies use Databricks and Snowflake together, so they aren’t really direct competitors per se. That being said they are rising Giants that are overlapping. Functionally, Databricks and Snowflake have been steadily moving into each other’s core markets - ETL and data processing, and data warehousing/lakehousing - for some time as they both try to become a data platform of choice for multiple workloads.

I think overtime Databricks and Snowflake will create a better bridge between DevOps and MLOps, among others. This will reduce friction between A.I. model creation and model deployment, thereby reducing cost and improving efficiency making A.I. easier to implement in the real world.

On the business side, I cannot wait for Databricks to go public with an IPO. Snowflake has a lot of great momentum. Incredibly it already has a market cap of $54.3 Billion, with gross margins of 64%. By the time it goes public, it could be worth approximately what Snowflake is worth or maybe a little less. Databricks is worth around $38 billion following its latest fundraise of $1.6 billion in August 2021, led by Counterpoint Global.

How do you see DevOps and MLops evolving together and the data science community forming on Substack?

Thanks for reading! If you want to support the channel and allow me to continue to write Newsletters feel free to get access to more content.

Subscribe now