...

How to Address the Digital Skills Gap to Build a Future-Proof Tech Workforce

Márton Hubay
May 20, 2023
Read: 7 min

If an organisation is to scale, it needs a data and cloud related talent strategy. A bold statement, I know, so let us look into its main points one by one first.

Boston Consulting Group reports that “[o]nly about 30% of companies navigate a digital transformation successfully.” Additionally, in their article “Lack of Skills Threatens Digital Transformation,” Gartner quote a TalentNeuron study that 53% of respondents have identified “the inability to identify needed skills was the No. 1 impediment to workforce transformation”.

When it comes to data and cloud specifically, these are future skills of the highest demand in the future also reflected by multiple reports, such as Randsdad’s Digital Skills: Unlock Opportunities for All and Gartner’s Critical Digital Skills to Accelerate Digital Transformation.

The talent development direction that the tech world has taken is quite clear and I am not going to go into the whys of it. Instead, I would like to discuss how organisations can build a future-proof tech workforce strategy. And it all starts by addressing the digital skills gap.

Measuring the cloud industry

Many organisations try to measure the cloud industry boom through estimations and forecasts about the data volume generated globally. We should always take such reports with a pinch of salt because of their measuring approach which tends to be more of a philosophy than actual measurement.

The International Data Corporation (IDC) even goes as far as talking about a “digital universe” and offering a metaphor on how universes expand. Their reports state that “data is doubling every two years” and the “global data volume will reach 175 zettabytes by 2025”.

Indeed, 175 ZB is an unfathomably large volume and although it is hard to imagine what we could do with that much information.

Yet, this certainly impacts the market. To begin with, the more data, the more needs and expectations from the business. More data generates more ideas and use cases. It then generates innovations and improvements on the product level, further increasing the competition on the organisational level.

Finally, it forms and transforms into industry trends and standards, constant dynamics and waves. As with other aspects of life, the only constant in the recent software engineering market is change.

How do you prepare for change?

The good news is that we do not have to reinvent the wheel; we can just listen to our good old “influencers” on how to deal with change. Kent Back did the homework for us when he summarised the most important lessons in 2004.

In his book Extreme Programming Explained: Embrace Change, he suggests that the software engineering essentials are values, principles and practices.

I have always imagined the way these three relate to one another and to change as the structure of a planet:

preparing for change in the tech industry

At the core, we see the values which are the least exposed to the corrosion that changes may cause. The nearer you get to the surface, the higher the mutability with technologies being prone to change the most.

Oddly, when we think about expertise, seniority or hiring, we are often biassed towards tech. However, this is the most volatile component and the one that is the easiest to learn and adapt to.

A quick glance to compare the technology landscape between 2012 and 2023 shows that the entire tech industry has changed dramatically.

Companies whose talent development strategy has focused entirely on a specific tech stack have suffered immensely as many of the tools have become obsolete in a decade. Moreover, the industry has been aflood with mergers and acquisitions, which often affect support and necessitate migrations.

Surely, a migration should not cause a company to sweep their entire engineering team. And it would not if the talent development strategy focused on the right components, viz. the ones at the core.

Practices

Values and principles are the compass that helps teams to navigate uncertainty and enables sustainable decision-making.

But if they are so important, how do we select them?

Engineering values and principles should overlap with company values or derive from them. While they are essential, they are also a vast topic that we are going to cover in a separate article.

When it comes to practices, however, they are much closer to technical terminologies and technical life than core components.

They are guidelines that govern technical design, architecture, software engineering lifecycles, standards and development environments.

If we take a high level view of software engineering, we can clearly see that application engineering as the mainstream branch is far more mature than its subbranches. This is a natural course and the result of the fact that the mainstream branch has the longest history and the largest community.

Data and AI subgroups on the other hand are the youngsters in the family with quantum computing being the toddler. Yet, new disciplines adopt their practices from mainstream ones and this can make the former resilient.

Let’s look at some examples.

Software engineering practices in data

Those of you who have trading experience are familiar with the concept of leading indicators. These are signals used to predict certain future events typically in an economic context.

When I am looking at the timeline of data’s adoption of some practices, I cannot help but think about leading indicators. I would even go on to refer to the current stage as the software engineering practices adoption era.

software engineering practices adoption lag in data

Test-driven development (TDD):

  • In software engineering, it started back in 2002 with Kent Beck’s book Test Driven Development: By Example;
  • In data engineering, the tool that enabled data engineering to adopt this practice was dbt with its release in 2016.

Continuous integration / continuous deployment (CI/CD):

Infrastructure as Code (IoC):

  • Puppet was founded in 2005, AWS Cloudformation was released in 2011 and Terraform came about in 2014;
  • Probably it is a good enough assumption that severalyears after the main product release from the data vendors, they also release the respective terraform module. Both Snowflake and Databricks started in 2020.

Domain-driven design (DDD): where we should note this in an approach rather than a practice

The practices I described above we introduced over 20 years ago. Enough time has passed for the data industry to learn its lessons and even start measuring performance.

The DevOps Research and Assessment (DORA) programme reports that the practices mentioned above have been contributing the most to the success of software engineering projects.

As I have summarised above, in data, we can see a lag, which spans from a couple of years in some areas to a significant period in others. Despite this lag, the very process of adoption of said practices has begun. There isn’t any report in data, such as DORA in the “general” software engineering, that would show what are the success criteria for data-specific projects, but why would it be that much different?

How to use the newly adopted practices in talent strategy?

Talent Strategy consists of the following components:

  • Recruiting and retaining talent;
  • Learning and talent development;
  • Performance management.

What these three pillars have in common is the methodology to determine seniority at a company.

Every discussion around talent development revolves around the same questions: What are the requirements for the levels we are hiring for? How does the company empower the employee to become more senior? What are the assessment criteria for the level in question?

The incorrect seniority calibration (a.k.a. Progression framework) can spawn the wrong culture, inequity and churn, all of which are the opposite of future-proof.

This is where practices come in. If we integrate software engineering practices in the progression framework’s criteria system, we are going to have successful formulas in place.

This will effectively prevent seniority being equated to knowing the technical documentation of a product by heart or having spend 15 years using a specific tool.

Talent strategy components

Here is how we can successfully leverage the progression framework in the three main areas:

Recruitment

The more overlap we have between recruitment criteria and progression framework, the less friction we are going to see later on. Recruitment should be an ultrafast edition of the (past) performance assessment. If we do not synchronise it, we can easily misjudge the seniority and the maturity of the person, which not just makes the onboarding for new joiners difficult but also causes frustration among the internal employees.

Performance management

When it comes to seniority, a fair measurement involves everyone being assessed against criteria of genuine contribution to productivity on the product development or service delivery side. Here, inequities can cause compensation differences and make people even more frustrated. Of course, a performance framework is comprised of many components, and here I have not covered the company’s cultural and organisational aspects.

Learning and development

Placing the progression framework at the heart of the learning and development strategy helps employees adopt the right skills that also enable them to advance their careers more effectively. It shows guidance on where and how the employee can have the biggest impact on the organisation.

Here is a pragmatic example: Company X has a long-term goal to build a future-proof engineering team. Based on this, the talent management strategy emphasises the importance of multiple, proven software engineering best practices such CI/CD. As a consequence, CI/CD is allocated significant weight in the progression framework, which is transparent to the engineers.

Engineers are aware that the more they follow the progression framework, the more impact they will have on the organisation. So they prefer to learn CI/CD as opposed to a new programming language and they try to apply the practice as much as they can on their projects. As a result, they deliver high-quality products, they focus their time on future-proof initiatives and ultimately they make a bigger impact on their company than they would have if they had focused on something else.

Let’s talk

Setting up the right foundations is essential for a long-term, future-proof talent strategy. The data industry has started adopting some software engineering practices and it is important to become and remain conscious of them. The success of those practices is already proven, as well as their robustness against changes.

Integrating these practices into the core of the talent development strategy will contribute to a consistent and equitable company culture, which is essential for building a future-proof tech team.

If you are ready to start talking about your talent development strategy, get in touch.

Meanwhile, make sure to read up on how an effective tech strategy helps improve diversity in data.

More on the topic

Everything we know, we are happy to share. Head to the blog to see how we leverage the tech.

Multitable SCD2 joins
Multitable SCD2 Joins: How to Unify Historical Changes
In the realm of data management, historical changes are conventionally stored in separate Slowly Changing Dimension Type 2 (SCD2) tables. However, extracting point-in-time insights from...
September 25, 2023
VaultSpeed Sapphire Partner
Infinite Lambda Becomes VaultSpeed Sapphire Partner
We are thrilled to share the exciting news that Infinite Lambda is now recognised as a VaultSpeed Sapphire partner. This development is extremely important to...
August 9, 2023
dbt Labs Premier Consulting Partner
Infinite Lambda Becomes dbt Labs Premier Consulting Partner
We are delighted to announce our new Premier Consulting Partner status with dbt Labs, marking a significant step in our successful collaboration. As two pioneers...
August 8, 2023
data contracts explained for non-tech audiences
Data Contracts for Non-Tech Readers: a Restaurant Analogy
Introducing the data kitchen Data contracts have been steadily gaining in popularity within the data community, providing a structured approach to manage and regulate the...
July 17, 2023
dbt Data Quality Tests Implementation
dbt Data Quality Tests Implementation
Data quality definition In this blog post, I am going to show you how to leverage dbt data quality tests to build robust pipelines that...
June 20, 2023
composability in data platforms
Towards Greater Composability in Data Platforms
This post is based on a talk of the same name that the author delivered at Data Mash #7 – London Edition in March 2023....
June 15, 2023

Everything we know, we are happy to share. Head to the blog to see how we leverage the tech.