Understanding the Job Market to Prepare for the Future of Work with Fork
During July and August 2021, Harvard Computer Society Tech for Social Good worked with Fork which is a startup incubating out of Harvard Innovation Lab that is on a mission to advance the careers of 10 million individuals by 2030. It is doing so by creating a career planning application for employees vulnerable to labor disruptions, such as due to automation and/or the pandemic, to enable them to grow into higher wage and demand roles across the span of their career.
Problem
Underemployed employees, who are often in lower wage service roles are more vulnerable to labor market disruptions like COVID and automation. That is why it is important to support blue-to-white collar transitions by providing these workers both the job pathway recommendations along with broader education on how to plan a career which often these workers do not get elsewhere unlike those in higher wage professional starting roles. A big part of this is understanding the relevant skills for different jobs so one could learn more and become more competitive on the job market as a result of this. However, current market alternatives that provide analysis of skills by job titles are limited and sometimes out of date.
Solution
Harvard Computer Society Tech for Social Good developed an algorithm that leverages natural language processing tools to parse through job descriptions and detect keywords associated with skills and other markers that are relevant to the job market. It was built on top of multiple open source job skills libraries that covered over 4000 skills relevant to numerous job fields. The solution also provided options to determine if a job description text would indicate remote work, undergraduate degree, and travel requirements.
The team used libraries such as nltk, spacy, and pandas to conduct data cleaning, text manipulation, and pattern recognition. For text preprocessing, the biggest emphasis was on methodologies of lemmatizing and sentiment analysis. The biggest challenge was the complexity of human language as such tools assume clear structure and unified use of words which is not always the case for human produced text. Some other aspects to be targeted were words that were commonly used in a specific context within a job description but which in the sphere of skill detection have another meaning. For example, the phrase health insurance in job descriptions was often mentioned as a benefit the employer offers not to imply that the job is in the insurance field. Those challenges made the development process fascinating and provided opportunities to improve the algorithm at every step of the journey.
Reflections
This project provided a great opportunity to support the reallocation processes of jobs which is now more important than ever due to the growth in unemployment as a consequence of the COVID-19 pandemic. The team enjoyed working with Fork and leveraging state of the art text preprocessing and manipulation tools to build a generalized solution whose applicability is not limited to certain types of job descriptions. Cooperation with Fork also proved that technological solutions offer numerous ways to tackle different challenges within the sphere of social improvement.