The debris of war is more than landmines and explosives. Its remains permeate communities for years, continuing to destroy lives and elongate the tail of conflict. One of the most significant factors in making a difference is ridding the world of uncontrolled Small Arms and Light Weapons (SALW) within crisis zones and security challenged areas. In Africa alone, it is estimated that there are over 100 million uncontrolled SALWs.
The effect of this is multi-faceted, and not limited to continued fighting. People are displaced, fleeing their homes through fear of injury and death. The risk of Gender Based Violence (GBV) rises alarmingly, with 45.6% of women in Africa being subjected to GBV as a result of armed conflict. And finally, the predominance of uncontrolled weapons erodes social cohesion and communal trust, tipping the balance towards violent confrontations every day.
Just before the close of 2020, we were fortunate enough to be introduced to the HALO Trust. HALO is a charity that has, for over thirty years, been clearing the world of landmines and explosives left behind by war. They have saved countless lives, created safe environments for children to grow up in, and given whole communities the opportunity to rebuild. However, conflict is complex, ever-changing and HALO knows it needs to change with it.
“The use of machine learning and artificial intelligence has great potential to support the developing world where the lack of digital data hampers the delivery of humanitarian assistance and development. We ought to remain innovative to continue
delivering value for money to our donors and beneficiaries”
Luan Jaupi, Head of ICT, HALO
“Our vision when starting Infinite Lambda was to make data skills and data solutions more accessible to an increasingly more data-dependant world. We firmly believe that by doing so, we empower the world to be more data-driven, and a better place to live.”
Nas Radev, CEO, Infinite Lambda
This quote is taken straight from our company handbook. It’s not just an aspiration or a soundbite but a firm belief that the expertise within our company has the potential to make a positive and enduring impact on life. As a data and cloud consultancy, we don’t naturally collide with some of the gravest challenges of our time. However, we believe we should if we are to make change happen.
As a fast growing data and cloud consultancy, conflict, disaster and geopolitics are not something we are used to discussing. HALO helped us understand the challenges they face in seeking out evidence of uncontrolled SALWs.
The NGO currently applies a manual process that involves searching a wide variety of sources online that indicate whether arms have either been seized or stolen. This process produces high quality incident reports, but is limited in its scalability due to the heavy reliance on experts, number of incidents and the geographical reach required.
As we learned more, we began to see where we could help design data-driven ways to assist the HALO experts in identifying arms diversion globally in a rapid and scalable manner. With a team of data engineers and data scientists, we began to explore ways to extract real-time information from multiple news and social media sources. The tool we created allows the user to search multiple sources with multiple search terms and geographies relevant to SALWs. Moreover, the user can schedule automated searches across these sources according to requirement.
We then developed machine learning algorithms to further enhance the accuracy, ensuring only the most relevant information is presented. Once the searches are complete, the raw data is then fed through the ML engine in order to refine the results with articles of relevance. Results are then displayed in order of relevance, allowing the user to verify the results they see as relevant or not. This human intervention completes this learning loop and allows the tool to become increasingly accurate.
This tool has the potential to seek out uncontrolled arms and assist HALO in reducing the numbers of SALWs worldwide, which will ultimately help save lives and livelihoods. The techniques we have used blaze the trail to finding new ways to make data work for the humanitarian sector in an affordable and scalable manner.
We are at the very beginning of this endeavour with HALO, but already we have shown the true potential of the good use of data.
We caught up with Priyan Chandrapala to find out more about the work he has being doing.
We extracted data from numerous REST APIs and converted them into a standardised JSON format. This applied to both the data and meta-data components related to the data sources and search terms. All the standardised JSON data was synced up with a BigQuery table. This table gets updated on the fly whenever there is new data from a datasource. Data sources included Twitter, NewsAPI and Google RSS.
“We developed a multi threaded cloud application which can crawl news websites and collect their content into a unified format. This application can scale from a few websites to a multiple hundreds effortlessly.”
Priyan Chandrapala, Data Engineer
Machine learning expert Nikolay Petrov gives a sneak peek into the world of machine learning.
Taking advantage of several thousands of manually collected, thoroughly reviewed and categorised articles we’ve built a pipeline which can give relevance scores and categorise any new article given as an input.
To achieve this we use lower dimensional mathematical representation of the article texts called embeddings. This in essence translates the semantics of the articles into numerical form which allows us to perform mathematical operations on them. Thus, we can train classifiers and capture similarity between documents and different characteristics of the texts.
“Finding information on the internet at the level which allows Halo Trust to build their impactful reports proves to be a cumbersome task. The possibility of vastly scaling their expert knowledge in executing that task is fascinating. In our first efforts we have managed to execute a pipeline which scored and categorized thousands of articles within a few minutes highlighting only the highly relevant ones. The potential where we can go from here is very exciting.”
Nikolay Petrov, Data Science / Engineering
“This has been a complete data project. On one hand, it had the intricacies of data ingestion focusing on scalability. On the other hand, it involved very sophisticated AI work, solving the different text length categorisation challenge. This was not obvious at all. Using DevOps principles and the best cloud engineering practices, combined with Google Cloud nuances, we created the right formula here. What looked like a well-known path in the beginning turned out to be a journey brimming with challenges and learning for all of us.”
Marton Hubay, CTO, Infinite Lambda
“I’m truly enjoying the journey of us, humans, learning to collaborate with computer intelligence. As we learn to dream bigger and expand our horizons collectively, we see the need to partner with the machines and tap into the higher intelligence that is within their potential. It feels amazing to be on the forefront of this endeavour being backed by a great team of engineers and friends.”
“I’m very glad I got to work with the HALO Trust as my first project at Infinite Lambda. People were motivated to do great work and we got to use the latest and best tools for the task at hand so that we moved fast. And we sure did. I was happy that I got my hands on Google Cloud and was impressed how fast we were able to create a nice UI using Retool.”
“I have worked on numerous data extraction projects but this was a novel experience. I was impressed by the speed at which a small team with bleeding edge cloud technology can prototype solutions. I also feel privileged to contribute to an inspiring cause such as HALO is undertaking.”