Having video cameras on fishing boats to detect accidental captures of seabirds isn’t a new idea. It’s potentially a very useful, cost-effective way to supplement or replace the use of human observers on vessels, and increase coverage across a fleet.
Dragonfly, with Pisces Research Limited and the support of members of the fishing industry, has been involved in researching and developing this technology in Aotearoa for more than a decade.
David Middleton, director of Pisces, describes the many challenges of collecting footage. “We need to get good quality footage, at an appropriate frame rate and then manage data storage and retrieval. That’s on top of keeping water out of the cameras, managing lighting and making sure they are recording at the right times.”
There had been early trials to automate the video review, but for the 6 years the programme has been in operation, detecting seabirds has relied on human review with some footage being reviewed several times by different people. David admits that, from an efficiency perspective, this wasn’t ideal.
“A longline haul lasts several hours and bird captures are rare and quick. If you’re reviewing hours of video, there’s a risk of missing these events. You have to put in a lot of effort for not much return.”
In 2020 Tech Futures Lab student Abel Butler began looking into the use of cameras on fishing vessels as part of his master’s degree.
“I was wondering why it was taking so long to equip boats with cameras. The legislation had been passed a while ago but not a lot seemed to be happening.”
He was put in touch with Pisces and Dragonfly after contacting the Ministry for Primary Industries.
Abel wanted to find out if a machine learning algorithm could be used instead of manual identification of captures. This became the topic of his master’s project and has catalysed efforts at Pisces and Dragonfly to move the technology forward.
“With the permission of the vessel operators, they were able to provide existing human-analysed footage that I could label when the capture events took place. This has given us a dataset that we’ve used to train the machine learning algorithm what a capture looks like.”
“It’s been fantastic to work with the talented staff at Dragonfly, especially Kiarie Ndegwa and Henry Zwart who’ve made huge progress with the machine learning in a short space of time.”
Initial results are promising, with the model identifying captures with accuracy comparable to that of human reviewers. While there remains work to do to fine-tune the model to improve its precision, the speed of automated review is remarkable.
“The algorithm can move through a 7-hour haul in about 10 minutes,” says David. “Human reviewers would be lucky to achieve a halving of the actual haul time.”
“Abel’s involvement focussed us on the big prize of using machine learning to detect the seabird captures, which has proved to be much more successful than we anticipated. His encouragement and the way everyone has contributed their skills in such a nice low-key way has been very productive.”
Abel is grateful for the opportunity to contribute to the project. He plans to work with Dragonfly to further refine the model before retesting it on some new footage.
“Starting out, we really didn’t know if this would work. To get to this point where we have some solid proof of concept and the potential to apply it in working practice, is the best progress I could have hoped for.”
The project team would like to acknowledge and thank the participating fishers for their ongoing, valuable contributions, and recognise the early funding from Southern Seabird Solutions and DOC which kicked off the project in 2015.
More information: Summary and evaluation of the electronic monitoring programmes in the SNA 1 trawl and bottom longline fisheries
Contract details: Dragonfly Data Science has been contracted by the Ministry for Primary Industries (MPI) since 2019 to deliver the video observation programme for the bottom longline fishery in Fisheries Management Area 1. Pisces is subcontracted to Dragonfly to manage the project. Prior to 2019, the programme was directly contracted from Trident Systems by Leigh Fisheries, Moana New Zealand and Sanford Ltd, with MPI and DOC additionally funding parts of the analysis.
Top image: View of the line hauling area of a vessel showing a woven object used to mimic a seabird. These proxies were used to test the viability of video observation in this fishery and the footage was reused during the initial training of the algorithm. Image used with permission of the footage owner, please contact Dragonfly before any re-use.