Overview:

Climate Change, is a highly debated high school topic, but do we really care about it? We see politicians, scientists, researchers, philanthropists, conservationists and almost everyone blaming each other for every climate catastrophe occurring today. The real question is what climate change or change are we talking about?

https://www.iberdrola.com/wcorp/gc/prod/en_US/comunicacion/cambio_climatico_economia_2_res/Infographic_Impact_Climate_Change.jpg

Change is never easy for any species or life form but it happens in order to survive and evolve and this is the change we need in order to challenge climate change - the change could be in form of laws to reduce carbon emissions, conserve biodiversity, advanced technologies which can outperform existing techs in terms of greener energy or total innovative revamping of traditional ways of all human activities.

According to soil scientists, at current rates of soil destruction (i.e. decarbonization, erosion, desertification, chemical pollution), within 50 years we will not only suffer serious damage to public health due to a qualitatively degraded food supply characterized by diminished nutrition and loss of important trace minerals, but we will literally no longer have enough arable topsoil to feed ourselves. Without protecting and regenerating the soil on our 4 billion acres of cultivated farmland, 8 billion acres of pastureland, and 10 billion acres of forest land, it will be impossible to feed the world, and keep global warming below 2 degrees Celsius, or halt the loss of biodiversity.

My innovation and solution here are to 'adapt and mitigate' and not to 'detect and forget'.

Three out of four crops across the globe producing fruits, or seeds for use as human food depend, at least in part, on bees and other pollinators.

What is our CliSensio?

CliSensio - Climate Sensing and Insect Infestation Control is a new adaptive device that runs SensiML AutoML audio recognition on the edge to recognize disease spreading vectors, hungry pests and threatful species. CliSensio device firmware can be flashed for mosquito species identification, pest, rodent identification and elimination responses.

1. Infestations of Insects turning Farmland into Deserts: A peer-reviewed study stated that the threat of rising temperatures and unpredictable weather patterns has multiplied tremendously. By 2050, droughts and insect infestations could destroy 26.2 per cent of the World's largest tea exporter, Kenya's tea-growing areas. A research study concluded that climate change is more dangerous than we can imagine, it could empty 'World's Grain Bowls' - Rising temperatures will stimulate insects' appetites -- and make some prone to reproducing more quickly -- spelling danger for key staples like wheat, corn and rice which feed billions of people.

Thousands of acres of crops could be destroyed by locusts at a time when the global economy is struggling to recover after the COVID-19 crisis in Asian Nations.

Thousands of acres of crops could be destroyed by locusts at a time when the global economy is struggling to recover after the COVID-19 crisis in Asian Nations.

While the world is still reeling from the coronavirus pandemic, several countries Brazil, Argentina, India, Pakistan and other East African nations are facing locust attacks that threaten to trigger a food crisis.

"When it gets warmer, pest metabolism increases, " said Scott Merrill, a researcher at the University of Vermont and co-author of the study in the journal Science. And when pest metabolism increases, insect pests eat more food, which is not good for crops. Overall, "global yield losses of these grains are projected to increase by 10 to 25 per cent per degree of global mean surface warming, " said the report. - http://13.235.153.162/world/global-warming-will-make-insects-hungrier-eating-up-key-crops-study-162721

• France stands to lose about 9.4 per cent of its maize to pests in a world that is 2 degrees C warmer, compared to about 6.6 per cent of yield losses today due to pests.
• In Europe, currently, the most productive wheat producing region in the world, annual pest-induced yield losses could reach 16 million tons.
• Eleven European countries are predicted to see 75 per cent or higher losses in wheat from pests, compared to current pest damage.
• In the United States, the world's largest maize producer, insect-induced maize losses could rise 40 per cent under current climate warming trajectories, meaning 20 fewer tons of maize per year.
• China, home to one-third of the world's rice production, could see losses of 27 million tons annually.
• Pandemic, war, and climate change fuel fears of global hunger, and food scarcity. The first challenge is to accept the problem. The second one is to accept responsibility. Climate change is unlike any challenge humanity has faced because, unlike pandemics and wars, here we have a warning. We have time to prepare for the impending disaster, but we are not exactly making good use of it. We're arguing and grandstanding, instead of acting. When the oceans rise, there is no fix. When the air becomes unbreathable, masks will not work. - The world needs to come together and find solutions to the climate disaster problems:http://13.235.153.162/world/climate-change-the-action-is-missing-despite-the-high-profile-summits-379354

2. Losing Grassland Birds is another Widespread Ecological Crisis: Growing unnecessary and competitive use of pesticides and insecticides that sweep out all insects affects the entire food chain of birds. Moreover, the insects are becoming resistant to the chemicals day by day but the birds which eat those insects barely survive for reproduction. Using lots of chemicals to boost farm yields but we tend to forget that due to excessive use of those pesticides, insecticides and inorganic manures, cross-pollinating agents, the arthropods( 90% of living organisms found on earth are insects) don't get attracted to those plants or otherwise die due to excessive chemical exposure and pollution. More than 2, 000 pollinating insects are extinct by now and only 500 are left to become extinct if not protected - "Even if there is no insect disease in farms, farmers uselessly spray chemicals and kill useful insects like honeybees and other birds too."

More than 90 per cent of the losses are from 12 species including sparrows, warblers, blackbirds, and finches.

"Now, we see fields of corn and other crops right up to the horizon, everything is sanitized and mechanized, there's no room left for birds, fauna and nature." - Ken Rosenberg, an ornithologist at Cornell University and principal co-author of the paper in Science told AFP. The National Observatory of Biodiversity estimates a 30 per cent decline in grassland birds between 1989 and 2017. The number of birds in the United States and Canada has fallen by an astonishing 29 per cent, or almost three billion, since 1970, scientists reported: https://www.wionews.com/world/north-american-bird-population-fell-by-quarter-over-50-years-study-250699

3. Eliminating threat species to our pollinators(our best friend, bees): Insects are pollinators and waste management experts, controlling other insects and keeping the soil healthy while also serving the most important function – being food for those higher up the food chain. In the absence of pollinators, many plant species will disappear.

Honeybees are very important to us. Flowers can hear buzzing bees—and it makes their nectar sweeter

Sure, the argument can be made that humans will be able to recreate such plants in the laboratory. Approximately 87 per cent of the over 300, 000 species of plants need external pollinators, where would one begin? Pollinators are our secret weapon in the fight against global warming.

Also, scientists are now on a hunt for the hornets, hoping to eradicate the species before they wipe out US bees. According to the scientists at the Washington State University (WSU), "it's a health hazard, and more importantly, the shockingly large hornets are a significant predator of honeybees." They will begin trapping queen murder hornets this spring, aiming to detect and eradicate the species.

Populations of honeybees and other pollinators in the US were already under pressure.

The United States has been fighting a deadly virus. It is now the world's worst-affected country and now it is preparing to fight another enemy, except this one is not invisible. On the contrary, it is huge. The world's largest hornets have been spotted for the first time in America.

4. Billions at risk of mosquito-borne diseases: Climate Change Will Expose Half of World’s Population to disease-spreading Mosquitoes By 2050. According to a study from PLOS Neglected Tropical Diseases by the end of this century, almost all of the world's population would be exposed to mosquito-borne diseases. Aedes aegypti and Aedes albopictus are two species of mosquitoes known to carry and transmit potentially deadly viruses like dengue, chikungunya, Zika and yellow fever. These bugs can thrive in urban environments, often travelling alongside humans as we ship goods and travel around the world.

Let's Build it!

Hardware Setup:

The heart of the device is QuickFeather EOS3 https://www.quicklogic.com/products/soc/eos-s3-microcontroller/ All SensiML AI algorithms are going to be performed on this device. See all the below images, for device setup.

How is liquid-chemical atomizer utilized in Clisensio?

The liquid atomizer has two components, a piezoelectric atomising head and a driver circuit. A wick is used for supplying the chemical liquid to a piezoelectric atomising head arranged in the spray apparatus. In CliSensio, the devices are used for chemical liquid spraying for targeted species,i.e. insect removal, insect growth retardation, insect repellent, disinfection based upon SensiML audio recognition and feature extraction models trained to detect harmful pests and insects.

1 / 4

Things required. Though I planned to use motor driver and haptic responses but proved to be of no use for this project.

Fixing the solar panel on the lid. Also, made holes so that the insects can enter the box or can enter holes if they are attracted by some lure compounds present in the enclosure.

Furthermore, the present innovation relates to a method for controlling harmful organisms by effectively and economically releasing active ingredients having the functions of insect removal, insect growth retardation and insect repelling in the air. Stop harmful insect and pest breeding in their growth period, saving farmers and communities from any perils. The method would massively reduce the overdosage of chemicals used in traditional agriculture or insect control. Device Firmware Setup:

The device firmware is based on QORC-SDK, https://github.com/QuickLogic-Corp/qorc-sdk all instructions are well documented here.

Clone this QORC SDK repository using the shown command, and check if make and GCC compiler is set correctly or not (if not follow the steps in GitHub link README.md from the above link).

$ git clone --recursive https://github.com/QuickLogic-Corp/qorc-sdk
# run gcc -v and make -v to find gcc compiler toolchain and make utility
# versions on your system
$ gcc -v & make -v

You can use any code editor, Eclipse C/C++ IDE is preferred but other IDEs can be used too(I used VS Code as it is the easier and swiss knife for my projects).

All the data extraction and training have been done using SensiML Analytics Studio and SensiML data capture lab: https://sensiml.com/products/ Currently data capture lab is only for Windows platforms, that's one downside for Linux and Mac users, but I managed it since I have both systems at my hand(windows and Linux separately not WSL). Our CliSensiodevice can run two different firmware:

1. Mosquito Species Detection and Control 2. Pest Detection and Control1. Mosquito Species Detection and Control

Data Collection: Since we are using the microphone as the only data source for our CliSensio (I had planned to bring in audio as well as sensor data into the project but currently either sensors or audio ML models are supported at a time, not both together.) The audio dataset was already collected, this project is inspired by the other projects I made in past to solve pest and insects problems, links here: 1. https://www.hackster.io/vilaksh01/tensorcrop-crop-quality-and-farm-control-3831b8 2. https://www.hackster.io/vilaksh01/sigfy-sigfox-farm-yield-with-crop-and-health-monitoring-3d8116All audio data used is uploaded on this Github page: https://github.com/vilaksh01/CliSens.io/tree/main/CliSensioMosquito-audio_data However, I recommend you to record it via your own device through SensiML data capture lab to achieve more accuracy, as those audio files are not recorded on same microphone architecture and bit rate.

Data Segmentation: This was performed through the Data capture lab after the data collection step. I recommend using Audacity to check spectrograms for the audio data before proceeding to segmentation, it gives a clear analysis of noise-to-valuable data in the audio file.

1 / 4 ? Aedes Aegypti Audio data segmentation, 55 segments

Aedes Aegypti Audio data segmentation, 55 segments

Anopheles Mosquito Audio

Culex Mosquito Audio

Training and Saving model: Log into your SensiML cloud or Data Analytics Studio to check our SensiML data and proceed training steps. We trained on two pipelines to make model more efficient and accurate. Pipeline 1 Check below images carefully for all steps.

1 / 3

Segmentation distribution for our audio data

Pipeline 1 achieved an accuracy of 77% after training

Pipeline 2 achieved accuracy of 75%

Visualizing Model Summary: 

This is one of the most important and highly rewarding features for ML training engineers(they get to know their models better). Below is the summary for pipeline 1 and pipeline 2.

Pipeline 2 Summary

Pipeline 2 Summary

Pipeline 1 Summary

Download the model: This is the final step to download Knowledge pack for Mosquito Breed Classification and Control for our firmware.

Our generated model JSON file for Pipeline 2, check the class maps- 1. Aedes Aegypti- is known to transmit the dengue virus, yellow fever virus, chikungunya virus, and Zika virus. It is suggested to be a potential vector of Venezuelan Equine Encephalitis virus and is capable of transmitting the West Nile virus., 2. AedesAlbopictus- This mosquito species is a known vector of chikungunya virus, dengue virus and dirofilariasis., 3. AnophelesMinimus- association with malaria, filariasis and arbovirus infections., 4. Background- Noises like wind, white noise, etc., 5. Culex tarsalis- spread the viruses that cause West Nile fever, St. Louis encephalitis, and Japanese encephalitis, as well as viral diseases of birds and horses., 0. Unknown Classifier;

Integrating Knowledge Pack with QORC-SDK and our firmware: The app structure is similar to qf-apps/qf_ssi_ai_app in QORC-SDK. The first step is to set up our GPIO pin for the liquid-atomizer module in the src/pincfg_table.c file.

{ // setup Water Atomizer Digital Output
.ucPin = PAD_23,
.ucFunc = PAD23_FUNC_SEL_GPIO_7,
.ucCtrl = PAD_CTRL_SRC_A0,
.ucMode = PAD_MODE_OUTPUT_EN,
.ucPull = PAD_NOPULL,
.ucDrv = PAD_DRV_STRENGTH_4MA,
.ucSpeed = PAD_SLEW_RATE_SLOW,
.ucSmtTrg = PAD_SMT_TRIG_DIS,
},

After setting up GPIO let's modify the src/sml_output.c function to direct the action response based upon audio classification or inference performed. Also, from here we can see the serial communication prints out classifications which can be used to send to the dashboard for Spatio-Temporal Vector Ecology map over LoRa or cellular, helping the government to be prepared for unforeseen events(epidemic or pandemic).

static void sml_output_serial(uint16_t model, uint16_t classification)
{
int count;
int wbytes = 0;
int buflen = sizeof(sensor_ssss_ai_result_buf)-1;
int ret;
kb_get_feature_vector(model, sensor_ssss_ai_fv_arr, &sensor_ssss_ai_fv_len);
// you can send other sensors or GPS data in below line for dashboard
count = snprintf(sensor_ssss_ai_result_buf, buflen,
"{\"ModelNumber\":%d,\"Classification\":%d", (int)model, (int)classification);
wbytes += count;
buflen -= count;
switch((int)classification)
{
// case if unknown audio found
case 0:
HAL_GPIO_Write(GPIO_4, 1); //blue LED for unknown
HAL_GPIO_Write(GPIO_5, 0); //green LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;

// case if mosquito species 'Aedes' 'Anopheles' 'Culex' detected
case (1 || 2 || 3 || 5):
HAL_GPIO_Write(GPIO_7, 1); // chemical atomizer GPIO 7 on
HAL_DelayUSec(15*1000*1000); // wait 15 seconds for atomizer
HAL_GPIO_Write(GPIO_7, 0); // chemical atomizer GPIO 7 off
HAL_DelayUSec(15*1000*1000); // wait 15 before resuming
break;

case 4:
HAL_GPIO_Write(GPIO_5, 1); //green LED for background
HAL_GPIO_Write(GPIO_4, 0); //blue LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;
}

After everything is in place, let's compile and flash the firmware to the device using the below command in the project directory.

$ make clean
$ make
$ qfprog --port /dev/ttyACM0 --m4app ./qf_ssi_ai_clisensio_app.bin --reset --mode m4

2. Pest Detection and Control

Testing the device audio classification for cricket insect sound playing on a smartphone. Data Collection: We are going to collect audio datasets for different pests, currently for this POC, I am more focused on locust sounds here (as India’s agriculture ministry is hoping to control the invasion before monsoon season hits north India at the end of June when locusts mature and breed. If the infestation is not controlled, it may threaten summer crops such as rice, maize and sorghum.) However, gathering locust swarm audio data was difficult so I tested with cricket sound files available for free here(Locusts belong to the same order of insects as grasshoppers, katydids and crickets - the Orthoptera). Here, are all the audio data for cricket sound https://mixkit.co/free-sound-effects/crickets/

1 / 5 ? Testing the device in rain ;)

Data Segmentation: We have segmented the audio data into two categories, viz. Cricket and Background.

1 / 2

Training and Saving model:

 In SensiML Analytics Studio, we are going to create the model and train the data for classification. Our query name is Q_PESTS_SPECIES. Training for Pipeline_1 is shown in one of the below images. Here, I tested two pipelines as before, the other pipeline is also shown below.

1 / 4

Integrating Knowledge Pack with QORC-SDK and our firmware: 

Similar to the previous firmware, the first step is to set up our GPIO pin for the liquid-atomizer module in the src/pincfg_table.c file.

{ // setup Water Atomizer Digital Output
.ucPin = PAD_23,
.ucFunc = PAD23_FUNC_SEL_GPIO_7,
.ucCtrl = PAD_CTRL_SRC_A0,
.ucMode = PAD_MODE_OUTPUT_EN,
.ucPull = PAD_NOPULL,
.ucDrv = PAD_DRV_STRENGTH_4MA,
.ucSpeed = PAD_SLEW_RATE_SLOW,
.ucSmtTrg = PAD_SMT_TRIG_DIS,
},

After setting up GPIO let's modify the src/sml_output.c function to direct the action response based upon audio classification or inference performed.

static void sml_output_serial(uint16_t model, uint16_t classification)
{
int count;
int wbytes = 0;
int buflen = sizeof(sensor_ssss_ai_result_buf)-1;
int ret;
kb_get_feature_vector(model, sensor_ssss_ai_fv_arr, &sensor_ssss_ai_fv_len);
// you can send other sensors or GPS data in below line for dashboard
count = snprintf(sensor_ssss_ai_result_buf, buflen,
"{\"ModelNumber\":%d,\"Classification\":%d", (int)model, (int)classification);
wbytes += count;
buflen -= count;

switch((int)classification)
{
// case if unknown detected
case 0:
HAL_GPIO_Write(GPIO_4, 1); //blue LED for unknown
HAL_GPIO_Write(GPIO_5, 0); //green LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;

// case if 'cricket' detected
case 2:
HAL_GPIO_Write(GPIO_7, 1); // chemical atomizer GPIO 7 on
HAL_DelayUSec(15*1000*1000); // wait 15 seconds for atomizer
HAL_GPIO_Write(GPIO_7, 0); // chemical atomizer GPIO 7 off
HAL_DelayUSec(15*1000*1000); // wait 15 before resuming
break;

// case if 'background' detected
case 1:
HAL_GPIO_Write(GPIO_5, 1); //green LED for background
HAL_GPIO_Write(GPIO_4, 0); //blue LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;
}

After everything is in place, let's compile and flash the firmware to the device using the below command in the project directory.

$ make clean
$ make
$ qfprog --port /dev/ttyACM0 --m4app ./qf_ssi_ai_clisensio_pest_app.bin --reset --mode m4

This pest detection and control use case can be improved for detecting killer Hornets and killing those in their breeding seasons before they reproduce in surplus and pose threat to our friend, bees. If pests are not coming near your device, you can put some luring compounds to attract the pests, the attraction will lead them to make a wingbeat sound near the device and the device in turn would activate a chemical-atomizer killing those mature pests before they make breeding grounds everywhere.

Device Power Requirements:

The device runs on solar power and SensiML compute consumes very low power along with a microphone, so it can last several months provided, the chemical repellent is changed frequently or better in a newer version of this Clisensio, I will be adding sensors to alert when to refill using LoRa or cellular.

Also, I am working to add LoRa and Cellular connectivity for the device to extend the notifications and demographic status features (Just the same IoT stuff, sending the data from RX/TX and visualizing on the dashboard) but for a POC this is well-understood......More work ahead :)