233: The Gap Between Space and Farm: Ground Truthing Satellite Data Models

The goal of the NASA Acres Consortium is to bridge the gap between space and farms to create sustainable food systems now and in the future. Yu Jiang, Assistant Professor of Systems Engineering and Data Analytics, School of Integrative Plant Science Horticulture Section Cornell AgriTech explains how this group of researchers are using land-based robots to ground truth data from satellites and aerial imaging to create predictive models. The project aims to bring cost-effective solutions for disease management, breeding, pruning, and more to farmers of all sizes.

Resources:

Vineyard Team Programs:

Get More

Subscribe wherever you listen so you never miss an episode on the latest science and research with the Sustainable Winegrowing Podcast. Since 1994, Vineyard Team has been your resource for workshops and field demonstrations, research, and events dedicated to the stewardship of our natural resources.

Learn more at www.vineyardteam.org.

Transcript

Craig Macmillan 0:00

Our guest today is Yu Jiang. He is an assistant professor of systems engineering and data analytics in the School of integrative plant science horticulture section at Cornell agritech. Thank you for being on the podcast

Yu Jiang 0:12

Thanks Craig for having me for these podcasts.

Craig Macmillan 0:15

I found out about you, because you're connected to the NASA acres Consortium, which is doing a bunch of really cool stuff for all kinds of crops around the world and winegrapes turning out to be part of it. What is what is NASA acres,

Yu Jiang 0:28

So I got to adopt the some of the official description about a NASA acre so our audience can better understand what's our mission and what's our approach. So NASA acres consortium is commissioned under NASA Applied Sciences program, and brings the value of Earth observation technology down to earth. NASA acres consortium established the march 2023 And then led by Dr. Alissa Witcraft from the University of Maryland. NASA acres is NASA's second consortium devoted to strengthening food security and agriculture, followed by the success of NASA harvest, a global focus a consortium but this time, NASA Acers specifically emphasizes on the US own agriculture land in NASA acres, we bridge the gap from space to farm and adaptation to impact to gather with US farmers, ranchers, and other agri food system decision makers who are charged with addressing the most pressing challenges to sustainable, productive, resilient agriculture now and in the future. to ensure our missions, NASA acres utilize a consortium structure to bring together a geographically, semantically and personally diverse group of agriculture actors, and partners from both public and private sectors collaborated within a model that matches ivory cultures own highly dynamic and diverse needs, and flexible partnerships and rapid actions on tools in NASA acres that will help ensure that a satellite based Earth Observations applications are user driven and free for all the preppers we envision .

Craig Macmillan 2:25

a huge mission. There's a bunch of different technologies that are involved here. And you're involved in a bunch of them. One that I'm particularly curious about was we had a guest on the podcast from Cornell Katie Gold, she was working with hyperspectral imaging and the detection of plant stress, but as a plant disease. And that's the that's the sky. Right? That's the information coming from satellites or whatever. You are the boots on the ground person. Is that right?

Yu Jiang 2:49

Yes, correct. I'm on the ground, I'm doing the groundwork.

Craig Macmillan 2:53

All right, we're literally grounding. So as far as that project goes, I understand that you're using robots and with sensors and artificial intelligence and whatnot to detect and predict disease spread. You tell me more about that.

Yu Jiang 3:09

For my account of a personal program, and the involvement of with NASA acres, you know, project, we bring in new, especially ground robots, we use various internet of of things, sensing network technologies, that we can offer the information as the ground truth matterments that many of these you know satellite or Earth observation data streams can use to try and various models for prediction, or estimation of various things of interest. And disease is definitely one of the biggest things for the ineyard management's currently adding in the future.

Craig Macmillan 3:50

Absolutely. If I understand what this work is on the ground as its ground truthing what the hyperspectral imaging is telling us is that right?

Yu Jiang 4:00

Roughly yes, if you can see there, all the current paradigm of doing remote sensing work. Most of the time, people are really focusing on the modeling, or how we can find the best and model to link or connect the hyperspectral signals collect data from, you know, satellite based or airborne based imagery systems, we use the ground truth data collected by a human on the ground. And these have been proven very successful in the past to produce various models that we are using right now including weather forecast, but with the very rapid and unprecedented climate challenges, and also the induced disease pressures. We are kind of lagging behind with the speed or pace we need to develop new models to tackle these problems. And that's a reason we want the robot to do so so that we can catch up with the disease. This can Have a fashion or progression speed, but also offer new tools for our viewers to use for their management decision making.

Craig Macmillan 5:08

So tell me about the robots, what are the robots doing?

Yu Jiang 5:12

So we developed a customized robot called the phytopathobot short for PPP. So basically phytopathology there's, my colleague, Katie Gold right is a scientist, that who really work on plant disease, and the bot is just the short name for the robots. And we put these two together, and basically just shows we integrated the kind of advantages offered robotic or automation systems with the new AI capability. So this robots can really bring the human experience and intelligence to all the fields that can do for example, if you see scouting recommendation, or some other, you know, checking functions that otherwise currently we have no human resources to do so for every single farm at the present time.

Craig Macmillan 6:06

Right? Is it fair to say that the training part would be described as artificial intelligence? Or should we call it neural network hearing? Or what would be the appropriate technical term for that part of it? Because I have a question about that.

Yu Jiang 6:17

Yeah, I think, broadly speaking, is a part of the artificial intelligence.

Craig Macmillan 6:23

Okay.

Yu Jiang 6:23

And that is more off the AI application for agriculture.

Craig Macmillan 6:28

What's happening is there's cameras then or there's some kind of a, either hyperspectral, or there's something that's getting information that's mounted on the robot, right?

Yu Jiang 6:40

Yes, correct. Our robot is currently equipped in ways both RGB multispectral thermal and the hyperspectral sensors, which many more on the road.

Craig Macmillan 6:52

And then you get readings. And then you know, human, I would assume says yes, this is disease, or yes, this is not. And then over many, many iterations, then the artificial intelligence learns what that is. And then it can be autonomous, you can send it out and it'll find it on its own, identify it on its own.

Yu Jiang 7:14

Yes, so I would see the autonomy is achieved at two levels. First is all the AI system for disease identification and quantification. We have a twin various models, with the expertise from our like, it's 30 plus year career technicians. And now we just a brand Hey, spray into the AI system that we can rely on to detect the disease in the field, specifically for a grape downey and powdery mildews at the moment. But at the same time, we also train the AI systems to guide the robots, autonomously navigating in the vineyard. is much more like the similar technology Tesla or other you know, EV car manufacturers are using for autonomous driving, but now just say, equipped those technologies with this ag robot that can do with autonomous navigation in vanguard in alternative in many of the different fields for agriculture purposes.

Craig Macmillan 8:14

the future of this technology, or the robots gonna continue to be a part of it, or are we going to be at a point where we're relying solely upon the aerial or orbit based imagery?

Yu Jiang 8:26

That's a great question. And I actually want to set up some of the context. information for our audience,

Craig Macmillan 8:33

please.

Yu Jiang 8:33

So yeah, the robots we kind of referred to here, actually those intelligent, you know, agent that can perform certain tasks in your backyard, or do the actual right to do all these operations, like a spring harvesting, you know, picking samples, all these, then when we consider how are we going to strategically and effectively deploy those robots? That's a big question is not a trivial because each robot at the current, you know, time would cost roughly 50,000 to $60,000. I think for many of the large farms, or wineries, the company will be able to afford that. For many of the small to medium sized farms, these can be a barrier for them to adopt the latest digital technology, which I hate, you know, that part as technologist. So one of the possibility is actually linked to the NASA acres project and the mission is a how we can use all sorts of information that can be affordably available to the growers to really use that for decision making. And a while of the concept we propose here is to make a closed loop joint training system that can connect the proximal sensing from the robots and other drone systems, we use the Earth observation data offered by federal agencies such as NASA, so that later all the growers can really enjoy, you know, using a very low cost or affordable platform offered from NASA or NASA acres consortia to make decisions on their individual farms. But largely training, the costs of a training such a model is taking over by large growers, largely, you know, stakeholders and some sort of a, you know, public and research institute that can balance the way or how the disadvantages you know, community can't adopt the latest technology.

Craig Macmillan 10:44

That is fascinating. You mentioned tasks, what kind of tasks are you talking about?

Yu Jiang 10:49

The current account of the PPP robots can do two tasks. First thing is for disease recognition, and the qualification, as I mentioned, for downey, and powerdy, and then now PPP can also generate a map right after the scanning off your vineyard, where those disease really severely infected your plants right now. And we working in progress try to use these PPP derive the map to correlate with the satellite maps or hyperspectral imaging so we can get so we can find which hyperspectral signals gone and correlated with diseases infection on the ground. And this is especially important for crops like grapes because of manual for the disease, or occurred from the bottom of the canopy, or the side of the canopy, where many off of the you know, satellite or Earth observation systems may not easily see at the beginning. But those signals will be embedded in the hyperspectral signatures.

Craig Macmillan 11:55

Got it. Okay. So I could get a map that would allow me to spray pesticide a fungicide very, very targeted way is kind of where we're going with this.

Yu Jiang 12:06

Yes, correct. I'm actually gonna just share some other ongoing effort here. Also, while also my colleague Dr. Devika Daughtrey from plants, Plant Pathology at Cornell agri tech, who identified the use of the UV, as treatment, powdery mildew or Downy Mildew for our grapes. And our account of ongoing efforts is to synchronize that map generated by PPP and the transfer to the UV robots. So now UV robots are gonna rely on that map to apply the UV treatment to balance the power usage and the hopefully to also maximize the contents of the disease spreading in the vineyard.

Craig Macmillan 12:52

That's really exciting. I understand the USDA also has some some role in this technology or related technologies.

Yu Jiang 12:59

Yeah, you ask the actually is a big partner of the whole team, especially for the grape genetics research unit, here in Geneva, New York. And we have a very multidisciplinary team, I will see I can see is from like a plant breeding to genetic to plant pathology now, including myself from engineering and robotics. And we also have about informatics, and we some colleagues from other universities on economy and marketing. So the whole team's efforts is back to a systems engineering approach, I would say. So when we look at the whole production, right, it's not just that, yeah, we have this robot that can do proceed and spray or deliver the UV treatment can solve all these questions. It's just hard to imagine that simple. So then we when we look at the whole agriculture production system, we started with the best plant material. And if we started with the building a candidate or a successful candidate data, usually just to make the rest of the whole production management much easier than ever before.

Craig Macmillan 14:14

Yeah, absolutely.

Yu Jiang 14:15

That's where, you know, all the scientists on the team really excited about how we can breed a new plant materials that have more like a natural resistance to plant the disease or maybe other stresses so that later on the in season management, it can be much more easily, you know, controlled or conducted by the growers. That Castile enable sustainable, you know, agriculture while maximizing the profitability for many of the growers in the future.

Craig Macmillan 14:45

I understand that one of the projects you've worked on had to do with phenotyping. So if I'm reading plants, there's a particular trait that I want and there's a particular expression of that trait that I want, whether it's disease tolerance or drought tolerance or salt tolerance. answer whatever it is, but that aspect of plant breeding is very difficult and takes a long time traditionally, and takes a high level of expertise. What is this idea of high throughput? phenotyping? What's that all about?

Yu Jiang 15:13

If you can have a think about the whole history of plant breeding, all the way you treat the back to mon Tao, we are human phenotyping is the best way, we just go to the field, plant and various plant materials, and just watch their performance in the field and find the best suitable for us. Right? So so then we recognize the traditional breeding, it becomes a numbers game, the more we test, the higher the possibility, we're going to find something, going t obe suitable for us, right? So we say it's a matter of who can email you this account of a traditional breeding way that requires the highest throughput phenotyping. Because the more you testing in the field, the higher the possibility we got to get something successful, and how to evaluate in the field is the biggest question right now. And that's where the high throughput plant phenotyping plays a vital role to address that bottleneck. So instead of for a breeder, to raw, only, you know, hundreds of 1000, you know, testing materials, the now can run, you know, 10,000, or even 100,000 in a year. That's how we hope to speed up the entire breeding cycles.

Craig Macmillan 16:25

So tell me the details of the tech of the details of the so I get some, I breed some plants, I've got some seeds, I'm gonna plant some seeds, right, I've got genetic recombination, now we gotta cross. How does this technology actually play a role? I put a bunch of plants in front of it, or how does it work?

Yu Jiang 16:46

Yep, so So in my understanding, there are actually two different paths ways to use that. One is along the traditional ways, as we just described, basically, we just find the best performancer from the field, right, and the system would just behave like a human in the field, we just find the tallest one, then we just a mirror the height of the plants in the field using the AI system with the robot, or if we want find more disease resistance is more like a what the PPB is helping right now, go to the field check a differente. And though gene all types off with a group of eyes, and then we find the least the infection as the candidate for the next one, right, this is a more like a traditional way. But now the second pathway is even more exciting is through the genetic studies. So once we kind of forget these phenotypes, especially there are differences, we have many different ways now can sequence them to understand their DNA markers and sequences, so that we will be able to work with the bell informaticians, to find which genes are associated with the phenotypic trees have a desire. Okay, so certain genes in my show, okay, the high disease resistance always associated with certain region in your DNA, and that's very likely being the gene or the region really control the resistance right to that particular disease. And if we ran multiple of these experiment, we get more and more as a candidate of Regents, and lead her on instead of keep running the field of trials, which still consume a lot of resources and the timing, because you need to wait until the plants are mature, and, you know, go through the entire season, we can now rely on those genetic, you know, information to identify the next around of a candidate, if the content of those gene regions is very likely, they're gonna have some, you know, resistance to certain disease. And that's another whole pathway, in my opinion, to facilitate the cultivar development in the future.

Craig Macmillan 18:58

And what is the role of AI in that?

Yu Jiang 19:00

So AI, please several rules there. So first, is to help the phenotyping itself, right. So basically, in the past, we sent a large group of it, you know, people go to the field and check the planet, hide diseases, infection, fruit size, you name it. And now we can just use, you know, robots to take images or even our cell phone to take an image. And then the AI will just mimic a human behavior to identify Oh, where the plant is, how tall the plant is, what's the number of leaves within that image or a number of a fruit fruit the size, a little versus, you know, trees and AI definitely now, at least, that being comparable with human performance for many of these tasks. And the other way is actually, to use AI as another tool to make a better prediction of relationship between the phenotypic trees and their genetic variants, right as we discuss for the second impassively is basically made to find that the association between genetic and phenotypic variants, and the AI also now plays a vital role to help us to find those relationships. It goes beyond traditional statistics human developed, and the find many interesting and hidden relationships that are currently statistic based approach cannot find.

Craig Macmillan 20:24

Wow, that's amazing. There's a couple of other things that that I that I was researching you that I noticed that were very, very, like practical right now, today, please, can I have some kind of technologies. One is improving the efficiency of pruning grapevines? And then I think I read this right, using facial recognition, AI technology to recognize powdery mildew infections. I would love to know about those two things, because those are two things that I would if I had it, I would use it today.

Yu Jiang 20:51

For sure. Let's start with the disease part. Yeah, cuz that's just allow what we just discussed why we developed that tool is basically a request actually, from my colleagues from the breeding and genetics slide. Okay. So in the past, my colleague, Dr. Lance Candle-Davison, at the USDA ARS develop a protocol that can use a one centimeter leaf disk as an assay to evaluate the disease progression, on the group leaf tissues, and then later on that can help him as a pathogen geneticists, to find the genes related to the disease resistance to powdery and downy mildews. But the challenge is, in the past, we have to train a bunch of, you know, technicians and the postdocs, even some of the other grad and graduate students at Cornell, to sit in front of a optical microscope and put the sample on our eight turn to like a tax 100x. And then manually identify how the pathogen really grew in the past a couple of days during the experiment, right, and then counted the number of a hyphal, which is a particular organ of the pathogen being grown, right. And then at the end of the day, they turn all these numbers back, and they will be able to run some quantitative genetic analysis, try to find the relationship. And I tried to once to be honest.

Craig Macmillan 22:27

Okay, yeah, I spent a lot of I spent a lot hours with a dissecting scope. So I hear you Isn't that fun?

Yu Jiang 22:34

Well, I want to see, for the first a couple of new samples. Yeah, it's it's a new experience for anyone, right? And if it's like, oh, yeah, I get that. After trial, you know, 10 samples. I'm done today. I don't want to see the front end of the microscope that day. And don't ask me to do this again. Right. It's quite tedious. And as a person, you'll feel fatigued very quickly. Yep. Very quickly, because you need to, to be super concentrated on what are you observing right now? And then also make the columns in your brain? I don't know how I did that. But I did. But after 10 samples, no, no more?

Craig Macmillan 23:16

Yeah. Yeah.

Yu Jiang 23:17

So that's the motivation for us to consider how the AI system can really help us, right? Because basically, what do we want the AI to do is giving you know, an image? Can you tell me? Which part content of the hypho And then tell me how I mean, how many of these hyphos are within that image? That's all right. So it's very much like the facial recognition technology we're using every single day. So our smartphone or maybe other security checking, you know, systems, right? And that just to give us motivation, hey, why not? Let me just build the robot and some of the AI tools that we can automate this whole process. So later on, instead of asking our students to do that very tedious work of observing the dissecting microscope, we will be able to allow them to do more intelligent work, how to find or improve the approval from the genetics, the perspective or the breeding perspective, rather than letting them doing this repeated and boring work. And that's the whole motivation here. And that's a reason why we can't have a proposed out method and that really got some success and to speed up that process. And now, just want to share with you in the past the year 2023 Last group, by using this technology was able to find a 60 more quantitative trait, a low sigh, which you can see there are data that gene regions related to certain, you know, phenotypic traits. And here in this study, that's more for the powdery mildew resistance. just named as single year, his team found 60 More as compare with, we fund probably 40 In the past four decades.

Craig Macmillan 25:08

Wow, wow, that's fantastic. There's so much here. There's so much stuff going on in it, as I have guests on that are working in these areas. It's just is it every day, I'm just learning so much new stuff, but I can't let you go without talking about pruning. I just, I just have to know about that I've I pruned a lot of grape vines personally, and I've trained people and you know, and there's, there's this, well, I'll just break it down for you. Pruning grape vines is an art form. And I don't care what kind of Trellis I don't care what kind of grape, whatever it is. And even if you're mechanized, that you gotta tune this thing up, and you got to collect data, and you got to figure out how this is gonna work. And when you have vines that are being pruned, you're trained, every single time somebody that I've been working with, usually above me was like, do these people really know what they're doing? Because they can't screw it up. Right? So now, is this going to help me? I mean, this is do you have technology? That's gonna help me you? I mean, I need this help.

Yu Jiang 26:02

Yes. Also, simple answer is A Yes, yes. And yes. So we are developing actually, the technology for the broader pruning a system for both apples and grapes as perennial crops, because they do need this type of technology to help based on my personal experience in the past three years, with both the pruning for apples and pruning for grapes, I share your burden Craig, it's not only you, but as an observer, and both the person who did the pruning, okay, using the knives, I have a strong feeling, I don't know what I'm doing.

Craig Macmillan 26:44

Right?

Yu Jiang 26:44

Right, I have a lot of criteria being you know, taught, say you need to find a branch that thick or that long, then you need to cut to certain lengths or just a cut them entirely, so that you can have new shoots coming with more healthy groups and the more productive grooves in the year. But to be honest, and once you get into the field, maybe perhaps the first several you keep that in mind. And then otherwise, oh, yeah, I just feel like these two needs to be cut. Don't ask me why I just feel that way. Right. And this is a kind of shows the non uniformity among the workforce. If I'm a beginner, I have less experience, I gotta be low in my working efficiency, I am going to create more problems, and rather than more success pruning, for the management, and obviously, the more counter for trend and people needs to be you know, pay them more because they have those experiences. So that all comes through the labor shortage issue, then it's just really hard to find those skilled people. So in my group, we kind of develop we are developing new 3d imaging technologies. Oh, wow. Yeah, that can get the very high fidelity of the 3d models of your grape vines and the apple trees in the field. And then once we get to some models, we can extract the skeleton is much more like how human described that, oh, yeah, that's my skin, and then I have to shoes and how they grow. And then we just be able to do that in a granular detail with all the needed information, like what's the diameter, or what's the length for each of these branch. And then due to all we can, based on our predefined the pruning criteria, to decide where are the cutting points, so that either a person or maybe a machine, or maybe a robot in the future, can go to the field directly cut based on the information we already get. Yeah, and the good thing is now with this whole kind of a new approach, instead of based on our existing you know, criteria, we can also form all different sets of criteria to really prune it in whatever way we want because that's a digital system. It won't hurt anything rather than using some of the power from you right? And then we can count off a get a difference you though proven the vineyard to take a look which we better serve our purpose. And we are also working with some offer collaborators try to incorporate to the growth models for grape vine. Try to see with different pruning strategy how the group vine or apple trees gonna grow during the growing season. And how I mean for me differently you know, branch structures and maybe different fruits load and the distribution with a hope you know, if we know this information beforehand, we can let the universe to determine what might be the best strategy we want to do as the you Though time progress to the green season, so do you have much more information in advance? Rather than Oh, yeah, I got it just to do the pruning. But that's the best I can do.

Craig Macmillan 27:15

Right? Right, right. So it sounds like that could be kind of an iterative process, you have a robot go through, and you get your 3d model, and you bring it back. And then you develop an algorithm essentially, that says, Keep this, don't keep this keep this, don't keep this, cut it here, cut it there, then you could execute that. Exactly, basically, to the vine.

Yu Jiang 30:29

Yes. Correct.

Craig Macmillan 30:30

And then you could have it grow. And then you can come back the following year, and say, Okay, well, what happened? And you could refine that model over time.

Yu Jiang 30:39

Yes, correct. That that's exactly the concept called a digital twin. Wow. Yeah, we see is a product actually from NASA, used to use that for you know, making the Mars rovers or the moon rovers, because they need to simulate so many different things before they put the actual manufacturing, right. But now we want to adopt these concepts for agriculture, before we do any of the decision making on pruning or harvesting. We want to see how they progress in the digital world, because it just takes us so minimal cost, and then we can have better understanding which way might be the best, we want to move forward.

Craig Macmillan 31:20

Wow, that's really exciting stuff. This technology is probably still in its infancy, I would guess.

Yu Jiang 31:27

Yes. Correct. I mean, although now we have more and more 3d imaging technologies and even more like a loose AI driven approaches. But it still is early stage, we are having some challenges from the field. So that's a reason we are, working hard to make progress. And I hope to share more things, you know, in the coming years with the audience here and hopefully demonstrated to the grape industry someday.

Craig Macmillan 31:53

Yeah, absolutely. Keep going. We're out of time. But I want to what is it one thing you would recommend to grape growers around this kind of topic, these topics, I guess I should say,

Yu Jiang 32:05

Can I share two actually?

Craig Macmillan 32:07

Please yeah, to is great.

Yu Jiang 32:09

Why I really want to share with with all the growers as we are at the point where many of these digital technologies are being more and more available and affordable. So please keep your eyes and the for example, at Cornell, my extension program focuses on the digital agriculture trials for adoption short for data aims to fill in the gap between you know, the growers and the startup companies who deliver those new da tools for production management, and also tried to offer more knowledge base to our growers, they can learn and better use these tools by themselves. So this is very important, as many of these tools go and just a calming and you don't want to miss the opportunity offer using the best of the tool to shop yourself and make better management. The second thing I also really want to share with our audience here is pleased to share all these exciting lands from digital agriculture to our case, to younger generations who are working in your, you know, vineyard or winery. I'm a strong believer the best investment is always you know, for the future generations. If they got excited if the et buy in all these ideas and put more efforts to start, you know, learn and develop new technologies back to agriculture and the food sectors. I believe we're gonna have a sustainable and resilient agriculture in the future for sure.

Craig Macmillan 33:39

That is fantastic. Where can people find out more about you.

Yu Jiang 33:42

you can check on my labs website is a se a i r dot c a l s dot cornell.edu. I will provide you the link so that you can share with the audience.

Craig Macmillan 33:58

Fantastic. So our guest today with Yun Jiang. He's a system professor of systems engineering and data analytics in the School of integrative plant science the whole crypto section of Cornell agritech thank you so much for being on the podcast. This was really fun.

Yu Jiang 34:13

Thanks so much Craig for having me today and as my priority to share our ongoing efforts and research with the broader audience here for grapes. Thanks, everyone.

Transcribed by https://otter.ai

Nearly perfect transcription by https://otter.ai