PV-Hawk: Computer Vision Pipeline for the Automated Inspection of Photovoltaic Plants

and this talks about my PhD thesis project and I thought I recorded for anyone who's interested in the topic who works in the field currently or who

wants operationalize the software that I developed or developed similar software and um yeah so my my project was about developing a computer vision pipeline for the automated inspection of

large-scale thrower type plans as shown here and um in this talk I will quickly give a motivation for why we would want to do inspections on these plans then I will

give a short overview of the core problem that was solved in this thesis and then I will talk in detail about the developed computer vision pipeline um the state of the art the develop

methods and the core results then we will conclude this and I will briefly mention the software that I developed in the course of this project and finally give an Outlook which should

be interesting for anyone working in this field who wants to improve um yeah the software or build the next

or go to the next step in this um so yeah first of all why would we want to do an inspection of life skill forward hack plans so it's mostly because solar PV is

getting very popular So currently we have about 775 gigawatt Peak deployed or actually in 2020 we had 70 175 gigawatt deployed which is about 3.2 percent

share of the global electricity production and the International Energy agency in a conservative scenario um predicts that another 940 gigawatt Peak

will be added until 2026.

also 160 gigawatt Peak annual lean apart and um these plants are massive I mean the lashes ones currently are about 550 square kilometer in size so they have

millions of modules and components cables connectors um and of course they can fail as shown here so there is a whole spectrum of things that can go wrong in these plants um burned out

um junction boxes like bypass diodes broken cable connectors material degradation um

yeah exposure to the elements like hailed um thermal Cycles mechanical load Cycles through wind and whatever and yeah Studies have found that about

eight percent of all modules and two percent of all strings in these plants are affected and it costs about six percent average power loss and um yeah we want to prevent of course these power losses and yield losses

and that's why we want to do inspection Advocate identify these failures and possibly repair or mitigate the causes and that's important to meet help meet

the investment targets of these plants to to ensure the economic success of solar PV and because only if it's economically successful it can grow in future

and um of course it's awesome and safety aspect because in some cases these failures could cause fires and um especially if the plants are mounted on rooftops we want to prevent that from

happening so it's also a regulatory requirement in some cases and also a solar preview inspection is very future topic um because currently we see a lot of new

deployments as you can see from here right we currently see very new plants that work well but let them age for five to ten years and the number of defects

will increase so inspection will become ever more important in the next coming years um yeah so I said these plans are massive in size so we can't possibly manually inspect them so what's being

done very often is infrared tomography with a small consumer Market drones such as this DJI drone and um our tomography is popular because

it has several advantages over other techniques so um for instance it's non-contact so it doesn't interfere with the plant operation you just imaged a plant with a thermal camera and then you

will see the anomalous modules as having an increased temperature and um that's as opposed for example to electroluminescence or full photo luminous sense Imaging where we have to

reverse power the plant or shine with a very uh bright LEDs onto the modules um it's safer and I said it doesn't interfere with the client operation also it allows to record during daytime

because we need a solar strong solar e Radiance of about 600 watt per square meter um so that's Advantage here is that you don't have to record during the night as for El or PL so you don't need to pay

nighttime surcharges or get special work permits for night operation and infrared tomography is also sensitive to both module and system anomalies so anomalies that affect individual modules but also

entire parts of the plant like the entire strings or whatever is and these small drones are popular because they are relatively low cost I mean you get this for about 20 000 Euro

so that's as compared to a helicopter or other equipment very cheap um they can be fully automated you can do Waypoint flights um and they helped does they help to achieve high

throughputs of about 48 000 modules which is equivalent to what 11 megawatt Peak per hour and it's about 10 to 15 times faster than walking on ground-based systems

yeah so this is how it looks like um when you record our data so we fly in this case manually over the plant we scan the rows sequentially

then we get this kind of footage now in RGB and now in infrared terminal infrared so you see this one has a slightly lower frame rate of what nine Hertz um you can now later see some here some animal

leaves popping up um yeah that's the kind of footage you work with that leads us to the problem statement of this project which is basically this large amount of challenging data being produced so the

data is challenging because it's a long duration it's highly repetitive and has no visual reference points so it's impossible for human to manually work with this kind of data

but instead we can use computer vision to to automate this procedure and what do we want to achieve here we want to actually get an overview map that can be easily interpreted by a human so we want to get a map in which we highlight

anomalous modules for further inspection or even directly forward peps um I said this can be done with computer vision and that's also what I did in this thesis

so I said I developed this computer vision pipeline um and it's can be broken down into some subtasks and also the following presentation will be grouped into the

subclass for every subtitles I will talk about the state of the art My Method um and the core result so in the beginning is the data acquisition and it's about planning the flights

acquiring the video data um yeah then the next step would be to to identify in the video where do we have PD modules um so which pixels represent preview modules then we would

have to de-reference the modules so identify which of these um where in the PV plant are these modules localized or located um The Next Step will be to identify

which of the modules have some anomalies are they healthy or do they show some kind of possible defects or which are visible through thermal anomalies and then finally would report this on a map

as shown earlier to allow our targeted repairs and yeah give an overview of the plant's Health state so we start with the

first uh so in this thesis I published three papers so um the first one is about the entire pipeline so in implemented every single step of the pipeline kind of the initial

solution to everything um then the second publication I looked in depth into the anomaly detection I spotted a problem here which is called domain shift and I proposed a solution to it

um and then finally in the last publication I reiterate on the module localization method who actually improved this and also developed a new anomaly detection method and the

following will be mostly about these two papers because I said the third one is an improvement of the first one so I will not go too much into the details of the first method um yeah

so let's start with the data acquisition how this works so um I said it's about planning and executing the flight and recording here so um we are relatively flexible here we don't really make assumptions about the

plant layout but it does help to just sequentially scan along the roads you don't have to be sequential but it just makes it easier to deal with the data later sort the data

um you can fly basically in a configuration where the row passes horizontally through the frame where it passes vertically through the frame you can also scan multiple rows at the time we test it with two and three rows as I

will show later also um yeah that's about it we did some manual slides but no one stops you from planning a waypoint flight with your preferred flight

controller and flying automatically the idea of here is that you can use the method both for small PV plans to perform ad hoc inspections like you don't want to like do a full-blown

flight planning for yeah I found it that is just a few hundred modules I just want to go there fly your drone quickly five minutes over the plant and we're done but on the other hand when you have

very large plants with thousands of modules you want to probably be able to do flight planning and automatically execute the flight so it is both possible with this method

from now the module detection um again that's the task of identifying which pixels in each frame represent a PV module and which one's smart

background or relevant components um so here the state of the art in this so as again this I'm not the first one who does this topic there were other works before me mostly they look into

individual aspects so they propose either a module detection method Anonymous detection method it or a localization methods the couple of works that also look into integrating

everything but I will now look into the aspect of module detection here so we see a lot of Works using classic image processing methods such as Kenny

Edge detection and hoofline transform um or also a segmentation by thresholding with fixed threshold values adaptive threshold values um the little

problem I have with these works is that they don't really use a lot of data to build the methods and test the methods I will show the numbers later but and that makes a bit questionable if these

methods actually generalized in practice to multiple plans because these PV plans have they can be quite different different right they have different modules different layouts can be having

grass in the background as well to send whatever so I'm a bit skeptical if these methods actually work on on these variety variety of plants but then we see um some new works using deep

learning um for instance one work used segmented segmentation to remove the background or classify Vector on versus PV modules in the foreground

um here's a little problems it won't be applicable to our method because it can't distinguish individual PV modules um then we also see object detection being used like YOLO it's popular here um

yeah so this one will give us access aligned bounding boxes and again that's not so good for our case because we don't really get the accurate positions of the preview module Corners because

the the boxes can be are only aligned to the image axes so is that what's usefulness or what we use as instant segmentation so we get a bounding box for HPV module and also a pixel accurate

mask this one allows us to to localize the module corners um there are a couple of work sets using the stimulation on an entire tables and rows but I think that's a bit problematic

because it wouldn't generalize if the role layout or table layout changes um so again that could be like multiple um it could be only like say two modules stack vertically but could be also three

or four stacked and this method would either you train it on all these configurations otherwise it wouldn't uh generalize whereas here you only identify individual PV modules and it doesn't really matter to the neural

network or the global arrangement of these modules is so this one is the way to go in my eyes and that's I said how we do it also so this is not our method for the module detection we first take our videos we

split the video into individual video frames because we work on a per frame basis uh so for each frame we apply in this case a mask rcnn

um out of not really modified very much um yeah just apply it to get the module masks for each frame um we did choose Mars as you know because it just works well in practice

there are like mature implementations available um libraries like um we have the tensorflow object detection API if it's a bit old data Maybe um mm detection has implementations for

mask rcnn so now we apply this I get our mask for every module for every frame um and now we can actually harness the um because we have video frames with

nine Hertz so we can connect the same or mass of the same PV module over subsequent video frames I'm using a much object tracker

and this will also stand to crop out um these regions we Rectify them with a homography together yeah rectangular shaped module and because of this

tracking we have multiple patches per module and this allows us to do some more advanced analytics Downstream because we have multiple patches available for the same module also allows us to for example filter out some

Reflections that may occur um it also makes let's let's not make more accurate predictions because of just more images available for each module now this is like how it looks like in

videos so this is our input video then this is the individual detections of the preview modules and masks and here the result of the tracking so if you see these blue numbers they stay constant

over the visibility of the respective module and overall we retrieved a mean average Precision of about 90 percent um and I know the number doesn't say

much but the results are sufficient I mean it's the robots are accurate enough to allow us to skip any kind of post-processing we just work directly with the output of mass rcnn

um and again this is useful because we introduce post-processing it would introduce heuristics again and we're not sure how well it would generalize it so it's good to just work out of the box

with two results of the Mask rcnn as in this experiment to know Common results for the module detection so let's experimentally um analyze just the amount of data that we need for training how that affects the overall I mean

average position that's achieved also what combinations of training and test plans would lead to what map um and we found here actually the most only interesting aspect that we find

here is that it makes a difference whether the modules are in landscape or portrait mode um as soon as you take that into account while training um we also found that uh I mean that's not very surprising but the num the more

data you include in the training set the higher the map that's achieved and the lower the standard deviation and kind of saturates a little bit didn't try on enough plants I would say

to really see this complete flattening of this curve but now we just said if we use training on the few plants it's sufficient um yeah a bit more interesting results probably

the data sets that we acquired um so in total we acquired um your these kind of image patches that I talked about earlier of 140 or more than

142 000 different PV modules stemming from 10 different PV plants and in total we have a 6.55 million of these little patches here because we on average that

for about 40 patches per module um and then for the First Data so we actually have two data sets because our camera broke down in the middle of the work so and we didn't have calibration

data for the old cameras so we worked with two different cameras and two different data sets um yeah but for the first data set we obtained a two uh 10 anomaly labels um

as shown here um yeah so these ones basically actually started with 26 anomaly labels and or anomaly classes and then because for most classes there were very few

examples available and that's quite bad for doing a supervised classification which I did in the first paper so I grouped together the classes depending on their appearance

um not so much depending on the underlying physical cost so um this entire study actually works with appearance Because deep learning can only look at the appearance of the images it doesn't really understand the physical causes for this Union engineer

and someone who knows the electrical engineering that goes into these modules and um those regions also grouped it according to appearance and we have these kind of classes that um but some

of the classes also allowing one-to-one mapping to the physical cost for example this uh homogeneously warmer module would mean that the module is disconnected from the plant you have basically all the incoming solar energy

needs to be dissipated away in the form of heat uh can't be transported away in the form of electrical energy so um kind of for some of these classes you can see

there's one physical cause for others you have multiple possible causes the second data set we have only binary labels so healthy versus some kind of anomaly

yeah and from this data set because it derives some insights for example we saw them I can see here that that's basically the percentage of anomalies for every plant in our study and um it's

actually I think that's from the first data set one but uh yeah so we see that not all plants have an equal number of or percentage of anomalies and there are some plants that are better kept like for example this one there are others

that are more permatic Dev maybe older plants more problematic modules that suffer from larger a degree of degradation so we do

see some variation here from say one percent to 12 to 12 um also we see a quite as a class imbalance with respect to the distribution of the difficult and different anomaly classes so we have

very frequent classes uh and very infrequent classes sure and another interesting Insight is that um yeah most plants suffer from one or two predominant kind of anomalies there's no plant that has the entire

spectrum of anomalies um each plant has its um one or two problem points um yeah that was it about the module detection now I'll talk about the module

localization and the reporting so localization was the task of geo-referencing each individual module in the in the video and then reporting is the task of showing it on the map the

results um yeah so how did I do the module localization or how do others do it first um so there are a variety of methods that were developed in the recent years so a very

simple one is simply just lock the control position whenever you see an anomaly in the frame but it doesn't really allow for accurate localization because usually have multiple modules profile um another method that's being used is

Panorama stitching um so here we would basically manually group the frames that belong to one row of the plant we would stitch them together into one yeah a larger

Mosaic of the entire row and then we could detect the modules in this model you can also count them and look like them relative to each other um this method so it does only use visual queue space here to build this um

because it doesn't really use any additional GPS measurements or something and it seems to work well for this smaller um yeah sequences that I've shown in

shorter rows that were analyzed not sure how well it works with like really large plants like because you would get this buildup of error during the mosaican procedure as you have to first manually group The Frames according to the rows

um yeah I'm not sure how well this would generalize and also how welders can cope with artifacts that are introduced during this Mosaic King I mean you have to do some kind of image blending like Feathering or something to to fuse these

individual images so I'm not sure how this could actually affect Downstream anomaly detection or module detection and it's actually the same from that odd photo methods have so these ones they fly a bit higher and then they create

one single odd photo of the entire plant um here again I'm a bit skeptical because you would introduce also like some kind of visual artifacts when when building this Auto photo

um so I'm not sure how the module detection will be then performed on our photos so I'm not sure how well these things would code I mean these artifacts could potentially trigger the animal detector but they're not real animalies

such as artifacts that were introduced in this procedure um another method that's being used is directly referencing so here we used um yeah gns position of the flying

vehicle and the attitude of the vehicle to assign a GPS position to each pixel in the camera um and can we can reduce doing some transformations of the reference frames

here um here I see the problem a little bit that if the GPS position is off because we have some kind of glitch or some I mean if you don't see enough satellites in a given moment you would have a

higher error in the position um positioning so you would also directly translate this arrow to the locations of the review modules

um a method that we used in our works and that I've seen in one other work is structure of a motion so this one here you would build a 3D con reconstruction of the plant using both the visual peels

so you kind of yeah I'm doing multi-view geometry from from my reviews of the scenery but you will also take into account the GPS positions so you would basically basically match the best of these two

worlds of Panorama stitching and material for instance you'll take the visual cues into account and you would do a Geo referencing um so you get a good local accuracy but also good Global accuracy

um so this method what it does is actually reconstruct the plan and then performs module and anomaly detection in the 3D reconstruction that's a bit problematic again you would have artifacts in this

reconstruction and also it's promoting because you need specialized algorithms that work in these Point clouds um I think what we do and that's probably what we do is we we do use the

structural motion to create a map of the plant but we do perform anomalies detection and motor detection in the source footage or original images this allows us to use standard algorithms like

typical deep learning algorithms for object detectioners and segmentation classification um but it would also give us the advantages of having this 3D reconstruction so we pair the vessel

both words here and that's how we do it so this is our um yeah that's what I showed earlier the module detection and now this is how the module localization fits into this

framework so for each video frame we would extract a GPS coordinates latitude and longitude the altitude is now reliable so we don't use that um we do linear installation because the

GPS locks with one Hertz the camera of 9 Hertz will linearly interpolate to match these rates and then we need to select a sparse subset of frames because we can't do it possibly on app we can perform

structure from motion on every um yeah frame because we need more we need a sufficient amount of Parallax and this is computationally expensive reduce every frame this would take weeks to

compute so we we basically select a set of frames based on the travel distance and the intersection over Union of the frames um we also need some kind of calibration information for the structure of emotion

and if we have these two things through the frames and the calibration data we can perform structural Motion in this case we use opensfm um cell open source library for

structure for motion and this gives us a sparse 3D Point cloud of the scenery that we actually don't need anymore but what we need is this 6D Pros trajectory

so the translation and rotation of each camera frame each keyframe basically I with respect to one Global coordinate origin and this is in um

uh geocornates um yeah because we use in this opensfm and the Geo the GPS coordinates of our trajectory this basically creates a GE reference

reconstruction um yeah then we take the 3D reconstruction so the D60 um posters are treated triangulate the

known Quantum points of the three PV modules from the video frame so from image coordinates into these word coordinates and fruitarian relation and some further refinement so we triangulate from every possible pair of

keyframes that we see the module in then we get let me average over these multiple the reconstructions that we get that we perform some iterative improvement of the resulting grid but basically this

gives us a q reference 3D reconstruction of the entire RPV plant then we just map that to the ground plane to get a

map like this I have to say that we don't really take the terrain elevation into account here so we just assume it's flat because in Germany it's quite flat here so it works not sure whether it

works in other regions so you may have to do a little bit more sophisticated here with having a digital surface model or digital terrain model and project the Reconstruction onto that instead of just

a flat plane but for our cases this worked well um yeah this is an example so here is basically the reconstructed camera trajectory in black and then the

reconstructed PV modules in red and their Center points in green just for a small subset of one larger cloud um this here is a

entire map of a relatively large cloud with about 10 megawatt Peaks so about 13 600 PV modules and there are some some artifacts some

glitches but um yeah it's more local not more like Global glitches the average success rate here is 99.3 percent so we miss only a very few uh a very small

fraction of modules but I have to say that this procedure is not very robust I mean um so it's kind of like an All or Nothing thing here if the Reconstruction works out then it looks like this it's fine you've got

most modules but in many cases also fails because we work infrared images which have low resolution and not many features there's blur through the

terminal um yeah heat moving in a certain pattern so these these are not very optimal conditions for structure for motion so either you get all or you don't get anything and you have to rerun the with

different settings to open sfm until you get this so let's show you something that the community could look into improving um yeah again but if it works it works pretty well then that's how it looks and here I

also analyzed what's the accuracy of this so um we have a total RMS used mean square error of about 5.9 meters here um that's in the range of GPS which is

expected a 4.9 meter rmse um yeah so that's okay I would say um and as if we look locally so pero the RMS is smaller it's 22 centimeters to 82

centimeters so we have quite a good local accuracy here that's totally fine for for our use case um you also see that accuracy here is a bit higher than here so we have kind of some error drift over the entire

reconstruction um I think it did it's fine um here that's what I said earlier so we don't really depend on a particular plan layout we don't really make any assumptions so we have the ability to

fly in different configurations like scan one row two Row three rows at a time and this gives us a means to do a speed accuracy trade-off so of course if

we fly uh only one row we have a higher accuracy because we get higher resolved patches of about 140 times 100 pixels on average um but we have only a throughput of 3.4

modules per second and we can actually ramp that up to almost 13 modules per second but then we also have a resolution that's only about a third of the size so here we have to decide what's what's

best the stability of the open SM reconstruction was actually best for this case and this one was best to fly it's like easiest tool if you want to like jump basically if you're done with

the row you're here you have to like go further for the next three rows it's a bit trickier to commentary rows and it was a bit more difficult this one works pretty well so this one is actually our

favorite configuration for practical applications I would say um yeah now that the last aspect so I talked about data acquisition how we find the modules and the for radio

frames and I talked about how we localize them is to create a map of the PVD plant and also interesting of course which of the modules are defective you still can't look at all the data it's just too many images uh I would take hours of

days so you want to do this automatically and yeah we have deep learning we have machine learning we can all use all these methods to automate this um and operated that means maybe first

what did the others do again we see a lot of Works using classic image processing especially with older Works um so again here Ash detection of Kenny Edge detection or thresholding with fixed thresholds adaptive thresholds

again the same problem they don't really use much data so it's not clear how well these methods generalize and we do see new works using deep learning and for example this word uses

semantic segmentation to pick out disconnected strings so entire rows of modules that are warmer than the surroundings we do see object detection being used so

that basically a little bit differently they detect the module hence the anomalies in one step they don't do it in two steps How We Do It um can have advantages I mean you have

only one neural network not two um but in my eyes as the disadvantage you're less flexible because you can't really separately tweak the performance of module detection and unwantly

detection because both is in one neural network so if you change the the network with regard to the anomalies you will automatically change them with regard to the detection of the modules so I guess it has both pros and cons what

we use and some other works have done that also supervised classification so we put in the entire image patch and then we predict a um single label single class label

um I might as that's the best method because you can distinguish different anomalies um you can um yeah see also not here you see only localized Unleashed so they're like Hotspots for example but this one

is also able to really look at the entire module and then most unreleased as a spread audio entire module area so I think that classification works works best for this kind of use case and

now this is our data set in numbers um we were lucky to have a lot of raw data available from Project partners so that allowed us to actually achieve state-of-the-art size of our data sets

we um maybe to give some reference like most half of the relative Works use less than 1000 modules and images so relatively small amount of data even there are many words are used in the

tens of images so that's really different yeah I mean that all the works but this is where I say I'm skeptical whether these methods even work or generalize um yeah but okay everything from here is

probably fine that the more the newer Works already have larger amounts of data in the hundreds of thousands of modules so that's a good development I would say um yeah what data said we had a 142

000 modules and because we have multiple images per module we actually have almost two orders of magnitude more images like the best one has also 150 000 images I think about

a little bit less and and we have 6.55 million images so yeah that's quite a jump here in terms of the number of annually classes that were analyzed we had 10 different classes

um as another popular data set that has 11 different classes um I guess it's important to look at these number of classes because you want to really spot all the anomalies that the chromatic thing is that many related

Works actually I look at only one class so they're only sensitive to one class or even two but that's too little in my eyes I mean you wouldn't really pick out all the important anomalies if you're only sensitive for one or two kinds of anomalies

um there's some words that do binary predictions so they are not able to tell you what anomaly it is but at least they are sensitive to more anomalies so that's also fine I would say um so yeah we want to look at either we're classifying many different animals

or being sensitive to many different anomalies and making binary prediction also the number of PV plans 70 of the studies used only one PV plant again that's critical because you don't

know how it generalizes if the plant layout changes background vegetation whatever um there's some new or especially new

works who use many more people funds um that's good we also use 10 plants so I think there's some works that use even more um I would say the more the better

also an aspect that has to be looked into is the the resolution of the images we have a GSD to ground something distance of about 1.13 centimeters per pixel

um I think these two data sets that I've shown here um they have um about 3 to 15 centimeters um so we have to take into account there

is this like IEC TS whatever Norm for Android tomography and PV plans and um they recommend a worst case resolution of three centimeters so we shouldn't go beyond this so if you have

15 centimeters a resolution that's too little for doing serious demography um yeah this is how this animal detection fits into the over the framework so here we had the module detection here we have the module

localization and I'll just how we do the annual detection so really basically basically take into account these different patches per module um do our analysis and then visualize the

results on the respective position in the map we know for example there's a bunch of images belongs to this module on the map we analyze here some defect and we can plot it here um yeah so I developed three different

annually detection methods in this work in each publication one the first one is how most Works have done it in the recent times I've seen the second one is on this problem of

domain shift that we'll talk about later and the third one is news deep learning it's just a physically inspired method that has also its advantages yeah so again our our task is to

basically export or classify either multi-class certification or binary classification these different defects based on these image patches um our first work is resonate 50

classifier with a soft Max layer cross entropy loss so basically the standard set of of imagenet classification to classify these 10 different classes and um

no surprises here we have our confusion Matrix we see that some classes are easier confused than others yeah it doesn't have nothing surprising um our overall accuracy is 84

um we can actually so if you use a sync that's running an experiment that we did is we used a single patch so one patch per module we get 84 accuracy and now as I said we have multiple images per

module we can actually boost the accuracy by almost seven percent um so there's an advantage of having multiple purchase per module um here the difference between patch level module is that here we really take

all the data into account independent of the modular assignment and here we do a majority voting off based on the images that belong to the module so we take this assignment between patches and modules into account

um maybe there's some class imbalance accuracy maybe not even be the best metric to look into uh the weighted F1 scores probably better for this case but we see a very similar pattern here

I don't know how to prove computed Class Activation Maps using credit plus plus um so basically we looked at the regions in the image which the you know the network looks at to come to its

conclusion when doing the prediction we find that it looks at the warm regions um that's what we would expect that we that's what we want to see so that's all good here um

but one one huge problem that we spotted and I think that's something the community really has to take into account is the problem of domain shift so basically I identified this by plotting the or embedding the images so

we see all these image patches into a low dimensional space um using umap dimensionality reduction and I found that um yeah for each PV plant we

have one distinctive cluster here of images so each point is one image and this clustering is much more pronounced in the clustering into normal versus abnormal image patches

and it is chromatic I mean what we want to see actually is one large cluster for normal and one large class of abnormal images that doesn't really uh yeah take into account where the image comes from for which plant

and the prayer is that and so shown on the left that's how we did it in our first work that I've just shown and also how most related Works have done it or all related words that I've seen that use some kind of neural network for

classification they sample training and testing uniformly from all clusters we use let's say 70 of each cluster for training and 30 of each cluster for testing so our train and test

distributions are the same and or have the same parameters like the same mean the same variation um and that's not realistic in my eyes so

there will lead of course the good accuracy because we have the same distribution so training and testing is what deep learning is good at but it's unrealistic I mean it will be more realistic to train on one plant or even

medical plants but then assume that the testing plants are the one we want to make the predictions on for which we don't have the labels yet is a distinctive class a different class because in practice we want to have our trained Network and then we just want to

apply it to a new data without having to label parts of that data um yeah so we have this distributional shift between the training data and the target data

um or test data yeah so the quality is source and Target because this problem is called unsupervised domain adaptation and they usually talk about source and Target data because you want to train on the source and then transfer to the Target

and yeah so so it's important to take this into account and I think this is something the community should stop doing it like this you should really do it like this instead I'm dead it's a yeah whole breadth of

methods that could be used to kind of tackle this uh domain shift here I did the first step I mean it didn't really solve the problem I only proposed one solution that worked a little bit

better than the normal cross entropy classify higher uh it's definitely not a the final solution and I think there's a lot of work that could be done still here to really tackle that

problem and improve accuracy and considering this domain cap some solution or partial solution is to use a contrastive loss instead of a cross-entropy loss because uh there's a

very popular work that has shown that contrastive representations are more informative than cross-entropy representations and this is higher informativeness

should lead to a better yeah better performance in the presence of this distributional shift and um as for how I did it it's

basically I trained a CNN on our source images or training images that are labeled and I trained it to produce embeddings that cluster or data into Normal and abnormal clusters independent

of their origin plant that the images come from I did so using a supervised contrastive loss um and yeah then basically I use this transition and to embed the target

images for which we want to predict the labels I also embedded the training images and then I have this space where I have both the unlabeled Target features and the labeled training features and now we just did k n

classification to get the yeah labels of each or the predictions for each Target image based on the neighboring training images

and um yeah again I mean this should do better with the contrastive with the domain shift because these contrasting representations are more informative than as if we had used the cross entropy

loss here because cross entropy loss tries to really extremely maximize to the in inter class a margin and reduce the intra class

variance so really tries to compress the data into one point and put the point to the different classes very far away but it's these are some features that are not very representative the contrastive

features are a bit more spread out and it does carry more information not as compressed I would say there's also what does this work found um yeah and then if you look at the results

so these are the feature spaces that we obtained after training with the contrasting loss so now we have our distinctive nominal cluster abnormal cluster which doesn't really show these different original or plans from which

the data originates from anymore there are some cases like here we do see still the Clusters from the original plant but we also have this clear separation between nominal and abnormal

and we also looked here at um different pairings of Target and Source plans I found that this matters of course so you have to make a clever choice of your training plan if you want to do this and

basically just choose the plan with more data more anomalies the busy use the plan for training that is most challenging and then you would expect higher accuracy on the testing

um yeah here for example we use this plant to show poorer performance and we found it it's because it has just four point times six less data at 4.6 times less data than plants A or B

yeah so it's something to take into account um here we actually quantitatively compare it with the cross entropy classifier and we see an improvement in the accuracy for some of our tasks so

this is always a pairing of source and Target plant right so we train on plan a we test it on Plan B for these two cases we see an improvement in many cases

not always and for Plan F which I said has a lot less data we actually see that the cross entry works better than our methods so I said I partially solved this problem

with the domain shift and not 100 sure if this contraceptive method is really a solution to it um I think that the core message of this work was really like to look at this

domain shift and try to find solutions for it um yeah so my solution should be taken with a cranial song Yes uh what another aspect that I think is very important that the community has to look into is

this detection of anomalies um so I said there the what I showed earlier the distribution of anomalies is very imbalanced so we have some very infrequent anomalies we have very

frequent ones and so it's um yeah we should assume that we would spot new anomalies in the future that we haven't encountered during training yet and um so it's important that the method

actually also picks out these unknown anomalies that are not part of the training data set but that may occur in practice and in order to test this ability of this cross-entry method now I

picked out uh this anomalies so five of the ten classes from the training set so our training set contains only five anomaly classes the test set still contains all are many classes and then I train the method and evaluated on the

entire full set of anomalies and interestingly I don't see a drop in accuracy here so it shows the method actually works well on um yeah taking all this data it would be

interesting to see how the cross entry classifier does here I didn't test it it could be interesting to figure out if it can do the same right if it's possible to train only on five classes but still

be able to to pick out all 10 classes um also some some aspect I have to mention here I picked out these five classes because they have uh I think in total only about five percent is only five

percent of the data in our training set the other 95 belong to the other five classes and I did so because I didn't want to um skew the experiment by picking out too much data because if I had picked out

the classes that are very represented then I would have removed large amounts of the training data and it probably would have uh then dropped the accuracy but then we would have seen only this drop due to the lower amount of data but

not due to being and these classes not being part of the data set anymore um and now the last method for my third publication on module animal detection is that is a non-deep learning method a

physically inspired method and it simply looks at the distribution of the module temperatures over the entire plant and which is shown here for this large plan with 13 600 modules and now we show

the maximum module temperatures maximum here means the maximum over the entire image patch of each module so basically we have our 140 times 100 pixel image patch now we look at the brightness pixel and this is the value that we show

here and now what we see here is first of all a global pattern so this entire recording took about two hours um I mean here so this started at about

10 30 am it went all the way to 12 30 pm and of course the air temperature change the cloud cover changed um so we see that in the morning it was cooler and then later it got a bit warmer uh

probably also the cloud cover was different that's maybe these smaller Hot patches here is a bit warmer he's a bit warmer he's a bit warmer here again um yeah so there's some Global pattern

that kind of prevents us from saying okay we want to make a cutoff at 45 degrees Celsius and everything gets higher than that is an anomaly we can't do that um instead we have to maybe run as well

before I can explain this one aspect is in this version we can actually spot system anomalies like open strings like here which we couldn't do with the Deep learning because the Deep learning looks at individual modules only my developing

methods I mean there would be us there are other methods look at entire strings and stuff but my method will only be able to pick out individual module and when it's not system anomalies um but now again we can't say we have

this threshold value and then we cut off and then everything gets warmer it's anomaly and so we have to get rid of this global distribution and now without this global distribution um we have basically a homogeneous mean

temperature now we can say okay make a cut off at 2.5 Kelvin difference and then it's all animals how do they compute this basically I just subtract it from each module temperature that was

shown here the medium [Music] of the surrounding modules surrounding means a circle of seven meter radius it was arbitrarily Chosen and this gives us basically a local

temperature difference so basically this now indicates how much warmer is this module than the median of the surrounding modules and this picks out very nicely to the um yeah modules that

have hot spots or hot or some terminal anomalies and now we can really say we have threshold value of 2.5 Kelvin or something and then we get all the detected um so now I compared this temperature

distribution with the Deep learning classifier the stimulating class file is from my second publication the cross entropy classifier um and we see actually the overall area

under the receiver operating curve is not much worse about four percent um interestingly we see that um they complement each other so the developing transfer as good as some classes where

the temperature distribution is not as good as detecting and vice versa so for instance the debugging one is good at detecting PID or substring anomalies that basically spread over the entire module area but have a low temperature

gradient and because of this low temperature gradient this method doesn't work as well but this one works well on on a kind of animals that have high gradients like but are small in spatial

extent how the CNN is not as good in detecting the anomalies with small spatial extent like the hot junction box hot spots or soiling or something and because they're just too small in image

but the temperature variant is high so the this method is good at this so ideally would combine both methods and if we want to really make a real world software that takes out all the

ways we should use both and I also have to say this one is only slightly better but it's also relatively low performance of the neural network we use the resnet 18 here I think it's also only trained

on one PV plant um so there could be a lot of gain by using more advanced neural network here with more data so whereas here we kind

of we hit the limit we can't prove this anymore on the event this one can detect open string anomalies um and so this one is set per module this one can also detect system anomalies and it doesn't really suffer

from the domain shift because it doesn't use any data um but then again because it doesn't use the data it is using manual heuristics like this seven meter radius and things so yeah there's a trailer after we made

here um and then let's uh let me conclude so this entire method that I developed it allowed us in our Institute to use the

available raw infrared videos um that previously were just dumped on the server and what could work with it I made it accessible to research I developed an entire software solution

for performing targeted repairs in large scale PV plans um and this method is high throughput it achieves about 10.6 megawatt per hour in throughput

which is much more than any manual method or Chrome based robot could have done I guess um yeah and I published this not not only the scientific results in my

Publications but also these two software packages that are available on my GitHub page as shown here um one is called PV Hawk that's the computer vision pipeline is a command line tool written in python and a

desktop app called PV hog viewer that can be used to browse the resulting data sets annotate module defects annotate electrical connectivity um

yeah this is a desktop implemented with QT for python um I didn't provide a web app um probably the next step yeah but for this case the desert was

just fine um now to give an Outlook that's probably the most interesting aspect because it shows what the field could do to improve this method um it's it's structured into module detection animal detection module

localization and Reporting again um one aspect is um yeah we did perform instant segmentation only on per frame basis but we could have also used some video detection method to do it to take

this temporal information into account maybe not sure if this is a really good idea but we could also provide that end-to-end solution to get picking out the patches from the video instead of having these different algorithms like

mass arsene and then multi-object tracker then uh some kind of mapping could probably be done in one neural network not sure that's a good idea um animal detection here is important

that we um yeah make this method future proof so let it be or make it able to work on different module Technologies we only took poly crystalline silicon modules into account

that are currently the most popular I think 95 percent uh share but in the future we probably see more thin film half cell shingling modules uh two-phase

modules whatever the method should be extended to this also interesting aspects are not only these film anomalies but also degree of soiling or geometric misalignments that occur

during the installation of the plant um also there is a huge family of animal detection modules the models that perform segmentation in an unsupervised

manner of the anomalies for example pardim it's one of these could be a good starting point to look at this paper um yeah it could be used instead of the supervised classification of anomalies

um also as I said the domain adaptation I've only proposed an initial solution a partial solution there are many active domain adaptation methods using for example um

the maximum mean discrepancy loss copy a starting point for this um also I didn't really take bias or class imbalance into account bias in the sense really only one person labeled our data set a class imbalance asset

anomalies have a high somewhat frequent somewhere less frequent um also one huge aspect is this electrical connectivity to take this into account because this would allow us

as shown here to to report um or match the image you read to the electrical parameters of the plant in order to get these electrical um

yeah information how these modules relates to each other in the plant currently we need to manually enter this information because there's no standardized cut formats that are

digital formats that would describe this connection and it's something the industry should develop um to give us a way to automatically ingest this kind of information

also I only looked at the typical plants that we have in Germany so these like row based plants um I did try a little bit with a rooftop plant but not really seriously so it would be important for us to look at

rooftop plants and and floating PV files which are popular in Southeast Asia for example um yeah these methods on my method should be extended to this again then a very important aspect is not only

spotting which modules are animals but also identifying how serious is that normally so how much do we lose actually in terms of power and yield and for this we need to look at both the imagery and the electrical data and we

also need to know how the modules are connected it's this aspect of the electrical layout then we need to find correlations here and kind of analyze how important are these anomalies actually because we don't want to actually give an alert if

that normally doesn't actually lead to a power loss or you've lost also um now we had this map shown earlier where we showed which modules are defective

um it would probably be cool to have a handheld device a phone app or tablet app and also localize the repack crew and the plan so that you the crew can already walk to the defective module and

knows where where they are on the plant because the plants are so huge you don't know where you are if you're standing on one of those um so that's that's a little Improvement that could be made

um also I only work with infrared later on I actually extended it to RGB already some students extended approach to electroluminescence and voluminous sense and one aspect that I want to mention

that would be very useful I guess is to perform all these computer vision methods like module detection structure of emotion perform all this in the RGB footage because the RGB has

higher resolution has more textures this uh sharper than the thermal image so it would be very helpful to do all these analysis in the visual footage and

then have a mapping from visual to our to terminal and only crop out from the terminal in the end that would be the the ideal and the way to go unluckily didn't work with our camera because the

camera was not really wasn't possible to temporarily synchronize it and spatially register the two video streams because it's a microbiometer or terminal camera so it's free running it the frame rate

is not consonant performance flat field calibration in intermittent intervals so there's some kind of jump in the footage this is really really tough problems and synchronize these but if that was

possible due to better sensors or a better software then that would be the way to go I would say yeah so that's basically let me conclude this that was the talk

um thank you very much for listening have a good day