D2.1 Data management plan
Data generated through project research activities are openly available with some limitations for the private data from Denmark, where owners-farmers do not allow it to be shared. EO data (S2-L1C, S2-L2A, S1-VV, VH) are available through Sentinel Hub. We have processed the L2A products for Slovenia, Austria and Denmark. Non-EO data (LPIS) are available through Geopedia . Crop Yield is available through FTP server. During the project, Data management plan was regularly updated on FTP server.
The tables below show data, which have been included.
|Cultivated AREA Crop Type|
|Cultivated AREA Crop Type|
Available through eo-learn task through Geopedia layers:
|Cultivated AREA Crop Type|
Yield Data from 30 Danish farmers
Crop Damage Slovenia
Updated Non -EO- data
Updated EO- data
D2.2 EO-data collection
We have uploaded all the EO-data to the project FTP server. The database was made to be updated during the whole project as the needs could change in future tasks. All Sentinel-1 and Sentinel-2 L1C from 2016 to 2017 over Austria, Slovenia and Denmark were uploaded on the server.
D2.3 Non-EO data collection
30 individual farmers were contacted by phone and thereafter a contract, regarding use of data covering approximately 6000 ha, were mailed, signed and return by each farmer. The non-EO data made available consist in part of detailed information on individual fields (e.g. management practice, crops, soil attributes) and in part of field trial results, all owned by the Danish farmers. To ensure confidentiality of these data and information, a Confidentiality Agreements have been negotiated between the partners of the Perceptive Sentinel project. The Confidentiality Agreement has been signed by all partners. We have provided detailed data on management practice on individual fields from the Danish field database and from Danish government databases for 2.6 million ha of Danish farmland. In addition, data from filed trials from the Nordic Field Trial Database were also provided.
D2.4 Data verification
For verification of EO and non-EO data purposes, data from field trials studying N application rates in winter wheat were obtained. Data verification for the algorithms were verified using drones equipped with multiband cameras. Using drones equipped with multispectral cameras, images were acquired throughout (6-7 times) the growth season of winter wheat in field trials. From image analysis vegetation, related indices were calculated. Additionally, the growth stage of the wheat was recorded at each imagery acquisition. For ground-truthing of drone-acquired imagery, plant material in trial plots was cut and quantified on 3 occasions during the growth seasons and analysed for N-content. From the Nordic Field Trial System information on management practice, geolocation and trial conditions were obtained.
D2.5 Non-EO data sets
Data from several different sources have been used: LPIS data, public datasets from Denmark, Slovenia, Austria, and field trial datasets from Denmark and Slovenia. The non-EO datasets were collected on common FTP address available to all partners of the Perceptive Sentinel project: ftp://ftp.sinergise.com/. LPIS data is also available over Geopedia.
D3.1 Algorithm theoretic baseline document
The ATBD was build on the reflection of already implemented algorithms and on the investigates biophysical indices. Commonly used algorithms were analysed and advantages and disadvantages have been gathered. Various Open Source tools – like SNAP, Sen2Cor, MAJA etc. have been considered. A selection of mandatory or very important methods and functions has been made and these will subsequently has been implemented into the processing services of the project.
D3.2 Pre-processing algorithms
Various domain specific pre-processing methods and algorithms for optical as well as RADAR data have been implemented into the eo-learn library, which serves as the main processing engine of the project. For optical data various composite methods, a cloud masking method, Feathering and a histogram matching method were implemented. For Sentinel-1 a pre-processing workflow based on SNAP was implemented.
D3.3 EO modelling data (biophysical indices)
After the biophysical indices of SNAP were also already provided, a validation based on the example of the Leaf Area Index was performed. For this purpose, the index produced with the eo-learn library was compared with the outcome of SNAP to evaluate the functionality and reliability of the implementation of Sinergise.
D4.1 Streaming learning baseline document
The deliverable report digs into the field of data-driven approach to processing Earth Observation data. In the report, we have focused on stream mining techniques and corresponding software solutions. We have provided a comprehensive list of challenges that are associated with applying stream mining to EO scenarios. Report also provides a prototype of a "user-friendly” data mining platform (the principles have been in parallel introduced also in the eo -learn library).
D4.2 Deep learning baseline document
The deliverable is a document describing the different deep learning approaches available for crop detection. First supervised methods with conventional neural networks are presented, then the deep learning methodology is described (datasets, annotations, data quality, data augmentation and various methods to improve learning). Semi-supervised learning is also shown before presenting how deep learning can be used in the specific case of this project.
D4.3 Spatial and radiometric features
Development of spatial and radiometric features describes the different features useful for data mining and highlevel information extraction which have been defined and developed. The following features extraction algorithms were then implemented in the eo-learn library and documented in the final document:
- Haralick features,
- Local binary pattern,
- Histogram of gradient,
- Blob extraction.
Those algorithms directly implemented inside the PerceptiveSentinel / eo-learn library can be easily used by all collaborators and be integrated in other studies or inside other algorithms.
D4.4 Heterogeneous Data Pre-Processing for Stream Mining
In this stage we have proceeded beyond understanding of the principles learned and tried to (beside extending the knowledge from D4.1) implement features, that were used within state-of-the-art papers. We have prepared working prototypes and pushed the feature extractors into the PreceptiveSentinel - eo-learn platform.
D4.5 Deep learning algorithms
The deep learning algorithms have been implemented for “cultivated areas” extraction and “crop type” detection on Sentinel-2 data. The deliverable provided the status of the current study.
D4.6 EO-QMiner: Stream Mining Models for Earth Observation
Computationally efficient data analysis methods are presented in this deliverable. The methods provide an alternative to current state-of-the-art methods, which focus solely on accuracy of the algorithms and where extensive research has been accomplished during the last years. We introduce incremental learning to the domain of EO data processing, we present a novel methodology for feature selection based on genetic algorithms and multi-objective optimization and we provide initial experiments on the integration of weather data into EO analysis. All the provided methodologies are integrated/compatible with EO-learn library.
D5.1 PerceptiveSentinel design document
We have defined external, internal interfaces and standards for source code. In user requirements we have followed up on EO Toolset user requirements and design. Before that we have collected information from partners on additional needs. We needed to defined the source for each of the EO data-source, where we could get the data and under which conditions. The purpose of activities in this phase was to produced and refined the top-level architectural design of the PerceptiveSentinel platform, i.e. the top-level structure and software components.
D5.2 Data Gateways
In general, two types of data have been used in PerceptiveSentinel:
- EO data – stored on DIAS-es and AWS and distributed by Sentinel Hub
- non-EO data – stored and distributed by Geopedia
Outcome of first iteration were new EO data gateways constructed for data ingestion of SENTINEL-1, SENTINEL-2, SENTINEL -3, ENVISAT, MODIS. Based on initial analysis the Copernicus open imagery is sufficient for the analysis and use-cases. Commercial imagery is for the moment too expensive for end-users to purchase, especially due to repeated observations required, so the ingestion data would not be constructed with Planet and RapidEye data.
D2.6 Demonstration data set
For this deliverable the crop type grouping was used that was given by the various partners as it should group crops with similar characteristics and help us to choose which data to use. This grouping was provided for all the reported crops from 2016 to 2017 in Austria and Slovenia.
D3.4 Pre-processing chains
Various pre-processing methods for optical as well as RADAR satellite imagery have been implemented in the first part of the project. Sentinel-2 Level-2A data as well as orthorectified Sentinel-1 data, both considered good enough for majority of applications, are readily available. The EO-Toolset already covers all standard pre-processing steps necessary to create valuable services and products. The workflow option of the eo-learn library can be used to easily create chains of individual tasks.
D4.5 Deep learning algorithms
The deliverable describes deep learning algorithms implemented for “cultivated areas” extraction and “crop type” detection on Sentinel-2 data. It presents the studied deep learning architectures and explains the optimization in neural network training. The goal was to detect crop type by segmenting Sentinel-2 images using Deep learning algorithms, where we wanted to associate a class to every pixel of an image. However, one pixel does not contain enough information to be used alone for our problem, because we also need to use the neighbourhood of the pixel. By analysing the local texture and spectral variation we could extract a lot more useful information to classify the pixel. Crops can be very different from each other and it is very difficult to group them into categories that make sense in relation to the observation. The vegetable or orchards type contains for example very different products that do not look similar on observation. However Sentinel-2 various spectral bands are supposed to add a lot of additional information and while waiting for better crop separation we try to get the best results using the previous categories and the proposed models.
Using eo-learn we developed a training-dataset creation pipe:
First we load the previously generated masks for each tile. These masks are simple rasterization of the LPIS data where an RGB value is assigned to every category. These masks are made for each year and are used as is since we don’t have any corresponding time information.
We get these kinds of images:
Here is some of the image/mask couple from the learning dataset. We can see that the cloud detection fails in this case and the grass labellisation is not complete. To get better result we should remove those kinds of patches from the dataset but it cannot be done manually as the number of patches to filter is huge.
D4.7 Streaming-learning validation report
The deliverable summarizes the work accomplished in PerceptiveSentinel machine learning. The focus has been on effective Big Data processing, which has been achieved through the following approaches: (1) effective feature selection with FASTENER genetic algorithm, (2) usage of stream learning techniques and development of ml-rapids library and (3) with smart feature engineering so that accurate models can be built with as little features as possible. The biggest achievement of it is the FASTENER genetic algorithm which has proven to outperform the current state of the art in the field.
D5.3 PerceptiveSentinel core libraries
PerceptiveSentinel core libraries are founded on EO-Toolset's processing library, which have been upgraded with the introduction of pre-processing algorithms incorporated into eo-learn library. These are integrated with EO-QMiner to support streaming machine learning, new data gateways and cases showing how to use eo-learn library with external services.
D5.4 EO-QMiner Integration layer
Integration between the PerceptiveSentinel system and EO-QMiner is an essential integrative part of the platform and enables a bi-directional data exchange. The platform provides learning and interpretation data, while EO-QMiner provides the interpreted data. The integration of developed tasks into eo-learn has been accomplished for complete feature extraction and feature selection while other relevant tasks have also been tightly integrated within the eo-learn project platform.
D5.5 Integration services
Integration services support chaining into 3rd party front-offices as data integration services; WMS, WMTS and WFS data-source services, based on OGC standards. Integration between PerceptiveSentinel platform and external services enables data exchange in both ways. User can provide external service and run the service together which is shown with two external examples of services. Open-Source integration as tiled web-map services are provided for easy inclusion in open-source frameworks such as Open Layers, Leaflet. APIs are provided for integration of advanced functionality, such as time-lapse videos, statistical query results, change detection products, etc. and for integration with 3rd party products.
All software components have been integrated into one system: PerceptiveSentinel platform. Comprehensive testing, first integration and functional testing have been performed. System roll-out enables PerceptiveSentinel platform to run in DEMO operational mode to begin the validation and demonstration activities. Users receive all support and help, required to use the system and design processing chains. A bug reporting system has been established.
List of currently implemented EOTask is here . If the task is not yet implemented, a user can create a new EOTask without any overhead.
5.6. 1 NEW SCRIPTS IN EO LEARN
Perceptive Sentinel’s eo-learn repository on GitHub has until now received contribution from 23 developers. Out of all contributors around one third are not affiliated to any of the Perceptive Sentinel’s project partners. In addition, 175 GitHub users have made their own fork of the eo-learn repository, which is usually done when code modifications or additions are done. In future we expect to receive contributions in form of pull requests from some of these users.
The material for the community how to contribute to eo-learn has been prepared here. If the community would like to contribute to eo-learn could check out the contribution guidelines. Eo-learn is distributed under the MIT license. When contributing code to the library, the community agree to its terms and conditions. If the community would like to keep parts of its own contribution private, can contact to firstname.lastname@example.org or through our Forum
Example of use cases - Use of Sinergise Sentinel Hub on the CREODIAS EO Data Hub
.In order to run eo-learn on CREODIAS EO Data Hub, users need to be registered on the CREODIAS platform. After registration, an application for EO Data Hub access needs to be made and OGC settings must be configurated with the WMS configurator. More information on the EO Data Hub access and the WMS configurator are here: (https://CREODIAS.eu/use-of-wms-configurator). After a configuration has been set, data stored on EO Data Hub can be accessed through OGC access (WMS, WMTS, WFS or WCS) and be used in a number of applications, like QGIS, Google Earth or custom scripting. In this example, an existing eo-learn Python Jupyter Notebook is adapted so data stored on EO Cloud is used. All information could be found here.
The material has also been prepared for the Nordic Remote Sensing 2019 conference with everything set up beforehand. The main purpose is to encourage the community to run the tutorial by themselves. The community is able to run the tutorial and see eo-learn obtained meaningful information from satellite data with just a few lines of Python code.
eo-learn use-cases: Land/Crop-type mapping
• Land- and Crop-type mapping at scale
• Turkey: 800 000 km2
• Done in less than 2 weeks
• Human in the loop still a bottleneck
Division of area of interest into 33000 EOPatches (multiple UTM zones)
Produced Land Cover map
eo-learn use-cases: Building detection
• New building detection in very high resolution imagery
New buildings (red) that are not yet part of existing reference data (blue
eo-learn use-cases: Field delineation
• Automatic Field delineation using Sentinel-2
• Executed at a country scale
• Individuals new to Remote Sensing, Earth Observations
• EO4FoodSecurity challenge supported by EO4Agri provided environment installed with eo-learn
• ICLR Workshop Challenge #2: Radiant Earth Computer Vision for Crop Detection from Satellite Imagery: competitors started using eo-learn
• Companies working in Remote Sensing, Earth Observation or companies who find value in Earth Observation data
• Enterprise level subscribers of Sentinel Hub
• Generate over 80% of requests
• Promoted by Radiant Earth Foundation
• eo-learn (Perceptive Sentinel platform) is
• Enables application of Streaming Machine-Learning models
• Is already in use by project partners and other users to extract value from Earth Observation data
• eo-learn is open and free
• eo-learn will be used in other projects and its capabilities will be further developed
D6.1 EO VAS: Cultivated AREA
The deliverable presents the deep learning algorithms implemented by Magellium for “cultivated areas” extraction and “crop type” detection on Sentinel-2 data.
It presents deep learning architectures and explains the optimization in neural network training and describes the performed tests and the obtained results.
D6.2 EO VAS: Crop TYPE
A Jupyter-Notebook for the automatic classification of crop types with Sentinel-2 data has been developed. The application possibilities of a crop type algorithm are extremely diverse (e.g. identification of single class vs eligibility check in the CAP framework) and the scientific problem with it as well. For many cases, highly specialized algorithms might be needed, which in return would lower the applicability to other use cases. As a result, it was decided to provide a “toolbox” for crop type identification. This approach, manifested in a Jupyter Notebook, follows the full path from the raw data to the final result in a very transparent manner. It leaves the crop type grouping easily changeable and allows to be adapted by every user to be applicable to various use cases. Two classification approaches were implemented, a classical Light model as a reference, as well as a temporal CNN.
D6.3 EO VAS: Crop CYCLE
The Crop cycle service has been developed, which enables user to compare differences in growth of selected crop between particular regions or between selected polygons (field, larger territorial unit) with the region average. Results could serve in several ways: to predict relative differences in crop growth, as a tool to predict expected yield, to observe the extent of eventual damage event or to confirm less favorable conditions for crop growth. Crop Cycle Service, plays an important role for other Value Added Services (VAS) such as Crop Damage, Crop Yield, or indirectly also Soil moisture. The crop cycle has been illustrated as a comparison of crop development between regions or as comparison between areas of interest to the average of crop development in the region.
D6.4 EO VAS: Crop DAMAGE
A multitemporal crop damage algorithm was developed using the eo-learn library and implemented as a REST API web service. Due to a lack of representative and specific in-situ data a generic fitness map was derived first. It was translated into a crop damage and status map by evaluating the temporal evolution of each pixel in comparison to other parts of the field, or if available among various other fields of the same crop. The service can be called over a REST API, an eo task to trigger the REST API is provided.
D6.5 EO VAS: Moisture CONTENT
The quantification of moisture content (soil and vegetation) is one of the key benefits that remote sensing techniques can provide for agriculture. A Sentinel-1/Sentinel-2 Hybrid Algorithm for the detection of soil moisture was developed and the resulting data for three distinct regions and two consecutive years 2018 and 2019 was modelled (Lower Austria, Mura region in Slovenia and middle Jutland in Denmark, excluding areas close to the ocean). A Jupyter notebook is provided which can be used to access the data.
D6.6 EO VAS: Crop YIELD
Deliverable summarizes the development of EO VAS Crop Yield service - a model forecasting winter wheat yields during the growth season. The main task was to collect data and structure features used in the machine learning model and subsequently test the effect of feature engineering on the performance of the model.
By combining geo-referenced sensor data, optical satellite images, terrain height model and weather data for predicting yield within the field, a new tool for precision agriculture in the web-solution Crop Manager was developed. The Crop Yield Service can be implemented in other European Countries with similar soil types and cultivation practices.
D7.1 Demonstration USE-CASE
The deliverable presents integration of the PerceptiveSentinel Service ‘Crop Yield’ into the Danish precision farming system Crop Manager. The integration into Crop Manager is used for demonstrating external chaining capabilities of the PerceptiveSentinel platform. The model predicting winter wheat yield within the field during the growth season is a sophisticated product, whose value will increase through its usage, as new learning data (the data about actual yield) will enter the system at the end of each season. Crop Manager serves as an external front-office, consuming PerceptiveSentinel Services, upgrading them with specific agricultural services and providing a data link with Mark Online, thus delivering high added value to Danish farmers. The winter wheat prediction model provides the farmer with a valuable tool to improve the management of his farm, both in the field and in budgeting.
Mockup from user story
User story above describes that the user needs an overview enabling the farmer or farm advisor to select a specific field, that the yield prognosis should show. The option will give the user quick access to the specific field having the requested yield prognosis data. The design is also prepared for the next version of yield prognosis services, so that yield prognoses can be shown for other crops as well. The finished design looks like the mock-up. One difference is that the implemented design uses the field colours in the users’ norm set, which gives each crop type its own color.
Screenshot of user story implemented in CropManager
Screenshot of user story implemented in CropManager
In order to raise awareness about the project and promote PerceptiveSentinel platform and its added value, specific tasks have been devoted to promotional activities. Each of the partners contributing in its own way and addressing their part of user community. Dissemination activities have been undertaken with a primary goal to raise awareness about project outputs, to receive recognition by expert population and thus attract attention from intermediate-user community. End-user community have been approached as well. Dissemination actions have started early in the project execution and are continuing with the same pace also in the post-project period.
Project partners have successfully planned, coordinated and facilitated several communication activities tailored to target specific audience throughout the project. Through communication activities, project partners have promoted the project to specific target groups, attracted potential intermediate users and end users and established the circumstances for successful market entry. Awareness about the yield prediction model in Crop Manager has successfully been raised and added value promoted to all target groups, especially to the farmers and farm advisors.
D8.3 Initial PerceptiveSentinel Community
This deliverable presents a report on the overall community activities carried out in providing descriptions and statistics on all the different ways in which the consortium approached stakeholders in view of the community building targets. A key resource for the success of eo-learn is, of course, the community, both of remote sensing and machine learning experts. All of the consortium dissemination and communication activities have lead towards establishment of Initial PerceptiveSentinel community, and have provided a push required to successfully enter the market. Dissemination activities have been undertaken with a primary goal to raise awareness about project outputs, to receive recognition by expert population and thus attract attention from intermediate-user community. End-user community has been approached.