Setup
Django BASIN-3D is a Django app that acts as a Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets.
The setup instructions below assume familiarity with Django. For more details, see https://www.djangoproject.com/start/
Custom plugins are developed for a broker instance. See ~/example-django/ directory containing the app “mybroker” for a broker instance example with datasource “Alpha” at https://github.com/BASIN-3D/django-basin3d.
Install
If you haven’t created a Django project for your custom broker, create one. See https://www.djangoproject.com/start/ for details. Once the custom Django broker project is created, install the BASIN-3D source distribution to Python environment for your Django project.
Install a source distribution with pip:
$ pip install django-basin3d
Make sure your installation was successful:
$ python
>>> import django_basin3d
>>>
Django Settings
In the Django settings, add “django_basin3d” and its dependencies to your INSTALLED_APPS setting like this:
INSTALLED_APPS = [
'<your app>',
'rest_framework',
'django_basin3d',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles'
]
URLConf
Include the basin3d URLconf in your project urls.py like this:
from django.conf.urls import include, url
from django_basin3d import urls as db_urls
url(r'^', include('db_urls.urls')),
See ~/example-django/mybroker/urls.py for an example.
Implement Data Source plugins
Create one or more plugins in a plugins module in your-app/plugins.py. The following files must be placed in your-app/ directory along side plugins.py
basin3d_observed_property_vocabulary.csv
Hydrology variables have been defined in basin3d. These broker variables are in a comma separated values (csv) file named basin3d_observed_property_vocabulary.csv and can be found at https://github.com/BASIN-3D/basin3d.
<plugin_name>_mapping.csv
Map your measurement variables for your plugin variables. The name of the file should be <plugin_name>_mapping.csv. This file must be placed this in your-app/ directory (e.g your-app/alpha_mapping.csv) .
attr_type,basin3d_vocab,datasource_vocab,datasource_desc
OBSERVED_PROPERTY:SAMPLING_MEDIUM,ACT:WATER,Acetate,acetate
OBSERVED_PROPERTY:SAMPLING_MEDIUM,Ag:WATER,Ag,sliver concentration in water
OBSERVED_PROPERTY:SAMPLING_MEDIUM,Ag:GAS,Ag_gas,silver concentration vaporized (bogus)
OBSERVED_PROPERTY:SAMPLING_MEDIUM,Al:WATER,Aluminum,aluminum concentration in water
OBSERVED_PROPERTY:SAMPLING_MEDIUM,Al:WATER,Al,aluminum (Al) concentration in water
OBSERVED_PROPERTY:SAMPLING_MEDIUM,As:SOLID_PHASE,As,arsenic in soil
OBSERVED_PROPERTY:SAMPLING_MEDIUM,PPT:WATER,precip,precipitation
OBSERVED_PROPERTY:SAMPLING_MEDIUM,PPT_TOT_DAY:WATER,daily precip,daily total precipitation
STATISTIC,MEAN,mean,
STATISTIC,MAX,max,
STATISTIC,MIN,min,
RESULT_QUALITY,VALIDATED,VALIDATED,
RESULT_QUALITY,UNVALIDATED,UNVALIDATED,
RESULT_QUALITY,REJECTED,REJECTED,
AGGREGATION_DURATION,DAY,DAY,
AGGREGATION_DURATION,NONE,NONE,
Extend the broker source plugin with the described attributes. The following example is from ~example-django/mybroker/plugins.py.
import logging
from typing import Any, List
from basin3d.core.models import AbsoluteCoordinate, AltitudeCoordinate, Coordinate, DepthCoordinate, \
GeographicCoordinate, MeasurementTimeseriesTVPObservation, MonitoringFeature, RelatedSamplingFeature, \
RepresentativeCoordinate, SpatialSamplingShapes, VerticalCoordinate, ResultListTVP
from basin3d.core.plugin import DataSourcePluginPoint, basin3d_plugin, DataSourcePluginAccess
from basin3d.core.schema.enum import FeatureTypeEnum, TimeFrequencyEnum
from basin3d.core.schema.query import QueryMonitoringFeature, QueryMeasurementTimeseriesTVP
logger = logging.getLogger(__name__)
class AlphaMeasurementTimeseriesTVPObservationAccess(DataSourcePluginAccess):
"""
MeasurementTimeseriesTVPObservation Access class
"""
synthesis_model_class = MeasurementTimeseriesTVPObservation
def list(self, query: QueryMeasurementTimeseriesTVP):
"""
Generate a list of MeasurementTimeseriesTVPObservation objects
"""
synthesis_messages: List[str] = []
data: List[Any] = []
quality: List[Any] = []
# query = kwargs.get('query')
# assert query
if query.monitoring_feature == ['region']:
return StopIteration({"message": "FOO"})
supported_monitoring_features = [f'{num}' for num in range(1, 5)]
if not any([loc_id in supported_monitoring_features for loc_id in query.monitoring_feature]):
return StopIteration({"message": "No data from data source matches monitoring features specified."})
location_indices = []
for loc_id in query.monitoring_feature:
if loc_id in supported_monitoring_features:
location_indices.append(int(loc_id.split('-')[-1]))
from datetime import datetime
for num in range(1, 10):
data.append((datetime(2016, 2, num), num * 0.3454))
data = [data, data, [], data]
rqe1 = 'VALIDATED'
rqe2 = 'UNVALIDATED'
rqe3 = 'REJECTED'
quality = [[rqe1, rqe1, rqe1, rqe1, rqe1, rqe1, rqe1, rqe1, rqe1],
[rqe2, rqe2, rqe2, rqe2, rqe2, rqe2, rqe2, rqe3, rqe3],
[],
[rqe1, rqe2, rqe3, rqe1, rqe1, rqe1, rqe1, rqe1, rqe1]]
qualities = [[rqe1],
[rqe2, rqe3],
[],
[rqe1, rqe2, rqe3]]
observed_property_variables = ["Acetate", "Acetate", "Aluminum", "Al"]
units = ['nm', 'nm', 'mg/L', 'mg/L']
statistics = ['mean', 'max', 'mean', 'max']
for num in location_indices:
observed_property_variable = observed_property_variables[num - 1]
feature_id = f'A-{str(num - 1)}'
if query:
if observed_property_variable not in query.observed_property:
continue
if query.statistic:
if statistics[num - 1] not in query.statistic:
continue
result_value = data[num - 1]
result_value_quality = quality[num - 1]
result_qualities = qualities[num - 1]
if query.result_quality:
filtered_value = []
filtered_quality = []
has_filtered_data_points = 0
for v, q in zip(result_value, result_value_quality):
if q in query.result_quality:
filtered_value.append(v)
filtered_quality.append(q)
else:
has_filtered_data_points += 1
if has_filtered_data_points > 0:
synthesis_messages.append(f'{feature_id} - {observed_property_variable}: {str(has_filtered_data_points)} timestamps did not match data quality query.')
if len(filtered_value) == 0:
synthesis_messages.append(f'{feature_id} - {observed_property_variable}: No data values matched result_quality query.')
print(f'{feature_id} - {observed_property_variable}')
continue
result_value = filtered_value
result_value_quality = filtered_quality
if len(result_value_quality) > 0:
result_qualities = list(set(result_value_quality))
else:
result_qualities = []
yield MeasurementTimeseriesTVPObservation(
plugin_access=self,
id=num,
observed_property=observed_property_variable,
utc_offset=-8 - num,
feature_of_interest=MonitoringFeature(
plugin_access=self,
id=num,
name="Point Location " + str(num),
description="The point.",
feature_type=FeatureTypeEnum.POINT,
shape=SpatialSamplingShapes.SHAPE_POINT,
coordinates=Coordinate(
absolute=AbsoluteCoordinate(
horizontal_position=GeographicCoordinate(
units=GeographicCoordinate.UNITS_DEC_DEGREES,
latitude=70.4657, longitude=-20.4567),
vertical_extent=AltitudeCoordinate(
datum=AltitudeCoordinate.DATUM_NAVD88,
value=1500,
distance_units=VerticalCoordinate.DISTANCE_UNITS_FEET)),
representative=RepresentativeCoordinate(
vertical_position=DepthCoordinate(
datum=DepthCoordinate.DATUM_LOCAL_SURFACE,
value=-0.5 - num * 0.1,
distance_units=VerticalCoordinate.DISTANCE_UNITS_METERS)
)
),
observed_properties=["Ag", "Acetate", "Aluminum", "Al"],
related_sampling_feature_complex=[
RelatedSamplingFeature(
plugin_access=self,
related_sampling_feature="Region1",
related_sampling_feature_type=FeatureTypeEnum.REGION,
role=RelatedSamplingFeature.ROLE_PARENT)]
),
feature_of_interest_type=FeatureTypeEnum.POINT,
unit_of_measurement=units[num - 1],
aggregation_duration=TimeFrequencyEnum.DAY,
result_quality=result_qualities,
time_reference_position=None,
statistic=statistics[num - 1],
result=ResultListTVP(
plugin_access=self,
value=result_value, result_quality=result_value_quality)
)
return StopIteration(synthesis_messages)
class AlphaMonitoringFeatureAccess(DataSourcePluginAccess):
"""
MonitoringFeature access class
"""
synthesis_model_class = MonitoringFeature
def list(self, query: QueryMonitoringFeature):
"""
Generate list of MonitoringFeature objects
"""
feature_type = query.feature_type
monitoring_feature_list = query.monitoring_feature
obj_region = MonitoringFeature(
plugin_access=self,
id="Region1",
name="AwesomeRegion",
description="This region is really awesome.",
feature_type=FeatureTypeEnum.REGION,
shape=SpatialSamplingShapes.SHAPE_SURFACE,
coordinates=Coordinate(representative=RepresentativeCoordinate(
representative_point=AbsoluteCoordinate(
horizontal_position=GeographicCoordinate(
units=GeographicCoordinate.UNITS_DEC_DEGREES,
latitude=70.4657, longitude=-20.4567),
vertical_extent=AltitudeCoordinate(
datum=AltitudeCoordinate.DATUM_NAVD88,
value=1500,
distance_units=VerticalCoordinate.DISTANCE_UNITS_FEET)),
representative_point_type=RepresentativeCoordinate.REPRESENTATIVE_POINT_TYPE_CENTER_LOCAL_SURFACE)
)
)
if monitoring_feature_list and 'Region1' in monitoring_feature_list:
if feature_type == FeatureTypeEnum.REGION:
yield obj_region
elif feature_type == FeatureTypeEnum.REGION:
yield obj_region
obj_point = MonitoringFeature(
plugin_access=self,
id="1",
name="Point Location 1",
description="The first point.",
feature_type=FeatureTypeEnum.POINT,
shape=SpatialSamplingShapes.SHAPE_POINT,
coordinates=Coordinate(
absolute=AbsoluteCoordinate(
horizontal_position=GeographicCoordinate(
units=GeographicCoordinate.UNITS_DEC_DEGREES,
latitude=70.4657, longitude=-20.4567),
vertical_extent=AltitudeCoordinate(
datum=AltitudeCoordinate.DATUM_NAVD88,
value=1500,
distance_units=VerticalCoordinate.DISTANCE_UNITS_FEET)),
representative=RepresentativeCoordinate(
vertical_position=DepthCoordinate(
datum=DepthCoordinate.DATUM_LOCAL_SURFACE,
value=-0.5,
distance_units=VerticalCoordinate.DISTANCE_UNITS_METERS)
)
),
observed_properties=["Ag", "Acetate"],
related_sampling_feature_complex=[
RelatedSamplingFeature(
plugin_access=self,
related_sampling_feature="Region1",
related_sampling_feature_type=FeatureTypeEnum.REGION,
role=RelatedSamplingFeature.ROLE_PARENT)]
)
if monitoring_feature_list and '1' in monitoring_feature_list:
if feature_type == FeatureTypeEnum.POINT:
yield obj_point
elif feature_type == FeatureTypeEnum.POINT:
yield obj_point
def get(self, query: QueryMonitoringFeature):
"""
Get a Monitoring Feature objects
"""
# query.id will always be a string at this point with validation upstream, thus ignoring the type checking
prefixed_monitoring_feature = f'{self.datasource.id_prefix}-{query.id}' # type: ignore[list-item]
query.monitoring_feature = [query.id] # type: ignore[list-item]
for s in self.list(query):
if s.id == prefixed_monitoring_feature:
return s
return None
@basin3d_plugin
class AlphaDataSourcePlugin(DataSourcePluginPoint):
title = 'Alpha Data Source Plugin'
plugin_access_classes = (AlphaMeasurementTimeseriesTVPObservationAccess, AlphaMonitoringFeatureAccess)
feature_types = ['REGION', 'POINT', 'TREE', 'HORIZONTAL_PATH']
class DataSourceMeta:
"""
This is an internal metadata class for defining DataSource attributes.
"""
# Data Source attributes
location = 'https://asource.foo/'
id = 'Alpha' # unique id for the datasource
id_prefix = 'A'
name = id # Human Friendly Data Source Name
Migrate the App
Run python manage.py migrate to create the Django BASIN-3d models. This will create the database and load the app’s plugins.
Run the Server
Start the development server:
$ bin/python manage.py runserver
Visit http://127.0.0.1:8000/ to view the REST API.
Visit http://127.0.0.1:8000/admin/ to manage a BASIN-3D models (you’ll need the Admin app enabled in the project’s urls.py):
url(r'^admin/', include(admin.site.urls)),
To create an admin user:
./manage.py createsuperuser
To exit running the server, control + C.