Quick Guide

Hook up Data Sources

To access a data source, its BASIN-3D plugin must be configured. Custom plugins can be built. Documentation will be forthcoming in future versions. Current available plugins:

If a plugin exists for the data source, import it before registering a synthesizer.

from basin3d.plugins import <plugin_name>

Register a Synthesizer

basin3d.synthesis.register(plugins=None)[source]

Register the specified plugins or implicitly register loaded plugins

>>> from basin3d import synthesis
>>> synthesizer = synthesis.register(['basin3d.plugins.usgs.USGSDataSourcePlugin'])
>>> synthesizer.datasources
[DataSource(id='USGS', name='USGS', id_prefix='USGS', location='https://waterservices.usgs.gov/nwis/', credentials={})]
Parameters:

plugins (Optional[List[str]]) – [Optional] plugins to registered

Returns:

DataSynthesizer(plugin_dict, catalog)

Get a List of Locations for a Data Source

basin3d.synthesis.DataSynthesizer.monitoring_features(self, query=None, **kwargs)

Search for all Monitoring Features, Monitoring Features by parent monitoring features, or Monitoring Feature by id(s).

To see feature types for a given plugin: <plugin_module>.<plugin_class>.feature_types

Search for a single monitoring feature by id:

>>> from basin3d.plugins import usgs, epa
>>> from basin3d import synthesis
>>> synthesizer = synthesis.register()
>>> response = synthesizer.monitoring_features(id='USGS-0101')
>>> mf = response.data
>>> print(f"{mf.id} - {mf.description}")
USGS-0101 - SUBREGION: St. John

Search for all USGS monitoring features:

>>> for mf in synthesizer.monitoring_features(datasource='USGS', feature_type='region'): 
...     print(f"{mf.id} - {mf.description}")
USGS-01 - REGION: New England
USGS-02 - REGION: Mid Atlantic
USGS-03 - REGION: South Atlantic-Gulf
...

Search for USGS points by parent (subbasin) monitoring features:

>>> for mf in synthesizer.monitoring_features(feature_type='point',parent_feature=['USGS-17040101']): 
...    print(f"{mf.id} {mf.coordinates and [(p.x, p.y) for p in mf.coordinates.absolute.horizontal_position]}")
USGS-13010000 [(-110.6647222, 44.1336111)]
USGS-13010065 [(-110.6675, 44.09888889)]
USGS-13010450 [(-110.5874305, 43.9038296)]
...

Search for USGS points by monitoring features identifiers:

>>> for mf in synthesizer.monitoring_features(feature_type='point', monitoring_feature=['USGS-13010000', 'USGS-13010450']): 
...    print(f"{mf.id} {mf.coordinates and [(p.x, p.y) for p in mf.coordinates.absolute.horizontal_position]}")
USGS-13010000 [(-110.6647222, 44.1336111)]
USGS-13010450 [(-110.5874305, 43.9038296)]

Unsupported feature types warning:

The code below is an example of what you will see if a registered plugin does not support the requested feature type.

>>> response_itr = synthesizer.monitoring_features(feature_type='horizontal_path')
>>> for mf in response_itr:
...   print(mf)
...

Output warning messages from the returned iterator

This is an example of checking the synthesis response messages in the basin3d.core.synthesis.DataSourceModelIterator.

>>> response_itr.synthesis_response.messages
[SynthesisMessage(msg='Feature type HORIZONTAL_PATH not supported by USGS.', level='WARN', where=['USGS', 'MonitoringFeature']), SynthesisMessage(msg='Feature type HORIZONTAL_PATH not supported by EPA Water Quality eXchange.', level='WARN', where=['EPA', 'MonitoringFeature'])]
Parameters:
Return type:

Union[DataSourceModelIterator, SynthesisResponse]

Returns:

a single SynthesisResponse for a query by id or a DataSourceModelIterator for multple basin3d.core.models.MonitoringFeature objects.

Note

Monitoring Feature Query parameters basin3d.core.schema.query.QueryMonitoringFeature

All parameters are optional.

* datasource A single data source id prefix.
* feature_type The basin3d.core.schema.enum.FeatureTypeEnum of the desired Monitoring Feature(s). Data Sources may not support all Feature Types.

Only one of the following can be specified in a query:

* id A single Monitoring Feature ID for the Monitoring Feature desired. Returns a single Synthesis Response object.
* monitoring_feature List of Monitoring Feature IDs for the Monitoring Features desired. Returns iterator of MonitoringFeature objects.
* parent_feature List of Monitoring Feature IDs for the parent features of the desired Monitoring Features. Returns iterator of MonitoringFeature objects.

Note

Monitoring Feature attributes basin3d.core.models.MonitoringFeature

Attributes values are dependent on data source features.

* id Unique feature identifier, prefixed by data source id
* name Feature name
* description Description of the Monitoring Feature
* feature_type basin3d.core.schema.enum.FeatureTypeEnum REGION, SUBREGION, BASIN, SUBBASIN, WATERSHED, SUBWATERSHED, SITE, PLOT, HORIZONTAL PATH, VERTICAL PATH, POINT
* observed_properties List of observed properties basin3d.core.models.ObservedProperty collected at the feature.
* related_sampling_feature_complex List of basin3d.core.models.RelatedSamplingFeature. PARENT features are currently supported.
* shape Shape of the feature: POINT, CURVE, SURFACE, SOLID
* coordinates Location of feature in absolute and/or representative datum: basin3d.core.models.Coordinate
* description_reference Additional information about the feature
* related_party List of people or organizations responsible for the feature
* utc_offset Coordinate Universal Time offset in hours (offset in hours), e.g., +9
* datasource The feature’s data source basin3d.core.models.DataSource

See Monitoring Feature Object Structure

Get a List of Attribute Mappings for a Data Source

basin3d.synthesis.DataSynthesizer.attribute_mappings(self, datasource_id=None, attr_type=None, attr_vocab=None, from_basin3d=False)
>>> from basin3d.plugins import usgs
>>> from basin3d import synthesis
>>> synthesizer = synthesis.register()
>>> response = synthesizer.attribute_mappings()  # list all attribute mappings registered
>>> for attr_mapping in response:
...     print(f'{attr_mapping.attr_type} | {attr_mapping.basin3d_vocab} -- {attr_mapping.datasource_vocab}')
OBSERVED_PROPERTY:SAMPLING_MEDIUM | PH:WATER -- 00400
OBSERVED_PROPERTY:SAMPLING_MEDIUM | RDC:WATER -- 00060
OBSERVED_PROPERTY:SAMPLING_MEDIUM | WLE:WATER -- 63161
OBSERVED_PROPERTY:SAMPLING_MEDIUM | WT:WATER -- 00010
OBSERVED_PROPERTY:SAMPLING_MEDIUM | DO:WATER -- 00300
...

>>> response = synthesizer.attribute_mappings(datasource_id='USGS', attr_type='STATISTIC')
>>> for attr_mapping in response:
...     print(f'{attr_mapping.attr_type} | {attr_mapping.basin3d_vocab} -- {attr_mapping.datasource_vocab}')
STATISTIC | MEAN -- 00003
STATISTIC | MIN -- 00002
STATISTIC | MAX -- 00001
STATISTIC | TOTAL -- 00006

>>> response = synthesizer.attribute_mappings(datasource_id='USGS', attr_type='RESULT_QUALITY', attr_vocab=['VALIDATED', 'ESTIMATED'], from_basin3d=True)
>>> for attr_mapping in response:
...     print(f'{attr_mapping.attr_type} | {attr_mapping.basin3d_vocab} -- {attr_mapping.datasource_vocab}, {attr_mapping.datasource_desc}')
RESULT_QUALITY | ESTIMATED -- e, Value has been edited or estimated by USGS personnel and is write protected
RESULT_QUALITY | ESTIMATED -- E, Value was computed from estimated unit values.
RESULT_QUALITY | VALIDATED -- A, Approved for publication -- Processing and review completed.

Return all the basin3d.core.models.AttributMapping registered or those that match the specified fields.

Parameters:

  • datasource_id – str, The datasource identifier

  • attr_type – str, The attribute type (e.g., OBSERVED_PROPERTY, STATISTIC, etc)

  • attr_vocab – str, The attribute vocabulary, either the BASIN-3D vocabulary or the datasource vocabulary

  • from_basin3d – bool, True = the specified attr_vocab is a BASIN-3D vocabulary, False: the specified attr_vocab is from the datasource

Returns:

iterator of basin3d.core.models.AttributeMapping objects

Get a List of Variables Supported by a BASIN-3D

basin3d.synthesis.DataSynthesizer.observed_properties(self)
>>> from basin3d.plugins import usgs
>>> from basin3d import synthesis
>>> synthesizer = synthesis.register()
>>> response = synthesizer.observed_properties()
>>> for opv in response:
...     print(f'{opv.basin3d_vocab} -- {opv.full_name} -- {opv.units}')
ACT -- Acetate (CH3COO) -- mM
Br -- Bromide (Br) -- mM
Cl -- Chloride (Cl) -- mM
DIN -- Dissolved Inorganic Nitrogen (Nitrate + Nitrite) -- mg/L
DTN -- Dissolved Total Nitrogen (DTN) -- mg/L
F -- Fluoride (F) -- mM
...

BASIN-3D observed properties. An observed property defines what is being measured. Data source observed property vocabularies are mapped and thus synthesized to the BASIN-3D observed property vocabulary.

Returns:

an iterator of basin3d.core.models.ObservedProperty objects

Get Time Series Data

basin3d.synthesis.DataSynthesizer.measurement_timeseries_tvp_observations(self, query=None, **kwargs)

Search for Measurement Timeseries TVP Observations for the specified query arguments.

Aggregation Duration for DAY (default) and NONE are both supported.

Search with aggregation duration DAY:

>>> from basin3d.plugins import usgs
>>> from basin3d import synthesis
>>> synthesizer = synthesis.register()
>>> timeseries = synthesizer.measurement_timeseries_tvp_observations(monitoring_feature=['USGS-09110990'],observed_property=['RDC','WT'],start_date='2019-10-01',end_date='2019-10-30',aggregation_duration='DAY')
>>> for timeseries in timeseries:
...    print(f"{timeseries.feature_of_interest.id} - {timeseries.observed_property.get_basin3d_vocab()}")
USGS-09110990 - RDC

Search with aggregation duration NONE:

>>> from basin3d.plugins import usgs
>>> from basin3d import synthesis
>>> synthesizer = synthesis.register()
>>> timeseries = synthesizer.measurement_timeseries_tvp_observations(monitoring_feature=["USGS-09110990", "USGS-09111250"],observed_property=['RDC','WT'],start_date='2020-04-01',end_date='2020-04-30',aggregation_duration='NONE')
>>> for timeseries in timeseries:
...    print(f"{timeseries.feature_of_interest.id} - {timeseries.observed_property.get_basin3d_vocab()}")
USGS-09110990 - RDC
USGS-09111250 - RDC
Parameters:
Return type:

DataSourceModelIterator

Returns:

a DataSourceModelIterator that yields basin3d.core.models.MeasurementTimeseriesTVPObservation objects

Note

Measurement Timeseries TVP Query parameters basin3d.core.schema.query.QueryMeasurementTimeseriesTVP

Required arguments:

* monitoring_feature List of monitoring features id(s)
* observed_property List of observed property(ies), i.e., BASIN-3D observed property vocabulary. See basin3d.synthesis.DataSynthesizer.observed_properties()
* start_date Start date YYYY-MM-DD

Optional arguments:

* end_date End date YYYY-MM-DD
* aggregation_duration A single aggregation duration basin3d.core.schema.enum.AggregationDurationEnum (YEAR|MONTH|DAY|HOUR|MINUTE|SECOND|NONE)
* statistic List of statistic(s) basin3d.core.schema.enum.StatisticEnum (MEAN|MIN|MAX|INSTANTANEOUS)
* result_quality List of result quality(ies) basin3d.core.schema.enum.ResultQualityEnum (VALIDATED|UNVALIDATED|SUSPECTED|REJECTED|ESTIMATED)
* sampling_medium List of sampling medium(s) basin3d.core.schema.enum.SamplingMediumEnum (SOLID_PHASE|WATER|GAS|OTHER)
* datasource A single data source id prefix

Note

Measurement Timeseries TVP Observation attributes basin3d.core.models.MeasurementTimeseriesTVPObservation

Attributes values are dependent on data source features.

basin3d.core.models.MappedAttribute are returned for several attributes so that the data source values are also available.

* id Observation identifier
* type Type of observation: MEASUREMENT_TVP_TIMESERIES
* observed_property The observation’s observed property basin3d.core.models.MappedAttribute
* datasource The data source basin3d.core.models.DataSource
* sampling_medium Observed property sampling medium basin3d.core.models.MappedAttribute (SOLID_PHASE, WATER, GAS, OTHER)
* phenomenon_time Datetime of the observation, for a timeseries the start and end times can be provided
* utc_offset Coordinate Universal Time offset in hours (offset in hours), e.g., +9
* feature_of_interest Monitoring Feature object basin3d.core.models.MonitoringFeature, feature on which the observation is being made
* feature_of_interest_type Feature type of the feature of interest, basin3d.core.schema.enum.FeatureTypeEnum
* result Observed values of the observed property being assessed, and (opt) their result quality, basin3d.core.models.ResultListTVP
* time_reference_position Position of timestamp in aggregated_duration (START, MIDDLE, END)
* aggregation_duration Time period represented by observation basin3d.core.models.MappedAttribute (YEAR, MONTH, DAY, HOUR, MINUTE, SECOND)
* unit_of_measurement Units in which the observation is reported
* statistic Statistical property of the observation result basin3d.core.models.MappedAttribute (MEAN, MIN, MAX, TOTAL)
* result_quality List quality assessment found in the results basin3d.core.models.MappedAttribute (VALIDATED, UNVALIDATED, SUSPECTED, REJECTED, ESTIMATED)

See Measurement Timeseries TVP Observations Object Structure

Logging

Use basin3d.monitor.configure() to configure python logging.

The example below changes the log level to DEBUG. Please refer to the Python logging documentation (https://docs.python.org/3/howto/logging.html#configuring-logging) on logging configuration options. configure() returns the logging configuration as a dictionary for review.

>>> from basin3d import monitor, synthesis
>>> from basin3d.plugins import usgs
>>> monitor.configure(loggers={"basin3d": {"level": "DEBUG"}})
{'version': 1, 'incremental': False, 'disable_existing_loggers': True,
...

Once logging is configured, all BASIN-3D logging is outputted.

>>> synthesizer = synthesis.register()
2022-04-14T16:59:31.082 INFO * basin3d.core.synthesis * - Loading Plugin = USGSDataSourcePlugin
2022-04-14T16:59:31.083 DEBUG * basin3d.core.catalog * - Initializing CatalogTinyDb metadata catalog
2022-04-14T16:59:31.084 INFO * basin3d.core.catalog * - Loading metadata catalog for Plugin USGS
2022-04-14T16:59:31.085 DEBUG * basin3d.core.catalog * - Mapping file mapping_usgs.csv for plugin package basin3d.plugins
2022-04-14T16:59:31.086 DEBUG * basin3d.core.catalog * - Mapped 00400 to pH
2022-04-14T16:59:31.087 DEBUG * basin3d.core.catalog * - Mapped 00060 to River Discharge
...

To create and output custom log messages:

  1. Use basin3d.monitor.get_logger() to create a custom logger.

  2. Use basin3d.monitor.configure() to configure the custom logger output.

>>> from basin3d import monitor
>>> monitor.configure(loggers={"my_logger": {"level": "DEBUG",
...         'handlers': ['error-console', 'console'], 'propagate': True}})
{'version': 1, 'incremental': False, 'disable_existing_loggers': True,
...
>>> logger = monitor.get_logger("my_logger")
>>> logger.info("My logging message")
2022-04-14T16:57:36.163 INFO * my_logger * - My logging message

Write log messages to a file called basin3d.log:

>>> from basin3d import monitor, synthesis
>>> from basin3d.plugins import usgs
>>> monitor.configure(
...    handlers={  "file":
...       { "class":"logging.FileHandler", "formatter": "simple", "filename": ".}/basin3d.log"}},
...   loggers={"basin3d": {"handlers": ["console", "file"]}})
{'version': 1, 'incremental': False, 'disable_existing_loggers': True,
...
>>> synthesizer = synthesis.register()
2022-04-14T17:32:50.502 INFO * basin3d.core.synthesis * - Loading Plugin = USGSDataSourcePlugin
2022-04-14T17:32:50.507 INFO * basin3d.core.catalog * - Loading metadata catalog for Plugin USGS
2022-04-14T17:32:50.509 INFO * basin3d.core.catalog * - Initialized CatalogTinyDb metadata catalog