1. 08 Feb, 2017 1 commit
    • Daniel Scheffler's avatar
      Added support for atmospheric correction of Landsat-7 and Landsat-8. · 5f91570c
      Daniel Scheffler authored
      algorithms.gms_object.GMS_object:
      - arr.setter: catched an error during setting of arr.bandnames
      - revised ac_options
      algorithms.L1C_P:
      - L1C_object:
          - revised logger.deleter
          - added property 'options'
          - added _get_mask_clouds()
          - revised run_atmospheric_correction()
          - revised _join_results_to_inObjs()
      database:
      - removed folder ac_options -> AC options are now directly imported fom S2SCAPEM (thus version controlled)
      misc.path_generator:
      - revised get_path_ac_options() -> AC options are now directly imported fom S2SCAPEM (thus version controlled)
      - updated __version__
      Former-commit-id: a75ee7c3
      Former-commit-id: 0657813b
      5f91570c
  2. 27 Jan, 2017 1 commit
  3. 26 Jan, 2017 2 commits
    • Daniel Scheffler's avatar
      revised calculation of acquisition geometry arrays for Landsat · 8c7a5323
      Daniel Scheffler authored
      algorithms.GEOPROCESSING:
      - calc_VAA_using_fullSceneCornerLonLat():
          - implemented keyword 'orbit_params' as fallback; added docstring
          -added 90 degrees to VAA
      - calc_VZA_array(): added some notes
      - calc_RAA_array(): added docstring
      algorithms.L1A_P.L1A_object():
      - calc_mean_VAA(): updated calc_VAA_using_fullSceneCornerLonLat call
      algorithms.L1C_P.L1C_object():
      - VZA_arr.getter: added parameter 'meshwidth'
      - revised VAA_arr.getter
      algorithms.METADATA:
      - get_orbit_params(): added some notes
      - updated __version__
      Former-commit-id: b8a055bf
      Former-commit-id: d08a8a57
      8c7a5323
    • Daniel Scheffler's avatar
      Fixed unexpected value in cloud mask of Sentinel-2 · c8d34ab1
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      algorithms.L1C_P.AtmCorr:
      - _join_results_to_inObjs(): fixed unexpected value in cloud mask of Sentinel-2
      - updated __version__
      Former-commit-id: 89a43619
      Former-commit-id: 93bc4d81
      c8d34ab1
  4. 25 Jan, 2017 6 commits
    • Daniel Scheffler's avatar
      Bugfix for logging errors during creation of MGRS tiles · 096d38a1
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      algorithms.gms_object.GMS_object:
      - to_MGRS_tiles(): loggers are now properly closed
      - to_ENVI(): added logger close command
      processing.process_controller:
      - L1C_processing(): increased CPUs for L1C_map to 15
      - updated __version__
      Former-commit-id: 70c42a32
      Former-commit-id: 0260f23e
      096d38a1
    • Daniel Scheffler's avatar
      Bugfix for exception when merging multiple sensor subsystems · 0fdfaaed
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      algorithms.GEOPROCESSING:
      - added get_common_extent()
      algorithms.L1B_P,L1B_object:
      - implemented clipextent and clipextent_prj keywords
      processing.pipeline.L2A_map:
      - updated correct_spatial_shifts call
      - updated __version__
      Former-commit-id: 3aec298d
      Former-commit-id: c860ecc3
      0fdfaaed
    • Daniel Scheffler's avatar
      Fixed logging errors in multiprocessing · ce2e9a10
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      algorithms.gms_object.GMS_object:
      - from_sensor_subsystems(): loggers of input objects are now properly closed
      processing.pipeline:
      - L2A_map(): now returns a list because generators cannot be pickled
      processing.process_controller:
      - L1C_processing(): set CPUs to 10 for testing
      - updated __version__
      Former-commit-id: 9a399df5
      Former-commit-id: 5a80e7b3
      ce2e9a10
    • Daniel Scheffler's avatar
      Silenced DEM generator · 4673ef7f
      Daniel Scheffler authored
      io.Input_reader:
      - get_dem_by_extent(): silenced stdout output
      - updated __version__
      Former-commit-id: cb07972c
      Former-commit-id: 785e70df
      4673ef7f
    • Daniel Scheffler's avatar
      Fixed missing data in DEM · 0e6f5dbf
      Daniel Scheffler authored
      misc.database_tools:
      - get_pgSQL_geospatial_query_cond(): changed default queryfunc
      - get_overlapping_MGRS_tiles(): replaced geo conditions by ST_Intersects
      - updated __version__
      Former-commit-id: d06817da
      Former-commit-id: e0107c5c
      0e6f5dbf
    • Daniel Scheffler's avatar
      Fixed pickling issues after failed AC. · 2636cfcb
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      algorithms.L1C_P.AtmCorr:
      - run_atmospheric_correction(): AC input arrays are now deleted before object is passed to exception handler
      processing.pipeline:
      - L1C_map(): disabled auto-dumping of AC inputs in case of error
      processing-process_controller:
      - L1C_processing(): set CPUs to 5 for testing
      - updated __version__
      Former-commit-id: 2a997e23
      Former-commit-id: 064a13b1
      2636cfcb
  5. 24 Jan, 2017 10 commits
  6. 23 Jan, 2017 1 commit
  7. 21 Jan, 2017 2 commits
    • Daniel Scheffler's avatar
      added some resources for atmospheric correction · f48afc34
      Daniel Scheffler authored
      - added ac_options for Sentinel-2
      - added cloud classier for Sentinel-2
      - added SNR model for Sentinel-2
      config:
      - reactivated exception handler
      - updated __version__
      Former-commit-id: dc2977bc
      Former-commit-id: 986c4d1d
      f48afc34
    • Daniel Scheffler's avatar
      first version fully operable for Sentinel-2 (including atmospheric correction and cloud masks) · 3282d365
      Daniel Scheffler authored
      algorithms.gms_object.GMS_object:
      - GMS_identifier: is only a getter now, not a singleton
      - added property 'MetaObj' / 'meta_odict' -> self-synchronizing
      - added LayerBandsAssignment.setter
      - revised arr.setter
      - revised mask_nodata.setter
      - mask_clouds.getter: bugfix
      - revised mask_clouds.setter
      - added property 'mask_clouds_confidence'
      - revised masks.setter
      - revised dem.setter
      - ac_options: number of CPUs are now passed (temporarily disabled)
      - revised ac_errors.setter
      - added LBA2bandnames
      - attributes2dict: bugfix for not addin meta_odict
      - revised from_tiles()
      - added from_sensor_subsystems(): a function for merging multiple subsystems of the same sensor (needed for Sentinel-2 and ASTER)
      - revised build_combined_masks_array()
      - get_subset_obj(): multiple bugfixes
      - to_GMS_file(): updated in the context of new property architecture
      - delete_previous_proc_level_results: added functionality to delete subsystem products after subsystems have been merged
      algorithms.gms_object.failed_GMS_object:
      - removed logger creation
      algorithms.L1A_P.L1A_object:
      - import_metadata(): updated METADATA call
      - calc_TOARadRefTemp: added warning when using Sentinel-2 Gains
      - calc_corner_positions(): added UTC timezone to datetime object
      algorithms.L1B_P.L1B_object:
      - _get_reference_image_params_pgSQL(): replaced temp_logger by self.logger
      - revised correct_spatial_shifts()
      algorithms.L1C_P:
      - L1C_object:
          - removed deprecated atm_corr()
      - AtmCorr:
          - __init__: added warning
          - revised data property
          - _meta_get_aux_data(): changed lonlat_arr to float16
          - run_atmospheric_correction(): added 'dump' keyword
          - revised _join_results_to_inObjs()
      algorithms.L2B_P.L2B_object:
      - spectral_homogenization(): bugfix
      algorithms.METADATA:
      - METADATA:
          - __init__(): is now initialized by GMS_identifier; does not directly run metadata reader
          - added read_meta()
          - Read_Sentinel2A_xmls(): updated setting of Gains
          - refactored to_meta_odict() to to_odict() and revised the function
          - added from_odict()
          - revised filter_layerdependent_metadata()
          - added 'map_odictKeys_objAttrnames'
          - get_LayerBandsAssignment():
              - added functionality to return full LBA for Sentinel-2 and ASTER
              - now properly handles bands removed after L1C and L2B
      io.Output_writer:
      - mask_to_ENVI_Classification(): bugfix
      misc.definition_dicts:
      - get_GMS_sensorcode(): added codes for Sentinel-2 full and ASTER full
      - get_outFillZeroSaturated(): added bool dtype
      - is_dataset_provided_as_fullScene(): added ASTER full and Sentinel-2 full
      misc.exception_handler:
      - log_uncaught_exceptions(): bugfix for 'disable_exception_handler'
      misc.helper_functions:
      - cut_GMS_obj_into_blocks(): bugfix
      misc.logging.GMS_logger:
      - added scene ID to formatter
      misc.mgrs_tile:
      - replaced deprecated reference
      misc.path_generator:
      - get_baseN(): bugfix
      - added get_path_ac_input_dump()
      processing.multiproc:
      - MAP: added functionality to disable multiprocessing
      processing.pipeline:
      - revised L2A_map(): new L2A calls
      processing.process_controller:
      - add_local_availability(): changed get_LayerBandsAssignment call
      - get_DB_objects(): bugfix for copied memory link during GMS object init
      - L1C_processing() raises NotImplementedError in tiled mode
      - L2A_processing(): added grouping of subsystems
      - updated __version__
      3282d365
  8. 16 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      third version of wrapper for atmospheric correction (first working version) · 6fa974e0
      Daniel Scheffler authored
      algorithms.GEOPROCESSING:
      - GEOPROCESSING: conversion_type_optical value renamed from 'Ref' to 'TOA_Ref' and 'BOA_Ref'
      - added keyword 'meshwidth' to the following functions (allows much faster processing):
          - zoom_2Darray_to_shapeFullArr()
          - adjust_acquisArrProv_to_shapeFullArr()
          - get_lonlat_coord_array():revised calculation of meshgrid
          - calc_VZA_array()
          - calc_AcqTime_array()
          - calc_SZA_SAA_array()
      - calc_RAA_array(): now receives a VAA_array instead of VAA_mean
      algorithms.gms_object:
      - added attributes 'fullSceneCornerPos' and 'fullSceneCornerPos'
      - logger: added assertion
      - added property 'log'
      - added dem deleter
      - revised property 'ac_options'
      - added property 'ac_errors'
      - added property 'subset'
      - refactored attribute 'acquisition_date' to 'acq_datetime' containing a full datetime timestamp in UTC time zone
      - to_GMS_file(): updated timestamp format
      algorithms.L1A_P.L1A_object:
      - get_MetaObj(): now also updates 'acq_datetime'
      - refactored get_MetaObj() to import_metadata()
      - included set_arr_desc_from_MetaObj() in import_metadata()
      - calc_TOARadRefTemp(): updated in the context of "conversion_type_optical" value change to 'TOA_Ref'
      - calc_cloud_mask(): temporarily excluded Sentinel-2 here; added code draft of S2A cloud mask calculation
      - calc_corner_positions(): major revision -> now calculates trueDataCornerPos/-LonLat AND fullSceneCornerPos/-LonLat
      - calc_center_AcqTime() now also updates 'acq_datetime'
      algorithms.L1B_P.ref_Scene:
      - moved _get_reference_image_params_pgSQL() and _sceneIDList_to_filt_overlap_scenes() to L1B_object
      algorithms.L1C_P.L1C_object:   -> major revision
      - added properties 'lonlat_arr', 'VZA_arr', 'VAA_arr', 'SZA_arr', 'SAA_arr', 'RAA_arr' based on get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
      - removed deprecated functions get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
      - revised delete_ac_input_arrays()
      algorithms.L1C_P.AtmCorr:
      - revised property 'logger'
      - metadata: added some tests
      - revised _meta_get_viewing_zenith(), _meta_get_viewing_azimuth(), _meta_get_relative_viewing_azimuth()
      - added _meta_get_aux_data()
      - added _get_dem()
      - added dummy version of _get_srf()
      - run_atmospheric_correction(): added docstring; some minor revisions
      - _join_results_to_inObjs(): now working
      algorithms.METADATA:
      - added property AcqDateTime: returns a full datetime object with UTC timezone
      - revised setters for AcqDate, AcqTime and AcqDateTime -> timezone now properly handled
      - refactored 'Meta2ODict' to 'to_meta_odict'
      - calc_center_acquisition_time(): now also sets AcqDateTime
      - get_LayerBandsAssignment(): processing level is now properly handled (in the context of missing bands after atmospheric correction)
      misc.database_tools:
      - renamed keyword 'trueDataCornerLonLat' to 'tgt_corners_lonlat' in the following functions:
          - get_pgSQL_geospatial_query_cond()
          - get_overlapping_scenes_from_postgreSQLdb()
          - get_overlapping_MGRS_tiles()
          - get_overlapping_MGRS_tiles2()
      misc.definitions_dicts:
      - added is_dataset_provided_as_fullScene()
      misc.exception_handler:
      - log_uncaught_exceptions:  exception handling is now optional and can be turned off via config
      misc.helper_functions:
      - modified some docstrings
      misc.logging.GMS_logger:
      - added property 'captured_stream' (not yet working)
      - added draft of StringIO handler (not yet working)
      misc.logging.path-generator:
      - get_path_srf_file: bugfix
      processing.pipeline:
      - L1A_map(): updated calls
      - L1A_map_1(): updated calls
      - revised L1C_map()
      processing.process_controller:
      - add_local_availability(): added comments and revised structure
      config:
      - Job:
          - added attribute 'disable_exception_handler'
          - added attribute 'scale_factor_BOARef'
          - added attribute 'scale_factor_errors_ac'
          - added assertion
      pg_SQLdb:
      - table config:
          - added fields 'scale_factor_BOARef' and 'scale_factor_errors_ac'
          - changed value of 'conversion_type_optical' to 'BOA_Ref'
      - updated __version__
      6fa974e0
  9. 11 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      second (not completely working) version of wrapper for atmospheric correction · d06523f9
      Daniel Scheffler authored
      algorithms.GEOPROCESSING:
      - revised imports
      algorithms.gms_object:
      - gms_object:
          - revised imports
          - added property 'dem': gms_object can now directly provide a corresponding SRTM DEM
          - arr: bandnames of property 'arr' are now in the form [B01, .., B8A,] and correspond to LayerBandsAssignment
          - added property 'ac_options': getter or options dictionary needed for atmospheric correction
          - from_disk(): added time zone to 'acquisition_date' datetime object
      - added class failed_GMS_object (based on earlier version from helper functions)
      algorithms.L1A_P.L1A_object:
      - revised imports
      - calc_TOARadRefTemp(): bugfix for wrong nodata value in out returned array in case of Sentinel-2
      - update_spec_vals_according_to_dtype: bugfix for not updating L1A_object.arr.nodata
      algorithms.L1B_P:
      - revised imports
      algorithms.L1C_P:
      - L1C_object:
          - get_lonlat_coord_array(): changed handling of return values
          - calc_acquisition_illumination_geometry(): changed handling of return values
      - AtmCorr:
          -  added attribute 'ac_input' containing input args/kwargs of atmospheric correction
          - data: now uses L1C_obj.arr.bandnames for lopping over bands
          - added property 'nodata'
          - added property 'tile_name'
          - added property 'band_spatial_sampling'
          - added property 'nodata'
          - added property 'nodata'
          - revised property 'metadata'
          - added _meta_get_spatial_samplings()
          - added _meta_get_solar_irradiance()
          - added _meta_get_viewing_zenith()
          - added _meta_get_viewing_azimuth()
          - added _meta_get_relative_viewing_azimuth()
          - revised run_atmospheric_correction()
          - revised join_results_to_inObjs()
      algorithms.METADATA
      - revised imports
      - added 'ScaleFactor' to meta_odict
      io.Input_reader:
      - fixed some bad type hints
      - SRF_reader(): moved path generator functionality to path_generator
      - added open_specific_file_within_archive() (moved)
      - added get_dem_by_extent(): new function for reading SRTM DEM data and warping to a given pixel grid
      io.Output_writer:
      - revised imports
      - fixed some bad type hints
      - added 'ScaleFactor' to enviHdr_keyOrder
      misc.__init__:
      - added __all__
      misc.database_tools:
      - fixed some bad type hints
      - get_overlapping_scenes_from_postgreSQLdb(): bugfix for wrong indexing
      misc.definition_dicts:
      - new module, consisting of earlier functions from helper_functions
      misc.exception_handler:
      - new module, consisting of earlier functions from helper_functions
      misc.helper_functions:
      - moved trace_unhandled_exceptions(), log_uncaught_exceptions() to misc.exception_handler
      - moved failed_GMS_object to gms_object
      - moved get_job_summary to process_controller
      - fixed some bad type hints
      - moved get_GMS_sensorcode(), get_mask_classdefinition(), get_outFillZeroSaturated(), get_mask_colormap() to misc.definition_dicts
      - moved open_specific_file_within_archive() to Input_reader
      misc.path_generator:
      - path_generator:
          - revised get_path_rawdata()
          - revised get_local_archive_path_baseN(): 'image_type' is not used anymore; removed deprecated warning
          - get_path_cloud_class_obj(): added cloud classificator files for Sentinel-2
          - added get_path_srf_file()
          - added get_path_snr_model()
          - added get_path_ac_options()
          - added get_path_ac_table()
      processing.multiproc:
      - revised MAP(): added new keyword 'flatten_ouput'
      processing.pipeline:
      - updated imports
      - revised L1C_map(): input represents one OR multiple L1B_objects belonging to the same scene_ID (atmospheric correction has to be applied to ALL subsystems of a scene at once)
      processing.process_controller:
      - changed some map calls due to new keyword  'flatten_ouput' of processing.multiproc.MAP
      - revised L1C_processing(): added grouping of L1B_objects by scene_ID
      - revised create_job_summary(): bugfix for emtpy input list of get_job_summary()
      - added get_job_summary(): moved from helper_functions
      config.Job:
      - added new attributes 'path_ac_options', 'path_ac_tables', 'path_SNR_models', 'path_dem_proc_srtm_90m', 'path_ECMWF_db'
      pgSGL_db table 'config':
      - added keys 'path_SNR_models', 'path_ac_options', 'path_dem_proc_srtm_90m', 'path_ECMWF_db',  'path_ac_tables'
      Fileserver:
      - added input datasets for atmospheric correction
      - updated __version__
      d06523f9
  10. 06 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      first (not yet working) version of wrapper for atmospheric correction · dd8f8334
      Daniel Scheffler authored
      algorithms.__init__:
      - added some imports
      algorithms.gms_object:
       - removed deprecated code
      algorithms.L1C_P:
      - removed deprecated code
      - added class 'AtmCorr': a wrapper class for atmospheric correction
      misc.database_tools:
      - fixed some broken type hints
      misc.helper_functions
      - fixed broken type hint
      - replaced deprecated function call
      processing.pipeline:
      - fixed some broken type hints
      - updated __version__
      dd8f8334
  11. 05 Jan, 2017 2 commits
    • Daniel Scheffler's avatar
      algorithms.gms_object.GMS_Object: · 428ee490
      Daniel Scheffler authored
      - __getstate__(): added code for deleting array attributes in order to wok around pickle object size limitation
      - added deleters for 'arr', 'mask_nodata', 'mask_clouds', 'masks'
      - fixed some broken type hints
      algorithms.L1B_P.L1B_Object:
      - correct_spatial_shifts(): bugfix for missing masks attribute in Flink mode
      algorithms.L1C_P.L1C_Object:
      - atm_corr(): bugfix for not deleting lonlat_arr
      misc.database_tools:
      - fixed some broken type hints
      misc.helper_functions:
      - fixed broken type hint
      processing.multiproc:
      - MAP(): bugfix for not always returning a list
      processing.process_controller:
      - _is_inMEM(): fixed broken type hint
      - updated __version__
      428ee490
    • Daniel Scheffler's avatar
      major revision of previous commit; module gms_object now fully implemented;... · d48276ad
      Daniel Scheffler authored
      major revision of previous commit; module gms_object now fully implemented; fixed differences in results between Flink and Python execution mode
      algorithms.GEOPROCESSING.GEOPROCESSING:
      - Layerstacking(): added possible bug hint; update_dataset_related_attributes() now always executed
      - get_lonlat_coord_array(): adjusted to handle GeoArray instances
      - calc_VZA_array(): adjusted to handle GeoArray instances
      - calc_SZA_SAA_array(): adjusted to handle GeoArray instances
      - calc_RAA_array(): adjusted to handle GeoArray instances
      algorithms.gms_object:
      - __init__(): added attribute 'MGRS_info'
      - renamed attribute mask_1bit to 'mask_nodata'
      - revised set_pathes()
      - revised __getstate__()
      - revised __deepcopy__()
      - revised coreg_needed()
      - added property 'resamp_needed'
      - added property 'arr'
      - added property 'mask_nodata'
      - added property 'mask_clouds'
      - added property 'masks'
      - added property 'pathGen'
      - revised attributes2dict()
      - revised from_disk()
      - revised calc_mask_nodata()
      - revised build_combined_masks_array()
      - combine_tiles_to_ObjAttr(): added DeprecationWarning
      - get_subset_obj(): adjusted to properly handle GeoArray instances
      - added to_ENVI(), based on Obj2ENVI from OUT_W
      algorithms.L1A_P.L1A_object:
      - import_rasterdata() now also sets shape_fullArr
      - archive_to_rasObj(): bugfix for overwriting Layerstacking result
      - revised get_MetaObj()
      - removed deprecated get_shape_fullArr()
      - calc_TOARadRefTemp(): revised to properly handle GeoArray instances
      - reference_data(): revised to properly handle GeoArray instances
      - calc_cloud_mask(): revised to properly handle GeoArray instances
      - calc_corner_positions(): added assertion; now robust to different WKT string formats
      - revised add_rasterInfo_to_MetaObj()
      - update_spec_vals_according_to_dtype(): now does not set nodata value anymore (property)
      algorithms.L1B_P:
      - Scene_finder:
          - ensured Python 2.7 compatibility
      - L1B_object:
          - correct_spatial_shifts(): major revision fixing some bugs
          - removed deprecated join_deshift_results()
          - removed deprecated apply_deshift_results()
      algorithms.L1C_P.L1C_Object:
      - get_lonlat_coord_array(): commented deprecated code out
      - calc_acquisition_illumination_geometry(): commented deprecated code out
      algorithms.L2A_P.L2A_Object:
      - get_DESHIFTER_configs(): some simplifications
      - removed deprecated class DESHIFTER (now fully imported from external library CoReg_Sat)
      algorithms.L2B_P.L2B_Object:
      - revised __init__()
      io.Output_writer:
      - moved Obj2ENVI to gms_object
      misc.helper_functions:
      - get_job_summary(): bugfix for counting inputs given as tiles of the same sceneid multiple times
      misc.path_generator:
      - added __getstate__() that automatically closes loggers
      - added __setstate__()
      processing.pipeline:
      - adjusted output writer calls
      - L1A_map(): removed calls to get_shape_fullArr() and calc_mask_nodata()
      - L1A_map_1(): removed calls to get_shape_fullArr() and calc_mask_nodata()
      - L2A_map(): added overwrite kwarg to calc_mask_nodata()
      - removed deprecated L2A_map_1()
      - removed deprecated L2A_map_2()
      - renamed L2B_map_1 to L2B_map
      - renamed L2C_map_1 to L2C_map
      processing.process_controller.process_controller:
      - added keyword 'db_host'
      - simplified L2A_processing()
      - create_job_summary(): bugfix for always passing L2C objects to get_job_summary() instead of highest requested processing level
      config:
      - set_config(): added keyword 'db_host'
      - Job:
          - added keyword 'db_host'
          - added attribute 'allow_subMultiprocessing'
          - database connection string is now dynamically adjusted to database host
      - updated __version__
      d48276ad
  12. 08 Dec, 2016 1 commit
  13. 07 Dec, 2016 1 commit
    • Daniel Scheffler's avatar
      implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P... · 007965bf
      Daniel Scheffler authored
      implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P to config, and META; deleted L0B_P
      algorithms:
      - added new module 'gms_object', based on parts of L1A_object
          - GMS_object now provides the complete basic functionality of all GMS objects in later processing levels
      - L0A_P has been dissolved/moved to config and METADATA
      - L0B_P removed (deprecated)
      algorithms.L1A_P.L1A_object:
      - revised __init__() (L1A_object now inherits from GMS_object)
      - added import_rasterdata()
      - moved functions providing basic GMS functionality to GMS_object
      - refactored fill_from_disk() to from_disk()
      algorithms.L1B_P - L2C-P:
      - updated __init__() functions due to new object structure
      io.Output_writer:
      - Obj2ENVI: data_DB_updater now receives a dictionary including current state of all properties
      misc.database_tools.GMS_JOB:
      - added __repr__()
      - bugfix for using CFG.job.conn_database instead of self.conn
      - revised from_job_ID(): self.dataframe is now properly filled
      misc.helper_functions:
      - failed_GMS_object now inherits from GMS_object
      - get_GMS_sensorcode() now does not need a logger anymore
      processing.pipeline:
      - refactored L0A_L1A_map to L1A_map and L0B_L1A_map_1 to L1A_map_1
      - revised L1A_map and L1A_map_1 according to new structure of L1A_object
      processing.process_controller:
      - added attribute '_DB_job_record'
      - added property 'DB_job_record'
      - removed get_data_list() (deprecated)
      - stop(): added docstring
      - L1A_processing(): edited docstring and added logger output; changed reduce call (!)
      - L1B_processing(): edited docstring and added logger output
      - L1C_processing(): edited docstring and added logger output; changed reduce call (!)
      - L2A_processing(): edited docstring and added logger output
      - L2B_processing(): edited docstring and added logger output; changed reduce call (!)
      - L2C_processing(): edited docstring and added logger output
      config:
      - Job: added attribute 'DB_job_record'
      - Usecase:
          - added attributes '_job' and 'data_list'
          - moved get_usecase_coord_grid() from top level of module here
          - added get_entity_IDs_within_AOI() from L0A_P
          - added get_data_list_of_current_jobID() from L0A_P
      - updated __version__
      007965bf
  14. 02 Dec, 2016 3 commits
    • Daniel Scheffler's avatar
      moved add_local_availability() to process_controller; changed output file names of image data · c588839b
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      L0A_P:
      - moved add_local_availability() to process_controller
      L1B_P.L1B_object:
      - coregister_spatially(): fixed deprecated keyword name due to API changes within CoReg_Sat
      misc.path_generator:
      - get_path_imagedata(): output file names of image data now end with '_image_data_<procLevel>.bsq'
      - get_outPath_hdr(): changed outNameSuffix
      processing.process_controller.process_controller:
      - get_data_list(): edited docstring
      - added add_local_availability from L0A_P
      - updated __version__
      c588839b
    • Daniel Scheffler's avatar
      Bugfix · 3cc34718
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      processing.pipeline:
      - L2C_map_1(): fix for using exec__L2BP instead of exec__L2CP
      3cc34718
    • Daniel Scheffler's avatar
      fixed a memory issue during L2C processing; revised process_controller · 096bddfd
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      algorithms.L1A_P.L1A_object:
      - get_subset_obj(): minor revisions; removed deprecated code
      - to_MGRS_tiles() now returns a generator object instead of a list => Output writer can be used directly after generating each tile
      misc.path_generator:
      - fixed some PyCharm warnings
      processing.process_controller:
      - _is_already_present(): added docstring
      - added get_DB_objects() -> simplifies L1A-L2C processor functions
      - run_all_processors(): KeyBoardInterrupt now caused process controller to shutdown gracefully
      - added stop()
      - simplified L1A-L2C processor functions
      config:
      - set_config(): added 'reset' kwarg
      - updated __version__
      096bddfd
  15. 01 Dec, 2016 6 commits
    • Daniel Scheffler's avatar
      cleaned up · db515a3b
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      - moved testing/job_logs to logs/job_logs
      - removed deprecated testing.process_controller_for_testing and testing.process_controller_for_testing__andre
      config:
      - adjusted Job.path_job_logs
      - updated __version__
      db515a3b
    • Daniel Scheffler's avatar
      updated .gitignore · acbb02f6
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      acbb02f6
    • Daniel Scheffler's avatar
      added new attribute 'tiling_block_size_XY' to config.Job · 29dda4ba
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      processing.pipeline:
      - tiling block size of several mappers now use CFG.Job.tiling_block_size_XY as default block size
      config.Job:
      - added attribute 'tiling_block_size_XY'
      - updated __version__
      29dda4ba
    • Daniel Scheffler's avatar
      merged the rest of old process_controller_for_testing and removed deprecated modules · 7c2c3a7b
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      processing.process_controller:
      - moved the rest of process_controller_for_testing here
      - cleaned up deprecated code snippets
      run_gms:
      - added possibility to pass job ID via console call
      - removed deprecated modules 'config_old', 'process_controller_for_testing' and 'unified_config'
      - updated __version__
      7c2c3a7b
    • Daniel Scheffler's avatar
      two little bugfixes · c24961ca
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      misc.helper_functions:
      - log_uncaught_exceptions.wrapped_GMS_mapper(): Bugfix for not passing kwargs to decorator output
      config:
      - GMS_configuration.usecase.setter: Bugfix for writing to GMS_job instead of GMS_usecase
      c24961ca
    • Daniel Scheffler's avatar
      Flink compatibility update / major revision of process controller · 5138279d
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      summary:
      - revised config -> database must be queried once per machine
      - all modules of the package are now directly importable
      - process controller is now a Python class containing a couple of methods for running the job
      all modules:
      - revised import statements
      - replaced functions 'CFG.get_job()' and 'CFG.get_usecase()' by property 'CFG.job' and 'CFG.usecase'
      misc.helper_functions:
      - moved MAP() to new module processing.multiproc
      misc.SpatialIndexMediator.SpatialIndexMediatorServer:
      - bugfix for returning 'port' instead of 'process_id'
      added new package 'processing'
      - added __init__
      - added module 'multiproc'
      - added new module 'process_controller' containing new class 'process_controller' with several methods for running the GMS job
      __init__:
      - revised
      config:
      - added set_config() replacing unified_config.set_config()
      - added class GMS_configuration containing properties for 'job' and 'usecase' that replace unified_config.get_job() and unified_config.get_usecase()
      - Job:
          - added attributes 'end_time' and 'computation_time'
          - revised class structure
      - Usecase:
          - revised class structure
      - removed deprecated code
      copied old version of config to config_old (still used by deprecated process_controller_for_testing)
      added run_gms.py for running GeoMultiSens process controller from console
      - updated __version__
      5138279d
  16. 28 Nov, 2016 1 commit