1. 15 Sep, 2017 1 commit
  2. 30 Aug, 2017 1 commit
  3. 26 Jun, 2017 2 commits
  4. 09 Jun, 2017 1 commit
  5. 23 May, 2017 1 commit
  6. 11 Apr, 2017 1 commit
  7. 05 Apr, 2017 1 commit
  8. 29 Mar, 2017 1 commit
    • Daniel Scheffler's avatar
      job status is now reset to 'pending'. · c0f057b4
      Daniel Scheffler authored
       misc.database_tools:
       - get_postgreSQL_value(): now replaces all " ' " with "" due to compatibility issues to postgreSQL syntax
       - update_records_in_postgreSQLdb(): now uses execute_pgSQL_query for better error reporting
       - GMS_JOB.reset_job_progress(): now resets status to 'pending'
       config.Job:
       - status is now set to 'pending'
      - updated __version__
      Former-commit-id: c5525fec
      Former-commit-id: 2cbb0083
      c0f057b4
  9. 25 Jan, 2017 1 commit
  10. 21 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      first version fully operable for Sentinel-2 (including atmospheric correction and cloud masks) · 3282d365
      Daniel Scheffler authored
      algorithms.gms_object.GMS_object:
      - GMS_identifier: is only a getter now, not a singleton
      - added property 'MetaObj' / 'meta_odict' -> self-synchronizing
      - added LayerBandsAssignment.setter
      - revised arr.setter
      - revised mask_nodata.setter
      - mask_clouds.getter: bugfix
      - revised mask_clouds.setter
      - added property 'mask_clouds_confidence'
      - revised masks.setter
      - revised dem.setter
      - ac_options: number of CPUs are now passed (temporarily disabled)
      - revised ac_errors.setter
      - added LBA2bandnames
      - attributes2dict: bugfix for not addin meta_odict
      - revised from_tiles()
      - added from_sensor_subsystems(): a function for merging multiple subsystems of the same sensor (needed for Sentinel-2 and ASTER)
      - revised build_combined_masks_array()
      - get_subset_obj(): multiple bugfixes
      - to_GMS_file(): updated in the context of new property architecture
      - delete_previous_proc_level_results: added functionality to delete subsystem products after subsystems have been merged
      algorithms.gms_object.failed_GMS_object:
      - removed logger creation
      algorithms.L1A_P.L1A_object:
      - import_metadata(): updated METADATA call
      - calc_TOARadRefTemp: added warning when using Sentinel-2 Gains
      - calc_corner_positions(): added UTC timezone to datetime object
      algorithms.L1B_P.L1B_object:
      - _get_reference_image_params_pgSQL(): replaced temp_logger by self.logger
      - revised correct_spatial_shifts()
      algorithms.L1C_P:
      - L1C_object:
          - removed deprecated atm_corr()
      - AtmCorr:
          - __init__: added warning
          - revised data property
          - _meta_get_aux_data(): changed lonlat_arr to float16
          - run_atmospheric_correction(): added 'dump' keyword
          - revised _join_results_to_inObjs()
      algorithms.L2B_P.L2B_object:
      - spectral_homogenization(): bugfix
      algorithms.METADATA:
      - METADATA:
          - __init__(): is now initialized by GMS_identifier; does not directly run metadata reader
          - added read_meta()
          - Read_Sentinel2A_xmls(): updated setting of Gains
          - refactored to_meta_odict() to to_odict() and revised the function
          - added from_odict()
          - revised filter_layerdependent_metadata()
          - added 'map_odictKeys_objAttrnames'
          - get_LayerBandsAssignment():
              - added functionality to return full LBA for Sentinel-2 and ASTER
              - now properly handles bands removed after L1C and L2B
      io.Output_writer:
      - mask_to_ENVI_Classification(): bugfix
      misc.definition_dicts:
      - get_GMS_sensorcode(): added codes for Sentinel-2 full and ASTER full
      - get_outFillZeroSaturated(): added bool dtype
      - is_dataset_provided_as_fullScene(): added ASTER full and Sentinel-2 full
      misc.exception_handler:
      - log_uncaught_exceptions(): bugfix for 'disable_exception_handler'
      misc.helper_functions:
      - cut_GMS_obj_into_blocks(): bugfix
      misc.logging.GMS_logger:
      - added scene ID to formatter
      misc.mgrs_tile:
      - replaced deprecated reference
      misc.path_generator:
      - get_baseN(): bugfix
      - added get_path_ac_input_dump()
      processing.multiproc:
      - MAP: added functionality to disable multiprocessing
      processing.pipeline:
      - revised L2A_map(): new L2A calls
      processing.process_controller:
      - add_local_availability(): changed get_LayerBandsAssignment call
      - get_DB_objects(): bugfix for copied memory link during GMS object init
      - L1C_processing() raises NotImplementedError in tiled mode
      - L2A_processing(): added grouping of subsystems
      - updated __version__
      3282d365
  11. 16 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      third version of wrapper for atmospheric correction (first working version) · 6fa974e0
      Daniel Scheffler authored
      algorithms.GEOPROCESSING:
      - GEOPROCESSING: conversion_type_optical value renamed from 'Ref' to 'TOA_Ref' and 'BOA_Ref'
      - added keyword 'meshwidth' to the following functions (allows much faster processing):
          - zoom_2Darray_to_shapeFullArr()
          - adjust_acquisArrProv_to_shapeFullArr()
          - get_lonlat_coord_array():revised calculation of meshgrid
          - calc_VZA_array()
          - calc_AcqTime_array()
          - calc_SZA_SAA_array()
      - calc_RAA_array(): now receives a VAA_array instead of VAA_mean
      algorithms.gms_object:
      - added attributes 'fullSceneCornerPos' and 'fullSceneCornerPos'
      - logger: added assertion
      - added property 'log'
      - added dem deleter
      - revised property 'ac_options'
      - added property 'ac_errors'
      - added property 'subset'
      - refactored attribute 'acquisition_date' to 'acq_datetime' containing a full datetime timestamp in UTC time zone
      - to_GMS_file(): updated timestamp format
      algorithms.L1A_P.L1A_object:
      - get_MetaObj(): now also updates 'acq_datetime'
      - refactored get_MetaObj() to import_metadata()
      - included set_arr_desc_from_MetaObj() in import_metadata()
      - calc_TOARadRefTemp(): updated in the context of "conversion_type_optical" value change to 'TOA_Ref'
      - calc_cloud_mask(): temporarily excluded Sentinel-2 here; added code draft of S2A cloud mask calculation
      - calc_corner_positions(): major revision -> now calculates trueDataCornerPos/-LonLat AND fullSceneCornerPos/-LonLat
      - calc_center_AcqTime() now also updates 'acq_datetime'
      algorithms.L1B_P.ref_Scene:
      - moved _get_reference_image_params_pgSQL() and _sceneIDList_to_filt_overlap_scenes() to L1B_object
      algorithms.L1C_P.L1C_object:   -> major revision
      - added properties 'lonlat_arr', 'VZA_arr', 'VAA_arr', 'SZA_arr', 'SAA_arr', 'RAA_arr' based on get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
      - removed deprecated functions get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
      - revised delete_ac_input_arrays()
      algorithms.L1C_P.AtmCorr:
      - revised property 'logger'
      - metadata: added some tests
      - revised _meta_get_viewing_zenith(), _meta_get_viewing_azimuth(), _meta_get_relative_viewing_azimuth()
      - added _meta_get_aux_data()
      - added _get_dem()
      - added dummy version of _get_srf()
      - run_atmospheric_correction(): added docstring; some minor revisions
      - _join_results_to_inObjs(): now working
      algorithms.METADATA:
      - added property AcqDateTime: returns a full datetime object with UTC timezone
      - revised setters for AcqDate, AcqTime and AcqDateTime -> timezone now properly handled
      - refactored 'Meta2ODict' to 'to_meta_odict'
      - calc_center_acquisition_time(): now also sets AcqDateTime
      - get_LayerBandsAssignment(): processing level is now properly handled (in the context of missing bands after atmospheric correction)
      misc.database_tools:
      - renamed keyword 'trueDataCornerLonLat' to 'tgt_corners_lonlat' in the following functions:
          - get_pgSQL_geospatial_query_cond()
          - get_overlapping_scenes_from_postgreSQLdb()
          - get_overlapping_MGRS_tiles()
          - get_overlapping_MGRS_tiles2()
      misc.definitions_dicts:
      - added is_dataset_provided_as_fullScene()
      misc.exception_handler:
      - log_uncaught_exceptions:  exception handling is now optional and can be turned off via config
      misc.helper_functions:
      - modified some docstrings
      misc.logging.GMS_logger:
      - added property 'captured_stream' (not yet working)
      - added draft of StringIO handler (not yet working)
      misc.logging.path-generator:
      - get_path_srf_file: bugfix
      processing.pipeline:
      - L1A_map(): updated calls
      - L1A_map_1(): updated calls
      - revised L1C_map()
      processing.process_controller:
      - add_local_availability(): added comments and revised structure
      config:
      - Job:
          - added attribute 'disable_exception_handler'
          - added attribute 'scale_factor_BOARef'
          - added attribute 'scale_factor_errors_ac'
          - added assertion
      pg_SQLdb:
      - table config:
          - added fields 'scale_factor_BOARef' and 'scale_factor_errors_ac'
          - changed value of 'conversion_type_optical' to 'BOA_Ref'
      - updated __version__
      6fa974e0
  12. 11 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      second (not completely working) version of wrapper for atmospheric correction · d06523f9
      Daniel Scheffler authored
      algorithms.GEOPROCESSING:
      - revised imports
      algorithms.gms_object:
      - gms_object:
          - revised imports
          - added property 'dem': gms_object can now directly provide a corresponding SRTM DEM
          - arr: bandnames of property 'arr' are now in the form [B01, .., B8A,] and correspond to LayerBandsAssignment
          - added property 'ac_options': getter or options dictionary needed for atmospheric correction
          - from_disk(): added time zone to 'acquisition_date' datetime object
      - added class failed_GMS_object (based on earlier version from helper functions)
      algorithms.L1A_P.L1A_object:
      - revised imports
      - calc_TOARadRefTemp(): bugfix for wrong nodata value in out returned array in case of Sentinel-2
      - update_spec_vals_according_to_dtype: bugfix for not updating L1A_object.arr.nodata
      algorithms.L1B_P:
      - revised imports
      algorithms.L1C_P:
      - L1C_object:
          - get_lonlat_coord_array(): changed handling of return values
          - calc_acquisition_illumination_geometry(): changed handling of return values
      - AtmCorr:
          -  added attribute 'ac_input' containing input args/kwargs of atmospheric correction
          - data: now uses L1C_obj.arr.bandnames for lopping over bands
          - added property 'nodata'
          - added property 'tile_name'
          - added property 'band_spatial_sampling'
          - added property 'nodata'
          - added property 'nodata'
          - revised property 'metadata'
          - added _meta_get_spatial_samplings()
          - added _meta_get_solar_irradiance()
          - added _meta_get_viewing_zenith()
          - added _meta_get_viewing_azimuth()
          - added _meta_get_relative_viewing_azimuth()
          - revised run_atmospheric_correction()
          - revised join_results_to_inObjs()
      algorithms.METADATA
      - revised imports
      - added 'ScaleFactor' to meta_odict
      io.Input_reader:
      - fixed some bad type hints
      - SRF_reader(): moved path generator functionality to path_generator
      - added open_specific_file_within_archive() (moved)
      - added get_dem_by_extent(): new function for reading SRTM DEM data and warping to a given pixel grid
      io.Output_writer:
      - revised imports
      - fixed some bad type hints
      - added 'ScaleFactor' to enviHdr_keyOrder
      misc.__init__:
      - added __all__
      misc.database_tools:
      - fixed some bad type hints
      - get_overlapping_scenes_from_postgreSQLdb(): bugfix for wrong indexing
      misc.definition_dicts:
      - new module, consisting of earlier functions from helper_functions
      misc.exception_handler:
      - new module, consisting of earlier functions from helper_functions
      misc.helper_functions:
      - moved trace_unhandled_exceptions(), log_uncaught_exceptions() to misc.exception_handler
      - moved failed_GMS_object to gms_object
      - moved get_job_summary to process_controller
      - fixed some bad type hints
      - moved get_GMS_sensorcode(), get_mask_classdefinition(), get_outFillZeroSaturated(), get_mask_colormap() to misc.definition_dicts
      - moved open_specific_file_within_archive() to Input_reader
      misc.path_generator:
      - path_generator:
          - revised get_path_rawdata()
          - revised get_local_archive_path_baseN(): 'image_type' is not used anymore; removed deprecated warning
          - get_path_cloud_class_obj(): added cloud classificator files for Sentinel-2
          - added get_path_srf_file()
          - added get_path_snr_model()
          - added get_path_ac_options()
          - added get_path_ac_table()
      processing.multiproc:
      - revised MAP(): added new keyword 'flatten_ouput'
      processing.pipeline:
      - updated imports
      - revised L1C_map(): input represents one OR multiple L1B_objects belonging to the same scene_ID (atmospheric correction has to be applied to ALL subsystems of a scene at once)
      processing.process_controller:
      - changed some map calls due to new keyword  'flatten_ouput' of processing.multiproc.MAP
      - revised L1C_processing(): added grouping of L1B_objects by scene_ID
      - revised create_job_summary(): bugfix for emtpy input list of get_job_summary()
      - added get_job_summary(): moved from helper_functions
      config.Job:
      - added new attributes 'path_ac_options', 'path_ac_tables', 'path_SNR_models', 'path_dem_proc_srtm_90m', 'path_ECMWF_db'
      pgSGL_db table 'config':
      - added keys 'path_SNR_models', 'path_ac_options', 'path_dem_proc_srtm_90m', 'path_ECMWF_db',  'path_ac_tables'
      Fileserver:
      - added input datasets for atmospheric correction
      - updated __version__
      d06523f9
  13. 05 Jan, 2017 2 commits
    • Daniel Scheffler's avatar
      algorithms.gms_object.GMS_Object: · 428ee490
      Daniel Scheffler authored
      - __getstate__(): added code for deleting array attributes in order to wok around pickle object size limitation
      - added deleters for 'arr', 'mask_nodata', 'mask_clouds', 'masks'
      - fixed some broken type hints
      algorithms.L1B_P.L1B_Object:
      - correct_spatial_shifts(): bugfix for missing masks attribute in Flink mode
      algorithms.L1C_P.L1C_Object:
      - atm_corr(): bugfix for not deleting lonlat_arr
      misc.database_tools:
      - fixed some broken type hints
      misc.helper_functions:
      - fixed broken type hint
      processing.multiproc:
      - MAP(): bugfix for not always returning a list
      processing.process_controller:
      - _is_inMEM(): fixed broken type hint
      - updated __version__
      428ee490
    • Daniel Scheffler's avatar
      major revision of previous commit; module gms_object now fully implemented;... · d48276ad
      Daniel Scheffler authored
      major revision of previous commit; module gms_object now fully implemented; fixed differences in results between Flink and Python execution mode
      algorithms.GEOPROCESSING.GEOPROCESSING:
      - Layerstacking(): added possible bug hint; update_dataset_related_attributes() now always executed
      - get_lonlat_coord_array(): adjusted to handle GeoArray instances
      - calc_VZA_array(): adjusted to handle GeoArray instances
      - calc_SZA_SAA_array(): adjusted to handle GeoArray instances
      - calc_RAA_array(): adjusted to handle GeoArray instances
      algorithms.gms_object:
      - __init__(): added attribute 'MGRS_info'
      - renamed attribute mask_1bit to 'mask_nodata'
      - revised set_pathes()
      - revised __getstate__()
      - revised __deepcopy__()
      - revised coreg_needed()
      - added property 'resamp_needed'
      - added property 'arr'
      - added property 'mask_nodata'
      - added property 'mask_clouds'
      - added property 'masks'
      - added property 'pathGen'
      - revised attributes2dict()
      - revised from_disk()
      - revised calc_mask_nodata()
      - revised build_combined_masks_array()
      - combine_tiles_to_ObjAttr(): added DeprecationWarning
      - get_subset_obj(): adjusted to properly handle GeoArray instances
      - added to_ENVI(), based on Obj2ENVI from OUT_W
      algorithms.L1A_P.L1A_object:
      - import_rasterdata() now also sets shape_fullArr
      - archive_to_rasObj(): bugfix for overwriting Layerstacking result
      - revised get_MetaObj()
      - removed deprecated get_shape_fullArr()
      - calc_TOARadRefTemp(): revised to properly handle GeoArray instances
      - reference_data(): revised to properly handle GeoArray instances
      - calc_cloud_mask(): revised to properly handle GeoArray instances
      - calc_corner_positions(): added assertion; now robust to different WKT string formats
      - revised add_rasterInfo_to_MetaObj()
      - update_spec_vals_according_to_dtype(): now does not set nodata value anymore (property)
      algorithms.L1B_P:
      - Scene_finder:
          - ensured Python 2.7 compatibility
      - L1B_object:
          - correct_spatial_shifts(): major revision fixing some bugs
          - removed deprecated join_deshift_results()
          - removed deprecated apply_deshift_results()
      algorithms.L1C_P.L1C_Object:
      - get_lonlat_coord_array(): commented deprecated code out
      - calc_acquisition_illumination_geometry(): commented deprecated code out
      algorithms.L2A_P.L2A_Object:
      - get_DESHIFTER_configs(): some simplifications
      - removed deprecated class DESHIFTER (now fully imported from external library CoReg_Sat)
      algorithms.L2B_P.L2B_Object:
      - revised __init__()
      io.Output_writer:
      - moved Obj2ENVI to gms_object
      misc.helper_functions:
      - get_job_summary(): bugfix for counting inputs given as tiles of the same sceneid multiple times
      misc.path_generator:
      - added __getstate__() that automatically closes loggers
      - added __setstate__()
      processing.pipeline:
      - adjusted output writer calls
      - L1A_map(): removed calls to get_shape_fullArr() and calc_mask_nodata()
      - L1A_map_1(): removed calls to get_shape_fullArr() and calc_mask_nodata()
      - L2A_map(): added overwrite kwarg to calc_mask_nodata()
      - removed deprecated L2A_map_1()
      - removed deprecated L2A_map_2()
      - renamed L2B_map_1 to L2B_map
      - renamed L2C_map_1 to L2C_map
      processing.process_controller.process_controller:
      - added keyword 'db_host'
      - simplified L2A_processing()
      - create_job_summary(): bugfix for always passing L2C objects to get_job_summary() instead of highest requested processing level
      config:
      - set_config(): added keyword 'db_host'
      - Job:
          - added keyword 'db_host'
          - added attribute 'allow_subMultiprocessing'
          - database connection string is now dynamically adjusted to database host
      - updated __version__
      d48276ad
  14. 08 Dec, 2016 1 commit
  15. 07 Dec, 2016 1 commit
    • Daniel Scheffler's avatar
      implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P... · 007965bf
      Daniel Scheffler authored
      implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P to config, and META; deleted L0B_P
      algorithms:
      - added new module 'gms_object', based on parts of L1A_object
          - GMS_object now provides the complete basic functionality of all GMS objects in later processing levels
      - L0A_P has been dissolved/moved to config and METADATA
      - L0B_P removed (deprecated)
      algorithms.L1A_P.L1A_object:
      - revised __init__() (L1A_object now inherits from GMS_object)
      - added import_rasterdata()
      - moved functions providing basic GMS functionality to GMS_object
      - refactored fill_from_disk() to from_disk()
      algorithms.L1B_P - L2C-P:
      - updated __init__() functions due to new object structure
      io.Output_writer:
      - Obj2ENVI: data_DB_updater now receives a dictionary including current state of all properties
      misc.database_tools.GMS_JOB:
      - added __repr__()
      - bugfix for using CFG.job.conn_database instead of self.conn
      - revised from_job_ID(): self.dataframe is now properly filled
      misc.helper_functions:
      - failed_GMS_object now inherits from GMS_object
      - get_GMS_sensorcode() now does not need a logger anymore
      processing.pipeline:
      - refactored L0A_L1A_map to L1A_map and L0B_L1A_map_1 to L1A_map_1
      - revised L1A_map and L1A_map_1 according to new structure of L1A_object
      processing.process_controller:
      - added attribute '_DB_job_record'
      - added property 'DB_job_record'
      - removed get_data_list() (deprecated)
      - stop(): added docstring
      - L1A_processing(): edited docstring and added logger output; changed reduce call (!)
      - L1B_processing(): edited docstring and added logger output
      - L1C_processing(): edited docstring and added logger output; changed reduce call (!)
      - L2A_processing(): edited docstring and added logger output
      - L2B_processing(): edited docstring and added logger output; changed reduce call (!)
      - L2C_processing(): edited docstring and added logger output
      config:
      - Job: added attribute 'DB_job_record'
      - Usecase:
          - added attributes '_job' and 'data_list'
          - moved get_usecase_coord_grid() from top level of module here
          - added get_entity_IDs_within_AOI() from L0A_P
          - added get_data_list_of_current_jobID() from L0A_P
      - updated __version__
      007965bf
  16. 02 Dec, 2016 1 commit
    • Daniel Scheffler's avatar
      moved add_local_availability() to process_controller; changed output file names of image data · c588839b
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      L0A_P:
      - moved add_local_availability() to process_controller
      L1B_P.L1B_object:
      - coregister_spatially(): fixed deprecated keyword name due to API changes within CoReg_Sat
      misc.path_generator:
      - get_path_imagedata(): output file names of image data now end with '_image_data_<procLevel>.bsq'
      - get_outPath_hdr(): changed outNameSuffix
      processing.process_controller.process_controller:
      - get_data_list(): edited docstring
      - added add_local_availability from L0A_P
      - updated __version__
      c588839b
  17. 01 Dec, 2016 1 commit
    • Daniel Scheffler's avatar
      Flink compatibility update / major revision of process controller · 5138279d
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      summary:
      - revised config -> database must be queried once per machine
      - all modules of the package are now directly importable
      - process controller is now a Python class containing a couple of methods for running the job
      all modules:
      - revised import statements
      - replaced functions 'CFG.get_job()' and 'CFG.get_usecase()' by property 'CFG.job' and 'CFG.usecase'
      misc.helper_functions:
      - moved MAP() to new module processing.multiproc
      misc.SpatialIndexMediator.SpatialIndexMediatorServer:
      - bugfix for returning 'port' instead of 'process_id'
      added new package 'processing'
      - added __init__
      - added module 'multiproc'
      - added new module 'process_controller' containing new class 'process_controller' with several methods for running the GMS job
      __init__:
      - revised
      config:
      - added set_config() replacing unified_config.set_config()
      - added class GMS_configuration containing properties for 'job' and 'usecase' that replace unified_config.get_job() and unified_config.get_usecase()
      - Job:
          - added attributes 'end_time' and 'computation_time'
          - revised class structure
      - Usecase:
          - revised class structure
      - removed deprecated code
      copied old version of config to config_old (still used by deprecated process_controller_for_testing)
      added run_gms.py for running GeoMultiSens process controller from console
      - updated __version__
      5138279d
  18. 28 Nov, 2016 1 commit
  19. 14 Oct, 2016 1 commit
    • Daniel Scheffler's avatar
      Bugfix for unexpected pixel values in cloud masks (bug from last commit);... · b1ead53b
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      Bugfix for unexpected pixel values in cloud masks (bug from last commit); bugfix for wrong colors in some cloud classification files
      L1A_P:
      - L1A_object.logger: bugfix for not creating new logger if there is a logger without handlers
      L1B_P:
      - correct_spatial_shifts(): bugfix for wrong resampling algorithm in case of masks attribute
      OUT_W:
      - mask_to_ENVI_Classification(): bugfix for returning array with wrong pixel values in case of missing no data value
      b1ead53b
  20. 13 Oct, 2016 1 commit
    • Daniel Scheffler's avatar
      revised logging practices of whole GMS package (fixes Permission errors during... · 11e1bc1c
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      revised logging practices of whole GMS package (fixes Permission errors during logging); added direct call of external DeShifter module
      L0B_P:
      - updated calls for logger closing
      L1A_P.L1A_object:
      - converted L1A_object.logger to property 'logger'
      - added close_GMS_loggers()
      - updated __getstate__ and __setstate__
      - log_for_fullArr_or_firstTile(): added 'close logger'-calls
      - MetaObj2ODict(): bugfix for not deleting MetaObj
      - added to_GMS_file() on the basis of OUT_W.ASCII_writer() which is now deprecated
      - delete_tempFiles(): added 'close logger'-calls
      L1B_P:
      - reordered imports
      - updated logger getters and 'close logger'-statements
      - L1B_object:
          - removed its own logger setup
          - correct_spatial_shifts(): added working version
      L2A_P:
      - DESHIFTER: updated logger getters and 'close logger'-statements
      INP_R:
      - pickle_SRF_DB(): updated logger getters and 'close logger'-statements
      OUT_W:
      - ASCII_writer(): now deprecated; added 'unclosed logfile' warnings
      - Obj2ENVI: updated writer calls for GMS file
      HLP_F:
      - removed setup_logger()
      - moved close_logger() to new module 'logging'
      - moved GMS_logger() to new module 'logging'
      - failed_GMS_object: updated logger getter
      - find_nearest() is now imported from external package 'py_tools_ds'
      - removed duplicate version of corner_coord_to_minmax()
      logging:
      - added new module logging, based on previous logging functions from HLP_F
      MGRS_tile:
      - added higher timeout for getting geometry from pgSQL database
      SpatialIndexMediator:
      - SpatialIndexMediatorServer:
          - status: revised return value
          - start(): revised warnings
          - stop(): revised return value
      PC:
      - job.logger is now an instance of GMS_logger
      - added new version of L2A_map()
      - added shutdown_loggers statement
      11e1bc1c
  21. 12 Oct, 2016 2 commits
    • Daniel Scheffler's avatar
      Bugfix for not properly closed logfiles; bugfix for · db54bd81
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      L0B_P:
      - L0B_object: bugfix for not properly closed logfiles
      L1A_P:
      - added assignment of L1A_object.GMS_identifier['logger']
      - added verbose mode to get_MetaObj() and disabled it
      - __getstate__(): bugfix
      L1B_P:
      - Scene_finder.filter_possib_ref_scenes(): bugfix for not filtering by projection
      ENV:
      - added dummy function check_paths()
      HLP_F:
      - close_logger(): bugfix for not checking if given logger has handlers
      CFG:
      - deleted two deprecated comments
      PC:
      - added some job IDs
      - added 'job comment info' during startup
      db54bd81
    • Daniel Scheffler's avatar
      Bugfix for not properly closed logfiles; added environment checker for Spatial... · 77a105fc
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      Bugfix for not properly closed logfiles; added environment checker for Spatial Index Mediator Server
       L1A_P:
       - L1A_Obj.__getstate__(): bugfix for not properly closed logfiles
      L1B_P:
      - get_reference_image_params_pgSQL(): bugfix for not properly closed logfile
      L2A_P:
      - correct_shifts(): bugfix for not properly closed logfile
      ENV:
      - check_dependencies(): added functionality to check current status of Spatial Index Mediator Server and to start the server if needed
      - added _log_or_print()
      HLP_F:
      - added close_logger()
      - added draft of class GMS_logger (not usable so far)
      - subcall_with_output(): added keywords no_stdout and no_stderr
      SpatialIndexMediator:
      - added class SpatialIndexMediatorServer: a class for interacting with java server (status, start, stop, restart, is_running, port, ...)
      CFG:
      - added job.path_spatIdxSrv
      PC:
      - added calls of environment checkers
      pgSQL_DB:
      - added 'path_spatial_index_mediator_server' to config table
      77a105fc
  22. 07 Oct, 2016 5 commits
    • Daniel Scheffler's avatar
      added a dummy environment checker module · 8e827622
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      8e827622
    • Daniel Scheffler's avatar
      808ab70a
    • Daniel Scheffler's avatar
      removed deprecated code in L1B_P; bugfix for wrong pixel size in reprojected... · c44bbeb4
      Daniel Scheffler authored
      removed deprecated code in L1B_P; bugfix for wrong pixel size in reprojected MGRS tiles (fixed in 'py_tools_ds'-package)
      L1B_P:
      - refactored Scene_getter to Scene_finder
      - moved COREG_OLD.get_reference_image_params_pgSQL() and COREG_OLD.sceneIDList_to_filt_overlap_scenes() to Scene_finder
      - removed deprecated class COREG_OLD
      c44bbeb4
    • Daniel Scheffler's avatar
      moved a couple of functions from GEOPROCESSING to external package... · a7c6c26f
      Daniel Scheffler authored
      moved a couple of functions from GEOPROCESSING to external package 'py_tools_ds' and added direct imports within respective modules
      GEOP:
      - the following functions moved to 'py_tools_ds':
          - transform_utm_to_wgs84
          - transform_wgs84_to_utm
          - transform_any_prj
          - reproject_shapelyPoly
          - lonlat_to_pixel
          - latLonToPixel
          - pixelToLatLon
          - pixelToMapYX
          - mapXY2imXY
          - imXY2mapXY
          - isProjectedOrGeographic
          - EPSG2Proj4
          - EPSG2WKT
          - WKT2EPSG
          - get_UTMzone
          - geotransform2mapinfo
          - mapinfo2geotransform
          - get_corner_coordinates
          - get_prjLonLat
          - get_proj4info
          - proj4_to_dict
          - prj_equal
          - corner_coord_to_minmax
          - get_footprint_polygon
          - get_overlap_polygon
          - find_line_intersection_point
          - calc_FullDataset_corner_positions
          - is_point_on_grid
          - is_coord_grid_equal
          - snap_bounds_to_pixGrid
      IO:
      - deleted deprecated module envifilehandling
      - renamed envifilehandling_BD to envifilehandling
      - deleted deprecated module sysenvironment
      a7c6c26f
    • Daniel Scheffler's avatar
      - replaced the duplicated fork of COREG within L1B_P by an imports from external packages · be17e860
      Daniel Scheffler authored
      - GeoMultiSens now depends on CoReg_Sat and py_tools_ds!
      GEOP:
      - moved warp_ndarray to external package 'py_tools_ds'
      - updated warp_ndarray calls
      L1A_P:
      - added property coreg_needed
      - added cubic resampling for MGRS tiles that have to be reprojected to the next UTM zone
      - to_MGRS_tiles(): added verbose mode
      L1B_P:
      - COREG and DESHIFTER are now imported from external package CoReg_Sat
      - deleted deprecated draft of COREG_GMS
      - added class Scene_getter(): a class used to find a proper geospatial reference scene for a given target scene
      - added class ref_Scene
      - L1B_object:
          - revised the whole class (it now generates its coreg_info and deshift_results on its own)
          - added property spatRef_available
          - added get_spatial_reference_scene()
          - added get_opt_bands4matching (based on an earlier version from COREG)
          - added coregister_spatially()
          - added correct_spatial_shifts(): not yet working
      L1C_P:
      - L1C_object: updated __init__() args of super class L1B_object
      L2A_P:
      - updated warp_ndarray calls
      L2B_P:
      - spectral homogenization is now only executed if target CWLs are different to source CWLs
      INP_R:
      - fixed an unclosed file within GMSfile2dict
      OUT_W:
      - fixed an unclosed file within Obj2ENVI
      PC:
      - added a new version of L1B_map_1()
      be17e860
  23. 26 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      Revision of algorithm for getting spatial refererence scene during L1B_P · c7e3832e
      Daniel Scheffler authored
      L1B_P:
      - COREG:
          - __init__(): changed condition that determines whether coregistration in executed or not
          - major revision of get_reference_image_params(): reference scene is now chosen out of ALL scenes included in the spatial query result (not only the first one); revised selection criteria; impementation now fully based on geopandas
          - calculate_spatial_shifts(): bugfix for not leaving while loop in case no match has been found
          - added class reference_scene_getter(): so far only a prototype that is not used yet
      PC:
      - changed job starting message
      - disabled L2C compression
      c7e3832e
  24. 19 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      Bugfix for "missing L2B mask files" during continued processing that earlier... · afcb0fa5
      Daniel Scheffler authored
      Bugfix for "missing L2B mask files" during continued processing that earlier stopped in L2B processing; implemented proper MGRS tiling scheme according to project agreements
      GEOP:
      - updated some deprecated 3rd-party-library references
      - added transform_any_prj(): a function fort transforming XY-coordinates from any source to any target projection
      - warp_ndarray():
          - implemented extra option 'outExtent_within' allowing to output arrays with a larger geographical extent than the input image (needed for MGRS tiling scheme)
      - improved some docstrings
      - added prj_equal(): a function to quickly check if two projections are equal
      - added snap_bounds_to_pixGrid(): a function to snap map bounds to a given pixel grid
      - added clip_array_using_mapBounds(): a function for clipping arrays with a map info using a given bounding box
      L0A_P:
      - add_local_availability():
          - replaced a hard coded list of processing levels
          - bugfix for returning an invalid processing level if processing is already done
      L1A_P:
      - fill_from_disk(): bugfix for copying memory addresses  when fill_from_disk() is called in multiprocessing
      - log_for_fullArr_or_firstTile(): bugfix for not handling empty arr_pos
      - added get_subset_obj(), based on older code from helper_functions: a function to generate subsets of GMS objects based on bounding box that can contain image AND map coordinates whereas map coordinates can also have a different projection (needed for proper generation of MGRS tiles)
       - added to_MGRS_tiles(, based on older code from helper_functions: a function for cutting a GMS object into MGRS tiles
       L1B_P:
       - L1B_object.apply_deshift_results():
           - bugfix for not updating geoinformations of 'masks' attribute after having applied spatial shift corrections
      META:
      - added 'data ignore value' to metadata
      IO:
      - refactored module 'gms_io' to 'io'
      ARR:
      - added submodule 'GeoArray' to io:
          - added class 'GeoArray': a class for simplifying array access, regardless to their actual memory location (in memory or on disk)
          - added _clip_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box in the same projection like the array itself
          - added  get_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box that can have any projection
      OUT_W:
      - added 'data ignore value' to enviHdr_keyOrder
      - added 'data ignore value' to mask_to_ENVI_Classification()
      - added set_output_nodataVal(): a function for adding a data ignore value to an already written file
      - Obj2ENVI: bugfix for not handling empty arr_pos
      DB_T:
      - delete_processing_results(), GMS_JOB.__delete_procdata(), GMS_JOB.delete_procdata_of_entire_job(), GMS_JOB.delete_procdata_of_failed_sceneIDs(), added functionality to delete a specific processing level from disk
      HLP_F:
      - revised find_nearest(): it now supports an automatic rounding algorithm that rounds a value to the nearest neighbour
      - replaced a deprecated 3rd-party function reference
      - get_arrSubsetBounds_from_shapelyPolyLonLat(): added flag 'ensure_valid_coords'
      MRGS_tile:
      - added submodule "mgrs_tile" for easily retrieving informations about MGRS tiles (bounding box, projection conversions, buffering, etc.)
      SpatialIndexMediator:
      - replaced wrong host address
      PC:
      - updated calls for MGRS tiling within L2C_map_1()
      afcb0fa5
  25. 06 Sep, 2016 1 commit
  26. 01 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      added handling of unexpected exceptions within all GMS mappers; added... · 38dad1ca
      Daniel Scheffler authored
      added handling of unexpected exceptions within all GMS mappers;  added creation of job success summary
      L1A_P:
      - removed deprecated attributes path_Outfile_L1A and path_Outfile_L1A_masks
      - calc_cloud_mask(): bugfix - wrong arguments passed to log_for_fullArr_or_firstTile
      L1B_P:
      - revised get_reference_image_params()
           -> spatial query works now but it still does not handle the case that no reference scene could be found or reference scene has a different projection
      L1C_P:
      - removed deprecated import
      INP_R:
      - get_list_GMSfiles(): bugfix for ignoring the given target processing level if job.call_type=='webapp'
      OUT_W:
      - Obj2ENVI(): bugfix for not assigning the correct output path to attribute 'mask_1bit' in case job.exec_mode=='Python'
       DB_T:
       - moved SpatialIndexMediator to a separate module within misc
       - simplified datetime reference
       - GMS_JOB: added comment keyword in order to allow short descriptions and comments for the job
      HLP_F:
      - added trace_unhandled_exceptions(): a dummy decorator function for handling unexpected exceptions
      - added class 'failed_GMS_object': a class to represent failed GMS objects (allows to get information from them - even after processing failed)
      - added log_uncaught_exceptions(): a decorator function for logging unhandled exceptions within GMS mapper functions
      - added get_job_summary(): a function that returns a detailed and a less detailed pandas dataframe summarizing job processing success
      - cut_GMS_obj_into_MGRS_tiles(): added logging
      - get_mask_classdefinition(): added mask_1bit class definition
      SpatialIndexMediator (SIM):
      - added new module SpatialIndexMediator for performing fast spatial queries targeting the 'scenes' table of postgreSQL database
      PC:
      - added decorators to all GMS mapper functions that enable handling of unexpected exceptions
      - added new list "failed_objects" in order to collect information about all GMS objects where processing failed
      - added job database entry updates regarding failed scenes and finish time
      - added creation and logging of job summary
      VC:
      - added cloud classifier objects for Landsat-5/7/8 to version control
      38dad1ca
  27. 30 Aug, 2016 1 commit
    • Daniel Scheffler's avatar
      Landsat-7 processing now available in exec_mode='Python' and... · f1f3607b
      Daniel Scheffler authored
      Landsat-7 processing now available in exec_mode='Python' and parallelization_level='scenes'; bugfixes in MGRS tile creation; database query for spatial reference not fully operable yet
      GEOP:
      - calc_FullDataset_corner_positions(): bugfix for returning wrong data corner positions for MGRS tiles
      - transform_utm_to_wgs84(): bugfix for returning "wrong UTM zone" error in case of negative zone number (south)
      - geotransform2mapinfo(): added NotImplementedError in case of rotated datasets
      - mapinfo2geotransform(): bugfix for improper handling of UL coordinate if UL coordinate is not 1/1
      - moved find_line_intersection_point() from calc_FullDataset_corner_positions() to top level
      L1A_P:
      - log_for_fullArr_or_firstTile(): added check of  MGRS_tile_obj.logAtThisTile in case of MGRS tile -> needed for proper logging for MGRS tiles if xmin=0/ymin=0 not exists
      - delete_tempFiles() bugfix for running delete_previous_proc_level_results() for the wrong processing level
      - added method __getstate__(): for defining custom pickling settings for logger
      - added method __setstate__(): for defining custom unpickling settings for logger
      - replaced all temp_logger-calls by self.logger calls
      - log_for_fullArr_or_firstTile(self, log_msg, subset=None, logger=None): deleted deprecated logger argument
      L1B_P:
      - calculate_spatial_shifts(): updated a warning if calculated shifts are out of given threshold
      - calculate_spatial_shifts(): bugfix for returning an updated map info although shift calculation did not pass validity test
      - L1B_obj: replaced all temp_logger-calls by self.logger calls
      - added a new version of get_reference_image_params() -> not fully operable yet
      - renamed old version of get_reference_image_params() to get_reference_image_params_pgSQL()
      - added editor fold to class GMS_COREG and commented out imports
      - commented out deprecated imports and extensions of sys.path
      L1C_P:
      - L1C_obj: replaced all temp_logger-calls by self.logger calls
      L2A_P:
      - correct_shifts(): bugfix for not processing scenes if no reference scene was available
      - correcr_shifts(): out_gsd is now properly set
      - L2A_obj: replaced all temp_logger-calls by self.logger calls
       META:
       - added docstring to metaDict_to_metaODict()
       OUT_W:
      - Obj2ENVI():
           - better documentation
           - replaced all temp_logger-calls by InObj.logger calls
      - ASCII_writer(): proper exclusion of loggers from JSON serialization
       DB_T:
       - updated GMS_JOB.jobs_table_columns due to two deleted columns in 'jobs' table
       - updated data_DB_updater() due to changes in attribute namings og GMS object
       - added class SpatialIndexMediator(): for fast geospatial queries targeting 'scenes' table of postgreSQL database
       HLP_F:
       - get_subset_GMS_obj(): major revision
          - some renamed object attributes -> new attributes: data_corners_LonLat, data_corners_utm, fullScene_corner_lonlat, bounds_LonLat, bounds_utm, corner_utm, corner_lonlat
          - bugfix for wrong map info of the returned subsetted GMS objects
          - much better documentation
      - cut_GMS_obj_into_MGRS_tiles():
          - better documentation
          - moved calculation of tile attributes to get_subset_GMS_obj()
          - added MGRS_tile_obj,logAtThisTile attribute for proper logging for MGRS tiles
      - corner_coord_to_minmax():
          - added documentation
          - updated first timestamp of Landsat-7 SLC-off data
      PG:
      - added job 26185268  # 25x L7 SLC off
      f1f3607b
  28. 24 Aug, 2016 1 commit
    • Daniel Scheffler's avatar
      First version that is able to produce MGRS cubes in mass production (so far... · 56edb6e3
      Daniel Scheffler authored
      First version that is able to produce MGRS cubes in mass production (so far only fully operable for Landsat-8)
      GEOP:
      - merged calc_FullDataset_corner_positionsOLD() and calc_FullDataset_corner_positions() to calc_FullDataset_corner_positions()
      	- the function now features two algorithms: 'numpy' (for Landsat-7 ETM+ SLC-off) and 'shapely' (for all remaining scenes)
      - deleted calc_FullDataset_corner_positionsOLD()
      - added is_point_on_grid(): Checks if a given point is exactly on the given coordinate grid. Needed to check if scene has to be coregistered or not.
      - added is_coord_grid_equal(): Needed to check if scene has to be coregistered or not.
      L0A_P:
      - updated name of jobs table in postgreSQL database
      L1A_P:
      - added attribute dataset_ID -> Needed to check if scene has to be coregistered or not.
      - added attributes scenes_proc_ID and mgrs_tiles_proc_ID -> needed for database entries
      - archive_to_rasObj() now asserts that the read archive contains any files
      - bugfix for Landsat-7 SLC-off scenes: archive_to_rasObj() now expects more files in the archive due to SLC off masks
      - bugfix calc_mask_nodata(): data had been read from wrong source if job.exec_mode=='Python'
      - revised calc_cloud_mask() -> the resulting array is now contains correct pixel values and is properly included in GMS_obj.masks
      - bugfix for Landsat-7 SLC-off scenes: calc_corner_positions() now calls a different algorithm that can deal with SLC-off gaps
      - bugfix calc_corner_positions(): incorrect getter for projection
      - bugfix build_combined_masks_array(): it now out always outputs unsigned integer 8bit
      - added delete_previous_proc_level_results(): needed to save disk space by deleting results of previous processing levels that are not needed anymore
      - delete_tempFiles() now also deletes results of previous processing levels on demand (depending on job.exec_L1AP[2])
      L1B_P:
      - added class COREG_GMS (not yet working) -> will replace class COREG soon and will synchronize its changes with external library CoReg_Sat
      - COREG.__init__():
      	- added attributes dataset_ID and coreg_needed -> Needed to check if scene has to be coregistered or not.
      	-> the class now only runs coregistration if its really needed. otherwise its skipped
      - COREG.get_updated_map_info(): now no output stream anymore in quiet mode
      - L1B_object:
      	- slighty changed logging behavior
      	- apply_deshift_results(): - bugfix for not properly updated map info and projection
      	- apply_deshift_results(): - bugfix for not properly updated dimensions of the mask attributes
      L1C_P:
      - delete_tempfiles() now also deletes results of previous processing levels  on demand
      L2A_P:
      - bugfix in get_DESHIFTER_configs(): not properly closed gdal dataset
      - added attributes shift_dataset_ID, x_shift_px, y_shift_px and deshift_needed in order to check if deshifting is really needed or not
      - correct_shifts():
      	- added proper logging
      	- bugfix for missing updated_projection in case coregistration has been skipped
      	- bugfix for not properly closed gdal dataset
      	- bugfix for wrong resampling method for cloud mask (cubic results in wrong output pixel values) -> now nearest neighbour
      	- added 'is clipped to extent' to deshift_results
      L1C_P:
      - bugfix spectral_homogenization(): now it also accepts string type array attributes if exec_mode=='Python'
      - revised interpolate_cube_linear()
      META:
      - CS_DATUM, CS_UTM_ZONE, CS_TYPE are now properly written to gms-files
      - added metaDict_to_metaODict() needed by output writer
      INP_R:
      - read_mask_subset(): added missing 'MGRS_tile' subset type
      OUT_W:
      - added silent_envi_write_image(): a monkey patch in order to silence spectral.io.envi._write_image
      - added write_ordered_envi_header(): in order to write properly sorted ENVI header files
      - revised Tiles_Writer(): previous version caused output errors for mask attributes of GMS objects
      - revised mask_to_ENVI_Classification(): bugfixes for wrong conversions of numpy arrays to ENVI classdification ready array and metadata
      - added check_header_not_empty()
      - major revision of Obj2ENVI(): now it properly writes MRGS tiles if exec_mode=='Python'
      - added database update for MGRS tiles (new records created in new database table 'mgrs_tiles_proc')
      DB_T:
      - splitted get_postgreSQL_matchingExp() to get_postgreSQL_value() and get_postgreSQL_matchingExp()
      	-> much more python variable types can now be converted to postgreSQL types
      - added create_record_in_postgreSQLdb(): needed to create new records for MGRS tiles
      - added class GMS_JOB: a job manager for GMS jobs
      	-> current features:
      		- create a job in the postgreSQL database, based on a list of archive files
      		- get the details about an existing job
      - added add_missing_filenames_in_pgSQLdb(): useful for finding and fixing database errors
      - import_shapefile_into_postgreSQL_database() updated name of postgreSQL database
      - major revision of data_DB_updater(): now it properly updates both tables 'scenes_proc' and 'mgrs_scenes_proc'
      HLP_F:
      - added attribute proc_chain
      - get_subset_GMS_obj(): bugfix for clipping the wrong tile dimension
      - get_subset_GMS_obj(): added updated tile corners and data corners to MGRS tiles
      - cut_GMS_obj_into_MGRS_tiles(): added calculation of tile bounds as well as data corner positions
      - _numba_array_merger(): added a type assertion
      - get_valid_arrSubsetBounds(): bugfix for returning wrong tile dimensions (1 row/column too big)
      - sceneID_to_trueDataCornerLonLat(): added valid polygon assertion
      - added class Landsat_entityID_decrypter: needed by class JOB_GMS for finding correct sensor parameters for Landsat data
      PG:
      - added get_path_rawdata()
      - get_path_procdata(): changed returned path structure for MGRS tiles saved to disk
      - get_baseN(): changed returned basename for GMS objects on disk
      CFG:
      - updated postgreSQL database connection credentials
      - added a "delete output of previous porcessing levels"-flag to exec__L**P
      - usecase: added datasetid_spatial_ref, virtual_sensor_name, target_gsd, EPSG, spatial_ref_gridx, spatial_ref_gridy
      PC:
      - added multiple job IDs containing pre-defined database jobs
      - updated L1B_map_1(): shift calculation is now skipped if no reference scene is needed or available
      - added L2A_map(): needed to execute processing in mass production, controlled by Python process controller
      - revised tempfile deletion within L1C_map_1(), L2A_map_1(), L2B_map_1()
      - updated L2A mapper calls -> exec_mode=='Python' is now supported
      - added hardcoded thread count restriction to L2B mapper call due to memory errors in Python execution mode with parallelization level 'scenes'
      pgSQLdb:
      - renamed database from usgs_cache to geomultisens
      - added table 'mrgs_tiles_proc'
      - renamed table 'scenes_jobs' to 'jobs'
      - updated jobs table with new columns
      - fixed missing filenames in scenes table
      - ...
      56edb6e3
  29. 12 Aug, 2016 1 commit
    • Daniel Scheffler's avatar
      GEOP: · a7920668
      Daniel Scheffler authored
      - warp_ndarray(): fixed a bug in the validation of the output dimensions calculated by rasterio that always expected a quadratic input array for performing a correction of out_cols and out_rows
      - added reproject_shapelyPoly()
      - added mapXY2imXY()
      - added imXY2mapXY()
      - added proj4_to_dict()
      L1A_P:
      - fixed a bug in calc_cloud_mask()
      - fixed a bug in calc_corner_positions()
      L1B_P:
      - fixed a bug in get_reference_image_params()
      - fixed a bug in apply_deshift_results() when applying results to a 2D array
      L2A_P:
      - changed titlebar
      L2B_P:
      - changed titlebar
      L2C_P:
      - changed titlebar
      - added a dummy version of L2C_object
      L2D_P:  Level 2D processor has been deleted due to changes in the L2 processing architecture
      META:
      - added META.CS_EPSG attribute
      OUT_W:
      - Obj2ENVI(): added functionality to write MGRS tiles as compressed ENVI files (compression mode not yet working)
      - dirty hack in mask_to_ENVI_Classification() -> still not working
      - added get_dtypeStr()
      - added new version of write_shp()
      DB_T:
      - moved some code from get_scene_and_dataset_infos_from_postgreSQLdb() to new function execute_pgSQL_query()
      - added get_pgSQL_geospatial_query_cond()
      - added get_overlapping_MGRS_tiles()
      - added get_overlapping_MGRS_tiles2()
      - added pdDataFrame_to_sql_k()
      - added import_shapefile_into_postgreSQL_database()
      - added postgreSQL_table_to_csv() (not yet working)
      HLP_F:
      - added docstring to setup_logger()
      - added gzipfile()
      - added ENVIfile_to_ENVIcompressed()
      - added get_subset_GMS_obj()
      - added cut_GMS_obj_into_MGRS_tiles()
      - added an extra assertion to postgreSQL_poly_to_cornerLonLat()
      - added some type hints
      - added postgreSQL_geometry_to_shapelyPolygon()
      - added shapelyPolygon_to_postgreSQL_geometry()
      - added get_imageCoords_from_shapelyPoly()
      - added get_valid_arrSubsetBounds()
      - added get_arrSubsetBounds_from_shapelyPolyLonLat() -> needed to get image coordinates of MGRS tiles
      - added get_UL_LR_from_shapefile_features()
      PG:
      - added functionality to get pathes for MGRS-tiled products
      CFG:
      - added additional output folder for MGRS-tiled products
      PC:
      - removed L2D_P import
      - added cloud mask and corner coordinates calculation to L2A_map_2
      - added L2C_map_1()
      - adjusted L2B calls
      - added first version of L2C calls
      a7920668
  30. 13 Jul, 2016 1 commit
    • Daniel Scheffler's avatar
      First prototype of algorithms for geometrical and spectral resolution working... · 9142f468
      Daniel Scheffler authored
      First prototype of algorithms for geometrical and spectral resolution working in map-reduce context (L2A, L2B).
      GEOP:
          - renamed GEOPROCESSING_BD.py to GEOPROCESSING.py
          - moved get_prjLonLat(), get_proj4info(), corner_coord_to_minmax() to GEOP
          - added docstrings to DN2Rad(), DN2TOARef(), TOARad2Kelvin_fastforward(), DN2DegreesCelsius_fastforward()
      L1A_P:
          - L1A_object.fill_arr_from_disk(): further silencing of console outputs
      L1B_P:
          - moved get_DESHIFTER_configs() and class DESHIFTER() to L2A_P
          - adjusted initial values for COREG attributes related to reference image (not None anymore in order to make L2A_P work if shift calculation failed)
          - increased database statement timeouts for queries within get_reference_image_params() to 25sek
          - L1B_object():
              - added attribute "deshift_results"
              - removed deprecated code
              - added join_deshift_results()
              - revised apply_deshift_results()
      L2A_P:
          -  added get_DESHIFTER_configs() and class DESHIFTER() from L1B_P
          - fixed two bugs in DESHIFTER.correct_shifts() where DESHIFTER.band2process was not respected and whole image cube was read instead of only one band
          - added class L2A_object()
      L2B_P:
          - added class L2B_object()
          - L2B_object():
              - added interpolate_cube_linear()
              - added spectral_homogenization()
      META:
          - renamed METADATA_BD.py to METADATA.py
      INP_R:
          - added quiet mode to read_ENVIfile()
      OUT_W:
          - added enviHdr_keyOrder using list from reorder_envi_header()
          - fixed a bug in reorder_ENVI_header() that caused repetitions of header keys
          - adjusted print_dict within mask_to_ENVI_Classification() in order to also support L2A and L2B
      HLP_F:
          - added parent objects for L2A and L2B in parentObjDict
          - added type hints to cut_GMS_obj_into_blocks() and merge_GMS_tiles_to_GMS_obj()
      GITIGNORE:
          - updated .gitignore file
      CFG:
          - added virtual_sensor_id, datasetid_spectral_ref, target_CWL, target_FWHM to usecase class by querying the database
      PC:
          - added type hints to mapper functions
          - revised L2A_map_2()
          - added L2B_map_1()
          - revised/added L2A algorithm calls (only Flink mode is supported so far)
          - added L2B algorithm calls (only Flink mode is supported so far)
      pgDB:
          - added Sentinel-2A virtual sensors to virtual_sensors table (different spatial resolutions)
          - added wavelengths positions and band widths to virtual_sensors table
      9142f468
  31. 10 Jul, 2016 1 commit
    • Daniel Scheffler's avatar
      Python execution mode now fully operational || implemented functions for... · 8abcf2ed
      Daniel Scheffler authored
      Python execution mode now fully operational || implemented functions for adding externally downloaded scenes to GMS database
      GEOP:
          - fixed imports
          - fixed a deprecated keyword in warp_ndarray()
      L1A_P:
          - fixed imports
          - added 'Python' exec_mode to L1A_object.fill_from_disk()
          - fixed return values calc_cloud_mask()
          - added some docstrings and type hints
          - fixed a bug where MetaObj.SpecVals are not updated after applying no data mask to object attribute
          - revised build_L1A_masks(9 -> new version called build_combined_masks_array()
              -> not longer called from process controller but only from output writer
          - improved delete_tempfiles()
      L1B_P:
          - revised COREG.__init__() to a much clearer version -> all attributes are now always existing
          - revised L1B_object.__init__()
      L1C_P:
          - fixed a bug where no data mask has been read from wrong processing level
      L1D_P:
           - L1D_P was deprecated -> not existing anymore
      INP_R:
          - added som docstrings and type hints
          - read_ENVIfile(), read_ENVI_image_data_as_array() and read_mask_subset() now also support .bsq files as input paths
      OUT_W:  major revision!
          - deleted param_dic
          - Tiles_writer(): fixed a bug that prevented writing of 2D arrays
          - revision of reorder_ENVI_header() -> more stable, more effective
          - revision of mask_to_ENVI_Classification()
          - revision of Obj2ENVI():
              - now also supports a tempfile mode that allows to directly write arrays of TILES from within the mapper function
              - added image_type_dict, metaAttr_dict, out_dtype_dict
              - now directly creates InObj.masks on demand
              - revised writing of array attributes
      DB_T:  multiple new functions for adding externally downloaded satellite scenes to GMS database
          - added many docstrings and type hints
          - added get_postgreSQL_matchingExp()
          - get_info_from_postgreSQLdb()
              - added a custom timeout keyword to get_info_from_postgreSQLdb() in order to allow non-static query timings
              - added the possibility to pass a list of values within conditions dictionary
          - added set_info_in_postgreSQLdb()
          - added get_dict_satellite_name_id()
          - added get_dict_sensor_name_id()
          - added get_entityIDs_from_filename()
          - added get_filename_by_entityID()
          - added get_notDownloadedsceneIDs()
          - added add_externally_downloaded_data_to_GMSDB()
          - added archive_exists_on_fileserver()
      HLP_F:
          - added silentremove()
          - fixed a missing colormap in get_mask_colormap()
      PG:
          - revised path_generator.__init__()
          - added get_path_cloudmaskdata()
          - added get_outPath_hdr()
      CFG:
          - removed static database statement timeout -> now directly adjustable by called functions
      PC:
          - L1A_map_2: array attributes are now directly written to disk in exec_mode 'Python' @ parallelization_level 'tiles'
            in order to save a lot of time for pickling big objects
          - L1A_obj.build_L1A_masks() is no longer called from PC
          - fixed a bug that prevented appending L1B_newTiles to L1B database objects
      8abcf2ed
  32. 28 Jun, 2016 1 commit
    • Daniel Scheffler's avatar
      Major update with further developments of map-reduce processing chain (working... · ee3fd0eb
      Daniel Scheffler authored
      Major update with further developments of map-reduce processing chain (working until L1C) and full Sentinel-2 support for all processing levels
      GEOP:
          - caught a bug in rasterio library that caused GEOP.warp_ndarray() to return warped images with wrong dimension (1 row more than expected),
            which caused L1B_P.COREG() to fail
          - added zoom_2Darray_to_shapeFullArr() and adjust_acquisArrProv_to_shapeFullArr()
            for adjusting dimensions of acquisition geometry arrays provided by data providers (e.g. Sentinel-2)
          - revision of calc_FullDataset_corner_positions() - now it does not fail anymore if only 2 image corners are visible in an image
            (granule with nodata on the one side)
      L0A_P:
          - fixed a bug in get_AllWrittenProcL_dueLog() returning the same processing level multiple times
            if same processing level has been written out more than once
      L1A_P:
          - revised L1A_object.fill_from_disk() in order to make it useable by all GMS objects in all processing levels
          - calc_mean_VAA(): corrected assignment of VAA_mean in case of Sentinel-2 granules
            where VAA cannot be calculated using true corner coordinates
          - moved cut_L1A_obj_into_blocks(), merge_L1A_tiles_to_L1A_obj(), _numba_array_merger() to HLP_F
            in order to make them usable everywhere in the processing chain, not only in L1A processing
      L1B_P:
          - added logging of database queries
          - added L1B_object.set_coreg_info() and moved assignment of L1B_object.coreg_info there
      L1C_P:
          - logging pattern changed so that log messages are only sent in case the full array or the first tile is processed
          - added special case of calculating viewing zenith angle for Sentinel-2 by zooming the respective provider array to the right dimension
          - moved dummy_atm_corr() to L1C_object.atm_corr() and updated log messages
          - added L1C_object.delete_tempFiles()
      META:
          - removed deprecated Sentinel-2 test mode in METADATA.Read_Sentinel2A_xmls()
          - added correct assignment of METADATA.gResolution for all supported datasets
          - added 'ViewingAngle_arrProv' and 'IncidenceAngle_arrProv' to Meta2ODict()
      INP_R:
          - added docstring to read_ENVI_image_data_as_array()
          - revised read_ENVI_image_data_as_array(), read_mask_subset(), GMSfile2dict()
          - revised ()
      OUT_W:
          - added L1C data to param_dic
          - added the possibility to include small numpy arrays as lists in *.gms files
            (needed for Sentinel-2's 'ViewingAngle_arrProv' and 'IncidenceAngle_arrProv')
          - added docstrings to Tiles_Writer(), Obj2ENVI()
          - added add_ENVIclassificationMeta_to_meta() for modifying metadata dictionaries
            so that respective images are recognized as classification images by ENVI
          - Obj2ENVI(): added the possibility to write GMS objects that only represent a tile of a full object
            which allows writing of processed data directly out of the Flink mappers making slow back-pickling of mapper results not needed anymore
      DB_T:
          - added docstrings to get_info_from_SQLdb() and get_info_from_postgreSQLdb()
          - improved error handling in get_info_from_postgreSQLdb()
      HLP_F:
          - added parentObjDict and initArgsDict
            that are needed to make GMS object cutter and tile joiner generally usable in the entire processing chain
          - added exclude_val argument to find_nearest()
          - added docstring to get_image_tileborders()
          - added cut_GMS_obj_into_blocks(), merge_GMS_tiles_to_GMS_obj() and _numba_array_merger() based on similar methods from L1A_P
      CONFIG:
          - increased postgreSQL database statement timeout to 15 sec
          - added a validation of output writer configurations depending on job.exec_mode
      PC:
          - revised L1B_map_1()
          - added L1C_map_1()
          - revised L2A_map_1() - not fully working yet
          - revised run_processController_in_multiprocessing()
              -> now it fully supports L1A-L1C processing and includes a first version L2A processing (not fully working yet)
      ee3fd0eb
  33. 18 Jun, 2016 1 commit
    • Daniel Scheffler's avatar
      Sentinel-2 compatibility update - L1A/L1B processing now fully compatible with Sentinel-2 data · 5ff92e51
      Daniel Scheffler authored
      L0A_P: added subset splitting for Sentinel-2 and Aster
      L0B_P: added filename attribute that now give a clear hint about filename on disk
      L1A_P:
          - archive_to_rasObj(): added support for Sentinel-2
          - added set_arr_desc_from_MetaObj()
          - added log_for_fullArr_or_firstTile()
          - added rescale_array()
          - DN2TOARadRefTemp() renamed to calc_TOARadRefTemp()
          - calc_TOARadRefTemp(): revised and facilitated
              - supports now DNs and Refs as Input
              - supports radiometric rescaling of arrays
          - calc_cloud_mask(): some bugfixes and code improvements
          - apply_nodata_mask_to_ObjAttr(): can now call apply_nodata_mask_to_saved_ENVIfile()
          - build_L1A_masks(): bugfix
      L1B_P:
          - fixed a bug that prevented adding of coreg_info to L1B_object if no reference scene had been available within COREG
          - added inheritance
      L1C_P:
          - added inheritance
      METADATA:
          - added Read_Sentinel2A_xmls(): metadata parser for Sentinel-2 data
          - some bugfixes and code improvements
          - added a ScaleFactor attribute
          - revised SPOT GMS_sensorcodes
          - revised self.PhysUnit
          - removed deprecated self.Satellite_GMS()
          - added Sentinel-2 metadata to isPAN(), isTHERMAL(), get_orbit_params(), get_special_values(), get_LayerBandsAssignment()
      OUT_W: bugfix
      INP_R: bugfix
      HLP_F:
          - revised SPOT GMS_sensorcodes
          - improved group_objects_by_attributes()
          - added find_in_xml_root(), find_in_xml(), get_values_from_xml(), stack_detectors()
      PG:
          - added Sentinel-2 compatibility
          - changed path derivation using entityid to filename
      CFG: added usecase.scale_factor_TOARef
      - added Sentinel-2 SRFs to database
      5ff92e51