1. 28 Sep, 2017 1 commit
  2. 27 Sep, 2017 2 commits
  3. 26 Sep, 2017 1 commit
  4. 25 Sep, 2017 1 commit
  5. 22 Sep, 2017 1 commit
  6. 21 Sep, 2017 1 commit
  7. 15 Sep, 2017 1 commit
  8. 30 Aug, 2017 1 commit
  9. 18 Jul, 2017 1 commit
  10. 06 Jul, 2017 1 commit
  11. 30 Jun, 2017 1 commit
  12. 01 Jun, 2017 1 commit
  13. 31 May, 2017 1 commit
    • Daniel Scheffler's avatar
      Bugfix · 3f982dec
      Daniel Scheffler authored
      - modified run_from_entityids()
      - delete_processing_results(): Fix for warning ".fuse_hidden.. remaining"
      updated __version__ and __versionalias__
      Former-commit-id: 5acce159
      Former-commit-id: 7e18bab0
  14. 24 May, 2017 1 commit
  15. 23 May, 2017 1 commit
  16. 11 Apr, 2017 1 commit
    • Daniel Scheffler's avatar
      added a first version of a console argument parser · 7d61e2de
      Daniel Scheffler authored
      - converted pyhdf import to soft import
      - GMS_JOB: fix
      - added from_entityIDlist(): dummy version
      run_at_geoms.sh / run_at_geoms_deployed.sh:
      - updated call
      - added run_from_jobid()
      - added run_from_sceneids()
      - added run_from_entityids()
      - added run_from_constraints()
      - added argument parser
      updated __version__
      Former-commit-id: e083eae3
      Former-commit-id: c8909644
  17. 30 Mar, 2017 1 commit
  18. 29 Mar, 2017 1 commit
    • Daniel Scheffler's avatar
      job status is now reset to 'pending'. · c0f057b4
      Daniel Scheffler authored
       - get_postgreSQL_value(): now replaces all " ' " with "" due to compatibility issues to postgreSQL syntax
       - update_records_in_postgreSQLdb(): now uses execute_pgSQL_query for better error reporting
       - GMS_JOB.reset_job_progress(): now resets status to 'pending'
       - status is now set to 'pending'
      - updated __version__
      Former-commit-id: c5525fec
      Former-commit-id: 2cbb0083
  19. 28 Mar, 2017 1 commit
    • Daniel Scheffler's avatar
      revised job statistics updater · 2b66ea9d
      Daniel Scheffler authored
       - added increment_decrement_arrayCol_in_postgreSQLdb()
       - GMS_JOB:
           - _populate_jobAttrs_from_sceneInfoGDF() / create(): updated statistics
           - added reset_job_progress()
           - implemented update_db_entry()
       - updated db_jobs_statistics_def
       - log_uncaught_exceptions.wrapped_GMS_mapper(): revised statistics updating scheme (now also properly working for scenes with subsystems
       - run_all_processors(): added reset of job progress in database
      - updated __version__
      Former-commit-id: 811477d3
      Former-commit-id: 0b237688
  20. 27 Mar, 2017 1 commit
    • Daniel Scheffler's avatar
      statistics column of jobs table in pgSQL database is now properly updated during processing · f65899ca
      Daniel Scheffler authored
      - added attributes 'status' and 'statistics'
      - db_entry: added docstring
      - _populate_jobAttrs_from_sceneInfoGDF(): now also sets self.statistics
      - create(): now also sets self.statistics
      - added dictionary 'db_jobs_statistics_def': added automatic de- and incrementing of scene count included in statistics column of jobs table in database
      - trace_unhandled_exceptions.wrapped_GMS_mapper():
      - updated __version__
      Former-commit-id: d5593e78
      Former-commit-id: cd1e7a26
  21. 23 Feb, 2017 1 commit
    • Daniel Scheffler's avatar
      Bugfix for issues writing MGRS tiles in case of scenes that have an overlap to... · ba54ca0f
      Daniel Scheffler authored
      Bugfix for issues writing MGRS tiles in case of scenes that have an overlap to neighbour UTM zone; added feature for job creation from scene IDs
      - get_subset_obj(): implemented keyword out_prj
      - to_MGRS_tiles(): fix for not respecting output projection
      - from_dictlist(): moved parts to _get_validated_sceneInfoGDFs() and _populate_jobAttrs_from_sceneInfoGDF()
      - added from_sceneIDlist()
      - added _get_validated_sceneInfoGDFs()
      - added _populate_jobAttrs_from_sceneInfoGDF()
      - added get_pathes_all_procdata(): not fully implemented yet
      - MAP: bugfix for ignoring CFG.job.CPUs in case CPUs is given
      - updated __version__
      Former-commit-id: 9b5925c8
      Former-commit-id: 12c9f4fe
  22. 25 Jan, 2017 2 commits
  23. 24 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      algorithms.gms_object.GMS_object: · cef2a152
      Daniel Scheffler authored
      - from_tiles(): bugfix for not updating array-dependent attributes
      - from_sensor_subsystems(): bugfix for not recreating loggers after subsystem merging
      - removed deprecated MetaObj2ODict()
      - get_subset_obj(): now operable from imBounds; fixed inconsistent attribute names
      - added to_tiles()
      - added log message
      - get_LayerbandsAssignment: bugfix
      - cut_GMS_obj_into_blocks(): now calls GMS_object.to_tiles()
      - revised cornerLonLat_to_postgreSQL_poly()
      - EPSG: bugfix
      - L1A_map_1() and L2A_map(): changed call for object conversion into tiles
      - removed /database/cloud_classifier from version controlling -> now external resource
      - updated __version__
      Former-commit-id: c9f38cb9
      Former-commit-id: 9b805282
  24. 16 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      third version of wrapper for atmospheric correction (first working version) · 6fa974e0
      Daniel Scheffler authored
      - GEOPROCESSING: conversion_type_optical value renamed from 'Ref' to 'TOA_Ref' and 'BOA_Ref'
      - added keyword 'meshwidth' to the following functions (allows much faster processing):
          - zoom_2Darray_to_shapeFullArr()
          - adjust_acquisArrProv_to_shapeFullArr()
          - get_lonlat_coord_array():revised calculation of meshgrid
          - calc_VZA_array()
          - calc_AcqTime_array()
          - calc_SZA_SAA_array()
      - calc_RAA_array(): now receives a VAA_array instead of VAA_mean
      - added attributes 'fullSceneCornerPos' and 'fullSceneCornerPos'
      - logger: added assertion
      - added property 'log'
      - added dem deleter
      - revised property 'ac_options'
      - added property 'ac_errors'
      - added property 'subset'
      - refactored attribute 'acquisition_date' to 'acq_datetime' containing a full datetime timestamp in UTC time zone
      - to_GMS_file(): updated timestamp format
      - get_MetaObj(): now also updates 'acq_datetime'
      - refactored get_MetaObj() to import_metadata()
      - included set_arr_desc_from_MetaObj() in import_metadata()
      - calc_TOARadRefTemp(): updated in the context of "conversion_type_optical" value change to 'TOA_Ref'
      - calc_cloud_mask(): temporarily excluded Sentinel-2 here; added code draft of S2A cloud mask calculation
      - calc_corner_positions(): major revision -> now calculates trueDataCornerPos/-LonLat AND fullSceneCornerPos/-LonLat
      - calc_center_AcqTime() now also updates 'acq_datetime'
      - moved _get_reference_image_params_pgSQL() and _sceneIDList_to_filt_overlap_scenes() to L1B_object
      algorithms.L1C_P.L1C_object:   -> major revision
      - added properties 'lonlat_arr', 'VZA_arr', 'VAA_arr', 'SZA_arr', 'SAA_arr', 'RAA_arr' based on get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
      - removed deprecated functions get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
      - revised delete_ac_input_arrays()
      - revised property 'logger'
      - metadata: added some tests
      - revised _meta_get_viewing_zenith(), _meta_get_viewing_azimuth(), _meta_get_relative_viewing_azimuth()
      - added _meta_get_aux_data()
      - added _get_dem()
      - added dummy version of _get_srf()
      - run_atmospheric_correction(): added docstring; some minor revisions
      - _join_results_to_inObjs(): now working
      - added property AcqDateTime: returns a full datetime object with UTC timezone
      - revised setters for AcqDate, AcqTime and AcqDateTime -> timezone now properly handled
      - refactored 'Meta2ODict' to 'to_meta_odict'
      - calc_center_acquisition_time(): now also sets AcqDateTime
      - get_LayerBandsAssignment(): processing level is now properly handled (in the context of missing bands after atmospheric correction)
      - renamed keyword 'trueDataCornerLonLat' to 'tgt_corners_lonlat' in the following functions:
          - get_pgSQL_geospatial_query_cond()
          - get_overlapping_scenes_from_postgreSQLdb()
          - get_overlapping_MGRS_tiles()
          - get_overlapping_MGRS_tiles2()
      - added is_dataset_provided_as_fullScene()
      - log_uncaught_exceptions:  exception handling is now optional and can be turned off via config
      - modified some docstrings
      - added property 'captured_stream' (not yet working)
      - added draft of StringIO handler (not yet working)
      - get_path_srf_file: bugfix
      - L1A_map(): updated calls
      - L1A_map_1(): updated calls
      - revised L1C_map()
      - add_local_availability(): added comments and revised structure
      - Job:
          - added attribute 'disable_exception_handler'
          - added attribute 'scale_factor_BOARef'
          - added attribute 'scale_factor_errors_ac'
          - added assertion
      - table config:
          - added fields 'scale_factor_BOARef' and 'scale_factor_errors_ac'
          - changed value of 'conversion_type_optical' to 'BOA_Ref'
      - updated __version__
  25. 11 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      second (not completely working) version of wrapper for atmospheric correction · d06523f9
      Daniel Scheffler authored
      - revised imports
      - gms_object:
          - revised imports
          - added property 'dem': gms_object can now directly provide a corresponding SRTM DEM
          - arr: bandnames of property 'arr' are now in the form [B01, .., B8A,] and correspond to LayerBandsAssignment
          - added property 'ac_options': getter or options dictionary needed for atmospheric correction
          - from_disk(): added time zone to 'acquisition_date' datetime object
      - added class failed_GMS_object (based on earlier version from helper functions)
      - revised imports
      - calc_TOARadRefTemp(): bugfix for wrong nodata value in out returned array in case of Sentinel-2
      - update_spec_vals_according_to_dtype: bugfix for not updating L1A_object.arr.nodata
      - revised imports
      - L1C_object:
          - get_lonlat_coord_array(): changed handling of return values
          - calc_acquisition_illumination_geometry(): changed handling of return values
      - AtmCorr:
          -  added attribute 'ac_input' containing input args/kwargs of atmospheric correction
          - data: now uses L1C_obj.arr.bandnames for lopping over bands
          - added property 'nodata'
          - added property 'tile_name'
          - added property 'band_spatial_sampling'
          - added property 'nodata'
          - added property 'nodata'
          - revised property 'metadata'
          - added _meta_get_spatial_samplings()
          - added _meta_get_solar_irradiance()
          - added _meta_get_viewing_zenith()
          - added _meta_get_viewing_azimuth()
          - added _meta_get_relative_viewing_azimuth()
          - revised run_atmospheric_correction()
          - revised join_results_to_inObjs()
      - revised imports
      - added 'ScaleFactor' to meta_odict
      - fixed some bad type hints
      - SRF_reader(): moved path generator functionality to path_generator
      - added open_specific_file_within_archive() (moved)
      - added get_dem_by_extent(): new function for reading SRTM DEM data and warping to a given pixel grid
      - revised imports
      - fixed some bad type hints
      - added 'ScaleFactor' to enviHdr_keyOrder
      - added __all__
      - fixed some bad type hints
      - get_overlapping_scenes_from_postgreSQLdb(): bugfix for wrong indexing
      - new module, consisting of earlier functions from helper_functions
      - new module, consisting of earlier functions from helper_functions
      - moved trace_unhandled_exceptions(), log_uncaught_exceptions() to misc.exception_handler
      - moved failed_GMS_object to gms_object
      - moved get_job_summary to process_controller
      - fixed some bad type hints
      - moved get_GMS_sensorcode(), get_mask_classdefinition(), get_outFillZeroSaturated(), get_mask_colormap() to misc.definition_dicts
      - moved open_specific_file_within_archive() to Input_reader
      - path_generator:
          - revised get_path_rawdata()
          - revised get_local_archive_path_baseN(): 'image_type' is not used anymore; removed deprecated warning
          - get_path_cloud_class_obj(): added cloud classificator files for Sentinel-2
          - added get_path_srf_file()
          - added get_path_snr_model()
          - added get_path_ac_options()
          - added get_path_ac_table()
      - revised MAP(): added new keyword 'flatten_ouput'
      - updated imports
      - revised L1C_map(): input represents one OR multiple L1B_objects belonging to the same scene_ID (atmospheric correction has to be applied to ALL subsystems of a scene at once)
      - changed some map calls due to new keyword  'flatten_ouput' of processing.multiproc.MAP
      - revised L1C_processing(): added grouping of L1B_objects by scene_ID
      - revised create_job_summary(): bugfix for emtpy input list of get_job_summary()
      - added get_job_summary(): moved from helper_functions
      - added new attributes 'path_ac_options', 'path_ac_tables', 'path_SNR_models', 'path_dem_proc_srtm_90m', 'path_ECMWF_db'
      pgSGL_db table 'config':
      - added keys 'path_SNR_models', 'path_ac_options', 'path_dem_proc_srtm_90m', 'path_ECMWF_db',  'path_ac_tables'
      - added input datasets for atmospheric correction
      - updated __version__
  26. 06 Jan, 2017 1 commit
    • Daniel Scheffler's avatar
      first (not yet working) version of wrapper for atmospheric correction · dd8f8334
      Daniel Scheffler authored
      - added some imports
       - removed deprecated code
      - removed deprecated code
      - added class 'AtmCorr': a wrapper class for atmospheric correction
      - fixed some broken type hints
      - fixed broken type hint
      - replaced deprecated function call
      - fixed some broken type hints
      - updated __version__
  27. 05 Jan, 2017 2 commits
    • Daniel Scheffler's avatar
      algorithms.gms_object.GMS_Object: · 428ee490
      Daniel Scheffler authored
      - __getstate__(): added code for deleting array attributes in order to wok around pickle object size limitation
      - added deleters for 'arr', 'mask_nodata', 'mask_clouds', 'masks'
      - fixed some broken type hints
      - correct_spatial_shifts(): bugfix for missing masks attribute in Flink mode
      - atm_corr(): bugfix for not deleting lonlat_arr
      - fixed some broken type hints
      - fixed broken type hint
      - MAP(): bugfix for not always returning a list
      - _is_inMEM(): fixed broken type hint
      - updated __version__
    • Daniel Scheffler's avatar
      major revision of previous commit; module gms_object now fully implemented;... · d48276ad
      Daniel Scheffler authored
      major revision of previous commit; module gms_object now fully implemented; fixed differences in results between Flink and Python execution mode
      - Layerstacking(): added possible bug hint; update_dataset_related_attributes() now always executed
      - get_lonlat_coord_array(): adjusted to handle GeoArray instances
      - calc_VZA_array(): adjusted to handle GeoArray instances
      - calc_SZA_SAA_array(): adjusted to handle GeoArray instances
      - calc_RAA_array(): adjusted to handle GeoArray instances
      - __init__(): added attribute 'MGRS_info'
      - renamed attribute mask_1bit to 'mask_nodata'
      - revised set_pathes()
      - revised __getstate__()
      - revised __deepcopy__()
      - revised coreg_needed()
      - added property 'resamp_needed'
      - added property 'arr'
      - added property 'mask_nodata'
      - added property 'mask_clouds'
      - added property 'masks'
      - added property 'pathGen'
      - revised attributes2dict()
      - revised from_disk()
      - revised calc_mask_nodata()
      - revised build_combined_masks_array()
      - combine_tiles_to_ObjAttr(): added DeprecationWarning
      - get_subset_obj(): adjusted to properly handle GeoArray instances
      - added to_ENVI(), based on Obj2ENVI from OUT_W
      - import_rasterdata() now also sets shape_fullArr
      - archive_to_rasObj(): bugfix for overwriting Layerstacking result
      - revised get_MetaObj()
      - removed deprecated get_shape_fullArr()
      - calc_TOARadRefTemp(): revised to properly handle GeoArray instances
      - reference_data(): revised to properly handle GeoArray instances
      - calc_cloud_mask(): revised to properly handle GeoArray instances
      - calc_corner_positions(): added assertion; now robust to different WKT string formats
      - revised add_rasterInfo_to_MetaObj()
      - update_spec_vals_according_to_dtype(): now does not set nodata value anymore (property)
      - Scene_finder:
          - ensured Python 2.7 compatibility
      - L1B_object:
          - correct_spatial_shifts(): major revision fixing some bugs
          - removed deprecated join_deshift_results()
          - removed deprecated apply_deshift_results()
      - get_lonlat_coord_array(): commented deprecated code out
      - calc_acquisition_illumination_geometry(): commented deprecated code out
      - get_DESHIFTER_configs(): some simplifications
      - removed deprecated class DESHIFTER (now fully imported from external library CoReg_Sat)
      - revised __init__()
      - moved Obj2ENVI to gms_object
      - get_job_summary(): bugfix for counting inputs given as tiles of the same sceneid multiple times
      - added __getstate__() that automatically closes loggers
      - added __setstate__()
      - adjusted output writer calls
      - L1A_map(): removed calls to get_shape_fullArr() and calc_mask_nodata()
      - L1A_map_1(): removed calls to get_shape_fullArr() and calc_mask_nodata()
      - L2A_map(): added overwrite kwarg to calc_mask_nodata()
      - removed deprecated L2A_map_1()
      - removed deprecated L2A_map_2()
      - renamed L2B_map_1 to L2B_map
      - renamed L2C_map_1 to L2C_map
      - added keyword 'db_host'
      - simplified L2A_processing()
      - create_job_summary(): bugfix for always passing L2C objects to get_job_summary() instead of highest requested processing level
      - set_config(): added keyword 'db_host'
      - Job:
          - added keyword 'db_host'
          - added attribute 'allow_subMultiprocessing'
          - database connection string is now dynamically adjusted to database host
      - updated __version__
  28. 08 Dec, 2016 1 commit
  29. 07 Dec, 2016 1 commit
    • Daniel Scheffler's avatar
      implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P... · 007965bf
      Daniel Scheffler authored
      implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P to config, and META; deleted L0B_P
      - added new module 'gms_object', based on parts of L1A_object
          - GMS_object now provides the complete basic functionality of all GMS objects in later processing levels
      - L0A_P has been dissolved/moved to config and METADATA
      - L0B_P removed (deprecated)
      - revised __init__() (L1A_object now inherits from GMS_object)
      - added import_rasterdata()
      - moved functions providing basic GMS functionality to GMS_object
      - refactored fill_from_disk() to from_disk()
      algorithms.L1B_P - L2C-P:
      - updated __init__() functions due to new object structure
      - Obj2ENVI: data_DB_updater now receives a dictionary including current state of all properties
      - added __repr__()
      - bugfix for using CFG.job.conn_database instead of self.conn
      - revised from_job_ID(): self.dataframe is now properly filled
      - failed_GMS_object now inherits from GMS_object
      - get_GMS_sensorcode() now does not need a logger anymore
      - refactored L0A_L1A_map to L1A_map and L0B_L1A_map_1 to L1A_map_1
      - revised L1A_map and L1A_map_1 according to new structure of L1A_object
      - added attribute '_DB_job_record'
      - added property 'DB_job_record'
      - removed get_data_list() (deprecated)
      - stop(): added docstring
      - L1A_processing(): edited docstring and added logger output; changed reduce call (!)
      - L1B_processing(): edited docstring and added logger output
      - L1C_processing(): edited docstring and added logger output; changed reduce call (!)
      - L2A_processing(): edited docstring and added logger output
      - L2B_processing(): edited docstring and added logger output; changed reduce call (!)
      - L2C_processing(): edited docstring and added logger output
      - Job: added attribute 'DB_job_record'
      - Usecase:
          - added attributes '_job' and 'data_list'
          - moved get_usecase_coord_grid() from top level of module here
          - added get_entity_IDs_within_AOI() from L0A_P
          - added get_data_list_of_current_jobID() from L0A_P
      - updated __version__
  30. 01 Dec, 2016 1 commit
    • Daniel Scheffler's avatar
      Flink compatibility update / major revision of process controller · 5138279d
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      - revised config -> database must be queried once per machine
      - all modules of the package are now directly importable
      - process controller is now a Python class containing a couple of methods for running the job
      all modules:
      - revised import statements
      - replaced functions 'CFG.get_job()' and 'CFG.get_usecase()' by property 'CFG.job' and 'CFG.usecase'
      - moved MAP() to new module processing.multiproc
      - bugfix for returning 'port' instead of 'process_id'
      added new package 'processing'
      - added __init__
      - added module 'multiproc'
      - added new module 'process_controller' containing new class 'process_controller' with several methods for running the GMS job
      - revised
      - added set_config() replacing unified_config.set_config()
      - added class GMS_configuration containing properties for 'job' and 'usecase' that replace unified_config.get_job() and unified_config.get_usecase()
      - Job:
          - added attributes 'end_time' and 'computation_time'
          - revised class structure
      - Usecase:
          - revised class structure
      - removed deprecated code
      copied old version of config to config_old (still used by deprecated process_controller_for_testing)
      added run_gms.py for running GeoMultiSens process controller from console
      - updated __version__
  31. 28 Nov, 2016 1 commit
  32. 14 Oct, 2016 1 commit
  33. 19 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      Bugfix for "missing L2B mask files" during continued processing that earlier... · afcb0fa5
      Daniel Scheffler authored
      Bugfix for "missing L2B mask files" during continued processing that earlier stopped in L2B processing; implemented proper MGRS tiling scheme according to project agreements
      - updated some deprecated 3rd-party-library references
      - added transform_any_prj(): a function fort transforming XY-coordinates from any source to any target projection
      - warp_ndarray():
          - implemented extra option 'outExtent_within' allowing to output arrays with a larger geographical extent than the input image (needed for MGRS tiling scheme)
      - improved some docstrings
      - added prj_equal(): a function to quickly check if two projections are equal
      - added snap_bounds_to_pixGrid(): a function to snap map bounds to a given pixel grid
      - added clip_array_using_mapBounds(): a function for clipping arrays with a map info using a given bounding box
      - add_local_availability():
          - replaced a hard coded list of processing levels
          - bugfix for returning an invalid processing level if processing is already done
      - fill_from_disk(): bugfix for copying memory addresses  when fill_from_disk() is called in multiprocessing
      - log_for_fullArr_or_firstTile(): bugfix for not handling empty arr_pos
      - added get_subset_obj(), based on older code from helper_functions: a function to generate subsets of GMS objects based on bounding box that can contain image AND map coordinates whereas map coordinates can also have a different projection (needed for proper generation of MGRS tiles)
       - added to_MGRS_tiles(, based on older code from helper_functions: a function for cutting a GMS object into MGRS tiles
       - L1B_object.apply_deshift_results():
           - bugfix for not updating geoinformations of 'masks' attribute after having applied spatial shift corrections
      - added 'data ignore value' to metadata
      - refactored module 'gms_io' to 'io'
      - added submodule 'GeoArray' to io:
          - added class 'GeoArray': a class for simplifying array access, regardless to their actual memory location (in memory or on disk)
          - added _clip_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box in the same projection like the array itself
          - added  get_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box that can have any projection
      - added 'data ignore value' to enviHdr_keyOrder
      - added 'data ignore value' to mask_to_ENVI_Classification()
      - added set_output_nodataVal(): a function for adding a data ignore value to an already written file
      - Obj2ENVI: bugfix for not handling empty arr_pos
      - delete_processing_results(), GMS_JOB.__delete_procdata(), GMS_JOB.delete_procdata_of_entire_job(), GMS_JOB.delete_procdata_of_failed_sceneIDs(), added functionality to delete a specific processing level from disk
      - revised find_nearest(): it now supports an automatic rounding algorithm that rounds a value to the nearest neighbour
      - replaced a deprecated 3rd-party function reference
      - get_arrSubsetBounds_from_shapelyPolyLonLat(): added flag 'ensure_valid_coords'
      - added submodule "mgrs_tile" for easily retrieving informations about MGRS tiles (bounding box, projection conversions, buffering, etc.)
      - replaced wrong host address
      - updated calls for MGRS tiling within L2C_map_1()
  34. 10 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      Bugfix for wrong scene counts in job summary; added functions for deleting processed data · 0d17e8bb
      Daniel Scheffler authored
       - added delete_processing_results(): a function for deleting processing results of a given scene ID
       - GMS_JOB: added __delete_procdata, delete_procdata_of_entire_job and delete_procdata_of_failed_sceneIDs
       - added proper counting for succeeded and failed objects
       - adjusted maximum count of workers for L1A and L2B processing
       - added some jobs needed for bugfixing
       - updated GMS beta usecase ## dev GMS41.ipynb
  35. 06 Sep, 2016 1 commit
  36. 01 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      added handling of unexpected exceptions within all GMS mappers; added... · 38dad1ca
      Daniel Scheffler authored
      added handling of unexpected exceptions within all GMS mappers;  added creation of job success summary
      - removed deprecated attributes path_Outfile_L1A and path_Outfile_L1A_masks
      - calc_cloud_mask(): bugfix - wrong arguments passed to log_for_fullArr_or_firstTile
      - revised get_reference_image_params()
           -> spatial query works now but it still does not handle the case that no reference scene could be found or reference scene has a different projection
      - removed deprecated import
      - get_list_GMSfiles(): bugfix for ignoring the given target processing level if job.call_type=='webapp'
      - Obj2ENVI(): bugfix for not assigning the correct output path to attribute 'mask_1bit' in case job.exec_mode=='Python'
       - moved SpatialIndexMediator to a separate module within misc
       - simplified datetime reference
       - GMS_JOB: added comment keyword in order to allow short descriptions and comments for the job
      - added trace_unhandled_exceptions(): a dummy decorator function for handling unexpected exceptions
      - added class 'failed_GMS_object': a class to represent failed GMS objects (allows to get information from them - even after processing failed)
      - added log_uncaught_exceptions(): a decorator function for logging unhandled exceptions within GMS mapper functions
      - added get_job_summary(): a function that returns a detailed and a less detailed pandas dataframe summarizing job processing success
      - cut_GMS_obj_into_MGRS_tiles(): added logging
      - get_mask_classdefinition(): added mask_1bit class definition
      SpatialIndexMediator (SIM):
      - added new module SpatialIndexMediator for performing fast spatial queries targeting the 'scenes' table of postgreSQL database
      - added decorators to all GMS mapper functions that enable handling of unexpected exceptions
      - added new list "failed_objects" in order to collect information about all GMS objects where processing failed
      - added job database entry updates regarding failed scenes and finish time
      - added creation and logging of job summary
      - added cloud classifier objects for Landsat-5/7/8 to version control
  37. 30 Aug, 2016 1 commit
    • Daniel Scheffler's avatar
      Landsat-7 processing now available in exec_mode='Python' and... · f1f3607b
      Daniel Scheffler authored
      Landsat-7 processing now available in exec_mode='Python' and parallelization_level='scenes'; bugfixes in MGRS tile creation; database query for spatial reference not fully operable yet
      - calc_FullDataset_corner_positions(): bugfix for returning wrong data corner positions for MGRS tiles
      - transform_utm_to_wgs84(): bugfix for returning "wrong UTM zone" error in case of negative zone number (south)
      - geotransform2mapinfo(): added NotImplementedError in case of rotated datasets
      - mapinfo2geotransform(): bugfix for improper handling of UL coordinate if UL coordinate is not 1/1
      - moved find_line_intersection_point() from calc_FullDataset_corner_positions() to top level
      - log_for_fullArr_or_firstTile(): added check of  MGRS_tile_obj.logAtThisTile in case of MGRS tile -> needed for proper logging for MGRS tiles if xmin=0/ymin=0 not exists
      - delete_tempFiles() bugfix for running delete_previous_proc_level_results() for the wrong processing level
      - added method __getstate__(): for defining custom pickling settings for logger
      - added method __setstate__(): for defining custom unpickling settings for logger
      - replaced all temp_logger-calls by self.logger calls
      - log_for_fullArr_or_firstTile(self, log_msg, subset=None, logger=None): deleted deprecated logger argument
      - calculate_spatial_shifts(): updated a warning if calculated shifts are out of given threshold
      - calculate_spatial_shifts(): bugfix for returning an updated map info although shift calculation did not pass validity test
      - L1B_obj: replaced all temp_logger-calls by self.logger calls
      - added a new version of get_reference_image_params() -> not fully operable yet
      - renamed old version of get_reference_image_params() to get_reference_image_params_pgSQL()
      - added editor fold to class GMS_COREG and commented out imports
      - commented out deprecated imports and extensions of sys.path
      - L1C_obj: replaced all temp_logger-calls by self.logger calls
      - correct_shifts(): bugfix for not processing scenes if no reference scene was available
      - correcr_shifts(): out_gsd is now properly set
      - L2A_obj: replaced all temp_logger-calls by self.logger calls
       - added docstring to metaDict_to_metaODict()
      - Obj2ENVI():
           - better documentation
           - replaced all temp_logger-calls by InObj.logger calls
      - ASCII_writer(): proper exclusion of loggers from JSON serialization
       - updated GMS_JOB.jobs_table_columns due to two deleted columns in 'jobs' table
       - updated data_DB_updater() due to changes in attribute namings og GMS object
       - added class SpatialIndexMediator(): for fast geospatial queries targeting 'scenes' table of postgreSQL database
       - get_subset_GMS_obj(): major revision
          - some renamed object attributes -> new attributes: data_corners_LonLat, data_corners_utm, fullScene_corner_lonlat, bounds_LonLat, bounds_utm, corner_utm, corner_lonlat
          - bugfix for wrong map info of the returned subsetted GMS objects
          - much better documentation
      - cut_GMS_obj_into_MGRS_tiles():
          - better documentation
          - moved calculation of tile attributes to get_subset_GMS_obj()
          - added MGRS_tile_obj,logAtThisTile attribute for proper logging for MGRS tiles
      - corner_coord_to_minmax():
          - added documentation
          - updated first timestamp of Landsat-7 SLC-off data
      - added job 26185268  # 25x L7 SLC off