1. 01 Dec, 2016 2 commits
    • Daniel Scheffler's avatar
      two little bugfixes · c24961ca
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      misc.helper_functions:
      - log_uncaught_exceptions.wrapped_GMS_mapper(): Bugfix for not passing kwargs to decorator output
      config:
      - GMS_configuration.usecase.setter: Bugfix for writing to GMS_job instead of GMS_usecase
      c24961ca
    • Daniel Scheffler's avatar
      Flink compatibility update / major revision of process controller · 5138279d
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      summary:
      - revised config -> database must be queried once per machine
      - all modules of the package are now directly importable
      - process controller is now a Python class containing a couple of methods for running the job
      all modules:
      - revised import statements
      - replaced functions 'CFG.get_job()' and 'CFG.get_usecase()' by property 'CFG.job' and 'CFG.usecase'
      misc.helper_functions:
      - moved MAP() to new module processing.multiproc
      misc.SpatialIndexMediator.SpatialIndexMediatorServer:
      - bugfix for returning 'port' instead of 'process_id'
      added new package 'processing'
      - added __init__
      - added module 'multiproc'
      - added new module 'process_controller' containing new class 'process_controller' with several methods for running the GMS job
      __init__:
      - revised
      config:
      - added set_config() replacing unified_config.set_config()
      - added class GMS_configuration containing properties for 'job' and 'usecase' that replace unified_config.get_job() and unified_config.get_usecase()
      - Job:
          - added attributes 'end_time' and 'computation_time'
          - revised class structure
      - Usecase:
          - revised class structure
      - removed deprecated code
      copied old version of config to config_old (still used by deprecated process_controller_for_testing)
      added run_gms.py for running GeoMultiSens process controller from console
      - updated __version__
      5138279d
  2. 28 Nov, 2016 3 commits
  3. 17 Oct, 2016 1 commit
  4. 14 Oct, 2016 2 commits
    • Daniel Scheffler's avatar
      Bugfix for wrong no data value in case of cloud classification masks that have... · d02c9f19
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      Bugfix for wrong no data value in case of cloud classification masks that have no no-data values inside
      OUT_W:
      - mask_to_ENVI_Classification(): Bugfix for wrong no data value in case of cloud classification masks that have no no-data values inside
      DB_T:
      removed deprecated import statement
      PC:
      - deleted deprecated versions of process_controller modules
      d02c9f19
    • Daniel Scheffler's avatar
      Bugfix for unexpected pixel values in cloud masks (bug from last commit);... · b1ead53b
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      Bugfix for unexpected pixel values in cloud masks (bug from last commit); bugfix for wrong colors in some cloud classification files
      L1A_P:
      - L1A_object.logger: bugfix for not creating new logger if there is a logger without handlers
      L1B_P:
      - correct_spatial_shifts(): bugfix for wrong resampling algorithm in case of masks attribute
      OUT_W:
      - mask_to_ENVI_Classification(): bugfix for returning array with wrong pixel values in case of missing no data value
      b1ead53b
  5. 13 Oct, 2016 1 commit
    • Daniel Scheffler's avatar
      revised logging practices of whole GMS package (fixes Permission errors during... · 11e1bc1c
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      revised logging practices of whole GMS package (fixes Permission errors during logging); added direct call of external DeShifter module
      L0B_P:
      - updated calls for logger closing
      L1A_P.L1A_object:
      - converted L1A_object.logger to property 'logger'
      - added close_GMS_loggers()
      - updated __getstate__ and __setstate__
      - log_for_fullArr_or_firstTile(): added 'close logger'-calls
      - MetaObj2ODict(): bugfix for not deleting MetaObj
      - added to_GMS_file() on the basis of OUT_W.ASCII_writer() which is now deprecated
      - delete_tempFiles(): added 'close logger'-calls
      L1B_P:
      - reordered imports
      - updated logger getters and 'close logger'-statements
      - L1B_object:
          - removed its own logger setup
          - correct_spatial_shifts(): added working version
      L2A_P:
      - DESHIFTER: updated logger getters and 'close logger'-statements
      INP_R:
      - pickle_SRF_DB(): updated logger getters and 'close logger'-statements
      OUT_W:
      - ASCII_writer(): now deprecated; added 'unclosed logfile' warnings
      - Obj2ENVI: updated writer calls for GMS file
      HLP_F:
      - removed setup_logger()
      - moved close_logger() to new module 'logging'
      - moved GMS_logger() to new module 'logging'
      - failed_GMS_object: updated logger getter
      - find_nearest() is now imported from external package 'py_tools_ds'
      - removed duplicate version of corner_coord_to_minmax()
      logging:
      - added new module logging, based on previous logging functions from HLP_F
      MGRS_tile:
      - added higher timeout for getting geometry from pgSQL database
      SpatialIndexMediator:
      - SpatialIndexMediatorServer:
          - status: revised return value
          - start(): revised warnings
          - stop(): revised return value
      PC:
      - job.logger is now an instance of GMS_logger
      - added new version of L2A_map()
      - added shutdown_loggers statement
      11e1bc1c
  6. 12 Oct, 2016 3 commits
    • Daniel Scheffler's avatar
      Bugfix for not properly closed logfiles; bugfix for · db54bd81
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      L0B_P:
      - L0B_object: bugfix for not properly closed logfiles
      L1A_P:
      - added assignment of L1A_object.GMS_identifier['logger']
      - added verbose mode to get_MetaObj() and disabled it
      - __getstate__(): bugfix
      L1B_P:
      - Scene_finder.filter_possib_ref_scenes(): bugfix for not filtering by projection
      ENV:
      - added dummy function check_paths()
      HLP_F:
      - close_logger(): bugfix for not checking if given logger has handlers
      CFG:
      - deleted two deprecated comments
      PC:
      - added some job IDs
      - added 'job comment info' during startup
      db54bd81
    • Daniel Scheffler's avatar
      Bugfix for not properly closed logfiles; added environment checker for Spatial... · 77a105fc
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      Bugfix for not properly closed logfiles; added environment checker for Spatial Index Mediator Server
       L1A_P:
       - L1A_Obj.__getstate__(): bugfix for not properly closed logfiles
      L1B_P:
      - get_reference_image_params_pgSQL(): bugfix for not properly closed logfile
      L2A_P:
      - correct_shifts(): bugfix for not properly closed logfile
      ENV:
      - check_dependencies(): added functionality to check current status of Spatial Index Mediator Server and to start the server if needed
      - added _log_or_print()
      HLP_F:
      - added close_logger()
      - added draft of class GMS_logger (not usable so far)
      - subcall_with_output(): added keywords no_stdout and no_stderr
      SpatialIndexMediator:
      - added class SpatialIndexMediatorServer: a class for interacting with java server (status, start, stop, restart, is_running, port, ...)
      CFG:
      - added job.path_spatIdxSrv
      PC:
      - added calls of environment checkers
      pgSQL_DB:
      - added 'path_spatial_index_mediator_server' to config table
      77a105fc
    • Daniel Scheffler's avatar
      removed a deprecated import statement · e9f82e2a
      Daniel Scheffler authored
      L1A_P:
      - see above
      e9f82e2a
  7. 07 Oct, 2016 6 commits
    • Daniel Scheffler's avatar
      added a dummy environment checker module · 8e827622
      Daniel Scheffler authored and Mathias Peters's avatar Mathias Peters committed
      8e827622
    • Daniel Scheffler's avatar
      808ab70a
    • Daniel Scheffler's avatar
      40cb3727
    • Daniel Scheffler's avatar
      removed deprecated code in L1B_P; bugfix for wrong pixel size in reprojected... · c44bbeb4
      Daniel Scheffler authored
      removed deprecated code in L1B_P; bugfix for wrong pixel size in reprojected MGRS tiles (fixed in 'py_tools_ds'-package)
      L1B_P:
      - refactored Scene_getter to Scene_finder
      - moved COREG_OLD.get_reference_image_params_pgSQL() and COREG_OLD.sceneIDList_to_filt_overlap_scenes() to Scene_finder
      - removed deprecated class COREG_OLD
      c44bbeb4
    • Daniel Scheffler's avatar
      moved a couple of functions from GEOPROCESSING to external package... · a7c6c26f
      Daniel Scheffler authored
      moved a couple of functions from GEOPROCESSING to external package 'py_tools_ds' and added direct imports within respective modules
      GEOP:
      - the following functions moved to 'py_tools_ds':
          - transform_utm_to_wgs84
          - transform_wgs84_to_utm
          - transform_any_prj
          - reproject_shapelyPoly
          - lonlat_to_pixel
          - latLonToPixel
          - pixelToLatLon
          - pixelToMapYX
          - mapXY2imXY
          - imXY2mapXY
          - isProjectedOrGeographic
          - EPSG2Proj4
          - EPSG2WKT
          - WKT2EPSG
          - get_UTMzone
          - geotransform2mapinfo
          - mapinfo2geotransform
          - get_corner_coordinates
          - get_prjLonLat
          - get_proj4info
          - proj4_to_dict
          - prj_equal
          - corner_coord_to_minmax
          - get_footprint_polygon
          - get_overlap_polygon
          - find_line_intersection_point
          - calc_FullDataset_corner_positions
          - is_point_on_grid
          - is_coord_grid_equal
          - snap_bounds_to_pixGrid
      IO:
      - deleted deprecated module envifilehandling
      - renamed envifilehandling_BD to envifilehandling
      - deleted deprecated module sysenvironment
      a7c6c26f
    • Daniel Scheffler's avatar
      - replaced the duplicated fork of COREG within L1B_P by an imports from external packages · be17e860
      Daniel Scheffler authored
      - GeoMultiSens now depends on CoReg_Sat and py_tools_ds!
      GEOP:
      - moved warp_ndarray to external package 'py_tools_ds'
      - updated warp_ndarray calls
      L1A_P:
      - added property coreg_needed
      - added cubic resampling for MGRS tiles that have to be reprojected to the next UTM zone
      - to_MGRS_tiles(): added verbose mode
      L1B_P:
      - COREG and DESHIFTER are now imported from external package CoReg_Sat
      - deleted deprecated draft of COREG_GMS
      - added class Scene_getter(): a class used to find a proper geospatial reference scene for a given target scene
      - added class ref_Scene
      - L1B_object:
          - revised the whole class (it now generates its coreg_info and deshift_results on its own)
          - added property spatRef_available
          - added get_spatial_reference_scene()
          - added get_opt_bands4matching (based on an earlier version from COREG)
          - added coregister_spatially()
          - added correct_spatial_shifts(): not yet working
      L1C_P:
      - L1C_object: updated __init__() args of super class L1B_object
      L2A_P:
      - updated warp_ndarray calls
      L2B_P:
      - spectral homogenization is now only executed if target CWLs are different to source CWLs
      INP_R:
      - fixed an unclosed file within GMSfile2dict
      OUT_W:
      - fixed an unclosed file within Obj2ENVI
      PC:
      - added a new version of L1B_map_1()
      be17e860
  8. 26 Sep, 2016 8 commits
  9. 23 Sep, 2016 1 commit
  10. 22 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      Implemented ENVI compressed output format · e6fca93d
      Daniel Scheffler authored
      GeoArray:
      - GeoArray: improved error handling
      OUT_W:
      - added 'file compression' key to enviHdr_keyOrder
      - added write_ENVI_compressed()
      - Obj2ENVI: added calls of write_ENVI_compressed
      HLP_F:
      - log_uncaught_exceptions: replaced a hard coded path reference
      PC:
      - L2C_map_1: enabled file compression for L2C MGRS tiles output
      e6fca93d
  11. 20 Sep, 2016 2 commits
  12. 19 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      Bugfix for "missing L2B mask files" during continued processing that earlier... · afcb0fa5
      Daniel Scheffler authored
      Bugfix for "missing L2B mask files" during continued processing that earlier stopped in L2B processing; implemented proper MGRS tiling scheme according to project agreements
      GEOP:
      - updated some deprecated 3rd-party-library references
      - added transform_any_prj(): a function fort transforming XY-coordinates from any source to any target projection
      - warp_ndarray():
          - implemented extra option 'outExtent_within' allowing to output arrays with a larger geographical extent than the input image (needed for MGRS tiling scheme)
      - improved some docstrings
      - added prj_equal(): a function to quickly check if two projections are equal
      - added snap_bounds_to_pixGrid(): a function to snap map bounds to a given pixel grid
      - added clip_array_using_mapBounds(): a function for clipping arrays with a map info using a given bounding box
      L0A_P:
      - add_local_availability():
          - replaced a hard coded list of processing levels
          - bugfix for returning an invalid processing level if processing is already done
      L1A_P:
      - fill_from_disk(): bugfix for copying memory addresses  when fill_from_disk() is called in multiprocessing
      - log_for_fullArr_or_firstTile(): bugfix for not handling empty arr_pos
      - added get_subset_obj(), based on older code from helper_functions: a function to generate subsets of GMS objects based on bounding box that can contain image AND map coordinates whereas map coordinates can also have a different projection (needed for proper generation of MGRS tiles)
       - added to_MGRS_tiles(, based on older code from helper_functions: a function for cutting a GMS object into MGRS tiles
       L1B_P:
       - L1B_object.apply_deshift_results():
           - bugfix for not updating geoinformations of 'masks' attribute after having applied spatial shift corrections
      META:
      - added 'data ignore value' to metadata
      IO:
      - refactored module 'gms_io' to 'io'
      ARR:
      - added submodule 'GeoArray' to io:
          - added class 'GeoArray': a class for simplifying array access, regardless to their actual memory location (in memory or on disk)
          - added _clip_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box in the same projection like the array itself
          - added  get_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box that can have any projection
      OUT_W:
      - added 'data ignore value' to enviHdr_keyOrder
      - added 'data ignore value' to mask_to_ENVI_Classification()
      - added set_output_nodataVal(): a function for adding a data ignore value to an already written file
      - Obj2ENVI: bugfix for not handling empty arr_pos
      DB_T:
      - delete_processing_results(), GMS_JOB.__delete_procdata(), GMS_JOB.delete_procdata_of_entire_job(), GMS_JOB.delete_procdata_of_failed_sceneIDs(), added functionality to delete a specific processing level from disk
      HLP_F:
      - revised find_nearest(): it now supports an automatic rounding algorithm that rounds a value to the nearest neighbour
      - replaced a deprecated 3rd-party function reference
      - get_arrSubsetBounds_from_shapelyPolyLonLat(): added flag 'ensure_valid_coords'
      MRGS_tile:
      - added submodule "mgrs_tile" for easily retrieving informations about MGRS tiles (bounding box, projection conversions, buffering, etc.)
      SpatialIndexMediator:
      - replaced wrong host address
      PC:
      - updated calls for MGRS tiling within L2C_map_1()
      afcb0fa5
  13. 10 Sep, 2016 1 commit
    • Daniel Scheffler's avatar
      Bugfix for wrong scene counts in job summary; added functions for deleting processed data · 0d17e8bb
      Daniel Scheffler authored
       DB_T:
       - added delete_processing_results(): a function for deleting processing results of a given scene ID
       - GMS_JOB: added __delete_procdata, delete_procdata_of_entire_job and delete_procdata_of_failed_sceneIDs
       PC:
       - added proper counting for succeeded and failed objects
       - adjusted maximum count of workers for L1A and L2B processing
       - added some jobs needed for bugfixing
       IPYNB:
       - updated GMS beta usecase ## dev GMS41.ipynb
      0d17e8bb
  14. 08 Sep, 2016 2 commits
    • Daniel Scheffler's avatar
      Bugfix for import error during unpickling cloud classifiers due to not... · 02c0b1e0
      Daniel Scheffler authored
      Bugfix for import error during unpickling cloud classifiers due to not properly set environment variable
      L1A_P:
      - calc_cloud_mask():  bugfix for appending the wrong folder to PATH -> caused import errors during cloud classifier unpickling
      02c0b1e0
    • Daniel Scheffler's avatar
      Bugfix for processing each scene twice in processing levels above L1B (fixes missing *masks_L1B.*) · eb0468dd
      Daniel Scheffler authored
      META:
      - get_EarthSunDistance(): bugfix for not returning 1.0 if job.path_earthSunDist is not available
      INP_R:
      - revision of get_list_GMSfiles() - now accepts 2 arguments instead of one argument tuple
      CFG:
      - adjusted database connection parameters for cluster "gms41" -> ATTENTION: localhost could be a problem on other nodes than gms41
      PC:
      - added function is_inMEM() for checking if a dataset has been recently processed or has to be read from disk
      - updated INP_R.get_list_GMSfiles() calls
      - bugfix for also trying to give a job summary based on processed scenes although no scene has been processed
      IPYNB:
      - added iPython Notebook GMS beta usecase ## dev GMS41.ipynb
      eb0468dd
  15. 07 Sep, 2016 1 commit
  16. 06 Sep, 2016 1 commit
  17. 02 Sep, 2016 4 commits