1. 24 Aug, 2016 2 commits
    • Daniel Scheffler's avatar
      added Jupyter notebook "GMS beta usecase" containing code for executing GMS... · 4d148f9f
      Daniel Scheffler authored
      added Jupyter notebook "GMS beta usecase" containing code for executing GMS beta usecase on another system
      4d148f9f
    • Daniel Scheffler's avatar
      First version that is able to produce MGRS cubes in mass production (so far... · 56edb6e3
      Daniel Scheffler authored
      First version that is able to produce MGRS cubes in mass production (so far only fully operable for Landsat-8)
      GEOP:
      - merged calc_FullDataset_corner_positionsOLD() and calc_FullDataset_corner_positions() to calc_FullDataset_corner_positions()
      	- the function now features two algorithms: 'numpy' (for Landsat-7 ETM+ SLC-off) and 'shapely' (for all remaining scenes)
      - deleted calc_FullDataset_corner_positionsOLD()
      - added is_point_on_grid(): Checks if a given point is exactly on the given coordinate grid. Needed to check if scene has to be coregistered or not.
      - added is_coord_grid_equal(): Needed to check if scene has to be coregistered or not.
      L0A_P:
      - updated name of jobs table in postgreSQL database
      L1A_P:
      - added attribute dataset_ID -> Needed to check if scene has to be coregistered or not.
      - added attributes scenes_proc_ID and mgrs_tiles_proc_ID -> needed for database entries
      - archive_to_rasObj() now asserts that the read archive contains any files
      - bugfix for Landsat-7 SLC-off scenes: archive_to_rasObj() now expects more files in the archive due to SLC off masks
      - bugfix calc_mask_nodata(): data had been read from wrong source if job.exec_mode=='Python'
      - revised calc_cloud_mask() -> the resulting array is now contains correct pixel values and is properly included in GMS_obj.masks
      - bugfix for Landsat-7 SLC-off scenes: calc_corner_positions() now calls a different algorithm that can deal with SLC-off gaps
      - bugfix calc_corner_positions(): incorrect getter for projection
      - bugfix build_combined_masks_array(): it now out always outputs unsigned integer 8bit
      - added delete_previous_proc_level_results(): needed to save disk space by deleting results of previous processing levels that are not needed anymore
      - delete_tempFiles() now also deletes results of previous processing levels on demand (depending on job.exec_L1AP[2])
      L1B_P:
      - added class COREG_GMS (not yet working) -> will replace class COREG soon and will synchronize its changes with external library CoReg_Sat
      - COREG.__init__():
      	- added attributes dataset_ID and coreg_needed -> Needed to check if scene has to be coregistered or not.
      	-> the class now only runs coregistration if its really needed. otherwise its skipped
      - COREG.get_updated_map_info(): now no output stream anymore in quiet mode
      - L1B_object:
      	- slighty changed logging behavior
      	- apply_deshift_results(): - bugfix for not properly updated map info and projection
      	- apply_deshift_results(): - bugfix for not properly updated dimensions of the mask attributes
      L1C_P:
      - delete_tempfiles() now also deletes results of previous processing levels  on demand
      L2A_P:
      - bugfix in get_DESHIFTER_configs(): not properly closed gdal dataset
      - added attributes shift_dataset_ID, x_shift_px, y_shift_px and deshift_needed in order to check if deshifting is really needed or not
      - correct_shifts():
      	- added proper logging
      	- bugfix for missing updated_projection in case coregistration has been skipped
      	- bugfix for not properly closed gdal dataset
      	- bugfix for wrong resampling method for cloud mask (cubic results in wrong output pixel values) -> now nearest neighbour
      	- added 'is clipped to extent' to deshift_results
      L1C_P:
      - bugfix spectral_homogenization(): now it also accepts string type array attributes if exec_mode=='Python'
      - revised interpolate_cube_linear()
      META:
      - CS_DATUM, CS_UTM_ZONE, CS_TYPE are now properly written to gms-files
      - added metaDict_to_metaODict() needed by output writer
      INP_R:
      - read_mask_subset(): added missing 'MGRS_tile' subset type
      OUT_W:
      - added silent_envi_write_image(): a monkey patch in order to silence spectral.io.envi._write_image
      - added write_ordered_envi_header(): in order to write properly sorted ENVI header files
      - revised Tiles_Writer(): previous version caused output errors for mask attributes of GMS objects
      - revised mask_to_ENVI_Classification(): bugfixes for wrong conversions of numpy arrays to ENVI classdification ready array and metadata
      - added check_header_not_empty()
      - major revision of Obj2ENVI(): now it properly writes MRGS tiles if exec_mode=='Python'
      - added database update for MGRS tiles (new records created in new database table 'mgrs_tiles_proc')
      DB_T:
      - splitted get_postgreSQL_matchingExp() to get_postgreSQL_value() and get_postgreSQL_matchingExp()
      	-> much more python variable types can now be converted to postgreSQL types
      - added create_record_in_postgreSQLdb(): needed to create new records for MGRS tiles
      - added class GMS_JOB: a job manager for GMS jobs
      	-> current features:
      		- create a job in the postgreSQL database, based on a list of archive files
      		- get the details about an existing job
      - added add_missing_filenames_in_pgSQLdb(): useful for finding and fixing database errors
      - import_shapefile_into_postgreSQL_database() updated name of postgreSQL database
      - major revision of data_DB_updater(): now it properly updates both tables 'scenes_proc' and 'mgrs_scenes_proc'
      HLP_F:
      - added attribute proc_chain
      - get_subset_GMS_obj(): bugfix for clipping the wrong tile dimension
      - get_subset_GMS_obj(): added updated tile corners and data corners to MGRS tiles
      - cut_GMS_obj_into_MGRS_tiles(): added calculation of tile bounds as well as data corner positions
      - _numba_array_merger(): added a type assertion
      - get_valid_arrSubsetBounds(): bugfix for returning wrong tile dimensions (1 row/column too big)
      - sceneID_to_trueDataCornerLonLat(): added valid polygon assertion
      - added class Landsat_entityID_decrypter: needed by class JOB_GMS for finding correct sensor parameters for Landsat data
      PG:
      - added get_path_rawdata()
      - get_path_procdata(): changed returned path structure for MGRS tiles saved to disk
      - get_baseN(): changed returned basename for GMS objects on disk
      CFG:
      - updated postgreSQL database connection credentials
      - added a "delete output of previous porcessing levels"-flag to exec__L**P
      - usecase: added datasetid_spatial_ref, virtual_sensor_name, target_gsd, EPSG, spatial_ref_gridx, spatial_ref_gridy
      PC:
      - added multiple job IDs containing pre-defined database jobs
      - updated L1B_map_1(): shift calculation is now skipped if no reference scene is needed or available
      - added L2A_map(): needed to execute processing in mass production, controlled by Python process controller
      - revised tempfile deletion within L1C_map_1(), L2A_map_1(), L2B_map_1()
      - updated L2A mapper calls -> exec_mode=='Python' is now supported
      - added hardcoded thread count restriction to L2B mapper call due to memory errors in Python execution mode with parallelization level 'scenes'
      pgSQLdb:
      - renamed database from usgs_cache to geomultisens
      - added table 'mrgs_tiles_proc'
      - renamed table 'scenes_jobs' to 'jobs'
      - updated jobs table with new columns
      - fixed missing filenames in scenes table
      - ...
      56edb6e3
  2. 12 Aug, 2016 1 commit
    • Daniel Scheffler's avatar
      GEOP: · a7920668
      Daniel Scheffler authored
      - warp_ndarray(): fixed a bug in the validation of the output dimensions calculated by rasterio that always expected a quadratic input array for performing a correction of out_cols and out_rows
      - added reproject_shapelyPoly()
      - added mapXY2imXY()
      - added imXY2mapXY()
      - added proj4_to_dict()
      L1A_P:
      - fixed a bug in calc_cloud_mask()
      - fixed a bug in calc_corner_positions()
      L1B_P:
      - fixed a bug in get_reference_image_params()
      - fixed a bug in apply_deshift_results() when applying results to a 2D array
      L2A_P:
      - changed titlebar
      L2B_P:
      - changed titlebar
      L2C_P:
      - changed titlebar
      - added a dummy version of L2C_object
      L2D_P:  Level 2D processor has been deleted due to changes in the L2 processing architecture
      META:
      - added META.CS_EPSG attribute
      OUT_W:
      - Obj2ENVI(): added functionality to write MGRS tiles as compressed ENVI files (compression mode not yet working)
      - dirty hack in mask_to_ENVI_Classification() -> still not working
      - added get_dtypeStr()
      - added new version of write_shp()
      DB_T:
      - moved some code from get_scene_and_dataset_infos_from_postgreSQLdb() to new function execute_pgSQL_query()
      - added get_pgSQL_geospatial_query_cond()
      - added get_overlapping_MGRS_tiles()
      - added get_overlapping_MGRS_tiles2()
      - added pdDataFrame_to_sql_k()
      - added import_shapefile_into_postgreSQL_database()
      - added postgreSQL_table_to_csv() (not yet working)
      HLP_F:
      - added docstring to setup_logger()
      - added gzipfile()
      - added ENVIfile_to_ENVIcompressed()
      - added get_subset_GMS_obj()
      - added cut_GMS_obj_into_MGRS_tiles()
      - added an extra assertion to postgreSQL_poly_to_cornerLonLat()
      - added some type hints
      - added postgreSQL_geometry_to_shapelyPolygon()
      - added shapelyPolygon_to_postgreSQL_geometry()
      - added get_imageCoords_from_shapelyPoly()
      - added get_valid_arrSubsetBounds()
      - added get_arrSubsetBounds_from_shapelyPolyLonLat() -> needed to get image coordinates of MGRS tiles
      - added get_UL_LR_from_shapefile_features()
      PG:
      - added functionality to get pathes for MGRS-tiled products
      CFG:
      - added additional output folder for MGRS-tiled products
      PC:
      - removed L2D_P import
      - added cloud mask and corner coordinates calculation to L2A_map_2
      - added L2C_map_1()
      - adjusted L2B calls
      - added first version of L2C calls
      a7920668
  3. 13 Jul, 2016 1 commit
    • Daniel Scheffler's avatar
      First prototype of algorithms for geometrical and spectral resolution working... · 9142f468
      Daniel Scheffler authored
      First prototype of algorithms for geometrical and spectral resolution working in map-reduce context (L2A, L2B).
      GEOP:
          - renamed GEOPROCESSING_BD.py to GEOPROCESSING.py
          - moved get_prjLonLat(), get_proj4info(), corner_coord_to_minmax() to GEOP
          - added docstrings to DN2Rad(), DN2TOARef(), TOARad2Kelvin_fastforward(), DN2DegreesCelsius_fastforward()
      L1A_P:
          - L1A_object.fill_arr_from_disk(): further silencing of console outputs
      L1B_P:
          - moved get_DESHIFTER_configs() and class DESHIFTER() to L2A_P
          - adjusted initial values for COREG attributes related to reference image (not None anymore in order to make L2A_P work if shift calculation failed)
          - increased database statement timeouts for queries within get_reference_image_params() to 25sek
          - L1B_object():
              - added attribute "deshift_results"
              - removed deprecated code
              - added join_deshift_results()
              - revised apply_deshift_results()
      L2A_P:
          -  added get_DESHIFTER_configs() and class DESHIFTER() from L1B_P
          - fixed two bugs in DESHIFTER.correct_shifts() where DESHIFTER.band2process was not respected and whole image cube was read instead of only one band
          - added class L2A_object()
      L2B_P:
          - added class L2B_object()
          - L2B_object():
              - added interpolate_cube_linear()
              - added spectral_homogenization()
      META:
          - renamed METADATA_BD.py to METADATA.py
      INP_R:
          - added quiet mode to read_ENVIfile()
      OUT_W:
          - added enviHdr_keyOrder using list from reorder_envi_header()
          - fixed a bug in reorder_ENVI_header() that caused repetitions of header keys
          - adjusted print_dict within mask_to_ENVI_Classification() in order to also support L2A and L2B
      HLP_F:
          - added parent objects for L2A and L2B in parentObjDict
          - added type hints to cut_GMS_obj_into_blocks() and merge_GMS_tiles_to_GMS_obj()
      GITIGNORE:
          - updated .gitignore file
      CFG:
          - added virtual_sensor_id, datasetid_spectral_ref, target_CWL, target_FWHM to usecase class by querying the database
      PC:
          - added type hints to mapper functions
          - revised L2A_map_2()
          - added L2B_map_1()
          - revised/added L2A algorithm calls (only Flink mode is supported so far)
          - added L2B algorithm calls (only Flink mode is supported so far)
      pgDB:
          - added Sentinel-2A virtual sensors to virtual_sensors table (different spatial resolutions)
          - added wavelengths positions and band widths to virtual_sensors table
      9142f468
  4. 10 Jul, 2016 1 commit
    • Daniel Scheffler's avatar
      Python execution mode now fully operational || implemented functions for... · 8abcf2ed
      Daniel Scheffler authored
      Python execution mode now fully operational || implemented functions for adding externally downloaded scenes to GMS database
      GEOP:
          - fixed imports
          - fixed a deprecated keyword in warp_ndarray()
      L1A_P:
          - fixed imports
          - added 'Python' exec_mode to L1A_object.fill_from_disk()
          - fixed return values calc_cloud_mask()
          - added some docstrings and type hints
          - fixed a bug where MetaObj.SpecVals are not updated after applying no data mask to object attribute
          - revised build_L1A_masks(9 -> new version called build_combined_masks_array()
              -> not longer called from process controller but only from output writer
          - improved delete_tempfiles()
      L1B_P:
          - revised COREG.__init__() to a much clearer version -> all attributes are now always existing
          - revised L1B_object.__init__()
      L1C_P:
          - fixed a bug where no data mask has been read from wrong processing level
      L1D_P:
           - L1D_P was deprecated -> not existing anymore
      INP_R:
          - added som docstrings and type hints
          - read_ENVIfile(), read_ENVI_image_data_as_array() and read_mask_subset() now also support .bsq files as input paths
      OUT_W:  major revision!
          - deleted param_dic
          - Tiles_writer(): fixed a bug that prevented writing of 2D arrays
          - revision of reorder_ENVI_header() -> more stable, more effective
          - revision of mask_to_ENVI_Classification()
          - revision of Obj2ENVI():
              - now also supports a tempfile mode that allows to directly write arrays of TILES from within the mapper function
              - added image_type_dict, metaAttr_dict, out_dtype_dict
              - now directly creates InObj.masks on demand
              - revised writing of array attributes
      DB_T:  multiple new functions for adding externally downloaded satellite scenes to GMS database
          - added many docstrings and type hints
          - added get_postgreSQL_matchingExp()
          - get_info_from_postgreSQLdb()
              - added a custom timeout keyword to get_info_from_postgreSQLdb() in order to allow non-static query timings
              - added the possibility to pass a list of values within conditions dictionary
          - added set_info_in_postgreSQLdb()
          - added get_dict_satellite_name_id()
          - added get_dict_sensor_name_id()
          - added get_entityIDs_from_filename()
          - added get_filename_by_entityID()
          - added get_notDownloadedsceneIDs()
          - added add_externally_downloaded_data_to_GMSDB()
          - added archive_exists_on_fileserver()
      HLP_F:
          - added silentremove()
          - fixed a missing colormap in get_mask_colormap()
      PG:
          - revised path_generator.__init__()
          - added get_path_cloudmaskdata()
          - added get_outPath_hdr()
      CFG:
          - removed static database statement timeout -> now directly adjustable by called functions
      PC:
          - L1A_map_2: array attributes are now directly written to disk in exec_mode 'Python' @ parallelization_level 'tiles'
            in order to save a lot of time for pickling big objects
          - L1A_obj.build_L1A_masks() is no longer called from PC
          - fixed a bug that prevented appending L1B_newTiles to L1B database objects
      8abcf2ed
  5. 28 Jun, 2016 1 commit
    • Daniel Scheffler's avatar
      Major update with further developments of map-reduce processing chain (working... · ee3fd0eb
      Daniel Scheffler authored
      Major update with further developments of map-reduce processing chain (working until L1C) and full Sentinel-2 support for all processing levels
      GEOP:
          - caught a bug in rasterio library that caused GEOP.warp_ndarray() to return warped images with wrong dimension (1 row more than expected),
            which caused L1B_P.COREG() to fail
          - added zoom_2Darray_to_shapeFullArr() and adjust_acquisArrProv_to_shapeFullArr()
            for adjusting dimensions of acquisition geometry arrays provided by data providers (e.g. Sentinel-2)
          - revision of calc_FullDataset_corner_positions() - now it does not fail anymore if only 2 image corners are visible in an image
            (granule with nodata on the one side)
      L0A_P:
          - fixed a bug in get_AllWrittenProcL_dueLog() returning the same processing level multiple times
            if same processing level has been written out more than once
      L1A_P:
          - revised L1A_object.fill_from_disk() in order to make it useable by all GMS objects in all processing levels
          - calc_mean_VAA(): corrected assignment of VAA_mean in case of Sentinel-2 granules
            where VAA cannot be calculated using true corner coordinates
          - moved cut_L1A_obj_into_blocks(), merge_L1A_tiles_to_L1A_obj(), _numba_array_merger() to HLP_F
            in order to make them usable everywhere in the processing chain, not only in L1A processing
      L1B_P:
          - added logging of database queries
          - added L1B_object.set_coreg_info() and moved assignment of L1B_object.coreg_info there
      L1C_P:
          - logging pattern changed so that log messages are only sent in case the full array or the first tile is processed
          - added special case of calculating viewing zenith angle for Sentinel-2 by zooming the respective provider array to the right dimension
          - moved dummy_atm_corr() to L1C_object.atm_corr() and updated log messages
          - added L1C_object.delete_tempFiles()
      META:
          - removed deprecated Sentinel-2 test mode in METADATA.Read_Sentinel2A_xmls()
          - added correct assignment of METADATA.gResolution for all supported datasets
          - added 'ViewingAngle_arrProv' and 'IncidenceAngle_arrProv' to Meta2ODict()
      INP_R:
          - added docstring to read_ENVI_image_data_as_array()
          - revised read_ENVI_image_data_as_array(), read_mask_subset(), GMSfile2dict()
          - revised ()
      OUT_W:
          - added L1C data to param_dic
          - added the possibility to include small numpy arrays as lists in *.gms files
            (needed for Sentinel-2's 'ViewingAngle_arrProv' and 'IncidenceAngle_arrProv')
          - added docstrings to Tiles_Writer(), Obj2ENVI()
          - added add_ENVIclassificationMeta_to_meta() for modifying metadata dictionaries
            so that respective images are recognized as classification images by ENVI
          - Obj2ENVI(): added the possibility to write GMS objects that only represent a tile of a full object
            which allows writing of processed data directly out of the Flink mappers making slow back-pickling of mapper results not needed anymore
      DB_T:
          - added docstrings to get_info_from_SQLdb() and get_info_from_postgreSQLdb()
          - improved error handling in get_info_from_postgreSQLdb()
      HLP_F:
          - added parentObjDict and initArgsDict
            that are needed to make GMS object cutter and tile joiner generally usable in the entire processing chain
          - added exclude_val argument to find_nearest()
          - added docstring to get_image_tileborders()
          - added cut_GMS_obj_into_blocks(), merge_GMS_tiles_to_GMS_obj() and _numba_array_merger() based on similar methods from L1A_P
      CONFIG:
          - increased postgreSQL database statement timeout to 15 sec
          - added a validation of output writer configurations depending on job.exec_mode
      PC:
          - revised L1B_map_1()
          - added L1C_map_1()
          - revised L2A_map_1() - not fully working yet
          - revised run_processController_in_multiprocessing()
              -> now it fully supports L1A-L1C processing and includes a first version L2A processing (not fully working yet)
      ee3fd0eb
  6. 18 Jun, 2016 1 commit
    • Daniel Scheffler's avatar
      Sentinel-2 compatibility update - L1A/L1B processing now fully compatible with Sentinel-2 data · 5ff92e51
      Daniel Scheffler authored
      L0A_P: added subset splitting for Sentinel-2 and Aster
      L0B_P: added filename attribute that now give a clear hint about filename on disk
      L1A_P:
          - archive_to_rasObj(): added support for Sentinel-2
          - added set_arr_desc_from_MetaObj()
          - added log_for_fullArr_or_firstTile()
          - added rescale_array()
          - DN2TOARadRefTemp() renamed to calc_TOARadRefTemp()
          - calc_TOARadRefTemp(): revised and facilitated
              - supports now DNs and Refs as Input
              - supports radiometric rescaling of arrays
          - calc_cloud_mask(): some bugfixes and code improvements
          - apply_nodata_mask_to_ObjAttr(): can now call apply_nodata_mask_to_saved_ENVIfile()
          - build_L1A_masks(): bugfix
      L1B_P:
          - fixed a bug that prevented adding of coreg_info to L1B_object if no reference scene had been available within COREG
          - added inheritance
      L1C_P:
          - added inheritance
      METADATA:
          - added Read_Sentinel2A_xmls(): metadata parser for Sentinel-2 data
          - some bugfixes and code improvements
          - added a ScaleFactor attribute
          - revised SPOT GMS_sensorcodes
          - revised self.PhysUnit
          - removed deprecated self.Satellite_GMS()
          - added Sentinel-2 metadata to isPAN(), isTHERMAL(), get_orbit_params(), get_special_values(), get_LayerBandsAssignment()
      OUT_W: bugfix
      INP_R: bugfix
      HLP_F:
          - revised SPOT GMS_sensorcodes
          - improved group_objects_by_attributes()
          - added find_in_xml_root(), find_in_xml(), get_values_from_xml(), stack_detectors()
      PG:
          - added Sentinel-2 compatibility
          - changed path derivation using entityid to filename
      CFG: added usecase.scale_factor_TOARef
      - added Sentinel-2 SRFs to database
      5ff92e51
  7. 10 Jun, 2016 1 commit
    • Daniel Scheffler's avatar
      Flink compatibility update 2 · 588c814d
      Daniel Scheffler authored
      GEOP:
          - moved get_lonlat_coord_array(), calc_VZA_array(), calc_AcqTime_array(), calc_SZA_SAA(), calc_SZA_SAA_array(), calc_RAA_array() outside of GEOPROCESSING object which makes them callable without disk access
          - added GEOP.get_subsetProps_from_shapeFullArr_arrPos()
          - added GEOP.get_subsetProps_from_subsetArg()
      L1A_P.L1A_object / L1C_P.L1C_object:
          - combined calc_VZA_array(), calc_SZA_SAA_array(), calc_RAA_array() to L1C_P.L1C_object.calc_acquisition_illumination_geometry()
          - moved get_lonlat_coord_array() to L1C_P.L1C_object
          - L1A_P: some bugfixes
      META: refactored Meta2SpyfileheaderMeta() to Meta2ODict()
      INP_R:
          - added read_mask_subset()
          - added quiet mode to read_ENVI_image_data_as_array()
      PG:
          - added docstings
          - added get_path_maskdata()
      588c814d
  8. 07 Jun, 2016 3 commits
  9. 01 Jun, 2016 1 commit
    • Daniel Scheffler's avatar
      Flink compatibility update: · f930ca7e
      Daniel Scheffler authored
      - code changes in order to prevent temporal disk outputs that are possibly not available to the next Flink mapper
      - L1A_P now fully supports execution of all functions without accessing harddisk (except from the data reader)
      - config now features a "Flink" mode that can be activated using job.exec_mode='Flink'
      - revision of L1A processor in order to facilitate processing chain
      - more efficient vectorization in several functions
      - some bugfixes
      - added some docstrings
      - GEOP: DN2Rad(), DN2TOARef(), TOARad2Kelvin_fastforward(), DN2DegreesCelsius_fastforward(), calc_FullDataset_corner_positions() are now callable directy from GEOP without instancing GEOP.GEOPROCESSING
      f930ca7e
  10. 13 Apr, 2016 3 commits
  11. 05 Apr, 2016 2 commits
    • Daniel Scheffler's avatar
      5c52a4fd
    • Daniel Scheffler's avatar
      process controller: extensive revision in order to clean up old stuff and to... · e17d387b
      Daniel Scheffler authored
      process controller: extensive revision in order to clean up old stuff and to make it more understandable
      - added process_controller_for_testing_OLD_20160405.py: a snapshot of the previous version
      L1B_P.COREG: revised get_reference_image_params():
       -> the algorithm now respects satellite, overlap percentage, cloudcover, acquisition date with different priorities and prints a download command to output stream in order to start the download an processing of a not yet existing but needed reference scene
       L1B_P.COREG: improved get_opt_winpos_winsize():
       - the algorithm now recognizes if the chosen matching window in not completely within the overlap area of the input images and automatically reduces window size
      DB_T: revised get_overlapping_scenes_from_postgreSQLdb()
      HLP_F: improved reorder_CornerLonLat() and sceneID_to_trueDataCornerLonLat()
      CFG: added java command queries and datasetid_spatial_ref
      e17d387b
  12. 16 Mar, 2016 1 commit
  13. 15 Mar, 2016 1 commit
    • Daniel Scheffler's avatar
      L1B_P.COREG now supports direct warping from arrays instead of reading inputs from disk · 5abead96
      Daniel Scheffler authored
      -> now nearly no physical tempfiles are needed anymore
      - revised code structure
      - GDAL variant of get_image_windows_to_match() is now deprecated and has been replaced by rasterio variant
      - COREG: added non-working code of an automatic getter for matching specttral bands
      - added a function to clean up tempfiles
       GEOP.warp_ndarray: fixed wrong output when a numpy view is taken as input for warping instead if an independent array
       L1A_P: minor code improvements
       INP_R: fixed a bug that caused wrong output of read_ENVIfile()
       HLP_F: added a function for subprocess calls including proper returning of error messages
       PG: fixed a bug in PG.__init__()
       config:  added database connection and statement timeouts
       PC: added code for reading of L1A cubes as input for L1B_P
      5abead96
  14. 09 Mar, 2016 2 commits
    • Daniel Scheffler's avatar
      L1B_P.DESHIFTER.correct_shifts now corrects shifts directly,... · 0037b20a
      Daniel Scheffler authored
      L1B_P.DESHIFTER.correct_shifts now corrects shifts directly, align_coordinate_grids() and resample_without_grid_aligning() are deprecated and not existing anymore
      L1B_P.DESHIFTER.correct_shifts: fixed a bug that caused wrong output with a non-standard output gsd
      L1B_P.DESHIFTER.correct_shifts: minor bugfixes
      L1B_P.COREG: scenes without existing reference scene are now automatically skipped wit a warning
      HLP_F: added group_tuples_by_keys_of_tupleElements() and group_dicts_by_key() that are used in process controller
      0037b20a
    • Daniel Scheffler's avatar
      L1B_P: third running version: · 81b8ecd8
      Daniel Scheffler authored
      - full multiprocessing compatibility
      - added a DESHIFTER class, that can be called from each processing level >= L1B
      - added a function to generate configurations for DESHIFTER in order to call DESHIFTER in multiprocessing from process controller
      - much faster IO
      - added a method to L1B object to perform deshifting without calling DESHIFTER from process controller
      - added a method to L1B object to apply deshifting results to object attributes
      - some bugfixes
      - config module: added values for setting global configurations for DESHIFTER (align_coord_grids, match_gsd, target_gsd) -> allow global spatial homogenization
      GEOP:
      - added warp_ndarray(): a method for warping numpy array that are associated to geo information (coordinate transformation) WITHOUT writing temporary files to disk
      - added isProjectedOrGeographic()
      - added wkt2epsg() for conversion of WKT projections to EPSG
      - added geotransform2mapinfo() and mapinfo2geotransform()
      - moved get_corner_coordinates() from L1B_P to GEOP
      - some bugfixes
      L0P: fixed a bug that caused a complete reprocessing when only L1B output is was deleted
      METADATA: fixed a bug when retrieving radiance and reflectance gains from Landsat MTL files
      HLP_F: added find_nearest(): a function to get the nearest value of a 1D-array in relation to a specific value
      PG: added get_tempfile(): a function to generate tempfiles that can be passed to functions that require a physical file on the disk
      process controller: updated L1B_P calls
      81b8ecd8
  15. 29 Feb, 2016 1 commit
  16. 25 Feb, 2016 1 commit
    • Daniel Scheffler's avatar
      L1B_P: second running version: · 6c280204
      Daniel Scheffler authored
      - much more stability and robustness
      - many nw features that have been developed in the context of the standalone version
      L1A_P:
      - Bugfix: fixed a bug that prevented further processing when array attributes contain only a path
      OUT_W:
      - added a function to write shapefiles from shapely polygons
      6c280204
  17. 24 Feb, 2016 1 commit
    • Daniel Scheffler's avatar
      L1B_P: added a function to get updated map info · bef92b9d
      Daniel Scheffler authored
      L1B_P: added an object to create L1B product
      GEOP: some style adjustments
      GEOP: pixelToMapYX: added the possibility to pass geotransform and projection
      OUT_W: minor bugfixes
      PC: added code for writing L1B products to disk
      bef92b9d
  18. 10 Feb, 2016 1 commit
    • Daniel Scheffler's avatar
      - L1B_P: L1B-shift calculation now working but input data is still read from disk · 944b84d1
      Daniel Scheffler authored
      - L1B_P: implemented dynamic postgresql-query for getting a matching reference image for an image to be shifted
      - all database related functions moved to new module called database_tools
      - all path related functions moved to a new module called path_generator
      - added  some docstrings
      - GEOP: added pixelToMapYX()
      - GEOP Bugfix: get_overlap_polygon() now returns an overlap percentage of 0 if no overlap exists (instead of an error)
      - L0A_P: derivation of scene basic metadata needed for job creation moved to DB_T.get_scene_and_dataset_infos_from_postgreSQLdb(sceneid)
      944b84d1
  19. 05 Feb, 2016 1 commit
    • Daniel Scheffler's avatar
      Some major bugfixes within preparation of L1B-P. · 891590f2
      Daniel Scheffler authored
      - moved generation of absolute paths from L0B-P to an external path_generator class within HLP_F
      - renamed GMS_proc_level to proc_level
      - converted L1A_obj.__call__() to L1A_obj.fill_from_disk()
      - added multiple docstrings
      - Bugfix: L1A_obj-attribute names read from ENVI header now compatible with further processing
      - Bugfix: fillvals are now passed correctly to calc_VZA_array, calc_SZA_SAA_array, calc_RAA_array
      - Bugfix: added missing argument to numba_array_merger
      - added a dummy version of L1B_P.get_opt_winpos_winsize()
      - GEOP: added functions for getting the overlap between two footprint polygons
      - INP_R: get_list_GMSfiles() now compatible with webapp GMS_call_type
      - Bugfix: OUT_W: fixed a bug that cause overwriting of L1A_obj attributes by OUT_W
      - HLP_F: added functions for conversion of postgreSQL geometry, postgreSQL polygon and internal CornerLonLat lists
      - added detection of debugger that runs process controller
      - commented multiple sections of process controller
      - some style adjustments
      - revised settings of environment variables in run.sh
      891590f2
  20. 03 Feb, 2016 1 commit
  21. 17 Dec, 2015 1 commit
    • Daniel Scheffler's avatar
      Processing can now be started from web interface AND console. · 1e39d909
      Daniel Scheffler authored
      - Integration of PostgreSQL database complete
      - removed the subsystem for Landsat data and added a sensormode
      - renamed LXXobj.scene_ID to LXXobj.entity_ID in order to respect "sceneid" identifier of postgreSQL database
      - config: added database queries for config params in case the process controller is started from web interface
         - access to config params now via __builtins__
      - L0A_P: revised add_local_availability()
      - L0B_P: added getter functions for path_procdata, baseN, path_logfile
      - acquisition date is now a datetime instance
      1e39d909
  22. 10 Dec, 2015 9 commits
  23. 09 Dec, 2015 3 commits