cmemsapi package#

Submodules#

cmemsapi.cmemsapi module#

Main module.

cmemsapi.cmemsapi.set_target_directory(local_storage_directory=None)[source]#

Returns working directory where data is saved.

Default value (None) creates a directory (copernicus-tmp-data) in the current working directory.

Parameters

local_storage_directory (path or str, optional) – A path object or string. The default is None.

Returns

target_directory – A path to the directory where data is saved.

Return type

path

cmemsapi.cmemsapi.multireplace(tobereplaced, substitute)[source]#

Returns replaced string given string and substitute map.

Parameters

  • tobereplaced (str) – String to execute replacements on.

  • substitute (dict) – Substitute dictionary {value to find: value to replace}.

Returns

Replaced string.

Return type

str

cmemsapi.cmemsapi.query(question, default='yes')[source]#

Returns answer from a yes/no question, read from user’s input.

Parameters

  • question (str) – String written as a question, displayed to user.

  • default (str, optional) – String value to be presented to user to help . The default is “yes”.

Raises

ValueError – Raise error to continue asking question until user inputs one of the valid choice.

Returns

Returns True if user validates question, False otherwise.

Return type

bool

cmemsapi.cmemsapi.get_config_constraints()[source]#

Returns constraints configuration as dict from which data requests will be stacked.

Returns

split_dict – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.get_credentials(file_rc=None, sep='=')[source]#

Returns Copernicus Marine Credentials.

Credentials can be specified in a file or if ommitted, manually by user’s input.

Parameters

  • file_rc (str or path, optional) – Location of the file storing credentials. The default is None.

  • sep (str, optional) – Character used to separate credential and its value. The default is =.

Raises

SystemExit – Raise an error to exit program at fatal error (wrong credentials etc).

Returns

  • copernicus_username (str) – Copernicus Marine username.

  • copernicus_password (str) – Copernicus Marine password.

cmemsapi.cmemsapi.check_credentials(user, pwd)[source]#

Check provided Copernicus Marine Credentials are correct.

Parameters
Returns

Returns True if credentials are correst, False otherwise.

Return type

bool

cmemsapi.cmemsapi.get_viewscript()[source]#

Ask the user to input the VIEW_SCRIPT command.

Returns

view_myscript – String representing the TEMPLATE COMMAND generated by the webportal. Example is available at https://tiny.cc/get-viewscript-from-web

Return type

str

cmemsapi.cmemsapi.viewscript_string_to_dict(viewmy_script)[source]#

Convert the VIEW SCRIPT string displayed by the webportal to a dict.

Parameters

viewmy_script (TYPE) – DESCRIPTION.

Returns

vs_dict – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.get_dates_stack(vs_dict, check_stack, size=None, renew=None)[source]#

Update a dict containing VIEW SCRIPT values with dates for sub-requests.

Parameters

  • vs_dict (TYPE) – DESCRIPTION.

  • check_stack (TYPE) – DESCRIPTION.

  • size (TYPE, optional) – DESCRIPTION. The default is None.

  • renew (TYPE, optional) – DESCRIPTION. The default is None.

Returns

vs_dict – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.viewscript_dict_to_string(size=None, strict=None, cmd=None, **kwargs)[source]#

Convert the dict containing keys and values of the VIEW SCRIPT, into a string as displayed by the webportal.

Parameters

  • size (TYPE, optional) – DESCRIPTION. The default is None.

  • strict (TYPE, optional) – DESCRIPTION. The default is None.

  • cmd (TYPE, optional) – DESCRIPTION. The default is None.

  • **kwargs (TYPE) – DESCRIPTION.

Returns

command – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.get_data(command=None, user=None, pwd=None, size=None)[source]#

Returns status of binary netCDF file or, if size is specified, potential result file size, whose units is kBytes.

Parameters

  • command (TYPE, optional) – DESCRIPTION. The default is None.

  • user (TYPE, optional) – DESCRIPTION. The default is None.

  • pwd (TYPE, optional) – DESCRIPTION. The default is None.

  • size (TYPE, optional) – DESCRIPTION. The default is None.

Returns

  • returncode (TYPE) – DESCRIPTION.

  • message (TYPE) – DESCRIPTION.

cmemsapi.cmemsapi.check_data(returncode, message, command=None, user=None, stack=None, size=None)[source]#

Returns True if status of the submitted request is successful, False otherwise.

Parameters

  • returncode (TYPE) – DESCRIPTION.

  • message (TYPE) – DESCRIPTION.

  • command (TYPE, optional) – DESCRIPTION. The default is None.

  • user (TYPE, optional) – DESCRIPTION. The default is None.

  • stack (TYPE, optional) – DESCRIPTION. The default is None.

  • size (TYPE, optional) – DESCRIPTION. The default is None.

Raises

SystemExit – Raise an error to exit program at fatal error due to server maintenance.

Returns

valid_check – DESCRIPTION.

Return type

bool

cmemsapi.cmemsapi.process_viewscript(target_directory, view_myscript=None, user=None, pwd=None, forcestack=None)[source]#

Generates as many data requests as required to match initial VIEW_SCRIPT.

Parameters

  • target_directory (str or path) – DESCRIPTION.

  • view_myscript (str, optional) – DESCRIPTION. The default is None.

  • user (str, optional) – DESCRIPTION. The default is None.

  • pwd (str, optional) – DESCRIPTION. The default is None.

  • forcestack (bool, optional) – DESCRIPTION. The default is None.

Raises

ValueError – DESCRIPTION.

Returns

On success, returns path of the output file matching the VIEW_SCRIPT data request, False otherwise.

Return type

TYPE

cmemsapi.cmemsapi.convert_size_hr(size_in_bytes)[source]#

Get size from bytes and displays to user in human readable.

Parameters

size_in_bytes (TYPE) – DESCRIPTION.

Returns

DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.get_disk_stat(drive=None)[source]#

Get disk size statistics.

Parameters

drive (TYPE, optional) – DESCRIPTION. The default is None.

Returns

disk_stat – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.get_file_size(files)[source]#

Get size of file(s) in bytes.

Parameters

files (TYPE) – DESCRIPTION.

Returns

mds_size – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.check_file_size(mds_size, default_nc_size=None)[source]#

Check size of file(s).

Parameters

  • mds_size (TYPE) – DESCRIPTION.

  • default_nc_size (TYPE, optional) – DESCRIPTION. The default is None.

Returns

check_fs – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.display_disk_stat(mds_size)[source]#

Display hard drive statistics to user.

Parameters

mds_size (TYPE) – DESCRIPTION.

Returns

mds_size_hr – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.get_file_pattern(outname, sep='-', rem=-1, advanced=True)[source]#

Retrieve a file_pattern from a filename and advanced regex.

Parameters

  • outname (str) – Filename from which a pattern must be extracted.

  • sep (str, optional) – Separator. The default is ‘-‘.

  • rem (TYPE, optional) – Removal parts. The default is -1.

  • advanced (TYPE, optional) – Advanced regex. The default is True.

Returns

file_pattern – The file_pattern extracted from filename.

Return type

str

cmemsapi.cmemsapi.get_years(ncfiles, sep='-')[source]#

Retrieve a list of years from a list of netCDF filenames.

Parameters

  • ncfiles (list) – List of filenames from which years will be extracted.

  • sep (TYPE, optional) – Separator. The default is ‘-‘.

Returns

years – List of years.

Return type

set

cmemsapi.cmemsapi.get_ncfiles(target_directory, file_pattern=None, year=None)[source]#

Retrieve list of files, based on parameters.

Parameters

  • target_directory (str) – DESCRIPTION.

  • file_pattern (TYPE, optional) – DESCRIPTION. The default is None.

  • year (TYPE, optional) – DESCRIPTION. The default is None.

Returns

ncfiles – List of strings containing absolute path to files.

Return type

list

cmemsapi.cmemsapi.set_outputfile(file_pattern, target_directory, target_out_directory=None, start_year=None, end_year=None)[source]#

Set output filename based on variables.

Parameters

  • file_pattern (TYPE) – DESCRIPTION.

  • target_directory (TYPE) – DESCRIPTION.

  • target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.

  • start_year (TYPE, optional) – DESCRIPTION. The default is None.

  • end_year (TYPE, optional) – DESCRIPTION. The default is None.

Returns

outputfile – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.over_write(outputfile)[source]#

Ask user if overwrite action should be performed.

Parameters

outputfile (TYPE) – DESCRIPTION.

Returns

ow – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.del_ncfiles(ncfiles)[source]#

Delete files.

Parameters

ncfiles (TYPE) – DESCRIPTION.

Returns

DESCRIPTION.

Return type

bool

cmemsapi.cmemsapi.to_nc4(mds, outputfile)[source]#

Convert file(s) to one single netCDF-4 file, based on computer limits.

Parameters

  • mds (TYPE) – DESCRIPTION.

  • outputfile (TYPE) – DESCRIPTION.

Returns

nc4 – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.to_csv(mds, outputfile)[source]#

Convert file(s) to one single csv file, based on computer limits.

Parameters

  • mds (TYPE) – DESCRIPTION.

  • outputfile (TYPE) – DESCRIPTION.

Returns

csv – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.to_mfcsv(input_directory, output_directory, max_depth_level=None)[source]#

Convert netcdf file(s) to multiple csv files, based on MS Excel Limits.

Parameters

  • input_directory (TYPE) – DESCRIPTION.

  • output_directory (TYPE) – DESCRIPTION.

  • max_depth_level (TYPE, optional) – DESCRIPTION. The default is None.

Returns

mfcsv – DESCRIPTION.

Return type

TYPE

cmemsapi.cmemsapi.to_nc4_csv(ncfiles, outputfile, skip_csv=False, default_nc_size=None)[source]#

Convert file(s) to both netCDF-4 and csv files, based on computer limits.

Parameters

  • ncfiles (TYPE) – DESCRIPTION.

  • outputfile (TYPE) – DESCRIPTION.

  • skip_csv (TYPE, optional) – DESCRIPTION. The default is False.

  • default_nc_size (TYPE, optional) – DESCRIPTION. The default is None.

Returns

  • nc4 (bool) – DESCRIPTION.

  • csv (bool) – DESCRIPTION.

  • check_ow (bool) – DESCRIPTION.

cmemsapi.cmemsapi.post_processing(outname, target_directory, target_out_directory=None, delete_files=True)[source]#

Post-process the data already located on disk.

Concatenate a complete timerange in a single netCDF-4 file, or if not possible, stack periods on minimum netCDF-4 files (either by year or by month). There is a possibility to delete old files to save space, thanks to convertion from nc3 to nc4 and to convert to CSV, if technically feasible.

Parameters

  • outname (TYPE) – DESCRIPTION.

  • target_directory (TYPE) – DESCRIPTION.

  • target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.

  • delete_files (TYPE, optional) – DESCRIPTION. The default is True.

Raises

SystemExit – DESCRIPTION.

Returns

processing – DESCRIPTION.

Return type

bool

See also

get_file_pattern

called from this method

get_ncfiles

called from this method

get_years

called from this method

set_outputfile

called from this method

to_nc4_csv

called from this method

del_ncfiles

called from this method

cmemsapi.cmemsapi.get(local_storage_directory=None, target_out_directory=None, view_myscript=None, user=None, pwd=None, forcestack=False, delete_files=True)[source]#

Download and post-process files to both compressed and tabular formats, if applicable.

Download as many subsets of dataset required to fulfill an initial data request based on a template command, called VIEW SCRIPT generated by Copernicus Marine website (https://marine.copernicus.eu). Then, all files are post-processed locally. e.g to concatenate in a single file, to save space (thanks to nc3 -> nc4), to convert to CSV (if technically possible), and to delete old files. End-user is guided throughout the process if no parameter is declared. To get started, this function is the main entry point.

Parameters

  • local_storage_directory (TYPE, optional) – DESCRIPTION. The default is None.

  • target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.

  • view_myscript (TYPE, optional) – DESCRIPTION. The default is None.

  • user (TYPE, optional) – DESCRIPTION. The default is None.

  • pwd (TYPE, optional) – DESCRIPTION. The default is None.

  • forcestack (TYPE, optional) – DESCRIPTION. The default is False.

  • delete_files (TYPE, optional) – DESCRIPTION. The default is True.

Return type

True.

See also

process_viewscript

Method to parse VIEW SCRIPT

post_processing

Method to convert downloaded data to other format

Examples

Ex 1. Let the user be guided by the script with interactive questions:

>>> cmemstb get

Ex 2. Get data matching a VIEW SCRIPT template command passed as parameter:

>>> cmemstb get --view_myscript="python -m motuclient --motu https://nrt.cmems-du.eu/motu-web/Motu --service-id GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS --product-id global-analysis-forecast-phy-001-024 --longitude-min -20 --longitude-max 45 --latitude-min 25 --latitude-max 72 --date-min \"2019-08-18 12:00:00\" --date-max \"2020-08-31 12:00:00\" --depth-min 0.493 --depth-max 0.4942 --variable thetao --out-dir <OUTPUT_DIRECTORY> --out-name <OUTPUT_FILENAME> --user <USERNAME> --pwd <PASSWORD>"

Notes

For Windows Operating System Users and when using the --view_myscript as parameter, you might want to double check that double quote around dates are well escaped (see above example).

cmemsapi.cmemsapi.cli()[source]#

Method to enable Command Line Interface and to expose only useful method for beginners.

Return type

None.

Module contents#

Top-level package for Copernicus Marine ToolBox.