backscatter.dmap.dmap module

exception backscatter.dmap.dmap.DmapDataError

Bases: Exception

Raised if there is an error in parsing of data

exception backscatter.dmap.dmap.EmptyFileError

Bases: Exception

Raised if the dmap file is empty or corrupted

class backscatter.dmap.dmap.RawDmapArray(name, dmap_type, data_type_fmt, mode, dimension, arr_dimensions, data)

Bases: object

Holds all the same data that the original C dmap array struct holds + some additional type information

get_arr_dimensions()

Returns a list of array dimensions

Returns:

arr_dimensions

get_data()

Returns the array data

Returns:

data

get_datatype_fmt()

Returns the string format identifier of the scaler that corresponds to the DMAP type

Returns:

data_type_fmt

get_dimension()

Returns the number of dimensions in the array

Returns:

dimension

get_mode()

Returns the mode of the array

Returns:

mode

get_name()

Returns the name of the array

Returns:

name

get_type()

Returns the DMAP type of the array

Returns:

dmap_type

set_arr_dimensions(arr_dimensions)

Sets the list of dimensions for the array

Parameters:

arr_dimensions – list of dimensions for the array

set_data(data)

Sets the array data

Parameters:

data – the data associated with the array

set_datatype_fmt(fmt)

Sets the DMAP type string format identifier of the array

Parameters:

fmt – the string format identifier

set_dimension(dimension)

Sets the number of array dimensions

Parameters:

dimension – total array dimensions

set_mode(mode)

Sets the mode of the array

Parameters:

mode – the mode of the array

set_name(name)

Sets the name of the array

Parameters:

name – name of the array

set_type(data_type)

Sets the DMAP type of the array

Parameters:

data_type – DMAP type of the array

class backscatter.dmap.dmap.RawDmapRead(dmap_data, stream=False)

Bases: object

Contains members and methods relating to parsing files into raw Dmap objects. Takes in a buffer path to decode. Default is open a file, but can optionally use a stream such as from a real time socket connection

build_n_dimension_list(dim, data_type_fmt)

This is used to build a list of multiple dimensions without knowing them ahead of time. This method is used to manually parse arrays from a dmap

Parameters:

• dim – list of dimensions

• data_type_fmt – string format identifier of the DMAP data type

Returns:

n dimensional list of data parsed from buffer

convert_datatype_to_fmt(data_type)

Converts a parsed data type header field from the dmap record to a data type character format

Parameters:

data_type – DMAP data type from parsed record

Returns:

a string format identifier for the DMAP data type

get_num_bytes(data_type_fmt)

Returns the number of bytes associated with each type

Parameters:

data_type_fmt – a string format identifier for the DMAP data type

Returns:

size in bytes of the data type

get_records()

Returns the list of parsed DMAP records

Returns:

dmap_records

parse_array(record_size)

Parses a new dmap array from bytearray

Returns:

new RawDmapArray with parsed data

parse_record()

Parses a single dmap record from the buffer

parse_scaler()

Parses a new dmap scaler from bytearray

Returns:

new RawDmapScaler with parsed data

read_data(data_type_fmt)

Reads an individual data type from the buffer

Given a format identifier, a number of bytes are read from the buffer and turned into the correct data type

Parameters:

data_type_fmt – a string format identifier for the DMAP data type

Returns:

parsed data

read_numerical_array(data_type_fmt, dimensions, total_elements)

Reads a numerical array from bytearray using numpy

Instead of reading array elements one by one, this method uses numpy to read an entire section of the buffer into a numpy array and then reshapes it to the correct dimensions. This method is prefered due to massive performance increase

Parameters:

• data_type_fmt – a string format identifier for the DMAP data type

• dimensions – a list of each array dimension

• total_elements – total elements in the array

Returns:

parsed numpy array in the correct shape

test_initial_data_integrity()

Quickly parses the data to add up data sizes and determine if the records are intact. There still may be errors, but this is a quick initial check

class backscatter.dmap.dmap.RawDmapRecord

Bases: object

Contains the arrays and scalers associated with a dmap record.

add_array(new_array)

Adds a new array to the DMAP record

Parameters:

new_array – new RawDmapArray to add

add_scaler(new_scaler)

Adds a new scaler to the DMAP record

Parameters:

new_scaler – new RawDmapScaler to add

get_arrays()

Returns the list of arrays in the DMAP record

Returns:

arrays

get_num_arrays()

Returns the number of arrays in the DMAP record

Returns:

num_arrays

get_num_scalers()

Returns the number of scalers in the DMAP record

Returns:

num_scalers

get_scalers()

Returns the list of scalers in the DMAP record

Returns:

scalers

set_arrays(arrays)

Sets the DMAP array list to a new list

Parameters:

arrays – new list of arrays

set_num_arrays(num_arrays)

Sets the number of arrays in this DMAP record

Parameters:

num_arrays – number of arrays

set_num_scalers(num_scalers)

Sets the number of scalers in this DMAP record

Parameters:

num_scalers – number of scalers

set_scalers(scalers)

Sets the DMAP scaler list to a new list

Parameters:

scalers – new list of scalers

class backscatter.dmap.dmap.RawDmapScaler(name, dmap_type, data_type_fmt, mode, data)

Bases: object

Holds all the same data that the original C dmap scaler struct holds + some additional type format identifiers

get_data()

Returns the scaler data

Returns:

data

get_datatype_fmt()

Returns the string format identifier of the scaler that corresponds to the DMAP type

Returns:

data_type_fmt

get_mode()

Returns the mode of the scaler

Returns:

mode

get_name()

Returns the name of the scaler

Returns:

name

get_type()

Returns the DMAP type of the scaler

Returns:

dmap_type

set_data(data)

Sets the data of the scaler

Parameters:

data – data for the scaler to contain

set_datatype_fmt(fmt)

Sets the string format identifier of the scaler that corresponds to the DMAP type of the scaler

Parameters:

fmt – DMAP type string format of the scaler

set_mode(mode)

Sets the mode of the scaler

Parameters:

mode – scaler mode

set_name(name)

Sets the name of the scaler

Parameters:

name – scaler name

set_type(data_type)

Sets the DMAP type of the scaler

Parameters:

data_type – DMAP type of the scaler

class backscatter.dmap.dmap.RawDmapWrite(data_dicts, file_path, ud_types={})

Bases: object

Contains members and methods relating to encoding dictionaries into a raw dmap buffer.

The ud_types are use to override the default types for riding. Useful if you want to write a number as a char instead of an int for example

convert_fmt_to_dmap_type(fmt)

Converts a format specifier to a dmap type to be written as part of buffer

Parameters:

fmt – a string format identifier for the DMAP data type

Returns:

DMAP type

data_dict_to_dmap_rec(data_dict)

This method converts a data dictionary to a dmap record.

The user defined dictionary specifies if any default types are to be overridden with your own type. This functions runs through each key/val element of the dictionary and creates a RawDmapArray or RawDmapScaler and adds them to a RawDmapRecord. Any python lists are converted to numpy arrays for fast and efficient convertion to bytes

Parameters:

data_dict – a dictionary of data to encode

dmap_array_to_bytes(array)

This method converts a RawDmapArray to the byte format to be written out.

The format is name,then type, number of dimensions, dimensions, array data.

Parameters:

array – a RawDmapArray

Returns:

total bytes the array will take up

dmap_record_to_bytes(record)

This method converts a dmap record to the byte format that is written to file.

Format is code,length of record,number of scalers,number of arrays, followed by the scalers and then the arrays

Parameters:

record – a RawDmapRecord

dmap_scaler_to_bytes(scaler)

This method converts a RawDmapScaler to the byte format written out.

The bytes are written as a name, then type, then data

Parameters:

scaler – a RawDmapScaler

Returns:

total bytes the scaler will take up

find_datatype_fmt(data)

Input could be an array of any dimensions so will recurse until fundamental type is found

Parameters:

data – data for which to find its type format

Returns:

a string format identifier for the python data type

type_to_fmt(data)

Finds data types and converts them to a format specifier for struct or numpy packing methods

Parameters:

data – data for which to find type

Returns:

a string format identifier for the python data type

backscatter.dmap.dmap.dicts_to_file(data_dicts, file_path, file_type='')

This function abstracts the type overrides for the main SuperDARN file types. These dictionaries write out the types to be compatible with C DMAP reading

Parameters:

• data_dicts – python dictionaries to write out

• file_path – path for which to write the data to file

• file_type – type of SuperDARN file with what the data is

backscatter.dmap.dmap.dmap_rec_to_dict(rec)

Converts the dmap record data to a easy to use dictionary

Parameters:

rec – a RawDmapRecord

Returns:

a dictionary of all data contained in the record

backscatter.dmap.dmap.parse_dmap_format_from_file(filepath, raw_dmap=False)

Creates a new dmap object from file and then formats the data results

into a nice list of dictionaries

Parameters:

• filepath – file path to get DMAP data from

• raw_dmap – a flag signalling to return the RawDmapRead object

instead of data dictionaries :returns: list of data dictionaries

backscatter.dmap.dmap.parse_dmap_format_from_stream(stream, raw_dmap=False)

Creates a new dmap object from a stream and then formats the data results into a nice list of dictionaries

Parameters:

• stream – buffer of raw bytes to convert

• raw_dmap – a flag signalling to return the RawDmapRead object

instead of data dictionaries :returns: list of data dictionaries