python-benedict

python-benedict is a dict subclass with keylist/keypath support, I/O shortcuts (base64, csv, ini, json, pickle, plist, query-string, toml, xls, xml, yaml) and many utilities... for humans, obviously.

Features

• 100% backward-compatible, you can safely wrap existing dictionaries.
• Keylist support using list of keys as key.
• Keypath support using keypath-separator (dot syntax by default).
• Keypath list-index support (also negative) using the standard [n] suffix.
• Normalized I/O operations with most common formats: base64, csv, ini, json, pickle, plist, query-string, toml, xls, xml, yaml.
• NEW Multiple I/O operations backends: filepath (read/write), url (read-only), s3 (read/write).
• Many utility and parse methods to retrieve data as needed (check the API section).
• Well tested. ;)

Index

• Installation
• Usage
  • Basics
  • Keylist
  • Keypath
    • Custom keypath separator
    • Change keypath separator
    • Disable keypath functionality
    • List index support
  • API
    • Utility methods
    • I/O methods
    • Parse methods
• Testing
• License

Installation

• Run pip install python-benedict

Usage

Basics

benedict is a dict subclass, so it is possible to use it as a normal dictionary (you can just cast an existing dict).

from benedict import benedict

# create a new empty instance
d = benedict()

# or cast an existing dict
d = benedict(existing_dict)

# or create from data source (filepath, url or data-string) in a supported format:
# Base64, CSV, JSON, TOML, XML, YAML, query-string
d = benedict('https://localhost:8000/data.json', format='json')

# or in a Django view
params = benedict(request.GET.items())
page = params.get_int('page', 1)

Keylist

Wherever a key is used, it is possible to use also a list (or a tuple) of keys.

d = benedict()

# set values by keys list
d['profile', 'firstname'] = 'Fabio'
d['profile', 'lastname'] = 'Caccamo'
print(d) # -> { 'profile':{ 'firstname':'Fabio', 'lastname':'Caccamo' } }
print(d['profile']) # -> { 'firstname':'Fabio', 'lastname':'Caccamo' }

# check if keypath exists in dict
print(['profile', 'lastname'] in d) # -> True

# delete value by keys list
del d['profile', 'lastname']
print(d['profile']) # -> { 'firstname':'Fabio' }

Keypath

. is the default keypath separator.

If you cast an existing dict and its keys contain the keypath separator a ValueError will be raised.

In this case you should use a custom keypath separator or disable keypath functionality.

d = benedict()

# set values by keypath
d['profile.firstname'] = 'Fabio'
d['profile.lastname'] = 'Caccamo'
print(d) # -> { 'profile':{ 'firstname':'Fabio', 'lastname':'Caccamo' } }
print(d['profile']) # -> { 'firstname':'Fabio', 'lastname':'Caccamo' }

# check if keypath exists in dict
print('profile.lastname' in d) # -> True

# delete value by keypath
del d['profile.lastname']

Custom keypath separator

You can customize the keypath separator passing the keypath_separator argument in the constructor.

If you pass an existing dict to the constructor and its keys contain the keypath separator an Exception will be raised.

d = benedict(existing_dict, keypath_separator='/')

Change keypath separator

You can change the keypath_separator at any time using the getter/setter property.

If any existing key contains the new keypath_separator an Exception will be raised.

d.keypath_separator = '/'

Disable keypath functionality

You can disable the keypath functionality passing keypath_separator=None in the constructor.

d = benedict(existing_dict, keypath_separator=None)

You can disable the keypath functionality using the getter/setter property.

d.keypath_separator = None

List index support

List index are supported, keypaths can include indexes (also negative) using [n], to perform any operation very fast:

# Eg. get last location cordinates of the first result:
loc = d['results[0].locations[-1].coordinates']
lat = loc.get_decimal('latitude')
lng = loc.get_decimal('longitude')

API

• Utility methods

  • clean
  • clone
  • dump
  • filter
  • find
  • flatten
  • groupby
  • invert
  • items_sorted_by_keys
  • items_sorted_by_values
  • keypaths
  • match
  • merge
  • move
  • nest
  • remove
  • rename
  • search
  • standardize
  • subset
  • swap
  • traverse
  • unflatten
  • unique

• I/O methods

  • from_base64
  • from_csv
  • from_ini
  • from_json
  • from_pickle
  • from_plist
  • from_query_string
  • from_toml
  • from_xls
  • from_xml
  • from_yaml
  • to_base64
  • to_csv
  • to_ini
  • to_json
  • to_pickle
  • to_plist
  • to_query_string
  • to_toml
  • to_xml
  • to_yaml

• Parse methods

  • get_bool
  • get_bool_list
  • get_date
  • get_date_list
  • get_datetime
  • get_datetime_list
  • get_decimal
  • get_decimal_list
  • get_dict
  • get_email
  • get_float
  • get_float_list
  • get_int
  • get_int_list
  • get_list
  • get_list_item
  • get_phonenumber
  • get_slug
  • get_slug_list
  • get_str
  • get_str_list
  • get_uuid
  • get_uuid_list

Utility methods

These methods are common utilities that will speed up your everyday work.

Utilities that accept key argument(s) also support keypath(s).

Utilities that return a dictionary always return a new benedict instance.

• clean

# Clean the current dict instance removing all empty values: None, '', {}, [], ().
# If strings or collections (dict, list, set, tuple) flags are False,
# related empty values will not be deleted.
d.clean(strings=True, collections=True)

• clone

# Return a clone (deepcopy) of the dict.
c = d.clone()

• dump

# Return a readable representation of any dict/list.
# This method can be used both as static method or instance method.
s = benedict.dump(d.keypaths())
print(s)
# or
d = benedict()
print(d.dump())

• filter

# Return a filtered dict using the given predicate function.
# Predicate function receives key, value arguments and should return a bool value.
predicate = lambda k, v: v is not None
f = d.filter(predicate)

• find

# Return the first match searching for the given keys/keypaths.
# If no result found, default value is returned.
keys = ['a.b.c', 'm.n.o', 'x.y.z']
f = d.find(keys, default=0)

• flatten

# Return a new flattened dict using the given separator to join nested dict keys to flatten keypaths.
f = d.flatten(separator='_')

• groupby

# Group a list of dicts at key by the value of the given by_key and return a new dict.
g = d.groupby('cities', by_key='country_code')

• invert

# Return an inverted dict where values become keys and keys become values.
# Since multiple keys could have the same value, each value will be a list of keys.
# If flat is True each value will be a single value (use this only if values are unique).
i = d.invert(flat=False)

• items_sorted_by_keys

# Return items (key/value list) sorted by keys.
# If reverse is True, the list will be reversed.
items = d.items_sorted_by_keys(reverse=False)

• items_sorted_by_values

# Return items (key/value list) sorted by values.
# If reverse is True, the list will be reversed.
items = d.items_sorted_by_values(reverse=False)

• keypaths

# Return a list of all keypaths in the dict.
# If indexes is True, the output will include list values indexes.
k = d.keypaths(indexes=False)

• match

# Return a list of all values whose keypath matches the given pattern (a regex or string).
# If pattern is string, wildcard can be used (eg. [*] can be used to match all list indexes).
# If indexes is True, the pattern will be matched also against list values.
m = d.match(pattern, indexes=True)

• merge

# Merge one or more dictionary objects into current instance (deepupdate).
# Sub-dictionaries keys will be merged toghether.
# If overwrite is False, existing values will not be overwritten.
# If concat is True, list values will be concatenated toghether.
d.merge(a, b, c, overwrite=True, concat=False)

• move

# Move an item from key_src to key_dst.
# It can be used to rename a key.
# If key_dst exists, its value will be overwritten.
d.move('a', 'b', overwrite=True)

• nest

# Nest a list of dicts at the given key and return a new nested list
# using the specified keys to establish the correct items hierarchy.
d.nest('values', id_key='id', parent_id_key='parent_id', children_key='children')

• remove

# Remove multiple keys from the dict.
# It is possible to pass a single key or more keys (as list or *args).
d.remove(['firstname', 'lastname', 'email'])

• rename

# Rename a dict item key from 'key' to 'key_new'.
# If key_new exists, a KeyError will be raised.
d.rename('first_name', 'firstname')

• search

# Search and return a list of items (dict, key, value, ) matching the given query.
r = d.search('hello', in_keys=True, in_values=True, exact=False, case_sensitive=False)

• standardize

# Standardize all dict keys, e.g. "Location Latitude" -> "location_latitude".
d.standardize()

• subset

# Return a dict subset for the given keys.
# It is possible to pass a single key or more keys (as list or *args).
s = d.subset(['firstname', 'lastname', 'email'])

• swap

# Swap items values at the given keys.
d.swap('firstname', 'lastname')

• traverse

# Traverse a dict passing each item (dict, key, value) to the given callback function.
def f(d, key, value):
    print(f'dict: {d} - key: {key} - value: {value}')
d.traverse(f)

• unflatten

# Return a new unflattened dict using the given separator to split dict keys to nested keypaths.
u = d.unflatten(separator='_')

• unique

# Remove duplicated values from the dict.
d.unique()

I/O methods

It is possible to create a benedict instance directly from data-source (filepath, url, s3 or data-string) by passing the data source and the data format (optional, default 'json') in the constructor.

# filepath
d = benedict('/root/data.yml', format='yaml')

# url
d = benedict('https://localhost:8000/data.xml', format='xml')

# s3
d = benedict('s3://my-bucket/data.xml', s3_options={"aws_access_key_id": "...", "aws_secret_access_key": "..."})

# data-string
d = benedict('{"a": 1, "b": 2, "c": 3, "x": 7, "y": 8, "z": 9}')

These methods simplify I/O operations with most common formats: base64, csv, ini, json, pickle, plist, query-string, toml, xls, xml, yaml.

In all from_* methods, the first argument can be: url, filepath or data-string.

In all to_* methods, if filepath='...' kwarg is specified, the output will be also saved at the specified filepath.

• from_base64

# Try to load/decode a base64 encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to choose the subformat used under the hood:
# (`csv`, `json`, `query-string`, `toml`, `xml`, `yaml`), default: 'json'.
# It's possible to choose the encoding, default 'utf-8'.
# A ValueError is raised in case of failure.
d = benedict.from_base64(s, subformat='json', encoding='utf-8', **kwargs)

• from_csv

# Try to load/decode a csv encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to specify the columns list, default: None (in this case the first row values will be used as keys).
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/csv.html
# A ValueError is raised in case of failure.
d = benedict.from_csv(s, columns=None, columns_row=True, **kwargs)

• from_ini

# Try to load/decode a ini encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/configparser.html
# A ValueError is raised in case of failure.
d = benedict.from_ini(s, **kwargs)

• from_json

# Try to load/decode a json encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/json.html
# A ValueError is raised in case of failure.
d = benedict.from_json(s, **kwargs)

• from_pickle

# Try to load/decode a pickle encoded in Base64 format and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/pickle.html
# A ValueError is raised in case of failure.
d = benedict.from_pickle(s, **kwargs)

• from_plist

# Try to load/decode a p-list encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/plistlib.html
# A ValueError is raised in case of failure.
d = benedict.from_plist(s, **kwargs)

• from_query_string

# Try to load/decode a query-string and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# A ValueError is raised in case of failure.
d = benedict.from_query_string(s, **kwargs)

• from_toml

# Try to load/decode a toml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://pypi.org/project/toml/
# A ValueError is raised in case of failure.
d = benedict.from_toml(s, **kwargs)

• from_xls

# Try to load/decode a xls file (".xls", ".xlsx", ".xlsm") from url, filepath or data-string.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# - https://openpyxl.readthedocs.io/ (for .xlsx and .xlsm files)
# - https://pypi.org/project/xlrd/ (for .xls files)
# A ValueError is raised in case of failure.
d = benedict.from_xls(s, sheet=0, columns=None, columns_row=True, **kwargs)

• from_xml

# Try to load/decode a xml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://github.com/martinblech/xmltodict
# A ValueError is raised in case of failure.
d = benedict.from_xml(s, **kwargs)

• from_yaml

# Try to load/decode a yaml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://pyyaml.org/wiki/PyYAMLDocumentation
# A ValueError is raised in case of failure.
d = benedict.from_yaml(s, **kwargs)

• to_base64

# Return the dict instance encoded in base64 format and optionally save it at the specified 'filepath'.
# It's possible to choose the subformat used under the hood:
# ('csv', json', `query-string`, 'toml', 'xml', 'yaml'), default: 'json'.
# It's possible to choose the encoding, default 'utf-8'.
# It's possible to pass decoder specific options using kwargs.
# A ValueError is raised in case of failure.
s = d.to_base64(subformat='json', encoding='utf-8', **kwargs)

• to_csv

# Return a list of dicts in the current dict encoded in csv format and optionally save it at the specified filepath.
# It's possible to specify the key of the item (list of dicts) to encode, default: 'values'.
# It's possible to specify the columns list, default: None (in this case the keys of the first item will be used).
# A ValueError is raised in case of failure.
s = d.to_csv(key='values', columns=None, columns_row=True, **kwargs)

• to_ini

# Return the dict instance encoded in ini format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/configparser.html
# A ValueError is raised in case of failure.
s = d.to_ini(**kwargs)

• to_json

# Return the dict instance encoded in json format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/json.html
# A ValueError is raised in case of failure.
s = d.to_json(**kwargs)

• to_pickle

# Return the dict instance as pickle encoded in Base64 format and optionally save it at the specified filepath.
# The pickle protocol used by default is 2.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/pickle.html
# A ValueError is raised in case of failure.
s = d.to_pickle(**kwargs)

• to_plist

# Return the dict instance encoded in p-list format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/plistlib.html
# A ValueError is raised in case of failure.
s = d.to_plist(**kwargs)

• to_query_string

# Return the dict instance as query-string and optionally save it at the specified filepath.
# A ValueError is raised in case of failure.
s = d.to_query_string(**kwargs)

• to_toml

# Return the dict instance encoded in toml format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://pypi.org/project/toml/
# A ValueError is raised in case of failure.
s = d.to_toml(**kwargs)

• to_xml

# Return the dict instance encoded in xml format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://github.com/martinblech/xmltodict
# A ValueError is raised in case of failure.
s = d.to_xml(**kwargs)

• to_yaml

# Return the dict instance encoded in yaml format.
# If filepath option is passed the output will be saved ath
# It's possible to pass encoder specific options using kwargs:
# https://pyyaml.org/wiki/PyYAMLDocumentation
# A ValueError is raised in case of failure.
s = d.to_yaml(**kwargs)

Parse methods

These methods are wrappers of the get method, they parse data trying to return it in the expected type.

• get_bool

# Get value by key or keypath trying to return it as bool.
# Values like `1`, `true`, `yes`, `on`, `ok` will be returned as `True`.
d.get_bool(key, default=False)

• get_bool_list

# Get value by key or keypath trying to return it as list of bool values.
# If separator is specified and value is a string it will be splitted.
d.get_bool_list(key, default=[], separator=',')

• get_date

# Get value by key or keypath trying to return it as date.
# If format is not specified it will be autodetected.
# If choices and value is in choices return value otherwise default.
d.get_date(key, default=None, format=None, choices=[])

• get_date_list

# Get value by key or keypath trying to return it as list of date values.
# If separator is specified and value is a string it will be splitted.
d.get_date_list(key, default=[], format=None, separator=',')

• get_datetime

# Get value by key or keypath trying to return it as datetime.
# If format is not specified it will be autodetected.
# If choices and value is in choices return value otherwise default.
d.get_datetime(key, default=None, format=None, choices=[])

• get_datetime_list

# Get value by key or keypath trying to return it as list of datetime values.
# If separator is specified and value is a string it will be splitted.
d.get_datetime_list(key, default=[], format=None, separator=',')

• get_decimal

# Get value by key or keypath trying to return it as Decimal.
# If choices and value is in choices return value otherwise default.
d.get_decimal(key, default=Decimal('0.0'), choices=[])

• get_decimal_list

# Get value by key or keypath trying to return it as list of Decimal values.
# If separator is specified and value is a string it will be splitted.
d.get_decimal_list(key, default=[], separator=',')

• get_dict

# Get value by key or keypath trying to return it as dict.
# If value is a json string it will be automatically decoded.
d.get_dict(key, default={})

• get_email

# Get email by key or keypath and return it.
# If value is blacklisted it will be automatically ignored.
# If check_blacklist is False, it will be not ignored even if blacklisted.
d.get_email(key, default='', choices=None, check_blacklist=True)

• get_float

# Get value by key or keypath trying to return it as float.
# If choices and value is in choices return value otherwise default.
d.get_float(key, default=0.0, choices=[])

• get_float_list

# Get value by key or keypath trying to return it as list of float values.
# If separator is specified and value is a string it will be splitted.
d.get_float_list(key, default=[], separator=',')

• get_int

# Get value by key or keypath trying to return it as int.
# If choices and value is in choices return value otherwise default.
d.get_int(key, default=0, choices=[])

• get_int_list

# Get value by key or keypath trying to return it as list of int values.
# If separator is specified and value is a string it will be splitted.
d.get_int_list(key, default=[], separator=',')

• get_list

# Get value by key or keypath trying to return it as list.
# If separator is specified and value is a string it will be splitted.
d.get_list(key, default=[], separator=',')

• get_list_item

# Get list by key or keypath and return value at the specified index.
# If separator is specified and list value is a string it will be splitted.
d.get_list_item(key, index=0, default=None, separator=',')

• get_phonenumber

# Get phone number by key or keypath and return a dict with different formats (e164, international, national).
# If country code is specified (alpha 2 code), it will be used to parse phone number correctly.
d.get_phonenumber(key, country_code=None, default=None)

• get_slug

# Get value by key or keypath trying to return it as slug.
# If choices and value is in choices return value otherwise default.
d.get_slug(key, default='', choices=[])

• get_slug_list

# Get value by key or keypath trying to return it as list of slug values.
# If separator is specified and value is a string it will be splitted.
d.get_slug_list(key, default=[], separator=',')

• get_str

# Get value by key or keypath trying to return it as string.
# Encoding issues will be automatically fixed.
# If choices and value is in choices return value otherwise default.
d.get_str(key, default='', choices=[])

• get_str_list

# Get value by key or keypath trying to return it as list of str values.
# If separator is specified and value is a string it will be splitted.
d.get_str_list(key, default=[], separator=',')

• get_uuid

# Get value by key or keypath trying to return it as valid uuid.
# If choices and value is in choices return value otherwise default.
d.get_uuid(key, default='', choices=[])

• get_uuid_list

# Get value by key or keypath trying to return it as list of valid uuid values.
# If separator is specified and value is a string it will be splitted.
d.get_uuid_list(key, default=[], separator=',')

Testing

# clone repository
git clone https://github.com/fabiocaccamo/python-benedict.git && cd python-benedict

# create virtualenv and activate it
python -m venv venv && . venv/bin/activate

# upgrade pip
python -m pip install --upgrade pip

# install requirements
pip install -r requirements.txt -r requirements-test.txt

# run tests using tox
tox

# or run tests using unittest
python -m unittest

License

Released under MIT License.

---

Supporting

• ⭐ Star this project on GitHub
• :octocat: Follow me on GitHub
• 💙 Follow me on Twitter
• 💰 Sponsor me on Github

See also

• python-fontbro - friendly font operations. 🧢

• python-fsutil - file-system utilities for lazy devs. 🧟‍♂️