python-benedict

python-benedict is a dict subclass with keylist/keypath/keyattr support, I/O shortcuts (base64, cli, csv, html, ini, json, pickle, plist, query-string, toml, xls, xml, yaml) and many utilities... for humans, obviously.

Features

• 100% backward-compatible, you can safely wrap existing dictionaries.
• NEW Keyattr support for get/set items using keys as attributes.
• Keylist support using list of keys as key.
• Keypath support using keypath-separator (dot syntax by default).
• Keypath list-index support (also negative) using the standard [n] suffix.
• Normalized I/O operations with most common formats: base64, cli, csv, html, ini, json, pickle, plist, query-string, toml, xls, xml, yaml.
• Multiple I/O operations backends: file-system (read/write), url (read-only), s3 (read/write).
• Many utility and parse methods to retrieve data as needed (check the API section).
• Well tested. ;)

Index

• Installation
  • Optional Requirements
• Usage
  • Basics
  • Keyattr my_dict.x.y.z
  • Keylist my_dict[["x", "y", "z"]]
  • Keypath my_dict["x.y.z"]
    • Custom keypath separator
    • Change keypath separator
    • Disable keypath functionality
    • List index support
  • I/O
  • API
    • Utility methods
    • I/O methods
    • Parse methods
• Testing
• License

Installation

If you want to install everything:

• Run pip install "python-benedict[all]"

alternatively you can install the main package:

• Run pip install python-benedict, then install only the optional requirements you need.

Optional Requirements

Here the hierarchy of possible installation targets available when running pip install "python-benedict[...]" (each target installs all its sub-targets):

• [all]
  • [io]
    • [html]
    • [toml]
    • [xls]
    • [xml]
    • [yaml]
  • [parse]
  • [s3]

Usage

Basics

benedict is a dict subclass, so it is possible to use it as a normal dictionary (you can just cast an existing dict).

from benedict import benedict

# create a new empty instance
d = benedict()

# or cast an existing dict
d = benedict(existing_dict)

# or create from data source (filepath, url or data-string) in a supported format:
# Base64, CSV, JSON, TOML, XML, YAML, query-string
d = benedict("https://localhost:8000/data.json", format="json")

# or in a Django view
params = benedict(request.GET.items())
page = params.get_int("page", 1)

Keyattr

It is possible to get/set items using keys as attributes (dotted notation).

d = benedict(keyattr_dynamic=True) # default False
d.profile.firstname = "Fabio"
d.profile.lastname = "Caccamo"
print(d) # -> { "profile":{ "firstname":"Fabio", "lastname":"Caccamo" } }

By default, if the keyattr_dynamic is not explicitly set to True, this functionality works for get/set only already existing items.

Disable keyattr functionality

You can disable the keyattr functionality passing keyattr_enabled=False option in the constructor.

d = benedict(existing_dict, keyattr_enabled=False) # default True

or using the getter/setter property.

d.keyattr_enabled = False

Dynamic keyattr functionality

You can enable the dynamic attributes access functionality passing keyattr_dynamic=True in the constructor.

d = benedict(existing_dict, keyattr_dynamic=True) # default False

or using the getter/setter property.

d.keyattr_dynamic = True

Warning - even if this feature is very useful, it has some obvious limitations: it works only for string keys that are unprotected (not starting with an _) and that don't clash with the currently supported methods names.

Keylist

Wherever a key is used, it is possible to use also a list of keys.

d = benedict()

# set values by keys list
d[["profile", "firstname"]] = "Fabio"
d[["profile", "lastname"]] = "Caccamo"
print(d) # -> { "profile":{ "firstname":"Fabio", "lastname":"Caccamo" } }
print(d["profile"]) # -> { "firstname":"Fabio", "lastname":"Caccamo" }

# check if keypath exists in dict
print([["profile", "lastname"]] in d) # -> True

# delete value by keys list
del d[["profile", "lastname"]]
print(d["profile"]) # -> { "firstname":"Fabio" }

Keypath

. is the default keypath separator.

If you cast an existing dict and its keys contain the keypath separator a ValueError will be raised.

In this case you should use a custom keypath separator or disable keypath functionality.

d = benedict()

# set values by keypath
d["profile.firstname"] = "Fabio"
d["profile.lastname"] = "Caccamo"
print(d) # -> { "profile":{ "firstname":"Fabio", "lastname":"Caccamo" } }
print(d["profile"]) # -> { "firstname":"Fabio", "lastname":"Caccamo" }

# check if keypath exists in dict
print("profile.lastname" in d) # -> True

# delete value by keypath
del d["profile.lastname"]

Custom keypath separator

You can customize the keypath separator passing the keypath_separator argument in the constructor.

If you pass an existing dict to the constructor and its keys contain the keypath separator an Exception will be raised.

d = benedict(existing_dict, keypath_separator="/")

Change keypath separator

You can change the keypath_separator at any time using the getter/setter property.

If any existing key contains the new keypath_separator an Exception will be raised.

d.keypath_separator = "/"

Disable keypath functionality

You can disable the keypath functionality passing keypath_separator=None option in the constructor.

d = benedict(existing_dict, keypath_separator=None)

or using the getter/setter property.

d.keypath_separator = None

List index support

List index are supported, keypaths can include indexes (also negative) using [n], to perform any operation very fast:

# Eg. get last location cordinates of the first result:
loc = d["results[0].locations[-1].coordinates"]
lat = loc.get_decimal("latitude")
lng = loc.get_decimal("longitude")

I/O

For simplifying I/O operations, benedict supports a variety of input/output methods with most common formats: base64, cli, csv, html, ini, json, pickle, plist, query-string, toml, xls, xml, yaml.

Input via constructor

It is possible to create a benedict instance directly from data-source (filepath, url, s3 or data string) by passing the data source and the data format (optional, default "json") in the constructor.

# filepath
d = benedict("/root/data.yml", format="yaml")

# url
d = benedict("https://localhost:8000/data.xml", format="xml")

# s3
d = benedict("s3://my-bucket/data.xml", s3_options={"aws_access_key_id": "...", "aws_secret_access_key": "..."})

# data
d = benedict('{"a": 1, "b": 2, "c": 3, "x": 7, "y": 8, "z": 9}')

Input methods

• All input methods can be accessed as class methods and are prefixed by from_* followed by the format name.
• In all input methods, the first argument can represent a source: file path, url, s3 url, or data string.

Input sources

All supported sources (file, url, s3, data) are allowed by default, but in certains situations when the input data comes from untrusted sources it may be useful to restrict the allowed sources using the sources argument:

# url
d = benedict("https://localhost:8000/data.json", sources=["url"]) # -> ok
d = benedict.from_json("https://localhost:8000/data.json", sources=["url"]) # -> ok

# s3
d = benedict("s3://my-bucket/data.json", sources=["url"]) # -> raise ValueError
d = benedict.from_json("s3://my-bucket/data.json", sources=["url"]) # -> raise ValueError

Output methods

• All output methods can be accessed as instance methods and are prefixed by to_* followed by the format name.
• In all output methods, if filepath="..." kwarg is specified, the output will be also saved at the specified filepath.

Supported formats

Here are the details of the supported formats, operations and extra options docs.

format	input	output	extra options docs
base64	✅	✅	-
cli	✅	❌	argparse
csv	✅	✅	csv
html	✅	❌	bs4 (Beautiful Soup 4)
ini	✅	✅	configparser
json	✅	✅	json
pickle	✅	✅	pickle
plist	✅	✅	plistlib
query-string	✅	✅	-
toml	✅	✅	toml
xls	✅	❌	openpyxl - xlrd
xml	✅	✅	xmltodict
yaml	✅	✅	PyYAML

API

• Utility methods

  • clean
  • clone
  • dump
  • filter
  • find
  • flatten
  • groupby
  • invert
  • items_sorted_by_keys
  • items_sorted_by_values
  • keypaths
  • match
  • merge
  • move
  • nest
  • remove
  • rename
  • search
  • standardize
  • subset
  • swap
  • traverse
  • unflatten
  • unique

• I/O methods

  • from_base64
  • from_cli
  • from_csv
  • from_ini
  • from_html
  • from_json
  • from_pickle
  • from_plist
  • from_query_string
  • from_toml
  • from_xls
  • from_xml
  • from_yaml
  • to_base64
  • to_csv
  • to_ini
  • to_json
  • to_pickle
  • to_plist
  • to_query_string
  • to_toml
  • to_xml
  • to_yaml

• Parse methods

  • get_bool
  • get_bool_list
  • get_date
  • get_date_list
  • get_datetime
  • get_datetime_list
  • get_decimal
  • get_decimal_list
  • get_dict
  • get_email
  • get_float
  • get_float_list
  • get_int
  • get_int_list
  • get_list
  • get_list_item
  • get_phonenumber
  • get_slug
  • get_slug_list
  • get_str
  • get_str_list
  • get_uuid
  • get_uuid_list

Utility methods

These methods are common utilities that will speed up your everyday work.

Utilities that accept key argument(s) also support keypath(s).

Utilities that return a dictionary always return a new benedict instance.

clean

# Clean the current dict instance removing all empty values: None, "", {}, [], ().
# If strings or collections (dict, list, set, tuple) flags are False,
# related empty values will not be deleted.
d.clean(strings=True, collections=True)

clone

# Return a clone (deepcopy) of the dict.
c = d.clone()

dump

# Return a readable representation of any dict/list.
# This method can be used both as static method or instance method.
s = benedict.dump(d.keypaths())
print(s)
# or
d = benedict()
print(d.dump())

filter

# Return a filtered dict using the given predicate function.
# Predicate function receives key, value arguments and should return a bool value.
predicate = lambda k, v: v is not None
f = d.filter(predicate)

find

# Return the first match searching for the given keys/keypaths.
# If no result found, default value is returned.
keys = ["a.b.c", "m.n.o", "x.y.z"]
f = d.find(keys, default=0)

flatten

# Return a new flattened dict using the given separator to join nested dict keys to flatten keypaths.
f = d.flatten(separator="_")

groupby

# Group a list of dicts at key by the value of the given by_key and return a new dict.
g = d.groupby("cities", by_key="country_code")

invert

# Return an inverted dict where values become keys and keys become values.
# Since multiple keys could have the same value, each value will be a list of keys.
# If flat is True each value will be a single value (use this only if values are unique).
i = d.invert(flat=False)

items_sorted_by_keys

# Return items (key/value list) sorted by keys.
# If reverse is True, the list will be reversed.
items = d.items_sorted_by_keys(reverse=False)

items_sorted_by_values

# Return items (key/value list) sorted by values.
# If reverse is True, the list will be reversed.
items = d.items_sorted_by_values(reverse=False)

keypaths

# Return a list of all keypaths in the dict.
# If indexes is True, the output will include list values indexes.
k = d.keypaths(indexes=False)

match

# Return a list of all values whose keypath matches the given pattern (a regex or string).
# If pattern is string, wildcard can be used (eg. [*] can be used to match all list indexes).
# If indexes is True, the pattern will be matched also against list values.
m = d.match(pattern, indexes=True)

merge

# Merge one or more dictionary objects into current instance (deepupdate).
# Sub-dictionaries keys will be merged together.
# If overwrite is False, existing values will not be overwritten.
# If concat is True, list values will be concatenated together.
d.merge(a, b, c, overwrite=True, concat=False)

move

# Move an item from key_src to key_dst.
# It can be used to rename a key.
# If key_dst exists, its value will be overwritten.
d.move("a", "b", overwrite=True)

nest

# Nest a list of dicts at the given key and return a new nested list
# using the specified keys to establish the correct items hierarchy.
d.nest("values", id_key="id", parent_id_key="parent_id", children_key="children")

remove

# Remove multiple keys from the dict.
# It is possible to pass a single key or more keys (as list or *args).
d.remove(["firstname", "lastname", "email"])

rename

# Rename a dict item key from "key" to "key_new".
# If key_new exists, a KeyError will be raised.
d.rename("first_name", "firstname")

search

# Search and return a list of items (dict, key, value, ) matching the given query.
r = d.search("hello", in_keys=True, in_values=True, exact=False, case_sensitive=False)

standardize

# Standardize all dict keys, e.g. "Location Latitude" -> "location_latitude".
d.standardize()

subset

# Return a dict subset for the given keys.
# It is possible to pass a single key or more keys (as list or *args).
s = d.subset(["firstname", "lastname", "email"])

swap

# Swap items values at the given keys.
d.swap("firstname", "lastname")

traverse

# Traverse a dict passing each item (dict, key, value) to the given callback function.
def f(d, key, value):
    print(f"dict: {d} - key: {key} - value: {value}")
d.traverse(f)

unflatten

# Return a new unflattened dict using the given separator to split dict keys to nested keypaths.
u = d.unflatten(separator="_")

unique

# Remove duplicated values from the dict.
d.unique()

I/O methods

These methods are available for input/output operations.

from_base64

# Try to load/decode a base64 encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to choose the subformat used under the hood:
# ('csv', 'json', 'query-string', 'toml', 'xml', 'yaml'), default: 'json'.
# It's possible to choose the encoding, default 'utf-8'.
# A ValueError is raised in case of failure.
d = benedict.from_base64(s, subformat="json", encoding="utf-8", **kwargs)

from_cli

# Load and decode data from a string of CLI arguments.
# ArgumentParser specific options can be passed using kwargs:
# https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser
# Return a new dict instance. A ValueError is raised in case of failure.
d = benedict.from_cli(s, **kwargs)

from_csv

# Try to load/decode a csv encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to specify the columns list, default: None (in this case the first row values will be used as keys).
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/csv.html
# A ValueError is raised in case of failure.
d = benedict.from_csv(s, columns=None, columns_row=True, **kwargs)

from_html

# Try to load/decode a html data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://beautiful-soup-4.readthedocs.io/
# A ValueError is raised in case of failure.
d = benedict.from_html(s, **kwargs)

from_ini

# Try to load/decode a ini encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/configparser.html
# A ValueError is raised in case of failure.
d = benedict.from_ini(s, **kwargs)

from_json

# Try to load/decode a json encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/json.html
# A ValueError is raised in case of failure.
d = benedict.from_json(s, **kwargs)

from_pickle

# Try to load/decode a pickle encoded in Base64 format and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/pickle.html
# A ValueError is raised in case of failure.
d = benedict.from_pickle(s, **kwargs)

from_plist

# Try to load/decode a p-list encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/plistlib.html
# A ValueError is raised in case of failure.
d = benedict.from_plist(s, **kwargs)

from_query_string

# Try to load/decode a query-string and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# A ValueError is raised in case of failure.
d = benedict.from_query_string(s, **kwargs)

from_toml

# Try to load/decode a toml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://pypi.org/project/toml/
# A ValueError is raised in case of failure.
d = benedict.from_toml(s, **kwargs)

from_xls

# Try to load/decode a xls file (".xls", ".xlsx", ".xlsm") from url, filepath or data-string.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# - https://openpyxl.readthedocs.io/ (for .xlsx and .xlsm files)
# - https://pypi.org/project/xlrd/ (for .xls files)
# A ValueError is raised in case of failure.
d = benedict.from_xls(s, sheet=0, columns=None, columns_row=True, **kwargs)

from_xml

# Try to load/decode a xml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://github.com/martinblech/xmltodict
# A ValueError is raised in case of failure.
d = benedict.from_xml(s, **kwargs)

from_yaml

# Try to load/decode a yaml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://pyyaml.org/wiki/PyYAMLDocumentation
# A ValueError is raised in case of failure.
d = benedict.from_yaml(s, **kwargs)

to_base64

# Return the dict instance encoded in base64 format and optionally save it at the specified 'filepath'.
# It's possible to choose the subformat used under the hood:
# ('csv', json', 'query-string', 'toml', 'xml', 'yaml'), default: 'json'.
# It's possible to choose the encoding, default 'utf-8'.
# It's possible to pass decoder specific options using kwargs.
# A ValueError is raised in case of failure.
s = d.to_base64(subformat="json", encoding="utf-8", **kwargs)

to_csv

# Return a list of dicts in the current dict encoded in csv format and optionally save it at the specified filepath.
# It's possible to specify the key of the item (list of dicts) to encode, default: 'values'.
# It's possible to specify the columns list, default: None (in this case the keys of the first item will be used).
# A ValueError is raised in case of failure.
s = d.to_csv(key="values", columns=None, columns_row=True, **kwargs)

to_ini

# Return the dict instance encoded in ini format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/configparser.html
# A ValueError is raised in case of failure.
s = d.to_ini(**kwargs)

to_json

# Return the dict instance encoded in json format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/json.html
# A ValueError is raised in case of failure.
s = d.to_json(**kwargs)

to_pickle

# Return the dict instance as pickle encoded in Base64 format and optionally save it at the specified filepath.
# The pickle protocol used by default is 2.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/pickle.html
# A ValueError is raised in case of failure.
s = d.to_pickle(**kwargs)

to_plist

# Return the dict instance encoded in p-list format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/plistlib.html
# A ValueError is raised in case of failure.
s = d.to_plist(**kwargs)

to_query_string

# Return the dict instance as query-string and optionally save it at the specified filepath.
# A ValueError is raised in case of failure.
s = d.to_query_string(**kwargs)

to_toml

# Return the dict instance encoded in toml format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://pypi.org/project/toml/
# A ValueError is raised in case of failure.
s = d.to_toml(**kwargs)

to_xml

# Return the dict instance encoded in xml format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://github.com/martinblech/xmltodict
# A ValueError is raised in case of failure.
s = d.to_xml(**kwargs)

to_yaml

# Return the dict instance encoded in yaml format.
# If filepath option is passed the output will be saved ath
# It's possible to pass encoder specific options using kwargs:
# https://pyyaml.org/wiki/PyYAMLDocumentation
# A ValueError is raised in case of failure.
s = d.to_yaml(**kwargs)

Parse methods

These methods are wrappers of the get method, they parse data trying to return it in the expected type.

get_bool

# Get value by key or keypath trying to return it as bool.
# Values like `1`, `true`, `yes`, `on`, `ok` will be returned as `True`.
d.get_bool(key, default=False)

get_bool_list

# Get value by key or keypath trying to return it as list of bool values.
# If separator is specified and value is a string it will be splitted.
d.get_bool_list(key, default=[], separator=",")

get_date

# Get value by key or keypath trying to return it as date.
# If format is not specified it will be autodetected.
# If choices and value is in choices return value otherwise default.
d.get_date(key, default=None, format=None, choices=[])

get_date_list

# Get value by key or keypath trying to return it as list of date values.
# If separator is specified and value is a string it will be splitted.
d.get_date_list(key, default=[], format=None, separator=",")

get_datetime

# Get value by key or keypath trying to return it as datetime.
# If format is not specified it will be autodetected.
# If choices and value is in choices return value otherwise default.
d.get_datetime(key, default=None, format=None, choices=[])

get_datetime_list

# Get value by key or keypath trying to return it as list of datetime values.
# If separator is specified and value is a string it will be splitted.
d.get_datetime_list(key, default=[], format=None, separator=",")

get_decimal

# Get value by key or keypath trying to return it as Decimal.
# If choices and value is in choices return value otherwise default.
d.get_decimal(key, default=Decimal("0.0"), choices=[])

get_decimal_list

# Get value by key or keypath trying to return it as list of Decimal values.
# If separator is specified and value is a string it will be splitted.
d.get_decimal_list(key, default=[], separator=",")

get_dict

# Get value by key or keypath trying to return it as dict.
# If value is a json string it will be automatically decoded.
d.get_dict(key, default={})

get_email

# Get email by key or keypath and return it.
# If value is blacklisted it will be automatically ignored.
# If check_blacklist is False, it will be not ignored even if blacklisted.
d.get_email(key, default="", choices=None, check_blacklist=True)

get_float

# Get value by key or keypath trying to return it as float.
# If choices and value is in choices return value otherwise default.
d.get_float(key, default=0.0, choices=[])

get_float_list

# Get value by key or keypath trying to return it as list of float values.
# If separator is specified and value is a string it will be splitted.
d.get_float_list(key, default=[], separator=",")

get_int

# Get value by key or keypath trying to return it as int.
# If choices and value is in choices return value otherwise default.
d.get_int(key, default=0, choices=[])

get_int_list

# Get value by key or keypath trying to return it as list of int values.
# If separator is specified and value is a string it will be splitted.
d.get_int_list(key, default=[], separator=",")

get_list

# Get value by key or keypath trying to return it as list.
# If separator is specified and value is a string it will be splitted.
d.get_list(key, default=[], separator=",")

get_list_item

# Get list by key or keypath and return value at the specified index.
# If separator is specified and list value is a string it will be splitted.
d.get_list_item(key, index=0, default=None, separator=",")

get_phonenumber

# Get phone number by key or keypath and return a dict with different formats (e164, international, national).
# If country code is specified (alpha 2 code), it will be used to parse phone number correctly.
d.get_phonenumber(key, country_code=None, default=None)

get_slug

# Get value by key or keypath trying to return it as slug.
# If choices and value is in choices return value otherwise default.
d.get_slug(key, default="", choices=[])

get_slug_list

# Get value by key or keypath trying to return it as list of slug values.
# If separator is specified and value is a string it will be splitted.
d.get_slug_list(key, default=[], separator=",")

get_str

# Get value by key or keypath trying to return it as string.
# Encoding issues will be automatically fixed.
# If choices and value is in choices return value otherwise default.
d.get_str(key, default="", choices=[])

get_str_list

# Get value by key or keypath trying to return it as list of str values.
# If separator is specified and value is a string it will be splitted.
d.get_str_list(key, default=[], separator=",")

get_uuid

# Get value by key or keypath trying to return it as valid uuid.
# If choices and value is in choices return value otherwise default.
d.get_uuid(key, default="", choices=[])

get_uuid_list

# Get value by key or keypath trying to return it as list of valid uuid values.
# If separator is specified and value is a string it will be splitted.
d.get_uuid_list(key, default=[], separator=",")

Testing

# clone repository
git clone https://github.com/fabiocaccamo/python-benedict.git && cd python-benedict

# create virtualenv and activate it
python -m venv venv && . venv/bin/activate

# upgrade pip
python -m pip install --upgrade pip

# install requirements
pip install -r requirements.txt -r requirements-test.txt

# install pre-commit to run formatters and linters
pre-commit install --install-hooks

# run tests using tox
tox

# or run tests using unittest
python -m unittest

License

Released under MIT License.

---

Supporting

• ⭐ Star this project on GitHub
• :octocat: Follow me on GitHub
• 💙 Follow me on Twitter
• 💰 Sponsor me on Github

See also

• python-fontbro - 🧢 friendly font operations on top of fontTools.

• python-fsutil - 🧟‍♂️ high-level file-system operations for lazy devs.