Python is a powerful language for data preparation, but there are some common mistakes or pitfalls folks may encounter. In this blog post, I’ll discuss five of the most common issues folks encounter when using Python for data preparation.
1. Considering missing values (`NaN`) as false.
False, None, and 0 (of any numeric type) all evaluate to False.
This set of objects and values are known as “falsy” and will evaluate to false. NaN or missing values are not falsy and therefore will not evaluate to false. This can cause much confusion and unexpected behavior by many operations.
2. Attempting to compare missing values
It seems simple enough that
NaN == NaN will return true. Both values “look” the same.
However, as it’s impossible to know if the two missing values are the same, this operation will always return false.
3. Thinking that all() only returns true when all elements are true.
all() method returns true if all elements of the iterable are true (or if the iterable is empty).
Don’t think of it as “Return true if all the elements of the iterable are true,” but instead “Return true if there are no false elements in the iterable.”
When the iterable is empty, there can be no false elements within it, meaning
all() evaluates to True.
Your keys to a better career
Get started with ACG and transform your career with courses and real hands-on labs in AWS, Microsoft Azure, Google Cloud, and beyond.
4. Converting to bool values
Pandas follows the numpy convention of raising an error when you try to convert something to a bool. This happens in an if or when using the Boolean operations, and, or, or not.
It is not clear what the result should be. Should it be True because it is not zero-length? False because there are False values?
It is unclear, so instead, Pandas raises a
ValueError: The truth value of a Series is ambiguous.
Use a.empty, a.bool() a.item(),a.any() or a.all().
5. Understanding the results of the isin() operation.
isin() operation returns a Boolean series showing whether each element in the Series is exactly contained in the passed sequence of values.
s = pd.Series(['dog', 'cat', 'fish']) >>> s.isin(['bird']) 0 False 1 False 2 False dtype: bool
Note that ‘bird’ does not exist in the series.
>>> s.isin(['bird', 'cat']) 0 False 1 True 2 False dtype: bool
Note ‘cat’ does exist in the 2nd value of the series.
Learn more aboutusing Python for data preparation
Python is a powerful language, but confusion can arise around missing and boolean values. Keep in mind that missing values are considered false and they cannot be compared.
When using the
all() method, remember that it returns true when there are no false values in the iterable. If all values are missing, like in the case of an empty array,
all() also returns true as missing values are not considered false.
If you receive a
ValueError when attempting to convert to bool values, be sure to take the helpful advice and use one of the suggested methods.
Looking to learn more about using Python for data preparation? Check out our Data Preparation (Import and Cleaning) for Python course.