Pandas is cool and awesome, but what if you just need a small transformation of data, a punctual change and you don’t want to install heavy things on your server? Or maybe you just launching Lambdas (or any other serverless functions) that have limited capacities in resources? In such scenarios, it is much better to stick to Python’s standard library and not load it with unnecessary stuff.
Also, knowing a few tricks will make your code much more readable and help you avoid redundancy. So, in this article, I will show you a few data manipulation tricks using only a standard library that might save you time and resources.
We will work with the following
.csv file as it is the most common format – a dummy shopping cart in a clothing store (you can just copy-paste it):
name,color,category,price,quantity t-shirt,black,top,20,1 pants,white,bottom,50,1 blazer,yellow,top,100,1 t-shirt,red,top,15,2 t-shirt,orange,top,25,1 sneakers,white,footwear,100,1 bracelet,green,accessories,5,3
1. Function enumerate()
It is not about data manipulation yet, but just a cool thing to consider. The function works like a normal iterator, but it just adds an index to every iteration. It comes in handy especially when reading from files as it allows to track the line numbers. Try this out:
In the end, those 2 lines do not make a huge difference in performance, but it looks more aesthetic and you avoid loose variables here and there.
2. Reading .csv files without pandas
Reading .csv files with standard file reader in Python might get complicated in some cases as it will return the content of the file as the string and we will need to add additional steps to handle that huge string. For these cases it is quite convenient to use module
csv. This reader gives you option to select a delimiter, quotechar, escapechar and few other options to make .csv import much easier. Try this:
And you will get the following output:
One thing to take into account from this output is that every value from a column will be considered as string, so if you want to perform some operations on those values you will have to adjust the types.
For the data manipulations part, we will use a module
collections. From the documentation: “This module implements specialized container datatypes providing alternatives to Python’s general-purpose built-in containers,
If you have a dataset with clothing items and want to find the total price of t-shirts or count how many different t-shirts you have there, how would you do that? A class
Counter() will help.
Check out the snippet below and see the difference. 1st version looks much better than the 3rd one, even though both are valid and do the same thing.
Counter() has a few handy methods like
total() available in Python 3.10 only).
>>> total_clothes.most_common(3) [('jacket', 100), ('t-shirt', 55), ('pants', 50)] >>> total_clothes.total() 220
Another cool thing that can be done using
Counter() is finding the most common words in a text or the most common character in a word. Let’s start with an easy one:
>>> Counter('abracadabra').most_common(3) # that's an example from Python docs actually [('a', 5), ('b', 2), ('r', 2)]
Now let’s use a common dataset for words count – Shakespear’s ‘King Lear’. This dataset will be in my repo for articles, folder
/data. Check the following little program:
And here we have all our words counted.
What we have done here? We read the file line by line, removed all the punctuation from each line (about this later), split that line into words, created a
Counter() for the words in each line and in the end added a counter per line to a total counter. Also we can check how many words in total are there:
>>> count_words.total() # only if you have Python 3.10 27762
Obviously, NLP libraries will tackle this more effectively, but hey!, it is pretty awesome to know you can do that with standard library and just few lines of code.
3.1.1. Removing all punctuation from the string
This one I have found while writing the program to count words XD. Honestly. I was really looking for a regex pattern to apply for this transformation, but apparently standard library and
string module already has quite a simple solution.
line = line.translate(str.maketrans('', '', string.punctuation))
What do we do here?
str.translate()– returns a copy of a given string in which each character has been replaced using mapping (translation) table
str.maketrans()– creates a translation table, you can read more in detail how exactly it works in Python docs, but for this particular example we have to know that the third parameter is always mapped to
None, or, for those like me, that need plain language to understand things – it removes any character specified in the third argument.
string.punctuation– predefined list of all punctuation characters. You can check that list in docs or simply by running:
So basically we get a new line, but without punctuation and without regex. Now let’s get back to
Now, if you don’t want to add up different values for same keys but just map it to see what is going on there –
defaultdict will take care of it:
And again as you can see, with regular dict that won’t work. Or you will have to write that ugly workaround and check if a key exists.
The first argument of the defaultdict,
default_factory, sets up a default value for each key. In our first example it was a list. But if we pass an
int there, we will be able to use it as a counter too. Try the following:
But the usage of the
Counter() is much wider, so bear this in mind. For example, with
defaultdicts you cannot perform the sum, like we did with the counters when counting words.
Imagine you have to keep a history of N last elements or N last lines of the file,
deque is a perfect solution for this. Now you can run window functions without turning into complex SQL queries. Rather ugly example, but still a food for thought. Imagine you have a dataset with the results of a given football team and you want to calculate their form in recent matches. Dummy dataset is available in
data folder. Check this out:
And the result:
Again, this one lacks some tuning, but I hope you get a general idea.
We’ve got so used to all these fancy packages that we forget the standard library completely. And standard library still got some aces in its sleeve. So the next time you need to perform a quick data manipulation with 0 dependencies on external packages try some of those tips.
Hope you find it useful! Cheers!