Pandas and the python os module use case — appending source file information from CSV’s.

In my day job, I use Python to automate several redundant tasks. Between the Pandas library and the CSV module, there is always something available for me to reach for. I typically process several different CSV’s each day with a planned final destination in an SQL database. While contemplating the schema design, I determined it would be best to store the actual source file information from which the data is derived, using the source files’ name and appending it to the end of each row. How did Python help me accomplish this? Continue reading to find out…

OS and Python used:

Self-Promotion:

If you enjoy the content written here, by all means, share this blog and your favorite post(s) with others who may benefit from or like it as well. Since coffee is my favorite drink, you can even buy me one if you would like!

While I won’t necessarily share any of the GIS survey data I process (for proprietary reasons), the concept remains the same and should work across any CSV file.

To get started, I’ll import pandas along with the os module, to take advantage of its operating system and file/folder functionalities:

>>> import pandas as pd
>>> import os