Recently I finished two courses on data import in Python at DataCamp and I was really surprised of the amount of sources that can be used to get data. Here I would like to summarize all those methods and at the same time keen my knowledge. Also I think for others it will be useful as well. So, let’s begin.<\/p>\n\n\n\n
There is huge variety of files that can be used as the data source:\u00a0<\/p>\n\n\n\n
Flat files – txt, csv – are easy and there are few ways to import them using numpy or pandas.<\/p>\n\n\n\n
numpy.recfromcsv<\/em> – Load ASCII data stored in a comma-separated file. The returned array is a record array (if usemask=False, see recarray) or a masked record array (if usemask=True, see ma.mrecords.MaskedRecords).<\/p>\n\n\n\n numpy.loadtxt<\/em> –\u00a0This function aims to be a fast reader for simply formatted files. The genfromtxt<\/em>\u00a0function provides more sophisticated handling of, e.g., lines with missing values.<\/p>\n\n\n\n numpy.genfromtxt<\/em> – Load data from a text file, with missing values handled as specified. Much more sophisticated function that has a lot of parameters to control your import.<\/p>\n\n\n\n With pandas it’s even easier – one line and you have your file in a DataFrame ready.\u00a0 Also supports optionally iterating or breaking of the file into chunks. <\/p>\n\n\n\n What the hell is pickle? It is used for serializing and de-serializing a Python object structure. Any object in python can be pickled so that it can be saved on disk. What pickle does is that\u00a0it \u201cserialises\u201d the object first before writing it to file. Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script. The code below will print a dictionary that was created somewhere and stored in the file – pretty cool, isn’t it?<\/p>\n\n\n\n With pandas.read_excel<\/em> that reads an Excel table into a pandas DataFrame and has a lot of customization importing data have never been more pleasant (sounds like TV commercial :D). But it is really true – documentation for this function is clear and you are actually able to do whatever you want with that Excel file.<\/p>\n\n\n\n SAS\u00a0stands for Statistical Analysis Software.\u00a0A SAS data set contains data values that are organized as a table of observations (rows) and variables (columns). To open this type of files and import data from it the code sample below will help:<\/p>\n\n\n\n Stata is a powerful statistical software that enables users to analyze, manage, and produce graphical visualizations of data. It is primarily used by researchers in the fields of economics, biomedicine, and political science to examine data patterns. Data stored in .dta files and the best way to import it is pandas.read_stata<\/em><\/p>\n\n\n\n Hierarchical Data Format (HDF) is a set of file formats (HDF4, HDF5) designed to store and organize large amounts of data.\u00a0HDF5 is a unique technology suite that makes possible the management of extremely large and complex data collections.\u00a0\u00a0HDF5 simplifies the file structure to include only two major types of object:<\/p>\n\n\n\n This results in a truly hierarchical, filesystem-like data format. In fact, resources in an HDF5 file are even accessed using the POSIX-like syntax \/path\/to\/resource<\/em>. Metadata is stored in the form of user-defined, named attributes attached to groups and datasets. More complex storage APIs representing images and tables can then be built up using datasets, groups and attributes.<\/p>\n\n\n\n To import HDF5 file we’ll need h5py<\/em> library. Code sample below made everything easier and totally understandable for me.<\/p>\n\n\n\n A lot of people work with MATLAB and store data in .mat files. So what those files are? These files contain list of variables and objects assigned to them in MATLAB workspace. It’s not surprising that it is imported in Python as dictionary in which keys are MATLAB variables and values – objects assigned to these variables. To write and read MATLAB files scipy.io<\/em> package is used.<\/p>\n\n\n\n Using drivers to connect to a database we can grab data directly from there. Usually it means: create connection, connect, run the query, fetch the data, close connection. It is possible to do it step by step, but in pandas we have an awesome function that does it for us, so why bother ourselves? It only requires a connection that can be created with sqlalchemy<\/em> package. Below is the example on connecting to sqlite database engine and getting data from it:<\/p>\n\n\n\n A separate article should be written on this, but I will highlight few things to at least know where to start. First of all, if we have a direct url to a file we can just use standard pandas.read_csv\/pandas.read_excel<\/em> functions specifying it in the parameter “file=”<\/p>\n\n\n\n Apart from this, to get data from the web we need to use HTTP protocol and especially GET method (there are a lot of them, but for the import we don’t need more). And package requests<\/em> does an incredible job doing this. To access a text from the response received by requests.get<\/em> we just need to use method .text.<\/p>\n\n\n\n r.text will give us a web-page with all html-tags on it – not very useful, isn’t it? But here is where the fun begins. We have a BeautifulSoup<\/em> package that can parse that HTML and extract the information we need, in this case all hyperlinks (continuing previous example):<\/p>\n\n\n\n In computer programming, an application programming interface (API) is a set of subroutine definitions, communication protocols, and tools for building software. In general terms, it is a set of clearly defined methods of communication between various components. There are a lot of different APIs and the first thing that has to be done is documentation checked, but the truth is almost all APIs return data in JSON format. And we have to be able to catch that result. And again package requests<\/em> will help us with it. (we have to send HTTP GET request to get data from API).<\/p>\n\n\n\n At this point my little summary gets to its end. Hopefully it will be useful not only for me.<\/p>\n","protected":false},"excerpt":{"rendered":" Recently I finished two courses on data import in Python at DataCamp and I was really surprised of the amount…<\/p>\n","protected":false},"author":1,"featured_media":508,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,5,7],"tags":[],"translation":{"provider":"WPGlobus","version":"3.0.0","language":"es","enabled_languages":["gb","es","uk"],"languages":{"gb":{"title":true,"content":true,"excerpt":false},"es":{"title":false,"content":false,"excerpt":false},"uk":{"title":false,"content":false,"excerpt":false}}},"_links":{"self":[{"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/posts\/499"}],"collection":[{"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/comments?post=499"}],"version-history":[{"count":5,"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/posts\/499\/revisions"}],"predecessor-version":[{"id":507,"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/posts\/499\/revisions\/507"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/media\/508"}],"wp:attachment":[{"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/media?parent=499"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/categories?post=499"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sergilehkyi.com\/es\/wp-json\/wp\/v2\/tags?post=499"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}data = np.recfromcsv(file)<\/pre>\n\n\n\n
data = np.loadtxt('file.csv', delimiter=',', skiprows=1, usecols=[0,2])\n<\/pre>\n\n\n\ndata = np.genfromtxt('titanic.csv', delimiter=',', names=True, dtype=None)\n<\/pre>\n\n\n\ndata = pd.read_csv(file, nrows=5, header=None, sep='\\t', comment='#',\nna_values='Nothing')<\/pre>\n\n\n\n
Pickle<\/h4>\n\n\n\n
import pickle
Excel<\/h4>\n\n\n\n
df = pd.read_excel('file.xlsx', sheet_name='sheet1')<\/pre>\n\n\n\nSAS and Stata<\/h4>\n\n\n\n
from sas7bdat import SAS7BDAT
df = pd.read_stata('file.dta')<\/pre>\n\n\n\nHDF5<\/h4>\n\n\n\n
import h5py
MATLAB<\/h4>\n\n\n\n
import scipy.io\nmat = scipy.io.loadmat('some_project.mat')Relational Databases<\/h4>\n\n\n\n
from sqlalchemy import create_engine\nimport pandas as pd
Data from Web<\/h4>\n\n\n\n
df = pd.read_csv('https:\/\/www.example.com\/data.csv', sep=';')<\/pre>\n\n\n\nimport requests\nr = requests.get('http:\/\/www.example.com\/some_html_page')\nprint(r.text)<\/pre>\n\n\n\nfrom bs4 import BeautifulSoup
API<\/h4>\n\n\n\n
import requests\nr = requests.get('https:\/\/www.example.com\/some_endpoint')