Language
- PEP (Python Enhancement Proposals)
PEP | Title | Remarks |
---|---|---|
PEP 8 | Style Guide for Python Code | |
PEP 3119 | Introducing Abstract Base Classes |
- PyPI - Python Package Index : a repository of software for the Python programming language
- PyPI Tutorial
- Python Packaging User Guide
References
- Best Python tutorials & best Python books
- Learning with Python 3 (October 2012)
- Python Tips : Intermediate Python
- Awesome Python : A curated list of awesome Python frameworks, libraries, software and resources.
- Automate the Boring Stuff with Python
- The Big Book of Small Python Projects
- Beyond the Basic Stuff with Python
Python 3 API
Module | Description | Sources | Remarks |
---|---|---|---|
collections |
Container datatypes | Lib/collections/__init__.py | |
collections.abc |
Abstract base classes for containers | Lib/_collections_abc.py | |
functools |
Higher-order functions and operations on callable objects | Lib/functools.py | reduce() |
statistics
|
Provides functions for calculating mathematical statistics of numeric (Real-valued) data. | Lib/statistics.py | |
random
|
Implements pseudo-random number generators for various distributions. | Lib/random.py | randrange(), choice(), choices() |
datetime
|
Supplies classes for manipulating dates and times. | Lib/datetime.py | |
time
|
Provides various time-related functions. | ||
calendar
|
Allows you to output calendars like the Unix cal program, and provides additional useful functions related to the calendar. | Lib/calendar.py | |
os.path
|
Implements some useful functions on pathnames. | Lib/posixpath.py, Lib/ntpath.py | |
pathlib
|
Offers classes representing filesystem paths with semantics appropriate for different operating systems. | Lib/pathlib.py | since v3.4 |
urllib
|
Collects several modules for working with URLs. | Lib/urllib/ | urllib.request, urllib.error, urllib.parse, urllib.robotparser |
urllib.request
|
Defines functions and classes which help in opening URLs (mostly HTTP) in a complex world — basic and digest authentication, redirections, cookies and more. | Lib/urllib/request.py | |
zipfile
|
Provides tools to create, read, write, append, and list a ZIP file. | Lib/zipfile.py | .ZIP File Format Spec |
unittest
|
supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework. | Lib/unittest/__init__.py |
- W3Schools / Python math Module
- W3Schools / Python statistics Module
- W3Schools / Python random Module
- W3Schools / Python requests Module
Built-in Functions
Built-in Fn. | Description | Remarks |
---|---|---|
len(s)
|
Return the length (the number of items) of an object. | |
hex(x)
|
Convert an integer number to a lowercase hexadecimal string prefixed with “0x”. | int() |
int(x, base)
|
Return an integer object constructed from a number or string x | hex() |
enumerate(iterable, start=0)
|
Return an enumerate object. | |
map(function, iterable, ...)
|
Return an iterator that applies function to every item of iterable, yielding the results. | |
filter(function, iterable)
|
Construct an iterator from those elements of iterable for which function returns true. | |
input()
|
Reads a line from input, converts it to a string (stripping a trailing newline), and returns that. | |
type(object)
|
Return the type of an object. | Reflection |
dir([object])
|
Without arguments, return the list of names in the current local scope. With an argument, attempt to return a list of valid attributes for that object. | Reflection |
Built-in Types
Category | Types | Remarks |
---|---|---|
Numeric Types | int, float, complex | |
Sequence Types | list, tuple, range | |
Text Sequence Type | str | |
Binary Sequence Types | bytes, bytearray, memoryview | |
Set Types | set, frozenset | |
Mapping Types | dict |
Type | Description | Literal | Remarks |
---|---|---|---|
list | ordered mutable collection | ['cat', 'dog', 'snake', 'fish', 'elephant'] | |
tuple | ordered immutable collection | ('Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun') | |
dict | collection of key-value pairs | {'dog': 'bark', 'cat': 'meow', 'snake': 'hiss'} | map, associative array |
set | unordered mutable collection | {1, 2, 3, 5, 7, 11, 13} |
Data Type | Reference | W3Schools Reference | W3School Tutorial | Remarks |
---|---|---|---|---|
str | String Method | Python String Methods | Python Strings | |
list | More on Lists | Python List/Array Methods | Python Lists | |
set | Set Types — set, frozenset | Python Set Methods | ||
dict | Mapping Types — dict | Python Dictionary Methods |
collections Module
Class | Description | Methods | Remarks |
---|---|---|---|
Iterator | __next__, __iter__ |
- Understanding slice notation (Feb 3 '09)
- Map, Filter and Reduce : three functions which facilitate a functional approach to programming
unittest Module
Method | Checks | Remarks |
---|---|---|
assertEqual(a, b) |
a == b |
|
assertNotEqual(a, b) |
a != b |
|
assertIs(a, b) |
a is b |
|
assertIsNot(a, b) |
a is not b |
|
assertRaise(ex, func, args, kwds) |
Readings
Grammar
Datatypes
- Python String Formatting Best Practices
- Python 3's f-Strings
f"Hello, {name}. You are {age}."
- Multiline f-Strings
- How to get the epoch timestamp in Python? (Oct 24, 2020)
int(time.time())
Container Types
abc
– Abstract Base Classes¶- How do I get list of methods in a Python class? (Dec 15 '09) :
dir()
- List Comprehensions
- Python : How to Check if an item exists in list ? (February 17, 2018)
list.count(item)
- How to loop with indexes in Python (APR 25TH, 2016)
presidents = ["Washington", "Adams", "Jefferson", "Madison", "Monroe", "Adams", "Jackson"]
for num, name in enumerate(presidents, start=1):
- How to Iterate Through a Dictionary in Python
for k in scores
for k, v in scores.items()
for k in scores.keys()
for v in scores.values()
Map, Filter, Reduce
- Python most powerful functions: map(), filter(), and reduce() in 5 minutes (July 16, 2021)
- Map, Filter and Reduce
# sum of odd number from 0 to 100
sum = reduce(lambda p, c: p + c if c % 2 == 1 else p, range(100))
# same summation using list comprehensions
sum2 = sum([a for a in range(100) if a % 2 == 1])
Data Analysis
misc
- Will Python Overtake Java? (May. 01, 17)
- Logging in Python
- Python provides a logging system as a part of its standard library, so you can quickly add logging to your application.
- An elegant way to run periodic tasks in python (Oct 30, 2018)
- Using
time.sleep
- Using
threading.Timer
- Using
threading.Event
- Using
Frameworks
Django
References
- Django 3.1 Documentation
- Writing your first Django app, part 1
- CLI (django-admin, manage.py)
Category | Command | Description | Remarks |
---|---|---|---|
Setup | startproject | ||
startapp | |||
Migration | makemigrations | Creates new migrations based on the changes detected to your models. | |
sqlmigrate | Prints the SQL for the named migration. | ||
migrate | Synchronizes the database state with the current set of models and migrations. | ||
Run | runserver | Starts a lightweight development Web server on the local machine. |
Readings
Jinja2
- http://jinja.pocoo.org/
- Desc. : a full featured template engine for Python
- License :
- Sources : https://github.com/pallets/jinja
- Readings
pytest
- http://docs.pytest.org/
- Desc. : a mature full-featured Python testing tool that helps you write better programs.
- License :
Libraries
Math
NumPy
- https://numpy.org/
- Desc. : The fundamental package for scientific computing with Python
- Sources :
References
Class | Description | Remarks |
---|---|---|
ndarray | a (usually fixed-size) multidimensional container of items of the same type and size. |
ndarray.ndim # int, number of array dimensions
ndarray.shape # tuple, tuple of array dimensions
ndarray.dtype # dtype object, data-type of the array’s elements
Category | Routine | Description | Remarks |
---|---|---|---|
Creation | numpy.array()
|
Create an array. | |
numpy.eye()
|
Return a 2-D array with ones on the diagonal and zeros elsewhere. | ||
Manipulation | numpy.reshape()
|
Gives a new shape to an array without changing its data | |
Statistics | numpy.mean()
|
Compute the arithmetic mean along the specified axis. | |
numpy.median()
|
Compute the median along the specified axis. |
SymPy
- https://www.sympy.org/
- Desc. : a Python library for symbolic mathematics.
- License : BSD
- Written in : Python
- Sources : https://github.com/sympy/sympy
SciPy
- https://www.scipy.org/
- Desc. : a Python-based ecosystem of open-source software for mathematics, science, and engineering.
- Modules
Module | Title | Description | Remarks |
---|---|---|---|
scipy.stats | Statistical functions | contains a large number of probability distributions as well as a growing library of statistical functions. |
Data Analysis
pandas
- https://pandas.pydata.org/
- Desc. : a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.
General Functions
- pandas general functions (pandas.funct)
Function | Description | Remarks |
---|---|---|
pandas.read_csv()
|
Read a comma-separated values (csv) file into DataFrame. | |
pandas.to_datetime()
|
Convert argument to datetime. | |
pandas.crosstab()
|
Compute a simple cross tabulation of two (or more) factors. | |
pandas.merge()
|
Merge DataFrame or named Series objects with a database-style join. | inner/left outer/right outer/full outer join |
pandas.get_dummies()
|
Convert categorical variable into dummy/indicator variables. |
pandas.crosstab()
Pandas | SQL | Remarks |
---|---|---|
crosstab(index=drink["Attend"], columns="count")
|
select ATTEND, count(*) from DRINK group by ATTEND |
|
crosstab(index=drink["Attend"], columns=drink["Name"])
|
select ATTNED, NAME, count(*) from DRINK group by ATTEND, NAME |
Modules
API | Source | Description | Remarks |
---|---|---|---|
Series
|
class Series(base.IndexOpsMixin, NDFrame)
|
One-dimensional ndarray with axis labels (including time series) |
|
DataFrame
|
class DataFrame(NDFrame, OpsMixin)
|
Two-dimensional, size-mutable, potentially heterogeneous tabular data | |
GroupBy
|
class GroupBy(BaseGroupBy[NDFrameT])
|
Objects are returned by groupby calls |
|
SeriesGroupBy |
class SeriesGroupBy(GroupBy[Series])
|
||
DataFrameGroupBy |
class DataFrameGroupBy(GroupBy[DataFrame])
|
pandas.DataFrame
Category | Property | Description | Remarks |
---|---|---|---|
Scheme | pandas.DataFrame.shape |
Return a tuple representing the dimensionality of the DataFrame. | |
pandas.DataFrame.columns |
The column labels of the DataFrame. | ||
pandas.DataFrame.dtypes |
Return the dtypes in the DataFrame. | ||
pandas.DataFrame.info() |
Prints information about a DataFrame including the index dtype and columns, non-null values and memory usage. | ||
Query | pandas.DataFrame.head() |
Return the first n rows. | return DataFrame object |
pandas.DataFrame.tail() |
Return the last n rows. | return DataFrame object | |
pandas.DataFrame.astype()
|
Cast a pandas object to a specified dtype. | ||
Filter | pandas.DataFrame.loc()
|
Access a group of rows and columns by label(s) or a boolean array. | foo.loc[0:10, ['bar', 'baz']] . return DataFrame , Series , or single value.
|
pandas.DataFrame.iloc()
|
Purely integer-location based indexing for selection by position. | foo.iloc[0:11, 3:8] , foo.iloc[[0, 3, 5], [1, 2, 5]]
| |
pandas.DataFrame.at()
|
Access a single value for a row/column label pair. | ||
Manipulate | pandas.DataFrame.rename()
|
Alter axes labels. | |
pandas.DataFrame.sort_values() |
Sort by the values along either axis | ||
pandas.DataFrame.drop()
|
Remove rows or columns by specifying label names and corresponding axis, or by specifying directly index or column names. | ||
pandas.DataFrame.copy()
|
Make a copy of this object’s indices and data. | ||
pandas.DataFrame.reset_index()
|
Reset the index, or a level of it. | inplace , drop
| |
Cleaning | pandas.DataFrame.isna()
|
Detect missing values. | pandas.DataFrame.isnull() ]
|
pandas.DataFrame.dropna()
|
Remove missing values. | ||
pandas.DataFrame.fillna()
|
Fill NA/NaN values using the specified method. | method
| |
Aggregation | pandas.DataFrame.groupby()
|
Group DataFrame using a mapper or by a Series of columns. | as_index , axis
|
pandas.DataFrame.any()
|
Return whether any element is True, potentially over an axis. | ||
Statisttics | pandas.DataFrame.sum()
|
Return the sum of the values over the requested axis. | |
pandas.DataFrame.mean()
|
Return the mean of the values over the requested axis. | inplace
| |
pandas.DataFrame.std()
|
Return sample standard deviation over requested axis. | ||
pandas.DataFrame.describe()
|
Generate descriptive statistics. |
pandas.Series
Property | Description | Remarks |
---|---|---|
pandas.Series.unique()
|
Return unique values of Series object. | |
pandas.Series.value_counts()
|
Return a Series containing counts of unique values. | |
pandas.Series.sum()
|
Return the sum of the values over the requested axis. | |
pandas.Series.mean() |
Return the mean of the values over the requested axis. | |
pandas.Series.max() |
Return the maximum of the values over the requested axis. | |
pandas.Series.rolling()
|
Provide rolling window calculations | |
pandas.Series.shift()
|
Shift index by desired number of periods with an optional time |
Readings
- Speed up Dataframe Operations using Map, Filter, and Reduce (Jan 6, 2021)
- Create a Pandas Dataframe by appending one row at a time (May 23, 2012)
df.loc[len(df)] = [ ... ]
pandas datareader
- https://github.com/pydata/pandas-datareader
- Desc. : Up to date remote data access for pandas
tsfresh
- https://github.com/blue-yonder/tsfresh
- Desc. : Provides systematic time-series feature extraction by combining established algorithms from statistics, time-series analysis, signal processing, and nonlinear dynamics with a robust feature selection algorithm.
Cesium
- https://cesium-ml.org/
- Desc. : an end-to-end machine learning platform for time-series, from calculation of features to model-building to predictions.
Visualization
Matplotlib
- https://matplotlib.org/
- Desc. : a comprehensive library for creating static, animated, and interactive visualizations in Python.
Module | Function | Description | Remarks |
---|---|---|---|
matplotlib.pyplot | provides a MATLAB-like way of plotting. | ||
scatter() | A scatter plot of y vs. x with varying marker size and/or color. | ||
show() | Display all open figures. |
Seaborn
- https://seaborn.pydata.org/
- Desc. : a Python data visualization library based on matplotlib.
Tools
pip
- https://pip.pypa.io/
- Desc. : a package management system used to install and manage software packages written in Python.
Readings
Commands
Command | Description | Remarks |
---|---|---|
pip help command |
||
pip list -vvv
|
List installed packages, including editables. | No name filtering. Use grep pipe.
|
pip show
|
Show information about one or more installed packages. | |
pip search [options] <query>
|
Search for PyPI packages whose name or summary contains <query>. | |
pip install --upgrade [options] <package>
|
Upgrade all specified packages to the newest available version. |
pyenv
- https://github.com/pyenv/pyenv
- Desc. : lets you easily switch between multiple versions of Python.
Readings
Command | Description | Remarks |
---|---|---|
pyenv versions |
Lists all Python versions known to pyenv. | |
pyenv install --list |
List the all available versions of Python, including Anaconda, Jython, pypy, and stackless. | |
pyenv install |
Install a Python version. |
Setuptools
- https://github.com/pypa/setuptools
- Desc. : a fully-featured, actively-maintained, and stable library designed to facilitate packaging Python projects.
- License : MIT License
Readings
PyBuilder
- https://pybuilder.io/
- Desc. : a software build tool written in pure Python which mainly targets Python applications.
- License : Apache License 2.0
- Sources : http://pybuilder.github.io/
Colab
- https://colab.research.google.com/
- Desc. : allows you to write and execute Python in your browser