Python, the language

• chr(): ASCII to char
• ord(): char to ASCII
• float(x)
• long(x)
• bool(x): convert x to bool
• int(): string to integer
• str(): integer to string
  • hex(x): convert integer to lowercase hex string prefix with '0x'
  • oct(x): integer to octal string
  • bin(x): an integer to binary string

The default value of an argument is evaluated once at the function definition. Thus, the object is shared for all the invoke of the function. This is typically not desired behavior.

def foo(a=[]):
    a.append(3)
    return a
foo()
foo()
# => [3,3] !!!

Python function pass-by-object. If you pass a list, you can modify the list, and the original list is modified.

a = [1,2]
def foo(x):
    x.append(3)
foo(a)
a # => [1,2,3]

lambda x : x+2
lambda x: x%2==0

The usage of lambda is often in map and filter.

• map(lambda_exp, mylist) will execute the lambda expression on each element of the list, and return a list containing the results.

use *args syntax, and args will be a tuple:

  def foo(*args):
    for a in args:
      print a

use **args to capture all keyword arguments.

def bar(**kwargs):
  for a in kwargs:
    print a, kwargs[a]

Combine them together:

def foobar(kind, *args, **kwargs):
  pass

Also, there's a concept for the reverse thing: unpack argument list from a list, with *list:

def foo(a,b):
  pass

l = [1,2]
foo(*l)

on python3, this syntax can appear on left side

first, *rest = [1,2,3,4]
first,*l,last = [1,2,3,4]

BaseException
 +-- SystemExit
 +-- KeyboardInterrupt
 +-- GeneratorExit
 +-- Exception
      +-- StopIteration
      +-- StandardError
      |    +-- BufferError
      |    +-- ArithmeticError
      |    |    +-- FloatingPointError
      |    |    +-- OverflowError
      |    |    +-- ZeroDivisionError
      |    +-- AssertionError
      |    +-- AttributeError
      |    +-- EnvironmentError
      |    |    +-- IOError
      |    |    +-- OSError
      |    |         +-- WindowsError (Windows)
      |    |         +-- VMSError (VMS)
      |    +-- EOFError
      |    +-- ImportError
      |    +-- LookupError
      |    |    +-- IndexError
      |    |    +-- KeyError
      |    +-- MemoryError
      |    +-- NameError
      |    |    +-- UnboundLocalError
      |    +-- ReferenceError
      |    +-- RuntimeError
      |    |    +-- NotImplementedError
      |    +-- SyntaxError
      |    |    +-- IndentationError
      |    |         +-- TabError
      |    +-- SystemError
      |    +-- TypeError
      |    +-- ValueError
      |         +-- UnicodeError
      |              +-- UnicodeDecodeError
      |              +-- UnicodeEncodeError
      |              +-- UnicodeTranslateError
      +-- Warning
           +-- DeprecationWarning
           +-- PendingDeprecationWarning
           +-- RuntimeWarning
           +-- SyntaxWarning
           +-- UserWarning
           +-- FutureWarning
	   +-- ImportWarning
	   +-- UnicodeWarning
	   +-- BytesWarning

The local structure directory must contain the __init__.py file to be able to import.

|-- main.py
|-- mypackage
    |-- __init__.py
    |-- a.py
    |-- b.py
    |-- subdir
        |-- __init__.py
        |-- c.py

The import statements should be:

from mypackage import a
from mypackage.b import foo as myfoo
from mypackage.subdir import c

export PYTHONPATH="$PYTHONPATH:/home/hebi/github/reading/models"

Add some path so that I can import from there:

sys.path.append('/home/hebi/github/reading/InferSent/')
# assume in root of that directory, models.py defines InferSent class
from models import InferSent

Packaging:

setup.py:

from setuptools import setup, find_packages
setup(
    name="InferSent-Mirror",
    version="0.1",
    # packages=find_packages(),
    packages=['p1', 'p2'],
)

Directory structure:

mypackage/
  p1/
    __init__.py
    xxx.py
  p2/
    __init__.py
    yyy.py

Install locally:

python3 setup.py install --user

Install from git repo:

pip install --user git+https://github.com/lihebi/InferSent

Import:

from p1 import xxx
from p2.yyy import foo

sort a dictionary by value:

sorted(dict1, key=dict1.get) # => list
sorted(dict1, key=dict1.get, reverse=True)

The slicing syntax is l[start:end:step]. The slicing will return a new list. Change to that list will not change the original one.

l[4]
l[4:]
l[::2]
l[:-1]

However, assign to the slicing itself will change the original one:

l[1:2] = [4,5,6]

Also, assign to a new variable only assign the reference:

a = [1,2,3]
b = a # only a reference

• range(stop)
• range(start, stop[, step])

Creating a matrix:

newmat=[[-1 for x in range(height)] for y in range(width)]

list comprehension

even_squares = [x**2 for x in l if x%2 == 0]

• list.append
• list.pop

Lists are mutable. The behavior of slicing is a bit confusing. If the slicing is used directly as the target of an assignment statement, it will modify the object in place. E.g.

a = [1,2,3,4]
a[1:3] = []
a # => [1,4]

That also means all other references to a will be modified:

a = [1,2,3,4]
a[1:3] = []
# although tuple is immutable, it can still contain reference to
# mutable objects.
c=(a,)
# this will also modify a
a.append(5)
c # => ([1,4,5])

However, if the slicing is assigned to another variable (either assignment or pass-by-object function call), it is copied. Modifying this copy will not affect the original list.

a = [1,2,3,4]
b = a[1:3]
b[0] = 9
a # => [1,2,3,4]
def foo(x):
    x[1] = 8

# changing b
foo(b)
b # => [9,8]
a # => [1,2,3,4]

If you convert a list to a tuple, the elements are shallow-copied.

a = [1,2,3]
b = [a]
# this is shallow copied. Still contains reference to the object "a"
c = tuple(b)
# no reference anymore, just a tuple of (1,2,3). Will never change
# whatsoever.
d = tuple(a)

# testing:
a[2] = 8
b # => [[1,2,8]]
c # => [[1,2,8]]
d # => [1,2,3]

String is immutable sequence, thus cannot be assigned. Thus it is fairly safe to use string.

• concatenate two strings directly by +.
• need to convert integer to string before concatenate: s + str(35)
• "".join(lst) works

str.split(sep=None)

default by white space

str.strip()

strip out white space at both begin and end

str.replace(old, new)

replace all.

str.startswith(s)

str.endswith(s)

String is an immutable object. It can use slicing. E.g. reversing a string is as easy as "hello"[::-1]!

However, notice that when using a negative step, the slicing should be lst[end:begin:-1]. This is because x = i + n*k:

with a third “step” parameter: a[i:j:k] selects all items of a with index x where x = i + n*k, n >= 0 and i <= x < j.

Also, the negative step does not always work as expect. E.g. the i index is included and j is not; the j can not be negative, then how can I include the first one in the list??

Thus if want to get a reverse of a sub-string, I would get sub-string first and then reverse it.

• os.environ['HOME']
• os.getenv(name)
• os.putenv(name, value)
• os.unsetenv(name)

os.system

simply run command

os.system("some command")

os.popen

access to input output

stream = os.popen("some command")
stream.read()

• subprocess.Popen

p = subprocess.Popen("echo Hello World", shell=True, stdout=subprocess.PIPE)
p.stdout.read()
s = subprocess.check_output('wc -l', stdin=p.stdout)

subprocess.call

this is the same as subprocess.Popen except that it waits and gives return code.

return_code = subprocess.call("echo Hello World", shell=True, stdout=subprocess.DEVNULL)

• os.abort()
• os.execl(path, arg0, arg1, …)
• os.execle(path, arg0, arg1, …, env)
• os.execlp(file, arg0, arg1, …)
• os.execlpe(file, arg0, arg1, …, env)
• os.execv(path, args)
• os.execve(path, args, env)
• os.execvp(file, args)
• os.execvpe(file, args, env)
• os.folk
• os.wait()

• os.system(cmd): run cmd, return exit code
• os.times(): 5-tuple
  • user time
  • system time
  • childrens user time
  • childrens system time
  • elapsed real time

Reading:

• read()
• readline(size=1)
• readlines()

Seeking:

• seek(offset=0)
  • 0 start
  • 1 current
  • 2 end
• tell(): current position

Writing:

• write(s): finally the string!
• writelines(lines): write a list of lines
• flush()

  f = open('text.txt')
  f.read() # return all content

  f = open('text.txt')
  for line in f:
      print(line)

  with open('a.txt') as f:
      for line in f:
          print(line)

Other IO:

• f = io.StringIO("some string"): in memory text stream
• f = io.BytesIO(b"some binary data \x00\x01")

• pprint.pprint(object, stream=None): pretty print
• 'string {0}, {hello}'.format('yes', hello=2)

print('xxx', end='')

read from stdin:

for line in sys.stdin:
  print(line)

from contextlib import redirect_stdout
with open('xxx.txt', 'w') as f:
    redirect_stdout(f)

Or:

sys.stdout = f

The file handle can be:

f = open(os.devnull, 'w')

It can also be a predefined handle, like sys.stderr:

with redirect_stdout(sys.stderr):
    help(dir)

import os
for root,dirs,files in os.walk('.'):
  for f in files:
    print f

• os.path.abspath('relative/path/to/file')
• os.path.exists("/path/to/file")
• os.rename('old', 'new')
• os.path.isfile

• os.getcwd(): current working directory
• os.chdir(path): change cwd
• os.mkdir(path)
• os.listdir(path='.'): list all in this dir. E.g. for item in os.listdir('/path'): print (item)
• os.makedirs(path): GOOD this is the way to go the make directories
• os.remove(path): remove a file
• os.rmdir(): remove an empty dir.
• os.removedirs(path): foo/bar/aaa will try to remove aaa, than bar, then foo. Don't use! To recursively remove all contents, use shutil.rmtree
• os.rename(src, dst)
• os.renames(old, new)
• os.rmdir(path): only work if dir is empty
• os.tempnam(): a reasonable absolute name for creating temporary file
  • seems to be vulnerable
• os.walk(top, topdown=True): for each directory including top itself, it yields 3-tuple (dirpath, dirnames, filenames). E.g. for root,dirs,files in os.walk('/path'): for f in files: print (f);

• copy(src,dst)
• copytree(src, dst): recursive
• rmtree(path): rm -r
• move(src, dst)

popen family is deprecated. Use subprocess.

If parameter is not listed, it means a single path.

• exists: GOOD. check whether a path exists
• split: return a pair (head, tail). tail is the last component, without slash. If path ends with slash, tail is empty
  • basename: the tail of the split output
  • dirname: head of split output
• normpath: collapse redundant separators and up level references
• abspath: from relative to absolute path. normpath(join(os.getcwd(), path))
• commonprefix(list): return the longest path prefix
• expanduser: replace the initial component of ~ by the users directory.
• getsize: in bytes
• isabs: predicate for absolute
• isfile:
• isdir
• islink
• join(path, *paths): join intelligently
• realpath: canonical path by following symbolic links

Object-oriented filesystem paths. https://docs.python.org/3/library/pathlib.html

pathlib.Path is the class. pathlib.PosixPath is a subclass for non-windows paths, but seems just for implementation purpose, makes no contribution for user.

Actually not very interesting, this table tells everything:

Some interesting APIs that don't have counterparts:

• Path.glob(pattern) that returns a list of all files matching the shell pattern, e.g. p.glob('*/*.py')
• slash operator: you can directly use p / 'foo' / 'bar'
• Path.iterdir() gives a list of directory items
• Path.parts gives a list of string

• mkstemp creates temp file, but this file is opened. The return value is the file descriptor (int) of the opened file, the same as that gets returned by os.open, thus not easy to work with
• mkdtemp creates temp dir. I would just use this when creating temporary files.

folder = tempfile.mkdtemp()
fd, fname = tempfile.mkstemp()

• date has year, month, day
• time has hour, minute, second
• datetime has both

import datetime

t1 = datetime.date.fromisoformat('2019-12-04')
t2 = datetime.date.fromisoformat('2018-11-24')

delta = t1 - t2
delta.days

t3 = datetime.date.today()
t4 = datetime.date.date(2019, 12, 20)

t0 = datetime.date.fromtimestamp(time.time())

from threading import Thread

class MyThread(Thread):
  def __init__(self, arg):
    Thread.__init__(self)
    self.arg = arg
  def run(self):
    pass

t = MyThread(arg)
t.start()

The package name is threading, the object is Thread.

Functions

• threading.active_count(): number of Thread object
• threading.current_thread(): current Thread object
• threading.enumerate(): return a list of all Thread objects
• threading.meain(): the main Thread object
• threading.local(): the instance of local storage. Different for different threads. Typical usage: mydata = threading.local()

Two ways to specify what to run:

• pass a callable object to the target argument when constructing Thread
• define a subclass of Thread and override the run method.

Methods:

• start: start the thread. It will call run method in a separate thread. The thread terminate when run terminate
• join(timeout=None): the calling thread will block until this thread terminate
  • timeout should be float in seconds
• is_alive: test whether the thread terminate

class threading.Lock

• acquire()
• release()

class threading.RLock

• this is recursive lock. The same thread can acquire the lock multiple times. They will be nested and only when the last release is called, the lock can be acquired by another thead
• acquire()
• release()

class threading.Condition(lock=None)

• the lock must be a Lock or RLock. If none, a RLock is created
• acquire()
• release()
• wait(timeout=None): wait until notified
  • release underlying lock
  • block until notify
  • re-acquire the lock and return
  • typical usage: while not item_is_available(): cv.wait()
  • often use with statement: =with cv: cv.wait_for(pred); get();
• wait_for(predicate, timeout=None)
  • this is same as while not predicate(): cv.wait(), thus more convenient than wait
• notify(n=1): notify one thread
• notify_all(): notify all threads waiting on this condition

class threading.Semaphore: this class manage resources with limited capacity.

• acquire(): decrease capacity
• release(): increase capacity

class threading.Event

• is_set():
• set(): set flag to true
• clear(): set flag to false
• wait(timeout=None): block until internal flag is true

class threading.Timer(interval, function) : Thread

• interval is float in seconds, function is callable. use start method to start the thread, and the function will be called after the delay.
• cancel(): stop the timer and cancel the execution. Only work if the the timer is still waiting.

class threading.Barrier(parties, action=None, timeout=None)

• parties is integer. Every thread calling wait will block, until parties number of such call is called. Then all players unblock and do things simultaneously.
• wait(timeout=None)
• reset(): reset the barrier. The thread waiting for it will receive BrokenBarrierError
• abort(): all current and future wait call for it will get BrokenBarrierError
• parties: number of parties
• n_waiting: number of current waiting
• broken: True or False

This provide multiprocessing.Process class, having similar API with Thread. It seems to use fork but don't have explicit exec on the document?? Wired and seems just do something thread can do (except the sharing of memory of course).

• subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, shell=False, timeout=None, check=False)
  • run the command and wait for it to complete. Return a CompleteProcess instance.
  • if check is True, raise CalledProcessError exception if return code non-zero. This replace the check_call and check_output.

class subprocess.CompletedProcess

• args
• returncode
• stdout: captured if PIPE is passed to stdout
• stderr: captured if PIPE is passed to stderr
• check_returncode(): if returncode is non-zero, raise CalledProcessError

Variables:

• subprocess.DEVNULL
• subprocess.PIPE
• subprocess.STDOUT: this is only used in the place of stderr to redirect it to stdout

class subprocess.CalledProcessError

• returncode
• cmd
• output: same as stdout
• stdout
• stderr

The followings are from 2.7, now only use run.

• subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
  • args: a list of argument, including arg0
  • it can also be a string due to that *
  • it will wait, then return returncode
  • do not use stdout=PIPE, use communicate() instead TODO
  • use shell=True is bad, but it can give me
    • shell pipes
    • filename wildcard
    • env variable expansion
    • ~ expansion
• check_call(args, *, …): same as call, except it will raise exception if return non-0
• check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False)
  • if return non-0, raise exception. Otherwise return the stdout

Popen object

• Popen constructor
  • args, bufsize=0, executable=None,
  • stdin=None, stdout=None, stderr=None,
  • preexec_fn=None, close_fds=False,
  • shell=False, cwd=None, env=None,
  • universal_newlines=False, startupinfo=None, creationflags=0
• Popen.poll(): check if child process has terminated. Set and return returncode.
• Popen.wait(): wait for process to terminate. Don't use PIPE with this.
• Popen.communicate(input=None): to use this, the corresponding stdin, stdout, stderr should be set to PIPE.
  • send data to stdin (string)
  • read data from stdout and stderr (it returns a tuple (out, err))
  • wait for termination
• Popen.sned_signal(signal)
• Popen.terminate(): send SIGTERM
• Popen.kill(): send SIGKILL
• Popen.pid
• Popen.returncode
  • set by poll and wait (and indirectly by communicate)
  • None indicate hasn't terminated
  • -N means terminated by signal N

package urllib.request

Functions

• urlopen(url, data=None)
  • url can be a string or Request object
  • for http and https, returns a http.client.HTTPResponse object
  • for FTP, file, data urls, return a urllib.response.addinfourl object
• pathname2url(path): do quoting
• url2pathname(path): do unquoting

class Request

• constructor: (url, data=None, headers={}, method=None)
  • url: a string
  • headers: a dictionary.
  • method: a string. 'GET' is default. Available values: 'HEAD', 'POST'

methods:

• get_method()
• add_header(key, val)
• has_header(key)
• get_header(key)
• remove_header(key)
• get_fullurl()
• header_items(): return a list of tuples (key, value)

  req = request.Request(query)
  req.add_header("Authorization", "token " + token)
  response = request.urlopen(req)
  s = response.read().decode('utf8')
  langj = json.loads(s);
  # deprecated
  urllib.request.urlretrieve(url[, filename])

• quote(string)
• quote_plus(string)
• unquote(string)
• unquote_plus(string)
• urlencode(query)

• plt.bar
• plt.scatter
• plt.plot: line plot
• plt.hist
• plt.pie

plt.plot([1,2,3,4])

Image via plt.imshow():

# plot a mnist digit
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
# since the data is just an array (28,28), imshow must have converted
# it to image pixel properly
plt.imshow(x_train[7777], cmap='Greys')
# must call plt.show() to open the figure window. Or, execute
# %matplotlib in the REPL, you can get the image directly after
# imshow().
plt.show()

Texts:

• plt.xlabel()
• plt.ylabel()
• plt.title()
• plt.axis()
• plt.text()
• plt.annotate
• plt.grid(True)
• plt.table(): attach a table to an axis!

Scale:

• plt.xscale('linear')
• plt.yscale('log')

plt.ioff()
figure = plt.figure()
figure.canvas.set_window_title('My Grid Visualization')
for x in range(height):
    for y in range(width):
        # print(x,y)
        figure.add_subplot(height, width, x*width + y + 1)
        plt.axis('off')
        plt.imshow(convert_image_255(images[x*width+y]), cmap='gray')
# plt.show()
plt.savefig(filename)

Or better, create figure and axis, and plot for each axis:

import matplotlib.pyplot as plt
import numpy as np

np.random.seed(19680801)
data = np.random.randn(2, 100)

fig, axs = plt.subplots(2, 2, figsize=(5, 5))
axs[0, 0].hist(data[0])
axs[1, 0].scatter(data[0], data[1])
axs[0, 1].plot(data[0], data[1])
axs[1, 1].hist2d(data[0], data[1])

plt.show()

Visualize using OS GUI toolkit:

plt.show()

Plot to a file:

pylab.ioff()
plot([1, 2, 3])
savefig("/tmp/test.png")