alexlib package¶
Submodules¶
alexlib.toolbox module¶
A collection of classes extending the functionality of Python’s builtins. email programmer@usa.com
-
class
alexlib.toolbox.
Artist
(*args, ax=None, figname='Graph', title='', label='curve', style='seaborn', create_new_axes=False, figure_policy=<FigurePolicy.add_new: 'Create a new figure with same name but with added suffix'>, figsize=(7, 4), **kwargs)[source]¶
-
class
alexlib.toolbox.
Base
[source]¶ Bases:
object
-
save
(path)[source]¶ this method goes with default .npy format (because it is generic). # P(path).parent.create()
-
-
class
alexlib.toolbox.
Compression
[source]¶ Bases:
object
Provides consistent behaviour across all methods … Both files and folders when compressed, default is being under the root of archive.
-
static
compress_folder
(ip_path, op_path, arcname, format_='zip', **kwargs)[source]¶ base_dir has to be relevant to op_path. If you want to compress a folder in Downloads/myfolder/compress_this Then, say that your rootdir is where you want the archive structure to include, then mention the folder you want to actually archive relatively to that root. fromat_: zip, tar, gztar, bztar, xztar
-
static
zip_file
(ip_path, op_path, arcname, **kwargs)[source]¶ arcname determines the directory of the file being archived inside the archive. Defaults to same as original directory except for drive. When changed, it should still include the file name in its end. If arcname = filename without any path, then, it will be in the root of the archive.
-
static
-
class
alexlib.toolbox.
FigureManager
(info_loc=None, figure_policy=<FigurePolicy.same: 'Grab the figure of the same name'>)[source]¶ Bases:
object
Handles figures of matplotlib.
-
static
get_fig_static
(figure_policy, figname='', suffix=None, **kwargs)[source]¶ - Parameters
figure_policy –
figname –
suffix – only relevant if figure_policy is add_new
kwargs –
- Returns
-
static
update
(fig_name, obj_name, data=None)[source]¶ Fastest update ever. But, you need access to label name. Using this function external to the plotter. But inside the plotter you need to define labels to objects The other alternative is to do the update inside the plotter, but it will become very verbose.
- Parameters
fig_name –
obj_name –
data –
- Returns
-
static
-
class
alexlib.toolbox.
FigurePolicy
(value)[source]¶ Bases:
enum.Enum
An enumeration.
-
add_new
= 'Create a new figure with same name but with added suffix'¶
-
close_create_new
= 'Close the previous figure that has the same figname and create a new fresh one'¶
-
same
= 'Grab the figure of the same name'¶
-
-
class
alexlib.toolbox.
ImShow
(*images_list: Union[list, numpy.ndarray], sup_titles=None, sub_labels=None, labels=None, save_type=<class 'alexlib.toolbox.SaveType.Null'>, save_name=None, save_dir=None, save_kwargs=None, subplots_adjust=None, gridspec=None, tight=True, info_loc=None, nrows=None, ncols=None, ax=None, figsize=None, figname='im_show', figure_policy=<FigurePolicy.add_new: 'Create a new figure with same name but with added suffix'>, auto_brightness=True, delay=200, pause=False, **kwargs)[source]¶ Bases:
alexlib.toolbox.FigureManager
-
artist
= 'internal'¶
-
static
cm
(im, nrows=3, ncols=7, **kwargs)[source]¶ Useful for looking at one image in multiple cmaps
- Parameters
im –
nrows –
ncols –
kwargs –
- Returns
-
parser
= 'internal'¶
-
stream
= 'update'¶
-
-
alexlib.toolbox.
L
¶ alias of
alexlib.toolbox.List
-
class
alexlib.toolbox.
List
(obj_list=None)[source]¶ Bases:
list
,alexlib.toolbox.Base
Use this class to keep items of the same type.
-
apply
(func, *args, lest=None, jobs=None, depth=1, **kwargs)[source]¶ - Parameters
jobs –
func – func has to be a function, possibly a lambda function. At any rate, it should return something.
args –
lest –
depth – apply the function to inner Lists
kwargs – a list of outputs each time the function is called on elements of the list.
- Returns
-
find
(patt, match='fnmatch')[source]¶ Looks up the string representation of all items in the list and finds the one that partially matches the argument passed. This method is a short for self.filter(lambda x: string_ in str(x)) If you need more complicated logic in the search, revert to filter method.
-
property
len
¶
-
modify
(func, lest=None)[source]¶ Modifies objects rather than returning new list of objects, hence the name of the method. :param func: a string that will be executed, assuming idx, x and y are given. :param lest: :return:
-
property
np
¶
-
-
class
alexlib.toolbox.
Manipulator
[source]¶ Bases:
object
-
static
merge_adjacent_axes
(array, ax1, ax2)[source]¶ Multiplies out two axes to generate reduced order array. :param array: :param ax1: :param ax2: :return:
-
static
-
class
alexlib.toolbox.
P
(*args, **kwargs)[source]¶ Bases:
pathlib.PosixPath
,pathlib.Path
,alexlib.toolbox.Base
Path Class: Designed with one goal in mind: any operation on paths MUST NOT take more than one line of code.
-
append
(name='', suffix=None)[source]¶ Add extra text after file name, and optionally add extra suffix. e.g: blahlah.extenion ==> becomes ==> blah/blah_name.extension
-
property
browse
¶
-
property
browse2
¶
-
copy
(target=None, contents=False, verbose=False)[source]¶ contents: copy the parent directory or its contents.
-
create
(parents=True, exist_ok=True, parent_only=False)[source]¶ Creates directory while returning the same object
-
find
(*args, r=True, **kwargs)[source]¶ short for globbing then using next method to get the first result
-
property
len
¶
-
myglob
(pattern='*', r=False, list_=True, files=True, folders=True, dotfiles=False, return_type=None, absolute=True, filters=None, win_order=False)[source]¶ - Parameters
win_order –
self –
filters –
dotfiles –
pattern – regex expression.
r – recursive search
list – output format, list or generator.
files – include files in search.
folders – include directories in search.
return_type – output type, Pathlib objects or strings.
absolute – return relative paths or abosolute ones.
- Returns
search results.
# :param visible: exclude hidden files and folders (Windows)
-
prepend
(name, stem=False)[source]¶ Add extra text before file name e.g: blahlah.extenion ==> becomes ==> blah/name_blah.extension
-
relativity_transform
(reference='deephead', abs_reference=None)[source]¶ Takes in a path defined relative to reference, transform it to a path relative to execution directory, then makes it absolute path.
Warning
reference must be included in the execution directory. Otherwise, absolute path of reference should be provided.
-
split
(at=None, index=None)[source]¶ Splits a path at a given string or index :param self: :param at: :param index: :return: two paths
-
property
string
¶
-
static
tmp
(folder=None, fn=None, path='home')[source]¶ folder is created. file name is not created, only appended.
-
property
trunk
¶ useful if you have multiple dots in file name where .stem fails.
-
-
class
alexlib.toolbox.
Read
[source]¶ Bases:
object
-
class
alexlib.toolbox.
Save
[source]¶ Bases:
object
-
static
json
(path, obj)[source]¶ This format is compatible with simple dictionaries that hold strings or numbers but nothing more than that. E.g. arrays or any other structure. An example of that is settings dictionary. It is useful because it can be inspected using any text editor.
-
static
-
class
alexlib.toolbox.
SaveType
[source]¶ Bases:
object
Programming philosophy: this class only worries about saving, and saving only. In other words, the figure must be fully prepared beforehand. Names here are only used for the purpose of saving, never putting titles on figures.
-
class
GIF
(interval=100, **kwargs)[source]¶ Bases:
alexlib.toolbox.SaveType.GenericSave
Requirements: same axis must persist, only new objects are drawn inside it. This is not harsh as no one wants to add multiple axes on top of each other. Next, the objects drawn must not be removed, or updated, instead they should pile up in axis.
# do not pass names in the add method. names will be extracted from figures. # usually it is smoother when adding animate=True to plot or imshow commands for GIF purpose
Works for images only. Add more .imshow to the same axis, and that’s it. imshow will conver up previous images. For lines, it will superimpose it and will look ugly.
If you clear the axis, nothing will be saved. This should not happend. The class will automatically detect new lines by their “neo” labels. and add them then hide them for the next round. Limitation of ArtistAnimation: works on lines and images list attached to figure axes. Doesn’t work on axes, unless you add large number of them. As such, titles are not incorporated etc.
-
class
GIFFileBasedAuto
(plotter_class, data, fps=4, dpi=150, bitrate=2500, _type='GIFFileBasedAuto', **kwargs)[source]¶
-
class
GenericAuto
(plotter_class, data, names_list=None, **kwargs)[source]¶ Bases:
alexlib.toolbox.SaveType.GenericSave
-
save_type
= 'auto'¶
-
-
class
GenericSave
(save_dir=None, save_name=None, watch_figs=None, max_calls=2000, delay=100, **kwargs)[source]¶ Bases:
object
You can either pass the figures to be tracked or, pass them dynamically at add method, or, add method will capture every figure and axis
-
stream
= 'clear'¶
-
-
class
Null
(*args, **kwargs)[source]¶ Bases:
alexlib.toolbox.SaveType.GenericSave
Use this when you do not want to save anything. This class will help plot to work faster by removing lines of previous plot, so you get live animation cheaply.
-
class
PDF
(*args, **kwargs)[source]¶ Bases:
alexlib.toolbox.SaveType.GenericSave
For pdf, you just need any figure manager, [update, clear, accumalate], preferabbly fastest.
-
class
-
class
alexlib.toolbox.
Struct
(dictionary=None, **kwargs)[source]¶ Bases:
alexlib.toolbox.Base
Use this class to keep bits and sundry items.
-
property
df
¶
-
property
dict
¶
-
property
-
class
alexlib.toolbox.
VisibilityViewer
(artist=None, hide_artist_axes=True)[source]¶ Bases:
alexlib.toolbox.FigureManager
-
artist
= 'external'¶
-
parser
= 'external'¶
-
stream
= 'accumulate'¶ Viewer Building Philosophy:
Viewer should act as Saver and Browser:
How is the data viewed:
Can either be an artist himself, as in ImShow.
external artist is required to view the data (especially non image data)
Data parsing:
- internal for loop to go through all the dataset passed.
# Allows manual control over parsing.
- external for loop. It should have add method.
# Manual control only takes place after the external loop is over. #TODO parallelize this.
Refresh mechanism.
Clear the axis.
accumulate, using visibility to hide previous axes.
The artist has an update method.
The artist has to have:
fig, ax, txt attributes. ax and txt should be lists.
the ax and txt attributes should always belong to the same figure.
Here and in all Visibility classes, the artist is assumed to be always creating new axes along the way.
-
-
class
alexlib.toolbox.
VisibilityViewerAuto
(data=None, artist=None, memorize=False, transpose=True, save_type=<class 'alexlib.toolbox.SaveType.Null'>, save_dir=None, save_name=None, delay=1, titles=None, legends=None, x_labels=None, pause=True, **kwargs)[source]¶
-
alexlib.toolbox.
accelerate
(func, ip)[source]¶ Conditions for this to work: * Must run under __main__ context * func must be defined outside that context.
To accelerate IO-bound process, use multithreading. An example of that is somthing very cheap to process, but takes a long time to be obtained like a request from server. For this, multithreading launches all threads together, then process them in an interleaved fashion as they arrive, all will line-up for same processor, if it happens that they arrived quickly.
To accelerate processing-bound process use multiprocessing, even better, use Numba. Method1 use: multiprocessing / multithreading. Method2: using joblib (still based on multiprocessing) from joblib import Parallel, delayed Fast method using Concurrent module
-
alexlib.toolbox.
browse
(path, depth=2, width=20)[source]¶ - Parameters
width – if there are more than this items in a directory, dont’ parse the rest.
depth – to prevent crash, limit how deep recursive call can happen.
path – absolute path
- Returns
constructs a class dynamically by using object method.
-
alexlib.toolbox.
run_globally
(name, asis=False)[source]¶ Takes in a function name, reads it source code and returns a new version of it that can be run in the main. This is useful to debug functions and class methods alike.
-
alexlib.toolbox.
tmp
(folder=None, fn=None, path='home')¶ folder is created. file name is not created, only appended.
alexlib.deeplearning module¶
-
class
alexlib.deeplearning.
BaseModel
(hp=None, data=None, model=None, compiler=None, history=None)[source]¶ Bases:
abc.ABC
-
build
(shape=None, dtype=<class 'numpy.float32'>)[source]¶ Building has two main uses.
Useful to baptize the model, especially when its layers are built lazily. Although this will eventually happen as the first batch goes in. This is a must before showing the summary of the model.
Doing sanity check about shapes when designing model.
Sanity check about values and ranges when random normal input is fed.
- Parameters
dtype –
shape –
- Returns
-
compile
(loss=None, optimizer=None, metrics=None, compile_model=True)[source]¶ Updates compiler attributes. This acts like a setter.
Must be run prior to fit method.
Can be run only after defining model attribute.
-
evaluate
(x_test=None, y_test=None, names_test=None, idx=None, viz=True, return_loss=False, **kwargs)[source]¶
-
infer
(x)[source]¶ This method assumes numpy input, datatype-wise and is also preprocessed. NN is put in eval mode.
- Parameters
x –
- Returns
prediction as numpy
-
-
class
alexlib.deeplearning.
DataReader
(hp=None, data_specs=None, split=None)[source]¶ Bases:
object
-
data_split
(*args, strings=None, **kwargs)[source]¶ - Parameters
args – whatever to be sent to train_test_split
kwargs – whatever to be sent to train_test_split
strings –
- Returns
-
static
from_saved
(path)[source]¶ This method offers an alternative constructer for DataReader class. Use this when loading training data is not required. It requires saved essential parameters to be stored. Those parameters are required by models to work.
- Parameters
path – full path to the saved .npy file containing a dictionary of attributes names and values.
- Returns
An object with attributes similar to keys and values as in dictionary loaded.
-
-
class
alexlib.deeplearning.
Device
(value)[source]¶ Bases:
enum.Enum
An enumeration.
-
auto
= 'auto'¶
-
cpu
= 'cpu'¶
-
gpu0
= 'gpu0'¶
-
gpu1
= 'gpu1'¶
-
two_gpus
= '2gpus'¶
-
-
class
alexlib.deeplearning.
Ensemble
(hp_class=None, data_class=None, model_class=None, n=15, _from_saved=False)[source]¶ Bases:
object
-
class
alexlib.deeplearning.
HPTuning
[source]¶ Bases:
object
-
class
alexlib.deeplearning.
HyperParam
[source]¶ Bases:
object
Benefits of this way of organizing the hyperparameters:
one place to control everything.
When doing multiple experiments, one command in console reminds you of settings used in that run (hp.__dict__).
Ease of saving settings of experiments! and also replicating it later.
-
property
save_dir
¶ Ensures that the folder created is directly under deephead root, no matter what directory the run was done from. This is especially useful during imports, resulting in predicted behaviour.