Creating and bundling custom profilers

[1]:

from typing import Union, Tuple, Optional
import itertools as it

import numpy as np
import pandas as pd
import statsmodels.tsa.api as sm

Profiling Functions

Creating a customer profiler is very straightforward. For a function to qualify as a profiler it needs:

  • To be type annotated

  • To have one and only one positional argument, the timeseries, which needs to be a pandas DataFrame or Series (or a union of both)

  • To be decorated with @ProfilingFunction

Let’s use a few dummy examples to better illustrate this.

[3]:

from tslumen.profile import ProfilingFunction

[4]:

@ProfilingFunction
def shape(data: Union[pd.Series, pd.DataFrame]) -> Tuple[int, ...]:
    return data.shape

shape

[4]:
ProfilingFunction shape at 0x7fcae639aa60 | Frame ✓ , Series ✓
Signature: (data: Union[pandas.core.series.Series, pandas.core.frame.DataFrame]) -> Tuple[int, ...]

As shown by the checkmarks, the shape function we just created is applied to the DataFrame as a whole, as well as each individual series.

Under the hood, the @ProfilingFunction decorator inspects the function and wraps it inside a class which, among other purposes, identifies the scope of action of the profiler (i.e. single Series, DataFrame or both), particularly useful when orchestrating the execution of multiple profilers. More on that later.

[5]:

@ProfilingFunction
def acovf(data: pd.Series, adjusted: bool = False, demean: bool = True,
          fft: bool = True, missing: str = 'none', nlag: Optional[int] = None) -> pd.Series:
    est = sm.acovf(data.values, adjusted=adjusted, demean=demean, fft=fft, missing=missing, nlag=nlag)
    return pd.Series(est, index=data.index)

acovf

[5]:
ProfilingFunction acovf at 0x7fcae639ac10 | Frame ✘, Series ✓
Signature: (data: pandas.core.series.Series, adjusted: bool = False, demean: bool = True, fft: bool = True, missing: str = 'none', nlag: Optional[int] = None) -> pandas.core.series.Series 
[6]:

@ProfilingFunction
def euc_sim(data: pd.DataFrame, measure: str = 'rmse') -> pd.DataFrame:
    fn = lambda d: d.mean() if measure == 'mean' else np.sqrt((d**2).mean())
    res = {(a, b): fn(data[b]-data[a])
           for a, b in it.product(data.columns, data.columns)}
    return pd.Series(res).unstack(0)

euc_sim

[6]:
ProfilingFunction euc_sim at 0x7fcae639a940 | Frame ✓ , Series ✘
Signature: (data: pandas.core.frame.DataFrame, measure: str = 'rmse') -> pandas.core.frame.DataFrame

Now the acovf function is for Series only, whereas euc_sim is for DataFrames only.

When wrapping functionality from another library, it is a good idea to expose (some of) its parameters as to make the profiler more configurable. Naturally, in some cases, it’s actually a good idea to hide them from the end-user, particularly when the wrapper itself will have the logic to select the parameters.

Note that any arguments the profiling function exposes (other than the first one) need to have defaults.

Let’s load a dataset and try out these profilers

[7]:

df = pd.read_csv('https://datahub.io/core/s-and-p-500/r/data.csv', parse_dates=[0], index_col=0)
df = df[(df.index >= '1990-01-01') & (df.index < '2015-01-01')]
df.head()

[7]:
SP500 Dividend Earnings Consumer Price Index Long Interest Rate Real Price Real Dividend Real Earnings PE10
Date
1990-01-01 339.97 11.14 22.49 127.4 8.21 666.69 21.85 44.10 17.05
1990-02-01 330.45 11.23 22.08 128.0 8.47 644.99 21.92 43.10 16.51
1990-03-01 338.46 11.32 21.67 128.7 8.59 657.03 21.97 42.07 16.83
1990-04-01 338.18 11.44 21.53 128.9 8.79 655.46 22.17 41.74 16.81
1990-05-01 350.25 11.55 21.40 129.2 8.76 677.28 22.34 41.38 17.39 
[8]:

shape(df)

[8]:
ProfileResult at 0x7fcae630cbe0 | name: shape scope: frame | execution: ✓ 0:00:00.000042
(300, 9) 
[9]:

shape(df.SP500)

[9]:
ProfileResult at 0x7fcae6307700 | name: shape scope: series target: SP500 | execution: ✓ 0:00:00.000047
(300,) 
[10]:

acovf(df.SP500)

[10]:
ProfileResult at 0x7fcae6285f10 | name: acovf scope: series target: SP500 | execution: ✓ 0:00:00.000727
Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
0 179217.457496 175888.867311 172240.056519 168900.130792 165215.340345 161487.048775 157530.850357 153705.459385 149985.479096 146125.05111 ... -21330.750303 -19308.50963 -17303.253341 -15288.805447 -13242.503765 -11143.461295 -9034.245959 -6815.498383 -4723.390484 -2356.619013

1 rows × 300 columns

[11]:

euc_sim(df, 'rmse')

[11]:
ProfileResult at 0x7fcae6289730 | name: euc_sim scope: frame | execution: ✓ 0:00:00.023375
Consumer Price Index Dividend Earnings Long Interest Rate PE10 Real Dividend Real Earnings Real Price SP500
Consumer Price Index 0.000000 165.486344 135.796101 181.364722 161.318178 159.546683 122.081817 1247.349170 935.855699
Dividend 165.486344 0.000000 35.661179 17.019169 11.701936 6.940290 48.498664 1407.033108 1093.831545
Earnings 135.796101 35.661179 0.000000 51.538956 34.956319 31.895157 15.897771 1375.411112 1060.978612
Long Interest Rate 181.364722 17.019169 51.538956 0.000000 21.497434 22.065529 64.027759 1422.551992 1110.110884
PE10 161.318178 11.701936 34.956319 21.497434 0.000000 8.751860 45.141560 1401.371581 1089.657346
Real Dividend 159.546683 6.940290 31.895157 22.065529 8.751860 0.000000 43.418904 1401.303869 1088.640134
Real Earnings 122.081817 48.498664 15.897771 64.027759 45.141560 43.418904 0.000000 1361.394243 1048.086806
Real Price 1247.349170 1407.033108 1375.411112 1422.551992 1401.371581 1401.303869 1361.394243 0.000000 364.090297
SP500 935.855699 1093.831545 1060.978612 1110.110884 1089.657346 1088.640134 1048.086806 364.090297 0.000000

All decorated functions return a ProfileResult object, which is a mere wrapper around the function’s result to capture additional details like the name of the function, execution times, exceptions, etc.

It can be used as a class or cast as a dictionary.

[12]:

pr = shape(df.SP500)
dict(pr)

[12]:

{'name': 'shape',
 'scope': 'series',
 'target': 'SP500',
 'start': datetime.datetime(2022, 11, 22, 17, 54, 44, 867712),
 'end': datetime.datetime(2022, 11, 22, 17, 54, 44, 867761),
 'success': True,
 'warnings': [],
 'exception': None,
 'result': (300,)}

[13]:

pr.start

[13]:

datetime.datetime(2022, 11, 22, 17, 54, 44, 867712)

Bundling profilers together

Single profilers (i.e. ProfilingFunction decorated functions) can be bundled together in one single class, which will deal with orchestrating their execution and collecting the results.

A DefaultProfiler is included in tslumen, essentially bundling all the profiling functions included in the package.

Creating a custom bundle is simply a matter of creating a subclass of BundledProfiler and defining the class variable _profilers as a list of the profiling functions that are to be included. Note that these need to have unique names, as the name is used to store the output of the function.

Let’s start by having a look at the DefaultProfiler.

[14]:

from tslumen.profile import DefaultProfiler

[15]:

DefaultProfiler.get_profilers()

[15]:

{'df_scaled': <tslumen.profile.base.ProfilingFunction at 0x7fcae9159a00>,
 'dt_end': <tslumen.profile.base.ProfilingFunction at 0x7fcae914bc10>,
 'dt_start': <tslumen.profile.base.ProfilingFunction at 0x7fcae914bcd0>,
 'freq': <tslumen.profile.base.ProfilingFunction at 0x7fcae914b7f0>,
 'infinite': <tslumen.profile.base.ProfilingFunction at 0x7fcae9159f10>,
 'length': <tslumen.profile.base.ProfilingFunction at 0x7fcae9159d00>,
 'missing': <tslumen.profile.base.ProfilingFunction at 0x7fcae9165f40>,
 'n_series': <tslumen.profile.base.ProfilingFunction at 0x7fcae914b8b0>,
 'period': <tslumen.profile.base.ProfilingFunction at 0x7fcae914b7c0>,
 'sample': <tslumen.profile.base.ProfilingFunction at 0x7fcae9159760>,
 'sz_total': <tslumen.profile.base.ProfilingFunction at 0x7fcae9159bb0>,
 'zeros': <tslumen.profile.base.ProfilingFunction at 0x7fcae9159d90>,
 'cov': <tslumen.profile.base.ProfilingFunction at 0x7fcae916ef10>,
 'iqr': <tslumen.profile.base.ProfilingFunction at 0x7fcae9173640>,
 'kurtosis': <tslumen.profile.base.ProfilingFunction at 0x7fcae91733d0>,
 'mad': <tslumen.profile.base.ProfilingFunction at 0x7fcae916e130>,
 'maximum': <tslumen.profile.base.ProfilingFunction at 0x7fcae9173460>,
 'mean': <tslumen.profile.base.ProfilingFunction at 0x7fcae91652e0>,
 'median': <tslumen.profile.base.ProfilingFunction at 0x7fcae91650d0>,
 'minimum': <tslumen.profile.base.ProfilingFunction at 0x7fcae916ed60>,
 'q25': <tslumen.profile.base.ProfilingFunction at 0x7fcae916ef40>,
 'q50': <tslumen.profile.base.ProfilingFunction at 0x7fcae916e160>,
 'q75': <tslumen.profile.base.ProfilingFunction at 0x7fcae916ee80>,
 'skew': <tslumen.profile.base.ProfilingFunction at 0x7fcae9179a90>,
 'std': <tslumen.profile.base.ProfilingFunction at 0x7fcae9165700>,
 'var': <tslumen.profile.base.ProfilingFunction at 0x7fcae9165250>,
 'adfuller_stationarity': <tslumen.profile.base.ProfilingFunction at 0x7fcae90546a0>,
 'jarque_bera_normality': <tslumen.profile.base.ProfilingFunction at 0x7fcae9054580>,
 'kpss_stationarity': <tslumen.profile.base.ProfilingFunction at 0x7fcae9061d90>,
 'levene_constant_variance': <tslumen.profile.base.ProfilingFunction at 0x7fcae9179af0>,
 'ljungbox_autocorrelation': <tslumen.profile.base.ProfilingFunction at 0x7fcae9179bb0>,
 'omnibus_normality': <tslumen.profile.base.ProfilingFunction at 0x7fcae9054b50>,
 'binned': <tslumen.profile.base.ProfilingFunction at 0x7fcae9002d60>,
 'pd_percentiles': <tslumen.profile.base.ProfilingFunction at 0x7fcae9061700>,
 'pd_quantiles': <tslumen.profile.base.ProfilingFunction at 0x7fcae900cf70>,
 'acf': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94640>,
 'acf_1d': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94d90>,
 'acf_2d': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94130>,
 'corr_kendall': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94520>,
 'corr_pearson': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94cd0>,
 'corr_spearman': <tslumen.profile.base.ProfilingFunction at 0x7fcae8da02e0>,
 'granger_causality': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94ee0>,
 'lag_corr': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94c40>,
 'pacf': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94a90>,
 'pacf_1d': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94160>,
 'pacf_2d': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94f40>,
 'seasonal_split': <tslumen.profile.base.ProfilingFunction at 0x7fcae8da0070>,
 'stl': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d94760>,
 'lowess': <tslumen.profile.base.ProfilingFunction at 0x7fcae8da0820>,
 'rolling_avg': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db1670>,
 'supsmu': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db5220>,
 'ft_acf': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db20d0>,
 'ft_adfuller': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db5550>,
 'ft_cross_pts': <tslumen.profile.base.ProfilingFunction at 0x7fcae8d4e130>,
 'ft_entropy': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db2070>,
 'ft_kpss': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db52e0>,
 'ft_pacf': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db5f40>,
 'ft_stl': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db5430>,
 'ft_tilewin': <tslumen.profile.base.ProfilingFunction at 0x7fcae8db5af0>}

The class method get_profilers returns a dictionary with all the ProfilingFunction bundled in this BundledProfiler

As the name suggests, the class method get_config_defaults returns a dictionary with the default parameters of each ProfilingFunction. It is useful for understanding exactly which parameter defaults can be overridden.

[16]:

DefaultProfiler.get_config_defaults()

[16]:

{'df_scaled': {},
 'dt_end': {},
 'dt_start': {},
 'freq': {},
 'infinite': {},
 'length': {},
 'missing': {},
 'n_series': {},
 'period': {},
 'sample': {'sample_size': 10},
 'sz_total': {'memory_deep': True},
 'zeros': {},
 'cov': {},
 'iqr': {},
 'kurtosis': {},
 'mad': {},
 'maximum': {},
 'mean': {},
 'median': {},
 'minimum': {},
 'q25': {},
 'q50': {},
 'q75': {},
 'skew': {},
 'std': {},
 'var': {},
 'adfuller_stationarity': {'confidence_level': 0.05},
 'jarque_bera_normality': {'confidence_level': 0.05},
 'kpss_stationarity': {'confidence_level': 0.05},
 'levene_constant_variance': {'confidence_level': 0.05},
 'ljungbox_autocorrelation': {'n_lags': None, 'confidence_level': 0.05},
 'omnibus_normality': {'confidence_level': 0.05},
 'binned': {'nbins': None},
 'pd_percentiles': {},
 'pd_quantiles': {},
 'acf': {'lags': 40,
  'adjusted': False,
  'fft': False,
  'alpha': 0.05,
  'missing': 'none'},
 'acf_1d': {'lags': 40,
  'adjusted': False,
  'fft': False,
  'alpha': 0.05,
  'missing': 'none'},
 'acf_2d': {'lags': 40,
  'adjusted': False,
  'fft': False,
  'alpha': 0.05,
  'missing': 'none'},
 'corr_kendall': {},
 'corr_pearson': {},
 'corr_spearman': {},
 'granger_causality': {'test': 'ssr_chi2test',
  'addconst': True,
  'maxlag': 5,
  'max_diff': 3,
  'adf_confidence': 0.1},
 'lag_corr': {'lags': ()},
 'pacf': {'lags': 40, 'method': 'ywadjusted', 'alpha': 0.05},
 'pacf_1d': {'lags': 40, 'method': 'ywadjusted', 'alpha': 0.05},
 'pacf_2d': {'lags': 40, 'method': 'ywadjusted', 'alpha': 0.05},
 'seasonal_split': {},
 'stl': {'period': None,
  'seasonal': 7,
  'trend': None,
  'low_pass': None,
  'seasonal_deg': 0,
  'trend_deg': 0,
  'low_pass_deg': 0,
  'robust': False,
  'seasonal_jump': 1,
  'trend_jump': 1,
  'low_pass_jump': 1},
 'lowess': {'fracs': (0.05, 0.1, 0.15),
  'it': 3,
  'delta': 0.0,
  'missing': 'drop'},
 'rolling_avg': {'wins': (), 'max_win_frac': 10},
 'supsmu': {'alpha': None,
  'period': None,
  'primary_spans': (0.05, 0.2, 0.5),
  'middle_span': 0.2,
  'final_span': 0.05},
 'ft_acf': {'n_diff': (0, 1, 2), 'n_size': (1, 10)},
 'ft_adfuller': {},
 'ft_cross_pts': {},
 'ft_entropy': {'sampling_frequency': 1.0, 'n_per_segment': None},
 'ft_kpss': {},
 'ft_pacf': {'n_diff': (0, 1, 2), 'n_size': (5,)},
 'ft_stl': {'freq': None},
 'ft_tilewin': {}}

Let us now create our own profiler by bundling the functions defined above.

[17]:

from tslumen.profile import BundledProfiler


class MyProfiler(BundledProfiler):
    _profilers = [shape, acovf, euc_sim]

[18]:

MyProfiler.get_profilers()

[18]:

{'shape': <tslumen.profile.base.ProfilingFunction at 0x7fcae639aa60>,
 'acovf': <tslumen.profile.base.ProfilingFunction at 0x7fcae639ac10>,
 'euc_sim': <tslumen.profile.base.ProfilingFunction at 0x7fcae639a940>}

[19]:

MyProfiler.get_config_defaults()

[19]:

{'shape': {},
 'acovf': {'adjusted': False,
  'demean': True,
  'fft': True,
  'missing': 'none',
  'nlag': None},
 'euc_sim': {'measure': 'rmse'}}

When instantiating the bundled profiler, the configs will be set, essentially by taking the defaults of each ProfilingFunction and overriding any parameters explicitly provided as an argument.

[20]:

# default configs were set
profiler1 = MyProfiler()
profiler1.config

[20]:

{'shape': {},
 'acovf': {'adjusted': False,
  'demean': True,
  'fft': True,
  'missing': 'none',
  'nlag': None},
 'euc_sim': {'measure': 'rmse'}}

[21]:

# euc_sim's measure was overriden
profiler2 = MyProfiler(config={'euc_sim': {'measure': 'mean'}})
profiler2.config

[21]:

{'shape': {},
 'acovf': {'adjusted': False,
  'demean': True,
  'fft': True,
  'missing': 'none',
  'nlag': None},
 'euc_sim': {'measure': 'mean'}}

With our profiler instantiated, we can now profile our data. The bundled profiler will take care of orchestrating the execution of the profiling functions and collecting the results. Depending on the function’s scope, each will be called with the dataset or once per series.

[22]:

pr1 = profiler1.profile(df)

[23]:

pr2 = profiler2.profile(df)

[24]:

pr1

[24]:
BundledResult at 0x7fcae6289df0 | name: MyProfiler scope: both | execution: ✓ 0:00:00.077626
exec_details
Profiler Scope Target Start End Duration Succeeded Exceptions # Runs
0 shape frame 2022-11-22 17:54:44.995921 2022-11-22 17:54:44.995990 0 days 00:00:00.000069 True None 1
1 euc_sim frame 2022-11-22 17:54:44.998409 2022-11-22 17:54:45.020264 0 days 00:00:00.021855 True None 1
2 shape series SP500 2022-11-22 17:54:45.021628 2022-11-22 17:54:45.021671 0 days 00:00:00.000043 True None 1
3 shape series Dividend 2022-11-22 17:54:45.022741 2022-11-22 17:54:45.022775 0 days 00:00:00.000034 True None 1
4 shape series Earnings 2022-11-22 17:54:45.023813 2022-11-22 17:54:45.023843 0 days 00:00:00.000030 True None 1
5 shape series Consumer Price Index 2022-11-22 17:54:45.024853 2022-11-22 17:54:45.024883 0 days 00:00:00.000030 True None 1
6 shape series Long Interest Rate 2022-11-22 17:54:45.025892 2022-11-22 17:54:45.025921 0 days 00:00:00.000029 True None 1
7 shape series Real Price 2022-11-22 17:54:45.026930 2022-11-22 17:54:45.026960 0 days 00:00:00.000030 True None 1
8 shape series Real Dividend 2022-11-22 17:54:45.027986 2022-11-22 17:54:45.028015 0 days 00:00:00.000029 True None 1
9 shape series Real Earnings 2022-11-22 17:54:45.029048 2022-11-22 17:54:45.029078 0 days 00:00:00.000030 True None 1
10 shape series PE10 2022-11-22 17:54:45.030177 2022-11-22 17:54:45.030206 0 days 00:00:00.000029 True None 1
11 acovf series SP500 2022-11-22 17:54:45.031225 2022-11-22 17:54:45.031938 0 days 00:00:00.000713 True None 1
12 acovf series Dividend 2022-11-22 17:54:45.032988 2022-11-22 17:54:45.033491 0 days 00:00:00.000503 True None 1
13 acovf series Earnings 2022-11-22 17:54:45.034576 2022-11-22 17:54:45.035131 0 days 00:00:00.000555 True None 1
14 acovf series Consumer Price Index 2022-11-22 17:54:45.036193 2022-11-22 17:54:45.036596 0 days 00:00:00.000403 True None 1
15 acovf series Long Interest Rate 2022-11-22 17:54:45.037695 2022-11-22 17:54:45.038267 0 days 00:00:00.000572 True None 1
16 acovf series Real Price 2022-11-22 17:54:45.039317 2022-11-22 17:54:45.039790 0 days 00:00:00.000473 True None 1
17 acovf series Real Dividend 2022-11-22 17:54:45.040876 2022-11-22 17:54:45.041296 0 days 00:00:00.000420 True None 1
18 acovf series Real Earnings 2022-11-22 17:54:45.042320 2022-11-22 17:54:45.042742 0 days 00:00:00.000422 True None 1
19 acovf series PE10 2022-11-22 17:54:45.043797 2022-11-22 17:54:45.044399 0 days 00:00:00.000602 True None 1
config
  • shape
  • acovf
    • adjusted 0
    • demean 1
    • fft 1
    • missing none
    • nlag
  • euc_sim
    • measure rmse
frame
  • shape
    • 300
    • 9
  • euc_sim
    Consumer Price Index Dividend Earnings Long Interest Rate PE10 Real Dividend Real Earnings Real Price SP500
    Consumer Price Index 0.000000 165.486344 135.796101 181.364722 161.318178 159.546683 122.081817 1247.349170 935.855699
    Dividend 165.486344 0.000000 35.661179 17.019169 11.701936 6.940290 48.498664 1407.033108 1093.831545
    Earnings 135.796101 35.661179 0.000000 51.538956 34.956319 31.895157 15.897771 1375.411112 1060.978612
    Long Interest Rate 181.364722 17.019169 51.538956 0.000000 21.497434 22.065529 64.027759 1422.551992 1110.110884
    PE10 161.318178 11.701936 34.956319 21.497434 0.000000 8.751860 45.141560 1401.371581 1089.657346
    Real Dividend 159.546683 6.940290 31.895157 22.065529 8.751860 0.000000 43.418904 1401.303869 1088.640134
    Real Earnings 122.081817 48.498664 15.897771 64.027759 45.141560 43.418904 0.000000 1361.394243 1048.086806
    Real Price 1247.349170 1407.033108 1375.411112 1422.551992 1401.371581 1401.303869 1361.394243 0.000000 364.090297
    SP500 935.855699 1093.831545 1060.978612 1110.110884 1089.657346 1088.640134 1048.086806 364.090297 0.000000
series
  • SP500
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 179217.457496 175888.867311 172240.056519 168900.130792 165215.340345 161487.048775 157530.850357 153705.459385 149985.479096 146125.05111 ... -21330.750303 -19308.50963 -17303.253341 -15288.805447 -13242.503765 -11143.461295 -9034.245959 -6815.498383 -4723.390484 -2356.619013

      1 rows × 300 columns

  • Dividend
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 49.400977 48.606554 47.794709 46.966367 46.124609 45.273997 44.416014 43.551058 42.681026 41.808392 ... -4.913877 -4.491505 -4.052579 -3.600032 -3.134073 -2.653572 -2.15576 -1.640504 -1.108437 -0.561729

      1 rows × 300 columns

  • Earnings
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 677.698755 669.673559 657.92764 643.170637 626.132013 607.439338 587.328768 566.06651 543.918129 521.181079 ... -49.219151 -44.510434 -39.753329 -34.898247 -29.945646 -24.880609 -19.760028 -14.58367 -9.542138 -4.681179

      1 rows × 300 columns

  • Consumer Price Index
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 1046.925821 1037.008199 1026.620322 1015.986037 1005.218336 994.466492 983.714687 972.977294 962.481925 952.30536 ... -95.760634 -86.981831 -77.911929 -68.49001 -58.772746 -48.980822 -39.139278 -29.207523 -19.34701 -9.600817

      1 rows × 300 columns

  • Long Interest Rate
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 3.206453 3.150692 3.079734 3.009203 2.937006 2.864374 2.802541 2.745641 2.686821 2.627148 ... -0.301043 -0.272852 -0.24273 -0.212285 -0.18481 -0.156489 -0.124337 -0.092837 -0.060652 -0.029786

      1 rows × 300 columns

  • Real Price
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 185372.546946 182138.908686 178410.497863 175010.963782 171288.50299 167479.202707 163330.402897 159296.874 155213.734452 150767.462332 ... -16587.827481 -14952.145329 -13356.148569 -11798.267161 -10280.053473 -8718.552261 -7130.097809 -5417.025787 -3824.172983 -1908.431674

      1 rows × 300 columns

  • Real Dividend
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 23.784269 23.307033 22.811945 22.305853 21.788526 21.257242 20.717799 20.169604 19.609093 19.035693 ... -1.788915 -1.635697 -1.476435 -1.319282 -1.16362 -1.000958 -0.828 -0.641519 -0.43859 -0.225194

      1 rows × 300 columns

  • Real Earnings
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 627.922808 620.911342 608.835159 592.677006 573.408179 551.651944 527.755051 502.042683 474.838709 446.575623 ... -32.995829 -29.667612 -26.376891 -23.058698 -19.706581 -16.260724 -12.815635 -9.372833 -6.049005 -2.927201

      1 rows × 300 columns

  • PE10
    • shape
      • 300
    • acovf
      Date 1990-01-01 1990-02-01 1990-03-01 1990-04-01 1990-05-01 1990-06-01 1990-07-01 1990-08-01 1990-09-01 1990-10-01 ... 2014-03-01 2014-04-01 2014-05-01 2014-06-01 2014-07-01 2014-08-01 2014-09-01 2014-10-01 2014-11-01 2014-12-01
      0 46.669335 46.126725 45.410262 44.710637 43.980611 43.22205 42.378318 41.554839 40.681552 39.682067 ... -0.104916 -0.110232 -0.115056 -0.117495 -0.110682 -0.100873 -0.093304 -0.077496 -0.08024 -0.04124

      1 rows × 300 columns

Results are held in a BundledResult object, which includes the same details as ProfileResult. Under the result attribute you’ll find:

  • exec_details a DataFrame with all the executions

  • config the configurations passed onto this execution

  • frame a dictionary with the results of each profiling function whose scope is DataFrame

  • series a dictionary with each of the series in the DataFrame, and for each series a dictionary with the results of the series profiling functions

[25]:

pr1.result.frame['euc_sim']

[25]:
Consumer Price Index Dividend Earnings Long Interest Rate PE10 Real Dividend Real Earnings Real Price SP500
Consumer Price Index 0.000000 165.486344 135.796101 181.364722 161.318178 159.546683 122.081817 1247.349170 935.855699
Dividend 165.486344 0.000000 35.661179 17.019169 11.701936 6.940290 48.498664 1407.033108 1093.831545
Earnings 135.796101 35.661179 0.000000 51.538956 34.956319 31.895157 15.897771 1375.411112 1060.978612
Long Interest Rate 181.364722 17.019169 51.538956 0.000000 21.497434 22.065529 64.027759 1422.551992 1110.110884
PE10 161.318178 11.701936 34.956319 21.497434 0.000000 8.751860 45.141560 1401.371581 1089.657346
Real Dividend 159.546683 6.940290 31.895157 22.065529 8.751860 0.000000 43.418904 1401.303869 1088.640134
Real Earnings 122.081817 48.498664 15.897771 64.027759 45.141560 43.418904 0.000000 1361.394243 1048.086806
Real Price 1247.349170 1407.033108 1375.411112 1422.551992 1401.371581 1401.303869 1361.394243 0.000000 364.090297
SP500 935.855699 1093.831545 1060.978612 1110.110884 1089.657346 1088.640134 1048.086806 364.090297 0.000000 
[26]:

pr2.result.frame['euc_sim']

[26]:
Consumer Price Index Dividend Earnings Long Interest Rate PE10 Real Dividend Real Earnings Real Price SP500
Consumer Price Index 0.000000 163.441100 134.428367 178.143767 157.812267 156.987300 119.670567 -1177.380400 -847.525000
Dividend -163.441100 0.000000 -29.012733 14.702667 -5.628833 -6.453800 -43.770533 -1340.821500 -1010.966100
Earnings -134.428367 29.012733 0.000000 43.715400 23.383900 22.558933 -14.757800 -1311.808767 -981.953367
Long Interest Rate -178.143767 -14.702667 -43.715400 0.000000 -20.331500 -21.156467 -58.473200 -1355.524167 -1025.668767
PE10 -157.812267 5.628833 -23.383900 20.331500 0.000000 -0.824967 -38.141700 -1335.192667 -1005.337267
Real Dividend -156.987300 6.453800 -22.558933 21.156467 0.824967 0.000000 -37.316733 -1334.367700 -1004.512300
Real Earnings -119.670567 43.770533 14.757800 58.473200 38.141700 37.316733 0.000000 -1297.050967 -967.195567
Real Price 1177.380400 1340.821500 1311.808767 1355.524167 1335.192667 1334.367700 1297.050967 0.000000 329.855400
SP500 847.525000 1010.966100 981.953367 1025.668767 1005.337267 1004.512300 967.195567 -329.855400 0.000000 
[27]:

pr1.result.frame['euc_sim'].equals(pr2.result.frame['euc_sim'])

[27]:

False

As we can see, the parameter is being passed on correctly to euc_sim.

Closing thoughts

This notebook illustrates how simple it is to create custom profilers and bundles.

It’s up to the user to mix and match whichever profilers are relevant to their needs, pre-canned ones available under tslumen.profile or user-defined functions, provided they meet the conventions listed above and are decorated using @ProfilingFunction.