4. Create training dataset from Online Feature Store enabled feature groups

Create a training dataset

overview-4.png

Establish a connection with your Hopsworks feature store.

import hsfs
connection = hsfs.connection()
# get a reference to the feature store, you can access also shared feature stores by providing the feature store name
fs = connection.get_feature_store()
Starting Spark application
IDYARN Application IDKindStateSpark UIDriver log
62application_1623853832952_0044pysparkidleLinkLink
SparkSession available as 'spark'.
Connected. Call `.close()` to terminate connection gracefully.

Create transformation function (optional)

Transformation functions are python functions that receive a feature value as input and returns the result of applying a specific transformation on it. It’s possible to defined your own python functions to transform feature values. These functions are created at the feature store level and can be used to generate training datasets by attaching them to specific features composing the dataset.

To be able to attach a transformation function to a training dataset it has to be either part of a library installed in Hopsworks or attached when starting the Jupyter notebook. For more information about transformation functions, please see https://docs.hopsworks.ai/latest/generated/transformation_functions/

The example code snippet below shows how to create a transformation function in the Hopsworks Feature Store.

from hsfs_transformers import transformers
normalize_meta = fs.create_transformation_function(
            transformation_function=transformers.normalize,
            output_type=int,
            version=1)
normalize_meta.save()

Get feature groups

card_transactions_10m_agg = fs.get_feature_group("card_transactions_10m_agg", version = 1)
card_transactions_1h_agg = fs.get_feature_group("card_transactions_1h_agg", version = 1)
card_transactions_12h_agg = fs.get_feature_group("card_transactions_12h_agg", version = 1)

Create training dataset

query = card_transactions_10m_agg.select(["stdev_amt_per_10m", "avg_amt_per_10m", "num_trans_per_10m"])\
                                 .join(card_transactions_1h_agg.select(["stdev_amt_per_1h", "avg_amt_per_1h", "num_trans_per_1h"]))\
                                 .join(card_transactions_12h_agg.select(["stdev_amt_per_12h", "avg_amt_per_12h", "num_trans_per_12h"]))
td_meta = fs.create_training_dataset(name="card_fraud_model",
                               description="Training dataset to train card fraud model",
                               data_format="tfrecord",
                               statistics_config={"enabled": True, "histograms": True, "correlations": False},
                               version=1,
#                              NOTE: To attach transformation function to training dataset provide transformation functions as dict,
#                                    where key is feature name and value is online transformation function name.
#                              transformation_functions={"stdev_amt_per_10m":normalize_meta,
#                                                        "avg_amt_per_10m":normalize_meta,
#                                                        ...
#                                                       }
                               )
td_meta.save(query)
<hsfs.training_dataset.TrainingDataset object at 0x7f98230f4410>
td_meta.read().show()
+---------------+---------------+----------------+--------------+----------------+-----------------+-----------------+-----------------+-----------------+
|avg_amt_per_10m|avg_amt_per_12h|stdev_amt_per_1h|avg_amt_per_1h|num_trans_per_1h|num_trans_per_12h|stdev_amt_per_12h|num_trans_per_10m|stdev_amt_per_10m|
+---------------+---------------+----------------+--------------+----------------+-----------------+-----------------+-----------------+-----------------+
|       1.005665|        1.03796|       1.0266877|     1.0247675|          1.0015|           1.0065|        1.0697409|            1.001|           1.0005|
|        1.03907|       1.139612|          1.0005|       1.02391|           1.001|            1.003|        1.1538386|            1.001|           1.0005|
|        1.04755|       1.318325|          1.0005|       5.18205|           1.001|            1.009|        1.9687029|            1.001|           1.0005|
|       1.043745|      1.0632564|          1.0005|       1.02109|           1.001|            1.006|        1.0999271|            1.001|           1.0005|
|       1.006885|       1.101762|          1.0005|      1.045685|           1.001|           1.0055|        1.1411338|            1.001|           1.0005|
|         1.0099|      1.1035554|          1.0005|      1.046075|           1.001|            1.006|        1.1777302|            1.001|           1.0005|
|        1.75089|      1.0588467|          1.0005|        1.2385|           1.001|           1.0035|        1.0908353|            1.001|           1.0005|
|       1.001085|      1.0586371|       3.8128898|       2.65805|           1.002|            1.004|        1.0761116|            1.001|           1.0005|
|       1.000665|       1.358466|       1.2214001|      1.206005|          1.0015|           1.0075|        1.8494561|            1.001|           1.0005|
|        1.00264|      1.1111832|          1.0005|       1.35051|           1.001|           1.0045|        1.1633795|            1.001|           1.0005|
|        1.05099|      1.1732512|       1.0785328|     1.1408975|          1.0015|            1.005|        1.1760855|            1.001|           1.0005|
|        1.30523|      1.6181775|       2.3883066|     2.0242176|          1.0015|           1.0065|        2.3270547|            1.001|           1.0005|
|        1.00766|       1.109039|       1.0183969|       1.01898|          1.0015|           1.0055|         1.171347|            1.001|           1.0005|
|      1.0175675|      1.5450006|          1.0005|      1.032555|           1.001|            1.005|        2.3324296|           1.0015|        1.0237107|
|       1.039175|      1.5020307|          1.0005|       2.68918|           1.001|            1.004|        1.7150314|            1.001|           1.0005|
|       1.437465|       1.587175|       1.1406308|     1.2778575|          1.0015|            1.004|        2.3232758|            1.001|           1.0005|
|        1.45408|      2.0839994|          1.0005|       1.02899|           1.001|            1.005|        2.4732418|            1.001|           1.0005|
|        1.23047|      1.5117222|          1.0005|       1.04728|           1.001|            1.005|        2.1051917|            1.001|           1.0005|
|        1.01566|       1.185298|       1.0169014|     1.0172175|          1.0015|           1.0055|        1.1831121|            1.001|           1.0005|
|       1.031375|       1.082731|       1.1941308|     1.1309226|          1.0025|           1.0055|        1.1124095|            1.001|           1.0005|
+---------------+---------------+----------------+--------------+----------------+-----------------+-----------------+-----------------+-----------------+
only showing top 20 rows

Check descriptive statistics

td_meta = fs.get_training_dataset("card_fraud_model", 1)
statistics = td_meta.get_statistics()

for feat_list in statistics.content.items():
    for stats in feat_list[1]:
        print("Feature: " + str(stats['column']))
        print(stats, end="\n\n")
Feature: num_trans_per_1h
{'dataType': 'Fractional', 'column': 'num_trans_per_1h', 'sum': 100.1280027627945, 'completeness': 1, 'histogram': [{'count': 10, 'value': '1.002', 'ratio': 0.1}, {'count': 1, 'value': '1.0025', 'ratio': 0.01}, {'count': 33, 'value': '1.0015', 'ratio': 0.33}, {'count': 56, 'value': '1.001', 'ratio': 0.56}], 'distinctness': 0.04, 'entropy': 0.9668672345930647, 'approximateNumDistinctValues': 4, 'isDataTypeInferred': 'false', 'uniqueness': 0.01, 'mean': 1.001280027627945, 'maximum': 1.002500057220459, 'stdDev': 0.0003557872363590212, 'minimum': 1.0010000467300415, 'approxPercentiles': []}
Feature: stdev_amt_per_10m
{'dataType': 'Fractional', 'column': 'stdev_amt_per_10m', 'sum': 103.46918630599976, 'completeness': 1, 'histogram': [{'count': 1, 'value': '1.0158194', 'ratio': 0.01}, {'count': 1, 'value': '1.0237107', 'ratio': 0.01}, {'count': 1, 'value': '4.2807717', 'ratio': 0.01}, {'count': 96, 'value': '1.0005', 'ratio': 0.96}, {'count': 1, 'value': '1.1008879', 'ratio': 0.01}], 'distinctness': 0.05, 'entropy': 0.22339592217896861, 'approximateNumDistinctValues': 5, 'isDataTypeInferred': 'false', 'uniqueness': 0.04, 'mean': 1.0346918630599975, 'maximum': 4.280771732330322, 'stdDev': 0.32640657836797243, 'minimum': 1.000499963760376, 'approxPercentiles': []}
Feature: stdev_amt_per_1h
{'dataType': 'Fractional', 'column': 'stdev_amt_per_1h', 'sum': 120.96687138080597, 'completeness': 1, 'histogram': [{'count': 1, 'value': '1.0512278', 'ratio': 0.01}, {'count': 1, 'value': '1.2470504', 'ratio': 0.01}, {'count': 1, 'value': '1.1666249', 'ratio': 0.01}, {'count': 1, 'value': '1.0686651', 'ratio': 0.01}, {'count': 1, 'value': '1.0186267', 'ratio': 0.01}, {'count': 1, 'value': '1.0060296', 'ratio': 0.01}, {'count': 1, 'value': '2.3883066', 'ratio': 0.01}, {'count': 1, 'value': '1.0785328', 'ratio': 0.01}, {'count': 1, 'value': '1.105774', 'ratio': 0.01}, {'count': 1, 'value': '1.1941308', 'ratio': 0.01}, {'count': 1, 'value': '1.0181073', 'ratio': 0.01}, {'count': 1, 'value': '1.3772634', 'ratio': 0.01}, {'count': 1, 'value': '1.1348182', 'ratio': 0.01}, {'count': 1, 'value': '1.1406308', 'ratio': 0.01}, {'count': 1, 'value': '1.0169014', 'ratio': 0.01}, {'count': 1, 'value': '3.8128898', 'ratio': 0.01}, {'count': 1, 'value': '1.2591444', 'ratio': 0.01}, {'count': 1, 'value': '1.1521285', 'ratio': 0.01}, {'count': 1, 'value': '2.1266055', 'ratio': 0.01}, {'count': 1, 'value': '1.1044518', 'ratio': 0.01}, {'count': 1, 'value': '1.1864266', 'ratio': 0.01}, {'count': 1, 'value': '1.0185984', 'ratio': 0.01}, {'count': 56, 'value': '1.0005', 'ratio': 0.56}, {'count': 1, 'value': '1.0266877', 'ratio': 0.01}, {'count': 1, 'value': '1.2785627', 'ratio': 0.01}, {'count': 1, 'value': '1.0254892', 'ratio': 0.01}, {'count': 1, 'value': '1.3304042', 'ratio': 0.01}, {'count': 1, 'value': '1.0392388', 'ratio': 0.01}, {'count': 1, 'value': '1.0129344', 'ratio': 0.01}, {'count': 1, 'value': '2.3644984', 'ratio': 0.01}, {'count': 1, 'value': '1.010601', 'ratio': 0.01}, {'count': 1, 'value': '1.0314606', 'ratio': 0.01}, {'count': 1, 'value': '1.0275433', 'ratio': 0.01}, {'count': 1, 'value': '3.9862726', 'ratio': 0.01}, {'count': 1, 'value': '1.0159959', 'ratio': 0.01}, {'count': 1, 'value': '1.018503', 'ratio': 0.01}, {'count': 1, 'value': '1.0214374', 'ratio': 0.01}, {'count': 1, 'value': '1.2214001', 'ratio': 0.01}, {'count': 1, 'value': '2.6181552', 'ratio': 0.01}, {'count': 1, 'value': '2.932789', 'ratio': 0.01}, {'count': 1, 'value': '1.019666', 'ratio': 0.01}, {'count': 1, 'value': '1.0103146', 'ratio': 0.01}, {'count': 1, 'value': '1.0183969', 'ratio': 0.01}, {'count': 1, 'value': '3.1567621', 'ratio': 0.01}, {'count': 1, 'value': '3.0988252', 'ratio': 0.01}], 'distinctness': 0.45, 'entropy': 2.3509732391764073, 'approximateNumDistinctValues': 41, 'isDataTypeInferred': 'false', 'uniqueness': 0.44, 'mean': 1.2096687138080597, 'maximum': 3.9862725734710693, 'stdDev': 0.5797821926947725, 'minimum': 1.000499963760376, 'approxPercentiles': []}
Feature: num_trans_per_10m
{'dataType': 'Fractional', 'column': 'num_trans_per_10m', 'sum': 100.10200452804565, 'completeness': 1, 'histogram': [{'count': 4, 'value': '1.0015', 'ratio': 0.04}, {'count': 96, 'value': '1.001', 'ratio': 0.96}], 'distinctness': 0.02, 'entropy': 0.167944147734173, 'approximateNumDistinctValues': 2, 'isDataTypeInferred': 'false', 'uniqueness': 0, 'mean': 1.0010200452804565, 'maximum': 1.0015000104904175, 'stdDev': 9.797248822435412e-05, 'minimum': 1.0010000467300415, 'approxPercentiles': []}
Feature: num_trans_per_12h
{'dataType': 'Fractional', 'column': 'num_trans_per_12h', 'sum': 100.5114996433258, 'completeness': 1, 'histogram': [{'count': 11, 'value': '1.0055', 'ratio': 0.11}, {'count': 7, 'value': '1.0045', 'ratio': 0.07}, {'count': 14, 'value': '1.004', 'ratio': 0.14}, {'count': 1, 'value': '1.0095', 'ratio': 0.01}, {'count': 1, 'value': '1.008', 'ratio': 0.01}, {'count': 2, 'value': '1.007', 'ratio': 0.02}, {'count': 11, 'value': '1.005', 'ratio': 0.11}, {'count': 12, 'value': '1.0065', 'ratio': 0.12}, {'count': 10, 'value': '1.0035', 'ratio': 0.1}, {'count': 1, 'value': '1.002', 'ratio': 0.01}, {'count': 3, 'value': '1.0075', 'ratio': 0.03}, {'count': 3, 'value': '1.0025', 'ratio': 0.03}, {'count': 11, 'value': '1.006', 'ratio': 0.11}, {'count': 3, 'value': '1.0085', 'ratio': 0.03}, {'count': 1, 'value': '1.0015', 'ratio': 0.01}, {'count': 7, 'value': '1.003', 'ratio': 0.07}, {'count': 2, 'value': '1.009', 'ratio': 0.02}], 'distinctness': 0.17, 'entropy': 2.5169209984742564, 'approximateNumDistinctValues': 16, 'isDataTypeInferred': 'false', 'uniqueness': 0.04, 'mean': 1.005114996433258, 'maximum': 1.0095000267028809, 'stdDev': 0.001644469927753096, 'minimum': 1.0015000104904175, 'approxPercentiles': []}
Feature: avg_amt_per_12h
{'dataType': 'Fractional', 'column': 'avg_amt_per_12h', 'sum': 131.92136085033417, 'completeness': 1, 'histogram': [{'count': 1, 'value': '1.0800072', 'ratio': 0.01}, {'count': 1, 'value': '1.224373', 'ratio': 0.01}, {'count': 1, 'value': '1.0785679', 'ratio': 0.01}, {'count': 1, 'value': '1.9062842', 'ratio': 0.01}, {'count': 1, 'value': '1.0294117', 'ratio': 0.01}, {'count': 1, 'value': '1.1111832', 'ratio': 0.01}, {'count': 1, 'value': '1.3303187', 'ratio': 0.01}, {'count': 1, 'value': '1.3356564', 'ratio': 0.01}, {'count': 1, 'value': '1.082731', 'ratio': 0.01}, {'count': 1, 'value': '1.018247', 'ratio': 0.01}, {'count': 1, 'value': '1.6378772', 'ratio': 0.01}, {'count': 1, 'value': '1.0942267', 'ratio': 0.01}, {'count': 1, 'value': '1.6897763', 'ratio': 0.01}, {'count': 1, 'value': '1.358466', 'ratio': 0.01}, {'count': 1, 'value': '1.0573936', 'ratio': 0.01}, {'count': 1, 'value': '1.1732512', 'ratio': 0.01}, {'count': 1, 'value': '1.101762', 'ratio': 0.01}, {'count': 1, 'value': '1.212781', 'ratio': 0.01}, {'count': 1, 'value': '1.0294558', 'ratio': 0.01}, {'count': 1, 'value': '1.0327667', 'ratio': 0.01}, {'count': 1, 'value': '1.5450006', 'ratio': 0.01}, {'count': 1, 'value': '1.185298', 'ratio': 0.01}, {'count': 1, 'value': '1.375337', 'ratio': 0.01}, {'count': 1, 'value': '1.1364158', 'ratio': 0.01}, {'count': 1, 'value': '2.0839994', 'ratio': 0.01}, {'count': 1, 'value': '1.1302421', 'ratio': 0.01}, {'count': 1, 'value': '1.1191986', 'ratio': 0.01}, {'count': 1, 'value': '1.1362562', 'ratio': 0.01}, {'count': 1, 'value': '1.318325', 'ratio': 0.01}, {'count': 1, 'value': '1.4493444', 'ratio': 0.01}, {'count': 1, 'value': '1.01326', 'ratio': 0.01}, {'count': 1, 'value': '1.076624', 'ratio': 0.01}, {'count': 1, 'value': '1.1089278', 'ratio': 0.01}, {'count': 1, 'value': '1.2688416', 'ratio': 0.01}, {'count': 1, 'value': '1.0222229', 'ratio': 0.01}, {'count': 1, 'value': '1.0324908', 'ratio': 0.01}, {'count': 1, 'value': '1.2453928', 'ratio': 0.01}, {'count': 1, 'value': '1.0905614', 'ratio': 0.01}, {'count': 1, 'value': '1.7009267', 'ratio': 0.01}, {'count': 1, 'value': '1.01186', 'ratio': 0.01}, {'count': 1, 'value': '1.2581811', 'ratio': 0.01}, {'count': 1, 'value': '1.4389635', 'ratio': 0.01}, {'count': 1, 'value': '2.105617', 'ratio': 0.01}, {'count': 1, 'value': '1.402288', 'ratio': 0.01}, {'count': 1, 'value': '1.3024898', 'ratio': 0.01}, {'count': 1, 'value': '1.0821495', 'ratio': 0.01}, {'count': 1, 'value': '1.050738', 'ratio': 0.01}, {'count': 1, 'value': '1.6836816', 'ratio': 0.01}, {'count': 1, 'value': '1.3569534', 'ratio': 0.01}, {'count': 1, 'value': '2.0048218', 'ratio': 0.01}, {'count': 1, 'value': '1.5020307', 'ratio': 0.01}, {'count': 1, 'value': '1.3985956', 'ratio': 0.01}, {'count': 1, 'value': '1.6181775', 'ratio': 0.01}, {'count': 1, 'value': '1.9060491', 'ratio': 0.01}, {'count': 1, 'value': '1.109039', 'ratio': 0.01}, {'count': 1, 'value': '1.0637444', 'ratio': 0.01}, {'count': 1, 'value': '1.333153', 'ratio': 0.01}, {'count': 1, 'value': '1.1665922', 'ratio': 0.01}, {'count': 1, 'value': '1.0619062', 'ratio': 0.01}, {'count': 1, 'value': '1.139612', 'ratio': 0.01}, {'count': 1, 'value': '1.63622', 'ratio': 0.01}, {'count': 1, 'value': '1.4508038', 'ratio': 0.01}, {'count': 1, 'value': '1.0632564', 'ratio': 0.01}, {'count': 1, 'value': '1.1547081', 'ratio': 0.01}, {'count': 1, 'value': '1.3129858', 'ratio': 0.01}, {'count': 1, 'value': '1.05302', 'ratio': 0.01}, {'count': 1, 'value': '1.5756481', 'ratio': 0.01}, {'count': 1, 'value': '2.583555', 'ratio': 0.01}, {'count': 1, 'value': '1.0586371', 'ratio': 0.01}, {'count': 1, 'value': '1.2744927', 'ratio': 0.01}, {'count': 1, 'value': '1.1035554', 'ratio': 0.01}, {'count': 1, 'value': '1.5529757', 'ratio': 0.01}, {'count': 1, 'value': '1.052545', 'ratio': 0.01}, {'count': 1, 'value': '1.671495', 'ratio': 0.01}, {'count': 1, 'value': '1.0588467', 'ratio': 0.01}, {'count': 1, 'value': '1.1476625', 'ratio': 0.01}, {'count': 1, 'value': '1.9633837', 'ratio': 0.01}, {'count': 1, 'value': '1.5117222', 'ratio': 0.01}, {'count': 1, 'value': '1.0902817', 'ratio': 0.01}, {'count': 1, 'value': '1.7777075', 'ratio': 0.01}, {'count': 1, 'value': '1.3016328', 'ratio': 0.01}, {'count': 1, 'value': '1.5121435', 'ratio': 0.01}, {'count': 1, 'value': '1.0565587', 'ratio': 0.01}, {'count': 1, 'value': '1.4155555', 'ratio': 0.01}, {'count': 1, 'value': '1.7569113', 'ratio': 0.01}, {'count': 1, 'value': '1.3465623', 'ratio': 0.01}, {'count': 1, 'value': '1.1179966', 'ratio': 0.01}, {'count': 1, 'value': '1.3834157', 'ratio': 0.01}, {'count': 1, 'value': '1.03796', 'ratio': 0.01}, {'count': 1, 'value': '1.062928', 'ratio': 0.01}, {'count': 1, 'value': '1.6132754', 'ratio': 0.01}, {'count': 1, 'value': '1.1268175', 'ratio': 0.01}, {'count': 1, 'value': '1.3831307', 'ratio': 0.01}, {'count': 1, 'value': '1.0567205', 'ratio': 0.01}, {'count': 1, 'value': '1.587175', 'ratio': 0.01}, {'count': 1, 'value': '1.5426328', 'ratio': 0.01}, {'count': 1, 'value': '1.0445257', 'ratio': 0.01}, {'count': 1, 'value': '1.2837173', 'ratio': 0.01}, {'count': 1, 'value': '1.5119296', 'ratio': 0.01}, {'count': 1, 'value': '1.0664245', 'ratio': 0.01}], 'distinctness': 1, 'entropy': 4.605170185988085, 'approximateNumDistinctValues': 97, 'isDataTypeInferred': 'false', 'uniqueness': 1, 'mean': 1.3192136085033417, 'maximum': 2.583554983139038, 'stdDev': 0.30051319257604864, 'minimum': 1.0118600130081177, 'approxPercentiles': []}
Feature: avg_amt_per_1h
{'dataType': 'Fractional', 'column': 'avg_amt_per_1h', 'sum': 132.38604545593262, 'completeness': 1, 'histogram': [{'count': 1, 'value': '2.40911', 'ratio': 0.01}, {'count': 1, 'value': '1.0080475', 'ratio': 0.01}, {'count': 1, 'value': '1.106885', 'ratio': 0.01}, {'count': 1, 'value': '1.0127', 'ratio': 0.01}, {'count': 1, 'value': '1.002485', 'ratio': 0.01}, {'count': 1, 'value': '1.1340275', 'ratio': 0.01}, {'count': 1, 'value': '1.189925', 'ratio': 0.01}, {'count': 1, 'value': '1.018255', 'ratio': 0.01}, {'count': 1, 'value': '1.151655', 'ratio': 0.01}, {'count': 1, 'value': '1.020265', 'ratio': 0.01}, {'count': 1, 'value': '1.0463924', 'ratio': 0.01}, {'count': 1, 'value': '1.032555', 'ratio': 0.01}, {'count': 1, 'value': '1.015035', 'ratio': 0.01}, {'count': 1, 'value': '1.0274175', 'ratio': 0.01}, {'count': 1, 'value': '1.046075', 'ratio': 0.01}, {'count': 1, 'value': '3.235095', 'ratio': 0.01}, {'count': 1, 'value': '1.01423', 'ratio': 0.01}, {'count': 1, 'value': '1.040795', 'ratio': 0.01}, {'count': 1, 'value': '2.5441875', 'ratio': 0.01}, {'count': 1, 'value': '1.0206333', 'ratio': 0.01}, {'count': 1, 'value': '1.261845', 'ratio': 0.01}, {'count': 1, 'value': '1.03678', 'ratio': 0.01}, {'count': 1, 'value': '1.35051', 'ratio': 0.01}, {'count': 1, 'value': '5.18205', 'ratio': 0.01}, {'count': 1, 'value': '1.08986', 'ratio': 0.01}, {'count': 1, 'value': '3.597025', 'ratio': 0.01}, {'count': 1, 'value': '1.04728', 'ratio': 0.01}, {'count': 1, 'value': '1.03759', 'ratio': 0.01}, {'count': 1, 'value': '1.034705', 'ratio': 0.01}, {'count': 1, 'value': '1.8659483', 'ratio': 0.01}, {'count': 1, 'value': '1.0372375', 'ratio': 0.01}, {'count': 1, 'value': '1.016445', 'ratio': 0.01}, {'count': 1, 'value': '2.48497', 'ratio': 0.01}, {'count': 1, 'value': '1.02899', 'ratio': 0.01}, {'count': 1, 'value': '1.0226876', 'ratio': 0.01}, {'count': 1, 'value': '1.04702', 'ratio': 0.01}, {'count': 1, 'value': '1.0172175', 'ratio': 0.01}, {'count': 1, 'value': '1.2194575', 'ratio': 0.01}, {'count': 1, 'value': '1.14429', 'ratio': 0.01}, {'count': 1, 'value': '1.2094333', 'ratio': 0.01}, {'count': 1, 'value': '1.033145', 'ratio': 0.01}, {'count': 1, 'value': '1.2428775', 'ratio': 0.01}, {'count': 1, 'value': '1.01006', 'ratio': 0.01}, {'count': 1, 'value': '1.02391', 'ratio': 0.01}, {'count': 1, 'value': '1.02716', 'ratio': 0.01}, {'count': 1, 'value': '1.206005', 'ratio': 0.01}, {'count': 1, 'value': '1.004125', 'ratio': 0.01}, {'count': 1, 'value': '2.577445', 'ratio': 0.01}, {'count': 1, 'value': '1.035445', 'ratio': 0.01}, {'count': 1, 'value': '1.00495', 'ratio': 0.01}, {'count': 1, 'value': '1.02109', 'ratio': 0.01}, {'count': 1, 'value': '1.10703', 'ratio': 0.01}, {'count': 1, 'value': '1.039485', 'ratio': 0.01}, {'count': 1, 'value': '1.00059', 'ratio': 0.01}, {'count': 1, 'value': '1.05304', 'ratio': 0.01}, {'count': 1, 'value': '1.00381', 'ratio': 0.01}, {'count': 1, 'value': '2.68918', 'ratio': 0.01}, {'count': 1, 'value': '1.1309226', 'ratio': 0.01}, {'count': 1, 'value': '1.221885', 'ratio': 0.01}, {'count': 1, 'value': '1.0371475', 'ratio': 0.01}, {'count': 1, 'value': '1.021155', 'ratio': 0.01}, {'count': 1, 'value': '1.3620833', 'ratio': 0.01}, {'count': 1, 'value': '1.27822', 'ratio': 0.01}, {'count': 1, 'value': '1.12946', 'ratio': 0.01}, {'count': 1, 'value': '3.13422', 'ratio': 0.01}, {'count': 1, 'value': '1.180885', 'ratio': 0.01}, {'count': 1, 'value': '1.62963', 'ratio': 0.01}, {'count': 1, 'value': '1.311395', 'ratio': 0.01}, {'count': 1, 'value': '1.08343', 'ratio': 0.01}, {'count': 1, 'value': '1.01898', 'ratio': 0.01}, {'count': 1, 'value': '1.9659175', 'ratio': 0.01}, {'count': 1, 'value': '1.020295', 'ratio': 0.01}, {'count': 1, 'value': '1.01963', 'ratio': 0.01}, {'count': 1, 'value': '1.1165', 'ratio': 0.01}, {'count': 1, 'value': '1.0504625', 'ratio': 0.01}, {'count': 1, 'value': '1.09707', 'ratio': 0.01}, {'count': 1, 'value': '1.0247675', 'ratio': 0.01}, {'count': 1, 'value': '2.0242176', 'ratio': 0.01}, {'count': 1, 'value': '1.045685', 'ratio': 0.01}, {'count': 1, 'value': '1.1408975', 'ratio': 0.01}, {'count': 1, 'value': '1.043165', 'ratio': 0.01}, {'count': 1, 'value': '1.168355', 'ratio': 0.01}, {'count': 1, 'value': '1.2778575', 'ratio': 0.01}, {'count': 1, 'value': '2.65805', 'ratio': 0.01}, {'count': 1, 'value': '1.2385', 'ratio': 0.01}, {'count': 1, 'value': '1.01753', 'ratio': 0.01}, {'count': 1, 'value': '1.006585', 'ratio': 0.01}, {'count': 1, 'value': '1.00862', 'ratio': 0.01}, {'count': 1, 'value': '1.04812', 'ratio': 0.01}, {'count': 1, 'value': '1.017795', 'ratio': 0.01}, {'count': 1, 'value': '1.398705', 'ratio': 0.01}, {'count': 1, 'value': '1.030905', 'ratio': 0.01}, {'count': 1, 'value': '2.07267', 'ratio': 0.01}, {'count': 1, 'value': '1.0252367', 'ratio': 0.01}, {'count': 1, 'value': '1.001415', 'ratio': 0.01}, {'count': 1, 'value': '1.17003', 'ratio': 0.01}, {'count': 1, 'value': '1.0416', 'ratio': 0.01}, {'count': 1, 'value': '1.1626', 'ratio': 0.01}, {'count': 1, 'value': '1.023965', 'ratio': 0.01}, {'count': 1, 'value': '1.043025', 'ratio': 0.01}], 'distinctness': 1, 'entropy': 4.605170185988084, 'approximateNumDistinctValues': 99, 'isDataTypeInferred': 'false', 'uniqueness': 1, 'mean': 1.3238604545593262, 'maximum': 5.1820502281188965, 'stdDev': 0.6606924539531908, 'minimum': 1.0005899667739868, 'approxPercentiles': []}
Feature: stdev_amt_per_12h
{'dataType': 'Fractional', 'column': 'stdev_amt_per_12h', 'sum': 161.8382396697998, 'completeness': 1, 'histogram': [{'count': 1, 'value': '1.8019531', 'ratio': 0.01}, {'count': 1, 'value': '2.803783', 'ratio': 0.01}, {'count': 1, 'value': '1.1666672', 'ratio': 0.01}, {'count': 1, 'value': '2.1051917', 'ratio': 0.01}, {'count': 1, 'value': '1.8459213', 'ratio': 0.01}, {'count': 1, 'value': '2.4104125', 'ratio': 0.01}, {'count': 1, 'value': '1.6034094', 'ratio': 0.01}, {'count': 1, 'value': '2.4050531', 'ratio': 0.01}, {'count': 1, 'value': '2.3270547', 'ratio': 0.01}, {'count': 1, 'value': '1.6114789', 'ratio': 0.01}, {'count': 1, 'value': '1.1411338', 'ratio': 0.01}, {'count': 1, 'value': '1.8494561', 'ratio': 0.01}, {'count': 1, 'value': '1.7284646', 'ratio': 0.01}, {'count': 1, 'value': '1.1920608', 'ratio': 0.01}, {'count': 1, 'value': '2.3982677', 'ratio': 0.01}, {'count': 1, 'value': '1.8245932', 'ratio': 0.01}, {'count': 1, 'value': '2.5743628', 'ratio': 0.01}, {'count': 1, 'value': '1.171347', 'ratio': 0.01}, {'count': 1, 'value': '1.5216314', 'ratio': 0.01}, {'count': 1, 'value': '3.7083027', 'ratio': 0.01}, {'count': 1, 'value': '1.5772926', 'ratio': 0.01}, {'count': 1, 'value': '1.9620116', 'ratio': 0.01}, {'count': 1, 'value': '1.0174568', 'ratio': 0.01}, {'count': 1, 'value': '1.026732', 'ratio': 0.01}, {'count': 1, 'value': '2.4732418', 'ratio': 0.01}, {'count': 1, 'value': '1.0785458', 'ratio': 0.01}, {'count': 1, 'value': '1.0439519', 'ratio': 0.01}, {'count': 1, 'value': '1.1426051', 'ratio': 0.01}, {'count': 1, 'value': '1.0761116', 'ratio': 0.01}, {'count': 1, 'value': '1.0640783', 'ratio': 0.01}, {'count': 1, 'value': '1.0908353', 'ratio': 0.01}, {'count': 1, 'value': '2.2777712', 'ratio': 0.01}, {'count': 1, 'value': '1.1538386', 'ratio': 0.01}, {'count': 1, 'value': '1.9687029', 'ratio': 0.01}, {'count': 1, 'value': '1.1172085', 'ratio': 0.01}, {'count': 1, 'value': '1.1493987', 'ratio': 0.01}, {'count': 1, 'value': '1.1760855', 'ratio': 0.01}, {'count': 1, 'value': '2.2774072', 'ratio': 0.01}, {'count': 1, 'value': '1.0697409', 'ratio': 0.01}, {'count': 1, 'value': '1.2650838', 'ratio': 0.01}, {'count': 1, 'value': '1.1520736', 'ratio': 0.01}, {'count': 1, 'value': '2.0978363', 'ratio': 0.01}, {'count': 1, 'value': '1.9432347', 'ratio': 0.01}, {'count': 1, 'value': '1.0634649', 'ratio': 0.01}, {'count': 1, 'value': '2.3441398', 'ratio': 0.01}, {'count': 1, 'value': '1.0057397', 'ratio': 0.01}, {'count': 1, 'value': '1.1633795', 'ratio': 0.01}, {'count': 1, 'value': '1.1272135', 'ratio': 0.01}, {'count': 1, 'value': '2.3324296', 'ratio': 0.01}, {'count': 1, 'value': '1.0225426', 'ratio': 0.01}, {'count': 1, 'value': '1.052131', 'ratio': 0.01}, {'count': 1, 'value': '2.3587024', 'ratio': 0.01}, {'count': 1, 'value': '1.1000363', 'ratio': 0.01}, {'count': 1, 'value': '1.647339', 'ratio': 0.01}, {'count': 1, 'value': '1.115488', 'ratio': 0.01}, {'count': 1, 'value': '1.1288215', 'ratio': 0.01}, {'count': 1, 'value': '1.1124095', 'ratio': 0.01}, {'count': 1, 'value': '2.5451388', 'ratio': 0.01}, {'count': 1, 'value': '1.1777302', 'ratio': 0.01}, {'count': 1, 'value': '2.3027701', 'ratio': 0.01}, {'count': 1, 'value': '1.1534002', 'ratio': 0.01}, {'count': 1, 'value': '2.1149588', 'ratio': 0.01}, {'count': 1, 'value': '1.6962408', 'ratio': 0.01}, {'count': 1, 'value': '2.5884216', 'ratio': 0.01}, {'count': 1, 'value': '1.723834', 'ratio': 0.01}, {'count': 1, 'value': '2.234932', 'ratio': 0.01}, {'count': 1, 'value': '2.3698418', 'ratio': 0.01}, {'count': 1, 'value': '1.9768231', 'ratio': 0.01}, {'count': 1, 'value': '1.0877242', 'ratio': 0.01}, {'count': 1, 'value': '1.156441', 'ratio': 0.01}, {'count': 1, 'value': '1.8140973', 'ratio': 0.01}, {'count': 1, 'value': '1.6870944', 'ratio': 0.01}, {'count': 1, 'value': '1.0999271', 'ratio': 0.01}, {'count': 1, 'value': '1.1455579', 'ratio': 0.01}, {'count': 1, 'value': '1.0127324', 'ratio': 0.01}, {'count': 1, 'value': '1.5816444', 'ratio': 0.01}, {'count': 1, 'value': '1.0111563', 'ratio': 0.01}, {'count': 1, 'value': '2.2827802', 'ratio': 0.01}, {'count': 1, 'value': '1.1831121', 'ratio': 0.01}, {'count': 1, 'value': '2.0209146', 'ratio': 0.01}, {'count': 1, 'value': '1.0338311', 'ratio': 0.01}, {'count': 1, 'value': '1.064192', 'ratio': 0.01}, {'count': 1, 'value': '1.1652536', 'ratio': 0.01}, {'count': 1, 'value': '1.4046191', 'ratio': 0.01}, {'count': 1, 'value': '2.3232758', 'ratio': 0.01}, {'count': 1, 'value': '1.7027466', 'ratio': 0.01}, {'count': 1, 'value': '1.1275098', 'ratio': 0.01}, {'count': 1, 'value': '2.6044703', 'ratio': 0.01}, {'count': 1, 'value': '1.1268175', 'ratio': 0.01}, {'count': 1, 'value': '1.7150314', 'ratio': 0.01}, {'count': 1, 'value': '1.6744218', 'ratio': 0.01}, {'count': 1, 'value': '2.3715672', 'ratio': 0.01}, {'count': 1, 'value': '1.1484296', 'ratio': 0.01}, {'count': 1, 'value': '1.1338341', 'ratio': 0.01}, {'count': 1, 'value': '1.1629473', 'ratio': 0.01}, {'count': 1, 'value': '2.293469', 'ratio': 0.01}, {'count': 1, 'value': '2.3112187', 'ratio': 0.01}, {'count': 1, 'value': '1.3514063', 'ratio': 0.01}, {'count': 1, 'value': '1.1651056', 'ratio': 0.01}, {'count': 1, 'value': '1.0251925', 'ratio': 0.01}], 'distinctness': 1, 'entropy': 4.6051701859880865, 'approximateNumDistinctValues': 94, 'isDataTypeInferred': 'false', 'uniqueness': 1, 'mean': 1.618382396697998, 'maximum': 3.7083027362823486, 'stdDev': 0.575247446872259, 'minimum': 1.005739688873291, 'approxPercentiles': []}
Feature: avg_amt_per_10m
{'dataType': 'Fractional', 'column': 'avg_amt_per_10m', 'sum': 133.25509345531464, 'completeness': 1, 'histogram': [{'count': 1, 'value': '1.03886', 'ratio': 0.01}, {'count': 1, 'value': '1.382755', 'ratio': 0.01}, {'count': 1, 'value': '1.02562', 'ratio': 0.01}, {'count': 1, 'value': '1.04755', 'ratio': 0.01}, {'count': 1, 'value': '1.032345', 'ratio': 0.01}, {'count': 1, 'value': '1.04332', 'ratio': 0.01}, {'count': 1, 'value': '1.044605', 'ratio': 0.01}, {'count': 1, 'value': '1.0240825', 'ratio': 0.01}, {'count': 1, 'value': '1.034525', 'ratio': 0.01}, {'count': 1, 'value': '1.75089', 'ratio': 0.01}, {'count': 1, 'value': '1.00093', 'ratio': 0.01}, {'count': 1, 'value': '1.0099', 'ratio': 0.01}, {'count': 1, 'value': '1.04687', 'ratio': 0.01}, {'count': 1, 'value': '1.19041', 'ratio': 0.01}, {'count': 1, 'value': '1.021545', 'ratio': 0.01}, {'count': 1, 'value': '4.3122', 'ratio': 0.01}, {'count': 1, 'value': '1.98826', 'ratio': 0.01}, {'count': 1, 'value': '3.430245', 'ratio': 0.01}, {'count': 1, 'value': '1.30523', 'ratio': 0.01}, {'count': 1, 'value': '1.05099', 'ratio': 0.01}, {'count': 1, 'value': '1.018075', 'ratio': 0.01}, {'count': 1, 'value': '1.45408', 'ratio': 0.01}, {'count': 1, 'value': '1.010845', 'ratio': 0.01}, {'count': 1, 'value': '1.07717', 'ratio': 0.01}, {'count': 1, 'value': '1.023185', 'ratio': 0.01}, {'count': 1, 'value': '1.04164', 'ratio': 0.01}, {'count': 1, 'value': '1.030305', 'ratio': 0.01}, {'count': 1, 'value': '1.30727', 'ratio': 0.01}, {'count': 1, 'value': '1.000555', 'ratio': 0.01}, {'count': 1, 'value': '1.005625', 'ratio': 0.01}, {'count': 1, 'value': '1.01566', 'ratio': 0.01}, {'count': 1, 'value': '1.002315', 'ratio': 0.01}, {'count': 1, 'value': '1.048055', 'ratio': 0.01}, {'count': 1, 'value': '1.01661', 'ratio': 0.01}, {'count': 1, 'value': '1.04681', 'ratio': 0.01}, {'count': 1, 'value': '1.000845', 'ratio': 0.01}, {'count': 1, 'value': '1.04419', 'ratio': 0.01}, {'count': 1, 'value': '3.04391', 'ratio': 0.01}, {'count': 1, 'value': '1.03907', 'ratio': 0.01}, {'count': 1, 'value': '1.01528', 'ratio': 0.01}, {'count': 1, 'value': '1.255555', 'ratio': 0.01}, {'count': 1, 'value': '1.00766', 'ratio': 0.01}, {'count': 1, 'value': '1.00088', 'ratio': 0.01}, {'count': 1, 'value': '1.103115', 'ratio': 0.01}, {'count': 1, 'value': '1.013595', 'ratio': 0.01}, {'count': 1, 'value': '1.02465', 'ratio': 0.01}, {'count': 1, 'value': '1.32599', 'ratio': 0.01}, {'count': 1, 'value': '1.02506', 'ratio': 0.01}, {'count': 1, 'value': '1.47876', 'ratio': 0.01}, {'count': 1, 'value': '1.001085', 'ratio': 0.01}, {'count': 1, 'value': '1.437465', 'ratio': 0.01}, {'count': 1, 'value': '1.044715', 'ratio': 0.01}, {'count': 1, 'value': '1.35467', 'ratio': 0.01}, {'count': 1, 'value': '1.05097', 'ratio': 0.01}, {'count': 1, 'value': '1.033705', 'ratio': 0.01}, {'count': 1, 'value': '3.3391125', 'ratio': 0.01}, {'count': 1, 'value': '1.005665', 'ratio': 0.01}, {'count': 1, 'value': '1.139815', 'ratio': 0.01}, {'count': 1, 'value': '1.00245', 'ratio': 0.01}, {'count': 1, 'value': '1.015125', 'ratio': 0.01}, {'count': 1, 'value': '1.00125', 'ratio': 0.01}, {'count': 1, 'value': '1.031375', 'ratio': 0.01}, {'count': 1, 'value': '1.02627', 'ratio': 0.01}, {'count': 1, 'value': '1.343955', 'ratio': 0.01}, {'count': 1, 'value': '1.010665', 'ratio': 0.01}, {'count': 1, 'value': '1.00189', 'ratio': 0.01}, {'count': 1, 'value': '1.654585', 'ratio': 0.01}, {'count': 1, 'value': '1.357625', 'ratio': 0.01}, {'count': 1, 'value': '1.006885', 'ratio': 0.01}, {'count': 1, 'value': '1.007555', 'ratio': 0.01}, {'count': 1, 'value': '1.00105', 'ratio': 0.01}, {'count': 1, 'value': '1.039175', 'ratio': 0.01}, {'count': 1, 'value': '1.448285', 'ratio': 0.01}, {'count': 1, 'value': '1.000665', 'ratio': 0.01}, {'count': 1, 'value': '1.23047', 'ratio': 0.01}, {'count': 1, 'value': '2.4998', 'ratio': 0.01}, {'count': 1, 'value': '1.02681', 'ratio': 0.01}, {'count': 1, 'value': '1.377555', 'ratio': 0.01}, {'count': 1, 'value': '1.047155', 'ratio': 0.01}, {'count': 1, 'value': '1.030375', 'ratio': 0.01}, {'count': 1, 'value': '1.034375', 'ratio': 0.01}, {'count': 1, 'value': '3.48816', 'ratio': 0.01}, {'count': 1, 'value': '1.00264', 'ratio': 0.01}, {'count': 1, 'value': '1.04393', 'ratio': 0.01}, {'count': 1, 'value': '1.046425', 'ratio': 0.01}, {'count': 1, 'value': '1.00715', 'ratio': 0.01}, {'count': 1, 'value': '1.00108', 'ratio': 0.01}, {'count': 1, 'value': '1.09949', 'ratio': 0.01}, {'count': 1, 'value': '1.018315', 'ratio': 0.01}, {'count': 1, 'value': '1.36298', 'ratio': 0.01}, {'count': 1, 'value': '1.043745', 'ratio': 0.01}, {'count': 1, 'value': '1.046455', 'ratio': 0.01}, {'count': 1, 'value': '1.000745', 'ratio': 0.01}, {'count': 1, 'value': '1.02304', 'ratio': 0.01}, {'count': 1, 'value': '2.00156', 'ratio': 0.01}, {'count': 1, 'value': '5.254005', 'ratio': 0.01}, {'count': 1, 'value': '1.38861', 'ratio': 0.01}, {'count': 1, 'value': '1.10394', 'ratio': 0.01}, {'count': 1, 'value': '4.444815', 'ratio': 0.01}, {'count': 1, 'value': '1.0175675', 'ratio': 0.01}], 'distinctness': 1, 'entropy': 4.6051701859880865, 'approximateNumDistinctValues': 99, 'isDataTypeInferred': 'false', 'uniqueness': 1, 'mean': 1.3325509345531463, 'maximum': 5.254004955291748, 'stdDev': 0.769244159280294, 'minimum': 1.0005550384521484, 'approxPercentiles': []}