split_data
Description
Splitting the preprocessed data into training, validation, and testing sets with regard to user preference. The function allows temporal splitting of the data, wherein the test set is selected from the last temporal units in the data, and a number of previous temporal units of the test set (here denoted by the Gap) is excluded from the data to simulate the situation of real prediction in which we do not have access to the information of the forecast_horizon - 1 units before the time point of the target variable. Fig. 2.a shows the temporal splitting of a sample preprocessed data. The cross validation method is also supported in the function, which is depicted in Fig. 2.b.
Usage
- predict.split_data(data, splitting_type='instance', instance_testing_size=None, instance_validation_size=None, instance_random_partitioning=False, fold_total_number=None, fold_number=None, forecast_horizon=1, granularity=1, verbose=0)
Parameters
# |
Input Name |
Input Description |
|---|---|---|
1
|
data
|
type: dataframe or string
default: -
details: a data frame or address of a data frame containing
preprocessed data. This data frame must have a column name format
conforming to Fig. 5.
|
2
|
splitting_type
|
type: {‘instance’, ‘fold’}
default: ‘instance’
details: type of splitting
if ‘instance’, the splitting is performed based on
instace_testing_size, instance_validation_size, and
instance_random_partitioning
if ‘fold’, the splitting is performed based on fold_total_number and
fold_number
|
3
|
instance_testing_size
|
type: int or float or None
default: None
details: The number of temporal units to be considered as testing
set
If int, is considered as the number of temporal units to be included
in the testing set
If float, is considered as the proportion of temporal units in the
dataset to be included in the testing split, thus should be between
0.0 and 1.0.
this input is considered if the splitting_type input is ‘instance’
example: 7 or 0.3
|
4
|
instance_validation
_size
|
type: int or float or None
default: None
details: The number of temporal units to be considered as a
validation set.
If int, is considered as the number of temporal units to be included
in the validation set.
If float, is considered as the proportion of temporal units in the
dataset to be included in the validation split, thus should be between
0.0 and 1.0.
This input is considered if the splitting_type input is ‘instance’.
Note that if the instance_testing_size have value, and
instance_validation_size is of type float, it is considered as the
proportion of the temporal units in the remaining data after removing
the testing set.
example: 7 or 0.3
|
5
|
instance_random
_partitioning
|
type: bool
default: False
details: type of partitioning the validation data
If False, the validation part will be selected from the last temporal
units in the data, otherwise the temporal units of validation part are
selected randomly from the recorded temporal units in the data.
this input is considered if the splitting_type input is ‘instance’
|
6
|
fold_total_number
|
type: int or None
default: None
details: total number of folds
If int, is considered as the total number of folds in the cross
validation method of partitioning
this input is considered if the splitting_type input is ‘fold’
|
7
|
fold_number
|
type: int or None
default: None
details: the fold number which validation data will be selected
from
this input is considered if the splitting_type input is ‘fold’
|
8
|
forecast_horizon
|
type: int
default: 1
details: forecast horizon to gap consideration in data splitting
process
By the gap, we mean the number of temporal units which are excluded
from data to simulate the situation of real prediction in which we do
not have access to the information of the forecast horizon - 1 units
before the time point of the target variable.
|
9
|
granularity
|
type: int
default: 1
details: If the augmentation is applied in the transformation of
temporal scale in the data_preprocess function, the number of smaller
scale units in a bigger scale unit (granularity) is needed for
detecting the sequence of the temporal units in the input data. If
augmentation is not applied, the granularity must be set to one.
|
10
|
verbose
|
type: int
default: 0
details: The level of details in produced logging information
available options:
0: no logging
1: only important information logging
|
Returns
# |
Output Name |
Output Description |
|---|---|---|
1
|
training_data
|
type: data frame
details: the training part of data after splitting process
|
2
|
validation_data
|
type: data frame or None
details: the validation part of data after splitting process. If
splitting_type is ‘instance’ and input_validation_size is not
specified, None is returned.
|
3
|
testing_data
|
type: data frame or None
details: The testing part of data after splitting process. If
splitting_type is ‘instance’ and input_testing_size is not
specified, None is returned.
|
4
|
gap_data
|
type: data frame or None
details: The data on temporal units between the part of data used
for fit and evaluate the model (training + validation) and testing set
which are excluded from data to simulate the situation of real
prediction in which we do not have access to the information of
forecast horizon - 1 units before the time point of the target
variable.
If instance_testing_size input is 0 this part is not considered and
None is returned.
|
Example
import pandas as pd
from stpredict.predict import split_data
df = pd.read_csv('./historical_data h=1.csv')
training_data, validation_data, testing_data, gap_data = split_data(data = df,
splitting_type = 'instance', instance_testing_size = 0.2,
instance_validation_size = 0.3, forecast_horizon = 2)
Fig. 2 Different types of data splitting. (a) Temporal splitting of a sample preprocessed data with forecast_horizon = 2, and instance_testing_size = 2, (b) Cross validation with fold_total_number = 3, fold_number = 2.