Question Regarding Genetic Feature Selection

[boguskiP]

Is there a simple/built-in method of ensuring that if you extract 2+ subsets from the same dataset (using Genetic Feature Selection), that the features found in the first subset are not found in another subset? I.e., Obtain a series of series of subsets that are "symmetrically different" mathematically.

I'm wondering since I want to have each subset have it's own 'branch' before coming all together and the presence of duplicated features makes understanding the 'feature importance' more difficult. Just asking to ensure I'm not coding for something that exists but I couldn't find it.

[perib]

There are two options for doing evolutionary feature selection in TPOT (which you can learn more about in Tutorial 4: https://github.com/EpistasisLab/tpot/blob/main/Tutorial/4_Genetic_Feature_Selection.ipynb)
Each FSSNode node picks one predefined subset of features. And GeneticFeatureSelectorNode gets a list of features and dynamically learns any subset of those features per node. A single GeneticFeatureSelectorNode will never select the same feature twice.

So for instance FSSNode you pass in
"subset a": [ 1,2,3],
"subset b": [ 5,6,7],
"subset c": [ 8,9,4],

And it will pick one of the three sets.

For GeneticFeatureSelectorNode, you pass in all features ([0,1,2,3,4,5,6,7,8,9]) and it will pick any subset. for example [1] or [3, 5, 7,8,9] or [6, 8].

Where you position these in your pipeline determines whether or not repeat features are selected. Here are a few examples using the FFSNode, but the same idea applied to the GeneticFeatureSelectorNode.

With the FSSNode, each node will select one of the predefined subsets of features. When using graph pipelines, TPOT will never actually create two duplicates of the same feature in a given pipeline. If crossover or mutation yields a pipeline with say 2 feature set B nodes in the leaves, those will actually be merged into one node with two (or more) outputs to their corresponding nodes. So repeat features wouldn't occur. (This may not work exactly with the GeneticFeatureSelectorNode since there is much more diversity in possible features and they have to exactly match to be merged. When using graph pipelines, you should not use GeneticFeatureSelectorNode in the leaves. Instead you should either use FFSNore in the leaves or use a sequential pipeline of a GeneticFeatureSelectorNode followed by a graphpipeline)

If you want to have example two branches, you could set up a search space such that each branch searches an independent set of features. (see tutorial 2 for more info https://github.com/EpistasisLab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb)

As an example outline (in pseudo-code):

branch1 = SequentialPipeline([ 
     FFSNode( <first subset of features>), 
     sometransformerpipeline 
])

branch2 = SequentialPipeline([ 
     FFSNode( <second subset of features>), 
     sometransformerpipeline 
])

final_pipeline = SequentialPipeline([
       UnionPipeline([branch1, branch2 ]),
       classification_search_space 
)]

This pipeline guarantees two branches, each with one feature set, and a transformer. The transformed subsets get concatenated by UnionPipeline and passed to the final classifier. If the feature sets in each branch are not overlapping, then you won't have repeated features.

You can also modify this in several ways. By using a DynamicUnionPipeline with multiple branches, TPOT will select between 1 and N branches, essentially deciding whether to use one or more feature sets.

Should also note that when using a DynamicUnionPipeline, TPOT will never select two pipelines that are exactly the same. So if you start your pipeline with:

 SequentialPipeline([
       DynamicUnionPipeline([FSS_search_space ]),
       classification_search_space 
)]

TPOT can dynamically select 1 or more feature sets, but it will never choose the same one twice. (TPOT matches based on the name of the feature set, so as long as you do not have features that overlap between your predetermined sets, you wouldn't have repeat features.

Hope this helps! Let me know if you have more questions

[perib]

perhaps some of this information could be made clearer in the tutorials and documentation. It might not be immediately obvious that TPOT does this kind of merging of identical pipelines in the background. In fact, it doesn't just merge identical nodes, but it checks if entire pipelines are identical and can be merged.

For example:

branch = SequentialPipeline([ 
     FFSNode( <all subsets of features>), 
     select_transformer_node,
])

 SequentialPipeline([
       DynamicUnionPipeline([branch ]),
       classification_search_space 
)]

This pipeline will create a set of 1 or more transformed subsets. But by default DynamicUnionPipeline will only merge pipelines that are exactly identical - so the same exact feature set with the same exact transformer. So in this case it is possible to have the same feature set show up twice in the pipeline, however, each will have a unique transformation.

To merge feature sets in this situation will require a graph structure, which would be incompatible with future mutation/crossovers. However, there is a kinda hidden feature which is the export_flattened_graphpipeline of the SklearnIndividual class. This converts all final pipelines to a graph pipeline, and in this case the identical leave nodes can be merged together to create an identical but simpler pipeline. So rather than two branches with the same feature set. one feature set will point to two transformers.

[boguskiP]

Thanks for the speedy reply! It'll take time to mull this information over but it helped a lot.

[boguskiP]

Okay, I have a new question that is slightly different but still relates to the topic.

Is there a way to ensure one branch of a Union can only crossover with the "same" corresponding branch of another individual. I.e., I wish to make it so that branch-1 of individual A could also crossover with branch-1 of an individual that is not A. Another way to think of it is that each branch is treated like a chromosome and you only want to swap with the corresponding chromosome.

I'm asking this because of the pipeline I've constructed has resulted in this error after the first generation:
File "/scratch2/users/mpb703/conda_environments/tpot_v1/lib/python3.10/site-packages/tpot/search_spaces/nodes/fss_node.py", line 113, in mutate self.selected_subset_name = rng.choice(names) File "numpy/random/_generator.pyx", line 803, in numpy.random._generator.Generator.choice ValueError: a cannot be empty unless no samples are taken
And the only reason why this could be happening is that the crossover process is swapping branches that are not corresponding due to how I've set them up.

Of my N branches, I've set up (N-1) branches in such a way to simulate what I wanted from the original question of the thread. Choice between ("FSSNode takes in ONE subset only" -> "GFS selects the features" -> "Augmentation of the data") |OR| ("SkipTransformer") to ensure that no features of this subset are selected is an option. The reason as to only N-1 branches is to ensure that there's at least one sample going through to stop the error since it stops the fitting process entirely.

Setting the crossover_rate of the GFSNodes to 0 did not help since the Sequential pipeline has it's own crossover with other sequential pipelines. If I set the crossover_rate of the Sequential pipelines to 0, then the branches will never crossover which sort of defeats the purpose of using a genetic pipeline.

[perib]

A Union should only crossover the same branch in the same location. For example if you have UnionPipeline([Branch1, Branch2]), the search space of Branch1 will never crossover with Branch2. Crossover only happens at the same index of both pipelines. So it will either do b1.crossover(b1) or b2.crossover(b2). see: https://github.com/EpistasisLab/tpot/blob/main/tpot/search_spaces/pipelines/union.py#L82

If the crossover rate for an inner search space is 0, but the outer one is non-zero, TPOT should choose to perform a crossover operation other than the one set to zero. For example a sequential pipeline could chose to swap its sequence of nodes with another sequential pipeline, but it would skip FSSNode.crossover(FSSNode) if the rate of that function is 0.

can you share the code you used to construct the search space and the FFSnode? If you do it would help me better understand the problem. In this error it looks like the issue is that the list of names is empty, perhaps there was an issue passing in the parameters to FSSNode?

[boguskiP]

Here's the construction of the entire pipeline. I will note that I am also trying to run a VERY basic linear pipeline as a choice, to see if the simpler approach is more fruitful than the complex method I'm testing out. My guess is that I'm missing the inclusion of setting crossover_rate to 0 somewhere.

X_sense = X[['dnaSenseStrand', 'teSenseStrand']]
X_length = X[['teQLength', 'geneLength', 'teRMLength','teQRMLeft']]
X_dist = X[['ssGeneTEDist', 'seGeneTEDist', 'esGeneTEDist', 'eeGeneTEDist', 'meanGeneTEDist','teQRMStart', 'teQRMEnd']]
X_score = X[['Smth0Wtrmn', 'teDvrg', 'teDel', 'teInsrt']]
X_type = X[['teClass', 'teFamily']]
X_misc = X[['teSplitIDCount', 'teTotality','geneTeOvrlp']]

subset_1_dict  = { "sense_strands" :  list(X_sense.columns) }
subset_2_dict  = { "lengths"       :  list(X_length.columns)}
subset_3_dict  = { "distances"     :  list(X_dist.columns)  }
subset_4_dict  = { "te_scores"     :  list(X_score.columns) }
subset_5_dict  = { "te_types"      :  list(X_type.columns)  }
subset_6_dict  = { "misc"          :  list(X_misc.columns)  }

fss_1 = FSSNode(subsets=subset_1_dict)
fss_2 = FSSNode(subsets=subset_2_dict)
fss_3 = FSSNode(subsets=subset_3_dict)
fss_4 = FSSNode(subsets=subset_4_dict)
fss_5 = FSSNode(subsets=subset_5_dict)
fss_6 = FSSNode(subsets=subset_6_dict)

single_GFSelector   = GeneticFeatureSelectorNode(n_features=X.shape[1],        crossover_rate=0, mutation_rate=1.75)
subset_1_GFSelector = GeneticFeatureSelectorNode(n_features=X_sense.shape[1],  crossover_rate=0, mutation_rate=1.75)
subset_2_GFSelector = GeneticFeatureSelectorNode(n_features=X_length.shape[1], crossover_rate=0, mutation_rate=1.75)
subset_3_GFSelector = GeneticFeatureSelectorNode(n_features=X_dist.shape[1],   crossover_rate=0, mutation_rate=1.75)
subset_4_GFSelector = GeneticFeatureSelectorNode(n_features=X_score.shape[1],  crossover_rate=0, mutation_rate=1.75)
subset_5_GFSelector = GeneticFeatureSelectorNode(n_features=X_type.shape[1],   crossover_rate=0, mutation_rate=1.75)
subset_6_GFSelector = GeneticFeatureSelectorNode(n_features=X_misc.shape[1],   crossover_rate=0, mutation_rate=1.75)

scale_space = get_search_space("scalers")
pass_space = get_search_space("Passthrough")
dynamic_scaler = DynamicLinearPipeline(search_space = scale_space, max_length=5)
augument_choice = ChoicePipeline([dynamic_scaler, pass_space])

single_seq   = SequentialPipeline([      single_GFSelector,   augument_choice ])
subset_1_seq = SequentialPipeline([fss_1,subset_1_GFSelector, augument_choice ])
subset_2_seq = SequentialPipeline([fss_2,subset_2_GFSelector, augument_choice ])
subset_3_seq = SequentialPipeline([fss_3,subset_3_GFSelector, augument_choice ])
subset_4_seq = SequentialPipeline([fss_4,subset_4_GFSelector, augument_choice ])
subset_5_seq = SequentialPipeline([fss_5,subset_5_GFSelector, augument_choice ])
subset_6_seq = SequentialPipeline([fss_6,subset_6_GFSelector, augument_choice ])

skip_space = get_search_space("SkipTransformer")
subset_1_choice = ChoicePipeline([subset_1_seq, skip_space])
subset_2_choice = ChoicePipeline([subset_2_seq, skip_space])
subset_3_choice = ChoicePipeline([subset_3_seq, skip_space])
subset_4_choice = subset_4_seq                              #need at least ONE pipeline each run no matter what
subset_5_choice = ChoicePipeline([subset_5_seq, skip_space])
subset_6_choice = ChoicePipeline([subset_6_seq, skip_space])

subset_union = UnionPipeline([subset_1_choice, subset_2_choice, subset_3_choice, subset_4_choice, subset_5_choice, subset_6_choice])

rng_set_classifiers = get_search_space("classifiers", random_state=314)

single_pipeline = SequentialPipeline([single_seq, rng_set_classifiers])
multi_pipeline = SequentialPipeline([subset_union, rng_set_classifiers])

finalized_pipeline_space = ChoicePipeline([single_pipeline,multi_pipeline])

[perib]

This is a bug! I submitted a PR addressing this in PR #1368 .

The issue is that when FSSnode only has a single feature set, mutations cause an error because it tries to select a "new" feature that doesn't exist. For now you can copy my fix into your environment. Or you could pass in a duplicate feature set with the same cols but a second name to avoid it, but then some pipelines won't get mutated at all (The fix should allow tpot to simply skip these mutations and correctly perform some other one).

With this fix, I was able to get several mutation/crossover without errors...

ind1 = finalized_pipeline_space.generate()
ind2 = finalized_pipeline_space.generate()

for i in range(10000):
    ind1.mutate()
    ind2.mutate()

    ind1.crossover(ind2)

ind1.export_pipeline()

A couple of other notes:

1. single_pipeline and multi_pipeline have different lengths and cannot be crossovered (usually, if they happen to line up its possible you will get some pipeline in between the two search spaces). If you don't care about crossover, that's fine too though. But it might be better to evolve only single_pipeline or only multi_pipeline rather than evolving both simultaneously with Choice.

2. I see what you mean now about needed to keep one pipeline no matter what. If you do always want that exact feature set then I think this pipeline still works great. But if you wanted a situation where it would a) always keep at least one feature set but b) never crossover mixmatched sets there are a couple of ways of doing this.

The simplest would be to modify the UnionPipeline here. It would still take in a list of search spaces, but it would also hold a boolean array of which pipelines are currently used/unused. You would have to add a mutation/crossover function to modify this and update export_pipeline to only include the selected pipelines in the output (basically the same thing that GeneticFeatureSelectorNode does). Then you would not need to include 'skip_space' in your pipeline.

Final note: This is a cool search space! Exactly the kind of thing I was hoping to empower with this new version. I'm very excited to see people use it. :)

[boguskiP]

Thank you very much for the help and the recommendation for the single pipeline vs multi-pipline. I feel vindicated knowing the main issue was a bug and not something on my end, things weren't just adding up right. I got the new fss_nody.py in my environment so I'll see how that goes with just a multi-pipeline run.