Skip to content

Vis fuse2 #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 32 commits into
base: fuse2
Choose a base branch
from
Open

Vis fuse2 #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
6cb1d5b
added itai visualization
Apr 18, 2022
8f8f165
removed files
Apr 18, 2022
7e34332
simplify code
Apr 18, 2022
ea9985f
represent polygon
Apr 18, 2022
ffc8cd2
convert RLE to map type
Apr 25, 2022
2a5cf1e
updated test
Apr 25, 2022
c3482c8
update
Apr 25, 2022
7b37c98
added save to nifti
Apr 27, 2022
4eac47f
added SaveVisual class
Apr 28, 2022
d00c1ef
moved test to jupyter
Apr 28, 2022
53b0e84
added run files
Apr 28, 2022
694b0ba
Delete .nfs00000000013866a0000003dc
itaijj Apr 28, 2022
fbf16f6
Delete .nfs00000000013866a3000003dd
itaijj Apr 28, 2022
c03d41b
updated jupyter
Apr 28, 2022
655519d
Merge branch 'itai_visualization' of https://github.com/IBM/fuse-med-…
Apr 28, 2022
351eb24
showing files
Apr 28, 2022
2561ce3
visualizer
May 2, 2022
80bd89c
Merge branch 'fuse2' into vis_fuse2
mosheraboh May 17, 2022
03a33bc
fixed conflicts
May 25, 2022
2b53c78
fixed PR comments on code
May 25, 2022
cf8d3c6
jupyter update
May 25, 2022
afa3425
Update README.md
itaijj May 26, 2022
823efde
Merge pull request #93 from IBM/itaijj-patch-1
itaijj May 26, 2022
0996384
Update README.md
itaijj May 26, 2022
01a2f87
Update README.md
itaijj May 26, 2022
5bffcd6
Update README.md
itaijj May 26, 2022
e5d219f
change file names
May 26, 2022
b85454f
Merge branch 'vis_fuse2' of https://github.com/IBM/fuse-med-ml into v…
May 26, 2022
d4fc12d
Update README.md
itaijj May 30, 2022
f57cfa3
Update README.md
itaijj May 30, 2022
416b019
removed type detector usage
May 31, 2022
8b5fdea
removed type detrctor
May 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 72 additions & 1 deletion fuse/data/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,73 @@ A pipeline is created from a list of tuples. Each tuple includes an op and op ar
In this example "sample_id" is a running index. OpKits21SampleIDDecode() is a custom op for Kits21 challenge converting the index to image path and segmentation path which are then loaded by OpLoadImage().
In other case than Kits21 you would have to implement your custome MySampleIDDecode() operator.
Finally, OpClip() and OpToRange() pre-process the image.


## Visualization - VisProbOp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it 2D image visualization? or more general? We should mention it here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here I just talk in general on the operation, it's not specific for 2D images.

This op can be placed between different steps at the pipeline , it registers an object in the pipeline to be visualized now/later.

Each op call can have an optional name which will appear in the image or file produced by the visualizer.

This op requires the following inputs in each call :
1. give the input keys that define a "namespace" which included your input ( e.g `sample_dict[“data.input.img”]` and `sample_dict[“data.input.seg"]`)
2. type detector object of class TypeDetectorPatternsBased - used to detect the input type from previous step and convert it to the right class
3. visualizer - instance of a class that inherits from VisualizerBase ( SaveToFormat - to save to nifti file , Imaging2dVisualizer - to save multiple 2d images in one file)
4. output_path - path to save the images
5. VisFlag - indicates what to do in the current step

A. VisFlag.VISUALIZE_CURRENT - visualize only current step

B. VisFlag.COLLECT - save the image in stack and show later

C. VisFlag.VISUALIZE_COLLECTED - visualize current step and all collected before it

D. VisFlag.CLEAR - clear the stack of collected images

The default behaviour is to save to file the visualization,
Alternatively you can change to show it online by using original_flag | VisFlag.ONLINE

## Basic example - using Imaging2dVisualizer
in each step we call VisProbe with same flag VisFlag.COLLECT and in the end we call with VisFlag.SHOW_COLLECTED

because we are using ONLINE mode images will be displayed in console.

The follwing example donwloads kits21 image and segmentation and visualize it's rotation and in a specific slice ( so we can use 2d visualizer)

**The original code is in fuseimg/utils/visualization/visualization_example.ipynb**
```python
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give a shorter example here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's a 3D vis, maybe it makes more sense to demonstrate it on a 2D use case? The select slice might be confusing.
or alternatively, implement a 3D visualizer as well and use it in the example.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can have less calls to OpVisProb,
we need medical image with segmentation I think,
you have other example with ready pipeline?


sample , data_dir = create_sample_1(views=1)
visual = Imaging2dVisualizer(cmap = 'gray')
VProbe = partial(VisProbe,
keys= ["data.viewpoint1.img", "data.viewpoint1.seg" ],
type_detector=type_detector_imaging,
visualizer = visual)
repeat_for = [dict(key="data.viewpoint1.img"), dict(key="data.viewpoint1.seg")]
slice_idx = 190
pipeline = PipelineDefault('test_pipeline', [
(OpLoadImage(data_dir), dict(key_in = 'data.viewpoint1.img_filename', key_out='data.viewpoint1.img', format="nib")),
(OpLoadImage(data_dir), dict(key_in = 'data.viewpoint1.seg_filename', key_out='data.viewpoint1.seg', format="nib")),
(OpSelectSlice(), dict(key="data.viewpoint1.img", slice_idx = slice_idx)),
(OpSelectSlice(), dict(key="data.viewpoint1.seg", slice_idx = slice_idx)),
(OpToIntImageSpace(), dict(key="data.viewpoint1.img") ),
(OpRepeat(OpToTensor(), kwargs_per_step_to_add=repeat_for), dict(dtype=torch.float32)),
(VProbe( VisFlag.COLLECT , name = "first"), {}),
(OpToRange(), dict(key="data.viewpoint1.img", from_range=(-500, 500), to_range=(0, 1))),
(VProbe( VisFlag.COLLECT , name = "second"), {}),
(OpSampleAndRepeat(OpAugAffine2D (), kwargs_per_step_to_add=repeat_for), dict(
rotate=30.0
)),
(VProbe( VisFlag.COLLECT, name = "third"), {}),
(OpSampleAndRepeat(OpAugAffine2D (), kwargs_per_step_to_add=repeat_for), dict(
rotate=30.0
)),
(VProbe( flags=VisFlag.VISUALIZE_COLLECTED | VisFlag.ONLINE , name = "last" ), {}),

])

sample = pipeline(sample)

```
The images will be displayed to the screen using matplotlib library.

## Caching
**The original code is in example_cache_pipeline() in fuse/data/examples/examples_readme.py**
Expand Down Expand Up @@ -310,6 +376,11 @@ The following operators are useful when implementing a common pipeline:
* OpToNumpy - convert many different types to NumPy array
* OpToTensor - convert many different types to PyTorch tensor

**Visualization and debugging operators**

* OpVisProb - Handle visualization, saves, shows and compares the sample with respect to the current state inside a pipeline
In most cases VisProbe can be used regardless of the domain, and the domain specific code will be implemented
as a Visualizer inheriting from VisualizerBase.
**Imaging operators**
See fuseimg package

167 changes: 49 additions & 118 deletions fuse/data/ops/ops_visprobe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,71 +1,54 @@
from typing import Callable, Dict, List, Optional, OrderedDict, Sequence, Tuple, Union
import copy
from typing import List, Optional, Union , Dict , Any , Callable
import enum
import numpy as np
from copy import deepcopy

from fuse.utils.ndict import NDict
from fuse.data.visualizer.visualizer_base import VisualizerBase
from .op_base import OpBase
from fuse.data.key_types import TypeDetectorBase

from .op_base import OpReversibleBase
from fuse.data.utils.sample import get_sample_id
class VisFlag(enum.IntFlag):
COLLECT = 1 #save current state for future comparison
SHOW_CURRENT = 2 #show current state
SHOW_COLLECTED = 4 #show comparison of all previuosly collected states
CLEAR = 8 #clear all collected states until this point in the pipeline
ONLINE = 16 #show operations will prompt the user with the releveant plot
OFFLINE = 32 #show operations will write to disk (using the caching mechanism) the relevant info (state or states for comparison)
FORWARD = 64 #visualization operation will be activated on forward pipeline execution flow
REVERSE = 128 #visualization operation will be activated on reverse pipeline execution flow
SHOW_ALL_COLLECTED = 256 #show comparison of all previuosly collected states
COLLECT = 1 #save current state for future visualization
VISUALIZE_CURRENT = 2 #show current state
VISUALIZE_COLLECTED = 4 #show visualization of all previuosly collected states
CLEAR = 8 #clear all collected states until this point in the pipeline
ONLINE = 16 #show operations will prompt the user with the releveant plot
# REVERSE = 32 #visualization operation will be activated on reverse pipeline execution flow


class VisProbe(OpBase):

class VisProbe(OpReversibleBase):
"""
Handle visualization, saves, shows and compares the sample with respect to the current state inside a pipeline
In most cases VisProbe can be used regardless of the domain, and the domain specific code will be implemented
as a Visualizer inheriting from VisualizerBase. In some cases there might be need to also inherit from VisProbe.

Important notes:
- running in a cached environment is dangerous and is prohibited
- this Operation is not thread safe ans so multithreading is also discouraged

"
"""

def __init__(self,flags: VisFlag,
keys: Union[List, dict] ,
type_detector: TypeDetectorBase,
id_filter: Union[None, List] = None,
visualizer: VisualizerBase = None,
cache_path: str = "~/"):
visualizer: VisualizerBase,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about sample_ids?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we agreed we don't need this here

name : str ="",
output_path: str = "~/"):
"""
:param flags: operation flags (or possible concatentation of flags using IntFlag), details:
COLLECT - save current state for future comparison
SHOW_CURRENT - show current state
SHOW_COllected - show comparison of all previuosly collected states
COLLECT - save current state for future visualization
VISUALIZE_CURRENT - visualize current state
VISUALIZE_COLLECTED - visualize all previuosly collected states
CLEAR - clear all collected states until this point in the pipeline
ONLINE - show operations will prompt the user with the releveant plot
OFFLINE - show operations will write to disk (using the caching mechanism) the relevant info (state or states for comparison)
FORWARD - visualization operation will be activated on forward pipeline execution flow
REVERSE - visualization operation will be activated on reverse pipeline execution flow
ONLINE - visualize operations will prompt the user with the releveant plot
:param keys: for which sample keys to handle visualization, also can be grouped in a dictionary
:param id_filter: for which sample id's to be activated, if None, active for all samples
:param visualizer: the actual visualization handler, depands on domain and use case, should implement Visualizer Base
:param cache_path: root dir to save the visualization outputs in offline mode

few issues to be aware of, detailed in github issues regarding static cached pipeline and multiprocessing
note - if both forward and reverse are on, then by default, on forward we do collect and on reverse we do show_collected to
compare reverse operations
for each domain we inherit for VisProbe like ImagingVisProbe,...
:param name : image name to represnt, if not given a number will be shown.
:param output_path: root dir to save the visualization outputs in offline mode
"""
super().__init__()
self._id_filter = id_filter
self._keys = keys
self._flags = flags
self._cacher = None
self._collected_prefix = "data.$vis"
self._cache_path = cache_path
self._name = name
self._output_path = output_path
self._visualizer = visualizer
self._type_detector = type_detector

def _extract_collected(self, sample_dict: NDict):
res = []
Expand All @@ -76,111 +59,59 @@ def _extract_collected(self, sample_dict: NDict):
res.append(vdata)
return res

def _extract_data(self, sample_dict: NDict, keys, op_id):
if type(keys) is list:
# infer keys groups
keys.sort()
first_type = self._type_detector.get_type(sample_dict, keys[0])
num_of_groups = len([self._type_detector.get_type(sample_dict, k) for k in keys if self._type_detector.get_type(sample_dict, k) == first_type])
keys_per_group = len(keys) // num_of_groups
keys = {f"group{i}": keys[i:i + keys_per_group] for i in range(0, len(keys), keys_per_group)}

def _extract_data(self, sample_dict: NDict, keys : dict , name : str):
res = NDict()
for group_id, group_keys in keys.items():
for key in group_keys:
prekey = f'groups.{group_id}.{key.replace(".", "_")}'
res[f'{prekey}.value'] = sample_dict[key]
res[f'{prekey}.type'] = self._type_detector.get_type(sample_dict, key)
res['$step_id'] = op_id
for arg_name, key in keys.items():
res[arg_name] = deepcopy(sample_dict[key])
res['name'] = name
return res


def _save(self, vis_data: Union[List, dict]):
# use caching to save all relevant vis_data
print("saving vis_data", vis_data)

def _handle_flags(self, flow, sample_dict: NDict, op_id: Optional[str]):
def _handle_flags(self, sample_dict: NDict, op_id: Optional[str]):
"""
See super class
"""
# sample was filtered out by its id
if self._id_filter and self.get_idx(sample_dict) not in self._id_filter:
return None
if flow not in self._flags:
return None

# grouped key dictionary with the following structure:
#vis_data = {"cc_group":
# {
# "key1": {
# "value": ndarray,
# "type": DataType.Image,
# "op_id": "test1"}
# "key2": {
# "value": ndarray,
# "type": DataType.BBox,
# "op_id": "test1"}
# },
# "mlo_goup":
# {
# "key3": {
# "value": ndarray,
# "type": DataType.Image,
# "op_id": "test1"}
# "key4": {
# "value": ndarray,
# "type": DataType.BBox,
# "op_id": "test1"}
# },
# }
vis_data = self._extract_data(sample_dict, self._keys, op_id)
both_fr = (VisFlag.REVERSE | VisFlag.FORWARD) in self._flags
dir_forward = flow == VisFlag.FORWARD
dir_reverse = flow == VisFlag.REVERSE
any_show_collected = VisFlag.SHOW_ALL_COLLECTED|VisFlag.SHOW_COLLECTED

if VisFlag.COLLECT in self._flags or (dir_forward and both_fr):
vis_data = self._extract_data(sample_dict, self._keys ,self._name)
name_prefix = op_id
if VisFlag.COLLECT in self._flags:
if not self._collected_prefix in sample_dict:
sample_dict[self._collected_prefix] = []
sample_dict[self._collected_prefix].append(vis_data)


if VisFlag.SHOW_CURRENT in self._flags:
if VisFlag.VISUALIZE_CURRENT in self._flags:
if VisFlag.ONLINE in self._flags:
self._visualizer.show(vis_data)
if VisFlag.OFFLINE in self._flags:
self._save(vis_data)
else :
self._visualizer.save(vis_data , name_prefix , self._output_path)

if (VisFlag.SHOW_ALL_COLLECTED in self._flags or VisFlag.SHOW_COLLECTED in self._flags) and (
(both_fr and dir_reverse) or not both_fr):
if VisFlag.VISUALIZE_COLLECTED in self._flags:
vis_data = self._extract_collected(sample_dict) + [vis_data]
if both_fr:
if VisFlag.SHOW_COLLECTED in self._flags:
vis_data = vis_data[-2:]
if VisFlag.ONLINE in self._flags:
self._visualizer.show(vis_data)
if VisFlag.OFFLINE in self._flags:
self.save(vis_data)
else :
self._visualizer.save(vis_data , name_prefix, self._output_path)

if VisFlag.CLEAR in self._flags:
sample_dict[self._collected_prefix] = []

if VisFlag.SHOW_COLLECTED in self._flags and both_fr and dir_reverse:
sample_dict[self._collected_prefix].pop()
# TODO - support reverse?
# if VisFlag.SHOW_COLLECTED in self._flags and VisFlag.REVERSE in self._flags:
# sample_dict[self._collected_prefix].pop()

return sample_dict


def __call__(self, sample_dict: NDict, op_id: Optional[str], **kwargs) -> Union[None, dict, List[dict]]:
res = self._handle_flags(VisFlag.FORWARD, sample_dict, op_id)
res = self._handle_flags(sample_dict, op_id)
return res

def reverse(self, sample_dict: NDict, op_id: Optional[str], key_to_reverse: str, key_to_follow: str) -> dict:
"""
See super class
See super class - not implemented for now
"""
res = self._handle_flags(VisFlag.REVERSE, sample_dict, op_id)
if res is None:
res = sample_dict
return res
# res = self._handle_flags(VisFlag.REVERSE, sample_dict, op_id)
# if res is None:
# res = sample_dict
# return res
return sample_dict

Loading