Skip to content

Comments

Merge the changes in old EMR example#168

Open
guoquan wants to merge 20 commits intodevelopfrom
examples/conll04/bert
Open

Merge the changes in old EMR example#168
guoquan wants to merge 20 commits intodevelopfrom
examples/conll04/bert

Conversation

@guoquan
Copy link
Collaborator

@guoquan guoquan commented Aug 26, 2020

No description provided.

guoquan added 14 commits June 13, 2020 22:49
Conflicts:
	regr/graph/dataNode.py
	regr/program/learningbaseprogram.py
	regr/program/model/pytorch.py
	regr/sensor/pytorch/sensors.py
	test_regr/examples/conll04/emr/data.py
Conflicts:
	regr/program/learningbaseprogram.py
	regr/program/model/pytorch.py
Conflicts:
	examples/emr/config.py
	examples/emr/emr/utils.py
	regr/program/loss.py
	regr/program/metric.py
	regr/utils.py
@guoquan
Copy link
Collaborator Author

guoquan commented Aug 26, 2020

I merge the latest changes in conll04 example in examples/emr. main.py is using the old design and can be run directly.
main2.py move the tokenizer to the sensor and has some issue with some special cases in match spans used in annotation and tokenized spans.

Since the goal change to replace everything in this example with the new interface, we can merge it now and solve any remaining issue using the new interface.

@kordjamshidi
Copy link
Member

Should this PR be merged? Or it is WIP?

else:
return [(*output_raw, input_item) for input_item in input_raw]


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems these are generic sensors not specifically for conll/EMR?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,
Yes, sensors in regr.sensor.torch are written in a generic manner. The only issue is their base class is regr.sensor.torch.Sensor. We later decided to use regr.sensor.pytorch.Sensor and subclasses instead of regr.sensor.torch.Sensor.
However, these sensors can be migrated easily to regr.sensor.pytorch.Sensor. I did some already, based on whatever I need in ACE event example.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. So, this PR is not for merging then and needs cleaning up?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR itself is good to merge because I just wanted to merge the latest changes in CONLL to develop.
I did not try to rewrite CONLL with the current interface in this PR. However, we can start another issue for this purpose.

Copy link
Member

@kordjamshidi kordjamshidi Sep 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you better to clean this up and then merge. Those general sensors in EMR folder does not make sense. The develop branch becomes more chaotic given the lack of documentation.



class LearningBasedProgram():
logger = logging.getLogger(__name__)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this class is under EMR? this class of LearningBasedProgram is only for EMR?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Program class was first written for CONLL and I had made it generic and moved it to the base package. I did not realize there is this class remaining. I can remove it before merging.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes please clean this up, we should just merge the clean documented code, the repo becomes unmanageable it seems given the current approach to development.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I am using this week to clean up and adding documentation, especially for Sensors and Programs.
I will clean up this class before merging this PR also.

def populate_one(self, data_item, inference=True):
for key, value in data_item:
data_item[key] = [value]
return next(self.populate(data_item, inference))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment, are these populate functions only for EMR? It would be helpful to add some documentation inside each class to see what is the goal, otherwise I am not sure if reviewing this code is possible. Thanks!

@guoquan guoquan mentioned this pull request Nov 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants