Skip to content

Comments

Added metric Brier Score#275

Merged
lvwerra merged 13 commits intohuggingface:mainfrom
kadirnar:brier_score
Aug 30, 2022
Merged

Added metric Brier Score#275
lvwerra merged 13 commits intohuggingface:mainfrom
kadirnar:brier_score

Conversation

@kadirnar
Copy link
Contributor

@kadirnar kadirnar commented Aug 25, 2022

I added Brier Score metric. #235

@kadirnar
Copy link
Contributor Author

kadirnar commented Aug 25, 2022

Hi @lvwerra,

Can you share the reformatter code?

@lvwerra
Copy link
Member

lvwerra commented Aug 25, 2022

Hi @kadirnar, thanks for that awesome contribution! If you are inside the repo (main folder) you can run make style && make quality. Let me know if you run into any issue!

@kadirnar
Copy link
Contributor Author

kadirnar commented Aug 25, 2022

make style && make quality

Thank you for your support. I am running vscode application as administrator.
Run Code:

make style && make quality

151 files left unchanged.
isort tests src metrics measurements
WARNING: Unable to parse file tests due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\tests'
WARNING: Unable to parse file src due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\src'
WARNING: Unable to parse file metrics due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\metrics'
WARNING: Unable to parse file measurements due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\measurements'
black --check --line-length 119 --target-version py36 tests src metrics comparisons measurements
All done! ✨ 🍰 ✨
151 files would be left unchanged.
isort --check-only tests src metrics measurements
WARNING: Unable to parse file tests due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\tests'
WARNING: Unable to parse file src due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\src'
WARNING: Unable to parse file metrics due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\metrics'
WARNING: Unable to parse file measurements due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\measurements'
flake8 tests src metrics
metrics\brier_score\brier_score.py:47:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:50:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:53:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:61:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:63:1: W293 blank line contains whitespace
make: *** [Makefile:8: quality] Error 1

The codes in the tokenizer_13a.py file have also changed

@lvwerra
Copy link
Member

lvwerra commented Aug 25, 2022

Seems to be an issue of isort and permissions on Windows (which I don't have access to): PyCQA/isort#835

Which version are you using? Can you run:

pip install .[dev]

to make sure all the right versions for black and isort are installed?

lvwerra
lvwerra reviewed Aug 25, 2022
View reviewed changes
lvwerra
lvwerra reviewed Aug 26, 2022
View reviewed changes
Copy link
Member

@lvwerra lvwerra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few fixes that should hopefully make the tests pass.

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Aug 26, 2022 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@kadirnar
Copy link
Contributor Author

kadirnar commented Aug 26, 2022

Hi @lvwerra ,
I updated the codes. It works when I test it with the sklearn library. Sample codes of the sklearn library:

    >>> import numpy as np
    >>> from sklearn.metrics import brier_score_loss
    >>> y_true = np.array([0, 1, 1, 0])
    >>> y_prob = np.array([0.1, 0.9, 0.8, 0.3])
    >>> brier_score_loss(y_true, y_prob)

Why doesn't it pass the tests?

lvwerra
lvwerra reviewed Aug 26, 2022
View reviewed changes
Copy link
Member

@lvwerra lvwerra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added two suggestions that should fix the issues.

@lvwerra
Copy link
Member

lvwerra commented Aug 26, 2022

Side comment: if there are code suggestions you can just click on "Commit suggestion" so you don't need to do an extra commit locally.

@kadirnar
Copy link
Contributor Author

kadirnar commented Aug 26, 2022

Side comment: if there are code suggestions you can just click on "Commit suggestion" so you don't need to do an extra commit locally.

"Commit suggestion

Side comment: if there are code suggestions you can just click on "Commit suggestion" so you don't need to do an extra commit locally.

I didn't know that. Thank you for your advice. I will try.

@kadirnar @lvwerra
Update metrics/brier_score/brier_score.py
7782d44 
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
lvwerra
lvwerra reviewed Aug 30, 2022
View reviewed changes
Copy link
Member

@lvwerra lvwerra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for working on this! Left a few minor nits and then I think we can merge!

kadirnar and others added 2 commits August 30, 2022 12:49
@kadirnar @lvwerra
Update metrics/brier_score/README.md
f233fe9 
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
@kadirnar @lvwerra
Update metrics/brier_score/README.md
9be89a9 
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
lvwerra
lvwerra reviewed Aug 30, 2022
View reviewed changes
lvwerra
lvwerra reviewed Aug 30, 2022
View reviewed changes
Copy link
Member

@lvwerra lvwerra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one more suggestion to fix the YAML in the README.

@kadirnar @lvwerra
Update metrics/brier_score/README.md
6a61928 
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
@kadirnar
Copy link
Contributor Author

kadirnar commented Aug 30, 2022

Left one more suggestion to fix the YAML in the README.

I didn't know how to fix this error. Thank you very much for your help.

@lvwerra
Copy link
Member

lvwerra commented Aug 30, 2022

Thanks a lot, looks good now! I'll approve and merge it after we merge another bug fix.

@lvwerra lvwerra merged commit 2dfe5d9 into huggingface:main Aug 30, 2022
@kadirnar kadirnar deleted the brier_score branch August 30, 2022 15:35
mathemakitten pushed a commit that referenced this pull request Sep 23, 2022
@kadirnar @lvwerra @mathemakitten
Added metric Brier Score (#275)
3635016 
* Added metric Brier Score

* added numpy library and reformatter

* Updated the args description.

* examples codes updated.

* word error fixed

* parameter error fixed.

* parameter and reference codes have been updated.

* The print code has been updated.

* output value have been updated.

* Update metrics/brier_score/brier_score.py

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* Update metrics/brier_score/README.md

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* Update metrics/brier_score/README.md

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* Update metrics/brier_score/README.md

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lvwerra lvwerra lvwerra approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kadirnar @lvwerra @HuggingFaceDocBuilderDev