[python] refine pandas support

[wxchan]

to do list:

• re-write PANDAS_DTYPE_MAPPER: more concise?
• auto extract feature_name from DataFrame: check consistency among train, valid, test datasets?
• auto transfer category from DataFrame: check consistency? save for prediction?
• check pandas input valid ([Python] Confusing error when trying to predict on empty dataframe slice  #1395):
  _data_from_pandas:
  _label_from_pandas: support category?
• re-write the format of pandas_categorical section into model files (more readable) (Refactoring the presentation of pandas_categorical section in model files #1201)
• test_pandas.py

[StrikerRUS]

I think, we can explicitly invoke pandas features (e.g. column names, categorical columns, etc.) at sklearn wrapper level and then pass data as numpy array. In this case all sklearn checks will be performed for pandas too.

[wxchan]

@StrikerRUS I am not sure I get it. The pandas support should be independent to sklearn support. You can take a try if you have a good idea.

[StrikerRUS]

@wxchan I'm not sure it's "good idea"...
I want to say, that pandas support should be implemented "more deeper" than sklearn wrapper. But all data checks in sklearn wrapper cannot handle pandas (now pandas just passed as is without any checks).
https://github.com/Microsoft/LightGBM/blob/master/python-package/lightgbm/sklearn.py#L415

So, I offer to extract all pandas features (column names, categorical columns, etc.) before sklearn checks and it will allow to perform these checks without loosing pandas features.

What do you think?

[wxchan]

@StrikerRUS I edit some tasks I can think of. You can add more you think it's necessary.
I think it's better that each of them be implemented separately, not like now one function doing multiple things.

[StrikerRUS]

@wxchan Yeah, great idea about small separate functions!

[StrikerRUS]

I haven't taken a detailed look at this yet, but it could be useful.

[muleina]

Why still receiving a warning from lightgbm ver 2.2.2 with python 3.6.6 in ubuntu.

lightgbm/basic.py:752: UserWarning: categorical_feature in param dict is overridden. warnings.warn('categorical_feature in param dict is overridden.')

Whether category dtypes were set in pandas dataframe or as a list of categorical_feature in lightgbm Dataset or training.

Why for the warning after the fix of 792?

[StrikerRUS]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.