Add extra links for google dataproc

[yesemsanthoshkumar]

Add extra links for google cloud operators - Dataproc

How it is implemented

1. DataprocOperator pushes the details to xcom
2. The value pushed into Xcom is fetched to construct the link for job

Note: I haven't worked with dataproc workflows so I haven't implemented links for WorkFlow related operators.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://apache-airflow-slack.herokuapp.com/

[yesemsanthoshkumar]

@mik-laj
I've added extra link for dataproc jobs.

However, I've the following questions.

1. While running the pre-commit hooks, I see both pylint and fake8 running checks. I've marked both of them to disable line too long error for some lines inside the files. Why do we have both of them?
2. I'm directing the link to the cluster if the job hasn't run and not generated any JobId. Is this approach valid? Do you want me to change this behaviour?
3. I coudn't find the cluster_name argument in DataprocSubmitJobOperator for which I've added the links. Shoud I add the links to DataprocJobBaseOperator? Or Should I add on both?

[turbaszek]

I suppose you did this because of adding xcom_push in execute, right?

[yesemsanthoshkumar]

Yes.

[yesemsanthoshkumar]

Have added a mock task instance for testing the xcom push. This change is regarding the operator extra links. Please take a look at the updated tests.

[turbaszek]

We should also add the console link to

airflow/airflow/serialization/serialized_objects.py

Lines 44 to 49 in 558be73

	BUILTIN_OPERATOR_EXTRA_LINKS: List[str] = [
	"airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleLink",
	"airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleIndexableLink",
	"airflow.providers.google.cloud.operators.mlengine.AIPlatformConsoleLink",
	"airflow.providers.qubole.operators.qubole.QDSLink"
	]

And as mentioned in a comment - we should add the link in DataprocSubmitJobOperator as other ops are deprecated. So, we may even consider to add this only to this one operator 😉

[yesemsanthoshkumar]

We should also add the console link to

airflow/airflow/serialization/serialized_objects.py

Lines 44 to 49 in 558be73

	BUILTIN_OPERATOR_EXTRA_LINKS: List[str] = [
	"airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleLink",
	"airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleIndexableLink",
	"airflow.providers.google.cloud.operators.mlengine.AIPlatformConsoleLink",
	"airflow.providers.qubole.operators.qubole.QDSLink"
	]

And as mentioned in a comment - we should add the link in DataprocSubmitJobOperator as other ops are deprecated. So, we may even consider to add this only to this one operator 😉

@turbaszek I've added to those other operator as well already.
:D

[yesemsanthoshkumar]

@mik-laj

2. I'm directing the link to the cluster if the job hasn't run and not generated any JobId. Is this approach valid? Do you want me to change this behaviour?

The link is not being redirected to the cluster URL if there is no detail of the job in the system.

3. I coudn't find the cluster_name argument in DataprocSubmitJobOperator for which I've added the links. Shoud I add the links to DataprocJobBaseOperator? Or Should I add on both?

Implemented on both the operators. For subclasses of DataprocJobBaseOperators, I've added tests only for DataprocSubmitSparkJob operator as the other operators are anyway deprecated. Please have a look at the above comments by @turbaszek regarding this.

[turbaszek]

LGTM! 🚀

[turbaszek]

@yesemsanthoshkumar can you please rebase? We started using black on providers package 👍

[turbaszek]

Nice to see this as part of #10014

[turbaszek]

Tested both links. This rocks! Thanks @yesemsanthoshkumar 🚀

The quote changess were reverted :).

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[boring-cyborg]

Awesome work, congrats on your first merged pull request!

[potiuk]

Wooho! 🎉 -> took some time :)