We provide some example scripts of running FATE jobs with FATE-Pipeline.
Please refer to the document linked above for details on FATE-Pipeline and FATE-Flow CLI v2. DSL version of provided Pipeline examples can be found here.
Here is a general guide to quick start a FATE job.
(optional) create virtual env
python -m venv venv source venv/bin/activate pip install -U pip
install fate_client
# this step installs FATE-Pipeline, FATE-Flow CLI v2, and FATE-Flow SDK pip install fate_client pipeline config --help
configure server information
# provide ip and port information # optionally, set logs directory pipeline config --ip 127.0.0.1 --port 9380
upload data with FATE-Pipeline
# upload data used by demo, optionally provides directory where examples/data locates python demo/pipeline-upload.py --base /data/projects/fate
If upload job is invoked correctly, job id will be printed to terminal and a upload bar is shown. If FATE-Board is available, job progress can be monitored on Board as well.
UPLOADING:||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.00% 2020-10-16 10:44:26.578 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - Job id is 20201016104426367594590 Job is still waiting, time elapse: 0:00:01 Running component upload_0, time elapse: 0:00:03 2020-10-16 10:44:31.042 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:133 - Job is complete!!! Job id is 20201016104426367594590 2020-10-16 10:44:31.043 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:04
run FATE-Pipeline fit and predict jobs
python demo/pipeline-mini-demo.py
As job runs, progress of job execution will also be printed as modules are executed. A message indicating final status ("complete") will also be printed at the end of the job.
2020-10-16 13:14:56.316 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - Job id is 20201016131456016425640 Job is still waiting, time elapse: 0:00:00 Running component reader_0, time elapse: 0:00:03 Running component dataio_0, time elapse: 0:00:05 Running component intersection_0, time elapse: 0:00:10 Running component hetero_lr_0, time elapse: 0:00:36 2020-10-16 13:15:33.703 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:133 - Job is complete!!! Job id is 20201016131456016425640 2020-10-16 13:15:33.703 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:37
Once fit job completes, demo script will print coefficients and other training information of model.
After having completed the fit job, script will invoke a predict job with the trained model. Note that
Evaluation
component is added to the predict job workflow. For more information on using FATE-Pipeline, please refer to this guide.2020-10-16 13:15:34.282 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - Job id is 20201016131533727391641 Job is still waiting, time elapse: 0:00:02 Running component reader_1, time elapse: 0:00:05 Running component dataio_0, time elapse: 0:00:07 Running component intersection_0, time elapse: 0:00:12 Running component hetero_lr_0, time elapse: 0:00:17 Running component evaluation_0, time elapse: 0:00:23 2020-10-16 13:15:58.206 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:133 - Job is complete!!! Job id is 20201016131533727391641 2020-10-16 13:15:58.207 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:230-10-16 10:46:09.350 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:23